Saturday, 20 March 2010

Exchange: 13 month old email received on Blackberry

A client keeps on receiving batches of 10 or so emails on their Blackberry via Exchange... those forwarded to me (on the 19th March 2010) were dated 16th Feb 2009 and 19th October 2009.

The client has Microsoft Exchange 2003, AVG for Exchange, MAC Entourage, Blackberrys using Blackberry Internet Service (BIS).

I have connected up to Outlook Web Access and checked that the emails received are still in the user's Inbox... so there is some kind of periodic sweep over a batch of emails in the Mailbox every now and again...

First look... is AVG again... perhaps biased by not looking at AVG in the last problem (with iisadmin and https services not restarting)... version is 9.0.272... all fairly up-to-date...

Email Scanner for Exchange settings VSAPI (in Advanced Settings -> Server Components) selected components are:
  • Background Scan - On
  • Proactive Scan - Off
  • Scan RTF - On
  • Number of Scanning Threads: 9
  • Scan Timeout: 180
A background scan could sweep through email messages... I need to find out what AVG says this does...
Background scanning is one of the features of the VSAPI application interface. It provides threaded scanning of the Exchange Messaging Databases. Whenever an item that has not been scanned before is encountered in the users mailbox folders, it is submitted to E-mail Scanner for MS Exchange to be scanned. Scanning and searching for the not examined objects runs in parallel. Note: A specific low priority thread is used for each database, which guarantees other tasks (e.g. e-mail messages storage in the Microsoft Exchange database) are always carried out preferentially.
So... there's a background process, running on a low-priority thread, meaning it'll give up processor time to anything with a higher priority... the timeout is 3 minutes per email - (that's the maximum time the scanner can spend on any one email)... on a busy server that could take a long time to scan a few large emails (at the least 20 emails per hour (unless Exchange is busy sending and receiving emails or responding to lots of Entourage requests from around the Office?)...

So my theory at the moment is that the low priority thread that VSAPI background scanning is working on is having to give way to other higher priority threads...

Another factor is that the client's office is all Mac .. all Entourage... and Entourage talks to Exchange differently than Outlook does... does this explain why other clients don't have a problem with their Blackberry synchronising? I guess I have to look into VSAPI a little...

This from the MS Exchange Team Blog is useful background (Parts 1,2,3):

First, this advises (in Part 2) switching on Medium Diagnostics Logging on Antivirus Scanning...

In Exchange Admin -> Servers -> Server -> Properties -> Diagnostic Logging tab -> in services: MSExchangeIS -> System -> in categories: Antivirus -> Set Medium or Maximum logging level -> Click OK to exit...

Wow... just noticed while in Exchange Admin ( -> First Storage Group -> Mailboxes) that this particular user's Mailbox is 10GB... the largest of all the users on that network, next largest is 6GB then 4GB... So that is now a potential factor in slow Background Scanning of this mailbox and old emails on her Blackberry...

Diagnostic logging on ... also the Exchange Blog mentions that when an email is scanned it is stamped - "At the completion of the scanning process ptagVirusScanningStamp is updated reflecting the results of the scan. This property holds information such as the vendor, version, scan results, and miscellaneous information regarding the last scan of the item" - Perhaps if the email has never been scanned, this stamp affects the sync?

It's been a few minutes now... so time to check Event Viewer for Antivirus events... (darn.. I reckon I need to restart MSExchangeIS service!)

These came in long before I switched on Max logging level:

Event Type: Error
Event Source: MSExchangeIS
Event Category: Virus Scanning
Event ID: 9581
Date: 20/03/2010
Time: 12:44:23
User: N/A
Computer: SERVER
Description:
Error code -536768764 returned from virus scanner initialization routine. Virus scanner was not loaded.

So if the scanner has been unable to initialise on a regular basis... I should take a note of the dates and times of all Events with ID 9581 - they might tally with old emails being sent - or they might not... I find it's good to have some solid dates and times... It might also happen at a particular time of the day or periodically - so making a note of times and dates helps to see a pattern if one exists... Or simply filter all Events in the Application Log by ID 9581...

ok... these messages occur at 00:44am and 12:43pm every day for the last 3 days 17-20th March 2010; then on the 5th Feb 2010; 16th to 22nd Dec 2009; and 9th/10th Dec 2009

I've got to presume that the rest of the time MSExchangeIS antivirus scanning was working fine... On the days it doesn't work, something regular is interfering... like backups? ... but there are also other times... on 4:44am on 18th Dec... and two this morning at 5am and 8am... which is perhaps when I ran the Exchange Best Practice Analyzer... or Trace Analyzer...

Microsoft Best Practice Analyzers web site...

.. what is happening at 12:43 and 00:44? First check backup times, particularly the time that the Exchange backup begins...

It appears that Backup Exec (12) is half way through backing up (with GRT enabled) at 00:44... but it starts an hour earlier and finishes afterwards... and it can't explain the 12:43pm failure...

Ah... AVG Antivirus Update Manager is set to check for virus updates every four hours and the last one was at 12:44pm ... That explains the 4:44am one on the 18th Dec, maybe the 8:42am this morning... but not the 5:09am... perhaps it was when I was playing around with AVG Email settings... ? possibly...

Is event 9581 in 'MSExchangeIS\Virus Scanning' relevant? Time to ask the client for as many dates and times that they received these old emails on their Blackberry... if they can... See if the dates tally with the dates of these errors at all... and also find out whether any of the old emails received were dated after the 9th December... (perhaps they used Blackberry Enterprise Server (BES) before and Internet Service (BIS) after then...?)

Whilst digging around on the Exchange Team Blog... I discovered a post about a change made in 2006 to Exchange that could affect Blackberrys and other services - connected to 'Send As' permissions... I'm not sure that applies here but for anyone with that problem here...

they point to kb article - 912918 - Users cannot send emails from a mobile device or from a shared mailbox in Exchange 2000 or 2003 - it mentions that it would break sending emails if you use BES... (perhaps that accounts for an error I saw back in 2008...)

And - Send As permission behaviour change in Exchange 2003
A fix has been released that changes the behavior of the "Full Mailbox Access" feature in Microsoft Exchange Server 2003. Prior to this change, any user with the “Full Mailbox Access” permission for a mailbox also had the ability to “Send As” the mailbox owner.
... the script is not straightforward.. you have to read the first kb article to know how to use it properly... there's no '-?' switch... - doesn't appear to do anything on our server.. no output...

I'm going to leave the server running for a bit with MSExchangeIS Antivirus logging enabled... Seems that Antivirus starts scanning the Mailbox and Public Folders at 00:44 - Public Folders is over in 6 minutes and Mailbox goes on for between 4 and 19 hours. It then starts again... but is quicker. Could be that it only does a full scan if the virus signatures are updated in AVG - then it has to re-scan everything that it has marked suspect.

Going to leave this till I hear back from the user... with diagnostics running on Virus Scanning I'll be able to see a connection next time... or not...

No comments:

Post a comment