Orphaned Mailboxes and Users after migration from SBS 2003 to SBS 2011

I got a call today from a colleague doing a migration from SBS2003 to SBS2011 ... Exchange 2003 to 2010.

Mailboxes had been copied and he had decommissioned the old server to find that the mailboxes were not there... they were not showing up in the new server... had deleting them removed them from both servers?

At that stage you will have disconnected the connector between the two servers...

First .. go to the disk and check the size of the Exchange Store on the disk... The new store should be the same number of Gb as the old one...

Second.. open the Exchange Mgmt Console and type

• Get-MailboxStatistics -Server servername

This should list all the mailboxes on the server.  You need to add a little bit to get more details... 

• Get-MailboxStatistics -Server servername | select DisplayName, DisconnectReason, LegacyDN, ItemCount, MailboxGuid, Database

Moved mailboxes all had a LegacyDN that contains 'first adminstrative group' - this is normal and there's no need to spend hours, as I did, thinking it was an error and trying to correct it... Apparently that doesn't matter...

What really mattered was time... wait...  The mailboxes we had transferred over were large - around 2-4GB... they were neither disabled nor anything... the new server simply had not finished processing them.  I came back to the problem about 12 hours later and they were showing up... perhaps it took less time, but they eventually showed up... in the meantime, I had created some problems for myself...

Users were showing up in AD Users and Computers, but not showing up in SBS Console - I tried remedying that using the 'Change User Roles' trick   But that just created a second mailbox that started collecting emails ... and didn't connect to the old mailbox...  Worry about users not showing up in SBS Console AFTER the mailboxes show up.  When all your users are connected to their mailboxes then run the 'Change User Role' trick.

If you have created a duplicate mailbox, the ItemCount has increased and you really don't want to lose a single email, try connecting via OWA - you might be able to hook up the 'Archive' function and archive those emails out for later retrieval...  The mailboxes I created only picked up 1 or 2 emails and so I disabled the mailbox and used the Get-DisconnectedMailbox username | Remove-DisconnectedMailbox code from Mike Pfeiffer to get rid of the newly created mailbox.

Soon.. all mailboxes were showing up - they appeared in Disconnected Mailboxes in Exchange Manager.  So I selected the mailbox and tried to connect them to a user... but the user was not showing up... The mailbox name was correct - but Exchange could not see the users in Active Directory.. strange...

• Get-User

Very simple... it produces a list of users and their RecipientType.  The users that Exchange could not see had a type of 'User' but those Exchange can see have a type 'UserMailbox'....  how to enable them?

• Get-User | where-object{$_.Name -eq "User's Name"}   - this should list one user (just to check before changing anything...)  if ok, run this:
• Get-MailboxDatabase -Server servername     - this gets the name of the database to use in this:
• Get-User | where-object($_.Name -eq "user's name") | Enable-Mailbox -Database "database name"

Now run Get-User and check the user has changed RecipientType to UserMailbox.

Now you can go back to Disconnected Mailboxes in Exchange Manager and run Connect to connect them to their mailbox... (if it didn't do it for you when you ran Enable-Mailbox).

Once you have connected all your users up.. go to SBS Console -> Users and Groups -> Users... and run the 'Change User Roles' task - select 'Standard User' in the roles and below select 'Add Role to Users' (NOT 'Replace' - then you are not changing anything).  On the next page you click the checkbox at the bottom 'Show all users from AD' and select all the users (one to start with to test) who you don't see in SBS Console...  run it all through and refresh the view.. all the users should now show up.

One last thing I forgot... and perhaps a trick to get around this issue too... when you run the 'Change User Roles' task each user is given an email address based on the current 'Email Address Policy' ... this is in Exchange Manager.. under Organization Configuration -> Hub Transport -> E-mail Address Policies -> Windows SBS Email Address Policy ... unfortunately 'First Name Only' is NOT an option.. duh!  But we can get around that...

1. Start -> Run -> adsiedit.msc  <- take care with this... if you get this far you probably are fine...
2. right-click on AdsiEdit -> Connect To... -> Select a Well-known Naming Context -> Configuration
3. Select Configuration in the console left panel -> then Configuration -> Services -> Microsoft Exchange ->  servername -> Recipient Policies
4. In the right-hand panel for Recipient Policies - right-click Windows SBS Email Address Policy select Properties
5. Click the Filter button and choose to show only attributes that have values
6. gatewayProxy has a value of SMTP:%m@mydomain.com chang
7. Click gatewayProxy and Edit - change to SMTP:%g@mydomain.com - %g is FirstName
8. click ok and exit out of AdsiEdit.msc

And finally .. a warning if you have Blackberry Enterprise Server running... I think I read somewhere there was a knowledgebase article from RIM about what to do in the migration.. .. it was for Exchange 2003 to 2007.. but best to take heed of any advice before attempting to migrate...

http://www.enowconsulting.com/ese/2009/04/bes-co-existence-during-exchange-2003.html

iPhone and ActiveSync: Message received

So... the user received my test email ...

I think the order of checks if you have problems with iPhone and ActiveSync on Exchange:

1) Check the ActiveSync is working using Microsoft's Online Test Tool
2) Switch on Proactive Scanning and Background Scanning in MSExchangeIS\VirusScan.
3) Check timeout values on all routers under your power. If possible, make the value greater than the MaxHeartbeatInterval registry setting in Exchange. (See previous post). (If not possible, perhaps make the Exchange MaxHeartbeatInterval just lower than the router timeout?)

No doubt he'll update to iOS4.1 soon and we'll get yet another problem...

iPhone and ActiveSync: Antivirus Proactive Scanning

And now I don't just have intermittent scanning... Now it only gets emails when the user opens Outlook...

Something has been cleared up by setting the router timeout to a larger value...

And opening Outlook causes the Antivirus to kick in on the messages. Exchange won't do anything with the messages until they have been scanned.. and they won't be scanned without being accessed first.

Checking the registry for the VirusScan settings...

HKLM\System\CurrentControlSet\Services\MSExchangeIS\VirusScan

The ProactiveScanning should read 1 - BackgroundScanning should read 1... then change ReloadNow to 1 .. and MSExchangeIS will reload... wait a little bit and press F5... ReloadNow will have been changed back to 0.

Now let's see if this works... Finally... (Combined with the timeouts in my last post)

iPhone and Exchange ActiveSync: Heartbeats

In an earlier post I mentioned there are two Heartbeat settings Min and Max:

• MinHeartbeatInterval - Dword - Decimal 60 (seconds) 
• MaxHeartbeatInterval - Dword - Decimal 2700 (seconds) 
• HbiSampleSize - Dword - Decimal 200 (samples) 
• HbiAlertThreshold - Dword - Decimal 480 (seconds) 

My Vigor 2900 router HTTP TCP SYN setting was 900 seconds. But clearly if the maximum Interval can be 2700 seconds (45 minutes), then it is going to fail...? Should I set the router's TCP SYN setting to 2700 seconds?

Ok... at the end of the Heartbeat period... the EXCHANGE server replies to the PHONE with a 200 OK message. This should force the iPhone to send another ping with a new (but more than likely same) heartbeat value.

What if the iPhone does not receive this 200 OK message. Perhaps the signal has dropped in a tunnel, for example? TCP/IP is best effort after all... that is the last message required of the Exchange Server.

The iPhone should assume that the period is up... and resend a new Ping anyway when it can? No? Right.. sort of .. the iPhone also calculates the best Heartbeat based on it's history of connections to the Exchange server.

So what if ... I am in the office... and my phone connects up... iPhone can create a longer and longer heartbeat period, up to 45 minutes (2700s)... and then I leave the office... and now the iPhone is not only waiting 2700s for the 200 OK... the phone has disconnected... the server tries to contact the iPhone in the office... nothing happens.. there must be confusion when switching between WiFi and OTA (over the air) ...

But also the HeartBeat period is continuously being cut off by the Router - set to 900s (15 mins)... So how long will it take before the iPhone's connection algorithm chooses a Heartbeat Interval that is smaller than the shortest timeout along the route to the server? Does it say... ok nothing back in 2700s .. lets try 2640s... lets try 2580s .. lets try 2520s ... and so on... which could take more than 6 hours! until finally reaches the correct value.... or does drop back to 60s then increase until it finds the best timeout value?

If for 6 hours or more... the timeout is greater than the value of the router timeout, then if emails arrive in the inbox after 15 minutes.. the server cannot tell you... because the router has shut the session... and the iPhone won't ask until the Heartbeat is up...

There'll just be the assumption for the iPhone that no emails arrived... and perhaps the Exchange server will just shrug and say, "well, I sent you the emails... don't look at me..."

I'm going to up the router to bigger than 2700s...

iPhone and Exchange ActiveSync: still problems - Heartbeats

ok.. well the one user who had problems has upgraded to iOS 4.0.2 (Apple-iPhone2C1/802.117) and now he has intermittent issues again... (with MS Exchange 2003 SP2)

So time to dig deeper... and the frustration has grown... when you think you have fixed it all and no-one calls for weeks on end...  And then it's the same problem yet again...  Time to dig even deeper...

HBISampleSize / Heartbeat Intervals

PHONE -> EXCHANGE SERVER - the phone opens a session (connection) with the server, nudges it and gives the server the heartbeat length...

The phone will hold OPEN that session for the length of the heartbeat...  and IF you get an email, the Exchange server will use the session to send the email to your phone.  IF no emails arrive, the phone will poll the Exchange server again... (not sure if that is immediately or not - or what if not successful?  if no signal - does it try?)

If you have a router or firewall between the PHONE and the Exchange server... like your office ISP router... or home router - WiFi...  or you are connecting over WiFi in a cafe...  and the Firewall/Router times out the HTTP(S) session because it is set to a maximum of 5 minutes not 30 minutes ... (Heartbest could be 15 mins so 30 mins is a safe value)  or your WiFi signal gets low and drops, killing the session ... then that session will be lost ... IF Exchange receives an email, then it cannot open a new session to your iPhone to send the emails... it must wait for your iPhone to poll the server again....

So it is vital to check all points along the connection from your iPhone to your Exchange server.  We really hope here that there aren't routers along the way at our Phone provider or other third-party, with HTTP(S) timeout values that are "almost 15 minutes"... we'll assume here, first, that if we're having problems then it is under our control (or our IT department)... (Having checked all our hops thoroughly, then we can pester the people responsible for the bits we don't have any control over.)

It's not true PUSH... no... but true push is not simple enough for the average iPhone user to set-up and maintain.  And it's really time-consuming if you have 30 phones on different networks ...

I'm going off to investigate time-out values again on the Vigor 2900...

What determines the Heartbeat on the iPhone?  Does it get the value from Exchange?  I'll let you know in my next post.

iPhone and Exchange ActiveSync: Router Timeout - cracked it?

The Draytek Vigor 2910 has now had its firmware upgraded to the latest version.


The portmaptime function is now available.  Running 'portmaptime -l' to list the timeout values shows TCP SYN as 60 seconds... well below the recommended 540 seconds for ActiveSync.


I have set it, according to some users, to 900:  'portmaptime -s 900' (s for 'tcp Syn')


The iPhone user reports that whatever change I made, appears to have cracked it... I told him I'm going to wait a couple of weeks before I start celebrating... :)

Exchange ActiveSync and iPhone: Messages stopped again

Client reports that ActiveSync has stopped working yet again...  checked PoolMon for Non-paged Pool and it's fine...

There's an event in the Application Log for ActiveSync - ID 3033.

Following this article:
http://support.microsoft.com/kb/905013/en-us

I checked the suggested registry key...

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MasSync\Parameters

And all four DWORDs are missing... I have set them to:

• MinHeartbeatInterval - Dword - Decimal 60 (seconds)
• MaxHeartbeatInterval - Dword - Decimal 2700 (seconds)
• HbiSampleSize - Dword - Decimal 200 (samples)
• HbiAlertThreshold - Dword - Decimal 480 (seconds)

It suggests changing the HTTP(S) timeout values on the router firewall but I can't find them... so I'm going to do a simple IISRESET and see if that resolves the problem.  Trouble is, an IISRESET might just resolve the issue, despite the new registry entries... so I can only warn the client that's a possibility... (when he returns)...

If that doesn't resolve the issue for now... back to the drawing board...

iPhone and Exchange ActiveSync: Incorrect AVG Version

Well, having discovered the server was losing non-paged pool resources I used poolmon and watched the resources over a couple of days...

AvgU was rising along with File and Irp.

If you open a cmd window and cd to c:\Windows\System32\drivers ... you can run

'findstr /m /l AvgU *.sys'

That should return just the file name of any sys file that contains the literal string 'AvgU'. It returned 'Avgtdix.sys'.

Further googling... Avgtdix.sys is a Network Connection Watcher. This file should NOT be installed on an Exchange Mail Server.

I have uninstalled AVG from the server and installed the Exchange Mail Server version of AVG (paid).

No Avg tags are showing up in the Non-Paged Pool... there's no Online Shield or Email Scanner in this version...

I will have to monitor the server for another few days...

iPhone and Exchange ActiveSync: not solved - reoccurred

So I got a call that the iPhone stopped getting emails... I didn't check till they went home in case I needed to restart their server out-of-hours ...

When I did connect I decided to run MS Exchange Best Practices Analyser... (search on ExBPA).

I knew one of the Best Practices was having an Application Log size in Event Viewer of 40Mb - so I decided to alter that myself first whilst I was waiting... I must have dozed off at that point ... suddenly I was disconnected... RealVNC was suggesting I was connected - it wasn't asking for my username and password - but my connection was getting refused by the server - 'Read/state - Connection disconnected by peer (10054)'

Somewhere I read that 10054 'read/state' might mean that the Application log size was too small and so the server could not accept connections... that was close? Did I type 40kb? I couldn't connect via VNC... but I couldn't connect either via VPN... could that cause that too?

I had to wait until the morning and catch the first person into the office... when I called they said they couldn't log in either... they could log into the server terminal... a quick look at the System log revealed srv errors related to non-paged pool memory.... I got them to reboot so we could all log in ...

When non-paged memory has only 20Mb left then Windows shuts all connections down... IIS6, HTTP.SYS, users are logged out, VPN connections shut down, RealVNC connections... and so on... it does that to prevent resources becoming so low that the system crashes... you are forced to log into the server and sort it out.

David Wang's HowTo post helped point me to Poolmon.exe - a tool that monitors which components are using Paged and Non-Paged memory... Poolmon.exe is one of the Windows Support Tools and can be downloaded from Microsoft.

Once installed, open a cmd window and type: poolmon -b to list by bytes - which is the column to watch...

Because my client's server has just booted this will just tell me initial values... so I am saving to a file: poolmon -b -n datetime.txt. Then I'll import the file into Excel and time stamp each row. I'll run poolmon every now and again and see which values are changing and which aren't.

So far I've got several culprits:

• Irp - - Io, IRP packets
• File - - File objects
• AvgU - an AVG component
• Ntfr - ntfs.sys - ERESOURCE (not to be confused with NtfR)
• MmCa - nt!mm - Mm control areas for mapped files

the largest riser is AvgU, followed by File. That's not surprising considering everyone had been logging in and were now accessing their files...

An hour later:

• FMsl - - fltmgr.sys - STREAM_LIST_CTRL structure
• File - - File objects
• AvgA - an AVG component
• AvgU - an AVG component
• Ntfr - ntfs.sys - ERESOURCE
• MmCa - nt!mm - Mm control areas for mapped files

My bet is that AVG is going to be the culprit... I will stop working on this now. Later I will take one more reading, but I may just remove AVG and install the correct server version - this version looks different to the other servers that I get to look at.

Exchange: 13 month old email received on Blackberry

A client keeps on receiving batches of 10 or so emails on their Blackberry via Exchange... those forwarded to me (on the 19th March 2010) were dated 16th Feb 2009 and 19th October 2009.

The client has Microsoft Exchange 2003, AVG for Exchange, MAC Entourage, Blackberrys using Blackberry Internet Service (BIS).

I have connected up to Outlook Web Access and checked that the emails received are still in the user's Inbox... so there is some kind of periodic sweep over a batch of emails in the Mailbox every now and again...

First look... is AVG again... perhaps biased by not looking at AVG in the last problem (with iisadmin and https services not restarting)... version is 9.0.272... all fairly up-to-date...

Email Scanner for Exchange settings VSAPI (in Advanced Settings -> Server Components) selected components are:

• Background Scan - On
• Proactive Scan - Off
• Scan RTF - On
• Number of Scanning Threads: 9
• Scan Timeout: 180

A background scan could sweep through email messages... I need to find out what AVG says this does...

Background scanning is one of the features of the VSAPI application interface. It provides threaded scanning of the Exchange Messaging Databases. Whenever an item that has not been scanned before is encountered in the users mailbox folders, it is submitted to E-mail Scanner for MS Exchange to be scanned. Scanning and searching for the not examined objects runs in parallel. Note: A specific low priority thread is used for each database, which guarantees other tasks (e.g. e-mail messages storage in the Microsoft Exchange database) are always carried out preferentially.

So... there's a background process, running on a low-priority thread, meaning it'll give up processor time to anything with a higher priority... the timeout is 3 minutes per email - (that's the maximum time the scanner can spend on any one email)... on a busy server that could take a long time to scan a few large emails (at the least 20 emails per hour (unless Exchange is busy sending and receiving emails or responding to lots of Entourage requests from around the Office?)...

So my theory at the moment is that the low priority thread that VSAPI background scanning is working on is having to give way to other higher priority threads...

Another factor is that the client's office is all Mac .. all Entourage... and Entourage talks to Exchange differently than Outlook does... does this explain why other clients don't have a problem with their Blackberry synchronising? I guess I have to look into VSAPI a little...

This from the MS Exchange Team Blog is useful background (Parts 1,2,3):

http://blogs.technet.com/exchange/archive/2004/10/20/245157.aspxhttp://blogs.technet.com/exchange/archive/2004/11/02/251177.aspx

http://blogs.technet.com/exchange/archive/2004/11/15/257737.aspx

First, this advises (in Part 2) switching on Medium Diagnostics Logging on Antivirus Scanning...

In Exchange Admin -> Servers -> Server -> Properties -> Diagnostic Logging tab -> in services: MSExchangeIS -> System -> in categories: Antivirus -> Set Medium or Maximum logging level -> Click OK to exit...

Wow... just noticed while in Exchange Admin ( -> First Storage Group -> Mailboxes) that this particular user's Mailbox is 10GB... the largest of all the users on that network, next largest is 6GB then 4GB... So that is now a potential factor in slow Background Scanning of this mailbox and old emails on her Blackberry...

Diagnostic logging on ... also the Exchange Blog mentions that when an email is scanned it is stamped - "At the completion of the scanning process ptagVirusScanningStamp is updated reflecting the results of the scan. This property holds information such as the vendor, version, scan results, and miscellaneous information regarding the last scan of the item" - Perhaps if the email has never been scanned, this stamp affects the sync?

It's been a few minutes now... so time to check Event Viewer for Antivirus events... (darn.. I reckon I need to restart MSExchangeIS service!)

These came in long before I switched on Max logging level:

Event Type: Error

Event Source: MSExchangeIS

Event Category: Virus Scanning

Event ID: 9581

Date: 20/03/2010

Time: 12:44:23

User: N/A

Computer: SERVER

Description:

Error code -536768764 returned from virus scanner initialization routine. Virus scanner was not loaded.

So if the scanner has been unable to initialise on a regular basis... I should take a note of the dates and times of all Events with ID 9581 - they might tally with old emails being sent - or they might not... I find it's good to have some solid dates and times... It might also happen at a particular time of the day or periodically - so making a note of times and dates helps to see a pattern if one exists... Or simply filter all Events in the Application Log by ID 9581...

ok... these messages occur at 00:44am and 12:43pm every day for the last 3 days 17-20th March 2010; then on the 5th Feb 2010; 16th to 22nd Dec 2009; and 9th/10th Dec 2009

I've got to presume that the rest of the time MSExchangeIS antivirus scanning was working fine... On the days it doesn't work, something regular is interfering... like backups? ... but there are also other times... on 4:44am on 18th Dec... and two this morning at 5am and 8am... which is perhaps when I ran the Exchange Best Practice Analyzer... or Trace Analyzer...

Microsoft Best Practice Analyzers web site...

http://technet.microsoft.com/en-us/exchange/bb288481.aspx

.. what is happening at 12:43 and 00:44? First check backup times, particularly the time that the Exchange backup begins...

It appears that Backup Exec (12) is half way through backing up (with GRT enabled) at 00:44... but it starts an hour earlier and finishes afterwards... and it can't explain the 12:43pm failure...

Ah... AVG Antivirus Update Manager is set to check for virus updates every four hours and the last one was at 12:44pm ... That explains the 4:44am one on the 18th Dec, maybe the 8:42am this morning... but not the 5:09am... perhaps it was when I was playing around with AVG Email settings... ? possibly...

Is event 9581 in 'MSExchangeIS\Virus Scanning' relevant? Time to ask the client for as many dates and times that they received these old emails on their Blackberry... if they can... See if the dates tally with the dates of these errors at all... and also find out whether any of the old emails received were dated after the 9th December... (perhaps they used Blackberry Enterprise Server (BES) before and Internet Service (BIS) after then...?)

Whilst digging around on the Exchange Team Blog... I discovered a post about a change made in 2006 to Exchange that could affect Blackberrys and other services - connected to 'Send As' permissions... I'm not sure that applies here but for anyone with that problem here...

http://blogs.technet.com/exchange/archive/2006/04/28/426707.aspx

they point to kb article - 912918 - Users cannot send emails from a mobile device or from a shared mailbox in Exchange 2000 or 2003 - it mentions that it would break sending emails if you use BES... (perhaps that accounts for an error I saw back in 2008...)

http://support.microsoft.com/kb/912918/

And - Send As permission behaviour change in Exchange 2003

http://support.microsoft.com/kb/895949/

A fix has been released that changes the behavior of the "Full Mailbox Access" feature in Microsoft Exchange Server 2003. Prior to this change, any user with the “Full Mailbox Access” permission for a mailbox also had the ability to “Send As” the mailbox owner.

... the script is not straightforward.. you have to read the first kb article to know how to use it properly... there's no '-?' switch... - doesn't appear to do anything on our server.. no output...

I'm going to leave the server running for a bit with MSExchangeIS Antivirus logging enabled... Seems that Antivirus starts scanning the Mailbox and Public Folders at 00:44 - Public Folders is over in 6 minutes and Mailbox goes on for between 4 and 19 hours. It then starts again... but is quicker. Could be that it only does a full scan if the virus signatures are updated in AVG - then it has to re-scan everything that it has marked suspect.

Going to leave this till I hear back from the user... with diagnostics running on Virus Scanning I'll be able to see a connection next time... or not...

Exchange ActiveSync and the iPhone: Problems Connecting (solved)

I think I have just about tried every trick in the book to get these iPhones connected to my client's SBS 2003 server... from deleting the Virtual Servers in Exchange Admin, removing them from the IIS MetaBase, DS2MB and letting SBS recreate them, checking through all the authentication and access settings, testing SSL, check User-level ActiveSync settings/Outlook Mobile Access and Global-level ActiveSync/OMA settings...

Up for a breather... and ... every time I ran IISRESET it would hang on HTTP SSL service - HTTP Filter.. so then there's no way that IISADMIN would restart... and then IIS is down because W3SVC won't come back up... so it meant a reboot every time...

[One of the problems on the SBS was that 2 updates had failed... .NET 2.0 SP2 (KB976569) was failing over and over - it appeared to take hours (14 hours - I tried to cancel the update and reboot the server - I should have made sure the Update was stopped before rebooting - since I was working on a server 200 miles away I had to wait till the client came in to find the server stuck on shutdown before I could get access again.) - after some investigation into the log files I found it failed because IISADMIN did not restart when asked. The other update was an Intelligent Message Filter for Exchange 2003 SP2 update from Feb 2010 (KB907747). Neither told me why they failed. Both failed due to IISADMIN not restarting (due to HTTP SSL not stopping)... I had to disable IISADMIN, reboot, install the updates, re-enable IISADMIN, reboot. And try to figure out why HTTP SSL was not stopping in a timely fashion.]

So my main focus became digging around in IIS... making sure the OWA and OMA applications in Default Website were attached to the correct ApplicationPool, ExchangeApplicationPool or the ExchangeMobile one...

Errors 3005 3007 for ActiveSync... really just tell me something's up... maybe a timeout... maybe the server is overloaded... but for a big server like that with a handful of users ... hardly likely... must be a setting somewhere... Number of concurrent connections .. in Performance on Default Website properties... that did something.

Check the log files under C:\Windows\System32\LogFiles\HTTPerr and ..\W3SVC1 - go to the end of each file and look for PROPFIND and POST and GET statements from ActiveSync... go to the end of each line and check if you get 409, 207, 403 ... if you're not getting a nice round number like 200, 400 etc then something's up... and it's another pointer in some direction...

You can view which devices have connected by connecting and visiting (using any browser):

https://mail.mydomain.com/exchange//NON_IPM_SUBTREE/Microsoft-Server-ActiveSync

Having fixed a few glitches here and there... and having been up all night and day... mostly waiting for reboots... something was holding HTTP open... and slowing this process down.. people don't reboot every time they make a change to IIS? duh....

I started to run combinations of the installed programs through Google... and there .. lo and behold ...

Antivirus... AVG ... You should not install AVG Online Shield, AVG Firewall and Email Scanner on a Windows Server running Exchange (and definitely not Exchange ActiveSync)

The Online Shield scans HTTP, HTTPS traffic - it has a hook into the HTTP SSL/HTTP Filter service ... and was not specified as a DependsOnService (or vice versa...) so there's no call to it to stop and start... HTTP SSL sits and waits for AVG Online Shield to stop using it for AVG to stop ... but that won't happen...

Avg Email Scanner just adds another layer around POP3 and SMTP ... with it scanning ports 25 and 110 on a machine with Exchange running... adds another layer in the timeout values possibly...

Once Online Shield and Email Scanner were switched off ... IISRESET worked without rebooting .. what a joy I did it several times over and over ...

An hour or two later the client texted me to say his iPhone was getting emails ... marvellous...

These websites were helpful:

Henrik Walther's in-depth Chapter 5 from his book "Securing Exchange Server 2003 & Outlook Web Access" - perfect for understanding the nitty-gritty bits of Exchange HTTP Virtual Folders etc...

AVG - What AVG components are not designed for server operating systems?

http://www.avg.com/us-en/faq.num-1337#num-1337

Microsoft's Exchange ActiveSync Test Website:

https://www.testexchangeconnectivity.com/

Microsoft Exchange ActiveSync Administration Tool ... a tool that you can use to delete old devices from ActiveSync Administration Tool... I should let my client know about it since his old iPhone that he replaced in December is still in the system.

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=e6851d23-d145-4dbf-a2cc-e0b4c6301453

Microsoft Exchange ActiveSync Certificate-Based Authentication Tool:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=82510e18-7965-4883-a8c3-f73f1f4733ac

GoDaddy SSL Certificates - works well with most PDAs, iPhone, Windows Mobile, Android and Exchange ActiveSync - also quite cheap at the moment for secure authentication...

http://www.godaddy.com/ssl/ssl-certificates.aspx?ci=8979

If anyone still needs help ... leave a comment...