Showing posts with label BES. Show all posts
Showing posts with label BES. Show all posts

Saturday, 11 June 2011

Orphaned Mailboxes and Users after migration from SBS 2003 to SBS 2011

I got a call today from a colleague doing a migration from SBS2003 to SBS2011 ... Exchange 2003 to 2010.

Mailboxes had been copied and he had decommissioned the old server to find that the mailboxes were not there... they were not showing up in the new server... had deleting them removed them from both servers?

At that stage you will have disconnected the connector between the two servers...

First .. go to the disk and check the size of the Exchange Store on the disk... The new store should be the same number of Gb as the old one...

Second.. open the Exchange Mgmt Console and type

  • Get-MailboxStatistics -Server servername
This should list all the mailboxes on the server.  You need to add a little bit to get more details... 
  • Get-MailboxStatistics -Server servername | select DisplayName, DisconnectReason, LegacyDN, ItemCount, MailboxGuid, Database
Moved mailboxes all had a LegacyDN that contains 'first adminstrative group' - this is normal and there's no need to spend hours, as I did, thinking it was an error and trying to correct it... Apparently that doesn't matter...

What really mattered was time... wait...  The mailboxes we had transferred over were large - around 2-4GB... they were neither disabled nor anything... the new server simply had not finished processing them.  I came back to the problem about 12 hours later and they were showing up... perhaps it took less time, but they eventually showed up... in the meantime, I had created some problems for myself...

Users were showing up in AD Users and Computers, but not showing up in SBS Console - I tried remedying that using the 'Change User Roles' trick   But that just created a second mailbox that started collecting emails ... and didn't connect to the old mailbox...  Worry about users not showing up in SBS Console AFTER the mailboxes show up.  When all your users are connected to their mailboxes then run the 'Change User Role' trick.

If you have created a duplicate mailbox, the ItemCount has increased and you really don't want to lose a single email, try connecting via OWA - you might be able to hook up the 'Archive' function and archive those emails out for later retrieval...  The mailboxes I created only picked up 1 or 2 emails and so I disabled the mailbox and used the Get-DisconnectedMailbox username | Remove-DisconnectedMailbox code from Mike Pfeiffer to get rid of the newly created mailbox.

Soon.. all mailboxes were showing up - they appeared in Disconnected Mailboxes in Exchange Manager.  So I selected the mailbox and tried to connect them to a user... but the user was not showing up... The mailbox name was correct - but Exchange could not see the users in Active Directory.. strange...
  • Get-User
Very simple... it produces a list of users and their RecipientType.  The users that Exchange could not see had a type of 'User' but those Exchange can see have a type 'UserMailbox'....  how to enable them?
  • Get-User | where-object{$_.Name -eq "User's Name"}   - this should list one user (just to check before changing anything...)  if ok, run this:
  • Get-MailboxDatabase -Server servername     - this gets the name of the database to use in this:
  • Get-User | where-object($_.Name -eq "user's name") | Enable-Mailbox -Database "database name"
Now run Get-User and check the user has changed RecipientType to UserMailbox.

Now you can go back to Disconnected Mailboxes in Exchange Manager and run Connect to connect them to their mailbox... (if it didn't do it for you when you ran Enable-Mailbox).

Once you have connected all your users up.. go to SBS Console -> Users and Groups -> Users... and run the 'Change User Roles' task - select 'Standard User' in the roles and below select 'Add Role to Users' (NOT 'Replace' - then you are not changing anything).  On the next page you click the checkbox at the bottom 'Show all users from AD' and select all the users (one to start with to test) who you don't see in SBS Console...  run it all through and refresh the view.. all the users should now show up.

One last thing I forgot... and perhaps a trick to get around this issue too... when you run the 'Change User Roles' task each user is given an email address based on the current 'Email Address Policy' ... this is in Exchange Manager.. under Organization Configuration -> Hub Transport -> E-mail Address Policies -> Windows SBS Email Address Policy ... unfortunately 'First Name Only' is NOT an option.. duh!  But we can get around that...
  1. Start -> Run -> adsiedit.msc  <- take care with this... if you get this far you probably are fine...
  2. right-click on AdsiEdit -> Connect To... -> Select a Well-known Naming Context -> Configuration
  3. Select Configuration in the console left panel -> then Configuration -> Services -> Microsoft Exchange ->  servername -> Recipient Policies
  4. In the right-hand panel for Recipient Policies - right-click Windows SBS Email Address Policy select Properties
  5. Click the Filter button and choose to show only attributes that have values
  6. gatewayProxy has a value of SMTP:%m@mydomain.com chang
  7. Click gatewayProxy and Edit - change to SMTP:%g@mydomain.com - %g is FirstName
  8. click ok and exit out of AdsiEdit.msc
And finally .. a warning if you have Blackberry Enterprise Server running... I think I read somewhere there was a knowledgebase article from RIM about what to do in the migration.. .. it was for Exchange 2003 to 2007.. but best to take heed of any advice before attempting to migrate...

Monday, 6 June 2011

BES Hell and the Case of the Missing Window

I was called on Friday... BES Express server down... numskull pulled the plug out of the wall... possibly during a SQL write... pages broken... help!

BESX - version 5.0.2 (Bundle 14)

Where do I start?

Logged into SQL Manager and ran "dbcc checkdb" on all the databases.... consistency errors in some of the tables in BESX... ran "dbcc checkdb (<dbname>, repair_allow_data_loss)" ran "dbcc rebuildindex <tablename>" on the tables... 12 consistency errors fixed...  I accept some data loss in this...

BESX Administration Service - "cannot find the requested page" or "Internet Explorer cannot display the webpage." - the same goes for Web Desktop Manager.  The joy of tying up all of your software's capabilities into a single jvm.dll based web UI... you can't be blamed if it doesn't load up....

So what is to blame?  I don't know.. I search high and low... In the end I uninstalled BESX.  To my regret.

One step back... I wanted to repair install BESX.  So I ran the installer - it stopped - error 2803.  I stopped all the services and tried again.  This time to install BESX "over the top".  I got through to the "Use Existing Configuration Database" .. cool... I can simply install this over the top...

Too soon! "The BlackBerry Configuration Database that you specified is associated with a previous installation of the BlackBerry Enterprise Server or BlackBerry Professional Software. You must create a new BlackBerry Configuration Database for BlackBerry Enterprise Server Express or specify a database that is associated with a previous installation of BlackBerry Enterprise Server Express."

I got fed up with it here - it IS a Config DB for BESX not BES - what are RIM talking about with this error message?... time was running out and I it had to be ready for Monday morning... so I uninstalled BESX.  I regret.

Oh joy... I bet this has something to do with AVG... (lol... I did check but I think AVG was acting normally... it had 58,124 "Virus Found FakeAlert" messages that took forever to not delete.. so when I came back much later a) the program had crashed and b) they had not been deleted ... i so love AVG - not)... 

I found somewhere online mention that I should check that the user BESADMIN has "Allow Log on Locally" as a right before I install BESX (on a side note - I checked through the Release Notes for BESX 5.0.3 (Bundle 12) and decided to download that - in case they'd fixed the 'configuration database is associated with BES and not BESX' error message (bug)... so .. Local Security Policy... er... what's going on here?
MMC could not create the snap-in
MMC could not create the snap-in The snap-in might not have been installed correctly.
Name: Group Policy object Editor
CLSID:{8FC0B734-A0E1-11D1-A7D3-0000F87571E3}
Darn it... something messed up with secpol.msc... Apparently we need gpedit.dll and gptext.dll... they are in System32 folder... and they are registered.. ("regsvr32 gptext.dll" worked but gpedit.dll threw a wobbly...) ... But that didn't solve it... I also need FrameDyn.dll from the wbem folder... it's there... and the program has to find that on the environment variable, PATH

Right... System Properties -> Advanced -> Environment Variables -> PATH= ... er... "...C:\System32;C:\System32\wbem;..."  WTF?!  Where has the "\Window" bit gone?  It goes to show that Windows Server can survive, just about, with this missing ... if you search around online for "c:\system32\wbem" it should not exist ... but it does exist - in one place someone uses it in a solution for the FrameDyn.dll ..  we are pointed to this kb http://support.microsoft.com/?kbid=826282  - be careful with implementing the solution... check that you don't just have a missing 'Window' next to the wbem path.

I wonder now... in retrospect, whether this missing Window was actually the cause of BESX not being able to load the Admin Service or Web Manager.   It can't check the security principals etc... It can't find WMI... 

BESX installed... we couldn't restore the old database over the top.. or copy data from the old db to the new... so we have had to wipe all the users' Blackberrys and reload them... thanks RIM...

The night was so long I can't remember if that was all there was... Space was an issue... an old server, so of course the Windows\installer folder was taking up several gigabytes which can't be removed safely... and the size of the system drive was only 12GB... from the days of small disks...  time for a complete re-install perhaps... 

I doubt anyone who reads this will have exactly the same problem .. this is a Blackberry Enterprise Server Express 5.02 SP1 issue I think.. plus the misfortune of a power failure (or a corrupted BESMgmt SQL DB from any other cause)... together with a malformed PATH variable... you have to be pretty unfortunate to have exactly the same symptoms... :)

I did wonder if it was a recent Windows Update that wrote the PATH variable incorrectly...  it appears to be somewhat common... I don't think people write "C:\System32\wbem" .. but someone has written: "%SYSTEMDRIVE%\system32\wbem" instead of "%SYSTEMROOT%\system32\wbem"

Let me know if that's the work of a virus ... that's the one thing I didn't bother to look into...

Saturday, 20 March 2010

Exchange: 13 month old email received on Blackberry

A client keeps on receiving batches of 10 or so emails on their Blackberry via Exchange... those forwarded to me (on the 19th March 2010) were dated 16th Feb 2009 and 19th October 2009.

The client has Microsoft Exchange 2003, AVG for Exchange, MAC Entourage, Blackberrys using Blackberry Internet Service (BIS).

I have connected up to Outlook Web Access and checked that the emails received are still in the user's Inbox... so there is some kind of periodic sweep over a batch of emails in the Mailbox every now and again...

First look... is AVG again... perhaps biased by not looking at AVG in the last problem (with iisadmin and https services not restarting)... version is 9.0.272... all fairly up-to-date...

Email Scanner for Exchange settings VSAPI (in Advanced Settings -> Server Components) selected components are:
  • Background Scan - On
  • Proactive Scan - Off
  • Scan RTF - On
  • Number of Scanning Threads: 9
  • Scan Timeout: 180
A background scan could sweep through email messages... I need to find out what AVG says this does...
Background scanning is one of the features of the VSAPI application interface. It provides threaded scanning of the Exchange Messaging Databases. Whenever an item that has not been scanned before is encountered in the users mailbox folders, it is submitted to E-mail Scanner for MS Exchange to be scanned. Scanning and searching for the not examined objects runs in parallel. Note: A specific low priority thread is used for each database, which guarantees other tasks (e.g. e-mail messages storage in the Microsoft Exchange database) are always carried out preferentially.
So... there's a background process, running on a low-priority thread, meaning it'll give up processor time to anything with a higher priority... the timeout is 3 minutes per email - (that's the maximum time the scanner can spend on any one email)... on a busy server that could take a long time to scan a few large emails (at the least 20 emails per hour (unless Exchange is busy sending and receiving emails or responding to lots of Entourage requests from around the Office?)...

So my theory at the moment is that the low priority thread that VSAPI background scanning is working on is having to give way to other higher priority threads...

Another factor is that the client's office is all Mac .. all Entourage... and Entourage talks to Exchange differently than Outlook does... does this explain why other clients don't have a problem with their Blackberry synchronising? I guess I have to look into VSAPI a little...

This from the MS Exchange Team Blog is useful background (Parts 1,2,3):

First, this advises (in Part 2) switching on Medium Diagnostics Logging on Antivirus Scanning...

In Exchange Admin -> Servers -> Server -> Properties -> Diagnostic Logging tab -> in services: MSExchangeIS -> System -> in categories: Antivirus -> Set Medium or Maximum logging level -> Click OK to exit...

Wow... just noticed while in Exchange Admin ( -> First Storage Group -> Mailboxes) that this particular user's Mailbox is 10GB... the largest of all the users on that network, next largest is 6GB then 4GB... So that is now a potential factor in slow Background Scanning of this mailbox and old emails on her Blackberry...

Diagnostic logging on ... also the Exchange Blog mentions that when an email is scanned it is stamped - "At the completion of the scanning process ptagVirusScanningStamp is updated reflecting the results of the scan. This property holds information such as the vendor, version, scan results, and miscellaneous information regarding the last scan of the item" - Perhaps if the email has never been scanned, this stamp affects the sync?

It's been a few minutes now... so time to check Event Viewer for Antivirus events... (darn.. I reckon I need to restart MSExchangeIS service!)

These came in long before I switched on Max logging level:

Event Type: Error
Event Source: MSExchangeIS
Event Category: Virus Scanning
Event ID: 9581
Date: 20/03/2010
Time: 12:44:23
User: N/A
Computer: SERVER
Description:
Error code -536768764 returned from virus scanner initialization routine. Virus scanner was not loaded.

So if the scanner has been unable to initialise on a regular basis... I should take a note of the dates and times of all Events with ID 9581 - they might tally with old emails being sent - or they might not... I find it's good to have some solid dates and times... It might also happen at a particular time of the day or periodically - so making a note of times and dates helps to see a pattern if one exists... Or simply filter all Events in the Application Log by ID 9581...

ok... these messages occur at 00:44am and 12:43pm every day for the last 3 days 17-20th March 2010; then on the 5th Feb 2010; 16th to 22nd Dec 2009; and 9th/10th Dec 2009

I've got to presume that the rest of the time MSExchangeIS antivirus scanning was working fine... On the days it doesn't work, something regular is interfering... like backups? ... but there are also other times... on 4:44am on 18th Dec... and two this morning at 5am and 8am... which is perhaps when I ran the Exchange Best Practice Analyzer... or Trace Analyzer...

Microsoft Best Practice Analyzers web site...

.. what is happening at 12:43 and 00:44? First check backup times, particularly the time that the Exchange backup begins...

It appears that Backup Exec (12) is half way through backing up (with GRT enabled) at 00:44... but it starts an hour earlier and finishes afterwards... and it can't explain the 12:43pm failure...

Ah... AVG Antivirus Update Manager is set to check for virus updates every four hours and the last one was at 12:44pm ... That explains the 4:44am one on the 18th Dec, maybe the 8:42am this morning... but not the 5:09am... perhaps it was when I was playing around with AVG Email settings... ? possibly...

Is event 9581 in 'MSExchangeIS\Virus Scanning' relevant? Time to ask the client for as many dates and times that they received these old emails on their Blackberry... if they can... See if the dates tally with the dates of these errors at all... and also find out whether any of the old emails received were dated after the 9th December... (perhaps they used Blackberry Enterprise Server (BES) before and Internet Service (BIS) after then...?)

Whilst digging around on the Exchange Team Blog... I discovered a post about a change made in 2006 to Exchange that could affect Blackberrys and other services - connected to 'Send As' permissions... I'm not sure that applies here but for anyone with that problem here...

they point to kb article - 912918 - Users cannot send emails from a mobile device or from a shared mailbox in Exchange 2000 or 2003 - it mentions that it would break sending emails if you use BES... (perhaps that accounts for an error I saw back in 2008...)

And - Send As permission behaviour change in Exchange 2003
A fix has been released that changes the behavior of the "Full Mailbox Access" feature in Microsoft Exchange Server 2003. Prior to this change, any user with the “Full Mailbox Access” permission for a mailbox also had the ability to “Send As” the mailbox owner.
... the script is not straightforward.. you have to read the first kb article to know how to use it properly... there's no '-?' switch... - doesn't appear to do anything on our server.. no output...

I'm going to leave the server running for a bit with MSExchangeIS Antivirus logging enabled... Seems that Antivirus starts scanning the Mailbox and Public Folders at 00:44 - Public Folders is over in 6 minutes and Mailbox goes on for between 4 and 19 hours. It then starts again... but is quicker. Could be that it only does a full scan if the virus signatures are updated in AVG - then it has to re-scan everything that it has marked suspect.

Going to leave this till I hear back from the user... with diagnostics running on Virus Scanning I'll be able to see a connection next time... or not...