BES Hell and the Case of the Missing Window

I was called on Friday... BES Express server down... numskull pulled the plug out of the wall... possibly during a SQL write... pages broken... help!

BESX - version 5.0.2 (Bundle 14)

Where do I start?

Logged into SQL Manager and ran "dbcc checkdb" on all the databases.... consistency errors in some of the tables in BESX... ran "dbcc checkdb (<dbname>, repair_allow_data_loss)" ran "dbcc rebuildindex <tablename>" on the tables... 12 consistency errors fixed...  I accept some data loss in this...

BESX Administration Service - "cannot find the requested page" or "Internet Explorer cannot display the webpage." - the same goes for Web Desktop Manager.  The joy of tying up all of your software's capabilities into a single jvm.dll based web UI... you can't be blamed if it doesn't load up....

So what is to blame?  I don't know.. I search high and low... In the end I uninstalled BESX.  To my regret.

One step back... I wanted to repair install BESX.  So I ran the installer - it stopped - error 2803.  I stopped all the services and tried again.  This time to install BESX "over the top".  I got through to the "Use Existing Configuration Database" .. cool... I can simply install this over the top...

Too soon! "The BlackBerry Configuration Database that you specified is associated with a previous installation of the BlackBerry Enterprise Server or BlackBerry Professional Software. You must create a new BlackBerry Configuration Database for BlackBerry Enterprise Server Express or specify a database that is associated with a previous installation of BlackBerry Enterprise Server Express."

I got fed up with it here - it IS a Config DB for BESX not BES - what are RIM talking about with this error message?... time was running out and I it had to be ready for Monday morning... so I uninstalled BESX.  I regret.

Oh joy... I bet this has something to do with AVG... (lol... I did check but I think AVG was acting normally... it had 58,124 "Virus Found FakeAlert" messages that took forever to not delete.. so when I came back much later a) the program had crashed and b) they had not been deleted ... i so love AVG - not)... 

I found somewhere online mention that I should check that the user BESADMIN has "Allow Log on Locally" as a right before I install BESX (on a side note - I checked through the Release Notes for BESX 5.0.3 (Bundle 12) and decided to download that - in case they'd fixed the 'configuration database is associated with BES and not BESX' error message (bug)... so .. Local Security Policy... er... what's going on here?

MMC could not create the snap-in
MMC could not create the snap-in The snap-in might not have been installed correctly.
Name: Group Policy object Editor
CLSID:{8FC0B734-A0E1-11D1-A7D3-0000F87571E3}

Darn it... something messed up with secpol.msc... Apparently we need gpedit.dll and gptext.dll... they are in System32 folder... and they are registered.. ("regsvr32 gptext.dll" worked but gpedit.dll threw a wobbly...) ... But that didn't solve it... I also need FrameDyn.dll from the wbem folder... it's there... and the program has to find that on the environment variable, PATH

Right... System Properties -> Advanced -> Environment Variables -> PATH= ... er... "...C:\System32;C:\System32\wbem;..."  WTF?!  Where has the "\Window" bit gone?  It goes to show that Windows Server can survive, just about, with this missing ... if you search around online for "c:\system32\wbem" it should not exist ... but it does exist - in one place someone uses it in a solution for the FrameDyn.dll ..  we are pointed to this kb http://support.microsoft.com/?kbid=826282  - be careful with implementing the solution... check that you don't just have a missing 'Window' next to the wbem path.

I wonder now... in retrospect, whether this missing Window was actually the cause of BESX not being able to load the Admin Service or Web Manager.   It can't check the security principals etc... It can't find WMI... 

BESX installed... we couldn't restore the old database over the top.. or copy data from the old db to the new... so we have had to wipe all the users' Blackberrys and reload them... thanks RIM...

The night was so long I can't remember if that was all there was... Space was an issue... an old server, so of course the Windows\installer folder was taking up several gigabytes which can't be removed safely... and the size of the system drive was only 12GB... from the days of small disks...  time for a complete re-install perhaps... 

I doubt anyone who reads this will have exactly the same problem .. this is a Blackberry Enterprise Server Express 5.02 SP1 issue I think.. plus the misfortune of a power failure (or a corrupted BESMgmt SQL DB from any other cause)... together with a malformed PATH variable... you have to be pretty unfortunate to have exactly the same symptoms... :)

I did wonder if it was a recent Windows Update that wrote the PATH variable incorrectly...  it appears to be somewhat common... I don't think people write "C:\System32\wbem" .. but someone has written: "%SYSTEMDRIVE%\system32\wbem" instead of "%SYSTEMROOT%\system32\wbem"

Let me know if that's the work of a virus ... that's the one thing I didn't bother to look into...