Dell Server PE1800 2GB RAM - Windows SBS Server 2003 R2: One Hour to Reboot

This server has quite a few issues ... the main issue is that the Backup Exec 12 backups are failing each night...

This blog isn't meant to be an anti-AVG blog... but AVG on this client's server is again the wrong version.  There should be safety blocks, detect Exchange running or something in the installer of the normal version to prevent it being installed... I've no idea why it's on there... perhaps it was an upgrade path at one time...

Way to check on an Exchange server - (which took me ages to finally discover but minutes to check now) - Run Poolmon.exe -b from the Server Support Tools for Windows Server 2003... look for the tag, AvgU in the 'NonP' (non-paged memory pool)... it shouldn't be there on an Exchange server.

So now I've uninstalled AVG and am installing the correct Exchange server version.

Rebooting was interesting...  If only for this precise jump in time:

Event Source: EventLog

Event ID: 6006

Date: 17/04/2010      Time: 05:01:12

Computer: SERVER1

Description:       The Event log service was stopped.

--------------------------------------------------------

Event Source: EventLog

Event ID: 6009

Date: 17/04/2010      Time: 06:00:07

Computer:         SERVER1

Description:       Microsoft (R) Windows (R) 5.02. 3790 Service Pack 2 Multiprocessor Free.

--------------------------------------------------------

Event Source: EventLog

Event ID: 6005

Date: 17/04/2010      Time: 06:00:07

Computer: SERVER1

Description:     The Event log service was started.

Anyone seen anything weird like that?  Precisely one hour between Event Log stopping and starting?  Could it be some settting in the Dell ACPI implementation for cooling the fans in the server?  I'm working remote so there's no way to check that without getting my colleague to look on his next visit.

Perhaps so few people notice this because they are sitting next to the machine and get impatient after 15 minutes and hard reset it ... I sat watching Task Manager ordered by Processor Percentage, as you do sometimes (yawn), and noticed dsm_sa_datamgr32.exe popping up into second place every now and again.

That kind of activity is going to play cruel with Virtual Shadow Copy during Adv. File Option activated backups... perhaps why the backup sits either in Snapshot Pre-processing or on-Queue all night and all day... there's not a quiet period during which it can start taking a snapshot... PERC overload...

Services - DSM SA Data Manager: "Provides a common interface and object model to access management information about the operating system, devices, enclosures and management devices. If this service is stopped, several management features will not function properly."

Version installed: 5.8.0.4938 - 3rd Dec 2007... that's a bit old, me reckons... There most certainly should be an update... have to upgrade to Dell Open Manage 6.2 ...

In the mean time I've been warned by AVG 9.0 that the version of Roxio Installed on this server is out of date and might cause problems with AVG 9.0 and other programs... so I found a fix here (tho this might disappear soon) - it asked for reboot and I don't want to until I've installed AVG 9.0 so I'm going to have to ignore the warning from AVG 9.0... and install it .. I won't know if that link solved the problem with Roxio.  Best thing would be to remove Roxio 5... and install.. what is it now? 6, 7, 8, 9, 2010?   I don't know if they are using it or not, so I can't just remove it ..

Strange .. I have found OpenManage 6.1.0 files ... but 5.8 is installed... 6.1.0 files were from 21st Oct 2009... but not installed?  Was a disk space issue apparently...  After doing a WSUS Cleanup there's 6GB free on F: ... so should be ok to install this time...

Seems like Dell brought out an update to Open Manage in Dec 2009... 6.2.0 .. fixes a leak or crash in DSM SA Data Manager ... so I'll update that...

Server updated and now rebooting...  Will it take another hour between the EventLog Stop and Start events in the Application Log?  Let's see...

iPhone and Exchange ActiveSync: not solved - reoccurred

So I got a call that the iPhone stopped getting emails... I didn't check till they went home in case I needed to restart their server out-of-hours ...

When I did connect I decided to run MS Exchange Best Practices Analyser... (search on ExBPA).

I knew one of the Best Practices was having an Application Log size in Event Viewer of 40Mb - so I decided to alter that myself first whilst I was waiting... I must have dozed off at that point ... suddenly I was disconnected... RealVNC was suggesting I was connected - it wasn't asking for my username and password - but my connection was getting refused by the server - 'Read/state - Connection disconnected by peer (10054)'

Somewhere I read that 10054 'read/state' might mean that the Application log size was too small and so the server could not accept connections... that was close? Did I type 40kb? I couldn't connect via VNC... but I couldn't connect either via VPN... could that cause that too?

I had to wait until the morning and catch the first person into the office... when I called they said they couldn't log in either... they could log into the server terminal... a quick look at the System log revealed srv errors related to non-paged pool memory.... I got them to reboot so we could all log in ...

When non-paged memory has only 20Mb left then Windows shuts all connections down... IIS6, HTTP.SYS, users are logged out, VPN connections shut down, RealVNC connections... and so on... it does that to prevent resources becoming so low that the system crashes... you are forced to log into the server and sort it out.

David Wang's HowTo post helped point me to Poolmon.exe - a tool that monitors which components are using Paged and Non-Paged memory... Poolmon.exe is one of the Windows Support Tools and can be downloaded from Microsoft.

Once installed, open a cmd window and type: poolmon -b to list by bytes - which is the column to watch...

Because my client's server has just booted this will just tell me initial values... so I am saving to a file: poolmon -b -n datetime.txt. Then I'll import the file into Excel and time stamp each row. I'll run poolmon every now and again and see which values are changing and which aren't.

So far I've got several culprits:

• Irp - - Io, IRP packets
• File - - File objects
• AvgU - an AVG component
• Ntfr - ntfs.sys - ERESOURCE (not to be confused with NtfR)
• MmCa - nt!mm - Mm control areas for mapped files

the largest riser is AvgU, followed by File. That's not surprising considering everyone had been logging in and were now accessing their files...

An hour later:

• FMsl - - fltmgr.sys - STREAM_LIST_CTRL structure
• File - - File objects
• AvgA - an AVG component
• AvgU - an AVG component
• Ntfr - ntfs.sys - ERESOURCE
• MmCa - nt!mm - Mm control areas for mapped files

My bet is that AVG is going to be the culprit... I will stop working on this now. Later I will take one more reading, but I may just remove AVG and install the correct server version - this version looks different to the other servers that I get to look at.