KRB_AP_ERR_MODIFIED - Event 4 - SBS11

Today I'm seeing several Security-Kerberos event id 4 messages on an SBS 2011 stand-alone server:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server COMPUTER1$. The target name used was RPCSS/COMPUTER2.mydomain.local. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (MYDOMAIN.LOCAL) is different from the client domain (MYDOMAIN.LOCAL), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

It would appear that the IP address for COMPUTER1$ in the DNS, is actually being used by COMPUTER2...  so try to find the IP address for COMPUTER2...

Open Administrator Tools -> DNS
Navigate to DNS->servername->Forward Lookup Zones->mydomain.local
Order by the Data column... which may contain mostly IP addresses
Look down the Data column for duplicated IP addresses.  In my case COMPUTER1 and COMPUTER2 had the IP address 192.168.0.15

Run a CMD window (Windows Key+R->type 'cmd'->OK) and type: ping -a 192.168.0.15 (or whatever the duplicate IP address.  Also you can run nbtstat -A 192.168.0.15 ... this resolved to COMPUTER2...

In the DNS Forward Lookup Zones for mydomain.local, I deleted anything with an IP address of 192.168.0.15 that was not COMPUTER2 - one of them was COMPUTER1.

That should prevent this error message appearing again.

For each of these computers I was also seeing a DistributedCOM Event Id 10009

DCOM was unable to communicate with the computer COMPUTER1.mydomain.local using any of the configured protocols.

These appear every 30 minutes since the last reboot.  But not in the last 90 minutes since I deleted duplicate forward lookup entries... Problem solved?   Now to figure out why the server locked us all out earlier today... WSUS overgrowth on C drive is the first contender... the SSDB is 22 GB....

BES Hell and the Case of the Missing Window

I was called on Friday... BES Express server down... numskull pulled the plug out of the wall... possibly during a SQL write... pages broken... help!

BESX - version 5.0.2 (Bundle 14)

Where do I start?

Logged into SQL Manager and ran "dbcc checkdb" on all the databases.... consistency errors in some of the tables in BESX... ran "dbcc checkdb (<dbname>, repair_allow_data_loss)" ran "dbcc rebuildindex <tablename>" on the tables... 12 consistency errors fixed...  I accept some data loss in this...

BESX Administration Service - "cannot find the requested page" or "Internet Explorer cannot display the webpage." - the same goes for Web Desktop Manager.  The joy of tying up all of your software's capabilities into a single jvm.dll based web UI... you can't be blamed if it doesn't load up....

So what is to blame?  I don't know.. I search high and low... In the end I uninstalled BESX.  To my regret.

One step back... I wanted to repair install BESX.  So I ran the installer - it stopped - error 2803.  I stopped all the services and tried again.  This time to install BESX "over the top".  I got through to the "Use Existing Configuration Database" .. cool... I can simply install this over the top...

Too soon! "The BlackBerry Configuration Database that you specified is associated with a previous installation of the BlackBerry Enterprise Server or BlackBerry Professional Software. You must create a new BlackBerry Configuration Database for BlackBerry Enterprise Server Express or specify a database that is associated with a previous installation of BlackBerry Enterprise Server Express."

I got fed up with it here - it IS a Config DB for BESX not BES - what are RIM talking about with this error message?... time was running out and I it had to be ready for Monday morning... so I uninstalled BESX.  I regret.

Oh joy... I bet this has something to do with AVG... (lol... I did check but I think AVG was acting normally... it had 58,124 "Virus Found FakeAlert" messages that took forever to not delete.. so when I came back much later a) the program had crashed and b) they had not been deleted ... i so love AVG - not)... 

I found somewhere online mention that I should check that the user BESADMIN has "Allow Log on Locally" as a right before I install BESX (on a side note - I checked through the Release Notes for BESX 5.0.3 (Bundle 12) and decided to download that - in case they'd fixed the 'configuration database is associated with BES and not BESX' error message (bug)... so .. Local Security Policy... er... what's going on here?

MMC could not create the snap-in
MMC could not create the snap-in The snap-in might not have been installed correctly.
Name: Group Policy object Editor
CLSID:{8FC0B734-A0E1-11D1-A7D3-0000F87571E3}

Darn it... something messed up with secpol.msc... Apparently we need gpedit.dll and gptext.dll... they are in System32 folder... and they are registered.. ("regsvr32 gptext.dll" worked but gpedit.dll threw a wobbly...) ... But that didn't solve it... I also need FrameDyn.dll from the wbem folder... it's there... and the program has to find that on the environment variable, PATH

Right... System Properties -> Advanced -> Environment Variables -> PATH= ... er... "...C:\System32;C:\System32\wbem;..."  WTF?!  Where has the "\Window" bit gone?  It goes to show that Windows Server can survive, just about, with this missing ... if you search around online for "c:\system32\wbem" it should not exist ... but it does exist - in one place someone uses it in a solution for the FrameDyn.dll ..  we are pointed to this kb http://support.microsoft.com/?kbid=826282  - be careful with implementing the solution... check that you don't just have a missing 'Window' next to the wbem path.

I wonder now... in retrospect, whether this missing Window was actually the cause of BESX not being able to load the Admin Service or Web Manager.   It can't check the security principals etc... It can't find WMI... 

BESX installed... we couldn't restore the old database over the top.. or copy data from the old db to the new... so we have had to wipe all the users' Blackberrys and reload them... thanks RIM...

The night was so long I can't remember if that was all there was... Space was an issue... an old server, so of course the Windows\installer folder was taking up several gigabytes which can't be removed safely... and the size of the system drive was only 12GB... from the days of small disks...  time for a complete re-install perhaps... 

I doubt anyone who reads this will have exactly the same problem .. this is a Blackberry Enterprise Server Express 5.02 SP1 issue I think.. plus the misfortune of a power failure (or a corrupted BESMgmt SQL DB from any other cause)... together with a malformed PATH variable... you have to be pretty unfortunate to have exactly the same symptoms... :)

I did wonder if it was a recent Windows Update that wrote the PATH variable incorrectly...  it appears to be somewhat common... I don't think people write "C:\System32\wbem" .. but someone has written: "%SYSTEMDRIVE%\system32\wbem" instead of "%SYSTEMROOT%\system32\wbem"

Let me know if that's the work of a virus ... that's the one thing I didn't bother to look into...

Macs - Entourage - Split DNS - SBS2003 - Cisco 1841

Here's a question...

In this client's domain, there are a couple of servers.  Both SBS2003.  It is set up with an internal mydomain.local address.

The server has an external name - mail.mydomain.com.

The Macs have been set up with WebDAV and Entourage... and point to mail.mydomain.com... so on the SBS2003 DNS Server, mail.mydomain.com has to be defined as the 192.168.0.2 address of the server.  If it is not then the request for mail.mydomain.com is forwarded off to external DNS servers, which returns the address of the external interface of the Cisco 1841 router...

Clients on the outside can see mail.mydomain.com ... but they often open a VPN connection to the Cisco 1841 router... the router gives them a DNS of something.. it's own DNS proxy perhaps?

So the client has a network connection somewhere in the world... with DNS local to that ISP... they connect to the VPN and are given an IP address of 192.168.0.x ... and DNS is something...

but they can no longer connect to mail.mydomain.com ...

Is there an easy answer?   My only answer so far is to look up CCNA courses nearby... :)  The company who should look after this router want £NNN for a solution they can't guarantee will work... And it's not a client we want to mess around with... it has to work or not.

iPhone and ActiveSync: Message received

So... the user received my test email ...

I think the order of checks if you have problems with iPhone and ActiveSync on Exchange:

1) Check the ActiveSync is working using Microsoft's Online Test Tool
2) Switch on Proactive Scanning and Background Scanning in MSExchangeIS\VirusScan.
3) Check timeout values on all routers under your power. If possible, make the value greater than the MaxHeartbeatInterval registry setting in Exchange. (See previous post). (If not possible, perhaps make the Exchange MaxHeartbeatInterval just lower than the router timeout?)

No doubt he'll update to iOS4.1 soon and we'll get yet another problem...

iPhone and ActiveSync: Antivirus Proactive Scanning

And now I don't just have intermittent scanning... Now it only gets emails when the user opens Outlook...

Something has been cleared up by setting the router timeout to a larger value...

And opening Outlook causes the Antivirus to kick in on the messages. Exchange won't do anything with the messages until they have been scanned.. and they won't be scanned without being accessed first.

Checking the registry for the VirusScan settings...

HKLM\System\CurrentControlSet\Services\MSExchangeIS\VirusScan

The ProactiveScanning should read 1 - BackgroundScanning should read 1... then change ReloadNow to 1 .. and MSExchangeIS will reload... wait a little bit and press F5... ReloadNow will have been changed back to 0.

Now let's see if this works... Finally... (Combined with the timeouts in my last post)

iPhone and Exchange ActiveSync: Router Timeout - cracked it?

The Draytek Vigor 2910 has now had its firmware upgraded to the latest version.


The portmaptime function is now available.  Running 'portmaptime -l' to list the timeout values shows TCP SYN as 60 seconds... well below the recommended 540 seconds for ActiveSync.


I have set it, according to some users, to 900:  'portmaptime -s 900' (s for 'tcp Syn')


The iPhone user reports that whatever change I made, appears to have cracked it... I told him I'm going to wait a couple of weeks before I start celebrating... :)

Blacklisted, Router Spam, Rootkits, Worms

A client has been infected by something nasty... email is not getting through to clients due to blacklisting...

1)  www.mxtoolbox.com  - Lookup the mail server MX record, check for blacklisting.  Find out the date that it happened and any reasons you can find.

2)  Find/fix the infected PC...  As I work remote, I set up a batch file that users can run... 

tasklist /svc > s:\%computername%-tasks.txt
netstat -nabvo > s:\%computername%-netstat.txt
;pause    [ remove commenting semi-colon if you want to]

This assumes there is an S: drive - replace with a network drive letter that you have access to - and save the file as something like 's:\commands.bat'.  Ask someone to run it on all computers.

Check each -netstat.txt file for numerous connections being made where there is a :25 after the Foreign Address (search the text file for ':25').

I've found one PC making such connections...

TCP    10.0.0.12:27163        208.123.68.19:25       SYN_SENT        700
  C:\WINDOWS\System32\mswsock.dll
  C:\WINDOWS\system32\WS2_32.dll
  -- unknown component(s) --
  C:\WINDOWS\system32\kernel32.dll
  [services.exe]

This suggests a dodgy mswsock.dll or software running that is spamming out.  I have checked the other PCs and none of them are infected.  So now to Remote Connect (RDP) into the PC and have a look.

No processes running ... rootkit? .. Ran MalwareBytes' Anti-Malware and found Adware.Starware ... Security Center warning if no Firewall or Antivirus was disabled... AVG looks like it got a random letter named dll file...  and I found Run settings ('rundll32 /dll, startup' in both Local Machine and Local User / Software/Microsoft/Windows/CurrentVersion/Run ... AVG reports the Hiloti... does that send out on smtp?  there's also another file reported as Trojan Horse Generic 17.BEDK.   Adobe monxga32.exe... not Adobe but in a folder called 'Startup' in the Adobe menu group in the Start Menu.

ThePhone.coop has still not responded with a Smart Host for relaying emails... which means their clients will not get their emails... Surely the service assistant didn't need to ask an engineer to call me back just to tell me the name of their Smart Host?

Anti-Malware found 109 issues.. but nothing substantial...  now running AVG's Rootkit Search...  That has not fixed it...  Computer rebooted and still infected.

I have connected to the router and added a new rule to block all SMTP Port 25 traffic from all LAN addresses except the server to any address on the internet.  That will stop the blacklisting organisations from blocking us... and will stop anyone else getting dodgy emails from us.

Now I can remote desktop back into the computer and have another good go at it ...

Strangely 'tasklist /svc' reveals that it is Process ID 708 that is sending smtp packets out.  ID 708 is Services.exe.  Tasklist reveals that Services.exe is a host for Eventlog and PlugPlay.

Sysinternals' Process Explorer with a filter of 'Path contains :smtp' also reveals ID 708.   But it also shows that the operation is TCP Disconnect and TCP Reconnect - which means the router block is working.   There's one inetinfo.exe in there sending smtp to localhost ... which throws the possibility of IISADMIN, SMTPSVC and W3SVC involvement.

Sysinternals' RootkitRevealer (RKR) has found quite a few results...

HKU\1-5-21-nnn..nnnn\UserAssist\{75048700-EF1F-11D0-9888-006097DEACF9}\Count\... ...HRZR_EHACVQY:uggc://vzntrf.tbbtyr.pbz/ ... etc

Further investigation discounts this as suspicious...
1) {75048700-EF1F-11D0-9888-006097DEACF9} is the CLSID for ActiveDesktop - so all these entries are Operating System entries.
2) HRZR_EHACVQY:uggc://vzntrf.tbbtyr.pbz/ is ROT13 encoding for UEME_RUNPIDL:http://images.google.com/

Also see Didier Stevens UserAssist Tool for an easier decryption of these UserAssist entries.

Null containing registry entries found by RKR such as in the hklm\security hive SAC* SAI* and SCM* have the same date (maybe) as the computer was installed... something to do with password hiding perhaps...

HKLM\Software .. Microsoft SQL Server .. apparently this is logged by RKR because SQL often changes quickly whilst the registry is being read.. so comparing it can often show innocent discrepancies.

HKLM\System\ControlSet1\Services\eyviuy
HKLM\System\ControlSet2\Services\eyviuy

These have the same names as the driver file (.sys) I found in the system32\drivers folder...

AVG has popped up .. The Trojans, Hiloti.AL Hiloti.AM have infected two files in System Volume Information\_restore folder...

I have given in and taken it back to a previous system restore point .. early last week.  If it gets re-infected then I'll have to ask someone to go and run Recovery Console and delete the offending system driver file and registry entry without Windows running.  

iPhone and Exchange ActiveSync: not yet solved - router update

Following all the links in the chain...

iPhone --  Router NAT -- (Routing and Remote Access Firewall) -- ActiveSync -- Exchange

Error 3033 for ActiveSync suggested checking the router HTTP(S) timeout.  The router is a Draytek Vigor 2900.  There appeared to be no way to alter the HTTP(S) timeout.  I have since been told that there is a command I can run...

portmaptime -s 900

So I telnetted into the router... type ? for help... the command is not listed... I had a look at the commands whilst I was there... nothing... firmware is 3.1.2 ... the service tech tells me that the portmaptime command was introduced in a later firmware.

Portmaptime -s will alter the TCP SYN Timeout value.  From the Draytek forums it appears that this command should work on the 2800 Series (2820) and other Vigor routers too.

Microsoft recommend a value of 900 seconds.  But there's still no guarantee that upgrading the router firmware and altering the timeout is going to resolve the iPhone<->Exchange problem altogether.  Have to wait and see.

Exchange ActiveSync and iPhone: Messages stopped again

Client reports that ActiveSync has stopped working yet again...  checked PoolMon for Non-paged Pool and it's fine...

There's an event in the Application Log for ActiveSync - ID 3033.

Following this article:
http://support.microsoft.com/kb/905013/en-us

I checked the suggested registry key...

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MasSync\Parameters

And all four DWORDs are missing... I have set them to:

• MinHeartbeatInterval - Dword - Decimal 60 (seconds)
• MaxHeartbeatInterval - Dword - Decimal 2700 (seconds)
• HbiSampleSize - Dword - Decimal 200 (samples)
• HbiAlertThreshold - Dword - Decimal 480 (seconds)

It suggests changing the HTTP(S) timeout values on the router firewall but I can't find them... so I'm going to do a simple IISRESET and see if that resolves the problem.  Trouble is, an IISRESET might just resolve the issue, despite the new registry entries... so I can only warn the client that's a possibility... (when he returns)...

If that doesn't resolve the issue for now... back to the drawing board...

Dell Server PE1800 2GB RAM - Windows SBS Server 2003 R2: One Hour to Reboot

This server has quite a few issues ... the main issue is that the Backup Exec 12 backups are failing each night...

This blog isn't meant to be an anti-AVG blog... but AVG on this client's server is again the wrong version.  There should be safety blocks, detect Exchange running or something in the installer of the normal version to prevent it being installed... I've no idea why it's on there... perhaps it was an upgrade path at one time...

Way to check on an Exchange server - (which took me ages to finally discover but minutes to check now) - Run Poolmon.exe -b from the Server Support Tools for Windows Server 2003... look for the tag, AvgU in the 'NonP' (non-paged memory pool)... it shouldn't be there on an Exchange server.

So now I've uninstalled AVG and am installing the correct Exchange server version.

Rebooting was interesting...  If only for this precise jump in time:

Event Source: EventLog

Event ID: 6006

Date: 17/04/2010      Time: 05:01:12

Computer: SERVER1

Description:       The Event log service was stopped.

--------------------------------------------------------

Event Source: EventLog

Event ID: 6009

Date: 17/04/2010      Time: 06:00:07

Computer:         SERVER1

Description:       Microsoft (R) Windows (R) 5.02. 3790 Service Pack 2 Multiprocessor Free.

--------------------------------------------------------

Event Source: EventLog

Event ID: 6005

Date: 17/04/2010      Time: 06:00:07

Computer: SERVER1

Description:     The Event log service was started.

Anyone seen anything weird like that?  Precisely one hour between Event Log stopping and starting?  Could it be some settting in the Dell ACPI implementation for cooling the fans in the server?  I'm working remote so there's no way to check that without getting my colleague to look on his next visit.

Perhaps so few people notice this because they are sitting next to the machine and get impatient after 15 minutes and hard reset it ... I sat watching Task Manager ordered by Processor Percentage, as you do sometimes (yawn), and noticed dsm_sa_datamgr32.exe popping up into second place every now and again.

That kind of activity is going to play cruel with Virtual Shadow Copy during Adv. File Option activated backups... perhaps why the backup sits either in Snapshot Pre-processing or on-Queue all night and all day... there's not a quiet period during which it can start taking a snapshot... PERC overload...

Services - DSM SA Data Manager: "Provides a common interface and object model to access management information about the operating system, devices, enclosures and management devices. If this service is stopped, several management features will not function properly."

Version installed: 5.8.0.4938 - 3rd Dec 2007... that's a bit old, me reckons... There most certainly should be an update... have to upgrade to Dell Open Manage 6.2 ...

In the mean time I've been warned by AVG 9.0 that the version of Roxio Installed on this server is out of date and might cause problems with AVG 9.0 and other programs... so I found a fix here (tho this might disappear soon) - it asked for reboot and I don't want to until I've installed AVG 9.0 so I'm going to have to ignore the warning from AVG 9.0... and install it .. I won't know if that link solved the problem with Roxio.  Best thing would be to remove Roxio 5... and install.. what is it now? 6, 7, 8, 9, 2010?   I don't know if they are using it or not, so I can't just remove it ..

Strange .. I have found OpenManage 6.1.0 files ... but 5.8 is installed... 6.1.0 files were from 21st Oct 2009... but not installed?  Was a disk space issue apparently...  After doing a WSUS Cleanup there's 6GB free on F: ... so should be ok to install this time...

Seems like Dell brought out an update to Open Manage in Dec 2009... 6.2.0 .. fixes a leak or crash in DSM SA Data Manager ... so I'll update that...

Server updated and now rebooting...  Will it take another hour between the EventLog Stop and Start events in the Application Log?  Let's see...