Configure Domain Controller Time Syncronization with External Source


The PDC Emulator in your domain is the only system you should need to run this on. The rest of the systems should sync with this DC. I’ve included the commands below that you can run against your secondary DCs just to make sure they know what they should be doing. This is a rare thing to have to do … but I ended up having to do it today because the time source configured on our domain (long ago) stopped serving up time. WARNING: To much time skew between your current time and the updated time could cause serious headaches for your clients. WARNING!

On your PDC Emulator from an elevated command prompt:

Continue reading “Configure Domain Controller Time Syncronization with External Source”

Upgrade Steps Outline – DPM 2010 to 2012 SP1

System Center

What follows is the basic outline I followed to upgrade System Data Protection Manager 2010 – DPM – running on a Windows Server 2008 R2 host to DPM 2012 SP1. Obviously each step has small libraries of documentation you could read, warnings you could ignore, and whatnot – but if you’re simply looking to make sure you’ve got the right steps planned out, this is what I did to upgrade our setup:

Please note – I did this same set of steps across both of our systems (primary site and secondary site).

  1. Ensure Windows is up to date.
  2. Ensure DPM 2010 is up to date.
  3. Ensure you are running SQL Server 2008 R2 (I have SP1 — 10.50.2550.0)
  4. You are set to install DPM 2012 at this point. Install was simple – put in the disk, provide some passwords… next, next, next. Continue reading “Upgrade Steps Outline – DPM 2010 to 2012 SP1”

Upgrade Steps Outline – SCVMM 2008 SP1 to 2012 SP1


System Center

What follows is the basic outline I followed to upgrade System Center Virtual Machine Manager 2008 SP1 running on a Windows Server 2008 R2 host to SCVMM 2012 SP1. Obviously each step has small libraries of documentation you could read, warning you could ignore, and whatnot – but if you’re simply looking to make sure you’ve got the right steps planned out, this is what I did to upgrade our setup:

  1. Make sure your OS / Data drive are at least 60 GB. You’ll need this for all the upgrades.
  2. Download SCVMM 2012 as well as SCVMM 2012 SP1. You can’t go straight to 2012 SP1 from 2008 SP1
  3. Upgrade SQL 2008 to 2008 R2 if you aren’t running it already. SCVMM 2012 SP1 doesn’t support 2008.
  5. Upgrade SCVMM to 2012
  6. Install SCVMM 2012 RU3  (
  7. Uninstall SCVMM 2012.
    I know, I know, but SCVMM 2012 does not support installation on Server 2012 and because there’s no direct upgrade path to SCVMM 2012 SP1 from 2008, you have to update the SCVMM database. Continue reading “Upgrade Steps Outline – SCVMM 2008 SP1 to 2012 SP1”

Delete An Email From All Mailboxes – Exchange 2007


Another day, another user who decided to give away their username and password to a phishing email. Thankfully this time it happened on a Monday morning, the spammer was kind enough to send to my internal users, and even better sent a spam to our helpdesk email. In other words – they basically told me they were sending spam from one of our mailboxes.

I did the usual – disabled the account/changed the password, blocked the spam/phishing site, purged our (growing) mail queues. But this time I really wanted to get rid of the email. So…
Powershell (Exchange Management Shell) to the rescue. Continue reading “Delete An Email From All Mailboxes – Exchange 2007”

Script to Compare List of Email Addresses in Exchange

I was provided a list of email addresses of employees in a system that doesn’t interface with AD/Exchange and asked to validate those email addresses exist within our Exchange server. I figured the easiest way was to just script it (the list was close to 1000 people).  Total run time to compare the list was just a few seconds.

Here’s the (quite simplistic) script to accomplish the task.

$logFile = 'c:\scripts\IsAccountValid.log'

# Uncomment the entry below if not running from the EMS
# Add-PSSnapin Microsoft.Exchange.Management.PowerShell.Admin

# Import the email addresses from text file
$Import=Get-Content "c:\Scripts\emailaddresses.txt"

ForEach ($address in $import) {
     $valid = get-mailbox -an $address
          If ($valid) {
          "$address is Valid" >> $logFile
          } else {
          "$address is Not Valid" >> $logFile

WSUS 3.0 SP2 will not run after installing update 2720211

Let me clarify that post subject up there. KB2720211 FAILED to install on my server and at the same time absolutely FUBAR’d the entire thing to the point where Update Services wouldn’t even start. I had our lead programmer stop by this morning and tell me that his computer kept complaining about not being able to download updates… (Note to self — add that into Nagios as I appear to have forgotten).

Anway, what a mess. I logged onto the server and along with the MMC not opening, and being greeted with an IIS worker crash warning the event log was chock full of errors:

Event ID 1011: A process serving application pool ‘WsusPool’ suffered a fatal communication error with the World Wide Web Publishing Service. The process id was ‘7540’. The data field contains the error number.
Event ID 7031: The Update Services service terminated unexpectedly.  It has done this 1 time(s).  The following corrective action will be taken in 300000 milliseconds: Restart the service.
Event ID 18456: Login failed for user ‘NT AUTHORITY\NETWORK SERVICE’. [CLIENT: <named pipe>]
Event ID 7032: The WSUS administration console was unable to connect to the WSUS Server via the remote API.

I’m sure you get the point…

Anyway, it appears this was a fairly widespread issue (great QA work MS). I found one solution that worked for me – copied here (just in case):
Continue reading “WSUS 3.0 SP2 will not run after installing update 2720211”

Bulk Migrate User Home Directory in Active Directory

This summer I’m decommissioning an old file server that is no longer under warranty, and we have plans for it along with the iSCSI SAN it’s connected to. Slight problem though – even though I’ve configured our user shares to replicate around using DFS, for years the user creation script that was used for creating accounts hard coded the user home directory to this file server. With thousands of accounts – I’m not about to go and do all this by hand.

Enter Quest ActiveRoles Management Shell (free). Sure I could have used other tools – but I’ve got this one and I like it. Anyway, here’s the script I wrote to do this. Note:  It is likely not perfect, maybe not the most efficient way to handle this (definitely a bit slow), but it works and was ran against a few thousand user accounts.
Continue reading “Bulk Migrate User Home Directory in Active Directory”

Monday Sucks: DPM BMR Restore of Hyper-V Cluster Nodes

First of all – let me tell you that I did not get to this point lightly. As a matter of fact it was quite honestly the LAST damned place I wanted to be. So how did I get here?

A while back Dell, who we purchased our EqualLogic (EQL for short) units from, contacted us about apparently requiring some annual maintenance as part of our service contract. So they email you up out of the blue and basically tell you that you have 30 days to schedule a time with an engineer who will assist you in updating your firmware on your EQL units. Oh and by the way, they want you to update your Host Integration Tools (HIT) Kit on all iSCSI connected servers as well at the same time. So, I went about collecting the information they wanted and getting some MX time setup for a weekend. I understand (and completely agree with) updating firmware and software that fixes big bugs so I don’t really blame Dell for wanting this done and quite honestly once a year is not too big of a deal.

So, the MX day comes along and all goes pretty well aside from the fact that it took about 12 hours to get the whole thing done because of the EQL firmware, switch firmware and the long list of servers that needed to be done, not to mention that (per Dell) HIT kit 3.5.1 needs to be uninstalled before moving to 4.0. Long day. However one little issue sprung up. We run Windows Core for our Hyper-V hosts. The tech who was running the show couldn’t find the documentation for how to remove 3.5.1 from Core – so the install of 4.0 was run directly over 3.5.1. Turns out… that was a BAD BAD move. Why? Well my Core Hyper-V hosts are basically headless and configured to boot directly from iSCSI not from internal disks. After said 4.0 installation – the system for some reason continues to generate more and more iSCSI connections to the boot LUN until…. the server crashes, the VMs on said node failover (ungracefully) to another host and spin back up.  All told my hyper-v hosts crash once a day, at least.

H0w do I know Dell’s HIT Kit is to blame? The logs… it’s all in the logs. I went from a few Event ID 116 (Removal of device xxxx on iSCSI session xxxx was vetoed by xxxx) events a week to  200+ a day per host .. and the logs started going nuts right after the HIT Kit update reboot.

Note: Typically there’s about 40-50 per EQL Unit not 200 per unit. When Nagios starts sending me these alarms I’m guaranteed a host crash within minutes.

Anybody else see the pattern that starts to develop at some random time after boot? This could happen 12 hours after boot or 3 hours. No rhyme or reason. Just starts creating new connections to the LUN and never releases the old ones until the system goes down in flames.

Case opened with Dell, spent almost 4 hours on the phone while they had tech after tech look at it and come up with bubkis. At Dell’s recommendation I’ve uninstalled 4.0 and gone back to 3.5.1. I’ve gone through the registry bit by bit looking for residual 4.0 stuffs. I’ve spent all kinds of time on technet and google looking for anything I can find to try and solve the issue. No joy. I’ve sent a multiple DSETs, a couple Lasso files. Nada. The final straw this morning (summary of email conversation – I was much more polite than this):

Tech: Please send more DSETs so we can send to Engineers.

Me: WTF happened to the last ones I sent on Friday?! Nothing has changed except for more crashes.

Tech: Oh… there they are teehee … let me get them to the engineers.

Me: Like what should have happened two days ago?

… sounds of birds chirping as the emails stopped coming …

Anyway, part of this was my own stupid. I should have (and dammit I thought about it and then forgot to add it to my checklist) but I should have snapshot the boot LUNs prior to the MX. Whoops. Thankfully I was smart enough to make sure I had BMR backups being done on the Hyper-V hosts (and had tested it before actually trusting it).

The process for running a BMR restore is pretty simple – and even though these are clustered hosts the process remains pretty much the same (even from an iSCSI boot perspective).

The generic process can be found here:

The only gotchas with my setup were:

1.I had to transition all the VMs from one host to the other and put the host in MX mode (SCVMM).

2. I had to allow the iSCSI connections to start and then cancel booting from the LUN. I was then prompted to boot from CD. This way the Boot LUN was attached to the server and Windows Setup could see it. Obviously doing the restore setup from Admin Tools wasn’t an option on Core.

3. Since my BMR was apparently a little too old (2 weeks?!) I had to disjoin the host from the domain and rejoin it. Not a big deal and the cluster picked the host right back up as if nothing had changed.

4. Only do one host at a time (depends on your cluster’s ability to tolerate failure obviously).

Now I get to spend many hours monitoring every little thing to make sure the host stays stable and it doesn’t go LUN happy again. Needless to say HIT Kit 4.0 is no longer on my ‘update’ list. Here’s hoping this fixes it…

UPDATE: No Go. This did not fix the issue. At this point I’m at a loss. Going back to a pre-install BMR recovery and still having the same problem — Which did NOT exist when the BMR backup was taken. Temporary solution now is I’m migrating one host to use physical disks inside the chassis (post on how I’m doing that upcoming) – so at least one host is stable and can host the critical VMs along with the ones that can’t handle spontaneous reboots very well (like SQL).