At AIS, we keep several virtual machines images “on ice” to support our test bench validation of the various software packages we develop. Hyper-V is a perfect tool in the software development tool chest as it allows a development team to quickly spin up the appropriate back office tools (Such as IIS or SQL server) to validate everything plays nicely in the sandbox.
Over the last couple of weeks, we were performing validation against MS Exchange 2007 and SQL Server in support of a Time Matters conversion. During our testing, the Exchange server began to act very funny. Issues such as:
- Unable to connect to the virtual host except as the local administrator
- Various errors in the error log for failed services
- E-Mails failing to route correctly
After some research, we realized the Exchange server was no longer validating against the test bench domain controller. Odd….
It turns out that the machine password was out of date on the exchange server resulting in the test bench domain controller refusing to validate the server on the domain. Now the pieces were falling into place. Here is what happened:
- We created the exchange virtual machine and created a backup of the VM
- During our testing, we would reset the test bench baseline by thawing out the backup VM and restarting the VM.
- Every 7 days, the exchange server invalidated and reset its machine password with the Domain Controller. Domain controllers will allow machines to log on with the current or the last use machine password. By allowing the last password to be valid, MS gets around several domain replication issues.
- After 14 days, the machine password in the frozen image was no longer valid resulting in the domain rejecting the machine’s access to the domain.
Fortunately, it is very easy to reset the machine password.
Take a look at this KB article from Microsoft for details
As a parting thought, this will also impact users who backup their VM images as a disaster recovery process. If you recover a virtual machine image which is more than a couple of weeks old, you will very likely notice odd security and domain level issues with the restored image. Resetting the machine password should be part of your SOP for recovery operations.