Monthly Archives: December 2012

SharePoint 2010, Managed Accounts, and failed password changes

SharePoint managed accounts are COOL!  The ability to have SharePoint manage your service account credentials auto-magically reduces repetitive tasks such as mandatory password changes on a large server farm.  System Admins just love managed accounts….

until they fail.

Failure Points
SharePoint keeps an encrypted copy of the credentials in the configuration store.  This works great *if* the credentials stay in sync with the credentials stored in Active Directory.  There are two common reasons why this credentials would get out-of-sync:

  • The password change fails but SharePoint thinks it was successful
  • The password is manually changed within AD but not within SharePoint

Regardless of the reason, the result is the same:  SharePoint has invalid credentials

Common Symptoms

  • SharePoint sites become unavailable displaying the ever popular
    “An unexpected error has occurred”
  • The server logs fill up with various authentication failed errors
  • The Central Administration –> Managed Accounts page errors out with
    “Object reference not set to an instance of an object” error
    (This is our personal favorite symptom)

[Queue the soft sounds of sobbing in the System Administrator’s office]

Steps to resolve

1st:  Reset the managed accounts to a known password
You must get the service accounts being managed by SharePoint into a known state.  Change the passwords for each impacted service account to a known password.

2nd: Get access to the Central Administration –> Managed Accounts page
The Managed Accounts page will be in an error state.  This is because SharePoint is still attempting to manage the service accounts but fails because of invalid credentials.  To fix, we need to shutoff automatic password changes for all managed accounts.

Enter your local, friendly PowerShell console!

  1. Launch SharePoint’s PowerShell
  2. Enter the following command

Get-SPManagedAccount | where {$_.AutomaticChange -eq $true} |% {$_.AutomaticChange=$false; $_.Update();}

This command will force all managed accounts to NOT attempt an automatic password change.  Once completed, the Managed Accounts page will once again be available.

3rd: Reset impacted Managed Account passwords

  1. Launch Central Administration
  2. Navigate: Security –> Configure Managed Accounts
  3. Click the Edit icon for the impacted service account
  4. Click Change Password Now and select the Use Existing Password option
  5. Enter the new “known” password you created in the 1st step “Reset the managed accounts to a known password”

4th: Reset IIS
Reset IIS using the IISRESET or (our personal favorite) rebooting the server.

Final Thoughts

  1. You will need to reset each managed account’s automatic password change to meet your own corporate policies.
  2. Make a backup!

Creating SQL Server mirror fails with 1488 / 1499 error.

Recently, the AIS support engineers encountered an interesting (and very perplexing) issue with mirroring under SQL Server.  The customer involved experienced a complex failure involving both custom software and hardware on their mirrored SQL Servers.  Our engineers recovered the hardware and principal database within minimal issues or fanfare.

The fun really began during the rebuild of the mirror:

  1. Logons created and associated with the instances?      CHECK!
  2. Principal database online and functioning?                    CHECK!
  3. Backups created from Principal                                      CHECK!
  4. Backups restored with NORECOVERY onto Mirror         CHECK!
  5. Partner configured against Mirror instance                     FAIL!

Database mirroring error: status 1488, severity 16, state 1, string [DB Name]. (Microsoft SQL Server, Error: 1499)

A quick look at
SELECT * FROM sysmessages WHERE msglangid = 1033
in the master database (and finding error 1488) revealed:

Database mirroring cannot be enabled because the “%.*ls” database is in single user mode.

But wait:  Our primary IS in MULTI_USER (and actually servicing customers)

Oh Joy!

[Extensive Research Happens Here]

Root Issue (at least in our case)
It turns out the Principal database was kinda-sorta in single-user mode and kinda-sorta in multi-user mode.  Somewhere buried deep in the metadata stored in the system tables within the database, a bit was still set signaling single-user mode which was respected by the backup/restore engine but not the database engine itself.

We proved this by restoring the backup created from the principal to a test instance without mirroring.  The backup recovered into single-user mode every time.

Basically, the Principal database was in multi-user mode but would create backups sets which were tagged as single-user mode resulting in a mirror failure.

Corrective Action

  1. Disconnected all clients from the Principal database
  2. Forced the Principal database into single-user mode
  3. Counted to 10 (No really, we did)
  4. Forced the Principal back into multi-user mode
  5. Created backup set for the mirror instance
  6. Rebuilt the mirror with no additional issues