See a heap corruption error from lsass.exe on domain controllers after installing Google Apps Password Sync

domain-controllerg-suitepasswordwindows-server-2003

We're moving to Google Apps and have just installed Google Apps Password Sync. We have yet to roll this out to users and are experiencing some problems.

Occasionally after a user changes their password an error message pops up on a domain controller:

Microsoft Visual C++ Debug Library

Debug Error!

Program: C:\WINDOWS\system32\lsass.exe

HEAP CORRUPTION DETECTED: after Normal block(#1234) at 0x00000000A123456E.
CRT detected that the application wrote to memory after end of heap buffer.

Press Retry to debug the application

Abort Retry Ignore

This message only shows up on the console session (will not show up through an RDP session.) If left unanswered, RDP eventually stops responding. The domain controller also stops replicating and cannot be shut down normally (a You do not have permissions to shutdown error appears).

I've had to perform a hard shutdown on DCs because of this error.

One of the requirements to activate a user account on Google Apps is a password change. I do not want to implement our new password policy and reset every user account until I know this problem is fixed.

What is causing this error and how can I fix it?

Best Answer

These errors only occur when the password length is below eight characters.

Eight characters also happens to be the password length required by Google Apps. If you replace your terribly short and insecure password policy with a proper one, lsass will stop crashing.

You can do this now without forcing all your users to change their passwords immediately, as per the accepted answer in this question.

This also might only affect Windows Server 2003 (the only search result I found on this said the error occured only on their 2003 install and not their 2008.)

It seems the errors happen more frequently as the password gets shorter. Setting a one character or blank password will crash lsass.exe every time. I got through 20,000 password changes using an eight character password before giving up on trying to crash it.

Trying to Google this error sucks. This is a self answer in hopes that I can help some poor soul from the future.