Security – Tracking down source of Active Directory user lockouts

active-directorySecuritywindows-server-2008-r2

The Problem

Suddenly, many users of this organization's systems began reporting that they had been locked out of their Active Directory accounts.

The Environment

The entire network is physical and local. There is no direct internet access. There is a firewall that connects the local network via VPN to a broader corperate network. Here are the components of the local network:

  • NY: Active Directory Domain.
  • server1: Windows Server 2008 R2; Active Directory Domain Controller for NY; Remote Desktop Licensing Server; HyperV VM Host; only administrators are allowed to connect.
  • server2: Windows Server 2008 R2; Remote Desktop Session Host; member of NY; users connect to this machine regularly to work on documents.
  • NL: Windows XP; VM running on HyperV hosted on server1; member of NY; used by only a select few users (maybe 3) at regular times each week.
  • FINXFER: Windows XP; VM running on HyperV hosted on server1; member of NY; runs a propriety piece of software that transfers data over the network at regular intervals.
  • Various other Windows XP VMs running on server1 that mostly run background services and aren't usually connected to using Remote Desktop.
  • Various HP thin client machines used for Remote Desktop connection to the above.

My Investigation

I started by looking at the event log on server1 (the domain controller). I filtered for event 4740 "A user account was locked out" and found that there was an occurrence of this event once every 2 to 3 minutes:

Frequent account lockouts

Each occurrence of the event looks like the following:

A user account was locked out.

Subject:
    Security ID:        SYSTEM
    Account Name:       SERVER1$
    Account Domain:     NY
    Logon ID:       0x3e7

Account That Was Locked Out:
    Security ID:        NY\JoeSmith
    Account Name:       JoeSmith

Additional Information:
    Caller Computer Name:   NL

Each occurrence has a different Active Directory username, but the rest is the same in every case.

This is an immediate red flag for me, as the frequency and repetition of the lockouts seem to suggest that someone or something is running down a list of usernames and trying to guess passwords until those users are locked out.

I notice that each of the events contains the line Caller Computer Name: NL, the Microsoft documentation for 4740 say that it contains:

the name of computer account from which logon attempt was received and after which target account was locked out.

As far as I can tell, this means that someone or something is trying to log in to NL using NY credentials, or that something on the NL machine itself is trying to authenticate using NY credentials.

To try and find the source, I enabled the following audit policies under "Local Security Policies" on NL:

Newly enabled security audit policies

As a temporary fix, we started keeping the NL VM off when not in use, and the lockouts do stop while the machine is offline. This has been going on for several weeks now.

Recently I left the NL VM online overnight to let the logs build up, and what I found didn't look good. I found the following two events occurring on NL multiple times a second, throughout the entire night:

Event Type: Failure Audit
Event Source:   Security
Event Category: Account Logon 
Event ID:   680
Date:       1/20/2020
Time:       8:31:24 PM
User:       NT AUTHORITY\SYSTEM
Computer:   NL
Description:
Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0
 Logon account:  Finance
 Source Workstation: FINXFER
 Error Code: 0xC000006A

Followed By:

Event Type: Failure Audit
Event Source:   Security
Event Category: Logon/Logoff 
Event ID:   529
Date:       1/20/2020
Time:       8:31:24 PM
User:       NT AUTHORITY\SYSTEM
Computer:   NL
Description:
Logon Failure:
    Reason:     Unknown user name or bad password
    User Name:  Finance
    Domain:     FINXFER
    Logon Type: 3
    Logon Process:  NtLmSsp 
    Authentication Package: NETWARE_AUTHENTICATION_PACKAGE_V1_0
    Workstation Name:   FINXFER

After these repeated a number of times, I would finally get this event:

Event Type: Success Audit
Event Source:   Security
Event Category: Account Management 
Event ID:   644
Date:       1/20/2020
Time:       8:31:29 PM
User:       NT AUTHORITY\SYSTEM
Computer:   NL
Description:
User Account Locked Out:
    Target Account Name:    Administrator
    Target Account ID:  NL\Administrator
    Caller Machine Name:    FINXFER
    Caller User Name:   NL$
    Caller Domain:  NY
    Caller Logon ID:    (0x0,0x3E7)

This is slightly different than I expected, because it's only trying the "administrator" login instead of random users, which is what I was seeing before. Of course, the administrator account can't actually get locked out, so if this is an attacker it makes sense that that's the user it would try to guess.

Now, the FINXFER VM does transfer information over the network, but it should not be doing anything with the NL machine and definitely not multiple times a second.

Ideas?

What other tools can I use, or what other logs can I search through/enable, in order to hunt down the source of these login attempts that are resulting in account lockouts? How can I find out what program on FNIXFER is initiating the login attempts?

Best Answer

You should really be using Advanced Audit Policies now, and set them for all computers via group policy. The Advanced policies are more granular and provide more specific info.

The article on the Audit Account Lockout policy describes the recommended GPO settings - basically, audit failure for everything, DCs, member servers, workstations. (The categories for "stronger success and stronger failure" are if your environment requires more stringent auditing - "general success/failure" are settings that should be applied anywhere.

I recommend starting at the top of the section and reviewing all of your audit policies in line with their recommendations. At the very least, you should be enabling any logon, Kerberos or authentication audit settings.

On your member server, though, do you see any logon events for the specific users at all? And if so, what kind of logons are they? Interactive, network...? What application(s) are running on the box - is it a web server, terminal server, etc? Are there any Scheduled Tasks running under a domain user context? (rather than SYSTEM) Or any that might be querying AD?

Related Topic