Ubuntu – PBIS Open AD authentication stops working on ubuntu with errors: “user accout has expired” and “is your account locked?”

active-directoryauthenticationlikewise-openpamUbuntu

We have been using PowerBroker Identity Services Open to authenticate active directory users from ubuntu hosts successfully for six months.

Recently AD authentication stopped working on several workstations after users performed an apt-get upgrade of 200+ packages at once. Authentication attempts give the errors, "invalid password," "user account has expired," or "is your account locked?"

I have not been able to link the issue to a specific package upgrade but workstations build from scratch with the same package versions do not experience the issue. I have tried reinstalling PBIS and validated all the config files but I'm missing something…. I'm at a loss and would love any advice anyone has. I'd rather not have to rebuild another box the next time this happens!

Authentication Attempts

I first verified that the the AD user account was enabled, was not locked and had not expired. Local user authentication works fine through lightdm and ssh.

  1. lightdm

    • valid credentials
      • error returned to user "Invalid password, please try again."
      • auth.log: nothing
      • syslog: nothing

    • incorrect password

      • error returned to user "Invalid password, please try again."

      • auth.log:

        lightdm: [lsass-pam] [module:pam_lsass]pam_sm_authenticate error [login:username][error code:40022]

      • syslog:

        lsass: [LwKrb5GetTgtImpl /builder/src-buildserver/Platform-8.0/src/linux/lwadvapi/threaded/krbtgt.c:276] KRB5 Error code: -1765328360 (Message: Preauthentication failed)
lsass: [lsass] Failed to authenticate user (name = 'username') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 17768

  2. ssh

    • valid credentials

      • ssh disconnect with "Connection closed by IP_ADDRESS."

      • auth.log:

        sshd[18237]: error: PAM: User account has expired for DOMAIN\\USER from HOSTNAME
sshd[18237]: error: Received disconnect from IP_ADDRESS: 13: Unable to authenticate [preauth]
      • syslog: nothing

    • incorrect password

      • ssh disconnect with "Connection closed by IP_ADDRESS."

      • auth.log:

        sshd[18276]: [lsass-pam] [module:pam_lsass]pam_sm_authenticate error [login:domain\username][error code:40022]
sshd[18272]: error: PAM: Authentication failure for domain\\username from hostname

      • syslog

        lsass: [LwKrb5GetTgtImpl /builder/src-buildserver/Platform-8.0/src/linux/lwadvapi/threaded/krbtgt.c:276] KRB5 Error code: -1765328360 (Message: Preauthentication failed)
lsass: [lsass] Failed to authenticate user (name = 'domain\username') -> error = 40022, symbol = LW_ERROR_PASSWORD_MISMATCH, client pid = 18276

  3. Just trying some crazy local stuff (and no, the account is not locked in AD)

            root@hostname:~# su - domain\\username
        su: Authentication failure
        (Ignored)
        reenter password for pam_mount:
        DOMAIN\username@hostname:~$ sudo cat /etc/fstab
        [sudo] password for DOMAIN\username:
        sudo: account validation failure, is your account locked?
        DOMAIN\username@hostname:~$

Configuration

  • Ubuntu 14.04
  • PBIS Open 8.0.1.2029 (pbis-open-8.0.1.2029.linux.x86_64.deb.sh)

  • /opt/pbis/bin/config –dump

    AllowDeleteTo ""  
AllowReadTo ""  
AllowWriteTo ""
MaxDiskUsage 104857600
MaxEventLifespan 90
MaxNumEvents 100000
DomainSeparator "\\"
SpaceReplacement "^"
EnableEventlog false
Providers "ActiveDirectory"
DisplayMotd false
PAMLogLevel "error"
UserNotAllowedError "Access denied"
AssumeDefaultDomain true
CreateHomeDir true
CreateK5Login true
SyncSystemTime true
TrimUserMembership true
LdapSignAndSeal false
LogADNetworkConnectionEvents true
NssEnumerationEnabled true
NssGroupMembersQueryCacheOnly true
NssUserMembershipQueryCacheOnly false
RefreshUserCredentials true
CacheEntryExpiry 14400
DomainManagerCheckDomainOnlineInterval 300
DomainManagerUnknownDomainCacheTimeout 3600
MachinePasswordLifespan 2592000
MemoryCacheSizeCap 0
HomeDirPrefix "/home"
HomeDirTemplate "%H/%D/%U"
RemoteHomeDirTemplate ""
HomeDirUmask "022"
LoginShellTemplate "/bin/bash"
SkeletonDirs "/etc/skel"
UserDomainPrefix "DOMAIN.COM"
DomainManagerIgnoreAllTrusts false
DomainManagerIncludeTrustsList
DomainManagerExcludeTrustsList
RequireMembershipOf "DOMAIN\\DOMAIN-GROUP"
Local_AcceptNTLMv1 true
Local_HomeDirTemplate "%H/local/%D/%U"
Local_HomeDirUmask "022"
Local_LoginShellTemplate "/bin/sh"
Local_SkeletonDirs "/etc/skel"
UserMonitorCheckInterval 1800
LsassAutostart true
EventlogAutostart true

  • /opt/pbis/bin/get-status

    LSA Server Status:

Compiled daemon version: 8.0.1.2029
Packaged product version: 8.0.2029.67662
Uptime:        1 days 1 hours 4 minutes 26 seconds

[Authentication provider: lsa-activedirectory-provider]

        Status:        Online
        Mode:          Un-provisioned
        Domain:        DOMAIN.COM
        Domain SID:    S-1-5-21-3537566271-1428921453-776812789
        Forest:        domain.com
        Site:          NYC
        Online check interval:  300 seconds
        [Trusted Domains: 1]


        [Domain: DOMAIN]

                DNS Domain:       domain.com
                Netbios name:     DOMAIN
                Forest name:      domain.com
                Trustee DNS name:
                Client site name: NYC
                Domain SID:       S-1-5-21-3537566271-1428921453-776812789
                Domain GUID:      0b6b6d88-ea48-314a-8bad-a997a57bc1f4
                Trust Flags:      [0x001d]
                                  [0x0001 - In forest]
                                  [0x0004 - Tree root]
                                  [0x0008 - Primary]
                                  [0x0010 - Native]
                Trust type:       Up Level
                Trust Attributes: [0x0000]
                Trust Direction:  Primary Domain
                Trust Mode:       In my forest Trust (MFT)
                Domain flags:     [0x0001]
                                  [0x0001 - Primary]

                [Domain Controller (DC) Information]

                        DC Name:              dc2.nyc.domain.com
                        DC Address:           10.x.x.50
                        DC Site:              NYC
                        DC Flags:             [0x0000f1fc]
                        DC Is PDC:            no
                        DC is time server:    yes
                        DC has writeable DS:  yes
                        DC is Global Catalog: yes
                        DC is running KDC:    yes

                [Global Catalog (GC) Information]

                        GC Name:              dc1.nyc.domain.com
                        GC Address:           10.x.x.50
                        GC Site:              NYC
                        GC Flags:             [0x0000f3fd]
                        GC Is PDC:            yes
                        GC is time server:    yes
                        GC has writeable DS:  yes
                        GC is running KDC:    yes

  • /opt/pbis/bin/find-objects –user USERNAME

    User object [1 of 1] (S-1-5-21-3537566271-1428921453-776812789-1107)
============
Enabled: yes
Distinguished name: CN=USERNAME,OU=User,OU=User Accounts,DC=domain,DC=com
SAM account name: username
NetBIOS domain name: DOMAIN
UPN: username@DOMAIN.COM
Display Name: First Last
Alias: <null>
UNIX name: DOMAIN\username
GECOS: First LAst
Shell: /bin/bash
Home directory: /home/DOMAIN/username
Windows home directory: \\domain.com\dfs\NYC\Users\username
Local windows home directory:
UID: 1023411283
Primary group SID: S-1-5-21-3537566271-1428921453-776812789-513
Primary GID: 1023410689
Password expired: no
Password never expires: yes
Change password on next logon: no
User can change password: yes
Account disabled: no
Account expired: no
Account locked: no

  • /etc/pbis/pbis-krb5-ad.conf

    [libdefaults]
    default_tgs_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
    default_tkt_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
    preferred_enctypes = AES256-CTS AES128-CTS RC4-HMAC DES-CBC-MD5 DES-CBC-CRC
    dns_lookup_kdc = true
    pkinit_kdc_hostname = <DNS>
    pkinit_anchors = DIR:/var/lib/pbis/trusted_certs
    pkinit_cert_match = &&<EKU>msScLogin<PRINCIPAL>
    pkinit_eku_checking = kpServerAuth
    pkinit_win2k_require_binding = false
    pkinit_identities = PKCS11:/opt/pbis/lib/libpkcs11.so

  • /etc/pam.d/common-session

    session [default=1]                     pam_permit.so
session requisite                       pam_deny.so
session required                        pam_permit.so
session optional                        pam_umask.so
session required                        pam_unix.so
session optional                        pam_mount.so
session [success=ok default=ignore]     pam_lsass.so
session optional                        pam_systemd.so

  • /etc/pam.d/common-auth

    auth    [success=2 default=ignore]      pam_unix.so nullok_secure
auth    [success=1 default=ignore]      pam_lsass.so try_first_pass
auth    requisite                       pam_deny.so
auth    required                        pam_permit.so
auth    optional                        pam_cap.so
auth    optional                        pam_mount.so

  • /opt/pbis/share/pbis.pam-auth-update 

    Name: Likewise
Default: yes
Priority: 250
Conflicts: winbind
Auth-Type: Primary
Auth:
        [success=end default=ignore]    pam_lsass.so try_first_pass
Auth-Initial:
        [success=end default=ignore]    pam_lsass.so
Account-Type: Primary
Account:
        [success=ok new_authtok_reqd=ok default=ignore]         pam_lsass.so unknown_ok
        [success=end new_authtok_reqd=done default=ignore]      pam_lsass.so
Session-Type: Additional
Session:
        sufficient      pam_lsass.so
Password-Type: Primary
Password:
        [success=end default=ignore]    pam_lsass.so use_authtok try_first_pass
Password-Initial:
        [success=end default=ignore]    pam_lsass.so

  • /usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf

    [SeatDefaults]
user-session=ubuntu
greeter-show-manual-login=true

  • /usr/share/lightdm/lightdm.conf.d/50-unity-greeter.conf

    [SeatDefaults]
allow-guest=false
greeter-show-remote-login=false
greeter-show-manual-login=true
greeter-session=unity-greeter

Best Answer

The key line is this:

sshd[18237]: error: PAM: User account has expired for DOMAIN\\USER from HOSTNAME

This indicates that a PAM module believes the account has expired. I'd focus less on auth/session and more on account, which is the facility focused on account properties not related to authentication. Your first task is to identify the module causing the problem. Once you know that, it should be much easier to identify why the module thinks the user should be blocked.

Review the applicable account modules one by one, and try adding the debug flag to individual entries to expand the logging output if you need more hints. If truly stumped and it wouldn't violate the security of a critical environment, you can also try commenting the account lines one at a time until you identify your culprit.

As for what changed, more than likely your PAM config was modified when these packages were installed. Chances are that the users in question were in this state all along, but the database associated with the misbehaving account module was being bypassed. (skipped, commented, not present at all, etc.)

Related Topic