Ssh – FreeIPA client on LXC container sshd access denied

freeipalxcproxmoxssh

On Proxmox 4.4 I installed Centos 7 VM with FreeIPA server:

ipa-server-install --idstart 10000 --setup-dns

I can use IPA users and log in to other VMs on Proxmox, but when I try same for Centos 7 LXC container I got errors:

May  6 13:15:50 aaaaaa sshd[424]: Authorized to user, krb5 principal user@HOMELAB.LOCAL (ssh_gssapi_krb5_cmdok)
May  6 13:15:50 aaaaaa sshd[424]: pam_sss(sshd:account): Access denied for user user: 4 (System error)
May  6 13:15:50 aaaaaa sshd[424]: fatal: Access denied for user user by PAM account configuration [preauth]

But:

[root@aaaaaa ~]# su - user
Creating home directory for user.
[user@aaaaaa ~]$

And there is now such user on /etc/passwd on this server, so it is from IPA.

[user@aaaaaa ~]$ id
uid=10001(user) gid=10000(admins) groups=10000(admins)
[user@aaaaaa ~]$ getent passwd user
user:*:10001:10000:Name Surename:/home/user:/bin/bash

Also, I can't log in as a root user to this container after I enrolled it to FreeIPA server.

[root@aaaaaa ~]# kinit admin
Password for admin@HOMELAB.LOCAL:
[root@aaaaaa ~]# klist
Ticket cache: KEYRING:persistent:0:krb_ccache_rirBgUU
Default principal: admin@HOMELAB.LOCAL

Valid starting       Expires              Service principal
05/06/2017 14:56:21  05/07/2017 14:56:19  krbtgt/HOMELAB.LOCAL@HOMELAB.LOCAL

So, kerberos worked, but the only problem I have is ssh. I change on Proxmox /etc/subgid and /etc/subuid (as propose here) to get more IDs, but it was desperate move. My IPA ids range is not to high, starting from 10000 and I can do su - user so its not the case.

I think I checked everything including remove sssd db, but its doesn't change anything.

Here is my sssd.conf:

[domain/homelab.local]

cache_credentials = True
krb5_store_password_if_offline = True
ipa_domain = homelab.local
id_provider = ipa
auth_provider = ipa
access_provider = permit
ipa_hostname = aaaaaa.homelab.local
chpass_provider = ipa
dyndns_update = True
ipa_server = _srv_, ipa.homelab.local
dyndns_iface = eth0
ldap_tls_cacert = /etc/ipa/ca.crt
[sssd]
services = nss, sudo, pam, ssh

domains = homelab.local
[nss]
homedir_substring = /home

[pam]

[sudo]

[autofs]

[ssh]

[pac]

[ifp]

And my system-auth:

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        [default=1 success=ok] pam_localuser.so
auth        [success=done ignore=ignore default=die] pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        sufficient    pam_sss.so forward_pass
auth        required      pam_deny.so

account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     [default=bad success=ok user_unknown=ignore] pam_sss.so
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    sufficient    pam_sss.so use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     optional      pam_oddjob_mkhomedir.so umask=0077
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so
session     optional      pam_sss.so

And my sshd_config file:

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
SyslogFacility AUTHPRIV
AuthorizedKeysFile      .ssh/authorized_keys
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials no
UsePAM yes
X11Forwarding yes
UsePrivilegeSeparation sandbox          # Default for new installations.
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
Subsystem       sftp    /usr/libexec/openssh/sftp-server

and below is output from ssh where I can connect to LXC container:

 ssh -vvv user@aaaaaa
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 60: Applying options for *
debug1: Executing proxy command: exec /usr/bin/sss_ssh_knownhostsproxy -p 22 aaaaaa
debug1: permanently_drop_suid: 10001
debug1: identity file /home/user/.ssh/id_rsa type -1
debug1: identity file /home/user/.ssh/id_rsa-cert type -1
debug1: identity file /home/user/.ssh/id_dsa type -1
debug1: identity file /home/user/.ssh/id_dsa-cert type -1
debug1: identity file /home/user/.ssh/id_ecdsa type -1
debug1: identity file /home/user/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/user/.ssh/id_ed25519 type -1
debug1: identity file /home/user/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 6 setting O_NONBLOCK
debug2: fd 5 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "aaaaaa" from file "/home/user/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug3: load_hostkeys: loading entries for host "aaaaaa" from file "/var/lib/sss/pubconf/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:2
debug3: load_hostkeys: found key type RSA in file /var/lib/sss/pubconf/known_hosts:4
debug3: load_hostkeys: found key type DSA in file /var/lib/sss/pubconf/known_hosts:6
debug3: load_hostkeys: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:8
debug3: load_hostkeys: loaded 4 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-rsa,ssh-dss,
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 4f:71:72:5c:46:e5:58:3b:cf:17:75:c9:52:35:38:e9
debug3: load_hostkeys: loading entries for host "aaaaaa" from file "/home/user/.ssh/known_hosts"
debug3: load_hostkeys: loaded 0 keys
debug3: load_hostkeys: loading entries for host "aaaaaa" from file "/var/lib/sss/pubconf/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /var/lib/sss/pubconf/known_hosts:2
debug3: load_hostkeys: found key type RSA in file /var/lib/sss/pubconf/known_hosts:4
debug3: load_hostkeys: found key type DSA in file /var/lib/sss/pubconf/known_hosts:6
debug3: load_hostkeys: found key type ED25519 in file /var/lib/sss/pubconf/known_hosts:8
debug3: load_hostkeys: loaded 4 keys
debug1: Host 'aaaaaa' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/sss/pubconf/known_hosts:2
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/user/.ssh/id_rsa ((nil)),
debug2: key: /home/user/.ssh/id_dsa ((nil)),
debug2: key: /home/user/.ssh/id_ecdsa ((nil)),
debug2: key: /home/user/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup gssapi-keyex
debug3: remaining preferred: gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-keyex
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug3: authmethod_lookup gssapi-with-mic
debug3: remaining preferred: publickey,keyboard-interactive,password
debug3: authmethod_is_enabled gssapi-with-mic
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
Connection closed by UNKNOWN

Of course on server user is not the real login I use.

Any more advice I can check? I struggling with this from few days, and can't find any clue how to solve this. I hope someone here can help me.

UPDATE:

I made a mistake writing that I cant login as a root user. I can. But still can't manage to login via ssh as other user from IPA. Also when I su - user I can't do sudo from this user, but on other VMs this user from IPA can run any command via sudo.

UPDATE2:
I find that on container when I execute kinit user and then klist I got:

Ticket cache: KEYRING:persistent:0:0

But same on VM looks like:

Ticket cache: KEYRING:persistent:10001:krb_ccache_K1JScvu

Best Answer

su - user typically avoids actual authentication because /etc/pam.d/su contains auth sufficient pam_rootok.so as a first line which short-circuits any authentication for root. So SSSD does not get involved at all.

Don't use keyring ccache storage inside containers as keyring is not namespaced. Remove default_ccache_name = KEYRING:persistent:%{uid} from /etc/krb5.conf in the container. libkrb5 will default to FILE:... ccache in /tmp.

Finally, when SSSD reports 'system error', you should use troubleshooting guide to increase debug level for domain section to 9 and analyze logs in the container. See https://web.archive.org/web/20170102152322/https://fedorahosted.org/sssd/wiki/Troubleshooting for details. SSSD project has recently migrated to pagure.io and didn't yet regenerated their documentation after Fedora Hosted infrastructure was retired, thus a link to a wayback machine.