SSH Key Authentication Issue When Not Physically Logged In

authenticationsshssh-keys

UPDATE: This issue was caused by home directory encryption

I have a strange problem. I have created and installed keys and disabled password login for SSH. If I try to SSH to the server I get a "Permission denied (publickey).". HOWEVER, if I hook a monitor and keyboard to the server, and login in the server itself (with the user I am connecting via SSH), this same ssh (key auth) from my laptop suddenly works.

How is it possible that remote SSH works if I am physically logged into the machine but not if I am not physically logged in?

The only info I get in case of the FAILED ssh attempt in /var/log/auth.log (no activity in syslog) is:

sshd[..] Connection closed by authenticating user <myuser> <ip> port <port> [preauth]

I have created keys on my laptop, copied them to the server, and changed the permissions:chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys chown username. ~/.ssh/authorized_keys

I have disabled password login by changing sshd_config:

ChallengeResponseAuthentication no
PasswordAuthentication no
UsePAM no

Full log of the failed remote SSH before the physical login on the server:

ssh -vvv <myuser>@<myhost>
:qOpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "<myhost>" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <myhost> [85.148.172.169] port 22.
debug1: Connection established.
debug1: identity file /home/<myuser>/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<myuser>/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6p1 Ubuntu-4
debug1: match: OpenSSH_7.6p1 Ubuntu-4 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to <myhost>:22 as '<myuser>'
debug3: hostkeys_foreach: reading file "/home/<myuser>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/<myuser>/.ssh/known_hosts:12
debug3: load_hostkeys: loaded 1 keys from <myhost>
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:5P8C9BEdtx1FZJ+TqzHNOxnlpJSw1ufmItzAhRW8uTY
debug3: hostkeys_foreach: reading file "/home/<myuser>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/<myuser>/.ssh/known_hosts:12
debug3: load_hostkeys: loaded 1 keys from <myhost>
debug3: hostkeys_foreach: reading file "/home/<myuser>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/<myuser>/.ssh/known_hosts:13
debug3: load_hostkeys: loaded 1 keys from 85.148.172.169
debug1: Host '<myhost>' is known and matches the ECDSA host key.
debug1: Found key in /home/<myuser>/.ssh/known_hosts:12
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /home/<myuser>/.ssh/id_rsa (0x5635443895b0), agent
debug2: key: /home/<myuser>/.ssh/id_dsa ((nil))
debug2: key: /home/<myuser>/.ssh/id_ecdsa ((nil))
debug2: key: /home/<myuser>/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:tLacXPO8rQRQIoy6xrKfJn2Q9o6bzEhVSnZHAzXtkNM /home/<myuser>/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/<myuser>/.ssh/id_dsa
debug3: no such identity: /home/<myuser>/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/<myuser>/.ssh/id_ecdsa
debug3: no such identity: /home/<myuser>/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/<myuser>/.ssh/id_ed25519
debug3: no such identity: /home/<myuser>/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
<myuser>@<myhost>: Permission denied (publickey).

Best Answer

First of all, please set:

UsePAM yes

in your sshd_config and restart ssh:

# systemctl restart sshd

That does not enable password authentication and it's needed because PAM does much more than just authenticate a user. Chances are that will solve the problem.

If not, you probably have something that mounts the user's home directory after login. Maybe the server has pam_mount or ecryptfs or something similar configured?

If that's the case, you can configure out-of-homedir, per user, authorized keys files, by adding the following to your /etc/ssh/sshd_config on your server:

AuthorizedKeysFile  /etc/ssh/authorized_keys/%u

then create the directory if it does not exist yet:

# mkdir /etc/ssh/authorized_keys/
# chmod 755 /etc/ssh/authorized_keys/

and restart ssh:

# systemctl restart sshd

Now you can install the authorized key for user myuser into /etc/ssh/authorized_keys/myuser, with 644 permission bits:

myuser@server:~$ sudo cp .ssh/authorized_keys /etc/ssh/authorized_keys/myuser
myuser@server:~$ sudo chmod 644 /etc/ssh/authorized_keys/myuser
myuser@server:~$ sudo chown root.root /etc/ssh/authorized_keys/myuser

Best Answer

Related Solutions

SSH Key – How to Create a Public Key from a Private Key

SSH Authentication – Use Only Password and Ignore SSH Key

Related Topic