Linux – SSH Public Key Authentication only works if active session exists before

kvm-virtualizationlinuxremote-accesssshssh-keys

I have a rather strange problem with my SSH configuration. I set up my server with the help of a Remote Access Card and configured everything with a KVM viewer.

So while being logged into the server via the KVM Viewer I configured SSH with only pubkey and tried to login from my local laptop. It worked fine.

If I quit the KVM Session (or logout with the user in the KVM session) I cannot login via ssh anymore (pubkey denied). SSH login only works as long as the user is somewhere still logged in.

Any hints what the problem might be?


Console output for a failed login (all personal data exchanged):

OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/mylocaluser/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 103: Applying options for *
debug1: Connecting to 100.100.100.100 [100.100.100.100] port 12345.
debug1: Connection established.
debug1: identity file /Users/mylocaluser/.ssh/id_rsa type 1
debug1: identity file /Users/mylocaluser/.ssh/id_rsa-cert type -1
debug1: identity file /Users/mylocaluser/.ssh/id_dsa type -1
debug1: identity file /Users/mylocaluser/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA ab:12:23:34:45:56:67:78:89:90:12:23:34:45:56:67
debug1: Host '[100.100.100.100]:12345' is known and matches the RSA host key.
debug1: Found key in /Users/mylocaluser/.ssh/known_hosts:36
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/mylocaluser/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: /Users/mylocaluser/.ssh/id_rsa2
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/mylocaluser/.ssh/id_dsa
debug1: No more authentication methods to try.
Permission denied (publickey).

Console output for a successfull login (only possible while "active session" exists):
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/mylocaluser/.ssh/config
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 103: Applying options for *
debug1: Connecting to 100.100.100.100 [100.100.100.100] port 12345.
debug1: Connection established.
debug1: identity file /Users/mylocaluser/.ssh/id_rsa type 1
debug1: identity file /Users/mylocaluser/.ssh/id_rsa-cert type -1
debug1: identity file /Users/mylocaluser/.ssh/id_dsa type -1
debug1: identity file /Users/mylocaluser/.ssh/id_dsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA ab:12:23:34:45:56:67:78:89:90:12:23:34:45:56:67
debug1: Host '[100.100.100.100]:12345' is known and matches the RSA host key.
debug1: Found key in /Users/mylocaluser/.ssh/known_hosts:36
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/mylocaluser/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 100.100.100.100 ([100.100.100.100]:12345).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = de_DE.UTF-8
Welcome to Ubuntu 14.04.1 LTS

Best Answer

When the user's home directory is encrypted with ecryptfs sshd cannot read the authorized_keys file from the user's home directory before the home directory has been mounted.

During login sshd will use pam to authenticate the user, and pam will use the password entered by the user to mount the encrypted home directory.

This is problematic if you want to restrict sshd to only permit public key authentication.

However you can place an unencrypted authorized_keys file on the server as well. This will permit the user to login using a key, but since this does not invoke pam, the home directory will not be mounted, and mounting the home directory without knowing the password won't work either.

Since the unencrypted home directory gets hidden by the encrypted home directory, placing the unencrypted authorized_keys file in the first place can be a bit tricky. A bind mount of the underlying file system can help with that.

If for example /home is just a directory on the root file system, you can do as follows:

mkdir /mnt/rootfs
mount --bind / /mnt/rootfs

And then you can create /mnt/rootfs/home/$USER/.ssh/authorized_keys

There is more you can do. Since the encrypted and unencrypted version of authorized_keys are two different files, you can put different contents in them. For example the unencrypted version can invoke a script in order to mount the encrypted home directory:

command="/usr/local/bin/ecryptfs-mount-from-ssh" ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDM1Ot12ThbTcPOGpfh7AiRqp3P4BMm3DNo4mDg7gDFPwCmM9rKRHTH0fBVSqkSGlXm84q29bckDukg7vfqkbTpbkP3e2YmTkP6p1J2SoX2QMUnBRRgL9It/ZiAfA2I4QzUrcywVvokO1F2DqcRLy5e5wKTUFfvIm6D2QfBmGbnW2Kkpn16hQyLT1ClXjFC1qXUhazePv0cAtWUCUGjRcLr/ipOphS7eOB46cGhYqtbMkKx0t93ZG4f6jM0o32cYy3RqprpZpTmCeG1gDyG+IlSLBYXYggr72iwTKsTZ9pMDTCBQ8Pb7l317TPOcJzTtDxnpgpGE3x4Vu/Ww+zhsIeT kasperd 2014 May 24

The important part is the command specified before the key. This get invoked instead of the shell. But that happens only when this particular public key is used, and only if the user's home directory is not mounted.

If the user's home directory is already mounted, this authorized_keys file is hidden and the encrypted version is used instead. The encrypted version of authorized_keys does not have the command, so the script to mount the home directory is not run.

So, what goes in the script. Here is my version:

#!/bin/bash -e

if [ $# = 1 ]
then
    PUBKEY="$(
        grep "$1" "$HOME/.ssh/authorized_keys" |
            sed -e 's/.* ssh-rsa //;s/ .*//')"
    /usr/local/bin/ssh-agent-ecryptfs-decryption.py "$PUBKEY" "$1" |
        ecryptfs-unwrap-passphrase "$HOME/.ecryptfs-ssh-wrapped/$1" - |
        ecryptfs-add-passphrase --fnek
fi
ecryptfs-mount-private
cd "$HOME"

if [ "$SSH_ORIGINAL_COMMAND" != "" ]
then
    exec /bin/bash -c "$SSH_ORIGINAL_COMMAND"
fi

exec /bin/bash -l

In the example above, the authorized_keys file is invoked without arguments, so the first if block is skipped. The ecryptfs-mount-private command will thus ask for the user's password. But this does not require sshd to have password authentication enabled, and thus will work on sshd with public key authentication only.

The next command will change to the user's encrypted home directory (until then the script would be running inside the unencrypted home directory).

The last part of the script will run the command given as argument to the ssh command if any, or the users login shell if no command was given.

One caveat is that this does not work with X11 forwarding, because the home directory is not available yet, when the cookie would be stored. But any other session opened while the home directory is already mounted, will be able to handle X11 forwarding.

Using ~/.ssh/rc instead could possibly solve the X11 forwarding issue. This is something I have not looked into yet.

The first if block is a bit of a hack, which I came up with to allow the user's home directory to be mounted without needing a password. Instead it uses a forwarded ssh-agent to mount the user's home directory. That part comes with disclaimers about not having had any peer review, so trusting the cryptography in the ssh-agent-ecryptfs-decryption.py is entirely at your own risk.

The python script looks like this:

#!/usr/bin/env python

from sys import argv
from os import environ
import socket

s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(environ['SSH_AUTH_SOCK'])

def encode_int(v):
    return ('%08x' % v).decode('hex')

def encode_string(s):
    return encode_int(len(s)) + s

def encode_mpint(v):
    h = '%x' % v
    if len(h) & 1: h = '0' + h
    return ('%04x%s' % (len(h) * 4, h)).decode('hex')

key_blob = argv[1].decode('base64')
msg = 'ecryptfs-decrypt ' + argv[2]

s.send(encode_string(chr(13) +
                     encode_string(key_blob) +
                     encode_string(msg) +
                     encode_int(0)))

response = s.recv(1024)

assert response == encode_string(chr(14) + response[5:]), argv[1]

passphrase = response[-48:].encode('base64').replace('\n', '')

print passphrase

So how does the decryption work? First of all the argument to the script as provided by authorized_keys is any random value. A uuid generated with uuidgen could work. The shell script uses grep to find the relevant line in the authorized_keys file to extract the public key.

The base64 encoded public key as well as the uuid are given to the python script. The public key used is exactly the one, which the user authenticated with. The python script asks the forwarded agent for a signature on a specific message using the public key in question (because signing messages is exactly what ssh-agent can do). Part of the signature is then encoded with base64 to produce a password.

This password is used to decrypt an ecryptfs wrapped password file, but the primary file is encrypted using the user's login password. This one is encrypted with a password generated from the ssh key.