Linux – SSH Connection Refused – Debug using Recovery Console

linuxssh

I have found a ton of questions answered about debugging why one cannot connect via SSH, but they all seem to require that you can still access the system – or say that without that nothing can be done. In my case, I cannot access the system directly, but I do have access to the filesystem using a recovery console.

So this is the situation: My provider made some kernel update today and in the process also rebooted my server. For some reason, I cannot connect via SSH anymore, but instead get a
ssh: connect to host mydomain.de port 22: Connection refused

I do not know whether sshd is just not running, or whether something (e.g. iptables) blocks my ssh connection attempts. I looked at the logfiles, none of the files in /var/log contain any mentioning on ssh, and /var/log/auth.log is empty. Before the kernel update, I could log in just fine and used certificates so that I would not need a password everytime I connect from my local machine.

What I tried so far:

  1. I looked in /etc/rc*.d/ for a link to the /etc/init.d/ssh script and found none. So I am expecting that sshd is not started properly on boot. Since I cannot run any programs in my system, I cannot use update-rc to change this. I tried to make a link manually using ln -s /etc/init.d/ssh /etc/rc6.d/K09sshd and restarted the server – this did not fix the problem. I do not know wether it is at all possible to do it like this and whether it is correct to create it in rc6.d and whether the K09 is correct. I just copied that from apache.

  2. I also tried to change my /etc/iptables.rules file to allow everything:

# Generated by iptables-save v1.4.0 on Thu Dec 10 18:05:32 2009
*mangle
:PREROUTING ACCEPT [7468813:1758703692]
:INPUT ACCEPT [7468810:1758703548]
:FORWARD ACCEPT [3:144]
:OUTPUT ACCEPT [7935930:3682829426]
:POSTROUTING ACCEPT [7935933:3682829570]
COMMIT
# Completed on Thu Dec 10 18:05:32 2009
# Generated by iptables-save v1.4.0 on Thu Dec 10 18:05:32 2009
*filter
:INPUT ACCEPT [7339662:1665166559]
:FORWARD ACCEPT [3:144]
:OUTPUT ACCEPT [7935930:3682829426]
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 993 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 143 -j ACCEPT
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 8080 -s localhost -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
-A INPUT -j ACCEPT
-A FORWARD -j ACCEPT
-A OUTPUT -j ACCEPT
COMMIT
# Completed on Thu Dec 10 18:05:32 2009
# Generated by iptables-save v1.4.0 on Thu Dec 10 18:05:32 2009
*nat
:PREROUTING ACCEPT [101662:5379853]
:POSTROUTING ACCEPT [393275:25394346]
:OUTPUT ACCEPT [393273:25394250]
COMMIT
# Completed on Thu Dec 10 18:05:32 2009

I am not sure this is done correctly or has any effect at all. I also did not find any mentioning of iptables in any file in /var/log.

So what else can I do?
Thank you for your help.

p.s.: After adding the crontab line as suggested by lain, I can find the following in the file /var/logauth.log

Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: sshd version OpenSSH_5.3p1 Debian-3ubuntu5
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: read PEM private key done: type RSA
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: private host key: #0 type 1 RSA
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: read PEM private key done: type DSA
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024
Mar  7 21:13:58 mysubdomain sshd[64900]: debug1: private host key: #1 type 2 DSA

Best Answer

Connection refused suggests that sshd is not running. Try running sshd in debug mode from the command line to see if there are any error messages.

/usr/sbin/sshd -f /etc/ssh/sshd_config -D -d

EDIT:

As you don't have access to run the above try putting it in a @reboot cron job

Add

@reboot root /bin/mkdir -p -m0755 /var/run/sshd && /usr/sbin/sshd -f /etc/ssh/sshd_config -d &>/var/log/sshd_debug

into /etc/crontab

Ubuntu sshd requires that the /var/run/sshd directory exists.