Ubuntu – SSHD : “connection closed by remote host”

I am not quite sure what is going on, but I am not able to log in via SSH. I have access to root from an emergency console, so I can still troubleshoot.

Here is my log when I get disconnected less -500 /var/log/auth.log | grep 'sshd'

Jun 20 07:03:09 veksen sshd[4638]: Accepted password for veksen from [myip] port 50535 ssh2
Jun 20 07:03:09 veksen sshd[4638]: pam_unix(sshd:session): session opened for user veksen by (uid=0)
Jun 20 07:03:09 veksen sshd[4638]: pam_unix(sshd:session): session closed for user veksen
Jun 20 07:05:14 veksen sshd[2399]: Received signal 15; terminating.
Jun 20 07:05:54 veksen sshd[4683]: Server listening on 0.0.0.0 port 22.
Jun 20 07:05:54 veksen sshd[4683]: Server listening on :: port 22.

I also notice quite a lot of failed log in that are not from me :

Jun 19 21:59:13 veksen sshd[4073]: Failed password for invalid user gitolite from 50.57.132.36 port 39869 ssh2
Jun 19 22:00:03 veksen sshd[4079]: reverse mapping checking getaddrinfo for 50-57-132-36.static.cloud-ips.com [50.57.132.36] failed - POSSIBLE BREAK-IN ATTEMPT!
Jun 19 22:00:03 veksen sshd[4079]: Invalid user gitolite from 50.57.132.36
Jun 19 22:00:03 veksen sshd[4079]: pam_unix(sshd:auth): check pass; user unknown
Jun 19 22:00:03 veksen sshd[4079]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=50.57.132.36
Jun 19 22:00:04 veksen sshd[4079]: Failed password for invalid user gitolite from 50.57.132.36 port 48132 ssh2
Jun 19 22:12:13 veksen sshd[4101]: reverse mapping checking getaddrinfo for 50-57-132-36.static.cloud-ips.com [50.57.132.36] failed - POSSIBLE BREAK-IN ATTEMPT!
Jun 19 22:12:13 veksen sshd[4101]: Invalid user jenkins from 50.57.132.36
Jun 19 22:12:13 veksen sshd[4101]: pam_unix(sshd:auth): check pass; user unknown
Jun 19 22:12:13 veksen sshd[4101]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=50.57.132.36
Jun 19 22:12:15 veksen sshd[4101]: Failed password for invalid user jenkins from 50.57.132.36 port 56393 ssh2

Here are my sshd config :

# Package generated configuration file
# See the sshd_config(5) manpage for details

# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes

# Lifetime and size of ephemeral version 1 server key
KeyRegenerationInterval 3600
ServerKeyBits 768

# Logging
SyslogFacility AUTH
LogLevel INFO

# Authentication:
LoginGraceTime 120
PermitRootLogin  without-password
StrictModes yes

RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile     %h/.ssh/authorized_keys

# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# For this to work you will also need host keys in /etc/ssh_known_hosts
RhostsRSAAuthentication no
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes

# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords  no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Change to no to disable tunnelled clear text passwords
#PasswordAuthentication yes

# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes

X11Forwarding yes
X11DisplayOffset 10
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no

#MaxStartups 10:30:60
#Banner /etc/issue.net

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

Subsystem sftp /usr/lib/openssh/sftp-server

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes

verbose login ssh -v

debug1: Authentication succeeded (password).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Linux veksen 2.6.27-xenU-4265-i386 #2 SMP Thu Dec 9 09:23:05 UTC 2010 i686 GNU/Linux
Ubuntu 10.04.3 LTS

Welcome to Ubuntu!
 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law.

Last login: Thu Jun 20 07:03:09 2013 from [me].
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
debug1: channel 0: free: client-session, nchannels 1
Connection to [server] closed.
Transferred: sent 1744, received 2536 bytes, in 0.1 seconds
Bytes per second: sent 12140.4, received 17653.6
debug1: Exit status 1

Firewall settings :

Chain INPUT (policy ACCEPT 2379 packets, 261K bytes)
 pkts bytes target     prot opt in     out     source               destination                              
 1467  123K fail2ban-ssh  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0                                        multiport dports 22
    0     0 fail2ban-postfix  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           multiport dports 25,465

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination                              

Chain OUTPUT (policy ACCEPT 2299 packets, 409K bytes)
 pkts bytes target     prot opt in     out     source               destination                              

Chain fail2ban-postfix (1 references)
 pkts bytes target     prot opt in     out     source               destination                              
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0                                

Chain fail2ban-ssh (1 references)
 pkts bytes target     prot opt in     out     source               destination                              
  876 87641 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0