I just built a linux bastion, let's call it "bastion1" (IP: 66.66.66.6) on RHEL 8 to replace an older RHEL 6 bastion "bastion0" (IP: 77.77.77.7) that is doing the exact same function. The two servers are configured the same (we use salt to push configurations, etc), The IPtables setup is also fine (all necessary entries have been duplicated for the new IP, etc). For this issue, let's assume my VPN IP is 55.55.55.5, and my username is "user1".
I can successfully ssh from my linux laptop to "bastion1", then ssh from "bastion1" to other servers on our network (in this example, let's call it "host1.ournetwork.com"). So far so good.
We use a config locally (ie: on my laptop) to make ssh "jump" through the bastion to get to another host. This is what's not working. When I say "ssh host1.ournetwork.com" it goes to the bastion, asks for my login, accept it successfully, then tries to get to "host1", and fails. It throws this error…
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Looking at the logs, "host1" shows nothing at all in the logs. "bastion1" shows this in the secure log…
Dec 29 17:25:23 bastion1 sshd[607500]: Accepted password for user1 from 55.55.55.5 port 39028 ssh2
Dec 29 17:25:23 bastion1 sshd[607500]: pam_unix(sshd:session): session opened for user user1 by (uid=0)
Dec 29 17:25:23 bastion1 sshd[607505]: error: connect to host1.ournetwork.com port 22 failed: Permission denied
Dec 29 17:25:23 bastion1 sshd[607500]: pam_unix(sshd:session): session closed for user user1
Obviously I have anonymized the specific info.
My local ssh config file has these entries in it….
# US2 bastion.
Host bastion1
HostName 66.66.66.6
User user1
port 22
ForwardAgent yes
Pubkeyauthentication yes
CertificateFile ~/.ssh/id_rsa-cert.pub
Host *.ournetwork.com
ProxyCommand ssh -A -W %h:%p bastion1
port 22
User user1
Pubkeyauthentication yes
CertificateFile ~/.ssh/id_rsa-cert.pub
So when I type locally "ssh host1.ournetwork.com" it tries to ssh to "bastion1" (66.66.66.6) and asks for password. When it authenticated successfully, it then jumps to "host1.ournetwork.com", where it asks for my password again. This setup has worked successfully for a long time w/ our current rhel6 bastion. Let's assume that it's IP was "77.77.77.7". So all I did locally once "bastion1" came online was change the IP in my local ssh config from 77.77.77.7 to 66.66.66.6
Here's what I get when I try to ssh now…
→ ssh host1.ournetwork.com
WARNING!
========================================================
All access to this machine is monitored. The following
actions are criminal offences and it is our company
policy to prosecute against:
** Unauthorised access to this computer
** Unauthorised viewing, copying or deleting data
** Unauthorised tampering of data
** Unauthorised use of this computer to access other computers.
========================================================
user1@66.66.66.6's password:
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Here's what I should be seeing, and what I see using the old bastion "bastion0"…
→ ssh host1.ournetwork.com
WARNING!
========================================================
All access to this machine is monitored. The following
actions are criminal offences and it is our company
policy to prosecute against:
** Unauthorised access to this computer
** Unauthorised viewing, copying or deleting data
** Unauthorised tampering of data
** Unauthorised use of this computer to access other computers.
========================================================
user1@66.66.66.6's password:
WARNING!
========================================================
All access to this machine is monitored. The following
actions are criminal offences and it is our company
policy to prosecute against:
** Unauthorised access to this computer
** Unauthorised viewing, copying or deleting data
** Unauthorised tampering of data
** Unauthorised use of this computer to access other computers.
========================================================
user1@host1.ournetworks.com's password:
Last login: Tue Dec 29 17:01:29 2020 from 66.66.66.6
I'm guessing that I'm just missing something simple, but I'm not great w/ ssh tunnels, etc so I can't figure out what I missed. Thoughts?
Edited to add…
Figured that someone would ask for a "-v" output, so here it is.
Here's what I see using the new "bastion1"…
→ ssh -v host1.ournetwork.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/user1/.ssh/config
debug1: /home/user1/.ssh/config line 30: Applying options for *.ournetwork.com
debug1: /home/user1/.ssh/config line 51: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Executing proxy command: exec ssh -A -W host1.ournetwork.com:22 bastion1
debug1: identity file /home/user1/.ssh/id_rsa type -1
debug1: identity file /home/user1/.ssh/id_dsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user1/.ssh/id_ed25519 type -1
debug1: identity file /home/user1/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user1/.ssh/id_xmss type -1
debug1: certificate file /home/user1/.ssh/id_rsa-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
WARNING!
========================================================
All access to this machine is monitored. The following
actions are criminal offences and it is our company
policy to prosecute against:
** Unauthorised access to this computer
** Unauthorised viewing, copying or deleting data
** Unauthorised tampering of data
** Unauthorised use of this computer to access other computers.
========================================================
user1@66.66.66.6's password:
channel 0: open failed: connect failed: open failed
stdio forwarding failed
kex_exchange_identification: Connection closed by remote host
Here's what I see using "bastion0" which actually works…
→ ssh -v host1.ournetwork.com
OpenSSH_8.2p1 Ubuntu-4ubuntu0.1, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /home/user1/.ssh/config
debug1: /home/user1/.ssh/config line 30: Applying options for *.ournetwork.com
debug1: /home/user1/.ssh/config line 51: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Executing proxy command: exec ssh -A -W host1.ournetwork.com:22 bastion1
debug1: identity file /home/user1/.ssh/id_rsa type -1
debug1: identity file /home/user1/.ssh/id_dsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa type -1
debug1: identity file /home/user1/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/user1/.ssh/id_ed25519 type -1
debug1: identity file /home/user1/.ssh/id_ed25519_sk type -1
debug1: identity file /home/user1/.ssh/id_xmss type -1
debug1: certificate file /home/user1/.ssh/id_rsa-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.1
WARNING!
========================================================
All access to this machine is monitored. The following
actions are criminal offences and it is our company
policy to prosecute against:
** Unauthorised access to this computer
** Unauthorised viewing, copying or deleting data
** Unauthorised tampering of data
** Unauthorised use of this computer to access other computers.
========================================================
user1@77.77.77.7's password:
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000002
debug1: Authenticating to host1.ournetwork.com:22 as 'user1'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:12Twz9Tp+BLbi91KWZ1gIyA3kNKns64hIK6BXkZcsls
debug1: Host 'host1.ournetwork.com' is known and matches the RSA host key.
debug1: Found key in /home/user1/.ssh/known_hosts:37
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/user1/.ssh/id_rsa-cert.pub RSA-CERT SHA256:ABJwputoncHL/SXD48hdFTH7gomP59BQEJxW/gGNa28 explicit
debug1: Will attempt key: /home/user1/.ssh/id_rsa
debug1: Will attempt key: /home/user1/.ssh/id_dsa
debug1: Will attempt key: /home/user1/.ssh/id_ecdsa
debug1: Will attempt key: /home/user1/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/user1/.ssh/id_ed25519
debug1: Will attempt key: /home/user1/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/user1/.ssh/id_xmss
debug1: SSH2_MSG_SERVICE_ACCEPT received
WARNING!
========================================================
All access to this machine is monitored. The following
actions are criminal offences and it is our company
policy to prosecute against:
** Unauthorised access to this computer
** Unauthorised viewing, copying or deleting data
** Unauthorised tampering of data
** Unauthorised use of this computer to access other computers.
========================================================
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user1/.ssh/id_rsa-cert.pub RSA-CERT SHA256:ABJwputoncHL/SXD48hdFTH7gomP59BQEJxW/gGNa28 explicit
debug1: Server accepts key: /home/user1/.ssh/id_rsa-cert.pub RSA-CERT SHA256:ABJwputoncHL/SXD48hdFTH7gomP59BQEJxW/gGNa28 explicit
debug1: Trying private key: /home/user1/.ssh/id_rsa
debug1: Trying private key: /home/user1/.ssh/id_dsa
debug1: Trying private key: /home/user1/.ssh/id_ecdsa
debug1: Trying private key: /home/user1/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/user1/.ssh/id_ed25519
debug1: Trying private key: /home/user1/.ssh/id_ed25519_sk
debug1: Trying private key: /home/user1/.ssh/id_xmss
debug1: Next authentication method: password
user1@host1.ournetwork.com's password:
debug1: Authentication succeeded (password).
Authenticated to host1.ournetwork.com (via proxy).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
Last login: Tue Dec 29 18:25:58 2020 from 77.77.77.7
Best Answer
I found the cause. selinux was blockingf me. I had missed the error when I trolled the audit log previously, although I don't know how I could have missed it.
All I had to do was set the "nis_enabled" boolean to enabled, and the problem is gone. :)