Ssh – VPC SSH port forward into private subnet

amazon-vpcamazon-web-servicesiptablesport-forwardingssh

Ok, so I've been racking my brain for DAYS on this dilema. I have a VPC setup with a public subnet, and a private subnet. The NAT is in place of course. I can connect from SSH into a instance in the public subnet, as well as the NAT. I can even ssh connect to the private instance from the public instance. I changed the SSHD configuration on the private instance to accept both port 22 and an arbitrary port number 1300. That works fine.

But I need to set it up so that I can connect to the private instance directly using the 1300 port number, ie.

ssh -i keyfile.pem user@1.2.3.4 -p 1300

and 1.2.3.4 should route it to the internal server 10.10.10.10.

Now I heard iptables is the job for this, so I went ahead and researched and played around with some routing with that. These are the rules I have setup on the public instance (not the NAT). I didn't want to use the NAT for this since AWS apperantly pre-configures the NAT instances when you set them up and I heard using iptables can mess that up.

*filter
:INPUT ACCEPT [129:12186]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [84:10472]
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -p tcp -m state --state NEW -m tcp --dport 1300 -j ACCEPT
-A INPUT -d 10.10.10.10/32 -p tcp -m limit --limit 5/min -j LOG --log-prefix "SSH Dropped: "
-A FORWARD -d 10.10.10.10/32 -p tcp -m tcp --dport 1300 -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
COMMIT
# Completed on Wed Apr 17 04:19:29 2013
# Generated by iptables-save v1.4.12 on Wed Apr 17 04:19:29 2013
*nat
:PREROUTING ACCEPT [2:104]
:INPUT ACCEPT [2:104]
:OUTPUT ACCEPT [6:681]
:POSTROUTING ACCEPT [7:745]
-A PREROUTING -i eth0 -p tcp -m tcp --dport 1300 -j DNAT --to-destination 10.10.10.10:1300
-A POSTROUTING -p tcp -m tcp --dport 1300 -j MASQUERADE
COMMIT

So when I try this from home. It just times out. No connection refused messages or anything. And I can't seem to find any log messages about dropped packets.

My security groups and ACL settings allow communications on these ports in both directions in both subnets and on the NAT. I'm at a loss. What am I doing wrong?

Best Answer

For posterity, this is a 5 step process. Bastion host refers to the public facing server, which you'll want to harden against potential attacks and through which your connections will travel to servers within your private subnet.

Step 1: Enable IP Forwarding (Bastion Host)

SSH to the bastion host and at the prompt, execute the following command:

echo 1 > /proc/sys/net/ipv4/ip_forward

Step 2: Modify IP Tables (Bastion Host)

SSH to the bastion host and at the prompt, execute the following commands:

sudo iptables -t nat -A PREROUTING -p tcp -m tcp --dport {PUBLIC_SSH_PORT} -j DNAT --to-destination {PRIVATE_IP_ADDRESS}:22
sudo iptables -t nat -A POSTROUTING -p tcp -m tcp --dport {PUBLIC_SSH_PORT} -j MASQUERADE

where:

{PUBLIC_SSH_PORT} is the port on the bastion host you plan to connect through
{PRIVATE_IP_ADDRESS} is the address of the server in the private subnet you plan to connect to

For more detailed information on what these configurations mean, you can run the following commands at the command prompt to view the man pages:

man iptables
man iptables-extensions

Step 3: Configure Security Groups (Amazon AWS Console)

Login to your Amazon AWS console and navigate to the Security Groups dashboard. Edit the security group assigned to your bastion host and add:

Custom TCP Rule to allow inbound and outbound connections on {PUBLIC_SSH_PORT} from whatever IP address you will be connecting from
Custom TCP Rule to allow inbound and outbound connections for All Traffic to/from the security group of the private server (type in the security group ID in the Custom IP field) - you could restrict this to just SSH traffic but if you have other services and servers running that you need to access, this is the simplest configuration

Edit the security group assigned to your private server and add:

Custom TCP Rule to allow inbound and outbound connections for All Traffic to/from the security group of the bastion host (type in the security group ID in the Custom IP field) - you could restrict this to just SSH traffic but if you have other services and servers running that you need to access, this is the simplest configuration

Step 4: Disable Source/Destination check (Amazon AWS Console)

Login to your Amazon AWS console and navigate to the Instances dashboard. Click on the bastion host that you just configured to forward connections. Disable source/destination checks from the action menu. This is the only device for which source/destination checks need to be disabled.

Step 5: Make Configurations Persistent (Across Server Restarts)

Modify the /etc/sysctl.conf file, and update the following line (change 0 to 1):

# Controls IP packet forwarding
net.ipv4.ip_forward = 1

Save the iptables configuration (to /etc/sysconfig/iptables) so it will be reloaded on boot:

sudo service iptables save

And finally, make sure that iptables itself starts up on boot:

sudo chkconfig iptables on

NOTE: You don't need to setup and configure a separate Amazon NAT instance to gain SSH access to private servers. The NAT instance is ideal if you plan to manage Internet access for various devices you may have within your private subnet.

Related Solutions

Linux – iptables port forward forwarding

You're sending the traffic to 10.52.208.221. Given the config you posted, your problem is the webserver, not the firewall. Your rules look to be correct. FORWARD and INPUT are redirected to RH-Firewall-1-INPUT where your first rule is to allow all traffic. As somebody else pointed out, you could be allowing all traffic on eth1, while the world is actually coming in eth0. Secondary, you have to NAT the traffic as it goes back out to the world

iptables --table nat --append POSTROUTING --proto tcp --dport 80 --jump MASQUERADE -o OUT_INTERFACE

Your traffic originating from the router will never hit the input or forward chains, but instead traverse the output chain on to the webserver. Other systems in that subnet will similarly go directly to the webserver. Systems out on the world at large however will traverse the INPUT / FORWARD chains and need SNAT as well as DNAT so that it appears to the world to be one machine. You still cannot test from within your network. you must test from the opposite interface from the webserver. Get me your IP addresses and I'll point you to the proper configs.

Iptables – ftp tls firewalled :(

In Passive mode, when the client wants to get a file from the server or send a file to the server, the FTP server will pick a random port and send that port to the FTP client.

When you're not using encryption, a properly configured firewall (using the ip_conntrack_ftp helper kernel module, which may be what you're missing for non-TLS connections) would "listen in" on the connection and mark these connections as RELATED. With encryption the firewall can't listen in.

The quick and dirty solution to this is to configure the FTP server to choose a small range of ports for passive connections, and then allow access to all of these ports. For instance, in vsftpd:

pasv_min_port=12000
pasv_max_port=12049

Then in iptables:

iptables -A INPUT -p tcp -m tcp -i eth0 --dport 12000:12049 -j ACCEPT

Allowing anyone to access these ports opens one possible exploit: if someone were to be scanning them over and over they might get lucky and be able to "beat" the real user to the data port and grab the file. Ideally your FTP server would check and make sure the connection is coming from the same place as the original connection, but thanks to things like "FXP" (transferring files from one server to another server by convincing one to make an active connection to the other's passive data port) some servers don't check the connection by default. You should check your configuration file and see if there is an option to disable FXP, and use it. (vsftpd calls this "promiscuous" and is disabled by default.)