Apache Reverse Proxy – Accessing VirtualHost URL from Localhost with IPTables

Apache2iptablesvirsh

I have an issue with a guest using virsh behind a server running with iptables firewall. This guest hosts websites, one is mattermost with reverse proxy.

Everything is working well. I then installed collabora online 1: https://www.collaboraoffice.com/code/. Super cool to open documents and they have a plugin for mattermost

I tested this plugin from a remote server also running mattermost that reaches out to this local box behind my iptables, and it is working perfectly. However, when I test this plugin from the guest local server itself, behind iptables, so NAT basically, it cannot find itself. I get timed out.

So, my guest behind iptables, with proper rules setup to pass traffic, holds mattermost AND CollaboraOnline, but if I point the public URL from mattermost to fetch CollaboraOnline, both localhost they cannot find each other. I can't do localhost:9980 or 127.0.0.1:9980 (which is where CollaboraOnline is) in the plugin, it does not like the port in the address…

If I do curl https://CollaboraOnline.domain.ca I can see it times out.

If I edit /etc/hosts file on localhost server to 127.0.0.1 CollaboraOnline.domain.ca, then curl works, mattermost-plugin does find CollaboraOnline, but it still will not work because I end up with an SSL type verification error like this.

AH02032: Hostname provided via SNI and hostname provided via HTTP have no compatible SSL setup how to bypass

So now I am running out of bright ideas. I have another public box not behind virsh with iptables, if I do curl to one of its localhost, everything is fine. This only leads me to believe iptables might need a rule for my guest running mattermost and CollaboraOnline to be able to loop back to itself when requesting a public URL it serves itself?!?

Does anyone have any idea about this?

My guest VM is 192.168.122.126 and my parent server hosting Vrish guests and iptables is 192.168.122.1

Here are the iptables rules ( removed some clutter like fail2ban stuff)

iptables -L
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             192.168.122.126      state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

iptables -L -t nat 
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:9980 to:192.168.122.126:9980
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:12000 to:192.168.122.126:12000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:11000 to:192.168.122.126:11000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:submission to:192.168.122.126:587
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:imap2 to:192.168.122.126:143
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:smtp to:192.168.122.126:25
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:webmin to:192.168.122.126:10000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:4443 to:192.168.122.126:4443
DNAT       udp  --  anywhere             148.59.149.79        udp dpt:10000 to:192.168.122.126:10000
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:http to:192.168.122.126:80
DNAT       tcp  --  anywhere             148.59.149.79        tcp dpt:https to:192.168.122.126:443

Best Answer

The culprit here has been a faulty/incomplete NAT rule. During a TCP connection, each endpoint has fixed source / destination IPs, and if an IP packet is received on this port with a different source / destination IP, the packet gets discarded. This is true for both endpoints: client ( curl ) and the server.

To understand the issue, I will follow the packet flow, starting from the client:

  • curl from within the VM sends a packet to the public IP address. Source IP: 192.168.122.126, Destination IP: 148.59.149.79
  • The kernel of the VM checks its routing table, and sends the ip packet to its default gateway, which is 192.168.122.1
  • the kernel of the host receives the packet and passes it through the iptables chains (remember which chain gets traversed first - PREROUTING goes before POSTROUTING )
  • inside the PREROUTING chain, the destination IP gets replaced with 192.168.122.126
  • here, the POSTROUTING chain would get traversed.
  • because of the destination IP, the packet goes back to the VM
  • at the server socket, the IP packet arrives. Destination AND source IP are 192.168.122.126. No Problem so far.
  • The server socket sends the reply - this time, since the destination IP is its own interface, the NAT rules of the host will NOT get applied.
  • The client socket receives the reply from 192.168.122.126 - which is not the IP, to which the client socket has sent its connection request, therefore the packet gets dropped.

The solution is the change of the source IP address in a way, that the reply from the VM's server socket has to pass the host NAT rules:

iptables -t nat -I POSTROUTING 1 -s 192.168.122.126 -d 192.168.122.126 -j SNAT --to-source 192.168.122.1

Remember, that the SNAT target is only applicable inside the POSTROUTING chain - therefore, the DNAT from the PREROUTING chain already has been applied.

With this rule, the server socket inside the VM receives an incoming connection request from 192.168.122.1 - sends the reply, and the reply will get the SNAT / DNAT rules reverse applied, as soon as it arrives at the host: source to 192.168.122.126, and destination to 148.59.149.79.

This reply matches the initial connection request, and the connection attempt succeeds.

Related Topic