Centos 7 fails to forward port

centos7firewalldport-forwarding

I have two servers running CentOS 7 with public IPs that are also connected to the same LAN. Server A has some VMs, one of which I'm trying to forward port 80 to. I'm doing this using firewall-cmd, and my WAN NIC is joined to the external zone in both cases (LAN is internal zone). This of course means masquerade is enabled (since that's the default for external). I have also checked (at least a dozen times) to make sure that ip_forwarding is set to 1, and that the setting persists after reboot.

Here is the output of firewall-cmd --list-all on Server A:

external (active)
  target: default
  icmp-block-inversion: no
  interfaces: em1
  sources: 
  services: 
  ports: 
  protocols: 
  masquerade: yes
  forward-ports: port=80:proto=tcp:toport=80:toaddr=192.168.11.1
  source-ports: 
  icmp-blocks: 
  rich rules: 

As far as I am aware, this should work, and yet it doesn't. To test, I run nc -vl 80 on the VM, and nc -v [external IP] 80 on an off-site server. The connection times out.

The same configuration on Server B works perfectly fine. I'm able to send and receive text using netcat without issue.


Here's a network diagram for some additional detail:

             WAN
              |
       _____Switch_____
      /                \
    em1               enp1s0      
     |                  |         zone: external
+----------+       +----------+  ---------------
| Server A |       | Server B |
+----------+       +----------+  ---------------
     |                  |         zone: internal
  bridge1            bridge1
   |   |                |
   |  em2            enp3s0
   |   \____Switch_____/
   |
  vnet0
+--------+
| Web VM |
+--------+
  • Network configuration is done via scripts with NM_CONTROLLED=no on all interfaces for both servers. Both servers also have effectively the same configuration (obvious difference being IP addresses).

  • Setting SELinux to permissive changes nothing (it's enabled on Server B without issue, and Server A doesn't have any relevant denials showing up in the audit logs anyway)

  • I can pipe the traffic through netcat like so on Server A: nc -vl 80 | nc -v 192.168.11.1 80, and it works just fine. I can also simply open port 80 on Server A and listen from there without issue. So it's strictly forwarding that's the problem here.

  • I ran iptables -t nat -S to view the loaded rules, and I see the following:

    -A PRE_external_allow -p tcp -m mark --mark 0x66 -j DNAT --to-destination 192.168.11.1:80
    
  • When I run iptables -t nat -vnL | grep :80 I can see the packet counter increase when I try to connect (although again, the connection times out), so I know the rule is being hit.

  • I have tried forwarding the port to a different device on the network not located on either server, with the same results (Server B works, Server A fails).

  • Server A is only just now being connected to WAN for the first time, so I don't know if forwarding was ever working in the past.


As for major differences, I can only think of the following:

  • Both servers are running different kernel versions (Server A is on 3.10.0-693.11.1.el7.x86_64, Server B is on 3.10.0-514.26.2.el7.x86_64). However, I did try running the older kernel on Server A, and that didn't solve the issue (so I'm relatively certain this isn't a kernel bug, unless it's related to the NIC drivers, although that would still be odd).

  • Server A is running firewalld v0.4.4.4, while Server B is on v0.4.3.2. I haven't yet tried rolling back to v0.4.3.2 on Server A (yum can't find that version of the package).

  • As you may have guessed, Server B hasn't had a major update since August. As much as I should, I'm hesitant to update it at this point out of fear the forwarding will break on there as well (rest assured, I will update eventually. Hopefully soon). Unfortunately, this means that there is an exceedingly large number of packages which have newer versions on Server A. Simply doing a diff of package versions would result in a lot of info to sift through.

  • I also noticed that some of the iptables rules were a bit different:

    Server A

    -A POSTROUTING_ZONES -o em1 -g POST_external
    -A POSTROUTING_ZONES -o bridge1 -g POST_internal
    -A POSTROUTING_ZONES -o em2 -g POST_internal
    -A POSTROUTING_ZONES -g POST_external
    
    -A PREROUTING_ZONES -i em1 -g PRE_external
    -A PREROUTING_ZONES -i bridge1 -g PRE_internal
    -A PREROUTING_ZONES -i em2 -g PRE_internal
    -A PREROUTING_ZONES -g PRE_external
    

    Server B

    -A POSTROUTING_ZONES -o enp1s0 -g POST_external
    -A POSTROUTING_ZONES -o bridge1 -j POST_internal
    -A POSTROUTING_ZONES -o enp3s0 -j POST_internal
    -A POSTROUTING_ZONES -g POST_external
    
    -A PREROUTING_ZONES -i enp1s0 -g PRE_external
    -A PREROUTING_ZONES -i bridge1 -j PRE_internal
    -A PREROUTING_ZONES -i enp3s0 -j PRE_internal
    -A PREROUTING_ZONES -g PRE_external
    

    Server B is using jump for the internal zone, while Server A is using goto. This could be due to the difference in firewalld version, and is the only thing I can think of at this point which may be causing this problem (though I still find this unlikely since this is related to the internal zone).

  • Both servers have docker installed and configured. Containers on Server A are strictly internal only. Some containers on Server B have port forwarding to the internet (configured via docker run). Server B also runs OpenVPN.

  • The only other thing I can think of to note is that the hardware is very different between the servers (Server A is a Dell T420, Server B is LGA 775 custom, so just desktop hardware).


I'm starting to go a tad insane at this point. I've even tried enabling masquerading on the internal zone, as I've seen suggested elsewhere (which makes absolutely no sense, and surprise, didn't fix the issue).

What am I missing here?


Update

Tried forwarding locally to a different port on Server A (i.e. firewall-cmd --zone=external --add-forward-port=port=80:proto=tcp:toaddr=192.168.11.3:toport:81). I can connect on port 80 when listening with netcat on Server A on port 81, so local forwarding is working fine. Just can't forward to other IP addresses.

Best Answer

Setting the target for the internal zone to ACCEPT fixed this issue for me.

I.E. firewall-cmd --permanent --zone=internal --set-target=ACCEPT

I'll admit that I somehow missed that setting on Server B. Still, I find it a little odd that outgoing connections aren't accepted by default under the default target.