IPTables – Why Some DNAT Rules Don’t Work Until Reboot?

dnatiptableslinuxnetworkingudp

My iptables DNAT rules don't work until reboot. If I reboot my server, all of the rules work.

Desciption of the architecture :

Tens of hosts (senders) send some UDP packets (one-way on a specific port 9999) to my Linux router. This Linux router use iptables to forward those packets to several hosts (receivers).

senderX 10.0.0.X ====> Linux router with iptables ====> receiverY 10.0.1.Y

The linux router have two network cards eth1 10.0.0.1/24 (senders side) and eth0 10.0.1.1/24 (receivers side).

Iptables setup :

  • ip_forwarding is activated
  • all of the default policies are set to ACCEPT
  • one iptables rules exist per sender, here is an example :
iptables -t nat -A PREROUTING -s 10.0.0.2 -i eth1 -j DNAT --to-destination 10.0.1.123

Network setup :

ip addr show :

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 54:9f:35:0a:16:38 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 brd 10.0.1.255 scope global eth0
    inet6 fe80::569f:35ff:fe0a:1638/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 54:9f:35:0a:16:3a brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global eth1
    inet6 fe80::569f:35ff:fe0a:163a/64 scope link
       valid_lft forever preferred_lft forever

Symptom :

After adding a set of rules, some of the rules doesn't work. And I can see with tcpdump that UDP packets are no more routed and packets are rejected.

tcpdump -n -i eth1 host 10.0.0.2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
16:12:58.241225 IP 10.0.0.2.56859 > 10.0.0.1.9999: UDP, length 1464
16:12:58.241285 IP 10.0.0.1 > 10.0.0.2: ICMP 10.0.0.1 udp port 9999 unreachable, length 556
  • If I flush all the rules and reinject them in iptables, rules that was not working still not work.
  • If I reboot my server all the rules work fine.

Analysis done :

I have added a rule to log a specific sender which is not working :

iptables -t nat -A PREROUTING -s 10.0.0.2 -i eth1 -j LOG --log-prefix='PREROUTING LOG :'

But this rule doesn't log anything. Packets are coming because I see those in tcpdump but they are not logged. Also with the -v option in iptables, I don't see counters increasing for this rule.

If I apply the same rule before it stop working, I have some logs.

Question :

  • Is there any limits on UDP forwarding in iptables ?
  • How I can troubleshoot this issue ?

Best Answer

The symptoms you describe match those seen when there is a conflict between a NAT rule and a connection tracking entry.

For example when a packet is matched by

-A PREROUTING -s 10.0.0.2 -i eth1 -j DNAT --to-destination 10.0.1.123

a new connection tracking entry need to be created. This will map a tuple of source and destination IP and port on the incoming side to a similar tuple on the outgoing side.

There cannot be an existing connection tracking entry matching the incoming side, because if there was it would have been used instead of the rule. However once the destination IP of the tuple has been replaced to construct the tuple for the outgoing side, the tuple may conflict with an existing connection tracking entry.

If you install the conntrack utility, you can type conntrack -L to see a list of existing connection tracking entries. That utility also has features to list only connection tracking entries matching specific criteria as well as remove selected entries.

If this is indeed the problem you are facing, then removing the offending connection tracking entry will make the problem go away. A permanent fix usually involves configuring relevant NAT rules for packets in both directions, such that you always get the desired connection tracking entry, even if the first packet happens to be send in the opposite direction than is usually the case.

Related Topic