iptables – Tcpdump Shows Different Redirection Port After Adding REDIRECT Rule

To be clear: OP's test is done from the system 192.168.1.100 to itself, not from a remote system, and that's the cause of the problem. The port wasn't changed in this case because no NAT rule matched, while it would have matched if done from a remote system.

The schematic below shows how order of operations are performed on a packet:

The reason is how NAT works on Linux: iptables sees a packet in the nat table only for the first packet of a new conntrack flow (which is thus in NEW state).

This rule works fine when from a remote system. In this case the first packet seen will be an incoming packet:

to port 8000 --> AF_PACKET (tcpdump) --> conntrack --> nat/PREROUTING (iptables REDIRECT): to port 30000
--> routing decision --> ... --> local process receiving on port 30000

All following packets in the same flow will have conntrack handle directly the port change (or port reversion for replies) and will skip any iptables rule in the nat table (as written in the schematic: nat table only consulted for NEW connections). So, (skipping the reply packet part), the next incoming packet will undergo this instead:

to port 8000 --> AF_PACKET (tcpdump) --> conntrack: to port 30000
--> routing decision --> ... --> local process receiving on port 30000

For a test on the system to itself, the first packet isn't an incoming packet but an outgoing packet. This happens instead, using the outgoing lo interface:

local process client curl --> routing decision --> conntrack --> nat/OUTPUT (no rule here)
--> reroute check --> AF_PACKET (tcpdump) --> to port 8000

And now this packet is looped back on the lo interface, it reappears as a packet which isn't anymore the first packet in a connection so follows second case as above: conntrack alone takes care of the NAT and doesn't call nat/PREROUTING. Except it wasn't instructed in the step before to do any NAT:

to port 8000 --> AF_PACKET (tcpdump) --> conntrack
--> routing decision --> ... -->nolocal process receiving on port8000

as there's nothing listening on port 8000, the OS sends back a TCP RST.

For this to work on the local system, a REDIRECT rule must also be put in the nat/OUTPUT chain:

iptables -t nat -I OUTPUT -d 192.168.1.100 -p tcp --dport 8000 -j REDIRECT --to-port 30000

Additional notes

• if the case is intended for remote use, don't test from the local system: rules traversed by the test aren't the same. This makes the test not reflecting reality.

  Just use a network namespace to create a pocket remote system in case no other system is available. Example that should work with a system having only OP's nat/PREROUTING rule and doing curl http://192.168.1.100/ (which doesn't require DNS):

  ip netns add remotesystem
  ip link add name vethremote up type veth peer netns remotesystem name eth0
  ip address add 192.0.2.1/24 dev vethremote
  ip -n remotesystem address add 192.0.2.2/24 dev eth0
  ip -n remotesystem link set eth0 up
  ip -n remotesystem route add 192.168.1.100 via 192.0.2.1
  ip netns exec remotesystem curl http://192.168.1.100:8000/
  

• tcpdump and NAT

  tcpdump happens at the AF_PACKET steps in the schematic above: very early for ingress and very late for egress. That means for a remote system case, it will never capture the port 30000 even when it's working. For the local system case, once the nat/OUTPUT rule is added, it will capture port 30000.

  Just don't trust blindly the address/port displayed by tcpdump when doing NAT: it depends on the case and where the capture happens.