Linux – TCP packets not reaching app, iptables dropping

iptableslinuxnetworking

We are occasionally, but not consistently, experiencing a weird network stack issue. Rebooting the server in question clears it up.

It happens as follows (gleaned through tcpdump on the server):

HTTP client starts sending request to Nginx.
Server responds normally, acking every packet it gets.
On the final client send, the packet never reaches the receiving socket on the server.
The client resends the packet several times, then the server finally times out and disconnects.

Also, strace of Nginx confirms that the data is not reaching Nginx.

Here is an edited version of the tcpdump output. I have simplified the exchange and anonymized some details.

Turning on iptables logging shows some packets being blocked, which may be relevant:

IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=39670 DPT=80 WINDOW=0 RES=0x00 RST URGP=0 
IN= OUT=eth0 SRC=server DST=client LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=39669 WINDOW=31 RES=0x00 ACK URGP=0

However, our iptables setup is pedestrian. We block everything except RELATED,ESTABLISHED, and we allow the port in question, 80. I don't see why iptables is blocking this, unless the packets are somehow falling outside the states of RELATED and ESTABLISHED.

I have also included our sysctl settings in the above gist. Anything else I can look at?

Linux 3.8.0 on Ubuntu 12.04.3, on DigitalOcean.

Edit 3: Disabled iptables, same problem, so it's not caused by bad iptables rules.

Edit 2: Above I show iptables blocking RST packets, but more importantly it's blocking a lot of ACKs. I just picked a random log entry, ACK seems more common.

Edit 1: I added iptables tracing. This seems to the part that drops a packet (though, again, not sure if this is related to my problem):

TRACE: raw:OUTPUT:rule:2 IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=41572 DPT=8001 SEQ=2118637628 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 
TRACE: raw:OUTPUT:policy:3 IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=41572 DPT=8001 SEQ=2118637628 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 
TRACE: filter:OUTPUT:rule:3 IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=41572 DPT=8001 SEQ=2118637628 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 
TRACE: filter:block:rule:1 IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=41572 DPT=8001 SEQ=2118637628 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 
TRACE: filter:logging:rule:1 IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=41572 DPT=8001 SEQ=2118637628 ACK=0 WINDOW=0 RES=0x00 RST URGP=0 
iptables: reject: IN= OUT=lo SRC=client DST=server LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=41572 DPT=8001 WINDOW=0 RES=0x00 RST URGP=0

No idea why lo is involved here. Server is accepting traffic on eth0.

Best Answer

Your log definitely shows communication happening on the lo interface.

Disable iptables by changing INPUT table default policy to ACCEPT and deactivate any REJECT or DROP rule which might stand in the way
Test again and check that it works. If not, the problem comes from somewhere else

I would bet $1000 on the fact your filtering rules accepting traffic are bound to some eth0 interface, thus rejecting traffic incoming on lo.

I would pay attention to where the testing client in relation to the server. If you are running your test client on the same machine, it most probably either uses the 127.0.0.1 IP address or the localhost domain name which usually resolves to the same IP address.

That will send traffic on the special loopback interface (lo) and not on the eth0 one.

Unless you bound nginx to a specific interface by asking it to listen on its IP address, nginx will listen by default on 0.0.0.0, which is every interface. Thus, you won't notice if it accepts connections on lo or not. You could try to force nginx listening on your eth0 IP address just to be sure.

When locally testing your server, ensure you use your one of your external interfaces (eth[0..]) IP address, or a domain name resolving to one of them.

Related Solutions

Iptables drops some packets on port 80 and i don’t know the cause

I have seen a very similar issue on my own systems and I may have an answer as to what is happening.

Apparently, TCP allows the client to send a FIN packet, and you to ACK that FIN packet, but still keep your end of the connection open and push more data through it, sending your own FIN packet at some time in the future. I remember reading that this was implemented at some point in Kernel 2.6 although my memory on this point may be inaccurate/irrelevant.

Many firewalls, including IPTables, don't seem to have implemented this yet or implement it incorrectly and consider the connection closed as soon as they see a FIN followed by an ACK. The result of this is that every so often, my server sends a packet out to a client that the firewall thinks is not part of an established connection and is not new and so drops it.

In practice, I haven't seen any important data in those final few packets that are being dropped, so the effect is a few extra reset packets getting thrown around and some extra connections left in the FIN_WAIT state but no broken or half-loaded pages.

I don't know about the 20 minute timings you are seeing but I would guess that it's a client of some sort that runs regularly every 20 minutes and does a request that causes your server to do the FIN ACK PSH FIN RST sequence.

This post may offer more information about exactly why IPTables drops these packets: http://lists.netfilter.org/pipermail/netfilter/2005-August/062059.html

According to Chris Brenton, it's a mismatch between the timeouts of the client, the server and the firewall for packets in the one-side-closed state.

Iptables ESTABLISHED,RELATED chain problems

Netfilter has dropped the connection from the state table. One of the timeouts in /proc/sys/net/netfilter/nf_conntrack_tcp_timeout_* has expired. I'm guessing it is nf_conntrack_tcp_timeout_last_ack.

Here is a scenario: Your web server sends a final packet with some data and fin set. The client goes comatose for 30 seconds. Netfilter removes the entry from the state table. The client wakes up and sends a fin ack that is not part of a tracked connection anymore.

I see this all the time on web servers. It is client problem.

If you want to verify that you could record the state table (/proc/net/nf_conntrack) and correlate that with the firewall log.

Edit: I had the direction reversed but the concept applies. In this case his server is making outgoing https connections so it is actually the client and the remote web server may be slow to respond.

The rule

/sbin/iptables -A INPUT -p tcp -m state --state NEW --dport 443 -j ACCEPT

has nothing to do with it. It is about DST port 443 and this is SPT 443.

Best Answer

Related Solutions

Iptables drops some packets on port 80 and i don’t know the cause

Iptables ESTABLISHED,RELATED chain problems

Related Topic