Nginx – Disable HTTP/2 Connection Reuse Across Domains

What is the possible explanation for this case?

A race condition between a ~5 seconds idle counter on the origin and variably client-side activity. The third involved variable is of course the network latency.

There seems to be a ~5 seconds idle timer on the origin, while your client is taking ~5 seconds to make a second request (POST) through the Nginx proxy. If the former is longer than the latter (including network latency) you have no problems. If it takes just a teeny bit more for the client request to be sent then you have a problem.

You can see how both the POST and the FIN,ACK from the Nginx are sent pretty much together: 2.4ms and 2.6ms after the origin's FIN,ACK respectively. This could've put you off track here, because I don't think the POST is a response to the origin's FIN,ACK at all. As it is sent 2.4ms after the origin's FIN,ACK

Why is the proxy (nginx) sending another request if it already received a FIN and is actually acknowledging it! (the ack number in the POST [PSH, ACK] packet is actually SEQ_NUMBER + 1 of the [FIN,ACK] - so it's acknowledging the phantom bit FIN.

The ACK number on the POST packet is for the "200 OK" packet most probably. There's no extra data coming from the server side after that HTTP response, so any ACK from the client will be ACKing the same number.

Update: We now know that the POST packet has the ACK number increased in 1, so Nginx knew about the [FIN,ACK]. Further investigation shows this is fine: A machine might send a request and end with a [FIN,ACK] if it's not planning to continue the connection after it receives the response from the remote side, who was to send the requested data back and end with continuing the [FIN,ACK] process.

This doesn't change the fact that there's a race condition where the origin decided to close the connection after 5 seconds idle, thus ignoring the POST packet that comes shortly after (and even sending a RST back - although it's not clear if this RST would've been sent regardless).

What are the possible reasons for the origin returning a [FIN,ACK] only after 5 seconds and not immediatelly? Read-timeout / idle-timeout?

You don't have to return a FIN,ACK immediately, especially since HTTP 1.1 and the introduction of persistent connections. These ~5 seconds seem to be an idle timer on the origin.

Both things are confirmed here: https://en.wikipedia.org/wiki/HTTP_persistent_connection - including a default 5 seconds idle timeout in Apache 2.2 or newer.

Suggested solutions

I can't really suggest a solution without knowing more about your infrastructure, but in rough terms you have a few options:

• Investigate why the client takes 5 seconds to send a second request. Drawbacks: Time consuming and probably implies application changes.
• Increase origin's (Apache?) timeout to maybe 10 seconds. Drawbacks: Problems to scale as you're keeping more resources idle. Might need application changes to dispose of the connections as soon as possible.
• Don't reuse the TCP connection for a second HTTP request, by issuing the "Connection: Close" header. Drawbacks: Higher cost per request as you have to establish a new TCP session. Might entail application changes to issue the header on all requests or changes on Nginx, thus deviating from your default configuration (increased admin costs).
• Use the "keepalive" option on Nginx inside your upstream configuration to set a keepalive lower than 5 seconds. Drawback: Lots of extra traffic/noise.

Hope this helps :)

I'd be surprised if the standards explicitly described and blessed such an operation (because it's vanishingly rare you can be sure the partial request you have received contains enough information to start producing a response), but I expect it would work in most cases. Any data you send back in response almost certainly won't be read by the client until its finished sending the request, so it'll sit there in the read buffer until the client starts reading, at which point it'll think, "blimey that was fast!" and start chewing away merrily. It won't realise that the data was ALREADY INSIDE THE BUILDING...

I'm sure there's some odd-ball HTTP client implementation that does weird things with its sending loop such that if it finds the socket to be readable before its finished writing the request it freaks out, but the only way to find those is to try it out and see what breaks. That is, pretty much, how everything is done in HTTP-land anyway -- and how new standard behaviours get made (someone does something insane, which becomes popular enough that everyone else has to adapt, so the IETF WG says "fine, we'll make that behaviour the standard, then").