Nginx + Jetty – thousands of connections stuck in LAST_ACK

freebsdjettynginx

I have a FreeBSD machine with jails — two in particular, one that runs nginx and another that runs a Java program that accepts requests via Jetty (embedded mode)

Jetty receives upwards of 500 requests/sec constantly and there has been an issue lately where I will constantly have over 60,000 connections in the LAST_ACK state between nginx and jetty.

Distribution of all connections (includes some other services, particularly php-fpm)

root@host:/root # netstat -an > conns.txt
root@host:/root # cat conns.txt | awk '{print $6}' | sort | uniq -c | sort -n
18 LISTEN
112 CLOSING
485 ESTABLISHED
650 FIN_WAIT_2
1425 FIN_WAIT_1
3301 TIME_WAIT
64215 LAST_ACK

Distribution of nginx -> jetty connections

root@host:/root # cat conns.txt | grep '10.10.1.57' | awk '{print $6}' | sort | uniq -c | sort -n
1 
3 CLOSE_WAIT
3 LISTEN
18 FIN_WAIT_2
125 ESTABLISHED
64193 LAST_ACK

I'd prefer every request to fully close the connection. Clients requests are about 10 minutes apart from each other so connections must be closed.

Some of the connections,

tcp4       0      0 10.10.1.50.46809       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46805       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46797       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46794       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46790       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46789       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46771       10.10.1.57.9050        LAST_ACK
etc..
  • On Jetty's end I've set maxIdleTime to 2000 — before this all connections were in ESTABLISHED but they are now LAST_ACK
  • On Jetty's end I've set Connection: close (i.e response.setHeader(HttpHeaders.CONNECTION, HttpHeaderValues.CLOSE);)
  • Jetty never reports a lot of open connections — always very few.
  • PF/IPFW is not currently being used
  • nginx – reset_timedout_connection is on

I cannot figure out how to get nginx or jetty to forcibly close the connection, is this simply something that needs to be fixed in Jetty so that it fully closes the socket after the request finishes?

Thanks a lot in advance

EDIT: forgot my nginx config for the proxy setup-

    proxy_pass              http://10.10.1.57:9050;
    proxy_set_header        HTTP_X_GEOIP $http_x_geoip;
    proxy_set_header        GEOIP_COUNTRY_CODE $geoip_country_code;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        Host $http_host;
    proxy_set_header        Connection "";
    proxy_http_version      1.1;

EDIT2: Forcing Jetty to close the connection via request.getConnection().getEndPoint().close() does nothing — it's obvious the connection IS being closed (as it's in LAST_ACK) but why isn't it getting past this? Is Nginx keeping the connection open to the backend for some reason?

Best Answer

OK well I was finally able to convince Jetty to play nicely.

Just to recap, here's what my connections look like now, machine-wide:

24 LAST_ACK
36 CLOSING
117 FIN_WAIT_2
175 ESTABLISHED
351 FIN_WAIT_1
4725 TIME_WAIT

And between nginx and Jetty

1 FIN_WAIT_2
3 LISTEN
14 ESTABLISHED

I've already been closing the entire connection and EndPoint (calling close() on the EndPoint closes the underlying SocketChannel) using this convenience method:

private void finishRequest(String message, Request baseRequest, HttpServletResponse response) throws IOException {
    ByteArrayISO8859Writer writer = new ByteArrayISO8859Writer(1500);
    writer.write(message);
    writer.flush();

    // set the content length
    response.setContentLength(writer.size());

    // write the response
    OutputStream outputStream = response.getOutputStream();
    writer.writeTo(outputStream);

    // close the streams
    outputStream.close();
    writer.close();
    baseRequest.getConnection().getEndPoint().close();
}

( I only send max one line to the client, so nothing fancy is needed )

However this still resulted in filling the entire server with LAST_ACK ... What made these finally disappear was enabling SO_LINGER (and making it forcefully close the socket immediately, with no timeout)

connector.setSoLingerTime(0);
Related Topic