Nginx – Why does nginx reverse proxy incoming bandwidth increase and surpass outgoing by far when proxy_buffering is enabled

I'm going to focus on slow client behavior, and how your configuration handles it, but don't be tempted to believe that is the only benefit. The same method that benefits slow clients also has benefits for fast clients, SSL handling, dealing with traffic surges, and other aspects of serving HTTP on the Internet.

Gunicorn is pre-forking software. For low latency communications, such as load balancer to app server or communications between services, pre-fork systems can be very successful. There is no cost in spinning up a process to handle the request, and a single process can be dedicated to handling a single request; the elimination of these things can lead to an overall faster, more efficient system until the number of simultaneous connections exceeds the number of available processes to handle them.

In your situation, you are dealing with high latency clients over the internet. These slow clients can tie up those same processes. When QPS matters, the application code needs to receive, handle, and resolve the request as quickly as possible so it can move on to another request. When slow clients communicate directly with your system, they tie up that process and slow it down. Instead of handling and disposing of the request as quickly as possible, that process now also has to wait around for the slow client. Effective QPS goes down.

Handling large numbers of connections with very little cpu and memory cost is what asynchronous servers like Nginx are good at. They aren't affected in the same negative manner by slow clients because they are adept at handling large numbers of clients simultaneously. In Nginx's case, running on modern hardware it can handle tens of thousands of connections at once.

Nginx in front of a pre-forking server is a great combination. Nginx handles communications with clients, and doesn't suffer a penalty for handling slow clients. It sends requests to the backend as fast as the backend can handle those requests, enabling the backend to be as efficient with server resources as possible. The backend returns the result as soon as it calculates it, and Nginx buffers that response to feed it to slow clients at their own pace. Meanwhile, the backend can move on to handling another request even as the slow client is still receiving the result.

Turn off port_in_redirect on the backend server.