Nginx and buffering of big responses

nginx

In our application's configuration, nginx acts as a reverse proxy in front of gunicorn.

Our application replies to frontend requests with generally speaking, small responses… but some endpoints generate responses that are larger than one memory page (4K).

When this happens, nginx logs this warning:

an upstream response is buffered to a temporary file 
/path/to/nginx/proxy_temp/4/86/0000000864 while reading upstream, 
client: 1.2.3.4, server: api.ourdomain.com, request: "GET /pdf/..."

Our nginx logs end up flooded with this warning – and as far as I can see, the only solutions to make this warning vanish from our logs, are bad solutions:

  • I can set nginx proxy_max_temp_file_size to 0 – basically disabling the buffering of large-ish responses. This would stop the buffering to files – but it would also mean that for endpoints generating large responses (e.g. PDF-generating ones that create 1-2MB responses) a slow consuming client would stall the corresponding gunicorn worker… In fact, if there are N gunicorn workers, it would only take N clients generating PDFs behind slow network connections, and our application would be down…

  • I can increase the proxy_buffer_size to more than 4K (one memory page, that is). I am pretty sure this would have a severe impact in nginx performance – 70% of our responses indeed fit in 4K, and we would force nginx to allocate … what? 2MB of buffers for each one of them, just to be ready for the occasional PDF-generating request? EDIT: In fact this is not an option at all – Michael (below) commented that the only allowed values are 4K and 8K.

  • I could turn proxy_buffering off – but that is just as bad as the first solution (slow clients => death).

In essence, nginx floods our logs about something that we WANT – temporary buffering to files when the responses are big.

We just want to stop this from flooding our logs, by "marking" it as not really a warning (I am not even sure why this is a warning – what "bad thing" is it warning us about?)

Other than editing nginx's source code and recompiling, is there any other solution that I am missing?

Best Answer

This is a result of the log level on Nginx's error_log directive.

https://www.nginx.com/resources/admin-guide/logging-and-monitoring/

error_log logs/error.log warn;

messages of warn, error crit, alert, and emerg levels are logged.

The default setting of the error log works globally. To override it, place the error_log directive in the main (top-level) configuration context. Settings in the main context are always inherited by other configuration levels. The error_log directive can be also specified at the http, stream, server and location levels and overrides the setting inherited from the higher levels.

That line about buffering is at the warn level:

[warn] 30055#0: *1428 an upstream response is buffered to a temporary file

So if you ensure your error log level isn't set to warn, i.e. leave it at the default or decrease it, then you won't see that warning anymore. Either of the solutions below will suppress them:

Leave it at the default (level 'error' and above):

error_log logs/error.log;

Same thing:

error_log logs/error.log error;

You can also increase the threshold beyond error, up to crit, alert, and emerg.

Related Topic