Nginx – Resource temporarily unavailable while connecting to upstream (php5-fpm)

502-errornginxphp-fpmsocket

I frequently get this error on random pages when the are like ~+10K connections (netstat -an |grep 80 |wc -l). This happens while there is still +10GB of free memory and server load is less than 3.

My relevant configs:

/etc/php5/fpm/php.ini

memory_limit = 1024M
default_socket_timeout = 120

/etc/php5/fpm/pool.d/www.conf

pm = dynamic 
pm.max_children = 30 
pm.start_servers = 5
pm.min_spare_servers = 2 
pm.max_spare_servers = 5 
;pm.max_requests =20000 #uncommented with any value didn't fixed
request_terminate_timeout = 120s

I also doubled the above vlues (i.e 60 children, 10 start, 10 max servers) with no avail.

/etc/nginx/nginx.conf

worker_connections 4024;
keepalive_timeout 10;

Nginx itself behind Varnish and memcached is enable. page rendering is pretty fast in general.

This seems to be a quite common problem you I could not find any helpful solution or even analysis of the possible causes. So your hints/solutions are greatly appreciated.

Best Answer

Unfortunately it is difficult to give definite answer to this question without more details.

I have a couple of guesses though:

My first guess is that your php5-fpm worker socket(s) have too small listen backlog for your load. The backlog queue length is 128 by default on Linux.

To increase it you should:

  1. Increase net.core.somaxconn sysctl value.
  2. Increase listen.backlog php5-fpm configuration setting.

You could try increasing both to 2048 for example to see if that helps.

My second guess is the per process maximum open file descriptors limit of the nginx workers. The proper way to increase it depends on your distribution. On Ubuntu/Debian you should have a look in /etc/default/nginx file and worker_rlimit_nofile nginx configuration setting.

Related Topic