Nginx + PHP-FPM = “Random” 502 Bad Gateway

502-errorfastcginginxphp-fpm

I am running Nginx and proxying php requests via FastCGI to PHP-FPM for processing. I will randomly receive 502 Bad Gateway error pages – I can reproduce this issue by clicking around my PHP websites very rapidly/refreshing a page for a minute or two. When I get the 502 error page all I have to do is refresh the browser and the page refreshes properly.

Here is my setup:

nginx/0.7.64 PHP 5.3.2 (fpm-fcgi)
(built: Apr 1 2010 06:42:04) Ubuntu
9.10 (Latest 2.6 Paravirt)

I compiled PHP-FPM using this ./configure directive

./configure –enable-fpm –sysconfdir=/etc/php5/conf.d –with-config-file-path=/etc/php5/conf.d/php.ini –with-zlib –with-openssl –enable-zip –enable-exif –enable-ftp –enable-mbstring –enable-mbregex –enable-soap –enable-sockets –disable-cgi –with-curl –with-curlwrappers –with-gd –with-mcrypt –enable-memcache –with-mhash –with-jpeg-dir=/usr/local/lib –with-mysql=/usr/bin/mysql –with-mysqli=/usr/bin/mysql_config –enable-pdo –with-pdo-mysql=/usr/bin/mysql –with-pdo-sqlite –with-pspell –with-snmp –with-sqlite –with-tidy –with-xmlrpc –with-xsl

My php-fpm.conf looks like this (the relevant parts):

 ...   
<value name="pm">
     <value name="max_children">3</value>
     ...
     <value name="request_terminate_timeout">60s</value>
       <value name="request_slowlog_timeout">30s</value>
       <value name="slowlog">/var/log/php-fpm.log.slow</value>
       <value name="rlimit_files">1024</value>
       <value name="rlimit_core">0</value>
       <value name="chroot"></value>
       <value name="chdir"></value>
       <value name="catch_workers_output">yes</value>
       <value name="max_requests">500</value>
...

I've tried increasing the max_children to 10 and it makes no difference. I've also tried setting it to 'dynamic' and setting max_children to 50, and start_server to '5' without any difference.

I have tried using both 1 and 5 nginx worker processes.

My fastcgi_params conf looks like:

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;
fastcgi_param  REDIRECT_STATUS    200;

Nginx logs the error as:

[error] 3947#0: *10530 connect()
failed (111: Connection refused) while
connecting to upstream, client:
68.40.xxx.xxx, server: www.domain.com, request: "GET /favicon.ico HTTP/1.1",
upstream: "fastcgi://127.0.0.1:9000",
host: "www.domain.com"

PHP-FPM logs the follow at the time of the error:

[NOTICE] pid 17161, fpm_unix_init_main(), line 255: getrlimit(nofile): max:1024, cur:1024 
[NOTICE] pid 17161, fpm_event_init_main(), line 93: libevent: using epoll 
[NOTICE] pid 17161, fpm_init(), line 50: fpm is    running, pid 17161 
[DEBUG] pid 17161, fpm_children_make(), line 403: [pool default] child 17162 started 
[DEBUG] pid 17161, fpm_children_make(), line   403: [pool default] child 17163    started 
[DEBUG] pid 17161,  fpm_children_make(), line 403: [pool default] child 17164 started 
[NOTICE] pid 17161, fpm_event_loop(), line 111: ready to handle connections

My CPU usage maxes out around 10-15% when I recreate the issue.
My Free mem (free -m) is > 130MB

I had this intermittent 502 Bad Gateway issue when in was using php5-cgi to service my php requests as well. Does anyone know how to fix this?

EDIT/UPDATE: I am using supervisor to launch php-fpm (not deamonized because of this).

Best Answer

I think your max_children is too low

<value name="max_children">3</value>

Also you can try switching from TCP port to unix socket for php-fpm:

php-fpm.conf

<value name="listen_address">/tmp/php.sock</value>

nginx.conf

fastcgi_pass unix:/tmp/php.sock;

Elaborated:

  1. Unix sockets are ~20% faster
  2. You don't use time-wait sockets for each connect
Related Topic