Nginx – Server running at high CPU usage (100%)

centosnginxphp-fpmWordpress

I have a nginx server on a digital ocean droplet, with 2 CPU's and 4gb mem.

I'm running a couple of minor WP-sites, without much traffic – but it seems like i can push the server to 100% CPU without much effort. I can actually just spam(1-2/sec) Hard Refresh, and the server hits 100% and throws a error 500.
I'm still very new to server management and Nginx, so i have tried to debug to the best of my knowledge – and i keep coming back to my config's not being enough good.

Server Information:

2 x CPU
4GB Memory
CentOS 7
VestaCP
Pure Nginx
Running WP sites only

Nginx conf:

# Server globals
 user                    nginx;
 worker_processes        auto;
 worker_rlimit_nofile    65535;
 error_log               /var/log/nginx/error.log crit;
 pid                     /var/run/nginx.pid;


 # Worker config
      events {
         worker_connections  1024;
         use                 epoll;
    multi_accept        on;
 }


 http {
# Main settings
sendfile                        on;
tcp_nopush                      on;
tcp_nodelay                     on;
client_header_timeout           3m;
client_body_timeout             3m;
send_timeout                    3m;
client_header_buffer_size       1k;
client_body_buffer_size         128k;
client_max_body_size            10m;
output_buffers                  1 32k;
postpone_output                 1460;
large_client_header_buffers     4   4k;
keepalive_timeout               30 30;
reset_timedout_connection       on;
server_tokens                   off;
server_name_in_redirect         off;
server_names_hash_max_size      512;
server_names_hash_bucket_size   512;


# Log format
log_format  main    '$remote_addr - $remote_user      [$time_local] $request '
                    '"$status" $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
log_format  bytes   '$body_bytes_sent';
#access_log          /var/log/nginx/access.log main;
access_log off;


# Mime settings
include             /etc/nginx/mime.types;
default_type        application/octet-stream;


# Compression
gzip                on;
gzip_static         on;
gzip_comp_level     2;
gzip_min_length     1000;
gzip_buffers        8 64k;
gzip_types          text/plain text/css text/javascript text/js text/xml application/json application/javascript application/x-javascript application/xml application/xml+rss application/x-font-ttf image/svg+xml font/opentype;
gzip_proxied        any;
gzip_disable        "MSIE [1-6]\.";


# Proxy settings
proxy_redirect      off;
proxy_set_header    Host            $host;
proxy_set_header    X-Real-IP       $remote_addr;
proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass_header   Set-Cookie;
proxy_connect_timeout   90;
proxy_send_timeout  90;
proxy_read_timeout  90;
proxy_buffers       32 4k;


# Cloudflare https://www.cloudflare.com/ips
set_real_ip_from   199.27.128.0/21;
set_real_ip_from   173.245.48.0/20;
set_real_ip_from   103.21.244.0/22;
set_real_ip_from   103.22.200.0/22;
set_real_ip_from   103.31.4.0/22;
set_real_ip_from   141.101.64.0/18;
set_real_ip_from   108.162.192.0/18;
set_real_ip_from   190.93.240.0/20;
set_real_ip_from   188.114.96.0/20;  
set_real_ip_from   197.234.240.0/22;
set_real_ip_from   198.41.128.0/17;
set_real_ip_from   162.158.0.0/15;
set_real_ip_from   104.16.0.0/12;
set_real_ip_from   172.64.0.0/13;
#set_real_ip_from   2400:cb00::/32;
#set_real_ip_from   2606:4700::/32;
#set_real_ip_from   2803:f800::/32;
#set_real_ip_from   2405:b500::/32;
#set_real_ip_from   2405:8100::/32;
real_ip_header     CF-Connecting-IP;


# SSL PCI Compliance
ssl_session_cache   shared:SSL:10m;
ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers        "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4";


# Error pages
error_page          403          /error/403.html;
error_page          404          /error/404.html;
error_page          502 503 504  /error/50x.html;


# Cache settings
proxy_cache_path /var/cache/nginx levels=2 keys_zone=cache:10m inactive=60m max_size=1024m;
proxy_cache_key "$host$request_uri $cookie_user";
proxy_temp_path  /var/cache/nginx/temp;
proxy_ignore_headers Expires Cache-Control;
proxy_cache_use_stale error timeout invalid_header http_502;
proxy_cache_valid any 1d;


# Cache bypass
map $http_cookie $no_cache {
    default 0;
    ~SESS 1;
    ~wordpress_logged_in 1;
}


# File cache settings
open_file_cache          max=10000 inactive=20s;
open_file_cache_valid    30s;
open_file_cache_min_uses 5;
open_file_cache_errors   off;


# Wildcard include
include             /etc/nginx/conf.d/*.conf;

}

PHP-fpm:

pm = ondemand
pm.process_idle_timeout = 10s
pm.max_children = 3
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 2
pm.max_requests = 300

 php_admin_value[memory_limit] = 256M

 env[HOSTNAME] = $HOSTNAME
 env[PATH] = /usr/local/bin:/usr/bin:/bin
 env[TMP] = /tmp
 env[TMPDIR] = /tmp
 env[TEMP] = /tmp

Top Screenshot – imgur.com/a/n02VZ

Last entries in Nginx error-log

Last entries in php error-log

Best Answer

The Problem

This is very unlikely to be a problem, but we don't know because you haven't included information like top/atop. This is because Wordpress / PHP is inefficient and take a lot of server resources. You could post a screenshot of top / atop when the server is busy if you like, but if it's all taken by PHP there's no need.

The Solution

One solution, probably be best solution, is caching. There are two types of caching.

Page Caching

Page caching is usually the best type in this situation. It's only applicable for anonymous users, not users who have logged in. On many sites anonymous users make up most of the traffic so caching them can have a massive benefit. A page is generated by a user, stored in the Nginx page cache, then given to the next user that comes along. The Nginx page cache is often in RAM so it's super fast.

Caching Headers

For page caching to work you need to set your headers correctly. This can be done using some Wordpress caching plugins, but I don't use them so I do it in Nginx. I had to build Nginx with the right modules added, which is pretty simple.

Wordpress Caching

Another type of caching is done by plugins within Wordpress. This can cache the database requests, which take time, and also does some kind of object caching that I don't fully understand. This can help both anonymous and logged in users. Sometimes it causes problems.

Example

I have a tutorial on this, here. It covers caching, building Nginx, and a bunch of other things like using a Content Distribution Network (CDN) which can also speed things up. Page one has downloadable config files. Nginx has an article on microcaching which is a good read.

CDN:

Speaking of CDNs, if you have a super high scale site, you can configure your pages to be cacheable and configure the CDN to cache the pages. That means your site rarely sees requests, only for logged in users and cache refreshes. That affects your server statistics, and increases the chance of serving users stale data.

Nginx Bug

Nginx 1.11.11 has a bug that it sometimes uses 100% of CPU. Nginx 1.11.12 fixes this.

Related Solutions

Nginx – Help needed setting up nginx to serve static files

I looked at the nginx error logs and found that it was trying to serve the media from /var/django/myproject/site_media/site_media instead of /var/django/myproject/site_media --weird.

I changed root /var/django/myproject/site_media; to root /var/django/myproject; and it works now.

Nginx – Configure php5-fpm for many concurrent users

I suspect your varnish cache is not caching anywhere near enough of the hits

here's what I would do in your situation:

Lower php max children to 100 or even 50 (if varnish does its job properly you don't need them) also remove the max requests line to allow the php processes not to respawn too quickly and thus prevent APC from being cleared too quickly which is also bad

also IF is not good according to nginx - http://wiki.nginx.org/IfIsEvil

I would change this line:

            if (!-e $request_filename) {
                    rewrite ^(.+)$ /index.php?q=$1 last;
            }

to:

try_files $uri $uri/ /index.php?$args;

If your version of nginx supports it (pretty certain if your nginx version is > 0.7.51 then it supports it)

you should also look at inserting the w3tc nginx rules direct into your vhost file to enabled proper disk enhanced caching of pages (which is faster than APC caching with nginx)

Take a look at the following varnish vcl which I use for sites - you will need to read through and edit a few things for your website - it also assumes that its only WP sites on the server and only 1 site on the server, it can easily be modified for more sites (take a look at the cookie section)

generic vcl: https://gist.github.com/b7332971a848bcb7ecef

With this config I would argue to remove fastcgi_cache to prevent any possible issues with a cache-chain occurring whereby trying locate any stray stale cache entries is more difficult

also tell w3tc that varnish is at 127.0.0.1 and it will purge it for you ;)

I deployed this to a server on Wednesday evening (with a few domain specific modifications) that was handling 2500 active site visitors it reduced load to less than 1 and the approx number of running php children was around 10-20 (this number does depend on number of logged in users and other factors like cookies) this server did have much more ram but the principle is the same, you should be able to easily handle the number of visitors you get at peaks

Best Answer

Related Solutions

Nginx – Help needed setting up nginx to serve static files

Nginx – Configure php5-fpm for many concurrent users

Related Topic