Nginx very slow. Changed from Apache2 to Nginx

I recently changed my server from Apache to Nginx. That's because the website was using file_get_contents to another website. When that external website was down or slow, the threads were maxed out and the server went down. It's like a domino effect.

I then upgraded the server from Apache2 to Nginx together with a reverse proxy also running Nginx. This way, I can run multiple servers with multiple domains on 1 single IP Address.

Today I noticed the visitors were dropping hard, and acted quick. The server is very slow and even gets a few Timeouts.

I have 3 servers:

1. Reverse Proxy – Nginx (Main connection to internet)
2. 1st Website – Nginx (about 10,000 unique visitors a day)
3. 2nd website – Apache2 (Experimental website, very quick response)

When connecting to the 1st website, it is VERY slow (about 28 seconds wait time), but when connecting to the 2nd website, it's very quick (about 0.8 seconds wait time)

I have very little knowledge of Nginx at the time, as I have been using Apache2 for 2 years now.

Reverse Proxy – /etc/nginx/sites-available/default

Yes, I have a bunch of domains on that first one

server {
    listen 80;

    server_name domain.com www.domain.com domain.org www.domain.org domain.co.uk www.domain.co.uk domain.co www.domain.co domain.me www.domain.me domain.se www.domain.se;

    location / {
        proxy_pass http://192.168.1.237;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        real_ip_header X-Real-IP;
        real_ip_recursive on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        }
}
server {
    listen 80;

    server_name www.domain2.com domain2.com;

    location / {
        proxy_pass http://192.168.1.230;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
     }
}
server {
    listen 80;

    server_name controller.domain.com;

    location / {
        allow 84.71.x.x/21;
        allow 192.168.0.0/16;
        deny all;
        proxy_pass http://192.168.1.1;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;

        }
}

1st webserver – /etc/nginx/nginx.conf

user www-data;
worker_processes 10;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 1024;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # nginx-naxsi config
        ##
        # Uncomment it if you installed nginx-naxsi
        ##

        #include /etc/nginx/naxsi_core.rules;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

1st webserver – /etc/nginx/sites-available/default

server {
        listen 80 default_server;
        listen [::]:80 default_server;

        root /var/www/html/pi;
        index index.php;

        # It seems like the server only needed 1 of these, as all the other domains redirects to the main domain
        server_name domain.com domain.org;

        error_page 404 /missing.php;
        error_page 403 /forbidden.php;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
                # Uncomment to enable naxsi on this location
                # include /etc/nginx/naxsi.rules;
        }

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_param REMOTE_ADDR $http_x_real_ip;

                # With php5-cgi alone:
                #fastcgi_pass 127.0.0.1:9000;
                # With php5-fpm:
                fastcgi_pass unix:/var/run/php5-fpm.sock;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}

I'm not sure where the problem lies, so I linked a few config files.

I used some of this article (I did not change the ports, but simply stopped apache, installed nginx and then uninstalled apache) to move from Apache to Nginx.

All computers are running Debian.

There are currently around 4600 requests per hour which I counted with this command: sudo grep -o '13/Feb/2018:09:' /var/log/nginx/access.log | wc -l

Do you have any recommendations or things I could try to solve this?

EDIT

The problem seems to occur when under high load.

UPDATE

The server seems to be working fine now.
I found this link and used the config as an example. Then I removed/edited to match the infrastructure (Reverse proxy, custom parameters).

Here is a visual representation (Monitorix) of the server status for the last week/day. I added a description on the images.

I'll let the server keep running as the configuration is at the moment, and see if I come across any future errors.

UPDATE 2

It has almost been a week, and I was hoping the problem were sovlved…
The problem has come back.

Now it constantly throws this error:

2018/02/17 18:52:21 [error] 18185#18185: *2065 connect() to unix:/var/run/php5-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 66.249.x.x, server: domain.org, request: "GET /index.php HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "domain.org"

I also found out that the server got very fast by restarting php5-fpm, but only for about 10 seconds.

I think it is weird that the server has been running fine for a week, and now suddenly this problem occurred again?

RAM is chilling at 1 GB / 4 GB, and the only noticeable different when looking at monitorix (except all network activity dropping) is this. The bump at the end, is when the website is down, but I don't think it matters.

Any recommendations?

UPDATE 3

I have been holding a close eye on the server for the past week and it has been incredible unstable.
Sometimes i has been very fast, other times it could around 20 seconds to load in the webpage.

I changed the configuration of nginx.conf but only seemed to help a little bit. I started to think the PHP was the problem, not the nginx. I found this question which linked to an article. I changed a few pm's in www.conf to this:

pm.max_children = 20
pm.start_servers = 8
pm.min_spare_servers = 8
pm.max_spare_servers = 15
pm.max_requests = 500

In the last hour, the response time has been around 400 ms.

I'll wait a while and see if the problem occurs again.