Nginx – How to create a non-caching fast nginx reverse-proxy

Nginx should be fast – at least people say so.

I can't get nginx fast. For benchmarking, i use siege in multiple configurations for simulating high loads. So i tried to load-balance a single server (i know load balancing a single server is rather useless, but for testing the load-balancer itself this is valid). I've set up an already optimized apache reverse proxy as reference. I get around 80tps via nginx and around 350tps using apache when i try getting the same – not cached – file from the same backend server. Of course the hardware/os is the same (current dual-core cpu, 2G ram, Ubuntu Server 16.04).

I tried already changing workers, maximum connections, poll-methods, buffer sizes for proxying, waiting times and buffers for clients. I can see a low load on the system, one nginx uses around 1 percent of the CPU and connections will wait sometime around 5 to 6 seconds. As i want to measure the performance of the reverse proxy i do not want to cache anything in this test.

So the question is: How to optimize the Performance of nginx as non-caching reverse proxy?

Example siege-command: siege -c 100 -b -r 100 -v <loadbalancer>/favicon.ico

Update: A bit of config, as requested. For keeping the question serverfault-conform, please answer generally about useful about parameters.

user www-data;
worker_processes 2;
pid /run/nginx.pid;
#thread_pool default threads=1500 max_queue=65536;
worker_rlimit_nofile 40000;

events {
    worker_connections 768;
    #multi_accept on;
    #use epoll;
}

http {

    #aio threads=default;
    #proxy_buffers           32 4m;
    #proxy_busy_buffers_size     25m;
    #proxy_buffer_size 512k;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    # server_tokens off;

    # server_names_hash_bucket_size 64;
    # server_name_in_redirect off;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    access_log /var/log/nginx/access.log combined buffer=1k;
    error_log /var/log/nginx/error.log;

    gzip off;
    # gzip_disable "msie6";

    # gzip_vary on;
    # gzip_proxied any;
    # gzip_comp_level 6;
    # gzip_buffers 16 8k;
    # gzip_http_version 1.1;
    # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    upstream kronos-backend {
        server kronos.example.com;
    }

    server {
        listen 80;
        listen [::]:80;

        ssl_certificate     /etc/ssl/certs/snakeoil.crt;
        ssl_certificate_key /etc/ssl/private/snakeoil.key;

        listen 443 ssl;
        listen [::]:443 ssl;

        keepalive_timeout 60;

        root /var/www/vhosts/default;

        index index.html;

        server_name <name>;

        server_tokens off; # Don't show that nginx is bringing out the website.

        location / {
            proxy_set_header Host            <name>;
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_pass http://kronos-backend;

            client_body_buffer_size 1M; # Lets keep all client-sent bodies in memory if less than 1 Megabyte. (Default for amd64: 16K)
            client_max_body_size 10M; # Maximum accepted body is 10M (Default is 1M).
        }
    }
}

This config is not very clean, i know. But it shows more or less what i already tried. It is derived from the default configuration from the nginx-package in Ubuntu.

The connection takes the following way:

       +--- apache2 ---+
       |               |
siege -+               +--> some webserver (not of interest)
       |               |
       +--- nginx -----+

So the data transmitted via nginx or apache2 does not matter as long there is no cache involved and the same url (in this case a static file) is used for testing. The caching mechanism was not activated intentionally at any of the webservers.

The config in apache uses mpm_worker.

ThreadLimit          600
StartServers         4
ServerLimit          11
MinSpareThreads      50
MaxSpareThreads      100
ThreadsPerChild      200
MaxClients           1000
MaxRequestsPerChild  20000

Example outputs of the above siege command:

$ siege -c 100 -b -r 10 <nginx>/wrapper_2_bg.png
** SIEGE 3.0.5
** Preparing 100 concurrent users for battle.
The server is now under siege..      done.

Transactions:                   1000 hits
Availability:                 100.00 %
Elapsed time:                  11.01 secs
Data transferred:               0.12 MB
Response time:                  0.84 secs
Transaction rate:              90.83 trans/sec
Throughput:                     0.01 MB/sec
Concurrency:                   76.24
Successful transactions:        1000
Failed transactions:               0
Longest transaction:            6.67
Shortest transaction:           0.03


$ siege -c 100 -b -r 10 <apache>/wrapper_2_bg.png
** SIEGE 3.0.5
** Preparing 100 concurrent users for battle.
The server is now under siege..      done.

Transactions:                   1000 hits
Availability:                 100.00 %
Elapsed time:                   3.16 secs
Data transferred:               0.12 MB
Response time:                  0.22 secs
Transaction rate:             316.46 trans/sec
Throughput:                     0.04 MB/sec
Concurrency:                   68.95
Successful transactions:        1000
Failed transactions:               0
Longest transaction:            3.06
Shortest transaction:           0.05