Nginx – RHEL Nginx SSL versus non SSL performance huge difference.

httphttpsnginxredhatssl

I'm in the process of setting up an Nginx 1.8 reverse proxy.

In short –

Serving HTML content HTTP traffic is up to 50x faster than HTTPS.

Serving ProxyPass HTTP traffic is up to 7x faster than HTTPS.

OS is RHEL7

Hardware: 

2 core VMWare Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz
cpu MHz         : 1897.802
cache size      : 15360 KB
bogomips        : 3795.60
1 Gbit network card

Benchmarking client is Apache bench, 1 hop away, ping 1ms.
Apache bench uses the following TLS protocol when running:

TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256

Server SSL certificate 2048-bit RSA.
OCSP Stapling is on and verified.

/etc/sysctl.conf has

net.ipv4.ip_local_port_range=1024 65000
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_fin_timeout=15
net.core.netdev_max_backlog=4096
net.core.rmem_max=16777216
net.core.somaxconn=4096
net.core.wmem_max=16777216
net.ipv4.tcp_max_syn_backlog=20480
net.ipv4.tcp_max_tw_buckets=400000
net.ipv4.tcp_no_metrics_save=1
net.ipv4.tcp_rmem=4096 87380 16777216
net.ipv4.tcp_syn_retries=2
net.ipv4.tcp_synack_retries=2
net.ipv4.tcp_wmem=4096 65536 16777216
vm.min_free_kbytes=65536

/etc/security/limits.conf has

nginx   soft    nofile  65536
nginx   hard    nofile  65536

Nginx is configured with 

server {
  listen 443 ssl deferred backlog=1024;
  listen 80 deferred backlog=1024;

  server_name SERVERNAME;

  client_max_body_size 10m;

  ssl_stapling on;
  ssl_stapling_verify on;
  ssl_trusted_certificate path_to_/certificateAndChain.cer;
  ssl_certificate path_to_/certificateAndChain.cer;
  ssl_certificate_key path_to_/private.key;
  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_ciphers "EECDH+AES:EECDH+AESGCM:EDH+AESGCM:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES128-GCM-SHA256:AES128+EECDH:D$
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:32m;
  ssl_session_timeout 1m;

  #resolver 8.8.8.8 8.8.8.4 valid=1m;
  #resolver_timeout 5s;

  location / {
   proxy_pass_header Server;
   proxy_set_header Host $http_host;
   proxy_set_header X-Real-IP $remote_addr;
   proxy_set_header X-Forwarded-For $remote_addr;
   proxy_set_header X-Scheme $scheme;
   proxy_connect_timeout 43200000;
   proxy_read_timeout 43200000;
   proxy_send_timeout 43200000;
   proxy_buffering off;
   proxy_http_version 1.1;
   proxy_set_header Connection "";

   proxy_pass http://IPADDRESS/;

  }

  location /localtest {
    root /var/www/localtest;
    index index.html;
  }
}

Actual results:

Serving local HTML content HTTP

ab -c200 -n20000 http://SERVERNAME/localtest/index.html
Requests per second:    12751.64 [#/sec] (mean)
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4   2.3      4      11
Processing:     2   12   7.3      9      96
Waiting:        1   10   7.7      7      96
Total:          2   16   6.6     14     100

HTTPS:

Requests per second:    252.28 [#/sec] (mean)
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       12  651 288.1    694    1470
Processing:     0  141 134.4    101    1090
Waiting:        0  101 124.3     65    1089
Total:         15  792 276.7    809    1641

Proxying to Apache, 1ms ping, 1 hop away.

HTTP

Requests per second:    1584.88 [#/sec] (mean)
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   2.3      1       8
Processing:     4  141 309.6     30    1244
Waiting:        4  141 309.7     29    1244
Total:         10  143 310.3     31    1248

HTTPS

Requests per second:    215.99 [#/sec] (mean)
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       14 1131 622.3   1137    2030
Processing:     4  474 413.2    313    1814
Waiting:        1  399 405.6    257    1679
Total:         26 1605 769.6   1699    3306

Best Answer

Benchmarks are lies, don't reflect the reality but might be a useful tool to detect bottlenecks. But you have to understand the benchmarks. Given that you omit essential details needed to understand the benchmark results it might be that you don't really understand what might affect the results of the benchmark.

Especially information about the size of the test payload and detailed CPU load information for server and client are missing. Thus it might be that you are reaching CPU limits on the client or on the server already. It might also mainly be a problem of the more round trips you need for the requests. Let's explain the aspects of HTTP vs. HTTPS in more detail:

ab -c200 -n20000 http://SERVERNAME/localtest/index.html

You've configured to use 200 concurrent requests. The size of the request is unknown so we can assume that there will be only minimal payload. You are also using no HTTP keep alive which means that there will a new TCP connection for each request. I doubt that apache bench is doing TLS session resume so that there will a full handshake each time. Which gives you:

  • HTTP: 1 RTT for the TCP handshake and another RTT for a minimal HTTP request and response. This might also include the connection close already (implementation dependent). This means 2 RTT and minimal data transfer.

  • HTTPS adds on top of this:

    • 2 RTT for a full TLS handshake and probably also 1 RTT for the TLS shutdown. Just because of a total of 5 RTT for HTTPS vs. 2 RTT for plain HTTP you should see a large drop in performance, i.e. from about 13000 req/s to 5200 req/s (i.e. 2/5).
    • The transferred data for TLS handshake alone might even be larger than what you have as payload within your simple HTTP only request (Edit: based on the commend the size of your responses vary from 60 byte to 50kb so this is probably not that relevant).
    • But then you have also lots of computations for the TLS handshake, both on the client and on the server side. And more of this because you are using ECDHE, see https://securitypitfalls.wordpress.com/2014/10/06/rsa-and-ecdsa-performance/.

The computations during the TLS handshake need lots of CPU time and that's why it would have been important to provide information about the CPU load. It might be that you are simply hitting the maximum the CPU could do, either at the server or at the client. Please also note that apache bench is single threaded, so it would be enough to max out the performance of a single CPU core even if the others are idle. And even if you use multiple thread the computation still takes time. Using openssl speed does not reflect what is really done inside the TLS handshake and it also tests only the maximum speed with a single thread, not with multiple computations in parallel and all the cache trashing etc involved.

Thus while this might be an interesting benchmark to see what is possible it does not reflect reality in most cases. The fact is that TLS can reduce performance a lot, but with common HTTP traffic you will have larger requests, HTTP keep alive and TLS session reuse which all reduce the impact of the costly TLS handshake.

But if the benchmark is actually limited on the server performance and not on the client performance the setup might reflect servers used for tracking, where you might have only a small response (i.e. 1x1 pixel) from lots of different sites without any kind of TLS session reuse or HTTP keep alive.

Related Topic