Nginx cpu consumption for ssl termination with rtmp stream

nginxrtmpsslstreaming

Here is my nginx config file

user  nginx;
worker_processes  auto;


error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}


stream {
    server {
        listen *:443 ssl;

        ssl_certificate             /etc/nginx/ssl/server.crt;
        ssl_certificate_key         /etc/nginx/ssl/server.key;

        ssl_session_cache shared:SSL:20m;
        ssl_session_timeout 60m;

        proxy_pass 127.0.0.1:1935;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers on;
        ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DHE+AES128:!ADH:!AECDH:!MD5;
    }
}

I am not happy with it because nginx consumes too much cpu.
How to tune it properly?
Here I have rtmp listener at 127.0.0.1:1935. So nginx is only terminating ssl and passes by rtmp further.

This is how cpu consumption looks for 1cpu server. About 50% is eaten by nginx and the rest by rtmp server. I would like to make nginx consume less CPU.
cpu consumption

Nginx version

nginx -V
nginx version: nginx/1.15.2
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) 
built with OpenSSL 1.1.0f  25 May 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.15.2/debian/debuild-base/nginx-1.15.2=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

The load you see is provided by 500 simultaneous stream uploads by command

ffmpeg -re -stream_loop -1 -i /dolbycanyon.mp4 -acodec copy -vcodec copy -f flv rtmps://rtmp:443/live/live139

from another server.

I generated certificate files /etc/nginx/ssl/server.crt and /etc/nginx/ssl/server.key by command

openssl req -config ./openssl.conf -x509 -nodes -days 365 -newkey rsa:2048 -keyout selfsigned.key -out selfsigned.crt

where openssl.conf is

[req]
prompt = no
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
C = US
ST = California
L = Los Angeles
O = Our Company Llc
#OU = Org Unit Name
CN = Our Company Llc
#emailAddress = info@example.com

[v3_req]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = example.com
DNS.2 = www.example.com

Update 1

So when I use nginx ssl termination, it can stand maximum 512 streams for 1 CPU. When I stream directly to RTMP port 1935 (without nginx) it's 970 simultaneous streams. So I would like to optimize nginx to get a number a bit closer to 970 streams rather than 512 streams.

Update 2

Looks like the problem is not in ssl termination but in nginx forwarding itself.
When I use simple forwarding (without ssl):

user  nginx;
worker_processes  auto;


error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}


stream {
    server {
        listen *:1936;
        proxy_pass 127.0.0.1:1935;
    }
}

It gives 511 simultaneous streams. It's almost the same amount as when I used ssl: 512.
And I have the same cpu consumption ratio: about 50% for nginx and about 50% for rtmp server itself.
In this case, I upload stream to 1936 (without ssl):

ffmpeg -re -stream_loop -1 -i /dolbycanyon.mp4 -acodec copy -vcodec copy -f flv rtmp://rtmp:1936/live/live6

Update 3

1) I checked /etc/security/limits.conf file. It was empty. I added a couple lines to it:

web soft nofile 65535
web hard nofile 65535

2) Also, I added file ULIMIT="-n 65535" to file /etc/default/nginx

3) And added line worker_rlimit_nofile 65535; to /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;


error_log  /var/log/nginx/error.log error;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

worker_rlimit_nofile 65535;

stream {
    server {
        listen *:1936;
        proxy_pass 127.0.0.1:1935;
    }
}

But the result is still the same: 512 connections.
By the way, I've done 1) for both host and Docker container. I run all this stuff inside a container which is living inside gcloud compute instance.

Best Answer

512 is the default number of the worker_connections value, which defines the number of connections per process (in this case, 1CPUx512Connections). My thoughts on that is you might be hitting a default limit even though you've obviously increased the default worker_connections as defined in the docs around it. http://nginx.org/en/docs/ngx_core_module.html#worker_connections

Have you tried increasing a worker_rlimit_nofile value or checking that the /etc/security/limits.conf file has any limits in it?