Nginx – Cannot allocate memory error in nginx with huge number of redirections

nginx

I faced with memory allocation error in nginx. I have configured a reverse proxy for a number of sites on my nginx, that I use as a simple load balancer between two backend nodes. Typical config for the site looks like this: 

upstream backend  {
  ip_hash;
  server <node-ip>;
  server <another-node-ip>;
}

server {
  server_name domain.subdomain.com;    

  # a HUGE bunch of redirection rules 
  include /etc/nginx/sites-available/root;

  location / {

    proxy_pass  http://backend  ;

    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

I have 12 sites with configuration like above. As you see config contains include of another file – sites-available/root. This file consists of a number of unclude directives to another files:

include /etc/nginx/sites-available/rules1;
include /etc/nginx/sites-available/rules2;
include /etc/nginx/sites-available/rules3;
...
include /etc/nginx/sites-available/rules16;

Every rules file contains a number of redirection rules like:

if ($request_uri ~* ^/some-url$) {
  return 302 /some-another-url/;
}

or

location ~ some-url {
  return 302 "some-another-url";
}

The total count of redirection rules is around 2300. I included root file to configurations of all 12 sites. After that time after time I get info message in /var/log/nginx/error.log:

[info] 23721#23721: Using 32768KiB of shared memory for push module in /etc/nginx/nginx.conf:66

The main problem is that sometimes command service nginx reload fails with errors in the log:

[alert] 22164#22164: fork() failed while spawning "worker process" (12: Cannot allocate memory)
2018/10/09 03:10:06

[alert] 22164#22164: sendmsg() failed (9: Bad file descriptor)

The issue is gone if I exclude redirection rules from config. Nginx is set up on simple AWS t2.small instance with Ubuntu 16.04. It has 1GB of RAM and I see (with free -m) that at least half of the memory is free. I have default nginx.conf. So the question is how to avoid cannot allocate memory error, that is caused by the huge number of redirection rules?

This question was originally posted here. I thought that somebody here can know the answer. Sorry for duplication.

Best Answer

Split the file up into smaller parts if it is a large one. I had a 1.8MB file of about 20k redirects which a low memory instance couldn't handle and gave the error you got. I fixed it by:

mkdir /etc/nginx/conf.d/redirects
split -l 1000 redirect_file redirect_folder/

...and then just including the folder in a server config e.g:

server {
    listen 127.0.0.1:80
    ...
    include conf.d/redirects/*;
    ...
}

Nginx then loads fine. I'm guessing but my hunch is that nginx releases the memory used to read a file once it has loaded it, with lots of smaller files the memory use is much lower than that needed to load one big one. As I say though that's a pure hunch, I could be wrong.

Related Topic