Nginx – AWS Elastic Beanstalk – Socket failure on Puma server when more than 300 connections at a given time

amazon-web-serviceselastic-beanstalknginxrubyruby-on-rails

I've been dealing with an, arguably, strange issue on a AWS Elastic Beanstalk environment. I'm getting the following nginx error when there are more than 300 connections on a single instance at a given time:

2018/03/23 20:56:53 [error] 5431#0: *4121 connect() to unix:///var/run/puma/my_app.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 🐙, server: _, request: "GET /api/v1/podcasts/ HTTP/1.1", upstream: "http://unix:///var/run/puma/my_app.sock:/api/v1/podcasts/", host: 🐙

My Puma config has

bind "unix:///var/run/puma/my_app.sock"

And the nginx config on the EC2 instance seems to have the socket properly set up (the default nginx config provided by EBS).

So yeah, I have no idea what's causing this. Any ideas? Thanks!

Best Answer

Well going through the issue and the configurations the failure is related to exceeded resource limit. Possibly two of the OS resources:

  • open file descriptors
  • processes available to a single user. Just to eliminate the process limitation keep your nproc value in EC2 instance to very high value or unlimited.

    sudo vi /etc/security/limits.conf

Refer this link for a brief idea. So since you cannot have worker_rlimit_nofile more than the ulimit set on the server first approch would be to calculate the nginx parameters and tweak as per requirement:

We can serve worker_processes * worker_connections / ( keepalive_timout * 2 ) users per second.

This scaling guide may help you quantifying the values you should increase and define. Once you are done defining the values for your web server you'll need to update the limits with higher or equal values. Check upto what user concurrency the server works to help you scale up.

Related Topic