Nginx – connect() failed upstream under load testing

load-testingnginxportresourcesreverse-proxy

I've been doing some load testing with wrk of my nginx reverse proxy -> my web app setup and I noticed that when I get to 1000+ concurrent connections, nginx starts returning 502s and the following error message:

2015/04/17 20:45:26 [crit] 6068#0: *1116212677 connect() to \
127.0.0.1:3004 failed (99: Cannot assign requested address) \
while connecting to upstream, client: xxx.xxx.xx.165, server: \
foo.bar.com, request: "GET /my/route HTTP/1.1", upstream: \
"http://127.0.0.1:3004/my/route", host: "foo.bar.com"

the wrk command was:

wrk -t10 -c500 -d5m "https://foo.bar.com/my/route" -H "Accept: application/json"

I'm trying to figure out what might have gone wrong here. My web application is listening to requests proxied by nginx at port 3004. Is nginx running out of ports? Is the web application not able to handle this many request? Are requests being timed out? I'm not clear on this and would love to have more insight into it.

Best Answer

Already answered here: https://stackoverflow.com/questions/14144396/nginx-proxy-connect-to-ip80-failed-99-cannot-assign-requested-address

The message suggests you've run out of local sockets/ports.

Try to increase networking limits:

echo "10240 65535" > /proc/sys/net/ipv4/ip_local_port_range
sysctl net.ipv4.tcp_timestamps=1
sysctl net.ipv4.tcp_tw_recycle=0
sysctl net.ipv4.tcp_tw_reuse=1
sysctl net.ipv4.tcp_max_tw_buckets=10000

Alternatively you may try unix sockets to see if it helps.

Related Topic