Nginx drops connections

linodeload balancingload-balancenginxphp-fpm

I'm having a setup where I use the linode nodebalancer (loadbalancer) for my nginx/php5-fpm servers. This balancer passive checks. These passive checks, check the status code of requests. If there are too many 5XX status codes the node (vps) is marked offline by the loadbalancer

The nodebalancer is putting my servers offline in a random way. When contacting linode support they came to the conclusion that there are no 500 errors, but connections are dropped (or timeout).

I can't find anything in my nginx logs. Is there any way to debug this problem and see what connections have been timed-out dropped by nginx?

EDIT
I can see a lot of 408 requests from the same IP/user agent. They come in by bulk.. Is this suspicious? How would you handle this situation? Snapshot from access.log

69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:29 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:30 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
69.30.*.* - - [22/Apr/2014:19:28:31 +0200] "POST /error/register-image-error/ HTTP/1.1" 408 0 "http://www.mysite.com/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"

Thanks!

Best Answer

Would be nice to know what HTTP code are you getting. I suspect it should be 408 (Request Timeout), so nodebalancer will think the server is gone and will mark it offline.

What you should try:

  • Increase the timeouts
  • Use TCP mode instead HTTP mode

Warning: Using TCP mode, cookie handling if OFF.

Related Topic