Nginx – Google Compute Engine Global HTTP load balancers

google-cloud-http-load-balancergoogle-cloud-platformgoogle-compute-enginenginx

I am using Google Compute Engine. I have setup deployment manager and it set's up a firewall that allows network LB to connect to web servers, the web servers themselves adds them to a Instance Group Manager and it set's up an Autoscaler that targets the Instance Group Manager, HTTP Health Check is setup that will execute against web server instances its adds the HTTP Health Checks to a backend service and adds the Instance Group Manager to the backend service, it set's up a URL Map that has the backend service as it's default Service the url mapper in its turn are added to the HTTP Proxy that is pointed at by a Forwarding Rule that have a global IP.

This setup is very similar to the setup described here https://cloud.google.com/solutions/scalable-and-resilient-apps

So now to the problem that I cans seem to solve for this setup. I have a Nginx server running on the web servers and it responds to requests and I am able to create event source connections to it but after exactly 1 minute the connection is closed with the error INCOMPLETE_CHUNKED_ENCODING. This do not happen if I connect directly to one of the web servers. I have changed the sysconf setting for tcp keepalive to:

net.ipv4.tcp_keepalive_time=600 
net.ipv4.tcp_keepalive_intvl=15 
net.ipv4.tcp_keepalive_probes=5

This after reading https://cloud.google.com/compute/docs/troubleshooting#networktraffic

I have tried countless things in the nginx config and can not seam to find a solution.

Do any one have any idea or similar problems?

Best Answer

Alex was right sharing that post link because it leads to the main issue, but it needs a little bit of an explanation.

You will need to change 'keepalive_timeout' value (default is 65) in your Nginx configuration file ( /etc/nginx/nginx.conf) to increase the HTTP connection timeout, so your timeout is longer than the 600 seconds timeout in the Load Balancer. This causes the load balancer to be the side that closes idle connections, rather than nginx.

Tune nginx keepalives to work with the Google Cloud Platform HTTP(S) Load Balancer.
Set “keepalive_timeout 650;” in nginx /etc/nginx/nginx.conf

keepalive_timeout 650;
keepalive_requests 10000;

More in depth information about http persistence.

Related Topic