Having just setup a project that is essentially identical to what you describe, I'll share my approach - no guarantees that it is 'the best', but it does work.
My server stack is
- Varnish (v3.0.2) - all interfaces, port 80
- Nginx (v1.0.14) - local interface, port 81
- Node.js (v0.6.13) - local interface, port 1337
- Operating system is CentOS 6.2 (or similar)
My Node.js app uses Websockets (sockets.io - v0.9.0) and Express (v2.5.8) - and is launched using forever. (The same server also has other sites on it - primarily PHP which use the same instances of Nginx and Varnish).
The basic intention of my approach is as follows:
- Single public port/address for both websocket and 'regular' data
- Cache some assets using Varnish
- Serve (uncached) static assets directly from nginx
- Pass requests for 'web pages' to nginx, and from their proxy to Node.js
- Pass web socket requests directly (from Varnish) to Node.js (bypass nginx).
Varnish config - /etc/varnish/default.vcl:
#Nginx - on port 81
backend default {
.host = "127.0.0.1";
.port = "81";
.connect_timeout = 5s;
.first_byte_timeout = 30s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
#Node.js - on port 1337
backend nodejs{
.host = "127.0.0.1";
.port = "1337";
.connect_timeout = 1s;
.first_byte_timeout = 2s;
.between_bytes_timeout = 60s;
.max_connections = 800;
}
sub vcl_recv {
set req.backend = default;
#Keeping the IP addresses correct for my logs
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For =
req.http.X-Forwarded-For + ", " + client.ip;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
#remove port, if included, to normalize host
set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");
#Part of the standard Varnish config
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
#Taken from the Varnish help on dealing with Websockets - pipe directly to Node.js
if (req.http.Upgrade ~ "(?i)websocket") {
set req.backend = nodejs;
return (pipe);
}
###Removed some cookie manipulation and compression settings##
if(req.http.Host ~"^(www\.)?example.com"){
#Removed some redirects and host normalization
#Requests made to this path, even if XHR polling still benefit from piping - pass does not seem to work
if (req.url ~ "^/socket.io/") {
set req.backend = nodejs;
return (pipe);
}
#I have a bunch of other sites which get included here, each in its own block
}elseif (req.http.Host ~ "^(www\.)?othersite.tld"){
#...
}
#Part of the standard Varnish config
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
#Everything else, lookup
return (lookup);
}
sub vcl_pipe {
#Need to copy the upgrade for websockets to work
if (req.http.upgrade) {
set bereq.http.upgrade = req.http.upgrade;
}
set bereq.http.Connection = "close";
return (pipe);
}
#All other functions should be fine unmodified (for basic functionality - most of mine are altered to my purposes; I find that adding a grace period, in particular, helps.
Nginx config - /etc/nginx/*/example.com.conf:
server {
listen *:81;
server_name example.com www.example.com static.example.com;
root /var/www/example.com/web;
error_log /var/log/nginx/example.com/error.log info;
access_log /var/log/nginx/example.com/access.log timed;
#removed error page setup
#home page
location = / {
proxy_pass http://node_js;
}
#everything else
location / {
try_files $uri $uri/ @proxy;
}
location @proxy{
proxy_pass http://node_js;
}
#removed some standard settings I use
}
upstream node_js {
server 127.0.0.1:1337;
server 127.0.0.1:1337;
}
I am not particularly crazy about the repetition of the proxy_pass statement, but haven't gotten around to finding a cleaner alternative yet, unfortunately. One approach may be to have a location block specifying the static file extensions explicitly and leave the proxy_pass statement outside of any location block.
A few settings from /etc/nginx/nginx.conf:
set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;
log_format timed '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'$request_time $upstream_response_time $pipe';
port_in_redirect off;
Among my other server blocks and settings, I also have gzip and keepalive enabled in my nginx config. (As an aside, I believe there is a TCP module for Nginx which would enable the use of websockets - however, I like using 'vanilla' versions of software (and their associated repositories), so that wasn't really an option for me).
A previous version of this setup resulted in an unusual 'blocking' behaviour with the piping in Varnish. Essentially, once a piped socket connection was established, the next request would be delayed until the pipe timed out (up to 60s). I haven't yet seen the same recur with this setup - but would be interested to know if you see a similar behaviour.
Best Answer
Nginx doesn't support WebSockets at all so you'll need to go with another load balancer. I've had good experiences with HAProxy. Check out this StackOverflow thread for a sample configuration: https://stackoverflow.com/questions/2419346/can-nginx-be-used-as-a-reverse-proxy-for-a-backend-websocket-server