Configuring lighttpd for many lightweight concurrent Keep-Alive http connections

httplighttpdreverse-proxy

I am using lighttpd as a front-end proxy to my custom HTTP based application server. I need to configure lighttpd for a large number (let's say around 5000) simultaneous http connections which have a large timeout and KeepAlive setting. Each connection will sit around mostly idle. Imagine an HTTP based chat server.

My HTTP server is using http-push Comet-like interaction (see Wikipedia http://en.wikipedia.org/wiki/Comet_(programming) ). Specifically, the AJAX client sends a GET request to which the server does not immediately reply. Instead, the server waits with the HTTPconnection open until it needs to message the client with new state, and then sends the HTTP reply to the GET request. The client processes the reply, and proceeds to send another GET request, which once again, the server will not immediately answer.

However, for the sake of the question, the exact nature of the requests is not necessary. What is needed is essentially a lighttpd configuration which allows a large number of low-bandwidth simultaneous HTTP proxy connections open at one time.

How do I configure lighttpd 1.4.19. I am running under Ubuntu 8.04. lighttpd is proxying requests to my app server as well as to my django backend.

  • Do I simply set
    server.max-keep-alive-requests =
    5000
    and call it a day?

  • Should server.max-fds = 5000 or
    some larger number?

  • What memory considerations are there?

  • Perhaps I should harden my app server
    so that it can be used without the
    lighttpd proxy (I am not that
    confident in my programming having it
    be directly world-facing without a
    reliable proxy)?

It is claimed that lighttpd can handle 10,000 simultanous connections. How do I configure it to do half of that number, most of which are mostly idle?

Best Answer

Remember that each proxy connection uses (a minimum of) two FDs, one to the client, one to the server being proxied. Also consider ~5-50 FD's for overhead.

As for the rest of your question I'd say you need to test your application in your environment to find the best settings for you.