How to get rid of TCP or lighttpd 4KB buffering on CGI output

cgilighttpdtcp

I have a stock lighttpd install on a Amazon Linux AMI running on EC2. The only config changes I did is to enable CGI support.

Then there is a custom CGI tool (written in C) put in /cgi-bin/ that is called from the outside world and outputs gzip'ed JSON. This tool outputs data regularly, say every 10-20 seconds, but only a few hundreds bytes each time.

The problem is that somewhere between the CGI stdout and what lighttpd returns to the client, things are buffered and arrive about 4KB at a time. Unfortunately, this system is behind the Amazon Elastic Load Balancer which doesn't allow idle connections for more than 60 seconds. Because of the buffering, it's as-if the server returns nothing the first 60 seconds, so the connection gets killed and the client doesn't get anything.

So how do I track down this buffering setting and reduce it significantly? I tried changing some lighttpd config parameters and even changing "net.ipv4.tcp_wmem" in the kernel, but nothing appears to work.

Best Answer

strace the lighttpd process with follow enabled:

strace -f -tt -p PIDOFLIGHTTPD

You will get output that shows each system call made by lighttpd and the CGI. The microsecond timestamp should indicate when the CGI is returning data and when lighttpd is writing it back to the client. This will also answer if the issue is upstream(if you see lighttpd sending back data almost instantly, etc).

Related Solutions

Python CGI on Amazon AWS EC2 micro-instance — a how-to!

(Strange post, so hopefully this won't be as strange a reply).

On the subject of security flaws: it is considered general bad practice to store cgi-bin scripts within the document root of the web server. Even the W3C eludes to it under "Are compiled languages such as C safer ..." in their World Wide Web Security FAQ:

Consider the following scenario. For convenience's sake, you've decided to identify CGI scripts to the server using the .cgi extension. Later on, you need to make a small change to an interpreted CGI script. You open it up with the Emacs text editor and modify the script. Unfortunately the edit leaves a backup copy of the script source code lying around in the document tree. Although the remote user can't obtain the source code by fetching the script itself, he can now obtain the backup copy by blindly requesting the URL:
    http://your-site/a/path/your_script.cgi~
(This is another good reason to limit CGI scripts to cgi-bin and to make sure that cgi-bin is separate from the document root.)

This is not as significant a threat as the ability to write a file within the document root. However, an attacker could obtain the source code of the cgi, devise a directed attack against it, and use it as a stepping stone into the server.

To mitigate this, you can add the following lines to the lighttpd.conf (or some variation therein) to direct cgi-bin to a directory separate from the /var/www/lighttpd document root.

$HTTP["url"] =~ "/cgi-bin/" { cgi.assign = ( "" => "" ) }
alias.url = ( "/cgi-bin/" => "/usr/lib/cgi-bin/" )

This requires both the cgi and alias modules for lighttpd.

Apache 2.2 CGI timeout configuration

The TimeOut directive defines the length of time Apache will wait for I/O in various circumstances:

When reading data from the client, the length of time to wait for a TCP packet to arrive if the read buffer is empty.
When writing data to the client, the length of time to wait for an acknowledgement of a packet if the send buffer is full.
In mod_cgi, the length of time to wait for output from a CGI script.
In mod_ext_filter, the length of time to wait for output from a filtering process.
In mod_proxy, the default timeout value if ProxyTimeout is not configured.

Best Answer

Related Solutions

Python CGI on Amazon AWS EC2 micro-instance — a how-to!

Apache 2.2 CGI timeout configuration

Related Topic