Vps – mod_fcgid: No space left on device

First google hit indicates it might be a Debian bug.

HA! I finally found the problem myself. It's more related to programming than server admin, but I decided to put the answer here anyway because by searching google I found I'm not the only one with that kind of problem (and since Apache hangs, the first guess is that there's a problem with the server).

The issue is not with Apache, but with my Wordpress. More specifically with my theme. I'm using a theme called Lightworld and it supports adding an image to the blog header. To allow that, it checks the image size by using PHP's function getimagesize(). Since this function was opening another http connection to the server to get the image, each request from ab was creating another request internally from PHP. As I was using all my server available slots, these PHP requests were put in the queue, but Apache could never get to them because all it's processes were locked with the original request waiting for a slot to complete the PHP internal request.

Basically, PHP was putting my server into a deadlock state, and Apache would only start working normally after these connections timed out waiting for their "child" request.

After I removed this function from my theme, now I can ab my server with as many concurrent connections as I want, and Apache is queueing them as expected.