We wanted to switch all our servers from lighttpd to nginx but while the Web server migration went fine, the static files server migration was disastrous.
We ended up with up to 10 seconds of latency for loading pictures, using this config with nginx:
- main config: https://gist.github.com/1089283
- on site example: https://gist.github.com/1089276
There are 4 sites, with about 120k visitors/day that consume massively streamed videos and picures.
A hint we had was that after rebooting nginx, loading picture was always really fast. The CPU or the memory consumption was always really minimal, so we though about the disk. We tried to tweak the file descriptors limit but it didn't change anything.
We switched back to lighttpd and we are back in business again, with nearly instantaneous picture loading.
We know we did something wrong, but what ? It would be nice to have only to maintain one server set up, and nginx is really easier to set up than lighttpd.
The configuration files you are seeing here are for nginx on a virtual machine dedicated to static files and video streaming (compile with the mp4 plugin).
Best Answer
I had a a somewhat similar experience with nginx recently. We were serving up static files and over time the server would slow down and eventually crash. It turned out nginx was caching the downloads to disk, eventually filling it and causing the server to crash.
Not saying this is your problem and unfortunately I don't have the fix we applied at the moment but perhaps it might set you in the right direction.