If you're using nginx, then you're talking just a few KB of overhead per active connection. If you're using something like Apache, you'll have one thread per connection, which means hundreds of KB or even megabytes per connection.
However, nginx does not support asynchronous disk IO on Linux (because async disk IO on Linux is basically horribly broken by design). So you will have to run many nginx worker processes, as every disk read could potentially block a whole worker process. If you're using FreeBSD, this isn't a problem, and nginx will work wonders with asynchronous disk and network IO. But you might want to stick with Apache if you're using Linux for this project.
But really, the most important thing is disk cache rather than the web server you choose. You want lots of free RAM so that the OS will cache those files and make reads really fast. If the "hot set" is more than say 8 GB, consider getting less RAM and an inexpensive SSD instead, as the cost/benefit ratio will likely be better.
Finally, consider using a CDN to offload this, and getting a really cheap server. Serving static files is what they do, and they do it very fast and very cheaply. SimpleCDN has the lowest prices, but MaxCDN, Rackspace, Amazon, etc. all are big players at the low end of the CDN space.
A few additional links and comments:
- There was a recent benchmark comparing Nginx, Apache, Varnish, and GWan for serving 100 byte static files.
- The Cherokee site has a few benchmarks comparing Nginx and Lighttpd.
- More links: One, Two, Three, Four, Five
- Be aware that benchmarks typical have a narrow band of validity, especially at this large a request rate. For example, the first benchmark is fine if all you're serving is 100 byte files but results may be significantly different if using 1kb, 10kb or a range of static file sizes. A lot of published benchmarks also have the issue that the author has good experience with only one of the products, meaning it has a very good configuration, and not of the other products, meaning they have a suboptimal or even bad configuration, which will skew the results.
- At this large of a request rate there are going be to many things that can effect the benchmark results. Not only the web server configuration but the OS configuration and hardware itself. For example, using a SSD drive may increase request rates but may work with web server X better than Y.
- Don't just look at the big maximum request rates from benchmarks but consider the average and minimums as well. For example, look at the Apache results from the first benchmark link.
- The best benchmark is doing to be one you do yourself using the files your serving on the hardware you're doing to use. Even quick benchmarks may reveal issues and effects not apparent from other published results.
- At this scale will it really matter if you can serve 15k or 20k requests/second? For example, at 20kr/s a 1.7kb file will require around 100TB of bandwidth a month. For the amount of money that the bandwidth will cost to buy another server (or 10) will be cheap by comparison. I would not choose solely by which server gives the biggest "number" but also consider how easy it is to setup/use, does it match the required feature set, is it well supported, etc....
Personally, I have used Lighttpd for years and couldn't be happier with it. I'm actually surprised at it performed compared to Nginx in the Cherokee benchmark results.
Best Answer
If you're looking for hardcore static performance, look into nginx. A web server that specializes specifically in serving static content.
You will find that Apache can serve static content quite quickly too if you don't use complex features or .htaccess files, etc.
Caching and connection handling are best left to the tunable features of the service daemon you choose.
Technically for any form of IO/general performance you want 0 swappiness - as that correlates to a memory problem (either too little memory, poor use, or legitimate consumption).