Samba – Resolving high iowait on Samba server

I know it's the time spent by the CPU waiting for a IO operations to complete, but what kind of IO operations precisely? What I am also not sure, is why it so important? Can't the CPU just do something else while the IO operation completes, and then get back to processing data?

Yes, the operating system will schedule other processes to run while one is blocked on IO. However inside that process, unless it's using asynchronous IO, it will not progress until whatever IO operation is complete.

Also what are the right tools to diagnose what process(es) did exactly wait for IO.

Some tools you might find useful

• iostat, to monitor the service times of your disks
• iotop (if your kernel supports it), to monitor the breakdown of IO requests per process
• strace, to look at the actual operations issued by a process

And what are the ways to minimize IO wait time?

• ensure you have free physical memory so the OS can cache disk blocks in memory
• keep your filesystem disk usage below 80% to avoid excessive fragmentation
• tune your filesystem
• use a battery backed array controller
• choose good buffer sizes when performing io operations

For the purely static content, you normally gain very little advantage by running Varnish in front of any mature web server such as nginx or Apache. (I can't speak for node.js because I have not yet used it.) The reason for this is that the kernel maintains a filesystem cache which stores recently accessed files in RAM. It's possible that Varnish does a slightly better job than the kernel at choosing exactly which files to keep in the cache and which to eject but it's also possible that it does a worse job. This will depend on what else your system does as well as the differences in the cache retention policies. The extra latency caused by having a proxy in front of your web server will far exceed any differences between Varnish and kernel filesystem caching.

You are correct about Varnish not caching responses sent with a Set-Cookie: header. It also skips looking at the cache if the request contains a Cookie: header. The reason for this is that there will be one such unique response for each and every user who visits any given page and each user is unlikely to re-visit the same page meaning that your cache would be full of entities that will never be used.

Varnish can cache dynamic content and this is where it shines. Let's say the front page of your website requires a PHP code compile and 20 database queries on cold caches. On warm caches (APC, memcached, Redis, MySQL query cache, etc.) it still requires lookups to memcached/Redis and doing a stat() on every PHP file included in the request. Assuming you have a reasonably well optimised site, this is still likely to be at least 1/10th of a second (and that's pretty well optimised; 1 - 3 whole seconds is far more common in my experience). Varnish will happily serve the same page in under 1/1000th of a second. But it means that your users can't be logged in on the front page of your website because they can't get their Cookie: headers but if this is acceptable to you, Varnish is an easy win.

Varnish also requires correct caching headers. If it isn't sure that it's safe to cache an object, it won't. If you meet all these requirements, Varnish might be the tool for you. That said, just putting the correct headers in place will cause the clients to cache the content themselves which can make a much larger difference than Varnish can.

If you lack the experience to benchmark your setup yourself, now is the time to get that experience. You are benchmarking to gauge the appropriateness of your configs. Try something and run ab over it, then change one thing and run ab in exactly the same way against that configuration. Now you know which one is "better". Rinse, repeat. (Also, always run ab twice and ignore the first result. This will make sure you only test warm caches and don't end up erroneously thinking one configuration is worse because it was tested against cold caches.)

Architecting a scalable solution for your website is a complex enough topic that it won't fit in a ServerFault answer. We can certainly help with individual parts of that (such as "How do I scale out my memcached server to a memcached cluster?") when you run into those problems.

Further reading:

I wrote an answer a while ago that had a section on finding the bottleneck when benchmarking.
Womble recently pointed me at an excellent post on finding bottlenecks.