Our database servers (mainly based on the Debian stable packages (=currently Wheezy) seem to have about 4 times more load for the same workload in kernel 3.2.0-4-amd64 then in it's previous 2.6.32-5-amd64 kernel. With all packages the same & booting in the other kernel we can clearly see the difference, and I'm at a loss as to why. The problem is, I don't see that much IO or CPU load difference.
Setting the default kernel.sched_min_granularity_ns & kernel.sched_latency_ns back to it's 2.6.32 values helps a little (thrice the load instead of 4 times), but not to the level we'd like. As a lot of kernel settings changed, we can hardly just blindly set the new kernel to the old default values of the 2.6 one.
Has anybody else had experience with this? If so, what caused this (and ideally: how could it be solved)?
As it's deep kernel-related, perhaps a difference in sysctl values might be of interest: here is a diff of the 2 (pastebinned to prevent an overly long question).
edit: currently we're investigating this SO answer to see if that applies.
Best Answer
Linux kernels 3.0 - 3.8 should be avoided or upgraded to address IO performance degradation
Linux kernel IO performance degradation demonstrated by Josh Berkus using a private benchmark workload running against PostgreSQL 9.3 on Ubuntu 12.04 with kernel 3.2.0.
"...you really need to avoid every kernel between 3.0 and 3.8. While RHEL has been sticking to the 2.6 kernels (which have their own issues, but not as bad as this), Ubuntu has released various 3.X kernels for 12.04...upgraded...to kernel 3.13.0, and ran the same exact workload...an 80% reduction in IO. We can thank the smart folks in the Linux FS/MM group for hammering down a whole slew of performance issues."
Please see http://www.databasesoup.com/2014/09/why-you-need-to-avoid-linux-kernel-32.html