Mysql high queries per second, with high cpu usage

The kind of numbers you are talking about are tiny, and as you observed load didn't change during that time period, so I don't expect this to be an issue of tuning apache or mysql directly.

In my experience of chasing issues like this down it's almost always ties into SQL. I would isolate what part of the page is consuming all of this time. Insert timer calls throughout the page in question and store the data somewhere you can review it (spit it to stderr and apache's error log if nothing else). Until you know the precise part of the code that is causing your slow down you will just be guessing.

Some possibilities:

• A query into a table that has grown very large and is now slow (lacks index? needs pruning?)
• A session table that is never pruned and has grown massive
• A very subtle corruption (run through and optimize all your mysql databases to fix)
• An ugly query that does heavy calculations every page load that could be optmized by saving the result in a cache.

I would also go back to basics and see if you can't find out what changed "a couple days ago". Was the page modified? Upgrades?

Another thing to check is I/O (iostat) to see if you may be I/O bound, though usually you will see symptoms in load for that.

Stop looking at the system and start looking at what's happening to MySQL. Even the briefest glance at the report makes it clear that you have some incredibly inefficient queries. Work on proper indexing and optimisation and I believe your problems will go away.