IIS w3wp using high cpu with no traffic

The CPU usage from Apache's server-status page is the average usage since Apache was started so it won't show spikes like this. When you get these load spikes you can check the server-status page to see what pages/clients are being server (ExtendedStatus must be on).

You can also use netstat to see what clients are currently accessing your machine:

 netstat -an | grep ESTABLISHED

If you run this over multiple hours and traffic spikes you may be able to spot a reoccuring IP address and potentially trace to a specific robot/crawler. If this does turn out to be the case you can look into using robots.txt to limit how well behaving robots should crawl your site.

Edit: On a busy server the above netstat command should show some entries like:

tcp        0      0 10.2.212.13:80              216.146.52.21:24979         ESTABLISHED
tcp        0      0 10.2.212.13:80              86.174.113.138:54901        ESTABLISHED
tcp        0      0 10.2.212.13:80              94.1.216.253:51204          ESTABLISHED
tcp        0      0 10.2.212.13:80              24.9.61.204:62936           ESTABLISHED

The client's IP address will be the one on the right. If you only see 1 or 2 lines it just means that at that moment there is just your ssh connection. Check again when your load increases. You can also remove the grep to list all connections although this will include a large number of old TIME_WAIT.

I would start with the extended server-status and see if that can reveal any obvious crawlers during traffic peaks.

As others have already pointed out, we can see from that screenshot that the CPU that's working so hard is spending all its time in kernel mode. (The red color.)

Running Powershell as administrator, type:

Get-Process | Select Name, PrivilegedProcessorTime | `
Sort-Object PrivilegedProcessorTime -Descending

The process at the top of the list is the process currently using the most kernel mode CPU time right now. If that process is not "System," then you've just figured out what user mode process is causing this CPU usage. If the process with the highest Privileged Processor Time is System, which I suspect it is, then it's a little more complicated.

Open Process Explorer. Optionally, set up your symbol server. Make sure you are running with full UAC elevation. Right click the System "process" and go to Properties. Then go to the Threads tab. Sort the threads by CPU usage. The thread that's causing all this kernel mode work should be here. If you look at the module listed under Start Address, it should give you a clue as to what the work is related to. If it's NDIS.sys, for instance, that's a network interface driver. If you set up the symbol server, you should see the name of a function within a module (unless the module is non-Microsoft,) else you'll just see a numerical offset from the module's start address.

Alternatively, use Xperf from the Windows Performance Toolkit to profile interrupts, DPCs, etc.

xperf -on PROC_THREAD+LOADER+DPC+INTERRUPT

and stop recording with xperf -d logfile.etl

Xperf replaces the old Kernrate tool, and can net you some extremely detailed data.

When a CPU is doing work in kernel mode, it's mostly running interrupt service routines. (ISRs) When an interrupt occurs, user mode work is suspended on that processor, and the CPU runs the ISR registered to that interrupt. If you find your CPU spending an inordinate amount of time on these interrupts, that usually indicates a faulty device driver that needs to be updated.

What bugs me (no pun intended) about this scenario though is that it appears as though whatever kernel thread that is doing this seems to be affinitized to that one core. I wonder why the dispatcher seems to be only scheduling the thread to run on that one seemingly arbitrary core. So I have a feeling that we need to find whoever wrote this device driver and show them how to do threaded DPCs, and not to explicitly set an affinity on kernel threads, etc.