How to limit excessive CPU usage from guest OS in KVM

A virtual CPU equates to 1 physical core, but when your VM attempts to process something, it can potentially run on any of the cores that happen to be available at that moment. The scheduler handles this, and the VM is not aware of it. You can assign multiple vCPUs to a VM which allows it to run concurrently across several cores.

Cores are shared between all VMs as needed, so you could have a 4-core system, and 10 VMs running on it with 2 vCPUs assigned to each. VMs share all the cores in your system quite efficiently as determined by the scheduler. This is one of the main benefits of virtualization - making the most use of under-subscribed resources to power multiple OS instances.

If your VMs are so busy that they have to contend for CPU time, the outcome is that VMs may have to wait for CPU time. Again, this is transparent to the VM and handled by the scheduler.

I'm not familiar with KVM but all of the above is generic behavior for most virtualization systems.

It turns out that the guest was under-reporting its CPU usage. I ran this same workload on another guest entirely and it used CPU resources at about the same level as my KVM host was reporting. This was not an apples to apples comparisan but it was a meaningful result.

Then, I had my real breakthrough. I noticed that, for this one guest I was mistakenly using the virtual "SCSI" controller. I changed it to Virtio and two things happened: host CPU usage went down a little bit, and the guest CPU went up - it reported almost exactly the same as the host, like we'd expect.