Tuning tips for CPU load for OpenBSD guest on qemu-kvm Server

kvm-virtualizationopenbsd

I have a CentOS KVM Server running 1 OpenBSD, 1 CentOS and 2 FreeBSD guests.

Only the openBSD guest has a higher CPU load on the KVM Server. 8-10% on Server whilst Guest is at idle.

The only service I am running on the guest is pfctl. if I disable this the CPU load remains the same.

Are there any performannce tuning alterations I can make to KVM Host / openBSd guest to improve this.

The only hardware difference to the other gusets is that the OpenBSD guest has a PassThru PCI NIC. Which I have disabled as a test and Cpu load renained at 8-10%.

I have been through a lot of settings in Virtual Manager, ie setting Disks to VirtIO, Display to VGA/None etc, CPU Settings. Maybe a setting I am missing as I am new to qemu-kvm/libvirt?

ps output OpenBSD guest.

ps aux |grep 26303
qemu     26303 10.0 10.1 1538672 1219800 ?     SLl  09:12   2:32 /usr/libexec/qemu-kvm -name skips -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Haswell,-rtm,-hle -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 0520a68a-de40-4187-8a9f-d1e05dc83255 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-skips/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive if=none,id=drive-ide0-0-1,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1 -drive file=/var/lib/libvirt/images/skips.qcow2,if=none,id=drive-ide0-1-0,format=qcow2 -device ide-hd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:59:fd:17,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 -spice port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -global qxl-vga.vgamem_mb=16 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x8 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
root     26309  0.0  0.0      0     0 ?        S    09:12   0:00 [kvm-pit/26303]
root     27346  0.0  0.0 112644   960 pts/4    S+   09:37   0:00 grep --color=auto 26303

CENTOS KVM HOST TOP OUTPUT:

top - 08:59:25 up 1 day, 13:12,  2 users,  load average: 0.11, 0.10, 0.10
Tasks: 145 total,   1 running, 144 sleeping,   0 stopped,   0 zombie
top - 09:05:53 up 1 day, 13:18,  2 users,  load average: 0.02, 0.06, 0.08
Tasks: 143 total,   2 running, 141 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.2 us,  0.9 sy,  0.0 ni, 97.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 11995120 total,   246888 free,  4868624 used,  6879608 buff/cache
KiB Swap:  6094844 total,  6083828 free,    11016 used.  6806152 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
26303 qemu      20   0 1537648 1.174g  11728 S   8.3 10.3   1:48.18 qemu-kvm
28147 qemu      20   0 1746032 1.084g  11672 S   1.3  9.5  18:33.98 qemu-kvm  
 2759 qemu      20   0 1646104 1.124g  11840 S   1.0  9.8  20:18.35 qemu-kvm
15927 qemu      20   0 1694228 1.052g  11632 S   0.7  9.2  25:24.74 qemu-kvm

OPENBSD GUEST TOP OUTPUT:

load averages:  0.09,  0.09,  0.08  
26 processes: 25 idle, 1 on processor
CPU states:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Memory: Real: 18M/62M act/tot Free: 920M Cache: 17M Swap: 0K/1264M

dmesg: (note: have lowered RAM from 1024 to 256MB)

OpenBSD 5.8 (GENERIC) #1170: Sun Aug 16 02:26:00 MDT 2015
    deraadt@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC
real mem = 251650048 (239MB)
avail mem = 240300032 (229MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf1e70 (10 entries)
bios0: vendor Seabios version "0.5.1" date 01/01/2011
bios0: Red Hat KVM
acpi0 at bios0: rev 0
acpi0: sleep states S5
acpi0: tables DSDT FACP SSDT APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Core Processor (Haswell), 3292.72 MHz
cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
pvbus0 at mainbus0: KVM
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 1
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 1.5.> ATAPI 5/cdrom removable
cd0(pciide0:0:1): using PIO mode 4, DMA mode 2
wd0 at pciide0 channel 1 drive 0: <QEMU HARDDISK>
wd0: 16-sector PIO, LBA48, 20480MB, 41943040 sectors
wd0(pciide0:1:0): using PIO mode 4, DMA mode 2
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
iic0: addr 0x18 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x1a 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x29 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x2b 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x4c 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
iic0: addr 0x4e 00=00 01=00 02=00 03=00 04=00 05=00 06=00 07=00 08=00 words 00=0000 01=0000 02=0000 03=0000 04=0000 05=0000 06=0000 07=0000
vga1 at pci0 dev 2 function 0 "Red Hat QXL Video" rev 0x04
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
re0 at pci0 dev 3 function 0 "Realtek 8139" rev 0x20: RTL8139C+ (0x7480), apic 0 int 11, address 52:54:00:59:fd:17
rlphy0 at re0 phy 0: RTL internal PHY
azalia0 at pci0 dev 4 function 0 "Intel 82801FB HD Audio" rev 0x01: apic 0 int 11
azalia0: No codecs found
virtio0 at pci0 dev 5 function 0 "Qumranet Virtio Console" rev 0x00: Virtio Console Device
virtio0: no matching child driver; not configured
uhci0 at pci0 dev 6 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci1 at pci0 dev 6 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 11
uhci2 at pci0 dev 6 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
ehci0 at pci0 dev 6 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 10
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
virtio1 at pci0 dev 7 function 0 "Qumranet Virtio Memory" rev 0x00: Virtio Memory Balloon Device
viomb0 at virtio1
virtio1: apic 0 int 11
re1 at pci0 dev 8 function 0 "Realtek 8169" rev 0x10: RTL8169/8110SB (0x1000), apic 0 int 11, address 60:e3:27:03:9f:80
rgephy0 at re1 phy 7: RTL8169S/8110S/8211 PHY, rev. 3
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 1: density unknown
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
nvram: invalid checksum
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on wd0a (92a6f3df5ed1df79.a) swap on wd0b dump on wd0b
clock: unknown CMOS layout

systat

1 users    Load 0.11 0.12 0.08                     Tue Feb 23 23:26:40 2016

            memory totals (in KB)            PAGING   SWAPPING     Interrupts
           real   virtual     free           in  out   in  out      278 total
Active    19208     19208   174948   ops                            100 clock
All       67940     67940  1469272   pages                              pciide0
                                                                    110 re0
Proc:r  d  s  w    Csw   Trp   Sys   Int   Sof  Flt       forks      68 re1
           3       223     1    42   180   222    7       fkppw
                                                          fksvm
   0.8%Int   0.2%Sys   0.0%Usr   0.0%Nic  99.0%Idle       pwait
|    |    |    |    |    |    |    |    |    |    |       relck
                                                          rlkok
                                                          noram
Namei         Sys-cache    Proc-cache    No-cache         ndcpy
    Calls     hits    %    hits     %    miss   %         fltcp
                                                          zfod
                                                          cow
Disks   cd0   wd0   fd0                              2024 fmin
seeks                                                2698 ftarg
xfers                                                     itarg
speed                                                   1 wired
  sec                                                     pdfre
                                                          pdscn
                                                          pzidle
                                                       13 kmapent

uname -a (from centos 7 1511)

Linux benson 3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

uname -a (from freebsd 10.2)

 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r286666: Wed Aug 12 15:26:37 UTC 2015     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

appreciate any tips,

Philip

UPDATE: I checked a lot of the options and could not get cpu levels down. On a further note I am running an instance of openBSD under xen and I am not experiencing high levels.

Best Answer

I managed to reduce the CPU usage by following the instructions from https://marc.info/?l=openbsd-misc&m=151575775607633&w=2. Below example of my OpenBSD 6.4 virtual machine config file:

<domain type='kvm'>
  <name>openbsd</name>
  <uuid>6ab7f964-06f2-4526-b5a7-c5cc1ff22254</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://openbsd.org/openbsd/6.4"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.0'>hvm</type>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='hpet' present='yes'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>coredump-restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/lvm2/openbsd-vm'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='usb' index='0' model='none'/>
    <interface type='bridge'>
      <mac address='fb:70:55:fa:35:08'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>    
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <video>
      <model type='virtio' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Now the CPU usage on the host has dropped to around 2-4%. Previously, it was around 10-12%.

Related Topic