Linux KVM guest hanging after several weeks

kvm-virtualizationlinux

I've got a Linux KVM guest that hangs after several weeks. Looking at the virt-manager window shows 100% CPU usage. virsh reboot guest doesn't take any effect, the guest needs to be forced off. On the guest I can see no indication of what went wrong. I've scanned /var/log/messages and just about any other log file that can might tell me something. After rebooting the system is stable for several weeks and then hangs again.

I've tried the following things:

Add clocksource=acpi_pm to the kernel boot options
Changing the disk bus from Virtio to IDE
Changing the NIC to e1000

All to no avail. Right now I'm at a point where I reboot the server on a weekly basis.

Are there any other ways to diagnose what is going wrong here? Or any other changes that might be made?

Best Answer

try to get something from qemu log from /var/log/libvirt/qemu/<domain-id>.log.

If your VM hangs, try to use guestfs to inspect VM disk file and check /var/log/messages and/or /var/log/dmesg to see what happens inside the VM.

Related Solutions

Linux – How to investigate an unresponsive KVM guest

Investigating there kinds of problems is really difficult because you'd need to isolate different features of the setup and test them - which is very difficult on such a commplex setup and as the repro is a two weeks long process.

The first thing is try to do is to configure the syslog to send the logs over the network to a remote syslog service (possibly the one running on the host - you'd need to enable remote forwarding access on the syslog server) to allow you to catch errors that didn't make it into the guest log due to storage free space or sync issues.

If that doesn't give any useful info, you can try hooking into the guest serial console (see here for details) and log anything that happens there to a log file on the host.

KVM Guest auto start after install

probably not the cleanest solution, but this works (with the definitions defined at the top of the script)

virsh destroy $name-$ip_short
virsh undefine $name-$ip_short
rm -fr /kvm/disks/$name-$ip_short.img

virt-install \
--name=$name-$ip_short \
--arch=x86_64 \
--ram=$memory \
--os-type=linux \
--os-variant=virtio26 \
--hvm \
--connect=qemu:///system \
--network bridge:br0 \
--vcpus=$cpus \
--accelerate \
--autostart \
--disk path=$disk_directory/$name-$ip_short.img,size=$disk_size \
--location http://$domain/$location_path \
--vnc \
--noautoconsole \
-x "ks=http://$domain/$ks_path ksdevice=eth0 ip=$ip_long netmask=255.255.255.0 gateway=$gateway dns=$dns"


finished="0";

while [ "$finished" = "0" ]; do
        sleep 5;
        if [ `virsh list --all | grep "running" | grep "$name-$ip_short" | wc -c` -eq 0 ];
        then
                #echo "setup finished, start vm $name-$ip_short"
                finished=1;
                virsh start $name-$ip_short
        fi
done

Best Answer

Related Solutions

Linux – How to investigate an unresponsive KVM guest

KVM Guest auto start after install

Related Topic