OpenStack instances power off by itself

kvm-virtualizationlibvirtopenstack

Already asked this over on ask.openstack.org but I figured it would get more views here too.

I am running OpenStack Mitaka on a 5 node cluster, 1 controller/network node and 4 compute nodes. I am able to create VMs and use them normally throughout my day but when I come back in the morning some instances are powered off.

I've checked to make sure there is enough disk space (OK). I've checked if the problem is on any compute node (YES). I've also checked the RAM and CPU usage (OK).

I've tried to change the handle_virt_lifecycle_events variable in nova.conf to true and false with no success as some research has led me to.

It seems like the power state from the DB to the hypervisor (KVM) being mismatched is causing this.

Any ideas?

Here's the only relevant information that I could fine from /var/log/nova/nova-compute.log:

2016-06-08 21:20:18.208 1738 INFO nova.compute.manager [-] [instance: 493ad84c-1e46-410c-8fab-078c77ddea10] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.228 1738 INFO nova.compute.manager [-] [instance: 2dcb6116-55ab-4987-a4f5-ce3e0f1edda4] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.232 1738 INFO nova.compute.manager [-] [instance: d1d0172d-20aa-4333-b9e9-aabc064287bc] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.298 1738 WARNING nova.compute.manager [-] [instance: 493ad84c-1e46-410c-8fab-078c77ddea10] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
2016-06-08 21:20:18.314 1738 WARNING nova.compute.manager [-] [instance: 2dcb6116-55ab-4987-a4f5-ce3e0f1edda4] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
2016-06-08 21:20:18.334 1738 WARNING nova.compute.manager [-] [instance: d1d0172d-20aa-4333-b9e9-aabc064287bc] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4

Best Answer

In /etc/nova/nova.conf, set the following option to disable the synchronization

sync_power_state_interval=-1

This will prevent enforcing VM state from DB.

Quoting from here:

The synchronization task will see that the DB is out of sync and will try to 'correct' the inconsistency by powering off the VM.

Related Topic