Already asked this over on ask.openstack.org but I figured it would get more views here too.
I am running OpenStack Mitaka on a 5 node cluster, 1 controller/network node and 4 compute nodes. I am able to create VMs and use them normally throughout my day but when I come back in the morning some instances are powered off.
I've checked to make sure there is enough disk space (OK). I've checked if the problem is on any compute node (YES). I've also checked the RAM and CPU usage (OK).
I've tried to change the handle_virt_lifecycle_events variable in nova.conf to true and false with no success as some research has led me to.
It seems like the power state from the DB to the hypervisor (KVM) being mismatched is causing this.
Any ideas?
Here's the only relevant information that I could fine from /var/log/nova/nova-compute.log:
2016-06-08 21:20:18.208 1738 INFO nova.compute.manager [-] [instance: 493ad84c-1e46-410c-8fab-078c77ddea10] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.228 1738 INFO nova.compute.manager [-] [instance: 2dcb6116-55ab-4987-a4f5-ce3e0f1edda4] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.232 1738 INFO nova.compute.manager [-] [instance: d1d0172d-20aa-4333-b9e9-aabc064287bc] During _sync_instance_power_state the DB power_state (1) does not match the vm_power_state from the hypervisor (4). Updating power_state in the DB to match the hypervisor.
2016-06-08 21:20:18.298 1738 WARNING nova.compute.manager [-] [instance: 493ad84c-1e46-410c-8fab-078c77ddea10] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
2016-06-08 21:20:18.314 1738 WARNING nova.compute.manager [-] [instance: 2dcb6116-55ab-4987-a4f5-ce3e0f1edda4] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
2016-06-08 21:20:18.334 1738 WARNING nova.compute.manager [-] [instance: d1d0172d-20aa-4333-b9e9-aabc064287bc] Instance shutdown by itself. Calling the stop API. Current vm_state: active, current task_state: None, original DB power_state: 1, current VM power_state: 4
Best Answer
In /etc/nova/nova.conf, set the following option to disable the synchronization
This will prevent enforcing VM state from DB.
Quoting from here: