Docker – Q: Unable to create ‘live’ snapshot of running KVM using virsh

dockerkvm-virtualizationqemurhel7snapshot

I was kindly directed to here, Serverfault from Stackoverflow to pose my question.

I'm running the following command on a RHEL7 hypervisor to create a live backup (disc image) of a running kvm, which is running a production docker service in the kvm I want a live snapshot.

Many docs I have read lead me to this command to get a live snapshot of a running KVM:

# virsh snapshot-create-as \
    --domain undercloud-vm undercloud-vm-snapshot \
    --diskspec vda,file=/backup/images/undercloud-vm.qcow2 \
    --quiesce \
    --disk-only \
    --atomic

error: internal error: unable to execute QEMU agent command
'guest-fsfreeze-freeze': failed to open /var/lib/docker/overlay2:
Permission denied

Not wanting to mess with the KVM's /var/lib/docker/overlay2 directory permissions, I have resorted to asking this question here. And if anyone has come across this issue of trying to make a live snapshot of a KVM that happens to be running docker inside. The main thing here is that the KVM is production, and it can not simply be paused or frozen to create the snapshot, it must continue to run and process data.

Any pointers to get me a live backup would be greatly appreciated.

INFO…………

In the KVM: (running: /usr/libexec/qemu-kvm …)

  • OS: Red Hat Enterprise Linux Server release 7.5 (Maipo)
  • Channel: org.qemu.guest_agent.0 has been added as virtual H/W
  • YUM: yum install qemu-guest-agent
  • S/W: Docker version 1.13.1, build 6e3bb8e/1.13.1

Hypervisor:

  • HW: PowerEdge FC630, Dell FX2s 36 CPUs, 128GB RAM
  • OS: Red Hat Enterprise Linux Server release 7.6 (Maipo)
  • SW: qemu-img-rhev = 10:2.10.0-21.el7_5.4
  • qemu-kvm-(common-)rhev = 10:2.10.0-21.el7_5.4

Best Answer

I just ran into the same problem. It's an selinux issue: selinux (on the guest) is preventing the QEMU guest agent from quiescing the target filesystem. You can see a summary of the issue like this:

# ausearch -c qemu-ga --raw | audit2allow


#============= virt_qemu_ga_t ==============

allow virt_qemu_ga_t container_var_lib_t:dir { ioctl open read search };

allow virt_qemu_ga_t proc_net_t:file read;

You can generate an appropriate local security module that permits qemu-ga to operate correctly like this:

# ausearch -c qemu-ga --raw | audit2allow -M local
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i local.pp
# semodule -i local.pp

With this change in place, we are able to use the --quiesce option when generating snapshots.

Update: upon inspection, this solution actually shows up in the error message included in the bug report:

Jan 16 10:59:49 localhost python: SELinux is preventing /usr/bin/qemu-ga from search access on the directory /var/lib/docker.#012#012***** Plugin catchall (100. confidence) suggests **************************#012#012If you believe that qemu-ga should be allowed search access on the docker directory by default.#012Then you should report this as a bug.#012You can generate a local policy module to allow this access.#012Do#012allow this access for now by executing:#012# ausearch -c 'qemu-ga' --raw | audit2allow -M my-qemuga#012# semodule -i my-qemuga.pp#012

If you look at the end of the error there, it says:

You can generate a local policy module to allow this access.
allow this access for now by executing:
# ausearch -c 'qemu-ga' --raw | audit2allow -M my-qemuga
# semodule -i my-qemuga.pp

...which is exactly what I wrote here.

Related Topic