Hyper-V VM stuck in endless loop trying to merge deleted snapshot with insufficient disk space

hyper-v

I'm currently stuck with a problem of insufficient disk space on a colocated server and could do with some help. The situation is:

  • The server is 'mine' but it's located at a hosting facility so I don't have easy access to it.
  • The server has two physical 250GB disks in a RAID1 configuration (i.e. presenting one 250GB disk)
  • The disk is partitioned into C: (177GB) and E: (54GB)
  • C: has about 25GB free. It might be possible to free up a little over 50GB (total) if needed.
  • The server is Win Server 2008 (not R2)
  • I have several VMs on the server. The problem one is called SVR03.
  • SVR03 has a dynamically-expanding vhd file which is the only file on E:, and is currently taking all the space on that drive: 54GB.
  • SVR03 had a snapshot, which was stored in the normal location on C:\programdata\microsoft\windows\hyper-v\snapshots. We found that this snapshot was taking increasing amounts of space on C: and we didn't want the snapshot, so it was deleted via Hyper-V manager. However, as I understand it this snapshot needs to be merged into the vhd before it's really deleted, and this merge happens on shutdown of the VM.
  • Now after shutting down SVR03 Hyper-V tries to perform that merge of the snapshot with the vhd, but because there's insufficient space on E: it fails. It seems to get to about 50% before failing.
  • Originally it was only getting to about 25% before failing, then I found an extra 2GB unpartitioned space on the drive so expanded the E:, sadly that wasn't enough extra space and the vhd has expanded to take all the space but the snapshot hasn't successfully merged in.
  • After Hyper-V fails to merge it waits a little while and then tries again.
  • I'm unable to compact the vhd for SRV03 because it's in use from Hyper-V trying to merge it.
  • I am able to start SVR03 back up again, but the same issue occurs when I shut down.

How can I get out of this situation?

Is there a way to tell Hyper-V to not merge the snapshot for now, so I can move the files around? e.g. I could move the vhd file elsewhere so there is more space (either by clearing space on C: or by turning off RAID and creating a new F: with 250GB). If I were to move the vhd file will that be ok or is the snapshot somehow linked to the current location of the file?

Is there a way to un-delete the snapshot so I can shut down SVR03 and then move the vhd?

Many thanks!

Best Answer

Turning RAID off seems like a Bad Idea™, but you're pretty well suck unless you can plug a USB drive into the server or something like that. You can shutdown the machine, cancel the merge, the grab the vhd and avhd files. Once you move them Hyper-V will no longer merge them automatically, you'll have to do it manually. Once it's merged move the vhd back to the original location and Hyper-V will automatically pickup that the avhd file is gone.

Related Topic