Do any of the VMs you're trying to back up contain dynamic disks? (That's in disk management within the VM, dynamic disks as opposed to basic disks. Not dynamically expanding VHDs)
To quote Microsoft:
When you perform a backup of the virtual machines, you must back up all volumes that host files for the virtual machine, including the InitialStore.xml file (in C:\ProgramData\Microsoft\Windows\Hyper-V, by default) and the volume(s) containing the VHD(s) and configuration XML files. For example, if the virtual machine configuration files are stored on the D: volume, and the virtual machine virtual hard disk (VHD) files are stored on the E: volume, and InitialStore.xml file is stored on the C: volume, you must back up the C:, D: and E: volumes.
Virtual machines that do not have Integration Services installed will be put in a saved state while the VSS snapshot is created.
Virtual machines that are running operating systems that do not support VSS, such as Microsoft Windows 2000 or Windows XP, will be put in a saved state while the VSS snapshot is created.
Virtual machines that contain dynamic disks must be backed up offline.
Try running the DOS command "vssadmin list writers" and "vss admin list providers". Capture the output of these command, as they will help in debugging the issue. If a particular writer reports any unexpected state, search the web for conditions that may be causing this issue.
A quick way to reset the writers is to reboot the server.
If VSS errors continue to persist, try re-registering VSS. Follow the steps outlined in MS KB article 940184.
Doublecheck for other VSS providers, i.e. any backup software that installs any sort of Open File agent (examples would be St Bernard, older versions of Backup Exec Open File Option, etc). These can be shown by typing "vssadmin list providers" at a command prompt. Providers from uninstalled software can be deleted by removing the offending CLSID of the VSS provider and rebooting the machine.
Is the machine a 64-bit machine? Are you getting "A function call was invalid because of the state of either the backup extensions or the coordinator. For example calling AddToSnapshot set prior to calling StartSnapshotSet." ? If so, try deleting the Subscription key per MS KB 940184
1. Click Start, click Run, type Regedit, and then click OK.
2. Locate and then click the following registry subkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\EventSystem\{26c409cc-ae86-11d1-b616-00805fc79216\Subscriptions
3. On the Edit menu, click Delete, and then click Yes to confirm that you want to delete the subkey.
4. Exit Registry Editor.
5. Reboot the machine
Next you can try running a utility called vshadow. It's part of the windows SDK.
To do a single manual snapshot, run
vshadow [drive letter]
which would take and immediately destroy a VSS snapshot of C: and D:. If that works, VSS is probably OK; if not it's a VSS problem.
All of this is where I would start to troubleshoot VSS problems.
As with anything, be extremely careful when modifying the registry
Best Answer
Not exactly and answer:
Yep, I've done a few. But I always do it offline. PXE boot WinPE, mount a network VHD, and robocopy it over. A quick bcdedit touchup to detect the new HAL and it's good to boot (more driver installation, integration services, etc from there).
Perhaps there's an equivalent process for VMware images?