Xen server 6.2: lost a VM when moving

Is it possible the VM has been hacked? Are the tools you're using to monitor the list of processes and memory from a known-good, read-only medium? You could have a rootkit installed which hides the leaky processes.

Okay, I think the answer is that the COW space for the logical volume is full.

Using the command 'lvs' (which I just discovered), I see...

# lvs
/dev/dm-20: read failed after 0 of 4096 at 0: Input/output error
LV             VG      Attr   LSize   Origin          Snap%  Move Log Copy%  Convert
[...other LVs...]
newvm-cdrive   mrburns Swi-I-   2.00G original-cdrive 100.00
[...other LVs...]

That capital 'S' at the start of the 'Attr' column means 'invalid Snapshot'. (A lower-case 's' would mean (valid) snapshot.) And as you can see, Snap% is 100, i.e., it's used all of its COW space.

Annoyingly, lvdisplay doesn't provide this information, and it doesn't tell you that your snapshot logical volume is invalid. (All it says is that the snapshot status is 'INACTIVE', which I took as meaning 'not currently in use'.) And the lvs command is not very widely advertised. And the error message ("Input/output error") isn't very helpful--in fact there were no log messages or error messages which suggested 'snapshot is full'. (Later versions of LVM2 write messages to /var/log/messages when the space is starting to fill up, but the version in Debian Lenny doesn't. Boo.)

And to compound the problem, there's no discussion of this on the internet (or at least, not that I could find)!

I did wonder why COW snapshots can't be fixed by just adding more space to the LV (using lvextend, but, actually, the COW space will be required not only when you write to the snapshot destination, but also when you write to the snapshot source. So once your COW area is filled up, any writes to the source LV must necessarily make the snapshot LV invalid, and not easily recoverable.