For anybody else who ends up suffering the error 15 grief that I did, it turns out that the device naming scheme in grub (hd0, hd1, hd2...) ended up being different between when grub boots and when grub is running after the system is up and running.
I spent a week with root (hd2,0) because that's what grub told me the drive I wanted was called.
But when I dropped to the grub shell on bootup I was surprised to find out that what was hd2 when the machine is up, is hd1 on boot. So I changed the menu.lst to use root (hd1,0) and it started working.
I hope to save somebody else lots of hair pulling with that one.
You have your "backup" script being executed by cron... and you shut down cron in the script in order to prevent cron jobs from running during the "backup". You really can't see where is the problem here? Your script shuts down crond, but crond is running your script, so, shutting down crond will close the descriptors connected to your script, which will then die, either with a broken pipe or by a interruption signal from crond itself.
Since the script died, crond won't be restarted anymore. That is what we call "shot yourself on the foot".
Even after restarting crond, it won't have registered that the job completed, since it was shutdown during its execution and/or had to signal its termination. Either crond itself or anacron (depends on what cron scheduler you are using), it will have to run the job again, potentially going into an infinite loop.
Your problem is an excellent example of everything that is wrong with inventing your own "backup" solution if you have no real-life experience with reliability management and disaster recovery. Worse, a lack of knowledge of how the system works.
First, and most important, you do not make a raw disk dump on a live filesystem. Filesystems were invented so that you do not touch the raw disk contents directly. You want to save the files stored in the filesystem, that is what matters for you. So you have to access them through the filesystem, not the raw bytes stored on the disk. If the partition is mounted, there is absolute no guarantee that your data is actually stored on the disk and that the disk will stay in a consistent state during the copy.
Even if you could snapshot the state of the disk in a recoverable manner (like a sudden power failure, which could be quickly recoverable with a journaled filesystem like ext3), that is never true with a hot disk dump. A disk dump takes a long time to complete, there are virtually infinite intermediate states between the beginning and the end of the dump, and the dump will contain a mixture of these states, which is potentially unrecoverable even with a journaled filesystem.
And I still didn't mention everything else that is wrong with raw disk dump backups:
- There is no difference between used and free space. It doesn't matter if you have a single 100 kB file or 250 GB in tens of thousand files, everything will be copied. It is extremely inefficient. You use this approach only if you need an identical clone of your disk, and with the disk unmounted.
- You can't do differential or incremental backups. All your backups must be full backups. All kinds of inefficiencies:
- Since this takes a lot of space, you usually will keep only a single copy of all the data. If your files are damaged or deleted before the backup and you don't notice, the damaged or deleted data is copied over the previous backup, making it useless.
- Since you do this over the previous data, if your system fails in the middle of the dump (which takes a longer time since you are copying the whole disk), both your original system and the backup are lost on a single shot.
- If 100 kB of data changed since the previous backup, you will still dump the whole disk. In your case, this is at least a million times less efficient.
- You can't restore this dump to a disk with a different geometry. If your replacement disk is smaller there is no discussion; if your replacement disk is bigger, you may be able to restore either losing the extra space or doing some manual (and dangerous for the uninitiated) changes to the partition table and partition superblocks. Do you want to trust your files, your work, to such a hack?
- Even if you mount the raw image using a loop device and copy the files manually... you end up copying your files manually!! So what you earned from doing a raw disk dump?? Just copy your damn files!
Many people have been there and have a lot of experience to share regarding disaster recovery. Don't try to invent your own backup solution, you will end messing things up. Use proper backup tooks, like dump, tar or rsync. If you need something more robust, use Amanda or Bacula, or one of the other hundreds of solutions ready to use.
Probably not the answer you were expecting, but had to be said.
Best Answer
If you're already using a virtualization solution, why are you going through all this trouble? The simplest solution would be to use something like LVM on your host, which already has snapshot support, and present a snapshot to your VM. That is:
If you want to revert back to the clean filesystem, delete the snapshot and recreate it. Easy, and no fiddling about with initrd or aufs -- in fact, it doesn't require any modifications to the guest OS.