I'm trying to write reference material for a friend who has power outages a couple of times a year which affect their linux server. This person is not well versed in linux systems administration, so I want to include screenshots and a fairly comprehensive description of what to expect, and how to respond.
One of the more common questions that comes up (a couple times a year; they're in a lightning-prone area) is how to recover from a corrupt or missing superblock in the root filesystem.
I have documentation on how to fix the problem, but it assumes you have a nodding acquaintance with linux and console-based systems administration. What I'd like to do is setup a KVM on my server, crash the root filesystem on purpose, and take screenshots of the initial discovery along with the steps to recover.
I cannot seem to trigger that missing/corrupt superblock issue. I've been trying things like:
(/dev/sda2 is where the root LVM partition lives)
dd if=/dev/zero of=/dev/sda2 bs=1024 count=1 seek=1
(/dev/mapper/vg_sys-lv_root is the root LVM partition)
dd if=/dev/zero of=/dev/mapper/vg_sys-lv_root bs=4096 count=1
Neither of these work; they both seem to create mayhem, but in very different ways from the run-of-the-mill missing/corrupt superblock.
Is there any way to simulate this scenario on a KVM guest (i.e. without setting up a machine and then pulling the power cord out of it)?
Best Answer
Make sure you're deleting all superblocks, including the backup superblocks
dumpe2fs /dev/path_to_your_volume | grep -i superblockFrom there, you can kick all the superblocks down like so many tinkertoys.