Linux – How to Delete 100GB File Without Thrashing IO/Load

Your dmesg output should give you an indication as to why it's signalling failure to the PV; that shouldn't be happening. As far as getting the system writable again, kicking the VG and LV to read-only and then back to read-write works from memory, but the real resolution is getting md to stop worrying LVM unnecessarily.

If you're using ext3, it's quite possible that only the metadata is journaled, this is the default behaviour of ext3. If you check the manual page for shred:

In the case of ext3 file systems, the above disclaimer applies (and shred is thus of limited effectiveness) only in data=journal mode, which journals file data in addition to just metadata. In both the data=ordered (default) and data=writeback modes, shred works as usual. Ext3 journaling modes can be changed by adding the data=something option to the mount options for a particular file system in the /etc/fstab file, as documented in the mount man page (man mount).

And then, you check your /proc/mounts to see what the mount-flags for your mounted filesystems are.

An example:

$ grep -i data /proc/mounts
/dev/root / ext3 rw,data=ordered 0 0
/dev/sda4 /stash ext3 rw,data=ordered 0 0

On both my /stash and / filesystems, shred should do an adequate job of securely deleting data.