We have an internal web server (virtualized, hosting ReviewBoard, but not super relevant) and we have a relatively consistent failure mode with failed NFS mounts causing / to fill up. Distro is Ubuntu (don't ask) if a solution depends on a different distribution, it will be slower to implement.
Backups are being performed to /mnt/backup/, which is supposed to an NFS mount to another system. Unfortunately, when the mount fails, or drops off, backups get performed on the root filesystem, which as you can imagine doesn't take long before / is full, and then services start to fail.
A number possible solutions have been discussed.
-
Monitor /mnt/backups and ensure it's not root. Perhaps a cron job.
-
Use /mnt/protected/backups, and mount /protected first to a small filesystem, perhaps a loop mount to a local file so it is much less likely to fail.
-
Chmod a-rwx /mnt/backups (the root filesystem mount point). I'm not sure if mounting over protected director will work, I think it does.
-
On the mounted tree create a directory called "Backups", then soft link "ln – s /mnt/backup/Backups /Backups". Using /Backups for backups will fail unless the /mnt/backup is mounted, since the local tree doesn't contain the sub-directory.
-
Performing a check that the directory is correctly mounted in the backup script.
I'm interested in any feedback on these approaches, pros cons or any additional techniques people use as a standard way of protecting the root file system from this type of nastiness.
Best Answer
Number 5 - Put a test in your backup script to ensure that the directory is mounted before continuing. The script should fail if the mount is not available or present. Or you can just make sure things are mounted prior to running the backup.
Try the
mountpointcommand, which checks if a specified directory is a mountpoint:mountpoint -q /mnt/backups || mount /mnt/backups