Ubuntu rm not deleting files

This isn't a permissions problem. The two clues are:

• rm: cannot remove 'admin.php' : Read-only file system
• everything was working until yesterday afternoon. At that point a coworker took out the SAN that the virtual machine file was on, and the web server had a less than graceful shutdown.

Somehow the filesystem containing /var/www dropped to "read only" probably when the SAN went away. The output of the mount command should identify this filesystem with a (ro) flag at the end.

The fix is to figure out why it happened, make sure it's corrected, and remount the filesystem rw with this command:

mount -oremount,rw $filesystem

I'd recommend something that can do incremental backups, because that way you can go back to files that were changed several days ago. We use dirvish for this (it's available from the ubuntu repos) - basically it's a wrapper for rsync written in perl that makes it easier to setup a system to run regular incremental backups, and then expire them based on given rules.

On our system we keep backups for 7 days, but keep the first backup of the week for 1 month, and the first backup of the month for one year, so usually you can find files that were modified and deleted some time ago. For things that are small, but important, we have much longer expiry times.

As for what you should backup, imo if you have the space back up everything unless you're sure you don't need to back up. With incremental rsync backups you're only copying over the files that have changed since the last backup - I'd rather have an extra 3gb of static data belonging to the operating system on my backup drive than find it was missing one important file when I needed to recover it.

At a minimum I'd backup the entire contents of /etc/, and most of /var depending on what stuff was running on the server.

If your mysql database is fairly small then your current approach is probably acceptable, but since you need to lock the database to generate the dump this isn't practical on large databases. Because of this lots of people run a slave mysql server, which is just a read only copy of the main one on the backup server, and use this to generate backups.

The mysql documentation on how to do this is fairly good, I worked through it the first time in an evening, without much previous mysql experience.

We have our slave mysql server tied into dirvish, so each day it generates a dump of each individual database, as well as an entire copy of /var/lib/mysql (it's stops the mysql server before doing this)