Where I work, servers are managed by an external company, so when you want a new server you get the root password for a while to set it up, and then they change it and you are supposed to make do with a non-root user.
So a while ago I set up my server, it runs Red Hat Enterprise Linux server 5.4, Apache 2.2.3, Apache Tomcat 5.5.23, and PostgreSQL 8.1.22. I created a user, a database, populated it with data, no problem. They changed the root password and everything seemed to work okay.
A couple of months ago, the server was shutdown for electrical maintenance in the building. I don't know if they shut it down cleanly or not.
As the server is not much used, I only found out last week that the PostgreSQL server didn't start again at reboot. I had to ask them to start it up and when they did, surprise, everything is gone: the user I created, the database…. The Tomcat application itself is still there and working but without a database connection, it doesn't do much.
I managed to get a root password for the time being, so I can look around.
Is there a way to find out what deleted everything? How can I prevent it from happening again?
Everything in /var/lib/pgsql/ is dated the day they restarted the server, in fact everything looks like that day was the first time it was ever started.
Is it possible there are multiple PostgreSQL servers or locations on the machine and the old database is still there somewhere? Or did something wipe out PostgreSQL?
I have a backup so the data itself is not lost, but I would very much like to learn what happened.
Best Answer
Anything special about the disk/lvm configuration on this server?
Was the /var/lib/pgsql/data directory a separate filesystem, maybe on a SAN or just a separate set of disks? If someone had manually mounted data/ without also putting it in fstab you would have a big surprise after your first reboot.