Linux – Diagnosing cause of orphaned inodes on Linux, busy MySQL

corruptionfilesystemsinodelinuxMySQL

One of our servers recently experienced some file system corruption and our root file system was automatically remounted as read only. The steps I took to recover were:

  1. attempted to remount > mount -n -o remount / this failed
  2. rebooted the server
  3. was prompted to perform a manual fsck, there were 5 orphaned inodes which required fixing.

After performing these steps I was able to gain access and the file system was writable again. Unfortunately I don't have any informative logs as none were written or I would have included these.

One cause that was been suggested was that our database was too busy to write the data to disk properly and this caused the issue, the high level of cache memory was given as an indication that this might be the case. However I'm not sure about this as although cache is high we aren't using the swap at all (output of free below).

$ free -m
             total       used       free     shared    buffers     cached
Mem:          2041       1879        162          0         62       1599
-/+ buffers/cache:        216       1825
Swap:          471          0        471

Is there any way I can diagnose the fault after it has happened? Does MySQL look like a likely candidate?

If not are there any steps I should take in the future if this happens again?

Best Answer

First sanity check your server:

  • Are you using ECC memory?
  • Are you running RAID? Did you see any RAID cards errors? (dmesg would have shown these at the time, but now you've rebooted they're probably lost)

A high level of cache is desirable and shouldn't corrupt your file-system in any way.

Related Topic