Linux – Need any hints regarding Linux host’s file-system going to read-only mode

centoscitrixlinuxxen

I have a CentOS 5.x Linux guest running on a XEN (Citrix) server, and from time to time it mysteriously goes into the read-only mode.

I've checked the file-system for errors, nothing suspicious appeared… 🙁

The system log says something like:

Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735103
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735191
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735279
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735359
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735447
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735535
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735103
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 309735103
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 307662855
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316647
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316655
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316663
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316671
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 315316735
Mar  5 10:57:16 testsrv last message repeated 38 times
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 307662855
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 262717023
Mar  5 10:57:16 testsrv last message repeated 2 times
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 258482255
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 141041743
Mar  5 10:57:16 testsrv kernel: EXT3-fs error (device xvdb1): ext3_find_entry: reading directory #17629185 offset 0
Mar  5 10:57:16 testsrv kernel: Aborting journal on device xvdb1.
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 16519
Mar  5 10:57:16 testsrv kernel: Buffer I/O error on device xvdb1, logical block 2057
Mar  5 10:57:16 testsrv kernel: lost page write due to I/O error on xvdb1
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 63
Mar  5 10:57:16 testsrv kernel: Buffer I/O error on device xvdb1, logical block 0
Mar  5 10:57:16 testsrv kernel: lost page write due to I/O error on xvdb1
Mar  5 10:57:16 testsrv kernel: end_request: I/O error, dev xvdb, sector 59692223
Mar  5 10:57:16 testsrv last message repeated 33 times
Mar  5 10:57:16 testsrv kernel: ext3_abort called.
Mar  5 10:57:16 testsrv kernel: EXT3-fs error (device xvdb1): ext3_journal_start_sb: Detected aborted journal
Mar  5 10:57:16 testsrv kernel: Remounting filesystem read-only

Best Answer

@DejanLekic: tune2fs etc do only scan the file system for errors. The kernel error messages you observed are about hardware failures, which is one layer beneath the file system. Modern file systems and operating systems are taking hardware failures very seriously; ext* remounts itself as read-only to prevent any write access causing further damage, XFS shuts itself down completely and so on.

So you have some kind of hardware problem. Bad RAID controller, dead hard drive, unreliable SAN, something else.

Related Topic