Centos – Filesystem is read-write but acts like read-only… remounting fixed it

centosfilesystemsread-only

CentOS 5.5

I just came across an instance where my /storage directory was supposedly a read-only file system. I get the error:

mkdir: cannot create directory `test': Read-only file system

df:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      141G   21G  114G  16% /
/dev/sda1              99M   13M   82M  14% /boot
tmpfs                1001M     0 1001M   0% /dev/shm
/dev/sdb1             1.4T  307G  999G  24% /storage
/dev/sdd1             1.4T  329G  978G  26% /media/hddblack
/dev/sdc1             1.4T  329G  978G  26% /media/hddsilver

mount:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /storage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdd1 on /media/hddblack type ext3 (ro)
/dev/sdc1 on /media/hddsilver type ext3 (ro)

/storage is clearly a read-write file system. Next I tried this:

mount -o remount,rw /dev/disk/by-label/storage /storage

And now, the filesystem is writable again. Hooray. Except, why did it do this? Here is the mount now:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /storage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdd1 on /media/hddblack type ext3 (ro)
/dev/sdc1 on /media/hddsilver type ext3 (ro)

Pretty much the same thing… any ideas what would cause this or what logs I should look into to find what the problem was?

Best Answer

Most likely you had a disk IO error. Check /var/log/messages. Look for messages about aborting the journal and re-mounting read-only.

P.S.: If that is the case you best have backups of your data.