Here is the situation:
We have a file server that is not saving timestamps
to a small portion of its files (77 files out of several million). It instead writes a default mtime of 1969-12-31 19:00 (Unix epoch).
When this does happen, it happens irregularly with no set pattern (ie. one file here, a group of files in the same
directory there. etc.) The files that are timestamped this way have been uploaded to the server through NFS.
It should be noted that we are only seeing this happen in an NFS share that is on this server and it has not happened to any of the servers system files, it seems isolated to just the NFS share.
Some additional info:
- the server is running Ubuntu Server 9.04
- NFSv4 for file serving. Here is the exports line:
/home/file_storage 10.0.0.1/24(rw,sync,no_root_squash,no_subtree_check) - We recently migrated our files to this server using rsync from our old file server
- This server has a hardware RAID 5 array using a 3ware 9650SE controller.
- Filesystem is ext3 with LVM and LUKS/DMCRYPT on a single root partition
- The server is regularly syncing its time with ntpd.
Update
Updated the filesystem and partitioning info above as requested in comments.
Best Answer
Are the only mount point on the raid controller /home/file_storage? In other words, are the other file systems on that server on the same raid controller or on a different controller?
If the only thing on the raid controller is the nfs share, then I'd worry that there are blocks of 0 getting written, and you see them as the date zeroed. It might be worthwhile to run a bunch of md5sums across the different files, and then run them a week later and see which ones have changed.
If, on the other hand, you have a lot of file systems on the same raid controller, and the only one messing up is the nfs shared one, then it could be that network packets are getting corrupted. We have seen in the past places where files have random 8k blocks of 0s in them. Ie, everyone thought that nfs block was written, but, in fact, it wasn't. This probably is a network problem and you should see errors on the interface then.
I'd go for the first one though.