If the inode count is your actual issue, you can increase the amount of available inodes for the tmpfs filesystem with the nr_inodes mount option. If you set nr_inodes=0 , then there will be unlimited inodes.
All this information is in the tmpfs kernel documentation.
See womble's answer for remount example. For boot, you will need to edit your fstab, or whatever does this for your particular Linux build so future mounts are handled correctly.
AFAIK, find
searches the filesystem's directories. If that file was deleted but still existing because it's open (a common trick on unix), it won't be found by find
.
I haven't tried in Solaris, but here is a note about using lsof
to identify such 'deleted but open' files, and recovering via a cat /proc/<procid>/fd/<fdid> > /tmp/xxxx
Edit:
it seems you've already identified this is the case, but still wondering how is it possible. here's a short explanation:
on POSIX filesystem's, files are handled by its inode
, and the directories are little more than a "path => inode" mapping. You can have more than one path 'pointing' to the same inode (it's called a hardlink), and the inode keeps a count of how many links it has. The rm
command simply calls unlink()
on this path, which reduces the link count and 'possibly' deletes the file itself.
But a path on the directory tree isn't the only possible reference to an inode, an open fd
on a running process also counts, and a 'deleted' file won't be really removed until it goes to 0.
As i mentioned in passing above, it's a common trick: if you have a temporary file that you don't care to keep after your process finishes running, just open it and immediately 'delete' it. The opened handle will work reliably, and when your process finishes (either normally, killed or crashing), the system will remove the handle and cleanly delete the temporary file.
A logfile isn't a likely candidate for such a 'hidden autodeleting' file; but it's not hard to do accidentally.
Since your deleted logfile is still live and collecting data, it seems that simply copying the content wouldn't help much. so try creating a new hardlink to the /proc//fd/ file, something like ln /proc/4366/fd/1 /tmp/xxxx
. Note there's no -s
flag, so ln
should create a new hardlink with the same inode as the original, not a symbolic link (which is little more than a pointer to an existing path, and not what you want).
Edit:
The ln /proc/... /tmp/...
command can't work because /proc and /tmp are in different filesystems. Unfortunately, I don't know any way to create a pathname for an existing inode. One would want that the link()
syscall would take an inode number and a path, but it takes source and destination paths.
Best Answer
Check is your version of e2fsprogs (e.g.
e2fsck -V
). There was apparently a bug prior to version 1.40.2 that could cause e2fsck (which is called by fsck for EXT2/EXT3 filesystems) to incorrectly report filesystem corruption of this type:The RHEL 4 servers that I have access to have version 1.35 on them, so this could be the real cause of your problem. It might be worth installing a newer version and running fsck again to be sure.
If that doesn't help, it may indeed be a filesystem problem. According to this thread, it appears that this can happen if you have a directory containing a very large number of files (millions or more). "Fixing" it with fsck will result in the inode being truncated, as you saw. In the case of a directory inode, truncation will cause the directory to "lose" files, and those orphaned files will end up in the filesystem's lost+found directory. They will be named after their inode number, not their original names, so although you'll still have the files, it may be difficult to tell them apart.