Linux – “No space left on device” error when writing certain session filenames

apache-2.2ext3filesystemslinuxPHP

I have some filenames (session files) that can't be written ("No space left on device"). Other filenames (same directory) are fine. Disk is not full. The filesystem is ext3

PHP is giving an error when trying to create the files, but the error can be reproduced on the command line:

# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory
# touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
touch: cannot touch `/path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1': No space left on device
# touch /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0
# less /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj0
# ls -al /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1
ls: /path/to/session_data/sess_u2q1pfelfr0jof3mp38jb2eaj1: No such file or directory

Note the different filenames. There doesn't seem to be any pattern, but it does affect certain filenames only, and they don't seem to be ever able to be written to.

To possibly complicate things further, this is an OpenVZ server, but you can't write to/create those filenames from either the virtual server or the hardware node.

There are a lot of files in that directory (18,456,002 at the time of writing), but there is no inode problem, and the disk is definitely not full.

As requested, df outputs:

[root@web1 session_data]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/simfs            2.7T  2.1T  495G  81% /
/dev/root             2.7T  2.1T  495G  81% /path/to/session_data
[root@web1 session_data]# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/simfs           726761472 22843560 703917912    4% /
/dev/root            726761472 22843560 703917912    4% /path/to/session_data

(I've edited the path to the session data, but it really is mounted separately like that, to enable sharing of session data between multiple virtual servers)

Best Answer

Ext3 does hashing on file/dir-names.
With so many many files in one directory I wouldn't be surprised if you are running in hash-collision issues.
Having all the filenames of the same length and have the same structure makes the likelihood of a collision even worse.

If there is a collision the filename is effectively unusable, as you noticed.

I have no further experience with this sort of thing so I can't give you any more detailed advice.
(We ran into a similar problem in the office once and the resident Linux guru used some tunefs magic to make the problem go away. I never got the exact details what he did. Just that it had to do with hash-collisions on the filenames.)

You can apparently set the hashing behavior via tunefs. There is also a ext3 extension called "dirindex" that really helps dealing with directories containing so many files.

Related Topic