Linux – ext4 says no space left on device but df reports 16MB free space

We are using ext4 filesystems to store data in only one big file.

We are having a strange issue : df reports free space (16MB), dumpe2fs reports 4096 free blocks (of 4KB, matches what df reports), but when trying to append data to our file the system reports No space left on device.

[root@localhost raw_2]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdd1        11G   11G   16M 100% /opt/HIDDEN/storage/raw_2
[root@localhost raw_2]# df -i .
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sdd1       11264    12 11252    1% /opt/HIDDEN/storage/raw_2
[root@localhost raw_2]# echo test > testfile
-bash: echo: write error: No space left on device
[root@localhost storage]# umount /dev/sdd1
[root@localhost storage]# dumpe2fs -h /dev/sdd1
dumpe2fs 1.41.12 (17-May-2010)
Filesystem volume name:   <none>
Last mounted on:          /opt/HIDDEN/storage/raw_2
Filesystem UUID:          9d6ca417-0854-461d-993d-e23c2a2229d4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              11264
Block count:              2883580
Reserved block count:     0
Free blocks:              4096
Free inodes:              11251
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      703
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         128
Inode blocks per group:   8
Flex block group size:    2048
Filesystem created:       Tue Nov 15 11:32:52 2016
Last mount time:          Tue Dec  6 14:14:57 2016
Last write time:          Tue Dec  6 15:16:54 2016
Mount count:              8
Maximum mount count:      32
Last checked:             Tue Nov 15 11:32:52 2016
Check interval:           15552000 (6 months)
Next check after:         Sun May 14 12:32:52 2017
Lifetime writes:          3572 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      dc8da308-ef45-4bea-b156-5690eb399646
Journal backup:           inode blocks
Journal features:         (none)
Journal size:             128M
Journal length:           32768
Journal sequence:         0x0004a45b
Journal start:            0

This is not due to reserved blocks for root and obviously not related to free inodes.

I have read a lot of discussions but found nothing that could apply to our case.

This problem happens on a VM with 11GB virtual disks. We have also seen this same behavior when using 10GB files mounted as loopback FS. We are using RedHat 6 (linux 2.6.32).

Our software is running as root. I also tried a simple cat /dev/zero > testfile, it stopped at the same limit, leaving 16MB free space.

Any help would be appreciated.

EDIT : using statfs on 2 different systems shows a strange behavior.

On my original system, statfs does not report any difference f_bfree==f_bavail :

statfs("/opt/HIDDEN/storage/raw_1/", {f_type="EXT2_SUPER_MAGIC", f_bsize=4096, f_blocks=2882680, f_bfree=2665715, f_bavail=2665715, f_files=11264, f_ffree=11251, f_fsid={-2032429898, 1911081970}, f_namelen=255, f_frsize=4096}) = 0

On a debian testing box (linux 4.8.0) :

statfs("/home/", {f_type=EXT2_SUPER_MAGIC, f_bsize=4096, f_blocks=55137925, f_bfree=26426280, f_bavail=26422184, f_files=10114944, f_ffree=9685471, f_fsid={1010187930, 3946494061}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0

This fs has no reserved blocks, but there is a difference of 4096 blocks between f_bavail (free blocks available to unprivileged user) and f_bfree (free blocks in fs). Does anybody know what are these 4096 reserved blocks ?

Thanks.