Linux – rsync (and scp) failing with no space left on XFS device

centoscentos7linuxxfs

I have a fresh install of CentOS 7 with a primary partition of 400GB formatted for XFS. Less than one GB is currently in use. I am trying to rsync 65GB of files over and the copy starts failing between 14GB and 26GB. (I have tried multiple times.)

I have plenty of space:

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      400G  919M  400G   1% /
devtmpfs        7.3G     0  7.3G   0% /dev
tmpfs           7.2G     0  7.2G   0% /dev/shm
tmpfs           7.2G   41M  7.2G   1% /run
tmpfs           7.2G     0  7.2G   0% /sys/fs/cgroup
/dev/xvdb        37G   49M   35G   1% /mnt

I have plenty of inodes:

# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       400M   24K  400M    1% /
devtmpfs         1.9M   289  1.9M    1% /dev
tmpfs            1.8M     1  1.8M    1% /dev/shm
tmpfs            1.8M   280  1.8M    1% /run
tmpfs            1.8M    13  1.8M    1% /sys/fs/cgroup
/dev/xvdb        2.4M    11  2.4M    1% /mnt

I don't know what this means, but I have seen it referenced in other places, so I thought it would be helpful in diagnosing:

# xfs_db -r "-c freesp -s" /dev/xvda1
   from      to extents  blocks    pct
      1       1      82      82   0.00
      2       3      14      30   0.00
      4       7      12      63   0.00
      8      15      13     131   0.00
     16      31      17     369   0.00
     32      63       9     408   0.00
     64     127       8     774   0.00
    128     255       9    1702   0.00
    256     511       4    1605   0.00
    512    1023       6    4267   0.00
   1024    2047       1    1664   0.00
   2048    4095       2    7446   0.01
   8192   16383       2   24221   0.02
  32768   65535       1   33389   0.03
 131072  262143       1  205592   0.20
 262144  524224     200 104345607  99.73
total free extents 381
total free blocks 104627350
average free extent size 274612

I have no idea what is left to check, or why it keeps failing. The quoted sections above are from before the copy. After the copy, inodes are at 47K.

Thank you in advance for any help you can provide.

Edit:
Using CentOS 7, not CentOS 6.7. Also, the kernel version:

# uname -r
3.10.0-123.8.1.el7.x86_64

Edit 2:

Here are the results after another attempt (22GB/65GB completed before failure):

# touch test
touch: cannot touch ‘test’: No space left on device
# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      400G   22G  379G   6% /
devtmpfs        7.3G     0  7.3G   0% /dev
tmpfs           7.2G     0  7.2G   0% /dev/shm
tmpfs           7.2G   41M  7.2G   1% /run
tmpfs           7.2G     0  7.2G   0% /sys/fs/cgroup
/dev/xvdb        37G   49M   35G   1% /mnt
# df -ih
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       400M   47K  400M    1% /
devtmpfs         1.9M   289  1.9M    1% /dev
tmpfs            1.8M     1  1.8M    1% /dev/shm
tmpfs            1.8M   280  1.8M    1% /run
tmpfs            1.8M    13  1.8M    1% /sys/fs/cgroup
/dev/xvdb        2.4M    11  2.4M    1% /mnt
# xfs_db -r "-c freesp -s" /dev/xvda1
   from      to extents  blocks    pct
      1       1      60      60   0.00
     64     127       1      81   0.00
    128     255       1     186   0.00
    512    1023       3    2317   0.00
   1024    2047       2    2538   0.00
   2048    4095       1    2310   0.00
   4096    8191       1    8001   0.01
   8192   16383       2   25782   0.03
 262144  524224     189 99059740  99.96
total free extents 260
total free blocks 99101015
average free extent size 381158

Even as root I am unable to create any new files or directories.

Edit 3:
Additional requested information:

# xfs_info /dev/xvda1
meta-data=/dev/xvda1             isize=256    agcount=201, agsize=524224 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=104855999, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Best Answer

Given what you've posted, there are only a few more places in XFS that can cause this problem. Here are two:

  • Quotas. These aren't exactly easy to accidentally turn on, but it's another area worth checking if nothing else yields results. Look for a uquota mount-option. If you still have suspicions, noquota as an option will tell everything to stop quotaing.
  • Mount options. You should have inode64 in your mount options, which is the mount-flag to tell the kernel to use 64-bit inode allocation. If you don't have it, the kernel could throw ENOSPC (out of space) when attempting to create new files and directories. Your partition isn't big enough for this to be a thing, but we're out of easy things to check at this point.

There is one other thing to check, outside of XFS.

  • Open File-Handle Ulimit This might be what rsync is running into. If it was memory, you would be seeing a different error.
Related Topic