Linux – How to write a sparse Linux (EXT4) disk image without writing gigabytes of zeros

disk-imageext4filesystemslinux

I have a 64 GB Linux disk image with ~50 GB of unused space across the partitions. The file is sparse, so it only takes ~14 GB on disk.

But if I dd the image, it writes the full 64 GB, which takes quite a while.

Is there any way I can do the equivalent of dd if=os.img of=/dev/sdb with this image, without having to write 50 GB of zeros?

Is there any tool that is smart enough to do this i.e. an imaging tool that has an awareness of the EXT4 filesystem?

Best Answer

dd can handle this. You need to add conv=sparse to the command line.

From the man page:

              try to seek rather than write the output for NUL input blocks

Related Solutions

KVM Virtualization – How to Copy KVM Disk Image to LVM

You may simply have an off-by-one math error:

3146776576 (size, in bytes of your .raw file)  / 1024 (kb) / 1024 (mb) == 3001MB

But you made your LV exactly 3000MB ?

There's nothing wrong with DD'ing one partition onto another. Even though your vmbuilder partition list added up to 3000, your .qcow file is a hard drive image, which includes a partition table + mbr @ the front of the disk.

XFS – Why XFS Filesystems Are Consuming More Space and Full of Sparse Files

I traced this issue back to a discussion about a commit to the XFS source tree from December 2010. The patch was introduced in Kernel 2.6.38 (and obviously, later backported into some popular Linux distribution kernels).

The observed fluctuations in disk usage are a result of a new feature; XFS Dynamic Speculative EOF Preallocation.

This is a move to reduce file fragmentation during streaming writes by speculatively allocating space as file sizes increase. The amount of space preallocated per file is dynamic and is primarily a function of the free space available on the filesystem (to preclude running out of space entirely).

It follows this schedule:

freespace       max prealloc size
  >5%             full extent (8GB)
  4-5%             2GB (8GB >> 2)
  3-4%             1GB (8GB >> 3)
  2-3%           512MB (8GB >> 4)
  1-2%           256MB (8GB >> 5)
  <1%            128MB (8GB >> 6)

This is an interesting addition to the filesystem as it may help with some of the massively fragmented files I deal with.

The additional space can be reclaimed temporarily by freeing the pagecache, dentries and inodes with:

sync; echo 3 > /proc/sys/vm/drop_caches

The feature can be disabled entirely by defining an allocsize value during the filesystem mount. The default for XFS is allocsize=64k.

The impact of this change will probably be felt by monitoring/thresholding systems (which is how I caught it), but has also affected database systems and could cause unpredictable or undesired results for thin-provisioned virtual machines and storage arrays (they'll use more space than you expect).

All in all, it caught me off-guard because there was no clear announcement of the filesystem change at the distribution level or even in monitoring the XFS mailing list.

Edit:
Performance on XFS volumes with this feature is drastically improved. I'm seeing consistent < 1% fragmentation on volumes that previously displayed up to 50% fragmentation. Write performance is up globally!

Stats from the same dataset, comparing legacy XFS to the version in EL6.3.

Old:

# xfs_db -r -c frag /dev/cciss/c0d0p9
actual 1874760, ideal 1256876, fragmentation factor 32.96%

New:

# xfs_db -r -c frag /dev/sdb1
actual 1201423, ideal 1190967, fragmentation factor 0.87%

Best Answer

Related Solutions

KVM Virtualization – How to Copy KVM Disk Image to LVM

XFS – Why XFS Filesystems Are Consuming More Space and Full of Sparse Files

Related Topic