UNIX/Linux sparse files — in use? How to handle these

backupfilesfilesystems

Since the beginning the standard UNIX/Linux systems support sparse files, this is a file which contains unused space that is unallocated until needed. To review, to generate via a C program: create a file, position to 2G, write ONE byte, close file. Doing an ls -l shows the size to be 2G….however ls -ls shows the size in blocks to be closer to a one byte file. If you logically access the file (i.e. cp sparse_file xxx) the resulting file xxx will indeed contain a fully allocated 2Gbytes.

I have created sparse files in the past as a testing vehicle for some of applications. However, their existence has caused a few problems.

The important problem is that outside of the 'dump' program, backup programs and general procedures access these type of files logically and thus for a 1 byte sparse file one gets a backup w/ 2G of 0'd data. This has caused some upset backup folks when I do this.

Any good solutions for this type of situation?

Best Answer

GNU Tar has the --sparse (-S) options that make working with spares files simple.

Related Solutions

Linux/Unix – SSL Certificate Location Guide

For system-wide use, OpenSSL should provide you /etc/ssl/certs and /etc/ssl/private. The latter of which will be restricted 700 to root:root.

If you have an application that doesn’t perform initial privilege separation from root, then it might suit you to locate them somewhere local to the application with the relevantly restricted ownership and permissions.

Linux – RHEL5: Can’t create sparse file bigger than 256GB in tmpfs

The answer is found in the Linux source, specifically, /usr/src/linux/mm/shmem.c, starting around line 70 on my system (Gentoo 2.6.31-ish):

/*
 * The maximum size of a shmem/tmpfs file is limited by the maximum size of
 * its triple-indirect swap vector - see illustration at shmem_swp_entry().
 *
 * With 4kB page size, maximum file size is just over 2TB on a 32-bit kernel,
 * but one eighth of that on a 64-bit kernel.  With 8kB page size, maximum
 * file size is just over 4TB on a 64-bit kernel, but 16TB on a 32-bit kernel,
 * MAX_LFS_FILESIZE being then more restrictive than swap vector layout.

One-eighth of 2 TB is exactly 256 GB. Larger sizes are possible with a 32-bit kernel, as you discovered with your 32-bit FC6 test system.

It appears that changing the page size may be related to enabling HugeTLB filesystem support in the kernel. However, I don't know enough about the guts of the kernel to say how or why, or what steps you need to take to take advantage of it, or what other implications it might have. To enable it, run make menuconfig, navigate to File systems, and then Pseudo filesystems. The option in question is HugeTLB file system support. The online help for it says:

CONFIG_HUGETLBFS:

hugetlbfs is a filesystem backing for HugeTLB pages, based on
ramfs. For architectures that support it, say Y here and read
<file:Documentation/vm/hugetlbpage.txt> for details.

If unsure, say N.

It might be worth running this by StackOverflow, too. I hope this helps.

Best Answer

Related Solutions

Linux/Unix – SSL Certificate Location Guide

Linux – RHEL5: Can’t create sparse file bigger than 256GB in tmpfs

Related Topic