Linux – quick way to get the very last file in a large TAR

linuxtar

Let's assume I have a several gigabyte tar file, but I also happen to know that the very last file written to the archive is something important that I need. Since tar files are appended sequentially, is there a way I can make tar read into the archive from the end to find this file, instead of starting from the beginning and reading over gigabytes of irrelevant data?

Best Answer

No, unfortunately there is not. From Wikipedia

Another weakness of the tar format compared to other archive formats is that there is no centralized location for the information about the contents of the file (a "table of contents" of sorts). So to list the names of the files that are in the archive, one must read through the entire archive and look for places where files start. Also, to extract one small file from the archive, instead of being able to lookup the offset in a table and go directly to that location, like other archive formats, with tar, one has to read through the entire archive, looking for the place where the desired file starts. For large tar archives, this causes a big performance penalty, making tar archives unsuitable for situations that often require random access of individual files.

Related Solutions

Tar File Checksum – Creating a Tar File with Checksums Included

Before going ahead and rewriting tar, you may want to profile the quick-and-easy method of reading the data twice, as it may not be much slower than doing it in one pass.

The two pass method is implented here:

http://www.g-loaded.eu/2007/12/01/veritar-verify-checksums-of-files-within-a-tar-archive/

with the one-liner:

  tar -cvpf mybackup.tar myfiles/| xargs -I '{}' sh -c "test -f '{}' && 
  md5sum '{}'" | tee mybackup.md5

While its true that md5sum is reading each file from disk in parallel with tar, instead of getting the data streamed through the pipe, Linux disk cacheing should make this second read a simple read from a memory buffer, which shouldn't really be slower than a stdin read. You just need to make sure you have enough space in your disk cache to store enough of each file that the 2nd reader is always reading from the cache and not getting far enough behind to have to retrieve from disk

Force tar to ignore/skip errors while compressing files

Your assumption is correct, "File changed as we read it" is a notice, usually related to files in use (i.e. written to during the creation process) while tar is creating the archive. If consistency is vital, you're better off rsyncing the contents elsewhere i.e.

rsync -avz /my/home/ /somebackupdir/my/home/  # initial sync, followed by 
rsync -avz /my/home/ /somebackupdir/my/home/  # any subsequent sync, repeated
                                              # as often as you feel necessary

This gives you the benefit of having a backup location that will only need to update the diffs before creating the tarball.

Best Answer

Related Solutions

Tar File Checksum – Creating a Tar File with Checksums Included

Force tar to ignore/skip errors while compressing files

Related Topic