R – Do any common OS file systems use hashes to avoid storing the same content data more than once

filesystemshashoperating system

Many file storage systems use hashes to avoid duplication of the same file content data (among other reasons), e.g., Git and Dropbox both use SHA256. The file names and dates can be different, but as long as the content gets the same hash generated, it never gets stored more than once.

It seems this would be a sensible thing to do in a OS file system in order to save space. Are there any file systems for Windows or *nix that do this, or is there a good reason why none of them do?

This would, for the most part, eliminate the need for duplicate file finder utilities, because at that point the only space you would be saving would be for the file entry in the file system, which for most users is not enough to matter.

Edit: Arguably this could go on serverfault, but I feel developers are more likely to understand the issues and trade-offs involved.

Best Answer

ZFS supports deduplication since last month: http://blogs.oracle.com/bonwick/en_US/entry/zfs_dedup

Though I wouldn't call this a "common" filesystem (afaik, it is currently only supported by *BSD), it is definitely one worth looking at.

Related Solutions

Git – One way hash (not for crypto/security), use SHA256 (not MD5, SHA-1)

Yes, I would use SHA-256. SHA-256 had a lot more than security purposes in mind; in fact one of the reasons that SHA1 needed to be replaced was for the very reason you need a hash function. A hash algorithm produces a finite site output; while having an undetermined amount of input. Eventually there will be a collision. The larger the output; the less likely of a collision (when using a proper hash algorithm).

Git went with SHA1 because they use it as file names; and they wanted it to be small and compact. SHA256 produces a much larger digest; consuming more disk space and more data to transmit over the wire. This question specifically addresses what would happen if git were to encounter collisions.

To look at your points:

SHA256 has been in the wild long enough that if there were problems; we should have seen them by now.
It isn't "stronger" per-se, it's less likely to produce a collision (if that is your criteria for stronger; then yes it is stronger).
SHA-256 is slower; yes. Much slower? Depends on what your needs are. For 95% of people; it's performance is acceptable assuming you're using a proper implementation.
In general, truncating the hash of SHA2 is an okay thing to do.

Linux command to list file systems available for mounting

On Ubuntu you can use to show discs:

sudo lshw -class disk

or to check all partitions on your system

sudo blkid -o list | grep "not mounted"

or if you just want the device:

sudo blkid -o list | grep "not mounted" | awk '{print $1}'

Best Answer

Related Solutions

Git – One way hash (not for crypto/security), use SHA256 (not MD5, SHA-1)

Linux command to list file systems available for mounting

Related Topic