Windows deduplication doesn’t yield usable space

deduplicationwindows-server-2012-r2windows-server-2016

Dedup size picture

Hi all,
When I setup Windows depuplication, I get great savings. See the attached picture. 1.28 TB deduped down to 23 GB. However, the space savings are not real.

Lets say that volume is 2TB. According to the screenshot, there is only 28 GB used, so I should be able to add another 1.9TB of data. In reality, once I add ~800GB of data (to hit the real 2TB limit) the volume fills. The attached picture would then say data size 2TB, size on disk 50GB (or something like that).

What is the point of data deduplication if I cannot utilize the space savings? Or, is there some trick to setting this up I'm not seeing. I've had these same results on server 2012r2 and 2016. I've tried on the HyperV level with VHDXs, and on a backup server with large backup files.

Best Answer

The space savings are real. You can check the dedup ratio in Server Manager. Upgrading to 2012R2 and using deduplication has saved us from buying new storage for at least a year.

Don't try to add the logical file sizes and compare them with "Size on Disk" - the latter is only the sum of not deduplicated files/parts. Deduplicated data doesn't show up at all here. The free space of the volume can't be calculated that way, check properties of drive/volume.

This makes sense - when two files have 80% in common, what size on disk does each file occupy?

The background is that deduplicated parts (or whole files) are stored in the deduplication pool. The non-deduplicated parts of a file are stored as a sparse file with reparse points where the deduplicated parts are to be.

The dedup pool hides inside the System Volume Information folder - if you check its size you know what amount of deduplicated/compressed is stored on the drive. Add that to Size on Disk of your files and you'll be pretty close to actual volume utilization.

There's a pretty good primer from MS for this: https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/understand

Related Topic