How to verify that a deduplication has taken place

deduplicationwindows-server-2012

Microsoft Windows Server 2012 and onwards offers a de-duplication service that periodically scans files, find identical chunks and removes excessive copies to save space.

To the user browsing the files, they should all look the same.

My problem is that I have a piece of software that's reading these files and failing, when it reads a file processed by de-duplication. I set up a windows server with de-duplication service to develop and test a fix for this but I am not sure if my test files are being deduplicated and if my fix is really working.

Is there something in the file metadata about any deduplication taken place? Or perhaps the de-duplication service has an accessible data base with the augmented files?

I have already tried the obvious: create a file, copy that file in the same folder and then view the properties of the folder – but the size of the folder amounts to both files, while I was expecting it to amount to the size of only one file.

Best Answer

Deduplication is implemented as a filter driver on top of NTFS (and now ReFS) and should work transparent. You can always disable it for some particular file sets if it causes issues.

To get deduplication status stick with Get-DedupeStatus cmdlet. See:

https://docs.microsoft.com/en-us/powershell/module/deduplication/get-dedupstatus

There's a way to visualize what's there. See:

https://www.foldersizes.com/features/windowsdeduplicationdiskspace

You can exclude particular files from deduplication jobs. See:

https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/advanced-settings

ExcludeFileType is what you should look at.

ExcludeFileType File types that are excluded from optimization Array of file extensions Some file types, particularly multimedia or files that are already compressed, do not benefit very much from being optimized. This setting allows you to configure which types are excluded.

Related Solutions

How does ZFS Block Level Deduplication fit with Variable Block Size

ZFS deduplication works on blocks (recordlength) it does not know/care about files. Each block is checksummed using sha256 (by default changeable). If the checksum matches an other block it will just reference the same record and no new data will be written. One problem of deduplication with ZFS is that checksums are kept in memory so large pools will require a lot of memory. So you should only apply reduplication when using large record length

Assuming recordlength 128k

If I a randomly filled file of 1GB, then I write a second file that is the same except half way through, I change one of the bytes. Will that file be deduplicated (all except for the changed byte's block?)

Yes only one block will not be duplicated.

If I write a single byte file, will it take a whole 128 kilobytes? If not, will the blocks get larger in the event the file gets longer?

128k will be allocated, if the file size grows above 128k more blocks will be allocated as needed.

If a file takes two 64kilobyte blocks (would this ever happen?), then would an identical file get deduped after taking a single 128 kilobyte block

A file will take 128k the same file will be deduplicated

If a file is shortened, then part of its block would have been ignored, perhaps the data would not be reset to 0x00 bytes. Would a half used block get deduced?

If the exact same block is found yes

How to activate deduplication in Windows Server 2012

You need to install, enable, and configure the dedupe features.

Since there is overhead associated with deduplication, and there are scrubbing and other jobs that need to be scheduled, it's not automagically enabled on a default install.

You can do this from the GUI, or using PowerShell:

Import-Module ServerManager
Add-WindowsFeature -name FS-Data-Deduplication
Import-Module Deduplication

Enable-DedupVolume E:

Please note that the commands listed above are the bare minimum necessary to turn dedupe on. If you want to use this in production, read the entire link and understand all of the other components involved and configure them as necessary.

Best Answer

Related Solutions

How does ZFS Block Level Deduplication fit with Variable Block Size

How to activate deduplication in Windows Server 2012

Related Topic