Deduplication (Delta Compression) Settings

Deduplication, also known as delta compression, is a process that reduces redundancy when backing up a large file on a regular basis. Delta compression algorithms cut down storage usage and backup time by detecting the change that occurred between backup cycles.

In order to turn off deduplication globally, uncheck “Activate Deduplication (Delta Compression)” in the Deduplication tab.

The “delta type” setting switches between incremental and differential backup deduplication.

Delta block size is the granularity by which deduplication works, basically the smallest detectable change. If a large size is used, it speeds up the process and reduces overhead but wastes more storage space. A small block size is more space efficient but involves more overhead and hence takes more time to complete.

The number of deduplication workers controls parallel execution of the deduplication process. You can get significant speed gains by allowing more than one CPU core to deduplicate a single file. However, RAM and CPU consumption will rise significantly as more workers are deployed.

What is Incremental Deduplication?

Incremental deduplication is a fundamental backup term and refers to a backup chain that starts with a full copy, followed by increments. The increments are based on a file content comparison between the current file copy and the copy from the previous backup cycle. Hence, each successive increment depends on the previous one.

Advantages: Fast and efficient backups, increments are as small as possible.

Disadvantages: Each increment requires its own step when restoring; hence, restore takes longer than with differential deduplication. In addition, access to all increments as well as the last full copy is needed when restoring.

What is Differential Deduplication?

Similar to incremental deduplication, differential deduplication starts off with a full copy of the file. Then, increments are generated with each backup cycle but all refer to the last available full copy of the file. The difference to incremental deduplication is that the current file version is compared to the last full copy of the file. While incremental deduplication creates a backup chain (full copy, followed by interdependent increments), differential deduplication creates a star relationship: each differential depends on the last full copy, not the last differential.

Advantages: Faster restore time, since only the last full copy and the last differential delta are needed. The restore process is hence a one or max. two step process.

Disadvantages: Differentials tend to get bigger after each backup because the percentage of change between the last full copy and the current backup cycle (usually) grows larger.

Why and How to Create Intermittent Full Copies

There is no right or wrong answer and no strategy suits all situations.

The setting ‘Every X cycles, create a full copy’ is a useful remedy to balance the advantages and disadvantages of the incremental and differential backup strategies discussed above.

A setting of “5” creates a full copy every five backup cycles. For example, if your backup runs daily, every 5^th day a full copy will be generated rather than an incremental or differential delta file.

The advantage of this strategy is that it cuts down restore time for incremental backup schemes because there won’t be more than five incremental steps to restore.

The advantage for differential backup schemes is that the differential delta files will be much smaller.

The disadvantage of recreating full copies at regular intervals is that they use more space in the backup store.

It is the responsibility of the administrator / user to balance the above pros and cons and match them to his/her goals and preferences. All strategies have advantages and disadvantages and each setting may be different from another.

For most large files, such as databases and virtual machine images we recommend a setting of “5” to “10”, to keep restore time and backup store usage as short as possible, while providing efficiently small delta files.

Delta Block Size and its Meaning

When comparing files for the purpose of deduplication, the file is processed in chunks for easier processing. Smaller delta blocks lead to a more economic change detection and reduce waste; however, smaller delta blocks require more overhead and management resources; hence, the backup runs slower.

Larger delta blocks significantly reduce the overhead when deduplicating a file and therefore the backups run faster. On the other hand, larger chunks waste more space in the backup store and require more RAM during processing.

To sum up, use smaller deltas to create economic deltas with less RAM but at a slower rate, and use large deltas to speed up your backups, but be informed that large delta blocks require more RAM and create larger delta files. We found a delta block size between 4MB and 16MB suits most applications.

How to Turn off Deduplication for all Files

For each backup task you can completely switch off deduplication by unchecking “Activate Deduplication (Delta Compression)” , see above.

How to Turn on Deduplication for Specific File Types

See above section.

First, ensure the setting “Activate Deduplication (Delta Compression)” (see screenshot above) is checked. Then navigate to the File Versioning / Cleanup tab and check “deduplication” in the table for the file type:

The example above has deduplication switched ON for VHD, AVHD, and VHDX files.

It is recommended to compress files as well when using deduplication, unless you know beforehand that the original file is encrypted or does not compress well, for example if you are dealing with encrypted virtual disk images or encrypted database files.