Why Backups Grow So Fast & How Chunking Helps
Backups grow quickly because most backup tools store every change as a new version, and even small edits can cause large blocks of data to be re-copied. Content-defined chunking (CDC) solves this by splitting files into variable-sized pieces based on the content itself, so only the chunks that actually changed are stored. This article explains why backup bloat happens and how CDC reduces storage and bandwidth usage.
The Core Problem: Whole-File and Fixed-Size Backups
Traditional backup methods fall into two camps, and both waste space:
- Full backups copy every file each time. A 10 GB folder backed up daily for a month uses 300 GB, even if only one file changed.
- Incremental backups copy only changed files, but if a large file (like a database or VM image) is modified, the entire file is re-copied. A 50 GB database that changes 1% each day still consumes 50 GB per backup.
Fixed-size block backup (splitting files into 4 MB chunks, for example) helps, but it fails when a file is edited in the middle or at the beginning. Inserting a single byte shifts every subsequent byte, so every block after the edit has different content, and all of them must be stored again.
How Content-Defined Chunking Works
Content-defined chunking uses a rolling hash to find natural breakpoints in the data stream. Instead of using fixed offsets, the algorithm slides a window (typically 48–64 bytes) across the file and computes a hash of that window. When the hash meets a certain condition (e.g., the last 13 bits are zero), a chunk boundary is declared. The chunk is then stored, and the process restarts.
Because boundaries depend on the data, not on position, an insertion or deletion only affects the chunks near the edit. Everything else remains identical, and deduplication can recognize them.
Typical parameters:
- Minimum chunk size (e.g., 2 KB) – prevents too many tiny chunks.
- Maximum chunk size (e.g., 64 KB) – prevents huge chunks that reduce deduplication.
- Average chunk size (e.g., 8–16 KB) – set by the hash condition probability.
Why CDC Dramatically Reduces Backup Size
Consider a 100 MB virtual machine disk file. With fixed 4 MB blocks, you have 25 blocks. If you change one byte in the middle, all 25 blocks after that point change, so you store 100 MB again. With CDC, you might have ~12,500 chunks of 8 KB average. The edit affects only the chunk containing the byte and possibly the next one (due to boundary shift). You store maybe 16 KB instead of 100 MB – a 99.98% reduction.
In real-world workloads (databases, log files, virtual machines), CDC typically reduces backup storage by 50–90% compared to fixed-block or whole-file methods, depending on how often files are modified.
Deduplication and Compression: The Synergy
CDC is often paired with deduplication and compression:
- Deduplication identifies identical chunks across different files or versions. CDC makes this effective because the same content always produces the same chunk boundaries, even if it appears at different offsets.
- Compression (e.g., LZ4, Zstandard) reduces the size of each chunk further. Compressing after deduplication is more efficient because only unique chunks are compressed.
For example, a backup system might store a chunk only once, then reference it from multiple versions. This is how modern backup tools (like Borg, Restic, and many enterprise products) achieve very low incremental sizes.
Choosing Backup Software with CDC
If you’re selecting a backup tool, look for these features:
- Variable chunk sizes – not just fixed blocks.
- Deduplication across versions – not just within a single backup.
- Encryption before upload – so the provider can’t read your data.
- Configurable chunk size – smaller chunks mean better dedup but more overhead.
Popular open-source tools that use CDC include BorgBackup, Restic, and Kopia. Commercial tools like Veeam and Acronis also use chunking, but often with proprietary algorithms.
Common Misconceptions and Pitfalls
“CDC always makes backups smaller.” Not always. If files are already compressed (e.g., JPEG, ZIP), CDC may not help much because even small changes alter large parts of the compressed stream. In such cases, whole-file or fixed-block backups might be just as good.
“Chunk size doesn’t matter.” It does. Too small chunks increase metadata overhead and slow down processing. Too large chunks reduce deduplication efficiency. A good rule of thumb is an average of 8–16 KB for general files, and larger (64 KB–1 MB) for very large files like VM images.
“CDC is only for enterprise.” Many free tools implement it, so even home users can benefit.
Cost and Performance Trade-offs
CDC is CPU-intensive because it hashes every byte of the file. On modern hardware, it can process 100–500 MB/s per core, which is usually fine for nightly backups. However, for very large data sets, the overhead can be noticeable.
Storage savings can be substantial: if you have 1 TB of data that changes 10% per day, a full backup every day would use 30 TB per month. With CDC and deduplication, you might store only the initial 1 TB plus 100 GB per day, totaling ~4 TB per month – a 87% saving.
Bandwidth savings are similar for remote backups, which is often the bigger cost.
Alternatives to CDC
- Fixed-size block backup – simpler, but fails on insertions/deletions.
- File-level incremental – easy, but re-copies whole files.
- Delta encoding – computes differences within files, but requires both versions to be present; used in rsync.
- Snapshot-based (e.g., ZFS, btrfs) – uses copy-on-write to store only changed blocks, but at the filesystem level, not application level.
CDC is generally the most flexible for arbitrary files and is the basis for most modern deduplication backup systems.
Frequently Asked Questions
Why does my backup grow even when I don’t change anything?
Many backup tools run integrity checks or change file metadata (access times), causing files to appear modified. Also, if you use whole-file backups, simply opening a file can change its content (e.g., a database that updates timestamps). CDC reduces this because only truly changed chunks are stored.
Can I use CDC with any backup software?
No, you need software that implements CDC. Most modern backup tools do, but some simple ones don’t. Check the documentation for terms like “variable block size” or “content-defined chunking.”
Does CDC work on encrypted or compressed files?
It works, but the dedup benefit is limited because encrypted or compressed data has high entropy – small changes cause many chunks to differ. For encrypted files, you must decrypt before chunking, which defeats the purpose if you want to deduplicate across users. For compressed files, you might need to decompress first, but that’s often impractical.
How much storage can I expect to save with CDC?
It depends on your data. For typical office documents and source code, you might see 70–90% reduction in incremental backup size. For already-compressed media files, savings might be only 10–30%. The best way is to test with your own data.
Is CDC the same as deduplication?
No. CDC is a method to split data into chunks; deduplication is the process of eliminating duplicate chunks. CDC makes deduplication more effective, but you can have deduplication without CDC (e.g., fixed blocks) and CDC without deduplication (though that’s rare).
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.