Puan
28,728
Çözümler
0
- Konum
- Freanchey
- Mesajlar
- 3,049
- Katılım
- 2 Eki 2026
- Tepkime puanı
- 3,565
- Puan
- 28,728
- Web sitesi
- glomaxgpt.com
FacebookXPinterestWhatsAppLinkedinReddItEmail
You change one line in a large file. Your “incremental” backup re-uploads the whole thing anyway.
Nine times out of ten, it’s not the network, and it’s not your storage backend. It’s the algorithm deciding where to cut your data into pieces.
Every deduplication system works the same way: split data into pieces, hash each piece, keep one copy per distinct hash. A piece is named after its own content, so two identical pieces are, by definition, the same piece. Snapshot a filesystem today and again tomorrow, and if most of the data hasn’t changed, most pieces are already in the store. The second snapshot costs you almost nothing.
That’s not a nice-to-have. A backup that actually survives an incident needs to be encrypted, frequent, immutable, replicated and verified, and each of those requirements multiplies what you store. When deduplication is weak, those multipliers stack until backing up everything becomes unaffordable, and teams quietly retreat to backing up only what they call “critical.” Good deduplication is what makes backing up everything affordable again.
And all of it rests on one assumption: unchanged data should yield unchanged pieces.
Where fixed-size chunking falls apart
The obvious way to split a file is to cut every N bytes. It’s fast. It also wrecks deduplication the moment you touch the file.
Add one byte near the front and every boundary after it slides over by one. Nothing lines up with where it used to be, so every hash downstream is “new,” so the store treats the entire rest of the file as fresh data. Edit one line in a 40 GB file and you re-upload close to 40 GB of it. Nothing meaningful changed, only the offsets did, and fixed-size boundaries are glued to offsets.
How content-defined chunking fixes it
Content-Defined Chunking (CDC) picks boundaries based on what the data is, not where it happens to sit.
Slide a small window over the byte stream and keep a rolling hash, cheap enough to update incrementally that the whole scan stays linear, then cut a boundary wherever that hash matches an agreed pattern.
You change one line in a large file. Your “incremental” backup re-uploads the whole thing anyway.
Nine times out of ten, it’s not the network, and it’s not your storage backend. It’s the algorithm deciding where to cut your data into pieces.
Every deduplication system works the same way: split data into pieces, hash each piece, keep one copy per distinct hash. A piece is named after its own content, so two identical pieces are, by definition, the same piece. Snapshot a filesystem today and again tomorrow, and if most of the data hasn’t changed, most pieces are already in the store. The second snapshot costs you almost nothing.
That’s not a nice-to-have. A backup that actually survives an incident needs to be encrypted, frequent, immutable, replicated and verified, and each of those requirements multiplies what you store. When deduplication is weak, those multipliers stack until backing up everything becomes unaffordable, and teams quietly retreat to backing up only what they call “critical.” Good deduplication is what makes backing up everything affordable again.
And all of it rests on one assumption: unchanged data should yield unchanged pieces.
Where fixed-size chunking falls apart
The obvious way to split a file is to cut every N bytes. It’s fast. It also wrecks deduplication the moment you touch the file.
Add one byte near the front and every boundary after it slides over by one. Nothing lines up with where it used to be, so every hash downstream is “new,” so the store treats the entire rest of the file as fresh data. Edit one line in a 40 GB file and you re-upload close to 40 GB of it. Nothing meaningful changed, only the offsets did, and fixed-size boundaries are glued to offsets.
How content-defined chunking fixes it
Content-Defined Chunking (CDC) picks boundaries based on what the data is, not where it happens to sit.
Slide a small window over the byte stream and keep a rolling hash, cheap enough to update incrementally that the whole scan stays linear, then cut a boundary wherever that hash matches an agreed pattern.