Re: Offline Deduplication for Btrfs

[Gordan Bobic <gordan@bobich.net>]

Peter A wrote:
> On Wednesday, January 05, 2011 08:19:04 pm Spelic wrote:
>>> I'd just make it always use the fs block size. No point in making it 
>>> variable.
>> Agreed. What is the reason for variable block size?
> 
> First post on this list - I mostly was just reading so far to learn more on fs 
> design but this is one topic I (unfortunately) have experience with... 
> 
> You wouldn't believe the difference variable block size dedupe makes. For a 
> pure fileserver, its ok to dedupe on block level but for most other uses, 
> variable is king. One big example is backups. Netbackup and most others 
> produce one stream with all data even when backing up to disk. Imagine you 
> move a whole lot of data from one dir to another. Think a directory with huge 
> video files. As a filesystem it would be de-duped nicely. The backup stream 
> however may and may not have matching fs blocks. If the directory name before 
> and after has the same lengths and such - then yeah, dedupe works. Directory 
> name is a byte shorter? Everything in the stream will be offset by one byte - 
> and no dedupe will occur at all on the whole dataset. In real world just 
> compare the dedupe performance of an Oracle 7000 (zfs and therefore fs block 
> based) to a DataDomain (variable lenght) in this usage scenario. Among our 
> customers we see something like 3 to 17x dedupe ration on the DD, 1.02 - 1.05 
> in the 7000.

Can you elaborate what you're talking about here? How does the length of 
a directory name affect alignment of file block contents? I don't see 
how variability of length matters, other than to make things a lot more 
complicated. Have you some real research/scientifically gathered data 
(non-hearsay/non-anecdotal) on the underlying reasons for the 
discrepancy in the deduping effectiveness you describe? 3-17x difference 
doesn't plausibly come purely from fixed vs. variable length block sizes.

The only case where I'd bother consider variable length deduping is in 
file deduping (rather than block), in this case we can just make a COW 
hard-link and it's _really_ cheap and effective.

Gordan