Re: [RFC PATCH 0/2] Btrfs: make a source length of 0 imply EOF for dedupe

[Dave Chinner <david@fromorbit.com>]

On Wed, Nov 23, 2016 at 08:55:59AM -0500, Zygo Blaxell wrote:
> On Wed, Nov 23, 2016 at 03:26:32PM +1100, Dave Chinner wrote:
> > On Tue, Nov 22, 2016 at 09:02:10PM -0500, Zygo Blaxell wrote:
> > > On Thu, Nov 17, 2016 at 04:07:48PM -0800, Omar Sandoval wrote:
> > > > 3. Both XFS and Btrfs cap each dedupe operation to 16MB, but the
> > > >    implicit EOF gets around this in the existing XFS implementation. I
> > > >    copied this for the Btrfs implementation.
> > > 
> > > Somewhat tangential to this patch, but on the dedup topic:  Can we raise
> > > or drop that 16MB limit?
> > > 
> > > The maximum btrfs extent length is 128MB.  Currently the btrfs dedup
> > > behavior for a 128MB extent is to generate 8x16MB shared extent references
> > > with different extent offsets to a single 128MB physical extent.
> > > These references no longer look like the original 128MB extent to a
> > > userspace dedup tool.  That raises the difficulty level substantially
> > > for a userspace dedup tool when it tries to figure out which extents to
> > > keep and which to discard or rewrite.
> > 
> > That, IMO, is a btrfs design/implementation problem, not a problem
> > with the API. Applications are always going to end up doing things
> > that aren't perfectly aligned to extent boundaries or sizes
> > regardless of the size limit that is placed on the dedupe ranges.
> 
> Given that XFS doesn't have all the problems btrfs does, why does XFS
> have the same aribitrary size limit?  Especially since XFS demonstrably
> doesn't need it?

Creating a new-but-slightly-incompatible jsut for XFS makes no
sense - we have multiple filesystems that support this functionality
and so they all should use the same APIs and present (as far as is
possible) the same behaviour to userspace.

IOWs it's more important to use existing APIs than to invent a new
one that does almost the same thing. This way userspace applications
don't need to be changed to support new XFS functionality and we
make life easier for everyone. A shiny new API without warts would
be nice, but we've already got to support the existing one forever,
it does the job we need and so it's less burden on everyone if we
just use it as is.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com