[LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Darrick J. Wong]

[resend, email exploded, sorry...]

Hi,

I want to discuss a few FS related topics that I haven't already seen on
the mailing lists:

 * Shared pagecache pages for reflinked files (and by extension making dax
   work with reflink on xfs)

 * Providing a simple interface for scrubbing filesystem metadata in the
   background (the online check thing).  Ideally we'd make it easy to discover
   what kind of metadata there is to check and provide a simple interface to
   check the metadata, once discovered.  This is a tricky interface topic
   since FS design differs pretty widely.

 * Rudimentary online repair and rebuilding (xfs) from secondary metadata

 * Working out the variances in the btrfs/xfs/ocfs2/nfs reflink implementations
   and making sure they all get test coverage

I would also like participate in some of the proposed discussions:

 * The ext4 summit (and whatever meeting of XFS devs may happen)

 * Integrating existing filesystems into pmem, or hallway bofs about designing
   new filesystems for pmem

 * Actually seeing the fs developers (well, everyone!) in person again :)

--Darrick

Re: [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Steve French]

Since SMB3 protocol has at least three ways to do copy offload (server
side copy),
the reflink topic is of interest to me and likely useful to discuss
for Samba server as
well as client (cifs.ko)

On Wed, Feb 10, 2016 at 1:19 PM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>
> [resend, email exploded, sorry...]
>
> Hi,
>
> I want to discuss a few FS related topics that I haven't already seen on
> the mailing lists:
>
>  * Shared pagecache pages for reflinked files (and by extension making dax
>    work with reflink on xfs)
>
>  * Providing a simple interface for scrubbing filesystem metadata in the
>    background (the online check thing).  Ideally we'd make it easy to discover
>    what kind of metadata there is to check and provide a simple interface to
>    check the metadata, once discovered.  This is a tricky interface topic
>    since FS design differs pretty widely.
>
>  * Rudimentary online repair and rebuilding (xfs) from secondary metadata
>
>  * Working out the variances in the btrfs/xfs/ocfs2/nfs reflink implementations
>    and making sure they all get test coverage
>
> I would also like participate in some of the proposed discussions:
>
>  * The ext4 summit (and whatever meeting of XFS devs may happen)
>
>  * Integrating existing filesystems into pmem, or hallway bofs about designing
>    new filesystems for pmem
>
>  * Actually seeing the fs developers (well, everyone!) in person again :)
>
> --Darrick
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Thanks,

Steve

Re: [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3528 bytes --]

Didn't see the original mail, so reply here.

On 2020/1/21 上午8:58, Steve French wrote:
> Since SMB3 protocol has at least three ways to do copy offload (server
> side copy),
> the reflink topic is of interest to me and likely useful to discuss
> for Samba server as
> well as client (cifs.ko)
> 
> On Wed, Feb 10, 2016 at 1:19 PM Darrick J. Wong <darrick.wong@oracle.com> wrote:
>>
>> [resend, email exploded, sorry...]
>>
>> Hi,
>>
>> I want to discuss a few FS related topics that I haven't already seen on
>> the mailing lists:
>>
>>  * Shared pagecache pages for reflinked files (and by extension making dax
>>    work with reflink on xfs)

IIRC Goldwyn Rodrigues <rgoldwyn@suse.com> is working on this, mostly
for btrfs, but should also apply to other fses.

>>
>>  * Providing a simple interface for scrubbing filesystem metadata in the
>>    background (the online check thing).  Ideally we'd make it easy to discover
>>    what kind of metadata there is to check and provide a simple interface to
>>    check the metadata, once discovered.  This is a tricky interface topic
>>    since FS design differs pretty widely.

Although btrfs has already implemented scrub for a long time, and btrfs
is coming over the point where kernel can detects more problems than
btrfs-check, I still hesitate to call it "check the metadata".

It looks more like "extended metadata sanity verification", other than
full cross-ref check implemented in user-space tools.

Currently, btrfs (and I guess xfs too) can detect corrupted metadata by:
- checksum
  All modern fses have similar mechanism.

- internal fields checking at read time
  At least btrfs is trying to do a byte-per-byte check for each fields.
  And latest such check has killed tons of fuzzed images (I guess that's
  why a lot of such CVE/fuzzed image reporters only want to report on
  older kernels).

  But this is mostly based on the fact that btrfs on-disk fields has a
  lot of redundancy. E.g. btrfs uses bytenr, not block count, allowing
  us to detect bit flips in lower bits easily.
  So not all fs could follow this step.

  But this provides us a good centralized place to validate most tree
  blocks.

  Maybe other fs devs could share such details too?

- runtime sanity check across metadata boundaries
  This is the traditional methods.

So is that xfs online scrub just a similar thing, or a full cross-ref check?

And if so, can we find a less confusing naming for the interface first?

>>
>>  * Rudimentary online repair and rebuilding (xfs) from secondary metadata

My guess is, it's checksum and copy based.
Or just like btrfs, if checksum passes but internal checks still failed,
just try next copy?

>>
>>  * Working out the variances in the btrfs/xfs/ocfs2/nfs reflink implementations
>>    and making sure they all get test coverage

That would be great!

Thanks,
Qu

>>
>> I would also like participate in some of the proposed discussions:
>>
>>  * The ext4 summit (and whatever meeting of XFS devs may happen)
>>
>>  * Integrating existing filesystems into pmem, or hallway bofs about designing
>>    new filesystems for pmem
>>
>>  * Actually seeing the fs developers (well, everyone!) in person again :)
>>
>> --Darrick
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Lsf-pc] [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Amir Goldstein]

On Tue, Jan 21, 2020 at 3:19 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>
> Didn't see the original mail, so reply here.

Heh! Original email was from 2016, but most of Darrick's wish list is
still relevant in 2020 :)

I for one would be very interested in getting an update on the
progress of pagecache
page sharing if there is anyone working on it.

Thanks,
Amir.

Re: [Lsf-pc] [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Darrick J. Wong]

On Tue, Jan 21, 2020 at 09:35:22AM +0200, Amir Goldstein wrote:
> On Tue, Jan 21, 2020 at 3:19 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> >
> > Didn't see the original mail, so reply here.
> 
> Heh! Original email was from 2016, but most of Darrick's wish list is
> still relevant in 2020 :)

Grumble grumble stable behavior of clonerange/deduperange ioctls across
filesystems grumble grumble.

> I for one would be very interested in getting an update on the
> progress of pagecache
> page sharing if there is anyone working on it.

Me too.  I guess it's the 21st, I should really send in a proposal for
*this year's* LSFMMBPFLOLBBQ.

--D

> Thanks,
> Amir.

Re: [Lsf-pc] [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Matthew Wilcox]

On Tue, Jan 21, 2020 at 08:18:40AM -0800, Darrick J. Wong wrote:
> On Tue, Jan 21, 2020 at 09:35:22AM +0200, Amir Goldstein wrote:
> > On Tue, Jan 21, 2020 at 3:19 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >
> > > Didn't see the original mail, so reply here.
> > 
> > Heh! Original email was from 2016, but most of Darrick's wish list is
> > still relevant in 2020 :)
> 
> Grumble grumble stable behavior of clonerange/deduperange ioctls across
> filesystems grumble grumble.
> 
> > I for one would be very interested in getting an update on the
> > progress of pagecache
> > page sharing if there is anyone working on it.
> 
> Me too.  I guess it's the 21st, I should really send in a proposal for
> *this year's* LSFMMBPFLOLBBQ.

I still have Strong Opinions on how pagecache page sharing should be done
... and half a dozen more important projects ahead of it in my queue.
So I have no update on this.

Re: [Lsf-pc] [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Dan Williams]

On Tue, Jan 21, 2020 at 2:02 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Tue, Jan 21, 2020 at 08:18:40AM -0800, Darrick J. Wong wrote:
> > On Tue, Jan 21, 2020 at 09:35:22AM +0200, Amir Goldstein wrote:
> > > On Tue, Jan 21, 2020 at 3:19 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > > >
> > > > Didn't see the original mail, so reply here.
> > >
> > > Heh! Original email was from 2016, but most of Darrick's wish list is
> > > still relevant in 2020 :)
> >
> > Grumble grumble stable behavior of clonerange/deduperange ioctls across
> > filesystems grumble grumble.
> >
> > > I for one would be very interested in getting an update on the
> > > progress of pagecache
> > > page sharing if there is anyone working on it.
> >
> > Me too.  I guess it's the 21st, I should really send in a proposal for
> > *this year's* LSFMMBPFLOLBBQ.
>
> I still have Strong Opinions on how pagecache page sharing should be done
> ... and half a dozen more important projects ahead of it in my queue.
> So I have no update on this.

We should plan to huddle on this especially if I don't get an RFC for
dax-reflink support out before the summit.

Re: [Lsf-pc] [LFS/MM TOPIC] fs reflink issues, fs online scrub/check, etc

[Dave Chinner]

On Tue, Jan 21, 2020 at 04:47:27PM -0800, Dan Williams wrote:
> On Tue, Jan 21, 2020 at 2:02 PM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > On Tue, Jan 21, 2020 at 08:18:40AM -0800, Darrick J. Wong wrote:
> > > On Tue, Jan 21, 2020 at 09:35:22AM +0200, Amir Goldstein wrote:
> > > > On Tue, Jan 21, 2020 at 3:19 AM Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > > > >
> > > > > Didn't see the original mail, so reply here.
> > > >
> > > > Heh! Original email was from 2016, but most of Darrick's wish list is
> > > > still relevant in 2020 :)
> > >
> > > Grumble grumble stable behavior of clonerange/deduperange ioctls across
> > > filesystems grumble grumble.
> > >
> > > > I for one would be very interested in getting an update on the
> > > > progress of pagecache
> > > > page sharing if there is anyone working on it.
> > >
> > > Me too.  I guess it's the 21st, I should really send in a proposal for
> > > *this year's* LSFMMBPFLOLBBQ.
> >
> > I still have Strong Opinions on how pagecache page sharing should be done
> > ... and half a dozen more important projects ahead of it in my queue.
> > So I have no update on this.
> 
> We should plan to huddle on this especially if I don't get an RFC for
> dax-reflink support out before the summit.

It would be a good idea to share your ideas about how you plan to
solve this problem earlier rather than later so you don't waste time
going down a path that is incompatible with what the filesystems
need to/want to/can do.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com