Re: [NYE PATCHCYCLONE] xfs: free space defrag and autonomous self healing

["Darrick J. Wong" <djwong@kernel.org>]

On Thu, Jan 02, 2025 at 09:37:47AM +0800, Stephen Zhang wrote:
> Darrick J. Wong <djwong@kernel.org> 于2025年1月1日周三 07:25写道：
> >
> > Hi everyone,
> >
> > Thank you all for helping get online repair, parent pointers, and
> > metadata directory trees, and realtime allocation groups merged this
> > year!  We got a lot done in 2024.
> >
> > Having sent pull requests to Carlos for the last pieces of the realtime
> > modernization project, I have exactly two worthwhile projects left in my
> > development trees!  The stuff here isn't necessarily in mergeable state
> > yet, but I still believe everyone ought to know what I'm up to.
> >
> > The first project implements (somewhat buggily; I never quite got back
> > to dealing with moving eof blocks) free space defragmentation so that we
> > can meaningfully shrink filesystems; garbage collect regions of the
> > filesystem; or prepare for large allocations.  There's not much new
> > kernel code other than exporting refcounts and gaining the ability to
> > map free space.
> >
> > The second project initiates filesystem self healing routines whenever
> > problems start to crop up, which means that it can run fully
> > autonomously in the background.  The monitoring system uses some
> > pseudo-file and seqbuf tricks that I lifted from kmo last winter.
> >
> > Both of these projects are largely userspace code.
> >
> > Also I threw in some xfs_repair code to do dangerous fs upgrades.
> > Nobody should use these, ever.
> >
> > Maintainers: please do not merge, this is a dog-and-pony show to attract
> > developer attention.
> >
> 
> [Add Dave to the list]
> 
> Hi, Darrick and all,
> 
> Recently, I have been considering implementing the XFS shrink feature based
> on the AF concept, which was mentioned in this link:
> 
> https://lore.kernel.org/linux-xfs/20241104014439.3786609-1-zhangshida@kylinos.cn/
> 
> In the lore link, it stated:
> The rules used by AG are more about extending outwards.
> whilst
> The rules used by AF are more about restricting inwards.
> 
> where the AF concept implicitly and naturally involves the semantics of
> compressing/shrinking(restricting).
> 
> AG(for xfs extend) and AF(for xfs shrink) are constructed in a symmetrical way,
> in which it is more elegant and easier to build more complex features on it.
> 
> To elaborate further, for example, AG should not be seen as
> independent entities in
> the shrink context. That means each AG requires separate
> managements(flags or something to indicate the state of that
> AG/region), which would increase the system complexity compared to the
> idea behind AF. AF views several AGs as a whole.
> 
> And when it comes to growfs, things start to get a little more
> complicated, and AF
> can handle it easily and naturally.
> 
> However talk is too cheap, to validate our point, we truly hope to have the
> opportunity to participate in developing these features by integrating
> the existing
> infrastructure you have already established with the AF concept.

Hmm, now that's interesting -- using the AF ("allocation fencing"?)
capability to constrain allocations to a subset of AGs, and then slowly
rewriting files and whatnot to migrate data to other AGs.  Eventually
you end up with an AG that's empty and therefore ready for shrink.

That's definitely a different way to do that than what I did (add a
"mapfree" ioctl to pin space to a file).  I'll ponder these 2 approaches
a bit more.

--D

> Best regards,
> Shida
> 
> 
> 
> > --D
> >
> > PS: I'll be back after the holidays to look at the zoned/atomic/fsverity
> > patches.  And finally rebase fstests to 2024-12-08.
> >
>