Re: [RFC PATCH] xfs: add new policy guidelines for llm-assisted patches

[Dave Chinner <dgc@kernel.org>]

On Tue, Jul 14, 2026 at 09:03:32AM -0700, Darrick J. Wong wrote:
> On Tue, Jul 14, 2026 at 08:42:00AM -0400, Brian Foster wrote:
> > > > > +Also, every patch/series submitted must be exercised through xfstests suite
> > > > > +- at least - through the auto group (and others depending on the change) as a way
> > > > > +to add extra coverage through the already existing regression cases and help
> > > > > +reviewers/maintainers through the integration process.
> > > 
> > > What about patches to online fsck, in which case the auto group may or
> > > may not cover it other than incidentally through xfs/28[56]? ;)
> > > 
> > > How about:
> > > 
> > > "Every patch submitted must be exercised through the fstests suite
> > > because good test coverage makes review and maintenance processes
> > > easier.  Ideally, the change should be exercised by a fstest case in the
> > > "auto" group, but another group may be more appropriate depending on the
> > > change."
> > > 
> > 
> > Since some things are not practical to test directly with an fstest, I
> > wonder if we should require or recommend that in absence of a viable
> > fstest, the submitter of an LLM generated patch should include a brief
> > description of how some change was unit and/or code coverage tested.
> > 
> > That way for things that look like hard to hit races, require crafted fs
> > images for log recovery corruption cases, etc., we have at least some
> > indication that the change works and doesn't tickle some other
> > previously unknown problem (like busted error paths, etc.).
> > 
> > IME even these sorts of cases can be coverage tested with custom
> > instrumentation. For example, "I don't have a custom fs image that
> > reproduced this broken log record condition, so I added an if (1) at
> > LINE XYZ to trigger the error check on a standard dirty log test fs
> > image. I confirm it works as expected, doesn't explode the kernel, and
> > the mount fails gracefully."

Here's my take: LLMs are powerful, and they can easily extend
existing tools to do new things. 

We're getting lots of people doing random perutrbations on journals
to find issues, but outside of that we have no deterministic way of
modifying the contents of a journal. LLMs are being used to play the
old game of whack-a-mole; they do it faster than anything else, but
at the end of the day it is still just a game of whack-a-mole.

Lets step back for a moment and look at the bigger picture.

We already have a tool specifically for observing and manipulating
the on-disk format of the filesystem - xfs_db. It can also
deterministically fuzz any metadata structure in the filesystem, and
we have fstests that exercise this capability to validate the
filesystem handles metadata corruptions reliably.

However, xfs_db cannot parse or modify journal structures, so we
cannot use it to inspect, modify or fuzz journal contents.

xfs_logprint can parse the structures, but it can't modify anythign,
and that code is so, so terrible that is needs to be rewritten
completely before it could be used for anything like fuzzing
specific journals structures.

Whilst I've been working far away from XFS for the past year and so
not doing any real dev work in the field, I've been using time
devoted to learning tasks to do some experimental stuff.
As a "learn how to drive LLMs" exercise as well
and trying to learn a bit about Rust, I've been rebuilding
xfs_logprint with Rust and the field-table architecture of xfs_db
to allow robust introspection of the on-disk journal structures.

I did this, because I wanted to be able to decode fragments of
structures that are held in sparsely logged buf_log_items. Having
logprint be aware of what the log item holds (e.g. we have BLFT that
defines the contents) and being able to print the actual structure
that was logged in exactly the same way as xfs_db would print the
structure makes analysing the journal contents so much easier. i.e.
no more manual structure decoding from a hex dump to compare that to
the structure on disk to see what changed in the journal....

I've got logprint half way there - most of the big issues are
solved, I've now in the process of adding support for all the
on-disk structures - dir/attr is next, then intents, and then all
the RT/metadir stuff and it'll be largely complete. It's already
much more useful and robust than the existing logprint code, but
there's still a ways to go before I'm ready to publish the first
prototype.

Once that is done, however, it's not much of a stretch for a
structure aware journal structure parser to be able to perform
targeted modification and rewrite structures. At that point, we have
a deterministic journal fuzzing tool, just like the xfs_db fuzz
command....

Ultimately, I want a combined db/logprint utility to be able to
recover the log onto an ram-based overlay, so when triaging log
recovery issues we can look at the original on-disk structure, the
multiple verions of the modified structure in the journal and in
memory as each version is recovered, all side by side in the one
context...

And, well, if it can do this, then we can probably also do log
recovery in userspace pretty easily, too.

Yeah, that's the sort of experiment I call a "learning task"
because, in the process, I have learnt how to drive LLMs well enough
that I don't need to write code anymore. The LLM functions as my
code editor that is capable of extremely fancy DWIM predictive text
insertion. For someone who hates the process of typing out code,
this has been a revelation.

I can push out the logrpint code I have so far for people to look
at; but it's not really ready for prime time yet and I'm only
mentioning it because the both the reason behind wanting to create
this tool and the process I've been using to build it are directly
relevant to this discussion.

IMO, this is the infrastructure project we should be asking people
spending big bucks on LLM tokens to help us build instead
of playing whack-a-mole on randomly corrupted fs images. Swamping us
with bandaids that don't adidress the underlying issues doesn't
improve the status quo.

We need to be leveraging the vast LLM resources that are available
to the wider community to build the infrastructure needed to make
our software more reliable. We've long complained that we don't have
the resources to tackle these sorts of big issues, but we're now
failing to recognise that LLMs actually provide us with the
resources and capabilities we've been wanting. What is best (and
worst) about LLM capabilities is that it does not require a 
change in available andi/or relevant human expertise to leverage.

In this case, we should be asking the people running the LLMs to
redirect their time towards building infrastructure that allows
deterministic exploration of the -entire- on-disk format variable
space without needing a million monkeys to bash randomly on stuff
until they find something that breaks. The on-disk format is well
defined, and verification can largely be derived from that
specification. With such a verification infrastructure, we can use
it to:

	a) find *all the bugs* deterministically, reliably reproduce
	them and use the LLMs to automate writing and testing fixes;
	and

	b) provide a robust regression test for the on-disk format
	parsers that will highlight when we break something into the
	foreseeable future.

LLMs are a big force-multiplier - we should be encouraging people
with LLM resources and tokens to spend to build the tools and
architecture required to make our software robust. Using them to
play the same old whack-a-mole game might fix existing bugs faster,
but it does nothing to address the underlying vulnerabilities in our
infrastructure, tools and processes that allow the creation of the
bugs in the first place.

Cheers,

Dave.
-- 
Dave Chinner
dgc@kernel.org