Re: [RFC] reworking the review-prompts subsystem guide

[Roman Gushchin <roman.gushchin@linux.dev>]

Jakub Kicinski <kuba@kernel.org> writes:

> On Fri, 02 Oct 2026 15:04:00 -0400 Chris Mason wrote:
>> Hi everyone,
>> 
>> I've been updating the review prompts, and you can find my current work here in the subsystem-build branch:
>> 
>> https://github.com/masoncl/review-prompts.git subsystem-build
>> 
>> This is a pretty big change, and since a few projects are syncing automatically, I didn't want to just throw it in without discussion.
>> 
>> The review prompts started with a lot of framework to explain how
>> the kernel works, and the prompts have focused on documenting
>> subsystems in a way that both LLMs and people can use for reviews
>> and writing code.
>> 
>> As models have progressed, they actually need different information
>> than they used to.  Recent models already know how most of the
>> kernel works, but they have some gaps because the kernel is always
>> changing, or because they have some bad assumptions baked in.
>> 
>> My new branch builds subsystem guides by asking a long list of
>> questions, and then extensively reading the sources to find the
>> right answers.  The delta between the LLM's answers and the right
>> answers is the new guide.
>> 
>> This is both much less useful to human readers and much longer.  I'm
>> not sure what to say about the human reader part, but instead of
>> having LLMs read the whole subsystem guide, I shifted to an index
>> where they search for symbols.  This is a better fit for more
>> advanced models, which mostly need updates on how the kernel has
>> changed since they were trained.
>> 
>> The subsystem-build branch was built with both sonnet-5.5 and
>> opus-5.5.  It's the union of all the things either model got wrong,
>> and the tree is setup so we can add builds for other kernel versions
>> (ex: stable kernel series).
>> 
>> I started down this path convinced that I needed to get the
>> subsystem guides into the kernel git tree.  My idea was this was
>> documentation, and it should land somewhere in the kernel for that
>> reason.  I ended up with something that probably shouldn't be in the
>> tree...different models need different things on top of different
>> kernels, and so I'm trying out the build idea.
>> 
>> What I know for sure is the existing review prompts have drifted
>> from mainline Linus.  It's impacting the quality of the reviews, so
>> I plan on working out something in the near future.
>
> I didn't read the whole thing, but makes sense AFAIU.
>
> Closing the loop with Sashiko and/or ML would be great :(
> If the bots read the ML they could both learn false positives and false
> negatives automatically. I suspect most of us thought about this by now.
> If we're doing a redesign should the ingest of ML be part of it?

I plan to do this (and had a prototype in the past), but it's tricky if
we take security seriously. And we absolutely should!
Obviously just giving an agent an access to lore archive is opening a
can of worms in terms of possible prompt injections. So I think we
should do it really carefully. And outside of security considerations,
humans are simple wrong too. So my plan with sashiko is to force it to
try to verify the feedback against the codebase and if it's not possible
trust only maintainers or people with a long history of meaningful
contributions.
And sashiko can (and does in my prototype) generate prompts based on the
feedback and automatically verify that it helps by re-reviewing original
patches. This should mostly eliminate a need for human-generated
prompts.
This is all doable, but probably will take few more week to build and
roll out.