Re: [RFC] reworking the review-prompts subsystem guide

["Chris Mason" <mason@kernel.org>]

On Fri, Oct 9, 2026, at 10:17 AM, Fuad Tabba wrote:
> Hi Chris,
>
> On Fri, 02 Oct 2026 20:04:00 +0100, "Chris Mason" <mason@kernel.org> wrote:
> [...]
>> My new branch builds subsystem guides by asking a long list of
>> questions, and then extensively reading the sources to find the
>> right answers.  The delta between the LLM's answers and the right
>> answers is the new guide.
>
> I went through the KVM/arm64 and protected KVM (pKVM) guides on the
> branch, and checked a sample of their claims against the tree. They're
> much more accurate and detailed than the hand-written ones, and the
> checking against the source is what makes the difference.

Great, I only did limited verification in that area, so its good to hear.

>
> Some of what a reviewer needs isn't in the code, though: which bugs
> matter and how much, and which reports are false alarms. The old pKVM
> guide said that a hypervisor crash only the host kernel can trigger is a
> hardening item, while one a guest can trigger is a real bug. The design
> deliberately keeps that kind of instruction out of the guides, so it's
> gone from the built one. Where should it live instead? The "Conventions
> for new code" files look closest: they already hold what maintainers ask
> for and no code states, and a review finds them by the directory a patch
> touches. Something like that for KVM/arm64 would work, and I can write
> it.

There's already a way to pass specific parts verbatim, which feels more
correct than under a "new code" label.  We can play around with a few
ideas though.

>
> Leaving out what the tested models already knew also tunes the guide to
> those models, and the model doing the review may not be one of them.

Yes.  I generated the current built prompts using both sonnet and opus
with the assumption that other models would get roughly the same
things wrong as one of the two of them.  This is obviously flawed, but
the prompts have a way to pick the best build directory based on kernel
version, so we could extend that to the model doing the
review (more below).

> Hand-written guides have the same problem, and I don't see an easy fix.
> Your mail points at builds per model, so one option is for each project
> to build for the model it runs. Another is for the build to keep the
> full checked answer rather than only the difference, now that the index
> makes length matter less. Which way are you leaning?

My original plan was to get the built guides small enough to be included
whole, in the same way the original guides were.  Adding the keyword
index was basically me admitting defeat, and it just didn't occur to me
that we might be able to index our way to victory on the full output.
IOW, great idea, I didn't think of that at all ;)

I'll do another run and put the full build up.  We can see if it's too big
to be useful or if we can make the index strong enough to get around
needing the per-model analysis.

>
>> This is both much less useful to human readers and much longer.  I'm
>> not sure what to say about the human reader part, but instead of
>> having LLMs read the whole subsystem guide, I shifted to an index
>> where they search for symbols.  This is a better fit for more
>> advanced models, which mostly need updates on how the kernel has
>> changed since they were trained.
>
> A common KVM patch adds a new hypercall. A symbol search finds the
> answers about the existing calls the patch uses, but not the rule for
> where a new one goes in the list: that answer is filed under a marker
> the diff never touches, and the header the diff edits isn't the source
> file of any index line. Could some answers be keyed to the directory as
> well, like the short table you kept for a few guides?
>

Absolutely.  The index is pretty dumb, we can do a lot better.

> [...]
>> What I know for sure is the existing review prompts have drifted
>> from mainline Linus.  It's impacting the quality of the reviews, so
>> I plan on working out something in the near future.
>
> Built guides drift too, just in a different way. These were built at
> 7.3-rc5, and one fix in rc6 made four of the KVM/arm64 claims wrong
> without renaming anything, so looking names up in the tree doesn't catch
> it. Most index lines already record a source file, so a review on a
> newer tree could check whether that file changed since the build, and
> re-check or flag the answer if it did.

A related question is how often do I need to rebuild in order for
the guides to be useful?  I'd assume the absolute minimum is every
final release, but every RC is also reasonable.

>
> Related: when a maintainer finds a wrong answer, the only fix is to
> change the question and rebuild, which needs model access and can come
> out differently each time. Who looks after the question files, and could
> there be a small override that a maintainer edits between rebuilds?
> Feedback from the list, as Jakub and Roman discussed, could take the
> same route: suggested questions that go through the same checks against
> the tree, rather than guide text.

The build script can rebuild a single guide, or people can just have
their agents hand edit the build?  It's a good point, we should have
AGENTS.md record some best practices.

>
> Separately, some pKVM questions were dropped for size, including the one
> on protected guest system registers, and the measurement notes say
> they're the first to bring back if the size limit is raised. There's no
> length limit now, so I could send a patch to bring them back.

Great, please do.

-chris