Re: [RFC] reworking the review-prompts subsystem guide

[Fuad Tabba <fuad.tabba@linux.dev>]

Hi Chris,

On Fri, 02 Oct 2026 20:04:00 +0100, "Chris Mason" <mason@kernel.org> wrote:
[...]
> My new branch builds subsystem guides by asking a long list of
> questions, and then extensively reading the sources to find the
> right answers.  The delta between the LLM's answers and the right
> answers is the new guide.

I went through the KVM/arm64 and protected KVM (pKVM) guides on the
branch, and checked a sample of their claims against the tree. They're
much more accurate and detailed than the hand-written ones, and the
checking against the source is what makes the difference.

Some of what a reviewer needs isn't in the code, though: which bugs
matter and how much, and which reports are false alarms. The old pKVM
guide said that a hypervisor crash only the host kernel can trigger is a
hardening item, while one a guest can trigger is a real bug. The design
deliberately keeps that kind of instruction out of the guides, so it's
gone from the built one. Where should it live instead? The "Conventions
for new code" files look closest: they already hold what maintainers ask
for and no code states, and a review finds them by the directory a patch
touches. Something like that for KVM/arm64 would work, and I can write
it.

Leaving out what the tested models already knew also tunes the guide to
those models, and the model doing the review may not be one of them.
Hand-written guides have the same problem, and I don't see an easy fix.
Your mail points at builds per model, so one option is for each project
to build for the model it runs. Another is for the build to keep the
full checked answer rather than only the difference, now that the index
makes length matter less. Which way are you leaning?

> This is both much less useful to human readers and much longer.  I'm
> not sure what to say about the human reader part, but instead of
> having LLMs read the whole subsystem guide, I shifted to an index
> where they search for symbols.  This is a better fit for more
> advanced models, which mostly need updates on how the kernel has
> changed since they were trained.

A common KVM patch adds a new hypercall. A symbol search finds the
answers about the existing calls the patch uses, but not the rule for
where a new one goes in the list: that answer is filed under a marker
the diff never touches, and the header the diff edits isn't the source
file of any index line. Could some answers be keyed to the directory as
well, like the short table you kept for a few guides?

[...]
> What I know for sure is the existing review prompts have drifted
> from mainline Linus.  It's impacting the quality of the reviews, so
> I plan on working out something in the near future.

Built guides drift too, just in a different way. These were built at
7.3-rc5, and one fix in rc6 made four of the KVM/arm64 claims wrong
without renaming anything, so looking names up in the tree doesn't catch
it. Most index lines already record a source file, so a review on a
newer tree could check whether that file changed since the build, and
re-check or flag the answer if it did.

Related: when a maintainer finds a wrong answer, the only fix is to
change the question and rebuild, which needs model access and can come
out differently each time. Who looks after the question files, and could
there be a small override that a maintainer edits between rebuilds?
Feedback from the list, as Jakub and Roman discussed, could take the
same route: suggested questions that go through the same checks against
the tree, rather than guide text.

Separately, some pKVM questions were dropped for size, including the one
on protected guest system registers, and the measurement notes say
they're the first to bring back if the size limit is raised. There's no
length limit now, so I could send a patch to bring them back.

Cheers,
/fuad