Re: [RFC] reworking the review-prompts subsystem guide

[Ihor Solodrai <ihor.solodrai@linux.dev>]

On 10/23/26 1:22 PM, Roman Gushchin wrote:
> Ihor Solodrai <ihor.solodrai@linux.dev> writes:
> 
>> [...]
>>
>> Some stats:
>>   * Same outcome (review posted vs clean) on 90% of commits.
>>   * Medium/high issues: posted on 7.4% of commits, vs 5.9% before.
>>   * Low/nit-only reviews: 10.3% of commits, vs 15.8% before.
>>   * Cost +6%, same wall time.
>>
>> Interesting findings:
>>   * Old vs new prompts differ about as much as two runs of the old
>>     prompts do: on an unchanged patch reposted in the next version,
>>     production reproduces only about half of its own findings.
>>   * The new guides are consulted in every review, but only 3 of 114
>>     new findings cite a guide answer, and the old prompts found those
>>     too. No clear wins attributable to the guides yet.
>>   * The new prompts no longer produce the "A subsystem pattern flags
>>     this as potentially concerning" items (mostly selftests style).
> 
> Thanks, Ihor!
> 
> Quick question: once it's merged, how kernel maintainers are supposed to
> change prompts or the idea is to completely stop doing it?

haha, I like how you said it's a "quick question"

Just speculating below, I'm not an AI expert by any means.

I don't think we should stop editing the prompts. They at least need
to be up to date matching both the source code, project development
process and maintainers' expectations. If we outright stop, the
prompts will rot and become stale, which will mislead the AI and burn
tokens even if it self-corrects.

At the same time, it appears that attempts to improve / fine tune the
prompts to improve the *quality* of the output or efficiency of the
workflow quickly hit diminishing returns.

The low hanging fruits in my experience are picked up fast by giving
an LLM samples of the real-world traces and ask it to improve the
prompts. It works.

Again, lots of salt here, but out of curiosity I looked at things like
GEPA [1]. While projects like this advertise significant improvements,
evaluations that I saw show only marginal gains. My guess is that a
new big LLM release would immediately eat all those improvements for
most use-cases.

And that's setting aside the problem that it's very difficult to
evaluate the results if the output is open-ended; and the fact that
it's costly to run. If we do all that, why not just build and add a
"Linux code review corpus" into the RLHF pass of the base LLM? I'd be
surprised if it's not happening in some form already at the big labs.

[1] https://github.com/gepa-ai/gepa


> 
> E.g. are they supposed to add new questions? Or edit the final result
> directly?

Maintainers feedback is essential to make the auto-review systems
work. They are the ultimate consumers and judges of the reviews.
The question is how exactly to capture the feedback.

The first versions of the review-prompts read like an algorithm
written in english: phase 0, step 1, ... step 13 - skip if X, etc.

IMO if you are using a frontier LLM + harness, this is now
unnecessary. The AIs can figure out the steps themselves.

What is necessary:
  - describe the desired outcome as precisely as possible
  - explain where and how to search for context and details
  - configure / explain what it can and can not do in terms of tool
    usage: e.g. don't try curl if you're sandboxed
    AI is prone to try hard even when it's deliberately limited. That's
    how it's been trained so far.

I think the most impactful and important part is steering the "desired
outcome" through the prompt. And that one is subjective in many cases.

Do you want only bugs that are reproducible, or do you want suspicious
code flagged as well? Do you want nits or nah? Commit message style?
How about a high-level design review for a big series? Would you trust
an AI to do that?

So, replying to AI reviews (even with "this is bullshit") is useful.
Adding {subsystem,project}-specific rules about what to look for and
what to drop is also very useful.

Obsessing over wording or writing prompts like programs - too time
consuming vs the value you'd get out of it.

Opinions are my own :)


> 
> Thanks