Re: [RFC] reworking the review-prompts subsystem guide

[Ihor Solodrai <ihor.solodrai@linux.dev>]

On 10/6/26 2:22 PM, Ihor Solodrai wrote:
> On 10/2/26 12:04 PM, Chris Mason wrote:
>> Hi everyone,
>>
>> I've been updating the review prompts, and you can find my current work here in the subsystem-build branch:
>>
>> https://github.com/masoncl/review-prompts.git subsystem-build
> 
> Hi Chris,
> 
> I've set up "shadow" AI reviews for BPF, taking advantage of our
> existing bpf-rc infra. It's a copy of the entire BPF CI pipeline that
> we occasionally use for testing of the CI itself.
> 
> [...]
> 
> I just pushed, it'll take a few days before we get any real samples.

Hi all.

Got results from a couple of weeks of running the subsystem-build
branch prompts on bpf list: 272 patches in 50 series total.

TL;DR: BPF bot will switch to the subsystem-build prompts. There are
no regressions in findings, and overall the results are quite similar.

Chris, IMO you can just merge subsystem-build into main.

Some stats:
  * Same outcome (review posted vs clean) on 90% of commits.
  * Medium/high issues: posted on 7.4% of commits, vs 5.9% before.
  * Low/nit-only reviews: 10.3% of commits, vs 15.8% before.
  * Cost +6%, same wall time.

Interesting findings:
  * Old vs new prompts differ about as much as two runs of the old
    prompts do: on an unchanged patch reposted in the next version,
    production reproduces only about half of its own findings.
  * The new guides are consulted in every review, but only 3 of 114
    new findings cite a guide answer, and the old prompts found those
    too. No clear wins attributable to the guides yet.
  * The new prompts no longer produce the "A subsystem pattern flags
    this as potentially concerning" items (mostly selftests style).

Overall, it appears that prompts tuning becomes less important vs base
model improvements, at least in the context of searching for
regressions on patches. As long as the core intent is expressed in the
"root" prompts.

The analysis was done by AI, of course, so grains of salt everywhere.

If anyone's interested I can share the dataset and analysis details
separately. Let me know.

Thanks.

> 
> Thank you for working on the prompts!
> 
>>
>> [...]