Re: [RFC] reworking the review-prompts subsystem guide

[Ihor Solodrai <ihor.solodrai@linux.dev>]

On 10/5/26 2:14 PM, Roman Gushchin wrote:
> Jakub Kicinski <kuba@kernel.org> writes:
> 
>> [...]
>>
>> Closing the loop with Sashiko and/or ML would be great :(
>> If the bots read the ML they could both learn false positives and false
>> negatives automatically. I suspect most of us thought about this by now.
>> If we're doing a redesign should the ingest of ML be part of it?

FWIW the BPF bot has been reading lore when reviewing patches through
semcode since February:
https://github.com/kernel-patches/vmtest/pull/442

Basically, the container in which the AI session is running has access
to local semcode via MCP, and local semcode db has the full (bpf) lore
archive.

It definitely helps with quality to give AI more context, but it also
has side effects. For example, a bot may repeat an issue raised by
another bot in a previous revision. In many cases this is noise, but
sometimes it is appropriate: the review may have been inappropriately
ignored. Currently this is mostly hidden by rules in the prompts to
not repeat issues that have already been discussed, because it appears
that people ignore many of the AI reviews anyways.


> 
> I plan to do this (and had a prototype in the past), but it's tricky if
> we take security seriously. And we absolutely should!
> Obviously just giving an agent an access to lore archive is opening a
> can of worms in terms of possible prompt injections. So I think we
> should do it really carefully. [...]

Re prompt injections, this is of course always a concern, and my yolo
attitude may not appeal to everyone. But I have to say since almost a
year of running AI reviews on BPF this hasn't been a problem once.

This of course depends on how the infrastructure is set up. BPF bot is
quite well isolated:
  * dedicated AWS account
  * each job runs on an ephemeral runner: not only new container, but
    also ephemeral hosts (we use CodeBuild)
  * the API access to inference is limited to 1h in a job
  * the AWS and github access tokens have limited permissions

With all that, I sleep well.

One can imagine a sci-fi scenario with a prompt injection tiggering a
bot to take over the AWS account, but given corporate monitoring of
activity there, we'd detect it and turn everything off immediately.

I also think that as the end-users of the LLM APIs we automatically
benefit from all the protections on the provider side. Sometimes this
is even annoying, because some models refuse to investigate certain
issues due to the guardrails on the backend.


> And outside of security considerations,
> humans are simple wrong too. So my plan with sashiko is to force it to
> try to verify the feedback against the codebase and if it's not possible
> trust only maintainers or people with a long history of meaningful
> contributions.

I have small local datasets generated from mailing lists, and one way
I tried to extract signal is to check whether the feedback from the
bot was incorporated in the next revision of the patches. Some people
do that silently without crediting the bots. So yeah, code is the
authority. Reputation plays some role of course, but in the end the
bots should check the claims against the code and discussion history.

> And sashiko can (and does in my prototype) generate prompts based on the
> feedback and automatically verify that it helps by re-reviewing original
> patches. This should mostly eliminate a need for human-generated
> prompts.

I did this manually a few times over the year: analyse lore -> come up
with prompt improvements -> test them -> merge.

This is definitely automatable and can close the loop to a large
degree. Prompting and context management is not that far from actual
machine learning, as it turns out :)


> This is all doable, but probably will take few more week to build and
> roll out.