Re: Sashiko review emails to the list

[Roman Gushchin <roman.gushchin@linux.dev>]

> On Jun 19, 2026, at 9:45 AM, Oliver Upton <oupton@kernel.org> wrote:
> 
> Hey,
> 
>> On Fri, Jun 19, 2026 at 03:19:05PM +0100, Fuad Tabba wrote:
>> Hi folks,
>> 
>> I really like Sashiko and find it very useful. It's flagged real bugs
>> in series I and others have posted to the list (e.g. [1][2][3]), and I
>> run it locally before sending, which has saved me a few respins.
>> 
>> That said, it's been posting a lot lately, so it seemed worth asking
>> how the review emails to the list are working out, and whether we
>> should change anything.
> 
> So this is entirely my fault since I added the email configuration for
> the kvmarm list. Sashiko has been finding some truly nasty bugs, posting
> on-list is the easiest way to get attention from the right folks to get
> things fixed.
> 
> With that being said, the signal to noise ratio hasn't been ideal.
> 
>> These fixes will take a while to land, so the question is what to do
>> meanwhile. Some options, and surely others:
>> 
>>  - Leave it as-is while the fixes propagate.
>>  - Stop the emails but keep the reviews on sashiko.dev, so people can
>> look rather than have them pushed.
>>  - Disable the emails until the noise is down to a reasonable level,
>> then re-enable.
> 
> This sounds like the right approach. I'd like to re-enable emails once
> we're happy with the quality of reviews.

From my perspective there are 3 main factors affecting the quality of reviews:
1) llm model capabilities. we have little control here, but it’s reasonable to expect that things will get better.
2) sashiko’s common code/harness. we’re improving it, but I’m not sure we have a lot of room left here, probably some. also, there are many tradeoffs to make, e.g. if we start verifying each issue separately, it almost certainly will improve signal/noise, but it will require way more tokens and will be slower.
3) developing per-subsystem prompts. there is a lot of potential here, but this is where we mostly rely on maintainers and developers.

Also not trying to push back on the decision, but I think it’s worth asking what is a reasonable level?
I tried to measure the true positive rate several times based on human feedback and it always was at least ~80% (and usually more for critical/high severity bugs).
And I’m afraid that it won’t be ~100% without compromising on the ability to find bugs. The reality is that there is a significant percentage of issues which are not exactly black and white and even people don’t necessarily agree if it’s an issue or not. Also there is a non-trivial amount of cases when ai findings are incorrectly dismissed by humans.

I’m not trying to pretend it’s perfect (it’s not), and I certainly expect it to be better going forward (and I’m working on it),
but the point is that we might not see _dramatic_ improvements going forward simple because there is no room left for it,
it’s more like a long tail of grey zone issues.

> On another note: Roman, is it possible to separately report pre-existing
> issues from findings in a patch? Maintainers have a higher likelihood of
> caring about these than individual contributors anyway.

It’s certainly possible, but how exactly do you see it?

It’s already somewhat separated (separate counters, separate list on top of each email).
I don’t want to stop reporting them completely (but we can discuss it), because it produces
a constant stream of fixes in the upstream kernel (in hundreds already).

I plan to improve it to e.g. not report multiple times within the same patchset.

Thanks