Re: Stop false review statements

Re: Stop false review statements

[Roman Gushchin]

> On May 17, 2026, at 11:56 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> On Sun, 17 May 2026 11:17:06 -0700
> Roman Gushchin <roman.gushchin@linux.dev> wrote:
> 
>>> On May 17, 2026, at 9:40 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>>> On Sun, 17 May 2026 12:12:00 +0200
>>> Greg KH <gregkh@linuxfoundation.org> wrote:
>>>>> On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
>>>>> On Sat, 16 May 2026 14:59:44 -0700
>>>>> Roman Gushchin <roman.gushchin@linux.dev> wrote:
>>>>>>> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
>>>>>>> I find it opposite: clogging commits with useless information, because
>>>>>>> some arbitrary and completely closed-source tool did analysis means
>>>>>>> nothing to me one year later when I look at the commit in the Git history.      
>>>>>> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
>>>>>> and the code belongs to LF.     
>>>>>> Yes, the instance behind sashiko.dev is using
>>>>>> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation -
>>>>>> Sashiko is supporting various LLMs, including open models - it’s just a practical
>>>>>> choice: to my knowledge the quality of open models is not on par with frontier closed
>>>>>> models     
>>>>> I would very much prefer using an open source LLM, even if not in pair
>>>>> with latest paid models.
>>>>>> and it would require a non-trivial amount of hardware and infrastructure to run
>>>>>> an open model at the required scale.    
>>>>> IMHO the best would be to have them running on some infra that would accept
>>>>> open source models (*). If there aren't enough resources to have our own
>>>>> infra, there are offers out there which allows running open source models
>>>>> like https://ollama.com/pricing (I never used myself).
>>>>> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
>>>>>   Not bad on my tests running it locally.    
>>>> You can run the tool locally, with whatever model you want, if you want
>>>> to.
>>>> But for now, let's just take the free credits that Google is willing to
>>>> throw at this thing and let it give us reviews IF the maintainer of the
>>>> subsystem feels it is something they want to do.  No one is forcing
>>>> maintainers to do this.  
>>> If Google and/or others are willing to give free credits on their cloud,
>>> they could instead or in addition give free credits to run ollama
>>> there, allowing us to use different models.
>>> From my side, while I won't personally object getting reviews from
>>> Sashiko/Gemini, this is something I can't reproduce locally. I would
>>> very much want something where I can select my LLM preferred model
>>> and run on my ollama docker container on my own GPU, in a way that
>>> I could run it locally before even sending a patch series.  
>> 
>> 2 thoughts here:
>> 1) I actually tried to run it with ollama on my personal framework 13. Adding nominal support is trivial,
>> but the whole thing is not really useful: I can get maybe few hundreds tokens per second using
>> a quantified model with reduced quality; an average sashiko review is consuming 3.5 millions tokens
>> (with Gemini 3.1 pro, it’s also model-dependent).
> 
> Do you mean 3.5 millions tokens per patch series? If so, that
> sounds a lot! Why does it require too many tokens?

It’s an average per patch, not a series. Some are much cheaper, some are much more expensive.
Sashiko posts token cost nearby each review.

Why it uses many tokens? Because in many cases it has to dig deep into the code.
Long sessions with multiple tool calls are expensive. Also Sashiko has a multi-stage
architecture, effectively it reviews every patch multiple times from different angles.
It has a measurable influence on the quality of reviews. The current generation of LLMs
is not good at spotting various types of issues at once: once it sees a memory leak
it can’t think anymore on e.g. locking issues. Also just by running the same thing multiple times
and combining the result you can meaningfully improve the quality.

>> I’m personally all in on having the entire thing as open as possible and I believe Sashiko is what
>> is realistically the best at this moment - a fully open-source harness and set of prompts which
>> can work with a variety of models.
>> I’m happy to merge a support for any LLM model which can produce decent review results.
>> 
>> 2) Due to probabilistic nature of LLMs, nothing is reproducible in a strict sense of the word.
>> Even with exactly the same model/harness/prompts you’ll get different results every time you run it.
>> It’s unfortunate, but it is what it is at the moment.
> 
> By "reproduce locally", I didn't mean in strict sense. Sure, LLM answers
> won't be identical, but I suspect that at least most of the major issues
> on a patch series would be reported by any decent model.

I believe we’re not quite there yet. Models do differ in their abilities to spot
various types of bugs and also producing false positives. Some types of issues
(e.g. complex locking issues) are really hard for best of the current models.

> So, if we have something that one can locally run using its GPU, being
> able to get an answer in the range of a couple of minutes per patch
> should be enough to catch most of the issues.

I’m happy to be wrong here, but my understanding is that it’s not realistic now.
Sashiko reviews taking longer with production grade hardware.

Re: Stop false review statements

[Roman Gushchin]


> On May 17, 2026, at 11:57 AM, Theodore Tso <tytso@mit.edu> wrote:
> On Sun, May 17, 2026 at 11:17:06AM -0700, Roman Gushchin wrote:
>> 
>> I actually tried to run it with ollama on my
>> personal framework 13. Adding nominal support is trivial, but the
>> whole thing is not really useful: I can get maybe few hundreds
>> tokens per second using a quantified model with reduced quality; an
>> average sashiko review is consuming 3.5 millions tokens (with Gemini
>> 3.1 pro, it’s also model-dependent).
> 
> I'm curious.  What hardware and LLM model were you using?  A few
> hundred tokens per second seems surprising high.  My initial
> research[1] showes that an M5 Max Macbook Pro costing 5 or 6 kilobucks
> can do 31.6 tokens/second on a 27B 4-bit Quanitized model (Qwen 3.5).

I’ve framework 13 with amd 7840u. I’ve tried several models both on cpu and gpu. 
Sorry, it was a couple of months ago and I don’t remember all the details, so I won’t 
claim any specific numbers, but as I remember the best numbers were around 
a hundred tokens per second. In any case it’s few orders of magnitude slower than
 what is realistically required.

If someone has a powerful hardware and is willing to benchmark sashiko with open-source
models, I’m very interested in results.

> [1] https://www.reddit.com/r/LocalLLaMA/comments/1rzkw4x/m5_max_128g_performance_tests_i_just_got_my_new/
> 
> The model matters of course.  With Gemma 3 27B and a 6-bit
> quantization, it's 21 tokens/s, and with Deepseek R1 8B Q6_K, it's
> 72.8 tokens/second.  But unless you're using a really low-end model,
> or a really expensive, splufty hardware platform, I haven't seen
> reports of hundreds of tokens per second on hardware costing a
> reasonable amount of memory.  (I'll set aside the question of whether
> spending $6k for a fully spec'ed out M5 Max Macbook Pro, or $15k for a
> fully spec'ed out M3 Ultra Mac Studio is "reasonable".)
> 
> As a result I'm not entirely sure how realistic it is to do reviews
> using "free" (you still have to pay $$$ for the hardware) local,
> open-weight LLM's if an average review requires around 3.5 million
> tokens.

Fully agree. But it might change in few years, things are moving quickly.

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sun, 17 May 2026 12:42:12 -0700
Roman Gushchin <roman.gushchin@linux.dev> wrote:

> 
> > On May 17, 2026, at 11:57 AM, Theodore Tso <tytso@mit.edu> wrote:
> > On Sun, May 17, 2026 at 11:17:06AM -0700, Roman Gushchin wrote:  
> >> 
> >> I actually tried to run it with ollama on my
> >> personal framework 13. Adding nominal support is trivial, but the
> >> whole thing is not really useful: I can get maybe few hundreds
> >> tokens per second using a quantified model with reduced quality; an
> >> average sashiko review is consuming 3.5 millions tokens (with Gemini
> >> 3.1 pro, it’s also model-dependent).  
> > 
> > I'm curious.  What hardware and LLM model were you using?  A few
> > hundred tokens per second seems surprising high.  My initial
> > research[1] showes that an M5 Max Macbook Pro costing 5 or 6 kilobucks
> > can do 31.6 tokens/second on a 27B 4-bit Quanitized model (Qwen 3.5).  
> 
> I’ve framework 13 with amd 7840u. I’ve tried several models both on cpu and gpu. 
> Sorry, it was a couple of months ago and I don’t remember all the details, so I won’t 
> claim any specific numbers, but as I remember the best numbers were around 
> a hundred tokens per second. In any case it’s few orders of magnitude slower than
>  what is realistically required.
> 
> If someone has a powerful hardware and is willing to benchmark sashiko with open-source
> models, I’m very interested in results.

If you add the patch you used with ollama somewhere, I can try
running here and do some benchmarks - that is assuming that 
it won't try to run 3.5 millions of tokens.


> 
> > [1] https://www.reddit.com/r/LocalLLaMA/comments/1rzkw4x/m5_max_128g_performance_tests_i_just_got_my_new/
> > 
> > The model matters of course.  With Gemma 3 27B and a 6-bit
> > quantization, it's 21 tokens/s, and with Deepseek R1 8B Q6_K, it's
> > 72.8 tokens/second.  But unless you're using a really low-end model,
> > or a really expensive, splufty hardware platform, I haven't seen
> > reports of hundreds of tokens per second on hardware costing a
> > reasonable amount of memory.  (I'll set aside the question of whether
> > spending $6k for a fully spec'ed out M5 Max Macbook Pro, or $15k for a
> > fully spec'ed out M3 Ultra Mac Studio is "reasonable".)
> > 
> > As a result I'm not entirely sure how realistic it is to do reviews
> > using "free" (you still have to pay $$$ for the hardware) local,
> > open-weight LLM's if an average review requires around 3.5 million
> > tokens.  
> 
> Fully agree. But it might change in few years, things are moving quickly.


Thanks,
Mauro

Stop false review statements

[Krzysztof Kozlowski]

What the hell is that:

https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/

As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
not a damn human do be able to make such statement. You are a bot, a tool.

Stop faking tags.

And really, considering how many false positives Sashiko produces, how
poor review comments it gives, how many misleading comments, it's
unacceptable to me to consider that a review.

Amount of useless noise Sashiko produces already changed my mind how
useful that tool is.

I will be NAKing every damn tag produced by such tools.


Best regards,
Krzysztof

Re: Stop false review statements

[Guenter Roeck]

On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> What the hell is that:
> 
> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> 
> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> not a damn human do be able to make such statement. You are a bot, a tool.
> 

Where exactly do the rules say that ? I seem to miss that.

There is a policy document about _contributions_ made by AI, but I don't
see the one that says that AI agents must not provide Reviewed-by: tags.

> Stop faking tags.
> 
> And really, considering how many false positives Sashiko produces, how
> poor review comments it gives, how many misleading comments, it's
> unacceptable to me to consider that a review.
> 
> Amount of useless noise Sashiko produces already changed my mind how
> useful that tool is.

We seem to have completely different experiences. Yes, it does produce
false positives, just like humans do. However, I have seen it find many
real bugs, including many in patches which already had Reviewed-by: tags
from (presumably) human reviewers.

Again, it appears that our experience is completely different than mine,
but after several weeks of getting code reviews from sashiko I do have to
say that I trust its review feedback significantly more than human reviews.
Sure, it does not guarantee that a patch is indeed bug free. A human review
doesn't guarantee it either.

> 
> I will be NAKing every damn tag produced by such tools.

I'd like to see an official policy. Until then I'll ignore your NAK in my
scope of responsibility.

Thanks,
Guenter

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 14:11, Guenter Roeck wrote:
> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>> What the hell is that:
>>
>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>
>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>> not a damn human do be able to make such statement. You are a bot, a tool.
>>
> 
> Where exactly do the rules say that ? I seem to miss that.
> 
> There is a policy document about _contributions_ made by AI, but I don't
> see the one that says that AI agents must not provide Reviewed-by: tags.

Quotes from the existing policy:

1. "By offering my Reviewed-by: tag, I state that:"

Tool cannot use first person "I". Tool cannot "state that".

2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
 appropriate modification of the kernel without any remaining serious"

Tool cannot make a statement of opinion.

3. "Any interested reviewer (who has done the work) can offer a
Reviewed-by".

Tool is not a reviewer as a person, thus above does not grant the tool
permission to offer a tag.

> 
>> Stop faking tags.
>>
>> And really, considering how many false positives Sashiko produces, how
>> poor review comments it gives, how many misleading comments, it's
>> unacceptable to me to consider that a review.
>>
>> Amount of useless noise Sashiko produces already changed my mind how
>> useful that tool is.
> 
> We seem to have completely different experiences. Yes, it does produce
> false positives, just like humans do. However, I have seen it find many
> real bugs, including many in patches which already had Reviewed-by: tags
> from (presumably) human reviewers.

Of course it finds bugs. But it also produces - roughly - 80-90% false
positives, completely useless.

This is very poor review score.

> 
> Again, it appears that our experience is completely different than mine,
> but after several weeks of getting code reviews from sashiko I do have to
> say that I trust its review feedback significantly more than human reviews.
> Sure, it does not guarantee that a patch is indeed bug free. A human review
> doesn't guarantee it either.
> 
>>
>> I will be NAKing every damn tag produced by such tools.
> 
> I'd like to see an official policy. Until then I'll ignore your NAK in my
> scope of responsibility.

:)

Best regards,
Krzysztof

Re: Stop false review statements

[Guenter Roeck]

On 5/16/26 05:16, Krzysztof Kozlowski wrote:
> On 16/05/2026 14:11, Guenter Roeck wrote:
>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>> What the hell is that:
>>>
>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>
>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>
>>
>> Where exactly do the rules say that ? I seem to miss that.
>>
>> There is a policy document about _contributions_ made by AI, but I don't
>> see the one that says that AI agents must not provide Reviewed-by: tags.
> 
> Quotes from the existing policy:
> 
> 1. "By offering my Reviewed-by: tag, I state that:"
> 
> Tool cannot use first person "I". Tool cannot "state that".
> 
> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>   appropriate modification of the kernel without any remaining serious"
> 
> Tool cannot make a statement of opinion.
> 
> 3. "Any interested reviewer (who has done the work) can offer a
> Reviewed-by".
> 
> Tool is not a reviewer as a person, thus above does not grant the tool
> permission to offer a tag.
> 

I'd like to see that explicitly spelled out. Until then it is your opinion.

>>
>>> Stop faking tags.
>>>
>>> And really, considering how many false positives Sashiko produces, how
>>> poor review comments it gives, how many misleading comments, it's
>>> unacceptable to me to consider that a review.
>>>
>>> Amount of useless noise Sashiko produces already changed my mind how
>>> useful that tool is.
>>
>> We seem to have completely different experiences. Yes, it does produce
>> false positives, just like humans do. However, I have seen it find many
>> real bugs, including many in patches which already had Reviewed-by: tags
>> from (presumably) human reviewers.
> 
> Of course it finds bugs. But it also produces - roughly - 80-90% false
> positives, completely useless.
> 

Really ? The ones I have seen are - roughly, to use the same term - 80-90%
true positives. Maybe you should explicitly ask for no Sashiko reviews in
your scope of responsibility.

Thanks,
Guenter

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 14:23, Guenter Roeck wrote:
> On 5/16/26 05:16, Krzysztof Kozlowski wrote:
>> On 16/05/2026 14:11, Guenter Roeck wrote:
>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>>> What the hell is that:
>>>>
>>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>>
>>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>>
>>>
>>> Where exactly do the rules say that ? I seem to miss that.
>>>
>>> There is a policy document about _contributions_ made by AI, but I don't
>>> see the one that says that AI agents must not provide Reviewed-by: tags.
>>
>> Quotes from the existing policy:
>>
>> 1. "By offering my Reviewed-by: tag, I state that:"
>>
>> Tool cannot use first person "I". Tool cannot "state that".
>>
>> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>>   appropriate modification of the kernel without any remaining serious"
>>
>> Tool cannot make a statement of opinion.
>>
>> 3. "Any interested reviewer (who has done the work) can offer a
>> Reviewed-by".
>>
>> Tool is not a reviewer as a person, thus above does not grant the tool
>> permission to offer a tag.
>>
> 
> I'd like to see that explicitly spelled out. Until then it is your opinion.

It is not an opinion. It is written. I gave you quotes.

Do you want to spell the rules of English language? That tool is not a
person?

Shall I send the patch like:

  Any interested reviewer (who has done the work) can offer a
  Reviewed-by.
 +In English "reviewer" is a person [1].
 + [1] https://en.wiktionary.org/wiki/reviewer

Seriously, you expect to document the English language?

> 
>>>
>>>> Stop faking tags.
>>>>
>>>> And really, considering how many false positives Sashiko produces, how
>>>> poor review comments it gives, how many misleading comments, it's
>>>> unacceptable to me to consider that a review.
>>>>
>>>> Amount of useless noise Sashiko produces already changed my mind how
>>>> useful that tool is.
>>>
>>> We seem to have completely different experiences. Yes, it does produce
>>> false positives, just like humans do. However, I have seen it find many
>>> real bugs, including many in patches which already had Reviewed-by: tags
>>> from (presumably) human reviewers.
>>
>> Of course it finds bugs. But it also produces - roughly - 80-90% false
>> positives, completely useless.
>>
> 
> Really ? The ones I have seen are - roughly, to use the same term - 80-90%
> true positives. Maybe you should explicitly ask for no Sashiko reviews in
> your scope of responsibility.

I already sent a patch to stop receiving all these emails and I stopped
reading them completely, when fetched via b4 for review in mutt workflow.

But this is not the point.

Our docs clearly state what Reviewed-by means, regardless of the quality
of the actual review. Poor quality is just another reason, less
important, though.

Best regards,
Krzysztof

Re: Stop false review statements

[Laurent Pinchart]

On Sat, May 16, 2026 at 02:29:15PM +0200, Krzysztof Kozlowski wrote:
> On 16/05/2026 14:23, Guenter Roeck wrote:
> > On 5/16/26 05:16, Krzysztof Kozlowski wrote:
> >> On 16/05/2026 14:11, Guenter Roeck wrote:
> >>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> >>>> What the hell is that:
> >>>>
> >>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> >>>>
> >>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> >>>> not a damn human do be able to make such statement. You are a bot, a tool.
> >>>>
> >>>
> >>> Where exactly do the rules say that ? I seem to miss that.
> >>>
> >>> There is a policy document about _contributions_ made by AI, but I don't
> >>> see the one that says that AI agents must not provide Reviewed-by: tags.
> >>
> >> Quotes from the existing policy:
> >>
> >> 1. "By offering my Reviewed-by: tag, I state that:"
> >>
> >> Tool cannot use first person "I". Tool cannot "state that".
> >>
> >> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
> >>   appropriate modification of the kernel without any remaining serious"
> >>
> >> Tool cannot make a statement of opinion.
> >>
> >> 3. "Any interested reviewer (who has done the work) can offer a
> >> Reviewed-by".
> >>
> >> Tool is not a reviewer as a person, thus above does not grant the tool
> >> permission to offer a tag.
> > 
> > I'd like to see that explicitly spelled out. Until then it is your opinion.
> 
> It is not an opinion. It is written. I gave you quotes.
> 
> Do you want to spell the rules of English language? That tool is not a
> person?
> 
> Shall I send the patch like:
> 
>   Any interested reviewer (who has done the work) can offer a
>   Reviewed-by.
>  +In English "reviewer" is a person [1].
>  + [1] https://en.wiktionary.org/wiki/reviewer
> 
> Seriously, you expect to document the English language?
> 
> >>>> Stop faking tags.
> >>>>
> >>>> And really, considering how many false positives Sashiko produces, how
> >>>> poor review comments it gives, how many misleading comments, it's
> >>>> unacceptable to me to consider that a review.
> >>>>
> >>>> Amount of useless noise Sashiko produces already changed my mind how
> >>>> useful that tool is.

Note this isn't en entirely new situation. As a maintainer, you know how
much you trust each reviewer. You will consider some R-b tags as a sign
you don't even have to look at a patch, and will completely ignore some
others. There's a whole continuum in the middle. In some ways, reviews
by an LLM are similar. You will trust them or not trust them.

Except they're also very different.

The kernel needs more skilled reviewers (I don't think this is a
controversial statement). We can't expect all newcomers to start with
extensive experience from day one, so there's a learning curve. I
believe it's fine for more junior reviewers to send R-b tags even if
they miss some issue, as long as they genuinely try and improve (and, in
some unfortunate cases, decide to leave if patch review turns out not to
be for them). Those R-b tags may feel like a bit of noise in the
beginning, but that's compensated by their value increasing over time.

Bot reviews are not the same. Not only are they generated at a much
larger scale than human reviews, they also won't learn from feedback you
give them. Sure, the tools may be improved when cases of false positives
are identified, and new LLMs may be trained with more (and better ?)
data to improve the output, but they won't learn from the interactions.

How much value a maintainer sees in those reviews is up to individual
maintainers. I will personally not consider a R-b tag from an LLM to
mean that a patch is ready to be merged (and I believe you won't
either). As such, I think that a R-b from an LLM is misleading and
doesn't provide good value. At best it's free advertising for company
making closed-source tools, which I don't think we should encourage.

If some maintainers want LLM reviews and want to act on them, that's
their personal prerogative. They're free to decide on how much value
they see in those reviews, as well as on whether or not they consider
usage of those tools compatible with FOSS ethics. Those are personal
decisions. However, given that the ethical decision is personal, I am
strongly against forcing patch authors to act on automated LLM review.

> >>> We seem to have completely different experiences. Yes, it does produce
> >>> false positives, just like humans do. However, I have seen it find many
> >>> real bugs, including many in patches which already had Reviewed-by: tags
> >>> from (presumably) human reviewers.
> >>
> >> Of course it finds bugs. But it also produces - roughly - 80-90% false
> >> positives, completely useless.
> >>
> > 
> > Really ? The ones I have seen are - roughly, to use the same term - 80-90%
> > true positives. Maybe you should explicitly ask for no Sashiko reviews in
> > your scope of responsibility.
> 
> I already sent a patch to stop receiving all these emails and I stopped
> reading them completely, when fetched via b4 for review in mutt workflow.
> 
> But this is not the point.
> 
> Our docs clearly state what Reviewed-by means, regardless of the quality
> of the actual review. Poor quality is just another reason, less
> important, though.

-- 
Regards,

Laurent Pinchart

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 15:24, Laurent Pinchart wrote:
> On Sat, May 16, 2026 at 02:29:15PM +0200, Krzysztof Kozlowski wrote:
>> On 16/05/2026 14:23, Guenter Roeck wrote:
>>> On 5/16/26 05:16, Krzysztof Kozlowski wrote:
>>>> On 16/05/2026 14:11, Guenter Roeck wrote:
>>>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>>>>> What the hell is that:
>>>>>>
>>>>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>>>>
>>>>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>>>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>>>>
>>>>>
>>>>> Where exactly do the rules say that ? I seem to miss that.
>>>>>
>>>>> There is a policy document about _contributions_ made by AI, but I don't
>>>>> see the one that says that AI agents must not provide Reviewed-by: tags.
>>>>
>>>> Quotes from the existing policy:
>>>>
>>>> 1. "By offering my Reviewed-by: tag, I state that:"
>>>>
>>>> Tool cannot use first person "I". Tool cannot "state that".
>>>>
>>>> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>>>>   appropriate modification of the kernel without any remaining serious"
>>>>
>>>> Tool cannot make a statement of opinion.
>>>>
>>>> 3. "Any interested reviewer (who has done the work) can offer a
>>>> Reviewed-by".
>>>>
>>>> Tool is not a reviewer as a person, thus above does not grant the tool
>>>> permission to offer a tag.
>>>
>>> I'd like to see that explicitly spelled out. Until then it is your opinion.
>>
>> It is not an opinion. It is written. I gave you quotes.
>>
>> Do you want to spell the rules of English language? That tool is not a
>> person?
>>
>> Shall I send the patch like:
>>
>>   Any interested reviewer (who has done the work) can offer a
>>   Reviewed-by.
>>  +In English "reviewer" is a person [1].
>>  + [1] https://en.wiktionary.org/wiki/reviewer
>>
>> Seriously, you expect to document the English language?
>>
>>>>>> Stop faking tags.
>>>>>>
>>>>>> And really, considering how many false positives Sashiko produces, how
>>>>>> poor review comments it gives, how many misleading comments, it's
>>>>>> unacceptable to me to consider that a review.
>>>>>>
>>>>>> Amount of useless noise Sashiko produces already changed my mind how
>>>>>> useful that tool is.
> 
> Note this isn't en entirely new situation. As a maintainer, you know how
> much you trust each reviewer. You will consider some R-b tags as a sign
> you don't even have to look at a patch, and will completely ignore some
> others. There's a whole continuum in the middle. In some ways, reviews
> by an LLM are similar. You will trust them or not trust them.
> 
> Except they're also very different.
> 
> The kernel needs more skilled reviewers (I don't think this is a
> controversial statement). We can't expect all newcomers to start with
> extensive experience from day one, so there's a learning curve. I
> believe it's fine for more junior reviewers to send R-b tags even if
> they miss some issue, as long as they genuinely try and improve (and, in
> some unfortunate cases, decide to leave if patch review turns out not to
> be for them). Those R-b tags may feel like a bit of noise in the
> beginning, but that's compensated by their value increasing over time.

Yes, I agree. Reviews from inexperienced people are sometimes fruitless
or pointless per actual value they bring, but they allow a person
(again: person) to grow in the community with a credits being the reward.

> 
> Bot reviews are not the same. Not only are they generated at a much
> larger scale than human reviews, they also won't learn from feedback you
> give them. Sure, the tools may be improved when cases of false positives
> are identified, and new LLMs may be trained with more (and better ?)
> data to improve the output, but they won't learn from the interactions.
> 
> How much value a maintainer sees in those reviews is up to individual
> maintainers. I will personally not consider a R-b tag from an LLM to
> mean that a patch is ready to be merged (and I believe you won't
> either). As such, I think that a R-b from an LLM is misleading and
> doesn't provide good value. At best it's free advertising for company
> making closed-source tools, which I don't think we should encourage.

That's different aspect than I raised. I agree with above approach but
it is more subjective.

What I brought is object: our docs clearly state that reviewer can offer
reviewed-by tag. They do not allow non-reviewers to offer a tag and
English is clear on that - only a person is a reviewer.

Dog is not a reviewer.

Hammer is not a reviewer.

Tool is not a reviewer.

Guenter did not bring any counter arguments that our docs ALLOW
non-person to provide a reviewed-by tag. I brought that arguments as
excerpt from our documented policy.

Best regards,
Krzysztof

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sat, 16 May 2026 16:24:07 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote:

> On Sat, May 16, 2026 at 02:29:15PM +0200, Krzysztof Kozlowski wrote:
> > On 16/05/2026 14:23, Guenter Roeck wrote:  
> > > On 5/16/26 05:16, Krzysztof Kozlowski wrote:  
> > >> On 16/05/2026 14:11, Guenter Roeck wrote:  
> > >>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:  
> > >>>> What the hell is that:
> > >>>>
> > >>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> > >>>>
> > >>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> > >>>> not a damn human do be able to make such statement. You are a bot, a tool.
> > >>>>  
> > >>>
> > >>> Where exactly do the rules say that ? I seem to miss that.
> > >>>
> > >>> There is a policy document about _contributions_ made by AI, but I don't
> > >>> see the one that says that AI agents must not provide Reviewed-by: tags.  
> > >>
> > >> Quotes from the existing policy:
> > >>
> > >> 1. "By offering my Reviewed-by: tag, I state that:"
> > >>
> > >> Tool cannot use first person "I". Tool cannot "state that".
> > >>
> > >> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
> > >>   appropriate modification of the kernel without any remaining serious"
> > >>
> > >> Tool cannot make a statement of opinion.
> > >>
> > >> 3. "Any interested reviewer (who has done the work) can offer a
> > >> Reviewed-by".
> > >>
> > >> Tool is not a reviewer as a person, thus above does not grant the tool
> > >> permission to offer a tag.  
> > > 
> > > I'd like to see that explicitly spelled out. Until then it is your opinion.  
> > 
> > It is not an opinion. It is written. I gave you quotes.
> > 
> > Do you want to spell the rules of English language? That tool is not a
> > person?
> > 
> > Shall I send the patch like:
> > 
> >   Any interested reviewer (who has done the work) can offer a
> >   Reviewed-by.
> >  +In English "reviewer" is a person [1].
> >  + [1] https://en.wiktionary.org/wiki/reviewer

Agreed. Reviewed-by doesn't apply.

> > 
> > Seriously, you expect to document the English language?
> >   
> > >>>> Stop faking tags.
> > >>>>
> > >>>> And really, considering how many false positives Sashiko produces, how
> > >>>> poor review comments it gives, how many misleading comments, it's
> > >>>> unacceptable to me to consider that a review.
> > >>>>
> > >>>> Amount of useless noise Sashiko produces already changed my mind how
> > >>>> useful that tool is.  
> 
> Note this isn't en entirely new situation. As a maintainer, you know how
> much you trust each reviewer. You will consider some R-b tags as a sign
> you don't even have to look at a patch, and will completely ignore some
> others. There's a whole continuum in the middle. In some ways, reviews
> by an LLM are similar. You will trust them or not trust them.
> 
> Except they're also very different.

This is not different than Coverity, Smatch, syzbot or any other robots (*).

(*) except for the ones that report actual build failures after trying
    to compile the Kernel.

So far, I got just one report from sashiko-bot on a patch series
I wrote for linux-doc. It did get some issues, but it also had false 
positives and it was too verbose for my taste, trying to explain
me why I wrote patches or why I did some changes, and writing 9
patch replies to a 13-patch series.

Even as a developer, I would expect a much cleaner output - and
with my maintainer's hat, just a single e-mail briefly giving
a summary of issues detected at the patch series as a hole
(e.g. just one short e-mail).

>  As such, I think that a R-b from an LLM is misleading and
> doesn't provide good value. At best it's free advertising for company
> making closed-source tools, which I don't think we should encourage.

I disagree. The earliest an issue is reported/fixed, the better.

As a maintainer, I do want that patch developers take reports from 
any bots into account, no matter what technology they use or if
the bot is sponsored by someone or not. Sure, open-source tools
are preferred, but we should not close our eyes to issues,
specially if they can end adding vulnerabilities at the Kernel if
the patches end being applied.

With that in mind, even if LLMs are giving just 20-30% accuracy
of a reviewer, but it has high accuracy picking security issues
that would otherwise be exploited by some LLM tool, I'd say it is
valuable enough.

That said, for LLM tools, I'd say that the report to maintainers/ML
need to be very short:

- What problems a patch series may have in terms of security
  (OOM? privilege escalation? ...);
- What major issues a patch series may have;
- Other medium issues(*).

(*) Issues that are already reported by bots like LKP shall not
    be reported from LLMs to maintainers, as we want to minimize
    duplicated information sent to maintainers as much as possible,
    and LLMs don't replace code compilers nor tools like checkpatch,
    which are more specialized.


Thanks,
Mauro

Re: Stop false review statements

[Jonathan Corbet]

Guenter Roeck <linux@roeck-us.net> writes:

> On 5/16/26 05:16, Krzysztof Kozlowski wrote:
>> Quotes from the existing policy:
>> 
>> 1. "By offering my Reviewed-by: tag, I state that:"
>> 
>> Tool cannot use first person "I". Tool cannot "state that".
>> 
>> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>>   appropriate modification of the kernel without any remaining serious"
>> 
>> Tool cannot make a statement of opinion.
>> 
>> 3. "Any interested reviewer (who has done the work) can offer a
>> Reviewed-by".
>> 
>> Tool is not a reviewer as a person, thus above does not grant the tool
>> permission to offer a tag.
>
> I'd like to see that explicitly spelled out. Until then it is your opinion.

So I'm the person who wrote that text.  Automated review tools weren't
really on the radar at that time, so I can't argue that it expresses an
opinion either way as to whether an LLM could make such assertions.

That said, I was certainly considering *human* reviewers at the time,
and all of the people who agreed with the suggested policy were too.
Adding bots seems like a stretch to me.

I can't speak for subsystems that require Reviewed-by tags on their
commits, but I'm not sure that their maintainers would accept an
automated review as satisfying that requirement.

If we want to record this sort of processing, perhaps a tag like
"Scanned-by" would be appropriate?

jon

Re: Stop false review statements

[Konstantin Ryabitsev]

On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> > What the hell is that:
> > 
> > https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> > 
> > As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> > not a damn human do be able to make such statement. You are a bot, a tool.
> > 
> 
> Where exactly do the rules say that ? I seem to miss that.
> 
> There is a policy document about _contributions_ made by AI, but I don't
> see the one that says that AI agents must not provide Reviewed-by: tags.

From my perspective, AI agents must NOT use the Reviewed-by tag for the
following reasons:

- We consider this a "person-trailer" and it implies agency
- Adding yourself to a commit via a trailer is a *binding responsibility* for
  the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
  messages regarding code in this commit. If the address is bogus or doesn't
  go to a developer, this is both wasteful and potentially frustrating.

-K

Re: Stop false review statements

[Greg KH]

On Sat, May 16, 2026 at 11:20:33AM -0400, Konstantin Ryabitsev wrote:
> On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
> > On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> > > What the hell is that:
> > > 
> > > https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> > > 
> > > As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> > > not a damn human do be able to make such statement. You are a bot, a tool.
> > > 
> > 
> > Where exactly do the rules say that ? I seem to miss that.
> > 
> > There is a policy document about _contributions_ made by AI, but I don't
> > see the one that says that AI agents must not provide Reviewed-by: tags.
> 
> >From my perspective, AI agents must NOT use the Reviewed-by tag for the
> following reasons:
> 
> - We consider this a "person-trailer" and it implies agency
> - Adding yourself to a commit via a trailer is a *binding responsibility* for
>   the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
>   messages regarding code in this commit. If the address is bogus or doesn't
>   go to a developer, this is both wasteful and potentially frustrating.

I agree, any sort of "automated" tool shouldn't be adding these types of
tags.

thanks,

greg k-h

Re: Stop false review statements

[Roman Gushchin]

> On May 16, 2026, at 8:20 AM, Konstantin Ryabitsev <mricon@kernel.org> wrote:
> 
> On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>> What the hell is that:
>>> 
>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>> 
>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>> 
>> 
>> Where exactly do the rules say that ? I seem to miss that.
>> 
>> There is a policy document about _contributions_ made by AI, but I don't
>> see the one that says that AI agents must not provide Reviewed-by: tags.
> 
> From my perspective, AI agents must NOT use the Reviewed-by tag for the
> following reasons:
> 
> - We consider this a "person-trailer" and it implies agency
> - Adding yourself to a commit via a trailer is a *binding responsibility* for
>  the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
>  messages regarding code in this commit. If the address is bogus or doesn't
>  go to a developer, this is both wasteful and potentially frustrating.

Hi Konstantin!

The goal here is to inform maintainers that sashiko has successfully reviewed the patch
and there were no findings, otherwise maintainers have to go to the web site and check the status.

I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form. 
And we use Reported-by: tags with various tooling for years.

What do you think is the best form?

I’ll pause sending reviewed-by tags until we have a discussion and agreement here.

Thanks

Re: Stop false review statements

[Greg KH]

On Sat, May 16, 2026 at 08:41:43AM -0700, Roman Gushchin wrote:
> 
> > On May 16, 2026, at 8:20 AM, Konstantin Ryabitsev <mricon@kernel.org> wrote:
> > 
> > On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
> >>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> >>> What the hell is that:
> >>> 
> >>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> >>> 
> >>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> >>> not a damn human do be able to make such statement. You are a bot, a tool.
> >>> 
> >> 
> >> Where exactly do the rules say that ? I seem to miss that.
> >> 
> >> There is a policy document about _contributions_ made by AI, but I don't
> >> see the one that says that AI agents must not provide Reviewed-by: tags.
> > 
> > From my perspective, AI agents must NOT use the Reviewed-by tag for the
> > following reasons:
> > 
> > - We consider this a "person-trailer" and it implies agency
> > - Adding yourself to a commit via a trailer is a *binding responsibility* for
> >  the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
> >  messages regarding code in this commit. If the address is bogus or doesn't
> >  go to a developer, this is both wasteful and potentially frustrating.
> 
> Hi Konstantin!
> 
> The goal here is to inform maintainers that sashiko has successfully reviewed the patch
> and there were no findings, otherwise maintainers have to go to the web site and check the status.

That's fine.

> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form. 
> And we use Reported-by: tags with various tooling for years.

Reported-by: shows the existance of a problem that some tool found, a
subtle difference here.

> What do you think is the best form?
> 
> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.

Just say it in some other text form, that our tools will not pick up.
Like:
	Tool XXXX reports that all is good:
		https://....

or something like that?

thanks,

greg k-h

Re: Stop false review statements

[Roman Gushchin]

> On May 16, 2026, at 8:45 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> On Sat, May 16, 2026 at 08:41:43AM -0700, Roman Gushchin wrote:
>> 
>>>> On May 16, 2026, at 8:20 AM, Konstantin Ryabitsev <mricon@kernel.org> wrote:
>>> 
>>> On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
>>>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>>>> What the hell is that:
>>>>> 
>>>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>>> 
>>>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>>> 
>>>> 
>>>> Where exactly do the rules say that ? I seem to miss that.
>>>> 
>>>> There is a policy document about _contributions_ made by AI, but I don't
>>>> see the one that says that AI agents must not provide Reviewed-by: tags.
>>> 
>>> From my perspective, AI agents must NOT use the Reviewed-by tag for the
>>> following reasons:
>>> 
>>> - We consider this a "person-trailer" and it implies agency
>>> - Adding yourself to a commit via a trailer is a *binding responsibility* for
>>> the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
>>> messages regarding code in this commit. If the address is bogus or doesn't
>>> go to a developer, this is both wasteful and potentially frustrating.
>> 
>> Hi Konstantin!
>> 
>> The goal here is to inform maintainers that sashiko has successfully reviewed the patch
>> and there were no findings, otherwise maintainers have to go to the web site and check the status.
> 
> That's fine.
> 
>> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
>> And we use Reported-by: tags with various tooling for years.
> 
> Reported-by: shows the existance of a problem that some tool found, a
> subtle difference here.
> 
>> What do you think is the best form?
>> 
>> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
> 
> Just say it in some other text form, that our tools will not pick up.
> Like:
>    Tool XXXX reports that all is good:
>        https://....
> 
> or something like that?

Sure, works for me.

Thanks,
Roman

Re: Stop false review statements

[Arnaldo Carvalho de Melo]

On Sat, May 16, 2026 at 08:49:39AM -0700, Roman Gushchin wrote:
> > On May 16, 2026, at 8:45 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > On Sat, May 16, 2026 at 08:41:43AM -0700, Roman Gushchin wrote:
> >>>> On May 16, 2026, at 8:20 AM, Konstantin Ryabitsev <mricon@kernel.org> wrote:

> >>> On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
> >>>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> >>>>> What the hell is that:
> 
> >>>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
 
> >>>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> >>>>> not a damn human do be able to make such statement. You are a bot, a tool.
 
> >>>> Where exactly do the rules say that ? I seem to miss that.
 
> >>>> There is a policy document about _contributions_ made by AI, but I don't
> >>>> see the one that says that AI agents must not provide Reviewed-by: tags.
 
> >>> From my perspective, AI agents must NOT use the Reviewed-by tag for the
> >>> following reasons:

> >>> - We consider this a "person-trailer" and it implies agency
> >>> - Adding yourself to a commit via a trailer is a *binding responsibility* for
> >>> the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
> >>> messages regarding code in this commit. If the address is bogus or doesn't
> >>> go to a developer, this is both wasteful and potentially frustrating.

> >> Hi Konstantin!

> >> The goal here is to inform maintainers that sashiko has successfully reviewed the patch
> >> and there were no findings, otherwise maintainers have to go to the web site and check the status.

> > That's fine.

> >> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
> >> And we use Reported-by: tags with various tooling for years.

> > Reported-by: shows the existance of a problem that some tool found, a
> > subtle difference here.

> >> What do you think is the best form?

> >> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
 
> > Just say it in some other text form, that our tools will not pick up.
> > Like:
> >    Tool XXXX reports that all is good:
> >        https://....
 
> > or something like that?
 
> Sure, works for me.

Couldn't this be something like:

AI-analysed-by: bot-X

Sometimes we expect them to run and produce results, that we should
check, and in my experience act upon and sometimes ignore, as usual with
any type of comments, when I don't get those results I usually go and
look at the web interface, be it the public one or the private one I use
(while contributing to one of them, sashiko for full disclosure), to see
if it is just that it is taking longer to process that specific patch or
if it really finished.

If I go in the morning to look at my inbox and see everything there it
reduces some steps for me.

So, yeah, Reviewed-by is definetly for definetly persons, but having
some tag that states that it went thru automated reviewing^Wanalysis by
a definetly bot/thing/whatever that some people think is useful seems
useful.

And of course I think this should be all opt-in by each and every
maintainer, this definetly is not something to be uniformly agreed on,
so if some mechanism is put in place for, say, sashiko, to reply that it
didn't find any problem with a particular patch for the subsystem I'm
involved with, then I'd like for that info to be provided as a reply to
the message.

Cheers,

- Arnaldo

Re: Stop false review statements

[Derek Barbosa]

On Sat, May 16, 2026 at 03:28:47PM -0300, Arnaldo Carvalho de Melo wrote:
> 
> Couldn't this be something like:
> 
> AI-analysed-by: bot-X

+1

78d979db6cef5 ("docs: add AI Coding Assistants documentation") introduced the
coding-assistants document some time ago [0]. In that time span, we've
seen a quick rise in popularity for tools that leverage reviews through the use
of LLMs.

And while the document does call out general AI contributions with:

> Contributions should include an Assisted-by tag in the following format:

expanding it to include something like "AI-analyzed-by" would be nice.

> So, yeah, Reviewed-by is definetly for definetly persons, but having
> some tag that states that it went thru automated reviewing^Wanalysis by
> a definetly bot/thing/whatever that some people think is useful seems
> useful.

Agreed. I think there is a distinction (although a pedantic one) between a
Generated-by/Assisted-by and something like "AI-analyzed" :-)


[0] https://lore.kernel.org/lkml/20251223122110.2496946-1-sashal@kernel.org/

-- 
Derek <debarbos@redhat.com>

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 23:29, Derek Barbosa wrote:
> On Sat, May 16, 2026 at 03:28:47PM -0300, Arnaldo Carvalho de Melo wrote:
>>
>> Couldn't this be something like:
>>
>> AI-analysed-by: bot-X
> 
> +1

But why? What is the benefit of storing in Git log information that some
tool did work?

We do not store checkpatch result (another pattern matching tool),
Coverity, Smatch, LKP or syzkallers.

Instead just blank +1 please provide arguments why this is useful for us.

I find it opposite: clogging commits with useless information, because
some arbitrary and completely closed-source tool did analysis means
nothing to me one year later when I look at the commit in the Git history.

Best regards,
Krzysztof

Re: Stop false review statements

[Roman Gushchin]

> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> 
> I find it opposite: clogging commits with useless information, because
> some arbitrary and completely closed-source tool did analysis means
> nothing to me one year later when I look at the commit in the Git history.

This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
and the code belongs to LF. Yes, the instance behind sashiko.dev is using
Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation - 
Sashiko is supporting various LLMs, including open models - it’s just a practical
choice: to my knowledge the quality of open models is not on par with frontier closed
models and it would require a non-trivial amount of hardware and infrastructure to run
an open model at the required scale.

Thanks

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 23:59, Roman Gushchin wrote:
> 
> 
>> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
>>
>> I find it opposite: clogging commits with useless information, because
>> some arbitrary and completely closed-source tool did analysis means
>> nothing to me one year later when I look at the commit in the Git history.
> 
> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> and the code belongs to LF. Yes, the instance behind sashiko.dev is using
> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation - 
> Sashiko is supporting various LLMs, including open models - it’s just a practical
> choice: to my knowledge the quality of open models is not on par with frontier closed
> models and it would require a non-trivial amount of hardware and infrastructure to run
> an open model at the required scale.

Sashiko is open, but it is not the Sashiko which performs the review but
closed source LLM behind.

Information that closed source LLM did some analysis is no more useful
than all other cases I mentioned - LKP, Smatch, Coverity or checkpatch -
of which most are even open source...

Best regards,
Krzysztof

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sat, 16 May 2026 14:59:44 -0700
Roman Gushchin <roman.gushchin@linux.dev> wrote:

> > On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> > 
> > I find it opposite: clogging commits with useless information, because
> > some arbitrary and completely closed-source tool did analysis means
> > nothing to me one year later when I look at the commit in the Git history.  
> 
> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> and the code belongs to LF. 

> Yes, the instance behind sashiko.dev is using
> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation - 
> Sashiko is supporting various LLMs, including open models - it’s just a practical
> choice: to my knowledge the quality of open models is not on par with frontier closed
> models 

I would very much prefer using an open source LLM, even if not in pair 
with latest paid models.

> and it would require a non-trivial amount of hardware and infrastructure to run
> an open model at the required scale.

IMHO the best would be to have them running on some infra that would accept
open source models (*). If there aren't enough resources to have our own
infra, there are offers out there which allows running open source models
like https://ollama.com/pricing (I never used myself).

(*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
    Not bad on my tests running it locally.

Thanks,
Mauro

Re: Stop false review statements

[Willy Tarreau]

On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
> On Sat, 16 May 2026 14:59:44 -0700
> Roman Gushchin <roman.gushchin@linux.dev> wrote:
> 
> > > On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> > > 
> > > I find it opposite: clogging commits with useless information, because
> > > some arbitrary and completely closed-source tool did analysis means
> > > nothing to me one year later when I look at the commit in the Git history.  
> > 
> > This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> > and the code belongs to LF. 
> 
> > Yes, the instance behind sashiko.dev is using
> > Gemini 3.1 Pro LLM, which is not open-source, but it's not a fundamental limitation - 
> > Sashiko is supporting various LLMs, including open models - it's just a practical
> > choice: to my knowledge the quality of open models is not on par with frontier closed
> > models 
> 
> I would very much prefer using an open source LLM, even if not in pair 
> with latest paid models.
> 
> > and it would require a non-trivial amount of hardware and infrastructure to run
> > an open model at the required scale.
> 
> IMHO the best would be to have them running on some infra that would accept
> open source models (*). If there aren't enough resources to have our own
> infra, there are offers out there which allows running open source models
> like https://ollama.com/pricing (I never used myself).
> 
> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
>     Not bad on my tests running it locally.

FWIW that's what I'm using locally coupled with llama.cpp to find bugs.
And it does. Plenty of valid ones. It's greatly sufficient for most work.

Willy

Re: Stop false review statements

[Greg KH]

On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
> On Sat, 16 May 2026 14:59:44 -0700
> Roman Gushchin <roman.gushchin@linux.dev> wrote:
> 
> > > On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> > > 
> > > I find it opposite: clogging commits with useless information, because
> > > some arbitrary and completely closed-source tool did analysis means
> > > nothing to me one year later when I look at the commit in the Git history.  
> > 
> > This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> > and the code belongs to LF. 
> 
> > Yes, the instance behind sashiko.dev is using
> > Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation - 
> > Sashiko is supporting various LLMs, including open models - it’s just a practical
> > choice: to my knowledge the quality of open models is not on par with frontier closed
> > models 
> 
> I would very much prefer using an open source LLM, even if not in pair 
> with latest paid models.
> 
> > and it would require a non-trivial amount of hardware and infrastructure to run
> > an open model at the required scale.
> 
> IMHO the best would be to have them running on some infra that would accept
> open source models (*). If there aren't enough resources to have our own
> infra, there are offers out there which allows running open source models
> like https://ollama.com/pricing (I never used myself).
> 
> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
>     Not bad on my tests running it locally.

You can run the tool locally, with whatever model you want, if you want
to.

But for now, let's just take the free credits that Google is willing to
throw at this thing and let it give us reviews IF the maintainer of the
subsystem feels it is something they want to do.  No one is forcing
maintainers to do this.

The netdev, bpf, and drm developers have been doing much the same for a
while now, with who-knows-what model behind the thing.  The model
doesn't matter, we aren't advertising for them, we just want the results
that they can provide us.

thanks,

greg k-h

Re: Stop false review statements

[Theodore Tso]

It should also be noted that Intel's zero-day bot was (a) closed
source, and (b) was sending its test regression reports with the
linux-kernel mailing list cc'ed, and no one really complained because
it was so useful, and if Intel was willing to use very expensive
hardware in their data center to contribute reports, so long as the
reports were useful and the false-positive noise was low enough, we
decided to be grateful and not worry (too much) about the fact that
Intel's zero-day bot was closed source.  (There was indeed some
grumbling in the bar at Plumbers, of course.  :-)

In my opinion, we should be doing the same for Sashiko, and that's the
decision which the ext4 developers have made --- at least for ext4
patches, after an experiment where we only sent reviews to the patch
authors and the maintainer, people were satisifed that false positive
rate was low enough (with the caveats that I had previously mentioned,
but we were willing to live with them because at least for us, it was
useful enough), that we will be requesting that Sashiko reviews be
cc'ed to the ext4 mailing list.

I realize that there are some extra sensitivities around AI / LLM's,
but from the perspective of reviewing patches, I don't see any
difference between this and other closed source tools that we've used,
such as Coverity and the Zero-day bot.  Not everyone will agree, of
course, but at the moment, this is a decision that we are making on a
subsystem by subsystem basis, which again, has strong historical
precedence.

Cheers,

						- Ted

Re: Stop false review statements

[Laurent Pinchart]

On Sun, May 17, 2026 at 12:29:12PM -0400, Theodore Tso wrote:
> It should also be noted that Intel's zero-day bot was (a) closed
> source, and (b) was sending its test regression reports with the
> linux-kernel mailing list cc'ed, and no one really complained because
> it was so useful, and if Intel was willing to use very expensive
> hardware in their data center to contribute reports, so long as the
> reports were useful and the false-positive noise was low enough, we
> decided to be grateful and not worry (too much) about the fact that
> Intel's zero-day bot was closed source.  (There was indeed some
> grumbling in the bar at Plumbers, of course.  :-)

The 0-day but was a closed-source front-end to orchestrate analysis
tools that are open-source (compilers, static analyzers, ...). Sashiko
is an open-source front-end to orchestrate analysis tools that are
closed-source. That's the complete opposite, so I'm not sure how
relevant the comparison is. Comparing with Coverity may be more
relevant.

> In my opinion, we should be doing the same for Sashiko, and that's the
> decision which the ext4 developers have made --- at least for ext4
> patches, after an experiment where we only sent reviews to the patch
> authors and the maintainer, people were satisifed that false positive
> rate was low enough (with the caveats that I had previously mentioned,
> but we were willing to live with them because at least for us, it was
> useful enough), that we will be requesting that Sashiko reviews be
> cc'ed to the ext4 mailing list.
> 
> I realize that there are some extra sensitivities around AI / LLM's,
> but from the perspective of reviewing patches, I don't see any
> difference between this and other closed source tools that we've used,
> such as Coverity and the Zero-day bot.  Not everyone will agree, of
> course, but at the moment, this is a decision that we are making on a
> subsystem by subsystem basis, which again, has strong historical
> precedence.

-- 
Regards,

Laurent Pinchart

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sun, 17 May 2026 12:12:00 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
> > On Sat, 16 May 2026 14:59:44 -0700
> > Roman Gushchin <roman.gushchin@linux.dev> wrote:
> >   
> > > > On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> > > > 
> > > > I find it opposite: clogging commits with useless information, because
> > > > some arbitrary and completely closed-source tool did analysis means
> > > > nothing to me one year later when I look at the commit in the Git history.    
> > > 
> > > This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> > > and the code belongs to LF.   
> >   
> > > Yes, the instance behind sashiko.dev is using
> > > Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation - 
> > > Sashiko is supporting various LLMs, including open models - it’s just a practical
> > > choice: to my knowledge the quality of open models is not on par with frontier closed
> > > models   
> > 
> > I would very much prefer using an open source LLM, even if not in pair 
> > with latest paid models.
> >   
> > > and it would require a non-trivial amount of hardware and infrastructure to run
> > > an open model at the required scale.  
> > 
> > IMHO the best would be to have them running on some infra that would accept
> > open source models (*). If there aren't enough resources to have our own
> > infra, there are offers out there which allows running open source models
> > like https://ollama.com/pricing (I never used myself).
> > 
> > (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
> >     Not bad on my tests running it locally.  
> 
> You can run the tool locally, with whatever model you want, if you want
> to.
> 
> But for now, let's just take the free credits that Google is willing to
> throw at this thing and let it give us reviews IF the maintainer of the
> subsystem feels it is something they want to do.  No one is forcing
> maintainers to do this.

If Google and/or others are willing to give free credits on their cloud,
they could instead or in addition give free credits to run ollama
there, allowing us to use different models.

From my side, while I won't personally object getting reviews from
Sashiko/Gemini, this is something I can't reproduce locally. I would
very much want something where I can select my LLM preferred model
and run on my ollama docker container on my own GPU, in a way that
I could run it locally before even sending a patch series.

> The netdev, bpf, and drm developers have been doing much the same for a
> while now, with who-knows-what model behind the thing.  The model
> doesn't matter, we aren't advertising for them, we just want the results
> that they can provide us.

It is not about the model itself. It is about being able to easily
install a sashiko locally on a container and easily make it use my
ollma server with the model(s) of my choice. Right now, at least at 
from its README.md, it sounds that only closed source services
are supported.

Thanks,
Mauro

Re: Stop false review statements

[Guenter Roeck]

On 5/17/26 09:39, Mauro Carvalho Chehab wrote:
...
> 
> It is not about the model itself. It is about being able to easily
> install a sashiko locally on a container and easily make it use my
> ollma server with the model(s) of my choice. Right now, at least at
> from its README.md, it sounds that only closed source services
> are supported.
> 

The README file says:

- **Self-contained**: Doesn't depend on 3rd-party tools and can work with
   various LLM providers (Gemini, Claude, and GitHub Copilot CLI are currently
   supported).

Sashiko is open source. No one prevents you from adding support for different
LLM providers. I would suggest to submit patches to have it support whatever
underlying LLM you want to use that isn't currently supported.

Guenter

Re: Stop false review statements

[Roman Gushchin]

> On May 17, 2026, at 9:40 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> On Sun, 17 May 2026 12:12:00 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
>>> On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
>>> On Sat, 16 May 2026 14:59:44 -0700
>>> Roman Gushchin <roman.gushchin@linux.dev> wrote:
>>> 
>>>>> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
>>>>> 
>>>>> I find it opposite: clogging commits with useless information, because
>>>>> some arbitrary and completely closed-source tool did analysis means
>>>>> nothing to me one year later when I look at the commit in the Git history.    
>>>> 
>>>> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
>>>> and the code belongs to LF.   
>>> 
>>>> Yes, the instance behind sashiko.dev is using
>>>> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation -
>>>> Sashiko is supporting various LLMs, including open models - it’s just a practical
>>>> choice: to my knowledge the quality of open models is not on par with frontier closed
>>>> models   
>>> 
>>> I would very much prefer using an open source LLM, even if not in pair
>>> with latest paid models.
>>> 
>>>> and it would require a non-trivial amount of hardware and infrastructure to run
>>>> an open model at the required scale.  
>>> 
>>> IMHO the best would be to have them running on some infra that would accept
>>> open source models (*). If there aren't enough resources to have our own
>>> infra, there are offers out there which allows running open source models
>>> like https://ollama.com/pricing (I never used myself).
>>> 
>>> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
>>>    Not bad on my tests running it locally.  
>> 
>> You can run the tool locally, with whatever model you want, if you want
>> to.
>> 
>> But for now, let's just take the free credits that Google is willing to
>> throw at this thing and let it give us reviews IF the maintainer of the
>> subsystem feels it is something they want to do.  No one is forcing
>> maintainers to do this.
> 
> If Google and/or others are willing to give free credits on their cloud,
> they could instead or in addition give free credits to run ollama
> there, allowing us to use different models.
> 
> From my side, while I won't personally object getting reviews from
> Sashiko/Gemini, this is something I can't reproduce locally. I would
> very much want something where I can select my LLM preferred model
> and run on my ollama docker container on my own GPU, in a way that
> I could run it locally before even sending a patch series.

2 thoughts here:
1) I actually tried to run it with ollama on my personal framework 13. Adding nominal support is trivial,
but the whole thing is not really useful: I can get maybe few hundreds tokens per second using
a quantified model with reduced quality; an average sashiko review is consuming 3.5 millions tokens
(with Gemini 3.1 pro, it’s also model-dependent).
I’m personally all in on having the entire thing as open as possible and I believe Sashiko is what 
is realistically the best at this moment - a fully open-source harness and set of prompts which 
can work with a variety of models.
I’m happy to merge a support for any LLM model which can produce decent review results.

2) Due to probabilistic nature of LLMs, nothing is reproducible in a strict sense of the word.
Even with exactly the same model/harness/prompts you’ll get different results every time you run it.
It’s unfortunate, but it is what it is at the moment.

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sun, 17 May 2026 11:17:06 -0700
Roman Gushchin <roman.gushchin@linux.dev> wrote:

> > On May 17, 2026, at 9:40 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > 
> > On Sun, 17 May 2026 12:12:00 +0200
> > Greg KH <gregkh@linuxfoundation.org> wrote:
> >   
> >>> On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
> >>> On Sat, 16 May 2026 14:59:44 -0700
> >>> Roman Gushchin <roman.gushchin@linux.dev> wrote:
> >>>   
> >>>>> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> >>>>> 
> >>>>> I find it opposite: clogging commits with useless information, because
> >>>>> some arbitrary and completely closed-source tool did analysis means
> >>>>> nothing to me one year later when I look at the commit in the Git history.      
> >>>> 
> >>>> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> >>>> and the code belongs to LF.     
> >>>   
> >>>> Yes, the instance behind sashiko.dev is using
> >>>> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation -
> >>>> Sashiko is supporting various LLMs, including open models - it’s just a practical
> >>>> choice: to my knowledge the quality of open models is not on par with frontier closed
> >>>> models     
> >>> 
> >>> I would very much prefer using an open source LLM, even if not in pair
> >>> with latest paid models.
> >>>   
> >>>> and it would require a non-trivial amount of hardware and infrastructure to run
> >>>> an open model at the required scale.    
> >>> 
> >>> IMHO the best would be to have them running on some infra that would accept
> >>> open source models (*). If there aren't enough resources to have our own
> >>> infra, there are offers out there which allows running open source models
> >>> like https://ollama.com/pricing (I never used myself).
> >>> 
> >>> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
> >>>    Not bad on my tests running it locally.    
> >> 
> >> You can run the tool locally, with whatever model you want, if you want
> >> to.
> >> 
> >> But for now, let's just take the free credits that Google is willing to
> >> throw at this thing and let it give us reviews IF the maintainer of the
> >> subsystem feels it is something they want to do.  No one is forcing
> >> maintainers to do this.  
> > 
> > If Google and/or others are willing to give free credits on their cloud,
> > they could instead or in addition give free credits to run ollama
> > there, allowing us to use different models.
> > 
> > From my side, while I won't personally object getting reviews from
> > Sashiko/Gemini, this is something I can't reproduce locally. I would
> > very much want something where I can select my LLM preferred model
> > and run on my ollama docker container on my own GPU, in a way that
> > I could run it locally before even sending a patch series.  
> 
> 2 thoughts here:
> 1) I actually tried to run it with ollama on my personal framework 13. Adding nominal support is trivial,
> but the whole thing is not really useful: I can get maybe few hundreds tokens per second using
> a quantified model with reduced quality; an average sashiko review is consuming 3.5 millions tokens 
> (with Gemini 3.1 pro, it’s also model-dependent).

Do you mean 3.5 millions tokens per patch series? If so, that
sounds a lot! Why does it require too many tokens?

> I’m personally all in on having the entire thing as open as possible and I believe Sashiko is what 
> is realistically the best at this moment - a fully open-source harness and set of prompts which 
> can work with a variety of models.
> I’m happy to merge a support for any LLM model which can produce decent review results.
> 
> 2) Due to probabilistic nature of LLMs, nothing is reproducible in a strict sense of the word.
> Even with exactly the same model/harness/prompts you’ll get different results every time you run it.
> It’s unfortunate, but it is what it is at the moment.

By "reproduce locally", I didn't mean in strict sense. Sure, LLM answers
won't be identical, but I suspect that at least most of the major issues 
on a patch series would be reported by any decent model.

So, if we have something that one can locally run using its GPU, being
able to get an answer in the range of a couple of minutes per patch
should be enough to catch most of the issues.

Thanks,
Mauro

Re: Stop false review statements

[Greg KH]

On Sun, May 17, 2026 at 08:56:06PM +0200, Mauro Carvalho Chehab wrote:
> By "reproduce locally", I didn't mean in strict sense. Sure, LLM answers
> won't be identical, but I suspect that at least most of the major issues 
> on a patch series would be reported by any decent model.
> 
> So, if we have something that one can locally run using its GPU, being
> able to get an answer in the range of a couple of minutes per patch
> should be enough to catch most of the issues.

That should be possible now, you can submit a patch locally to the
system.  I have a "cheat-sheet" around here somewhere that explains how
to do that, no idea why it's not part of the documentation.  I can dig
it up after breakfast...

greg k-h

Re: Stop false review statements

[Theodore Tso]

On Sun, May 17, 2026 at 11:17:06AM -0700, Roman Gushchin wrote:
> 
> I actually tried to run it with ollama on my
> personal framework 13. Adding nominal support is trivial, but the
> whole thing is not really useful: I can get maybe few hundreds
> tokens per second using a quantified model with reduced quality; an
> average sashiko review is consuming 3.5 millions tokens (with Gemini
> 3.1 pro, it’s also model-dependent).

I'm curious.  What hardware and LLM model were you using?  A few
hundred tokens per second seems surprising high.  My initial
research[1] showes that an M5 Max Macbook Pro costing 5 or 6 kilobucks
can do 31.6 tokens/second on a 27B 4-bit Quanitized model (Qwen 3.5).

[1] https://www.reddit.com/r/LocalLLaMA/comments/1rzkw4x/m5_max_128g_performance_tests_i_just_got_my_new/

The model matters of course.  With Gemma 3 27B and a 6-bit
quantization, it's 21 tokens/s, and with Deepseek R1 8B Q6_K, it's
72.8 tokens/second.  But unless you're using a really low-end model,
or a really expensive, splufty hardware platform, I haven't seen
reports of hundreds of tokens per second on hardware costing a
reasonable amount of memory.  (I'll set aside the question of whether
spending $6k for a fully spec'ed out M5 Max Macbook Pro, or $15k for a
fully spec'ed out M3 Ultra Mac Studio is "reasonable".)

As a result I'm not entirely sure how realistic it is to do reviews
using "free" (you still have to pay $$$ for the hardware) local,
open-weight LLM's if an average review requires around 3.5 million
tokens.

Cheers,

						- Ted

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sun, 17 May 2026 14:57:01 -0400
"Theodore Tso" <tytso@mit.edu> wrote:

> On Sun, May 17, 2026 at 11:17:06AM -0700, Roman Gushchin wrote:
> > 
> > I actually tried to run it with ollama on my
> > personal framework 13. Adding nominal support is trivial, but the
> > whole thing is not really useful: I can get maybe few hundreds
> > tokens per second using a quantified model with reduced quality; an
> > average sashiko review is consuming 3.5 millions tokens (with Gemini
> > 3.1 pro, it’s also model-dependent).  
> 
> I'm curious.  What hardware and LLM model were you using?  A few
> hundred tokens per second seems surprising high.  My initial
> research[1] showes that an M5 Max Macbook Pro costing 5 or 6 kilobucks
> can do 31.6 tokens/second on a 27B 4-bit Quanitized model (Qwen 3.5).
> 
> [1] https://www.reddit.com/r/LocalLLaMA/comments/1rzkw4x/m5_max_128g_performance_tests_i_just_got_my_new/
> 
> The model matters of course.  With Gemma 3 27B and a 6-bit
> quantization, it's 21 tokens/s, and with Deepseek R1 8B Q6_K, it's
> 72.8 tokens/second.  But unless you're using a really low-end model,
> or a really expensive, splufty hardware platform, I haven't seen
> reports of hundreds of tokens per second on hardware costing a
> reasonable amount of memory.  (I'll set aside the question of whether
> spending $6k for a fully spec'ed out M5 Max Macbook Pro, or $15k for a
> fully spec'ed out M3 Ultra Mac Studio is "reasonable".)

Ted,

Here, I'm using a RX9060XT, with is a relatively budget hardware.

It is also at the range of dozens of tokens per second. If you're
interested, I ran a benchmark this weekend with 3 models (just
for the sake of testing a set of turboquant patches - those aren't
the models I normally use).

You can see results here:
	https://github.com/ollama/ollama/pull/15505#issuecomment-4467278354

llama3.2:3b with f16 speed gives 72.5 decode tokens/s, and 37 decode tokens/s
with tq4 (actually a modified version of it) which, according with the
PR author, has quality almost identical to f16.

The main issue on such hardware is to have only 16 GB VRAM, making
it a little bit slow for models like qwen3.6:35b, as it will partially 
use CPU. Still, you can get a pretty decent answer in a couple of
minutes, with thinking enabled.

> As a result I'm not entirely sure how realistic it is to do reviews
> using "free" (you still have to pay $$$ for the hardware) local,
> open-weight LLM's if an average review requires around 3.5 million
> tokens.

Yes, 3.5 million tokens is indeed too much. I wonder why. Maybe
Gemini spreads the same query to multiple instances, making it 
spend a lot more tokens?

Here, I did some tests asking some LLM models to review code,
getting answers on a reasonable time (but didn't try to use sashiko
prompts).

Thanks,
Mauro

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 17:49, Roman Gushchin wrote:
>>
>>> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
>>> And we use Reported-by: tags with various tooling for years.
>>
>> Reported-by: shows the existance of a problem that some tool found, a
>> subtle difference here.
>>
>>> What do you think is the best form?
>>>
>>> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
>>
>> Just say it in some other text form, that our tools will not pick up.
>> Like:
>>    Tool XXXX reports that all is good:
>>        https://....
>>
>> or something like that?
> 
> Sure, works for me.
Roman,
Before implementing such changes, send a RFC or just ask a few folks for
opinions. We do use the tool, among other tools, so we will gladly
provide a feedback.

Sashiko should in general not send such emails when not asked for. Why?
Because we have also other bots, like LKP, KernelCI, and imagine how
maintainer's mailbox will look like.

LKP allows opt-in for your own repo, which for example I am using, so I
get confirmation of the success. But people are not receiving them. I
cannot imagine all the people getting these LKP-successfully-built
emails on every email.

Best regards,
Krzysztof

Re: Stop false review statements

[Roman Gushchin]

> On May 16, 2026, at 11:29 AM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> 
> On 16/05/2026 17:49, Roman Gushchin wrote:
>>> 
>>>> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
>>>> And we use Reported-by: tags with various tooling for years.
>>> 
>>> Reported-by: shows the existance of a problem that some tool found, a
>>> subtle difference here.
>>> 
>>>> What do you think is the best form?
>>>> 
>>>> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
>>> 
>>> Just say it in some other text form, that our tools will not pick up.
>>> Like:
>>>   Tool XXXX reports that all is good:
>>>       https://....
>>> 
>>> or something like that?
>> 
>> Sure, works for me.
> Roman,
> Before implementing such changes, send a RFC or just ask a few folks for
> opinions. We do use the tool, among other tools, so we will gladly
> provide a feedback.
> 
> Sashiko should in general not send such emails when not asked for. Why?
> Because we have also other bots, like LKP, KernelCI, and imagine how
> maintainer's mailbox will look like.
> 
> LKP allows opt-in for your own repo, which for example I am using, so I
> get confirmation of the success. But people are not receiving them. I
> cannot imagine all the people getting these LKP-successfully-built
> emails on every email.

It’s opt-in on per-subsystem basis, as well as all other email-related features.
I do rely on corresponding maintainers to decide if they want it or not.
Even in the case which you was so unhappy about, I asked Guenter prior 
to enabling it for hwmon.

If you’re saying that it should not send any non-personal emails in general, I disagree here,
but happy to have a discussion, assuming it’s polite and constructive.

The reason why I disagree is simple: there are maintainers/subsystems who like Sashiko’s reviews 
and  before introducing the email interface they had to manually send links to Sashiko’s reviews
as replies to proposed patches. I’ve been explicitly asked to add an ability to send out
emails with reviews.

Thanks

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 20:56, Roman Gushchin wrote:
> 
> 
>> On May 16, 2026, at 11:29 AM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
>>
>> On 16/05/2026 17:49, Roman Gushchin wrote:
>>>>
>>>>> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
>>>>> And we use Reported-by: tags with various tooling for years.
>>>>
>>>> Reported-by: shows the existance of a problem that some tool found, a
>>>> subtle difference here.
>>>>
>>>>> What do you think is the best form?
>>>>>
>>>>> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
>>>>
>>>> Just say it in some other text form, that our tools will not pick up.
>>>> Like:
>>>>   Tool XXXX reports that all is good:
>>>>       https://....
>>>>
>>>> or something like that?
>>>
>>> Sure, works for me.
>> Roman,
>> Before implementing such changes, send a RFC or just ask a few folks for
>> opinions. We do use the tool, among other tools, so we will gladly
>> provide a feedback.
>>
>> Sashiko should in general not send such emails when not asked for. Why?
>> Because we have also other bots, like LKP, KernelCI, and imagine how
>> maintainer's mailbox will look like.
>>
>> LKP allows opt-in for your own repo, which for example I am using, so I
>> get confirmation of the success. But people are not receiving them. I
>> cannot imagine all the people getting these LKP-successfully-built
>> emails on every email.
> 
> It’s opt-in on per-subsystem basis, as well as all other email-related features.
> I do rely on corresponding maintainers to decide if they want it or not.

The trouble is that subsystem is mailing list, thus I still got all of
them via b4, which is used to get the discussion.

Send them only to the maintainer, for example. Or maintainer + authors.

Basically the same as LKP is doing.

> If you’re saying that it should not send any non-personal emails in general, I disagree here,
> but happy to have a discussion, assuming it’s polite and constructive.

I meant it should not be send to people who did not request that. Opt-in
should be explicit and no mailing lists must be Cced (because then it is
sending to everyone).

> 
> The reason why I disagree is simple: there are maintainers/subsystems who like Sashiko’s reviews 
> and  before introducing the email interface they had to manually send links to Sashiko’s reviews
> as replies to proposed patches. I’ve been explicitly asked to add an ability to send out
> emails with reviews.

Sure, I agree with the need for use-case.

Best regards,
Krzysztof

Re: Stop false review statements

[Guenter Roeck]

On 5/16/26 12:00, Krzysztof Kozlowski wrote:
...

>> It’s opt-in on per-subsystem basis, as well as all other email-related features.
>> I do rely on corresponding maintainers to decide if they want it or not.
> 
> The trouble is that subsystem is mailing list, thus I still got all of
> them via b4, which is used to get the discussion.
> 
> Send them only to the maintainer, for example. Or maintainer + authors.
> 

For hwmon and watchdog I most definitely want the response sent to the
mailing list and to the patch author. That was the original configuration
for hwmon. Roman took it out because people who were copied on the
original patch complained that they did _not_ get Sashiko's reply.
That is a perfect lose-lose situation.

Guenter

Re: Stop false review statements

[Guenter Roeck]

On 5/16/26 12:13, Guenter Roeck wrote:
> On 5/16/26 12:00, Krzysztof Kozlowski wrote:
> ...
> 
>>> It’s opt-in on per-subsystem basis, as well as all other email-related features.
>>> I do rely on corresponding maintainers to decide if they want it or not.
>>
>> The trouble is that subsystem is mailing list, thus I still got all of
>> them via b4, which is used to get the discussion.
>>
>> Send them only to the maintainer, for example. Or maintainer + authors.
>>
> 
> For hwmon and watchdog I most definitely want the response sent to the
> mailing list and to the patch author. That was the original configuration
> for hwmon. Roman took it out because people who were copied on the
> original patch complained that they did _not_ get Sashiko's reply.

Actually, turns out he didn't, he just moved it.

However, it turns out that Rob added krzk+dt@kernel.org as explicit
Cc: target for the devicetree subsystem. I would suggest to drop that.

Thanks,
Guenter

Re: Stop false review statements

[Roman Gushchin]

> On May 16, 2026, at 12:25 PM, Guenter Roeck <linux@roeck-us.net> wrote:
> 
> On 5/16/26 12:13, Guenter Roeck wrote:
>> On 5/16/26 12:00, Krzysztof Kozlowski wrote:
>> ...
>>>> It’s opt-in on per-subsystem basis, as well as all other email-related features.
>>>> I do rely on corresponding maintainers to decide if they want it or not.
>>> 
>>> The trouble is that subsystem is mailing list, thus I still got all of
>>> them via b4, which is used to get the discussion.
>>> 
>>> Send them only to the maintainer, for example. Or maintainer + authors.
>>> 
>> For hwmon and watchdog I most definitely want the response sent to the
>> mailing list and to the patch author. That was the original configuration
>> for hwmon. Roman took it out because people who were copied on the
>> original patch complained that they did _not_ get Sashiko's reply.
> 
> Actually, turns out he didn't, he just moved it.
> 
> However, it turns out that Rob added krzk+dt@kernel.org as explicit
> Cc: target for the devicetree subsystem. I would suggest to drop that.

I already did.
I haven’t changed the hwmon configuration.

Thanks

Re: Stop false review statements

[Roman Gushchin]

> On May 16, 2026, at 12:00 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> 
> On 16/05/2026 20:56, Roman Gushchin wrote:
>> 
>> 
>>>> On May 16, 2026, at 11:29 AM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
>>> 
>>> On 16/05/2026 17:49, Roman Gushchin wrote:
>>>>> 
>>>>>> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
>>>>>> And we use Reported-by: tags with various tooling for years.
>>>>> 
>>>>> Reported-by: shows the existance of a problem that some tool found, a
>>>>> subtle difference here.
>>>>> 
>>>>>> What do you think is the best form?
>>>>>> 
>>>>>> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
>>>>> 
>>>>> Just say it in some other text form, that our tools will not pick up.
>>>>> Like:
>>>>>  Tool XXXX reports that all is good:
>>>>>      https://....
>>>>> 
>>>>> or something like that?
>>>> 
>>>> Sure, works for me.
>>> Roman,
>>> Before implementing such changes, send a RFC or just ask a few folks for
>>> opinions. We do use the tool, among other tools, so we will gladly
>>> provide a feedback.
>>> 
>>> Sashiko should in general not send such emails when not asked for. Why?
>>> Because we have also other bots, like LKP, KernelCI, and imagine how
>>> maintainer's mailbox will look like.
>>> 
>>> LKP allows opt-in for your own repo, which for example I am using, so I
>>> get confirmation of the success. But people are not receiving them. I
>>> cannot imagine all the people getting these LKP-successfully-built
>>> emails on every email.
>> 
>> It’s opt-in on per-subsystem basis, as well as all other email-related features.
>> I do rely on corresponding maintainers to decide if they want it or not.
> 
> The trouble is that subsystem is mailing list, thus I still got all of
> them via b4, which is used to get the discussion.
> 
> Send them only to the maintainer, for example. Or maintainer + authors.
> 
> Basically the same as LKP is doing.

There are subsystems which want email reviews to be sent to the subsystem
mailing list. In fact, all currently configured email policies came from maintainers,
I don’t push anything based on my own preferences. 

Sashiko can be configured the way you describe it or in any other way, it’s up to corresponding 
maintainers.

I agree, it’s sometimes gets tricky when a patchset is sent to multiple mailing lists,
which policy to apply. I have some improvements in my plans, but it’s not always possible
to say how it should be handled. It’s not fundamentally new: landing changes touching 
multiple subsystems is always harder exactly because maintainers might have different
and sometimes conflicting views.

> 
>> If you’re saying that it should not send any non-personal emails in general, I disagree here,
>> but happy to have a discussion, assuming it’s polite and constructive.
> 
> I meant it should not be send to people who did not request that. Opt-in
> should be explicit and no mailing lists must be Cced (because then it is
> sending to everyone).

>> 
>> The reason why I disagree is simple: there are maintainers/subsystems who like Sashiko’s reviews
>> and  before introducing the email interface they had to manually send links to Sashiko’s reviews
>> as replies to proposed patches. I’ve been explicitly asked to add an ability to send out
>> emails with reviews.
> 
> Sure, I agree with the need for use-case.
> 
> Best regards,
> Krzysztof

Re: Stop false review statements

[Theodore Tso]

On Sat, May 16, 2026 at 12:15:12PM -0700, Roman Gushchin wrote:
> > The trouble is that subsystem is mailing list, thus I still got all of
> > them via b4, which is used to get the discussion.
> > 
> > Send them only to the maintainer, for example. Or maintainer + authors.
> > 
> > Basically the same as LKP is doing.
> 
> There are subsystems which want email reviews to be sent to the subsystem
> mailing list. In fact, all currently configured email policies came from maintainers,
> I don’t push anything based on my own preferences.

In the case of ext4, we have a weekly video conference of the core
developers, and last week I asked the ext4 core developers whether we
should start cc'ing the linux-ext4 list.  When I first asked Roman to
send the reviews to the me as the reviewer and the patch author, I
didn't want to cc the list in the case people would find annoying.

The discussion in our video chat was that the quality of the reviews
was quite good, and the only feedback from the ext4 developers was (a)
pre-existing problems that were unrelated the patch series, (b)
sometimes the problems that was pointed out are ones that we don't
care about (for example, there was a recent comment about readahead
detection being racy, and that was not ext4-specific, and readahead is
a hint and if two processes are reading the file at the same
time.... oh cares how the system handles the hueristic of something
which is a hint anyway), and (c) while Shashiko is good at pointing
out problems, its suggestted solutions aren't as good.

But that's OK, on the whole, the Sashiko is finding problems that
humans very familiar with code base had missed.  And so it's certainly
better than most human reviewers.

Based on that, the consensus of the ext4 core developers that it would
be better to make sure that the linux-ext4 list should be cc'ed.  So
that's a decision that didn't come from me as the ext4 maintainer, but
after consulting with core ext4 developers and reviewers.

> I agree, it’s sometimes gets tricky when a patchset is sent to
> multiple mailing lists, which policy to apply.

What I would suggest is that if we have a patch which is cc'ed to say,
linux-xfs, linux-ext4, and linux-fsdevel, as well as a dozen
developers suggested by get_maintainer.pl, and only the ext4 list has
requested the reviews, then only send it to the ext4 maintainer, the
ext4 mailing list, and the patch author.  The Sashiko review doesn't
need to be cc'ed to the other lists, or the dozen or so other
maintainers.

Cheers,

					- Ted

Re: Stop false review statements

[Danilo Krummrich]

On Sat May 16, 2026 at 9:15 PM CEST, Roman Gushchin wrote:
> I agree, it’s sometimes gets tricky when a patchset is sent to multiple
> mailing lists, which policy to apply. I have some improvements in my plans,
> but it’s not always possible to say how it should be handled.

Which improvements do you have in mind?

> It’s not fundamentally new: landing changes touching multiple subsystems is
> always harder exactly because maintainers might have different and sometimes
> conflicting views.

It can also be relevant in cases where only a single subsystem is touched.

For instance, in the case of Rust, the rust-for-linux list serves two purposes
-- when it is a Rust subsystem change and when Rust code of any other subsystem
is touched, i.e. the rust-for-linux list has more of a LKML character and also
receives patches for subsystems whose maintainers may not have opted in to
sashiko email delivery.

That said, I personally don't mind too much, I really like sashiko, which is
also why I asked for adding the driver-core list. My experience has been that it
does a very decent job in providing feedback for C code; my feeling is that
feedback for Rust code is not quite on par yet, but of course it also highly
depends on the complexity and scope of the corresponding changes.

However, I still have the same concern I raised previously when it comes to
email delivery: I think that when sashiko sends feedback to contributors
(without Cc'ing the mailing list and all other recipients), it should actively
ask the contributor to raise things on the list with all other recipients,
reviewers and maintainers before acting on them, such that changes subsequent to
the first submission on the list are aligned.

Can this be added please?

Thanks,
Danilo

Re: Stop false review statements

[Danilo Krummrich]

On Sun May 17, 2026 at 5:56 PM CEST, Danilo Krummrich wrote:
> However, I still have the same concern I raised previously when it comes to
> email delivery: I think that when sashiko sends feedback to contributors
> (without Cc'ing the mailing list and all other recipients), it should actively
> ask the contributor to raise things on the list with all other recipients,
> reviewers and maintainers before acting on them, such that changes subsequent to
> the first submission on the list are aligned.
>
> Can this be added please?

I'm also happy to send a PR of course.

Thanks,
Danilo

Re: Stop false review statements

[SeongJae Park]

On Sat, 16 May 2026 08:49:39 -0700 Roman Gushchin <roman.gushchin@linux.dev> wrote:

> 
> > On May 16, 2026, at 8:45 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> > 
> > On Sat, May 16, 2026 at 08:41:43AM -0700, Roman Gushchin wrote:
> >> 
> >>>> On May 16, 2026, at 8:20 AM, Konstantin Ryabitsev <mricon@kernel.org> wrote:
> >>> 
> >>> On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
> >>>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
[...]
> >> The goal here is to inform maintainers that sashiko has successfully reviewed the patch
> >> and there were no findings, otherwise maintainers have to go to the web site and check the status.

Yes, this will be helpful.  I also think notifying review failures (usually due
to patch applying failure) or general review results summary for every case
(maybe opt-in?) would also be helpful.

> > 
> > That's fine.
> > 
> >> I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form.
> >> And we use Reported-by: tags with various tooling for years.
> > 
> > Reported-by: shows the existance of a problem that some tool found, a
> > subtle difference here.
> > 
> >> What do you think is the best form?
> >> 
> >> I’ll pause sending reviewed-by tags until we have a discussion and agreement here.
> > 
> > Just say it in some other text form, that our tools will not pick up.
> > Like:
> >    Tool XXXX reports that all is good:
> >        https://....
> > 
> > or something like that?
> 
> Sure, works for me.

+1.  I was also feeling Reviewed-by: is at least controversial.


Thanks,
SJ

[...]

Re: Stop false review statements

[Mauro Carvalho Chehab]

On Sat, 16 May 2026 17:45:51 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> > I’m not attached to any specific form of it, I thought Reviewed-by is the most obvious form. 
> > And we use Reported-by: tags with various tooling for years.  
> 
> Reported-by: shows the existance of a problem that some tool found, a
> subtle difference here.

I'd say that, if an issue was found after a patch is merged,
I don't see why to distinguish. I mean:

if tool or a bot XYZ found a real issue, and a patch fixes it, 
reported-by applies - being a LLM tool/bot or not.

Now, if someone sends a patch series v1, get a bot report and send a
v2  of the same patch series due to some CI/bot/LLM/... feedback, IMO 
the right approach is to mention it on patch 0, just like we do with
any other feedback. Eventually, if such feedback is more relevant, it
can be also be mentioned inside patch description(s).

That's said, I would be fine with either a free text mention or with
some tag.

If one wants/needs to justify if/why some tool is relevant for kernel
development, a simple grep would be enough:

	$ git log|grep -i coverity|wc -l
	4267
	$ git log|grep -i smatch|wc -l
	13140
	$ git log|grep -i sashiko |wc -l
	138

IMO, there's no need for an special tag.

Thanks,
Mauro