Stop false review statements

Stop false review statements

[Krzysztof Kozlowski]

What the hell is that:

https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/

As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
not a damn human do be able to make such statement. You are a bot, a tool.

Stop faking tags.

And really, considering how many false positives Sashiko produces, how
poor review comments it gives, how many misleading comments, it's
unacceptable to me to consider that a review.

Amount of useless noise Sashiko produces already changed my mind how
useful that tool is.

I will be NAKing every damn tag produced by such tools.


Best regards,
Krzysztof

Re: Stop false review statements

[Guenter Roeck]

On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> What the hell is that:
> 
> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> 
> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> not a damn human do be able to make such statement. You are a bot, a tool.
> 

Where exactly do the rules say that ? I seem to miss that.

There is a policy document about _contributions_ made by AI, but I don't
see the one that says that AI agents must not provide Reviewed-by: tags.

> Stop faking tags.
> 
> And really, considering how many false positives Sashiko produces, how
> poor review comments it gives, how many misleading comments, it's
> unacceptable to me to consider that a review.
> 
> Amount of useless noise Sashiko produces already changed my mind how
> useful that tool is.

We seem to have completely different experiences. Yes, it does produce
false positives, just like humans do. However, I have seen it find many
real bugs, including many in patches which already had Reviewed-by: tags
from (presumably) human reviewers.

Again, it appears that our experience is completely different than mine,
but after several weeks of getting code reviews from sashiko I do have to
say that I trust its review feedback significantly more than human reviews.
Sure, it does not guarantee that a patch is indeed bug free. A human review
doesn't guarantee it either.

> 
> I will be NAKing every damn tag produced by such tools.

I'd like to see an official policy. Until then I'll ignore your NAK in my
scope of responsibility.

Thanks,
Guenter

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 14:11, Guenter Roeck wrote:
> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>> What the hell is that:
>>
>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>
>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>> not a damn human do be able to make such statement. You are a bot, a tool.
>>
> 
> Where exactly do the rules say that ? I seem to miss that.
> 
> There is a policy document about _contributions_ made by AI, but I don't
> see the one that says that AI agents must not provide Reviewed-by: tags.

Quotes from the existing policy:

1. "By offering my Reviewed-by: tag, I state that:"

Tool cannot use first person "I". Tool cannot "state that".

2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
 appropriate modification of the kernel without any remaining serious"

Tool cannot make a statement of opinion.

3. "Any interested reviewer (who has done the work) can offer a
Reviewed-by".

Tool is not a reviewer as a person, thus above does not grant the tool
permission to offer a tag.

> 
>> Stop faking tags.
>>
>> And really, considering how many false positives Sashiko produces, how
>> poor review comments it gives, how many misleading comments, it's
>> unacceptable to me to consider that a review.
>>
>> Amount of useless noise Sashiko produces already changed my mind how
>> useful that tool is.
> 
> We seem to have completely different experiences. Yes, it does produce
> false positives, just like humans do. However, I have seen it find many
> real bugs, including many in patches which already had Reviewed-by: tags
> from (presumably) human reviewers.

Of course it finds bugs. But it also produces - roughly - 80-90% false
positives, completely useless.

This is very poor review score.

> 
> Again, it appears that our experience is completely different than mine,
> but after several weeks of getting code reviews from sashiko I do have to
> say that I trust its review feedback significantly more than human reviews.
> Sure, it does not guarantee that a patch is indeed bug free. A human review
> doesn't guarantee it either.
> 
>>
>> I will be NAKing every damn tag produced by such tools.
> 
> I'd like to see an official policy. Until then I'll ignore your NAK in my
> scope of responsibility.

:)

Best regards,
Krzysztof

Re: Stop false review statements

[Guenter Roeck]

On 5/16/26 05:16, Krzysztof Kozlowski wrote:
> On 16/05/2026 14:11, Guenter Roeck wrote:
>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>> What the hell is that:
>>>
>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>
>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>
>>
>> Where exactly do the rules say that ? I seem to miss that.
>>
>> There is a policy document about _contributions_ made by AI, but I don't
>> see the one that says that AI agents must not provide Reviewed-by: tags.
> 
> Quotes from the existing policy:
> 
> 1. "By offering my Reviewed-by: tag, I state that:"
> 
> Tool cannot use first person "I". Tool cannot "state that".
> 
> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>   appropriate modification of the kernel without any remaining serious"
> 
> Tool cannot make a statement of opinion.
> 
> 3. "Any interested reviewer (who has done the work) can offer a
> Reviewed-by".
> 
> Tool is not a reviewer as a person, thus above does not grant the tool
> permission to offer a tag.
> 

I'd like to see that explicitly spelled out. Until then it is your opinion.

>>
>>> Stop faking tags.
>>>
>>> And really, considering how many false positives Sashiko produces, how
>>> poor review comments it gives, how many misleading comments, it's
>>> unacceptable to me to consider that a review.
>>>
>>> Amount of useless noise Sashiko produces already changed my mind how
>>> useful that tool is.
>>
>> We seem to have completely different experiences. Yes, it does produce
>> false positives, just like humans do. However, I have seen it find many
>> real bugs, including many in patches which already had Reviewed-by: tags
>> from (presumably) human reviewers.
> 
> Of course it finds bugs. But it also produces - roughly - 80-90% false
> positives, completely useless.
> 

Really ? The ones I have seen are - roughly, to use the same term - 80-90%
true positives. Maybe you should explicitly ask for no Sashiko reviews in
your scope of responsibility.

Thanks,
Guenter

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 14:23, Guenter Roeck wrote:
> On 5/16/26 05:16, Krzysztof Kozlowski wrote:
>> On 16/05/2026 14:11, Guenter Roeck wrote:
>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>>> What the hell is that:
>>>>
>>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>>
>>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>>
>>>
>>> Where exactly do the rules say that ? I seem to miss that.
>>>
>>> There is a policy document about _contributions_ made by AI, but I don't
>>> see the one that says that AI agents must not provide Reviewed-by: tags.
>>
>> Quotes from the existing policy:
>>
>> 1. "By offering my Reviewed-by: tag, I state that:"
>>
>> Tool cannot use first person "I". Tool cannot "state that".
>>
>> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>>   appropriate modification of the kernel without any remaining serious"
>>
>> Tool cannot make a statement of opinion.
>>
>> 3. "Any interested reviewer (who has done the work) can offer a
>> Reviewed-by".
>>
>> Tool is not a reviewer as a person, thus above does not grant the tool
>> permission to offer a tag.
>>
> 
> I'd like to see that explicitly spelled out. Until then it is your opinion.

It is not an opinion. It is written. I gave you quotes.

Do you want to spell the rules of English language? That tool is not a
person?

Shall I send the patch like:

  Any interested reviewer (who has done the work) can offer a
  Reviewed-by.
 +In English "reviewer" is a person [1].
 + [1] https://en.wiktionary.org/wiki/reviewer

Seriously, you expect to document the English language?

> 
>>>
>>>> Stop faking tags.
>>>>
>>>> And really, considering how many false positives Sashiko produces, how
>>>> poor review comments it gives, how many misleading comments, it's
>>>> unacceptable to me to consider that a review.
>>>>
>>>> Amount of useless noise Sashiko produces already changed my mind how
>>>> useful that tool is.
>>>
>>> We seem to have completely different experiences. Yes, it does produce
>>> false positives, just like humans do. However, I have seen it find many
>>> real bugs, including many in patches which already had Reviewed-by: tags
>>> from (presumably) human reviewers.
>>
>> Of course it finds bugs. But it also produces - roughly - 80-90% false
>> positives, completely useless.
>>
> 
> Really ? The ones I have seen are - roughly, to use the same term - 80-90%
> true positives. Maybe you should explicitly ask for no Sashiko reviews in
> your scope of responsibility.

I already sent a patch to stop receiving all these emails and I stopped
reading them completely, when fetched via b4 for review in mutt workflow.

But this is not the point.

Our docs clearly state what Reviewed-by means, regardless of the quality
of the actual review. Poor quality is just another reason, less
important, though.

Best regards,
Krzysztof

Re: Stop false review statements

[Laurent Pinchart]

On Sat, May 16, 2026 at 02:29:15PM +0200, Krzysztof Kozlowski wrote:
> On 16/05/2026 14:23, Guenter Roeck wrote:
> > On 5/16/26 05:16, Krzysztof Kozlowski wrote:
> >> On 16/05/2026 14:11, Guenter Roeck wrote:
> >>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> >>>> What the hell is that:
> >>>>
> >>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> >>>>
> >>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> >>>> not a damn human do be able to make such statement. You are a bot, a tool.
> >>>>
> >>>
> >>> Where exactly do the rules say that ? I seem to miss that.
> >>>
> >>> There is a policy document about _contributions_ made by AI, but I don't
> >>> see the one that says that AI agents must not provide Reviewed-by: tags.
> >>
> >> Quotes from the existing policy:
> >>
> >> 1. "By offering my Reviewed-by: tag, I state that:"
> >>
> >> Tool cannot use first person "I". Tool cannot "state that".
> >>
> >> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
> >>   appropriate modification of the kernel without any remaining serious"
> >>
> >> Tool cannot make a statement of opinion.
> >>
> >> 3. "Any interested reviewer (who has done the work) can offer a
> >> Reviewed-by".
> >>
> >> Tool is not a reviewer as a person, thus above does not grant the tool
> >> permission to offer a tag.
> > 
> > I'd like to see that explicitly spelled out. Until then it is your opinion.
> 
> It is not an opinion. It is written. I gave you quotes.
> 
> Do you want to spell the rules of English language? That tool is not a
> person?
> 
> Shall I send the patch like:
> 
>   Any interested reviewer (who has done the work) can offer a
>   Reviewed-by.
>  +In English "reviewer" is a person [1].
>  + [1] https://en.wiktionary.org/wiki/reviewer
> 
> Seriously, you expect to document the English language?
> 
> >>>> Stop faking tags.
> >>>>
> >>>> And really, considering how many false positives Sashiko produces, how
> >>>> poor review comments it gives, how many misleading comments, it's
> >>>> unacceptable to me to consider that a review.
> >>>>
> >>>> Amount of useless noise Sashiko produces already changed my mind how
> >>>> useful that tool is.

Note this isn't en entirely new situation. As a maintainer, you know how
much you trust each reviewer. You will consider some R-b tags as a sign
you don't even have to look at a patch, and will completely ignore some
others. There's a whole continuum in the middle. In some ways, reviews
by an LLM are similar. You will trust them or not trust them.

Except they're also very different.

The kernel needs more skilled reviewers (I don't think this is a
controversial statement). We can't expect all newcomers to start with
extensive experience from day one, so there's a learning curve. I
believe it's fine for more junior reviewers to send R-b tags even if
they miss some issue, as long as they genuinely try and improve (and, in
some unfortunate cases, decide to leave if patch review turns out not to
be for them). Those R-b tags may feel like a bit of noise in the
beginning, but that's compensated by their value increasing over time.

Bot reviews are not the same. Not only are they generated at a much
larger scale than human reviews, they also won't learn from feedback you
give them. Sure, the tools may be improved when cases of false positives
are identified, and new LLMs may be trained with more (and better ?)
data to improve the output, but they won't learn from the interactions.

How much value a maintainer sees in those reviews is up to individual
maintainers. I will personally not consider a R-b tag from an LLM to
mean that a patch is ready to be merged (and I believe you won't
either). As such, I think that a R-b from an LLM is misleading and
doesn't provide good value. At best it's free advertising for company
making closed-source tools, which I don't think we should encourage.

If some maintainers want LLM reviews and want to act on them, that's
their personal prerogative. They're free to decide on how much value
they see in those reviews, as well as on whether or not they consider
usage of those tools compatible with FOSS ethics. Those are personal
decisions. However, given that the ethical decision is personal, I am
strongly against forcing patch authors to act on automated LLM review.

> >>> We seem to have completely different experiences. Yes, it does produce
> >>> false positives, just like humans do. However, I have seen it find many
> >>> real bugs, including many in patches which already had Reviewed-by: tags
> >>> from (presumably) human reviewers.
> >>
> >> Of course it finds bugs. But it also produces - roughly - 80-90% false
> >> positives, completely useless.
> >>
> > 
> > Really ? The ones I have seen are - roughly, to use the same term - 80-90%
> > true positives. Maybe you should explicitly ask for no Sashiko reviews in
> > your scope of responsibility.
> 
> I already sent a patch to stop receiving all these emails and I stopped
> reading them completely, when fetched via b4 for review in mutt workflow.
> 
> But this is not the point.
> 
> Our docs clearly state what Reviewed-by means, regardless of the quality
> of the actual review. Poor quality is just another reason, less
> important, though.

-- 
Regards,

Laurent Pinchart

Re: Stop false review statements

[Krzysztof Kozlowski]

On 16/05/2026 15:24, Laurent Pinchart wrote:
> On Sat, May 16, 2026 at 02:29:15PM +0200, Krzysztof Kozlowski wrote:
>> On 16/05/2026 14:23, Guenter Roeck wrote:
>>> On 5/16/26 05:16, Krzysztof Kozlowski wrote:
>>>> On 16/05/2026 14:11, Guenter Roeck wrote:
>>>>> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
>>>>>> What the hell is that:
>>>>>>
>>>>>> https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
>>>>>>
>>>>>> As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
>>>>>> not a damn human do be able to make such statement. You are a bot, a tool.
>>>>>>
>>>>>
>>>>> Where exactly do the rules say that ? I seem to miss that.
>>>>>
>>>>> There is a policy document about _contributions_ made by AI, but I don't
>>>>> see the one that says that AI agents must not provide Reviewed-by: tags.
>>>>
>>>> Quotes from the existing policy:
>>>>
>>>> 1. "By offering my Reviewed-by: tag, I state that:"
>>>>
>>>> Tool cannot use first person "I". Tool cannot "state that".
>>>>
>>>> 2. "A Reviewed-by tag is *a statement of opinion* that the patch is an
>>>>   appropriate modification of the kernel without any remaining serious"
>>>>
>>>> Tool cannot make a statement of opinion.
>>>>
>>>> 3. "Any interested reviewer (who has done the work) can offer a
>>>> Reviewed-by".
>>>>
>>>> Tool is not a reviewer as a person, thus above does not grant the tool
>>>> permission to offer a tag.
>>>
>>> I'd like to see that explicitly spelled out. Until then it is your opinion.
>>
>> It is not an opinion. It is written. I gave you quotes.
>>
>> Do you want to spell the rules of English language? That tool is not a
>> person?
>>
>> Shall I send the patch like:
>>
>>   Any interested reviewer (who has done the work) can offer a
>>   Reviewed-by.
>>  +In English "reviewer" is a person [1].
>>  + [1] https://en.wiktionary.org/wiki/reviewer
>>
>> Seriously, you expect to document the English language?
>>
>>>>>> Stop faking tags.
>>>>>>
>>>>>> And really, considering how many false positives Sashiko produces, how
>>>>>> poor review comments it gives, how many misleading comments, it's
>>>>>> unacceptable to me to consider that a review.
>>>>>>
>>>>>> Amount of useless noise Sashiko produces already changed my mind how
>>>>>> useful that tool is.
> 
> Note this isn't en entirely new situation. As a maintainer, you know how
> much you trust each reviewer. You will consider some R-b tags as a sign
> you don't even have to look at a patch, and will completely ignore some
> others. There's a whole continuum in the middle. In some ways, reviews
> by an LLM are similar. You will trust them or not trust them.
> 
> Except they're also very different.
> 
> The kernel needs more skilled reviewers (I don't think this is a
> controversial statement). We can't expect all newcomers to start with
> extensive experience from day one, so there's a learning curve. I
> believe it's fine for more junior reviewers to send R-b tags even if
> they miss some issue, as long as they genuinely try and improve (and, in
> some unfortunate cases, decide to leave if patch review turns out not to
> be for them). Those R-b tags may feel like a bit of noise in the
> beginning, but that's compensated by their value increasing over time.

Yes, I agree. Reviews from inexperienced people are sometimes fruitless
or pointless per actual value they bring, but they allow a person
(again: person) to grow in the community with a credits being the reward.

> 
> Bot reviews are not the same. Not only are they generated at a much
> larger scale than human reviews, they also won't learn from feedback you
> give them. Sure, the tools may be improved when cases of false positives
> are identified, and new LLMs may be trained with more (and better ?)
> data to improve the output, but they won't learn from the interactions.
> 
> How much value a maintainer sees in those reviews is up to individual
> maintainers. I will personally not consider a R-b tag from an LLM to
> mean that a patch is ready to be merged (and I believe you won't
> either). As such, I think that a R-b from an LLM is misleading and
> doesn't provide good value. At best it's free advertising for company
> making closed-source tools, which I don't think we should encourage.

That's different aspect than I raised. I agree with above approach but
it is more subjective.

What I brought is object: our docs clearly state that reviewer can offer
reviewed-by tag. They do not allow non-reviewers to offer a tag and
English is clear on that - only a person is a reviewer.

Dog is not a reviewer.

Hammer is not a reviewer.

Tool is not a reviewer.

Guenter did not bring any counter arguments that our docs ALLOW
non-person to provide a reviewed-by tag. I brought that arguments as
excerpt from our documented policy.

Best regards,
Krzysztof

Re: Stop false review statements

[Konstantin Ryabitsev]

On Sat, May 16, 2026 at 05:11:28AM -0700, Guenter Roeck wrote:
> On Sat, May 16, 2026 at 10:05:02AM +0200, Krzysztof Kozlowski wrote:
> > What the hell is that:
> > 
> > https://lore.kernel.org/all/20260515190707.033BDC2BCB0@smtp.kernel.org/
> > 
> > As a bot you CANNOT MAKE a Reviewer's statement of oversight. You are
> > not a damn human do be able to make such statement. You are a bot, a tool.
> > 
> 
> Where exactly do the rules say that ? I seem to miss that.
> 
> There is a policy document about _contributions_ made by AI, but I don't
> see the one that says that AI agents must not provide Reviewed-by: tags.

From my perspective, AI agents must NOT use the Reviewed-by tag for the
following reasons:

- We consider this a "person-trailer" and it implies agency
- Adding yourself to a commit via a trailer is a *binding responsibility* for
  the change. A lot of tooling will cc the Reviewed-by addresses on follow-up
  messages regarding code in this commit. If the address is bogus or doesn't
  go to a developer, this is both wasteful and potentially frustrating.

-K