Re: Linking Patchwork with Sashiko?

Re: Linking Patchwork with Sashiko?

[Ihor Solodrai]

On 7/14/26 5:59 PM, Laurent Pinchart wrote:
> [...]
> 
> I expect maintainers who want to act on sashiko reviews to triage and
> verify them first before bothering authors, yes.

I can't help but chime in, because I find this opinion triggeringly
unreasonable.

As of today, the problem is *not* "thoughtful experts are bombarded
with low-quality feedback from llms", even though the experts might
feel like that frequently.

The problem is that the mailing list is flooded with slop to varying
degree depending on the subsystem. And review by humans, even with
powerful AI tools at hand, doesn't scale [1][2].

And the argument "I didn't sign up for AI reviews" is weak. If you've
sent a patch to the mailing list, and there are maintainer-sanctioned
bot messages in response, you've already opted-in to receive them.

Yes, it is annoying. It's annoying in the same way a dozen nits on
your patch from a human reviewer is annoying. But the contributors
generally accept this, otherwise their patches don't land.

Your job as a contributor is to make sure the patch is "good enough"
to land. Now, with ubiquitous AI tools, "good enough" means *at least*
that the AI bots don't find real bugs.

Yes, assessing whether AI report is "real" is work, but it's not
reasonable to push this work on already overwhelmed maintainers. And
if AI is wrong, again it's the job of the author to convince the
maintainers it's wrong.

I set up automated AI code reviews on BPF list using Chris Mason's
prompts in the fall last year [3], before sashiko was developed. And
originally we were very worried about the quality of the reviews.

Time has showed that we had to be more worried about the average
quality of incoming patch instead.

[1] https://lore.kernel.org/bpf/CAADnVQKsM8iOGO3ZL3LgbWigOBBMrGbTYZ_k_Ktz=+cVkvRLXg@mail.gmail.com/
[2] https://lore.kernel.org/all/20260703-vfs-summer-jam-22b2edbbbc44@brauner/
[3] https://lore.kernel.org/all/0d6aa077-1222-49a0-9554-dd922122a904@meta.com/

> I believe we should
> follow the first two recommendations of the Software Freedom Conservancy
> on using LLM-backed generative AI systems for FOSS contributions ([1]).
> 
> [1] https://sfconservancy.org/llm-gen-ai/llm-backed-generative-ai-recommendations.html
> 

Re: Linking Patchwork with Sashiko?

[Laurent Pinchart]

On Wed, Jul 15, 2026 at 09:28:03AM -0700, Ihor Solodrai wrote:
> On 7/14/26 5:59 PM, Laurent Pinchart wrote:
> > [...]
> > 
> > I expect maintainers who want to act on sashiko reviews to triage and
> > verify them first before bothering authors, yes.
> 
> I can't help but chime in, because I find this opinion triggeringly
> unreasonable.
> 
> As of today, the problem is *not* "thoughtful experts are bombarded
> with low-quality feedback from llms", even though the experts might
> feel like that frequently.
> 
> The problem is that the mailing list is flooded with slop to varying
> degree depending on the subsystem. And review by humans, even with
> powerful AI tools at hand, doesn't scale [1][2].

I certainly agree with you that this is a large problem. I don't think
we have a single problem though, but I can understand people who
consider the problem you're discribing as more urgent to address.

> And the argument "I didn't sign up for AI reviews" is weak. If you've
> sent a patch to the mailing list, and there are maintainer-sanctioned
> bot messages in response, you've already opted-in to receive them.

We're enabling bots on mailing lists that people are already subscribed
to. But I don't think that's a very important argument here.

> Yes, it is annoying. It's annoying in the same way a dozen nits on
> your patch from a human reviewer is annoying.

That I strongly disagree with. A humain reviewer sending a review with
dozens of nits usually doesn't annoy me, and doesn't make me feel
diminished as a human.

> But the contributors
> generally accept this, otherwise their patches don't land.
> 
> Your job as a contributor is to make sure the patch is "good enough"
> to land. Now, with ubiquitous AI tools, "good enough" means *at least*
> that the AI bots don't find real bugs.
> 
> Yes, assessing whether AI report is "real" is work, but it's not
> reasonable to push this work on already overwhelmed maintainers. And
> if AI is wrong, again it's the job of the author to convince the
> maintainers it's wrong.

We disagree on this as well. This would force contributors to constantly
justify their value against a machine that is known to produce a
non-negligible quantity of nonsense. Furthermore, if those generative AI
tools are as good as their supporters claim, the time they free for
maintainers should outweight the need workload to triage the comments.

> I set up automated AI code reviews on BPF list using Chris Mason's
> prompts in the fall last year [3], before sashiko was developed. And
> originally we were very worried about the quality of the reviews.
> 
> Time has showed that we had to be more worried about the average
> quality of incoming patch instead.
> 
> [1] https://lore.kernel.org/bpf/CAADnVQKsM8iOGO3ZL3LgbWigOBBMrGbTYZ_k_Ktz=+cVkvRLXg@mail.gmail.com/
> [2] https://lore.kernel.org/all/20260703-vfs-summer-jam-22b2edbbbc44@brauner/
> [3] https://lore.kernel.org/all/0d6aa077-1222-49a0-9554-dd922122a904@meta.com/
> 
> > I believe we should
> > follow the first two recommendations of the Software Freedom Conservancy
> > on using LLM-backed generative AI systems for FOSS contributions ([1]).
> > 
> > [1] https://sfconservancy.org/llm-gen-ai/llm-backed-generative-ai-recommendations.html

-- 
Regards,

Laurent Pinchart

Re: Linking Patchwork with Sashiko?

[Ihor Solodrai]

On 7/15/26 9:39 AM, Laurent Pinchart wrote:
> On Wed, Jul 15, 2026 at 09:28:03AM -0700, Ihor Solodrai wrote:
>> On 7/14/26 5:59 PM, Laurent Pinchart wrote:
>>> [...]
>>
>> Yes, assessing whether AI report is "real" is work, but it's not
>> reasonable to push this work on already overwhelmed maintainers. And
>> if AI is wrong, again it's the job of the author to convince the
>> maintainers it's wrong.
> 
> We disagree on this as well. This would force contributors to constantly
> justify their value against a machine that is known to produce a
> non-negligible quantity of nonsense.

I know what you mean and I can relate. Going through walls of
AI-generated paragraphs is not fun, to put it lightly.

However it seems that you are suggesting to simply delegate this work
to maintainers, and I very much disagree with that approach (even
though I'm not a maintainer myself).

I hope you're not saying "let's just not use AI", because that ship
has sailed a while ago.

And if automated AI reviews are there, someone has to do the
triage. AI can do the triage itself only to a limited degree. For
example, there are instructions in review-prompts where AI checks
whether an issue has already been discussed. But it will never be
perfect.

I think it makes much more sense for the author of the patch to do the
triage, as they have the best context of the work, and they are
usually the most invested in their patch being merged in.

> Furthermore, if those generative AI
> tools are as good as their supporters claim, the time they free for
> maintainers should outweight the need workload to triage the comments.

Even if the AI tools were ~perfect with respect to code review, I
think people would still be signing off patches and taking
responsibility for them.

An imaginary perfect AI reviewer that maintainer can trust to land
code without looking at it would essentially become an AI maintainer.
Independent of whether you think it's possible, it hasn't happened
yet, so we have to deal with imperfect AI in the meanwhile.


> >> I set up automated AI code reviews on BPF list using Chris Mason's
>> prompts in the fall last year [3], before sashiko was developed. And
>> originally we were very worried about the quality of the reviews.
>>
>> Time has showed that we had to be more worried about the average
>> quality of incoming patch instead.
>>
>> [1] https://lore.kernel.org/bpf/CAADnVQKsM8iOGO3ZL3LgbWigOBBMrGbTYZ_k_Ktz=+cVkvRLXg@mail.gmail.com/
>> [2] https://lore.kernel.org/all/20260703-vfs-summer-jam-22b2edbbbc44@brauner/
>> [3] https://lore.kernel.org/all/0d6aa077-1222-49a0-9554-dd922122a904@meta.com/
>>
>>> I believe we should
>>> follow the first two recommendations of the Software Freedom Conservancy
>>> on using LLM-backed generative AI systems for FOSS contributions ([1]).
>>>
>>> [1] https://sfconservancy.org/llm-gen-ai/llm-backed-generative-ai-recommendations.html
> 

Re: Linking Patchwork with Sashiko?

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1650 bytes --]

On Wed, Jul 15, 2026 at 10:12:03AM -0700, Ihor Solodrai wrote:
> On 7/15/26 9:39 AM, Laurent Pinchart wrote:
> > On Wed, Jul 15, 2026 at 09:28:03AM -0700, Ihor Solodrai wrote:

> >> Yes, assessing whether AI report is "real" is work, but it's not
> >> reasonable to push this work on already overwhelmed maintainers. And
> >> if AI is wrong, again it's the job of the author to convince the
> >> maintainers it's wrong.

> > We disagree on this as well. This would force contributors to constantly
> > justify their value against a machine that is known to produce a
> > non-negligible quantity of nonsense.

> I know what you mean and I can relate. Going through walls of
> AI-generated paragraphs is not fun, to put it lightly.

> However it seems that you are suggesting to simply delegate this work
> to maintainers, and I very much disagree with that approach (even
> though I'm not a maintainer myself).

This is something that already happens with human written reviews,
indeed I have a canned reply for one particular reviewer telling
submitters it's OK to ignore them.  It goes in both directions - it can
be ignoring some issues or postponing them for later, or it can be
making sure that there's input from particular people.

> I think it makes much more sense for the author of the patch to do the
> triage, as they have the best context of the work, and they are
> usually the most invested in their patch being merged in.

It's not clear cut - untrustworthy tooling often causes problems, I'm
fairly sure I'm seeing issues that are the result of submitters getting
mislead by LLM reviews in a similar manner to some compiler warnings.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Linking Patchwork with Sashiko?

[Jason Gunthorpe]

On Wed, Jul 15, 2026 at 07:39:21PM +0300, Laurent Pinchart wrote:
> > Yes, assessing whether AI report is "real" is work, but it's not
> > reasonable to push this work on already overwhelmed maintainers. And
> > if AI is wrong, again it's the job of the author to convince the
> > maintainers it's wrong.
> 
> We disagree on this as well. This would force contributors to constantly
> justify their value against a machine that is known to produce a
> non-negligible quantity of nonsense. Furthermore, if those generative AI
> tools are as good as their supporters claim, the time they free for
> maintainers should outweight the need workload to triage the comments.

I feel Sashiko and this whole AI world has created alot more work. It
is not saving time at all. Now I have to go and read every Sashiko
report to see if there is something important that normal review
missed. I have fix all the drive by 'this is already broken' stuff it
finds. We get way more patches than we did before. Now *everything*
has to be looked at more skeptically if came from some AI slop. I get
crazy real security reports that need special hand holding now that we
never got before.

Even on the submitting side now I have to run patches through all
kinds of AI tools to have a solid chance to navigate the new guantlet
upstream.

We aren't saving time, or creating less work. My hope is we are
creating better quality, higher security and less bugs.

Jason

Re: Linking Patchwork with Sashiko?

[Ihor Solodrai]

On 7/15/26 10:35 AM, Mark Brown wrote:
> On Wed, Jul 15, 2026 at 10:12:03AM -0700, Ihor Solodrai wrote:
>> On 7/15/26 9:39 AM, Laurent Pinchart wrote:
>>> On Wed, Jul 15, 2026 at 09:28:03AM -0700, Ihor Solodrai wrote:
> 
>>>> Yes, assessing whether AI report is "real" is work, but it's not
>>>> reasonable to push this work on already overwhelmed maintainers. And
>>>> if AI is wrong, again it's the job of the author to convince the
>>>> maintainers it's wrong.
> 
>>> We disagree on this as well. This would force contributors to constantly
>>> justify their value against a machine that is known to produce a
>>> non-negligible quantity of nonsense.
> 
>> I know what you mean and I can relate. Going through walls of
>> AI-generated paragraphs is not fun, to put it lightly.
> 
>> However it seems that you are suggesting to simply delegate this work
>> to maintainers, and I very much disagree with that approach (even
>> though I'm not a maintainer myself).
> 
> This is something that already happens with human written reviews,
> indeed I have a canned reply for one particular reviewer telling
> submitters it's OK to ignore them.  It goes in both directions - it can
> be ignoring some issues or postponing them for later, or it can be
> making sure that there's input from particular people.

My objection is to a suggested expectation for every review to be
triaged by maintainers/reviewers *before* it's sent or the author
engages with it.

I think the author should be the first one to engage, it's a feedback
on their patch after all. And of course reviewers/maintainers are free
to jump in whenever they can and feel it's necessary, and they do. Opt-in.

> 
>> I think it makes much more sense for the author of the patch to do the
>> triage, as they have the best context of the work, and they are
>> usually the most invested in their patch being merged in.
> 
> It's not clear cut - untrustworthy tooling often causes problems, I'm
> fairly sure I'm seeing issues that are the result of submitters getting
> mislead by LLM reviews in a similar manner to some compiler warnings.

Yes, this is a valid point. I'm not arguing automated AI feedback is
always good. This is why the triage is needed in the first place.

I think one small thing that may help with this is teaching
contributors to not iterate on the revisions too quickly based solely
on AI feedback. And also try to iterate with AI reviews off-list first.

But that's marginal. Fundamentally, a human has to read AI feedback on
the patch in the end. I don't see a way around it, barring the singularity™.

Re: Linking Patchwork with Sashiko?

[Laurent Pinchart]

On Wed, Jul 15, 2026 at 10:12:03AM -0700, Ihor Solodrai wrote:
> On 7/15/26 9:39 AM, Laurent Pinchart wrote:
> > On Wed, Jul 15, 2026 at 09:28:03AM -0700, Ihor Solodrai wrote:
> >> On 7/14/26 5:59 PM, Laurent Pinchart wrote:
> >>> [...]
> >>
> >> Yes, assessing whether AI report is "real" is work, but it's not
> >> reasonable to push this work on already overwhelmed maintainers. And
> >> if AI is wrong, again it's the job of the author to convince the
> >> maintainers it's wrong.
> > 
> > We disagree on this as well. This would force contributors to constantly
> > justify their value against a machine that is known to produce a
> > non-negligible quantity of nonsense.
> 
> I know what you mean and I can relate. Going through walls of
> AI-generated paragraphs is not fun, to put it lightly.
> 
> However it seems that you are suggesting to simply delegate this work
> to maintainers,

For authors who object to generative AI, yes. And most likely for junior
developers too, see below.

> and I very much disagree with that approach (even
> though I'm not a maintainer myself).

I had noticed that :-)

> I hope you're not saying "let's just not use AI", because that ship
> has sailed a while ago.

For the time being I'm not seeing this as a viable option in any way, so
I'm not calling for it.

> And if automated AI reviews are there, someone has to do the
> triage. AI can do the triage itself only to a limited degree. For
> example, there are instructions in review-prompts where AI checks
> whether an issue has already been discussed. But it will never be
> perfect.
> 
> I think it makes much more sense for the author of the patch to do the
> triage, as they have the best context of the work, and they are
> usually the most invested in their patch being merged in.

The discussion regarding enabling sashiko for the linux-media mailing
list started with the question of whether or not authors should be
CC'ed. This came from a consideration on the impact of false positives
and other hallucinations on junior developers. They may have the best
context of the work, but they are also most at risk of not questioning
the validity of the reviews. A one-size-fits-them-all approach may not
be good, which is why I think some sort of opt-in option could address
many concerns. I don't expect immediate and universal enthousiasm
though.

> > Furthermore, if those generative AI
> > tools are as good as their supporters claim, the time they free for
> > maintainers should outweight the need workload to triage the comments.
> 
> Even if the AI tools were ~perfect with respect to code review, I
> think people would still be signing off patches and taking
> responsibility for them.

My comment wasn't about signing off patches. I was questioning whether
the time that sashiko saves for a maintainer is smaller, similar, or
larger than the time it takes to triage the comments. If the tool saves
maintainers from having to review some of the code, or from considering
some classes of problems in their reviews, would triaging the comments
come even close to the time saved ?

> An imaginary perfect AI reviewer that maintainer can trust to land
> code without looking at it would essentially become an AI maintainer.
> Independent of whether you think it's possible, it hasn't happened
> yet, so we have to deal with imperfect AI in the meanwhile.
> 
> > >> I set up automated AI code reviews on BPF list using Chris Mason's
> >> prompts in the fall last year [3], before sashiko was developed. And
> >> originally we were very worried about the quality of the reviews.
> >>
> >> Time has showed that we had to be more worried about the average
> >> quality of incoming patch instead.
> >>
> >> [1] https://lore.kernel.org/bpf/CAADnVQKsM8iOGO3ZL3LgbWigOBBMrGbTYZ_k_Ktz=+cVkvRLXg@mail.gmail.com/
> >> [2] https://lore.kernel.org/all/20260703-vfs-summer-jam-22b2edbbbc44@brauner/
> >> [3] https://lore.kernel.org/all/0d6aa077-1222-49a0-9554-dd922122a904@meta.com/
> >>
> >>> I believe we should
> >>> follow the first two recommendations of the Software Freedom Conservancy
> >>> on using LLM-backed generative AI systems for FOSS contributions ([1]).
> >>>
> >>> [1] https://sfconservancy.org/llm-gen-ai/llm-backed-generative-ai-recommendations.html

-- 
Regards,

Laurent Pinchart

Re: Linking Patchwork with Sashiko?

[Laurent Pinchart]

On Wed, Jul 15, 2026 at 02:41:53PM -0300, Jason Gunthorpe wrote:
> On Wed, Jul 15, 2026 at 07:39:21PM +0300, Laurent Pinchart wrote:
> > > Yes, assessing whether AI report is "real" is work, but it's not
> > > reasonable to push this work on already overwhelmed maintainers. And
> > > if AI is wrong, again it's the job of the author to convince the
> > > maintainers it's wrong.
> > 
> > We disagree on this as well. This would force contributors to constantly
> > justify their value against a machine that is known to produce a
> > non-negligible quantity of nonsense. Furthermore, if those generative AI
> > tools are as good as their supporters claim, the time they free for
> > maintainers should outweight the need workload to triage the comments.
> 
> I feel Sashiko and this whole AI world has created alot more work. It
> is not saving time at all. Now I have to go and read every Sashiko
> report to see if there is something important that normal review
> missed. I have fix all the drive by 'this is already broken' stuff it
> finds. We get way more patches than we did before. Now *everything*
> has to be looked at more skeptically if came from some AI slop. I get
> crazy real security reports that need special hand holding now that we
> never got before.

I'd even go as far as questioning if the additional work is resulting in
less time being spent on changes that would address classes of problems.
There are many areas in the kernel where APIs are hard to use correctly.
Improvements there (and I know there's ongoing work such as Kees' object
allocator functions for instance) may pay off better than spending time
discussing about one-time memory leaks in error paths of probe
functions.

> Even on the submitting side now I have to run patches through all
> kinds of AI tools to have a solid chance to navigate the new guantlet
> upstream.
> 
> We aren't saving time, or creating less work. My hope is we are
> creating better quality, higher security and less bugs.

-- 
Regards,

Laurent Pinchart

Re: Linking Patchwork with Sashiko?

[Dmitry Torokhov]

On Wed, Jul 15, 2026 at 02:41:53PM -0300, Jason Gunthorpe wrote:
> On Wed, Jul 15, 2026 at 07:39:21PM +0300, Laurent Pinchart wrote:
> > > Yes, assessing whether AI report is "real" is work, but it's not
> > > reasonable to push this work on already overwhelmed maintainers. And
> > > if AI is wrong, again it's the job of the author to convince the
> > > maintainers it's wrong.
> > 
> > We disagree on this as well. This would force contributors to constantly
> > justify their value against a machine that is known to produce a
> > non-negligible quantity of nonsense. Furthermore, if those generative AI
> > tools are as good as their supporters claim, the time they free for
> > maintainers should outweight the need workload to triage the comments.
> 
> I feel Sashiko and this whole AI world has created alot more work. It
> is not saving time at all. Now I have to go and read every Sashiko
> report to see if there is something important that normal review
> missed. I have fix all the drive by 'this is already broken' stuff it
> finds. We get way more patches than we did before. Now *everything*
> has to be looked at more skeptically if came from some AI slop. I get
> crazy real security reports that need special hand holding now that we
> never got before.

No, we are just becoming aware how awful our code actually is and how
may issues have been overlooked by experienced developers and
maintainers, not even talking about junior contributors.

So save all these "pre-existing issue" reports in todo/ and go over them
when you have a chance.

> 
> Even on the submitting side now I have to run patches through all
> kinds of AI tools to have a solid chance to navigate the new guantlet
> upstream.

Yes, I experience the same. But that just shows that we are not perfect
and prone to making mistakes.

> 
> We aren't saving time, or creating less work. My hope is we are
> creating better quality, higher security and less bugs.

I believe we are. Hopefully as we are fixing all these pre-existing
issues we will be seeing less of them...

Thanks.

-- 
Dmitry

Re: Linking Patchwork with Sashiko?

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

On Wed, Jul 15, 2026 at 10:10:46PM +0300, Laurent Pinchart wrote:
> On Wed, Jul 15, 2026 at 02:41:53PM -0300, Jason Gunthorpe wrote:

> > I feel Sashiko and this whole AI world has created alot more work. It
> > is not saving time at all. Now I have to go and read every Sashiko
> > report to see if there is something important that normal review
> > missed. I have fix all the drive by 'this is already broken' stuff it
> > finds. We get way more patches than we did before. Now *everything*
> > has to be looked at more skeptically if came from some AI slop. I get
> > crazy real security reports that need special hand holding now that we
> > never got before.

> I'd even go as far as questioning if the additional work is resulting in
> less time being spent on changes that would address classes of problems.
> There are many areas in the kernel where APIs are hard to use correctly.
> Improvements there (and I know there's ongoing work such as Kees' object
> allocator functions for instance) may pay off better than spending time
> discussing about one-time memory leaks in error paths of probe
> functions.

Yeah, that's been one of the big issues I've noticed with people
handling the LLM reviews poorly - people tend to end up with very
tactical approachs for issues that are not at all tactical.  Which is
one of the symptoms of the stuff the LLMs generate themselves.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Linking Patchwork with Sashiko?

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1576 bytes --]

On Wed, Jul 15, 2026 at 11:39:41AM -0700, Ihor Solodrai wrote:
> On 7/15/26 10:35 AM, Mark Brown wrote:
> > On Wed, Jul 15, 2026 at 10:12:03AM -0700, Ihor Solodrai wrote:

> >> However it seems that you are suggesting to simply delegate this work
> >> to maintainers, and I very much disagree with that approach (even
> >> though I'm not a maintainer myself).

> > This is something that already happens with human written reviews,
> > indeed I have a canned reply for one particular reviewer telling
> > submitters it's OK to ignore them.  It goes in both directions - it can
> > be ignoring some issues or postponing them for later, or it can be
> > making sure that there's input from particular people.

> My objection is to a suggested expectation for every review to be
> triaged by maintainers/reviewers *before* it's sent or the author
> engages with it.

> I think the author should be the first one to engage, it's a feedback
> on their patch after all. And of course reviewers/maintainers are free
> to jump in whenever they can and feel it's necessary, and they do. Opt-in.

OTOH I don't really want to have to not only look at whatever random
tool is sending reports but also have to check replies from submitters
for routine stuff, I already have a perfectly adequate supply of email.
What I'm seeing is more at the level where I'm likely to want to provide
some additional feedback to people, or where there's a lot of false
reports to ignore.  If people look themselves and take appropriate
action then great, but it's not at the level of the build test bots.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Linking Patchwork with Sashiko?

[Theodore Tso]

On Wed, Jul 15, 2026 at 10:06:54PM -0500, Laurent Pinchart wrote:
> This came from a consideration on the impact of false positives
> and other hallucinations on junior developers.

You keep claiming that there is an awful lot of false positives and
other hallucinations.  That's an assertion that needs to be tested to
make sure that we're not relying on a few pieces of anecdata.  How
does this compare with false positives and bogus recommendations from
certain human reviewers?

After all, I can think of at *least* one human reviewer whose reviews
are far worse than Sashiko's in false positives.  That might or might
not be the same person as the one Mark has said that he has a stock
e-mail reply telling a junior developer that they should be free to
ignore reviews from that particular human being.  :-)

Each maintainer can decide how much they trust Shashiko reviews.  At
least for my subsystem, I haven't seen the hallucionations that you
are continually citing, certainly not as a common case.  I will note
that even experienced human reviewers have occasionally flagged an
issue, only to say, "my bad, you're right, it's fine" after a
discussion with the patch author.  And that's OK.  When a human being
makes a false positive criticism, no one responds with "I am feeling
demeaned as a human being!"  Perhaps we should be giving
tool-generated reviews (whether it is via checkpatch or Sashiko) with
a similar level of grace?

At least with tool-generated reviews, we can send bug reports asking
that the tool can be improved.  At least with one particular human
reviewer that I have in mind, no attempts asking that person to
improve their reviews hasn't been helpful.

Bottom line --- if you are making the claim the Sashiko reviews are
rife with false positives, all I can say is <<Citation Needed>>.
 
					- Ted

Re: Linking Patchwork with Sashiko?

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 4020 bytes --]

Le mercredi 15 juillet 2026 à 17:13 -0400, Theodore Tso a écrit :
> On Wed, Jul 15, 2026 at 10:06:54PM -0500, Laurent Pinchart wrote:
> > This came from a consideration on the impact of false positives
> > and other hallucinations on junior developers.
> 
> You keep claiming that there is an awful lot of false positives and
> other hallucinations.  That's an assertion that needs to be tested to
> make sure that we're not relying on a few pieces of anecdata.  How
> does this compare with false positives and bogus recommendations from
> certain human reviewers?

The difference is that it comes with a highly credible explanation, which takes
a lot more time to solve. On my end, having spent the evening through quite an
unusual amount of build tested AI review fixes, I'm starting to learn the common
broken pattern, so I guess I'll get better at filtering.

In V4L2 m2m notably, there is a "job" in the background, which allow to safely
(at least this is the theory) chain up work, and synchronize against pending
work. But Sashiko (and other AI) seems to not get it and state with confidence
and long explanation that the drivers are racy and should do X/Y/Z more on top
(and never blames the framework).

This is why I recommended not forwarding the reviews to the linux media list
just yet during the summit. Now, the tricky part will be to find time to try and
help narrow this down, I never checked the implementation, but I suppose you can
add more layers of prompts to force into looking deeper ?

> 
> After all, I can think of at *least* one human reviewer whose reviews
> are far worse than Sashiko's in false positives.  That might or might
> not be the same person as the one Mark has said that he has a stock
> e-mail reply telling a junior developer that they should be free to
> ignore reviews from that particular human being.  :-)

Hopefully I'm not in that list. We all make that mistake of being convincing in
the wrong direction from time to time, with the difference that when we do, we
usually offer some help, and try to be kind after that.

> 
> Each maintainer can decide how much they trust Shashiko reviews.  At
> least for my subsystem, I haven't seen the hallucionations that you
> are continually citing, certainly not as a common case.  I will note
> that even experienced human reviewers have occasionally flagged an
> issue, only to say, "my bad, you're right, it's fine" after a
> discussion with the patch author.  And that's OK.  When a human being
> makes a false positive criticism, no one responds with "I am feeling
> demeaned as a human being!"  Perhaps we should be giving
> tool-generated reviews (whether it is via checkpatch or Sashiko) with
> a similar level of grace?

I'm not sure I'd call it an hallucination, it does looks racy on the surface.
The tool simply lack bit of specialization. Its also something we observe with
reference passing in GStreamer project, where the AI does a perfect job in Rust,
where this is explicit in the language, and fail miserably in C where its all
about knowing the conventions (both when reviewing and coding).

> 
> At least with tool-generated reviews, we can send bug reports asking
> that the tool can be improved.  At least with one particular human
> reviewer that I have in mind, no attempts asking that person to
> improve their reviews hasn't been helpful.

And yes, I'd rather improve the tool then kill it. For me, I seconded the
decision to not send emails to the linux media list, so we could gain some time.
As its obvious that once its more visible, we'll start getting more and more of
the build tested fix based on AI report made by people that don't even own the
hardware to test their changes and are likely just interested in adding their
name in the changelog.

> 
> Bottom line --- if you are making the claim the Sashiko reviews are
> rife with false positives, all I can say is <<Citation Needed>>.

cheers,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Linking Patchwork with Sashiko?

[Theodore Tso]

On Wed, Jul 15, 2026 at 11:00:54PM -0500, Nicolas Dufresne wrote:
> In V4L2 m2m notably, there is a "job" in the background, which allow
> to safely (at least this is the theory) chain up work, and
> synchronize against pending work. But Sashiko (and other AI) seems
> to not get it and state with confidence and long explanation that
> the drivers are racy and should do X/Y/Z more on top (and never
> blames the framework).

Is there actually a race which is allowed by the framework?  Or is
Sashiko's proposed race actually not possible?

Or is it a case where it actually *is* racy, but it's something we
really don't care about?  I've seen this happen with KCSAN where
technically it really *is* a data race, but it's actually not
important --- so we just have to mark it with the data_race() macro[1].

[1] https://docs.kernel.org/dev-tools/lkmm/docs/access-marking.html

This sort of thing can happen any time we introduce a new tool,
whether it's KCSAN or lockdep.  It's not necesarily unique to AI
reviews.  In both AI and non-AI tools, we can have developers who fix
the problem in the wrong way, or where the fix actually makes things
worse than the putative "bug".

> This is why I recommended not forwarding the reviews to the linux
> media list just yet during the summit. Now, the tricky part will be
> to find time to try and help narrow this down, I never checked the
> implementation, but I suppose you can add more layers of prompts to
> force into looking deeper ?

Has a bug been filed with the Shashiko folks?  I can imagine any
number of solutions, including adding comments in the source, or some
kind of marker much like the data_race() macro to shut up KCSAN
warnings, or __attribute__((unused)) to shut up gcc or Clang warnings.
Yet another solution might be to add some kind of a AGENTS.md file in
the explains why a particular construct is safe.  Personally, I
consider it to be the less desirable solution, since having something
inline in the sources is likely to get out of sync with future code
changes, and having the annotation in the sources will hopefully be
useful to humans, not just AI bots.

But the exact solution is going to be very specific exact warning and
why the AI bot thinks there is a problem when it really isn't.

    	       	      	       	 	 - Ted

P.S.  There was a recent thread[1] where using Clang's context
analysis resulted in a false positive, and I think it's instructive to
see how it was handled compared to false positives from Sashiko.  We
had a discussion, and the result was two pull requests to LLVM to
avoid the false positive.  There was no complaints about how the false
positive somehow was demeaning to our humanity, or that we should be
accomodate those people who want to ban the use of static analyzers.
There is something about AI that seems to trigger a different sort of
energy, for better or for worse.

[1] https://lore.kernel.org/r/20260712165610.366474-2-timday@thelustrecollective.com

Re: Linking Patchwork with Sashiko?

[Jai Luthra]

Hi Ted,

Quoting Theodore Tso (2026-07-16 02:43:39)
> On Wed, Jul 15, 2026 at 10:06:54PM -0500, Laurent Pinchart wrote:
> > This came from a consideration on the impact of false positives
> > and other hallucinations on junior developers.
> 
> You keep claiming that there is an awful lot of false positives and
> other hallucinations.  That's an assertion that needs to be tested to
> make sure that we're not relying on a few pieces of anecdata.  How
> does this compare with false positives and bogus recommendations from
> certain human reviewers?
> 
> After all, I can think of at *least* one human reviewer whose reviews
> are far worse than Sashiko's in false positives.  That might or might
> not be the same person as the one Mark has said that he has a stock
> e-mail reply telling a junior developer that they should be free to
> ignore reviews from that particular human being.  :-)
> 

That's actually a point against Sashiko. I can maintain a mental score of 
different humans, their past review comments and behaviour, and other 
contextual things like the time of day of their response.

It's really hard to do that with a bot like Sashiko, which can show two 
extremes (even in the same review):

1. Find insane(ly good) security bugs that other humans missed
2. Hallucinate code that doesn't exist, or give false positives 
goose-chases [1][2] that make sense if you read the code like prose, but 
miss key details or context

Which makes me extra cautious and thorough when going through LLM generated 
reviews compared to a human reviewer I trust.

Later in this thread you've equated Sashiko to static-analyzer tools 
instead of humans (which is a bit convenient).

IMO false positives are possible everywhere, but LLMs are unpredictable in 
a very different way compared to static-analysis tools *especially* because 
they can be quite good at finding some issues and are non-deterministic.

I can understand the appeal of using them to find security bugs or as extra 
"eye balls to make all bugs shallow". And I am not advocating AI as bad or 
to ban it. But I do think usage of LLMs through Sashiko may give us free 
reviews, but require brand new work of reviewing the reviews. And this new 
work will be a burden on experienced folks, not newcomers, who will simply 
believe the LLM output because it is (or sounds) smarter than them.

[1] https://lore.kernel.org/all/20260520160638.7F2BA1F000E9@smtp.kernel.org/
[2] https://lore.kernel.org/all/20260715174326.0E5B51F00A3A@smtp.kernel.org/

See the pm_rumtime_put() suggestions above. Some people started giving the 
same review comments after possibly seeing Sashiko comments.

But on deeper inspection it turned out to be a false-positive:
lore.kernel.org/all/178080367434.9570.8101421417305606672@freya

My point isn't that static-analysis tools can't make the same mistake, but 
that they're open source and fixable in a way that current architecture of 
LLM models isn't in general, other than giving it an ever-increasing prompt 
and hoping the random-number generator gods bless me when I need them to.

Thanks,
Jai

> Each maintainer can decide how much they trust Shashiko reviews.  At
> least for my subsystem, I haven't seen the hallucionations that you
> are continually citing, certainly not as a common case.  I will note
> that even experienced human reviewers have occasionally flagged an
> issue, only to say, "my bad, you're right, it's fine" after a
> discussion with the patch author.  And that's OK.  When a human being
> makes a false positive criticism, no one responds with "I am feeling
> demeaned as a human being!"  Perhaps we should be giving
> tool-generated reviews (whether it is via checkpatch or Sashiko) with
> a similar level of grace?
> 
> At least with tool-generated reviews, we can send bug reports asking
> that the tool can be improved.  At least with one particular human
> reviewer that I have in mind, no attempts asking that person to
> improve their reviews hasn't been helpful.
> 
> Bottom line --- if you are making the claim the Sashiko reviews are
> rife with false positives, all I can say is <<Citation Needed>>.
>  

PS: There is a separate issue of how LF is advocating closed-weights paid 
models trained on public data, just because it is free (as in beer) today 
for *them* because they're a big organization. That demotivates a lot of 
people (including me) who like working on open source projects because 
they're open to inspect and tinker with.

Unfortunately if the "my way or the highway" approach is pushed too much, 
the kernel might end up with a lot of free Gemini tokens but not many 
enthusiastic humans to use them :-)

I've personally stopped using closed-weights model for anything, but 
somehow now required to interact with one while working on the biggest open 
source project in the world?? Maybe LF hasn't gotten the memo on how good 
the open-weights models are these days, so I'll stop my whining here and 
wait for a few months ;-)

>                                         - Ted
>

Re: Linking Patchwork with Sashiko?

[Mauro Carvalho Chehab]

On Wed, 15 Jul 2026 23:53:55 -0400
"Theodore Tso" <tytso@mit.edu> wrote:

> On Wed, Jul 15, 2026 at 11:00:54PM -0500, Nicolas Dufresne wrote:
> > In V4L2 m2m notably, there is a "job" in the background, which allow
> > to safely (at least this is the theory) chain up work, and
> > synchronize against pending work. But Sashiko (and other AI) seems
> > to not get it and state with confidence and long explanation that
> > the drivers are racy and should do X/Y/Z more on top (and never
> > blames the framework).  
> 
> Is there actually a race which is allowed by the framework?  Or is
> Sashiko's proposed race actually not possible?
> 
> Or is it a case where it actually *is* racy, but it's something we
> really don't care about?  I've seen this happen with KCSAN where
> technically it really *is* a data race, but it's actually not
> important --- so we just have to mark it with the data_race() macro[1].
> 
> [1] https://docs.kernel.org/dev-tools/lkmm/docs/access-marking.html
> 
> This sort of thing can happen any time we introduce a new tool,
> whether it's KCSAN or lockdep.  

Lockdep warnings are particularly tricky and hard to fix. We did
have false-positive lockdep warnings ("hallucinations") in the past
at subsystem level, which took years and a lot of work to find better
ways to fix them.

> It's not necesarily unique to AI
> reviews.  In both AI and non-AI tools, we can have developers who fix
> the problem in the wrong way, or where the fix actually makes things
> worse than the putative "bug".

Very true.

> > This is why I recommended not forwarding the reviews to the linux
> > media list just yet during the summit. Now, the tricky part will be
> > to find time to try and help narrow this down, I never checked the
> > implementation, but I suppose you can add more layers of prompts to
> > force into looking deeper ?  
> 
> Has a bug been filed with the Shashiko folks?  I can imagine any
> number of solutions, including adding comments in the source, or some
> kind of marker much like the data_race() macro to shut up KCSAN
> warnings, or __attribute__((unused)) to shut up gcc or Clang warnings.
> Yet another solution might be to add some kind of a AGENTS.md file in
> the explains why a particular construct is safe.  Personally, I
> consider it to be the less desirable solution, since having something
> inline in the sources is likely to get out of sync with future code
> changes, and having the annotation in the sources will hopefully be
> useful to humans, not just AI bots.
> 
> But the exact solution is going to be very specific exact warning and
> why the AI bot thinks there is a problem when it really isn't.

Agreed.

Just like any tool, LLM can produce very wrong or very good results
depending on how one uses it. I'm pretty much convinced that media
needs a custom set of prompts to avoid it to report false positives
and to make it point things that are more relevant to us.

> 
>     	       	      	       	 	 - Ted
> 
> P.S.  There was a recent thread[1] where using Clang's context
> analysis resulted in a false positive, and I think it's instructive to
> see how it was handled compared to false positives from Sashiko.  We
> had a discussion, and the result was two pull requests to LLVM to
> avoid the false positive.  There was no complaints about how the false
> positive somehow was demeaning to our humanity, or that we should be
> accomodate those people who want to ban the use of static analyzers.
> There is something about AI that seems to trigger a different sort of
> energy, for better or for worse.
> 
> [1] https://lore.kernel.org/r/20260712165610.366474-2-timday@thelustrecollective.com
> 



Thanks,
Mauro