Re: Stop false review statements

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

On Mon, 18 May 2026 08:16:01 -0400
"Theodore Tso" <tytso@mit.edu> wrote:

> On Mon, May 18, 2026 at 11:04:29AM +0300, Jani Nikula wrote:
> > > Sashiko is supporting various LLMs, including open models - it’s just a practical
> > > choice: to my knowledge the quality of open models is not on par with frontier closed
> > > models and it would require a non-trivial amount of hardware and infrastructure to run
> > > an open model at the required scale.  
> > 
> > In the context of the "Reviewed-by: Sashiko" discussion, this actually
> > makes it really hard to assess the quality of those reviews.  
> 
> Agreed.  There's a reason why the coding-assistants.rst specifies the
> model which is used:
> 
>   Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
> 
> The problem is that (as Jon has pointed out) coding-assistants.rst was
> intended for use when the tool was beging used to help create the code
> --- that is, "Coding Assistants".  What we're doing here is more of a
> reviewer assistance.  Something like:
> 
>   Scanned-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]
> 
> Would be more interesting, but it doesn't actually tell us anything
> about what the results were of the scan.

I don't like scanned-by, even for a tool that would always get the
same results like checkpatch. For LLM, this is even worse, as two runs
may give different results for the same code.

IMO what makes much more sense is to add information there when
a change was done due avoid a problem detected by sashiko, and what
was the fixed issue, e.g. a textual description like:

	Changed locking schema after Sashiko's report report about
	XXX race condition.

Also, except if 100% of the 

> One of the problems here is that there is a distinction between the
> infrastructure and review prompts in the Sashiko github repository,
> and the reviews that are being published by Sashiko the web service
> being run by Google that is being lost by some folks.  So I wonder if
> for now, we should just do something like:
> 
> Link: https://sashiko.dev/#/patchset/20260515091829.194810-1-me%40linux.beauty

Adding a link makes sense to me. It doesn't need to be to the
sashiko's email though: it can be to the entire thread.

> Or just have a link to lore where the review has responded to the
> Sashiko review stating where the Sashiko review reported a
> pre-existing condition (perhaps one that we don't care about because
> races in readahead logic is really Not A Big Deal, etc.)  We go for
> this strategy, it would actually be better for the Shashiko.dev review
> to get cc'ed to the mailing list.
> 
> Personally, I think that's probably be best way to go.  We already
> don't insert into the git commit an explanation of why some bullsh*t
> review by some wannabe human reviewer should be ignored, or why a
> discussion of some problem discovered by a human review in the source
> of the review would be handled in a future patch set.  That's what the
> discussion on lore.kernel.org is for.  And we shouldn't treat AI
> reviews any different from how we deal with human reviews.  So if we
> want to give credit to an AI review, then let's go with the
> Scanned-by. 

If the entire content of a Sashiko review is ignored, I don't think
it is worth adding anything. Just like we do with humans, IMO the
best is to just mention what changed due to some feedback received
by a human or by a bot, mentioning who/what bot helped to identify
the issue.

> Or we can just let people look at the mailing list, and
> if people want to have statistics, we can ask people to use a script
> running against public inbox to figure things out.
> 
> 						- Ted
> 



Thanks,
Mauro