Re: Stop false review statements

[Mauro Carvalho Chehab <mchehab+huawei@kernel.org>]

On Sun, 17 May 2026 11:17:06 -0700
Roman Gushchin <roman.gushchin@linux.dev> wrote:

> > On May 17, 2026, at 9:40 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > 
> > On Sun, 17 May 2026 12:12:00 +0200
> > Greg KH <gregkh@linuxfoundation.org> wrote:
> >   
> >>> On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
> >>> On Sat, 16 May 2026 14:59:44 -0700
> >>> Roman Gushchin <roman.gushchin@linux.dev> wrote:
> >>>   
> >>>>> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
> >>>>> 
> >>>>> I find it opposite: clogging commits with useless information, because
> >>>>> some arbitrary and completely closed-source tool did analysis means
> >>>>> nothing to me one year later when I look at the commit in the Git history.      
> >>>> 
> >>>> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
> >>>> and the code belongs to LF.     
> >>>   
> >>>> Yes, the instance behind sashiko.dev is using
> >>>> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation -
> >>>> Sashiko is supporting various LLMs, including open models - it’s just a practical
> >>>> choice: to my knowledge the quality of open models is not on par with frontier closed
> >>>> models     
> >>> 
> >>> I would very much prefer using an open source LLM, even if not in pair
> >>> with latest paid models.
> >>>   
> >>>> and it would require a non-trivial amount of hardware and infrastructure to run
> >>>> an open model at the required scale.    
> >>> 
> >>> IMHO the best would be to have them running on some infra that would accept
> >>> open source models (*). If there aren't enough resources to have our own
> >>> infra, there are offers out there which allows running open source models
> >>> like https://ollama.com/pricing (I never used myself).
> >>> 
> >>> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
> >>>    Not bad on my tests running it locally.    
> >> 
> >> You can run the tool locally, with whatever model you want, if you want
> >> to.
> >> 
> >> But for now, let's just take the free credits that Google is willing to
> >> throw at this thing and let it give us reviews IF the maintainer of the
> >> subsystem feels it is something they want to do.  No one is forcing
> >> maintainers to do this.  
> > 
> > If Google and/or others are willing to give free credits on their cloud,
> > they could instead or in addition give free credits to run ollama
> > there, allowing us to use different models.
> > 
> > From my side, while I won't personally object getting reviews from
> > Sashiko/Gemini, this is something I can't reproduce locally. I would
> > very much want something where I can select my LLM preferred model
> > and run on my ollama docker container on my own GPU, in a way that
> > I could run it locally before even sending a patch series.  
> 
> 2 thoughts here:
> 1) I actually tried to run it with ollama on my personal framework 13. Adding nominal support is trivial,
> but the whole thing is not really useful: I can get maybe few hundreds tokens per second using
> a quantified model with reduced quality; an average sashiko review is consuming 3.5 millions tokens 
> (with Gemini 3.1 pro, it’s also model-dependent).

Do you mean 3.5 millions tokens per patch series? If so, that
sounds a lot! Why does it require too many tokens?

> I’m personally all in on having the entire thing as open as possible and I believe Sashiko is what 
> is realistically the best at this moment - a fully open-source harness and set of prompts which 
> can work with a variety of models.
> I’m happy to merge a support for any LLM model which can produce decent review results.
> 
> 2) Due to probabilistic nature of LLMs, nothing is reproducible in a strict sense of the word.
> Even with exactly the same model/harness/prompts you’ll get different results every time you run it.
> It’s unfortunate, but it is what it is at the moment.

By "reproduce locally", I didn't mean in strict sense. Sure, LLM answers
won't be identical, but I suspect that at least most of the major issues 
on a patch series would be reported by any decent model.

So, if we have something that one can locally run using its GPU, being
able to get an answer in the range of a couple of minutes per patch
should be enough to catch most of the issues.

Thanks,
Mauro