Re: Stop false review statements

[Roman Gushchin <roman.gushchin@linux.dev>]

> On May 17, 2026, at 11:56 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> On Sun, 17 May 2026 11:17:06 -0700
> Roman Gushchin <roman.gushchin@linux.dev> wrote:
> 
>>> On May 17, 2026, at 9:40 AM, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>>> On Sun, 17 May 2026 12:12:00 +0200
>>> Greg KH <gregkh@linuxfoundation.org> wrote:
>>>>> On Sun, May 17, 2026 at 12:05:56PM +0200, Mauro Carvalho Chehab wrote:
>>>>> On Sat, 16 May 2026 14:59:44 -0700
>>>>> Roman Gushchin <roman.gushchin@linux.dev> wrote:
>>>>>>> On May 16, 2026, at 2:33 PM, Krzysztof Kozlowski <krzk@kernel.org> wrote:
>>>>>>> I find it opposite: clogging commits with useless information, because
>>>>>>> some arbitrary and completely closed-source tool did analysis means
>>>>>>> nothing to me one year later when I look at the commit in the Git history.      
>>>>>> This is simple not true: Sashiko is fully open-source, under Apache 2.0 license
>>>>>> and the code belongs to LF.     
>>>>>> Yes, the instance behind sashiko.dev is using
>>>>>> Gemini 3.1 Pro LLM, which is not open-source, but it’s not a fundamental limitation -
>>>>>> Sashiko is supporting various LLMs, including open models - it’s just a practical
>>>>>> choice: to my knowledge the quality of open models is not on par with frontier closed
>>>>>> models     
>>>>> I would very much prefer using an open source LLM, even if not in pair
>>>>> with latest paid models.
>>>>>> and it would require a non-trivial amount of hardware and infrastructure to run
>>>>>> an open model at the required scale.    
>>>>> IMHO the best would be to have them running on some infra that would accept
>>>>> open source models (*). If there aren't enough resources to have our own
>>>>> infra, there are offers out there which allows running open source models
>>>>> like https://ollama.com/pricing (I never used myself).
>>>>> (*) For instance, Qwen3.6 is brand new and licensed under apache-2.0.
>>>>>   Not bad on my tests running it locally.    
>>>> You can run the tool locally, with whatever model you want, if you want
>>>> to.
>>>> But for now, let's just take the free credits that Google is willing to
>>>> throw at this thing and let it give us reviews IF the maintainer of the
>>>> subsystem feels it is something they want to do.  No one is forcing
>>>> maintainers to do this.  
>>> If Google and/or others are willing to give free credits on their cloud,
>>> they could instead or in addition give free credits to run ollama
>>> there, allowing us to use different models.
>>> From my side, while I won't personally object getting reviews from
>>> Sashiko/Gemini, this is something I can't reproduce locally. I would
>>> very much want something where I can select my LLM preferred model
>>> and run on my ollama docker container on my own GPU, in a way that
>>> I could run it locally before even sending a patch series.  
>> 
>> 2 thoughts here:
>> 1) I actually tried to run it with ollama on my personal framework 13. Adding nominal support is trivial,
>> but the whole thing is not really useful: I can get maybe few hundreds tokens per second using
>> a quantified model with reduced quality; an average sashiko review is consuming 3.5 millions tokens
>> (with Gemini 3.1 pro, it’s also model-dependent).
> 
> Do you mean 3.5 millions tokens per patch series? If so, that
> sounds a lot! Why does it require too many tokens?

It’s an average per patch, not a series. Some are much cheaper, some are much more expensive.
Sashiko posts token cost nearby each review.

Why it uses many tokens? Because in many cases it has to dig deep into the code.
Long sessions with multiple tool calls are expensive. Also Sashiko has a multi-stage
architecture, effectively it reviews every patch multiple times from different angles.
It has a measurable influence on the quality of reviews. The current generation of LLMs
is not good at spotting various types of issues at once: once it sees a memory leak
it can’t think anymore on e.g. locking issues. Also just by running the same thing multiple times
and combining the result you can meaningfully improve the quality.

>> I’m personally all in on having the entire thing as open as possible and I believe Sashiko is what
>> is realistically the best at this moment - a fully open-source harness and set of prompts which
>> can work with a variety of models.
>> I’m happy to merge a support for any LLM model which can produce decent review results.
>> 
>> 2) Due to probabilistic nature of LLMs, nothing is reproducible in a strict sense of the word.
>> Even with exactly the same model/harness/prompts you’ll get different results every time you run it.
>> It’s unfortunate, but it is what it is at the moment.
> 
> By "reproduce locally", I didn't mean in strict sense. Sure, LLM answers
> won't be identical, but I suspect that at least most of the major issues
> on a patch series would be reported by any decent model.

I believe we’re not quite there yet. Models do differ in their abilities to spot
various types of bugs and also producing false positives. Some types of issues
(e.g. complex locking issues) are really hard for best of the current models.

> So, if we have something that one can locally run using its GPU, being
> able to get an answer in the range of a couple of minutes per patch
> should be enough to catch most of the issues.

I’m happy to be wrong here, but my understanding is that it’s not realistic now.
Sashiko reviews taking longer with production grade hardware.