Re: QEMU Sashiko

["Alex Bennée" <alex.bennee@linaro.org>]

Alistair <alistair@alistair23.me> writes:

> Hey,
>
> Sashiko just merged initial support for QEMU [1].
>
> For those who don't know, Sashiko is a LLM powered review bot used for
> the Linux kernel. When people send patches to LKML Sashiko reviews the
> patches and responds to the mailing list with any issue that it finds.
>
> Like all LLM tools, it is both useful, but also verbose, prone to
> hallucinations and misses things (like humans do to though). The
> general consensus in the kernel community is that it's a useful and
> valuable tool.

Linaro has been experimenting with AI reviews as well with our
patchpilot tool. You can see an example of its output here:

  https://patchpilot.linaro.org/run/2749c6de82c1-2#stage=review  

> Sashiko is just a Rust application and prompts, it doesn't include a
> LLM and you can use a range of different LLM backends.
>
> Google sponsors the development and runs a Linux instance. So you can
> see the hosted Linux instance here: https://sashiko.dev/ and it uses
> Gemini for the LLM which Google pays for. It seems like there is also a
> GCC one starting up.

I don't suppose anyone is interested in sponsoring the token cost? The
main reason we haven't deployed it beyond our engineers is the cost
implications. I assume the reason Google can afford to run it is a
vested interest in the kernel and owning their own inference
infrastructure.

> The cool thing about Sashiko though is you can run it locally as well,
> so you can use it to test your patches before sending them to LKML, as
> long as you have a LLM backend you can use.
>
> So, now that Sashiko also supports QEMU there are two exciting use
> cases:
>
> 1. Developers can run it locally to review their patches before sending
> them to the list. This can catch issues before sending patches
>
> 2. Maintainers can run an instance to review patches. How you set this
> up depends on the maintainer and how many tokens you have. Pointing it
> at the list directly and running it for all patches might be too much,
> but some combination of manually triggering it for certain patches
> seems to work well. The Settings.toml file can be setup based on what
> you want to do.

Be warned if you enable with an expensive model like Claude a moderate
patch review can very quickly add up to $20-$50 of tokens. 

> The hope with this is to save maintainers time as Sashiko can catch
> issues without requiring much time from a maintainer.

It depends. Some times you end up reading quite a big wall of model
reasoning text and you end up none the wiser to what it actually found
wrong. On the other hand given engineers are expensive you don't need to
save much of their time to make the token cost worthwhile.

That said I personally feel one under explored area is using cheaper
models and a common code smells skill that could be useful in
identifying the obvious nits and mistakes that often come up in the
first pass review. That would leave maintainers free to spend more time
thinking about the actual logic and design of a patch.

If it was embedded in skills we could hopefully get contributors to run
reviews locally with their own agents before posting a series and
avoiding the normal RFC feedback of nits and fixes. It would be
effectively a slightly smarter checkpatch.

>
> QEMU support in Sashiko is very new, so it will take some tweaking to
> improve things, but it is at least a good starting place.
>
> 1:
> https://github.com/sashiko-dev/sashiko/commit/e432036bba7ab10d4517385916e6c834d8176f70
>
> Alistair

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro