Re: Linking Patchwork with Sashiko?

["Theodore Tso" <tytso@mit.edu>]

On Thu, Jul 16, 2026 at 11:53:44AM -0500, Jai Luthra wrote:
> IMO false positives are possible everywhere, but LLMs are unpredictable in 
> a very different way compared to static-analysis tools *especially* because 
> they can be quite good at finding some issues and are non-deterministic.

Syzbot reports (whether it is of a lockdep or WARN spat) are also
non-determinstic.  Even when it claims to have a reproducer, the
reproducer is not neecessarily reliable.  I think the difference is
that we know what to expect with syzbot, and LLM reviews are new.

And as others have recently pointed out, there is a combination of
very deep subtle, real bugs that humans would not or have not
discovered --- and at the same time, hallucinations that aren't real.
So it's been hard to calibrate whether or not a particular report from
Sashiko is valid or not.  But this is nothing new, and I would claim
that it's not necessarily unique to LLM's.

> I can understand the appeal of using them to find security bugs or as extra 
> "eye balls to make all bugs shallow". And I am not advocating AI as bad or 
> to ban it. But I do think usage of LLMs through Sashiko may give us free 
> reviews, but require brand new work of reviewing the reviews. And this new 
> work will be a burden on experienced folks, not newcomers, who will simply 
> believe the LLM output because it is (or sounds) smarter than them.

Syzbot and oss-fuzz has been an extra burden on us as well.  I suspect
how much each tool impacts each subsystem varies, but I've had *way*
more syzbot false positives (e.g., reports that don't conform to our
threat models, such as reproducers using root privileges to modify the
loop device out from under a file system being reported as a "security
bug") than with Sashiko.  But it seems pretty clear that each
subsystem's experience has had different experiences with differnt
tools.

> My point isn't that static-analysis tools can't make the same mistake, but 
> that they're open source and fixable in a way that current architecture of 
> LLM models isn't in general, other than giving it an ever-increasing prompt 
> and hoping the random-number generator gods bless me when I need them to.

Sashiko is open source as well, and there are a number of ways of
fixing this, including adding annotations to the source code, etc.

Sure, figuring out prompts is tricky, but I'd argue so is making
changes to Clang for its context analysis.  Fortunately, there are
people who are smarter than me in terms of their Clang-fu who have
been improving Clang.  Similarly, Roman and his team are working on
improving Sashiko.  I don't think anyone has said that kernel
developers are responsible for changing the prompts (or changing
syzbot, Coverity, etc.)

Cheers,

						- Ted