Re: addressing sashiko's false positive rate

["Lorenzo Stoakes (ARM)" <ljs@kernel.org>]

Hi HyeongJun,

Thanks for doing this! :)

On Fri, Oct 02, 2026 at 05:56:28PM +0900, HyeongJun An wrote:
> On Tue, Sep 29, 2026, Lorenzo Stoakes (ARM) wrote:
> > 3. You could even get a weak signal from detecting whether code at the report
> >    site was altered or not, that wouldn't require voluntary input.
>
> On Tue, Sep 29, 2026, Roman Gushchin wrote:
> > Agree.
>
> I tried this on three lists to see how weak the signal is.
>
> I took Sashiko reviews from 2026-06-15 to 2026-08-31 on linux-input and
> bpf, from the review emails, and on netdev, from the sashiko.dev API.
> For each finding not marked pre-existing, I found the next version of
> the same patch on the list. If the code the finding points at changed
> there, I counted it as acted on. If it is unchanged, as ignored. Then an
> LLM checked 50 random findings per list against the next version.
>
>                                  input    bpf      netdev
>   findings judged                501      578      1899
>   acted on, LLM check            21/35    13/31    12/37
>   automatic label correct        27/35    29/31    30/37
>   LLM check undecided            15/50    19/50    13/50

So that's 30-60% hit rate which broadly matches my experience.

My data spanned 3 subsystems, so this brings that up to 6 - this is feeling
more and more like a verified measure.

Though of course it's relatively rough, that feels pretty confirmatory!

I think that does at this point pretty well rule out mm being different as
an explanation.

>
> This measures whether the author changed the code, not whether the
> finding was right, so it does not compare directly with Lorenzo's
> useful/wrong split. It also leaves out pre-existing findings and does
> not deduplicate across versions.

Ack yeah - that's all we can hope for with this really, as you only observe
the code changing, not why or if related, or duplicated etc.

These are inaccurate in that you don't know if ignored or unrelated change
or etc. etc. and mine are inaccurate in that my judgement might be wrong
also.

But the numbers are inaccurate in different directions and both seem to be
mutually confirmatory that the false positive rate is a lot more than 20%.

What's useful here is that this can be used to improve this in the verifier
- I've found that a bounded measure like this is the best way to get an LLM
to improve things and usually fairly rapidly too!

>
> The automatic label matched the LLM check 77% to 94% of the time. It
> mostly misses fixes that only reorder lines or that change code away
> from the snippet the finding quotes, so "unchanged" over-counts ignored
> findings. As a weak signal for picking candidates it works. As a
> verdict on its own it does not.

Yeah of course as above that's all we could expect.

>
> Lorenzo's point about repeats shows up here too. A finding that Make
> drops the space in $(if $(3), $(3)) was raised on v6, v7 and v8 of the
> selftests/bpf Makefile series. Running make shows the space is kept.
> https://lore.kernel.org/all/20260921075855.2065871-4-nickolay.lysenko@gmail.com/

Yeah duplication is very much an issue.

It also means repeats inflate your 'ignored' count so de-duping would
likely move your 'acted on' number up a bit.

>
> Some authors already mark findings as rejected themselves, in their
> changelogs. A bpf v3 changelog has a "To sashiko:" section, and a quic
> changelog starts with "Sashiko, do not report any of the following
> concerns". That text could feed the "rejected" marking Lorenzo asked
> for without anyone being asked to label.
> https://lore.kernel.org/all/20260808140720.293604-2-jiayuan.chen@linux.dev/
> https://lore.kernel.org/all/375ee7fe827d9df248a27a8072c84a7c167fc50a.1789393775.git.lucien.xin@gmail.com/

This is really useful - it's a free data source that's more accurate and
helps address the cross-version stuff Roman mentioned is currently not a
signal, and it's a sign there's already data out there to use.

(I've previously even replied to private sashiko reports - though cc'd to
sashiko-reviews I think - for instance, I'm sure others do similar, as well
as general replies on-list saying whether sashiko right/not).

Consuming these and treating as higher confidence could be very useful!

>
> On pre-existing issues, my data predates the split you just landed. In
> the 907 netdev patchsets I fetched, the preexisting flag was set on 38%
> of Critical and 37% of High findings, against 4% of Medium and none of
> Low. So separating them should remove a large share of the severe
> noise.

Yeah should be interesting to compare the numbers before/after!

>
> I would like to work on this in Sashiko, in small steps:
>
> 1. A script under scripts/ that reports this signal per subsystem from
>    Sashiko's own data, run by hand and never in CI.
> 2. Collecting findings that came back unchanged on a later version, or
>    that an author rebutted in a changelog, as candidate negative cases
>    for the benchmark. A person confirms each one before it is added,
>    which is one way to answer #426.

Obviously up to Roman and the team but sounds good to me :)

For 2. I included .csv reports for sashiko's 112 findings in my original
mail which could be used as seed data?

>
> Would that be welcome, or would you rather fold it into the continuous
> analysis you mentioned? Either way the scripts and the verdicts, each
> with a one-line reason, are ready to share. I used an LLM to help write
> the scripts.

I also think it could be helpful to have an LLM consume the retrieved data
and see how it can use it to improve the verifier.

Maybe have it do some A/B stuff too and churn away at it modifying the
verifier until it gets better results?

It's important to avoid overfitting to the data though - so you'd want to
use different subsets of scored data in each candidate dataset to avoid
that.

I was thinking of trying something like that myself but I have negative
available time for new stuff.

But could play with it on a weekend here or there or something... :)

>
> Thanks,
> HyeongJun

--
Cheers, Lorenzo