Auto-invalidating old syzbot reports?

Auto-invalidating old syzbot reports?

[Eric Biggers]

Currently the upstream Linux kernel has 888 open syzbot reports
(https://syzkaller.appspot.com/upstream).  However, nearly two-thirds of them
(577) were reported more than 1 year ago.  Old reports are often for bugs that
were already fixed.  They can also be reports that got overlooked, forgotten
about, not sent to the right place, etc.  Kernel maintainers also change over
time, so the current maintainer(s) might never have received the original report
even if syzbot sent the original report to the correct maintainer(s).

Having these old reports open is preventing syzbot from re-reporting any bugs
with the same crash signature (where a crash signature is something like
"KASAN: null-ptr-deref Read in percpu_ref_exit") if it is still being seen.

syzbot does auto-invalidate some old bugs, but only ones without a reproducer.

Given that humans aren't keeping up with these reports, has it been considered
to auto-invalidate all old syzbot reports -- not just ones without a reproducer?

- Eric

Re: Auto-invalidating old syzbot reports?

[Aleksandr Nogikh]

Hi Eric,

Thanks for contacting us!
These are very interesting points.

Syzbot indeed only closes old bugs without a reproducer, because if we
have a repro, then we can periodically do a fix bisection. And yes,
this mechanism unfortunately does not always work perfectly.

I think we could split the problem you described into two parts.
1) Some bugs that are "open" on the dashboard are actually no longer
relevant and should be closed.

If you know some old opened bugs with repro, which are actually
already fixed, could you please share them? It would be helpful to
figure out the exact reason why they are still open.
There are some cases when we can close bugs with a repro without
losing too much -- e.g. for bugs from -next there was a discussion in
https://github.com/google/syzkaller/issues/1957.
Also, if the fix bisection fails, but the repro no longer triggers the
crash on the HEAD, then we could probably "cancel" the repro and let
the bug be auto-closed (actually, auto-invalidated) later?

2) Some bugs were reported to the mailing lists, but became forgotten.

We could periodically take maintainers as per the latest commit and
send a reminder email to them. What do you think, would people go mad
if we did that for each bug e.g. every 6 months? :) Only if the bug
still happens on syzbot, of course.

At some point we were also considering sending aggregated reminders
(e.g. sth like "we still have X open bugs for the subsystem you
maintain/have actively contributed to, here they are:"), but to do
that, we first need to learn how to more or less reliably classify the
bugs into the subsystems.

--
Best Regards,
Aleksandr

On Tue, Jun 7, 2022 at 12:19 AM Eric Biggers <ebiggers@kernel.org> wrote:
>
> Currently the upstream Linux kernel has 888 open syzbot reports
> (https://syzkaller.appspot.com/upstream).  However, nearly two-thirds of them
> (577) were reported more than 1 year ago.  Old reports are often for bugs that
> were already fixed.  They can also be reports that got overlooked, forgotten
> about, not sent to the right place, etc.  Kernel maintainers also change over
> time, so the current maintainer(s) might never have received the original report
> even if syzbot sent the original report to the correct maintainer(s).
>
> Having these old reports open is preventing syzbot from re-reporting any bugs
> with the same crash signature (where a crash signature is something like
> "KASAN: null-ptr-deref Read in percpu_ref_exit") if it is still being seen.
>
> syzbot does auto-invalidate some old bugs, but only ones without a reproducer.
>
> Given that humans aren't keeping up with these reports, has it been considered
> to auto-invalidate all old syzbot reports -- not just ones without a reproducer?
>
> - Eric
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller/Yp59WCODvEDbpxOY%40sol.localdomain.

Re: Auto-invalidating old syzbot reports?

[Eric Biggers]

On Tue, Jun 07, 2022 at 04:37:42PM +0200, 'Aleksandr Nogikh' via syzkaller wrote:
> Hi Eric,
> 
> Thanks for contacting us!
> These are very interesting points.
> 
> Syzbot indeed only closes old bugs without a reproducer, because if we
> have a repro, then we can periodically do a fix bisection. And yes,
> this mechanism unfortunately does not always work perfectly.
> 
> I think we could split the problem you described into two parts.
> 1) Some bugs that are "open" on the dashboard are actually no longer
> relevant and should be closed.
> 
> If you know some old opened bugs with repro, which are actually
> already fixed, could you please share them? It would be helpful to
> figure out the exact reason why they are still open.
> There are some cases when we can close bugs with a repro without
> losing too much -- e.g. for bugs from -next there was a discussion in
> https://github.com/google/syzkaller/issues/1957.
> Also, if the fix bisection fails, but the repro no longer triggers the
> crash on the HEAD, then we could probably "cancel" the repro and let
> the bug be auto-closed (actually, auto-invalidated) later?
> 

Just to give the first definitive example I could find,
https://syzkaller.appspot.com/bug?id=06c43cd0a71aec08de8ea55ca5cda816c45ab4e0
("KMSAN: uninit-value in _mix_pool_bytes") is a 3-year old bug that is still
open even though it was fixed by commit f45a4248ea4c ("net: usb: rtl8150: set
random MAC address when set_ethernet_addr() fails").

From working on syzbot reports in the past, I can say that the "already fixed"
case for old reports is very common.  It is hard and time-consuming to actually
identify them as such though, since it requires root-causing the bug.  If it was
quick and easy to do so, there wouldn't be hundreds of such open reports...

> 2) Some bugs were reported to the mailing lists, but became forgotten.
> 
> We could periodically take maintainers as per the latest commit and
> send a reminder email to them. What do you think, would people go mad
> if we did that for each bug e.g. every 6 months? :) Only if the bug
> still happens on syzbot, of course.

That is greatly needed, but to get there we first need to get past the
assumption that every syzbot report will get properly handled by humans and thus
should never be automatically closed.  That assumption has been tried for the
last 5 years, and unfortunately it isn't working.  (If responding to syzbot
reports was being properly funded, it would be possible, but it's not.)  It
looks like you agree, as per your suggestion that only crashes that still happen
in syzbot should be reminded about.  I think syzbot actually needs to go further
and close the old bug reports, not merely suppress reminders about them...

In any case, reminders *must* include the latest crash details in a way that
clearly shows that the bug is still relevant.

> At some point we were also considering sending aggregated reminders
> (e.g. sth like "we still have X open bugs for the subsystem you
> maintain/have actively contributed to, here they are:"), but to do
> that, we first need to learn how to more or less reliably classify the
> bugs into the subsystems.

Well syzbot already identifies subsystems via the stack trace; it's just not as
good as a human expert, and probably never will be since the correct subsystem
can be very non-obvious.  For a short time, I was actually manually classifying
the subsystems for syzbot reports and sending out reminders --- see
https://lore.kernel.org/linux-kernel/?q=%22open+syzbot+bugs%22 --- but I gave up
due to lack of support from my job for doing that work, combined with receiving
somewhat less engagement than I had hoped.

Perhaps the best solution would be to crowdsource by providing a self-service
"#syz subsystem $FOO" command.

- Eric

Re: Auto-invalidating old syzbot reports?

[Aleksandr Nogikh]

On Wed, Jun 8, 2022 at 2:41 AM Eric Biggers <ebiggers@kernel.org> wrote:
>
> On Tue, Jun 07, 2022 at 04:37:42PM +0200, 'Aleksandr Nogikh' via syzkaller wrote:
> > Hi Eric,
> >
> > Thanks for contacting us!
> > These are very interesting points.
> >
> > Syzbot indeed only closes old bugs without a reproducer, because if we
> > have a repro, then we can periodically do a fix bisection. And yes,
> > this mechanism unfortunately does not always work perfectly.
> >
> > I think we could split the problem you described into two parts.
> > 1) Some bugs that are "open" on the dashboard are actually no longer
> > relevant and should be closed.
> >
> > If you know some old opened bugs with repro, which are actually
> > already fixed, could you please share them? It would be helpful to
> > figure out the exact reason why they are still open.
> > There are some cases when we can close bugs with a repro without
> > losing too much -- e.g. for bugs from -next there was a discussion in
> > https://github.com/google/syzkaller/issues/1957.
> > Also, if the fix bisection fails, but the repro no longer triggers the
> > crash on the HEAD, then we could probably "cancel" the repro and let
> > the bug be auto-closed (actually, auto-invalidated) later?
> >
>
> Just to give the first definitive example I could find,
> https://syzkaller.appspot.com/bug?id=06c43cd0a71aec08de8ea55ca5cda816c45ab4e0
> ("KMSAN: uninit-value in _mix_pool_bytes") is a 3-year old bug that is still
> open even though it was fixed by commit f45a4248ea4c ("net: usb: rtl8150: set
> random MAC address when set_ethernet_addr() fails").

Thanks for providing the example!
Yes, KMSAN bugs are unfortunately a bit special at the moment - we
cannot do a proper fix bisection because the KMSAN repo is regularly
rebased. Correct Reported-by tag and manual fix reporting (#syz fix)
are right now the only ways to get KMSAN bugs closed.

For this particular bug the problem was that the fixing commit
referenced (Reported-by:
syzbot+abbc768b560c84d92fd3@syzkaller.appspotmail.com) another bug
(https://syzkaller.appspot.com/bug?extid=abbc768b560c84d92fd3).

>
> From working on syzbot reports in the past, I can say that the "already fixed"
> case for old reports is very common.  It is hard and time-consuming to actually
> identify them as such though, since it requires root-causing the bug.  If it was
> quick and easy to do so, there wouldn't be hundreds of such open reports...
>
> > 2) Some bugs were reported to the mailing lists, but became forgotten.
> >
> > We could periodically take maintainers as per the latest commit and
> > send a reminder email to them. What do you think, would people go mad
> > if we did that for each bug e.g. every 6 months? :) Only if the bug
> > still happens on syzbot, of course.
>
> That is greatly needed, but to get there we first need to get past the
> assumption that every syzbot report will get properly handled by humans and thus
> should never be automatically closed.  That assumption has been tried for the
> last 5 years, and unfortunately it isn't working.  (If responding to syzbot
> reports was being properly funded, it would be possible, but it's not.)  It
> looks like you agree, as per your suggestion that only crashes that still happen
> in syzbot should be reminded about.  I think syzbot actually needs to go further
> and close the old bug reports, not merely suppress reminders about them...

I've filed an issue about making syzbot periodically re-test the
reproducers. If the repro doesn't trigger a bug anymore, syzkaller
will pretend there's no repro and close the bug once crashes are not
happening anymore. This should hopefully resolve the problem with old
irrelevant bugs.
https://github.com/google/syzkaller/issues/3193

>
> In any case, reminders *must* include the latest crash details in a way that
> clearly shows that the bug is still relevant.

That's a good point, thanks!

>
> > At some point we were also considering sending aggregated reminders
> > (e.g. sth like "we still have X open bugs for the subsystem you
> > maintain/have actively contributed to, here they are:"), but to do
> > that, we first need to learn how to more or less reliably classify the
> > bugs into the subsystems.
>
> Well syzbot already identifies subsystems via the stack trace; it's just not as
> good as a human expert, and probably never will be since the correct subsystem
> can be very non-obvious.  For a short time, I was actually manually classifying
> the subsystems for syzbot reports and sending out reminders --- see
> https://lore.kernel.org/linux-kernel/?q=%22open+syzbot+bugs%22 --- but I gave up
> due to lack of support from my job for doing that work, combined with receiving
> somewhat less engagement than I had hoped.

Hmm, yes, maybe we can indeed give even the existing subsystem
detection mechanism a try.
And it looks like it's better to only send grouped reminders -- given
how much negative reaction one can see under your link(s), I wonder
what would there be if we send per-bug reminders :)

Hopefully I'll be able to get to the implementation in some foreseeable future.

>
> Perhaps the best solution would be to crowdsource by providing a self-service
> "#syz subsystem $FOO" command.
>
> - Eric