Re: sysbot AI patches and wireless

[Johannes Berg <johannes@sipsolutions.net>]

Hi!

> 1. Every AI-generated patch sent by syzbot has been pre-reviewed and
> approved by a human engineer. The person who approved the patch is
> listed in the From: and Signed-off-by: fields.

Yes, I realize that. But the experience still seems to be one of me
effectively consuming pure LLM output, just via an intermediary. If I
had the time to do that, then I'd be perfectly able to talk to an LLM
directly, cutting out the intermediaries, and get the issues fixed that
way - much, much faster than writing everything in emails.

If, on the other hand, the intermediaries actually do the necessary
legwork etc. then I don't think this whole process is necessary; I don't
think using "b4" would be a significant hurdle in the process, and
that's really the only thing this helps with? Maybe LLM access (not
everyone has effectively free tokens), but that's still enabled by
syzbot providing the service over on the internal/upstream-moderation
list I guess.

Now, of course now that I say this (and you disabled it) I guess I'll
just see the patches pasted into an email manually instead, and I've
lost the signal that I could use to just ignore them entirely, but at
least I've been honest about it - and I guess I'll still silently drop
patches where the "human engineer" has no idea what they're doing. Even
having Reported-by: syzbot has been a signal of that to some extent.

I do think syzbot is a bit of a special case - at least if there's a
reproducer it's trivial to tell the system "hack the code until the
issue no longer reproduces" - but that's almost certainly guaranteed to
not be a useful patch yet.

The system, in a case like this, is almost certainly going to provide a
very narrow, targeted fix (with an annoying wall of text explaining
exactly that), but I think that at least the human in the loop should
actually take a step back from that and ask what the semantics of the
code should be ... I've played this game with Slawomir's first patch
myself, but that clearly cannot scale if the original intermediary
doesn't want to do that.

Maybe it's something you can even tell the LLM to do, somehow, so the
first draft is better. In this case, for example, why the hell did it
decide that it made any sense to have multiple branches of the same
switch statement - and there are even only two! - implement the same
validation? At the very least I'd expect the "human engineer" to take
that step back.

This is why I'm refusing these patches, because clearly nobody actually
even bothers to look at the semantics of the code before or during the
patching. Does pulling out the check outside of the switch change the
order of errors? Yes. Does that matter? No, the new order of errors for
NL80211_TDLS_ENABLE_LINK would actually - if you think about it (!) -
make a lot more sense! Am I surprised the LLM doesn't do that when you
tell it to make a targeted fix? Absolutely not.

But I really cannot make that judgement call myself for every single
issue like that, if I could, see above, I could be doing all of this
myself. Need the contributors to do that. Slawomir did that after I
prompted (pun intended!) him to do that, and it didn't work out so well
and we had a good discussion about it, but again, I can't provide that
support all time.

> 2. For the patches already sent to LKML, the bot *does not* send
> automatic replies and *does not* automatically submit newer versions.
> Once the patch is on LKML, follow-ups, comments, and iterations are
> handled entirely by the human developer who signed off on it, so you
> have not been interacting with an LLM.

Thanks for clarifying. I've definitely seen pure LLM replies (sometimes
even with the output saying things along the lines of "the reviewer said
this, I'll explain...") but that might have been in other contexts,
clearly it's not just syzbot which enables people doing things like
that.

> I apologize for the frustration this may have caused. We'll stop
> sending AI-assisted patches to the wireless subsystem.

Apology accepted, and thanks for disabling it.

Slawomir, Krystian, I assume you mean well and I apologise that you're
getting caught in the cross-fire here. Even the targeted fixes are still
fixes, so I understand from your perspective this still made sense, but
from mine it just absolutely cannot scale, both in terms of long-term
maintenance (sprinkling checks all over the code disregarding the
architecture) and also in terms of review bandwidth etc. I'm sorry
you're effectively the two first victims of this new process.

johannes