Re: AUTOSEL process

[Sasha Levin <sashal@kernel.org>]

On Sat, Mar 11, 2023 at 09:19:54PM +0100, Willy Tarreau wrote:
>On Sat, Mar 11, 2023 at 11:46:05AM -0800, Eric Biggers wrote:
>> (And please note, the key word here is *confidence*.  We all agree that it's
>> never possible to be absolutely 100% sure whether a commit is appropriate for
>> stable or not.  That's a red herring.
>
>In fact even developers themselves sometimes don't know, and even when they
>know, sometimes they know after committing it. Many times we've found that
>a bug was accidently resolved by a small change. Just for this it's important
>to support a post-merge analysis.
>> And I would assume, or at least hope, that the neural network thing being used
>> for AUTOSEL outputs a confidence rating and not just a yes/no answer.  If it
>> actually just outputs yes/no, well how is anyone supposed to know that and fix
>> that, given that it does not seem to be an open source project?)
>
>Honestly I don't know. I ran a few experiments with natural language
>processors such as GPT-3 on commit messages which contained human-readable
>instructions, and asking "what am I expected to do with these patches", and
>seeing the bot respond "you should backport them to this version, change
>this and that in that version, and preliminary take that patch". It
>summarized extremely well the instructions delivered by the developer,
>which is awesome, but was not able to provide any form of confidence
>level. I don't know what Sasha uses but wouldn't be surprised it shares
>some such mechanisms and that it might not always be easy to get such a
>confidence level. But I could be wrong.

It's actually pretty stupid: it uses the existence of ~10k of the most
common words in commit messages + metrics from cqmetrics
(github.com/dspinellis/cqmetrics) as input.

Although I get a score, which is already set pretty high, confidence is
really non-existant here: at the end it depends mostly on the writing
style of said commit author more than anything.

-- 
Thanks,
Sasha