Re: [PATCH 1/4] docs/devel: introduce a new policy on AI-generated contributions

[Paolo Bonzini <pbonzini@redhat.com>]

On 9/2/26 14:01, Peter Maydell wrote:
> A contributor
> who was unaware of or chose to deliberately ignore our existing
> AI policy is just as likely to be unaware of or to ignore an updated
> policy that says "yes, but only with limitations and must be
> pre-arranged for larger amounts of generated code".

Oh, they won't be unaware of it. :)  That's why I included the 
AGENTS.md, even though it's technically separate.  They'll be reminded 
every two minutes of the need to pre-arrange review, until they tell the 
LLM to shut up.

> (A contributor who deliberately ignores the policy is I hope going to
> be rare; contributors who are simply unaware of it are something that
> we can hopefully improve on by (a) having an AGENTS.md that will tell
> their LLM about it and (b) asking in case of doubt on the maintainer
> side.)

Exactly.  Also, we have contributors who are unware of it now and we 
live with it or look at it on a case-by-case basis.  People ignoring the 
policy will be the same.

> I think fundamentally my issue here is that to me it doesn't seem
> like the project has problems with "we aren't writing code fast
> enough"; the problems are more "we don't have enough code review",
> "our existing code is getting holes poked in it by automated
> bug finding" and "we have too much old code that's barely used and
> unmaintained". So an AI policy update whose major move is "allow
> more generated code" feels bad to me, because it's inevitably
> going to increase the workload on reviewers and maintainers.
> I would be much happier seeing more use of AI in the project to
> help us where we're struggling.

I agree.  The hope is to get a bit of that via better testing, and that 
"we aren't writing code fast enough" is applied to stuff that helps long 
term rather than to the itch of the day.  Whether it works or not it 
remains to be seen.

I certainly don't want an explosion of new device models and 
unreviewable qtests.

> The question I had with the Rust policy was to what extent their
> "experimentation" is actually going to be cautious experimentation,
> and how much it turns out to be "this is a back door that lets the
> people who want to write and review LLM-generated code do that anyway
> as long as the people who don't want to can mostly ignore it".
> 
> If this is really experimental then we should have some clear
> boundaries and criteria for our experimentation, covering for
> instance
> (a) when do we assess the success or failure of the experiment?
> (b) what are we looking at to decide success/failure?
> (c) what are our boundaries for what kinds of change we're willing
>      to make under this experiment and what we are not?
> (d) what is our plan for rolling back or otherwise recovering if
>      the experiment seems to have failed?

I agree these are good questions.  They don't have obvious answers but 
as a start:

(a) a year was more or less the distance between the old policy and the 
time when we started noticing AI work on the mailing list, so probably 
another 8-12 months?  Such as from November's QEMU Summit to the one in 
Sep-Nov 2027?

(b) Having to abort the experiment in advance is the obvious failure 
mode.  Everything goes unnoticed is success.  I would say that pervasive 
discussions on boundaries would be pretty bad too.  Success/failure may 
also be split across categories, say by forbidding the bugfix category.

(c) no idea.  The concept of pre-arrangement leaves this to individual 
maintainers.  As Daniel points out "different rules for different 
subsystems" is not a good thing in principle, but in the end there is 
already a large subjective element in what maintainers are willing to 
review and even from whom.

(d) worst case we could simply revert the policy, at least in effects 
though probably not in language.

I'll add (e) what are our boundaries for AI patches covering multiple 
maintainers?  The extreme case is Alex's "make QOM parenting mandatory" 
series 
(https://lore.kernel.org/qemu-devel/20260718213652.37673-1-graf@amazon.com/), 
but it's actually not the hardest---apart from being done with AI, we 
had similarly large changes in the past such as the Meson switch and 
qdev_new() refactoring.  In those cases a pre-arrangement was needed 
anyway with the community, more than with a specific sets of maintainers.

> The Rust policy makes some attempts at some of these e.g. with its
> "circuit breaker" provision and the requirement that LLM changes are
> "non-critical" ones.

I didn't include a "circuit breaker" because someone would have to 
implement it, but also because I don't expect any maintainer to be so 
eager as to flood QEMU with AI-generated work.  And we're a much smaller 
community so I hope we would be able to sort it out among ourselves.

>> +LLM-assisted and LLM-created contributions
>> +''''''''''''''''''''''''''''''''''''''''''
>> +
>> +Use of generative AI tools for code contributions generally falls into
>> +four buckets:
>> +
>> +- "background" assistance
>> +- small LLM-assisted bugfixes
>> +- use of LLMs to help generating parts of a larger patch
> 
> I guess I'm generally OK with these (though I might add an "If in doubt
> about whether your use here is too extensive, ask" to the last one:
> "a parser" is potentially a pretty big thing to be delegating to the LLM,
> for instance, and might either be "mostly boilerplate" or to shade over
> into the "writing large parts of functional code" category).

Sure I can remove the parser example.  The initial one I had was 
"switching to a new API".  That is more representative of the intent it 
would basically put Alex's 137 patch series almost entirely under this 
section and I didn't want to do that.

I can report that the LLM was very reluctant to let me use this third 
category, and even less to let me do it without "AI-used-for".  Even for 
a one-line change to a "#define" in a 1000-line patch, it insisted that 
perhaps I should have included the trailer.  The AGENTS.md makes it very 
meticulous. :)

> I do note that even for "small bugfixes below 10 lines of code" the
> code review effort can still be pretty huge where it's touching something
> like a device model, where you have to go and find the right datasheet
> or spec and confirm whether the proposed change is really the right one
> or if it just fixes whatever the assert/crash was but in the wrong way.
> [...] Having a hundred "fix minor bug in old code" patches on the list that
> are unreviewed isn't a lot better than having a hundred issues in the
> bug tracker (indeed, it's arguably worse, since we have no tracking
> system for patches on the mailing list; at least the issues won't just
> get lost in the deluge...)

Yes, and especially when we have them submitted to "Odd fixes" areas the 
risk of maintainer DDOS is there.  But if people start submitting too 
much you *can* tell them according to the policy that you need them to 
develop e.g. a test suite.  Which yes, will *also* be more work to 
review for maintainers, but probably something that was sorely needed.

The hope is that by not placing AI-generated code in a "don't ask, don't 
tell" area we actually have tools to push back and reach an equilibrium 
that is better for the project and for the maintainers.  How it will 
work, it remains to be seen.

> I'm also more willing to allow leeway and to trust the judgement on
> LLM use for somebody who is already a regular contributor to the
> project (and so has some idea of how the codebase works and better
> ability to spot when generated code has gone off in the wrong
> direction), versus patches from somebody who hasn't contributed
> before. Is that something we want to try to encode in policy
> (e.g. with limitations on the "pre-arranged larger contribution"
> case) ?
Pre-arrangement does not mean you're forced to say yes.  Saying "I've 
never done this and I don't want to give you false hopes, so I'll 
decline" is fine.

Paolo