Re: on ai generated and code provenance

[Kevin Wolf <kwolf@redhat.com>]

Am 27.05.2026 um 12:01 hat Paolo Bonzini geschrieben:
> On 5/27/26 10:41, Kevin Wolf wrote:
> > Am 26.05.2026 um 21:52 hat Warner Losh geschrieben:
> > > The QEMU Project currently may accept limited uses of AI that produce
> > > high quality patches that are limited in the creative content added.
> > > While maintainers will ultimately decide, changes like the following
> > > fall within this policy
> > > 1. Fixing obvious warnings in the obvious ways suggested by the tool
> > > 2. Tree wide API changes, and other similar mechanical changes done
> > >     today with perl/python/sed/coccinelle
> > 
> > As I said in the paragraph you quoted below, I don't think we should
> > encourage using AI for tasks that a deterministic tool could do.
> 
> In some cases such a tool does not exist.

Then it's not a task that a deterministic tool could do.

Of course, you can always write a new tool that does the exact thing you
want to change. But that's not what I was talking about here, I was
really talking about existing common tools.

> Much to my surprise, there is no tool to do static type inference on
> Python code, but AI is very good at doing it.

I think this is a special case that has a different balance anyway. When
reviewing such a patch, I would skim the change for the general approach
and if I like it, but checking for consistency and completeness is
something I would use mypy for - that is, a deterministic tool that can
verify the change. So I'd still use one, just at a different time.

(It actually also might be a rare instance where someone (TM) should
actually write the tool because it would be generally useful.)

> > Letting AI perform the change directly instead may be an acceptable
> > shortcut for a one-man hobby project that nobody else will ever look at,
> > but in the context of a community project like QEMU in which your
> > changes have to be reviewed and understood by others, it matters a lot
> > that the output of the tool is reproducible. Otherwise, you're creating
> > unnecessary work for others, and that isn't acceptable.
> 
> When applicable, going through coccinelle (with the aid of AI if needed! is
> indeed a good middle ground as it helps reviewers for large changes. If you
> have many slightly different but easily separated changes (e.g. you can
> split the patch by struct field), it may make things worse.
> 
> Its also worth noting that in other cases even sed or coccinelle, while
> deterministic, cannot produce 100% of the patch.

Agreed, it's all a case of "if possible, prefer this", not "you have to
do this 100% of the time".

> > So maybe we should even explicitly mention a recommendation like the
> > following:
> > 
> >      If you can use a deterministic tool, don't use AI instead. If you
> >      don't know how to use the deterministic tool, use the AI to tell you
> >      how to use it instead of trying to replace it.
> 
> I like it.
> 
> > > 3. Limited, small changes to fix bugs or add a small new feature whose
> > >     scope is less than about 100 lines and the originator can explain
> > >     them all or the meta issues about the patch.
> > 
> > Not sure if mentioning a number of lines is wise. 100 lines can be
> > mostly boilerplate and simple sequential code or they can be a deeply
> > nested complex algorithm.
> 
> I'd put the threshold at 20-50 at most.
> 
> > I think I would see more use in a tag like (better name welcome):
> > 
> >     AI-used-for: [code|tests|docs|commit message]...
> 
> I like this *a lot*.  No need for free advertisement, but some traceability
> is useful.
> 
> For tools such as sed or coccinelle, having the exact script in the patch or
> commit message useful.  Plus, the execution of the script more or lesss
> delimits the commit by itself (or 90%+ of it).  For LLMs it's a bit less
> clear cut because separating docs makes little sense.  And the exact model
> is pointless, it will be obsolete in 6 months and provide no useful
> information.
> 
> So, something like:
> 
> ------------------- 8< -------------------
> Use of AI-generated content
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> The QEMU project currently allows using AI/LLM tools to produce patches in
> scenarios with limited creative content:
> 
> Mechanical changes
>   If you can use a deterministic tool or a script, don't use AI instead.
>   If you don't know how to do the change deterministically, you may
>   ask the AI for help, rather than having it stand in for the tools.
> 
> Small bug fixes
>   These should be limited to 20 lines of code or less, not including
>   tests.  You are still expected to understand and explain your changes
>   and the rationale behind them.

I agree with "not including tests". But I think this would be more
consistent if we also add new tests (that come without a small bug fix
at the same time; either because the problem is already fixed or because
the fix is too complex to qualify) as another allowed category.

(To be honest, I'm a bit biased here because allowing tests is my single
biggest wish from an AI policy update.)

> These boundaries do not apply to other uses of AI, such as researching
> APIs or algorithms, static analysis, or debugging, provided their output
> is not included in contributions.  Larger uses of AI are allowed as an
> experiment, but they should be agreed upon with the maintainer prior to
> submission.
> 
> Use of AI does not remove the need for authors to comply with all other
> requirements for contribution.  In particular, the "Signed-off-by"
> label in a patch submission is a statement that the author takes
> responsibility for the entire contents of the patch, certifying that
> their patch submission is made in accordance with the rules of the
> `Developer's Certificate of Origin (DCO) <dco>`.
> 
> Commit messages for AI-assisted changes
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> 
> When AI/LLM tools produce or substantively shape your patch, add an
> ``AI-used-for:`` trailer.  The text of the trailer could be one or more of
> ``code``, ``tests``, ``docs``, ``research``, possibly followed by an
> explanation in parentheses::

Include a category for commit messages, or are we expecting that commit
messages are always written by a human? If so, that should be explicit.

>     AI-used-for: tests, docs
>     AI-used-for: code
>     AI-used-for: code (refactoring)
>     AI-used-for: code (prototype)
>     AI-used-for: research
> 
> The trailer is intended as a clarification of your DCO obligations as well
> as to guide reviewers.  It is not intended for minimal presence such as
> autocomplete or asking for a pre-review of the patch, and it does not remove
> your responsibility to understand the changes that you are submitting.
> 
> Include the prompt in the commit message if it helps a reviewer judge the
> result:
> 
> * yes: "move field ``foo`` from ``struct aa`` to ``struct bb``.  If a
> function already has a local variable or parameter of type ``struct bb``,
> use it instead of accessing ``aa.bb``."
> 
> * yes: "add an implementation of the trait for ``Mutex<T: MyTrait>``,
> forwarding the member functions to ``T`` while taking the lock around the
> calls".
> 
> * no: "write user-facing documentation for the new tool"
> 
> * no: "write testcases for the new functions"
> 
> Deterministic tooling (sed, coccinelle, formatters) is out of scope for the
> trailer, but should be mentioned in the commit message.

Apart from the above comments, this looks good to me.

Kevin