Re: [PATCH 1/4] docs/devel: relax policy on AI-generated contributions

["Alex Bennée" <alex.bennee@linaro.org>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Oct 06, 2026 at 10:44:37AM +0100, Alex Bennée wrote:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> 
>> > Am 05.10.2026 um 23:09 hat BALATON Zoltan geschrieben:
>> >> On Fri, 2 Oct 2026, Paolo Bonzini wrote:
>> >> > I agree completely with how you phrased the first and third.  I agree
>> >> > that the second bullet is a risk too, but it's mitigated by "LLM-created
>> >> > contributions will be held to a higher standard than human-created ones,
>> >> > because LLMs make it easier to write tests" (and by maintainers being
>> >> > able to
>> >> 
>> >> Tests need to be reviewed too to be able to trust them so that does not help
>> >> against overloading reviewers with generated patches but burdens them even
>> >> more: they will also have to review the tests before the code.
>> >
>> > The required level of trust for tests is comparatively low, so it's much
>> > easier to review tests to a sufficient level than say device emulation
>> > code.
>> >
>> > If a new test looks plausible, fails before the fix and passes
>> > afterwards, it's more useful that what we had before, even if it's
>> > technically buggy in some corner case.
>> 
>> And also easier to rip out in the unlikely event it infringes anything.
>> 
>> But broadly having a test* is better than having no test.
>> 
>> * modulo the effects on our coverage.
>
> The maintability of test code is really important too, just as much
> as the code being tested. Every test potentially constrains how we
> can change code in future as tests themselves are never perfect or
> exhaustive and often take short-cuts in their design to suit the
> current testing needs. As we enhance the code under test, the test
> suite needs to evolve with it and that requires that we understand
> the test code and why it is doing what it is doing.

I don't disagree - and in an ideal world every core feature would have
an exhaustive test suite which exercised all the limits and exceptions
of the sub-system. Writing those good test suites is hard and certainly
not something I would be comfortable yolo'ing into the code base with an
LLM.

> A test spat out by AI which is a string of 20 undocumented qtest
> commands may fail before the fix and pass afterwards, but that
> test is not automatically a win if we don't understand why the test
> is doing what it is doing - it becomes a burden for long term maint.

However generally the qtest sequences we see are exercising an
individual device and are not dissimilar to the sequences we see come
out of the fuzzers. It would be nice if we had clear documentation for
how hardware was meant to behave but for some of our older device models
they are more or less written to handle what the guest expects in normal
operation.

And I'd certainly prefer a simple patch to a devices qtest test than an
elaborate test harness to boot qemu and send qtest commands by hand.

So in short - short tests exercising one device are fine to come out of
an LLM if the alternative is nothing. Testing frameworks to exercise
core sub-systems should remain the prerogative of the thinking engineer
unless the maintainer is particularly keen to review a models attempt to
fix it.

>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro