Re: QEMU 5.1: Can we require each new device/machine to provided a test?

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, May 15, 2020 at 12:11:17PM +0200, Thomas Huth wrote:
> On 07/04/2020 12.59, Philippe Mathieu-Daudé wrote:
> > Hello,
> > 
> > Following Markus thread on deprecating unmaintained (untested) code
> > (machines) [1] and the effort done to gather the information shared in
> > the replies [2], and the various acceptance tests added, is it
> > feasible to require for the next release that each new device/machine
> > is provided a test covering it?
> > 
> > If no, what is missing?
> 
> If a qtest is feasible, yes, I think we should require one for new
> devices. But what about machines - you normally need a test image for
> this. In that case, there is still the question where testing images
> could be hosted. Not every developer has a web space where they could
> put their test images onto. And what about images that contain non-free
> code?

Yep, it isn't feasible to make this a hard rule.

IMHO this is where a support tier classification comes into play

 - Tier 1: actively maintained, qtest coverage available. Expected
           to work reliably at all times since every commit is CI
	   tested

  - Tier 2: actively maintained, no qtest coverage. Should usually
           work but regression may creep in due to reliance on the
	   maintainer to manually test on adhoc basis

  - Tier 3: not actively maintained, unknown state but liable to
            be broken indefinitely

Tier 1 is obviously the most desirable state we would like everthing to
be at. Contributors will have to fix problems their patches cause as
they will be blocked by CI.

Tier 2 is an admission that reality gets in the way. Ideally stuff in
this tier will graduate to Tier 1 at some point. Even if it doesn't
though, it is still valid to keep it in QEMU long term. Contributors
shouldn't gratuitously break stuff in these board, but if they do,
then the maintainer is ultimately responsible for fixing it, as the
contributors don't have a test rig for it.

Tier 3 is abandonware. If a maintainer doesn't appear, users should
not expect it to continue to exist long term. Contributors are free
to send patches which break this, and are under no obligation to
fix problems in these boards. We may deprecate & delete it after a
while


Over time we'll likely add more criteria to stuff in Tier 1. This
could lead to some things dropping from Tier 1 to Tier 2. This is
OK, as it doesn't make those things worse than they already were.
We're just saying that Tier 2 isn't as thoroughly tested as we
would like it to be in an ideal world.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|