Re: [Qemu-devel] QEMU Summit 2017: minutes

[Cornelia Huck <cohuck@redhat.com>]

On Tue, 28 Nov 2017 09:33:52 +0100
Thomas Huth <thuth@redhat.com> wrote:

> On 27.11.2017 23:03, John Snow wrote:
> > 
> > On 11/23/2017 11:31 AM, Peter Maydell wrote:  
> [...]
> >> Continuous Integration:
> >>  * Christian Borntraeger: qemu-iotests have broken a lot, they should be
> >>    run before patches are merged  
> > 
> > This, rather unfortunately, is a huge testing burden. I try to make sure
> > I do it for everything I submit, but for the volume of block patches it
> > really does rely CI. The more we add (to our pitifully sparse iotesting,
> > I might add) the longer it takes. Ensuring per-patch testing begins to
> > take prohibitively long.
> > 
> > Perhaps per-pull or per-merge becomes more feasible. Maybe if we do
> > implement a block-next amalgam we'd be able to batch our testing on a
> > weekly basis.  
> 
> I think you block-layer folks should do at least run the qemu-iotests
> before sending a pull request to Peter. The iotests should really not be
> broken in upstream master.

This is unlikely to cover the iotest failures on s390 (due to usage of
ccw, strange backing devices, etc.), though. We have basically two
options here:
- Continue to rely on the IBM folks finding those problems (which will
  likely be post-merge, but better than nothing.)
- Have patchew (which has a bot on s390) execute the iotests - which is
  time-consuming.

> 
> >>  * Peter Maydell: If it isn't tested by "make check" then it isn't tested:
> >>    so if something is regularly regressing then it needs to be added to
> >>    "make check".  
> > 
> > Is this tenable long term? We can't conceivably state that we will never
> > test things that aren't in "make check" -- we ought to have different
> > tiers, at least. The full testing suite should run for RC tags at least,
> > but it's not feasible (I think?) to run the entire battery of tests on
> > every commit... but that shouldn't stop us from running them /sometimes/...  
> 
> We've already got "make check SPEED=slow" for running tests that take a
> lot of time. So maybe you could do that in the iotests as well, so that
> the normal, quick tests can be run during "make check" and the full
> iotest suite is only run during "make check SPEED=slow" ?

+1 to that. Having a subset covered by default is better than nothing
at all.