Re: [PATCH-for-5.2? 1/1] Acceptance tests: bump Fedora to 32

["Daniel P. Berrangé" <berrange@redhat.com>]

On Thu, Dec 03, 2020 at 12:29:59PM -0500, Cleber Rosa wrote:
> On Thu, Dec 03, 2020 at 05:02:33PM +0000, Daniel P. Berrangé wrote:
> > I think the problem with the Fedora acceptance is that we'll be constantly
> > chasing a moving target. Every URL we pick will go away 6-12 months later.
> > IOW, while the acceptance test pass today, in 6 months time they'll be
> > failing.  IOW,  switching to F32 doesn't solve the root cause, it just
> > pushs the problem down the road for 6 months until F32 is EOL and hits
> > the same URL change problem.
> >
> 
> Just FIY, the tests will not FAIL when the images are removed from the
> official locations.  This is what happens Today:
> 
>    JOB ID     : e85527a9d75023070f15b833eac0f91f803afc83
>    JOB LOG    : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/job.log
>     (1/1) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm: CANCEL: Failed to download/prepare boot image (0.33 s)
>    RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1
>    JOB HTML   : /home/cleber/avocado/job-results/job-2020-12-03T12.21-e85527a/results.html
>    JOB TIME   : 0.76 s
> 
> And *normally*, we'd have 12+ months between updates, that is from
> Fedora 31 -> 33, 33 -> 35, etc.
> 
> > One way to avoid this is to *not* actually  test a current Fedora.
> > Instead intentionally point at an EOL Fedora release whose URL has
> > already moved to the archive site which is long term stable.
> >
> 
> So the tradeoff is, a patch every 6 or 12 months, versus using a more
> modern guest.  With other tests, such as virtiofs_submounts.py,
> already depending on the same decision (to avoid multiple guest images
> downloaded), I think this tradeoff decision needs more visibility.
> 
> IMO, the cost of such a simple patch every 6 or 12 months is very low
> provided we'll benefit from the newer guests.

I don't think changing the OS version typically changes the level of
coverage in aggregate.  The new OS may exercise new code paths, but
it will stop exercising old code paths, so most of the time it'll
be net-zero.  The ideal would be to test a representative selection
of both old and new versions but capacity limits.

The only time there's probably a notable difference is if we need to
access to a new type of device that the old OS doesn't have, but
that's relatively rare.



Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|