* [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 @ 2021-01-27 1:09 Daniele Buono 2021-01-27 1:09 ` [PATCH 1/1] " Daniele Buono 2021-01-28 20:19 ` [PATCH 0/1] " Wainer dos Santos Moschetta 0 siblings, 2 replies; 6+ messages in thread From: Daniele Buono @ 2021-01-27 1:09 UTC (permalink / raw) To: qemu-devel Cc: Philippe Mathieu-Daudé, Wainer dos Santos Moschetta, dbuono, Cleber Rosa Local acceptance tests run with "make check-acceptance" are now showing some cases canceled like the following: (01/39) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: CANCEL: Failed to download/prepare boot image (0.25 s) Turns out, every full-vm test in boot_linux.py is trying to use a Fedora 31 cloud image and is failing, with Avocado refusing to download it, presumably because Fedora 31 is EOL. This patch moves to Fedora 32, which is still supported. And seem to work fine The script has the image checksums hardcoded. I downloaded and verified the checksums with the Fedora 32 GPG key, but I would feel more confident if someone else wants to verify it too. The checksum files are here: https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/ppc64le/images/Fedora-Cloud-32-1.6-ppc64le-CHECKSUM https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/s390x/images/Fedora-Cloud-32-1.6-s390x-CHECKSUM https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/aarch64/images/Fedora-Cloud-32-1.6-aarch64-CHECKSUM https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-32-1.6-x86_64-CHECKSUM and the GPG keys are available here: https://getfedora.org/en/security/ NOTE: I tried moving to Fedora 33, but the aarch64 VM cannot boot properly. May be worth investigating but I have no experience with ARM so I'll leave that to someone else, if interested. Daniele Buono (1): tests/acceptance/boot_linux: Switch to Fedora 32 tests/acceptance/boot_linux.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) -- 2.30.0 ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/1] tests/acceptance/boot_linux: Switch to Fedora 32 2021-01-27 1:09 [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 Daniele Buono @ 2021-01-27 1:09 ` Daniele Buono 2021-01-28 20:19 ` [PATCH 0/1] " Wainer dos Santos Moschetta 1 sibling, 0 replies; 6+ messages in thread From: Daniele Buono @ 2021-01-27 1:09 UTC (permalink / raw) To: qemu-devel Cc: Philippe Mathieu-Daudé, Wainer dos Santos Moschetta, dbuono, Cleber Rosa Some acceptance test require a full VM image. The tests were using Fedora 31, which is now EOL. Probably as a consequence, the Avocado framework is not able to download such images anymore. Switch to Fedora 32, which is still supported. Signed-off-by: Daniele Buono <dbuono@linux.vnet.ibm.com> --- tests/acceptance/boot_linux.py | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/tests/acceptance/boot_linux.py b/tests/acceptance/boot_linux.py index 1da4a53d6a..b954f95f0b 100644 --- a/tests/acceptance/boot_linux.py +++ b/tests/acceptance/boot_linux.py @@ -48,7 +48,7 @@ class BootLinuxBase(Test): image_arch = 'ppc64le' try: boot = vmimage.get( - 'fedora', arch=image_arch, version='31', + 'fedora', arch=image_arch, version='32', checksum=self.chksum, algorithm='sha256', cache_dir=self.cache_dirs[0], @@ -111,7 +111,7 @@ class BootLinuxX8664(BootLinux): :avocado: tags=arch:x86_64 """ - chksum = 'e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0' + chksum = '423a4ce32fa32c50c11e3d3ff392db97a762533b81bef9d00599de518a7469c8' def test_pc_i440fx_tcg(self): """ @@ -160,8 +160,7 @@ class BootLinuxAarch64(BootLinux): :avocado: tags=machine:virt :avocado: tags=machine:gic-version=2 """ - - chksum = '1e18d9c0cf734940c4b5d5ec592facaed2af0ad0329383d5639c997fdf16fe49' + chksum = 'b367755c664a2d7a26955bbfff985855adfa2ca15e908baf15b4b176d68d3967' def add_common_args(self): self.vm.add_args('-bios', @@ -217,7 +216,7 @@ class BootLinuxPPC64(BootLinux): :avocado: tags=arch:ppc64 """ - chksum = '7c3528b85a3df4b2306e892199a9e1e43f991c506f2cc390dc4efa2026ad2f58' + chksum = 'dd989a078d641713c55720ba3e4320b204ade6954e2bfe4570c8058dc36e2e5d' def test_pseries_tcg(self): """ @@ -235,7 +234,7 @@ class BootLinuxS390X(BootLinux): :avocado: tags=arch:s390x """ - chksum = '4caaab5a434fd4d1079149a072fdc7891e354f834d355069ca982fdcaf5a122d' + chksum = '93e49b98fa016797a6864a95cbb7beaec86ffd61dbcd42a951158ae732dae1ec' @skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab') def test_s390_ccw_virtio_tcg(self): -- 2.30.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 2021-01-27 1:09 [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 Daniele Buono 2021-01-27 1:09 ` [PATCH 1/1] " Daniele Buono @ 2021-01-28 20:19 ` Wainer dos Santos Moschetta 2021-01-28 22:06 ` Daniele Buono 1 sibling, 1 reply; 6+ messages in thread From: Wainer dos Santos Moschetta @ 2021-01-28 20:19 UTC (permalink / raw) To: Daniele Buono, qemu-devel Cc: Daniel P. Berrangé, Philippe Mathieu-Daudé, Cleber Rosa Hi, On 1/26/21 10:09 PM, Daniele Buono wrote: > Local acceptance tests run with "make check-acceptance" are now > showing some cases canceled like the following: > > (01/39) tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: CANCEL: Failed to download/prepare boot image (0.25 s) > > Turns out, every full-vm test in boot_linux.py is trying to use a > Fedora 31 cloud image and is failing, with Avocado refusing to download > it, presumably because Fedora 31 is EOL. > > This patch moves to Fedora 32, which is still supported. And seem to > work fine While ago it was discussed about updating the Fedora version which, in my opinion, ended up without a conclusion. Please see the complete thread in: https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the discussion. Thanks! - Wainer > > The script has the image checksums hardcoded. I downloaded and verified > the checksums with the Fedora 32 GPG key, but I would feel more > confident if someone else wants to verify it too. > > The checksum files are here: > https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/ppc64le/images/Fedora-Cloud-32-1.6-ppc64le-CHECKSUM > https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/s390x/images/Fedora-Cloud-32-1.6-s390x-CHECKSUM > https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/aarch64/images/Fedora-Cloud-32-1.6-aarch64-CHECKSUM > https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-32-1.6-x86_64-CHECKSUM > and the GPG keys are available here: > https://getfedora.org/en/security/ > > NOTE: I tried moving to Fedora 33, but the aarch64 VM cannot boot > properly. May be worth investigating but I have no experience with ARM > so I'll leave that to someone else, if interested. > > Daniele Buono (1): > tests/acceptance/boot_linux: Switch to Fedora 32 > > tests/acceptance/boot_linux.py | 11 +++++------ > 1 file changed, 5 insertions(+), 6 deletions(-) > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 2021-01-28 20:19 ` [PATCH 0/1] " Wainer dos Santos Moschetta @ 2021-01-28 22:06 ` Daniele Buono 2021-02-05 16:54 ` Philippe Mathieu-Daudé 0 siblings, 1 reply; 6+ messages in thread From: Daniele Buono @ 2021-01-28 22:06 UTC (permalink / raw) To: Wainer dos Santos Moschetta, qemu-devel Cc: Daniel P. Berrangé, Philippe Mathieu-Daudé, Cleber Rosa On 1/28/2021 3:19 PM, Wainer dos Santos Moschetta wrote: > Hi, > > On 1/26/21 10:09 PM, Daniele Buono wrote: >> Local acceptance tests run with "make check-acceptance" are now >> showing some cases canceled like the following: >> >> (01/39) >> tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: >> CANCEL: Failed to download/prepare boot image (0.25 s) >> >> Turns out, every full-vm test in boot_linux.py is trying to use a >> Fedora 31 cloud image and is failing, with Avocado refusing to download >> it, presumably because Fedora 31 is EOL. >> >> This patch moves to Fedora 32, which is still supported. And seem to >> work fine > > While ago it was discussed about updating the Fedora version which, in > my opinion, ended up without a conclusion. Please see the complete > thread in: > > https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html Oops, didn't notice the previous thread. Apologies for the duplicate! > > I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the > discussion. > > Thanks! > > - Wainer > >> >> The script has the image checksums hardcoded. I downloaded and verified >> the checksums with the Fedora 32 GPG key, but I would feel more >> confident if someone else wants to verify it too. >> >> The checksum files are here: >> https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/ppc64le/images/Fedora-Cloud-32-1.6-ppc64le-CHECKSUM >> >> https://download-ib01.fedoraproject.org/pub/fedora-secondary/releases/32/Cloud/s390x/images/Fedora-Cloud-32-1.6-s390x-CHECKSUM >> >> https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/aarch64/images/Fedora-Cloud-32-1.6-aarch64-CHECKSUM >> >> https://download-ib01.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-32-1.6-x86_64-CHECKSUM >> >> and the GPG keys are available here: >> https://getfedora.org/en/security/ >> >> NOTE: I tried moving to Fedora 33, but the aarch64 VM cannot boot >> properly. May be worth investigating but I have no experience with ARM >> so I'll leave that to someone else, if interested. >> >> Daniele Buono (1): >> tests/acceptance/boot_linux: Switch to Fedora 32 >> >> tests/acceptance/boot_linux.py | 11 +++++------ >> 1 file changed, 5 insertions(+), 6 deletions(-) >> > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 2021-01-28 22:06 ` Daniele Buono @ 2021-02-05 16:54 ` Philippe Mathieu-Daudé 2021-02-05 17:28 ` Daniel P. Berrangé 0 siblings, 1 reply; 6+ messages in thread From: Philippe Mathieu-Daudé @ 2021-02-05 16:54 UTC (permalink / raw) To: Daniele Buono, Wainer dos Santos Moschetta, qemu-devel Cc: Daniel P. Berrangé, Cleber Rosa Hi Wainer, On 1/28/21 11:06 PM, Daniele Buono wrote: > On 1/28/2021 3:19 PM, Wainer dos Santos Moschetta wrote: >> Hi, >> >> On 1/26/21 10:09 PM, Daniele Buono wrote: >>> Local acceptance tests run with "make check-acceptance" are now >>> showing some cases canceled like the following: >>> >>> (01/39) >>> tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: >>> CANCEL: Failed to download/prepare boot image (0.25 s) >>> >>> Turns out, every full-vm test in boot_linux.py is trying to use a >>> Fedora 31 cloud image and is failing, with Avocado refusing to download >>> it, presumably because Fedora 31 is EOL. >>> >>> This patch moves to Fedora 32, which is still supported. And seem to >>> work fine >> >> While ago it was discussed about updating the Fedora version which, in >> my opinion, ended up without a conclusion. Please see the complete >> thread in: >> >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html > > Oops, didn't notice the previous thread. Apologies for the duplicate! > >> >> I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the >> discussion. The first question I'd like to figure out is how/where can we archive the artifacts being tested by the project. As we maintain stable tags, I'm more worried about regressions affecting LTS use rather than recently added features which get more coverage and activity. Is this too conservative for acceptance testing? Regards, Phil. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 2021-02-05 16:54 ` Philippe Mathieu-Daudé @ 2021-02-05 17:28 ` Daniel P. Berrangé 0 siblings, 0 replies; 6+ messages in thread From: Daniel P. Berrangé @ 2021-02-05 17:28 UTC (permalink / raw) To: Philippe Mathieu-Daudé Cc: Cleber Rosa, Wainer dos Santos Moschetta, Daniele Buono, qemu-devel On Fri, Feb 05, 2021 at 05:54:24PM +0100, Philippe Mathieu-Daudé wrote: > Hi Wainer, > > On 1/28/21 11:06 PM, Daniele Buono wrote: > > On 1/28/2021 3:19 PM, Wainer dos Santos Moschetta wrote: > >> Hi, > >> > >> On 1/26/21 10:09 PM, Daniele Buono wrote: > >>> Local acceptance tests run with "make check-acceptance" are now > >>> showing some cases canceled like the following: > >>> > >>> (01/39) > >>> tests/acceptance/boot_linux.py:BootLinuxX8664.test_pc_i440fx_tcg: > >>> CANCEL: Failed to download/prepare boot image (0.25 s) > >>> > >>> Turns out, every full-vm test in boot_linux.py is trying to use a > >>> Fedora 31 cloud image and is failing, with Avocado refusing to download > >>> it, presumably because Fedora 31 is EOL. > >>> > >>> This patch moves to Fedora 32, which is still supported. And seem to > >>> work fine > >> > >> While ago it was discussed about updating the Fedora version which, in > >> my opinion, ended up without a conclusion. Please see the complete > >> thread in: > >> > >> https://www.mail-archive.com/qemu-devel@nongnu.org/msg763986.html > > > > Oops, didn't notice the previous thread. Apologies for the duplicate! > > > >> > >> I'm CC'ing Daniel Berrrangé so that, perhaps, we could resume the > >> discussion. > > The first question I'd like to figure out is how/where can we archive > the artifacts being tested by the project. As we maintain stable tags, > I'm more worried about regressions affecting LTS use rather than > recently added features which get more coverage and activity. > Is this too conservative for acceptance testing? I think there's multiple issues to worry about, one short term, the others long term. The most immediate issue is that we are pointing to an EOL Fedora release with broken download URL. While we could update to F 32, that doesn't solve the problem. It will burn us again in just a few months when F34 comes out making F32 EOL. We need a long term solution. The broader long term question is what our goals are for the acceptance tests. I think the primary goal is detect regressions in QEMU where we break something which used to work. To achieve this we don't need to be chasing distro releases. It is fine for us to be testing an EOL distro. The fact that it is EOL doesn't invalidate the test behaviour. The only problem switch EOL distros is the URL breakge. That is trivially solved in the Fedora case by downloading from the archive.fedoraproject.org server instead of main server. Problem solved forever. The more important long term question is whether the selection of distros we are testing gives us coverage which exercises all the scenarios that we care about. For example, considering virtio devices, we need - Guest OS which *only* has virtio legacy mode implemented - pc for PCI - q35 for PCI-e Proves deices work in legacy mode on both PCI and PCIe - Guest OS which has both legacy and modern mode implemented - pc for PCI - q35 for PCI-e Proves devices work in transitional mode (pc) and modern-only mode (PCI-e) So for virtio coverage, we'll need 2 guest OS, each with two scenarios as a starting point. RHEL-6 is an example of a distro that was legacy only. Anything newer does the other case. But then we need to expand the axis to non-x86 too, most especially for ppc64 and s390, which have non-PCI based options as well as PCI. Periodically a new virtio device will arrive (eg virtiofs) and that might mean we need to add another test scenario, or upgrade the more modern OS in an existing test scenario. This is reasonably infrequent though, so in general we won't need to be chasing the bleeding edge distros. The matrix can grow pretty damn fast, and we need to keep it under control. One way to deal with that is to have one test case cover multiple features. eg don't run a separate test for each of virtio-net, virtio-blk, virtio-scsi. Launch a single VM in which we can test all of them at once. Also we don't need to test all of Ubuntu, Fedora, RHEL, SUSE if all those distros are basically exercising the same set of QEMU device code paths. ie just becuase the distro is different, doesn't mean it adds value to the test coverage. We should be strongly driven by whether the distro exercises a feature that isn't already covered by an existing test. So the next question is what needs to be in our matrix ? Most downstream vendors will say it needs to be all KVM related features. QEMU as a project though also cares more broadly about many more devices for emulating older platforms. There's some balance to be struck here. On many previous occassions we've talked about classifying QEMU features into support tiers. If we had a view of what our support tiers were for each feature, this would in turn show us where to spend effort in building up testing coverage. I made an abortive start at trying to define what tiers might mean but didn't get into really classifying features https://wiki.qemu.org/Support_Tiers don't pay attention to the host/guest classifications I did there - I was mostly just playing with how to represent it. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2021-02-05 18:12 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-01-27 1:09 [PATCH 0/1] tests/acceptance/boot_linux: Switch to Fedora 32 Daniele Buono 2021-01-27 1:09 ` [PATCH 1/1] " Daniele Buono 2021-01-28 20:19 ` [PATCH 0/1] " Wainer dos Santos Moschetta 2021-01-28 22:06 ` Daniele Buono 2021-02-05 16:54 ` Philippe Mathieu-Daudé 2021-02-05 17:28 ` Daniel P. Berrangé
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).