From: Cleber Rosa <crosa@redhat.com>
To: "Philippe Mathieu-Daudé" <philmd@redhat.com>
Cc: "Thomas Huth" <thuth@redhat.com>,
"Daniel P . Berrange" <berrange@redhat.com>,
"Eduardo Habkost" <ehabkost@redhat.com>,
"Bin Meng" <bin.meng@windriver.com>,
"qemu-devel@nongnu.org Developers" <qemu-devel@nongnu.org>,
"Niek Linnenbank" <nieklinnenbank@gmail.com>,
"Gerd Hoffmann" <kraxel@redhat.com>,
"Stefan Hajnoczi" <stefanha@redhat.com>,
"Bin Meng" <bmeng.cn@gmail.com>,
"Alex Bennée" <alex.bennee@linaro.org>
Subject: Re: Should we keep using Avocado for functional testing? (was: Re: [PULL 22/23] hw/sd: Fix incorrect populated function switch status data structure)
Date: Mon, 26 Oct 2020 12:14:50 -0400 [thread overview]
Message-ID: <20201026161450.GA2201333@localhost.localdomain> (raw)
In-Reply-To: <26e41113-e124-63f7-64c5-2d42011c5cfb@redhat.com>
[-- Attachment #1: Type: text/plain, Size: 6700 bytes --]
On Fri, Oct 23, 2020 at 11:34:29AM +0200, Philippe Mathieu-Daudé wrote:
> On 10/23/20 11:23 AM, Philippe Mathieu-Daudé wrote:
>
> Back to the cache problem, I started to ask 2 years ago,
> https://www.mail-archive.com/avocado-devel@redhat.com/msg00860.html
> note this is the exact situation we are having:
>
Hi Phil,
I think the issue on this thread was more on the server mirror side,
than on the *cache* side of things, right? Still, it's an open issue
that needs to be discussed indeed.
> >> - What will happens if I add tests downloading running on their
> >> compiled u-boot
> >> (https://downloads.gumstix.com/images/angstrom/developer/2012-01-22-1750/u-boot.bin)
> >> and the company decides to remove this old directory?
> >> Since sometimes old open-source software are hard to rebuild with
> >> recent compilers, should we consider to use a public storage to keep
> >> open-source (signed) blobs we can use for integration testing?
> >
> > For Avocado-VT, there are the JeOS images[1], which we keep on a test
> > "assets" directory. We have a lot of storage/bandwidth availability,
> > so it can be used for other assets proven to be necessary for tests.
> >
> > As long as distribution rights and licensing are not issues, we can
> > definitely use the same server for kernels, u-boot images and what
> > not.
> >
> > [1] - https://avocado-project.org/data/assets/
>
>
> We discussed about this again last year at the KVM forum. Various
> RFE have been filled:
> https://www.mail-archive.com/avocado-devel@redhat.com/msg01183.html
>
I understand it has been a year, which is a lot of time specially when
it comes to an urgent need such as more and better testing and CI.
But some things can be patched quickly, while others need further
structural work. More on that later...
>
> What we need is a QEMU community file server similar to the asset
> one used by the Avocado community. The problem is who is going to
> pay and sysadmin it. IIRC Gerd suggested to use GitLab Page, but
> the artifact file size is limited to a maximum of 1GiB:
> https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-pages
> Alternative is to use a git-lfs server:
> https://docs.gitlab.com/ee/topics/git/lfs/#hosting-lfs-objects-externally
> This is now a project management problem.
>
> Next week is the KVM forum 2020 and there might be a BoF session
> to talk about that:
> https://kvmforum2020.sched.com/overview/type/Birds+of+a+Feather+Sessions+(BoF)
>
>
> Now that the QEMU community started to use gitlab-ci more and more
> I received complains that Avocado is not practical (hard to
> understand how to reproduce command line to debug, options to use
> not clear, timeouts not clear, why download all artifacts to run
> a single test, various issues regarding caching, cache filled /home
> filesystem) so I have been asked to look at Avocado alternatives,
> because we want contributors run more tests and CI, not them be
> scared by it.
What I can say is that the Avocado developers have been working
on a major overhaul to address those issues. I'm not sure this
is the place/time to go over each of those items, but there may be
some confusion going on some of those issues. For instance, your
point about "why download all artifacts to run a single test".
It shouldn't, and it doesn't:
$ du -hcs ~/avocado/data/cache/
4.0K /home/cleber/avocado/data/cache/
4.0K total
$ ./tests/venv/bin/avocado run tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc
Fetching asset from tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc
JOB ID : a3c08d7a0acc613c207b27e4b291a0010965af2d
JOB LOG : /home/cleber/avocado/job-results/job-2020-10-26T11.50-a3c08d7/job.log
(1/1) tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: PASS (2.04 s)
RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 3.22 s
$ du -hcs ~/avocado/data/cache/
8.3M /home/cleber/avocado/data/cache/
8.3M total
About Avocado being hard to understand, I'm a bit saddened by that,
because the team spent a lot of effort in:
* Restructuring / rewriting / reviewing the documention
* Normalizing the command line options
* Deprecating and removing a lot of non-relevant functionality
I've probably said the same thing in a few different channels and
contexts, but for the last ~9 months, the Avocado project had been
working on a new architecture that was highly influenced by the
limitations we found while using Avocado in QEMU. Version 82.0 was
the result of that work, and it'd be very bad timing if, instead of
integrating this "new Avocado", QEMU would start "from scratch" again.
I've been looking for an opportunity to present what this "new
Avocado" is. Maybe this or another BoF would be a good place?
> I haven't looked at alternatives yet, from the various features
> we could have use, the biggest one is the ability to interact with
> the serial console. And that feature is duplicated with the VM
> testing class: tests/vm/basevm.py.
> The SSH session used in linux_ssh_mips_malta.py is similar to the
> one from basevm.py too (see 'make vm-boot-ssh-%').
> Some tests are not 'acceptance' but simple qtest written in Python,
> such cpu_queries.py / pc_cpu_hotplug_props.py / migration.py /
> pc_cpu_hotplug_props.py / x86_cpu_model_versions.py.
>
> The classes I find practical are downloading artifact, uncompressing
> or extracting archive, and eventually the cloudinit one. But we can probably
> use them directly.
>
I understand this is your PoV with regards to how much value you
believe exists in those particular utility modules. The challenge is
how to combine other people's PoV with regards to the same matter, and
manage them accordingly as a group. One example is how Pavel Dovgaluk
may find the "gdb" utility module important.
Now consider that Red Hat is vested to have some of the functional /
integration tests that run downstream (and which are based on
Avocado+Avocado-VT) migrated upstreamed. Then, all of a sudden, a
much larger group of "practical classes" arise.
Finally, these are, as you mentioned in another thread, utility
libraries (such as python/qemu) are commonly needed in other projects
(specially those related to QEMU). I understand that suggesting to
have more code infrastructure code living in the QEMU would actually
make it harder to solve similar problems once.
>
> Lot of material to discuss :)
>
Absolutely!
Best,
- Cleber.
> Regards,
>
> Phil.
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2020-10-26 16:18 UTC|newest]
Thread overview: 35+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-21 17:28 [PULL 00/23] SD/MMC patches for 2020-08-21 Philippe Mathieu-Daudé
2020-08-21 17:28 ` [PULL 01/23] hw/sd/pxa2xx_mmci: Do not create SD card within the SD host controller Philippe Mathieu-Daudé
2020-08-21 17:28 ` [PULL 02/23] hw/sd/pxa2xx_mmci: Trivial simplification Philippe Mathieu-Daudé
2020-08-21 17:28 ` [PULL 03/23] hw/lm32/milkymist: Un-inline milkymist_memcard_create() Philippe Mathieu-Daudé
2020-08-21 17:28 ` [PULL 04/23] hw/sd/milkymist: Create the SDBus at init() Philippe Mathieu-Daudé
2020-08-21 17:28 ` [PULL 05/23] hw/sd/milkymist: Do not create SD card within the SD host controller Philippe Mathieu-Daudé
2020-08-21 17:28 ` [PULL 06/23] hw/sd/pl181: Replace fprintf(stderr, "*\n") with error_report() Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 07/23] hw/sd/pl181: Rename pl181_send_command() as pl181_do_command() Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 08/23] hw/sd/pl181: Add TODO to use Fifo32 API Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 09/23] hw/sd/pl181: Use named GPIOs Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 10/23] hw/sd/pl181: Expose a SDBus and connect the SDCard to it Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 11/23] hw/sd/pl181: Do not create SD card within the SD host controller Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 12/23] hw/sd/pl181: Replace disabled fprintf()s by trace events Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 13/23] hw/sd/sdcard: Make sd_data_ready() static Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 14/23] hw/sd: Move sdcard legacy API to 'hw/sd/sdcard_legacy.h' Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 15/23] hw/sd: Rename read/write_data() as read/write_byte() Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 16/23] hw/sd: Rename sdbus_write_data() as sdbus_write_byte() Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 17/23] hw/sd: Rename sdbus_read_data() as sdbus_read_byte() Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 18/23] hw/sd: Add sdbus_write_data() to write multiples bytes on the data line Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 19/23] hw/sd: Use sdbus_write_data() instead of sdbus_write_byte when possible Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 20/23] hw/sd: Add sdbus_read_data() to read multiples bytes on the data line Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 21/23] hw/sd: Use sdbus_read_data() instead of sdbus_read_byte() when possible Philippe Mathieu-Daudé
2020-08-21 17:29 ` [PULL 22/23] hw/sd: Fix incorrect populated function switch status data structure Philippe Mathieu-Daudé
2020-10-20 15:16 ` Philippe Mathieu-Daudé
2020-10-21 9:57 ` Bin Meng
2020-10-21 10:07 ` Philippe Mathieu-Daudé
2020-10-22 14:47 ` Bin Meng
2020-10-22 15:20 ` Niek Linnenbank
2020-10-23 2:02 ` Bin Meng
2020-10-23 9:23 ` Philippe Mathieu-Daudé
2020-10-23 9:34 ` Should we keep using Avocado for functional testing? (was: Re: [PULL 22/23] hw/sd: Fix incorrect populated function switch status data structure) Philippe Mathieu-Daudé
2020-10-24 21:41 ` Niek Linnenbank
2020-10-26 16:14 ` Cleber Rosa [this message]
2020-08-21 17:29 ` [PULL 23/23] hw/sd: Correct the maximum size of a Standard Capacity SD Memory Card Philippe Mathieu-Daudé
2020-08-23 10:37 ` [PULL 00/23] SD/MMC patches for 2020-08-21 Peter Maydell
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201026161450.GA2201333@localhost.localdomain \
--to=crosa@redhat.com \
--cc=alex.bennee@linaro.org \
--cc=berrange@redhat.com \
--cc=bin.meng@windriver.com \
--cc=bmeng.cn@gmail.com \
--cc=ehabkost@redhat.com \
--cc=kraxel@redhat.com \
--cc=nieklinnenbank@gmail.com \
--cc=philmd@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@redhat.com \
--cc=thuth@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).