From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:57488) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ek9pN-0007xw-3l for qemu-devel@nongnu.org; Fri, 09 Feb 2018 09:39:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ek9pJ-0002EF-Sn for qemu-devel@nongnu.org; Fri, 09 Feb 2018 09:39:01 -0500 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:47582 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1ek9pJ-0002DR-Lb for qemu-devel@nongnu.org; Fri, 09 Feb 2018 09:38:57 -0500 References: <7e30d2cc-6c42-15e9-78ef-1085e081a386@redhat.com> From: =?UTF-8?B?THVrw6HFoSBEb2t0b3I=?= Message-ID: <011121a3-6c46-770d-161a-cafdeeca184c@redhat.com> Date: Fri, 9 Feb 2018 15:38:40 +0100 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="0ro7a2PR67i2ipewpM85yQnwHpLgAGxrT" Subject: Re: [Qemu-devel] Functional tests (AKA Avocado-based tests) List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alistair Francis , Cleber Rosa Cc: "qemu-devel@nongnu.org Developers" , Amador Pahim , Jeff Nelson This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --0ro7a2PR67i2ipewpM85yQnwHpLgAGxrT From: =?UTF-8?B?THVrw6HFoSBEb2t0b3I=?= To: Alistair Francis , Cleber Rosa Cc: "qemu-devel@nongnu.org Developers" , Amador Pahim , Jeff Nelson Message-ID: <011121a3-6c46-770d-161a-cafdeeca184c@redhat.com> Subject: Re: [Qemu-devel] Functional tests (AKA Avocado-based tests) References: <7e30d2cc-6c42-15e9-78ef-1085e081a386@redhat.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Dne 9.2.2018 v 00:41 Alistair Francis napsal(a): > On Thu, Feb 8, 2018 at 3:38 PM, Alistair Francis = wrote: >> On Mon, Feb 5, 2018 at 8:34 AM, Cleber Rosa wrote: >>> >>> >>> On 02/01/2018 07:10 PM, Alistair Francis wrote: >>>> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa wrot= e: >>>>> >>>>> >>>>> On 01/17/2018 06:41 PM, Alistair Francis wrote: >>>>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa w= rote: >>>>>>> TL;DR >>>>>>> =3D=3D=3D=3D=3D >>>>>>> >>>>>>> This is about how QEMU developers can get started with functional= >>>>>>> tests that are built on top of the Avocado libraries (and meant t= o be >>>>>>> run with the Avocado test runner). >>>>>>> >>>>>>> The past >>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D >>>>>>> >>>>>>> The Avocado project[1] has been working, for quite some time now,= on a >>>>>>> "set of tools and libraries" with the goal of making writing test= s >>>>>>> easier. It is supposed to be a framework agnostic to the exact >>>>>>> software that will be under test. >>>>>>> >>>>>>> But, at the same time, the Avocado project cannot deny its inheri= tance >>>>>>> and influences. Those come from Autotest[2], which had "KVM Auto= test" >>>>>>> as its largest and most developed "test". This large Autotest te= st >>>>>>> (KVM Autotest) became virt-test[3] and later got integrated into >>>>>>> Avocado and became Avocado-VT[4] which is quite relevant here, >>>>>>> together with its QEMU test provider[5]. >>>>>>> >>>>>>> Avocado-VT and the QEMU test provider attempt to provide coverage= >>>>>>> across platform and QEMU versions, which increases its complexity= =2E >>>>>>> Also, it's built on a legacy set of principles and tools that mak= es >>>>>>> some developers stir away from it. >>>>>>> >>>>>>> What's new? >>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>>>> >>>>>>> A few months ago, the Avocado developers returned to its >>>>>>> "virtualization origins", in an attempt to learn from the QEMU >>>>>>> project, and try to help with a way to have more functional tests= in >>>>>>> the upstream QEMU repo. >>>>>>> >>>>>>> We believe it's possible to expand the test coverage for QEMU by >>>>>>> facilitating >>>>>>> the creation of more functional tests QEMU. This is no different= than how >>>>>>> other types of tests are already included in the tree itself. >>>>>>> >>>>>>> How >>>>>>> =3D=3D=3D >>>>>>> >>>>>>> How we did it (so far) >>>>>>> ---------------------- >>>>>>> >>>>>>> We're aware that there's a dilemma here: to be able to easily wri= te >>>>>>> more powerful tests, a lot of the complexity has to be moved >>>>>>> elsewhere. Here, it means moving complexity from the test itself= to a >>>>>>> framework. The QEMU source tree itself has proofs of this approa= ch, >>>>>>> being the "scripts" and "tests/qemu-iotests" some of the examples= =2E >>>>>>> >>>>>>> Avocado itself[1] provides a lot of the code that should help to >>>>>>> absorb some of the complexities in writing tests, but not exactly= >>>>>>> everything that is needed for QEMU. The approach we believe will= have >>>>>>> the best balance is to reuse upstream Avocado libraries whenever = they >>>>>>> are useful and generic enough, and on top of that, libraries that= are >>>>>>> part of QEMU itself. >>>>>>> >>>>>>> How can you get started with it >>>>>>> ------------------------------- >>>>>>> >>>>>>> First of all, get Avocado installed. Besides the Avocado test ru= nner >>>>>>> itself, this will give you the basic libraries on which the other= part >>>>>>> of this work was built on. We want that to be simple and painles= s, so >>>>>>> here's our best bet for a one-liner installation: >>>>>>> >>>>>>> pip install --user avocado-framework >>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect >>>>>>> >>>>>>> That will install Avocado within the user's home directory. If y= ou >>>>>>> give up on it, it can be uninstalled with another simple one-line= r: >>>>>>> >>>>>>> pip uninstall -y avocado-framework >>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect >>>>>>> >>>>>>> Now, suppose you're working on a given feature, and want to try y= our >>>>>>> luck writing a test using this work. To avoid having you fetchin= g and >>>>>>> rebasing from our currently in development fork[6] and branch[7],= you >>>>>>> can just >>>>>>> add one commit to your tree with: >>>>>>> >>>>>>> curl >>>>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.= patch | >>>>>>> git am - >>>>>>> >>>>>>> This will get a simple patch from a snapshot branch[8]. You can,= of course, >>>>>>> do it "the git way", fetching from that repo[6] and using the >>>>>>> non-snapshotted branch. >>>>>>> >>>>>>> After that, we'd love for you to take a look at some of the exist= ing >>>>>>> tests[9][10] and then attempt to create test for your own use cas= e. >>>>>>> The basic README[11] file, and the Avocado documentation[12] are = also >>>>>>> important resources not to be missed. >>>>>>> >>>>>>> What's next? >>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>>>> >>>>>>> Initially, feedback is what we're looking for. It would be great= ly >>>>>>> appreciated to understand if/how this suits (or not) use cases ou= t >>>>>>> there. >>>>>>> >>>>>>> After feedback, further refinements, and more tests are written, = the >>>>>>> Avocado developers will follow up with an initial patch series fo= r >>>>>>> upstream QEMU. In such a proposal, we intend to have further >>>>>>> integration. Ideally in way that "configure" can be given a >>>>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "ma= ke >>>>>>> [functional-]check" would seamlessly include them. >>>>>> >>>>>> I have a few thoughts. >>>>>> >>>>>> We use pytest/pexpect internally to kick off QEMU runs and monitor= the >>>>>> output (no interaction with the QEMU source tree) and I think it i= s >>>>>> extremely useful. So I am all for using Python to test things and = this >>>>>> looks really well done! >>>>>> >>>>> >>>>> Thanks for checking it out, and for the positive words. Now, sorry= if >>>>> I'm missing some obvious information, but is this work of yours wit= h >>>>> pytest/pexpect publicly available? I'd like to also take a look at= >>>>> that, because it does look similar to the Avocado + aexpect approac= h >>>>> taken here. >>>> >>>> Unfortunately it's not and it would take months for us to be able to= >>>> make it available. >>>> >>> >>> I see. >>> >>>>> >>>>>> What I don't understand though is what this gives us compared to t= he >>>>>> existing QEMU test infrastructure? Besides being able to use Pytho= n >>>>>> and a better interface what are the main benefits? I think that is= >>>>>> something worth documenting somewhere. >>>>>> >>>>> >>>>> We currently intend to *add* to the QEMU test infrastructure, not >>>>> replace it. >>>> >>>> Is there a benefit of integrating it into the tree then? It's always= >>>> possible to have an out of tree testing framework. >>>> >>> >>> Upstream first is just the general modus operandi that we have. Then= , >>> we want to have as much testing as possible as early as possible. >>> >>> Finally, by having tests in-tree, they can be seen in different light= =2E >>> What I mean is that when tests are in-tree, they will (or should) rea= lly >>> map to the current state of the code they test (per commit). If a >>> feature changes behavior upstream, the respective test will also need= a >>> change at the same time, and as such, will mean an *intended* change = and >>> not a regression. With out-of-tree tests, it's pretty hard to keep t= his >>> synchronization and regressions will slip in, *at least* for some tim= e. >>> >>> I guess the question here is actually the opposite: do you see any >>> problems with in-tree tests? Since you keep out of tree tests, I wou= ld >>> honestly like to hear about your experience. >> >> For us the main reason is that we want to avoid editing the fork as >> much as possible, so we aren't going to add tests to our fork. >> >> The other main advantage I see is that it allows the tests to be >> updated more easily as the changes don't need to go through the more >> rigorous QEMU review process. >> >> You are correct though that synchronisation is an issue. >=20 > Actually another thing we can do is run our tests on a known working > QEMU version with new/different guest images to test other teams work. > For example we can find Linux bugs by running the latest internal > Linux kernel builds on our known working master QEMU. >=20 > That might be harder to setup and justify with an in tree framework, > but isn't really a justification to make it out of tree. >=20 Sure, but it means your tests are written the way they work with differen= t qemu versions. For those we have `avocado-vt` with lots of tests usuall= y written by QA guys (as Avocado-vt is not easy to start with and does no= t attract many qemu developers). The beauty of Avocado_qemu is it's suppo= se to be very simple to start with to allow writing simple, specific, usu= ally single-purpose tests, especially regression, tests that can be easil= y attached along with the fix. (at least that's what I see as the main go= al) Regards, Luk=C3=A1=C5=A1 >> >> Alistair >> >>> >>>>> >>>>> The benefits we envision are, besides hopefully easier and more cap= able >>>>> interfaces, to simply have more upstream tests. This means avoidin= g new >>>>> regressions and improving coverage. >>>>> >>>>>> Also, it looks like this will require images checked into git >>>>>> somewhere is that correct? Is there a good plan on how to handle t= hat? >>>>>> >>>>> >>>>> It won't require images checked into git. Right now, tests use the= >>>>> vmimage library: >>>>> >>>>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.u= tils.html#avocado.utils.vmimage.get >>>>> >>>>> Which downloads (and caches) images from external sources. >>>> >>>> Ah! That's cool. Managing images is one of the challenges we have at= the moment. >>>> >>>> Alistair >>>> >>> >>> Nice that you like it. We also find a library such as "vmimage" is >>> something simple enough, but still not quite something that would liv= e >>> in the QEMU tree. >>> >>> - Cleber. >>> >>>>> >>>>> Please let me know if you have more questions! >>>>> - Cleber. >>>>> >>>>>> Alistair >>>>>> >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> References >>>>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>>>> >>>>>>> [1] http://avocado-framework.github.io/ >>>>>>> [2] http://autotest.github.io/ >>>>>>> [3] https://github.com/autotest/virt-test >>>>>>> [4] https://github.com/avocado-framework/avocado-vt >>>>>>> [5] https://github.com/autotest/tp-qemu >>>>>>> [6] https://github.com/apahim/qemu >>>>>>> [7] https://github.com/apahim/qemu/tree/avocado_qemu >>>>>>> [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot >>>>>>> [9] >>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/te= st_info_memdev_host_nodes.py >>>>>>> [10] >>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/te= st_ovmf_with_240_vcpus.py >>>>>>> [11] >>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/RE= ADME.rst >>>>>>> [12] http://avocado-framework.readthedocs.io/ >>>>>>> >>>>>>> -- >>>>>>> Cleber Rosa >>>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>>>>>> [ Avocado Test Framework - avocado-framework.github.io ] >>>>>>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >>>>>>> >>>>> >>>>> -- >>>>> Cleber Rosa >>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>>>> [ Avocado Test Framework - avocado-framework.github.io ] >>>>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] >>> >>> -- >>> Cleber Rosa >>> [ Sr Software Engineer - Virtualization Team - Red Hat ] >>> [ Avocado Test Framework - avocado-framework.github.io ] >>> [ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ] --0ro7a2PR67i2ipewpM85yQnwHpLgAGxrT Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQFHBAEBCAAxFiEEpApMRcQDTeAqWtSDJrNi5H/PIsEFAlp9snATHGxkb2t0b3JA cmVkaGF0LmNvbQAKCRAms2Lkf88iwSFwB/9jHbgWpozmApyRD881Ah4pUYj5GSxP PiD6kqF7KoUglE/xZ5dCQFWXZGxx9VyNfuPZkLK+DdUqbITgPd4RRQO2xT3E+TjW V1/lFi5mrX2AbXGU+9j3kOOAPZssY9uZts3JS7Nctjdwa3uMh/16FeAhyixa0awi 3csZPWdFmcgswNVcmVeNjt+fNZADx1/tGiRr1RTcgc62aFOe6AsZNUQ060chAkz/ qgt8SsmnVNtOnAlrVEcEc2+hHkwyZcrp336wmlXCH8HdbeE5Q14WjJGYn4OOsqwh mE49RlgtjDjH6yQa+eM25C0vp8var3KpaYqUaG7bUWawl2RcGBbH6sia =3gV/ -----END PGP SIGNATURE----- --0ro7a2PR67i2ipewpM85yQnwHpLgAGxrT--