[Qemu-devel] Functional tests (AKA Avocado-based tests)

[Qemu-devel] Functional tests (AKA Avocado-based tests)

[Cleber Rosa]

TL;DR
=====

This is about how QEMU developers can get started with functional
tests that are built on top of the Avocado libraries (and meant to be
run with the Avocado test runner).

The past
========

The Avocado project[1] has been working, for quite some time now, on a
"set of tools and libraries" with the goal of making writing tests
easier.  It is supposed to be a framework agnostic to the exact
software that will be under test.

But, at the same time, the Avocado project cannot deny its inheritance
and influences.  Those come from Autotest[2], which had "KVM Autotest"
as its largest and most developed "test".  This large Autotest test
(KVM Autotest) became virt-test[3] and later got integrated into
Avocado and became Avocado-VT[4] which is quite relevant here,
together with its QEMU test provider[5].

Avocado-VT and the QEMU test provider attempt to provide coverage
across platform and QEMU versions, which increases its complexity.
Also, it's built on a legacy set of principles and tools that makes
some developers stir away from it.

What's new?
===========

A few months ago, the Avocado developers returned to its
"virtualization origins", in an attempt to learn from the QEMU
project, and try to help with a way to have more functional tests in
the upstream QEMU repo.

We believe it's possible to expand the test coverage for QEMU by
facilitating
the creation of more functional tests QEMU.  This is no different than how
other types of tests are already included in the tree itself.

How
===

How we did it (so far)
----------------------

We're aware that there's a dilemma here: to be able to easily write
more powerful tests, a lot of the complexity has to be moved
elsewhere.  Here, it means moving complexity from the test itself to a
framework.  The QEMU source tree itself has proofs of this approach,
being the "scripts" and "tests/qemu-iotests" some of the examples.

Avocado itself[1] provides a lot of the code that should help to
absorb some of the complexities in writing tests, but not exactly
everything that is needed for QEMU.  The approach we believe will have
the best balance is to reuse upstream Avocado libraries whenever they
are useful and generic enough, and on top of that, libraries that are
part of QEMU itself.

How can you get started with it
-------------------------------

First of all, get Avocado installed.  Besides the Avocado test runner
itself, this will give you the basic libraries on which the other part
of this work was built on.  We want that to be simple and painless, so
here's our best bet for a one-liner installation:

  pip install --user avocado-framework
avocado-framework-plugin-varianter-yaml-to-mux aexpect

That will install Avocado within the user's home directory.  If you
give up on it, it can be uninstalled with another simple one-liner:

  pip uninstall -y avocado-framework
avocado-framework-plugin-varianter-yaml-to-mux aexpect

Now, suppose you're working on a given feature, and want to try your
luck writing a test using this work.  To avoid having you fetching and
rebasing from our currently in development fork[6] and branch[7], you
can just
add one commit to your tree with:

  curl
https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
git am -

This will get a simple patch from a snapshot branch[8].  You can, of course,
do it "the git way", fetching from that repo[6] and using the
non-snapshotted branch.

After that, we'd love for you to take a look at some of the existing
tests[9][10] and then attempt to create test for your own use case.
The basic README[11] file, and the Avocado documentation[12] are also
important resources not to be missed.

What's next?
============

Initially, feedback is what we're looking for.  It would be greatly
appreciated to understand if/how this suits (or not) use cases out
there.

After feedback, further refinements, and more tests are written, the
Avocado developers will follow up with an initial patch series for
upstream QEMU.  In such a proposal, we intend to have further
integration.  Ideally in way that "configure" can be given a
"--with-functional-[avocado-]tests" parameter of sorts, and a "make
 [functional-]check" would seamlessly include them.

Thanks!

References
==========

 [1] http://avocado-framework.github.io/
 [2] http://autotest.github.io/
 [3] https://github.com/autotest/virt-test
 [4] https://github.com/avocado-framework/avocado-vt
 [5] https://github.com/autotest/tp-qemu
 [6] https://github.com/apahim/qemu
 [7] https://github.com/apahim/qemu/tree/avocado_qemu
 [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
 [9]
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
[10]
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
[11]
https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
[12] http://avocado-framework.readthedocs.io/

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
> TL;DR
> =====
>
> This is about how QEMU developers can get started with functional
> tests that are built on top of the Avocado libraries (and meant to be
> run with the Avocado test runner).
>
> The past
> ========
>
> The Avocado project[1] has been working, for quite some time now, on a
> "set of tools and libraries" with the goal of making writing tests
> easier.  It is supposed to be a framework agnostic to the exact
> software that will be under test.
>
> But, at the same time, the Avocado project cannot deny its inheritance
> and influences.  Those come from Autotest[2], which had "KVM Autotest"
> as its largest and most developed "test".  This large Autotest test
> (KVM Autotest) became virt-test[3] and later got integrated into
> Avocado and became Avocado-VT[4] which is quite relevant here,
> together with its QEMU test provider[5].
>
> Avocado-VT and the QEMU test provider attempt to provide coverage
> across platform and QEMU versions, which increases its complexity.
> Also, it's built on a legacy set of principles and tools that makes
> some developers stir away from it.
>
> What's new?
> ===========
>
> A few months ago, the Avocado developers returned to its
> "virtualization origins", in an attempt to learn from the QEMU
> project, and try to help with a way to have more functional tests in
> the upstream QEMU repo.
>
> We believe it's possible to expand the test coverage for QEMU by
> facilitating
> the creation of more functional tests QEMU.  This is no different than how
> other types of tests are already included in the tree itself.
>
> How
> ===
>
> How we did it (so far)
> ----------------------
>
> We're aware that there's a dilemma here: to be able to easily write
> more powerful tests, a lot of the complexity has to be moved
> elsewhere.  Here, it means moving complexity from the test itself to a
> framework.  The QEMU source tree itself has proofs of this approach,
> being the "scripts" and "tests/qemu-iotests" some of the examples.
>
> Avocado itself[1] provides a lot of the code that should help to
> absorb some of the complexities in writing tests, but not exactly
> everything that is needed for QEMU.  The approach we believe will have
> the best balance is to reuse upstream Avocado libraries whenever they
> are useful and generic enough, and on top of that, libraries that are
> part of QEMU itself.
>
> How can you get started with it
> -------------------------------
>
> First of all, get Avocado installed.  Besides the Avocado test runner
> itself, this will give you the basic libraries on which the other part
> of this work was built on.  We want that to be simple and painless, so
> here's our best bet for a one-liner installation:
>
>   pip install --user avocado-framework
> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>
> That will install Avocado within the user's home directory.  If you
> give up on it, it can be uninstalled with another simple one-liner:
>
>   pip uninstall -y avocado-framework
> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>
> Now, suppose you're working on a given feature, and want to try your
> luck writing a test using this work.  To avoid having you fetching and
> rebasing from our currently in development fork[6] and branch[7], you
> can just
> add one commit to your tree with:
>
>   curl
> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
> git am -
>
> This will get a simple patch from a snapshot branch[8].  You can, of course,
> do it "the git way", fetching from that repo[6] and using the
> non-snapshotted branch.
>
> After that, we'd love for you to take a look at some of the existing
> tests[9][10] and then attempt to create test for your own use case.
> The basic README[11] file, and the Avocado documentation[12] are also
> important resources not to be missed.
>
> What's next?
> ============
>
> Initially, feedback is what we're looking for.  It would be greatly
> appreciated to understand if/how this suits (or not) use cases out
> there.
>
> After feedback, further refinements, and more tests are written, the
> Avocado developers will follow up with an initial patch series for
> upstream QEMU.  In such a proposal, we intend to have further
> integration.  Ideally in way that "configure" can be given a
> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>  [functional-]check" would seamlessly include them.

I have a few thoughts.

We use pytest/pexpect internally to kick off QEMU runs and monitor the
output (no interaction with the QEMU source tree) and I think it is
extremely useful. So I am all for using Python to test things and this
looks really well done!

What I don't understand though is what this gives us compared to the
existing QEMU test infrastructure? Besides being able to use Python
and a better interface what are the main benefits? I think that is
something worth documenting somewhere.

Also, it looks like this will require images checked into git
somewhere is that correct? Is there a good plan on how to handle that?

Alistair

>
> Thanks!
>
> References
> ==========
>
>  [1] http://avocado-framework.github.io/
>  [2] http://autotest.github.io/
>  [3] https://github.com/autotest/virt-test
>  [4] https://github.com/avocado-framework/avocado-vt
>  [5] https://github.com/autotest/tp-qemu
>  [6] https://github.com/apahim/qemu
>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>  [9]
> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
> [10]
> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
> [11]
> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
> [12] http://avocado-framework.readthedocs.io/
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Cleber Rosa]

On 01/17/2018 06:41 PM, Alistair Francis wrote:
> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>> TL;DR
>> =====
>>
>> This is about how QEMU developers can get started with functional
>> tests that are built on top of the Avocado libraries (and meant to be
>> run with the Avocado test runner).
>>
>> The past
>> ========
>>
>> The Avocado project[1] has been working, for quite some time now, on a
>> "set of tools and libraries" with the goal of making writing tests
>> easier.  It is supposed to be a framework agnostic to the exact
>> software that will be under test.
>>
>> But, at the same time, the Avocado project cannot deny its inheritance
>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>> as its largest and most developed "test".  This large Autotest test
>> (KVM Autotest) became virt-test[3] and later got integrated into
>> Avocado and became Avocado-VT[4] which is quite relevant here,
>> together with its QEMU test provider[5].
>>
>> Avocado-VT and the QEMU test provider attempt to provide coverage
>> across platform and QEMU versions, which increases its complexity.
>> Also, it's built on a legacy set of principles and tools that makes
>> some developers stir away from it.
>>
>> What's new?
>> ===========
>>
>> A few months ago, the Avocado developers returned to its
>> "virtualization origins", in an attempt to learn from the QEMU
>> project, and try to help with a way to have more functional tests in
>> the upstream QEMU repo.
>>
>> We believe it's possible to expand the test coverage for QEMU by
>> facilitating
>> the creation of more functional tests QEMU.  This is no different than how
>> other types of tests are already included in the tree itself.
>>
>> How
>> ===
>>
>> How we did it (so far)
>> ----------------------
>>
>> We're aware that there's a dilemma here: to be able to easily write
>> more powerful tests, a lot of the complexity has to be moved
>> elsewhere.  Here, it means moving complexity from the test itself to a
>> framework.  The QEMU source tree itself has proofs of this approach,
>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>
>> Avocado itself[1] provides a lot of the code that should help to
>> absorb some of the complexities in writing tests, but not exactly
>> everything that is needed for QEMU.  The approach we believe will have
>> the best balance is to reuse upstream Avocado libraries whenever they
>> are useful and generic enough, and on top of that, libraries that are
>> part of QEMU itself.
>>
>> How can you get started with it
>> -------------------------------
>>
>> First of all, get Avocado installed.  Besides the Avocado test runner
>> itself, this will give you the basic libraries on which the other part
>> of this work was built on.  We want that to be simple and painless, so
>> here's our best bet for a one-liner installation:
>>
>>   pip install --user avocado-framework
>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>
>> That will install Avocado within the user's home directory.  If you
>> give up on it, it can be uninstalled with another simple one-liner:
>>
>>   pip uninstall -y avocado-framework
>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>
>> Now, suppose you're working on a given feature, and want to try your
>> luck writing a test using this work.  To avoid having you fetching and
>> rebasing from our currently in development fork[6] and branch[7], you
>> can just
>> add one commit to your tree with:
>>
>>   curl
>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>> git am -
>>
>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>> do it "the git way", fetching from that repo[6] and using the
>> non-snapshotted branch.
>>
>> After that, we'd love for you to take a look at some of the existing
>> tests[9][10] and then attempt to create test for your own use case.
>> The basic README[11] file, and the Avocado documentation[12] are also
>> important resources not to be missed.
>>
>> What's next?
>> ============
>>
>> Initially, feedback is what we're looking for.  It would be greatly
>> appreciated to understand if/how this suits (or not) use cases out
>> there.
>>
>> After feedback, further refinements, and more tests are written, the
>> Avocado developers will follow up with an initial patch series for
>> upstream QEMU.  In such a proposal, we intend to have further
>> integration.  Ideally in way that "configure" can be given a
>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>  [functional-]check" would seamlessly include them.
> 
> I have a few thoughts.
> 
> We use pytest/pexpect internally to kick off QEMU runs and monitor the
> output (no interaction with the QEMU source tree) and I think it is
> extremely useful. So I am all for using Python to test things and this
> looks really well done!
> 

Thanks for checking it out, and for the positive words.  Now, sorry if
I'm missing some obvious information, but is this work of yours with
pytest/pexpect publicly available?  I'd like to also take a look at
that, because it does look similar to the Avocado + aexpect approach
taken here.

> What I don't understand though is what this gives us compared to the
> existing QEMU test infrastructure? Besides being able to use Python
> and a better interface what are the main benefits? I think that is
> something worth documenting somewhere.
> 

We currently intend to *add* to the QEMU test infrastructure, not
replace it.

The benefits we envision are, besides hopefully easier and more capable
interfaces, to simply have more upstream tests.  This means avoiding new
regressions and improving coverage.

> Also, it looks like this will require images checked into git
> somewhere is that correct? Is there a good plan on how to handle that?
> 

It won't require images checked into git.  Right now, tests use the
vmimage library:

http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get

Which downloads (and caches) images from external sources.

Please let me know if you have more questions!
- Cleber.

> Alistair
> 
>>
>> Thanks!
>>
>> References
>> ==========
>>
>>  [1] http://avocado-framework.github.io/
>>  [2] http://autotest.github.io/
>>  [3] https://github.com/autotest/virt-test
>>  [4] https://github.com/avocado-framework/avocado-vt
>>  [5] https://github.com/autotest/tp-qemu
>>  [6] https://github.com/apahim/qemu
>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>  [9]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>> [10]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>> [11]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>> [12] http://avocado-framework.readthedocs.io/
>>
>> --
>> Cleber Rosa
>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>> [ Avocado Test Framework - avocado-framework.github.io ]
>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Amador Pahim]

On Thu, Jan 18, 2018 at 1:47 AM, Cleber Rosa <crosa@redhat.com> wrote:
>
>
> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>> TL;DR
>>> =====
>>>
>>> This is about how QEMU developers can get started with functional
>>> tests that are built on top of the Avocado libraries (and meant to be
>>> run with the Avocado test runner).
>>>
>>> The past
>>> ========
>>>
>>> The Avocado project[1] has been working, for quite some time now, on a
>>> "set of tools and libraries" with the goal of making writing tests
>>> easier.  It is supposed to be a framework agnostic to the exact
>>> software that will be under test.
>>>
>>> But, at the same time, the Avocado project cannot deny its inheritance
>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>> as its largest and most developed "test".  This large Autotest test
>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>> together with its QEMU test provider[5].
>>>
>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>> across platform and QEMU versions, which increases its complexity.
>>> Also, it's built on a legacy set of principles and tools that makes
>>> some developers stir away from it.
>>>
>>> What's new?
>>> ===========
>>>
>>> A few months ago, the Avocado developers returned to its
>>> "virtualization origins", in an attempt to learn from the QEMU
>>> project, and try to help with a way to have more functional tests in
>>> the upstream QEMU repo.
>>>
>>> We believe it's possible to expand the test coverage for QEMU by
>>> facilitating
>>> the creation of more functional tests QEMU.  This is no different than how
>>> other types of tests are already included in the tree itself.
>>>
>>> How
>>> ===
>>>
>>> How we did it (so far)
>>> ----------------------
>>>
>>> We're aware that there's a dilemma here: to be able to easily write
>>> more powerful tests, a lot of the complexity has to be moved
>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>> framework.  The QEMU source tree itself has proofs of this approach,
>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>
>>> Avocado itself[1] provides a lot of the code that should help to
>>> absorb some of the complexities in writing tests, but not exactly
>>> everything that is needed for QEMU.  The approach we believe will have
>>> the best balance is to reuse upstream Avocado libraries whenever they
>>> are useful and generic enough, and on top of that, libraries that are
>>> part of QEMU itself.
>>>
>>> How can you get started with it
>>> -------------------------------
>>>
>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>> itself, this will give you the basic libraries on which the other part
>>> of this work was built on.  We want that to be simple and painless, so
>>> here's our best bet for a one-liner installation:
>>>
>>>   pip install --user avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> That will install Avocado within the user's home directory.  If you
>>> give up on it, it can be uninstalled with another simple one-liner:
>>>
>>>   pip uninstall -y avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> Now, suppose you're working on a given feature, and want to try your
>>> luck writing a test using this work.  To avoid having you fetching and
>>> rebasing from our currently in development fork[6] and branch[7], you
>>> can just
>>> add one commit to your tree with:
>>>
>>>   curl
>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>> git am -
>>>
>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>> do it "the git way", fetching from that repo[6] and using the
>>> non-snapshotted branch.
>>>
>>> After that, we'd love for you to take a look at some of the existing
>>> tests[9][10] and then attempt to create test for your own use case.
>>> The basic README[11] file, and the Avocado documentation[12] are also
>>> important resources not to be missed.
>>>
>>> What's next?
>>> ============
>>>
>>> Initially, feedback is what we're looking for.  It would be greatly
>>> appreciated to understand if/how this suits (or not) use cases out
>>> there.
>>>
>>> After feedback, further refinements, and more tests are written, the
>>> Avocado developers will follow up with an initial patch series for
>>> upstream QEMU.  In such a proposal, we intend to have further
>>> integration.  Ideally in way that "configure" can be given a
>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>  [functional-]check" would seamlessly include them.
>>
>> I have a few thoughts.
>>
>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>> output (no interaction with the QEMU source tree) and I think it is
>> extremely useful. So I am all for using Python to test things and this
>> looks really well done!
>>
>
> Thanks for checking it out, and for the positive words.  Now, sorry if
> I'm missing some obvious information, but is this work of yours with
> pytest/pexpect publicly available?  I'd like to also take a look at
> that, because it does look similar to the Avocado + aexpect approach
> taken here.
>
>> What I don't understand though is what this gives us compared to the
>> existing QEMU test infrastructure? Besides being able to use Python
>> and a better interface what are the main benefits? I think that is
>> something worth documenting somewhere.
>>
>
> We currently intend to *add* to the QEMU test infrastructure, not
> replace it.
>
> The benefits we envision are, besides hopefully easier and more capable
> interfaces, to simply have more upstream tests.  This means avoiding new
> regressions and improving coverage.
>
>> Also, it looks like this will require images checked into git
>> somewhere is that correct? Is there a good plan on how to handle that?
>>
>
> It won't require images checked into git.  Right now, tests use the
> vmimage library:
>
> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>
> Which downloads (and caches) images from external sources.

Expanding on that, the vmimage lib downloads and caches the so called
"cloud images" from distros own repositories. Here's[1] some
documentation with usage examples, including one example on how to add
your own image provider.

Since those images usually come with no password, requiring cloudinit,
we added to the avocado_qemu "add_image()" API the ability to set the
default user and its password attaching a cloudinit cdrom to the VM,
if you include the option "cloudinit=True". You can see an example
here[2].

In the example [2], there's no username/password options in the
add_image() call, so the defaults (avocado/avocado) will be assumed.
Btw, besides setting username/password as options in the add_image(),
they can also come from parameters, so you don't have to hard code
that information in in each test. See the "image_user"/"image_pass"
parameters in the README[3] file.

[1] - http://avocado-framework.readthedocs.io/en/57.0/utils/vmimage.html
[2] - https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_nec-usb-xhci.py#L22-L23
[3] - https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst

--
apahim

>
> Please let me know if you have more questions!
> - Cleber.
>
>> Alistair
>>
>>>
>>> Thanks!
>>>
>>> References
>>> ==========
>>>
>>>  [1] http://avocado-framework.github.io/
>>>  [2] http://autotest.github.io/
>>>  [3] https://github.com/autotest/virt-test
>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>  [5] https://github.com/autotest/tp-qemu
>>>  [6] https://github.com/apahim/qemu
>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>  [9]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>> [10]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>> [11]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>> [12] http://avocado-framework.readthedocs.io/
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Fam Zheng]

On Thu, Jan 18, 2018 at 5:07 PM, Amador Pahim <apahim@redhat.com> wrote:
> Expanding on that, the vmimage lib downloads and caches the so called
> "cloud images" from distros own repositories. Here's[1] some
> documentation with usage examples, including one example on how to add
> your own image provider.

Linux distributions are easy anyway, and sometimes even Docker is
simpler and good enough ($QEMU/tests/docker). Other architectures and
OS families, such as non-x86_64, BSD, Windows, macOS are all difficult
topics. $QEMU/tests/vm tried to attack that but is still not complete.
What are your thoughts on that subject?

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <crosa@redhat.com> wrote:
>
>
> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>> TL;DR
>>> =====
>>>
>>> This is about how QEMU developers can get started with functional
>>> tests that are built on top of the Avocado libraries (and meant to be
>>> run with the Avocado test runner).
>>>
>>> The past
>>> ========
>>>
>>> The Avocado project[1] has been working, for quite some time now, on a
>>> "set of tools and libraries" with the goal of making writing tests
>>> easier.  It is supposed to be a framework agnostic to the exact
>>> software that will be under test.
>>>
>>> But, at the same time, the Avocado project cannot deny its inheritance
>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>> as its largest and most developed "test".  This large Autotest test
>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>> together with its QEMU test provider[5].
>>>
>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>> across platform and QEMU versions, which increases its complexity.
>>> Also, it's built on a legacy set of principles and tools that makes
>>> some developers stir away from it.
>>>
>>> What's new?
>>> ===========
>>>
>>> A few months ago, the Avocado developers returned to its
>>> "virtualization origins", in an attempt to learn from the QEMU
>>> project, and try to help with a way to have more functional tests in
>>> the upstream QEMU repo.
>>>
>>> We believe it's possible to expand the test coverage for QEMU by
>>> facilitating
>>> the creation of more functional tests QEMU.  This is no different than how
>>> other types of tests are already included in the tree itself.
>>>
>>> How
>>> ===
>>>
>>> How we did it (so far)
>>> ----------------------
>>>
>>> We're aware that there's a dilemma here: to be able to easily write
>>> more powerful tests, a lot of the complexity has to be moved
>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>> framework.  The QEMU source tree itself has proofs of this approach,
>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>
>>> Avocado itself[1] provides a lot of the code that should help to
>>> absorb some of the complexities in writing tests, but not exactly
>>> everything that is needed for QEMU.  The approach we believe will have
>>> the best balance is to reuse upstream Avocado libraries whenever they
>>> are useful and generic enough, and on top of that, libraries that are
>>> part of QEMU itself.
>>>
>>> How can you get started with it
>>> -------------------------------
>>>
>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>> itself, this will give you the basic libraries on which the other part
>>> of this work was built on.  We want that to be simple and painless, so
>>> here's our best bet for a one-liner installation:
>>>
>>>   pip install --user avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> That will install Avocado within the user's home directory.  If you
>>> give up on it, it can be uninstalled with another simple one-liner:
>>>
>>>   pip uninstall -y avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> Now, suppose you're working on a given feature, and want to try your
>>> luck writing a test using this work.  To avoid having you fetching and
>>> rebasing from our currently in development fork[6] and branch[7], you
>>> can just
>>> add one commit to your tree with:
>>>
>>>   curl
>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>> git am -
>>>
>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>> do it "the git way", fetching from that repo[6] and using the
>>> non-snapshotted branch.
>>>
>>> After that, we'd love for you to take a look at some of the existing
>>> tests[9][10] and then attempt to create test for your own use case.
>>> The basic README[11] file, and the Avocado documentation[12] are also
>>> important resources not to be missed.
>>>
>>> What's next?
>>> ============
>>>
>>> Initially, feedback is what we're looking for.  It would be greatly
>>> appreciated to understand if/how this suits (or not) use cases out
>>> there.
>>>
>>> After feedback, further refinements, and more tests are written, the
>>> Avocado developers will follow up with an initial patch series for
>>> upstream QEMU.  In such a proposal, we intend to have further
>>> integration.  Ideally in way that "configure" can be given a
>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>  [functional-]check" would seamlessly include them.
>>
>> I have a few thoughts.
>>
>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>> output (no interaction with the QEMU source tree) and I think it is
>> extremely useful. So I am all for using Python to test things and this
>> looks really well done!
>>
>
> Thanks for checking it out, and for the positive words.  Now, sorry if
> I'm missing some obvious information, but is this work of yours with
> pytest/pexpect publicly available?  I'd like to also take a look at
> that, because it does look similar to the Avocado + aexpect approach
> taken here.

Unfortunately it's not and it would take months for us to be able to
make it available.

>
>> What I don't understand though is what this gives us compared to the
>> existing QEMU test infrastructure? Besides being able to use Python
>> and a better interface what are the main benefits? I think that is
>> something worth documenting somewhere.
>>
>
> We currently intend to *add* to the QEMU test infrastructure, not
> replace it.

Is there a benefit of integrating it into the tree then? It's always
possible to have an out of tree testing framework.

>
> The benefits we envision are, besides hopefully easier and more capable
> interfaces, to simply have more upstream tests.  This means avoiding new
> regressions and improving coverage.
>
>> Also, it looks like this will require images checked into git
>> somewhere is that correct? Is there a good plan on how to handle that?
>>
>
> It won't require images checked into git.  Right now, tests use the
> vmimage library:
>
> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>
> Which downloads (and caches) images from external sources.

Ah! That's cool. Managing images is one of the challenges we have at the moment.

Alistair

>
> Please let me know if you have more questions!
> - Cleber.
>
>> Alistair
>>
>>>
>>> Thanks!
>>>
>>> References
>>> ==========
>>>
>>>  [1] http://avocado-framework.github.io/
>>>  [2] http://autotest.github.io/
>>>  [3] https://github.com/autotest/virt-test
>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>  [5] https://github.com/autotest/tp-qemu
>>>  [6] https://github.com/apahim/qemu
>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>  [9]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>> [10]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>> [11]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>> [12] http://avocado-framework.readthedocs.io/
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Cleber Rosa]

On 02/01/2018 07:10 PM, Alistair Francis wrote:
> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <crosa@redhat.com> wrote:
>>
>>
>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>>> TL;DR
>>>> =====
>>>>
>>>> This is about how QEMU developers can get started with functional
>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>> run with the Avocado test runner).
>>>>
>>>> The past
>>>> ========
>>>>
>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>> "set of tools and libraries" with the goal of making writing tests
>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>> software that will be under test.
>>>>
>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>> as its largest and most developed "test".  This large Autotest test
>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>> together with its QEMU test provider[5].
>>>>
>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>> across platform and QEMU versions, which increases its complexity.
>>>> Also, it's built on a legacy set of principles and tools that makes
>>>> some developers stir away from it.
>>>>
>>>> What's new?
>>>> ===========
>>>>
>>>> A few months ago, the Avocado developers returned to its
>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>> project, and try to help with a way to have more functional tests in
>>>> the upstream QEMU repo.
>>>>
>>>> We believe it's possible to expand the test coverage for QEMU by
>>>> facilitating
>>>> the creation of more functional tests QEMU.  This is no different than how
>>>> other types of tests are already included in the tree itself.
>>>>
>>>> How
>>>> ===
>>>>
>>>> How we did it (so far)
>>>> ----------------------
>>>>
>>>> We're aware that there's a dilemma here: to be able to easily write
>>>> more powerful tests, a lot of the complexity has to be moved
>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>
>>>> Avocado itself[1] provides a lot of the code that should help to
>>>> absorb some of the complexities in writing tests, but not exactly
>>>> everything that is needed for QEMU.  The approach we believe will have
>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>> are useful and generic enough, and on top of that, libraries that are
>>>> part of QEMU itself.
>>>>
>>>> How can you get started with it
>>>> -------------------------------
>>>>
>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>> itself, this will give you the basic libraries on which the other part
>>>> of this work was built on.  We want that to be simple and painless, so
>>>> here's our best bet for a one-liner installation:
>>>>
>>>>   pip install --user avocado-framework
>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>
>>>> That will install Avocado within the user's home directory.  If you
>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>
>>>>   pip uninstall -y avocado-framework
>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>
>>>> Now, suppose you're working on a given feature, and want to try your
>>>> luck writing a test using this work.  To avoid having you fetching and
>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>> can just
>>>> add one commit to your tree with:
>>>>
>>>>   curl
>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>> git am -
>>>>
>>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>>> do it "the git way", fetching from that repo[6] and using the
>>>> non-snapshotted branch.
>>>>
>>>> After that, we'd love for you to take a look at some of the existing
>>>> tests[9][10] and then attempt to create test for your own use case.
>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>> important resources not to be missed.
>>>>
>>>> What's next?
>>>> ============
>>>>
>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>> appreciated to understand if/how this suits (or not) use cases out
>>>> there.
>>>>
>>>> After feedback, further refinements, and more tests are written, the
>>>> Avocado developers will follow up with an initial patch series for
>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>> integration.  Ideally in way that "configure" can be given a
>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>  [functional-]check" would seamlessly include them.
>>>
>>> I have a few thoughts.
>>>
>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>> output (no interaction with the QEMU source tree) and I think it is
>>> extremely useful. So I am all for using Python to test things and this
>>> looks really well done!
>>>
>>
>> Thanks for checking it out, and for the positive words.  Now, sorry if
>> I'm missing some obvious information, but is this work of yours with
>> pytest/pexpect publicly available?  I'd like to also take a look at
>> that, because it does look similar to the Avocado + aexpect approach
>> taken here.
> 
> Unfortunately it's not and it would take months for us to be able to
> make it available.
> 

I see.

>>
>>> What I don't understand though is what this gives us compared to the
>>> existing QEMU test infrastructure? Besides being able to use Python
>>> and a better interface what are the main benefits? I think that is
>>> something worth documenting somewhere.
>>>
>>
>> We currently intend to *add* to the QEMU test infrastructure, not
>> replace it.
> 
> Is there a benefit of integrating it into the tree then? It's always
> possible to have an out of tree testing framework.
> 

Upstream first is just the general modus operandi that we have.  Then,
we want to have as much testing as possible as early as possible.

Finally, by having tests in-tree, they can be seen in different light.
What I mean is that when tests are in-tree, they will (or should) really
map to the current state of the code they test (per commit).  If a
feature changes behavior upstream, the respective test will also need a
change at the same time, and as such, will mean an *intended* change and
not a regression.  With out-of-tree tests, it's pretty hard to keep this
synchronization and regressions will slip in, *at least* for some time.

I guess the question here is actually the opposite: do you see any
problems with in-tree tests?  Since you keep out of tree tests, I would
honestly like to hear about your experience.

>>
>> The benefits we envision are, besides hopefully easier and more capable
>> interfaces, to simply have more upstream tests.  This means avoiding new
>> regressions and improving coverage.
>>
>>> Also, it looks like this will require images checked into git
>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>
>>
>> It won't require images checked into git.  Right now, tests use the
>> vmimage library:
>>
>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>
>> Which downloads (and caches) images from external sources.
> 
> Ah! That's cool. Managing images is one of the challenges we have at the moment.
> 
> Alistair
> 

Nice that you like it.  We also find a library such as "vmimage" is
something simple enough, but still not quite something that would live
in the QEMU tree.

- Cleber.

>>
>> Please let me know if you have more questions!
>> - Cleber.
>>
>>> Alistair
>>>
>>>>
>>>> Thanks!
>>>>
>>>> References
>>>> ==========
>>>>
>>>>  [1] http://avocado-framework.github.io/
>>>>  [2] http://autotest.github.io/
>>>>  [3] https://github.com/autotest/virt-test
>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>  [5] https://github.com/autotest/tp-qemu
>>>>  [6] https://github.com/apahim/qemu
>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>  [9]
>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>> [10]
>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>> [11]
>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>> [12] http://avocado-framework.readthedocs.io/
>>>>
>>>> --
>>>> Cleber Rosa
>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>
>>
>> --
>> Cleber Rosa
>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>> [ Avocado Test Framework - avocado-framework.github.io ]
>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Mon, Feb 5, 2018 at 8:34 AM, Cleber Rosa <crosa@redhat.com> wrote:
>
>
> On 02/01/2018 07:10 PM, Alistair Francis wrote:
>> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <crosa@redhat.com> wrote:
>>>
>>>
>>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>>>> TL;DR
>>>>> =====
>>>>>
>>>>> This is about how QEMU developers can get started with functional
>>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>>> run with the Avocado test runner).
>>>>>
>>>>> The past
>>>>> ========
>>>>>
>>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>>> "set of tools and libraries" with the goal of making writing tests
>>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>>> software that will be under test.
>>>>>
>>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>>> as its largest and most developed "test".  This large Autotest test
>>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>>> together with its QEMU test provider[5].
>>>>>
>>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>>> across platform and QEMU versions, which increases its complexity.
>>>>> Also, it's built on a legacy set of principles and tools that makes
>>>>> some developers stir away from it.
>>>>>
>>>>> What's new?
>>>>> ===========
>>>>>
>>>>> A few months ago, the Avocado developers returned to its
>>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>>> project, and try to help with a way to have more functional tests in
>>>>> the upstream QEMU repo.
>>>>>
>>>>> We believe it's possible to expand the test coverage for QEMU by
>>>>> facilitating
>>>>> the creation of more functional tests QEMU.  This is no different than how
>>>>> other types of tests are already included in the tree itself.
>>>>>
>>>>> How
>>>>> ===
>>>>>
>>>>> How we did it (so far)
>>>>> ----------------------
>>>>>
>>>>> We're aware that there's a dilemma here: to be able to easily write
>>>>> more powerful tests, a lot of the complexity has to be moved
>>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>>
>>>>> Avocado itself[1] provides a lot of the code that should help to
>>>>> absorb some of the complexities in writing tests, but not exactly
>>>>> everything that is needed for QEMU.  The approach we believe will have
>>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>>> are useful and generic enough, and on top of that, libraries that are
>>>>> part of QEMU itself.
>>>>>
>>>>> How can you get started with it
>>>>> -------------------------------
>>>>>
>>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>>> itself, this will give you the basic libraries on which the other part
>>>>> of this work was built on.  We want that to be simple and painless, so
>>>>> here's our best bet for a one-liner installation:
>>>>>
>>>>>   pip install --user avocado-framework
>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>
>>>>> That will install Avocado within the user's home directory.  If you
>>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>>
>>>>>   pip uninstall -y avocado-framework
>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>
>>>>> Now, suppose you're working on a given feature, and want to try your
>>>>> luck writing a test using this work.  To avoid having you fetching and
>>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>>> can just
>>>>> add one commit to your tree with:
>>>>>
>>>>>   curl
>>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>>> git am -
>>>>>
>>>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>>>> do it "the git way", fetching from that repo[6] and using the
>>>>> non-snapshotted branch.
>>>>>
>>>>> After that, we'd love for you to take a look at some of the existing
>>>>> tests[9][10] and then attempt to create test for your own use case.
>>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>>> important resources not to be missed.
>>>>>
>>>>> What's next?
>>>>> ============
>>>>>
>>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>>> appreciated to understand if/how this suits (or not) use cases out
>>>>> there.
>>>>>
>>>>> After feedback, further refinements, and more tests are written, the
>>>>> Avocado developers will follow up with an initial patch series for
>>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>>> integration.  Ideally in way that "configure" can be given a
>>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>>  [functional-]check" would seamlessly include them.
>>>>
>>>> I have a few thoughts.
>>>>
>>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>>> output (no interaction with the QEMU source tree) and I think it is
>>>> extremely useful. So I am all for using Python to test things and this
>>>> looks really well done!
>>>>
>>>
>>> Thanks for checking it out, and for the positive words.  Now, sorry if
>>> I'm missing some obvious information, but is this work of yours with
>>> pytest/pexpect publicly available?  I'd like to also take a look at
>>> that, because it does look similar to the Avocado + aexpect approach
>>> taken here.
>>
>> Unfortunately it's not and it would take months for us to be able to
>> make it available.
>>
>
> I see.
>
>>>
>>>> What I don't understand though is what this gives us compared to the
>>>> existing QEMU test infrastructure? Besides being able to use Python
>>>> and a better interface what are the main benefits? I think that is
>>>> something worth documenting somewhere.
>>>>
>>>
>>> We currently intend to *add* to the QEMU test infrastructure, not
>>> replace it.
>>
>> Is there a benefit of integrating it into the tree then? It's always
>> possible to have an out of tree testing framework.
>>
>
> Upstream first is just the general modus operandi that we have.  Then,
> we want to have as much testing as possible as early as possible.
>
> Finally, by having tests in-tree, they can be seen in different light.
> What I mean is that when tests are in-tree, they will (or should) really
> map to the current state of the code they test (per commit).  If a
> feature changes behavior upstream, the respective test will also need a
> change at the same time, and as such, will mean an *intended* change and
> not a regression.  With out-of-tree tests, it's pretty hard to keep this
> synchronization and regressions will slip in, *at least* for some time.
>
> I guess the question here is actually the opposite: do you see any
> problems with in-tree tests?  Since you keep out of tree tests, I would
> honestly like to hear about your experience.

For us the main reason is that we want to avoid editing the fork as
much as possible, so we aren't going to add tests to our fork.

The other main advantage I see is that it allows the tests to be
updated more easily as the changes don't need to go through the more
rigorous QEMU review process.

You are correct though that synchronisation is an issue.

Alistair

>
>>>
>>> The benefits we envision are, besides hopefully easier and more capable
>>> interfaces, to simply have more upstream tests.  This means avoiding new
>>> regressions and improving coverage.
>>>
>>>> Also, it looks like this will require images checked into git
>>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>>
>>>
>>> It won't require images checked into git.  Right now, tests use the
>>> vmimage library:
>>>
>>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>>
>>> Which downloads (and caches) images from external sources.
>>
>> Ah! That's cool. Managing images is one of the challenges we have at the moment.
>>
>> Alistair
>>
>
> Nice that you like it.  We also find a library such as "vmimage" is
> something simple enough, but still not quite something that would live
> in the QEMU tree.
>
> - Cleber.
>
>>>
>>> Please let me know if you have more questions!
>>> - Cleber.
>>>
>>>> Alistair
>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>> References
>>>>> ==========
>>>>>
>>>>>  [1] http://avocado-framework.github.io/
>>>>>  [2] http://autotest.github.io/
>>>>>  [3] https://github.com/autotest/virt-test
>>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>>  [5] https://github.com/autotest/tp-qemu
>>>>>  [6] https://github.com/apahim/qemu
>>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>>  [9]
>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>>> [10]
>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>>> [11]
>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>>> [12] http://avocado-framework.readthedocs.io/
>>>>>
>>>>> --
>>>>> Cleber Rosa
>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>>
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>
> --
> Cleber Rosa
> [ Sr Software Engineer - Virtualization Team - Red Hat ]
> [ Avocado Test Framework - avocado-framework.github.io ]
> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Thu, Feb 8, 2018 at 3:38 PM, Alistair Francis <alistair23@gmail.com> wrote:
> On Mon, Feb 5, 2018 at 8:34 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>
>>
>> On 02/01/2018 07:10 PM, Alistair Francis wrote:
>>> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <crosa@redhat.com> wrote:
>>>>
>>>>
>>>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>>>>> TL;DR
>>>>>> =====
>>>>>>
>>>>>> This is about how QEMU developers can get started with functional
>>>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>>>> run with the Avocado test runner).
>>>>>>
>>>>>> The past
>>>>>> ========
>>>>>>
>>>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>>>> "set of tools and libraries" with the goal of making writing tests
>>>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>>>> software that will be under test.
>>>>>>
>>>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>>>> as its largest and most developed "test".  This large Autotest test
>>>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>>>> together with its QEMU test provider[5].
>>>>>>
>>>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>>>> across platform and QEMU versions, which increases its complexity.
>>>>>> Also, it's built on a legacy set of principles and tools that makes
>>>>>> some developers stir away from it.
>>>>>>
>>>>>> What's new?
>>>>>> ===========
>>>>>>
>>>>>> A few months ago, the Avocado developers returned to its
>>>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>>>> project, and try to help with a way to have more functional tests in
>>>>>> the upstream QEMU repo.
>>>>>>
>>>>>> We believe it's possible to expand the test coverage for QEMU by
>>>>>> facilitating
>>>>>> the creation of more functional tests QEMU.  This is no different than how
>>>>>> other types of tests are already included in the tree itself.
>>>>>>
>>>>>> How
>>>>>> ===
>>>>>>
>>>>>> How we did it (so far)
>>>>>> ----------------------
>>>>>>
>>>>>> We're aware that there's a dilemma here: to be able to easily write
>>>>>> more powerful tests, a lot of the complexity has to be moved
>>>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>>>
>>>>>> Avocado itself[1] provides a lot of the code that should help to
>>>>>> absorb some of the complexities in writing tests, but not exactly
>>>>>> everything that is needed for QEMU.  The approach we believe will have
>>>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>>>> are useful and generic enough, and on top of that, libraries that are
>>>>>> part of QEMU itself.
>>>>>>
>>>>>> How can you get started with it
>>>>>> -------------------------------
>>>>>>
>>>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>>>> itself, this will give you the basic libraries on which the other part
>>>>>> of this work was built on.  We want that to be simple and painless, so
>>>>>> here's our best bet for a one-liner installation:
>>>>>>
>>>>>>   pip install --user avocado-framework
>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>>
>>>>>> That will install Avocado within the user's home directory.  If you
>>>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>>>
>>>>>>   pip uninstall -y avocado-framework
>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>>
>>>>>> Now, suppose you're working on a given feature, and want to try your
>>>>>> luck writing a test using this work.  To avoid having you fetching and
>>>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>>>> can just
>>>>>> add one commit to your tree with:
>>>>>>
>>>>>>   curl
>>>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>>>> git am -
>>>>>>
>>>>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>>>>> do it "the git way", fetching from that repo[6] and using the
>>>>>> non-snapshotted branch.
>>>>>>
>>>>>> After that, we'd love for you to take a look at some of the existing
>>>>>> tests[9][10] and then attempt to create test for your own use case.
>>>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>>>> important resources not to be missed.
>>>>>>
>>>>>> What's next?
>>>>>> ============
>>>>>>
>>>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>>>> appreciated to understand if/how this suits (or not) use cases out
>>>>>> there.
>>>>>>
>>>>>> After feedback, further refinements, and more tests are written, the
>>>>>> Avocado developers will follow up with an initial patch series for
>>>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>>>> integration.  Ideally in way that "configure" can be given a
>>>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>>>  [functional-]check" would seamlessly include them.
>>>>>
>>>>> I have a few thoughts.
>>>>>
>>>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>>>> output (no interaction with the QEMU source tree) and I think it is
>>>>> extremely useful. So I am all for using Python to test things and this
>>>>> looks really well done!
>>>>>
>>>>
>>>> Thanks for checking it out, and for the positive words.  Now, sorry if
>>>> I'm missing some obvious information, but is this work of yours with
>>>> pytest/pexpect publicly available?  I'd like to also take a look at
>>>> that, because it does look similar to the Avocado + aexpect approach
>>>> taken here.
>>>
>>> Unfortunately it's not and it would take months for us to be able to
>>> make it available.
>>>
>>
>> I see.
>>
>>>>
>>>>> What I don't understand though is what this gives us compared to the
>>>>> existing QEMU test infrastructure? Besides being able to use Python
>>>>> and a better interface what are the main benefits? I think that is
>>>>> something worth documenting somewhere.
>>>>>
>>>>
>>>> We currently intend to *add* to the QEMU test infrastructure, not
>>>> replace it.
>>>
>>> Is there a benefit of integrating it into the tree then? It's always
>>> possible to have an out of tree testing framework.
>>>
>>
>> Upstream first is just the general modus operandi that we have.  Then,
>> we want to have as much testing as possible as early as possible.
>>
>> Finally, by having tests in-tree, they can be seen in different light.
>> What I mean is that when tests are in-tree, they will (or should) really
>> map to the current state of the code they test (per commit).  If a
>> feature changes behavior upstream, the respective test will also need a
>> change at the same time, and as such, will mean an *intended* change and
>> not a regression.  With out-of-tree tests, it's pretty hard to keep this
>> synchronization and regressions will slip in, *at least* for some time.
>>
>> I guess the question here is actually the opposite: do you see any
>> problems with in-tree tests?  Since you keep out of tree tests, I would
>> honestly like to hear about your experience.
>
> For us the main reason is that we want to avoid editing the fork as
> much as possible, so we aren't going to add tests to our fork.
>
> The other main advantage I see is that it allows the tests to be
> updated more easily as the changes don't need to go through the more
> rigorous QEMU review process.
>
> You are correct though that synchronisation is an issue.

Actually another thing we can do is run our tests on a known working
QEMU version with new/different guest images to test other teams work.
For example we can find Linux bugs by running the latest internal
Linux kernel builds on our known working master QEMU.

That might be harder to setup and justify with an in tree framework,
but isn't really a justification to make it out of tree.

>
> Alistair
>
>>
>>>>
>>>> The benefits we envision are, besides hopefully easier and more capable
>>>> interfaces, to simply have more upstream tests.  This means avoiding new
>>>> regressions and improving coverage.
>>>>
>>>>> Also, it looks like this will require images checked into git
>>>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>>>
>>>>
>>>> It won't require images checked into git.  Right now, tests use the
>>>> vmimage library:
>>>>
>>>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>>>
>>>> Which downloads (and caches) images from external sources.
>>>
>>> Ah! That's cool. Managing images is one of the challenges we have at the moment.
>>>
>>> Alistair
>>>
>>
>> Nice that you like it.  We also find a library such as "vmimage" is
>> something simple enough, but still not quite something that would live
>> in the QEMU tree.
>>
>> - Cleber.
>>
>>>>
>>>> Please let me know if you have more questions!
>>>> - Cleber.
>>>>
>>>>> Alistair
>>>>>
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> References
>>>>>> ==========
>>>>>>
>>>>>>  [1] http://avocado-framework.github.io/
>>>>>>  [2] http://autotest.github.io/
>>>>>>  [3] https://github.com/autotest/virt-test
>>>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>>>  [5] https://github.com/autotest/tp-qemu
>>>>>>  [6] https://github.com/apahim/qemu
>>>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>>>  [9]
>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>>>> [10]
>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>>>> [11]
>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>>>> [12] http://avocado-framework.readthedocs.io/
>>>>>>
>>>>>> --
>>>>>> Cleber Rosa
>>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>>>
>>>>
>>>> --
>>>> Cleber Rosa
>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>
>> --
>> Cleber Rosa
>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>> [ Avocado Test Framework - avocado-framework.github.io ]
>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Lukáš Doktor]

[-- Attachment #1: Type: text/plain, Size: 11956 bytes --]

Dne 9.2.2018 v 00:41 Alistair Francis napsal(a):
> On Thu, Feb 8, 2018 at 3:38 PM, Alistair Francis <alistair23@gmail.com> wrote:
>> On Mon, Feb 5, 2018 at 8:34 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>>
>>>
>>> On 02/01/2018 07:10 PM, Alistair Francis wrote:
>>>> On Wed, Jan 17, 2018 at 4:47 PM, Cleber Rosa <crosa@redhat.com> wrote:
>>>>>
>>>>>
>>>>> On 01/17/2018 06:41 PM, Alistair Francis wrote:
>>>>>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>>>>>> TL;DR
>>>>>>> =====
>>>>>>>
>>>>>>> This is about how QEMU developers can get started with functional
>>>>>>> tests that are built on top of the Avocado libraries (and meant to be
>>>>>>> run with the Avocado test runner).
>>>>>>>
>>>>>>> The past
>>>>>>> ========
>>>>>>>
>>>>>>> The Avocado project[1] has been working, for quite some time now, on a
>>>>>>> "set of tools and libraries" with the goal of making writing tests
>>>>>>> easier.  It is supposed to be a framework agnostic to the exact
>>>>>>> software that will be under test.
>>>>>>>
>>>>>>> But, at the same time, the Avocado project cannot deny its inheritance
>>>>>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>>>>>> as its largest and most developed "test".  This large Autotest test
>>>>>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>>>>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>>>>>> together with its QEMU test provider[5].
>>>>>>>
>>>>>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>>>>>> across platform and QEMU versions, which increases its complexity.
>>>>>>> Also, it's built on a legacy set of principles and tools that makes
>>>>>>> some developers stir away from it.
>>>>>>>
>>>>>>> What's new?
>>>>>>> ===========
>>>>>>>
>>>>>>> A few months ago, the Avocado developers returned to its
>>>>>>> "virtualization origins", in an attempt to learn from the QEMU
>>>>>>> project, and try to help with a way to have more functional tests in
>>>>>>> the upstream QEMU repo.
>>>>>>>
>>>>>>> We believe it's possible to expand the test coverage for QEMU by
>>>>>>> facilitating
>>>>>>> the creation of more functional tests QEMU.  This is no different than how
>>>>>>> other types of tests are already included in the tree itself.
>>>>>>>
>>>>>>> How
>>>>>>> ===
>>>>>>>
>>>>>>> How we did it (so far)
>>>>>>> ----------------------
>>>>>>>
>>>>>>> We're aware that there's a dilemma here: to be able to easily write
>>>>>>> more powerful tests, a lot of the complexity has to be moved
>>>>>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>>>>>> framework.  The QEMU source tree itself has proofs of this approach,
>>>>>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>>>>>
>>>>>>> Avocado itself[1] provides a lot of the code that should help to
>>>>>>> absorb some of the complexities in writing tests, but not exactly
>>>>>>> everything that is needed for QEMU.  The approach we believe will have
>>>>>>> the best balance is to reuse upstream Avocado libraries whenever they
>>>>>>> are useful and generic enough, and on top of that, libraries that are
>>>>>>> part of QEMU itself.
>>>>>>>
>>>>>>> How can you get started with it
>>>>>>> -------------------------------
>>>>>>>
>>>>>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>>>>>> itself, this will give you the basic libraries on which the other part
>>>>>>> of this work was built on.  We want that to be simple and painless, so
>>>>>>> here's our best bet for a one-liner installation:
>>>>>>>
>>>>>>>   pip install --user avocado-framework
>>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>>>
>>>>>>> That will install Avocado within the user's home directory.  If you
>>>>>>> give up on it, it can be uninstalled with another simple one-liner:
>>>>>>>
>>>>>>>   pip uninstall -y avocado-framework
>>>>>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>>>>>
>>>>>>> Now, suppose you're working on a given feature, and want to try your
>>>>>>> luck writing a test using this work.  To avoid having you fetching and
>>>>>>> rebasing from our currently in development fork[6] and branch[7], you
>>>>>>> can just
>>>>>>> add one commit to your tree with:
>>>>>>>
>>>>>>>   curl
>>>>>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>>>>>> git am -
>>>>>>>
>>>>>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>>>>>> do it "the git way", fetching from that repo[6] and using the
>>>>>>> non-snapshotted branch.
>>>>>>>
>>>>>>> After that, we'd love for you to take a look at some of the existing
>>>>>>> tests[9][10] and then attempt to create test for your own use case.
>>>>>>> The basic README[11] file, and the Avocado documentation[12] are also
>>>>>>> important resources not to be missed.
>>>>>>>
>>>>>>> What's next?
>>>>>>> ============
>>>>>>>
>>>>>>> Initially, feedback is what we're looking for.  It would be greatly
>>>>>>> appreciated to understand if/how this suits (or not) use cases out
>>>>>>> there.
>>>>>>>
>>>>>>> After feedback, further refinements, and more tests are written, the
>>>>>>> Avocado developers will follow up with an initial patch series for
>>>>>>> upstream QEMU.  In such a proposal, we intend to have further
>>>>>>> integration.  Ideally in way that "configure" can be given a
>>>>>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>>>>>  [functional-]check" would seamlessly include them.
>>>>>>
>>>>>> I have a few thoughts.
>>>>>>
>>>>>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>>>>>> output (no interaction with the QEMU source tree) and I think it is
>>>>>> extremely useful. So I am all for using Python to test things and this
>>>>>> looks really well done!
>>>>>>
>>>>>
>>>>> Thanks for checking it out, and for the positive words.  Now, sorry if
>>>>> I'm missing some obvious information, but is this work of yours with
>>>>> pytest/pexpect publicly available?  I'd like to also take a look at
>>>>> that, because it does look similar to the Avocado + aexpect approach
>>>>> taken here.
>>>>
>>>> Unfortunately it's not and it would take months for us to be able to
>>>> make it available.
>>>>
>>>
>>> I see.
>>>
>>>>>
>>>>>> What I don't understand though is what this gives us compared to the
>>>>>> existing QEMU test infrastructure? Besides being able to use Python
>>>>>> and a better interface what are the main benefits? I think that is
>>>>>> something worth documenting somewhere.
>>>>>>
>>>>>
>>>>> We currently intend to *add* to the QEMU test infrastructure, not
>>>>> replace it.
>>>>
>>>> Is there a benefit of integrating it into the tree then? It's always
>>>> possible to have an out of tree testing framework.
>>>>
>>>
>>> Upstream first is just the general modus operandi that we have.  Then,
>>> we want to have as much testing as possible as early as possible.
>>>
>>> Finally, by having tests in-tree, they can be seen in different light.
>>> What I mean is that when tests are in-tree, they will (or should) really
>>> map to the current state of the code they test (per commit).  If a
>>> feature changes behavior upstream, the respective test will also need a
>>> change at the same time, and as such, will mean an *intended* change and
>>> not a regression.  With out-of-tree tests, it's pretty hard to keep this
>>> synchronization and regressions will slip in, *at least* for some time.
>>>
>>> I guess the question here is actually the opposite: do you see any
>>> problems with in-tree tests?  Since you keep out of tree tests, I would
>>> honestly like to hear about your experience.
>>
>> For us the main reason is that we want to avoid editing the fork as
>> much as possible, so we aren't going to add tests to our fork.
>>
>> The other main advantage I see is that it allows the tests to be
>> updated more easily as the changes don't need to go through the more
>> rigorous QEMU review process.
>>
>> You are correct though that synchronisation is an issue.
> 
> Actually another thing we can do is run our tests on a known working
> QEMU version with new/different guest images to test other teams work.
> For example we can find Linux bugs by running the latest internal
> Linux kernel builds on our known working master QEMU.
> 
> That might be harder to setup and justify with an in tree framework,
> but isn't really a justification to make it out of tree.
> 

Sure, but it means your tests are written the way they work with different qemu versions. For those we have `avocado-vt` with lots of tests usually written by QA guys (as Avocado-vt is not easy to start with and does not attract many qemu developers). The beauty of Avocado_qemu is it's suppose to be very simple to start with to allow writing simple, specific, usually single-purpose tests, especially regression, tests that can be easily attached along with the fix. (at least that's what I see as the main goal)

Regards,
Lukáš

>>
>> Alistair
>>
>>>
>>>>>
>>>>> The benefits we envision are, besides hopefully easier and more capable
>>>>> interfaces, to simply have more upstream tests.  This means avoiding new
>>>>> regressions and improving coverage.
>>>>>
>>>>>> Also, it looks like this will require images checked into git
>>>>>> somewhere is that correct? Is there a good plan on how to handle that?
>>>>>>
>>>>>
>>>>> It won't require images checked into git.  Right now, tests use the
>>>>> vmimage library:
>>>>>
>>>>> http://avocado-framework.readthedocs.io/en/57.0/api/utils/avocado.utils.html#avocado.utils.vmimage.get
>>>>>
>>>>> Which downloads (and caches) images from external sources.
>>>>
>>>> Ah! That's cool. Managing images is one of the challenges we have at the moment.
>>>>
>>>> Alistair
>>>>
>>>
>>> Nice that you like it.  We also find a library such as "vmimage" is
>>> something simple enough, but still not quite something that would live
>>> in the QEMU tree.
>>>
>>> - Cleber.
>>>
>>>>>
>>>>> Please let me know if you have more questions!
>>>>> - Cleber.
>>>>>
>>>>>> Alistair
>>>>>>
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> References
>>>>>>> ==========
>>>>>>>
>>>>>>>  [1] http://avocado-framework.github.io/
>>>>>>>  [2] http://autotest.github.io/
>>>>>>>  [3] https://github.com/autotest/virt-test
>>>>>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>>>>>  [5] https://github.com/autotest/tp-qemu
>>>>>>>  [6] https://github.com/apahim/qemu
>>>>>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>>>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>>>>>  [9]
>>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>>>>>> [10]
>>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>>>>>> [11]
>>>>>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>>>>>> [12] http://avocado-framework.readthedocs.io/
>>>>>>>
>>>>>>> --
>>>>>>> Cleber Rosa
>>>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>>>>>
>>>>>
>>>>> --
>>>>> Cleber Rosa
>>>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 512 bytes --]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Lukáš Doktor]

[-- Attachment #1: Type: text/plain, Size: 7609 bytes --]

Dne 18.1.2018 v 00:41 Alistair Francis napsal(a):
> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>> TL;DR
>> =====
>>
>> This is about how QEMU developers can get started with functional
>> tests that are built on top of the Avocado libraries (and meant to be
>> run with the Avocado test runner).
>>
>> The past
>> ========
>>
>> The Avocado project[1] has been working, for quite some time now, on a
>> "set of tools and libraries" with the goal of making writing tests
>> easier.  It is supposed to be a framework agnostic to the exact
>> software that will be under test.
>>
>> But, at the same time, the Avocado project cannot deny its inheritance
>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>> as its largest and most developed "test".  This large Autotest test
>> (KVM Autotest) became virt-test[3] and later got integrated into
>> Avocado and became Avocado-VT[4] which is quite relevant here,
>> together with its QEMU test provider[5].
>>
>> Avocado-VT and the QEMU test provider attempt to provide coverage
>> across platform and QEMU versions, which increases its complexity.
>> Also, it's built on a legacy set of principles and tools that makes
>> some developers stir away from it.
>>
>> What's new?
>> ===========
>>
>> A few months ago, the Avocado developers returned to its
>> "virtualization origins", in an attempt to learn from the QEMU
>> project, and try to help with a way to have more functional tests in
>> the upstream QEMU repo.
>>
>> We believe it's possible to expand the test coverage for QEMU by
>> facilitating
>> the creation of more functional tests QEMU.  This is no different than how
>> other types of tests are already included in the tree itself.
>>
>> How
>> ===
>>
>> How we did it (so far)
>> ----------------------
>>
>> We're aware that there's a dilemma here: to be able to easily write
>> more powerful tests, a lot of the complexity has to be moved
>> elsewhere.  Here, it means moving complexity from the test itself to a
>> framework.  The QEMU source tree itself has proofs of this approach,
>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>
>> Avocado itself[1] provides a lot of the code that should help to
>> absorb some of the complexities in writing tests, but not exactly
>> everything that is needed for QEMU.  The approach we believe will have
>> the best balance is to reuse upstream Avocado libraries whenever they
>> are useful and generic enough, and on top of that, libraries that are
>> part of QEMU itself.
>>
>> How can you get started with it
>> -------------------------------
>>
>> First of all, get Avocado installed.  Besides the Avocado test runner
>> itself, this will give you the basic libraries on which the other part
>> of this work was built on.  We want that to be simple and painless, so
>> here's our best bet for a one-liner installation:
>>
>>   pip install --user avocado-framework
>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>
>> That will install Avocado within the user's home directory.  If you
>> give up on it, it can be uninstalled with another simple one-liner:
>>
>>   pip uninstall -y avocado-framework
>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>
>> Now, suppose you're working on a given feature, and want to try your
>> luck writing a test using this work.  To avoid having you fetching and
>> rebasing from our currently in development fork[6] and branch[7], you
>> can just
>> add one commit to your tree with:
>>
>>   curl
>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>> git am -
>>
>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>> do it "the git way", fetching from that repo[6] and using the
>> non-snapshotted branch.
>>
>> After that, we'd love for you to take a look at some of the existing
>> tests[9][10] and then attempt to create test for your own use case.
>> The basic README[11] file, and the Avocado documentation[12] are also
>> important resources not to be missed.
>>
>> What's next?
>> ============
>>
>> Initially, feedback is what we're looking for.  It would be greatly
>> appreciated to understand if/how this suits (or not) use cases out
>> there.
>>
>> After feedback, further refinements, and more tests are written, the
>> Avocado developers will follow up with an initial patch series for
>> upstream QEMU.  In such a proposal, we intend to have further
>> integration.  Ideally in way that "configure" can be given a
>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>  [functional-]check" would seamlessly include them.
> 
> I have a few thoughts.
> 
> We use pytest/pexpect internally to kick off QEMU runs and monitor the
> output (no interaction with the QEMU source tree) and I think it is
> extremely useful. So I am all for using Python to test things and this
> looks really well done!
> 
> What I don't understand though is what this gives us compared to the
> existing QEMU test infrastructure? Besides being able to use Python
> and a better interface what are the main benefits? I think that is
> something worth documenting somewhere.
> 

Hello Alistar,

It was just briefly mentioned by Cleber in the main email, but to be more detailed there are 2 (3) benefits:

1. Using a test runner avoids boiler-plate code to create and provide results. Avocado is a test runner and can unify the execution as well as results. Avocado results are compatible with multiple results DBs/systems and even allows diffing (you can see what changed between different executions).

2. Avocado is not just a test runner, but also set of utils. This can simplify writing tests as one does not have to reinvent the wheel and can simply say "I require this pkg, install it", "I want to start this service" or run commands interactively without the need to worry about setting pty, reading-out the pipes etc.

3. There are at least three Red Hat employees based in virt team who work on Avocado development. This means there is a background to support you the community to simplify writing tests (avocado_qemu "helpers" are one example). In return we expect less regressions as more testing can be performed easily before releases/merges/during development.

Regards,
Lukáš


> Also, it looks like this will require images checked into git
> somewhere is that correct? Is there a good plan on how to handle that?
> 
> Alistair
> 
>>
>> Thanks!
>>
>> References
>> ==========
>>
>>  [1] http://avocado-framework.github.io/
>>  [2] http://autotest.github.io/
>>  [3] https://github.com/autotest/virt-test
>>  [4] https://github.com/avocado-framework/avocado-vt
>>  [5] https://github.com/autotest/tp-qemu
>>  [6] https://github.com/apahim/qemu
>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>  [9]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>> [10]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>> [11]
>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>> [12] http://avocado-framework.readthedocs.io/
>>
>> --
>> Cleber Rosa
>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>> [ Avocado Test Framework - avocado-framework.github.io ]
>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 502 bytes --]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Alistair Francis]

On Thu, Jan 18, 2018 at 1:25 AM, Lukáš Doktor <ldoktor@redhat.com> wrote:
> Dne 18.1.2018 v 00:41 Alistair Francis napsal(a):
>> On Wed, Jan 17, 2018 at 12:05 AM, Cleber Rosa <crosa@redhat.com> wrote:
>>> TL;DR
>>> =====
>>>
>>> This is about how QEMU developers can get started with functional
>>> tests that are built on top of the Avocado libraries (and meant to be
>>> run with the Avocado test runner).
>>>
>>> The past
>>> ========
>>>
>>> The Avocado project[1] has been working, for quite some time now, on a
>>> "set of tools and libraries" with the goal of making writing tests
>>> easier.  It is supposed to be a framework agnostic to the exact
>>> software that will be under test.
>>>
>>> But, at the same time, the Avocado project cannot deny its inheritance
>>> and influences.  Those come from Autotest[2], which had "KVM Autotest"
>>> as its largest and most developed "test".  This large Autotest test
>>> (KVM Autotest) became virt-test[3] and later got integrated into
>>> Avocado and became Avocado-VT[4] which is quite relevant here,
>>> together with its QEMU test provider[5].
>>>
>>> Avocado-VT and the QEMU test provider attempt to provide coverage
>>> across platform and QEMU versions, which increases its complexity.
>>> Also, it's built on a legacy set of principles and tools that makes
>>> some developers stir away from it.
>>>
>>> What's new?
>>> ===========
>>>
>>> A few months ago, the Avocado developers returned to its
>>> "virtualization origins", in an attempt to learn from the QEMU
>>> project, and try to help with a way to have more functional tests in
>>> the upstream QEMU repo.
>>>
>>> We believe it's possible to expand the test coverage for QEMU by
>>> facilitating
>>> the creation of more functional tests QEMU.  This is no different than how
>>> other types of tests are already included in the tree itself.
>>>
>>> How
>>> ===
>>>
>>> How we did it (so far)
>>> ----------------------
>>>
>>> We're aware that there's a dilemma here: to be able to easily write
>>> more powerful tests, a lot of the complexity has to be moved
>>> elsewhere.  Here, it means moving complexity from the test itself to a
>>> framework.  The QEMU source tree itself has proofs of this approach,
>>> being the "scripts" and "tests/qemu-iotests" some of the examples.
>>>
>>> Avocado itself[1] provides a lot of the code that should help to
>>> absorb some of the complexities in writing tests, but not exactly
>>> everything that is needed for QEMU.  The approach we believe will have
>>> the best balance is to reuse upstream Avocado libraries whenever they
>>> are useful and generic enough, and on top of that, libraries that are
>>> part of QEMU itself.
>>>
>>> How can you get started with it
>>> -------------------------------
>>>
>>> First of all, get Avocado installed.  Besides the Avocado test runner
>>> itself, this will give you the basic libraries on which the other part
>>> of this work was built on.  We want that to be simple and painless, so
>>> here's our best bet for a one-liner installation:
>>>
>>>   pip install --user avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> That will install Avocado within the user's home directory.  If you
>>> give up on it, it can be uninstalled with another simple one-liner:
>>>
>>>   pip uninstall -y avocado-framework
>>> avocado-framework-plugin-varianter-yaml-to-mux aexpect
>>>
>>> Now, suppose you're working on a given feature, and want to try your
>>> luck writing a test using this work.  To avoid having you fetching and
>>> rebasing from our currently in development fork[6] and branch[7], you
>>> can just
>>> add one commit to your tree with:
>>>
>>>   curl
>>> https://patch-diff.githubusercontent.com/raw/apahim/qemu/pull/17.patch |
>>> git am -
>>>
>>> This will get a simple patch from a snapshot branch[8].  You can, of course,
>>> do it "the git way", fetching from that repo[6] and using the
>>> non-snapshotted branch.
>>>
>>> After that, we'd love for you to take a look at some of the existing
>>> tests[9][10] and then attempt to create test for your own use case.
>>> The basic README[11] file, and the Avocado documentation[12] are also
>>> important resources not to be missed.
>>>
>>> What's next?
>>> ============
>>>
>>> Initially, feedback is what we're looking for.  It would be greatly
>>> appreciated to understand if/how this suits (or not) use cases out
>>> there.
>>>
>>> After feedback, further refinements, and more tests are written, the
>>> Avocado developers will follow up with an initial patch series for
>>> upstream QEMU.  In such a proposal, we intend to have further
>>> integration.  Ideally in way that "configure" can be given a
>>> "--with-functional-[avocado-]tests" parameter of sorts, and a "make
>>>  [functional-]check" would seamlessly include them.
>>
>> I have a few thoughts.
>>
>> We use pytest/pexpect internally to kick off QEMU runs and monitor the
>> output (no interaction with the QEMU source tree) and I think it is
>> extremely useful. So I am all for using Python to test things and this
>> looks really well done!
>>
>> What I don't understand though is what this gives us compared to the
>> existing QEMU test infrastructure? Besides being able to use Python
>> and a better interface what are the main benefits? I think that is
>> something worth documenting somewhere.
>>
>
> Hello Alistar,
>
> It was just briefly mentioned by Cleber in the main email, but to be more detailed there are 2 (3) benefits:
>
> 1. Using a test runner avoids boiler-plate code to create and provide results. Avocado is a test runner and can unify the execution as well as results. Avocado results are compatible with multiple results DBs/systems and even allows diffing (you can see what changed between different executions).
>
> 2. Avocado is not just a test runner, but also set of utils. This can simplify writing tests as one does not have to reinvent the wheel and can simply say "I require this pkg, install it", "I want to start this service" or run commands interactively without the need to worry about setting pty, reading-out the pipes etc.
>
> 3. There are at least three Red Hat employees based in virt team who work on Avocado development. This means there is a background to support you the community to simplify writing tests (avocado_qemu "helpers" are one example). In return we expect less regressions as more testing can be performed easily before releases/merges/during development.

Cool!

Sorry for the delay here.

If the test doesn't have to integrate with QEMU then does it make
sense to have a separate repo?

The way we handle this kind of testing is we just point to a built
QEMU tree and py.test runs tests on all of the binaries. That way they
can be developed independently, which has advantages and
disadvantages.

I don't have an answer, I was just wondering if you have thought about that.

Alistair

>
> Regards,
> Lukáš
>
>
>> Also, it looks like this will require images checked into git
>> somewhere is that correct? Is there a good plan on how to handle that?
>>
>> Alistair
>>
>>>
>>> Thanks!
>>>
>>> References
>>> ==========
>>>
>>>  [1] http://avocado-framework.github.io/
>>>  [2] http://autotest.github.io/
>>>  [3] https://github.com/autotest/virt-test
>>>  [4] https://github.com/avocado-framework/avocado-vt
>>>  [5] https://github.com/autotest/tp-qemu
>>>  [6] https://github.com/apahim/qemu
>>>  [7] https://github.com/apahim/qemu/tree/avocado_qemu
>>>  [8] https://github.com/apahim/qemu/tree/avocado_qemu_snapshot
>>>  [9]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_info_memdev_host_nodes.py
>>> [10]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/test_ovmf_with_240_vcpus.py
>>> [11]
>>> https://github.com/apahim/qemu/blob/avocado_qemu/tests/avocado/README.rst
>>> [12] http://avocado-framework.readthedocs.io/
>>>
>>> --
>>> Cleber Rosa
>>> [ Sr Software Engineer - Virtualization Team - Red Hat ]
>>> [ Avocado Test Framework - avocado-framework.github.io ]
>>> [  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]
>>>
>
>

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Wed, Jan 17, 2018 at 03:05:54AM -0500, Cleber Rosa wrote:
> This is about how QEMU developers can get started with functional
> tests that are built on top of the Avocado libraries (and meant to be
> run with the Avocado test runner).

Here is my guess at where this fits into QEMU's testing landscape:

Avocado can help us write a new kind of test - functional (acceptance)
tests - that run full-blown guest operating systems, invoke shell
commands inside guests, and also use QMP.  Avocado can probably test
guest OS installation and boot.

There is some overlap with what exists in tests/ today, but I guess it
won't replace check (unit tests), qtests, or qemu-iotests.

Is this how you envision Avocado usage in QEMU?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] Functional tests (AKA Avocado-based tests)

[Cleber Rosa]

On 01/25/2018 09:11 AM, Stefan Hajnoczi wrote:
> On Wed, Jan 17, 2018 at 03:05:54AM -0500, Cleber Rosa wrote:
>> This is about how QEMU developers can get started with functional
>> tests that are built on top of the Avocado libraries (and meant to be
>> run with the Avocado test runner).
> 
> Here is my guess at where this fits into QEMU's testing landscape:
> 
> Avocado can help us write a new kind of test - functional (acceptance)
> tests - that run full-blown guest operating systems, invoke shell
> commands inside guests, and also use QMP.  Avocado can probably test
> guest OS installation and boot.
> 

That is the main goal indeed.  Then at the same time, tests that won't
"run full-brown guest operating systems" should also be easy to
write[1].  For instance, I gave myself a few minutes, literally, to
prove this point and wrote a few VNC related tests[2].

> There is some overlap with what exists in tests/ today, but I guess it
> won't replace check (unit tests), qtests, or qemu-iotests.
> 

There's no *need* whatsoever to replace anything.  People may see value
in using the Avocado test runner to run other tests at some point, but
that's completely unrelated to your original question and our goal.

> Is this how you envision Avocado usage in QEMU?
> 
> Stefan
> 

In short, yes.  The long answer has already been given above.

- Cleber.

[1] - As opposed to say, Avocado-VT, where a lot of pre/post processing,
including guest operating system images, will usually happen by default.

[2] -
https://github.com/clebergnu/qemu/blob/vnc_tests/tests/avocado/test_vnc.py

-- 
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[  7ABB 96EB 8B46 B94D 5E0F  E9BB 657E 8D33 A5F2 09F3  ]