[Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

Hi guys. For a while we have been discussing ways to make the 
virtualization tests written on top of autotest useful for development 
level testing.

One of our main goals is to provide useful tools for the qemu community, 
since we have a good number of tests and libraries written to perform 
integration/QA testing for that tool, being successfuly used by a number 
of QA teams that work on qemu. Also, we recently provided a subset of 
that infrastructure to test libvirt, one of our virtualization projects 
of interest.

We realized that some (admittedly not very radical) changes have to be 
made on autotest itself, so we're inviting other users of autotest to 
give this a good read. This same document lives in the autotest wiki:

https://github.com/autotest/autotest/wiki/FuturePlans

Please note that splitting the virt tests from autotest is not discarded 
at the moment, and it's not incompatible with the plan outlined below.

====================================
Virt tests and autotest future goals
====================================

In order to make the autotest infrastructure, and the virt tests 
developed on top of autotest more useful for the people working on 
developing linux, virt and other platform software, as well as the QA 
teams, we are working on a number of goals, to streamline, simplify and 
make the available tools appropriate for *development level testing*.

Executing tests appropriate for *QA level testing* will continue to be 
supported, as it's one of the biggest strenghts of autotest.

The problem
-----------

Autotest provides as of today a local engine, used to run tests on your 
local machine (your laptop or a remote server). Currently, it runs tests 
properly wrapped and packaged under the autotest client/tests/ folder, 
using specific autotest rules.

For the virt tests that live inside autotest, we have even more rules to 
follow, causing a lot of frustration for people that are not used to how 
things are structured and how to execute and select tests.

The proposed solution
---------------------

A solution is needed for both scenarios (virt and other general purpose 
tests). The idea is to create specialized tools can run simple tests 
without packaging, code that:

  * Knows nothing about the underlying infrastructure
  * Written in any language (shell script, c, perl, you name it)

It'll be up to the test runner to make sense of the results, provided 
that the test writer follows some simple common sense principles while 
writing the code:

  1) Make the program to return 0 on success, !=0 on failure
  2) Make the program use a test output format, mainly TAP

For simple tests, we believe that option 1) will be far more popular. 
Autotest will harness the execution of the test and put the results 
under the test results directory, with all the sysinfo collection and 
other instrumentation transparently available for the user.

At some point, the test writer might want to start the framework 
features that need to be enabled explicitly, then he/she might want to 
learn how to use the python API to do so, but it'll not be a requirement.

More about the test runner
--------------------------

The test runner for both general and virt cases should have very simple 
and minimal output:

::

     Results stored in /path/to/autotest/results/default
     my-full-test.py -- PASS
     my-other-test.py -- PASS
     look-mom-i-can-use-shell.sh -- PASS
     look-mom-i-can-use-perl.pl -- FAIL
     test-name-is-the-description.sh -- PASS
     my-yet-another-test.sh -- SKIPPED
     i-like-python.py -- PASS
     whatever-test.pl -- PASS

Both will be specialized tools that use the infrastructure of
client/bin/autotest-local, but with special code to attend to the output 
spec above. They will know how to handle dependencies, and skip tests if 
needed.

Directory structure
-------------------

This is just to give a rough idea of how we won't depend the tests to be 
in the autotest source code folder:

::

     /path/to/autotest -> top level dir, that will make the autotest 
libs available
      - client/bin -> Contains the test runners and auxiliary scripts
      - client/virt/tests: Contains the virt tests that still live in 
autotest
      - client/tests/kvm/tests: Contains the qemu tests that still live 
in autotest

     /any/path/test1: Contains tests for software foo
     /any/path/test2: Contains tests for software bar

     /any/path/images: Contains minimal guest images for virtualization 
tests

Bootstrap procedure
-------------------

In order to comfortably use the framework features, some bootstrap steps 
will be needed, along the following lines:

::

     git clone git://github.com/autotest/autotest.git /path/to/autotest
     export PATH='/path/to/autotest/client/bin':$PATH
     export PYTHON_PATH='/path/to/autotest':$PYTHON_PATH
     export AUTOTEST_DATA='/path/to/images'

Writing tests
-------------

Simple tests, general case
~~~~~~~~~~~~~~~~~~~~~~~~~~

As previously mentioned, writing a trivial test is as simple as writing 
a program that returns either 0 (true) or any other value (false). 
Autotest returns PASS on true and FAIL on false.

Simple tests, virt case
~~~~~~~~~~~~~~~~~~~~~~~

The difference is that the program might be executed in the guest or the 
host, so a command line flag or environment variable might be set to 
indicate where the program should be executed (host, guest or both). 
Autotest returns PASS on true and FAIL on false. This functionality is 
inspired on qemu-test, thanks to Anthony Liguori.

Instrumented tests, general case
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The test author can learn how to create an autotest wrapper for the test 
suite and use the specialized tool to run it.

Instrumented tests, virt case
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The test author can learn how to create an instrumented test for 
virtualization, using the python APIs, or any other language using 
auxiliary scripts that encapsulate high level functionality for use on 
shell scripts or other languages. Ideas for auxiliary scripts:

::

     virt_run_migration [params] [options]
     virt_run_timedrift [params] [options]
     virt_run_nic_hotplug [params] [options]
     virt_run_block_hotplug [params] [options]

Where tests live
----------------

The tests won't need to be in the autotest tree, they can live anywhere. 
The reason for this is that projects need in tree tests, that can be 
maintained by the project maintainers.

Standard use case for virt is to have both trivial and instrumented 
tests living in the respective project's tree (qemu and libvirt). 
Trivial tests don't need autotest libs, while instrumented tests will 
need to, but that's OK provided that the appropriate bootstrap procedure 
was made.

Test Examples
-------------

simple, non instrumented
~~~~~~~~~~~~~~~~~~~~~~~~

::

     uptime.sh:
         #!/bin/sh
         exec uptime

     uptime.py:
         #!/usr/bin/python
         import os, sys
         sys.exit(os.system("uptime"))

     uptime.pl:
         #!/usr/bin/perl
         system("uptime");
         exit($?);

     qemu-img-convert.sh
         #!/bin/bash
         qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi 
$TEMPDIR/output.qcow2
         diff -b $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2 > 
/dev/null
         ...

uptime.py - instrummented using libautotest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

     #!/bin/python

     from autotest import utils, logging

     def run_uptime_host(test, params, env):
         uptime = utils.system_output("uptime")
         logging.info("Host uptime result is: %s", uptime)


uptime.py - host/guest mode, instrummented using libautotest
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

::

     #!/bin/python

     from autotest import utils, logging

     def run_uptime_host_and_guest(test, params, env):
         vm = env.get_vm(params["main_vm"])
         vm.verify_alive()
         session = vm.wait_for_login()

         uptime_guest = session.cmd("uptime")
         logging.info("Guest uptime result is: %s", uptime_guest)

         uptime_host = utils.system_output("uptime")
         logging.info("Host uptime result is: %s", uptime_host)


Virt/qemu tests: Minimal guest images
-------------------------------------

In order to make development level test possible, we need the tests to 
run fast. In order to do that, a set of minimal guest images is being 
developed and we have a version for x86_64 ready and functional:

https://github.com/autotest/buildroot-autotest

This is a repo based on the buildroot project, that tracks the upstream 
project and contain branches to generate minimal images, for different 
architectures (so far x86_64), so people can reproduce the images 
available here:

http://lmr.fedorapeople.org/jeos_images/

For now, we have a x86_64 image already done and buit:

http://lmr.fedorapeople.org/jeos_images/jeos_x86_64.tar.bz2

This is a 18 MB (bz2 tarball) image, that expands to about ~50 MB, with 
the latest stable linux, busybox, python, ssh and networking, that is 
fairly capable by its size, being able to run a fair amount of testing, 
with small boot times. This functionality is also inspired on qemu-test, 
by Anthony Liguori.

The specialized virt/qemu tool will use these images, downloading them 
if needed to run its tests.

Where we are, how to help
-------------------------

We have a prototype version of the virt specialized tool, that still 
does not implement the output spec, as well as a functional x86_64 guest 
together with a recipe to re-create it. There's still a lot of work to 
do, we'd like your input to help us. The work items are being tracked at 
the label future-vision on the autotest issue tracker:

https://github.com/autotest/autotest/issues?labels=future-vision&sort=created&direction=desc&state=open&page=1

You might want to help us by giving us feedback about this plan. Thank you!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <lmr@redhat.com> wrote:
> One of our main goals is to provide useful tools for the qemu community,
> since we have a good number of tests and libraries written to perform
> integration/QA testing for that tool, being successfuly used by a number of
> QA teams that work on qemu. Also, we recently provided a subset of that
> infrastructure to test libvirt, one of our virtualization projects of
> interest.

Thanks for sharing.

My feeling is that although this proposal opens some interesting
possibilities we're missing the key ingredient to attract people to
autotest as a virtualization test platform.

As a developer who writes tests and would like to test more, my number
one requirement is productivity.  It is frustrating when there is a
test case in my mind but I lack the vocabulary to express it as an
automated test.  In other words, we need an API to that makes it easy
to express virtualization tests.  This is the most important thing for
me and I won't make any other points in this post but will try to
expand more on it.

It shouldn't be a heavyweight framework that imposes constraints on
tests or have a high learning curve, it should be a library that
provides a vocabulary for configuring VMs, interacting with the serial
port, etc.

I think autotest can already do a lot of that today, but I don't know
how.  Actually I kind of do because I've looked at test source but it
doesn't look easy for me to consume.  It hasn't been made accessible
to the level where I know autotest is the right choice.  I worry that
digging into autotest will require significant time investment and
impose constraints on simple test cases I want to write.

I've preferred to write plain Python unittest tests that invoke
qemu-img, qemu-io, and qemu-kvm directly.  It's a pain to write them
from scratch but still quicker than battling with autotest to write a
test every now and then.

What would make me use autotest:
 * A clear productivity win over writing tests from scratch
 * Python API docs for launching a VM, creating disk image,
interacting with the serial port, sending QMP commands, etc

I think these unmet needs are why QEMU has been growing its own test
infrastructure.

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <lmr@redhat.com> wrote:
>> One of our main goals is to provide useful tools for the qemu community,
>> since we have a good number of tests and libraries written to perform
>> integration/QA testing for that tool, being successfuly used by a number of
>> QA teams that work on qemu. Also, we recently provided a subset of that
>> infrastructure to test libvirt, one of our virtualization projects of
>> interest.
>
> Thanks for sharing.

One thing I should have added is that my message is about what it
would take for me to use autotest and contribute tests.  But now I
realize that you might be going for a different model:

If you're aiming for a different model where autotest integrates
external test suites (i.e. tests wouldn't be written in autotest.git,
instead autotest.git would contain snapshots of external test suites),
then this proposal seems fine.  Upstream projects like QEMU would
develop their own test suite and it would be dropped into autotest or
a specific autotest instance.

I'm not 100% sure which of these two models you're going for?

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 11:54:31AM +0000, Stefan Hajnoczi wrote:
> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <lmr@redhat.com> wrote:
> >> One of our main goals is to provide useful tools for the qemu community,
> >> since we have a good number of tests and libraries written to perform
> >> integration/QA testing for that tool, being successfuly used by a number of
> >> QA teams that work on qemu. Also, we recently provided a subset of that
> >> infrastructure to test libvirt, one of our virtualization projects of
> >> interest.
> >
> > Thanks for sharing.
> 
> One thing I should have added is that my message is about what it
> would take for me to use autotest and contribute tests.  But now I
> realize that you might be going for a different model:
> 
> If you're aiming for a different model where autotest integrates
> external test suites (i.e. tests wouldn't be written in autotest.git,
> instead autotest.git would contain snapshots of external test suites),
> then this proposal seems fine.  Upstream projects like QEMU would
> develop their own test suite and it would be dropped into autotest or
> a specific autotest instance.
> 

Yes, that's the idea. We want autotest to be the framework, not
(just a) collection of tests. We also want each development team
to implement and maintain their own set of tests, using (or not)
the goodies from autotest at their discretion.

In summary, autotest is (or is going to be) a framework that
provides:

 - A test runner, with grid/cluster support and advanced
   instrumentation
 - A devel library and set of utilities for test writers
 - A set of pre-built images (JeOS – Just Enough OS) for
   test writers 

(attached is a picture showing what we want to achieve)

If a project has an internal library or set of utilities that can
be of general use, they can be submitted to autotest.git for
inclusion, thus reaching a broader audience.

A short summary of the plans:

 - Tests can live anywhere and each devel team implements and
   maintains their own set of tests
 - Usage of the autotest library by test writers is optional
 - Tests are scripts returning 0 or error (any language)
 - Tests can be run individually or in sets
 - Tests should run fast, our target is seconds or a few minutes
 - The test runner is smart and “just works” by default
   - Trivial standard output (FAIL, PASS, SKIPPED)
   - Collect logs, OS data and other stuff (e.g. --record-video!)
   - Skip unsupported tests based on the environment they're run
   - Multiplex configurations / platforms when on the grid
   - Support to run tests “in the cloud”

There are also lots of small changes and usability improvements
in the pipeline (and feedback right now is very valuable).

Thanks,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

[-- Attachment #1: Type: text/plain, Size: 3256 bytes --]

On Thu, Mar 08, 2012 at 09:17:42AM -0300, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 11:54:31AM +0000, Stefan Hajnoczi wrote:
> > On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues <lmr@redhat.com> wrote:
> > >> One of our main goals is to provide useful tools for the qemu community,
> > >> since we have a good number of tests and libraries written to perform
> > >> integration/QA testing for that tool, being successfuly used by a number of
> > >> QA teams that work on qemu. Also, we recently provided a subset of that
> > >> infrastructure to test libvirt, one of our virtualization projects of
> > >> interest.
> > >
> > > Thanks for sharing.
> > 
> > One thing I should have added is that my message is about what it
> > would take for me to use autotest and contribute tests.  But now I
> > realize that you might be going for a different model:
> > 
> > If you're aiming for a different model where autotest integrates
> > external test suites (i.e. tests wouldn't be written in autotest.git,
> > instead autotest.git would contain snapshots of external test suites),
> > then this proposal seems fine.  Upstream projects like QEMU would
> > develop their own test suite and it would be dropped into autotest or
> > a specific autotest instance.
> > 
> 
> Yes, that's the idea. We want autotest to be the framework, not
> (just a) collection of tests. We also want each development team
> to implement and maintain their own set of tests, using (or not)
> the goodies from autotest at their discretion.
> 
> In summary, autotest is (or is going to be) a framework that
> provides:
> 
>  - A test runner, with grid/cluster support and advanced
>    instrumentation
>  - A devel library and set of utilities for test writers
>  - A set of pre-built images (JeOS – Just Enough OS) for
>    test writers 
> 
> (attached is a picture showing what we want to achieve)

Facepalm right after pressing 'y'.

> 
> If a project has an internal library or set of utilities that can
> be of general use, they can be submitted to autotest.git for
> inclusion, thus reaching a broader audience.
> 
> A short summary of the plans:
> 
>  - Tests can live anywhere and each devel team implements and
>    maintains their own set of tests
>  - Usage of the autotest library by test writers is optional
>  - Tests are scripts returning 0 or error (any language)
>  - Tests can be run individually or in sets
>  - Tests should run fast, our target is seconds or a few minutes
>  - The test runner is smart and “just works” by default
>    - Trivial standard output (FAIL, PASS, SKIPPED)
>    - Collect logs, OS data and other stuff (e.g. --record-video!)
>    - Skip unsupported tests based on the environment they're run
>    - Multiplex configurations / platforms when on the grid
>    - Support to run tests “in the cloud”
> 
> There are also lots of small changes and usability improvements
> in the pipeline (and feedback right now is very valuable).
> 
> Thanks,
>   - Ademar
> 
> -- 
> Ademar de Souza Reis Jr.
> Red Hat
> 
> ^[:wq!

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

[-- Attachment #2: autotest-arch.jpg --]
[-- Type: image/jpeg, Size: 28710 bytes --]

Re: [Qemu-devel] [KVM-AUTOTEST] [RFC] Future goals for autotest and virtualization tests

[Osier Yang]

On 2012年03月08日 20:17, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 11:54:31AM +0000, Stefan Hajnoczi wrote:
>> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi<stefanha@gmail.com>  wrote:
>>> On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues<lmr@redhat.com>  wrote:
>>>> One of our main goals is to provide useful tools for the qemu community,
>>>> since we have a good number of tests and libraries written to perform
>>>> integration/QA testing for that tool, being successfuly used by a number of
>>>> QA teams that work on qemu. Also, we recently provided a subset of that
>>>> infrastructure to test libvirt, one of our virtualization projects of
>>>> interest.
>>>
>>> Thanks for sharing.
>>
>> One thing I should have added is that my message is about what it
>> would take for me to use autotest and contribute tests.  But now I
>> realize that you might be going for a different model:
>>
>> If you're aiming for a different model where autotest integrates
>> external test suites (i.e. tests wouldn't be written in autotest.git,
>> instead autotest.git would contain snapshots of external test suites),
>> then this proposal seems fine.  Upstream projects like QEMU would
>> develop their own test suite and it would be dropped into autotest or
>> a specific autotest instance.
>>
>
> Yes, that's the idea. We want autotest to be the framework, not
> (just a) collection of tests. We also want each development team
> to implement and maintain their own set of tests, using (or not)
> the goodies from autotest at their discretion.
>
> In summary, autotest is (or is going to be) a framework that
> provides:
>
>   - A test runner, with grid/cluster support and advanced
>     instrumentation
>   - A devel library and set of utilities for test writers
>   - A set of pre-built images (JeOS – Just Enough OS) for
>     test writers
>
> (attached is a picture showing what we want to achieve)
>
> If a project has an internal library or set of utilities that can
> be of general use, they can be submitted to autotest.git for
> inclusion, thus reaching a broader audience.
>
> A short summary of the plans:
>
>   - Tests can live anywhere and each devel team implements and
>     maintains their own set of tests
>   - Usage of the autotest library by test writers is optional
>   - Tests are scripts returning 0 or error (any language)
>   - Tests can be run individually or in sets
>   - Tests should run fast, our target is seconds or a few minutes
>   - The test runner is smart and “just works” by default
>     - Trivial standard output (FAIL, PASS, SKIPPED)
>     - Collect logs, OS data and other stuff (e.g. --record-video!)

Is there a way to collect the logs generated by external test
suite itself? It could be in the specific path.

Osier

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa]

On 03/08/2012 08:54 AM, Stefan Hajnoczi wrote:
> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi<stefanha@gmail.com>  wrote:
>> On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues<lmr@redhat.com>  wrote:
>>> One of our main goals is to provide useful tools for the qemu community,
>>> since we have a good number of tests and libraries written to perform
>>> integration/QA testing for that tool, being successfuly used by a number of
>>> QA teams that work on qemu. Also, we recently provided a subset of that
>>> infrastructure to test libvirt, one of our virtualization projects of
>>> interest.
>> Thanks for sharing.
> One thing I should have added is that my message is about what it
> would take for me to use autotest and contribute tests.  But now I
> realize that you might be going for a different model:
>
> If you're aiming for a different model where autotest integrates
> external test suites (i.e. tests wouldn't be written in autotest.git,
> instead autotest.git would contain snapshots of external test suites),
> then this proposal seems fine.  Upstream projects like QEMU would
> develop their own test suite and it would be dropped into autotest or
> a specific autotest instance.
>
> I'm not 100% sure which of these two models you're going for?

Autotest will continue to integrate and ship with external test suites,
even though that's an option at packaging time.

But the point here is that we also want to cover the other use cases,
which includes being able to run tests that are hosted within external
projects, such as QEMU itself. The idea is to put two things in a state
that's easier to be consumed by individual developers:

   * The test runner
   * The (optional) autotest API

So, by making the autotest API optional, and even the language of the
test script your own choice, you can keep you current test code, using
your own mini-framework and still use the autotest test runner for running
the tests and gathering the results and important system information.

By improving the API, which basically means making it more visible, better
organized and documented, we hope that users writing instrumented
tests (using serial or ssh sessions, sending either HMP or QMP monitor
commands, etc) will choose to use it.

CR.

>
> Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 12:28 PM, Cleber Rosa <crosa@redhat.com> wrote:
> On 03/08/2012 08:54 AM, Stefan Hajnoczi wrote:
>>
>> On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi<stefanha@gmail.com>
>>  wrote:
>>>
>>> On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues<lmr@redhat.com>
>>>  wrote:
>>>>
>>>> One of our main goals is to provide useful tools for the qemu community,
>>>> since we have a good number of tests and libraries written to perform
>>>> integration/QA testing for that tool, being successfuly used by a number
>>>> of
>>>> QA teams that work on qemu. Also, we recently provided a subset of that
>>>> infrastructure to test libvirt, one of our virtualization projects of
>>>> interest.
>>>
>>> Thanks for sharing.
>>
>> One thing I should have added is that my message is about what it
>> would take for me to use autotest and contribute tests.  But now I
>> realize that you might be going for a different model:
>>
>> If you're aiming for a different model where autotest integrates
>> external test suites (i.e. tests wouldn't be written in autotest.git,
>> instead autotest.git would contain snapshots of external test suites),
>> then this proposal seems fine.  Upstream projects like QEMU would
>> develop their own test suite and it would be dropped into autotest or
>> a specific autotest instance.
>>
>> I'm not 100% sure which of these two models you're going for?
>
>
> Autotest will continue to integrate and ship with external test suites,
> even though that's an option at packaging time.
>
> But the point here is that we also want to cover the other use cases,
> which includes being able to run tests that are hosted within external
> projects, such as QEMU itself. The idea is to put two things in a state
> that's easier to be consumed by individual developers:
>
>  * The test runner
>  * The (optional) autotest API
>
> So, by making the autotest API optional, and even the language of the
> test script your own choice, you can keep you current test code, using
> your own mini-framework and still use the autotest test runner for running
> the tests and gathering the results and important system information.
>
> By improving the API, which basically means making it more visible, better
> organized and documented, we hope that users writing instrumented
> tests (using serial or ssh sessions, sending either HMP or QMP monitor
> commands, etc) will choose to use it.

It sounds like you are trying to do both the aggregation of external
test suites and provide APIs/JEOS.  Aggregation will succeed but the
API will not because aggregation undermines the effort to get folks to
use autotest.  By encouraging upstream to have their own test suites,
the API and JEOS will be taken care of upstream.  It won't make sense
for upstream developers to go to autotest when they have a test suite
in their own repository.

Upstream needs to have APIs and JEOS in order to implement their
in-tree test suites - without the ability to create VMs, interact with
the serial console, etc tests couldn't really do anything useful.
QEMU is already growing these abilities right now.

This 50/50 split seems like a compromise so that QA teams can continue
working in autotest and developers can continue working upstream.  It
doesn't change the situation we have today, that's why my first email
described a QEMU/KVM testing library (not framework).  That could
change the game because upstream tests could use the utilities that
have been developed in autotest, but without the framework.  I think
this is what we really need to do in order to grow together rather
than growing apart.  With this proposal I don't see the incentives
that will bring test development closer together, instead it seems
like a license to grow apart.

That said, I don't want to be too serious or negative about it.  It's
not the end of the world but I wanted to share my thoughts because
I've been feeling the need for better testing infrastructure (mostly
with QEMU).

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> Hi guys. For a while we have been discussing ways to make the virtualization
> tests written on top of autotest useful for development level testing.
>
> One of our main goals is to provide useful tools for the qemu community, since
> we have a good number of tests and libraries written to perform integration/QA
> testing for that tool, being successfuly used by a number of QA teams that work
> on qemu. Also, we recently provided a subset of that infrastructure to test
> libvirt, one of our virtualization projects of interest.
>
> We realized that some (admittedly not very radical) changes have to be made on
> autotest itself, so we're inviting other users of autotest to give this a good
> read. This same document lives in the autotest wiki:
>
> https://github.com/autotest/autotest/wiki/FuturePlans
>
> Please note that splitting the virt tests from autotest is not discarded at the
> moment, and it's not incompatible with the plan outlined below.
>
> ====================================
> Virt tests and autotest future goals
> ====================================
>
> In order to make the autotest infrastructure, and the virt tests developed on
> top of autotest more useful for the people working on developing linux, virt and
> other platform software, as well as the QA teams, we are working on a number of
> goals, to streamline, simplify and make the available tools appropriate for
> *development level testing*.
>
> Executing tests appropriate for *QA level testing* will continue to be
> supported, as it's one of the biggest strenghts of autotest.
>
> The problem
> -----------
>
> Autotest provides as of today a local engine, used to run tests on your local
> machine (your laptop or a remote server). Currently, it runs tests properly
> wrapped and packaged under the autotest client/tests/ folder, using specific
> autotest rules.
>
> For the virt tests that live inside autotest, we have even more rules to follow,
> causing a lot of frustration for people that are not used to how things are
> structured and how to execute and select tests.
>
> The proposed solution
> ---------------------
>
> A solution is needed for both scenarios (virt and other general purpose tests).
> The idea is to create specialized tools can run simple tests without packaging,
> code that:
>
> * Knows nothing about the underlying infrastructure
> * Written in any language (shell script, c, perl, you name it)
>
> It'll be up to the test runner to make sense of the results, provided that the
> test writer follows some simple common sense principles while writing the code:
>
> 1) Make the program to return 0 on success, !=0 on failure
> 2) Make the program use a test output format, mainly TAP

We're using gtest in QEMU, not TAP--for better or worse.  If autotest could use 
the gtest protocol, it would integrate much better with QEMU's tests.

Within QEMU, everything should be gtest when possible.

> For simple tests, we believe that option 1) will be far more popular. Autotest
> will harness the execution of the test and put the results under the test
> results directory, with all the sysinfo collection and other instrumentation
> transparently available for the user.
>
> At some point, the test writer might want to start the framework features that
> need to be enabled explicitly, then he/she might want to learn how to use the
> python API to do so, but it'll not be a requirement.
>
> More about the test runner
> --------------------------
>
> The test runner for both general and virt cases should have very simple and
> minimal output:
>
> ::
>
> Results stored in /path/to/autotest/results/default
> my-full-test.py -- PASS
> my-other-test.py -- PASS
> look-mom-i-can-use-shell.sh -- PASS
> look-mom-i-can-use-perl.pl -- FAIL
> test-name-is-the-description.sh -- PASS
> my-yet-another-test.sh -- SKIPPED
> i-like-python.py -- PASS
> whatever-test.pl -- PASS
>
> Both will be specialized tools that use the infrastructure of
> client/bin/autotest-local, but with special code to attend to the output spec
> above. They will know how to handle dependencies, and skip tests if needed.
>
> Directory structure
> -------------------
>
> This is just to give a rough idea of how we won't depend the tests to be in the
> autotest source code folder:
>
> ::
>
> /path/to/autotest -> top level dir, that will make the autotest libs available
> - client/bin -> Contains the test runners and auxiliary scripts
> - client/virt/tests: Contains the virt tests that still live in autotest
> - client/tests/kvm/tests: Contains the qemu tests that still live in autotest
>
> /any/path/test1: Contains tests for software foo
> /any/path/test2: Contains tests for software bar
>
> /any/path/images: Contains minimal guest images for virtualization tests
>
> Bootstrap procedure
> -------------------
>
> In order to comfortably use the framework features, some bootstrap steps will be
> needed, along the following lines:
>
> ::
>
> git clone git://github.com/autotest/autotest.git /path/to/autotest
> export PATH='/path/to/autotest/client/bin':$PATH
> export PYTHON_PATH='/path/to/autotest':$PYTHON_PATH
> export AUTOTEST_DATA='/path/to/images'
>
> Writing tests
> -------------
>
> Simple tests, general case
> ~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> As previously mentioned, writing a trivial test is as simple as writing a
> program that returns either 0 (true) or any other value (false). Autotest
> returns PASS on true and FAIL on false.
>
> Simple tests, virt case
> ~~~~~~~~~~~~~~~~~~~~~~~
>
> The difference is that the program might be executed in the guest or the host,
> so a command line flag or environment variable might be set to indicate where
> the program should be executed (host, guest or both). Autotest returns PASS on
> true and FAIL on false. This functionality is inspired on qemu-test, thanks to
> Anthony Liguori.
>
> Instrumented tests, general case
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The test author can learn how to create an autotest wrapper for the test suite
> and use the specialized tool to run it.
>
> Instrumented tests, virt case
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> The test author can learn how to create an instrumented test for virtualization,
> using the python APIs, or any other language using auxiliary scripts that
> encapsulate high level functionality for use on shell scripts or other
> languages. Ideas for auxiliary scripts:
>
> ::
>
> virt_run_migration [params] [options]
> virt_run_timedrift [params] [options]
> virt_run_nic_hotplug [params] [options]
> virt_run_block_hotplug [params] [options]
>
> Where tests live
> ----------------
>
> The tests won't need to be in the autotest tree, they can live anywhere. The
> reason for this is that projects need in tree tests, that can be maintained by
> the project maintainers.
>
> Standard use case for virt is to have both trivial and instrumented tests living
> in the respective project's tree (qemu and libvirt). Trivial tests don't need
> autotest libs, while instrumented tests will need to, but that's OK provided
> that the appropriate bootstrap procedure was made.
>
> Test Examples
> -------------
>
> simple, non instrumented
> ~~~~~~~~~~~~~~~~~~~~~~~~
>
> ::
>
> uptime.sh:
> #!/bin/sh
> exec uptime
>
> uptime.py:
> #!/usr/bin/python
> import os, sys
> sys.exit(os.system("uptime"))
>
> uptime.pl:
> #!/usr/bin/perl
> system("uptime");
> exit($?);
>
> qemu-img-convert.sh
> #!/bin/bash
> qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2
> diff -b $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2 > /dev/null
> ...
>
> uptime.py - instrummented using libautotest
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> ::
>
> #!/bin/python
>
> from autotest import utils, logging
>
> def run_uptime_host(test, params, env):
> uptime = utils.system_output("uptime")
> logging.info("Host uptime result is: %s", uptime)
>
>
> uptime.py - host/guest mode, instrummented using libautotest
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> ::
>
> #!/bin/python
>
> from autotest import utils, logging
>
> def run_uptime_host_and_guest(test, params, env):
> vm = env.get_vm(params["main_vm"])
> vm.verify_alive()
> session = vm.wait_for_login()
>
> uptime_guest = session.cmd("uptime")
> logging.info("Guest uptime result is: %s", uptime_guest)
>
> uptime_host = utils.system_output("uptime")
> logging.info("Host uptime result is: %s", uptime_host)
>
>
> Virt/qemu tests: Minimal guest images
> -------------------------------------
>
> In order to make development level test possible, we need the tests to run fast.
> In order to do that, a set of minimal guest images is being developed and we
> have a version for x86_64 ready and functional:
>
> https://github.com/autotest/buildroot-autotest

I'm really not a fan of buildroot.  Note that in order to ship binaries, full 
source needs to be provided in order to comply with the GPL.  The FSF at least 
states that referring to another website for source that's not under your 
control doesn't satisfy the requirements of the GPL.

Just out of curiosity, did you try to use qemu-test?  Is there a reason you 
created something different?

I think it's good that you're thinking about how to make writing tests easier, 
but we have a growing test infrastructure in QEMU and that's what I'd prefer 
people focused on.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/08/2012 10:36 AM, Anthony Liguori wrote:

>> Virt/qemu tests: Minimal guest images
>> -------------------------------------
>>
>> In order to make development level test possible, we need the tests to
>> run fast.
>> In order to do that, a set of minimal guest images is being developed
>> and we
>> have a version for x86_64 ready and functional:
>>
>> https://github.com/autotest/buildroot-autotest
>
> I'm really not a fan of buildroot. Note that in order to ship binaries,
> full source needs to be provided in order to comply with the GPL. The
> FSF at least states that referring to another website for source that's
> not under your control doesn't satisfy the requirements of the GPL.

We have a full clone of the buildroot repository that points out to the 
sources, if it's necessary to have a clone of all the projects needed 
host there in order to be able to publish a binary image to help people, 
then we can do it.

> Just out of curiosity, did you try to use qemu-test? Is there a reason
> you created something different?

I did, and it does what it proposes to. Nothing against it, but we have 
code that can do more things, that has been developed for longer time.

It's similar to qemu-jeos vs buildbot, you have written scripts to 
create an image, which happens to be precisely why buildroot was written 
more than 10 years ago and it works very well, allowing me to put things 
on the image that are not possible with qemu-jeos. If the problem is to 
point out to all sub modules as git repos, we can make that happen too, 
rather than re-writing stuff that works.

For a long time I would like to see people working on a single code 
base, because that would allow things to progress further and people 
would have even better tools to use.

By implementing the features of qemu-test in autotest we could simply 
use the qemu-test tests and use autotest rather than qemu test, and 
that's why we have done it.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 08:01 AM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>
>>> Virt/qemu tests: Minimal guest images
>>> -------------------------------------
>>>
>>> In order to make development level test possible, we need the tests to
>>> run fast.
>>> In order to do that, a set of minimal guest images is being developed
>>> and we
>>> have a version for x86_64 ready and functional:
>>>
>>> https://github.com/autotest/buildroot-autotest
>>
>> I'm really not a fan of buildroot. Note that in order to ship binaries,
>> full source needs to be provided in order to comply with the GPL. The
>> FSF at least states that referring to another website for source that's
>> not under your control doesn't satisfy the requirements of the GPL.
>
> We have a full clone of the buildroot repository that points out to the sources,
> if it's necessary to have a clone of all the projects needed host there in order
> to be able to publish a binary image to help people, then we can do it.

This is harder than I think you anticipate but okay..

>
>> Just out of curiosity, did you try to use qemu-test? Is there a reason
>> you created something different?
>
> I did, and it does what it proposes to. Nothing against it, but we have code
> that can do more things, that has been developed for longer time.
>
> It's similar to qemu-jeos vs buildbot, you have written scripts to create an
> image, which happens to be precisely why buildroot was written more than 10
> years ago and it works very well, allowing me to put things on the image that
> are not possible with qemu-jeos. If the problem is to point out to all sub
> modules as git repos, we can make that happen too, rather than re-writing stuff
> that works.
>
> For a long time I would like to see people working on a single code base,

I agree, we just disagree on what that code base should be :-)

That code base should be qemu.git.  This discussion isn't about improving 
third-party QE--at least not to me.  Third party QE is a solved problem thanks 
to all of your wonderful work with kvm-autotest.  I'm sure you're looking for 
more participation/developers, but even if you had twice the developers working 
on kvm-autotest, I don't think it would fundamentally change our quality.

I'm interested in driving our development process toward test driven development 
such that all 200+ people that write patches for QEMU for a given release write 
and run tests as part of their normal development process.  The requirements to 
achieve this are different than the requirements you have been working against 
up until now.

Every barrier that we put up to writing and running tests will reduce than 
number of 200+ to something lower.

Submitting a patch to a different project than qemu.git is a barrier.  Now 
instead of getting a single set of feedback, you've got to deal with feedback 
from two projects.

Having to use setup another framework (that runs as root) is another barrier.  I 
change a file in QEMU, run make, then run make check.  I don't install anything, 
I don't sudo anything.  The whole process is relatively quick and painless.

Having to make a change to autotest, then install autotest, relaunch it, etc, is 
just too complicated to be part of a developers fast path.

Now I think we should talk about how to make tests that live in qemu.git and run 
as part of make check easily harnessed by autotest..  But I think the primary 
focus of future test work needs to be within qemu.git.

Regards,

Anthony Liguori

> because that would allow things to progress further and people would have even
> better tools to use.
>
> By implementing the features of qemu-test in autotest we could simply use the
> qemu-test tests and use autotest rather than qemu test, and that's why we have
> done it.
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 08:48:33AM -0600, Anthony Liguori wrote:
> On 03/08/2012 08:01 AM, Lucas Meneghel Rodrigues wrote:
> >On 03/08/2012 10:36 AM, Anthony Liguori wrote:
> >
> >>>Virt/qemu tests: Minimal guest images
> >>>-------------------------------------
> >>>
> >>>In order to make development level test possible, we need the tests to
> >>>run fast.
> >>>In order to do that, a set of minimal guest images is being developed
> >>>and we
> >>>have a version for x86_64 ready and functional:
> >>>
> >>>https://github.com/autotest/buildroot-autotest
> >>
> >>I'm really not a fan of buildroot. Note that in order to ship binaries,
> >>full source needs to be provided in order to comply with the GPL. The
> >>FSF at least states that referring to another website for source that's
> >>not under your control doesn't satisfy the requirements of the GPL.
> >
> >We have a full clone of the buildroot repository that points out to the sources,
> >if it's necessary to have a clone of all the projects needed host there in order
> >to be able to publish a binary image to help people, then we can do it.
> 
> This is harder than I think you anticipate but okay..
> 
> >
> >>Just out of curiosity, did you try to use qemu-test? Is there a reason
> >>you created something different?
> >
> >I did, and it does what it proposes to. Nothing against it, but we have code
> >that can do more things, that has been developed for longer time.
> >
> >It's similar to qemu-jeos vs buildbot, you have written scripts to create an
> >image, which happens to be precisely why buildroot was written more than 10
> >years ago and it works very well, allowing me to put things on the image that
> >are not possible with qemu-jeos. If the problem is to point out to all sub
> >modules as git repos, we can make that happen too, rather than re-writing stuff
> >that works.
> >
> >For a long time I would like to see people working on a single code base,
> 
> I agree, we just disagree on what that code base should be :-)
> 
> That code base should be qemu.git.  This discussion isn't about
> improving third-party QE--at least not to me.  Third party QE is a
> solved problem thanks to all of your wonderful work with
> kvm-autotest.  I'm sure you're looking for more
> participation/developers, but even if you had twice the developers
> working on kvm-autotest, I don't think it would fundamentally change
> our quality.

You got it wrong: we don't want people to work on autotest, we
want people to use autotest because we believe there's a lot of
benefit on using it.

And we *know* that the current autotest is not good enough for
developers, which is why we're now working on new usage
scenarios.

> 
> I'm interested in driving our development process toward test driven
> development such that all 200+ people that write patches for QEMU
> for a given release write and run tests as part of their normal
> development process.  The requirements to achieve this are different
> than the requirements you have been working against up until now.

Fully agree, these are new requirements.

> 
> Every barrier that we put up to writing and running tests will
> reduce than number of 200+ to something lower.
> 
> Submitting a patch to a different project than qemu.git is a
> barrier.  Now instead of getting a single set of feedback, you've
> got to deal with feedback from two projects.
> 

Fully agree, please check my previous email with the plans for
the new architecture of autotest.

> Having to use setup another framework (that runs as root) is another
> barrier.  I change a file in QEMU, run make, then run make check.  I
> don't install anything, I don't sudo anything.  The whole process is
> relatively quick and painless.

Fully agree, this is one of the requirement we're going after.

> 
> Having to make a change to autotest, then install autotest, relaunch
> it, etc, is just too complicated to be part of a developers fast
> path.

Fully agree, which is why we're not planning any of it.

> 
> Now I think we should talk about how to make tests that live in
> qemu.git and run as part of make check easily harnessed by
> autotest..  But I think the primary focus of future test work needs
> to be within qemu.git.

Fully agree, please check our previous e-mails with the plans for
the new architecture.

So as you can see, there are no disagreements. :-) Please keep
the feedback and requirements coming.

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Andreas Färber]

Hi,

Am 08.03.2012 16:00, schrieb Ademar Reis:
> Fully agree, please check my previous email with the plans for
> the new architecture of autotest.
[...]
> Fully agree, please check our previous e-mails with the plans for
> the new architecture.

FYI your mails are arriving hours late on qemu-devel.

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Fri, Mar 09, 2012 at 12:59:16AM +0100, Andreas Färber wrote:
> Hi,
> 
> Am 08.03.2012 16:00, schrieb Ademar Reis:
> > Fully agree, please check my previous email with the plans for
> > the new architecture of autotest.
> [...]
> > Fully agree, please check our previous e-mails with the plans for
> > the new architecture.
> 
> FYI your mails are arriving hours late on qemu-devel.

Thanks for the heads up. It appears to be normal now, as I see
the message I've just sent in the archives.

Anyway, the message I was talking about is this one:

http://lists.nongnu.org/archive/html/qemu-devel/2012-03/msg01312.html
http://lists.nongnu.org/archive/html/qemu-devel/2012-03/msg01313.html
(the last one contains the intended attachment)

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >Hi guys. For a while we have been discussing ways to make the virtualization
> >tests written on top of autotest useful for development level testing.
> >
> >One of our main goals is to provide useful tools for the qemu community, since
> >we have a good number of tests and libraries written to perform integration/QA
> >testing for that tool, being successfuly used by a number of QA teams that work
> >on qemu. Also, we recently provided a subset of that infrastructure to test
> >libvirt, one of our virtualization projects of interest.
> >
> >We realized that some (admittedly not very radical) changes have to be made on
> >autotest itself, so we're inviting other users of autotest to give this a good
> >read. This same document lives in the autotest wiki:
> >
> >https://github.com/autotest/autotest/wiki/FuturePlans
> >
> >Please note that splitting the virt tests from autotest is not discarded at the
> >moment, and it's not incompatible with the plan outlined below.
> >
> >====================================
> >Virt tests and autotest future goals
> >====================================
> >
> >In order to make the autotest infrastructure, and the virt tests developed on
> >top of autotest more useful for the people working on developing linux, virt and
> >other platform software, as well as the QA teams, we are working on a number of
> >goals, to streamline, simplify and make the available tools appropriate for
> >*development level testing*.
> >
> >Executing tests appropriate for *QA level testing* will continue to be
> >supported, as it's one of the biggest strenghts of autotest.
> >
> >The problem
> >-----------
> >
> >Autotest provides as of today a local engine, used to run tests on your local
> >machine (your laptop or a remote server). Currently, it runs tests properly
> >wrapped and packaged under the autotest client/tests/ folder, using specific
> >autotest rules.
> >
> >For the virt tests that live inside autotest, we have even more rules to follow,
> >causing a lot of frustration for people that are not used to how things are
> >structured and how to execute and select tests.
> >
> >The proposed solution
> >---------------------
> >
> >A solution is needed for both scenarios (virt and other general purpose tests).
> >The idea is to create specialized tools can run simple tests without packaging,
> >code that:
> >
> >* Knows nothing about the underlying infrastructure
> >* Written in any language (shell script, c, perl, you name it)
> >
> >It'll be up to the test runner to make sense of the results, provided that the
> >test writer follows some simple common sense principles while writing the code:
> >
> >1) Make the program to return 0 on success, !=0 on failure
> >2) Make the program use a test output format, mainly TAP
> 
> We're using gtest in QEMU, not TAP--for better or worse.  If
> autotest could use the gtest protocol, it would integrate much
> better with QEMU's tests.
> 
> Within QEMU, everything should be gtest when possible.
> 
> >For simple tests, we believe that option 1) will be far more popular. Autotest
> >will harness the execution of the test and put the results under the test
> >results directory, with all the sysinfo collection and other instrumentation
> >transparently available for the user.
> >
> >At some point, the test writer might want to start the framework features that
> >need to be enabled explicitly, then he/she might want to learn how to use the
> >python API to do so, but it'll not be a requirement.
> >
> >More about the test runner
> >--------------------------
> >
> >The test runner for both general and virt cases should have very simple and
> >minimal output:
> >
> >::
> >
> >Results stored in /path/to/autotest/results/default
> >my-full-test.py -- PASS
> >my-other-test.py -- PASS
> >look-mom-i-can-use-shell.sh -- PASS
> >look-mom-i-can-use-perl.pl -- FAIL
> >test-name-is-the-description.sh -- PASS
> >my-yet-another-test.sh -- SKIPPED
> >i-like-python.py -- PASS
> >whatever-test.pl -- PASS
> >
> >Both will be specialized tools that use the infrastructure of
> >client/bin/autotest-local, but with special code to attend to the output spec
> >above. They will know how to handle dependencies, and skip tests if needed.
> >
> >Directory structure
> >-------------------
> >
> >This is just to give a rough idea of how we won't depend the tests to be in the
> >autotest source code folder:
> >
> >::
> >
> >/path/to/autotest -> top level dir, that will make the autotest libs available
> >- client/bin -> Contains the test runners and auxiliary scripts
> >- client/virt/tests: Contains the virt tests that still live in autotest
> >- client/tests/kvm/tests: Contains the qemu tests that still live in autotest
> >
> >/any/path/test1: Contains tests for software foo
> >/any/path/test2: Contains tests for software bar
> >
> >/any/path/images: Contains minimal guest images for virtualization tests
> >
> >Bootstrap procedure
> >-------------------
> >
> >In order to comfortably use the framework features, some bootstrap steps will be
> >needed, along the following lines:
> >
> >::
> >
> >git clone git://github.com/autotest/autotest.git /path/to/autotest
> >export PATH='/path/to/autotest/client/bin':$PATH
> >export PYTHON_PATH='/path/to/autotest':$PYTHON_PATH
> >export AUTOTEST_DATA='/path/to/images'
> >
> >Writing tests
> >-------------
> >
> >Simple tests, general case
> >~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >As previously mentioned, writing a trivial test is as simple as writing a
> >program that returns either 0 (true) or any other value (false). Autotest
> >returns PASS on true and FAIL on false.
> >
> >Simple tests, virt case
> >~~~~~~~~~~~~~~~~~~~~~~~
> >
> >The difference is that the program might be executed in the guest or the host,
> >so a command line flag or environment variable might be set to indicate where
> >the program should be executed (host, guest or both). Autotest returns PASS on
> >true and FAIL on false. This functionality is inspired on qemu-test, thanks to
> >Anthony Liguori.
> >
> >Instrumented tests, general case
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >The test author can learn how to create an autotest wrapper for the test suite
> >and use the specialized tool to run it.
> >
> >Instrumented tests, virt case
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >The test author can learn how to create an instrumented test for virtualization,
> >using the python APIs, or any other language using auxiliary scripts that
> >encapsulate high level functionality for use on shell scripts or other
> >languages. Ideas for auxiliary scripts:
> >
> >::
> >
> >virt_run_migration [params] [options]
> >virt_run_timedrift [params] [options]
> >virt_run_nic_hotplug [params] [options]
> >virt_run_block_hotplug [params] [options]
> >
> >Where tests live
> >----------------
> >
> >The tests won't need to be in the autotest tree, they can live anywhere. The
> >reason for this is that projects need in tree tests, that can be maintained by
> >the project maintainers.
> >
> >Standard use case for virt is to have both trivial and instrumented tests living
> >in the respective project's tree (qemu and libvirt). Trivial tests don't need
> >autotest libs, while instrumented tests will need to, but that's OK provided
> >that the appropriate bootstrap procedure was made.
> >
> >Test Examples
> >-------------
> >
> >simple, non instrumented
> >~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >::
> >
> >uptime.sh:
> >#!/bin/sh
> >exec uptime
> >
> >uptime.py:
> >#!/usr/bin/python
> >import os, sys
> >sys.exit(os.system("uptime"))
> >
> >uptime.pl:
> >#!/usr/bin/perl
> >system("uptime");
> >exit($?);
> >
> >qemu-img-convert.sh
> >#!/bin/bash
> >qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2
> >diff -b $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2 > /dev/null
> >...
> >
> >uptime.py - instrummented using libautotest
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >::
> >
> >#!/bin/python
> >
> >from autotest import utils, logging
> >
> >def run_uptime_host(test, params, env):
> >uptime = utils.system_output("uptime")
> >logging.info("Host uptime result is: %s", uptime)
> >
> >
> >uptime.py - host/guest mode, instrummented using libautotest
> >~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> >
> >::
> >
> >#!/bin/python
> >
> >from autotest import utils, logging
> >
> >def run_uptime_host_and_guest(test, params, env):
> >vm = env.get_vm(params["main_vm"])
> >vm.verify_alive()
> >session = vm.wait_for_login()
> >
> >uptime_guest = session.cmd("uptime")
> >logging.info("Guest uptime result is: %s", uptime_guest)
> >
> >uptime_host = utils.system_output("uptime")
> >logging.info("Host uptime result is: %s", uptime_host)
> >
> >
> >Virt/qemu tests: Minimal guest images
> >-------------------------------------
> >
> >In order to make development level test possible, we need the tests to run fast.
> >In order to do that, a set of minimal guest images is being developed and we
> >have a version for x86_64 ready and functional:
> >
> >https://github.com/autotest/buildroot-autotest
> 
> I'm really not a fan of buildroot.  Note that in order to ship
> binaries, full source needs to be provided in order to comply with
> the GPL.  The FSF at least states that referring to another website
> for source that's not under your control doesn't satisfy the
> requirements of the GPL.
> 
> Just out of curiosity, did you try to use qemu-test?  Is there a
> reason you created something different?
> 
> I think it's good that you're thinking about how to make writing
> tests easier, but we have a growing test infrastructure in QEMU and
> that's what I'd prefer people focused on.
> 

You probably remember the long thread we had back in December on
qemu-devel on this topic. Back then our message was "we have a
growing test infrastructure in s/QEMU/autotest/ and that's what
we'd prefer people focused on". :-)

>From Dor:

(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)

"""
If you wish, you can challenge Lucas and Cleber w/ these type of
requirements and we'll all improve as a result.
"""

Your response was:

"""
Well consider qemu-test the challenge. It's an existence proof
that we can have a very easy to use framework for testing that
runs extremely fast and is very easy to write tests for.
"""

http://knowyourmeme.com/memes/challenge-accepted ;-)

I particularly agreed with basically everything you said on that
discussion regarding test simplification (I had just joined the
team back then). To me, autotest has been focusing on QE-level,
leaving the developer-level test requirements out. Now we're
attacking this new front, and a lot of the requirements are
indeed from that discussion.

By simplifying the design and bringing barriers down, we hope to
reach a broader audience and help developers write and maintain
tests, benefiting from all the instrumentation that autotest
brings. It's not going to be just about qemu (check the new test
examples).

We have a team fully dedicated to autotest and it's used not only
by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
(these all have code contributions in autotest)

That said, the current qemu-tests will probably be easily
integrated into (the new) autotest and we hope that, given enough
time, autotest will be good enough to relieve qemu from the
framework maintenance and code duplication with other projects.

We hope in the future you see autotest the way you see gtest
right now: a library that helps you write better tests. The fully
instrummented test-runner and the hability to run tests in the
cloud will hopefully bring a lot of value as well.

Please raise your requirements. We would love to avoid code
duplication and be integrated as much as possible.

Cheers,
   - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 08:49 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
>>> Virt/qemu tests: Minimal guest images
>>> -------------------------------------
>>>
>>> In order to make development level test possible, we need the tests to run fast.
>>> In order to do that, a set of minimal guest images is being developed and we
>>> have a version for x86_64 ready and functional:
>>>
>>> https://github.com/autotest/buildroot-autotest
>>
>> I'm really not a fan of buildroot.  Note that in order to ship
>> binaries, full source needs to be provided in order to comply with
>> the GPL.  The FSF at least states that referring to another website
>> for source that's not under your control doesn't satisfy the
>> requirements of the GPL.
>>
>> Just out of curiosity, did you try to use qemu-test?  Is there a
>> reason you created something different?
>>
>> I think it's good that you're thinking about how to make writing
>> tests easier, but we have a growing test infrastructure in QEMU and
>> that's what I'd prefer people focused on.
>>
>
> You probably remember the long thread we had back in December on
> qemu-devel on this topic. Back then our message was "we have a
> growing test infrastructure in s/QEMU/autotest/ and that's what
> we'd prefer people focused on". :-)
>
>  From Dor:
>
> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
>
> """
> If you wish, you can challenge Lucas and Cleber w/ these type of
> requirements and we'll all improve as a result.
> """
>
> Your response was:
>
> """
> Well consider qemu-test the challenge. It's an existence proof
> that we can have a very easy to use framework for testing that
> runs extremely fast and is very easy to write tests for.
> """
>
> http://knowyourmeme.com/memes/challenge-accepted ;-)
>
> I particularly agreed with basically everything you said on that
> discussion regarding test simplification (I had just joined the
> team back then). To me, autotest has been focusing on QE-level,
> leaving the developer-level test requirements out. Now we're
> attacking this new front, and a lot of the requirements are
> indeed from that discussion.

If you want to talk about this in terms of "requirements", my requirement is for 
"developer-level" tests to live in qemu.git and be integrated into make check.

Just as we've been discussing and working on since the previous set of discussions.

> By simplifying the design and bringing barriers down, we hope to
> reach a broader audience and help developers write and maintain
> tests, benefiting from all the instrumentation that autotest
> brings. It's not going to be just about qemu (check the new test
> examples).
>
> We have a team fully dedicated to autotest and it's used not only
> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> (these all have code contributions in autotest)
>
> That said, the current qemu-tests will probably be easily
> integrated into (the new) autotest and we hope that, given enough
> time, autotest will be good enough to relieve qemu from the
> framework maintenance and code duplication with other projects.

autotest should not be the focal point for integration.  qemu.git should be.

I'd be perfectly happy to review patches submitting the test infrastructure from 
kvm-autotest into qemu.git (provided it didn't have unreasonable external 
dependencies and fit into QEMU).

Developer-level tests need to live where the developers live.  The developers 
live in qemu.git.  See my other response on this thread for the explanation of 
why this is so important.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
> On 03/08/2012 08:49 AM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> >>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >>>Virt/qemu tests: Minimal guest images
> >>>-------------------------------------
> >>>
> >>>In order to make development level test possible, we need the tests to run fast.
> >>>In order to do that, a set of minimal guest images is being developed and we
> >>>have a version for x86_64 ready and functional:
> >>>
> >>>https://github.com/autotest/buildroot-autotest
> >>
> >>I'm really not a fan of buildroot.  Note that in order to ship
> >>binaries, full source needs to be provided in order to comply with
> >>the GPL.  The FSF at least states that referring to another website
> >>for source that's not under your control doesn't satisfy the
> >>requirements of the GPL.
> >>
> >>Just out of curiosity, did you try to use qemu-test?  Is there a
> >>reason you created something different?
> >>
> >>I think it's good that you're thinking about how to make writing
> >>tests easier, but we have a growing test infrastructure in QEMU and
> >>that's what I'd prefer people focused on.
> >>
> >
> >You probably remember the long thread we had back in December on
> >qemu-devel on this topic. Back then our message was "we have a
> >growing test infrastructure in s/QEMU/autotest/ and that's what
> >we'd prefer people focused on". :-)
> >
> > From Dor:
> >
> >(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
> >
> >"""
> >If you wish, you can challenge Lucas and Cleber w/ these type of
> >requirements and we'll all improve as a result.
> >"""
> >
> >Your response was:
> >
> >"""
> >Well consider qemu-test the challenge. It's an existence proof
> >that we can have a very easy to use framework for testing that
> >runs extremely fast and is very easy to write tests for.
> >"""
> >
> >http://knowyourmeme.com/memes/challenge-accepted ;-)
> >
> >I particularly agreed with basically everything you said on that
> >discussion regarding test simplification (I had just joined the
> >team back then). To me, autotest has been focusing on QE-level,
> >leaving the developer-level test requirements out. Now we're
> >attacking this new front, and a lot of the requirements are
> >indeed from that discussion.
> 
> If you want to talk about this in terms of "requirements", my
> requirement is for "developer-level" tests to live in qemu.git and
> be integrated into make check.
> 
> Just as we've been discussing and working on since the previous set of discussions.
> 
> >By simplifying the design and bringing barriers down, we hope to
> >reach a broader audience and help developers write and maintain
> >tests, benefiting from all the instrumentation that autotest
> >brings. It's not going to be just about qemu (check the new test
> >examples).
> >
> >We have a team fully dedicated to autotest and it's used not only
> >by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> >(these all have code contributions in autotest)
> >
> >That said, the current qemu-tests will probably be easily
> >integrated into (the new) autotest and we hope that, given enough
> >time, autotest will be good enough to relieve qemu from the
> >framework maintenance and code duplication with other projects.
> 
> autotest should not be the focal point for integration.  qemu.git should be.
> 
> I'd be perfectly happy to review patches submitting the test
> infrastructure from kvm-autotest into qemu.git (provided it didn't
> have unreasonable external dependencies and fit into QEMU).
> 
> Developer-level tests need to live where the developers live.  The
> developers live in qemu.git.  See my other response on this thread
> for the explanation of why this is so important.
> 

Excelent, we're in the same page then. This was my number 1
requirement when I was discussing the changes with Lucas and
Cleber. For convenience, I'll repeat here what I wrote in a
previous e-mail (no qemu-devel archive available yet to use as a
reference).

In summary, autotest is (or is going to be) a framework that
provides:

 - A test runner, with grid/cluster support and advanced
   instrumentation
 - A devel library and set of utilities for test writers
 - A set of pre-built images (JeOS – Just Enough OS) for
   test writers

(attached is a picture showing what we want to achieve)

If a project has an internal library or set of utilities that can
be of general use, they can be submitted to autotest.git for
inclusion, thus reaching a broader audience.

A short summary of the plans:

 - Tests can live anywhere and each devel team implements and
   maintains their own set of tests
 - Usage of the autotest library by test writers is optional
 - Tests are scripts returning 0 or error (any language)
 - Tests can be run individually or in sets
 - Tests should run fast, our target is seconds or a few minutes
 - The test runner is smart and “just works” by default
   - Trivial standard output (FAIL, PASS, SKIPPED)
   - Collect logs, OS data and other stuff (e.g. --record-video!)
   - Skip unsupported tests based on the environment they're run
   - Multiplex configurations / platforms when on the grid
   - Support to run tests “in the cloud”

There are also lots of small changes and usability improvements
in the pipeline (and feedback right now is very valuable).

Thanks,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 09:07 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
>> On 03/08/2012 08:49 AM, Ademar Reis wrote:
>>> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
>>>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
>>>>> Virt/qemu tests: Minimal guest images
>>>>> -------------------------------------
>>>>>
>>>>> In order to make development level test possible, we need the tests to run fast.
>>>>> In order to do that, a set of minimal guest images is being developed and we
>>>>> have a version for x86_64 ready and functional:
>>>>>
>>>>> https://github.com/autotest/buildroot-autotest
>>>>
>>>> I'm really not a fan of buildroot.  Note that in order to ship
>>>> binaries, full source needs to be provided in order to comply with
>>>> the GPL.  The FSF at least states that referring to another website
>>>> for source that's not under your control doesn't satisfy the
>>>> requirements of the GPL.
>>>>
>>>> Just out of curiosity, did you try to use qemu-test?  Is there a
>>>> reason you created something different?
>>>>
>>>> I think it's good that you're thinking about how to make writing
>>>> tests easier, but we have a growing test infrastructure in QEMU and
>>>> that's what I'd prefer people focused on.
>>>>
>>>
>>> You probably remember the long thread we had back in December on
>>> qemu-devel on this topic. Back then our message was "we have a
>>> growing test infrastructure in s/QEMU/autotest/ and that's what
>>> we'd prefer people focused on". :-)
>>>
>>>  From Dor:
>>>
>>> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
>>>
>>> """
>>> If you wish, you can challenge Lucas and Cleber w/ these type of
>>> requirements and we'll all improve as a result.
>>> """
>>>
>>> Your response was:
>>>
>>> """
>>> Well consider qemu-test the challenge. It's an existence proof
>>> that we can have a very easy to use framework for testing that
>>> runs extremely fast and is very easy to write tests for.
>>> """
>>>
>>> http://knowyourmeme.com/memes/challenge-accepted ;-)
>>>
>>> I particularly agreed with basically everything you said on that
>>> discussion regarding test simplification (I had just joined the
>>> team back then). To me, autotest has been focusing on QE-level,
>>> leaving the developer-level test requirements out. Now we're
>>> attacking this new front, and a lot of the requirements are
>>> indeed from that discussion.
>>
>> If you want to talk about this in terms of "requirements", my
>> requirement is for "developer-level" tests to live in qemu.git and
>> be integrated into make check.
>>
>> Just as we've been discussing and working on since the previous set of discussions.
>>
>>> By simplifying the design and bringing barriers down, we hope to
>>> reach a broader audience and help developers write and maintain
>>> tests, benefiting from all the instrumentation that autotest
>>> brings. It's not going to be just about qemu (check the new test
>>> examples).
>>>
>>> We have a team fully dedicated to autotest and it's used not only
>>> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
>>> (these all have code contributions in autotest)
>>>
>>> That said, the current qemu-tests will probably be easily
>>> integrated into (the new) autotest and we hope that, given enough
>>> time, autotest will be good enough to relieve qemu from the
>>> framework maintenance and code duplication with other projects.
>>
>> autotest should not be the focal point for integration.  qemu.git should be.
>>
>> I'd be perfectly happy to review patches submitting the test
>> infrastructure from kvm-autotest into qemu.git (provided it didn't
>> have unreasonable external dependencies and fit into QEMU).
>>
>> Developer-level tests need to live where the developers live.  The
>> developers live in qemu.git.  See my other response on this thread
>> for the explanation of why this is so important.
>>
>
> Excelent, we're in the same page then. This was my number 1
> requirement when I was discussing the changes with Lucas and
> Cleber. For convenience, I'll repeat here what I wrote in a
> previous e-mail (no qemu-devel archive available yet to use as a
> reference).
>
> In summary, autotest is (or is going to be) a framework that
> provides:
>
>   - A test runner, with grid/cluster support and advanced
>     instrumentation
>   - A devel library and set of utilities for test writers
>   - A set of pre-built images (JeOS – Just Enough OS) for
>     test writers

I don't think autotest is the right place for this to live.  We need this 
directly in qemu.git otherwise we're severely limited in what tests we can write.

I guess that doesn't preclude autotest having its own JeOS mechanism, but we 
clearly need one in qemu.git.

>
> (attached is a picture showing what we want to achieve)
>
> If a project has an internal library or set of utilities that can
> be of general use, they can be submitted to autotest.git for
> inclusion, thus reaching a broader audience.
>
> A short summary of the plans:
>
>   - Tests can live anywhere and each devel team implements and
>     maintains their own set of tests

Let me change this to:

- Autotest will learn how to harness the tests that each development team 
creates in their respective git repository.

I think this is what you mean, but it's not how I interpret the above.  I read 
this as, "tests can (and should) live anywhere across any set of repositories". 
  I think when you say "live anywhere", you mean this strictly from an autotest 
perspective and are trying to accomodate tests that live in upstream repositories.

>   - Usage of the autotest library by test writers is optional
>   - Tests are scripts returning 0 or error (any language)
>   - Tests can be run individually or in sets
>   - Tests should run fast, our target is seconds or a few minutes
>   - The test runner is smart and “just works” by default
>     - Trivial standard output (FAIL, PASS, SKIPPED)
>     - Collect logs, OS data and other stuff (e.g. --record-video!)
>     - Skip unsupported tests based on the environment they're run
>     - Multiplex configurations / platforms when on the grid
>     - Support to run tests “in the cloud”

This is where you start to lose me.  If all you're saying is, "QEMU can continue 
to use gtest to build out it's test infrastructure and autotest will learn how 
to use it", then we're in violent agreement.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
> On 03/08/2012 09:07 AM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 08:49 AM, Ademar Reis wrote:
> >>>On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> >>>>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >>>>>Virt/qemu tests: Minimal guest images
> >>>>>-------------------------------------
> >>>>>
> >>>>>In order to make development level test possible, we need the tests to run fast.
> >>>>>In order to do that, a set of minimal guest images is being developed and we
> >>>>>have a version for x86_64 ready and functional:
> >>>>>
> >>>>>https://github.com/autotest/buildroot-autotest
> >>>>
> >>>>I'm really not a fan of buildroot.  Note that in order to ship
> >>>>binaries, full source needs to be provided in order to comply with
> >>>>the GPL.  The FSF at least states that referring to another website
> >>>>for source that's not under your control doesn't satisfy the
> >>>>requirements of the GPL.
> >>>>
> >>>>Just out of curiosity, did you try to use qemu-test?  Is there a
> >>>>reason you created something different?
> >>>>
> >>>>I think it's good that you're thinking about how to make writing
> >>>>tests easier, but we have a growing test infrastructure in QEMU and
> >>>>that's what I'd prefer people focused on.
> >>>>
> >>>
> >>>You probably remember the long thread we had back in December on
> >>>qemu-devel on this topic. Back then our message was "we have a
> >>>growing test infrastructure in s/QEMU/autotest/ and that's what
> >>>we'd prefer people focused on". :-)
> >>>
> >>> From Dor:
> >>>
> >>>(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
> >>>
> >>>"""
> >>>If you wish, you can challenge Lucas and Cleber w/ these type of
> >>>requirements and we'll all improve as a result.
> >>>"""
> >>>
> >>>Your response was:
> >>>
> >>>"""
> >>>Well consider qemu-test the challenge. It's an existence proof
> >>>that we can have a very easy to use framework for testing that
> >>>runs extremely fast and is very easy to write tests for.
> >>>"""
> >>>
> >>>http://knowyourmeme.com/memes/challenge-accepted ;-)
> >>>
> >>>I particularly agreed with basically everything you said on that
> >>>discussion regarding test simplification (I had just joined the
> >>>team back then). To me, autotest has been focusing on QE-level,
> >>>leaving the developer-level test requirements out. Now we're
> >>>attacking this new front, and a lot of the requirements are
> >>>indeed from that discussion.
> >>
> >>If you want to talk about this in terms of "requirements", my
> >>requirement is for "developer-level" tests to live in qemu.git and
> >>be integrated into make check.
> >>
> >>Just as we've been discussing and working on since the previous set of discussions.
> >>
> >>>By simplifying the design and bringing barriers down, we hope to
> >>>reach a broader audience and help developers write and maintain
> >>>tests, benefiting from all the instrumentation that autotest
> >>>brings. It's not going to be just about qemu (check the new test
> >>>examples).
> >>>
> >>>We have a team fully dedicated to autotest and it's used not only
> >>>by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> >>>(these all have code contributions in autotest)
> >>>
> >>>That said, the current qemu-tests will probably be easily
> >>>integrated into (the new) autotest and we hope that, given enough
> >>>time, autotest will be good enough to relieve qemu from the
> >>>framework maintenance and code duplication with other projects.
> >>
> >>autotest should not be the focal point for integration.  qemu.git should be.
> >>
> >>I'd be perfectly happy to review patches submitting the test
> >>infrastructure from kvm-autotest into qemu.git (provided it didn't
> >>have unreasonable external dependencies and fit into QEMU).
> >>
> >>Developer-level tests need to live where the developers live.  The
> >>developers live in qemu.git.  See my other response on this thread
> >>for the explanation of why this is so important.
> >>
> >
> >Excelent, we're in the same page then. This was my number 1
> >requirement when I was discussing the changes with Lucas and
> >Cleber. For convenience, I'll repeat here what I wrote in a
> >previous e-mail (no qemu-devel archive available yet to use as a
> >reference).
> >
> >In summary, autotest is (or is going to be) a framework that
> >provides:
> >
> >  - A test runner, with grid/cluster support and advanced
> >    instrumentation
> >  - A devel library and set of utilities for test writers
> >  - A set of pre-built images (JeOS – Just Enough OS) for
> >    test writers
> 
> I don't think autotest is the right place for this to live.  We need
> this directly in qemu.git otherwise we're severely limited in what
> tests we can write.
> 
> I guess that doesn't preclude autotest having its own JeOS
> mechanism, but we clearly need one in qemu.git.
> 
> >
> >(attached is a picture showing what we want to achieve)
> >
> >If a project has an internal library or set of utilities that can
> >be of general use, they can be submitted to autotest.git for
> >inclusion, thus reaching a broader audience.
> >
> >A short summary of the plans:
> >
> >  - Tests can live anywhere and each devel team implements and
> >    maintains their own set of tests
> 
> Let me change this to:
> 
> - Autotest will learn how to harness the tests that each development
> team creates in their respective git repository.

Yep, given this simple requirement: "tests return 0 or an error
code". The output from the test runner will be a simple PASS/FAIL
and stdout/stderr will be properly collected by the test runner
and made available in a standard location. If using TAP, even
better.

This will allow trivial tests. As the test complexity grows, then
developers may start using parts of the autotest library or
utilities, thus requiring it installed (a trivial bootstrap
procedure is also a requirement, see the e-mail from Lucas).

> 
> I think this is what you mean, but it's not how I interpret the
> above.  I read this as, "tests can (and should) live anywhere across
> any set of repositories".  I think when you say "live anywhere", you
> mean this strictly from an autotest perspective and are trying to
> accomodate tests that live in upstream repositories.

> >  - Usage of the autotest library by test writers is optional
> >  - Tests are scripts returning 0 or error (any language)

I hope these two clarify the above.

> >  - Tests can be run individually or in sets
> >  - Tests should run fast, our target is seconds or a few minutes
> >  - The test runner is smart and “just works” by default
> >    - Trivial standard output (FAIL, PASS, SKIPPED)
> >    - Collect logs, OS data and other stuff (e.g. --record-video!)
> >    - Skip unsupported tests based on the environment they're run
> >    - Multiplex configurations / platforms when on the grid
> >    - Support to run tests “in the cloud”
> 
> This is where you start to lose me.  If all you're saying is, "QEMU
> can continue to use gtest to build out it's test infrastructure and
> autotest will learn how to use it", then we're in violent agreement.

Good :-)

Inside qemu.git, you can add as many instrumentation and helper
libraries as you want. As a developer it'll be up to you to setup
your environment to run the tests, and autotest will be just yet
another dependency.

QE will probably be interested in setting up a bot with several
tests from differente repositories, but that's their problem, not
yours.

BTW, please note that we're not trying to cover unit-tests in
autotest. At least not by the definition of unit-tests that I
know, which are code-level tests without running the application
itself (unit-tests test functions/methods only, not the
application as a whole).

Autotest is more about integration tests, which are complementary
to unit-tests in a "test driven development" environment: you
instantiate your program by using scripts and check if things
happen as you expect, as in this trivial example:

qemu-img-convert.sh
    #!/bin/bash
    qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2
    exec diff -q $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2

(of course, you could encapsulate your unit-test runner as one
test inside autotest)

The way I see test automation working at the developer level is
similar to what we have in WebKit (my previous project): dozens
of thousands of small test scripts (28k last time I counted), run
all the time on a grid of buildbots and on developer machines.
Tests are maintained by the devel team and constantly growing
because bugfixes and new features are acompanied by new tests.

And a broken test is considered as bad as a broken build.

BUT we'll need a lot of effort and time to get there. We hope
autotest will help somehow, but it's just a small baby step.

Cheers,
   - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 10:05 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
>> On 03/08/2012 09:07 AM, Ademar Reis wrote:
>>> On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
>>>> On 03/08/2012 08:49 AM, Ademar Reis wrote:
>>>>> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
>>>>>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
>>>>>>> Virt/qemu tests: Minimal guest images
>>>>>>> -------------------------------------
>>>>>>>
>>>>>>> In order to make development level test possible, we need the tests to run fast.
>>>>>>> In order to do that, a set of minimal guest images is being developed and we
>>>>>>> have a version for x86_64 ready and functional:
>>>>>>>
>>>>>>> https://github.com/autotest/buildroot-autotest
>>>>>>
>>>>>> I'm really not a fan of buildroot.  Note that in order to ship
>>>>>> binaries, full source needs to be provided in order to comply with
>>>>>> the GPL.  The FSF at least states that referring to another website
>>>>>> for source that's not under your control doesn't satisfy the
>>>>>> requirements of the GPL.
>>>>>>
>>>>>> Just out of curiosity, did you try to use qemu-test?  Is there a
>>>>>> reason you created something different?
>>>>>>
>>>>>> I think it's good that you're thinking about how to make writing
>>>>>> tests easier, but we have a growing test infrastructure in QEMU and
>>>>>> that's what I'd prefer people focused on.
>>>>>>
>>>>>
>>>>> You probably remember the long thread we had back in December on
>>>>> qemu-devel on this topic. Back then our message was "we have a
>>>>> growing test infrastructure in s/QEMU/autotest/ and that's what
>>>>> we'd prefer people focused on". :-)
>>>>>
>>>>>  From Dor:
>>>>>
>>>>> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
>>>>>
>>>>> """
>>>>> If you wish, you can challenge Lucas and Cleber w/ these type of
>>>>> requirements and we'll all improve as a result.
>>>>> """
>>>>>
>>>>> Your response was:
>>>>>
>>>>> """
>>>>> Well consider qemu-test the challenge. It's an existence proof
>>>>> that we can have a very easy to use framework for testing that
>>>>> runs extremely fast and is very easy to write tests for.
>>>>> """
>>>>>
>>>>> http://knowyourmeme.com/memes/challenge-accepted ;-)
>>>>>
>>>>> I particularly agreed with basically everything you said on that
>>>>> discussion regarding test simplification (I had just joined the
>>>>> team back then). To me, autotest has been focusing on QE-level,
>>>>> leaving the developer-level test requirements out. Now we're
>>>>> attacking this new front, and a lot of the requirements are
>>>>> indeed from that discussion.
>>>>
>>>> If you want to talk about this in terms of "requirements", my
>>>> requirement is for "developer-level" tests to live in qemu.git and
>>>> be integrated into make check.
>>>>
>>>> Just as we've been discussing and working on since the previous set of discussions.
>>>>
>>>>> By simplifying the design and bringing barriers down, we hope to
>>>>> reach a broader audience and help developers write and maintain
>>>>> tests, benefiting from all the instrumentation that autotest
>>>>> brings. It's not going to be just about qemu (check the new test
>>>>> examples).
>>>>>
>>>>> We have a team fully dedicated to autotest and it's used not only
>>>>> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
>>>>> (these all have code contributions in autotest)
>>>>>
>>>>> That said, the current qemu-tests will probably be easily
>>>>> integrated into (the new) autotest and we hope that, given enough
>>>>> time, autotest will be good enough to relieve qemu from the
>>>>> framework maintenance and code duplication with other projects.
>>>>
>>>> autotest should not be the focal point for integration.  qemu.git should be.
>>>>
>>>> I'd be perfectly happy to review patches submitting the test
>>>> infrastructure from kvm-autotest into qemu.git (provided it didn't
>>>> have unreasonable external dependencies and fit into QEMU).
>>>>
>>>> Developer-level tests need to live where the developers live.  The
>>>> developers live in qemu.git.  See my other response on this thread
>>>> for the explanation of why this is so important.
>>>>
>>>
>>> Excelent, we're in the same page then. This was my number 1
>>> requirement when I was discussing the changes with Lucas and
>>> Cleber. For convenience, I'll repeat here what I wrote in a
>>> previous e-mail (no qemu-devel archive available yet to use as a
>>> reference).
>>>
>>> In summary, autotest is (or is going to be) a framework that
>>> provides:
>>>
>>>   - A test runner, with grid/cluster support and advanced
>>>     instrumentation
>>>   - A devel library and set of utilities for test writers
>>>   - A set of pre-built images (JeOS – Just Enough OS) for
>>>     test writers
>>
>> I don't think autotest is the right place for this to live.  We need
>> this directly in qemu.git otherwise we're severely limited in what
>> tests we can write.
>>
>> I guess that doesn't preclude autotest having its own JeOS
>> mechanism, but we clearly need one in qemu.git.
>>
>>>
>>> (attached is a picture showing what we want to achieve)
>>>
>>> If a project has an internal library or set of utilities that can
>>> be of general use, they can be submitted to autotest.git for
>>> inclusion, thus reaching a broader audience.
>>>
>>> A short summary of the plans:
>>>
>>>   - Tests can live anywhere and each devel team implements and
>>>     maintains their own set of tests
>>
>> Let me change this to:
>>
>> - Autotest will learn how to harness the tests that each development
>> team creates in their respective git repository.
>
> Yep, given this simple requirement: "tests return 0 or an error
> code". The output from the test runner will be a simple PASS/FAIL
> and stdout/stderr will be properly collected by the test runner
> and made available in a standard location. If using TAP, even
> better.

We use gtest, so this will not be the case.  I think it would be best for 
autotest to learn how to interact with the gtest protocol.

>
> This will allow trivial tests. As the test complexity grows, then
> developers may start using parts of the autotest library or
> utilities, thus requiring it installed (a trivial bootstrap
> procedure is also a requirement, see the e-mail from Lucas).

What is the value of the autotest library to a test writer other than being part 
of the "autotest framework"?

>> This is where you start to lose me.  If all you're saying is, "QEMU
>> can continue to use gtest to build out it's test infrastructure and
>> autotest will learn how to use it", then we're in violent agreement.
>
> Good :-)
>
> Inside qemu.git, you can add as many instrumentation and helper
> libraries as you want. As a developer it'll be up to you to setup
> your environment to run the tests, and autotest will be just yet
> another dependency.

I think there's an implicit assumption here that the tests in qemu.git will have 
a dependency on autotest.  I'd like to understand why this dependency is necessary.

Normally, autotest executes third party tests, what makes QEMU special here?

> QE will probably be interested in setting up a bot with several
> tests from differente repositories, but that's their problem, not
> yours.

Right, and this is what autotest is very good at :-)

> BTW, please note that we're not trying to cover unit-tests in
> autotest. At least not by the definition of unit-tests that I
> know, which are code-level tests without running the application
> itself (unit-tests test functions/methods only, not the
> application as a whole).

There is never a clear line between unit-tests and integration tests as long as 
you're talking about a single project.

I expect QEMU to grow tests for anything that involves launching QEMU directly. 
  Where I would not see QEMU growing tests for is things like launching QEMU 
through libvirt.

> Autotest is more about integration tests, which are complementary
> to unit-tests in a "test driven development" environment: you
> instantiate your program by using scripts and check if things
> happen as you expect, as in this trivial example:
>
> qemu-img-convert.sh
>      #!/bin/bash
>      qemu-img convert -O qcow2 $DATA/qemu-imgs/reference.vdi $TEMPDIR/output.qcow2
>      exec diff -q $TEMPDIR/output.qcow2 $DATA/qemu-imgs/reference.qcow2
>
> (of course, you could encapsulate your unit-test runner as one
> test inside autotest)
>
> The way I see test automation working at the developer level is
> similar to what we have in WebKit (my previous project): dozens
> of thousands of small test scripts (28k last time I counted), run
> all the time on a grid of buildbots and on developer machines.
> Tests are maintained by the devel team and constantly growing
> because bugfixes and new features are acompanied by new tests.
>
> And a broken test is considered as bad as a broken build.
>
> BUT we'll need a lot of effort and time to get there. We hope
> autotest will help somehow, but it's just a small baby step.

Yes, I have a similar view of the long term goal.  And just like those thousands 
of tests live in the main webkit tree, I think we need thousands of tests in 
qemu.git.

Regards,

Anthony Liguori

> Cheers,
>     - Ademar
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 11:03:54AM -0600, Anthony Liguori wrote:
> On 03/08/2012 10:05 AM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 09:07 AM, Ademar Reis wrote:
> >>>On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
> >>>>On 03/08/2012 08:49 AM, Ademar Reis wrote:
> >>>>>On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
> >>>>>>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
> >>>>>>>Virt/qemu tests: Minimal guest images
> >>>>>>>-------------------------------------
> >>>>>>>
> >>>>>>>In order to make development level test possible, we need the tests to run fast.
> >>>>>>>In order to do that, a set of minimal guest images is being developed and we
> >>>>>>>have a version for x86_64 ready and functional:
> >>>>>>>
> >>>>>>>https://github.com/autotest/buildroot-autotest
> >>>>>>
> >>>>>>I'm really not a fan of buildroot.  Note that in order to ship
> >>>>>>binaries, full source needs to be provided in order to comply with
> >>>>>>the GPL.  The FSF at least states that referring to another website
> >>>>>>for source that's not under your control doesn't satisfy the
> >>>>>>requirements of the GPL.
> >>>>>>
> >>>>>>Just out of curiosity, did you try to use qemu-test?  Is there a
> >>>>>>reason you created something different?
> >>>>>>
> >>>>>>I think it's good that you're thinking about how to make writing
> >>>>>>tests easier, but we have a growing test infrastructure in QEMU and
> >>>>>>that's what I'd prefer people focused on.
> >>>>>>
> >>>>>
> >>>>>You probably remember the long thread we had back in December on
> >>>>>qemu-devel on this topic. Back then our message was "we have a
> >>>>>growing test infrastructure in s/QEMU/autotest/ and that's what
> >>>>>we'd prefer people focused on". :-)
> >>>>>
> >>>>> From Dor:
> >>>>>
> >>>>>(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
> >>>>>
> >>>>>"""
> >>>>>If you wish, you can challenge Lucas and Cleber w/ these type of
> >>>>>requirements and we'll all improve as a result.
> >>>>>"""
> >>>>>
> >>>>>Your response was:
> >>>>>
> >>>>>"""
> >>>>>Well consider qemu-test the challenge. It's an existence proof
> >>>>>that we can have a very easy to use framework for testing that
> >>>>>runs extremely fast and is very easy to write tests for.
> >>>>>"""
> >>>>>
> >>>>>http://knowyourmeme.com/memes/challenge-accepted ;-)
> >>>>>
> >>>>>I particularly agreed with basically everything you said on that
> >>>>>discussion regarding test simplification (I had just joined the
> >>>>>team back then). To me, autotest has been focusing on QE-level,
> >>>>>leaving the developer-level test requirements out. Now we're
> >>>>>attacking this new front, and a lot of the requirements are
> >>>>>indeed from that discussion.
> >>>>
> >>>>If you want to talk about this in terms of "requirements", my
> >>>>requirement is for "developer-level" tests to live in qemu.git and
> >>>>be integrated into make check.
> >>>>
> >>>>Just as we've been discussing and working on since the previous set of discussions.
> >>>>
> >>>>>By simplifying the design and bringing barriers down, we hope to
> >>>>>reach a broader audience and help developers write and maintain
> >>>>>tests, benefiting from all the instrumentation that autotest
> >>>>>brings. It's not going to be just about qemu (check the new test
> >>>>>examples).
> >>>>>
> >>>>>We have a team fully dedicated to autotest and it's used not only
> >>>>>by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
> >>>>>(these all have code contributions in autotest)
> >>>>>
> >>>>>That said, the current qemu-tests will probably be easily
> >>>>>integrated into (the new) autotest and we hope that, given enough
> >>>>>time, autotest will be good enough to relieve qemu from the
> >>>>>framework maintenance and code duplication with other projects.
> >>>>
> >>>>autotest should not be the focal point for integration.  qemu.git should be.
> >>>>
> >>>>I'd be perfectly happy to review patches submitting the test
> >>>>infrastructure from kvm-autotest into qemu.git (provided it didn't
> >>>>have unreasonable external dependencies and fit into QEMU).
> >>>>
> >>>>Developer-level tests need to live where the developers live.  The
> >>>>developers live in qemu.git.  See my other response on this thread
> >>>>for the explanation of why this is so important.
> >>>>
> >>>
> >>>Excelent, we're in the same page then. This was my number 1
> >>>requirement when I was discussing the changes with Lucas and
> >>>Cleber. For convenience, I'll repeat here what I wrote in a
> >>>previous e-mail (no qemu-devel archive available yet to use as a
> >>>reference).
> >>>
> >>>In summary, autotest is (or is going to be) a framework that
> >>>provides:
> >>>
> >>>  - A test runner, with grid/cluster support and advanced
> >>>    instrumentation
> >>>  - A devel library and set of utilities for test writers
> >>>  - A set of pre-built images (JeOS – Just Enough OS) for
> >>>    test writers
> >>
> >>I don't think autotest is the right place for this to live.  We need
> >>this directly in qemu.git otherwise we're severely limited in what
> >>tests we can write.
> >>
> >>I guess that doesn't preclude autotest having its own JeOS
> >>mechanism, but we clearly need one in qemu.git.
> >>
> >>>
> >>>(attached is a picture showing what we want to achieve)
> >>>
> >>>If a project has an internal library or set of utilities that can
> >>>be of general use, they can be submitted to autotest.git for
> >>>inclusion, thus reaching a broader audience.
> >>>
> >>>A short summary of the plans:
> >>>
> >>>  - Tests can live anywhere and each devel team implements and
> >>>    maintains their own set of tests
> >>
> >>Let me change this to:
> >>
> >>- Autotest will learn how to harness the tests that each development
> >>team creates in their respective git repository.
> >
> >Yep, given this simple requirement: "tests return 0 or an error
> >code". The output from the test runner will be a simple PASS/FAIL
> >and stdout/stderr will be properly collected by the test runner
> >and made available in a standard location. If using TAP, even
> >better.
> 
> We use gtest, so this will not be the case.  I think it would be
> best for autotest to learn how to interact with the gtest protocol.

I don't see a problem with this. It's very minor IMO, we can
probably speak gtest as well, but in the worst case, all logs
will be collected and are human-readable anyway.

> >
> >This will allow trivial tests. As the test complexity grows, then
> >developers may start using parts of the autotest library or
> >utilities, thus requiring it installed (a trivial bootstrap
> >procedure is also a requirement, see the e-mail from Lucas).
> 
> What is the value of the autotest library to a test writer other
> than being part of the "autotest framework"?

Well, that's the whole idea: there are tons of "goodies" in the
autotest library and in the utilities. You can either replicate
the code and maintain everything inside qemu.git or use what
autotest provides not just to QEMU but other projects as well.

For example, the autotest library provides ways to talk to
guests, check if they're alive, interact via ssh, open serial
channels, start/stop/install/kill/migrate VMs, record video, etc.
Why would you want to duplicate and maintain all this
functionality inside QEMU when this could be provided for free by
autotest, and shared with other projects?

Maybe we can split things a bit more and consider the creation of
a "libvirttest" (or whatever name makes sense) that would
implement all the virt testing infrastructure to be shared among
the interested parties:

 - QEMU
 - libvirt
 - spice
 - ...
 - QE teams (the ones who write complex and comprehensive tests
   in their own repo and will help a lot with the maintenance
   of such library)

But such code already exists and there are active contributors to
it in autotest.git. Our plan is to considerably improve it, but
if something is wrong with our plans, that's the moment to
improve things up by raising requirements.

> 
> >>This is where you start to lose me.  If all you're saying is, "QEMU
> >>can continue to use gtest to build out it's test infrastructure and
> >>autotest will learn how to use it", then we're in violent agreement.
> >
> >Good :-)
> >
> >Inside qemu.git, you can add as many instrumentation and helper
> >libraries as you want. As a developer it'll be up to you to setup
> >your environment to run the tests, and autotest will be just yet
> >another dependency.
> 
> I think there's an implicit assumption here that the tests in
> qemu.git will have a dependency on autotest.  I'd like to understand
> why this dependency is necessary.
> 
> Normally, autotest executes third party tests, what makes QEMU special here?

We're just offering what we already have, thanks to all the
resources invested in the "kvm-" part of "kvm-autotest" through
years. Again, we could split that part out of autotest, but in
the end it's just a naming convention.

A different question would be: what's the point of QEMU writting
code to handle interaction with VMs if such code can be shared
among different projects? Why rewrite everything if there's a
team allocated to this kind of task already, with a substancial
codebase and expertise? Why not raise requirements instead?

> 
> >QE will probably be interested in setting up a bot with several
> >tests from differente repositories, but that's their problem, not
> >yours.
> 
> Right, and this is what autotest is very good at :-)

Yep, but a lot of code can be shared and we can have a common
API, there's no need to duplicate efforts.

I clearly see difference motivations for tests written by QE and
developers, but I don't see why they can't share a common library
and why they can't exchange tests, expertise and contribute with
each other.

For example, you could get patches refactoring or fixing QEMU
tests submitted by members of a QE team. And tests could
move between the QE and QEMU repositories.

> 
> >BTW, please note that we're not trying to cover unit-tests in
> >autotest. At least not by the definition of unit-tests that I
> >know, which are code-level tests without running the application
> >itself (unit-tests test functions/methods only, not the
> >application as a whole).
> 
> There is never a clear line between unit-tests and integration tests
> as long as you're talking about a single project.

By the book, this would be the difference:

unit-test:
    void main()
    {
        my_function();
    }

integration test (or validation test):
    void main()
    {
        exec("my-application");
    }

But that's all semantics, not important for this discussion IMO.

> 
> I expect QEMU to grow tests for anything that involves launching
> QEMU directly.  Where I would not see QEMU growing tests for is
> things like launching QEMU through libvirt.

Agree. For QEMU developers, libvirt should not be on the way, the
interaction should be minimal or non-existent.

That's an area which will require some work in libautotest,
because due to previous QE requirements, it now invokes libvirt
methods instead of QEMU directly for a lot of stuff. But it can
be done and is part of the plan.

<snip>

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/08/2012 02:59 PM, Ademar Reis wrote:
> Agree. For QEMU developers, libvirt should not be on the way, the
> interaction should be minimal or non-existent.
>
> That's an area which will require some work in libautotest,
> because due to previous QE requirements, it now invokes libvirt
> methods instead of QEMU directly for a lot of stuff. But it can
> be done and is part of the plan.

Just a little correction here, we have 2 different backends, one that 
uses libvirt and another that uses qemu directly, so no issue here at all.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/08/2012 02:59 PM, Ademar Reis wrote:
> Agree. For QEMU developers, libvirt should not be on the way, the
> interaction should be minimal or non-existent.
>
> That's an area which will require some work in libautotest,
> because due to previous QE requirements, it now invokes libvirt
> methods instead of QEMU directly for a lot of stuff. But it can
> be done and is part of the plan.

Just a little correction here, we have 2 different backends, one that 
uses libvirt and another that uses qemu directly, so no issue here at all.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 11:59 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 11:03:54AM -0600, Anthony Liguori wrote:
>> On 03/08/2012 10:05 AM, Ademar Reis wrote:
>>> On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote:
>>>> On 03/08/2012 09:07 AM, Ademar Reis wrote:
>>>>> On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote:
>>>>>> On 03/08/2012 08:49 AM, Ademar Reis wrote:
>>>>>>> On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote:
>>>>>>>> On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote:
>>>>>>>>> Virt/qemu tests: Minimal guest images
>>>>>>>>> -------------------------------------
>>>>>>>>>
>>>>>>>>> In order to make development level test possible, we need the tests to run fast.
>>>>>>>>> In order to do that, a set of minimal guest images is being developed and we
>>>>>>>>> have a version for x86_64 ready and functional:
>>>>>>>>>
>>>>>>>>> https://github.com/autotest/buildroot-autotest
>>>>>>>>
>>>>>>>> I'm really not a fan of buildroot.  Note that in order to ship
>>>>>>>> binaries, full source needs to be provided in order to comply with
>>>>>>>> the GPL.  The FSF at least states that referring to another website
>>>>>>>> for source that's not under your control doesn't satisfy the
>>>>>>>> requirements of the GPL.
>>>>>>>>
>>>>>>>> Just out of curiosity, did you try to use qemu-test?  Is there a
>>>>>>>> reason you created something different?
>>>>>>>>
>>>>>>>> I think it's good that you're thinking about how to make writing
>>>>>>>> tests easier, but we have a growing test infrastructure in QEMU and
>>>>>>>> that's what I'd prefer people focused on.
>>>>>>>>
>>>>>>>
>>>>>>> You probably remember the long thread we had back in December on
>>>>>>> qemu-devel on this topic. Back then our message was "we have a
>>>>>>> growing test infrastructure in s/QEMU/autotest/ and that's what
>>>>>>> we'd prefer people focused on". :-)
>>>>>>>
>>>>>>>  From Dor:
>>>>>>>
>>>>>>> (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html)
>>>>>>>
>>>>>>> """
>>>>>>> If you wish, you can challenge Lucas and Cleber w/ these type of
>>>>>>> requirements and we'll all improve as a result.
>>>>>>> """
>>>>>>>
>>>>>>> Your response was:
>>>>>>>
>>>>>>> """
>>>>>>> Well consider qemu-test the challenge. It's an existence proof
>>>>>>> that we can have a very easy to use framework for testing that
>>>>>>> runs extremely fast and is very easy to write tests for.
>>>>>>> """
>>>>>>>
>>>>>>> http://knowyourmeme.com/memes/challenge-accepted ;-)
>>>>>>>
>>>>>>> I particularly agreed with basically everything you said on that
>>>>>>> discussion regarding test simplification (I had just joined the
>>>>>>> team back then). To me, autotest has been focusing on QE-level,
>>>>>>> leaving the developer-level test requirements out. Now we're
>>>>>>> attacking this new front, and a lot of the requirements are
>>>>>>> indeed from that discussion.
>>>>>>
>>>>>> If you want to talk about this in terms of "requirements", my
>>>>>> requirement is for "developer-level" tests to live in qemu.git and
>>>>>> be integrated into make check.
>>>>>>
>>>>>> Just as we've been discussing and working on since the previous set of discussions.
>>>>>>
>>>>>>> By simplifying the design and bringing barriers down, we hope to
>>>>>>> reach a broader audience and help developers write and maintain
>>>>>>> tests, benefiting from all the instrumentation that autotest
>>>>>>> brings. It's not going to be just about qemu (check the new test
>>>>>>> examples).
>>>>>>>
>>>>>>> We have a team fully dedicated to autotest and it's used not only
>>>>>>> by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc
>>>>>>> (these all have code contributions in autotest)
>>>>>>>
>>>>>>> That said, the current qemu-tests will probably be easily
>>>>>>> integrated into (the new) autotest and we hope that, given enough
>>>>>>> time, autotest will be good enough to relieve qemu from the
>>>>>>> framework maintenance and code duplication with other projects.
>>>>>>
>>>>>> autotest should not be the focal point for integration.  qemu.git should be.
>>>>>>
>>>>>> I'd be perfectly happy to review patches submitting the test
>>>>>> infrastructure from kvm-autotest into qemu.git (provided it didn't
>>>>>> have unreasonable external dependencies and fit into QEMU).
>>>>>>
>>>>>> Developer-level tests need to live where the developers live.  The
>>>>>> developers live in qemu.git.  See my other response on this thread
>>>>>> for the explanation of why this is so important.
>>>>>>
>>>>>
>>>>> Excelent, we're in the same page then. This was my number 1
>>>>> requirement when I was discussing the changes with Lucas and
>>>>> Cleber. For convenience, I'll repeat here what I wrote in a
>>>>> previous e-mail (no qemu-devel archive available yet to use as a
>>>>> reference).
>>>>>
>>>>> In summary, autotest is (or is going to be) a framework that
>>>>> provides:
>>>>>
>>>>>   - A test runner, with grid/cluster support and advanced
>>>>>     instrumentation
>>>>>   - A devel library and set of utilities for test writers
>>>>>   - A set of pre-built images (JeOS – Just Enough OS) for
>>>>>     test writers
>>>>
>>>> I don't think autotest is the right place for this to live.  We need
>>>> this directly in qemu.git otherwise we're severely limited in what
>>>> tests we can write.
>>>>
>>>> I guess that doesn't preclude autotest having its own JeOS
>>>> mechanism, but we clearly need one in qemu.git.
>>>>
>>>>>
>>>>> (attached is a picture showing what we want to achieve)
>>>>>
>>>>> If a project has an internal library or set of utilities that can
>>>>> be of general use, they can be submitted to autotest.git for
>>>>> inclusion, thus reaching a broader audience.
>>>>>
>>>>> A short summary of the plans:
>>>>>
>>>>>   - Tests can live anywhere and each devel team implements and
>>>>>     maintains their own set of tests
>>>>
>>>> Let me change this to:
>>>>
>>>> - Autotest will learn how to harness the tests that each development
>>>> team creates in their respective git repository.
>>>
>>> Yep, given this simple requirement: "tests return 0 or an error
>>> code". The output from the test runner will be a simple PASS/FAIL
>>> and stdout/stderr will be properly collected by the test runner
>>> and made available in a standard location. If using TAP, even
>>> better.
>>
>> We use gtest, so this will not be the case.  I think it would be
>> best for autotest to learn how to interact with the gtest protocol.
>
> I don't see a problem with this. It's very minor IMO, we can
> probably speak gtest as well, but in the worst case, all logs
> will be collected and are human-readable anyway.
>
>>>
>>> This will allow trivial tests. As the test complexity grows, then
>>> developers may start using parts of the autotest library or
>>> utilities, thus requiring it installed (a trivial bootstrap
>>> procedure is also a requirement, see the e-mail from Lucas).
>>
>> What is the value of the autotest library to a test writer other
>> than being part of the "autotest framework"?
>
> Well, that's the whole idea: there are tons of "goodies" in the
> autotest library and in the utilities. You can either replicate
> the code and maintain everything inside qemu.git or use what
> autotest provides not just to QEMU but other projects as well.
>
> For example, the autotest library provides ways to talk to
> guests, check if they're alive, interact via ssh, open serial
> channels, start/stop/install/kill/migrate VMs, record video, etc.
> Why would you want to duplicate and maintain all this
> functionality inside QEMU when this could be provided for free by
> autotest, and shared with other projects?
>
> Maybe we can split things a bit more and consider the creation of
> a "libvirttest" (or whatever name makes sense) that would
> implement all the virt testing infrastructure to be shared among
> the interested parties:
>
>   - QEMU
>   - libvirt
>   - spice
>   - ...
>   - QE teams (the ones who write complex and comprehensive tests
>     in their own repo and will help a lot with the maintenance
>     of such library)
>
> But such code already exists and there are active contributors to
> it in autotest.git. Our plan is to considerably improve it, but
> if something is wrong with our plans, that's the moment to
> improve things up by raising requirements.
>
>>
>>>> This is where you start to lose me.  If all you're saying is, "QEMU
>>>> can continue to use gtest to build out it's test infrastructure and
>>>> autotest will learn how to use it", then we're in violent agreement.
>>>
>>> Good :-)
>>>
>>> Inside qemu.git, you can add as many instrumentation and helper
>>> libraries as you want. As a developer it'll be up to you to setup
>>> your environment to run the tests, and autotest will be just yet
>>> another dependency.
>>
>> I think there's an implicit assumption here that the tests in
>> qemu.git will have a dependency on autotest.  I'd like to understand
>> why this dependency is necessary.
>>
>> Normally, autotest executes third party tests, what makes QEMU special here?
>
> We're just offering what we already have, thanks to all the
> resources invested in the "kvm-" part of "kvm-autotest" through
> years. Again, we could split that part out of autotest, but in
> the end it's just a naming convention.
>
> A different question would be: what's the point of QEMU writting
> code to handle interaction with VMs if such code can be shared
> among different projects? Why rewrite everything if there's a
> team allocated to this kind of task already, with a substancial
> codebase and expertise? Why not raise requirements instead?
>
>>
>>> QE will probably be interested in setting up a bot with several
>>> tests from differente repositories, but that's their problem, not
>>> yours.
>>
>> Right, and this is what autotest is very good at :-)
>
> Yep, but a lot of code can be shared and we can have a common
> API, there's no need to duplicate efforts.

I think the right way to address this is sending patches to qemu-devel.  If you 
think there are bits of autotest that could be integrated, wonderful.

But you are the autotest experts, not me.  You can tell me that there's 
wonderful things that should be reused but I have no idea what they are and am 
not in a position to figure them out.

Hopefully you'll agree, that autotest is not currently conducive to a buffet 
style consumption model.  As an outsider, it looks pretty much all or nothing to me.

> I clearly see difference motivations for tests written by QE and
> developers, but I don't see why they can't share a common library
> and why they can't exchange tests, expertise and contribute with
> each other.

I have no moral objections to sharing code.  But I feel strongly that we need to 
increase our testing in qemu.git.  If you think that there is code in autotest 
that could contribute to that, patches are more than welcome.

> By the book, this would be the difference:
>
> unit-test:
>      void main()
>      {
>          my_function();
>      }
>
> integration test (or validation test):
>      void main()
>      {
>          exec("my-application");
>      }
>
> But that's all semantics, not important for this discussion IMO.

Heh, the lines are not quite that clear.  We can debate the Unified Process here 
but I agree, it's not important for this discussion :-)

>> I expect QEMU to grow tests for anything that involves launching
>> QEMU directly.  Where I would not see QEMU growing tests for is
>> things like launching QEMU through libvirt.
>
> Agree. For QEMU developers, libvirt should not be on the way, the
> interaction should be minimal or non-existent.
>
> That's an area which will require some work in libautotest,
> because due to previous QE requirements, it now invokes libvirt
> methods instead of QEMU directly for a lot of stuff. But it can
> be done and is part of the plan.

If you're talking about libautotest as an abstraction layer for launching QEMU, 
I don't think that's something we should use in upstream QEMU.  Abstraction 
layers are okay when the things you are abstracting are well defined and 
relatively static.  We're talking about testing the tip of a development tree 
though and coordinating changes with an abstraction layer would be counter 
productive.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> On 03/08/2012 11:59 AM, Ademar Reis wrote:

<snip>

> 
> >>I expect QEMU to grow tests for anything that involves launching
> >>QEMU directly.  Where I would not see QEMU growing tests for is
> >>things like launching QEMU through libvirt.
> >
> >Agree. For QEMU developers, libvirt should not be on the way, the
> >interaction should be minimal or non-existent.
> >
> >That's an area which will require some work in libautotest,
> >because due to previous QE requirements, it now invokes libvirt
> >methods instead of QEMU directly for a lot of stuff. But it can
> >be done and is part of the plan.
> 
> If you're talking about libautotest as an abstraction layer for
> launching QEMU, I don't think that's something we should use in
> upstream QEMU.  Abstraction layers are okay when the things you are
> abstracting are well defined and relatively static.  We're talking
> about testing the tip of a development tree though and coordinating
> changes with an abstraction layer would be counter productive.
>

Valid point. If I'm a qemu developer and I want to make huge
changes, ideally I shouldn't have to submit patches to
libautotest because the tests in qemu.org will break.

I don't know how likely such scenario is (Lucas is maitaining
kvm-autotest for years now and the integration hardly breaks)
But we do have a conflict of interests, so lets evaluate a
few options:

My main interests are:
    - remove barriers for developers to write tests
    - share the testing effort between multiple projects (qemu,
      libvirt, ovirt, spice, QE, etc...)

Now the options:

1. We keep the invocation stuff in libautotest and when writting
a complex test in qemu.git, the test writter can choose to use it
because of the goodies there.

Pros:
  - Lots of codepaths implemented both in python and as cmd-line
    utilities: less lines of code to write a test, smaller
    barrier for the developer.
  - Mature code in the test library.
  - Goodies such as video-recording and system info collection
    in a standard way for different projects.
  - Code shared with other teams.
  - The same test code can run on old platforms because
    libautotest has a compatibility layer, thus enabling a
    larger grid of tests.
  - QE teams will be closer to the development, sharing
    some of their testing code and maintenance burden.
  - It can co-exist with qemu-test.

Cons:
  - You depend on an external library for some of your tests (in
    a way, a circular dependency)

2. We "move" (or implement) all the libautotest stuff related to
qemu invocation and interaction into qemu.git, forbidding the
dependency of autotest in qemu.git.

In a way, that's what tends to happen with qemu-test if things
keep going on AS IS in qemu.org (I'm absolutely sure that you'll
replicate a lot of what autotest does as you increase the test
coverage and complexity).

Pros:
  - Everything under qemu.git control: you break it, you fix it.
  - Organic growth of the test infra-structure in QEMU

Cons:
  - Lot of code will be duplicated to cover the main code paths:
    writting tests will require writting/supporting considerable
    ammount of code (that already exists in autotest).
  - QE will be alienated from the qemu test effort. There will be
    no integration between the QE efforts and the maintenance of
    the qemu developer-level tests.
  - You don't get the goodies from autotest (standardized system
    info collection, video recording, grid support, etc).
  - The new tests will work only on the master branch.


3. A mix of (1) and (2): we "move" everything under qemu.git, but
keep the compatibility layer and provide a "libqemutest" for
third-parties.

Anybody writting a test that interacts with qemu will use this
library: qemu, libvirt, spice, ovirt, QE tests, etc.

There's current code written in python (which can be encapsulated
into cmd line utilities), but the library can provide stuff in
shell, perl, whatever. You just have to provide an API for
third-parties.

To avoid the burden of the qemu-devel list for collaborators, a
maintainer collects the contributions from third-parties.

Pros:
  - Everything under qemu.git control.
  - API available for upper layers in the stack (libvirt, QE,
    autotest, ovirt, spice, etc), keeping an unified effort on
    testing.
  - Reuse the current code from autotest (refactorings are
    planned anyway).

Cons:
  - A lot of work to be done by qemu developers.
  - A lot of work to be done by autotest: third-parties will keep
    using autotest instead of libqemutest for a while.

(1) requires work only on autotest by now... you'll wait and see
how far we can go (just raise your concerns now if you plan to
block the autotest dependency in qemu.git). In parallel, you can
keep the work on qemu-test and let the decision up to the test
writter.

(2) is the status quo of qemu.org

(3) could start with a patch submission from the autotest
developers to qemu-devel in the near future. It's not much
different than (1).

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 03:02 PM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> On 03/08/2012 11:59 AM, Ademar Reis wrote:
>
> <snip>
>
>>
>>>> I expect QEMU to grow tests for anything that involves launching
>>>> QEMU directly.  Where I would not see QEMU growing tests for is
>>>> things like launching QEMU through libvirt.
>>>
>>> Agree. For QEMU developers, libvirt should not be on the way, the
>>> interaction should be minimal or non-existent.
>>>
>>> That's an area which will require some work in libautotest,
>>> because due to previous QE requirements, it now invokes libvirt
>>> methods instead of QEMU directly for a lot of stuff. But it can
>>> be done and is part of the plan.
>>
>> If you're talking about libautotest as an abstraction layer for
>> launching QEMU, I don't think that's something we should use in
>> upstream QEMU.  Abstraction layers are okay when the things you are
>> abstracting are well defined and relatively static.  We're talking
>> about testing the tip of a development tree though and coordinating
>> changes with an abstraction layer would be counter productive.
>>
>
> Valid point. If I'm a qemu developer and I want to make huge
> changes, ideally I shouldn't have to submit patches to
> libautotest because the tests in qemu.org will break.
>
> I don't know how likely such scenario is (Lucas is maitaining
> kvm-autotest for years now and the integration hardly breaks)
> But we do have a conflict of interests, so lets evaluate a
> few options:

Because kvm-autotest doesn't have thousands of tests and isn't part of 200+ 
developers fast paths :-)

>
> My main interests are:
>      - remove barriers for developers to write tests

This is my number one priority.

>      - share the testing effort between multiple projects (qemu,
>        libvirt, ovirt, spice, QE, etc...)

This is my number 99 priority.

I think you could achieve practical code sharing by decomposing kvm-autotest 
into a set of useful, independent tools.  For instance....

> Now the options:
>
> 1. We keep the invocation stuff in libautotest and when writting
> a complex test in qemu.git, the test writter can choose to use it
> because of the goodies there.
>
> Pros:
>    - Lots of codepaths implemented both in python and as cmd-line
>      utilities: less lines of code to write a test, smaller
>      barrier for the developer.

I've got an existence proof that this is not true.  The qemu-test tests are 
smaller than the corresponding autotest tests.

>    - Mature code in the test library.
>    - Goodies such as video-recording and system info collection
>      in a standard way for different projects.

Why does video-recording have to be part of libautotest?  Why can't you just 
have a simply utility that connects to a VNC session and records it?  Presumably 
that's all it's doing here.

Likewise, there are dozens of "system info collection" such as sosreport..  why 
fold this all into libautotest?

>    - Code shared with other teams.
>    - The same test code can run on old platforms because
>      libautotest has a compatibility layer, thus enabling a
>      larger grid of tests.

This is not a requirement from a QEMU perspective nor do I think it's useful.

>    - QE teams will be closer to the development, sharing
>      some of their testing code and maintenance burden.

The QE team could simply contribute tests to qemu.git to achieve this same goal.

> 2. We "move" (or implement) all the libautotest stuff related to
> qemu invocation and interaction into qemu.git, forbidding the
> dependency of autotest in qemu.git.

Let me just be really clear here.  In order to do TDD, tests have to be a 
mandatory part of the QEMU build process.  We cannot introduce new dependencies 
in order to build tests.

Making autotest an unconditional dependency of qemu.git is a non-starter for me.

> In a way, that's what tends to happen with qemu-test if things
> keep going on AS IS in qemu.org (I'm absolutely sure that you'll
> replicate a lot of what autotest does as you increase the test
> coverage and complexity).
>
> Pros:
>    - Everything under qemu.git control: you break it, you fix it.
>    - Organic growth of the test infra-structure in QEMU
>
> Cons:
>    - Lot of code will be duplicated to cover the main code paths:
>      writting tests will require writting/supporting considerable
>      ammount of code (that already exists in autotest).

Again, existence proof that this isn't true.

>    - QE will be alienated from the qemu test effort. There will be
>      no integration between the QE efforts and the maintenance of
>      the qemu developer-level tests.

I think we're a pretty friendly and open community :-)  There is no reason that 
QE should be "alienated" unless folks are choosing not to participate upstream.

>    - You don't get the goodies from autotest (standardized system
>      info collection, video recording, grid support, etc).
>    - The new tests will work only on the master branch.

This last one is a legitimate point that I have considered myself.  But I think 
the value of having tests in HEAD outweigh the cost here.

> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
> keep the compatibility layer and provide a "libqemutest" for
> third-parties.
>
> Anybody writting a test that interacts with qemu will use this
> library: qemu, libvirt, spice, ovirt, QE tests, etc.

I really see this all as over architecting to be honest.

Can someone explain in clear terms (without appealing to maturity, flexibility, 
or any other qualitative aspect) why it takes anything more than a few dozen 
shell functions to write all of the tests that are in kvm-autotest test today?

You launch qemu (by just using a command), and you interact with the monitor via 
QMP and HMP.  Why do you ever need anything more than that?

Why does libautotest even need to exist?

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >
> ><snip>
> >
> >>
> >>>>I expect QEMU to grow tests for anything that involves launching
> >>>>QEMU directly.  Where I would not see QEMU growing tests for is
> >>>>things like launching QEMU through libvirt.
> >>>
> >>>Agree. For QEMU developers, libvirt should not be on the way, the
> >>>interaction should be minimal or non-existent.
> >>>
> >>>That's an area which will require some work in libautotest,
> >>>because due to previous QE requirements, it now invokes libvirt
> >>>methods instead of QEMU directly for a lot of stuff. But it can
> >>>be done and is part of the plan.
> >>
> >>If you're talking about libautotest as an abstraction layer for
> >>launching QEMU, I don't think that's something we should use in
> >>upstream QEMU.  Abstraction layers are okay when the things you are
> >>abstracting are well defined and relatively static.  We're talking
> >>about testing the tip of a development tree though and coordinating
> >>changes with an abstraction layer would be counter productive.
> >>
> >
> >Valid point. If I'm a qemu developer and I want to make huge
> >changes, ideally I shouldn't have to submit patches to
> >libautotest because the tests in qemu.org will break.
> >
> >I don't know how likely such scenario is (Lucas is maitaining
> >kvm-autotest for years now and the integration hardly breaks)
> >But we do have a conflict of interests, so lets evaluate a
> >few options:
> 
> Because kvm-autotest doesn't have thousands of tests and isn't part
> of 200+ developers fast paths :-)
> 
> >
> >My main interests are:
> >     - remove barriers for developers to write tests
> 
> This is my number one priority.
> 
> >     - share the testing effort between multiple projects (qemu,
> >       libvirt, ovirt, spice, QE, etc...)
> 
> This is my number 99 priority.

You have the right to consider the interests of qemu only, but
qemu is used by several other teams who depend on it to implement
their own tests. Some collaboration would be extremely beneficial
for everybody.

Collaboration goes both ways: you get benefits from the
interaction and code reuse with different projects and teams, but
it may require a few tradeoffs.

> 
> I think you could achieve practical code sharing by decomposing
> kvm-autotest into a set of useful, independent tools.  For
> instance....
> 
> >Now the options:
> >
> >1. We keep the invocation stuff in libautotest and when writting
> >a complex test in qemu.git, the test writter can choose to use it
> >because of the goodies there.
> >
> >Pros:
> >   - Lots of codepaths implemented both in python and as cmd-line
> >     utilities: less lines of code to write a test, smaller
> >     barrier for the developer.
> 
> I've got an existence proof that this is not true.  The qemu-test
> tests are smaller than the corresponding autotest tests.

You're comparing developer-level tests with the existent QA-level
tests (much more complex).

We're talking about an approach where you'll need just a few
lines of code to instantiate a VM and interact with it.

> 
> >   - Mature code in the test library.
> >   - Goodies such as video-recording and system info collection
> >     in a standard way for different projects.
> 
> Why does video-recording have to be part of libautotest?  Why can't
> you just have a simply utility that connects to a VNC session and
> records it?  Presumably that's all it's doing here.
> 
> Likewise, there are dozens of "system info collection" such as
> sosreport..  why fold this all into libautotest?

For standardization: think outside of the qemu community. A lot
of our developers contribute code to multiple projects. It gets
easier if there's a standard way to achieve the same results when
writting tests for different projects.

Anyway, it's a minor issue.

> 
> >   - Code shared with other teams.
> >   - The same test code can run on old platforms because
> >     libautotest has a compatibility layer, thus enabling a
> >     larger grid of tests.
> 
> This is not a requirement from a QEMU perspective nor do I think it's useful.

Well, it's not a requirement, but you can't expect QE teams to
spend their time on tests that work only on HEAD. It makes no
sense from their point of view.


> 
> >   - QE teams will be closer to the development, sharing
> >     some of their testing code and maintenance burden.
> 
> The QE team could simply contribute tests to qemu.git to achieve this same goal.

They won't help you if you don't help them. Again, it doesn't
make sense for QE to work on HEAD. At least not the QE teams I'm
talking about.

And I say from my own experience: maintaining a set of thousands
of tests is far from trivial. You'll waste a lot of potential
contributions by alienating the fully-dedicated teams currently
working on QE.

> 
> >2. We "move" (or implement) all the libautotest stuff related to
> >qemu invocation and interaction into qemu.git, forbidding the
> >dependency of autotest in qemu.git.
> 
> Let me just be really clear here.  In order to do TDD, tests have to
> be a mandatory part of the QEMU build process.  We cannot introduce
> new dependencies in order to build tests.
> 
> Making autotest an unconditional dependency of qemu.git is a non-starter for me.

You have dependencies on gtest and other libraries. it's not that
different.

Plus it's not unconditional: the test runner will report tests
SKIPPED if a dependency is not present.

It's actually the oposite: you'll have to forbid developers from
submiting tests that use libautotest if you don't want the
dependency.

> 
> >In a way, that's what tends to happen with qemu-test if things
> >keep going on AS IS in qemu.org (I'm absolutely sure that you'll
> >replicate a lot of what autotest does as you increase the test
> >coverage and complexity).
> >
> >Pros:
> >   - Everything under qemu.git control: you break it, you fix it.
> >   - Organic growth of the test infra-structure in QEMU
> >
> >Cons:
> >   - Lot of code will be duplicated to cover the main code paths:
> >     writting tests will require writting/supporting considerable
> >     ammount of code (that already exists in autotest).
> 
> Again, existence proof that this isn't true.

Well, let's agree to disagree then. My bet is that qemu-test will
suffer a lot of growing pains if TDD is properly used, but I can't
prove it.

> 
> >   - QE will be alienated from the qemu test effort. There will be
> >     no integration between the QE efforts and the maintenance of
> >     the qemu developer-level tests.
> 
> I think we're a pretty friendly and open community :-)  There is no
> reason that QE should be "alienated" unless folks are choosing not
> to participate upstream.

For the exact same reasons you as a developer don't want to
implement tests inside autotest, QE won't want to implement tests
for qemu.git. It's out of their comfort zone, just put yourself
on their shoes.

> 
> >   - You don't get the goodies from autotest (standardized system
> >     info collection, video recording, grid support, etc).
> >   - The new tests will work only on the master branch.
> 
> This last one is a legitimate point that I have considered myself.
> But I think the value of having tests in HEAD outweigh the cost
> here.

Thinking about qemu only, agree.

> 
> >3. A mix of (1) and (2): we "move" everything under qemu.git, but
> >keep the compatibility layer and provide a "libqemutest" for
> >third-parties.
> >
> >Anybody writting a test that interacts with qemu will use this
> >library: qemu, libvirt, spice, ovirt, QE tests, etc.
> 
> I really see this all as over architecting to be honest.
> 
> Can someone explain in clear terms (without appealing to maturity,
> flexibility, or any other qualitative aspect) why it takes anything
> more than a few dozen shell functions to write all of the tests that
> are in kvm-autotest test today?

This is close to being offensive IMO, so I won't answer. :-)

> 
> You launch qemu (by just using a command), and you interact with the
> monitor via QMP and HMP.  Why do you ever need anything more than
> that?

Well, again, I'll agree to disagree.

> 
> Why does libautotest even need to exist?

You're looking at things from the sole point of view of a qemu
developer. If you don't want to see things from a different
perspective and accept any kind of tradeoffs for better
collaboration with other teams, then unfortunately there's not
much we can do. It's a pity, because collaboration goes both
ways.

We're trying very hard to satisfy your requirements and
interests. Don't you see any way you can help us satisfy
ours?

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 04:24 PM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> On 03/08/2012 03:02 PM, Ademar Reis wrote:
>>> On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>>>> On 03/08/2012 11:59 AM, Ademar Reis wrote:
>>>
>>> <snip>
>>>
>>>>
>>>>>> I expect QEMU to grow tests for anything that involves launching
>>>>>> QEMU directly.  Where I would not see QEMU growing tests for is
>>>>>> things like launching QEMU through libvirt.
>>>>>
>>>>> Agree. For QEMU developers, libvirt should not be on the way, the
>>>>> interaction should be minimal or non-existent.
>>>>>
>>>>> That's an area which will require some work in libautotest,
>>>>> because due to previous QE requirements, it now invokes libvirt
>>>>> methods instead of QEMU directly for a lot of stuff. But it can
>>>>> be done and is part of the plan.
>>>>
>>>> If you're talking about libautotest as an abstraction layer for
>>>> launching QEMU, I don't think that's something we should use in
>>>> upstream QEMU.  Abstraction layers are okay when the things you are
>>>> abstracting are well defined and relatively static.  We're talking
>>>> about testing the tip of a development tree though and coordinating
>>>> changes with an abstraction layer would be counter productive.
>>>>
>>>
>>> Valid point. If I'm a qemu developer and I want to make huge
>>> changes, ideally I shouldn't have to submit patches to
>>> libautotest because the tests in qemu.org will break.
>>>
>>> I don't know how likely such scenario is (Lucas is maitaining
>>> kvm-autotest for years now and the integration hardly breaks)
>>> But we do have a conflict of interests, so lets evaluate a
>>> few options:
>>
>> Because kvm-autotest doesn't have thousands of tests and isn't part
>> of 200+ developers fast paths :-)
>>
>>>
>>> My main interests are:
>>>      - remove barriers for developers to write tests
>>
>> This is my number one priority.
>>
>>>      - share the testing effort between multiple projects (qemu,
>>>        libvirt, ovirt, spice, QE, etc...)
>>
>> This is my number 99 priority.
>
> You have the right to consider the interests of qemu only, but
> qemu is used by several other teams who depend on it to implement
> their own tests. Some collaboration would be extremely beneficial
> for everybody.
>
> Collaboration goes both ways: you get benefits from the
> interaction and code reuse with different projects and teams, but
> it may require a few tradeoffs.

I think the problem here effectively boils down to "code sharing at any cost".

Let's look at it in purely practical terms.  How many lines of code would be 
saved here and what does that code do?

If the only sharing is some infrastructure bits in libautotest, then I suspect 
we're talking in the low 10ks of lines at the absolute maximum (and I don't 
think it's even close to that in reality).  Launching QEMU and connecting to QMP 
isn't that hard.  It's in the hundreds of lines of code.  What else is there 
that's need to write a test?

I don't think this is sufficiently compelling to add an unconditional autotest 
dependency to QEMU.

>> I think you could achieve practical code sharing by decomposing
>> kvm-autotest into a set of useful, independent tools.  For
>> instance....
>>
>>> Now the options:
>>>
>>> 1. We keep the invocation stuff in libautotest and when writting
>>> a complex test in qemu.git, the test writter can choose to use it
>>> because of the goodies there.
>>>
>>> Pros:
>>>    - Lots of codepaths implemented both in python and as cmd-line
>>>      utilities: less lines of code to write a test, smaller
>>>      barrier for the developer.
>>
>> I've got an existence proof that this is not true.  The qemu-test
>> tests are smaller than the corresponding autotest tests.
>
> You're comparing developer-level tests with the existent QA-level
> tests (much more complex).

Let's be specific then.  Look at device-add.sh in qemu-test.  It's 71LOC. 
pci_hotplug.py in autotest is 204LOC.

What's the "much more complex" part of pci_hotplug.py that is enabled by having 
the autotest framework?

> We're talking about an approach where you'll need just a few
> lines of code to instantiate a VM and interact with it.

This isn't that hard of a problem.  It just takes a few lines to launch QEMU in 
qemu-test too.

>>>    - Mature code in the test library.
>>>    - Goodies such as video-recording and system info collection
>>>      in a standard way for different projects.
>>
>> Why does video-recording have to be part of libautotest?  Why can't
>> you just have a simply utility that connects to a VNC session and
>> records it?  Presumably that's all it's doing here.
>>
>> Likewise, there are dozens of "system info collection" such as
>> sosreport..  why fold this all into libautotest?
>
> For standardization: think outside of the qemu community. A lot
> of our developers contribute code to multiple projects. It gets
> easier if there's a standard way to achieve the same results when
> writting tests for different projects.
>
> Anyway, it's a minor issue.

Yeah, I think this is outside of the scope of the current discussion as it's 
sort of like trying to boil the ocean :-)

>>>    - Code shared with other teams.
>>>    - The same test code can run on old platforms because
>>>      libautotest has a compatibility layer, thus enabling a
>>>      larger grid of tests.
>>
>> This is not a requirement from a QEMU perspective nor do I think it's useful.
>
> Well, it's not a requirement, but you can't expect QE teams to
> spend their time on tests that work only on HEAD. It makes no
> sense from their point of view.

They're certainly testing some sort of HEAD, even if it's not qemu.git.  I don't 
see any conflicting goals here.  Write a test against upstream, backport to a 
downstream if necessarily.

>
>>
>>>    - QE teams will be closer to the development, sharing
>>>      some of their testing code and maintenance burden.
>>
>> The QE team could simply contribute tests to qemu.git to achieve this same goal.
>
> They won't help you if you don't help them. Again, it doesn't
> make sense for QE to work on HEAD. At least not the QE teams I'm
> talking about.
>
> And I say from my own experience: maintaining a set of thousands
> of tests is far from trivial. You'll waste a lot of potential
> contributions by alienating the fully-dedicated teams currently
> working on QE.

And trying to maintain a thousand tests against any possible version is probably 
also unrealistic.

Is the expectation that all of the tests in WebKit work against any possible 
version of WebKit?  How do they enforce this since it's all in trunk?

>>> 2. We "move" (or implement) all the libautotest stuff related to
>>> qemu invocation and interaction into qemu.git, forbidding the
>>> dependency of autotest in qemu.git.
>>
>> Let me just be really clear here.  In order to do TDD, tests have to
>> be a mandatory part of the QEMU build process.  We cannot introduce
>> new dependencies in order to build tests.
>>
>> Making autotest an unconditional dependency of qemu.git is a non-starter for me.
>
> You have dependencies on gtest and other libraries. it's not that
> different.

We have a dependency on glib, which gtest happens to be a part of.  It was a big 
deal introducing that dependency and gtest was not the only (or primary) reason 
to introduce that dependency.

> Plus it's not unconditional: the test runner will report tests
> SKIPPED if a dependency is not present.

But then the tests aren't run so if most developers didn't have it installed, 
and most tests were written with it, most developers wouldn't be running most 
tests which defeats the purpose, no?

> It's actually the oposite: you'll have to forbid developers from
> submiting tests that use libautotest if you don't want the
> dependency.

I think part of the problem here is the abstract nature of this discussion. 
What is libautotest actually supposed to do?

This would be an easier discussion if we were talking about patches instead of 
proposals.

>>>    - QE will be alienated from the qemu test effort. There will be
>>>      no integration between the QE efforts and the maintenance of
>>>      the qemu developer-level tests.
>>
>> I think we're a pretty friendly and open community :-)  There is no
>> reason that QE should be "alienated" unless folks are choosing not
>> to participate upstream.
>
> For the exact same reasons you as a developer don't want to
> implement tests inside autotest, QE won't want to implement tests
> for qemu.git. It's out of their comfort zone, just put yourself
> on their shoes.

This is a really, really poor argument and I hope I don't need to go into 
details of why.  If the primary reason for libautotest is so the people writing 
tests for QEMU can avoid actually working with the developers of QEMU...  we've 
got a problem.

>>
>>>    - You don't get the goodies from autotest (standardized system
>>>      info collection, video recording, grid support, etc).
>>>    - The new tests will work only on the master branch.
>>
>> This last one is a legitimate point that I have considered myself.
>> But I think the value of having tests in HEAD outweigh the cost
>> here.
>
> Thinking about qemu only, agree.
>
>>
>>> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
>>> keep the compatibility layer and provide a "libqemutest" for
>>> third-parties.
>>>
>>> Anybody writting a test that interacts with qemu will use this
>>> library: qemu, libvirt, spice, ovirt, QE tests, etc.
>>
>> I really see this all as over architecting to be honest.
>>
>> Can someone explain in clear terms (without appealing to maturity,
>> flexibility, or any other qualitative aspect) why it takes anything
>> more than a few dozen shell functions to write all of the tests that
>> are in kvm-autotest test today?
>
> This is close to being offensive IMO, so I won't answer. :-)

I honestly don't mean it to be so I apologize if it comes off that way.  But I 
dislike arguing in such abstract terms.  What are the features that you think 
libautotest is needed for?

>> You launch qemu (by just using a command), and you interact with the
>> monitor via QMP and HMP.  Why do you ever need anything more than
>> that?
>
> Well, again, I'll agree to disagree.
>
>>
>> Why does libautotest even need to exist?
>
> You're looking at things from the sole point of view of a qemu
> developer. If you don't want to see things from a different
> perspective and accept any kind of tradeoffs for better
> collaboration with other teams, then unfortunately there's not
> much we can do. It's a pity, because collaboration goes both
> ways.

I need something more concrete then this.  What is it about libautotest that 
would allow collaboration?

> We're trying very hard to satisfy your requirements and
> interests. Don't you see any way you can help us satisfy
> ours?

I'm having a hard time understanding what your requirements are other than that 
we should write our tests using libautotest (which has not really been explained 
other than being levied as a requirement).

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
> On 03/08/2012 04:24 PM, Ademar Reis wrote:
> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >>>
> >>><snip>
> >>>
> >>>>
> >>>>>>I expect QEMU to grow tests for anything that involves launching
> >>>>>>QEMU directly.  Where I would not see QEMU growing tests for is
> >>>>>>things like launching QEMU through libvirt.
> >>>>>
> >>>>>Agree. For QEMU developers, libvirt should not be on the way, the
> >>>>>interaction should be minimal or non-existent.
> >>>>>
> >>>>>That's an area which will require some work in libautotest,
> >>>>>because due to previous QE requirements, it now invokes libvirt
> >>>>>methods instead of QEMU directly for a lot of stuff. But it can
> >>>>>be done and is part of the plan.
> >>>>
> >>>>If you're talking about libautotest as an abstraction layer for
> >>>>launching QEMU, I don't think that's something we should use in
> >>>>upstream QEMU.  Abstraction layers are okay when the things you are
> >>>>abstracting are well defined and relatively static.  We're talking
> >>>>about testing the tip of a development tree though and coordinating
> >>>>changes with an abstraction layer would be counter productive.
> >>>>
> >>>
> >>>Valid point. If I'm a qemu developer and I want to make huge
> >>>changes, ideally I shouldn't have to submit patches to
> >>>libautotest because the tests in qemu.org will break.
> >>>
> >>>I don't know how likely such scenario is (Lucas is maitaining
> >>>kvm-autotest for years now and the integration hardly breaks)
> >>>But we do have a conflict of interests, so lets evaluate a
> >>>few options:
> >>
> >>Because kvm-autotest doesn't have thousands of tests and isn't part
> >>of 200+ developers fast paths :-)
> >>
> >>>
> >>>My main interests are:
> >>>     - remove barriers for developers to write tests
> >>
> >>This is my number one priority.
> >>
> >>>     - share the testing effort between multiple projects (qemu,
> >>>       libvirt, ovirt, spice, QE, etc...)
> >>
> >>This is my number 99 priority.
> >
> >You have the right to consider the interests of qemu only, but
> >qemu is used by several other teams who depend on it to implement
> >their own tests. Some collaboration would be extremely beneficial
> >for everybody.
> >
> >Collaboration goes both ways: you get benefits from the
> >interaction and code reuse with different projects and teams, but
> >it may require a few tradeoffs.
> 
> I think the problem here effectively boils down to "code sharing at any cost".
> 

I may agree with the "sharing at any cost" if you reference parts
of the discussion from December. But if you re-read that thread
and compare with what we're proposing now, you'll see that we're
proposing something very different, based on the requirements
previously raised.

> Let's look at it in purely practical terms.  How many lines of code
> would be saved here and what does that code do?
> 
> If the only sharing is some infrastructure bits in libautotest, then
> I suspect we're talking in the low 10ks of lines at the absolute
> maximum (and I don't think it's even close to that in reality).
> Launching QEMU and connecting to QMP isn't that hard.  It's in the
> hundreds of lines of code.  What else is there that's need to write
> a test?
> 
> I don't think this is sufficiently compelling to add an
> unconditional autotest dependency to QEMU.
> 
> >>I think you could achieve practical code sharing by decomposing
> >>kvm-autotest into a set of useful, independent tools.  For
> >>instance....
> >>
> >>>Now the options:
> >>>
> >>>1. We keep the invocation stuff in libautotest and when writting
> >>>a complex test in qemu.git, the test writter can choose to use it
> >>>because of the goodies there.
> >>>
> >>>Pros:
> >>>   - Lots of codepaths implemented both in python and as cmd-line
> >>>     utilities: less lines of code to write a test, smaller
> >>>     barrier for the developer.
> >>
> >>I've got an existence proof that this is not true.  The qemu-test
> >>tests are smaller than the corresponding autotest tests.
> >
> >You're comparing developer-level tests with the existent QA-level
> >tests (much more complex).
> 
> Let's be specific then.  Look at device-add.sh in qemu-test.  It's
> 71LOC. pci_hotplug.py in autotest is 204LOC.
> 
> What's the "much more complex" part of pci_hotplug.py that is
> enabled by having the autotest framework?
> 
> >We're talking about an approach where you'll need just a few
> >lines of code to instantiate a VM and interact with it.
> 
> This isn't that hard of a problem.  It just takes a few lines to
> launch QEMU in qemu-test too.

Lucas or Cleber: could you list everything that is involved in
launching a VM? Which kind of goodies we get from autotest, even
a very lean (and fast) future version of it?

> 
> >>>   - Mature code in the test library.
> >>>   - Goodies such as video-recording and system info collection
> >>>     in a standard way for different projects.
> >>
> >>Why does video-recording have to be part of libautotest?  Why can't
> >>you just have a simply utility that connects to a VNC session and
> >>records it?  Presumably that's all it's doing here.
> >>
> >>Likewise, there are dozens of "system info collection" such as
> >>sosreport..  why fold this all into libautotest?
> >
> >For standardization: think outside of the qemu community. A lot
> >of our developers contribute code to multiple projects. It gets
> >easier if there's a standard way to achieve the same results when
> >writting tests for different projects.
> >
> >Anyway, it's a minor issue.
> 
> Yeah, I think this is outside of the scope of the current discussion
> as it's sort of like trying to boil the ocean :-)
> 
> >>>   - Code shared with other teams.
> >>>   - The same test code can run on old platforms because
> >>>     libautotest has a compatibility layer, thus enabling a
> >>>     larger grid of tests.
> >>
> >>This is not a requirement from a QEMU perspective nor do I think it's useful.
> >
> >Well, it's not a requirement, but you can't expect QE teams to
> >spend their time on tests that work only on HEAD. It makes no
> >sense from their point of view.
> 
> They're certainly testing some sort of HEAD, even if it's not
> qemu.git.  I don't see any conflicting goals here.  Write a test
> against upstream, backport to a downstream if necessarily.

Exactly: if you use a powerfull enough test framework/library
that allows they to extend the test. Otherwise they'll just write
their own version using autotest, which is far more powerfull.

> 
> >
> >>
> >>>   - QE teams will be closer to the development, sharing
> >>>     some of their testing code and maintenance burden.
> >>
> >>The QE team could simply contribute tests to qemu.git to achieve this same goal.
> >
> >They won't help you if you don't help them. Again, it doesn't
> >make sense for QE to work on HEAD. At least not the QE teams I'm
> >talking about.
> >
> >And I say from my own experience: maintaining a set of thousands
> >of tests is far from trivial. You'll waste a lot of potential
> >contributions by alienating the fully-dedicated teams currently
> >working on QE.
> 
> And trying to maintain a thousand tests against any possible version
> is probably also unrealistic.

That's why there's a large team of QEs fully allocated to it. And
what you described is exactly their job description.

> 
> Is the expectation that all of the tests in WebKit work against any
> possible version of WebKit?  How do they enforce this since it's all
> in trunk?

Webkit reached a TDD level where there's no need to support
multiple releases (think of Google Chrome). Each port does its
own QE, most of the time in trunk.

And a lot of engineers are fully dedicated to keep the testing
machinery running. It's very challenging (even though it's far
easier to test a webengine than to test an emulator/hypervisor).

In other words, there's no separation between QE tests and
developer-level tests in WebKit. But I don't expect qemu to get
there anytime soon.

> 
> >>>2. We "move" (or implement) all the libautotest stuff related to
> >>>qemu invocation and interaction into qemu.git, forbidding the
> >>>dependency of autotest in qemu.git.
> >>
> >>Let me just be really clear here.  In order to do TDD, tests have to
> >>be a mandatory part of the QEMU build process.  We cannot introduce
> >>new dependencies in order to build tests.
> >>
> >>Making autotest an unconditional dependency of qemu.git is a non-starter for me.
> >
> >You have dependencies on gtest and other libraries. it's not that
> >different.
> 
> We have a dependency on glib, which gtest happens to be a part of.
> It was a big deal introducing that dependency and gtest was not the
> only (or primary) reason to introduce that dependency.
> 
> >Plus it's not unconditional: the test runner will report tests
> >SKIPPED if a dependency is not present.
> 
> But then the tests aren't run so if most developers didn't have it
> installed, and most tests were written with it, most developers
> wouldn't be running most tests which defeats the purpose, no?

Part of a TDD approach is to have build and test bots frequently
running tests on multiple platforms with different
configurations.

You can't expect developers to run all tests all the time.

When I talk about the "grid mode", I'm talking about the
multiplexing of configuration and platforms that you get when you
run an instrummented test using autotest.

> 
> >It's actually the oposite: you'll have to forbid developers from
> >submiting tests that use libautotest if you don't want the
> >dependency.
> 
> I think part of the problem here is the abstract nature of this
> discussion. What is libautotest actually supposed to do?
> 
> This would be an easier discussion if we were talking about patches
> instead of proposals.

We're at the RFC level, patches are being worked on, but this
thread is helping us shape the work we'll do. For example, if the
usage of libautotest if forbidden in qemu.git, we can avoid a lot
of our to-do.

Ditto for the JeOS images. Based on this discussion we figured
out that we won't be able to distribute the binary images.
(thanks!)

> 
> >>>   - QE will be alienated from the qemu test effort. There will be
> >>>     no integration between the QE efforts and the maintenance of
> >>>     the qemu developer-level tests.
> >>
> >>I think we're a pretty friendly and open community :-)  There is no
> >>reason that QE should be "alienated" unless folks are choosing not
> >>to participate upstream.
> >
> >For the exact same reasons you as a developer don't want to
> >implement tests inside autotest, QE won't want to implement tests
> >for qemu.git. It's out of their comfort zone, just put yourself
> >on their shoes.
> 
> This is a really, really poor argument and I hope I don't need to go
> into details of why.  If the primary reason for libautotest is so
> the people writing tests for QEMU can avoid actually working with
> the developers of QEMU...  we've got a problem.

No, one of the benefits of having libautotest is to *collaborate*
with QE. I'll explain again:

- As a qemu developer, I don't want to spend my time learning and
  getting involved in autotest, which is a complex QE project
  (I heard this numerous times).

- As a Quality Engineer, I don't want to invest my time learning
  and getting involved into upstream qemu to test HEAD.

> >>
> >>>   - You don't get the goodies from autotest (standardized system
> >>>     info collection, video recording, grid support, etc).
> >>>   - The new tests will work only on the master branch.
> >>
> >>This last one is a legitimate point that I have considered myself.
> >>But I think the value of having tests in HEAD outweigh the cost
> >>here.
> >
> >Thinking about qemu only, agree.
> >
> >>
> >>>3. A mix of (1) and (2): we "move" everything under qemu.git, but
> >>>keep the compatibility layer and provide a "libqemutest" for
> >>>third-parties.
> >>>
> >>>Anybody writting a test that interacts with qemu will use this
> >>>library: qemu, libvirt, spice, ovirt, QE tests, etc.
> >>
> >>I really see this all as over architecting to be honest.
> >>
> >>Can someone explain in clear terms (without appealing to maturity,
> >>flexibility, or any other qualitative aspect) why it takes anything
> >>more than a few dozen shell functions to write all of the tests that
> >>are in kvm-autotest test today?
> >
> >This is close to being offensive IMO, so I won't answer. :-)
> 
> I honestly don't mean it to be so I apologize if it comes off that
> way.  But I dislike arguing in such abstract terms.  What are the
> features that you think libautotest is needed for?

As a manager relatively new to the team I lack the technical
expertise to put it in more practical terms. But check the other
e-mail from Lucas where there's a virtio-console test. I'll reply
to it.

> 
> >>You launch qemu (by just using a command), and you interact with the
> >>monitor via QMP and HMP.  Why do you ever need anything more than
> >>that?
> >
> >Well, again, I'll agree to disagree.
> >
> >>
> >>Why does libautotest even need to exist?
> >
> >You're looking at things from the sole point of view of a qemu
> >developer. If you don't want to see things from a different
> >perspective and accept any kind of tradeoffs for better
> >collaboration with other teams, then unfortunately there's not
> >much we can do. It's a pity, because collaboration goes both
> >ways.
> 
> I need something more concrete then this.  What is it about
> libautotest that would allow collaboration?
> 
> >We're trying very hard to satisfy your requirements and
> >interests. Don't you see any way you can help us satisfy
> >ours?
> 
> I'm having a hard time understanding what your requirements are
> other than that we should write our tests using libautotest (which
> has not really been explained other than being levied as a
> requirement).

I have no requirements other than the ones that interest you. I
have interests, listed in the beginning of this thread:

    - remove barriers for developers to write tests
    - share the testing effort between multiple projects
      (qemu, libvirt, ovirt, spice, QE, etc...)

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <areis@redhat.com> wrote:
> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
>> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
>> >>>   - QE will be alienated from the qemu test effort. There will be
>> >>>     no integration between the QE efforts and the maintenance of
>> >>>     the qemu developer-level tests.
>> >>
>> >>I think we're a pretty friendly and open community :-)  There is no
>> >>reason that QE should be "alienated" unless folks are choosing not
>> >>to participate upstream.
>> >
>> >For the exact same reasons you as a developer don't want to
>> >implement tests inside autotest, QE won't want to implement tests
>> >for qemu.git. It's out of their comfort zone, just put yourself
>> >on their shoes.
>>
>> This is a really, really poor argument and I hope I don't need to go
>> into details of why.  If the primary reason for libautotest is so
>> the people writing tests for QEMU can avoid actually working with
>> the developers of QEMU...  we've got a problem.
>
> No, one of the benefits of having libautotest is to *collaborate*
> with QE. I'll explain again:
>
> - As a qemu developer, I don't want to spend my time learning and
>  getting involved in autotest, which is a complex QE project
>  (I heard this numerous times).
>
> - As a Quality Engineer, I don't want to invest my time learning
>  and getting involved into upstream qemu to test HEAD.

I think this is the key point of the whole discussion - most of the
other topics have been distractions.  Both communities do testing but
we test different things and have different priorities.

For me this has been the big realization from this discussion.  I felt
kvm-autotest and qemu should share tests.  I was pushing for that but
after following this thread I don't think it makes sense, here's why:

The Quality Engineer you describe is not a QEMU upstream QE, instead
the QE has a broader and more downstream focus.  (This is why
comparisons with WebKit or other upstream projects doing testing are
not valid comparisons.)

There is not enough in common between upstream QEMU testing and
downstream KVM QE to make convergence a win-win.  libautotest sounds
like a technical solution to a people problem - the problem is that we
have different priorities.  We overlap but at the end of the day we do
different things.  We can make a best effort to converge but I don't
see incentives that will make this a success.  Creating an abstraction
library will be sub-optimal for both communities.

I think what's much more valuable is for qemu.git tests to be easily
hooked into autotest.  That way you get access to the testing that the
qemu community is doing for free.

And on the flip-side it would be awesome for kvm-autotest to cover
upstream.  As an example, QEMU uses buildbot with a public web
interface which shows the history of all runs and failure
notifications are sent to qemu-devel (it's not perfect but I think it
adds value to the community).  Can you can do daily/weekly qemu.git
upstream kvm-autotest runs, make the results easily accessible on the
public web, and perhaps hook into the qemu-devel mailing list?

That level of collaboration would allow both communities to do what
they want to do effectively, while still getting benefits from each
other.

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <areis@redhat.com> wrote:
> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >> >>>   - QE will be alienated from the qemu test effort. There will be
> >> >>>     no integration between the QE efforts and the maintenance of
> >> >>>     the qemu developer-level tests.
> >> >>
> >> >>I think we're a pretty friendly and open community :-)  There is no
> >> >>reason that QE should be "alienated" unless folks are choosing not
> >> >>to participate upstream.
> >> >
> >> >For the exact same reasons you as a developer don't want to
> >> >implement tests inside autotest, QE won't want to implement tests
> >> >for qemu.git. It's out of their comfort zone, just put yourself
> >> >on their shoes.
> >>
> >> This is a really, really poor argument and I hope I don't need to go
> >> into details of why.  If the primary reason for libautotest is so
> >> the people writing tests for QEMU can avoid actually working with
> >> the developers of QEMU...  we've got a problem.
> >
> > No, one of the benefits of having libautotest is to *collaborate*
> > with QE. I'll explain again:
> >
> > - As a qemu developer, I don't want to spend my time learning and
> >  getting involved in autotest, which is a complex QE project
> >  (I heard this numerous times).
> >
> > - As a Quality Engineer, I don't want to invest my time learning
> >  and getting involved into upstream qemu to test HEAD.
> 
> I think this is the key point of the whole discussion - most of the
> other topics have been distractions.  Both communities do testing but
> we test different things and have different priorities.
> 
> For me this has been the big realization from this discussion.  I felt
> kvm-autotest and qemu should share tests.  I was pushing for that but
> after following this thread I don't think it makes sense, here's why:
> 
> The Quality Engineer you describe is not a QEMU upstream QE, instead
> the QE has a broader and more downstream focus.  (This is why
> comparisons with WebKit or other upstream projects doing testing are
> not valid comparisons.)

Lucas, Cleber and the others red-hatters should remembers this
from my internal presentation, it was the first point I made:
QE and Developers have very different goals and interests.

Which is why we're pushing all these changes in autotest. We see
opportunities for collaboration, but we do realize the difference.

And look: Lucas and Cleber are not QE, they're developers working
on the autotest framework/library/whatever. We'll need similar
positions inside qemu as the test infra-structure grows.

> 
> There is not enough in common between upstream QEMU testing and
> downstream KVM QE to make convergence a win-win.  libautotest sounds
> like a technical solution to a people problem - the problem is that we
> have different priorities.  We overlap but at the end of the day we do
> different things.  We can make a best effort to converge but I don't
> see incentives that will make this a success.  Creating an abstraction
> library will be sub-optimal for both communities.

This is the part I don't agree, but in the end it's a matter of opinion.

> 
> I think what's much more valuable is for qemu.git tests to be easily
> hooked into autotest.  That way you get access to the testing that the
> qemu community is doing for free.

We get this by default in our plans: any application that can be
run and returns 0/error can be run as a test inside autotest.

What we're asking is the *possibility* of a developer using
libautotest when writting a complex test.

- We're not asking everybody to use autotest
- We're not saying autotest is better for everything
- We're not requiring you to install autotest on your machine
  (even though the process will be trivial)

- We're asking that, if a developer is going to write a test
  together with his patch to submit to qemu.git, he should be
  allowed to use libautotest at his own discretion...
- Ditto for using the test-runner: even if tests are simple
  scripts, there will be benefits in using our optional
  test-runner during development.

This is how I see the interaction with autotest-based tests:

    $ make check-full
    tests.d/my-test-using-qemu-tests.sh -- PASS
    tests.d/trivial-test.sh -- PASS
    tests.d/my-test-using-libautotest -- SKIP
    ...

    $ yum install libautotest
    # (or whatever the bootstrap procedure is)
    $ make check-full
    tests.d/my-test-using-qemu-tests.sh -- PASS
    tests.d/trivial-test.sh -- PASS
    tests.d/my-test-using-libautotest -- PASS
    ...

But for some reason that I still don't understand, the simple
acceptance of the optional dependency of libautotest in qemu is
seen as a bad thing.

If the ultimate requirement now is "we don't want to use an
external library, we don't want to maintain one ourselves and we
don't want tests to cover things out of qemu", then clearly
there's no space for autotest in qemu.git and we can move on.

For developers of libvirt, spice, ovirt and any other project up
in the stack, writting code similar to what's part of qemu-test
makes no sense at all and libautotest is a real need. We would
like qemu to embrace autotest as well thus providing
cross-project developers a similar experience and sharing the
maintenance burden, but that might be too ambitious.

Anyway, we're at the RFC stage, and this discussion has been
valuable. If for nothing else, to validate that there are no
significant technical requirements we're missing, which makes me
happy.

We have a lot of work to do and hopefully the value of autotest
to qemu will be seen as patches keep flowing and other developers
use it. In the next kvm-forum we can discuss this again in
person, maybe with some beers around us. :-)

> 
> And on the flip-side it would be awesome for kvm-autotest to cover
> upstream.  As an example, QEMU uses buildbot with a public web
> interface which shows the history of all runs and failure
> notifications are sent to qemu-devel (it's not perfect but I think it
> adds value to the community).  Can you can do daily/weekly qemu.git
> upstream kvm-autotest runs, make the results easily accessible on the
> public web, and perhaps hook into the qemu-devel mailing list?

That's also in the pipeline, but we might not have resources
anytime soon.

> That level of collaboration would allow both communities to do what
> they want to do effectively, while still getting benefits from each
> other.

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis <areis@redhat.com> wrote:
> On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
>> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <areis@redhat.com> wrote:
>> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
>> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
>> >> >>>   - QE will be alienated from the qemu test effort. There will be
>> >> >>>     no integration between the QE efforts and the maintenance of
>> >> >>>     the qemu developer-level tests.
>> >> >>
>> >> >>I think we're a pretty friendly and open community :-)  There is no
>> >> >>reason that QE should be "alienated" unless folks are choosing not
>> >> >>to participate upstream.
>> >> >
>> >> >For the exact same reasons you as a developer don't want to
>> >> >implement tests inside autotest, QE won't want to implement tests
>> >> >for qemu.git. It's out of their comfort zone, just put yourself
>> >> >on their shoes.
>> >>
>> >> This is a really, really poor argument and I hope I don't need to go
>> >> into details of why.  If the primary reason for libautotest is so
>> >> the people writing tests for QEMU can avoid actually working with
>> >> the developers of QEMU...  we've got a problem.
>> >
>> > No, one of the benefits of having libautotest is to *collaborate*
>> > with QE. I'll explain again:
>> >
>> > - As a qemu developer, I don't want to spend my time learning and
>> >  getting involved in autotest, which is a complex QE project
>> >  (I heard this numerous times).
>> >
>> > - As a Quality Engineer, I don't want to invest my time learning
>> >  and getting involved into upstream qemu to test HEAD.
>>
>> I think this is the key point of the whole discussion - most of the
>> other topics have been distractions.  Both communities do testing but
>> we test different things and have different priorities.
>>
>> For me this has been the big realization from this discussion.  I felt
>> kvm-autotest and qemu should share tests.  I was pushing for that but
>> after following this thread I don't think it makes sense, here's why:
>>
>> The Quality Engineer you describe is not a QEMU upstream QE, instead
>> the QE has a broader and more downstream focus.  (This is why
>> comparisons with WebKit or other upstream projects doing testing are
>> not valid comparisons.)
>
> Lucas, Cleber and the others red-hatters should remembers this
> from my internal presentation, it was the first point I made:
> QE and Developers have very different goals and interests.
>
> Which is why we're pushing all these changes in autotest. We see
> opportunities for collaboration, but we do realize the difference.
>
> And look: Lucas and Cleber are not QE, they're developers working
> on the autotest framework/library/whatever. We'll need similar
> positions inside qemu as the test infra-structure grows.

I don't understand this last paragraph.  If qemu.git upstream was
doing full-scale QE it would work fine because the differences that
I've described and you also have pointed out would be absent.

>> I think what's much more valuable is for qemu.git tests to be easily
>> hooked into autotest.  That way you get access to the testing that the
>> qemu community is doing for free.
>
> We get this by default in our plans: any application that can be
> run and returns 0/error can be run as a test inside autotest.

Yes, and I think that part of the plan makes sense.

> What we're asking is the *possibility* of a developer using
> libautotest when writting a complex test.
>
> - We're not asking everybody to use autotest
> - We're not saying autotest is better for everything
> - We're not requiring you to install autotest on your machine
>  (even though the process will be trivial)
>
> - We're asking that, if a developer is going to write a test
>  together with his patch to submit to qemu.git, he should be
>  allowed to use libautotest at his own discretion...
> - Ditto for using the test-runner: even if tests are simple
>  scripts, there will be benefits in using our optional
>  test-runner during development.
>
> This is how I see the interaction with autotest-based tests:
>
>    $ make check-full
>    tests.d/my-test-using-qemu-tests.sh -- PASS
>    tests.d/trivial-test.sh -- PASS
>    tests.d/my-test-using-libautotest -- SKIP
>    ...
>
>    $ yum install libautotest
>    # (or whatever the bootstrap procedure is)
>    $ make check-full
>    tests.d/my-test-using-qemu-tests.sh -- PASS
>    tests.d/trivial-test.sh -- PASS
>    tests.d/my-test-using-libautotest -- PASS
>    ...
>
> But for some reason that I still don't understand, the simple
> acceptance of the optional dependency of libautotest in qemu is
> seen as a bad thing.

I don't mind a libvirttest but I've expressed concerns that that if
qemu.git has it's own test library then the incentive to use or
contribute to an external library is not there.  That's why I
suggested a less ambitious approach.

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Fri, Mar 09, 2012 at 02:54:23PM +0000, Stefan Hajnoczi wrote:
> On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis <areis@redhat.com> wrote:
> > On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
> >> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <areis@redhat.com> wrote:
> >> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
> >> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
> >> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
> >> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
> >> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
> >> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
> >> >> >>>   - QE will be alienated from the qemu test effort. There will be
> >> >> >>>     no integration between the QE efforts and the maintenance of
> >> >> >>>     the qemu developer-level tests.
> >> >> >>
> >> >> >>I think we're a pretty friendly and open community :-)  There is no
> >> >> >>reason that QE should be "alienated" unless folks are choosing not
> >> >> >>to participate upstream.
> >> >> >
> >> >> >For the exact same reasons you as a developer don't want to
> >> >> >implement tests inside autotest, QE won't want to implement tests
> >> >> >for qemu.git. It's out of their comfort zone, just put yourself
> >> >> >on their shoes.
> >> >>
> >> >> This is a really, really poor argument and I hope I don't need to go
> >> >> into details of why.  If the primary reason for libautotest is so
> >> >> the people writing tests for QEMU can avoid actually working with
> >> >> the developers of QEMU...  we've got a problem.
> >> >
> >> > No, one of the benefits of having libautotest is to *collaborate*
> >> > with QE. I'll explain again:
> >> >
> >> > - As a qemu developer, I don't want to spend my time learning and
> >> >  getting involved in autotest, which is a complex QE project
> >> >  (I heard this numerous times).
> >> >
> >> > - As a Quality Engineer, I don't want to invest my time learning
> >> >  and getting involved into upstream qemu to test HEAD.
> >>
> >> I think this is the key point of the whole discussion - most of the
> >> other topics have been distractions.  Both communities do testing but
> >> we test different things and have different priorities.
> >>
> >> For me this has been the big realization from this discussion.  I felt
> >> kvm-autotest and qemu should share tests.  I was pushing for that but
> >> after following this thread I don't think it makes sense, here's why:
> >>
> >> The Quality Engineer you describe is not a QEMU upstream QE, instead
> >> the QE has a broader and more downstream focus.  (This is why
> >> comparisons with WebKit or other upstream projects doing testing are
> >> not valid comparisons.)
> >
> > Lucas, Cleber and the others red-hatters should remembers this
> > from my internal presentation, it was the first point I made:
> > QE and Developers have very different goals and interests.
> >
> > Which is why we're pushing all these changes in autotest. We see
> > opportunities for collaboration, but we do realize the difference.
> >
> > And look: Lucas and Cleber are not QE, they're developers working
> > on the autotest framework/library/whatever. We'll need similar
> > positions inside qemu as the test infra-structure grows.
> 
> I don't understand this last paragraph.  If qemu.git upstream was
> doing full-scale QE it would work fine because the differences that
> I've described and you also have pointed out would be absent.
> 

In order to have QEMU working in full "TDD Mode" (a current
goal), I predict developers assigned to the maintenance of the
in-house test infrastructure (qemu-test) will be needed, on
positions similar to what Lucas and Cleber currently do with
autotest. Only time will tell.

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Stefan Hajnoczi]

On Fri, Mar 9, 2012 at 3:01 PM, Ademar Reis <areis@redhat.com> wrote:
> On Fri, Mar 09, 2012 at 02:54:23PM +0000, Stefan Hajnoczi wrote:
>> On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis <areis@redhat.com> wrote:
>> > On Fri, Mar 09, 2012 at 09:41:05AM +0000, Stefan Hajnoczi wrote:
>> >> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis <areis@redhat.com> wrote:
>> >> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>> >> >> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> >> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>> >> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote:
>> >> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>> >> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote:
>> >> >> >>>   - QE will be alienated from the qemu test effort. There will be
>> >> >> >>>     no integration between the QE efforts and the maintenance of
>> >> >> >>>     the qemu developer-level tests.
>> >> >> >>
>> >> >> >>I think we're a pretty friendly and open community :-)  There is no
>> >> >> >>reason that QE should be "alienated" unless folks are choosing not
>> >> >> >>to participate upstream.
>> >> >> >
>> >> >> >For the exact same reasons you as a developer don't want to
>> >> >> >implement tests inside autotest, QE won't want to implement tests
>> >> >> >for qemu.git. It's out of their comfort zone, just put yourself
>> >> >> >on their shoes.
>> >> >>
>> >> >> This is a really, really poor argument and I hope I don't need to go
>> >> >> into details of why.  If the primary reason for libautotest is so
>> >> >> the people writing tests for QEMU can avoid actually working with
>> >> >> the developers of QEMU...  we've got a problem.
>> >> >
>> >> > No, one of the benefits of having libautotest is to *collaborate*
>> >> > with QE. I'll explain again:
>> >> >
>> >> > - As a qemu developer, I don't want to spend my time learning and
>> >> >  getting involved in autotest, which is a complex QE project
>> >> >  (I heard this numerous times).
>> >> >
>> >> > - As a Quality Engineer, I don't want to invest my time learning
>> >> >  and getting involved into upstream qemu to test HEAD.
>> >>
>> >> I think this is the key point of the whole discussion - most of the
>> >> other topics have been distractions.  Both communities do testing but
>> >> we test different things and have different priorities.
>> >>
>> >> For me this has been the big realization from this discussion.  I felt
>> >> kvm-autotest and qemu should share tests.  I was pushing for that but
>> >> after following this thread I don't think it makes sense, here's why:
>> >>
>> >> The Quality Engineer you describe is not a QEMU upstream QE, instead
>> >> the QE has a broader and more downstream focus.  (This is why
>> >> comparisons with WebKit or other upstream projects doing testing are
>> >> not valid comparisons.)
>> >
>> > Lucas, Cleber and the others red-hatters should remembers this
>> > from my internal presentation, it was the first point I made:
>> > QE and Developers have very different goals and interests.
>> >
>> > Which is why we're pushing all these changes in autotest. We see
>> > opportunities for collaboration, but we do realize the difference.
>> >
>> > And look: Lucas and Cleber are not QE, they're developers working
>> > on the autotest framework/library/whatever. We'll need similar
>> > positions inside qemu as the test infra-structure grows.
>>
>> I don't understand this last paragraph.  If qemu.git upstream was
>> doing full-scale QE it would work fine because the differences that
>> I've described and you also have pointed out would be absent.
>>
>
> In order to have QEMU working in full "TDD Mode" (a current
> goal), I predict developers assigned to the maintenance of the
> in-house test infrastructure (qemu-test) will be needed, on
> positions similar to what Lucas and Cleber currently do with
> autotest. Only time will tell.

I agree that engineers are needed to work on testing as testing
increases upstream.

Stefan

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Kevin Wolf]

Am 09.03.2012 00:51, schrieb Ademar Reis:
> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>>> Plus it's not unconditional: the test runner will report tests
>>> SKIPPED if a dependency is not present.
>>
>> But then the tests aren't run so if most developers didn't have it
>> installed, and most tests were written with it, most developers
>> wouldn't be running most tests which defeats the purpose, no?
> 
> Part of a TDD approach is to have build and test bots frequently
> running tests on multiple platforms with different
> configurations.
> 
> You can't expect developers to run all tests all the time.

I think this is one of the most important points: Not all developers
must run all the tests all the time.

Actually, Anthony agreed with me when I said that developers should run
some sanity tests for all of qemu and maybe a few more tests for the
subsystems they're touching. I agree that it would be bad to have a
autotest dependency for those basic tests that everyone should run and
that should take a few minutes at most.

For the rest of test cases, however, not everyone needs to run them and
I think an external dependency (that is reasonably easy to satisfy) is
not a problem there. A test bot would be great, but even if people just
run them occasionally by hand, that would already detect bugs that are
currently left in the code for months. If maintainers do it before
pushing code into master, you'll even catch everything before it goes
into master. This is as good as it gets.

The important thing is that tests exist in the first place, not who runs
them.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 05:14 AM, Kevin Wolf wrote:
> Am 09.03.2012 00:51, schrieb Ademar Reis:
>> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>>>> Plus it's not unconditional: the test runner will report tests
>>>> SKIPPED if a dependency is not present.
>>>
>>> But then the tests aren't run so if most developers didn't have it
>>> installed, and most tests were written with it, most developers
>>> wouldn't be running most tests which defeats the purpose, no?
>>
>> Part of a TDD approach is to have build and test bots frequently
>> running tests on multiple platforms with different
>> configurations.
>>
>> You can't expect developers to run all tests all the time.
>
> I think this is one of the most important points: Not all developers
> must run all the tests all the time.
>
> Actually, Anthony agreed with me when I said that developers should run
> some sanity tests for all of qemu and maybe a few more tests for the
> subsystems they're touching.

And a small number of randomly chosen test cases.

We don't want to have test cases that never run under normal circumstances or 
else they're prone to break.  That's why I've talked a lot about keeping 'make 
check' time bound.

> I agree that it would be bad to have a
> autotest dependency for those basic tests that everyone should run and
> that should take a few minutes at most.
>
> For the rest of test cases, however, not everyone needs to run them and
> I think an external dependency (that is reasonably easy to satisfy) is
> not a problem there.

I'd prefer to avoid external dependencies as it just encourages people not to 
test.  There may be exceptions for certain types of tests, but it should be an 
exception, not the rule.

> A test bot would be great, but even if people just
> run them occasionally by hand, that would already detect bugs that are
> currently left in the code for months. If maintainers do it before
> pushing code into master, you'll even catch everything before it goes
> into master. This is as good as it gets.

We'll integrate make check into buildbot which currently does look at 
submaintainer trees.

Regards,

Anthony Liguori

>
> The important thing is that tests exist in the first place, not who runs
> them.
>
> Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Kevin Wolf]

Am 09.03.2012 12:59, schrieb Anthony Liguori:
> On 03/09/2012 05:14 AM, Kevin Wolf wrote:
>> Am 09.03.2012 00:51, schrieb Ademar Reis:
>>> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>>>>> Plus it's not unconditional: the test runner will report tests
>>>>> SKIPPED if a dependency is not present.
>>>>
>>>> But then the tests aren't run so if most developers didn't have it
>>>> installed, and most tests were written with it, most developers
>>>> wouldn't be running most tests which defeats the purpose, no?
>>>
>>> Part of a TDD approach is to have build and test bots frequently
>>> running tests on multiple platforms with different
>>> configurations.
>>>
>>> You can't expect developers to run all tests all the time.
>>
>> I think this is one of the most important points: Not all developers
>> must run all the tests all the time.
>>
>> Actually, Anthony agreed with me when I said that developers should run
>> some sanity tests for all of qemu and maybe a few more tests for the
>> subsystems they're touching.
> 
> And a small number of randomly chosen test cases.
> 
> We don't want to have test cases that never run under normal circumstances or 
> else they're prone to break.  That's why I've talked a lot about keeping 'make 
> check' time bound.

This sounds horrible. make check results must be reproducible, not
depend on a randomly chosen set. If you do it like this, a passed make
check means exactly _nothing_.

>> I agree that it would be bad to have a
>> autotest dependency for those basic tests that everyone should run and
>> that should take a few minutes at most.
>>
>> For the rest of test cases, however, not everyone needs to run them and
>> I think an external dependency (that is reasonably easy to satisfy) is
>> not a problem there.
> 
> I'd prefer to avoid external dependencies as it just encourages people not to 
> test.  There may be exceptions for certain types of tests, but it should be an 
> exception, not the rule.
> 
>> A test bot would be great, but even if people just
>> run them occasionally by hand, that would already detect bugs that are
>> currently left in the code for months. If maintainers do it before
>> pushing code into master, you'll even catch everything before it goes
>> into master. This is as good as it gets.
> 
> We'll integrate make check into buildbot which currently does look at 
> submaintainer trees.

But make check will never be the full thing. And if you want to
integrate make check into automated testing, then choosing a random
subset of tests for each is an even worse idea than it is anyway.

I believe the only sane thing to do is to distinguish between quick
sanity tests that are run by make check, and a full test suite that is
not run by every developer, but ideally by some test bots.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 06:13 AM, Kevin Wolf wrote:
> Am 09.03.2012 12:59, schrieb Anthony Liguori:
>> On 03/09/2012 05:14 AM, Kevin Wolf wrote:
>>> Am 09.03.2012 00:51, schrieb Ademar Reis:
>>>> On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote:
>>>>>> Plus it's not unconditional: the test runner will report tests
>>>>>> SKIPPED if a dependency is not present.
>>>>>
>>>>> But then the tests aren't run so if most developers didn't have it
>>>>> installed, and most tests were written with it, most developers
>>>>> wouldn't be running most tests which defeats the purpose, no?
>>>>
>>>> Part of a TDD approach is to have build and test bots frequently
>>>> running tests on multiple platforms with different
>>>> configurations.
>>>>
>>>> You can't expect developers to run all tests all the time.
>>>
>>> I think this is one of the most important points: Not all developers
>>> must run all the tests all the time.
>>>
>>> Actually, Anthony agreed with me when I said that developers should run
>>> some sanity tests for all of qemu and maybe a few more tests for the
>>> subsystems they're touching.
>>
>> And a small number of randomly chosen test cases.
>>
>> We don't want to have test cases that never run under normal circumstances or
>> else they're prone to break.  That's why I've talked a lot about keeping 'make
>> check' time bound.
>
> This sounds horrible. make check results must be reproducible,

It is reproducible by fixing the random seed.  When you run qemu-test, you get 
output like:

$ ./qemu-test qemu-system-x86_64 tests/device-add.sh
Using RANDOM seed 56782
Formatting '.tmp-2398/disk.img', fmt=qcow2 size=10737418240 encryption=off 
cluster_size=65536
ls: cannot access .tmp-2398/logfile-2398.log: No such file or directory
/home/anthony/build/qemu/x86_64-softmmu/qemu-system-x86_64 -kernel 
bin/vmlinuz-3.0 -initrd .tmp-2398/initramfs-2398.img.gz -append console=ttyS0 
seed=56782 -nographic -enable-kvm -device virtio-balloon-pci,id=balloon0 
-pidfile .tmp-2398/pidfile-2398.pid -qmp 
unix:.tmp-2398/qmpsock-2398.sock,server,nowait
test: 70: =: unexpected operator
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.0.0 (anthony@titi) (gcc version 4.5.2 
(Ubuntu/Linaro 4.5.2-8ubuntu4) ) #7 SMP Mon Dec 19 15:54:15 CST 2011

To run the exact same test again, you would do:

$ QEMU_TEST_SEED=56782 ./qemu-test qemu-system-x86_64 tests/devices-add.sh

This lets you design high coverage tests without having to mess around with 
changing configuration files.  In the latest qemu-test, you can even fix your 
random choices if you want to configure a very specific test.

> not
> depend on a randomly chosen set. If you do it like this, a passed make
> check means exactly _nothing_.

You can have it both ways.  You can fix choices in order to have a deterministic 
test of a specific thing or you can allow choices to be made randomly (with 
weighting).

So you can make make check validate very precise things but chances are, there's 
a class of things that you would like people to test but maybe not every time 
make check runs.  Having a contingent of occasionally executed tests means 
you'll get better coverage for less commonly used features.

>>> A test bot would be great, but even if people just
>>> run them occasionally by hand, that would already detect bugs that are
>>> currently left in the code for months. If maintainers do it before
>>> pushing code into master, you'll even catch everything before it goes
>>> into master. This is as good as it gets.
>>
>> We'll integrate make check into buildbot which currently does look at
>> submaintainer trees.
>
> But make check will never be the full thing. And if you want to
> integrate make check into automated testing, then choosing a random
> subset of tests for each is an even worse idea than it is anyway.

make check-full.

> I believe the only sane thing to do is to distinguish between quick
> sanity tests that are run by make check, and a full test suite that is
> not run by every developer, but ideally by some test bots.

Yeah, I don't think we're disagreeing at all.

Regards,

Anthony Liguori

>
> Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa]

On 03/08/2012 08:21 PM, Anthony Liguori wrote:
> On 03/08/2012 04:24 PM, Ademar Reis wrote:
>> On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote:
>>> On 03/08/2012 03:02 PM, Ademar Reis wrote:
>>>> On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote:
>>>>> On 03/08/2012 11:59 AM, Ademar Reis wrote:
>>>>
>>>> <snip>
>>>>
>>>>>
>>>>>>> I expect QEMU to grow tests for anything that involves launching
>>>>>>> QEMU directly.  Where I would not see QEMU growing tests for is
>>>>>>> things like launching QEMU through libvirt.
>>>>>>
>>>>>> Agree. For QEMU developers, libvirt should not be on the way, the
>>>>>> interaction should be minimal or non-existent.
>>>>>>
>>>>>> That's an area which will require some work in libautotest,
>>>>>> because due to previous QE requirements, it now invokes libvirt
>>>>>> methods instead of QEMU directly for a lot of stuff. But it can
>>>>>> be done and is part of the plan.
>>>>>
>>>>> If you're talking about libautotest as an abstraction layer for
>>>>> launching QEMU, I don't think that's something we should use in
>>>>> upstream QEMU.  Abstraction layers are okay when the things you are
>>>>> abstracting are well defined and relatively static.  We're talking
>>>>> about testing the tip of a development tree though and coordinating
>>>>> changes with an abstraction layer would be counter productive.
>>>>>
>>>>
>>>> Valid point. If I'm a qemu developer and I want to make huge
>>>> changes, ideally I shouldn't have to submit patches to
>>>> libautotest because the tests in qemu.org will break.
>>>>
>>>> I don't know how likely such scenario is (Lucas is maitaining
>>>> kvm-autotest for years now and the integration hardly breaks)
>>>> But we do have a conflict of interests, so lets evaluate a
>>>> few options:
>>>
>>> Because kvm-autotest doesn't have thousands of tests and isn't part
>>> of 200+ developers fast paths :-)
>>>
>>>>
>>>> My main interests are:
>>>>      - remove barriers for developers to write tests
>>>
>>> This is my number one priority.
>>>
>>>>      - share the testing effort between multiple projects (qemu,
>>>>        libvirt, ovirt, spice, QE, etc...)
>>>
>>> This is my number 99 priority.
>>
>> You have the right to consider the interests of qemu only, but
>> qemu is used by several other teams who depend on it to implement
>> their own tests. Some collaboration would be extremely beneficial
>> for everybody.
>>
>> Collaboration goes both ways: you get benefits from the
>> interaction and code reuse with different projects and teams, but
>> it may require a few tradeoffs.
>
> I think the problem here effectively boils down to "code sharing at 
> any cost".
>
> Let's look at it in purely practical terms.  How many lines of code 
> would be saved here and what does that code do?
>
> If the only sharing is some infrastructure bits in libautotest, then I 
> suspect we're talking in the low 10ks of lines at the absolute maximum 
> (and I don't think it's even close to that in reality).  Launching 
> QEMU and connecting to QMP isn't that hard.  It's in the hundreds of 
> lines of code.  What else is there that's need to write a test?
>
> I don't think this is sufficiently compelling to add an unconditional 
> autotest dependency to QEMU.
>
>>> I think you could achieve practical code sharing by decomposing
>>> kvm-autotest into a set of useful, independent tools.  For
>>> instance....
>>>
>>>> Now the options:
>>>>
>>>> 1. We keep the invocation stuff in libautotest and when writting
>>>> a complex test in qemu.git, the test writter can choose to use it
>>>> because of the goodies there.
>>>>
>>>> Pros:
>>>>    - Lots of codepaths implemented both in python and as cmd-line
>>>>      utilities: less lines of code to write a test, smaller
>>>>      barrier for the developer.
>>>
>>> I've got an existence proof that this is not true.  The qemu-test
>>> tests are smaller than the corresponding autotest tests.
>>
>> You're comparing developer-level tests with the existent QA-level
>> tests (much more complex).
>
> Let's be specific then.  Look at device-add.sh in qemu-test.  It's 
> 71LOC. pci_hotplug.py in autotest is 204LOC.

pci_hotplug.py does much more than device-add.sh:

    * tests both pci_add and device_add commands
    * checks the monitor syntax for adding a new drive, that is, it 
works on HEAD and on other versions (such as the ones in some RHEL 
releases);
    * tests both nic and block hotplug
    * for block, tests with both virtio and scsi
    * also does device removal, both for pci_add and device_add syntaxes

>
> What's the "much more complex" part of pci_hotplug.py that is enabled 
> by having the autotest framework?
>
>> We're talking about an approach where you'll need just a few
>> lines of code to instantiate a VM and interact with it.
>
> This isn't that hard of a problem.  It just takes a few lines to 
> launch QEMU in qemu-test too.
>
>>>>    - Mature code in the test library.
>>>>    - Goodies such as video-recording and system info collection
>>>>      in a standard way for different projects.
>>>
>>> Why does video-recording have to be part of libautotest?  Why can't
>>> you just have a simply utility that connects to a VNC session and
>>> records it?  Presumably that's all it's doing here.
>>>
>>> Likewise, there are dozens of "system info collection" such as
>>> sosreport..  why fold this all into libautotest?
>>
>> For standardization: think outside of the qemu community. A lot
>> of our developers contribute code to multiple projects. It gets
>> easier if there's a standard way to achieve the same results when
>> writting tests for different projects.
>>
>> Anyway, it's a minor issue.
>
> Yeah, I think this is outside of the scope of the current discussion 
> as it's sort of like trying to boil the ocean :-)
>
>>>>    - Code shared with other teams.
>>>>    - The same test code can run on old platforms because
>>>>      libautotest has a compatibility layer, thus enabling a
>>>>      larger grid of tests.
>>>
>>> This is not a requirement from a QEMU perspective nor do I think 
>>> it's useful.
>>
>> Well, it's not a requirement, but you can't expect QE teams to
>> spend their time on tests that work only on HEAD. It makes no
>> sense from their point of view.
>
> They're certainly testing some sort of HEAD, even if it's not 
> qemu.git.  I don't see any conflicting goals here.  Write a test 
> against upstream, backport to a downstream if necessarily.
>
>>
>>>
>>>>    - QE teams will be closer to the development, sharing
>>>>      some of their testing code and maintenance burden.
>>>
>>> The QE team could simply contribute tests to qemu.git to achieve 
>>> this same goal.
>>
>> They won't help you if you don't help them. Again, it doesn't
>> make sense for QE to work on HEAD. At least not the QE teams I'm
>> talking about.
>>
>> And I say from my own experience: maintaining a set of thousands
>> of tests is far from trivial. You'll waste a lot of potential
>> contributions by alienating the fully-dedicated teams currently
>> working on QE.
>
> And trying to maintain a thousand tests against any possible version 
> is probably also unrealistic.
>
> Is the expectation that all of the tests in WebKit work against any 
> possible version of WebKit?  How do they enforce this since it's all 
> in trunk?
>
>>>> 2. We "move" (or implement) all the libautotest stuff related to
>>>> qemu invocation and interaction into qemu.git, forbidding the
>>>> dependency of autotest in qemu.git.
>>>
>>> Let me just be really clear here.  In order to do TDD, tests have to
>>> be a mandatory part of the QEMU build process.  We cannot introduce
>>> new dependencies in order to build tests.
>>>
>>> Making autotest an unconditional dependency of qemu.git is a 
>>> non-starter for me.
>>
>> You have dependencies on gtest and other libraries. it's not that
>> different.
>
> We have a dependency on glib, which gtest happens to be a part of.  It 
> was a big deal introducing that dependency and gtest was not the only 
> (or primary) reason to introduce that dependency.
>
>> Plus it's not unconditional: the test runner will report tests
>> SKIPPED if a dependency is not present.
>
> But then the tests aren't run so if most developers didn't have it 
> installed, and most tests were written with it, most developers 
> wouldn't be running most tests which defeats the purpose, no?
>
>> It's actually the oposite: you'll have to forbid developers from
>> submiting tests that use libautotest if you don't want the
>> dependency.
>
> I think part of the problem here is the abstract nature of this 
> discussion. What is libautotest actually supposed to do?
>
> This would be an easier discussion if we were talking about patches 
> instead of proposals.
>
>>>>    - QE will be alienated from the qemu test effort. There will be
>>>>      no integration between the QE efforts and the maintenance of
>>>>      the qemu developer-level tests.
>>>
>>> I think we're a pretty friendly and open community :-)  There is no
>>> reason that QE should be "alienated" unless folks are choosing not
>>> to participate upstream.
>>
>> For the exact same reasons you as a developer don't want to
>> implement tests inside autotest, QE won't want to implement tests
>> for qemu.git. It's out of their comfort zone, just put yourself
>> on their shoes.
>
> This is a really, really poor argument and I hope I don't need to go 
> into details of why.  If the primary reason for libautotest is so the 
> people writing tests for QEMU can avoid actually working with the 
> developers of QEMU...  we've got a problem.
>
>>>
>>>>    - You don't get the goodies from autotest (standardized system
>>>>      info collection, video recording, grid support, etc).
>>>>    - The new tests will work only on the master branch.
>>>
>>> This last one is a legitimate point that I have considered myself.
>>> But I think the value of having tests in HEAD outweigh the cost
>>> here.
>>
>> Thinking about qemu only, agree.
>>
>>>
>>>> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
>>>> keep the compatibility layer and provide a "libqemutest" for
>>>> third-parties.
>>>>
>>>> Anybody writting a test that interacts with qemu will use this
>>>> library: qemu, libvirt, spice, ovirt, QE tests, etc.
>>>
>>> I really see this all as over architecting to be honest.
>>>
>>> Can someone explain in clear terms (without appealing to maturity,
>>> flexibility, or any other qualitative aspect) why it takes anything
>>> more than a few dozen shell functions to write all of the tests that
>>> are in kvm-autotest test today?
>>
>> This is close to being offensive IMO, so I won't answer. :-)
>
> I honestly don't mean it to be so I apologize if it comes off that 
> way.  But I dislike arguing in such abstract terms.  What are the 
> features that you think libautotest is needed for?
>
>>> You launch qemu (by just using a command), and you interact with the
>>> monitor via QMP and HMP.  Why do you ever need anything more than
>>> that?
>>
>> Well, again, I'll agree to disagree.
>>
>>>
>>> Why does libautotest even need to exist?
>>
>> You're looking at things from the sole point of view of a qemu
>> developer. If you don't want to see things from a different
>> perspective and accept any kind of tradeoffs for better
>> collaboration with other teams, then unfortunately there's not
>> much we can do. It's a pity, because collaboration goes both
>> ways.
>
> I need something more concrete then this.  What is it about 
> libautotest that would allow collaboration?
>
>> We're trying very hard to satisfy your requirements and
>> interests. Don't you see any way you can help us satisfy
>> ours?
>
> I'm having a hard time understanding what your requirements are other 
> than that we should write our tests using libautotest (which has not 
> really been explained other than being levied as a requirement).
>
> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 05:20 AM, Cleber Rosa wrote:
>>> You're comparing developer-level tests with the existent QA-level
>>> tests (much more complex).
>>
>> Let's be specific then. Look at device-add.sh in qemu-test. It's 71LOC.
>> pci_hotplug.py in autotest is 204LOC.
>
> pci_hotplug.py does much more than device-add.sh:
>
> * tests both pci_add and device_add commands
> * checks the monitor syntax for adding a new drive, that is, it works on HEAD
> and on other versions (such as the ones in some RHEL releases);
> * tests both nic and block hotplug
> * for block, tests with both virtio and scsi
> * also does device removal, both for pci_add and device_add syntaxes

Ok, but clearly, there's no magic in autotest that makes this sufficiently 
easier.  It's just a matter of:

cmd=`named_choose command device_add pci_add`

if test $cmd = "device_add"; then
   qmp device_add --driver=virtio-blk-pci --drive=hd0
else
   hmp pci_add auto virtio-blk-pci,drive=hd0
fi

It's not there today because pci_add is deprecated.  There assertion was that 
autotest makes it easier to write tests.  How does it make it easier to write 
pci_hotplug?

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa]

On 03/09/2012 09:04 AM, Anthony Liguori wrote:
> On 03/09/2012 05:20 AM, Cleber Rosa wrote:
>>>> You're comparing developer-level tests with the existent QA-level
>>>> tests (much more complex).
>>>
>>> Let's be specific then. Look at device-add.sh in qemu-test. It's 71LOC.
>>> pci_hotplug.py in autotest is 204LOC.
>>
>> pci_hotplug.py does much more than device-add.sh:
>>
>> * tests both pci_add and device_add commands
>> * checks the monitor syntax for adding a new drive, that is, it works 
>> on HEAD
>> and on other versions (such as the ones in some RHEL releases);
>> * tests both nic and block hotplug
>> * for block, tests with both virtio and scsi
>> * also does device removal, both for pci_add and device_add syntaxes
>
> Ok, but clearly, there's no magic in autotest that makes this 
> sufficiently easier.  It's just a matter of:
>
> cmd=`named_choose command device_add pci_add`
>
> if test $cmd = "device_add"; then
>   qmp device_add --driver=virtio-blk-pci --drive=hd0
> else
>   hmp pci_add auto virtio-blk-pci,drive=hd0
> fi
>
> It's not there today because pci_add is deprecated.  There assertion 
> was that autotest makes it easier to write tests.  How does it make it 
> easier to write pci_hotplug?
>

Sure. I agree that it's fair from QEMU's PoV alone to forget about 
legacy things such as pci_add.

On this particular example, the one thing that strikes me the most is 
that (kvm-)autotest allows either a very static test run (as 
device_add.sh does), or a very configurable test run (as pci_hotplug.py 
does and QE needs).

I believe that with little compromises (required by any kind of 
collaboration) we can have the best 
tests/tools/libraries/framework/whatever by combining these similar code 
bases.

Again, we're willing to work and try to make that happen, but IMHO there 
must be a more open attitude here.

Cheers,
CR.

> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 06:40 AM, Cleber Rosa wrote:
> On 03/09/2012 09:04 AM, Anthony Liguori wrote:
>> On 03/09/2012 05:20 AM, Cleber Rosa wrote:
>>>>> You're comparing developer-level tests with the existent QA-level
>>>>> tests (much more complex).
>>>>
>>>> Let's be specific then. Look at device-add.sh in qemu-test. It's 71LOC.
>>>> pci_hotplug.py in autotest is 204LOC.
>>>
>>> pci_hotplug.py does much more than device-add.sh:
>>>
>>> * tests both pci_add and device_add commands
>>> * checks the monitor syntax for adding a new drive, that is, it works on HEAD
>>> and on other versions (such as the ones in some RHEL releases);
>>> * tests both nic and block hotplug
>>> * for block, tests with both virtio and scsi
>>> * also does device removal, both for pci_add and device_add syntaxes
>>
>> Ok, but clearly, there's no magic in autotest that makes this sufficiently
>> easier. It's just a matter of:
>>
>> cmd=`named_choose command device_add pci_add`
>>
>> if test $cmd = "device_add"; then
>> qmp device_add --driver=virtio-blk-pci --drive=hd0
>> else
>> hmp pci_add auto virtio-blk-pci,drive=hd0
>> fi
>>
>> It's not there today because pci_add is deprecated. There assertion was that
>> autotest makes it easier to write tests. How does it make it easier to write
>> pci_hotplug?
>>
>
> Sure. I agree that it's fair from QEMU's PoV alone to forget about legacy things
> such as pci_add.
>
> On this particular example, the one thing that strikes me the most is that
> (kvm-)autotest allows either a very static test run (as device_add.sh does), or
> a very configurable test run (as pci_hotplug.py does and QE needs).

With named_choose and profiles, you can actually configure it very specifically. 
  Most of the tests are not using named_choose right now, but it's easy enough 
to change that.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa]

On 03/09/2012 09:42 AM, Anthony Liguori wrote:
> On 03/09/2012 06:40 AM, Cleber Rosa wrote:
>> On 03/09/2012 09:04 AM, Anthony Liguori wrote:
>>> On 03/09/2012 05:20 AM, Cleber Rosa wrote:
>>>>>> You're comparing developer-level tests with the existent QA-level
>>>>>> tests (much more complex).
>>>>>
>>>>> Let's be specific then. Look at device-add.sh in qemu-test. It's 
>>>>> 71LOC.
>>>>> pci_hotplug.py in autotest is 204LOC.
>>>>
>>>> pci_hotplug.py does much more than device-add.sh:
>>>>
>>>> * tests both pci_add and device_add commands
>>>> * checks the monitor syntax for adding a new drive, that is, it 
>>>> works on HEAD
>>>> and on other versions (such as the ones in some RHEL releases);
>>>> * tests both nic and block hotplug
>>>> * for block, tests with both virtio and scsi
>>>> * also does device removal, both for pci_add and device_add syntaxes
>>>
>>> Ok, but clearly, there's no magic in autotest that makes this 
>>> sufficiently
>>> easier. It's just a matter of:
>>>
>>> cmd=`named_choose command device_add pci_add`
>>>
>>> if test $cmd = "device_add"; then
>>> qmp device_add --driver=virtio-blk-pci --drive=hd0
>>> else
>>> hmp pci_add auto virtio-blk-pci,drive=hd0
>>> fi
>>>
>>> It's not there today because pci_add is deprecated. There assertion 
>>> was that
>>> autotest makes it easier to write tests. How does it make it easier 
>>> to write
>>> pci_hotplug?
>>>
>>
>> Sure. I agree that it's fair from QEMU's PoV alone to forget about 
>> legacy things
>> such as pci_add.
>>
>> On this particular example, the one thing that strikes me the most is 
>> that
>> (kvm-)autotest allows either a very static test run (as device_add.sh 
>> does), or
>> a very configurable test run (as pci_hotplug.py does and QE needs).
>
> With named_choose and profiles, you can actually configure it very 
> specifically.  Most of the tests are not using named_choose right now, 
> but it's easy enough to change that.

And then the two code bases will end up having even more similar 
features, similar complexity, etc.

>
> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/08/2012 06:24 PM, Anthony Liguori wrote:
>>
>> Cons:
>> - Lot of code will be duplicated to cover the main code paths:
>> writting tests will require writting/supporting considerable
>> ammount of code (that already exists in autotest).
>
> Again, existence proof that this isn't true.

Case in point, the virtio test (that uses an auxiliary script to send 
data to the host). Can you tell me if both tests cover even remotely the 
same amount of functionality?

https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py

https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py

Here is the qemu-test version

http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

What the qemu-test version covers:
  * host starts qemu with one virtio console device backed by a file
  * guest verifies if the name of the device is correct
  * guest writes to the console device
  * host verifies if guest wrote to the virtio console

What the virtio-console covers:

  * Sends data between host and guest back and forth, validates the data 
being sent, for both small and large amounts of data, both random or 
sequential.
  * Tests write/send in blocking, polling, selecting mode, with port 
mode sync/async
  * Verifies if the maximum amount of ports was created and it's 
available in the guest
  * Tries lseek on the device
  * Verifies if concomitant access to a single port passes or fails
  * Verifies data throughput and resource utilization
  * Repeats the data transfer with live migration to see if the port 
connections survive
  * Keeps an eye on the guest OS to see if we have kernel panics, and if 
it does, it'll clean up and put the guest in a working state so other 
functionality can be checked
  * Probably something else I'm forgetting right now.

Good luck implementing that in a shell script. I'd love to see how you 
implement that amount of coverage in less lines in shell.

So, more coverage, more code. It's as simple as that. We don't write 
code just for the sake of it.

>> - QE will be alienated from the qemu test effort. There will be
>> no integration between the QE efforts and the maintenance of
>> the qemu developer-level tests.
>
> I think we're a pretty friendly and open community :-) There is no
> reason that QE should be "alienated" unless folks are choosing not to
> participate upstream.
>
>> - You don't get the goodies from autotest (standardized system
>> info collection, video recording, grid support, etc).
>> - The new tests will work only on the master branch.
>
> This last one is a legitimate point that I have considered myself. But I
> think the value of having tests in HEAD outweigh the cost here.
>
>> 3. A mix of (1) and (2): we "move" everything under qemu.git, but
>> keep the compatibility layer and provide a "libqemutest" for
>> third-parties.
>>
>> Anybody writting a test that interacts with qemu will use this
>> library: qemu, libvirt, spice, ovirt, QE tests, etc.
>
> I really see this all as over architecting to be honest.
>
> Can someone explain in clear terms (without appealing to maturity,
> flexibility, or any other qualitative aspect) why it takes anything more
> than a few dozen shell functions to write all of the tests that are in
> kvm-autotest test today?

Clearly as your requirements are different than the ones we had when the 
project was written, qemu-test doesn't need any of the stuff present 
there and you can do it all with a handful of shell script functions.

But to do cover the same things covered in autotest today with the same 
requirements, I really doubt you can do the same with the same handful 
of shell script functions. See the example about the virtio test above, 
not to mention other functionality we have with the tests, such as make 
possible to do tests run with a migration thread going on the 
background, while it's plugging/unplugging devices, running a stress 
test or ltp, or a benchmark, or measuring the time drift experienced by 
the guest.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 08:07:27PM -0300, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 06:24 PM, Anthony Liguori wrote:
> >>
> >>Cons:
> >>- Lot of code will be duplicated to cover the main code paths:
> >>writting tests will require writting/supporting considerable
> >>ammount of code (that already exists in autotest).
> >
> >Again, existence proof that this isn't true.
> 
> Case in point, the virtio test (that uses an auxiliary script to
> send data to the host). Can you tell me if both tests cover even
> remotely the same amount of functionality?
> 
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
> 
> https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py
> 
> Here is the qemu-test version
> 
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master
> 
> What the qemu-test version covers:
>  * host starts qemu with one virtio console device backed by a file
>  * guest verifies if the name of the device is correct
>  * guest writes to the console device
>  * host verifies if guest wrote to the virtio console
> 
> What the virtio-console covers:
> 
>  * Sends data between host and guest back and forth, validates the
> data being sent, for both small and large amounts of data, both
> random or sequential.
>  * Tests write/send in blocking, polling, selecting mode, with port
> mode sync/async
>  * Verifies if the maximum amount of ports was created and it's
> available in the guest
>  * Tries lseek on the device
>  * Verifies if concomitant access to a single port passes or fails
>  * Verifies data throughput and resource utilization
>  * Repeats the data transfer with live migration to see if the port
> connections survive
>  * Keeps an eye on the guest OS to see if we have kernel panics, and
> if it does, it'll clean up and put the guest in a working state so
> other functionality can be checked
>  * Probably something else I'm forgetting right now.
> 

Thanks for the example Lucas. :)

BTW, this is a QE-level test. Simpler tests may be sufficient for
developers and for TDD, but, if one implements even a much
simpler test using libautotest, he gets these benefits for free:

1. QE or somebody else may extend the test using the autotest
goodies the original developer was not aware of. Patches from QE
will flow to qemu.git. That's the integration with QE I was
talking about.

2. The test can run in the autotest grid, where a large matrix of
configurations and usage scenarios are used. That's possible
because we don't have much stuff hardcoded.

3. Autotest has a lot of checks for robustness. One of the main
problems of keeping thousands of tests running is to keep them
stable, without false positives.

Lucas: could you ellaborate more on what we get "for free" when
we use libautotest, even when writting a very simple
developer-level test?

> Good luck implementing that in a shell script. I'd love to see how
> you implement that amount of coverage in less lines in shell.
> 
> So, more coverage, more code. It's as simple as that. We don't write
> code just for the sake of it.
> 
> >>- QE will be alienated from the qemu test effort. There will be
> >>no integration between the QE efforts and the maintenance of
> >>the qemu developer-level tests.
> >
> >I think we're a pretty friendly and open community :-) There is no
> >reason that QE should be "alienated" unless folks are choosing not to
> >participate upstream.
> >
> >>- You don't get the goodies from autotest (standardized system
> >>info collection, video recording, grid support, etc).
> >>- The new tests will work only on the master branch.
> >
> >This last one is a legitimate point that I have considered myself. But I
> >think the value of having tests in HEAD outweigh the cost here.
> >
> >>3. A mix of (1) and (2): we "move" everything under qemu.git, but
> >>keep the compatibility layer and provide a "libqemutest" for
> >>third-parties.
> >>
> >>Anybody writting a test that interacts with qemu will use this
> >>library: qemu, libvirt, spice, ovirt, QE tests, etc.
> >
> >I really see this all as over architecting to be honest.
> >
> >Can someone explain in clear terms (without appealing to maturity,
> >flexibility, or any other qualitative aspect) why it takes anything more
> >than a few dozen shell functions to write all of the tests that are in
> >kvm-autotest test today?
> 
> Clearly as your requirements are different than the ones we had when
> the project was written, qemu-test doesn't need any of the stuff
> present there and you can do it all with a handful of shell script
> functions.
> 
> But to do cover the same things covered in autotest today with the
> same requirements, I really doubt you can do the same with the same
> handful of shell script functions. See the example about the virtio
> test above, not to mention other functionality we have with the
> tests, such as make possible to do tests run with a migration thread
> going on the background, while it's plugging/unplugging devices,
> running a stress test or ltp, or a benchmark, or measuring the time
> drift experienced by the guest.

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 06:24 PM, Anthony Liguori wrote:
>>>
>>> Cons:
>>> - Lot of code will be duplicated to cover the main code paths:
>>> writting tests will require writting/supporting considerable
>>> ammount of code (that already exists in autotest).
>>
>> Again, existence proof that this isn't true.
>
> Case in point, the virtio test (that uses an auxiliary script to send data to
> the host). Can you tell me if both tests cover even remotely the same amount of
> functionality?
>
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>
>
> https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py
>
>
> Here is the qemu-test version
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

I seem to recall us having this same discussion in the past....  But 
nevertheless, I started to respond in detail but decided to wait until tomorrow. 
  It's been a long day and I think this thread is a bit too heated already.  I 
think I'll be a bit less grumpy in the morning :-)

But there's one thing I want to point out...

> What the qemu-test version covers:
> * host starts qemu with one virtio console device backed by a file
> * guest verifies if the name of the device is correct
> * guest writes to the console device
> * host verifies if guest wrote to the virtio console
>
> What the virtio-console covers:
>
> * Sends data between host and guest back and forth, validates the data being
> sent, for both small and large amounts of data, both random or sequential.
> * Tests write/send in blocking, polling, selecting mode, with port mode sync/async

This bullet (and many of the bullets) below are not tests of QEMU.  They are 
tests of the Linux kernel.  This is an integration test, not a functional test 
of QEMU.

We absolutely need integration tests and hands down, that's where autotest is 
going to shine.  This is clearly not in the scope of qemu-test and there's been 
a few discussions on the ML on this including me rejecting a patch that crossed 
this boundary.

Right now autotest effectively does everything because we (QEMU) have not 
fulfilled our responsibility here.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 01:04, Anthony Liguori ha scritto:
>>
>> * Sends data between host and guest back and forth, validates the data
>> being
>> sent, for both small and large amounts of data, both random or
>> sequential.
>> * Tests write/send in blocking, polling, selecting mode, with port
>> mode sync/async
> 
> This bullet (and many of the bullets) below are not tests of QEMU.  They
> are tests of the Linux kernel.  This is an integration test, not a
> functional test of QEMU.

I think only these two other bullets are pure kernel tests:

>  * Tries lseek on the device
>  * Verifies if concomitant access to a single port passes or fails

In particular, migration and monitoring for panics are very useful for
QEMU, since QEMU bugs can crash the kernel.  Of course this is moot if
you have proper unit tests at the qtest level, but there's quite some
work to do before you can dismiss all the work in autotest.  And when
you have those, you can also dismiss some of the work in qemu-test.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 06:24 PM, Anthony Liguori wrote:
>>>
>>> Cons:
>>> - Lot of code will be duplicated to cover the main code paths:
>>> writting tests will require writting/supporting considerable
>>> ammount of code (that already exists in autotest).
>>
>> Again, existence proof that this isn't true.
>
> Case in point, the virtio test (that uses an auxiliary script to send data to
> the host). Can you tell me if both tests cover even remotely the same amount of
> functionality?
>
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py
>
>
> https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py
>
>
> Here is the qemu-test version
>
> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master

So virtio-serial is an exception in autotest:

2174 virtio_console.py
1875 cgroup.py
615 ksm_overcommit.py
439 qemu_img.py
407 qmp_basic.py
389 qmp_basic_rhel6.py
356 stepmaker.py
247 steps.py

// The next largest actual test of QEMU
203 pci_hotplug.py
190 cdrom.py
182 physical_resources_check.py
181 timedrift.py
170 enospc.py
150 balloon_check.py
138 multi_disk.py
121 unittest.py
117 migration.py
111 cpu_hotplug.py
107 migration_multi_host.py
104 nic_hotplug.py
103 timedrift_with_stop.py
96 timedrift_with_migration.py
...

So picking virtio-serial as your comparison point is not really representative 
of kvm-autotest but at any rate...

>
>
> What the qemu-test version covers:
> * host starts qemu with one virtio console device backed by a file
> * guest verifies if the name of the device is correct
> * guest writes to the console device
> * host verifies if guest wrote to the virtio console
>
> What the virtio-console covers:
>
> * Sends data between host and guest back and forth, validates the data being
> sent, for both small and large amounts of data, both random or sequential.

This is an improvement.  qemu-test has shared random seed so adding a mode like 
this wouldn't be hard.

> * Tests write/send in blocking, polling, selecting mode, with port mode sync/async

This is a Linux regression test, not a QEMU test.

> * Verifies if the maximum amount of ports was created and it's available in the
> guest

This is an improvement and would be trivial to add.

> * Tries lseek on the device

This is a Linux regression test.

> * Verifies if concomitant access to a single port passes or fails

I think this is also a Linux test.

> * Verifies data throughput and resource utilization

This is outside the scope of qemu-test.

> * Repeats the data transfer with live migration to see if the port connections
> survive

This is neat and an improvement.  I haven't attempted to do migration testing 
with qemu-test.

> * Keeps an eye on the guest OS to see if we have kernel panics, and if it does,
> it'll clean up and put the guest in a working state so other functionality can
> be checked

This is a Linux test.

> * Probably something else I'm forgetting right now.
>
> Good luck implementing that in a shell script. I'd love to see how you implement
> that amount of coverage in less lines in shell.
>
> So, more coverage, more code. It's as simple as that. We don't write code just
> for the sake of it.

I hope it's clear though that most of what you're doing here is an integration 
test with Linux.  This is exactly what autotest should be doing IMHO and exactly 
not what we should be doing in qemu.git.

>> I really see this all as over architecting to be honest.
>>
>> Can someone explain in clear terms (without appealing to maturity,
>> flexibility, or any other qualitative aspect) why it takes anything more
>> than a few dozen shell functions to write all of the tests that are in
>> kvm-autotest test today?
>
> Clearly as your requirements are different than the ones we had when the project
> was written, qemu-test doesn't need any of the stuff present there and you can
> do it all with a handful of shell script functions.
>
> But to do cover the same things covered in autotest today with the same
> requirements, I really doubt you can do the same with the same handful of shell
> script functions. See the example about the virtio test above, not to mention
> other functionality we have with the tests, such as make possible to do tests
> run with a migration thread going on the background, while it's
> plugging/unplugging devices, running a stress test or ltp, or a benchmark, or
> measuring the time drift experienced by the guest.

That's fine.  Your requirements are different and that's why you need autotest. 
  But we don't have the same requirements in qemu.git.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/09/2012 09:13 AM, Anthony Liguori wrote:
> On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
>> Here is the qemu-test version
>>
>> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master
>>
>
> So virtio-serial is an exception in autotest:
>
> 2174 virtio_console.py
> 1875 cgroup.py
> 615 ksm_overcommit.py
> 439 qemu_img.py
> 407 qmp_basic.py
> 389 qmp_basic_rhel6.py
> 356 stepmaker.py
> 247 steps.py
>
> // The next largest actual test of QEMU
> 203 pci_hotplug.py
> 190 cdrom.py
> 182 physical_resources_check.py
> 181 timedrift.py
> 170 enospc.py
> 150 balloon_check.py
> 138 multi_disk.py
> 121 unittest.py
> 117 migration.py
> 111 cpu_hotplug.py
> 107 migration_multi_host.py
> 104 nic_hotplug.py
> 103 timedrift_with_stop.py
> 96 timedrift_with_migration.py
> ...
>
> So picking virtio-serial as your comparison point is not really
> representative of kvm-autotest but at any rate...

We have a bunch of high level test functions in 
client/virt/virt_test_utils.py that contain some commonly used test 
functions such as migration and running autotest tests on vms, and other 
functions, that allow us to reuse those functions on the tests and save 
code, but we can reasonably assume that it doesn't change the order of 
magnitude of the actual qemu tests in size, so point taken.

You also have a point in the respect that a lot of the large tests are 
more linux-qemu integration tests, name cpuflags, cgroups, ksm_overcommit.

The point you tried to make and I replied to was 'qemu-test tests are 
all smaller than the equivalent kvm autotest tests'. Well, sure they 
tend to be, but in pretty much all cases more code means more 
functionality being covered, and making sure the same test works on 
RHEL5, RHEL6, upstream, on an Ubuntu, Fedora, RHEL or even Windows guests.

It is indeed a bit nerve wrecking to hear that all you can do with the 
stuff you have been working on the last 3 years can be done better with 
a dozen of shell script functions. It's similar to say that we just like 
to throw lines at a text editor just for the fun of it. I am sure you 
didn't mean it but that is how it sounded, and that's why I'd like to 
assure that the code there *does stuff*. It's just that this extra stuff 
is potentially not interesting to the goal of doing developer level 
regression testing of qemu alone.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 06:48 AM, Lucas Meneghel Rodrigues wrote:
> On 03/09/2012 09:13 AM, Anthony Liguori wrote:
>> On 03/08/2012 05:07 PM, Lucas Meneghel Rodrigues wrote:
>>> Here is the qemu-test version
>>>
>>> http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master
>>>
>>>
>>
>> So virtio-serial is an exception in autotest:
>>
>> 2174 virtio_console.py
>> 1875 cgroup.py
>> 615 ksm_overcommit.py
>> 439 qemu_img.py
>> 407 qmp_basic.py
>> 389 qmp_basic_rhel6.py
>> 356 stepmaker.py
>> 247 steps.py
>>
>> // The next largest actual test of QEMU
>> 203 pci_hotplug.py
>> 190 cdrom.py
>> 182 physical_resources_check.py
>> 181 timedrift.py
>> 170 enospc.py
>> 150 balloon_check.py
>> 138 multi_disk.py
>> 121 unittest.py
>> 117 migration.py
>> 111 cpu_hotplug.py
>> 107 migration_multi_host.py
>> 104 nic_hotplug.py
>> 103 timedrift_with_stop.py
>> 96 timedrift_with_migration.py
>> ...
>>
>> So picking virtio-serial as your comparison point is not really
>> representative of kvm-autotest but at any rate...
>
> We have a bunch of high level test functions in client/virt/virt_test_utils.py
> that contain some commonly used test functions such as migration and running
> autotest tests on vms, and other functions, that allow us to reuse those
> functions on the tests and save code, but we can reasonably assume that it
> doesn't change the order of magnitude of the actual qemu tests in size, so point
> taken.
>
> You also have a point in the respect that a lot of the large tests are more
> linux-qemu integration tests, name cpuflags, cgroups, ksm_overcommit.

And this is exactly the type of thing that autotest will always be the best tool 
for.

>
> The point you tried to make and I replied to was 'qemu-test tests are all
> smaller than the equivalent kvm autotest tests'. Well, sure they tend to be, but
> in pretty much all cases more code means more functionality being covered, and
> making sure the same test works on RHEL5, RHEL6, upstream, on an Ubuntu, Fedora,
> RHEL or even Windows guests.

But this is not the scope of qemu-test.  And we may end up duplicating some 
things here but does it really matter?  We're talking about a few dozen python 
scripts that are a ~100 lines.

> It is indeed a bit nerve wrecking to hear that all you can do with the stuff you
> have been working on the last 3 years can be done better with a dozen of shell
> script functions. It's similar to say that we just like to throw lines at a text
> editor just for the fun of it. I am sure you didn't mean it but that is how it
> sounded, and that's why I'd like to assure that the code there *does stuff*.

Look at how this discussion started.  We've been discussing testing on 
qemu-devel at excruciating length and detail and have finally come to something 
of a consensus.  AFAIK, no one from autotest has participated in those 
discussions which is fair as I'm sure ya'll don't read qemu-devel religiously.

Then we see this note that more or less declares, this is how QEMU should do all 
of its testing.  What reaction did you really expect there to be? :-)

> It's just that this extra stuff is potentially not interesting to the goal of
> doing developer level regression testing of qemu alone.

I think we need to focus this discussion on concrete technical proposals.  If 
the proposal is, QEMU should use libautotest, we need to start with an awful lot 
more detail about libautotest does and what functions it provides.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/09/2012 11:13 AM, Anthony Liguori wrote:
>> It is indeed a bit nerve wrecking to hear that all you can do with the
>> stuff you
>> have been working on the last 3 years can be done better with a dozen
>> of shell
>> script functions. It's similar to say that we just like to throw lines
>> at a text
>> editor just for the fun of it. I am sure you didn't mean it but that
>> is how it
>> sounded, and that's why I'd like to assure that the code there *does
>> stuff*.
>
> Look at how this discussion started. We've been discussing testing on
> qemu-devel at excruciating length and detail and have finally come to
> something of a consensus. AFAIK, no one from autotest has participated
> in those discussions which is fair as I'm sure ya'll don't read
> qemu-devel religiously.

All right, point taken.

> Then we see this note that more or less declares, this is how QEMU
> should do all of its testing. What reaction did you really expect there
> to be? :-)

It was an attempt to offer what we have rather than dictating how QEMU 
should do all its testing:

"""
One of our main goals is to provide useful tools for the qemu community,
since we have a good number of tests and libraries written to perform
integration/QA testing for that tool, being successfuly used by a number
of QA teams that work on qemu.
"""

I re-read the first message I sent, and certainly did not find where we 
declare that this is the way QEMU should do its testing.

If you judge that there's nothing interesting there for qemu, I'm fine 
with it. That all said, certainly I did not expect your repeated 
attempts to show that you can do it all better with a couple of shell 
script lines, oh boy, I did not.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Fri, Mar 09, 2012 at 08:13:45AM -0600, Anthony Liguori wrote:
> On 03/09/2012 06:48 AM, Lucas Meneghel Rodrigues wrote:
>
> Look at how this discussion started.  We've been discussing testing
> on qemu-devel at excruciating length and detail and have finally
> come to something of a consensus.  AFAIK, no one from autotest has
> participated in those discussions which is fair as I'm sure ya'll
> don't read qemu-devel religiously.

The discussion started back in December and this is a follow-up
RFC based on what was said back then.

(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg02434.html)

> 
> Then we see this note that more or less declares, this is how QEMU
> should do all of its testing.  What reaction did you really expect
> there to be? :-)

This is a clear misunderstanding: we've repeated numerous times
that libautotest and the test runner are optional. I don't know
how much I have to stress this out.

Technically speaking, the use of autotest would be enabled by
"import autotest", "source $AUTOTESTDIR/libautotest.sh" or
something similar. This can be easily detected by the test
runner, which could skip such tests if autotest is not available.

We're also offering a global test-runner that we believe will
bring advantages to anybody running tests that follow the simple
requirement of being a script returning 0/error.

> 
> >It's just that this extra stuff is potentially not interesting to the goal of
> >doing developer level regression testing of qemu alone.
> 
> I think we need to focus this discussion on concrete technical
> proposals.  If the proposal is, QEMU should use libautotest, we need
> to start with an awful lot more detail about libautotest does and
> what functions it provides.

The discussion diverged long ago. So much that the initial test
examples, bootstrap procedure, directory structure, test runner
output, and our request for requiements was all kind of
forgotten.

Anyway, we'll keep working on autotest targeting the other
projects which will benefit from it. As I said in the other
e-mail, we'll probably revisit this topic in the KVM forum in
person, with more code to show and hopefully with some beers
around us. :-)

Cheers,
  - Ademar

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 08/03/2012 22:24, Anthony Liguori ha scritto:
> The qemu-test tests are smaller than the corresponding autotest tests.

They also do much less.

It's true that a combination of qemu-test + qtests could do 99% of the
job more simply than autotest.  But the last 1% (including migration)
would require a large effort, while it would be just there in autotest.

> Making autotest an unconditional dependency of qemu.git is a non-starter for me.

If it were as simple as "yum install py-autotest" it would not be a
problem, would it?

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 07:07 AM, Paolo Bonzini wrote:
> Il 08/03/2012 22:24, Anthony Liguori ha scritto:
>> The qemu-test tests are smaller than the corresponding autotest tests.
>
> They also do much less.
>
> It's true that a combination of qemu-test + qtests could do 99% of the
> job more simply than autotest.  But the last 1% (including migration)
> would require a large effort, while it would be just there in autotest.

Can you clarify here?  When you say "migration", what I think you mean is that 
kvm-autotest has the ability to run autotest client tests within a guest while 
migration.

I do not believe (with the exception of virtio-serial) that it has the ability 
to run tests like pci_hotplug while migrating.

That means libautotest would have nothing to do with the migration test in 
kvm-autotest.

IOW, if we integrate qemu-test with gtest, and autotest learned how to speak the 
gtest protocol, what features would not be available to gtest-based tests verses 
ones written against libautotest?

>
>> Making autotest an unconditional dependency of qemu.git is a non-starter for me.
>
> If it were as simple as "yum install py-autotest" it would not be a
> problem, would it?

It if existed today, had clear value, was widely available, and had a stable 
interface, it wouldn't be an insurmountable problem.

But none of those points are true right now.  And right now, the most important 
one is "has clear value".  We really need to have a clear technical discussion 
about libautotest would actually do.

AFAICT, only two things have been suggested for what it would do:

1) it replaces gtest with its own protocol (but why is this protocol better than 
gtest?)

2) it adds a launching abstraction layer such that the same test can be launched 
with QEMU directly and with libvirt.

But (2) is an anti-feature from my point of view.  The last thing we need is yet 
another layer of abstraction for launching QEMU.

Regards,

Anthony Liguori

>
> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 14:56, Anthony Liguori ha scritto:
> On 03/09/2012 07:07 AM, Paolo Bonzini wrote:
>> Il 08/03/2012 22:24, Anthony Liguori ha scritto:
>>> The qemu-test tests are smaller than the corresponding autotest 
>>> tests.
>> 
>> They also do much less.
>> 
>> It's true that a combination of qemu-test + qtests could do 99% of 
>> the job more simply than autotest.  But the last 1% (including 
>> migration) would require a large effort, while it would be just 
>> there in autotest.
> 
> Can you clarify here?  When you say "migration", what I think you 
> mean is that kvm-autotest has the ability to run autotest client 
> tests within a guest while migration.

1) Migration is quite different from other tests in terms of VM
lifecycle, even if you use migrate-to-file + restore.  It requires
special infrastructure in the drivers.

2) Migration bugs are subtle and often nondeterministic, so they require
long running tests that do the same thing over and over again.  These
requirements are more similar to integration tests than unit tests.  So
integration tests cannot really be dismissed.

BTW, I don't see the problem with requiring a large "make check" time.
I probably already said this: when I was working on GCC, I ran tests
overnight every time I had piled up enough changes, and it was simply
not possible to skip that step because compiler changes often had
surprising ramifications.  Yes, in some cases I had a bad surprise the
following day, but it's unavoidable.

Even today, bootstrapping and testing GCC on embedded systems can be
done but it takes 4-5 *days*.  Of course you do it first in a
cross-compiler, but surprises still can happen.

But we're lucky: compilers and QEMU are pretty much the opposite, and a
bad surprise should be much more rare here.  QEMU tests and
infrastructure are harder, but the components are delimited much better
than in a compiler with 200 cascading passes, which is what makes qtest
possible at all.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 08:43 AM, Paolo Bonzini wrote:
> Il 09/03/2012 14:56, Anthony Liguori ha scritto:
>> On 03/09/2012 07:07 AM, Paolo Bonzini wrote:
>>> Il 08/03/2012 22:24, Anthony Liguori ha scritto:
>>>> The qemu-test tests are smaller than the corresponding autotest
>>>> tests.
>>>
>>> They also do much less.
>>>
>>> It's true that a combination of qemu-test + qtests could do 99% of
>>> the job more simply than autotest.  But the last 1% (including
>>> migration) would require a large effort, while it would be just
>>> there in autotest.
>>
>> Can you clarify here?  When you say "migration", what I think you
>> mean is that kvm-autotest has the ability to run autotest client
>> tests within a guest while migration.
>
> 1) Migration is quite different from other tests in terms of VM
> lifecycle, even if you use migrate-to-file + restore.  It requires
> special infrastructure in the drivers.
>
> 2) Migration bugs are subtle and often nondeterministic, so they require
> long running tests that do the same thing over and over again.  These
> requirements are more similar to integration tests than unit tests.  So
> integration tests cannot really be dismissed.

I think migration really requires special test support in QEMU.  I've long 
wanted to introduce a migration torture mode that would constantly tear down the 
device model and then restore it during normal operation.

I would then want to make -migration-torture an option to any test.

> BTW, I don't see the problem with requiring a large "make check" time.
> I probably already said this: when I was working on GCC, I ran tests
> overnight every time I had piled up enough changes, and it was simply
> not possible to skip that step because compiler changes often had
> surprising ramifications.  Yes, in some cases I had a bad surprise the
> following day, but it's unavoidable.
>
> Even today, bootstrapping and testing GCC on embedded systems can be
> done but it takes 4-5 *days*.  Of course you do it first in a
> cross-compiler, but surprises still can happen.

I think it's fair to have multiple levels of check.  There should be a short 
sanity check one at least but I have no problem with make check-full taking a 
long period of time.

Regards,

Anthony Liguori

> But we're lucky: compilers and QEMU are pretty much the opposite, and a
> bad surprise should be much more rare here.  QEMU tests and
> infrastructure are harder, but the components are delimited much better
> than in a compiler with 200 cascading passes, which is what makes qtest
> possible at all.
>
> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 08/03/2012 18:59, Ademar Reis ha scritto:
> unit-test:
>     void main()
>     {
>         my_function();
>     }
> 
> integration test (or validation test):
>     void main()
>     {
>         exec("my-application");
>     }
> 
> But that's all semantics, not important for this discussion IMO.

We do have some "real" unit tests in QEMU, but because a large part of
it is not suitable to inclusion as a library, there's also something in
the middle for QEMU.

Both qtest and qemu-io-tests are in some sense integration tests because
they need working binaries (respectively qemu and qemu-io).  However,
the protocol spoken to the binary, and the kind of stimuli in the test
are designed to affect a particular function or module of QEMU; that
makes the tests more similar to a unit test, especially for qtest which
is written using gtest or PyUnit.

This is where we want to have most of test case growth in the
not-so-long term.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Kevin Wolf]

Am 08.03.2012 15:56, schrieb Anthony Liguori:
>> I particularly agreed with basically everything you said on that
>> discussion regarding test simplification (I had just joined the
>> team back then). To me, autotest has been focusing on QE-level,
>> leaving the developer-level test requirements out. Now we're
>> attacking this new front, and a lot of the requirements are
>> indeed from that discussion.
> 
> If you want to talk about this in terms of "requirements", my requirement is for 
> "developer-level" tests to live in qemu.git and be integrated into make check.

Actually, I think make check should be something that runs _really_
quickly. We'll probably want another make check-full or something that
runs more extensive tests.

That's the reason why the patch I proposed earlier today runs only a few
qemu-iotests cases during make check. The whole thing would already take
too long for everyone to use it. If people run some quick sanity tests
for all of qemu plus the full tests in the subsystem in which they're
making changes, I think that would be the best we can realistically expect.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 04:46:09PM +0100, Kevin Wolf wrote:
> Am 08.03.2012 15:56, schrieb Anthony Liguori:
> >> I particularly agreed with basically everything you said on that
> >> discussion regarding test simplification (I had just joined the
> >> team back then). To me, autotest has been focusing on QE-level,
> >> leaving the developer-level test requirements out. Now we're
> >> attacking this new front, and a lot of the requirements are
> >> indeed from that discussion.
> > 
> > If you want to talk about this in terms of "requirements", my requirement is for 
> > "developer-level" tests to live in qemu.git and be integrated into make check.
> 
> Actually, I think make check should be something that runs _really_
> quickly. We'll probably want another make check-full or something that
> runs more extensive tests.

I like make check because it's part of make, but I would suggest:

$ test-runner <dir> | <file> ...

This would allow use-cases such as:

[qemu]$ test-runner tests.d/block/
  (run all block tests)

[qemu]$ test-runner tests.d/core/whatever-i-want-to-test.sh
  (run one individual test)

[qemu]$ test-runner tests.d/
  (run all tests)

[qemu]$ test-runner tests.d/devices/*pci*
  (you get the idea)

And, of course:
[qemu]$ test-runner --remote=autotest.qemu.org tests.d/block

(a test path/name is its ID to both the local test runner and the
remote server)

Cheers,
  - Ademar

> 
> That's the reason why the patch I proposed earlier today runs only a few
> qemu-iotests cases during make check. The whole thing would already take
> too long for everyone to use it. If people run some quick sanity tests
> for all of qemu plus the full tests in the subsystem in which they're
> making changes, I think that would be the best we can realistically expect.
> 
> Kevin

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 09:57 AM, Ademar Reis wrote:
> On Thu, Mar 08, 2012 at 04:46:09PM +0100, Kevin Wolf wrote:
>> Am 08.03.2012 15:56, schrieb Anthony Liguori:
>>>> I particularly agreed with basically everything you said on that
>>>> discussion regarding test simplification (I had just joined the
>>>> team back then). To me, autotest has been focusing on QE-level,
>>>> leaving the developer-level test requirements out. Now we're
>>>> attacking this new front, and a lot of the requirements are
>>>> indeed from that discussion.
>>>
>>> If you want to talk about this in terms of "requirements", my requirement is for
>>> "developer-level" tests to live in qemu.git and be integrated into make check.
>>
>> Actually, I think make check should be something that runs _really_
>> quickly. We'll probably want another make check-full or something that
>> runs more extensive tests.
>
> I like make check because it's part of make, but I would suggest:
>
> $ test-runner<dir>  |<file>  ...
>
> This would allow use-cases such as:
>
> [qemu]$ test-runner tests.d/block/
>    (run all block tests)

make check-block

>
> [qemu]$ test-runner tests.d/core/whatever-i-want-to-test.sh
>    (run one individual test)

./tests/core/whatever-i-want-to-test

>
> [qemu]$ test-runner tests.d/
>    (run all tests)

make check

>
> [qemu]$ test-runner tests.d/devices/*pci*
>    (you get the idea)

make check-pci

>
> And, of course:
> [qemu]$ test-runner --remote=autotest.qemu.org tests.d/block

I don't understand what this would do.

Are you proposing test-runner as a replacement for gtest?

Regards,

Anthony Liguori

> (a test path/name is its ID to both the local test runner and the
> remote server)
>
> Cheers,
>    - Ademar
>
>>
>> That's the reason why the patch I proposed earlier today runs only a few
>> qemu-iotests cases during make check. The whole thing would already take
>> too long for everyone to use it. If people run some quick sanity tests
>> for all of qemu plus the full tests in the subsystem in which they're
>> making changes, I think that would be the best we can realistically expect.
>>
>> Kevin
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Kevin Wolf]

Am 08.03.2012 17:10, schrieb Anthony Liguori:
>> And, of course:
>> [qemu]$ test-runner --remote=autotest.qemu.org tests.d/block
> 
> I don't understand what this would do.

>From the previous discussions on this topic, I suppose it would task the
autotest instance at autotest.qemu.org to run the block tests on my git
tree and send me an email with the results, or something like that. For
running full tests this would be a great thing to have, I don't want my
own development boxes to be busy for several hours. Not sure if it can
work for upstream, though.

And if you want to translate the syntax, I guess it would be make
remote=autotest.qemu.org check-block in your proposal. I'm fine with
either syntax.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 10:34 AM, Kevin Wolf wrote:
> Am 08.03.2012 17:10, schrieb Anthony Liguori:
>>> And, of course:
>>> [qemu]$ test-runner --remote=autotest.qemu.org tests.d/block
>>
>> I don't understand what this would do.
>
>  From the previous discussions on this topic, I suppose it would task the
> autotest instance at autotest.qemu.org to run the block tests on my git
> tree and send me an email with the results, or something like that. For
> running full tests this would be a great thing to have, I don't want my
> own development boxes to be busy for several hours. Not sure if it can
> work for upstream, though.

I don't think that's realistic from an upstream PoV.  We don't have the 
infrastructure today to host that and the companies that do have that 
infrastructure behind their firewall I imagine.

Regards,

Anthony Liguori

>
> And if you want to translate the syntax, I guess it would be make
> remote=autotest.qemu.org check-block in your proposal. I'm fine with
> either syntax.
>
> Kevin
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Ademar Reis]

On Thu, Mar 08, 2012 at 10:36:42AM -0600, Anthony Liguori wrote:
> On 03/08/2012 10:34 AM, Kevin Wolf wrote:
> >Am 08.03.2012 17:10, schrieb Anthony Liguori:
> >>>And, of course:
> >>>[qemu]$ test-runner --remote=autotest.qemu.org tests.d/block
> >>
> >>I don't understand what this would do.
> >
> > From the previous discussions on this topic, I suppose it would task the
> >autotest instance at autotest.qemu.org to run the block tests on my git
> >tree and send me an email with the results, or something like that. For
> >running full tests this would be a great thing to have, I don't want my
> >own development boxes to be busy for several hours. Not sure if it can
> >work for upstream, though.
> 
> I don't think that's realistic from an upstream PoV.  We don't have
> the infrastructure today to host that and the companies that do have
> that infrastructure behind their firewall I imagine.
> 

Sure, that's a goal for the long long run. We don't have anything
similar in autotest either. the host autotest.qemu.org was just an
arbitrary example.

> 
> >
> >And if you want to translate the syntax, I guess it would be make
> >remote=autotest.qemu.org check-block in your proposal. I'm fine with
> >either syntax.

Sure, make check-whatever looks good. It's less granular, but
equally good (and we can easily create the make targets if the
tool is flexible enough)

-- 
Ademar de Souza Reis Jr.
Red Hat

^[:wq!

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Kevin Wolf]

Am 08.03.2012 17:36, schrieb Anthony Liguori:
> On 03/08/2012 10:34 AM, Kevin Wolf wrote:
>> Am 08.03.2012 17:10, schrieb Anthony Liguori:
>>>> And, of course:
>>>> [qemu]$ test-runner --remote=autotest.qemu.org tests.d/block
>>>
>>> I don't understand what this would do.
>>
>>  From the previous discussions on this topic, I suppose it would task the
>> autotest instance at autotest.qemu.org to run the block tests on my git
>> tree and send me an email with the results, or something like that. For
>> running full tests this would be a great thing to have, I don't want my
>> own development boxes to be busy for several hours. Not sure if it can
>> work for upstream, though.
> 
> I don't think that's realistic from an upstream PoV.  We don't have the 
> infrastructure today to host that and the companies that do have that 
> infrastructure behind their firewall I imagine.

Yeah, that's what I imagine, too. It would still be useful to have
support for this kind of things in the infrastructure so that people who
do have access to a private test host can make use of it.

Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 09:46 AM, Kevin Wolf wrote:
> Am 08.03.2012 15:56, schrieb Anthony Liguori:
>>> I particularly agreed with basically everything you said on that
>>> discussion regarding test simplification (I had just joined the
>>> team back then). To me, autotest has been focusing on QE-level,
>>> leaving the developer-level test requirements out. Now we're
>>> attacking this new front, and a lot of the requirements are
>>> indeed from that discussion.
>>
>> If you want to talk about this in terms of "requirements", my requirement is for
>> "developer-level" tests to live in qemu.git and be integrated into make check.
>
> Actually, I think make check should be something that runs _really_
> quickly. We'll probably want another make check-full or something that
> runs more extensive tests.

Yeah, my qtest series has a few make check variants that all run with different 
speeds.

>
> That's the reason why the patch I proposed earlier today runs only a few
> qemu-iotests cases during make check. The whole thing would already take
> too long for everyone to use it. If people run some quick sanity tests
> for all of qemu plus the full tests in the subsystem in which they're
> making changes, I think that would be the best we can realistically expect.

Yup, agreed.

Regards,

Anthony Liguori

>
> Kevin

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

Before I forget, I'd like to ask you about this:

On 03/08/2012 10:36 AM, Anthony Liguori wrote:
> I'm really not a fan of buildroot. Note that in order to ship binaries,
> full source needs to be provided in order to comply with the GPL. The
> FSF at least states that referring to another website for source that's
> not under your control doesn't satisfy the requirements of the GPL.

About using buildroot, what is up with it, since it is mature and works 
well? You mentioned than providing all the sources is harder than it 
looks like, and I surely think this might be the case.

But in all my naiveness, if the problem is to ship the exact source with 
the images have been built, couldn't I just ask buildroot to fetch all 
the tarball sources (there's a function to perform source download only) 
and add them to the appropriate git branch? Granted, it sounds horribly 
inefficient space wise, but wouldn't it solve the requirements? Anyone 
can uncompress tarballs and see the source there.

I've built the images using the latest release tarballs of each project, 
such as linux, uclibc, busybox and such. I wanted something up to date 
enough (example, linux 3.2.6) but not downright git master HEAD, since 
lots of problems can creep up by doing it.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 09:19 AM, Lucas Meneghel Rodrigues wrote:
> Before I forget, I'd like to ask you about this:
>
> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>> I'm really not a fan of buildroot. Note that in order to ship binaries,
>> full source needs to be provided in order to comply with the GPL. The
>> FSF at least states that referring to another website for source that's
>> not under your control doesn't satisfy the requirements of the GPL.
>
> About using buildroot, what is up with it, since it is mature and works well?
> You mentioned than providing all the sources is harder than it looks like, and I
> surely think this might be the case.

buildroot is a full blown distribution.  But instead of distributing binaries, 
it only distributes source code.  Think of it like Gentoo--.

It relies on third party links to fetch said source code which means that it's 
not unusual at all for buildroot to straight out fail.  Because it needs a GCC 
that can build shared libraries, it also requires rebuilding GCC which increases 
the build time even for a minimalistic configuration.

Having a JeOS it not meant to replace having proper guests (like Fedora or 
Ubuntu).  As I've said in other notes, the point of having a JeOS is to 
essentially use Linux as a libOS when writing unit tests.

> But in all my naiveness, if the problem is to ship the exact source with the
> images have been built, couldn't I just ask buildroot to fetch all the tarball
> sources (there's a function to perform source download only) and add them to the
> appropriate git branch?

Okay, let's say I'm a developer and I want to use a new Linux kernel to test the 
new virtio-pci feature I added with buildroot.  Where do I begin?  What patch do 
I submit?

That's the use case I'm trying to go after.

Regards,

Anthony Liguori

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/08/2012 03:57 PM, Anthony Liguori wrote:
> On 03/08/2012 09:19 AM, Lucas Meneghel Rodrigues wrote:
>> Before I forget, I'd like to ask you about this:
>>
>> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>>> I'm really not a fan of buildroot. Note that in order to ship binaries,
>>> full source needs to be provided in order to comply with the GPL. The
>>> FSF at least states that referring to another website for source that's
>>> not under your control doesn't satisfy the requirements of the GPL.
>>
>> About using buildroot, what is up with it, since it is mature and
>> works well?
>> You mentioned than providing all the sources is harder than it looks
>> like, and I
>> surely think this might be the case.
>
> buildroot is a full blown distribution. But instead of distributing
> binaries, it only distributes source code. Think of it like Gentoo--.
>
> It relies on third party links to fetch said source code which means
> that it's not unusual

By this definition, qemu test fetch source code from 3rd party 
repositories just as much, after all you have to fetch your linux and 
busybox code from somewhere, I assume.

at all for buildroot to straight out fail. Because
> it needs a GCC that can build shared libraries, it also requires
> rebuilding GCC which increases the build time even for a minimalistic
> configuration.

But is this additional build time a problem? Is the person expected to 
be messing around with re-creating the minimal guest images all the time?

> Having a JeOS it not meant to replace having proper guests (like Fedora
> or Ubuntu). As I've said in other notes, the point of having a JeOS is
> to essentially use Linux as a libOS when writing unit tests.

That is why we would have a reference binary jeos image that can be 
downloaded (although not a replacement, having a small guest makes setup 
time smaller, as well as smaller time to run the tests). If people need 
to tweak kernel, etc for some reason, I'm assuming that person would 
have their own clone of the repo and rebuild the images with the 
modification, and use that for the tests, I guess just the way you would 
do with qemu-jeos.

>> But in all my naiveness, if the problem is to ship the exact source
>> with the
>> images have been built, couldn't I just ask buildroot to fetch all the
>> tarball
>> sources (there's a function to perform source download only) and add
>> them to the
>> appropriate git branch?
>
> Okay, let's say I'm a developer and I want to use a new Linux kernel to
> test the new virtio-pci feature I added with buildroot. Where do I
> begin? What patch do I submit?

You change the configuration file to build your linux from git rather 
than the tarball, change the linux config and type 'make'. At the end, 
you'll have a patch that can be sent and kept in another 
autotest-buildroot branch, if it's proven to be useful *for other people*.

All in all, I don't see what is the big difference between this workflow 
and qemu-jeos.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 01:34 PM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 03:57 PM, Anthony Liguori wrote:
>> On 03/08/2012 09:19 AM, Lucas Meneghel Rodrigues wrote:
>>> Before I forget, I'd like to ask you about this:
>>>
>>> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>>>> I'm really not a fan of buildroot. Note that in order to ship binaries,
>>>> full source needs to be provided in order to comply with the GPL. The
>>>> FSF at least states that referring to another website for source that's
>>>> not under your control doesn't satisfy the requirements of the GPL.
>>>
>>> About using buildroot, what is up with it, since it is mature and
>>> works well?
>>> You mentioned than providing all the sources is harder than it looks
>>> like, and I
>>> surely think this might be the case.
>>
>> buildroot is a full blown distribution. But instead of distributing
>> binaries, it only distributes source code. Think of it like Gentoo--.
>>
>> It relies on third party links to fetch said source code which means
>> that it's not unusual
>
> By this definition, qemu test fetch source code from 3rd party repositories just
> as much, after all you have to fetch your linux and busybox code from somewhere,
> I assume.

It uses git submodules with repositories hosted on git.qemu.org.

>
> at all for buildroot to straight out fail. Because
>> it needs a GCC that can build shared libraries, it also requires
>> rebuilding GCC which increases the build time even for a minimalistic
>> configuration.
>
> But is this additional build time a problem? Is the person expected to be
> messing around with re-creating the minimal guest images all the time?

I think it's common enough to want to update the guest kernel that yes, this is 
something that many people need to be able to do in order for the test cases to 
be effective.  If it takes 24hr to rebuild the images, that's problematic.

>> Having a JeOS it not meant to replace having proper guests (like Fedora
>> or Ubuntu). As I've said in other notes, the point of having a JeOS is
>> to essentially use Linux as a libOS when writing unit tests.
>
> That is why we would have a reference binary jeos image that can be downloaded
> (although not a replacement, having a small guest makes setup time smaller, as
> well as smaller time to run the tests). If people need to tweak kernel, etc for
> some reason, I'm assuming that person would have their own clone of the repo and
> rebuild the images with the modification, and use that for the tests, I guess
> just the way you would do with qemu-jeos.

Yes, but this is a common thing.  This is something that needs to be designed 
for upfront for the infrastructure to be useful.

>>> But in all my naiveness, if the problem is to ship the exact source
>>> with the
>>> images have been built, couldn't I just ask buildroot to fetch all the
>>> tarball
>>> sources (there's a function to perform source download only) and add
>>> them to the
>>> appropriate git branch?
>>
>> Okay, let's say I'm a developer and I want to use a new Linux kernel to
>> test the new virtio-pci feature I added with buildroot. Where do I
>> begin? What patch do I submit?
>
> You change the configuration file to build your linux from git rather than the
> tarball, change the linux config and type 'make'. At the end, you'll have a
> patch that can be sent and kept in another autotest-buildroot branch, if it's
> proven to be useful *for other people*.

Except you also need to update those tarballs too that you're storing in git, 
remember.

Herein lies the problem.  You forgot and it's your proposal :-)

To make this model work, you're going to need to heavily change buildroot to use 
a local repository of tarballs and make sure that people update tarballs 
appropriately.  Updating tarballs is cumbersome and there's obviously a question 
of pedigree.  Are you going to require the tarballs come with the original 
release signature?  How are you going to guard against people taking tarballs of 
their own local trees instead of Linus' tree?

We already have this problem with the linux-headers FWIW.

qemu-jeos exists exactly to solve these problems in a robust fashion, based on 
the experiences we've had maintaining ROM modules in qemu.git.

Regards,

Anthony Liguori

>
> All in all, I don't see what is the big difference between this workflow and
> qemu-jeos.
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Lucas Meneghel Rodrigues]

On 03/08/2012 04:43 PM, Anthony Liguori wrote:
> On 03/08/2012 01:34 PM, Lucas Meneghel Rodrigues wrote:
>> On 03/08/2012 03:57 PM, Anthony Liguori wrote:
>>> On 03/08/2012 09:19 AM, Lucas Meneghel Rodrigues wrote:
>>>> Before I forget, I'd like to ask you about this:
>>>>
>>>> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>>>>> I'm really not a fan of buildroot. Note that in order to ship
>>>>> binaries,
>>>>> full source needs to be provided in order to comply with the GPL. The
>>>>> FSF at least states that referring to another website for source
>>>>> that's
>>>>> not under your control doesn't satisfy the requirements of the GPL.
>>>>
>>>> About using buildroot, what is up with it, since it is mature and
>>>> works well?
>>>> You mentioned than providing all the sources is harder than it looks
>>>> like, and I
>>>> surely think this might be the case.
>>>
>>> buildroot is a full blown distribution. But instead of distributing
>>> binaries, it only distributes source code. Think of it like Gentoo--.
>>>
>>> It relies on third party links to fetch said source code which means
>>> that it's not unusual
>>
>> By this definition, qemu test fetch source code from 3rd party
>> repositories just
>> as much, after all you have to fetch your linux and busybox code from
>> somewhere,
>> I assume.
>
> It uses git submodules with repositories hosted on git.qemu.org.

The linux, uclibc, gcc and and busybox repositories are nowhere to be 
seen there. Also, from .gitmodules for qemu-jeos:

[submodule "busybox"]
         path = busybox
         url = git://busybox.net/busybox.git
[submodule "linux"]
         path = linux
         url = 
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
[submodule "uClibc"]
         path = uClibc
         url = git://uclibc.org/uClibc.git
[submodule "binutils"]
         path = binutils
         url = git://sources.redhat.com/git/binutils.git
[submodule "gcc"]
         path = gcc
         url = git://gcc.gnu.org/git/gcc.git

So still pretty much a lot of code outside the qemu.org realm.

>>
>> at all for buildroot to straight out fail. Because
>>> it needs a GCC that can build shared libraries, it also requires
>>> rebuilding GCC which increases the build time even for a minimalistic
>>> configuration.
>>
>> But is this additional build time a problem? Is the person expected to be
>> messing around with re-creating the minimal guest images all the time?
>
> I think it's common enough to want to update the guest kernel that yes,
> this is something that many people need to be able to do in order for
> the test cases to be effective. If it takes 24hr to rebuild the images,
> that's problematic.

I'd say more like 2 hours on commodity hardware to do a full rebuild. 
Incremental rebuilds are much quicker (under 20 minutes), since gcc and 
all userspace was already built, so no need to do a full rebuild each 
time you want to update the kernel.

>>> Having a JeOS it not meant to replace having proper guests (like Fedora
>>> or Ubuntu). As I've said in other notes, the point of having a JeOS is
>>> to essentially use Linux as a libOS when writing unit tests.
>>
>> That is why we would have a reference binary jeos image that can be
>> downloaded
>> (although not a replacement, having a small guest makes setup time
>> smaller, as
>> well as smaller time to run the tests). If people need to tweak
>> kernel, etc for
>> some reason, I'm assuming that person would have their own clone of
>> the repo and
>> rebuild the images with the modification, and use that for the tests,
>> I guess
>> just the way you would do with qemu-jeos.
>
> Yes, but this is a common thing. This is something that needs to be
> designed for upfront for the infrastructure to be useful.
>
>>>> But in all my naiveness, if the problem is to ship the exact source
>>>> with the
>>>> images have been built, couldn't I just ask buildroot to fetch all the1:30
>>>> tarball
>>>> sources (there's a function to perform source download only) and add
>>>> them to the
>>>> appropriate git branch?
>>>
>>> Okay, let's say I'm a developer and I want to use a new Linux kernel to
>>> test the new virtio-pci feature I added with buildroot. Where do I
>>> begin? What patch do I submit?
>>
>> You change the configuration file to build your linux from git rather
>> than the
>> tarball, change the linux config and type 'make'. At the end, you'll
>> have a
>> patch that can be sent and kept in another autotest-buildroot branch,
>> if it's
>> proven to be useful *for other people*.
>
> Except you also need to update those tarballs too that you're storing in
> git, remember.
>
> Herein lies the problem. You forgot and it's your proposal :-)

Ok, fair enough :) But still, qemu-jeos points out to external 
repositories, just as much as buildroot. It seems to me that the whole 
point about FSF requiring the source to be under your control is no 
longer valid here.

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Andreas Färber]

Am 08.03.2012 21:17, schrieb Lucas Meneghel Rodrigues:
> [...] qemu-jeos points out to external
> repositories, just as much as buildroot. It seems to me that the whole
> point about FSF requiring the source to be under your control is no
> longer valid here.

As long as no binary is distributed, there's no need to keep sources
around, is there?

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/08/2012 02:17 PM, Lucas Meneghel Rodrigues wrote:
> On 03/08/2012 04:43 PM, Anthony Liguori wrote:
>> On 03/08/2012 01:34 PM, Lucas Meneghel Rodrigues wrote:
>>> On 03/08/2012 03:57 PM, Anthony Liguori wrote:
>>>> On 03/08/2012 09:19 AM, Lucas Meneghel Rodrigues wrote:
>>>>> Before I forget, I'd like to ask you about this:
>>>>>
>>>>> On 03/08/2012 10:36 AM, Anthony Liguori wrote:
>>>>>> I'm really not a fan of buildroot. Note that in order to ship
>>>>>> binaries,
>>>>>> full source needs to be provided in order to comply with the GPL. The
>>>>>> FSF at least states that referring to another website for source
>>>>>> that's
>>>>>> not under your control doesn't satisfy the requirements of the GPL.
>>>>>
>>>>> About using buildroot, what is up with it, since it is mature and
>>>>> works well?
>>>>> You mentioned than providing all the sources is harder than it looks
>>>>> like, and I
>>>>> surely think this might be the case.
>>>>
>>>> buildroot is a full blown distribution. But instead of distributing
>>>> binaries, it only distributes source code. Think of it like Gentoo--.
>>>>
>>>> It relies on third party links to fetch said source code which means
>>>> that it's not unusual
>>>
>>> By this definition, qemu test fetch source code from 3rd party
>>> repositories just
>>> as much, after all you have to fetch your linux and busybox code from
>>> somewhere,
>>> I assume.
>>
>> It uses git submodules with repositories hosted on git.qemu.org.
>
> The linux, uclibc, gcc and and busybox repositories are nowhere to be seen
> there. Also, from .gitmodules for qemu-jeos:
>
> [submodule "busybox"]
> path = busybox
> url = git://busybox.net/busybox.git
> [submodule "linux"]
> path = linux
> url = git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> [submodule "uClibc"]
> path = uClibc
> url = git://uclibc.org/uClibc.git
> [submodule "binutils"]
> path = binutils
> url = git://sources.redhat.com/git/binutils.git
> [submodule "gcc"]
> path = gcc
> url = git://gcc.gnu.org/git/gcc.git
>
> So still pretty much a lot of code outside the qemu.org realm.

Yes, because I haven't mirrored any of this yet :-)  But it's trivial to do.

>>> You change the configuration file to build your linux from git rather
>>> than the
>>> tarball, change the linux config and type 'make'. At the end, you'll
>>> have a
>>> patch that can be sent and kept in another autotest-buildroot branch,
>>> if it's
>>> proven to be useful *for other people*.
>>
>> Except you also need to update those tarballs too that you're storing in
>> git, remember.
>>
>> Herein lies the problem. You forgot and it's your proposal :-)
>
> Ok, fair enough :) But still, qemu-jeos points out to external repositories,
> just as much as buildroot. It seems to me that the whole point about FSF
> requiring the source to be under your control is no longer valid here.

There aren't qemu-jeos binaries on qemu.org.  There won't be until I mirror the 
git repos.

It's the infrastructure that matters here.  Submodules provides a nice 
infrastructure to handle all of this and minimizing the external components 
makes the whole thing much more manageable.

Regards,

Anthony Liguori

>
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 08/03/2012 22:03, Anthony Liguori ha scritto:
>>>
>>>
>>> Herein lies the problem. You forgot and it's your proposal :-)
>>
>> Ok, fair enough :) But still, qemu-jeos points out to external
>> repositories,
>> just as much as buildroot. It seems to me that the whole point about FSF
>> requiring the source to be under your control is no longer valid here.
> 
> There aren't qemu-jeos binaries on qemu.org.  There won't be until I
> mirror the git repos.
> 
> It's the infrastructure that matters here.  Submodules provides a nice
> infrastructure to handle all of this and minimizing the external
> components makes the whole thing much more manageable.

How do you handle out-of-tree patches with submodules (as is the case
when working on new code)?

You want to require tests in order to commit to qemu, but this (for
tests where using qtest is not feasible for any reason) requires all
drivers to be upstream and accessible to qemu-jeos.

Perhaps for this it would make sense to associate a qemu-jeos commit id
with an upstream commit (from upstream) + a quilt patch queue.

But there's also the problem of embedded devices whose toolchain is not
available upstream at all, so you'd need to import those separately and
somehow add a different submodule.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 07:36 AM, Paolo Bonzini wrote:
> Il 08/03/2012 22:03, Anthony Liguori ha scritto:
>>>>
>>>>
>>>> Herein lies the problem. You forgot and it's your proposal :-)
>>>
>>> Ok, fair enough :) But still, qemu-jeos points out to external
>>> repositories,
>>> just as much as buildroot. It seems to me that the whole point about FSF
>>> requiring the source to be under your control is no longer valid here.
>>
>> There aren't qemu-jeos binaries on qemu.org.  There won't be until I
>> mirror the git repos.
>>
>> It's the infrastructure that matters here.  Submodules provides a nice
>> infrastructure to handle all of this and minimizing the external
>> components makes the whole thing much more manageable.
>
> How do you handle out-of-tree patches with submodules (as is the case
> when working on new code)?

It's very easy to update .gitmodules to point to a different tree on your local 
system and then update the ref to a local commit.

So from a development perspective, it's simple.  The harder question is what to 
do about qemu-test HEAD on qemu.org

Maintaining downstreams or patches is just too much work IMHO.  I think for 
qemu.org, we simply have to use Linus or Avi's tree and simply live with the 
consequences.

We could open up another tree on qemu.org with patches if someone was willing to 
maintain it but I think that's a bad strategy to take.  But it's a possibility 
if this really becomes a problem.

>
> You want to require tests in order to commit to qemu, but this (for
> tests where using qtest is not feasible for any reason) requires all
> drivers to be upstream and accessible to qemu-jeos.

We're really just talking about virtio here, no?  Maybe we can convince Rusty to 
have a proper virtio-next.git...

> Perhaps for this it would make sense to associate a qemu-jeos commit id
> with an upstream commit (from upstream) + a quilt patch queue.
>
> But there's also the problem of embedded devices whose toolchain is not
> available upstream at all, so you'd need to import those separately and
> somehow add a different submodule.

I think this ends up being outside the scope of qemu-test.  Perfect is the enemy 
of good here.

Regards,

Anthony Liguori

> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 15:01, Anthony Liguori ha scritto:
>> How do you handle out-of-tree patches with submodules (as is the case
>> when working on new code)?
> 
> It's very easy to update .gitmodules to point to a different tree on
> your local system and then update the ref to a local commit.
> 
> So from a development perspective, it's simple.  The harder question is
> what to do about qemu-test HEAD on qemu.org

Yep.  So you would commit patches to qemu.git with some kind of IOU for
tests, that will be committed as soon as kernel support hits the mainline?

>> You want to require tests in order to commit to qemu, but this (for
>> tests where using qtest is not feasible for any reason) requires all
>> drivers to be upstream and accessible to qemu-jeos.
> 
> We're really just talking about virtio here, no?  Maybe we can convince
> Rusty to have a proper virtio-next.git...
> 
> I think this ends up being outside the scope of qemu-test.  Perfect is
> the enemy of good here.

No, this ends up being outside the scope of Anthony, who only cares
about testing KVM/x86.  We add one virtio device every couple years,
while new ARM boards are added every couple months.  It kind of works
because a large part of the QEMU development community is testing on
KVM/x86, but not entirely.

It is not a problem. of course: you're human, you're one person and one
would need a dozen to set up everything properly.  OTOH, it's too easy
to dismiss buildroot when you are catering to a much smaller objective.
The complete objective would have a much bigger overlap with
buildroot's, and likely the design would too.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 08:30 AM, Paolo Bonzini wrote:
> Il 09/03/2012 15:01, Anthony Liguori ha scritto:
>>> How do you handle out-of-tree patches with submodules (as is the case
>>> when working on new code)?
>>
>> It's very easy to update .gitmodules to point to a different tree on
>> your local system and then update the ref to a local commit.
>>
>> So from a development perspective, it's simple.  The harder question is
>> what to do about qemu-test HEAD on qemu.org
>
> Yep.  So you would commit patches to qemu.git with some kind of IOU for
> tests, that will be committed as soon as kernel support hits the mainline?

Probably commit the test and mark it as an expected failure until support hits 
mainline.

>
>>> You want to require tests in order to commit to qemu, but this (for
>>> tests where using qtest is not feasible for any reason) requires all
>>> drivers to be upstream and accessible to qemu-jeos.
>>
>> We're really just talking about virtio here, no?  Maybe we can convince
>> Rusty to have a proper virtio-next.git...
>>
>> I think this ends up being outside the scope of qemu-test.  Perfect is
>> the enemy of good here.
>
> No, this ends up being outside the scope of Anthony, who only cares
> about testing KVM/x86.  We add one virtio device every couple years,
> while new ARM boards are added every couple months.  It kind of works
> because a large part of the QEMU development community is testing on
> KVM/x86, but not entirely.

qemu-test is a specific solution to a specific problem.  The problem is: how can 
you leverage an existing kernel to simplify writing device model tests without 
having a full blown libOS in qtest.

Linux is the only part that matters here.  The userspace in qemu-jeos is just 
there to give a small environment for Linux to function properly in.

> It is not a problem. of course: you're human, you're one person and one
> would need a dozen to set up everything properly.  OTOH, it's too easy
> to dismiss buildroot when you are catering to a much smaller objective.
> The complete objective would have a much bigger overlap with
> buildroot's, and likely the design would too.

You have a different objective than I do.  You want to build a reproducible 
guest that runs on a wider variety of architectures than common distributions do 
with a good enough userspace to write integration tests with.  That's a good 
goal if you care about the full stack.

I'm skeptical of the value of this from a QEMU point of view.  Do we really care 
if the buildroot version of udev propagates hotplug events correct in buildroot 
for ARM in a beagleboard machine?

I'm sure someone cares about that, but it's ultimately not a QEMU problem.  I 
think we need to focus on getting our house in order (in terms of quality) 
before we start trying to tackle a larger scope.

Regards,

Anthony Liguori

>
> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 15:43, Anthony Liguori ha scritto:
> Linux is the only part that matters here.  The userspace in qemu-jeos
> is just there to give a small environment for Linux to function
> properly in.

But again that's not okay for all testcases.  If I want to do SCSI
tests, I cannot write them in shell scripts because qemu-jeos does not
have sg3_utils.

Even libos would not help; relying on sg3_utils means that the client is
a completely clear reimplementation.  I don't care about the extra trips
to the SCSI specs that sg3_utils would save.  The problem is that since
qtest cannot run on real hardware, I don't have confirmation that I
didn't misinterpret the standards.  Writing all bits of the testsuite
makes it likely to have the same misinterpretations in both the tested
code and the tests, even if I try to write the tests from scratch with
only the spec in front of me.

> I'm skeptical of the value of this from a QEMU point of view.  Do we
> really care if the buildroot version of udev propagates hotplug
> events correct in buildroot for ARM in a beagleboard machine?

I don't care about udev.  But even just binutils+GCC+kernel+uclibc is
not as easy if you care about dozens of architectures.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 09:00 AM, Paolo Bonzini wrote:
> Il 09/03/2012 15:43, Anthony Liguori ha scritto:
>> Linux is the only part that matters here.  The userspace in qemu-jeos
>> is just there to give a small environment for Linux to function
>> properly in.
>
> But again that's not okay for all testcases.  If I want to do SCSI
> tests, I cannot write them in shell scripts because qemu-jeos does not
> have sg3_utils.

What SCSI tests are you trying to write?

Are these the sort of tests that would be interesting to also run on Fedora, 
Windows, and Ubuntu?

Regards,

Anthony Liguori

> Even libos would not help; relying on sg3_utils means that the client is
> a completely clear reimplementation.  I don't care about the extra trips
> to the SCSI specs that sg3_utils would save.  The problem is that since
> qtest cannot run on real hardware, I don't have confirmation that I
> didn't misinterpret the standards.  Writing all bits of the testsuite
> makes it likely to have the same misinterpretations in both the tested
> code and the tests, even if I try to write the tests from scratch with
> only the spec in front of me.
>
>> I'm skeptical of the value of this from a QEMU point of view.  Do we
>> really care if the buildroot version of udev propagates hotplug
>> events correct in buildroot for ARM in a beagleboard machine?
>
> I don't care about udev.  But even just binutils+GCC+kernel+uclibc is
> not as easy if you care about dozens of architectures.
>
> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 16:02, Anthony Liguori ha scritto:
>>
>> But again that's not okay for all testcases.  If I want to do SCSI
>> tests, I cannot write them in shell scripts because qemu-jeos does not
>> have sg3_utils.
> 
> What SCSI tests are you trying to write?

At the very least dump the inquiry pages, mode pages, etc. and see that
they make sense and correspond to the device properties.

> Are these the sort of tests that would be interesting to also run on
> Fedora, Windows, and Ubuntu?

They should give exactly the same output on any guest.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 09:17 AM, Paolo Bonzini wrote:
> Il 09/03/2012 16:02, Anthony Liguori ha scritto:
>>>
>>> But again that's not okay for all testcases.  If I want to do SCSI
>>> tests, I cannot write them in shell scripts because qemu-jeos does not
>>> have sg3_utils.
>>
>> What SCSI tests are you trying to write?
>
> At the very least dump the inquiry pages, mode pages, etc. and see that
> they make sense and correspond to the device properties.

Is this not something that's reasonably easy to do in qtest?

Is it possible to write a C program that does the ioctl and dump the inquiry 
page in a text format conducive to shell parsing?

>> Are these the sort of tests that would be interesting to also run on
>> Fedora, Windows, and Ubuntu?
>
> They should give exactly the same output on any guest.

Is it valuable to have a per-platform test or since this is mostly passthrough 
to the device (I assume), do you just need a single test?

Regards,

Anthony Liguori

>
> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Paolo Bonzini]

Il 09/03/2012 16:24, Anthony Liguori ha scritto:
>> At the very least dump the inquiry pages, mode pages, etc. and see that
>> they make sense and correspond to the device properties.
> 
> Is this not something that's reasonably easy to do in qtest?

Yes (at least with virtio-scsi the libos bits are relatively small; just
think of what it would have been like when the only HBA was LSI), but
with one gotcha...

> Is it possible to write a C program that does the ioctl and dump the
> inquiry page in a text format conducive to shell parsing?

... sg_utils also parses the pages and dumps them in human-readable
format.  This is useful because it provides a completely separate
implementation and avoids problems with misinterpretation of the
standard.  Of course it would work just as well if someone wrote tests
instead of me.

>>> Are these the sort of tests that would be interesting to also run on
>>> Fedora, Windows, and Ubuntu?
>>
>> They should give exactly the same output on any guest.
> 
> Is it valuable to have a per-platform test or since this is mostly
> passthrough to the device (I assume), do you just need a single test?

Ah, understood.  Yeah, a single test is enough for the purpose of
testing QEMU.  If you want to test the driver too, running under Windows
would be useful.

Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Anthony Liguori]

On 03/09/2012 09:34 AM, Paolo Bonzini wrote:
> Il 09/03/2012 16:24, Anthony Liguori ha scritto:
>>> At the very least dump the inquiry pages, mode pages, etc. and see that
>>> they make sense and correspond to the device properties.
>>
>> Is this not something that's reasonably easy to do in qtest?
>
> Yes (at least with virtio-scsi the libos bits are relatively small; just
> think of what it would have been like when the only HBA was LSI), but
> with one gotcha...
>
>> Is it possible to write a C program that does the ioctl and dump the
>> inquiry page in a text format conducive to shell parsing?
>
> ... sg_utils also parses the pages and dumps them in human-readable
> format.  This is useful because it provides a completely separate
> implementation and avoids problems with misinterpretation of the
> standard.  Of course it would work just as well if someone wrote tests
> instead of me.

I don't recommend it in the general case, but it should be trivial to add 
additional packages to qemu-jeos and reuse the toolchain to build them.

I don't think it's all that valuable here.  I think you really want to test this 
via qtest.  You could easy copy/paste code from sg_utils to do the parsing if 
you were so inclined...

>>>> Are these the sort of tests that would be interesting to also run on
>>>> Fedora, Windows, and Ubuntu?
>>>
>>> They should give exactly the same output on any guest.
>>
>> Is it valuable to have a per-platform test or since this is mostly
>> passthrough to the device (I assume), do you just need a single test?
>
> Ah, understood.  Yeah, a single test is enough for the purpose of
> testing QEMU.  If you want to test the driver too, running under Windows
> would be useful.

We should separate integration test (testing multiple components where we want 
to be able to use different versions/implementations of one component) from 
functional/unit tests that are strictly testing a single component.  There isn't 
always a clean separation and there may be overlap, but I don't think we should 
stress out about the overlap.

For instance, doing general purpose I/O testing is something where we want to 
test I/O with a Linux guest, a Windows guest, etc.  This is an integration test 
and we should focus on it.

But we probably want some sort of I/O test in qemu-test too since it provides a 
simple functional test.  Yes, there's overlap, but the functional form of the 
test is so simple that it really isn't that important.

But testing how QEMU handles malformatted virtio-blk requests is something 
that's clearly QEMU specific.  There's no value doing that with a Linux and 
Windows guest (even if it's somehow possible).  It's definitely a unit test 
that's strictly specific to QEMU.

I think autotest should be able to execute QEMU's unit and functional tests.  By 
using a test framework like gtest that exposes a test protocol, it should be 
trivial to add that support.

Regards,

Anthony Liguori

> Paolo

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Cleber Rosa]

On 03/09/2012 12:48 PM, Anthony Liguori wrote:
> On 03/09/2012 09:34 AM, Paolo Bonzini wrote:
>> Il 09/03/2012 16:24, Anthony Liguori ha scritto:
>>>> At the very least dump the inquiry pages, mode pages, etc. and see 
>>>> that
>>>> they make sense and correspond to the device properties.
>>>
>>> Is this not something that's reasonably easy to do in qtest?
>>
>> Yes (at least with virtio-scsi the libos bits are relatively small; just
>> think of what it would have been like when the only HBA was LSI), but
>> with one gotcha...
>>
>>> Is it possible to write a C program that does the ioctl and dump the
>>> inquiry page in a text format conducive to shell parsing?
>>
>> ... sg_utils also parses the pages and dumps them in human-readable
>> format.  This is useful because it provides a completely separate
>> implementation and avoids problems with misinterpretation of the
>> standard.  Of course it would work just as well if someone wrote tests
>> instead of me.
>
> I don't recommend it in the general case, but it should be trivial to 
> add additional packages to qemu-jeos and reuse the toolchain to build 
> them.
>

And then the two code bases (now buildroot and qemu-jeos) would have and 
increasing number of similar features.

> I don't think it's all that valuable here.  I think you really want to 
> test this via qtest.  You could easy copy/paste code from sg_utils to 
> do the parsing if you were so inclined...

And yet more code duplication. Even if a few KLOCs, but still... 
Hopefully I'm not the only one that fears those suggestions becoming real.

>
>>>>> Are these the sort of tests that would be interesting to also run on
>>>>> Fedora, Windows, and Ubuntu?
>>>>
>>>> They should give exactly the same output on any guest.
>>>
>>> Is it valuable to have a per-platform test or since this is mostly
>>> passthrough to the device (I assume), do you just need a single test?
>>
>> Ah, understood.  Yeah, a single test is enough for the purpose of
>> testing QEMU.  If you want to test the driver too, running under Windows
>> would be useful.
>
> We should separate integration test (testing multiple components where 
> we want to be able to use different versions/implementations of one 
> component) from functional/unit tests that are strictly testing a 
> single component.  There isn't always a clean separation and there may 
> be overlap, but I don't think we should stress out about the overlap.
>
> For instance, doing general purpose I/O testing is something where we 
> want to test I/O with a Linux guest, a Windows guest, etc.  This is an 
> integration test and we should focus on it.
>
> But we probably want some sort of I/O test in qemu-test too since it 
> provides a simple functional test.  Yes, there's overlap, but the 
> functional form of the test is so simple that it really isn't that 
> important.
>
> But testing how QEMU handles malformatted virtio-blk requests is 
> something that's clearly QEMU specific.  There's no value doing that 
> with a Linux and Windows guest (even if it's somehow possible).  It's 
> definitely a unit test that's strictly specific to QEMU.
>
> I think autotest should be able to execute QEMU's unit and functional 
> tests.  By using a test framework like gtest that exposes a test 
> protocol, it should be trivial to add that support.
>
> Regards,
>
> Anthony Liguori
>
>> Paolo
>

Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests

[Alon Levy]

On Thu, Mar 08, 2012 at 01:00:27AM -0300, Lucas Meneghel Rodrigues wrote:
[snip]
> 
> https://github.com/autotest/buildroot-autotest

Thanks, it was relatively easy to add qxl to this.

[snip]