Re: [Qemu-devel] KVM call minutes for Apr 5

[Lucas Meneghel Rodrigues <lmr@redhat.com>]

On Tue, 2011-04-05 at 16:29 +0100, Stefan Hajnoczi wrote:

> Features that I think are important for a qemu.git kvm-autotest:
> * Public results display (sounds like you're working on this)

^ Yes, we hope to get this setup soon.

> * Public notifications of breakage, qemu.git/master failures to
> qemu-devel mailing list.

^ The challenge is to get enough data to determine what is a new
breakage from a known issue, mainly. More related to have historical
data from test results than anything else, IMO.

> * A one-time contributor can get their code tested.  No requirement to
> set up a server because contributors may not have the resources.

Coming back to the point that many colleagues made: We need a sort of
'make test' on the qemu trees that would fetch autotest and could setup
basic tests that people could run, maybe suggest test sets...

The problem I see is, getting guests up and running using configs that
actually matter is not trivial (there are things such as ensuring that
all auxiliary utilities are installed in a distro agnostic fashion,
having bridges and DHCP server setup on possibly a disconnected work
laptop, and stuff). 

So, having a 'no brains involved at all' setup is quite a challenge,
suggestions welcome. Also, downloading isos, waiting for guests to
install and run thorough tests won't be fast. So J. Random Developer
might not bother to run tests even if we can provide a fool proof,
perfectly automated setup, because it'd take a long time at first to get
the tests run. This is also a challenge.

About the ideas of how a 'make tests' feature would work, please give me
some feedback. I picture something like:

-------------------------------------------------------------------
$ make test
Checking out autotest...
1 - Verifying directories (check if the directory structure expected by the default test config is there)
Creating /tmp/kvm_autotest_root/images
Creating /tmp/kvm_autotest_root/isos
Creating /tmp/kvm_autotest_root/steps_data
Do you want to setup NFS mounts or copy isos to some of those dirs? (y/n)

2 - Creating config files from samples (copy the default config samples to actual config files)
Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/build.cfg from sample
Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/cdkeys.cfg from sample
Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/tests_base.cfg from sample
Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/tests.cfg from sample
Creating config file /home/lmr/Code/autotest-git/client/tests/kvm/unittests.cfg from sample


3 - Do you want to test Linux guests? (y/n)

(if y)
 Verifying iso (make sure we have the OS ISO needed for the default test set) 
 Verifying iso Fedora-14-x86_64-DVD.iso
 /tmp/kvm_autotest_root/isos/linux/Fedora-14-x86_64-DVD.iso not found or corrupted
 Would you like to download it? (y/n)

4 - Do you want to test windows guests? (y/n)

(if y)
 Verifying winutils.iso (make sure we have the utility ISO needed for Windows testing)
 In order to run the KVM autotests in Windows guests, we provide you an ISO that this script can download
 Verifying iso winutils.iso
 /tmp/kvm_autotest_root/isos/windows/winutils.iso not found or corrupted
 Would you like to download it? (y/n)

5 - Checking for the KVM module (make sure kvm is loaded to accelerate qemu)
Running '/sbin/lsmod'
KVM module loaded

6 - Verify needed packages to get started
Please take a look at the online documentation http://www.linux-kvm.org/page/KVM-Autotest/Client_Install (session 'Install Prerequisite packages')
You can also edit the test config files (see output of step 2 for a list).

7 - What are the areas you want to verify (enter all that apply)
 A) Unit tests (does not require to install guests)
 B) Sanity (just install and boot guests)
 C) Networking
 D) Block device
 E) Migration
 F) Time drift

8 - Did you check your config and are ready to start testing (y/n)

Running tests...
-------------------------------------------------------------------

If someone answered n to both 3) and 4) only the unit tests would be
available to execute.

> Perhaps kvm-autotest is a good platform for the automated testing of
> ARM TCG.  Paul is CCed, I recently saw the Jenkins qemu build and boot
> tests he has set up.  Lucas, do you have ideas on how these efforts
> can work together to bring testing to upstream QEMU?
> http://validation.linaro.org/jenkins/job/qemu-boot-images/

I heard about jenkins before and it is indeed a nice project. What they
do here, from what I could assess browsing at the webpage you provided
is:

1) Build qemu.git every time there are commits
2) Boot pre-made 'pristine' images, one is a lenny arm image and the
other is a linaro arm image.

It is possible to do the same with kvm autotest, just a matter of not
performing guest install tests and executing only the boot tests with
pre-made images. What jenkins does here is a even quicker and shorter
version of our sanity jobs.

About how we can work together, I thought about some possibilities:

1) Modify the jenkins test step to execute a kvm autotest job after the
build, with the stripped down test set. We might gain some extra debug
info, that the current test step does not seem to provide
2) Do the normal test step and if that succeeds, trigger a kvm autotest
job that does more comprehensive testing, such as migration, time drift,
block layer, etc

The funny thing is that KVM autotest has infrastructure to do the same
as jenkins does, but jenkins is highly streamlined for the buildbot use
case (continuous build and integration), and I see that as a very nice
advantage. So I'd rather keep use jenkins and have kvm autotest plugged
into it conveniently.