Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/29/2011 08:38 AM, Dor Laor wrote:
> On 12/28/2011 07:21 PM, Avi Kivity wrote:
>> Would you add live migration testing to qemu-test? If yes, you're
>> duplicating some more. If not, you're not doing functional or coverage
>> tests for that functionality.
>
>  From the recent threads it looks to me that the 2 advantages of qemu-test over
> kvm-autotest are:
>
> 1. python is not used within the guest
> 2. qemu-test is smaller and simpler
>
> Except for the above 2, kvm autotest is superior.

Sorry, but that's non-sense.

qemu-test builds a custom guest, entirely from source.  This makes it possible 
to efficiently test non-native architectures.  The tests are written for this 
minimal environment (which is not straight forward to do).  This tests will 
never run on Windows as they are 100% tied to their environment.

autotest (let's be clear, there is no such thing as kvm autotest anymore) is a 
framework for executing third party tests.  It's got fancy magic to execute in 
client mode vs. server mode.  There is a suite of tests that are integrated in 
autotest that exercise various types of virtualization functionality.  This 
suite of tests use a special config format to determine what they do.

Included in this, is the ability to create a guest from an ISO either using step 
files or via a guest-specific mechanism (kickstart, etc.).  The tests are 
written to be mostly guest agnostic and are therefore written in Python in a 
cross platform way.

There is essentially no overlap between the two and calling kvm autotest 
superior is like calling the Sun superior to the Moon.

> My main motivation to merge
> qemu-test and kvm autotest is that I fear that qemu-test will be the only
> requirement for committing stuff for qemu and developers will be excused from
> their 'duty' by writing a 50 LOC shell script and assume they done w/ testing.

There is no requirement to write autotest tests to get things merged into QEMU. 
  That's is how things are today.

And I don't think there will ever a requirement to write anything that isn't 
merged directly into qemu.git.  I'm not going to hold up merging a feature until 
another project merges something into their tree.

So unless we're going to merge KVM autotest into qemu.git, I don't think there's 
every going to be a requirement to write a KVM autotest test in order to get 
something merged into qemu.git.

But since autotest is a framework for running third-party tests, it seems to 
make sense for qemu.git to have a test framework that autotest can execute.

And since what we call KVM autotest actually tests a heck of a lot more than 
just QEMU, it makes sense for KVM autotest to continue to focus on full stack 
testing where QEMU is but one of the many components that it tests.

> In addition to that, the 50 LOC will be too basic and only provide value for
> basic functionality tests and kvm autotest folks will need to rewrite all of the
> matching content and beyond.
>
> Those above 2 advantages can be solve:
>
> 1. python is not used within the guest
> - One option is just to use the same shell code in kvm autotest w/o
> no python code and use the same kernel/initramfs.

Yes, you can make a directory in kvm autotest that just imports qemu-test, but 
what's the point of doing that?  Isn't it better for this to live in qemu.git?

> - Another way is to use a plain linux distro w/ python and boot it
> into single user test mode and potentially use a S4 guest snapshot
> or external snapshot to shorten its boot time.

You cannot easily create a "plain linux distro" for an ARM target.  If you don't 
believe me, add ARM guest support to KVM autotest and see how well it works out :-)

> You'll get faster boot time and friendlier code.

Adding S4 resume for "simple tests" seems to be a bit odd to me...

> 2. qemu-test is smaller and simpler
> kvm autotest will have a fast, out of the box mode especially
> designed to answer your requirements.
> It's mainly a matter of preparing a fastTestMode.py that will setup
> all of the above in a similar way that today's
> https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py

I hope you see from the above that this isn't just about having something new 
that has fewer features and is therefore simpler for the time being.

The approach is fundamentally different.

Regards,

Anthony Liguori