Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/29/2011 10:53 AM, Avi Kivity wrote:
> On 12/29/2011 06:39 PM, Anthony Liguori wrote:
>>
>> qemu-test builds a custom guest, entirely from source.  This makes it
>> possible to efficiently test non-native architectures.  The tests are
>> written for this minimal environment (which is not straight forward to
>> do).  This tests will never run on Windows as they are 100% tied to
>> their environment.
>>
>> autotest (let's be clear, there is no such thing as kvm autotest
>> anymore) is a framework for executing third party tests.  It's got
>> fancy magic to execute in client mode vs. server mode.  There is a
>> suite of tests that are integrated in autotest that exercise various
>> types of virtualization functionality.  This suite of tests use a
>> special config format to determine what they do.
>>
>> Included in this, is the ability to create a guest from an ISO either
>> using step files or via a guest-specific mechanism (kickstart, etc.).
>> The tests are written to be mostly guest agnostic and are therefore
>> written in Python in a cross platform way.
>>
>> There is essentially no overlap between the two and calling kvm
>> autotest superior is like calling the Sun superior to the Moon.
>
> Why not extend autotest do the same thing.  Instead of downloading a
> fedora iso it would download a kernel tarball and (cross-)build it.

My plan is to post a qemu-jeos repo that has the build bits for creating the 
kernel/initramfs pair.

qemu-test will then live in qemu.git and just depend on the results of the 
qemu-jeos.git.  I'd maybe even go as far as including the binaries in qemu.git 
but I don't think that would work so well...

>>> My main motivation to merge
>>> qemu-test and kvm autotest is that I fear that qemu-test will be the
>>> only
>>> requirement for committing stuff for qemu and developers will be
>>> excused from
>>> their 'duty' by writing a 50 LOC shell script and assume they done w/
>>> testing.
>>
>> There is no requirement to write autotest tests to get things merged
>> into QEMU.  That's is how things are today.
>>
>> And I don't think there will ever a requirement to write anything that
>> isn't merged directly into qemu.git.  I'm not going to hold up merging
>> a feature until another project merges something into their tree.
>
> What about virtio features (we used to depend on the kernel, now on the
> spec)?  Seabios?

Those are rare occurrences and have always been a bit awkward.  OTOH, we're 
talking about the critical path of essentially every single feature that gets 
merged into QEMU.

>> So unless we're going to merge KVM autotest into qemu.git, I don't
>> think there's every going to be a requirement to write a KVM autotest
>> test in order to get something merged into qemu.git.
>
> Let's consider postcopy live migration being proposed.  In addition to
> various unit tests, it also wants an integration test.  So now we need
> to write a qemu-test postcopy live migration test, and also autotest
> test that tests non-Linux guests, migration under load, with memory
> hotplug, etc.

But the integration test is also going to depend on libvirt support for the 
feature.  So how does this all get orchestrated?  kvm-autotest merges the test 
case first, then QEMU merges the support, then libvirt?  What about VDSM and 
ovirt-engine?  Once kvm-autotest starts supporting VDSM that's yet another 
component to deal with.

Realistically, kvm-autotest is going to have to wait for the bits to get merged 
in all of the necessary pieces and then build a test for it on their own schedule.

>> But since autotest is a framework for running third-party tests, it
>> seems to make sense for qemu.git to have a test framework that
>> autotest can execute.
>>
>> And since what we call KVM autotest actually tests a heck of a lot
>> more than just QEMU, it makes sense for KVM autotest to continue to
>> focus on full stack testing where QEMU is but one of the many
>> components that it tests.
>
> It might have made sense to split the kvm-testing functionality of
> autotest, and have autotest drive that.  We could even have called it
> qemu-test.

I specifically advocated this during Lucas' KVM Forum talk and he was strongly 
opposed to it.

I think kvm autotest would get a lot more interest if the test cases were pulled 
out of autotest, made more stand alone.  They also should be more Unix like 
being divided into individual executables with independent command line options 
over a single do-everything configuration file.

But that's all independent of qemu-test.  qemu-test is really qtest + 
Linux/busybox as a libOS.  Even if the above happened, I would still think 
qemu-test needed to exist.

Regards,

Anthony Liguori