Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Avi Kivity <avi@redhat.com>]

On 12/29/2011 06:26 PM, Anthony Liguori wrote:
> On 12/28/2011 11:21 AM, Avi Kivity wrote:
>> On 12/28/2011 06:42 PM, Anthony Liguori wrote:
>>>> In fact using linux as a guest negates that.  First of all, which
>>>> linux
>>>> version? if it's fixed, you'll eventually miss functionality and
>>>> need to
>>>> migrate.  If it keeps changing, so does your test, and it will keep
>>>> breaking.
>>>
>>>
>>> The kernel is a git submodule so it's a very specific version.  Yes,
>>> we may need to bump the version down the road and obviously, if we
>>> have to change any tests in the process, we can.
>>
>> Having a full linux source as part of the build process detracts
>> somewhat from the advantages here.
>
> Except that binaries can be made available.  The initramfs is
> currently 512k while the kernel is about 5MB.
>
> OTOH, a Fedora install runs in the multiple GB.  Having
> kernels/initramfs for the dozen target types that QEMU supports is
> reasonable.  Having Fedora installs for all of them probably isn't for
> most people.

Then use DSL for the ping test.

>
>>> But there's certain things that I still consider to be unit testing
>>> (like basic networking tests) that I don't want to have to write with
>>> qtest.  I'm not up for writing a TCP/IP stack in Python...
>>
>> A ping test is not a unit test.
>>
>> The ping test is already covered by kvm-autotest; just set up a config
>> that runs just that; after the initial run you'll have a guest installed
>> so it'll be quick.  If we have a DSL guest it'll even be very quick.
>
> Achieving this with kvm-autotest is not easy.  I doubt that DSL has an
> automated install mechanism and the step file based guest installation
> is fincky at best.
>
> When we had the kvm-autotest day, none of the folks that tried to set
> it up who had never set it up before had something working by the end
> of the day.  And that was with lmr available on IRC to answer questions.
>
> There is a huge setup cost with kvm-autotest.

That needs to be fixed, not workarounded.  I previously advised Lucas
not to spend too much time on this, since the audience is limited and it
would be hard to amortize the effort.  Now I see that this was wrong.

>
>> To test the various NIC emulations, you don't need a full TCP stack,
>> just like you didn't need to write an NTP implementation for qtest's rtc
>> test.  Instead you just poke values until it sends out a packet.  If you
>> want to test virtio-net with both direct and indirect buffers, you can
>> only do that with a unit test, you can't do it using a full linux guest
>> since it has its own ideas of when to use indirects and when to avoid
>> them (for example, it may choose to always avoid them).
>
> I am not advocating that we don't write proper unit tests...

Right, and qtest is dangerously close to making writing tests fun.  But
qemu-test is a wierd cross between qtest and kvm-autotest.

>
>>> Acceptance testing is, "does Windows boot", "can I create three
>>> virtio-serial devices".
>>>
>>> Obviously, part of acceptance testing is, "does this set of functional
>>> tests pass".
>>
>> Seems like a very blurry line.  Especially as the functional test is
>> weaker than either qtest and kvm-autotest.  I now have to agree with the
>> others that it's duplicate functionality.  Does it really matter whether
>> you're creating an image by compiling Linux and assembling and
>> initramfs, or by downloading Fedora.iso and installing it?
>
> Yes.  One of them is built entirely from source and the other is doing
> something meant to be done on bare metal.
>
> How do you test ARM guests with kvm-autotest?  There is no such thing
> as an ISO installer for ARM.  Are we going to download random images
> from the internet?

Do exactly what you plan to do with qemu-test.

>
> Even if there is an ISO, installing a guest with TCG will take many
> hours.  My local qemu-test repo can now fully bootstrap an initramfs
> for non-x86 targets (which involves building gcc and uClibc).  Start
> to finish it takes about 30 minutes regardless of the target (since
> the build runs natively).

You can skip the install test and only do the boot test.  Seed your disk
with images from qemu.org or by running the install.  kvm-autotest does
this.

>
>> Would you add live migration testing to qemu-test?  If yes, you're
>> duplicating some more.  If not, you're not doing functional or coverage
>> tests for that functionality.
>
> I would add live migration schema testing, absolutely.  I wouldn't add
> an acceptance test though.
>
> An acceptance test with live migration would be creating a guest,
> running it, and then seeing it live migration worked.

Then it's duplicate functionality.  It's already been done, but the
setup procedure is hard, so let's do it again.

>
>>> I don't want to write a TCP/IP stack.  We aren't just grabbing a
>>> random distro kernel.  We're building one from scratch configured in a
>>> specific way.
>>
>> How does that help?
>
> Not sure I understand the question.
>

In what way is your specifically configured kernel's TCP stack better
than the random distro's kernel's?

-- 
error compiling committee.c: too many arguments to function