Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Dor Laor <dlaor@redhat.com>]

On 12/28/2011 07:21 PM, Avi Kivity wrote:
>> I think you're advocating for qtest.  This is another important part
>> >  of my testing strategy.  I haven't received a lot of input on that RFC...
>> >
>> >  http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com
>> >
>> >  But there's certain things that I still consider to be unit testing
>> >  (like basic networking tests) that I don't want to have to write with
>> >  qtest.  I'm not up for writing a TCP/IP stack in Python...
> A ping test is not a unit test.
>
> The ping test is already covered by kvm-autotest; just set up a config
> that runs just that; after the initial run you'll have a guest installed
> so it'll be quick.  If we have a DSL guest it'll even be very quick.
>
> To test the various NIC emulations, you don't need a full TCP stack,
> just like you didn't need to write an NTP implementation for qtest's rtc
> test.  Instead you just poke values until it sends out a packet.  If you
> want to test virtio-net with both direct and indirect buffers, you can
> only do that with a unit test, you can't do it using a full linux guest
> since it has its own ideas of when to use indirects and when to avoid
> them (for example, it may choose to always avoid them).
>
>>> >>  I suggest the following:
>>> >>
>>> >>     - qemu-test: qemu unit tests
>>> >>     - kvm-unit-tests: kvm unit tests
>>> >>     - kvm-autotest: unit test drivers + system tests
>> >
>> >
>> >  I would counter with:
>> >
>> >  - gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)

+1 for qtester

>> >  - qtest: low level, single device functional tests

+1 for it

>> >  - kvm-unit-tests: low level, instruction-set level functional tests
> Not really.  kvm-unit-tests tests things specific to kvm.
>
>> >  - qemu-test: higher level functional/coverage tests (multiple device
>> >                interaction)
>> >  - kvm-autotest: unit/functional test drivers + acceptance testing
>> >
>> >  Note that the line I'm drawing is acceptance vs. functional testing,
>> >  not unit vs. integration testing.  Technically, our unit tests are
>> >  things like test-visitor.  Everything else is an integration test.
>> >
>> >  But the separation between kvm-autotest is acceptance testing vs.
>> >  functional testing.
>> >
>> >  Acceptance testing is, "does Windows boot", "can I create three
>> >  virtio-serial devices".
>> >
>> >  Obviously, part of acceptance testing is, "does this set of functional
>> >  tests pass".
> Seems like a very blurry line.  Especially as the functional test is
> weaker than either qtest and kvm-autotest.  I now have to agree with the
> others that it's duplicate functionality.  Does it really matter whether
> you're creating an image by compiling Linux and assembling and
> initramfs, or by downloading Fedora.iso and installing it?  It's testing
> exactly the same thing, guest boot and functionality.
>
> Would you add live migration testing to qemu-test?  If yes, you're
> duplicating some more.  If not, you're not doing functional or coverage
> tests for that functionality.

 From the recent threads it looks to me that the 2 advantages of 
qemu-test over kvm-autotest are:

1. python is not used within the guest
2. qemu-test is smaller and simpler

Except for the above 2, kvm autotest is superior. My main motivation to 
merge qemu-test and kvm autotest is that I fear that qemu-test will be 
the only requirement for committing stuff for qemu and developers will 
be excused from their 'duty' by writing a 50 LOC shell script and assume 
they done w/ testing. In addition to that, the 50 LOC will be too basic 
and only provide value for basic functionality tests and kvm autotest 
folks will need to rewrite all of the matching content and beyond.

Those above 2 advantages can be solve:

1. python is not used within the guest
    - One option is just to use the same shell code in kvm autotest w/o
      no python code and use the same kernel/initramfs.
    - Another way is to use a plain linux distro w/ python and boot it
      into single user test mode and potentially use a S4 guest snapshot
      or external snapshot to shorten its boot time.
      You'll get faster boot time and friendlier code.

2. qemu-test is smaller and simpler
    kvm autotest will have a fast, out of the box mode especially
    designed to answer your requirements.
    It's mainly a matter of preparing a fastTestMode.py that will setup
    all of the above in a similar way that today's 
https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py

Lastly, each test can have a fast|basic|sanity overloaded function that 
will be called by this.

>> >
>>>> >>>  We shouldn't be discussing whether it's possible to merge the two
>>>> >>>  tools, but rather what the technical benefits of doing so would be.
>>>> >>>
>>>> >>>  Since at this point, there is almost no overlap between the two, I
>>>> >>>  don't see any actual technical benefit to merging them.  I see benefit
>>>> >>>  to autotest executing qemu-test, of course.
>>> >>
>>> >>  I'd say that running a ping test is a weak version of kvm-autotest's
>>> >>  system tests.
>> >
>> >  Consider the Linux kernel to be our library of functionality to write
>> >  our unit tests.
> I was about to say the same thing, but with a negative implication.
> Using Linux restricts your tests to what Linux does with the devices.
>
>> >  I don't want to write a TCP/IP stack.  We aren't just grabbing a
>> >  random distro kernel.  We're building one from scratch configured in a
>> >  specific way.
> How does that help?
>
>> >  In theory, we could even host the kernel source on git.qemu.org and
>> >  fork it to add more interesting things in the kernel (although I'd
>> >  prefer not to do this, obviously).
> That way lies madness, though it could be argued that it lies all around us.