Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU

[Anthony Liguori <anthony@codemonkey.ws>]

On 12/28/2011 09:01 AM, Avi Kivity wrote:
> On 12/28/2011 04:27 PM, Anthony Liguori wrote:
>>> Maybe I've used the wrong wording. I got the feeling that, besides
>>> testing qemu
>>> the way you need it, keeping qemu-test simple was really important.
>>
>>
>> Simple is always important.  In the case of qemu-test, there are some
>> important trade offs made to keep things simple and fast.  It doesn't
>> try to work with arbitrary guests which means the tests need to handle
>> only one environment.  The guest is pre-made to have exactly what is
>> needed for the tests so there is no setup required.  The guest is as
>> small as possible such that the test can run as quickly as possible.
>
> In fact using linux as a guest negates that.  First of all, which linux
> version? if it's fixed, you'll eventually miss functionality and need to
> migrate.  If it keeps changing, so does your test, and it will keep
> breaking.

The kernel is a git submodule so it's a very specific version.  Yes, we may need 
to bump the version down the road and obviously, if we have to change any tests 
in the process, we can.

>
> Using Linux as a guest allows you to do system tests (ping was an
> example) but doesn't allow you to do unit tests (test regression where
> if this bit in that register was set, but if them bits in thar registers
> were clear, then the packet would be encoded with ROT-26 before being
> sent out).
>
> I think qemu-test needs to use its own drivers which allow full control
> over what you do with the hardware.  Otherwise, a regression that only
> shows up in non-Linux guests will not be testable with qemu-test.

I think you're advocating for qtest.  This is another important part of my 
testing strategy.  I haven't received a lot of input on that RFC...

http://mid.gmane.org/1322765012-3164-1-git-send-email-aliguori@us.ibm.com

But there's certain things that I still consider to be unit testing (like basic 
networking tests) that I don't want to have to write with qtest.  I'm not up for 
writing a TCP/IP stack in Python...

>> There is no need to have a single tool that meets every possible
>> need.  In fact, the Unix tradition is to have separate single purposed
>> tools.
>
> Those single purpose tools, if successful, tend to grow more purposes
> (cf. qemu), and if unsuccessful, tend to lose purpose.
>
>> Having two test tools it not a bad thing provided that the overlap
>> isn't significant.
>
> This is important, if the boundary isn't clear, then it will grow more
> fuzzy in time.
>
> I suggest the following:
>
>   - qemu-test: qemu unit tests
>   - kvm-unit-tests: kvm unit tests
>   - kvm-autotest: unit test drivers + system tests

I would counter with:

- gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)
- qtest: low level, single device functional tests
- kvm-unit-tests: low level, instruction-set level functional tests
- qemu-test: higher level functional/coverage tests (multiple device
              interaction)
- kvm-autotest: unit/functional test drivers + acceptance testing

Note that the line I'm drawing is acceptance vs. functional testing, not unit 
vs. integration testing.  Technically, our unit tests are things like 
test-visitor.  Everything else is an integration test.

But the separation between kvm-autotest is acceptance testing vs. functional 
testing.

Acceptance testing is, "does Windows boot", "can I create three virtio-serial 
devices".

Obviously, part of acceptance testing is, "does this set of functional tests pass".

>> We shouldn't be discussing whether it's possible to merge the two
>> tools, but rather what the technical benefits of doing so would be.
>>
>> Since at this point, there is almost no overlap between the two, I
>> don't see any actual technical benefit to merging them.  I see benefit
>> to autotest executing qemu-test, of course.
>
> I'd say that running a ping test is a weak version of kvm-autotest's
> system tests.

Consider the Linux kernel to be our library of functionality to write our unit 
tests.  I don't want to write a TCP/IP stack.  We aren't just grabbing a random 
distro kernel.  We're building one from scratch configured in a specific way.

In theory, we could even host the kernel source on git.qemu.org and fork it to 
add more interesting things in the kernel (although I'd prefer not to do this, 
obviously).

>  Running a synthetic test that pokes values into memory
> and mmio and sees a packet coming out is a unit test (the latter can in
> fact be executed without a guest at all, just a table driving calls to
> the memory and irq APIs).
>
>
>> Just putting the code in kvm-autotest.git in a directory doesn't make
>> sense to me.  Beyond the lack of a technical reason to do so,
>> logistically, it makes it harder for me to ask people to submit test
>> cases with a patch series if I can't actually apply the test case when
>> I'm applying the patches to qemu.git.
>>
>> If qemu-test didn't use large submodules (like linux.git), I would
>> have made qemu-test part of qemu.git.  As far as I'm concerned,
>> qemu-test.git is just an extension to qemu.git.
>
> Why not just put it in qemu.git?

Actually, having thought a bit about it, I think we can move qemu-test into 
qemu.git and just build the initramfs in a separate repository.  That will make 
it easier to host binary packages of the later on qemu.org.

Regards,

Anthony Liguori

>