Re: [Qemu-devel] gtester questions/issues

[Michael Roth <mdroth@linux.vnet.ibm.com>]

On 06/09/2011 06:04 PM, Michael Roth wrote:
> On 06/09/2011 03:02 PM, Luiz Capitulino wrote:
>> On Thu, 09 Jun 2011 14:04:37 -0500
>> Anthony Liguori<aliguori@us.ibm.com> wrote:
>>
>>> On 06/09/2011 01:47 PM, Luiz Capitulino wrote:
>>>>
>>>> I've started writing some tests with the glib test framework (used
>>>> by the qapi
>>>> patches) but am facing some issues that doesn't seem to exist with
>>>> check (our
>>>> current framework).
>>>>
>>>> Of course that it's possible that I'm missing something, in this
>>>> case pointers
>>>> are welcome, but I must admit that my first impression wasn't positive.
>>>>
>>>> 1. Catching test abortion
>>>>
>>>> By default check runs each test on a separate process, this way it's
>>>> able to
>>>> catch any kind of abortion (such as an invalid pointer deference)
>>>> and it
>>>> prints a very developer friendly message:
>>>>
>>>> Running suite(s): Memory module test suite
>>>> 0%: Checks: 1, Failures: 0, Errors: 1
>>>> check-memory.c:20:E:Memory API:test_read_write_byte_simple:33:
>>>> (after this point) Received signal 11 (Segmentation fault)
>>>>
>>>> The glib suite doesn't seem to do that, at least not by default, so
>>>> this is
>>>> what you get on an invalid pointer:
>>>>
>>>> ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2
>>>> /qapi/visitor/input/int: Segmentation fault (core dumped)
>>>> ~/src/qmp-unstable/build (qapi-review)/
>>>>
>>>> Is it possible to have check's functionality someway? I read about the
>>>> g_test_trap_fork() function, but one would have to use it manually in
>>>> each test case, this is a no-no.
>>>
>>> I think this is a personal preference thing. I think having fork() be
>>> optional is great because it makes it easier to use common state for
>>> multiple test cases.
>>
>> Coupling test-cases like this is almost always a bad thing. Test-cases
>> have
>> to be independent from each other so that they can be run and debugged
>> individually, also a failing test won't bring the whole suite down, as
>> this
>> makes a failing report useless.
>>
>> That said, you can still do this sharing without sacrificing essential
>> features.
>> Like disabling the fork mode altogether or subdividing test cases.
>>
>> Anyway, If there's a non-ultra cumbersome way to use
>> g_test_trap_fork() (or any
>> other workaround) to catch segfaults and abortions, then fine.
>> Otherwise I
>> consider this a blocker, as any code we're going to test in qemu can
>> possibly
>> crash. This is really a very basic feature that a C unit-test
>> framework can
>> offer.
>>
>
> You kind of get the desired behavior if you run the test via something
> like:
>
> gtester -k -o test.xml test-visiter
>
> The gtester utility will log the return code after a test bombs, then
> restart and skip to the test following the one that bombed. And I'm sure
> gtester-report can process the resulting test.xml in manner similar to
> check... unfortunately it appears to be broken for me on Ubuntu 10.04 so
> here's the raw XML dump for reference:

(the epic_fail test segfaults, the epic_fail2 test fails an assert)

>
> <?xml version="1.0"?>
> <gtester>
> <testbinary path="./test-visiter">
> <binary file="./test-visiter"/>
> <random-seed>R02S13c4d9e6d35c23e8dd988917863a66b1</random-seed>
> <testcase path="/0.15/visiter_core">
> <duration>0.000346</duration>
> <status exit-status="0" n-forks="0" result="success"/>
> </testcase>
> <testcase path="/0.15/epic_fail">
> <duration>0.000000</duration>
> <status exit-status="-256" n-forks="0" result="failed"/>
> </testcase>
> <duration>0.015056</duration>
> </testbinary>
> <testbinary path="./test-visiter">
> <binary file="./test-visiter"/>
> <random-seed>R02S7acda18e321c5a41ccaee4f524877343</random-seed>
> <testcase path="/0.15/visiter_core" skipped="1"/>
> <testcase path="/0.15/epic_fail" skipped="1"/>
> <testcase path="/0.15/epic_fail2">
>
> <error>ERROR:/home/mdroth/w/qemu2.git/test-visiter.c:312:test_epic_fail2: assertion
> failed: (false)</error>
> <duration>0.000000</duration>
> <status exit-status="-256" n-forks="0" result="failed"/>
> </testcase>
> <duration>0.006355</duration>
> </testbinary>
> <testbinary path="./test-visiter">
> <binary file="./test-visiter"/>
> <random-seed>R02S73a208dd8f1b127c23b6a7883df9b78f</random-seed>
> <testcase path="/0.15/visiter_core" skipped="1"/>
> <testcase path="/0.15/epic_fail" skipped="1"/>
> <testcase path="/0.15/epic_fail2" skipped="1"/>
> <testcase path="/0.15/nested_structs">
> <duration>0.000318</duration>
> <status exit-status="0" n-forks="0" result="success"/>
> </testcase>
> <testcase path="/0.15/enums">
> <duration>0.000036</duration>
> <status exit-status="0" n-forks="0" result="success"/>
> </testcase>
> <testcase path="/0.15/nested_enums">
> <duration>0.000059</duration>
> <status exit-status="0" n-forks="0" result="success"/>
> </testcase>
> <duration>0.008079</duration>
> </testbinary>
> </gtester>
>
> XML or HTML...it's not pretty, but we can make use of it for automated
> tests. And for interactive use I don't think it's as much a problem
> since that'll for the most part be developers making sure they didn't
> break any tests before committing, or working on failures picked up by
> automated runs: not a big deal in those cases if the unit tests stop at
> the first abort.