[Qemu-devel] gtester questions/issues

[Luiz Capitulino <lcapitulino@redhat.com>]

I've started writing some tests with the glib test framework (used by the qapi
patches) but am facing some issues that doesn't seem to exist with check (our
current framework).

Of course that it's possible that I'm missing something, in this case pointers
are welcome, but I must admit that my first impression wasn't positive.

1. Catching test abortion

By default check runs each test on a separate process, this way it's able to
catch any kind of abortion (such as an invalid pointer deference) and it
prints a very developer friendly message:

 Running suite(s): Memory module test suite
 0%: Checks: 1, Failures: 0, Errors: 1
 check-memory.c:20:E:Memory API:test_read_write_byte_simple:33: (after this point) Received signal 11 (Segmentation fault)

The glib suite doesn't seem to do that, at least not by default, so this is
what you get on an invalid pointer:

 ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2 
 /qapi/visitor/input/int: Segmentation fault (core dumped)
 ~/src/qmp-unstable/build (qapi-review)/ 

Is it possible to have check's functionality someway? I read about the
g_test_trap_fork() function, but one would have to use it manually in
each test case, this is a no-no.

2. Memory leaks

If you write something as simple as:

int main(int argc, char **argv)
{
    g_test_init(&argc, &argv, NULL);
    return g_test_run();
}

And run it under valgrind, you'll see this leaks memory. If you add
tests cases to it you'll see that it floods memory. This makes it almost
impossible to debug memory leaks.

Is there a cleanup function I'm missing? I googled for it, but I found only
other people complaining about this too :(

Now, let me say that this will also happen with check if you it in fork mode
(which is the default). However, the leak goes away when you run it in
non-fork mode which is what you want to do if you want to do any kind of debug
with check (having the bug is still not acceptable though, but the fact is that
it won't bite you in practice).

3. Test output

The default output I get when I run a gtester test is:

 ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2 
 /qapi/visitor/input/int: OK
 /qapi/visitor/input/str: OK
 ~/src/qmp-unstable/build (qapi-review)/ 

Which is probably ok for a small amount of tests. However, you don't want to
look for a list of 10 or more lines to see if a test failed, you want something
more obvious, like what check does:

 ~/src/qmp-unstable/build (qapi-review)/ ./check-qint 
 Running suite(s): QInt test-suite
 100%: Checks: 5, Failures: 0, Errors: 0
 ~/src/qmp-unstable/build (qapi-review)/ 

Now, I read about the gtester program and the gtester-report and I can understand
the wonders of a xml + html report (like having on the web page, etc) but running
two programs and transforming xml is not what developers want to do when they're
running unit-tests every few minutes (not to mention that I'm getting all kinds of
crashes when I run gtester-report in fedora).

Ah, I just found out that check also has xml support but I've never
used it...