From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:45347) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QUoHZ-0000AE-J9 for qemu-devel@nongnu.org; Thu, 09 Jun 2011 19:04:58 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QUoHY-0005HE-6j for qemu-devel@nongnu.org; Thu, 09 Jun 2011 19:04:57 -0400 Received: from e33.co.us.ibm.com ([32.97.110.151]:56757) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QUoHX-0005H6-W1 for qemu-devel@nongnu.org; Thu, 09 Jun 2011 19:04:56 -0400 Received: from d03relay03.boulder.ibm.com (d03relay03.boulder.ibm.com [9.17.195.228]) by e33.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id p59MvRCY022698 for ; Thu, 9 Jun 2011 16:57:27 -0600 Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay03.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p59N4jcH135444 for ; Thu, 9 Jun 2011 17:04:46 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p59H4iIW020053 for ; Thu, 9 Jun 2011 11:04:44 -0600 Message-ID: <4DF1518C.5030101@linux.vnet.ibm.com> Date: Thu, 09 Jun 2011 18:04:44 -0500 From: Michael Roth MIME-Version: 1.0 References: <20110609154704.71665a02@doriath> <4DF11945.7000108@us.ibm.com> <20110609170241.1ba7fde6@doriath> In-Reply-To: <20110609170241.1ba7fde6@doriath> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] gtester questions/issues List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Luiz Capitulino Cc: Stefan Hajnoczi , Anthony Liguori , Markus Armbruster , qemu-devel On 06/09/2011 03:02 PM, Luiz Capitulino wrote: > On Thu, 09 Jun 2011 14:04:37 -0500 > Anthony Liguori wrote: > >> On 06/09/2011 01:47 PM, Luiz Capitulino wrote: >>> >>> I've started writing some tests with the glib test framework (used by the qapi >>> patches) but am facing some issues that doesn't seem to exist with check (our >>> current framework). >>> >>> Of course that it's possible that I'm missing something, in this case pointers >>> are welcome, but I must admit that my first impression wasn't positive. >>> >>> 1. Catching test abortion >>> >>> By default check runs each test on a separate process, this way it's able to >>> catch any kind of abortion (such as an invalid pointer deference) and it >>> prints a very developer friendly message: >>> >>> Running suite(s): Memory module test suite >>> 0%: Checks: 1, Failures: 0, Errors: 1 >>> check-memory.c:20:E:Memory API:test_read_write_byte_simple:33: (after this point) Received signal 11 (Segmentation fault) >>> >>> The glib suite doesn't seem to do that, at least not by default, so this is >>> what you get on an invalid pointer: >>> >>> ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2 >>> /qapi/visitor/input/int: Segmentation fault (core dumped) >>> ~/src/qmp-unstable/build (qapi-review)/ >>> >>> Is it possible to have check's functionality someway? I read about the >>> g_test_trap_fork() function, but one would have to use it manually in >>> each test case, this is a no-no. >> >> I think this is a personal preference thing. I think having fork() be >> optional is great because it makes it easier to use common state for >> multiple test cases. > > Coupling test-cases like this is almost always a bad thing. Test-cases have > to be independent from each other so that they can be run and debugged > individually, also a failing test won't bring the whole suite down, as this > makes a failing report useless. > > That said, you can still do this sharing without sacrificing essential features. > Like disabling the fork mode altogether or subdividing test cases. > > Anyway, If there's a non-ultra cumbersome way to use g_test_trap_fork() (or any > other workaround) to catch segfaults and abortions, then fine. Otherwise I > consider this a blocker, as any code we're going to test in qemu can possibly > crash. This is really a very basic feature that a C unit-test framework can > offer. > You kind of get the desired behavior if you run the test via something like: gtester -k -o test.xml test-visiter The gtester utility will log the return code after a test bombs, then restart and skip to the test following the one that bombed. And I'm sure gtester-report can process the resulting test.xml in manner similar to check... unfortunately it appears to be broken for me on Ubuntu 10.04 so here's the raw XML dump for reference: R02S13c4d9e6d35c23e8dd988917863a66b1 0.000346 0.000000 0.015056 R02S7acda18e321c5a41ccaee4f524877343 ERROR:/home/mdroth/w/qemu2.git/test-visiter.c:312:test_epic_fail2: assertion failed: (false) 0.000000 0.006355 R02S73a208dd8f1b127c23b6a7883df9b78f 0.000318 0.000036 0.000059 0.008079 XML or HTML...it's not pretty, but we can make use of it for automated tests. And for interactive use I don't think it's as much a problem since that'll for the most part be developers making sure they didn't break any tests before committing, or working on failures picked up by automated runs: not a big deal in those cases if the unit tests stop at the first abort.