From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:47037) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QUoJs-0000fy-2Q for qemu-devel@nongnu.org; Thu, 09 Jun 2011 19:07:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QUoJq-00068l-9M for qemu-devel@nongnu.org; Thu, 09 Jun 2011 19:07:19 -0400 Received: from e34.co.us.ibm.com ([32.97.110.152]:54642) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QUoJp-00068d-Q4 for qemu-devel@nongnu.org; Thu, 09 Jun 2011 19:07:18 -0400 Received: from d03relay05.boulder.ibm.com (d03relay05.boulder.ibm.com [9.17.195.107]) by e34.co.us.ibm.com (8.14.4/8.13.1) with ESMTP id p59MsU93005961 for ; Thu, 9 Jun 2011 16:54:30 -0600 Received: from d03av03.boulder.ibm.com (d03av03.boulder.ibm.com [9.17.195.169]) by d03relay05.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p59N76G3166504 for ; Thu, 9 Jun 2011 17:07:07 -0600 Received: from d03av03.boulder.ibm.com (loopback [127.0.0.1]) by d03av03.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p59H75Ew027861 for ; Thu, 9 Jun 2011 11:07:05 -0600 Message-ID: <4DF15219.7090303@linux.vnet.ibm.com> Date: Thu, 09 Jun 2011 18:07:05 -0500 From: Michael Roth MIME-Version: 1.0 References: <20110609154704.71665a02@doriath> <4DF11945.7000108@us.ibm.com> <20110609170241.1ba7fde6@doriath> <4DF1518C.5030101@linux.vnet.ibm.com> In-Reply-To: <4DF1518C.5030101@linux.vnet.ibm.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] gtester questions/issues List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Luiz Capitulino Cc: Stefan Hajnoczi , Anthony Liguori , Markus Armbruster , qemu-devel On 06/09/2011 06:04 PM, Michael Roth wrote: > On 06/09/2011 03:02 PM, Luiz Capitulino wrote: >> On Thu, 09 Jun 2011 14:04:37 -0500 >> Anthony Liguori wrote: >> >>> On 06/09/2011 01:47 PM, Luiz Capitulino wrote: >>>> >>>> I've started writing some tests with the glib test framework (used >>>> by the qapi >>>> patches) but am facing some issues that doesn't seem to exist with >>>> check (our >>>> current framework). >>>> >>>> Of course that it's possible that I'm missing something, in this >>>> case pointers >>>> are welcome, but I must admit that my first impression wasn't positive. >>>> >>>> 1. Catching test abortion >>>> >>>> By default check runs each test on a separate process, this way it's >>>> able to >>>> catch any kind of abortion (such as an invalid pointer deference) >>>> and it >>>> prints a very developer friendly message: >>>> >>>> Running suite(s): Memory module test suite >>>> 0%: Checks: 1, Failures: 0, Errors: 1 >>>> check-memory.c:20:E:Memory API:test_read_write_byte_simple:33: >>>> (after this point) Received signal 11 (Segmentation fault) >>>> >>>> The glib suite doesn't seem to do that, at least not by default, so >>>> this is >>>> what you get on an invalid pointer: >>>> >>>> ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2 >>>> /qapi/visitor/input/int: Segmentation fault (core dumped) >>>> ~/src/qmp-unstable/build (qapi-review)/ >>>> >>>> Is it possible to have check's functionality someway? I read about the >>>> g_test_trap_fork() function, but one would have to use it manually in >>>> each test case, this is a no-no. >>> >>> I think this is a personal preference thing. I think having fork() be >>> optional is great because it makes it easier to use common state for >>> multiple test cases. >> >> Coupling test-cases like this is almost always a bad thing. Test-cases >> have >> to be independent from each other so that they can be run and debugged >> individually, also a failing test won't bring the whole suite down, as >> this >> makes a failing report useless. >> >> That said, you can still do this sharing without sacrificing essential >> features. >> Like disabling the fork mode altogether or subdividing test cases. >> >> Anyway, If there's a non-ultra cumbersome way to use >> g_test_trap_fork() (or any >> other workaround) to catch segfaults and abortions, then fine. >> Otherwise I >> consider this a blocker, as any code we're going to test in qemu can >> possibly >> crash. This is really a very basic feature that a C unit-test >> framework can >> offer. >> > > You kind of get the desired behavior if you run the test via something > like: > > gtester -k -o test.xml test-visiter > > The gtester utility will log the return code after a test bombs, then > restart and skip to the test following the one that bombed. And I'm sure > gtester-report can process the resulting test.xml in manner similar to > check... unfortunately it appears to be broken for me on Ubuntu 10.04 so > here's the raw XML dump for reference: (the epic_fail test segfaults, the epic_fail2 test fails an assert) > > > > > > R02S13c4d9e6d35c23e8dd988917863a66b1 > > 0.000346 > > > > 0.000000 > > > 0.015056 > > > > R02S7acda18e321c5a41ccaee4f524877343 > > > > > ERROR:/home/mdroth/w/qemu2.git/test-visiter.c:312:test_epic_fail2: assertion > failed: (false) > 0.000000 > > > 0.006355 > > > > R02S73a208dd8f1b127c23b6a7883df9b78f > > > > > 0.000318 > > > > 0.000036 > > > > 0.000059 > > > 0.008079 > > > > XML or HTML...it's not pretty, but we can make use of it for automated > tests. And for interactive use I don't think it's as much a problem > since that'll for the most part be developers making sure they didn't > break any tests before committing, or working on failures picked up by > automated runs: not a big deal in those cases if the unit tests stop at > the first abort.