From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:37830) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QUkXE-0002ZX-B9 for qemu-devel@nongnu.org; Thu, 09 Jun 2011 15:04:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1QUkX9-0008Cy-4x for qemu-devel@nongnu.org; Thu, 09 Jun 2011 15:04:51 -0400 Received: from e1.ny.us.ibm.com ([32.97.182.141]:51390) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1QUkX8-0008Cr-Q0 for qemu-devel@nongnu.org; Thu, 09 Jun 2011 15:04:47 -0400 Received: from d01relay05.pok.ibm.com (d01relay05.pok.ibm.com [9.56.227.237]) by e1.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p59Ir7PS015424 for ; Thu, 9 Jun 2011 14:53:07 -0400 Received: from d03av06.boulder.ibm.com (d03av06.boulder.ibm.com [9.17.195.245]) by d01relay05.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p59J4hF2058062 for ; Thu, 9 Jun 2011 15:04:44 -0400 Received: from d03av06.boulder.ibm.com (loopback [127.0.0.1]) by d03av06.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p59J9sXN002985 for ; Thu, 9 Jun 2011 13:09:54 -0600 Message-ID: <4DF11945.7000108@us.ibm.com> Date: Thu, 09 Jun 2011 14:04:37 -0500 From: Anthony Liguori MIME-Version: 1.0 References: <20110609154704.71665a02@doriath> In-Reply-To: <20110609154704.71665a02@doriath> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] gtester questions/issues List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Luiz Capitulino Cc: qemu-devel , Stefan Hajnoczi , mdroth@linux.vnet.ibm.com, Markus Armbruster On 06/09/2011 01:47 PM, Luiz Capitulino wrote: > > I've started writing some tests with the glib test framework (used by the qapi > patches) but am facing some issues that doesn't seem to exist with check (our > current framework). > > Of course that it's possible that I'm missing something, in this case pointers > are welcome, but I must admit that my first impression wasn't positive. > > 1. Catching test abortion > > By default check runs each test on a separate process, this way it's able to > catch any kind of abortion (such as an invalid pointer deference) and it > prints a very developer friendly message: > > Running suite(s): Memory module test suite > 0%: Checks: 1, Failures: 0, Errors: 1 > check-memory.c:20:E:Memory API:test_read_write_byte_simple:33: (after this point) Received signal 11 (Segmentation fault) > > The glib suite doesn't seem to do that, at least not by default, so this is > what you get on an invalid pointer: > > ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2 > /qapi/visitor/input/int: Segmentation fault (core dumped) > ~/src/qmp-unstable/build (qapi-review)/ > > Is it possible to have check's functionality someway? I read about the > g_test_trap_fork() function, but one would have to use it manually in > each test case, this is a no-no. I think this is a personal preference thing. I think having fork() be optional is great because it makes it easier to use common state for multiple test cases. > > 2. Memory leaks > > If you write something as simple as: > > int main(int argc, char **argv) > { > g_test_init(&argc,&argv, NULL); > return g_test_run(); > } > > And run it under valgrind, you'll see this leaks memory. If you add > tests cases to it you'll see that it floods memory. This makes it almost > impossible to debug memory leaks. > > Is there a cleanup function I'm missing? I googled for it, but I found only > other people complaining about this too :( My version of glib/valgrind doesn't have this problem. Maybe there's a valgrind filter for gtester on ubuntu and not fedora? > > Now, let me say that this will also happen with check if you it in fork mode > (which is the default). However, the leak goes away when you run it in > non-fork mode which is what you want to do if you want to do any kind of debug > with check (having the bug is still not acceptable though, but the fact is that > it won't bite you in practice). > > 3. Test output > > The default output I get when I run a gtester test is: > > ~/src/qmp-unstable/build (qapi-review)/ ./test-visiter2 > /qapi/visitor/input/int: OK > /qapi/visitor/input/str: OK > ~/src/qmp-unstable/build (qapi-review)/ > > Which is probably ok for a small amount of tests. However, you don't want to > look for a list of 10 or more lines to see if a test failed, you want something > more obvious, like what check does: > > ~/src/qmp-unstable/build (qapi-review)/ ./check-qint > Running suite(s): QInt test-suite > 100%: Checks: 5, Failures: 0, Errors: 0 > ~/src/qmp-unstable/build (qapi-review)/ > > Now, I read about the gtester program and the gtester-report and I can understand > the wonders of a xml + html report (like having on the web page, etc) but running > two programs and transforming xml is not what developers want to do when they're > running unit-tests every few minutes (not to mention that I'm getting all kinds of > crashes when I run gtester-report in fedora). I actually like the way gtester does it and the html output is quite nice IMHO. But the main motivator between gtester is that it's there. It can be a non-optional build dependency. libcheck cannot because it's not widely available/used. It's also much harder to use libcheck since you have to create a test hierarchy programmatically. The check tests have bit rotted over time to the point that they're broken in the tree. I attribute this to the fact that they aren't built by default. Regards, Anthony Liguori > Ah, I just found out that check also has xml support but I've never > used it...