From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:53129) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfxGi-0006fR-Ul for qemu-devel@nongnu.org; Wed, 28 Dec 2011 12:26:29 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RfxGh-00061I-N5 for qemu-devel@nongnu.org; Wed, 28 Dec 2011 12:26:24 -0500 Received: from mx1.redhat.com ([209.132.183.28]:63333) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RfxGh-00061B-BO for qemu-devel@nongnu.org; Wed, 28 Dec 2011 12:26:23 -0500 Message-ID: <4EFB5138.5020502@redhat.com> Date: Wed, 28 Dec 2011 19:26:16 +0200 From: Avi Kivity MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB35AB.6040003@redhat.com> <4EFB4757.4020504@codemonkey.ws> In-Reply-To: <4EFB4757.4020504@codemonkey.ws> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Anthony Liguori Cc: "lmr@redhat.com" , Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa On 12/28/2011 06:44 PM, Anthony Liguori wrote: > On 12/28/2011 09:28 AM, Avi Kivity wrote: >> On 12/28/2011 05:01 PM, Avi Kivity wrote: >>> I'd say that running a ping test is a weak version of kvm-autotest's >>> system tests. Running a synthetic test that pokes values into memory >>> and mmio and sees a packet coming out is a unit test (the latter can in >>> fact be executed without a guest at all, just a table driving calls to >>> the memory and irq APIs). >>> >> >> Consider >> 98d23704138e0 >> 7b4252e83f6f7d >> f7e80adf3cc4 >> c16ada980f43 >> 4abf12f4ea8 >> >> (found by looking for 'fix' in the commit log and filtering out the >> commits that don't support my case) >> >> how can you reject such patches on the grounds that they're not >> accompanied by unit tests? > > That's why I've also proposed qtest. But having written quite a few > qtest unit tests by now, you hit the limits of this type of testing > pretty quickly. Can you describe those limits? > >> only by making it easy to add tests for >> them. I think it would be hard/impossible to test them with >> linux-as-a-guest, since they fix edge cases that linux doesn't invoke. >> But by having our own driver (often just using qmp to poke at memory), >> we can easily generate the sequence that triggers the error. >> >> We'd probably need a library to support setting up a pci device's BARs, >> but that's easy with qmp/python integration. You can even poke a small >> executable into memory and execute it directly, if you really need guest >> cpu interaction. > > Please review the qtest series. I think it offers a pretty good > approach to writing this style of test. But as I mentioned, you hit > the limits pretty quickly. I think it's great, it looks like exactly what I wanted, except it's been delivered on time. I'd really like to see it integrated quickly with some flesh around it, then replying -ENOTEST to all patches. This will improve qemu's quality a lot more than guest boot/ping tests, which we do regularly with kvm-autotest anyway. Think of how new instruction emulations are always accompanied by new kvm-unit-tests tests, I often don't even have to ask for them. -- error compiling committee.c: too many arguments to function