From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:38606) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgIaq-0008D9-LR for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:12:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RgIao-0006If-Ps for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:12:36 -0500 Received: from mail-yx0-f173.google.com ([209.85.213.173]:49278) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgIao-0006IZ-M1 for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:12:34 -0500 Received: by yenm6 with SMTP id m6so8938801yen.4 for ; Thu, 29 Dec 2011 08:12:34 -0800 (PST) Message-ID: <4EFC916E.4070902@codemonkey.ws> Date: Thu, 29 Dec 2011 10:12:30 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB35AB.6040003@redhat.com> <4EFB4757.4020504@codemonkey.ws> <4EFB5138.5020502@redhat.com> In-Reply-To: <4EFB5138.5020502@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Avi Kivity Cc: "lmr@redhat.com" , Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa On 12/28/2011 11:26 AM, Avi Kivity wrote: > On 12/28/2011 06:44 PM, Anthony Liguori wrote: >> On 12/28/2011 09:28 AM, Avi Kivity wrote: >>> On 12/28/2011 05:01 PM, Avi Kivity wrote: >>>> I'd say that running a ping test is a weak version of kvm-autotest's >>>> system tests. Running a synthetic test that pokes values into memory >>>> and mmio and sees a packet coming out is a unit test (the latter can in >>>> fact be executed without a guest at all, just a table driving calls to >>>> the memory and irq APIs). >>>> >>> >>> Consider >>> 98d23704138e0 >>> 7b4252e83f6f7d >>> f7e80adf3cc4 >>> c16ada980f43 >>> 4abf12f4ea8 >>> >>> (found by looking for 'fix' in the commit log and filtering out the >>> commits that don't support my case) >>> >>> how can you reject such patches on the grounds that they're not >>> accompanied by unit tests? >> >> That's why I've also proposed qtest. But having written quite a few >> qtest unit tests by now, you hit the limits of this type of testing >> pretty quickly. > > Can you describe those limits? I started writing a finger print test. While it's easy to do PCI enumeration, it gets pretty nasty if you want to actually access BARs (to read virtio registers) and forget about trying to get SMBIOS information as that involves mucking around with ACPI. OTOH, it was very simple to write with qemu-test: http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/finger-print.sh;h=fc715378a3bbae0b458cc419981c2493d98f5c3d;hb=HEAD And I ended up finding some long standing bugs in the process: http://mid.gmane.org/1324305949-20960-2-git-send-email-aliguori@us.ibm.com It's fairly nasty as it would only show up if doing migration from a new QEMU with a new guest to a old QEMU. I think it's a good example of the type of test that I'm targeting at qemu-test. It's really not interesting to run it across multiple distro types. But doing it with qtest would be prohibitively hard. >>> only by making it easy to add tests for >>> them. I think it would be hard/impossible to test them with >>> linux-as-a-guest, since they fix edge cases that linux doesn't invoke. >>> But by having our own driver (often just using qmp to poke at memory), >>> we can easily generate the sequence that triggers the error. >>> >>> We'd probably need a library to support setting up a pci device's BARs, >>> but that's easy with qmp/python integration. You can even poke a small >>> executable into memory and execute it directly, if you really need guest >>> cpu interaction. >> >> Please review the qtest series. I think it offers a pretty good >> approach to writing this style of test. But as I mentioned, you hit >> the limits pretty quickly. > > I think it's great, it looks like exactly what I wanted, except it's > been delivered on time. Can you take a look at how interrupts are handled? From what I gather, the only real limitation of this approach is that we won't be able to simulate MSI vectors but I don't think that's a huge problem. I looked at integrating interrupt handling at CPUs itself and that turned out to be fairly invasive. > I'd really like to see it integrated quickly > with some flesh around it, then replying -ENOTEST to all patches. This > will improve qemu's quality a lot more than guest boot/ping tests, which > we do regularly with kvm-autotest anyway. > > Think of how new instruction emulations are always accompanied by new > kvm-unit-tests tests, I often don't even have to ask for them. Yes, this is exactly where I'm heading with all of this. Regards, Anthony Liguori