From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:34441) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgOCR-0002gQ-NY for qemu-devel@nongnu.org; Thu, 29 Dec 2011 17:11:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RgOCQ-0002Ht-LE for qemu-devel@nongnu.org; Thu, 29 Dec 2011 17:11:47 -0500 Received: from mx1.redhat.com ([209.132.183.28]:57272) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgOCQ-0002Hd-ER for qemu-devel@nongnu.org; Thu, 29 Dec 2011 17:11:46 -0500 Message-ID: <4EFCE5A2.6050409@redhat.com> Date: Thu, 29 Dec 2011 20:11:46 -0200 From: Lucas Meneghel Rodrigues MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB35AB.6040003@redhat.com> <4EFB4757.4020504@codemonkey.ws> <4EFB5138.5020502@redhat.com> <4EFC916E.4070902@codemonkey.ws> <4EFC9706.4090500@redhat.com> <4EFCA2AF.5000806@redhat.com> <4EFCB30E.9030002@codemonkey.ws> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell Cc: Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Avi Kivity , Cleber Rosa , Gerd Hoffmann On 12/29/2011 05:04 PM, Peter Maydell wrote: > On 29 December 2011 18:35, Anthony Liguori wrote: >> On 12/29/2011 11:49 AM, Peter Maydell wrote: >>> The next obvious question is: are we going to make a serious attempt? >>> (For instance, in a hypothetical tests-required world, would we >>> tell those nice folks from Samsung "no you can't land your >>> Exynos patches unless you write 9000+ lines of test cases" ? >> >> The virtio-serial test case I posted was 50 lines in qemu-test. The >> virtio-serial driver is about ~1500 LOC. That's about 3%. > > This just means it's testing only a fraction of what virtio-serial > actually implements (as you note yourself in the comments for the > test case). For serious coverage you'd also need to cover reading(!), > larger quantities of data than single lines, what happens when one end > feeds in data but the other end isn't reading, vmstate save/load, > multiple simultaneous ports, behaviour on close-and-reopen, > whether it works on bigendian targets, benchmarking to identify > possible performance regressions, etc etc. I think that by the time > you've done all that you'll be closer to 1500 lines than 50. Agreed. There's a reason why virtio-serial testing in kvm autotest is 2.1 k LOC for the test itself and 0.9 k LOC for the auxiliary guest script. Most of the scenarios you've described are covered on kvm autotest version. >> I would expect that we at least have some sort of test that could verify >> that the Exynos platform more or less worked as expected. If that was just >> booting a Linux kernel, that would be fine by me. > > Yes. There's a large range between "no tests required at all" (essentially > what we have now) and "an equivalent level of device testing to what you > would carry out before sending your hardware design out to be fabbed into > silicon" (which I hope we'd all agree would be ludicrously high for QEMU); > I'm trying to establish where we're attempting to set our bar. > > I agree that we'd get a lot of bang-for-the-buck out of basic automated > "boot the guest and prod it" testing that covered most of our platforms. And that sort of automated testing can be done right now with KVM autotest. Granted, I never had time to make it work with ARM, but if people are interested, we are happily accepting patches :)