From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:38398) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgIoA-0000Gi-Sz for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:26:28 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RgIo9-0000NP-IE for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:26:22 -0500 Received: from mail-yw0-f45.google.com ([209.85.213.45]:37643) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgIo9-0000NK-Bt for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:26:21 -0500 Received: by yhgg71 with SMTP id g71so9664141yhg.4 for ; Thu, 29 Dec 2011 08:26:21 -0800 (PST) Message-ID: <4EFC94A9.1040300@codemonkey.ws> Date: Thu, 29 Dec 2011 10:26:17 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB46DD.4000905@codemonkey.ws> <4EFB5014.9030609@redhat.com> In-Reply-To: <4EFB5014.9030609@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Avi Kivity Cc: "lmr@redhat.com" , Stefan Hajnoczi , cleber@redhat.com, dlaor@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa On 12/28/2011 11:21 AM, Avi Kivity wrote: > On 12/28/2011 06:42 PM, Anthony Liguori wrote: >>> In fact using linux as a guest negates that. First of all, which linux >>> version? if it's fixed, you'll eventually miss functionality and need to >>> migrate. If it keeps changing, so does your test, and it will keep >>> breaking. >> >> >> The kernel is a git submodule so it's a very specific version. Yes, >> we may need to bump the version down the road and obviously, if we >> have to change any tests in the process, we can. > > Having a full linux source as part of the build process detracts > somewhat from the advantages here. Except that binaries can be made available. The initramfs is currently 512k while the kernel is about 5MB. OTOH, a Fedora install runs in the multiple GB. Having kernels/initramfs for the dozen target types that QEMU supports is reasonable. Having Fedora installs for all of them probably isn't for most people. >> But there's certain things that I still consider to be unit testing >> (like basic networking tests) that I don't want to have to write with >> qtest. I'm not up for writing a TCP/IP stack in Python... > > A ping test is not a unit test. > > The ping test is already covered by kvm-autotest; just set up a config > that runs just that; after the initial run you'll have a guest installed > so it'll be quick. If we have a DSL guest it'll even be very quick. Achieving this with kvm-autotest is not easy. I doubt that DSL has an automated install mechanism and the step file based guest installation is fincky at best. When we had the kvm-autotest day, none of the folks that tried to set it up who had never set it up before had something working by the end of the day. And that was with lmr available on IRC to answer questions. There is a huge setup cost with kvm-autotest. > To test the various NIC emulations, you don't need a full TCP stack, > just like you didn't need to write an NTP implementation for qtest's rtc > test. Instead you just poke values until it sends out a packet. If you > want to test virtio-net with both direct and indirect buffers, you can > only do that with a unit test, you can't do it using a full linux guest > since it has its own ideas of when to use indirects and when to avoid > them (for example, it may choose to always avoid them). I am not advocating that we don't write proper unit tests... >> Acceptance testing is, "does Windows boot", "can I create three >> virtio-serial devices". >> >> Obviously, part of acceptance testing is, "does this set of functional >> tests pass". > > Seems like a very blurry line. Especially as the functional test is > weaker than either qtest and kvm-autotest. I now have to agree with the > others that it's duplicate functionality. Does it really matter whether > you're creating an image by compiling Linux and assembling and > initramfs, or by downloading Fedora.iso and installing it? Yes. One of them is built entirely from source and the other is doing something meant to be done on bare metal. How do you test ARM guests with kvm-autotest? There is no such thing as an ISO installer for ARM. Are we going to download random images from the internet? Even if there is an ISO, installing a guest with TCG will take many hours. My local qemu-test repo can now fully bootstrap an initramfs for non-x86 targets (which involves building gcc and uClibc). Start to finish it takes about 30 minutes regardless of the target (since the build runs natively). > Would you add live migration testing to qemu-test? If yes, you're > duplicating some more. If not, you're not doing functional or coverage > tests for that functionality. I would add live migration schema testing, absolutely. I wouldn't add an acceptance test though. An acceptance test with live migration would be creating a guest, running it, and then seeing it live migration worked. >> I don't want to write a TCP/IP stack. We aren't just grabbing a >> random distro kernel. We're building one from scratch configured in a >> specific way. > > How does that help? Not sure I understand the question. Regards, Anthony Liguori >> In theory, we could even host the kernel source on git.qemu.org and >> fork it to add more interesting things in the kernel (although I'd >> prefer not to do this, obviously). > > That way lies madness, though it could be argued that it lies all around us. > >