From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:39705) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgJ13-0005TP-Bv for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:39:47 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RgJ0y-000338-GU for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:39:41 -0500 Received: from mail-yx0-f173.google.com ([209.85.213.173]:35622) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RgJ0y-00032y-D0 for qemu-devel@nongnu.org; Thu, 29 Dec 2011 11:39:36 -0500 Received: by yenm6 with SMTP id m6so8950595yen.4 for ; Thu, 29 Dec 2011 08:39:34 -0800 (PST) Message-ID: <4EFC97C3.4080603@codemonkey.ws> Date: Thu, 29 Dec 2011 10:39:31 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4EEF70B4.3070109@us.ibm.com> <4EF73EF5.8050606@redhat.com> <4EF88EC0.8020301@codemonkey.ws> <4EF8FC88.70809@redhat.com> <4EFA4829.4000207@redhat.com> <4EFA80EA.3050405@codemonkey.ws> <4EFAA2A2.4000107@redhat.com> <4EFB2764.7040006@codemonkey.ws> <4EFB2F36.2090408@redhat.com> <4EFB46DD.4000905@codemonkey.ws> <4EFB5014.9030609@redhat.com> <4EFC7B72.9050802@redhat.com> In-Reply-To: <4EFC7B72.9050802@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [ANNOUNCE] qemu-test: a set of tests scripts for QEMU List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: dlaor@redhat.com Cc: "lmr@redhat.com" , Stefan Hajnoczi , cleber@redhat.com, qemu-devel , Gerd Hoffmann , Cleber Rosa , Avi Kivity On 12/29/2011 08:38 AM, Dor Laor wrote: > On 12/28/2011 07:21 PM, Avi Kivity wrote: >> Would you add live migration testing to qemu-test? If yes, you're >> duplicating some more. If not, you're not doing functional or coverage >> tests for that functionality. > > From the recent threads it looks to me that the 2 advantages of qemu-test over > kvm-autotest are: > > 1. python is not used within the guest > 2. qemu-test is smaller and simpler > > Except for the above 2, kvm autotest is superior. Sorry, but that's non-sense. qemu-test builds a custom guest, entirely from source. This makes it possible to efficiently test non-native architectures. The tests are written for this minimal environment (which is not straight forward to do). This tests will never run on Windows as they are 100% tied to their environment. autotest (let's be clear, there is no such thing as kvm autotest anymore) is a framework for executing third party tests. It's got fancy magic to execute in client mode vs. server mode. There is a suite of tests that are integrated in autotest that exercise various types of virtualization functionality. This suite of tests use a special config format to determine what they do. Included in this, is the ability to create a guest from an ISO either using step files or via a guest-specific mechanism (kickstart, etc.). The tests are written to be mostly guest agnostic and are therefore written in Python in a cross platform way. There is essentially no overlap between the two and calling kvm autotest superior is like calling the Sun superior to the Moon. > My main motivation to merge > qemu-test and kvm autotest is that I fear that qemu-test will be the only > requirement for committing stuff for qemu and developers will be excused from > their 'duty' by writing a 50 LOC shell script and assume they done w/ testing. There is no requirement to write autotest tests to get things merged into QEMU. That's is how things are today. And I don't think there will ever a requirement to write anything that isn't merged directly into qemu.git. I'm not going to hold up merging a feature until another project merges something into their tree. So unless we're going to merge KVM autotest into qemu.git, I don't think there's every going to be a requirement to write a KVM autotest test in order to get something merged into qemu.git. But since autotest is a framework for running third-party tests, it seems to make sense for qemu.git to have a test framework that autotest can execute. And since what we call KVM autotest actually tests a heck of a lot more than just QEMU, it makes sense for KVM autotest to continue to focus on full stack testing where QEMU is but one of the many components that it tests. > In addition to that, the 50 LOC will be too basic and only provide value for > basic functionality tests and kvm autotest folks will need to rewrite all of the > matching content and beyond. > > Those above 2 advantages can be solve: > > 1. python is not used within the guest > - One option is just to use the same shell code in kvm autotest w/o > no python code and use the same kernel/initramfs. Yes, you can make a directory in kvm autotest that just imports qemu-test, but what's the point of doing that? Isn't it better for this to live in qemu.git? > - Another way is to use a plain linux distro w/ python and boot it > into single user test mode and potentially use a S4 guest snapshot > or external snapshot to shorten its boot time. You cannot easily create a "plain linux distro" for an ARM target. If you don't believe me, add ARM guest support to KVM autotest and see how well it works out :-) > You'll get faster boot time and friendlier code. Adding S4 resume for "simple tests" seems to be a bit odd to me... > 2. qemu-test is smaller and simpler > kvm autotest will have a fast, out of the box mode especially > designed to answer your requirements. > It's mainly a matter of preparing a fastTestMode.py that will setup > all of the above in a similar way that today's > https://github.com/autotest/autotest/blob/master/client/tests/kvm/get_started.py I hope you see from the above that this isn't just about having something new that has fewer features and is therefore simpler for the time being. The approach is fundamentally different. Regards, Anthony Liguori