From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:53753) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5blN-0005Aw-LQ for qemu-devel@nongnu.org; Thu, 08 Mar 2012 06:44:11 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S5blL-00014h-Lo for qemu-devel@nongnu.org; Thu, 08 Mar 2012 06:44:05 -0500 Received: from mail-lpp01m010-f45.google.com ([209.85.215.45]:48371) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5blL-00014N-C5 for qemu-devel@nongnu.org; Thu, 08 Mar 2012 06:44:03 -0500 Received: by lahe6 with SMTP id e6so416392lah.4 for ; Thu, 08 Mar 2012 03:44:00 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <4F582EDB.1040608@redhat.com> References: <4F582EDB.1040608@redhat.com> Date: Thu, 8 Mar 2012 11:44:00 +0000 Message-ID: From: Stefan Hajnoczi Content-Type: text/plain; charset=ISO-8859-1 Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Lucas Meneghel Rodrigues Cc: QEMU devel , Scott Zawalski , Cleber Rosa , Autotest Mailing List , "kvm-autotest@redhat.com" , Ademar Reis On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues wrote: > One of our main goals is to provide useful tools for the qemu community, > since we have a good number of tests and libraries written to perform > integration/QA testing for that tool, being successfuly used by a number of > QA teams that work on qemu. Also, we recently provided a subset of that > infrastructure to test libvirt, one of our virtualization projects of > interest. Thanks for sharing. My feeling is that although this proposal opens some interesting possibilities we're missing the key ingredient to attract people to autotest as a virtualization test platform. As a developer who writes tests and would like to test more, my number one requirement is productivity. It is frustrating when there is a test case in my mind but I lack the vocabulary to express it as an automated test. In other words, we need an API to that makes it easy to express virtualization tests. This is the most important thing for me and I won't make any other points in this post but will try to expand more on it. It shouldn't be a heavyweight framework that imposes constraints on tests or have a high learning curve, it should be a library that provides a vocabulary for configuring VMs, interacting with the serial port, etc. I think autotest can already do a lot of that today, but I don't know how. Actually I kind of do because I've looked at test source but it doesn't look easy for me to consume. It hasn't been made accessible to the level where I know autotest is the right choice. I worry that digging into autotest will require significant time investment and impose constraints on simple test cases I want to write. I've preferred to write plain Python unittest tests that invoke qemu-img, qemu-io, and qemu-kvm directly. It's a pain to write them from scratch but still quicker than battling with autotest to write a test every now and then. What would make me use autotest: * A clear productivity win over writing tests from scratch * Python API docs for launching a VM, creating disk image, interacting with the serial port, sending QMP commands, etc I think these unmet needs are why QEMU has been growing its own test infrastructure. Stefan