From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:38800) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5fvf-0007pX-D3 for qemu-devel@nongnu.org; Thu, 08 Mar 2012 11:11:03 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S5fva-0000Aj-FG for qemu-devel@nongnu.org; Thu, 08 Mar 2012 11:10:58 -0500 Received: from mail-pw0-f45.google.com ([209.85.160.45]:46182) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S5fva-0000AS-8a for qemu-devel@nongnu.org; Thu, 08 Mar 2012 11:10:54 -0500 Received: by pbcuo5 with SMTP id uo5so1955621pbc.4 for ; Thu, 08 Mar 2012 08:10:52 -0800 (PST) Message-ID: <4F58DA08.8080903@codemonkey.ws> Date: Thu, 08 Mar 2012 10:10:48 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4F582EDB.1040608@redhat.com> <4F58B5CB.8040503@codemonkey.ws> <20120308144958.GA25750@t420s.optimusnet> <4F58C897.5020405@codemonkey.ws> <4F58D441.60106@redhat.com> <20120308155703.GA30606@t420s.optimusnet> In-Reply-To: <20120308155703.GA30606@t420s.optimusnet> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Ademar Reis Cc: Kevin Wolf , Lucas Meneghel Rodrigues , QEMU devel , Scott Zawalski , Cleber Rosa , "kvm-autotest@redhat.com" On 03/08/2012 09:57 AM, Ademar Reis wrote: > On Thu, Mar 08, 2012 at 04:46:09PM +0100, Kevin Wolf wrote: >> Am 08.03.2012 15:56, schrieb Anthony Liguori: >>>> I particularly agreed with basically everything you said on that >>>> discussion regarding test simplification (I had just joined the >>>> team back then). To me, autotest has been focusing on QE-level, >>>> leaving the developer-level test requirements out. Now we're >>>> attacking this new front, and a lot of the requirements are >>>> indeed from that discussion. >>> >>> If you want to talk about this in terms of "requirements", my requirement is for >>> "developer-level" tests to live in qemu.git and be integrated into make check. >> >> Actually, I think make check should be something that runs _really_ >> quickly. We'll probably want another make check-full or something that >> runs more extensive tests. > > I like make check because it's part of make, but I would suggest: > > $ test-runner | ... > > This would allow use-cases such as: > > [qemu]$ test-runner tests.d/block/ > (run all block tests) make check-block > > [qemu]$ test-runner tests.d/core/whatever-i-want-to-test.sh > (run one individual test) ./tests/core/whatever-i-want-to-test > > [qemu]$ test-runner tests.d/ > (run all tests) make check > > [qemu]$ test-runner tests.d/devices/*pci* > (you get the idea) make check-pci > > And, of course: > [qemu]$ test-runner --remote=autotest.qemu.org tests.d/block I don't understand what this would do. Are you proposing test-runner as a replacement for gtest? Regards, Anthony Liguori > (a test path/name is its ID to both the local test runner and the > remote server) > > Cheers, > - Ademar > >> >> That's the reason why the patch I proposed earlier today runs only a few >> qemu-iotests cases during make check. The whole thing would already take >> too long for everyone to use it. If people run some quick sanity tests >> for all of qemu plus the full tests in the subsystem in which they're >> making changes, I think that would be the best we can realistically expect. >> >> Kevin >