From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:57119) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S617U-0000tx-QK for qemu-devel@nongnu.org; Fri, 09 Mar 2012 09:48:38 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S617K-0006Xm-Ox for qemu-devel@nongnu.org; Fri, 09 Mar 2012 09:48:36 -0500 Received: from mail-pz0-f45.google.com ([209.85.210.45]:63718) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S617K-0006XZ-IE for qemu-devel@nongnu.org; Fri, 09 Mar 2012 09:48:26 -0500 Received: by dadp14 with SMTP id p14so1733336dad.4 for ; Fri, 09 Mar 2012 06:48:24 -0800 (PST) Message-ID: <4F5A1834.8060107@codemonkey.ws> Date: Fri, 09 Mar 2012 08:48:20 -0600 From: Anthony Liguori MIME-Version: 1.0 References: <4F582EDB.1040608@redhat.com> <4F58B5CB.8040503@codemonkey.ws> <20120308144958.GA25750@t420s.optimusnet> <4F58C897.5020405@codemonkey.ws> <20120308150722.GA30576@t420s.optimusnet> <4F58CCBA.9000702@codemonkey.ws> <20120308160505.GA32360@t420s.optimusnet> <4F58E67A.3050708@codemonkey.ws> <20120308175907.GA4900@t420s.optimusnet> <4F5905AA.3060304@codemonkey.ws> <20120308210209.GA11998@t420s.optimusnet> <4F59237F.6010406@codemonkey.ws> <4F5A0076.8020704@redhat.com> <4F5A0C06.3040907@codemonkey.ws> <4F5A170B.10000@redhat.com> In-Reply-To: <4F5A170B.10000@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: Lucas Meneghel Rodrigues , QEMU devel , Scott Zawalski , Cleber Rosa , "kvm-autotest@redhat.com" , Ademar Reis On 03/09/2012 08:43 AM, Paolo Bonzini wrote: > Il 09/03/2012 14:56, Anthony Liguori ha scritto: >> On 03/09/2012 07:07 AM, Paolo Bonzini wrote: >>> Il 08/03/2012 22:24, Anthony Liguori ha scritto: >>>> The qemu-test tests are smaller than the corresponding autotest >>>> tests. >>> >>> They also do much less. >>> >>> It's true that a combination of qemu-test + qtests could do 99% of >>> the job more simply than autotest. But the last 1% (including >>> migration) would require a large effort, while it would be just >>> there in autotest. >> >> Can you clarify here? When you say "migration", what I think you >> mean is that kvm-autotest has the ability to run autotest client >> tests within a guest while migration. > > 1) Migration is quite different from other tests in terms of VM > lifecycle, even if you use migrate-to-file + restore. It requires > special infrastructure in the drivers. > > 2) Migration bugs are subtle and often nondeterministic, so they require > long running tests that do the same thing over and over again. These > requirements are more similar to integration tests than unit tests. So > integration tests cannot really be dismissed. I think migration really requires special test support in QEMU. I've long wanted to introduce a migration torture mode that would constantly tear down the device model and then restore it during normal operation. I would then want to make -migration-torture an option to any test. > BTW, I don't see the problem with requiring a large "make check" time. > I probably already said this: when I was working on GCC, I ran tests > overnight every time I had piled up enough changes, and it was simply > not possible to skip that step because compiler changes often had > surprising ramifications. Yes, in some cases I had a bad surprise the > following day, but it's unavoidable. > > Even today, bootstrapping and testing GCC on embedded systems can be > done but it takes 4-5 *days*. Of course you do it first in a > cross-compiler, but surprises still can happen. I think it's fair to have multiple levels of check. There should be a short sanity check one at least but I have no problem with make check-full taking a long period of time. Regards, Anthony Liguori > But we're lucky: compilers and QEMU are pretty much the opposite, and a > bad surprise should be much more rare here. QEMU tests and > infrastructure are harder, but the components are delimited much better > than in a compiler with 200 cascading passes, which is what makes qtest > possible at all. > > Paolo