From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37734) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dafkK-0001G6-Jk for qemu-devel@nongnu.org; Thu, 27 Jul 2017 06:10:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dafkG-0001LK-Gr for qemu-devel@nongnu.org; Thu, 27 Jul 2017 06:10:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:40310) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dafkG-0001Ki-9g for qemu-devel@nongnu.org; Thu, 27 Jul 2017 06:10:16 -0400 Date: Thu, 27 Jul 2017 11:09:58 +0100 From: "Daniel P. Berrange" Message-ID: <20170727100958.GG2555@redhat.com> Reply-To: "Daniel P. Berrange" References: <87o9so1hl7.fsf@dusky.pond.sub.org> <20170717103308.GI7163@stefanha-x1.localdomain> <8760epr9vw.fsf@dusky.pond.sub.org> <20170721153331.GL18014@stefanha-x1.localdomain> <026a093f-1f79-4af9-a5c9-d540342b472e@redhat.com> <8760eiuxu5.fsf@dusky.pond.sub.org> <9d4a3015-a3d2-e9d5-2254-0bcb77b37161@redhat.com> <871sp2gut9.fsf@dusky.pond.sub.org> <20170727091957.GC5117@lemon.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20170727091957.GC5117@lemon.lan> Subject: Re: [Qemu-devel] Improving QMP test coverage List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Fam Zheng Cc: Markus Armbruster , =?utf-8?B?THVrw6HFoQ==?= Doktor , Cleber Rosa , Stefan Hajnoczi , Amador Pahim , qemu-devel@nongnu.org, Cleber Rosa On Thu, Jul 27, 2017 at 05:19:57PM +0800, Fam Zheng wrote: > On Thu, 07/27 10:14, Markus Armbruster wrote: > > This brings some advantages of "verify output with diff" to tests that > > verify with code. Improvement if it simplifies the verification code. > > > > I'd still prefer *no* verification code (by delegating the job to diff) > > for tests where I can get away wit it. > > Python based iotests can be (re)done in such a way that they print actual logs > (interactions with qtest/monitor, stdout/stderr of QEMU, etc) instead of the > current dot dot dot summary, then we automatically have diff based verification, > no? The python test 149 that I wrote does exactly that. There's no reason why the others couldn't do the same. > One thing I feel painful with bash iotests is how harder it is to write > complicated test scenarios such as migration, incremental backup, etc. Yes, shell is an awful language if you need non-trivial control logic or data structures > On the other hand the iotests are more difficult to debug when things go wrong > because it eats the output which, if done with shell, should be very easy to > get. Even if the python tests are not doing verify-by-diff, it should be fairly easy to wire up an env variable IOTESTS_DEBUG=1 which would force them to propagate stdout/err of all commands run (or at least save it to a log file somewhere). Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|