From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:55368) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Whg11-0001KP-5h for qemu-devel@nongnu.org; Tue, 06 May 2014 10:06:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Whg0w-0003YG-5g for qemu-devel@nongnu.org; Tue, 06 May 2014 10:06:39 -0400 Received: from mx1.redhat.com ([209.132.183.28]:55042) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Whg0v-0003Xx-Tv for qemu-devel@nongnu.org; Tue, 06 May 2014 10:06:34 -0400 Date: Tue, 6 May 2014 16:06:28 +0200 From: Stefan Hajnoczi Message-ID: <20140506140628.GA19770@stefanha-thinkpad.redhat.com> References: <20140505152119.GA27180@stefanha-thinkpad.redhat.com> <20140505153209.GF3317@noname.str.redhat.com> <20140506082934.GB8923@stefanha-thinkpad.redhat.com> <20140506085119.GA3941@noname.str.redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20140506085119.GA3941@noname.str.redhat.com> Subject: Re: [Qemu-devel] [PATCH v5 0/3] Add common QEMU control functionality to qemu-iotests List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Kevin Wolf Cc: famz@redhat.com, benoit@irqsave.net, Stefan Hajnoczi , Jeff Cody , qemu-devel@nongnu.org, Max Reitz On Tue, May 06, 2014 at 10:51:19AM +0200, Kevin Wolf wrote: > Am 06.05.2014 um 10:29 hat Stefan Hajnoczi geschrieben: > > On Mon, May 05, 2014 at 05:32:09PM +0200, Kevin Wolf wrote: > > > Am 05.05.2014 um 17:21 hat Stefan Hajnoczi geschrieben: > > > > On Wed, Apr 30, 2014 at 10:55:07AM -0400, Jeff Cody wrote: > > > > > This adds some common functionality to control QEMU for qemu-iotests. > > > > > > > > > > Additionally, test 085 is updated to use this new functionality. > > > > > > > > > > Some minor fixups along the way, to clear up spaced pathname issues, > > > > > for common.rc, test 019, and test 086. > > > > > > > > > > > > > > > Jeff Cody (3): > > > > > block: qemu-iotests - add common.qemu, for bash-controlled qemu tests > > > > > > > > Once a test launches QEMU, it soon needs to parse QMP commands or wait > > > > for QMP events. That doesn't lend itself to the traditional > > > > qemu-iotests shell model. That is why iotests.py exists. > > > > > > > > Shell script is a poor language for test cases that go beyond > > > > pre-defined commands whose output is saved for diffing. The string > > > > manipulation is clumsy, JSON is not supported, tricks with fifos can > > > > easily deadlock or break when a process terminates unexpectedly, etc. > > > > > > > > If we go further in the direction of this patch series, we'll duplicate > > > > existing iotests.py code and have complex shell tests that are hard to > > > > extend. I think it's time to draw the line and convert any test cases > > > > that need to complexity to Python. > > > > > > > > Why not use iotests.py? > > > > > > Because it's hard to use. The "compare against reference output" thing > > > is the first thing that you lose with iotests.py, and it's the most > > > useful feature in qemu-iotests. > > > > > > When a Python test case fails, you get into real debugging. When a shell > > > script test case fails, you usually see immediately from the reference > > > output diff what's wrong. > > > > > > I accept iotest.py for anything that needs to evaluate QMP return > > > values, reluctantly, because we have nothing better. But that's it, I > > > don't actually _like_ using it. > > > > I agree with what you say but we're arguing about different things. > > > > I'm saying: > > * For command output diff tests, use shell. > > * For tests that interact dynamically with running QEMU, use > > iotests.py. > > That's a false dichotomy to start with. What about test cases that > interact dynamically with running QEMU _and_ we want to use a command > output diff? Use print and ./check will diff the output. > > You're saying: > > * Command output diff tests are easy to understand and debug. > > * Tests that interact dynamically with QEMU are harder to debug. > > No. I'm specifically saying that iotests.py tests are harder to debug. > Even if they don't do much dynamic interaction. Not true as mentioned above, use print if you want diffing. > > Just because we like the simple shell tests better doesn't mean writing > > complex interaction tests in shell will make them simple! > > > > These test cases in this patch aren't simple shell tests. Let's admit > > they are complex and use iotests.py, which is a more appropriate tool > > for managing a running QEMU process and interacting with it. > > What's wrong with the tests in this series? I find them quite easy to > read and work with. But yes, I guess you can actually make them complex > by using iotest.py... I guess we'll only know by converting the tests and comparing.