From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=53031 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PnGoo-0000oS-Up for qemu-devel@nongnu.org; Wed, 09 Feb 2011 15:39:20 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PnGom-0002Dx-Hn for qemu-devel@nongnu.org; Wed, 09 Feb 2011 15:39:17 -0500 Received: from e8.ny.us.ibm.com ([32.97.182.138]:45533) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PnGom-0002Dt-F5 for qemu-devel@nongnu.org; Wed, 09 Feb 2011 15:39:16 -0500 Received: from d01dlp02.pok.ibm.com (d01dlp02.pok.ibm.com [9.56.224.85]) by e8.ny.us.ibm.com (8.14.4/8.13.1) with ESMTP id p19GL1I2027456 for ; Wed, 9 Feb 2011 11:21:06 -0500 Received: from d01relay06.pok.ibm.com (d01relay06.pok.ibm.com [9.56.227.116]) by d01dlp02.pok.ibm.com (Postfix) with ESMTP id 26DAC4DE8026 for ; Wed, 9 Feb 2011 15:38:24 -0500 (EST) Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay06.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id p19KdCZI2154712 for ; Wed, 9 Feb 2011 15:39:12 -0500 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id p19KdAla011431 for ; Wed, 9 Feb 2011 15:39:12 -0500 Message-ID: <4D52FB6A.5030805@linux.vnet.ibm.com> Date: Wed, 09 Feb 2011 14:39:06 -0600 From: Michael Roth MIME-Version: 1.0 Subject: Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework References: <1296827392-1291-1-git-send-email-mdroth@linux.vnet.ibm.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Blue Swirl Cc: aliguori@linux.vnet.ibm.com, ryanh@us.ibm.com, agl@linux.vnet.ibm.com, qemu-devel@nongnu.org, stefanha@linux.vnet.ibm.com On 02/09/2011 01:42 PM, Blue Swirl wrote: > On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth wrote: >> These patches apply to master (2-04-2011), and can also be obtained from: >> git://repo.or.cz/qemu/mdroth.git qtest_v1 >> >> OVERVIEW: >> >> QEMU currently lacks a standard means to do targeted unit testing of the device model. Frameworks like kvm-autotest interact via guest OS, which provide a highly abstracted interface to the underlying machine, and are susceptable to bugs in the guest OS itself. This allows for reasonable test coverage of guest functionality as a whole, but reduces the accuracy and specificity with which we can exercise paths in the underlying devices. >> >> The following patches provide the basic beginnings of a test framework which replaces vcpu threads with test threads that interact with the underlying machine directly, allowing for directed unit/performance testing of individual devices. Test modules are built directly into the qemu binary, and each module provides the following interfaces: >> >> init(): >> Called in place of qemu's normal machine initialization to setup up devices explicitly. A full machine can be created here by calling into the normal init path, as well as minimal machines with a select set of buses/devices/IRQ handlers. >> >> run(): >> Test logic that interacts with the now-created machine. >> >> cleanup(): >> Currently unused, but potentially allows for chaining multiple tests together. Currently we run one module, then exit. >> >> As mentioned these are very early starting points. We're mostly looking for input from the community on the basic approach and overall requirements for an acceptable framework. A basic RTC test module is provided as an example. >> >> BUILD/EXAMPLE USAGE: >> >> $ ./configure --target-list=x86_64-softmmu --enable-qtest --enable-io-thread >> $ make >> $ ./x86_64-softmmu/qemu-system-x86_64 -test ? >> Available test modules: >> rtc >> $ ./x86_64-softmmu/qemu-system-x86_64 -test rtc >> ../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200 >> ../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999, exp_duration: 5000, drift ratio: 0.000200 >> >> GENERAL PLAN: >> >> - Provide libraries for common operations like PCI device enumeration, APIC configuration, default-configured machine setup, interrupt handling, etc. >> - Develop tests as machine/target specific, potentially make some tests re-usable as interfaces are better defined >> - Do port i/o via cpu_in/cpu_out commands >> - Do guest memory access via a CPUPhysMemoryClient interface >> - Allow interrupts to be sent by writing to an FD, detection in test modules via select()/read() >> >> TODO: >> >> - A means to propagate test returns values to main i/o thread >> - Better defined test harness for individual test cases and/or modules, likely via GLib >> - Support for multiple test threads in a single test module for scalability testing >> - Modify vl.c hooks so tests can configure their own timers/clocksources >> - More test modules, improve current rtc module >> - Further implementing/fleshing out of the overall plan >> >> Comments/feedback are welcome! > > Would it be possible to couple this with the tracing or Kemari somehow > so that you could capture, say, block device traces and feed them to > test setup? I would think so...it's a pretty open ended framework, a unit test could, say, read in block device traces in some pre-defined format and then execute those against a block device. We're also planning on adding command-line parameters for tests, so a unit test could actually be used as a general testing utility. for instance: qemu -test block-trace-virtio -test-opts tracefile=,target_img=,target_fmt=qcow2,comparison_img=