From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from [140.186.70.92] (port=44113 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PnFvm-0004mF-9L for qemu-devel@nongnu.org; Wed, 09 Feb 2011 14:42:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PnFvk-00010a-0g for qemu-devel@nongnu.org; Wed, 09 Feb 2011 14:42:25 -0500 Received: from mail-px0-f173.google.com ([209.85.212.173]:52214) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PnFvj-00010O-S2 for qemu-devel@nongnu.org; Wed, 09 Feb 2011 14:42:23 -0500 Received: by pxi16 with SMTP id 16so118232pxi.4 for ; Wed, 09 Feb 2011 11:42:22 -0800 (PST) MIME-Version: 1.0 In-Reply-To: <1296827392-1291-1-git-send-email-mdroth@linux.vnet.ibm.com> References: <1296827392-1291-1-git-send-email-mdroth@linux.vnet.ibm.com> From: Blue Swirl Date: Wed, 9 Feb 2011 21:42:00 +0200 Message-ID: Subject: Re: [Qemu-devel] [RFC][PATCH v6 00/04] qtest: qemu unit testing framework Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Michael Roth Cc: aliguori@linux.vnet.ibm.com, ryanh@us.ibm.com, agl@linux.vnet.ibm.com, qemu-devel@nongnu.org, stefanha@linux.vnet.ibm.com On Fri, Feb 4, 2011 at 3:49 PM, Michael Roth wr= ote: > These patches apply to master (2-04-2011), and can also be obtained from: > git://repo.or.cz/qemu/mdroth.git qtest_v1 > > OVERVIEW: > > QEMU currently lacks a standard means to do targeted unit testing of the = device model. Frameworks like kvm-autotest interact via guest OS, which pro= vide a highly abstracted interface to the underlying machine, and are susce= ptable to bugs in the guest OS itself. This allows for reasonable test cove= rage of guest functionality as a whole, but reduces the accuracy and specif= icity with which we can exercise paths in the underlying devices. > > The following patches provide the basic beginnings of a test framework wh= ich replaces vcpu threads with test threads that interact with the underlyi= ng machine directly, allowing for directed unit/performance testing of indi= vidual devices. Test modules are built directly into the qemu binary, and e= ach module provides the following interfaces: > > init(): > =C2=A0Called in place of qemu's normal machine initialization to setup up= devices explicitly. A full machine can be created here by calling into the= normal init path, as well as minimal machines with a select set of buses/d= evices/IRQ handlers. > > run(): > =C2=A0Test logic that interacts with the now-created machine. > > cleanup(): > =C2=A0Currently unused, but potentially allows for chaining multiple test= s together. Currently we run one module, then exit. > > As mentioned these are very early starting points. We're mostly looking f= or input from the community on the basic approach and overall requirements = for an acceptable framework. A basic RTC test module is provided as an exam= ple. > > BUILD/EXAMPLE USAGE: > > =C2=A0$ ./configure --target-list=3Dx86_64-softmmu --enable-qtest --enabl= e-io-thread > =C2=A0$ make > =C2=A0$ ./x86_64-softmmu/qemu-system-x86_64 -test ? > =C2=A0Available test modules: > =C2=A0rtc > =C2=A0$ ./x86_64-softmmu/qemu-system-x86_64 -test rtc > =C2=A0../qtest/qtest_rtc.c:test_drift():L94: hz: 2, duration_ms: 4999, ex= p_duration: 5000, drift ratio: 0.000200 > =C2=A0../qtest/qtest_rtc.c:test_drift():L111: hz: 1024, duration_ms: 4999= , exp_duration: 5000, drift ratio: 0.000200 > > GENERAL PLAN: > > =C2=A0- Provide libraries for common operations like PCI device enumerati= on, APIC configuration, default-configured machine setup, interrupt handlin= g, etc. > =C2=A0- Develop tests as machine/target specific, potentially make some t= ests re-usable as interfaces are better defined > =C2=A0- Do port i/o via cpu_in/cpu_out commands > =C2=A0- Do guest memory access via a CPUPhysMemoryClient interface > =C2=A0- Allow interrupts to be sent by writing to an FD, detection in tes= t modules via select()/read() > > TODO: > > =C2=A0- A means to propagate test returns values to main i/o thread > =C2=A0- Better defined test harness for individual test cases and/or modu= les, likely via GLib > =C2=A0- Support for multiple test threads in a single test module for sca= lability testing > =C2=A0- Modify vl.c hooks so tests can configure their own timers/clockso= urces > =C2=A0- More test modules, improve current rtc module > =C2=A0- Further implementing/fleshing out of the overall plan > > Comments/feedback are welcome! Would it be possible to couple this with the tracing or Kemari somehow so that you could capture, say, block device traces and feed them to test setup?