From mboxrd@z Thu Jan 1 00:00:00 1970 From: Uri Lublin Subject: Re: kvm-autotest -- introducing kvm_runtest_2 Date: Wed, 04 Mar 2009 10:58:54 +0200 Message-ID: <49AE42CE.7080105@redhat.com> References: <49AADD81.1070207@redhat.com> <20090302174504.GH11777@us.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Cc: KVM List To: Ryan Harper Return-path: Received: from mx2.redhat.com ([66.187.237.31]:54310 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753222AbZCDI7A (ORCPT ); Wed, 4 Mar 2009 03:59:00 -0500 In-Reply-To: <20090302174504.GH11777@us.ibm.com> Sender: kvm-owner@vger.kernel.org List-ID: Ryan Harper wrote: > * Uri Lublin [2009-03-01 13:10]: Ryan, Sorry for the late response. >> KVM-autotest is a test framework for kvm, based on autotest >> (http://autotest.kernel.org). > > I've been digging into kvm_runtest_2 and have some feedback, but first > to say, runtest_2 is huge cleanup and simplification from runtest, so > thanks. Now for some comments: > > - kvm_tests.cfg has a decent learning curve to wrap your head around. > It would have been useful to have some debugging that would dump out > what rules were filtering out guests... the dependencies aren't always > easy to find. My first experience with the dep hunt is just changing > 'only qcow2' in fc8_quick to raw and getting no output from > kvm_config.py. Turns out, that 'raw' has a smp2 requirement ... that > sort of filtering could be displayed with debugging output making > configuration changes easier. I agree, we need to add some debug messages to kvm_config.py. The smp - raw connection, of course, should not have left in the configuration file. It was just us testing the code. > - documentation of keywords and structure would be nice, explaining > what -variant , only and @ are doing for you, etc. Please read http://kvm.qumranet.com/kvmwiki/RegressionTests/ConfigFile2 > - it seems like the definition and rules ought to be separate from the > last section which defines which tests to run (the fc8_quick area), so > adding something as simple as include support to kvm_config.py would > be sufficient to support a common definition file but different > testing rules. An include support is one way to do it. We thought of a different way, which is to add rules from the control file. So the control file would pick the test-list it wants to run. Your suggestion is simpler but you need both a config file and a control file to change the test-list. We need to change only the control file. > > - kvm_runtest_2 as mentioned doesn't mess with your host networking and > relies on -net user and redir, would be good to plumb through -net tap > support that can be configured instead of always using -net user We want to add -net tap support, as that is what users usually use. kvm_runtest does exactly that (a part of kvm_host.cfg). The drawbacks of tap are (among others): - One must run tests as root, or play with sudo/chmod (True for /dev/kvm, but simpler) - You have to a have a dhcpd around. kvm_runtest by default runs a local dhcpd to serve kvm guests (part of setup/cleanup tests). - A bit more difficult to configure. > > - make -vnc parameter config/optional Agreed > > I noticed the references to the setup isos for windows that presumbly > install cygwin telnetd/sshd, are those available? if the isos > themselves aren't, if the build instructions are, that would be very > useful. You are right. We do have an installation iso images for telnetd/sshd. I did not want to commit iso images. Also, I am not sure about licensing, and I prefer that we would generate them on the user machine. We'll add the build instructions to the wiki. > > - guest install wizard using md5sum region matching ... ouch. This is > quite fickle. I've seen different kvms generate different md5sum for > the same region a couple of times. I know distributing screenshots of > certain OSes is a grey area, but it would be nice to plumb through > screenshot comparison and make that configurable. FWIW, I'll probably > look at pulling the screenshot comparison bits from kvmtest and getting > that integrated in kvm_runtest_2. Creating a step file is not as easy as it seems, exactly for that reason. One has to pick a part of the screenshot that only available when input is expected and that would be consistent. We were thinking of replacing the md5sum with a tiny compressed image of the part of the image that was picked. We had two other implementation for guest-wizard, one which only compares two/three consecutive screendumps (problems with e.g. clock ticking), and one similar to kvmtest. The kvmtest way is to let the user create his/her own screendumps to be used later. I did not want to add so many screendumps images to the repository. Step-Maker keeps the images it uses, so we can compare them upon failure. Step-Editor lets the user to change a single barrier_2 step (or more) by looking at the original image and picking a different area. > > > - kvm_runtest_2 looks a lot more like a regular autotest test, which is > a Good Thing(TM). There are still some things that would prevent it > going upstream autotest (which I assume is the long term goal) You assume correctly. > - a lot of the ssh and scp work to copy autotest client into a guest > is already handled by autoserv That is true, but we want to be able to run it as client test too. That way a user does not have to install the server to run kvm tests on his/her machine. > - vm.py has a lot of infrastructure that should be integrated into > autotest/server/kvm.py or possibly client-side common code to support > the next comment In the long term, there should be a client-server shared directory that deals with kvm guests (letting the host-client be the server for kvm-guests clients) > - kvm_tests.py defines new tests as functions, each of these tests > should be a separate client tests which sounds like a pain, but > should allow for easier test composition and hopefully make it easier > to add new tests that look like any other client side test with just > the implementation.py and control file > - this model moves toward eliminating kvm_runtest_2 and having a > server-side generate a set of tests to run and spawning those on a > target system. I am not sure that I follow. Why implementing a test as a function is a pain ? The plan is to keep kvm_runtest_2 in the client side, but move the configuration file + parser to the server (if one wants to dispatch test from the server). The server can dispatch the client test (using kvm_runtest_2 "test" + dictionary + tag). We have dependencies and we can spread unrelated kvm tests among similar hosts (e.g install guest OS and run tests for Fedora-10 64 bit on one machine, and install guest OS and run tests for Windows-XP 32 bit on another). We may have to add hosts into the configuration file, or add it while running (from the control file ?). We sure do not want to go back to the way kvm_runtest works (playing with symbolic links so that autotest would find and run the test), and we do not want too many kvm_test_NAME directories under client/tests/ . > > I do still like the idea of having a client-side test that can just > run on a developer/user's system to produce results without having to > configure all of the autotest server-side bits. Me too :-) Thanks for all the comments and suggestions, Uri.