Re: kvm-autotest -- introducing kvm_runtest_2

[Ryan Harper <ryanh@us.ibm.com>]

* Michael Goldish <mgoldish@redhat.com> [2009-03-12 02:26]:
> 
> > > Regarding the stepfiles you created for Linux -- I can't help much
> > > with those since I don't have the data. I do believe that if I had
> > the
> > > data and the stepfiles I could quickly identify the problem, so if
> > you
> > > think those can be sent to us, I'd like to have them.
> > 
> > I created a stepfile for RHEL5 and what I'm seeing is that one of the
> > screens I captured in stepmaker ended up having a focus ring around
> > something and on replay the focus isn't there.  This situation isn't
> > something that a new algo will fix as you pointed out.  I'm wondering
> > if
> > this is something you've seen.  I don't quite understand how it would
> > happen since stepmaker and the replace send the same keystrokes.  I
> > also
> > don't see how in general this can be avoided.
> 
> The problem sounds familiar. Does the ring appear around one of the
> GNOME menubars, i.e. around "Applications" or "System"? GNOME seems to
> be somewhat indeterministic with those rings. If you run the stepfile
> several times, you'll notice that in most cases you'll see a focus
> ring (or no focus ring, I don't quite remember) and the rest of the
> time you'll get the other case.

Ding Ding Ding! =)

> 
> This can be avoided either with experience, or a good wiki entry on
> picking the right barriers (which we plan to create). But you don't
> have to avoid making mistakes with stepmaker -- most types of mistakes
> are fixed very quickly and easily with stepeditor.

yep, used stepeditor to fix; defintely worth documenting where one
should be invoking stepeditor -- from the steps dir; if you don't run it
from there, it won't find the steps_data dir =(

> 
> The fix depends on exactly what you were trying to do:
> 
> - If you sent "alt-f1" to open the menu, and in the following step
> picked the open menu (including the "Applications" caption itself) to
> make sure it was open -- use stepeditor to modify the barrier so that
> it doesn't include the "Applications" caption or anything that might
> have a ring around it.

That worked for me.


> 
> The following text was copied from your previous e-mail:
> 
> > I do have the debug dir data from these runs.  Looking at the cropped
> > ppm and screendump ppm is how I determined that there must be
> > something
> > wrong with how the image is rendered since the cropped ppm matches
> > the
> > screendump output, but with whatever subtle difference that generates
> > a
> > different md5sum.
> 
> I'm not sure my previous e-mail was clear enough, so just in case it
> wasn't, let me rephrase: The cropped ppm is generated from the
> screendump ppm every time the stepfile running module receives a
> screendump from the guest in order to see if it matches a barrier.
> This is done for debugging purposes. If you somehow check, you'll see
> there is no subtle difference between those two files. It wouldn't
> make sense to find a subtle difference between them, and if you did
> find one, it certainly wouldn't indicate a stepfile problem, but
> rather a very strange bug in the framework.  You should be looking for
> subtle differences between the screendump ppm and the reference
> screendump ppm, as well as between the cropped screendump ppm and the
> reference cropped screendump ppm. By "reference" I mean coming from
> the stepmaker data. If you don't have the stepmaker data, you have no
> way of knowing what caused the difference in the md5sums.

Right -- the real win was comparing the full screendump to the reference
screendump - basically, without the reference dumps, the debug output
isn't useful.

I'll have to go back and re-read your email on where to put the
reference ppm files so one gets the refrence comparision.

> 
> 
> There are two other things I forgot to mention in my previous e-mail:
> 
> The Windows failures you're seeing might be caused by KVM bugs other
> than the one I mentioned. KVM-84 has a very strong tendency to crash
> during Windows installations. You can use the logs to find out if that
> happened in your case. If you have the latest git HEAD the exception
> info will look something like "Barrier timed out at step ... (VM is
> dead)", and if you have a slightly older version, you'll probably see
> "(guest is stuck)" at the end of the info string. You should also see
> the system consistently complaining that it can't fetch any
> screendumps from the guest (this will appear in stdout).

I've seen those on kvm-84.

> The other thing has to do with the ISO files. kvm_runtest has a very
> important feature that we innocently forgot to implement in
> kvm_runtest_2 -- md5sum verification of the ISO files. This means that
> the framework currently makes no use of the "md5sum" and "md5sum_1m"
> parameters in the config file. This means you might be using different
> ISOs than the ones we made our stepfiles with. In that case I wouldn't
> expect any stepfile to succeed. However, if you used these same ISOs
> with kvm_runtest then they should be fine. In any case, I'll add the
> feature ASAP to the git repository.

Right - I suppose it might be better if the names of the windows iso
disks matched how MS names them in MSDN, for example, kvm_runtest refers
to  Windows2008-x64.iso  which doesn't match any name from MSDN, what we
have is:
en_windows_server_2008_datacenter_enterprise_standard_x64_dvd_X14-26714.iso

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com