Re: kvm-autotest -- introducing kvm_runtest_2

[Ryan Harper <ryanh@us.ibm.com>]

* Michael Goldish <mgoldish@redhat.com> [2009-03-11 03:08]:
> > > I'd like to comment on this. I don't doubt that some fuzzy matching
> > > algorithm (such as calculating match percentages) would generally
> > be
> > > more robust. I do however doubt it would significantly lower the
> > false
> > > positive rate in our case (which is fairly low already). False
> > > positive failures in step files are typically caused by:
> > 
> > I've seen multiple failures during the windows guest installs which I
> > assume are well tested stepfiles.  For example, 2k8 installs and the
> > fails to pass the barrier when trying to set the user password for
> > the
> > first time.  The cropped region *looks* exactly like the the intended
> > location on the screendump, but md5sums to something different. 
> > 
> > A recent run of 2k3 and 2k8 installs resulted in the following
> > failures:
> > 
> > Win2k3-32bit -- screenshot of "Windows Setup" and Setup is starting
> > windows, cropped region is of "Setup is starting Windows" full screen
> > dump matches this text from a human pov
> > 
> > Win2k3-64-bit -- same as above
> > 
> > Win2k8-32-bit -- screenshot of "The user's password must be changed
> > before logging in the first time with OK and cancel buttons.  -
> > cropped
> > region is of the text "The user's password must be changed before
> > logging in the first time" - matching the full screen screendump fine
> > from a human POV
> > 
> > Win2k8-64-bit -- same as above
> > 
> > We've also been creating stepfiles for Linux guests as well that
> > aren't
> > here, various SLES and RHEL installs -- and I've repeatedly seen the
> > same issue where the cropped region *should* match but isn't, and it
> > isn't a result of any of the very correct reasons you've listed below
> > as
> > to why the stepfiles might fail.
> 
> The Windows failures you're describing sound like they could be caused
> by a known KVM bug, which results in Windows installations sometimes
> booting from CDROM, instead of the HDD, immediately following the
> installation.

No, but I have seen a bug that after an install and guest OS reboots,
KVM fails to boot from the harddrive; exiting KVM and then booting from
HD works fine.  We're looking into that one right now.

> 
> I assume you don't have the stepmaker data of those Windows stepfiles.

These are the stepfiles that came with kvm-autotest so no *I* don't have
stepmaker data, but whomever committed the windows stepfiles to
kvm-autotest *should* have the data ... 

> In that case, the images left by the stepfile test, scrdump.ppm and
> cropped_scrdump.ppm, are in fact the full screendump and a cropped
> region in it. They should always match perfectly, because the cropped
> one is generated from the full one at runtime. None of them reflects
> the expected guest behavior; they reflect what the stepfile test
> actually found. The only thing you have that reflects the expected
> guest behavior is the md5sum found in the stepfile.
> 
> If you happened to keep the "debug" dirs which contain the screendumps
> and test logs, and could somehow send them to me or Uri, I'd be able
> to tell you what went wrong with the test and whether it is indeed
> that KVM bug or a stepfile error. We probably could also use the
> stepfiles you were working with, because we might have changed ours
> recently, though that is unlikely because we don't change old
> stepfiles very often nowadays.

I do have the debug dir data from these runs.  Looking at the cropped
ppm and screendump ppm is how I determined that there must be something
wrong with how the image is rendered since the cropped ppm matches the
screendump output, but with whatever subtle difference that generates a
different md5sum.

I'll see about figuring out how to get the debug output to you.


> 
> Regarding the stepfiles you created for Linux -- I can't help much
> with those since I don't have the data. I do believe that if I had the
> data and the stepfiles I could quickly identify the problem, so if you
> think those can be sent to us, I'd like to have them.

OK

> 
> I'm not sure exactly what version of kvm_runtest_2 you're using (are
> you are using kvm_runtest_2?), but I think it should support

Yep, kvm_runtest_2; but I've seen the same issue on kvm_runtest.

> automatic comparison of the actual screendump with the expected
> screendump. If you have a slightly older version than the current git
> HEAD, then you should probably place your <stepfile>_data directory
> right next to <stepfile>, and whenever a stepfile test fails you'll
> get -- in addition to scrdump.ppm and cropped_scrdump.ppm --
> scrdump_reference.ppm and cropped_scrdump_reference.ppm, as well as a
> nice green-red comparison image which colors all matching pixels green
> and all mismatching ones red. That last image is very helpful when
> stepfiles require fixing. If you have the latest git HEAD, you should
> place all your <stepfile>_data dirs in a dir named "steps_data" which
> should reside next to "steps" (which should contain the stepfiles
> themselves).

Very useful information, should be added to the wiki; we should write a
section on using stepmaker/stepeditor and best practices on picking
barriers/cropped regions.

> > Well, either there is a *bug* right now that is triggering a higher
> > rate
> > of false positives, or using a better algorithm is a requirement;
> > distributing stepfiles and md5sums that don't work isn't productive,
> > so
> > in the case that it is a bug I still suggest we pursue a more
> > resilient
> > algorithm.
> 
> Do the Windows tests you mentioned fail consistently, or have you
> witnessed any of them succeed in some of the runs?

Consistently fail, no passes so far.


-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com