Re: kvm-autotest -- introducing kvm_runtest_2

[Ryan Harper <ryanh@us.ibm.com>]

* Michael Goldish <mgoldish@redhat.com> [2009-03-10 20:55]:
> 
> ----- "Ryan Harper" <ryanh@us.ibm.com> wrote:
> 
> > > >- guest install wizard using md5sum region matching ... ouch.  This
> > is
> > > >quite fickle.  I've seen different kvms generate different md5sum
> > for
> > > >the same region a couple of times.  I know distributing screenshots
> > of
> > > >certain OSes is a grey area, but it would be nice to plumb through
> > > >screenshot comparison and make that configurable.  FWIW, I'll
> > probably
> > > >look at pulling the screenshot comparison bits from kvmtest and
> > getting
> > > >that integrated in kvm_runtest_2.
> > > Creating a step file is not as easy as it seems, exactly for that
> > reason. 
> > > One has to pick a part of the screenshot that only available when
> > input is 
> > > expected and that would be consistent. We were thinking of replacing
> > the 
> > > md5sum with a tiny compressed image of the part of the image that
> > was 
> > > picked.
> > 
> > It isn't just that step file creation isn't easy is that even with a
> > good stepfile with smart region boxes, md5sum can *still* fail
> > because
> > KVM renders one pixel in the region differently than the system where
> > the
> > original was created; this creates false positives failures.
> 
> I'd like to comment on this. I don't doubt that some fuzzy matching
> algorithm (such as calculating match percentages) would generally be
> more robust. I do however doubt it would significantly lower the false
> positive rate in our case (which is fairly low already). False
> positive failures in step files are typically caused by:

I've seen multiple failures during the windows guest installs which I
assume are well tested stepfiles.  For example, 2k8 installs and the
fails to pass the barrier when trying to set the user password for the
first time.  The cropped region *looks* exactly like the the intended
location on the screendump, but md5sums to something different. 

A recent run of 2k3 and 2k8 installs resulted in the following failures:

Win2k3-32bit -- screenshot of "Windows Setup" and Setup is starting
windows, cropped region is of "Setup is starting Windows" full screen
dump matches this text from a human pov

Win2k3-64-bit -- same as above

Win2k8-32-bit -- screenshot of "The user's password must be changed
before logging in the first time with OK and cancel buttons.  - cropped
region is of the text "The user's password must be changed before
logging in the first time" - matching the full screen screendump fine
from a human POV

Win2k8-64-bit -- same as above

We've also been creating stepfiles for Linux guests as well that aren't
here, various SLES and RHEL installs -- and I've repeatedly seen the
same issue where the cropped region *should* match but isn't, and it
isn't a result of any of the very correct reasons you've listed below as
to why the stepfiles might fail.

> 
> - an unexpected popup window covering the test region
> - a dialog which has a different position every time (and varies by
>   many pixels)
> - a barrier that passes before the controls get input focus, which
>   causes the following keystrokes to have no effect
> - in text mode, sometimes a line of text is printed unexpectedly and
>   causes the entire screen to scroll up
> - addition/modification of a KVM feature which changes the course of
>   the installation

> 
> I may have left something out. In any case, all these problems are
> solved by picking better barrier regions, but none can be solved by
> using a more forgiving comparison method. I have encountered a single
> installation that rendered a single pixel in an indeterministic
> fashion, and though this problem was easily solved by correcting the
> barrier (using a stepfile editor), I do agree we might get a small
> decrease in the false positive rate if we use a more forgiving
> algorithm.

Well, either there is a *bug* right now that is triggering a higher rate
of false positives, or using a better algorithm is a requirement;
distributing stepfiles and md5sums that don't work isn't productive, so
in the case that it is a bug I still suggest we pursue a more resilient
algorithm.

> 
> However, there is also a risk: a more forgiving algorithm may forgive
> KVM for rendering errors. It may also make it risky to pick barriers
> that are meant to catch small text; I believe a button with a "Yes"
> caption and a button with a "No" caption would have a very high match
> percentage, especially if you have to pick the whole button, or maybe
> even some of its surroundings (and you often do).

Noted, though I think as you indicated above, smart selection of the
cropped region goes a long way toward avoiding these sorts of
collisions.

> 
> I still believe it's a good idea to look into other methods (we're
> already doing that) and start experimenting with them.

Cool.  Obviously without the original ppm files from the stepmaker run,
we can't determine if a different algo would help so we're generating
new stepfiles and ppm data and trying to reproduce the md5sum mismatch
issues.  If there is anything I can do to help with the algo work let me
know.

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
ryanh@us.ibm.com