Re: [PATCH] KVM: tests_base.cfg: major guest OS cleanup

[Marcelo Tosatti <mtosatti@redhat.com>]

On Sun, Feb 07, 2010 at 09:26:21PM -0200, Lucas Meneghel Rodrigues wrote:
> On Sun, 2010-02-07 at 13:24 -0500, Michael Goldish wrote:
> 
> Let me explain the reasoning behind this:
> 
> > Looks like you dropped all step file tests, and I don't think
> > this is a good idea.  Keeping them is harmless, so I don't think
> > they should be dropped unless they're completely useless.
> > If you don't want them to run, keep them out of the sample test
> > sets, but by removing them entirely from the sample config file,
> > you're making it much harder for those who use them to keep
> > using them.
> 
> Good point, I didn't mean to do it just for the sake of not having them
> anymore:
>  * In the vast majority of the cases, I have downloaded and tested the
> latest ISOS from the OS vendors, and tested it with the unattended
> files. I do admit that I didn't try the step files, but I assumed they
> wouldn't work so that's why I dropped them.
> 
> * Removing older versions of distributions like Fedora have good reason:
> The support cycle of them is very short (18 months), by their very
> nature (Fedora is a technology preview, we are allways moving on). 
> So it's not reasonable to keep testing guest OS that might have bugs
> that no one will care to fix.

Disagree. Testing of older guests can trigger KVM bugs which are not
seen during testing of newer ones. 

> * As for the longer living versions of OS, like windows and RHEL,
> there's a clear version for using the latest versions as well: The OS
> vendors usually provide the updates (I mean inside a same family of that
> OS, like RHEL 3.X, or windows2008) and they strongly advise System
> Adminsitrators to allways use the latest version when installing new
> systems.
> 
> In other words, let's say we find a bug on RHEL 5.3. The very first
> thing we'll be asked to is to try reproducing the problem on 5.4, and if
> it doesn't, the bug will be closed (same if you are using windows
> [version] SP X, they will allways ask you to try the latest SP).

Not really. If the test succeeded before, and it regresses after qemu or 
KVM changes, its a bug we should care about.

> So IMHO
> it's important to focus our testing resources on the versions of the OS
> that will be used on new installs by the general IT public. So I looked
> after the latest ISOS that the vendors had to offer and tested only with
> step install, since I didn't have to change the unattended files for
> doing so.
> 
> > Of course, if we become quite confident that no one will ever want
> > to use step files again, getting rid of them isn't bad.  I do
> > however think that some people will keep using them, and I even
> > think it's a good idea to run a quick step file test as part of
> > our daily routine on autotest.virt.bos, just to make sure KVM
> > displays stuff properly and responds to keyboard input.
> 
> Yes, I do think step installs are important:
> 
>  * Good method to verify input and image display, like you just
> mentioned;
>  * They are truly a universal way to get OSs installed. So if the given
> OS has no reasonable unattended install system, we have step file based
> install ready to rescue us.
> 
> So in terms of maintance, we are better of keeping unattended installs
> for the OSs we know how to do it, but keep up to date step files for the
> guest OSs that we can't use with unattended. Like, bump the versions for
> all the OSs that we can't do unattended version, and have up to date
> step files for those.

Why don't keep both? 

> > Regarding reorganizing duplicate information:
> > You don't need to get rid of step file tests in order to avoid
> > config code duplication.  You can reuse information like this:
> > 
> > - 11.32:
> >     image_name = fc11-32
> >     ...
> >     (common stuff)
> >     ...
> >     install:
> >         steps = Fedora-11-32.steps
> >         (install specific stuff)
> >     unattended_install:
> >         unattended_file = unattended/something
> >         (unattended_install specific stuff)
> 
> Sure, most of step file removals were due to the fact that I introduced
> new ISOs, like I explained before.
> 
> > Regarding version bumping:
> > Since bumping guest OS versions breaks step file installs, I think
> > we should either keep the old versions next to the new ones in the
> > same config file, and exclude them from sample test sets, or keep
> > them in a separate config file, e.g. tests_base_old.cfg.sample
> > (or tests_base.cfg.sample.old?).
> > In any case, I don't think we should remove the old versions.
> > This also applies to the old Fedoras.
> 
> We could keep them, but like I've said, we should focus our testing
> resources using the latest versions provided by the OS vendors.

Agree, but please keep the old step file method around in the reposity
for all guests. Not only they can be useful if problems arise with
unattended install, but the screen comparison method that is used can
find bugs that unattended install cannot (issues that involves drawing
the screen).