Re: [PATCH] KVM: tests_base.cfg: major guest OS cleanup

[Lucas Meneghel Rodrigues <lmr@redhat.com>]

On Sun, 2010-02-07 at 13:24 -0500, Michael Goldish wrote:

Let me explain the reasoning behind this:

> Looks like you dropped all step file tests, and I don't think
> this is a good idea.  Keeping them is harmless, so I don't think
> they should be dropped unless they're completely useless.
> If you don't want them to run, keep them out of the sample test
> sets, but by removing them entirely from the sample config file,
> you're making it much harder for those who use them to keep
> using them.

Good point, I didn't mean to do it just for the sake of not having them
anymore:
 * In the vast majority of the cases, I have downloaded and tested the
latest ISOS from the OS vendors, and tested it with the unattended
files. I do admit that I didn't try the step files, but I assumed they
wouldn't work so that's why I dropped them.

* Removing older versions of distributions like Fedora have good reason:
The support cycle of them is very short (18 months), by their very
nature (Fedora is a technology preview, we are allways moving on). So
it's not reasonable to keep testing guest OS that might have bugs that
no one will care to fix.

* As for the longer living versions of OS, like windows and RHEL,
there's a clear version for using the latest versions as well: The OS
vendors usually provide the updates (I mean inside a same family of that
OS, like RHEL 3.X, or windows2008) and they strongly advise System
Adminsitrators to allways use the latest version when installing new
systems.

In other words, let's say we find a bug on RHEL 5.3. The very first
thing we'll be asked to is to try reproducing the problem on 5.4, and if
it doesn't, the bug will be closed (same if you are using windows
[version] SP X, they will allways ask you to try the latest SP). So IMHO
it's important to focus our testing resources on the versions of the OS
that will be used on new installs by the general IT public. So I looked
after the latest ISOS that the vendors had to offer and tested only with
step install, since I didn't have to change the unattended files for
doing so.

> Of course, if we become quite confident that no one will ever want
> to use step files again, getting rid of them isn't bad.  I do
> however think that some people will keep using them, and I even
> think it's a good idea to run a quick step file test as part of
> our daily routine on autotest.virt.bos, just to make sure KVM
> displays stuff properly and responds to keyboard input.

Yes, I do think step installs are important:

 * Good method to verify input and image display, like you just
mentioned;
 * They are truly a universal way to get OSs installed. So if the given
OS has no reasonable unattended install system, we have step file based
install ready to rescue us.

So in terms of maintance, we are better of keeping unattended installs
for the OSs we know how to do it, but keep up to date step files for the
guest OSs that we can't use with unattended. Like, bump the versions for
all the OSs that we can't do unattended version, and have up to date
step files for those.

> Regarding reorganizing duplicate information:
> You don't need to get rid of step file tests in order to avoid
> config code duplication.  You can reuse information like this:
> 
> - 11.32:
>     image_name = fc11-32
>     ...
>     (common stuff)
>     ...
>     install:
>         steps = Fedora-11-32.steps
>         (install specific stuff)
>     unattended_install:
>         unattended_file = unattended/something
>         (unattended_install specific stuff)

Sure, most of step file removals were due to the fact that I introduced
new ISOs, like I explained before.

> Regarding version bumping:
> Since bumping guest OS versions breaks step file installs, I think
> we should either keep the old versions next to the new ones in the
> same config file, and exclude them from sample test sets, or keep
> them in a separate config file, e.g. tests_base_old.cfg.sample
> (or tests_base.cfg.sample.old?).
> In any case, I don't think we should remove the old versions.
> This also applies to the old Fedoras.

We could keep them, but like I've said, we should focus our testing
resources using the latest versions provided by the OS vendors.