Re: regression testing

[ebiederm@xmission.com (Eric W. Biederman)]

nbecker@fred.net writes:

> Hi.  I was wondering if there has been any discussion of kernel
> regression testing.  Wouldn't it be great if we didn't have to depend
> on human testers to verify every change didn't break something?

There is a some truth to this.  However for kernel development there
is one thing to keep in mind: most bugs are in the drivers (buggy
hardware with a software workaround counts as a driver bug).  Having
an army of human testers with weird machines with all kinds of
drivers is the only economical way of doing driver regression testing.

> OK, I'll admit I haven't given this a lot of thought.  What I'm
> wondering is whether the user-mode linux could help here (allow a way
> to simulate controlled activity).

The most devastating bugs are in the core kernel code, and a
regression test for that code is more likely.  Yes user-mode linux
could help here (you could stress test the core kernel without worry
that when it crashes your machine will crash as well).  

Additionally a good test suite that give the kernel a good going over
in a manner that exercises standard kinds of hardware could also help.  

Unless you are using profiling to verify you have covered all paths
in the code there will always be question such as does that code path
work in practice.

The other thing that can help a lot looks like the augmented static
analysis of the kernel.  Once that is generally available we should
be able to verify at compile time that a lot of little rules are
always obeyed.  It will never get the really hard things but the
compiler pointing out that there is an sti on all paths after a cli
could improve things significantly (especially with the drivers).

Eric