regression testing

regression testing

[nbecker]

Hi.  I was wondering if there has been any discussion of kernel
regression testing.  Wouldn't it be great if we didn't have to depend
on human testers to verify every change didn't break something?

OK, I'll admit I haven't given this a lot of thought.  What I'm
wondering is whether the user-mode linux could help here (allow a way
to simulate controlled activity).

Re: regression testing

[Richard B. Johnson]

On 22 Mar 2001 nbecker@fred.net wrote:

> Hi.  I was wondering if there has been any discussion of kernel
> regression testing.  Wouldn't it be great if we didn't have to depend
> on human testers to verify every change didn't break something?
> 
> OK, I'll admit I haven't given this a lot of thought.  What I'm
> wondering is whether the user-mode linux could help here (allow a way
> to simulate controlled activity).
> -

Regression testing __is__ what happens when 10,000 testers independently
try to break the software!

Canned so-called "regression-test" schemes will fail to test at least
90 percent of the code paths, while attempting to "test" 100 percent
of the code!


Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of
course you get it all back when you reboot..."; Actual explanation
obtained from the Micro$oft help desk.

Re: regression testing

[nbecker]

>>>>> "Richard" == Richard B Johnson <root@chaos.analogic.com> writes:

    Richard> On 22 Mar 2001 nbecker@fred.net wrote:
    >> Hi.  I was wondering if there has been any discussion of kernel
    >> regression testing.  Wouldn't it be great if we didn't have to depend
    >> on human testers to verify every change didn't break something?
    >> 
    >> OK, I'll admit I haven't given this a lot of thought.  What I'm
    >> wondering is whether the user-mode linux could help here (allow a way
    >> to simulate controlled activity).
    >> -

The problem I see, based on reading this list, is that the testing is
not so thorough as you say.  First of all, do we really know how many
people test the pre-relase kernels?  How about which features they
actually test?

In principle, we could find out how much test coverage there is.

The problem with an army of testers, is that after an individual has
gone through the build/install/screw-around-with-it process a few
hundred times, it starts to get a bit repetitive.  They start to get
sloppy and haphazard regarding what they test or whether they even try
out xxx-pre<put_big_number_here> at all.  If we could automate at
least some basic testing that would be a big win.

Re: regression testing

[Alan Cox]

> Regression testing __is__ what happens when 10,000 testers independently
> try to break the software!

Nope. Thats stress testing and a limited amount of coverage testing.

> Canned so-called "regression-test" schemes will fail to test at least
> 90 percent of the code paths, while attempting to "test" 100 percent
> of the code!

Then they are not well designed. Tools like gprof and the kernel profiling
will let you measure code path coverage of a test series

Re: regression testing

[Richard B. Johnson]

On Thu, 22 Mar 2001, Alan Cox wrote:

> > Regression testing __is__ what happens when 10,000 testers independently
> > try to break the software!
> 
> Nope. Thats stress testing and a limited amount of coverage testing.
> 
> > Canned so-called "regression-test" schemes will fail to test at least
> > 90 percent of the code paths, while attempting to "test" 100 percent
> > of the code!
> 
> Then they are not well designed. Tools like gprof and the kernel profiling
> will let you measure code path coverage of a test series
> 

Alan's back! 

Cheers,
Dick Johnson

Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).

"Memory is like gasoline. You use it up when you are running. Of
course you get it all back when you reboot..."; Actual explanation
obtained from the Micro$oft help desk.

Re: regression testing

[Cort Dougan]

} On 22 Mar 2001 nbecker@fred.net wrote:
} 
} > Hi.  I was wondering if there has been any discussion of kernel
} > regression testing.  Wouldn't it be great if we didn't have to depend
} > on human testers to verify every change didn't break something?
} > 
} > OK, I'll admit I haven't given this a lot of thought.  What I'm
} > wondering is whether the user-mode linux could help here (allow a way
} > to simulate controlled activity).
} > -
} 
} Regression testing __is__ what happens when 10,000 testers independently
} try to break the software!

No, in fact that is not a regression test.

} Canned so-called "regression-test" schemes will fail to test at least
} 90 percent of the code paths, while attempting to "test" 100 percent
} of the code!

A canned set of regression tests would actually do what they're supposed to
- prevent the kernel from regressing.  If you fix a bug - write a test for
that bug and keep running it.  Something we follow for RTLinux that has
helped us immensely.

Re: regression testing

[Wade Hampton]

nbecker@fred.net wrote:
> 
> Hi.  I was wondering if there has been any discussion of kernel
> regression testing.  Wouldn't it be great if we didn't have to depend
> on human testers to verify every change didn't break something?
IMHO, much of the strength of Linux is the very large, extremely 
diverse population of folks using it, testing it, beating on 
the latest release, etc.  

However, a lab dedicated to testing the linux kernel, properly 
funded, staffed, and containing the most common hardware and 
software would be a good idea.  Does anyone have any idea how
this could be accomplished?  Who could do it?  IBM?  What would
it cost to setup a reasonable lab?  My guess would be dozens 
of machines of various architectures, a staff of at least 10,
several thousand square feet of space, and a good budget....
Any takers?  

Much of the kernel COULD be tested such as file systems, network 
stack, SMP, compile options on various platforms, etc.  More
obscure hardware, some older hardware, etc., would be out of 
scope for such an effort.

Cheers,
-- 
W. Wade, Hampton  <whampton@staffnet.com>  
If Microsoft Built Cars:  Every time they repainted the 
lines on the road, you'd have to buy a new car.
Occasionally your car would just die for no reason, and 
you'd have to restart it, but you'd just accept this.

Re: regression testing

[Nathan Straz]

On Thu, Mar 22, 2001 at 10:13:04AM -0500, Wade Hampton wrote:
> nbecker@fred.net wrote:
> > Hi.  I was wondering if there has been any discussion of kernel
> > regression testing.  Wouldn't it be great if we didn't have to depend
> > on human testers to verify every change didn't break something?
> IMHO, much of the strength of Linux is the very large, extremely 
> diverse population of folks using it, testing it, beating on 
> the latest release, etc.  

The Linux community definately provides the configuration testing that
could never be done in a lab setting.  At least one that most companies
could afford.  

> However, a lab dedicated to testing the linux kernel, properly 
> funded, staffed, and containing the most common hardware and 
> software would be a good idea.  Does anyone have any idea how
> this could be accomplished?  Who could do it?  IBM?  What would
> it cost to setup a reasonable lab?  My guess would be dozens 
> of machines of various architectures, a staff of at least 10,
> several thousand square feet of space, and a good budget....
> Any takers?  

SGI is working on regression testing for Linux.  We have released some
of our tests and utilities under the Linux Test Project.  IMHO, a few
hundred tests aren't enough.  I need to make another big push with the
tests I've ported over the last few months.  

I believe there is some work in the Open Source Development Lab that
some IBMers could comment on.  I don't know if there is a web site
detailing their efforts yet.  

-- 
Nate Straz                                              nstraz@sgi.com
sgi, inc                                           http://www.sgi.com/
Linux Test Project                    http://oss.sgi.com/projects/ltp/

Re: regression testing

[Nathan Dabney]

On Thu, Mar 22, 2001 at 09:56:16AM -0600, Nathan Straz wrote:
> On Thu, Mar 22, 2001 at 10:13:04AM -0500, Wade Hampton wrote:
> > However, a lab dedicated to testing the linux kernel, properly 
> > funded, staffed, and containing the most common hardware and 
> > software would be a good idea.  Does anyone have any idea how
> > this could be accomplished?  Who could do it?  IBM?  What would
> > it cost to setup a reasonable lab?  My guess would be dozens 
> > of machines of various architectures, a staff of at least 10,
> > several thousand square feet of space, and a good budget....
> > Any takers?  
> 
> SGI is working on regression testing for Linux.  We have released some
> of our tests and utilities under the Linux Test Project.  IMHO, a few
> hundred tests aren't enough.  I need to make another big push with the
> tests I've ported over the last few months.  
> 
> I believe there is some work in the Open Source Development Lab that
> some IBMers could comment on.  I don't know if there is a web site
> detailing their efforts yet.  

www.osdlab.org is the closest you will get right now.

Since we have only recently opened we have been working to get the machines 
configured and ready for beating.  We have only recently started opening up the 
larger servers to developers and testers.

Any regression/scalability development and testing people want to do should be
workable.  Anyone interested in coordinating development or testing at our lab
can contact me for info.

-Nathan Dabney
Open Source Development Lab

Re: regression testing

[Rik van Riel]

On 22 Mar 2001 nbecker@fred.net wrote:

> Hi.  I was wondering if there has been any discussion of kernel
> regression testing.  Wouldn't it be great if we didn't have to depend
> on human testers to verify every change didn't break something?

This is definately a great idea.  A relatively easy start should
be testing things like:
- automated crashme runs
- automated heavy stress testing
- automated compilation of the kernel with random config
  options (done by Arjan v/d Ven?)
- automated testing of certain kernel behaviour (didn't
  SGI have a project to look at this?  could they use help?)
- ... ?

Note that some of these testing options are almost trivial to
set up and could be run by any sysadmin with a few spare/test
machines. It would be great if testing like this could be done
on the Linux kernel.

If there isn't already some place to coordinate this testing,
I'll be able to host everything needed on NL.linux.org.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: regression testing

[Cort Dougan]

We have a start for PPC.  It has the title "Regression Tester" but is
actually a "compiles and boots tester".  The aim is a automated regression
test.

Take a look at http://altus.drgw.net/

It pulls directly from our BitKeeper archive every time we push a change
and goes through the build targeted for a number of platforms.

} - automated compilation of the kernel with random config
}   options (done by Arjan v/d Ven?)
} - automated testing of certain kernel behaviour (didn't
}   SGI have a project to look at this?  could they use help?)
} - ... ?

Re: regression testing

[Jonathan Morton]

>- automated heavy stress testing

This would be an interesting one to me, from a benchmarking POV.  I'd like
to know what my hardware can really do, for one thing - it's all very well
saying this box can do X Whetstones and has a 100Mbit NIC, but it's a much
more solid thing to be able to say "my box handled the official Foobar
stress-test in Y hours, handling Z widgets per second".

The lmbench statistics are nice, but most of those numbers mean absolutely
nothing to all but übergeeks (what they say to my partially-trained ears is
that my gateway box sucks, but then again it's five-year-old Intel hardware
so that could probably have been predicted).  Are there already
stress-testing kits for user-level processes I could play with?

What would be *really* interesting would be to make such tests as portable
as practical, so that at least some of them can be run on "rival platforms"
(such as *cough* Redmondware) to see if they stand up to the challenge as
well.

However, if the focus is on *kernel* stress-testing, portability might be a
little difficult to arrange for many tests.  The basics could probably be
written portably or ported easily, though - things that lmbench already
(briefly) benchmarks, for example.

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi@cyberspace.org  (not for attachments)
big-mail: chromatix@penguinpowered.com
uni-mail: j.d.morton@lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*)
-----END GEEK CODE BLOCK-----

Re: regression testing

[Horst von Brand]

Jonathan Morton <chromi@cyberspace.org> said:
> >- automated heavy stress testing

> This would be an interesting one to me, from a benchmarking POV.  I'd like
> to know what my hardware can really do, for one thing - it's all very well
> saying this box can do X Whetstones and has a 100Mbit NIC, but it's a much
> more solid thing to be able to say "my box handled the official Foobar
> stress-test in Y hours, handling Z widgets per second".

Which would tell you exactly that, nothing more and nothing less. No idea
how many users it would support reading mail via IMAP, if playing Quake on
it is doable, or whether it is a good idea to use it as a firewall or
development machine. Computer systems are very complex, their uses much
more so (and extremely varied to boot); to believe some number (or even a
small set of numbers) will tell you everything there is to know about a
particular box is just an illusion. And a box rarely stands in isolation.
-- 
Dr. Horst H. von Brand                       mailto:vonbrand@inf.utfsm.cl
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: regression testing

[Rik van Riel]

On Fri, 23 Mar 2001, Horst von Brand wrote:
> Jonathan Morton <chromi@cyberspace.org> said:
> > >- automated heavy stress testing
> 
> > This would be an interesting one to me, from a benchmarking POV.  I'd like
> > to know what my hardware can really do, for one thing - it's all very well
> > saying this box can do X Whetstones and has a 100Mbit NIC, but it's a much
> > more solid thing to be able to say "my box handled the official Foobar
> > stress-test in Y hours, handling Z widgets per second".
> 
> Which would tell you exactly that, nothing more and nothing less.

But if we get 500 people to test _different_ things on a regular
basis, we might be able to get more meaningful results.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

		http://www.surriel.com/
http://www.conectiva.com/	http://distro.conectiva.com.br/

Re: regression testing

[Eric W. Biederman]

nbecker@fred.net writes:

> Hi.  I was wondering if there has been any discussion of kernel
> regression testing.  Wouldn't it be great if we didn't have to depend
> on human testers to verify every change didn't break something?

There is a some truth to this.  However for kernel development there
is one thing to keep in mind: most bugs are in the drivers (buggy
hardware with a software workaround counts as a driver bug).  Having
an army of human testers with weird machines with all kinds of
drivers is the only economical way of doing driver regression testing.

> OK, I'll admit I haven't given this a lot of thought.  What I'm
> wondering is whether the user-mode linux could help here (allow a way
> to simulate controlled activity).

The most devastating bugs are in the core kernel code, and a
regression test for that code is more likely.  Yes user-mode linux
could help here (you could stress test the core kernel without worry
that when it crashes your machine will crash as well).  

Additionally a good test suite that give the kernel a good going over
in a manner that exercises standard kinds of hardware could also help.  

Unless you are using profiling to verify you have covered all paths
in the code there will always be question such as does that code path
work in practice.

The other thing that can help a lot looks like the augmented static
analysis of the kernel.  Once that is generally available we should
be able to verify at compile time that a lot of little rules are
always obeyed.  It will never get the really hard things but the
compiler pointing out that there is an sti on all paths after a cli
could improve things significantly (especially with the drivers).

Eric

Re: regression testing

[Werner Almesberger]

Eric W. Biederman wrote:
> Yes user-mode linux
> could help here (you could stress test the core kernel without worry
> that when it crashes your machine will crash as well).  

A similar approach can be used for very detailed tests of specific
subsystems. E.g. that's what we've started doing, kind of as a by-product
of some other work, with tcsim, which runs most of the traffic control
subsystem in user space. (ftp://icaftp.epfl.ch/pub/linux/tcng/)

The advantage over using UML is that we have a much simpler environment,
allowing for more straightforward integration, and we can exercise tight
control over infrastructure parts like timers or network events. Once
UML is part of the mainstream kernel, it would be interesting to try to
obtain the same functionality with UML too.

Some added goodies include the ability to run kernel code with malloc
debuggers like Electric Fence, which has already helped to find a few
real bugs. (Does EFence work with UML ?)

- Werner

-- 
  _________________________________________________________________________
 / Werner Almesberger, ICA, EPFL, CH           Werner.Almesberger@epfl.ch /
/_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/

RE: regression testing

[Torrey Hoffman]

Rik van Riel wrote:

>This is definately a great idea. 
[...]
>Note that some of these testing options are almost trivial to
>set up 
[...]

If an effort in this direction produced a "kernel-tester.tar.gz"
package, I'm sure lots of people with spare hardware would install 
it, and then check it once a day for oops or crashes, and mail in 
the bug reports....

I have the spare hardware, I just don't have the time to constantly
be downloading, compiling, and testing.  I'm sure I'm not unique in 
that respect.  

An automated system would probably bring thousands of new machines 
on board for continuous testing, keeping it a community effort. 

Not that the testing work of IBM, SGI, Red Hat, et. al. is 
unappreciated, but a distributed, open, regression testing package
sort of fits the Linux philosophy and model...

Torrey Hoffman