Testing: What do you want?

Testing: What do you want?

[Scott Robert Ladd]

OSNews and LinuxJournal are urging people to test the Linux kernel. I've 
been doing so for several months now, running the latest kernel live on 
my primary development system. I figure the best way to know if the 
kernel works is to use it when I'm working. ;)

At the moment, my biggest contribution to kernel development is testing, 
given that I'm not yet comfortable hacking kernel code in public. My 
question is: What do the kernel developers want from testers? What sort 
of reports are helpful? Is there anything in particular that needs 
extensive testing?

I haven't found a "Guide to Testing the Kernel" anywhere -- perhaps such 
a document exists, but I've missed it. Such a document could provide 
basic information for "testing newbies."

My experience thus far:
For the most part, the 2.5 series has worked very well for me, albeit 
with a few glitches (radeonfb, for example, as reported last week.) I'll 
build the 2.5.65 kernel on my Sparc later today, and see how well it 
works there.

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Professional programming for science and engineering;
Interesting and unusual bits of very free code.

Re: Testing: What do you want?

[Jan-Benedict Glaw]

[-- Attachment #1: Type: text/plain, Size: 3534 bytes --]

On Mon, 2003-03-24 09:46:05 -0500, Scott Robert Ladd <coyote@coyotegulch.com>
wrote in message <3E7F1A2D.4050306@coyotegulch.com>:
> For the most part, the 2.5 series has worked very well for me, albeit 
> with a few glitches (radeonfb, for example, as reported last week.) I'll 
> build the 2.5.65 kernel on my Sparc later today, and see how well it 
> works there.

sparc32? If you get it to build or even to boot, please drop me a note
with at least this information I'm *really* interested in:

	- Machine type
	- CPU(s)
	- .config file
	- gcc -v
	- ld -v

Last time I looked at it, sparc32 wasn't in any good state (esp. SMP) in
2.5.x. This is because Dave S. Miller stopped spending a lot of hacking
time (he has to work for other things now and only merges patches he
gets sent, where he formerly did tons on active development for
sparc32).

I'm in the progress of a (private) attempt to build a Linux Test Centre.
(I've already mentioned that - read my last mail in the thread
aboutremoving .gz files from kernel.org.)

The idea is to have automatic kernel builds (for all available
architectures I've collected test hardware for:) and run tests. This
needs to be achieved with automatic cross-compilation of kernels (you
don't want to let a m68k compile it's own kernel:), installation and
choosing this for booting. I've got some quite nice ideas there
(including electronic power switches, serial console management etc.),
but yet, I'm not assured that I'll get the room I may get near
Halle/Westf (Germany).

What is _most_ important to testing is this:

	- *fast* response.
			Developers don't like to wait like a month
			before they hear they broke something. If there
			are (untested) patches timely in between, it may
			even get hard to sort the broken part out (cf.
			sparc32 at 2.5.x).

			So the basics are doing automated _compile_
			tests. This includes keeping tables (file name -
			responsible person, architecture - responsible
			person) for automated notification, as well as a
			quite good system to switch certain .config
			items off (so if you find some compile error,
			you have to automatically (if possible) switch
			off the corresponding feature and start again).

	- Decoded Oopses.
			With the in-kernel kksymoops, this is (most of
			the time) quite easy to do if you've got serial
			console working.

			Possibly implementing kgdb for more
			architectures could help also.

			If you then get an answer by a developer, you
			also need to response on a fast manner. Give any
			information to the developers as early as
			possible. They don't like asking for every
			piece. They like mails containing anything they
			need (structured and readable).

			If you then receive patches for review, you'd
			also be capable of automatically including them,
			starting a new compile/install/boot-up, ...

	- Runtime test (stability).
			Some Kernels first start booting, but freeze
			days later. These are the hard ones:( By
			possibility, you haven't got any information on
			this...
	
...and all this for as many machines/architectures with as different as
possibly hardware attached.

MfG, JBG

-- 
   Jan-Benedict Glaw       jbglaw@lug-owl.de    . +49-172-7608481
   "Eine Freie Meinung in  einem Freien Kopf    | Gegen Zensur | Gegen Krieg
    fuer einen Freien Staat voll Freier Bürger" | im Internet! |   im Irak!
      ret = do_actions((curr | FREE_SPEECH) & ~(IRAQ_WAR_2 | DRM | TCPA));

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Testing: What do you want?

[Alan Cox]

On Mon, 2003-03-24 at 14:46, Scott Robert Ladd wrote:
> question is: What do the kernel developers want from testers? What sort 
> of reports are helpful? Is there anything in particular that needs 
> extensive testing?

One of the best things people can do is just use it. OSDL and others run
stress tests but often its users configurations that find bugs not
stress and coverage runs.

Re: Testing: What do you want?

[Scott Robert Ladd]

Jan-Benedict Glaw wrote:
> sparc32? If you get it to build or even to boot, please drop me a note
> 
> Last time I looked at it, sparc32 wasn't in any good state (esp. SMP) in
> 2.5.x. This is because Dave S. Miller stopped spending a lot of hacking
> time (he has to work for other things now and only merges patches he
> gets sent, where he formerly did tons on active development for
> sparc32).

I'll let you know about whether it works or not. What you tell me about 
SPARC support, however, does not bode well. I was about to say "I'm 
surprised that Sun isn't providing any support" -- then my senses 
returned, and I realized I'm not surprised in the least.

..Scott

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)

Re: Testing: What do you want?

[Craig Thomas]

I would echo Alan's comments.  Testing on various platforms with
different configurations will show that linux is capable of running
on a larger variety of hardware.

OSDL has a set of stress tests that are run on new kernels as they
become available.  However, the architectures tested are limited
(compared with the variety that can be found in the world).  There 
are enough testers in the community to collectively test for 
correctness of the kernel.  Test suites such as LTP and LSB help
in this regard.

It would be nice to have a collaborative effort of testing and 
reporting of findings so that everyone can look at particular areas
of interest.  There is also need to generate test results quickly so
that defects can be found and corrected in a timely manner if they
exist.
 

On Mon, 2003-03-24 at 09:01, Alan Cox wrote:
> On Mon, 2003-03-24 at 14:46, Scott Robert Ladd wrote:
> > question is: What do the kernel developers want from testers? What sort 
> > of reports are helpful? Is there anything in particular that needs 
> > extensive testing?
> 
> One of the best things people can do is just use it. OSDL and others run
> stress tests but often its users configurations that find bugs not
> stress and coverage runs.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
-- 
Craig Thomas <craiger@osdl.org>
OSDL

Re: Testing: What do you want?

[Rob Radez]

On Mon, Mar 24, 2003 at 12:30:56PM -0500, Scott Robert Ladd wrote:
> Jan-Benedict Glaw wrote:
> >sparc32? If you get it to build or even to boot, please drop me a note
> >
> >Last time I looked at it, sparc32 wasn't in any good state (esp. SMP) in
> >2.5.x. This is because Dave S. Miller stopped spending a lot of hacking
> >time (he has to work for other things now and only merges patches he
> >gets sent, where he formerly did tons on active development for
> >sparc32).
> 
> I'll let you know about whether it works or not. What you tell me about 
> SPARC support, however, does not bode well. I was about to say "I'm 
> surprised that Sun isn't providing any support" -- then my senses 
> returned, and I realized I'm not surprised in the least.

I just wanted to let y'all know that I've been trying to keep an
up-to-date status page for sparc32 at http://osinvestor.com/sparc/ with
basic information on 2.4 and 2.5 compiling and booting (or not as may
be.)

And yes, 2.5.65 builds and boots UP on sparc32, for me at least.  SMP is
badly broken.

Regards,
Rob Radez

Re: Testing: What do you want?

[Jörn Engel]

On Mon, 24 March 2003 15:06:26 -0800, Craig Thomas wrote:
> 
> OSDL has a set of stress tests that are run on new kernels as they
> become available.  However, the architectures tested are limited
> (compared with the variety that can be found in the world).  There 
> are enough testers in the community to collectively test for 
> correctness of the kernel.  Test suites such as LTP and LSB help
> in this regard.
> 
> On Mon, 2003-03-24 at 09:01, Alan Cox wrote:
> > 
> > One of the best things people can do is just use it. OSDL and others run
> > stress tests but often its users configurations that find bugs not
> > stress and coverage runs.

And another thing I have found to be good at finding bugs is a sick
mind. If you intend to crash a system and do just about anything a
normal user wouldn't, you will stress all the code paths that are
usually not tested.

Jörn

-- 
To recognize individual spam features you have to try to get into the
mind of the spammer, and frankly I want to spend as little time inside
the minds of spammers as possible.
-- Paul Graham

Re: Testing: What do you want?

[Scott Robert Ladd]

Jörn Engel wrote:
> And another thing I have found to be good at finding bugs is a sick
> mind. If you intend to crash a system and do just about anything a
> normal user wouldn't, you will stress all the code paths that are
> usually not tested.

I'll throw a couple big evolutionary simulations on one of my boxes, and 
see how Linux digests it. Lovely thing about stochastic code -- you 
never know what it might do. :)

I could put my wife on Linux. She's a brilliant woman of great talent -- 
and, alas, she is endowed with an incredible ability to wreak 
unintentional destruction on technology. She has accomplished some truly 
remarkable feats of software implosion -- soemtimes, without touching 
the keyboard!

Let's put it this way: She got *really* angry at me once, and all of my 
systems (Linboxen included) REBOOTED spontaneously in reaction to her 
outburst!

Now that's what *I* call "stressing" the code!

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Professional programming for science and engineering;
Interesting and unusual bits of very free code.

Re: Testing: What do you want?

[Scott Robert Ladd]

Jörn Engel wrote:
> And another thing I have found to be good at finding bugs is a sick
> mind. If you intend to crash a system and do just about anything a
> normal user wouldn't, you will stress all the code paths that are
> usually not tested.

I'll throw a couple big evolutionary simulations on one of my boxes, and 
see how Linux digests it. Lovely thing about stochastic code -- you 
never know what it might do. :)

I could put my wife on Linux. She's a brilliant woman of great talent -- 
and, alas, she is endowed with an incredible ability to wreak 
unintentional destruction on technology. She has accomplished some truly 
remarkable feats of software implosion -- sometimes, without touching 
the keyboard!

Let's put it this way: She got *really* angry at me once, and all of my 
systems (Linboxen included) REBOOTED spontaneously in reaction to her 
outburst!

Now that's what *I* call "stressing" the code!

-- 
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Professional programming for science and engineering;
Interesting and unusual bits of very free code.

Re: Testing: What do you want?

[Craig Thomas]

On Mon, 2003-03-24 at 16:04, Rob Radez wrote:
> 
> I just wanted to let y'all know that I've been trying to keep an
> up-to-date status page for sparc32 at http://osinvestor.com/sparc/ with
> basic information on 2.4 and 2.5 compiling and booting (or not as may
> be.)
> 
> And yes, 2.5.65 builds and boots UP on sparc32, for me at least.  SMP is
> badly broken.
> 
> Regards,
> Rob Radez
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

This is great!  At OSDL, we are working on a Linux Stability project for
the 2.5 kernel.  The goal is to provide a web page of test result
content so the linux community can track the progress of the kernel
as it gets more stable with subsequent releases.  This page is
relatively new and I would like to make it a location to get all kinds
of information.  The sparc32 status page would be a great addition to
the stabilization web page, if its all right with you to add it.  It
would be a very good idea to collect all of the testing results
performed on the broad spectrum of architectures into one place to
view.

The OSDL Linux Stability results web page is located at:
http://www.osdl.org/projects/26lnxstblztn/results/  
I would be interested in discussing how the "testers" in the linux
community can share their results with everybody. 

-- 
Craig Thomas <craiger@osdl.org>
OSDL