Re: Kernel stress testing coverage

Re: Kernel stress testing coverage

[Paul Larson]

Alan Cox <alan@lxorguk.ukuu.org.uk> on 03/08/2001 02:06:06 PM

To:   Paul Larson/Austin/IBM@ibmus
cc:
Subject:  Re: Kernel stress testing coverage



>One thing I've been using for coverage (at least some coverage) is the
posix
>test suite

--------------------------

Are you talking about the same posix test suite that LSB is using?  I've
looked into that a little, but here are the two problems I'm wanting to
address:

1. How much of the kernel is getting hit on a run of any given test?  Even
an approximate percentage is fine as long as I can prove it.

2. I could run many many copies simultaneously I suppose and get some
stress, but I'd prefer to stress individual pieces one at a time.  Those
pieces could then be mixed together in later runs for mixed load stress.
Additional mixed load tests will be performed with general applications
(web servers, databases, etc) for more of a "real world" environment, but I
want to have focused tests as well.

I'm betting that there are probably a LOT of quick and dirty test programs
that kernel hackers have written to expose a problem or thoroughly test a
piece of the kernel that they modified.  These type of things would be
FYI this project will be going on sourceforge very soon.  I want to have a
little more to start out with though and finish putting together a good
project description, testplans, etc. to post as soon as we put it on there.
I hate it when people start projects and you don't see any good information
about it for weeks.

Thanks,
Paul Larson

Re: Kernel stress testing coverage

[Alan Cox]

> 1. How much of the kernel is getting hit on a run of any given test?  Even
> an approximate percentage is fine as long as I can prove it.

I've not measured it by percentage. You could use the profiling code in
the kernel to generate a profile and from that measure coverage at least
for non interrupt blocking code

> piece of the kernel that they modified.  These type of things would be
> FYI this project will be going on sourceforge very soon.  I want to have a
> little more to start out with though and finish putting together a good
> project description, testplans, etc. to post as soon as we put it on there.
> I hate it when people start projects and you don't see any good information
> about it for weeks.

VA Cerberus has mahy of these

Re: Kernel stress testing coverage

[Roger Larsson]

Hi,

Here is a link to some memory usage related test programs:

  http://carpanta.dc.fi.udc.es/~quintela/memtest/

They have proven their value many times...


/RogerL

On Thursday 08 March 2001 21:57, Paul Larson wrote:
> Alan Cox <alan@lxorguk.ukuu.org.uk> on 03/08/2001 02:06:06 PM
>
> To:   Paul Larson/Austin/IBM@ibmus
> cc:
> Subject:  Re: Kernel stress testing coverage
>
> >One thing I've been using for coverage (at least some coverage) is the
>
> posix
>
> >test suite
>
> --------------------------
>
> Are you talking about the same posix test suite that LSB is using?  I've
> looked into that a little, but here are the two problems I'm wanting to
> address:
>
> 1. How much of the kernel is getting hit on a run of any given test?  Even
> an approximate percentage is fine as long as I can prove it.
>
> 2. I could run many many copies simultaneously I suppose and get some
> stress, but I'd prefer to stress individual pieces one at a time.  Those
> pieces could then be mixed together in later runs for mixed load stress.
> Additional mixed load tests will be performed with general applications
> (web servers, databases, etc) for more of a "real world" environment, but I
> want to have focused tests as well.
>
> I'm betting that there are probably a LOT of quick and dirty test programs
> that kernel hackers have written to expose a problem or thoroughly test a
> piece of the kernel that they modified.  These type of things would be
> FYI this project will be going on sourceforge very soon.  I want to have a
> little more to start out with though and finish putting together a good
> project description, testplans, etc. to post as soon as we put it on there.
> I hate it when people start projects and you don't see any good information
> about it for weeks.
>
> Thanks,
> Paul Larson
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
Home page:
  none currently

Re: Kernel stress testing coverage

[Anton Blanchard]

 
Hi,

> Are you talking about the same posix test suite that LSB is using?  I've
> looked into that a little, but here are the two problems I'm wanting to
> address:
> 
> 1. How much of the kernel is getting hit on a run of any given test?  Even
> an approximate percentage is fine as long as I can prove it.
> 
> 2. I could run many many copies simultaneously I suppose and get some
> stress, but I'd prefer to stress individual pieces one at a time.  Those
> pieces could then be mixed together in later runs for mixed load stress.
> Additional mixed load tests will be performed with general applications
> (web servers, databases, etc) for more of a "real world" environment, but I
> want to have focused tests as well.

The POSIX tests are good for regression testing not necessarily for
stress testing, obviously both are important. The person to talk to
about this is Chris Yeoh <cyeoh@linuxcare.com.au>.

> I'm betting that there are probably a LOT of quick and dirty test programs
> that kernel hackers have written to expose a problem or thoroughly test a
> piece of the kernel that they modified.  These type of things would be
> FYI this project will be going on sourceforge very soon.  I want to have a
> little more to start out with though and finish putting together a good
> project description, testplans, etc. to post as soon as we put it on there.
> I hate it when people start projects and you don't see any good information
> about it for weeks.

Yes for example I have a set of scripts I use to test software raid,
loopback, ramdisk etc for sparc because there is a tendency for these
things to break and not get found until long afterwards.

Anton

Kernel stress testing coverage

[Paul Larson]

I'm looking for some advice from all of you that know and understand the
Linux kernel so well.  I'm not a kernel developer, but I want to do some
verification work on it, namely stress testing to begin with.  I'm working
on putting together a suite of tests to test the linux kernels under stress
loads for extended runs.  I'll be doing these tests on a mixture of
machines, but most of my focus will be on 2-way, 4-way and 8-way smp
machines.
We've been using some open source tools such as bonnie (for filesystem
stress), but I welcome suggestions for others that will provide good
coverage.  To fill in the gaps, we'll probably be developing our own
testcases.  We have no problems generating mixed loads, and even maxing out
8-way SMP servers for long periods of time, but to be useful, I want to
make sure that whatever tests we are running are getting good coverage of
the kernel code.
Is there any way to see what pieces of the kernel or even percentage we are
hitting with any given test?  I've heard of tools like gcov for doing this
with applications, but the kernel itself seems like it might require
something more.  Are there any ideas you can suggest for writing tests that
will hit as much code as possible in any given section of the kernel like
FS, MM, Scheduler, IPC?  What major sections (like those I previously
mentioned) should I be focused on trying to hit?
All the tests we are writing will be open source of course, and I welcome
any input you may provide.

Thanks,
Paul Larson
Please reply to: plars@us.ibm.com

Re: Kernel stress testing coverage

[Jeff Dike]

plars@us.ibm.com said:
>  I've heard of tools like gcov for doing this with applications, but
> the kernel itself seems like it might require something more.

Have a look at user-mode Linux (http://user-mode-linux.sourceforge.net).  It 
runs the kernel in userspace, so gprof and gcov can be used on it (although, 
at the moment, neither work on UML because of architectural changes I've made 
since I first made them work - these problems will be fixed in the 
medium-term).

Both gprof and gcov produce very interesting information when run on the 
kernel.

So, while you're gathering up tools and information, gather up UML and start 
playing with it.  And if you feel like it, make gcov work again on it - I 
accept patches :-)
				Jeff