2.6.0-test8/test9 repeatable lockups at high loads - please reproduce

2.6.0-test8/test9 repeatable lockups at high loads - please reproduce

[Michael Frank]

[-- Attachment #1: Type: text/plain, Size: 644 bytes --]

Using attached stress tests on a P4 with IDE, 2.6.0-test8/test9 lockup
repeatably within 5 minutes to an hour.

It is generally pssible to break into kdb debugger but not into kgdb.

Tests running fine with 2.4.22 and 2.4.21 for 100s of hours

Requirements:

gnu tool chain
X 3 or 4
configured ready to compile 2.4 or 2.6 kernel source tree linked to /usr/src/linuxtest
/tmp directory and/or $TEMP set to a user writable temporary directory
200MB spare disk space

How to use:

$ ln -s /usr/src/yourlinux /usr/src/linuxtest
$ tar jxf ti-tests.tar.bz2

to start the test:

$ ti-tests/ti 1

to stop the test

$ ti-tests/ti 0

Regards
Michael





[-- Attachment #2: ti-tests.tar.bz2 --]
[-- Type: application/x-tbz, Size: 35479 bytes --]

Re: 2.6.0-test8/test9 repeatable lockups at high loads - please reproduce

[Michael Frank]

On Tuesday 28 October 2003 05:44, Michael Frank wrote:
> Using attached stress tests on a P4 with IDE, 2.6.0-test8/test9 lockup
> repeatably within 5 minutes to an hour.
> 
> It is generally pssible to break into kdb debugger but not into kgdb.
> 
> Tests running fine with 2.4.22 and 2.4.21 for 100s of hours
> 
> Requirements:
> 
> gnu tool chain
> X 3 or 4
> configured ready to compile 2.4 or 2.6 kernel source tree linked to /usr/src/linuxtest
> /tmp directory and/or $TEMP set to a user writable temporary directory
> 200MB spare disk space
> 
> How to use:
> 
> $ ln -s /usr/src/yourlinux /usr/src/linuxtest
> $ tar jxf ti-tests.tar.bz2
> 
> to start the test:
> 
> $ ti-tests/ti 1
> 
> to stop the test
> 
> $ ti-tests/ti 0
> 

Sorry, README_TI file was missing. Enclosed

Regards
Michael

*****************************************************************************
ti-tests

Copyright 2003 Michael Frank, mhf@linuxmail.org

This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
Version 2 as published by the Free Software Foundation.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
http://www.gnu.org/copyleft/gpl.html
*****************************************************************************

Version: 1.0-beta

1) What does it do
------------------

ti executes one or more concurrent functions to assert system level stress.
ti utilizes dd, the gnu tool chain and unixbench system level functions: syscall, pipe,
   context1, spawn, execl, fstime, fsbuffer, fsdisk, shell, C.

2) Individual functions
-----------------------

Each test function is run in a dedicated xterm or VT when not running under X.
All test functions can be run in the current terminal by prepending the function
name with an underscore such as in _ddw

compk - kernel compile consisting of make clean dep bzImage modules

ddw instances count - "instances" dd write loops of "count" 4K blocks. Instances is
                      in range of 1 to 10 and count must be > 0

ddr instances count - "instances" dd read loop of "count" 4K blocks. Instances is
                      in range of 1 to 10 and count must be > 0

3) Combined test functions
--------------------------

Each combined test function is run in a dedicated xterm or VT when not running under X.
All combined test functions can be run in the current terminal by prepending the function
name with an underscore such as in _ub2

ub2   concurrently run 2 unixbenchs - CPU's below 100MHz:
         syscall / pipe / context1 / spawn  / execl
         fsbuffer / fstime / fsdisk / shell / C

ub5    concurrently run 5 unixbenchs - for CPU's below 500MHz:
         syscall / pipe
         context1 / spawn
         execl / fsbuffer
         fstime / fsdisk
         shell / C

ub10 - concurrently run 10 unixbenchs - GHz CPU's:
         syscall pipe context1 spawn execl
         fstime fsbuffer fsdisk shell C

ub17 - executes ub2, ub5 and ub10

4) Status functions
-------------------

Each status function is run in a dedicated xterm or VT when not running under X.
All status functions except 'vm' can be run in the current terminal by prepending the
function name with an underscore such as in _msg

dmesg - show dmesg every second

la - loadavg every second

msg - show /var/log/messages every second

ps - ps -A every second

vm - show vmstat every second

5) Combined status functions
----------------------------

stat - show msg, dmesg, vm, la, ps

6) Combined status and test functions
-------------------------------------

0 - stop all

1 - start stat compk ub17 ddw 4 5000

7) Requirements
---------------

- Recent POSIX compliant shell such as bash 2.05
- Complete installation of gnu tools such as gcc and make
- Configured ready to compile 2.4 or 2.6 kernel source tree linked to /usr/src/linuxtest
- /tmp directory and/or $TEMP set to a user writable temporary directory

8) Security
-----------

Save all files and exit all applications in case reset is
required

Should not be run as root

Be sure that $TEMP/$FILE* does not match any valuable files

Avoid setting ddw/ddr instances > 5 and if so increase instances in small steps

9) Files
--------

ti -  the main test script
ti-run - modified unixbench Run script
pgms/* - unixbench ready to run programs
src/* - unixbench routines source
tmp/ - unixbench temporary files
results/ - unixbench temporary files
README-TI - this file

10) Installation example
------------------------

$ cd
$ ln -s /usr/src/yourlinux /usr/src/linuxtest   # link in testkernel
$ tar jxf ti-tests.tar.bz2                      # install package
# ln -s ~/ti-tests/ti /usr/local/bin            # or suitable directory in path

More configuration is documented in ti-tests/ti

11) Typical usage
----------------

Run the status display
$ ti stat

Run ub5
$ ti ub5

Commands can also be combined

Run the status display, ub2
$ ti stat ub2

Run combined tests
$ ti 1

12) Logging

Each pass of ub and compk is logged to the system log.

13) Bugs, caveats and limitations
---------------------------------

ti can assert extreme loads. Loadaverage of 100 was observed.

ti may fill up the disk and slow the system down to the point of unusability.

Usage with PIO hardisk mode is not recommended.

ti 0 will kill _all_ processes listed in ti function dokillall,
which is a likely nuisance should one use the system for real work
while testing it.

ti may cause instable hardware or software to crash or hang.

ti may cause overclocked or undercooled or weak hardware to burn up.

Use ti for your benefit and at your own risk. Do not run as root.

14) History
-----------

most functions and the use of unixbench functions originate from testing swsusp.

dd loops were used by the tstinter kernel interactivity test script.

Re: 2.6.0-test8/test9 repeatable lockups at high loads - please reproduce

[Michael Frank]

And in case you ask...

15) Why is it done this way
---------------------------

Experience shows that randomness of activities and events over a broad spectrum
cause substantial stress in any system.

A simple example is the fact that an instable system often can perform one
function at a time (such as run an editor or a single compiler wo make), but
fails when several tasks are demanded such as building a kernel.

Another example is the 68000 based STD bus system which could run multiuser
turbodos perfectly but died with version 7 because the powersupply could not
handle the random fluctuations of power consumption and started oscillating
and going out of spec. The local unix guru at the time always said "you listen
it makes noise like doors on the enterprise" ;) and hooking up a storage scope
validated his opinion after some time and trial. Interestingly even some bad
grounding (loops) of the 10MB 8" drive made trouble specific to V7.

When developing tests for swsusp, criteria were applied as follows
- randomness was of first interest
- the desire to excercise a broad spectrum of system functions was next
- the desire to utilize mature functionality in order not to reinvent the wheel
- software must be available freely

The ancient unixbench suite meets these requirements and is complemented
by make, gcc for generic user level stress and dd for brute force IO loads.
A little bit of bash scripting completes the test suite.

It is noted that the CPU related unixbench functions such as dhrystone,
whetstone, etc are not used here as these have shown to do little more than burn
processor clocks on-die. A  bad (CPU) chip being used according to datasheet
is a tiny fraction of 1 in millions and we do not want to test for this anyway.

As to randomness, unixbench tests are timed, but by asynchronous events the
timing is not perfect, thus several of these tests gradually drift relative to
each other and greatly interfere looking from the system(call) perspective.

Opinion of the gut, and experience with failures is that the  spectrum of tests
excercises most system functions heavily.

The tests have however not been profiled as I have not yet learned how to use
profiling. There is always too much to do.