Re: Automated performance testing system was Re: Text form for STP tests

[Cliff White <cliffw@osdl.org>]

> Linux-MM fellows,
> 
> I've been talking to Cliff about the need for a set of benchmarks,
> covering as many different workloads as possible, for developers to have a 
> better notion of impact on performance changes. 
> 
> Usually when one does a change which affects performance, he/she runs one 
> or two benchmarks with a limited amount of hardware configurations.
> This is a very painful, boring and time consuming process, which can 
> result in misinterpretation and/or limited understading of the results 
> of such changes.
> 
> It is important to automate such process, with a set of benchmarks 
> covering as wide as possible range of workloads, running on common 
> and most used hardware variations.
> 
> OSDL's STP provides the base framework for this.
> 
[ snip ]
> bonnie++
> reaim (default, new_fserver, shared)
> dbench_long
> kernbench
> tiobench
> 
> Each of these running one the following combinations:
> 
> 1CPU, 2CPU, 4CPU, 8CPU (4 variants).
> 
> total memory, half memory, a quarter of total memory (3 variants).
> 
> Thats 12 results for each benchmark."
> 
The configuration files to do these tests are now written, and the humble
robots are running this test series against linux-2.6.7 ( for history data )
There will need to be some adjustments - some of these tests will no doubt
fail for reasons of script error or configuration ( i see already kernbench will 
have to be redunced for 1-cpu systems, as it runs > 13.5 hours :( )

And, the second part of the automation is already done, but needs input.
I can aim this test battery at any kernel patch, where 'any kernel patch'
is identified by a regexp. What kernels do you want this against? 
I've heard mention of 'baseline' - we call this baseline:

/^(patch|linux)-\d+\.\d+\.\d+$/

( starts with 'patch' or 'linux', then '-' followed by three decimals ) 

cliffw
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"aart@kvack.org"> aart@kvack.org </a>