Re: question about contest benchmark

[Haoqiang Zheng <haoqiang@gmail.com>]

Con,

Thanks so much for your response. The confusion was caused by what you
said at http://members.optusnet.com.au/ckolivas/contest/:
"The lower the time (and ratio) the better, and the higher the cpu
percentage the better. ".

>From http://kerneltrap.org/node/465, I also found: "
Time needs to be as low as possible (and therefore ratio as close to 1)
CPU% needs to be as high as possible
Loads needs to be as high as possible
LCPU% needs to be as high as possible.
". 

However, from what you described in the email, this doesn't sound to
be true anymore. I think it will be hard to interpret the results if
it's the "balance" that matters. For example, if CPU% is 40 in case A
and 60 in case B.  If lower is better, then A is better.  If it's
balance that matters, I would say A is about the same as B since in
both case MAX/MIN = 1.5.  (Suppose the total usage is 100%).  So which
answer is correct?

BTW, I like your clarification about interactivity vs. responsiveness.
 But considering the work you have done on improving Linux
interactivity, there's no wonder that people think CONTEST is also an
interactivity benchmark.  :-)

Haoqiang
On 5/3/05, Con Kolivas <kernel@kolivas.org> wrote:
> On Wed, 4 May 2005 04:11, Haoqiang Zheng wrote:
> > I am wondering how we should interpret the CONTEST benchmark results.
> > I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G,
> > 1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes
> > 70.1% and the ratio is 3.98.  Based on what Con says, the result is
> > bad since the ratio is high. I did some tracing and found the
> > background load (contest) runs at a dynamic priority of 115-120, which
> > is often higher than the dynamic priority of the kernel compiling
> > processes. This explains why the process_load consumes so much CPU.
> >
> >  My question is why is the result bad at all? One could certainly
> > argue that contest processes shouldn't consume so much CPU time since
> > they are considered to be background jobs. But why is kernel compiling
> > considered foreground jobs? Why making kernel compiling faster is
> > better? Actually, I am wondering if CONTEST is an appropriate
> > benchmark to report system responsiveness at all?
> 
> I don't think in my readme do I say anywhere what is the ideal balance.
> Process_load is a uniquely different load to the other loads which are
> various combinations of cpu and i/o. It spawns processes that wake up, hand
> their data off to another process and go to sleep. Thus the processes behave
> like interactive one with their frequent waiting, but share their effective
> group cpu usage amongst all the process_child processes running so none of
> them is actually seen as cpu bound. Furthermore there are massive numbers of
> context switches between them meaning there is a large in-kernel "system"
> load that is done on behalf of the process_child ren. The purpose of the
> process_load in contest is to ensure that an interactive design is not
> DoS'able by processes behaving like this. Process_load spawns 4 times as many
> processes as the timed 'make' in contest so theoretically ideal cpu balance
> between them should show process_load having 4x as much cpu as the make.
> Because their cpu binding is so intermittent it's hard to balance them
> perfectly. Anyway the balance in your output seems pretty good. When the
> interactive design goes horribly wrong process_load consumes 100 times as
> much cpu as the 'make'.
> 
> >
> >  Any comments?
> >
> >  BTW, what benchmark do you guys use to test system responsiveness?
> 
> Note that interactivity is not responsiveness which some people try to measure
> with contest, and there is still no interactivity benchmark. Responsiveness
> is the ability of the system to continue performing tasks at a reasonable
> pace under various system loads. Interactivity is having low scheduling
> latency and jitter in those tasks where human interaction would notice the
> latency and jitter - and what constitutes and interactive tasks has not been
> quantified although we all know what they are when using the pc.
> 
> Cheers,
> Con
> 
> 
>