question about contest benchmark

question about contest benchmark

[Haoqiang Zheng]

I am wondering how we should interpret the CONTEST benchmark results. 
I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G,
1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes
70.1% and the ratio is 3.98.  Based on what Con says, the result is
bad since the ratio is high. I did some tracing and found the
background load (contest) runs at a dynamic priority of 115-120, which
is often higher than the dynamic priority of the kernel compiling
processes. This explains why the process_load consumes so much CPU.
 
 My question is why is the result bad at all? One could certainly
argue that contest processes shouldn't consume so much CPU time since
they are considered to be background jobs. But why is kernel compiling
considered foreground jobs? Why making kernel compiling faster is
better? Actually, I am wondering if CONTEST is an appropriate
benchmark to report system responsiveness at all?
 
 Any comments?
 
 BTW, what benchmark do you guys use to test system responsiveness?
 
  Haoqiang

Re: question about contest benchmark

[Lee Revell]

On Tue, 2005-05-03 at 14:11 -0400, Haoqiang Zheng wrote:
>  My question is why is the result bad at all? One could certainly
> argue that contest processes shouldn't consume so much CPU time since
> they are considered to be background jobs. But why is kernel compiling
> considered foreground jobs? Why making kernel compiling faster is
> better?

I have reported the same problem before.  CPU bound processes like
kernel compiles will sometimes starve interactive processes like my mail
client (Evolution).

Actually, I discovered a horrible performance bug in Evolution
(suboptimal "hide junk" implementation, and searches for "" not properly
optimized away, see evolution-hackers for a rough patch) that was
responsible for the massive CPU suckage.

But, it seems to me that even if an interactive process briefly goes CPU
bound (due to bloat, bugs, or intent), it should still be scheduled
preferentially to a pure CPU bound process like a build.

Lee

Re: question about contest benchmark

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 613 bytes --]

On Tue, 03 May 2005 14:29:59 EDT, Lee Revell said:

> But, it seems to me that even if an interactive process briefly goes CPU
> bound (due to bloat, bugs, or intent), it should still be scheduled
> preferentially to a pure CPU bound process like a build.

So you want it to schedule that big image (Evolution) that's already used 5
minutes of CPU since it started (this morning, admittedly) in preference to
that cc1 process that will be gone before it's used 2 seconds of CPU, plus all
the disk I/O that cc1 performs (hopefully the cache will help here, but it may
indeed go to disk to read the source files)?


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: question about contest benchmark

[Lee Revell]

On Tue, 2005-05-03 at 16:09 -0400, Valdis.Kletnieks@vt.edu wrote:
> On Tue, 03 May 2005 14:29:59 EDT, Lee Revell said:
> 
> > But, it seems to me that even if an interactive process briefly goes CPU
> > bound (due to bloat, bugs, or intent), it should still be scheduled
> > preferentially to a pure CPU bound process like a build.
> 
> So you want it to schedule that big image (Evolution) that's already used 5
> minutes of CPU since it started (this morning, admittedly) in preference to
> that cc1 process that will be gone before it's used 2 seconds of CPU, plus all
> the disk I/O that cc1 performs (hopefully the cache will help here, but it may
> indeed go to disk to read the source files)?
> 

Yes.  Almost no one will notice whether that build took 2 or 4 seconds.
But a few seconds is an eternity when you are staring at a blank pane,
waiting for it to render the message list.  And I found that when
waiting for the message list to render, backgrounding the build causes
it to render in a second or two, while if I just wait for it it might
take 20 seconds.  This implies that the scheduler could do the same
thing.

Anyway, this was not a great example, as the problem turned out to be an
application bug.  If I can find a non-pathological case that
demonstrates a scheduler problem, I'll post it.

Lee

Re: question about contest benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 2917 bytes --]

On Wed, 4 May 2005 04:11, Haoqiang Zheng wrote:
> I am wondering how we should interpret the CONTEST benchmark results.
> I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G,
> 1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes
> 70.1% and the ratio is 3.98.  Based on what Con says, the result is
> bad since the ratio is high. I did some tracing and found the
> background load (contest) runs at a dynamic priority of 115-120, which
> is often higher than the dynamic priority of the kernel compiling
> processes. This explains why the process_load consumes so much CPU.
>
>  My question is why is the result bad at all? One could certainly
> argue that contest processes shouldn't consume so much CPU time since
> they are considered to be background jobs. But why is kernel compiling
> considered foreground jobs? Why making kernel compiling faster is
> better? Actually, I am wondering if CONTEST is an appropriate
> benchmark to report system responsiveness at all?

I don't think in my readme do I say anywhere what is the ideal balance. 
Process_load is a uniquely different load to the other loads which are 
various combinations of cpu and i/o. It spawns processes that wake up, hand 
their data off to another process and go to sleep. Thus the processes behave 
like interactive one with their frequent waiting, but share their effective 
group cpu usage amongst all the process_child processes running so none of 
them is actually seen as cpu bound. Furthermore there are massive numbers of 
context switches between them meaning there is a large in-kernel "system" 
load that is done on behalf of the process_child ren. The purpose of the 
process_load in contest is to ensure that an interactive design is not 
DoS'able by processes behaving like this. Process_load spawns 4 times as many 
processes as the timed 'make' in contest so theoretically ideal cpu balance 
between them should show process_load having 4x as much cpu as the make. 
Because their cpu binding is so intermittent it's hard to balance them 
perfectly. Anyway the balance in your output seems pretty good. When the 
interactive design goes horribly wrong process_load consumes 100 times as 
much cpu as the 'make'.

>
>  Any comments?
>
>  BTW, what benchmark do you guys use to test system responsiveness?

Note that interactivity is not responsiveness which some people try to measure 
with contest, and there is still no interactivity benchmark. Responsiveness 
is the ability of the system to continue performing tasks at a reasonable 
pace under various system loads. Interactivity is having low scheduling 
latency and jitter in those tasks where human interaction would notice the 
latency and jitter - and what constitutes and interactive tasks has not been 
quantified although we all know what they are when using the pc.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: question about contest benchmark

[Haoqiang Zheng]

Con,

Thanks so much for your response. The confusion was caused by what you
said at http://members.optusnet.com.au/ckolivas/contest/:
"The lower the time (and ratio) the better, and the higher the cpu
percentage the better. ".

>From http://kerneltrap.org/node/465, I also found: "
Time needs to be as low as possible (and therefore ratio as close to 1)
CPU% needs to be as high as possible
Loads needs to be as high as possible
LCPU% needs to be as high as possible.
". 

However, from what you described in the email, this doesn't sound to
be true anymore. I think it will be hard to interpret the results if
it's the "balance" that matters. For example, if CPU% is 40 in case A
and 60 in case B.  If lower is better, then A is better.  If it's
balance that matters, I would say A is about the same as B since in
both case MAX/MIN = 1.5.  (Suppose the total usage is 100%).  So which
answer is correct?

BTW, I like your clarification about interactivity vs. responsiveness.
 But considering the work you have done on improving Linux
interactivity, there's no wonder that people think CONTEST is also an
interactivity benchmark.  :-)

Haoqiang
On 5/3/05, Con Kolivas <kernel@kolivas.org> wrote:
> On Wed, 4 May 2005 04:11, Haoqiang Zheng wrote:
> > I am wondering how we should interpret the CONTEST benchmark results.
> > I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G,
> > 1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes
> > 70.1% and the ratio is 3.98.  Based on what Con says, the result is
> > bad since the ratio is high. I did some tracing and found the
> > background load (contest) runs at a dynamic priority of 115-120, which
> > is often higher than the dynamic priority of the kernel compiling
> > processes. This explains why the process_load consumes so much CPU.
> >
> >  My question is why is the result bad at all? One could certainly
> > argue that contest processes shouldn't consume so much CPU time since
> > they are considered to be background jobs. But why is kernel compiling
> > considered foreground jobs? Why making kernel compiling faster is
> > better? Actually, I am wondering if CONTEST is an appropriate
> > benchmark to report system responsiveness at all?
> 
> I don't think in my readme do I say anywhere what is the ideal balance.
> Process_load is a uniquely different load to the other loads which are
> various combinations of cpu and i/o. It spawns processes that wake up, hand
> their data off to another process and go to sleep. Thus the processes behave
> like interactive one with their frequent waiting, but share their effective
> group cpu usage amongst all the process_child processes running so none of
> them is actually seen as cpu bound. Furthermore there are massive numbers of
> context switches between them meaning there is a large in-kernel "system"
> load that is done on behalf of the process_child ren. The purpose of the
> process_load in contest is to ensure that an interactive design is not
> DoS'able by processes behaving like this. Process_load spawns 4 times as many
> processes as the timed 'make' in contest so theoretically ideal cpu balance
> between them should show process_load having 4x as much cpu as the make.
> Because their cpu binding is so intermittent it's hard to balance them
> perfectly. Anyway the balance in your output seems pretty good. When the
> interactive design goes horribly wrong process_load consumes 100 times as
> much cpu as the 'make'.
> 
> >
> >  Any comments?
> >
> >  BTW, what benchmark do you guys use to test system responsiveness?
> 
> Note that interactivity is not responsiveness which some people try to measure
> with contest, and there is still no interactivity benchmark. Responsiveness
> is the ability of the system to continue performing tasks at a reasonable
> pace under various system loads. Interactivity is having low scheduling
> latency and jitter in those tasks where human interaction would notice the
> latency and jitter - and what constitutes and interactive tasks has not been
> quantified although we all know what they are when using the pc.
> 
> Cheers,
> Con
> 
> 
>