[ANNOUNCE] Interbench v0.26

[ANNOUNCE] Interbench v0.26

[Con Kolivas]

This benchmark application is designed to benchmark interactivity in Linux.

Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.26.tar.bz2

Web site:
http://interbench.kolivas.org

Changes since v0.24:

v0.25:
The timekeeping thread of background load no longer runs SCHED_FIFO. The 
benchmark is allowed to proceed if it does not detect swap and instead 
disables mem_load. The documentation was updated.

v0.26:
Fixed the standard deviation measurements at last (thanks Peter Williams). 
There should be no practical limit to how long you can run a benchmark for 
now.

Cheers,
Con

Re: [ck] [ANNOUNCE] Interbench v0.26

[Gabriel Devenyi]

You haven't quite completely fixed the SD calculations it seems:


--- Benchmarking simulated cpu of Gaming in the presence of simulated---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
None       2.44 +/- nan         48.6            98.7
Video      12.8 +/- nan         55.2              89
X          89.7 +/- nan          494            52.8
Burn        400 +/- nan         1004            20.1
Write      49.2 +/- nan          343            67.2
Read       4.14 +/- nan         56.7            96.7
Compile     551 +/- nan         1369            15.4


--
Gabriel Devenyi
ace@staticwave.ca

Con Kolivas wrote:
> This benchmark application is designed to benchmark interactivity in Linux.
> 
> Direct download link:
> http://ck.kolivas.org/apps/interbench/interbench-0.26.tar.bz2
> 
> Web site:
> http://interbench.kolivas.org
> 
> Changes since v0.24:
> 
> v0.25:
> The timekeeping thread of background load no longer runs SCHED_FIFO. The 
> benchmark is allowed to proceed if it does not detect swap and instead 
> disables mem_load. The documentation was updated.
> 
> v0.26:
> Fixed the standard deviation measurements at last (thanks Peter Williams). 
> There should be no practical limit to how long you can run a benchmark for 
> now.
> 
> Cheers,
> Con
> _______________________________________________
> ck@vds.kolivas.org
> ck mailing list. Please reply-to-all when posting.
> If replying to an email please reply below the original message.
> http://bhhdoa.org.au/mailman/listinfo/ck
> 

Re: [ck] [ANNOUNCE] Interbench v0.26

[Con Kolivas]

On Wed, 3 Aug 2005 22:01, Gabriel Devenyi wrote:
> You haven't quite completely fixed the SD calculations it seems:
>
>
> --- Benchmarking simulated cpu of Gaming in the presence of simulated---
> Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
> None       2.44 +/- nan         48.6            98.7
> Video      12.8 +/- nan         55.2              89
> X          89.7 +/- nan          494            52.8
> Burn        400 +/- nan         1004            20.1
> Write      49.2 +/- nan          343            67.2
> Read       4.14 +/- nan         56.7            96.7
> Compile     551 +/- nan         1369            15.4

>:(

I keep trying

Con

Re: [ck] [ANNOUNCE] Interbench v0.26

[Peter Williams]

Con Kolivas wrote:
> On Wed, 3 Aug 2005 22:01, Gabriel Devenyi wrote:
> 
>>You haven't quite completely fixed the SD calculations it seems:
>>
>>
>>--- Benchmarking simulated cpu of Gaming in the presence of simulated---
>>Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
>>None       2.44 +/- nan         48.6            98.7
>>Video      12.8 +/- nan         55.2              89
>>X          89.7 +/- nan          494            52.8
>>Burn        400 +/- nan         1004            20.1
>>Write      49.2 +/- nan          343            67.2
>>Read       4.14 +/- nan         56.7            96.7
>>Compile     551 +/- nan         1369            15.4
> 
> 
>>:(
> 
> 
> I keep trying

The problem is a variation of the original one that I pointed out.  The 
value that's being added to the sum of the squares of the latency is not 
always the square of the value being added to the latency.

Would you like me to fix it and send you a patch?

Peter
-- 
Peter Williams                                   pwil3058@bigpond.net.au

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce

Re: [ck] [ANNOUNCE] Interbench v0.26

[Con Kolivas]

On Thu, 4 Aug 2005 09:25 am, Peter Williams wrote:
> Con Kolivas wrote:
> > On Wed, 3 Aug 2005 22:01, Gabriel Devenyi wrote:
> >>You haven't quite completely fixed the SD calculations it seems:
> >>
> >>
> >>--- Benchmarking simulated cpu of Gaming in the presence of simulated---
> >>Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
> >>None       2.44 +/- nan         48.6            98.7
> >>Video      12.8 +/- nan         55.2              89
> >>X          89.7 +/- nan          494            52.8
> >>Burn        400 +/- nan         1004            20.1
> >>Write      49.2 +/- nan          343            67.2
> >>Read       4.14 +/- nan         56.7            96.7
> >>Compile     551 +/- nan         1369            15.4
> >>
> >>:(
> >
> > I keep trying
>
> The problem is a variation of the original one that I pointed out.  The
> value that's being added to the sum of the squares of the latency is not
> always the square of the value being added to the latency.
>
> Would you like me to fix it and send you a patch?

I fixed that too in what is in front of me (sorry not the one I've released it 
was only clear to me when this report came back) and am still hitting some 
bug somewhere. I've yet to track it down.

Cheers,
Con

Re: [ck] [ANNOUNCE] Interbench v0.26

[Con Kolivas]

On Thu, 4 Aug 2005 09:25 am, Con Kolivas wrote:
> On Thu, 4 Aug 2005 09:25 am, Peter Williams wrote:
> > Con Kolivas wrote:
> > > On Wed, 3 Aug 2005 22:01, Gabriel Devenyi wrote:
> > >>You haven't quite completely fixed the SD calculations it seems:
> > >>
> > >>
> > >>--- Benchmarking simulated cpu of Gaming in the presence of
> > >> simulated--- Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
> > >>None       2.44 +/- nan         48.6            98.7
> > >>Video      12.8 +/- nan         55.2              89
> > >>X          89.7 +/- nan          494            52.8
> > >>Burn        400 +/- nan         1004            20.1
> > >>Write      49.2 +/- nan          343            67.2
> > >>Read       4.14 +/- nan         56.7            96.7
> > >>Compile     551 +/- nan         1369            15.4
> > >>
> > >>:(
> > >
> > > I keep trying
> >
> > The problem is a variation of the original one that I pointed out.  The
> > value that's being added to the sum of the squares of the latency is not
> > always the square of the value being added to the latency.
> >
> > Would you like me to fix it and send you a patch?
>
> I fixed that too in what is in front of me (sorry not the one I've released
> it was only clear to me when this report came back) and am still hitting
> some bug somewhere. I've yet to track it down.

Silly me. The gaming emulation doesn't use periodic_schedule so I wasn't 
storing the data anywhere for standard deviation. Will release a fixed 
version soon.

Cheers,
Con

[ANNOUNCE] Interbench 0.27

[Con Kolivas]

Interbench is a benchmark application is designed to benchmark interactivity 
in Linux.

Direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2

Web page:
http://interbench.kolivas.org

Changes:
Standard deviation and average latency calculation was corrected. Gaming 
standard deviation was implemented.

Cheers,
Con

Re: [ck] [ANNOUNCE] Interbench 0.27

[Gabriel Devenyi]

Hi Con,

You must hate me by now...

The "Gaming" benchmark has the same issue with nan coming out of the 
STDEV calculations, probably requires the same fix as before.

Secondly, the benchmarking of loops_per_ms is running forever, and I 
managed to determine where its happening.

In calibrate loops you run a while loop and iterate to get 1000 for 
run_time, then you calculate it one more time to ensure it was right 
*however* you put a sleep(1) before that. It seems to seriously skew the 
results, as it consistently adds ~500 to run_time, as run_time is now 
1500, it jumps back up to redo because of the goto statement, and runs 
the while loop again, continue ad nausium. I added some simple debugging 
output which prints run time at the end of each while loop, and right 
before the goto if statement, this is the output.

--cut--
loops_per_ms unknown; benchmarking...
while: 224
while: 649
while: 993
while: 1025
while: 976
while: 1001
while: 1000
1494
while: 671
while: 997
while: 1000
1496
--cut--

The solution I used is of course to simply comment out the sleep 
statement, then everything works nicely, however your comments appear to 
indicate that the sleep is there to make the system settle a little, 
perhaps another method needs to be used. Thanks for your time!

--
Gabriel Devenyi
ace@staticwave.ca

Con Kolivas wrote:
> Interbench is a benchmark application is designed to benchmark interactivity 
> in Linux.
> 
> Direct download link:
> http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2
> 
> Web page:
> http://interbench.kolivas.org
> 
> Changes:
> Standard deviation and average latency calculation was corrected. Gaming 
> standard deviation was implemented.
> 
> Cheers,
> Con
> _______________________________________________
> ck@vds.kolivas.org
> ck mailing list. Please reply-to-all when posting.
> If replying to an email please reply below the original message.
> http://bhhdoa.org.au/mailman/listinfo/ck
> 

Re: [ck] [ANNOUNCE] Interbench 0.27

[Con Kolivas]

On Thu, 4 Aug 2005 21:44, Gabriel Devenyi wrote:
> Hi Con,
>
> You must hate me by now...

No. A bug report is a bug report. I hate the fact that I coded up 2000 lines 
of code and am still suffering from a problem in the same 10 lines that I did 
in version .01. PEBKAC. 

> The "Gaming" benchmark has the same issue with nan coming out of the
> STDEV calculations, probably requires the same fix as before.

Anyway Peter Williams has promised to fix it for me (yay!).

> Secondly, the benchmarking of loops_per_ms is running forever, and I
> managed to determine where its happening.
>
> In calibrate loops you run a while loop and iterate to get 1000 for
> run_time, then you calculate it one more time to ensure it was right
> *however* you put a sleep(1) before that. It seems to seriously skew the
> results, as it consistently adds ~500 to run_time, as run_time is now
> 1500, it jumps back up to redo because of the goto statement, and runs
> the while loop again, continue ad nausium. I added some simple debugging
> output which prints run time at the end of each while loop, and right
> before the goto if statement, this is the output.

> The solution I used is of course to simply comment out the sleep
> statement, then everything works nicely, however your comments appear to
> indicate that the sleep is there to make the system settle a little,
> perhaps another method needs to be used. Thanks for your time!

I have to think about it. This seems a problem only on one type of cpu for 
some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 
followed by the check made results far less reliable. This way with the sleep 
1 I have not had spurious results returned by the calibration. I'm open to 
suggestions if anyone's got one.

Cheers,
Con

Re: [ck] [ANNOUNCE] Interbench 0.27

[Gabriel Devenyi]

Con Kolivas wrote:
> On Thu, 4 Aug 2005 21:44, Gabriel Devenyi wrote:
> 
>>Hi Con,
>>
>>You must hate me by now...
> 
> 
> No. A bug report is a bug report. I hate the fact that I coded up 2000 lines 
> of code and am still suffering from a problem in the same 10 lines that I did 
> in version .01. PEBKAC. 
I guess we all have our weaknesses, mine is off-by-one errors, which are 
a bad thing when writing code for a statistics class at school ;)

> 
> 
>>The "Gaming" benchmark has the same issue with nan coming out of the
>>STDEV calculations, probably requires the same fix as before.
> 
> 
> Anyway Peter Williams has promised to fix it for me (yay!).
> 
> 
>>Secondly, the benchmarking of loops_per_ms is running forever, and I
>>managed to determine where its happening.
>>
>>In calibrate loops you run a while loop and iterate to get 1000 for
>>run_time, then you calculate it one more time to ensure it was right
>>*however* you put a sleep(1) before that. It seems to seriously skew the
>>results, as it consistently adds ~500 to run_time, as run_time is now
>>1500, it jumps back up to redo because of the goto statement, and runs
>>the while loop again, continue ad nausium. I added some simple debugging
>>output which prints run time at the end of each while loop, and right
>>before the goto if statement, this is the output.
> 
> 
>>The solution I used is of course to simply comment out the sleep
>>statement, then everything works nicely, however your comments appear to
>>indicate that the sleep is there to make the system settle a little,
>>perhaps another method needs to be used. Thanks for your time!
> 
> 
> I have to think about it. This seems a problem only on one type of cpu for 
> some strange reason (lemme guess; athlon?) and indeed leaving out the sleep 1 
> followed by the check made results far less reliable. This way with the sleep 
> 1 I have not had spurious results returned by the calibration. I'm open to 
> suggestions if anyone's got one.
Yeah, thats right, it spins forever on both my athlon-tbird and my 
athlon64 (in x86_64 mode). I'll take another look at the code tonight, 
to see if I can figure out why its causing this, or another way of 
incurring the delay you're looking for.

> 
> Cheers,
> Con
> 

--
Gabriel Devenyi
ace@staticwave.ca

Re: [ck] [ANNOUNCE] Interbench 0.27

[Con Kolivas]

On Thu, 4 Aug 2005 22:05, Gabriel Devenyi wrote:
> Con Kolivas wrote:
> > I have to think about it. This seems a problem only on one type of cpu
> > for some strange reason (lemme guess; athlon?) and indeed leaving out the
> > sleep 1 followed by the check made results far less reliable. This way
> > with the sleep 1 I have not had spurious results returned by the
> > calibration. I'm open to suggestions if anyone's got one.
>
> Yeah, thats right, it spins forever on both my athlon-tbird and my
> athlon64 (in x86_64 mode). I'll take another look at the code tonight,
> to see if I can figure out why its causing this, or another way of
> incurring the delay you're looking for.

I'd appreciate it. It's almost like some power stepping that's responsible. 
I've never seen it happen on any intel processor (including the pentiumM ones 
which have truckloads of power saving features). I've asked many people if 
they're running some equivalent of cool'n'quiet or powernow* and they all 
insist they're not... I'm not that familiar with all the powersaving techs 
though.

Cheers,
Con

Re: [ck] [ANNOUNCE] Interbench 0.27

[Gabriel Devenyi]

Con Kolivas wrote:
> I'd appreciate it. It's almost like some power stepping that's responsible. 
> I've never seen it happen on any intel processor (including the pentiumM ones 
> which have truckloads of power saving features). I've asked many people if 
> they're running some equivalent of cool'n'quiet or powernow* and they all 
> insist they're not... I'm not that familiar with all the powersaving techs 
> though.

I'm certainly not running any powersaving features on my athlon-tbird(it 
doesn't have any AFAIK, and its the hottest running AMD processor there 
is) However I've realized I did have cool n' quiet with the ondemand 
governor running on my athlon64, so that might indicate an issue, again, 
I'll look into that tonight.

--
Gabriel Devenyi
ace@staticwave.ca

Re: [ck] [ANNOUNCE] Interbench 0.27

[Gabriel Devenyi]

After conducting some further research I've determined that cool n quiet has 
no effect on this "bug" if you can call it that. With the system running in 
init 1, and cool n quiet disabled in the bios, a sleep(N>0) results in the 
run_time value afterwards always being nearly the same value of ~995000 on my 
athlon64, similarly, my server an athlon-tbird, which definitely has no power 
saving features, hovers at ~1496000

Obviously since these values are nowhere near 10000, the loops_per_ms 
benchmark runs forever, has anyone seen/read about sleep on amd machines 
doing something odd? Can anyone else with an amd machine confirm this 
behavior? Con: should we attempt to get the attention of LKML to see why amd 
chips act differently?

--
Gabriel Devenyi
ace@staticwave.ca

Re: [ck] [ANNOUNCE] Interbench 0.27

[Con Kolivas]

On Sat, 6 Aug 2005 13:37, Gabriel Devenyi wrote:
> After conducting some further research I've determined that cool n quiet
> has no effect on this "bug" if you can call it that. With the system
> running in init 1, and cool n quiet disabled in the bios, a sleep(N>0)
> results in the run_time value afterwards always being nearly the same value
> of ~995000 on my athlon64, similarly, my server an athlon-tbird, which
> definitely has no power saving features, hovers at ~1496000

We know that sleep(1) doesn't give us accurate sleep of 1 second, only close 
to it limited by Hz, schedule_timeout and how busy the kernel otherwise is.

> Obviously since these values are nowhere near 10000, the loops_per_ms
> benchmark runs forever, has anyone seen/read about sleep on amd machines
> doing something odd? Can anyone else with an amd machine confirm this
> behavior? Con: should we attempt to get the attention of LKML to see why
> amd chips act differently?

None of that matters because the timing is done during a non sleep period 
using the real time clock:

	start_time = get_nsecs(&myts);
	burn_loops(loops);
	run_time = get_nsecs(&myts) - start_time;

So the time spent in sleep(1) should be irrelevant to the timing of 
loops_per_ms. Something else is happening to the cpu _during_ the sleep that 
makes the next lot of loops take a different length of time. That's the bit I 
haven't been able to figure out.

Cheers,
Con

Re: [ANNOUNCE] Interbench 0.27

[Bill Davidsen]

Con Kolivas wrote:
> Interbench is a benchmark application is designed to benchmark interactivity 
> in Linux.
> 
> Direct download link:
> http://ck.kolivas.org/apps/interbench/interbench-0.27.tar.bz2
> 
> Web page:
> http://interbench.kolivas.org
> 
> Changes:
> Standard deviation and average latency calculation was corrected. Gaming 
> standard deviation was implemented.

As you may or may not remember I have a response benchmark, which does 
different things... And one of the things I found is that when trying to 
determine if a tuning was "better" was to look at the 90 and/or 95 
percentile value. The max, average, and SD give you information which 
may be hard to really understand, but the "mostly better than X" times 
are pretty easy to understand.

I finally wound up using a dynamic percentile thing of my own creation, 
but there's no supporting theory, I just looked with response curve 
shapes and found a way to get numbers useful to me.

So you might find the percentile values pull additional information out 
of your data points, particularly for noisy results.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me