[ANNOUNCE] Interbench v0.24

[ANNOUNCE] Interbench v0.24

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1719 bytes --]

Interbench is a Linux Kernel Interactivity Benchmark.

Direct download:
http://ck.kolivas.org/apps/interbench/interbench-0.24.tar.bz2
Web:
http://interbench.kolivas.org

Changes:
3 new loads were added:

Gaming benchmark:
This simulates an unlocked frame rate cpu intensive 3d gaming environment. It 
measures the latencies mean/sd/max and desired cpu percentage only. These 
should give a marker of frame rate stability (latencies), and maximum frame 
rates under different loads (desired cpu percentage). As this simulates an 
unlocked frame rate the deadlines met is meaningless. This does not 
accurately emulate a 3d game which is gpu bound, only a cpu bound one.

Hackbench:
Taken from Rusty's hackbench code as suggested by Ingo Molnar, this will run 
'hackbench 50' repeatedly in the background when benchmarking real time 
performance.

Custom:
Based on the periodic scheduling used for audio/video, custom will allow you 
to specify a cpu percentage and frame rate of a custom workload, and this can 
be used to benchmark this workload's performance under normal scheduling, 
real time scheduling or it can be used as a background load.


Bugfixes:
Numerous floating point and overflow errors were tracked down and fixed. These 
are responsible for results like 'nan' and 4294... which is basically 2^32. 
Unfortunately the standard deviation reported in previous versions appears to 
have been bogus, but fortunately little value was placed on this result.

Error handling was made _much_ more robust - for example it was found that 
contrary to 'man sem_wait' but consistent with SUSv3, sem_wait can return -1 
with -EINTR.

Lots of little tweaks.


Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ck] [ANNOUNCE] Interbench v0.24

[Con Kolivas]

On Fri, 29 Jul 2005 21:11, Gabriel Devenyi wrote:
> Hello Con,
>
> Attempting to run this on my 2.6.12-ck3s system results in the following
> error:
>
> sawtooth interbench-0.24 # ./interbench
> loops_per_ms unknown; benchmarking...
> 690936 loops_per_ms saved to file interbench.loops_per_ms
>
> Could not get memory or swap size
>
> Bug perhaps? My configuration hasn't changed since interbench 0.23 AFAIK.

No swap?

Cheers,
Con

Re: [ck] [ANNOUNCE] Interbench v0.24

[Gabriel Devenyi]

Thats correct, does it need it?

--
Gabriel Devenyi
ace@staticwave.ca

Con Kolivas wrote:
> On Fri, 29 Jul 2005 21:11, Gabriel Devenyi wrote:
> 
>>Hello Con,
>>
>>Attempting to run this on my 2.6.12-ck3s system results in the following
>>error:
>>
>>sawtooth interbench-0.24 # ./interbench
>>loops_per_ms unknown; benchmarking...
>>690936 loops_per_ms saved to file interbench.loops_per_ms
>>
>>Could not get memory or swap size
>>
>>Bug perhaps? My configuration hasn't changed since interbench 0.23 AFAIK.
> 
> 
> No swap?
> 
> Cheers,
> Con
> 

Re: [ck] [ANNOUNCE] Interbench v0.24

[Con Kolivas]

On Fri, 29 Jul 2005 21:47, Gabriel Devenyi wrote:
> Thats correct, does it need it?

For the memload, yes it does. I guess for the next version I can drop memload 
if it can't read the swap information rather than not running at all.

Cheers,
Con

> Con Kolivas wrote:
> > On Fri, 29 Jul 2005 21:11, Gabriel Devenyi wrote:
> >>Hello Con,
> >>
> >>Attempting to run this on my 2.6.12-ck3s system results in the following
> >>error:
> >>
> >>sawtooth interbench-0.24 # ./interbench
> >>loops_per_ms unknown; benchmarking...
> >>690936 loops_per_ms saved to file interbench.loops_per_ms
> >>
> >>Could not get memory or swap size
> >>
> >>Bug perhaps? My configuration hasn't changed since interbench 0.23 AFAIK.
> >
> > No swap?
> >
> > Cheers,
> > Con

Re: [ANNOUNCE] Interbench v0.24

[Peter Williams]

Con Kolivas wrote:
> Interbench is a Linux Kernel Interactivity Benchmark.
> 
> Direct download:
> http://ck.kolivas.org/apps/interbench/interbench-0.24.tar.bz2
> Web:
> http://interbench.kolivas.org
> 
> Changes:
> 3 new loads were added:
> 
> Gaming benchmark:
> This simulates an unlocked frame rate cpu intensive 3d gaming environment. It 
> measures the latencies mean/sd/max and desired cpu percentage only. These 
> should give a marker of frame rate stability (latencies), and maximum frame 
> rates under different loads (desired cpu percentage). As this simulates an 
> unlocked frame rate the deadlines met is meaningless. This does not 
> accurately emulate a 3d game which is gpu bound, only a cpu bound one.
> 
> Hackbench:
> Taken from Rusty's hackbench code as suggested by Ingo Molnar, this will run 
> 'hackbench 50' repeatedly in the background when benchmarking real time 
> performance.
> 
> Custom:
> Based on the periodic scheduling used for audio/video, custom will allow you 
> to specify a cpu percentage and frame rate of a custom workload, and this can 
> be used to benchmark this workload's performance under normal scheduling, 
> real time scheduling or it can be used as a background load.
> 
> 
> Bugfixes:
> Numerous floating point and overflow errors were tracked down and fixed. These 
> are responsible for results like 'nan' and 4294... which is basically 2^32. 
> Unfortunately the standard deviation reported in previous versions appears to 
> have been bogus, but fortunately little value was placed on this result.
> 
> Error handling was made _much_ more robust - for example it was found that 
> contrary to 'man sem_wait' but consistent with SUSv3, sem_wait can return -1 
> with -EINTR.
> 
> Lots of little tweaks.

I've just been perusing the code and noticed that there is a bug in 
calculating the latency standard deviation caused by the fact that the 
latency that is inserted into the samples array is not necessarily the 
same as that added to total_latency and could be quite a bit larger.  So 
either the means are too small or the standard deviations are too large.

BTW, there's a method for calculating variances and means that avoids 
the need to keep an array of samples.  Basically, in addition to the sum 
of the samples (sum_x, say) you keep a sum of the squares of the samples 
(sum_x_sq, say) and the number of samples (n, say).  Then:

mean = sum_x / n
variance = (sum_x_sq - (mean * mean) / n) / (n - 1)
standard deviation = sqrt(variance)

Without the need to keep the array of samples there's no need worry 
about an arbitrary upper limit on the number of samples.

Peter
-- 
Peter Williams                                   pwil3058@bigpond.net.au

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce

Re: [ANNOUNCE] Interbench v0.24

[Con Kolivas]

On Tue, 2 Aug 2005 05:08 pm, Peter Williams wrote:
> Con Kolivas wrote:
> > Interbench is a Linux Kernel Interactivity Benchmark.
> >
> > Direct download:
> > http://ck.kolivas.org/apps/interbench/interbench-0.24.tar.bz2
> > Web:
> > http://interbench.kolivas.org
> >
> > Changes:
> > 3 new loads were added:
> >
> > Gaming benchmark:
> > This simulates an unlocked frame rate cpu intensive 3d gaming
> > environment. It measures the latencies mean/sd/max and desired cpu
> > percentage only. These should give a marker of frame rate stability
> > (latencies), and maximum frame rates under different loads (desired cpu
> > percentage). As this simulates an unlocked frame rate the deadlines met
> > is meaningless. This does not accurately emulate a 3d game which is gpu
> > bound, only a cpu bound one.
> >
> > Hackbench:
> > Taken from Rusty's hackbench code as suggested by Ingo Molnar, this will
> > run 'hackbench 50' repeatedly in the background when benchmarking real
> > time performance.
> >
> > Custom:
> > Based on the periodic scheduling used for audio/video, custom will allow
> > you to specify a cpu percentage and frame rate of a custom workload, and
> > this can be used to benchmark this workload's performance under normal
> > scheduling, real time scheduling or it can be used as a background load.
> >
> >
> > Bugfixes:
> > Numerous floating point and overflow errors were tracked down and fixed.
> > These are responsible for results like 'nan' and 4294... which is
> > basically 2^32. Unfortunately the standard deviation reported in previous
> > versions appears to have been bogus, but fortunately little value was
> > placed on this result.
> >
> > Error handling was made _much_ more robust - for example it was found
> > that contrary to 'man sem_wait' but consistent with SUSv3, sem_wait can
> > return -1 with -EINTR.
> >
> > Lots of little tweaks.
>
> I've just been perusing the code and noticed that there is a bug in
> calculating the latency standard deviation caused by the fact that the
> latency that is inserted into the samples array is not necessarily the
> same as that added to total_latency and could be quite a bit larger.  So
> either the means are too small or the standard deviations are too large.
>
> BTW, there's a method for calculating variances and means that avoids
> the need to keep an array of samples.  Basically, in addition to the sum
> of the samples (sum_x, say) you keep a sum of the squares of the samples
> (sum_x_sq, say) and the number of samples (n, say).  Then:
>
> mean = sum_x / n
> variance = (sum_x_sq - (mean * mean) / n) / (n - 1)
> standard deviation = sqrt(variance)
>
> Without the need to keep the array of samples there's no need worry
> about an arbitrary upper limit on the number of samples.

Lovely! Thank you very much as that was causing me much grief indeed and your 
approach simplifies the code greatly.

Cheers,
Con