[ANNOUNCE] Interbench v0.21

[ANNOUNCE] Interbench v0.21

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1032 bytes --]


Interbech is a an application is designed to benchmark interactivity in Linux.

Version 0.21 update

http://ck.kolivas.org/apps/interbench/interbench-0.21.tar.bz2


Changes:

Changed the design to run the benchmarked and background loads as separate 
processes that spawn their own threads instead of everything running as a 
thread of the same process. This was suggested to me originally by Ingo 
Molnar who noticed significant slowdown due to conflict over ->mm->mmap_sem, 
invalidating the benchmark results when run in real time mode. This makes a 
large difference to the latencies measured under mem_load particularly when 
running real time benchmarks on a RT-PREEMPT kernel.

Accounting changes to max_latency to only measure the largest latency of a 
single scheduling frame - this makes max_latency much smaller (and probably 
more realistic). Often you may see max latency exactly one frame wide now 
(consistent with one dropped frame) such as 16.7ms on video.

Minor cleanups.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

RE: [ANNOUNCE] Interbench v0.21

[Al Boldi]

Con Kolivas wrote: {
Version 0.21 update
Changed the design to run the benchmarked and background loads as separate
processes that spawn their own threads instead of everything running as a
thread of the same process.
}

Great!

Re: [ANNOUNCE] Interbench v0.21

[Lee Revell]

On Fri, 2005-07-15 at 14:01 +1000, Con Kolivas wrote:
> his makes a large difference to the latencies measured under mem_load
> particularly when running real time benchmarks on a RT-PREEMPT kernel

Here are some results from my 600MHz C3.  In realtime mode, the
PREEMPT_RT kernel performs as expected, max latencies are around 55
usecs with a very tight distribution.

Based on what we already know about the RT kernel I think this validates
the benchmark.  So the numbers Con posted showing an interactivity
regression from HZ=250 should be taken seriously.

Lee

Realtime mode:

rlrevell@mindpipe:~/kernel-source/interbench-0.21$ ./interbench -r -t 5
84648 loops_per_ms read from file interbench.loops_per_ms

Using 84648 loops per ms, running every load for 10 seconds
Benchmarking kernel 2.6.12-RT-V0.7.51-28 at datestamp 200507160215

--- Benchmarking Audio real time in the presence of loads ---
        Latency +/- SD (us)  Max Latency   % Desired CPU  % Deadlines Met
None         35 +/- 3.85          41             100            100
Video        36 +/- 4.15          51             100            100
X            37 +/- 4.39          49             100            100
Burn         34 +/- 4.19          50             100            100
Write        40 +/- 4.9           52             100            100
Read         38 +/- 3.36          46             100            100
Compile      40 +/- 4.11          54             100            100
Memload      41 +/- 4.24          51             100            100

--- Benchmarking Video real time in the presence of loads ---
        Latency +/- SD (us)  Max Latency   % Desired CPU  % Deadlines Met
None         29 +/- 4.15          42             100            100
X            28 +/- 3.52          46             100            100
Burn         28 +/- 3.37          41             100            100
Write        41 +/- 3.25          59             100            100
Read         37 +/- 3.07          43             100            100
Compile      36 +/- 4.99          54             100            100
Memload      38 +/- 3.39          48             100            100


Non realtime mode:

rlrevell@mindpipe:~/kernel-source/interbench-0.21$ ./interbench -t 5
84648 loops_per_ms read from file interbench.loops_per_ms

Using 84648 loops per ms, running every load for 10 seconds
Benchmarking kernel 2.6.12-RT-V0.7.51-28 at datestamp 200507160237

--- Benchmarking Audio in the presence of loads ---
        Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.042 +/- 0.0234     0.176             100            100
Video     0.121 +/- 0.926         13             100            100
X          1.35 +/- 2.93        19.4             100            100
Burn      0.067 +/- 0.215       3.02             100            100
Write     0.763 +/- 2.18        16.9             100            100
Read      0.263 +/- 1.12        9.01             100            100
Compile   0.216 +/- 1.06         9.2             100            100
Memload   0.541 +/- 1.86        13.1             100            100

--- Benchmarking Video in the presence of loads ---
        Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.428 +/- 1.07          17             100           97.2
X           4.6 +/- 4.23          50             100           68.3
Burn      0.394 +/- 1.18        16.8             100           97.8
Write      1.31 +/- 2.05        39.4             100             93
Read      0.462 +/- 0.809       18.2             100           96.8
Compile   0.991 +/- 1.67        42.3             100           94.2
Memload   0.558 +/- 0.949       18.9             100           97.2

--- Benchmarking X in the presence of loads ---
        Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None      0.599 +/- 0.599         24             100             95
Video      13.3 +/- 13.4          93             100             64
Burn      0.519 +/- 0.52          15             100             94
Write      1.91 +/- 1.91          77             100             91
Read      0.449 +/- 0.45          20             100             95
Compile    1.01 +/- 1.04          30             100             93
Memload    1.26 +/- 1.26          30             100             88

Re: [ANNOUNCE] Interbench v0.21

[Lee Revell]

On Fri, 2005-07-15 at 14:01 +1000, Con Kolivas wrote:
> Interbech is a an application is designed to benchmark interactivity in Linux.
> 
> Version 0.21 update
> 
> http://ck.kolivas.org/apps/interbench/interbench-0.21.tar.bz2
> 

I would suggest using microseconds for both the RT and non RT tests.  It
would allow easier comparison of results.  I have a pretty slow machine
and the max result would only be ~44000 usecs.

Also, if it's run with -r and sched_setscheduler fails, rather than
saying "you must be root for SCHED_FIFO" the error message should
encourage the user to try a 2.6.12+ kernel and add themselves to the
"audio" or "realtime" group, and to file a feature request if their
distro does not support the new realtime rlimit feature.

We should encourage more applications to take advantage of, and distros
to support, the non-root RT scheduling available in 2.6.12+.  I really
think the kernel is good enough at this point that we could achieve
OSX-like multimedia performance on the desktop if more apps like xmms,
xine, and mplayer were to adopt a multithreaded model with the
time-critical rendering threads running RT.  XMMS recently adopted such
a model, but I don't think the audio thread runs SCHED_FIFO yet.  These
benchmarks imply that it would be a massive improvement.

Lee

Re: [ANNOUNCE] Interbench v0.21

[Con Kolivas]

On Sat, 16 Jul 2005 06:41 pm, Lee Revell wrote:
> On Fri, 2005-07-15 at 14:01 +1000, Con Kolivas wrote:
> > Interbech is a an application is designed to benchmark interactivity in
> > Linux.
> >
> > Version 0.21 update
> >
> > http://ck.kolivas.org/apps/interbench/interbench-0.21.tar.bz2
>
> I would suggest using microseconds for both the RT and non RT tests.  It
> would allow easier comparison of results.  I have a pretty slow machine
> and the max result would only be ~44000 usecs.

I think the significance of usec values from the non-rt tests makes this an 
inappropriate thing to do. The variation in results will be greater than usec 
resolution.

> Also, if it's run with -r and sched_setscheduler fails, rather than
> saying "you must be root for SCHED_FIFO" the error message should
> encourage the user to try a 2.6.12+ kernel and add themselves to the
> "audio" or "realtime" group, and to file a feature request if their
> distro does not support the new realtime rlimit feature.
>
> We should encourage more applications to take advantage of, and distros
> to support, the non-root RT scheduling available in 2.6.12+.  I really
> think the kernel is good enough at this point that we could achieve
> OSX-like multimedia performance on the desktop if more apps like xmms,
> xine, and mplayer were to adopt a multithreaded model with the
> time-critical rendering threads running RT.  XMMS recently adopted such
> a model, but I don't think the audio thread runs SCHED_FIFO yet.  These
> benchmarks imply that it would be a massive improvement.

While I agree with you in principal on getting the rlimit feature working and 
supported, this benchmark is meant to be run in single user mode for most 
reproducible and valid results. However, clearly there will be people using 
it cautiously as a normal user first. I originally did not include the 
information that you need to be root in v.20 and said in the documentation 
"need rt privileges" but within about 5 minutes of posting it I had someone 
not understanding what "unable to get SCHED_FIFO" meant. I guess a more 
verbose message will be required explaining non-root RT as well.

Cheers,
Con