[ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 13699 bytes --]

	Interbench - The Linux Interactivity Benchmark v0.20

http://interbench.kolivas.org

direct download link:
http://ck.kolivas.org/apps/interbench/interbench-0.20.tar.bz2

	Introduction

This benchmark application is designed to benchmark interactivity in Linux.

	Interactivity, what is it?

There has been a lot of talk about what makes up a nice feeling desktop under
linux. It comes down to two different but intimately related parameters which
are not well defined. We often use the terms responsiveness and interactivity
in the same sentence, but I'd like to separate the two. As there is no formal
definition I prefer to define them as such:

Responsiveness: The rate at which your workloads can proceed under different
load conditions.

Interactivity: The scheduling latency and jitter present in tasks where the 
user would notice a palpable deterioration under different load conditions.

Responsiveness would allow you to continue using your machine without too much
interruption to your work, whereas interactivity would allow you to play audio
or video without any dropouts, or drag a gui window across the screen and have
it render smoothly across the screen without jerks .

Contest was a benchmark originally written by me to test system 
responsiveness, and interbench is a benchmark I wrote as a sequel to contest 
to test interactivity.

It is designed to measure the effect of changes in Linux kernel design or 
system
configuration changes such as cpu, I/O scheduler and filesystem changes and
options. With careful benchmarking, different hardware can be compared.


	What does it do?

It is designed to emulate the cpu scheduling behaviour of interactive tasks 
and
measure their scheduling latency and jitter. It does this with the tasks on
their own and then in the presence of various background loads, both with
configurable nice levels and the benchmarked tasks can be real time.


	How does it work?

First it benchmarks how best to reproduce a fixed percentage of cpu usage on 
the machine currently being used for the benchmark. It saves this to a file 
and then uses this for all subsequent runs to keep the emulation of cpu usage 
constant.

It runs a real time high priority timing thread that wakes up the thread or
threads of the simulated interactive tasks and then measures the latency in 
the time taken to schedule. As there is no accurate timer driven scheduling 
in linux the timing thread sleeps as accurately as linux kernel supports, and 
latency is considered as the time from this sleep till the simulated task 
gets scheduled.


	What interactive tasks are simulated and how?

X:
X is simulated as a thread that uses a variable amount of cpu ranging from 0 
to 100%. This simulates an idle gui where a window is grabbed and then 
dragged across the screen.

Audio:
Audio is simulated as a thread that tries to run at 50ms intervals that then
requires 5% cpu. This behaviour ignores any caching that would normally be 
done by well designed audio applications, but has been seen as the interval 
used to write to audio cards by a popular linux audio player. It also ignores 
any of the effects of different audio drivers and audio cards. Audio can also 
be run as a real time SCHED_FIFO task.

Video:
Video is simulated as a thread that tries to receive cpu 60 times per second
and uses 40% cpu. This would be quite a demanding video playback at 60fps. 
Like the audio simulator it ignores caching, drivers and video cards. As per 
audio, video can be run SCHED_FIFO.


	What loads are simulated?

None:
Otherwise idle system.

Video:
The video simulation thread is also used as a background load.

X:
The X simulation thread is used as a load.

Burn:
A configurable number of threads fully cpu bound (4 by default).

Write:
A streaming write to disk repeatedly of a file the size of physical ram.

Read:
Repeatedly reading a file from disk the size of physical ram (to avoid any
caching effects).

Compile:
Simulating a heavy 'make -j4' compilation by running Burn, Write and Read
concurrently.

Memload:
Simulating heavy memory and swap pressure by repeatedly accessing 110% of
available ram and moving it around and freeing it.


	What is measured and what does it mean?

1. The average scheduling latency (time to requesting cpu till actually 
getting
it) of deadlines met during the test period. 
2. The scheduling jitter is represented by calculating the standard deviation
of the latency
3. The maximum latency seen during the test period
4. Percentage of desired cpu
5. Percentage of deadlines met.

This data is output to console and saved to a file which is stamped with the
kernel name and date. Use fixed font for clarity:

	Sample:
--- Benchmarking X in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.495 +/- 0.495         45		 100	         96
Video	   11.7 +/- 11.7        1815		89.6	       62.7
Burn	   27.9 +/- 28.1        3335		78.5	         44
Write	   4.02 +/- 4.03         372		  97	       78.7
Read	   1.09 +/- 1.09         158		99.7	         88
Compile	   28.8 +/- 28.8        3351		78.2	       43.7
Memload	   2.81 +/- 2.81         187		98.7	         85

What can be seen here is that never during this test run were all the so 
called deadlines met by the X simulator, although all the desired cpu was 
achieved under no load. In X terms this means that every bit of window 
movement was drawn while moving the window, but some were delayed and there 
was enough time to catch up before the next deadline. In the 'Burn' column we 
can see that only 44% of the deadlines were met, and only 78.5% of the 
desired cpu was achieved. This means that some deadlines were so late 
(%deadlines met was low) that some redraws were dropped entirely to catch up. 
In X terms this would translate into jerky movement, in audio it would be a 
skip, and in video it would be a dropped frame. Note that despite the massive 
maximum latency of >3seconds, the average latency is still less than 30ms. 
This is because redraws are dropped in order to catch up usually by these 
sorts of applications.


	What is relevant in the data?

The results pessimise quite a lot what happens in real world terms because 
they ignore the reality of buffering, but this allows us to pick up subtle 
differences more readily. In terms of what would be noticed by the end user,
dropping deadlines would make noticable clicks in audio, subtle visible frame
time delays in video, and loss of "smooth" movement in X. Dropping desired cpu
would be much more noticeable with audio skips, missed video frames or jerks
in window movement under X. The magnitude of these would be best represented 
by the maximum latency. When the deadlines are actually met, the average 
latency represents how "smooth" it would look. Average humans' limit of 
perception for jitter is in the order of 7ms. Trained audio observers might 
notice much less.


	How to use it?

In response to critisicm of difficulty in setting up my previous benchmark, 
contest, I've made this as simple as possible.

	Short version:
make
./interbench

	Longer version:
Build with 'make'. It is a single executable once built so if you desire to
install it simply copy the interbench binary wherever you like.

To get good reproducible data from it you should boot into runlevel one so
that nothing else is running on the machine. All power saving (cpu throttling,
cpu frequency modifications) must be disabled on the first run to get an
accurate measurement for cpu usage. You may enable them later if you are
benchmarking their effect on interactivity on that machine. Root is almost
mandatory for this benchmark, or real time privileges at the very least. You
need free disk space in the directory it is being run in the order of 2* your
physical ram for the disk loads. A default run in v0.20 takes about 15
minutes to complete, longer if your disk is slow.

Command line options supported:
interbench [-l <int>] [-L <int>] [-t <int] [-B <int>] [-N <int>] [-b] [-c] 
[-h] [-n] [-r]
 -l     Use <int> loops per sec (default: use saved benchmark)
 -L     Use cpu load of <int> with burn load (default: 4)
 -t     Seconds to run each benchmark (default: 30)
 -B     Nice the benchmarked thread to <int> (default: 0)
 -N     Nice the load thread to <int> (default: 0)
 -b     Benchmark loops_per_ms even if it is already known
 -c     Output to console only (default: use console and logfile)
 -r     Perform real time scheduling benchmarks (default: non-rt)
 -h     Show this help

There is one hidden option which is not supported by default, -u
which emulates a uniprocessor when run on an smp machine. The support for cpu
affinity is not built in by default because there are multiple versions of
the sched_setaffinity call in glibc that not only accept different variable
types but across architectures take different numbers of arguments. For x86
support you can change the '#if 0' in interbench.c to '#if 1' to enable the
affinity support to be built in. The function on x86_64 for those very keen
does not have the sizeof argument.


So how does -ck perform? As much as I'd like to say it was a walkover I have 
to admit you need to squint hard to be convinced that -ck is better overall. 
Both mainline and -ck perform better in different load settings:

The SCHED_NORMAL nice 0 runs are as below, performed on a pentium M 1.7Ghz:

Benchmarking kernel 2.6.13-rc1 with datestamp 200507121411

--- Benchmarking Audio in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.003 +/- 0          0.005		 100	        100
Video	   1.02 +/- 0.487       1.68		 100	        100
X	   1.32 +/- 2.22          10		 100	        100
Burn	  0.518 +/- 306004        52		 100	         99
Write	  0.031 +/- 0.209       2.58		 100	        100
Read	  0.006 +/- 0.00173     0.01		 100	        100
Compile	   4.59 +/- 5.74         426		96.5	         94
Memload	  0.021 +/- 0.0697     0.659		 100	        100

--- Benchmarking Video in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.003 +/- 0          0.005		 100	        100
X	   3.27 +/- 3.2         41.3		88.8	       77.7
Burn	  0.003 +/- 0.001      0.005		 100	        100
Write	  0.151 +/- 0.67          50		99.5	         99
Read	  0.004 +/- 0.00173    0.037		 100	        100
Compile	  0.025 +/- 0.248       4.81		 100	        100
Memload	  0.018 +/- 0.0572     0.715		 100	        100

--- Benchmarking X in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.009 +/- 0.0966         1		 100	         99
Video	   4.46 +/- 4.43         572		91.9	         66
Burn	   1.58 +/- 1.58         156		 100	         98
Write	  0.002 +/- 0.0237         4		 100	         98
Read	  0.008 +/- 0.0797        15		 100	         96
Compile	  0.009 +/- 0.0896         2		 100	         99
Memload	  0.108 +/- 0.13          10		 100	         98


Benchmarking kernel 2.6.12-rc6-ck1 with datestamp 200507121345

--- Benchmarking Audio in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.003 +/- 0          0.005		 100	        100
Video	  0.003 +/- 0          0.004		 100	        100
X	   2.53 +/- 3.01          11		 100	        100
Burn	  0.294 +/- 1.47          11		 100	        100
Write	  0.025 +/- 0.116       1.02		 100	        100
Read	  0.007 +/- 0.001       0.01		 100	        100
Compile	  0.393 +/- 1.68          11		 100	        100
Memload	  0.095 +/- 0.545          6		 100	        100

--- Benchmarking Video in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.003 +/- 0.00245    0.052		 100	        100
X	   3.57 +/- 3.21        22.7		95.7	       91.3
Burn	  0.837 +/- 2.49          50		97.7	       95.5
Write	  0.094 +/- 0.596       16.7		 100	       99.8
Read	  0.005 +/- 0.00872    0.169		 100	        100
Compile	  0.543 +/- 1.91        33.3		98.8	       97.7
Memload	   0.21 +/- 0.836       16.7		99.7	       99.3

--- Benchmarking X in the presence of loads ---
	Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None	  0.009 +/- 0.0964         1		 100	         99
Video	   2.31 +/- 2.27         754		90.9	         65
Burn	  0.129 +/- 0.151         12		 100	         98
Write	  0.069 +/- 0.112          6		 100	         98
Read	  0.009 +/- 0.0896         1		 100	         99
Compile	  0.039 +/- 0.102          3		 100	         98
Memload	  0.004 +/- 0.0408         1		 100	         99


The full logs are available here (including niced runs and real time runs):
http://ck.kolivas.org/apps/interbench/2.6.13-rc1.log
http://ck.kolivas.org/apps/interbench/2.6.12-rc6-ck1.log

Thanks:
For help from Zwane Mwaikambo, Bert Hubert, Seth Arnold, Rik Van Riel,  
Nicholas Miell and John Levon. Aggelos Economopoulos for contest code, and
Bob Matthews for irman (mem_load) code.

This was quite some time in the making... I realise there's so much more that 
could be done trying to simulate the interactive tasks and the loads, but 
this is a start, it's quite standardised and the results are reproducible. 
Adding more code to simulate loads and threads to benchmark is quite easy if 
someone wishes to suggest or code up something I'm all ears. Of course 
bugfixes, comments and suggestions are most welcome.

Cheers,
Con Kolivas

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[David Lang]

this looks very interesting, however one thing that looks odd to me in 
this is the thought of comparing the results for significantly different 
hardware.

for some of the loads you really are going to be independant of the speed 
of the hardware (burn, compile, etc will use whatever you have) however 
for others (X, audio, video) saying that they take a specific percentage 
of the cpu doesn't seem right.

if I have a 400MHz cpu each of these will take a much larger percentage of 
the cpu to get the job done then if I have a 4GHz cpu for example.

for audio and video this would seem to be a fairly simple scaleing factor 
(or just doing a fixed amount of work rather then a fixed percentage of 
the CPU worth of work), however for X it is probably much more complicated 
(is the X load really linearly random in how much work it does, or is it 
weighted towards small amounts with occasional large amounts hitting? I 
would guess that at least beyond a certin point the liklyhood of that much 
work being needed would be lower)

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1423 bytes --]

On Tue, 12 Jul 2005 21:57, David Lang wrote:
> this looks very interesting, however one thing that looks odd to me in
> this is the thought of comparing the results for significantly different
> hardware.
>
> for some of the loads you really are going to be independant of the speed
> of the hardware (burn, compile, etc will use whatever you have) however
> for others (X, audio, video) saying that they take a specific percentage
> of the cpu doesn't seem right.
>
> if I have a 400MHz cpu each of these will take a much larger percentage of
> the cpu to get the job done then if I have a 4GHz cpu for example.
>
> for audio and video this would seem to be a fairly simple scaleing factor
> (or just doing a fixed amount of work rather then a fixed percentage of
> the CPU worth of work), however for X it is probably much more complicated
> (is the X load really linearly random in how much work it does, or is it
> weighted towards small amounts with occasional large amounts hitting? I
> would guess that at least beyond a certin point the liklyhood of that much
> work being needed would be lower)

Actually I don't disagree. What I mean by hardware changes is more along the 
lines of changing the hard disk type in the same setup. That's what I mean by 
careful with the benchmarking. Taking the results from an athlon XP and 
comparing it to an altix is silly for example.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[David Lang]

On Tue, 12 Jul 2005, Con Kolivas wrote:

> 
> On Tue, 12 Jul 2005 21:57, David Lang wrote:
>> this looks very interesting, however one thing that looks odd to me in
>> this is the thought of comparing the results for significantly different
>> hardware.
>>
>> for some of the loads you really are going to be independant of the speed
>> of the hardware (burn, compile, etc will use whatever you have) however
>> for others (X, audio, video) saying that they take a specific percentage
>> of the cpu doesn't seem right.
>>
>> if I have a 400MHz cpu each of these will take a much larger percentage of
>> the cpu to get the job done then if I have a 4GHz cpu for example.
>>
>> for audio and video this would seem to be a fairly simple scaleing factor
>> (or just doing a fixed amount of work rather then a fixed percentage of
>> the CPU worth of work), however for X it is probably much more complicated
>> (is the X load really linearly random in how much work it does, or is it
>> weighted towards small amounts with occasional large amounts hitting? I
>> would guess that at least beyond a certin point the liklyhood of that much
>> work being needed would be lower)
>
> Actually I don't disagree. What I mean by hardware changes is more along the
> lines of changing the hard disk type in the same setup. That's what I mean by
> careful with the benchmarking. Taking the results from an athlon XP and
> comparing it to an altix is silly for example.
>

however if you defined these loads in terms of the amount of work (number 
of loops) rather then percentage of the CPU I think you would be pretty 
close (especially if there was a way to tweak these for a test if needed). 
This sort of testing would be especially useful for low-end/embedded 
applications.

for example a series 1 DirectTv tivo manages to write two program streams 
to disk while reading and viewing a third, all on a 75MHz PPC with 64M of 
ram (on a tweaked 2.1 kernel) you are pretty close to being able to 
simulate this load

which brings up another possible config option/test case, changing the 
read/write tests to try to do X MB/sec rather then the max possible speed 
(probably defaulting to max if nothing is specified)

thanks again for working to define a good test case

David Lang


-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 518 bytes --]

On Tue, 12 Jul 2005 22:17, David Lang wrote:
> which brings up another possible config option/test case, changing the
> read/write tests to try to do X MB/sec rather then the max possible speed
> (probably defaulting to max if nothing is specified)

That's a good idea. I was planning on adding a configurable cpu%/interval 
benchmark as well so configurable read/write is a logical addition. 

> thanks again for working to define a good test case

You're welcome, and thanks for feedback.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Lee Revell]

On Tue, 2005-07-12 at 05:17 -0700, David Lang wrote:
> for example a series 1 DirectTv tivo manages to write two program
> streams to disk while reading and viewing a third

Actually it writes two streams to disk while reading and viewing one of
them, unless they released a model with three tuners...

Lee

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[David Lang]

On Tue, 12 Jul 2005, Lee Revell wrote:

> On Tue, 2005-07-12 at 05:17 -0700, David Lang wrote:
>> for example a series 1 DirectTv tivo manages to write two program
>> streams to disk while reading and viewing a third
>
> Actually it writes two streams to disk while reading and viewing one of
> them, unless they released a model with three tuners...

it can be recording two shows (one per tuner) while playing a completely 
different show. you don't have to be watching one of the two that are 
being recorded.

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Lee Revell]

On Tue, 2005-07-12 at 10:55 -0700, David Lang wrote:
> On Tue, 12 Jul 2005, Lee Revell wrote:
> 
> > On Tue, 2005-07-12 at 05:17 -0700, David Lang wrote:
> >> for example a series 1 DirectTv tivo manages to write two program
> >> streams to disk while reading and viewing a third
> >
> > Actually it writes two streams to disk while reading and viewing one of
> > them, unless they released a model with three tuners...
> 
> it can be recording two shows (one per tuner) while playing a completely 
> different show. you don't have to be watching one of the two that are 
> being recorded.

Heh, of course it can, I realized this after I hit send...  You would
think I'd remember that after watching like a million hours of TV on the
thing.

Lee

RE: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Al Boldi]

On Tue, 12 Jul 2005, Con Kolivas wrote:
> It runs a real time high priority timing thread that wakes up the thread

Nice, but why is it threaded?

Forking would be more realistic!

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 663 bytes --]

On Wed, 13 Jul 2005 06:55, Al Boldi wrote:
> On Tue, 12 Jul 2005, Con Kolivas wrote:
> > It runs a real time high priority timing thread that wakes up the thread
>
> Nice, but why is it threaded?

Because I'm an amateur, and I had to start somewhere.

> Forking would be more realistic!

Something for the future would be to fork the background workload separately 
from the benchmarked workload. Having them all in the same mm does appear to 
cause other interactions which pollute the real time data as Ingo has pointed 
out to me privately.

I'm a little burnt out from the effort so far so it can wait.

Thanks for feedback.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Bill Davidsen]

Con Kolivas wrote:
> On Tue, 12 Jul 2005 21:57, David Lang wrote:
> 
>>this looks very interesting, however one thing that looks odd to me in
>>this is the thought of comparing the results for significantly different
>>hardware.
>>
>>for some of the loads you really are going to be independant of the speed
>>of the hardware (burn, compile, etc will use whatever you have) however
>>for others (X, audio, video) saying that they take a specific percentage
>>of the cpu doesn't seem right.
>>
>>if I have a 400MHz cpu each of these will take a much larger percentage of
>>the cpu to get the job done then if I have a 4GHz cpu for example.
>>
>>for audio and video this would seem to be a fairly simple scaleing factor
>>(or just doing a fixed amount of work rather then a fixed percentage of
>>the CPU worth of work), however for X it is probably much more complicated
>>(is the X load really linearly random in how much work it does, or is it
>>weighted towards small amounts with occasional large amounts hitting? I
>>would guess that at least beyond a certin point the liklyhood of that much
>>work being needed would be lower)
> 
> 
> Actually I don't disagree. What I mean by hardware changes is more along the 
> lines of changing the hard disk type in the same setup. That's what I mean by 
> careful with the benchmarking. Taking the results from an athlon XP and 
> comparing it to an altix is silly for example.

I'm going to cautiously disagree. If the CPU needed was scaled so it 
represented a fixed number of cycles (operations, work units) then the 
effect of faster CPU would be shown. And the total power of all attached 
CPUs should be taken into account, using HT or SMP does have an effect 
of feel.

Disk tests should be at a fixed rate, not all you can do. That's NOT 
realistic.

-- 
    -bill davidsen (davidsen@tmr.com)
"The secret to procrastination is to put things off until the
  last possible moment - but no longer"  -me

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 2195 bytes --]

On Thu, 14 Jul 2005 03:54, Bill Davidsen wrote:
> Con Kolivas wrote:
> > On Tue, 12 Jul 2005 21:57, David Lang wrote:
> >>for audio and video this would seem to be a fairly simple scaleing factor
> >>(or just doing a fixed amount of work rather then a fixed percentage of
> >>the CPU worth of work), however for X it is probably much more
> >> complicated (is the X load really linearly random in how much work it
> >> does, or is it weighted towards small amounts with occasional large
> >> amounts hitting? I would guess that at least beyond a certin point the
> >> liklyhood of that much work being needed would be lower)
> >
> > Actually I don't disagree. What I mean by hardware changes is more along
> > the lines of changing the hard disk type in the same setup. That's what I
> > mean by careful with the benchmarking. Taking the results from an athlon
> > XP and comparing it to an altix is silly for example.
>
> I'm going to cautiously disagree. If the CPU needed was scaled so it
> represented a fixed number of cycles (operations, work units) then the
> effect of faster CPU would be shown. And the total power of all attached
> CPUs should be taken into account, using HT or SMP does have an effect
> of feel.

That is rather hard to do because each architecture's interpretation of fixed 
number of cycles is different and this doesn't represent their speed in the 
real world. The calculation when interbench is first run to see how many 
"loops per ms" took quite a bit of effort to find just how many loops each 
different cpu would do per ms and then find a way to make that not change 
through compiler optimised code. The "loops per ms" parameter did not end up 
being proportional to cpu Mhz except on the same cpu type.

> Disk tests should be at a fixed rate, not all you can do. That's NOT
> realistic.

Not true; what you suggest is another thing to check entirely, and that would 
be a valid benchmark too. What I'm interested in is what happens if you read 
or write a DVD ISO image for example to your hard disk and what this does to 
interactivity. This sort of reading or writing is not throttled in real life.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[David Lang]

On Thu, 14 Jul 2005, Con Kolivas wrote:

> On Thu, 14 Jul 2005 03:54, Bill Davidsen wrote:
>> Con Kolivas wrote:
>>> On Tue, 12 Jul 2005 21:57, David Lang wrote:
>>>> for audio and video this would seem to be a fairly simple scaleing factor
>>>> (or just doing a fixed amount of work rather then a fixed percentage of
>>>> the CPU worth of work), however for X it is probably much more
>>>> complicated (is the X load really linearly random in how much work it
>>>> does, or is it weighted towards small amounts with occasional large
>>>> amounts hitting? I would guess that at least beyond a certin point the
>>>> liklyhood of that much work being needed would be lower)
>>>
>>> Actually I don't disagree. What I mean by hardware changes is more along
>>> the lines of changing the hard disk type in the same setup. That's what I
>>> mean by careful with the benchmarking. Taking the results from an athlon
>>> XP and comparing it to an altix is silly for example.
>>
>> I'm going to cautiously disagree. If the CPU needed was scaled so it
>> represented a fixed number of cycles (operations, work units) then the
>> effect of faster CPU would be shown. And the total power of all attached
>> CPUs should be taken into account, using HT or SMP does have an effect
>> of feel.
>
> That is rather hard to do because each architecture's interpretation of fixed
> number of cycles is different and this doesn't represent their speed in the
> real world. The calculation when interbench is first run to see how many
> "loops per ms" took quite a bit of effort to find just how many loops each
> different cpu would do per ms and then find a way to make that not change
> through compiler optimised code. The "loops per ms" parameter did not end up
> being proportional to cpu Mhz except on the same cpu type.

right, but the amount of cpu required to do a specific task will also vary 
significantly between CPU families for the same task as well. as long as 
the loops don't get optimized away by the compiler I think you can setup 
some loops to do the same work on each CPU, even if they take 
significantly different amounts of time (as an off-the-wall, obviously 
untested example you could make your 'loop' be a calculation of Pi and for 
the 'audio' test you compute the first 100 digits, for the video test you 
compute the first 1000 digits, and for the X test you compute a random 
number of digits between 10 and 10000)

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 3151 bytes --]

On Thu, 14 Jul 2005 10:31, David Lang wrote:
> On Thu, 14 Jul 2005, Con Kolivas wrote:
> > On Thu, 14 Jul 2005 03:54, Bill Davidsen wrote:
> >> Con Kolivas wrote:
> >>> On Tue, 12 Jul 2005 21:57, David Lang wrote:
> >>>> for audio and video this would seem to be a fairly simple scaleing
> >>>> factor (or just doing a fixed amount of work rather then a fixed
> >>>> percentage of the CPU worth of work), however for X it is probably
> >>>> much more complicated (is the X load really linearly random in how
> >>>> much work it does, or is it weighted towards small amounts with
> >>>> occasional large amounts hitting? I would guess that at least beyond a
> >>>> certin point the liklyhood of that much work being needed would be
> >>>> lower)
> >>>
> >>> Actually I don't disagree. What I mean by hardware changes is more
> >>> along the lines of changing the hard disk type in the same setup.
> >>> That's what I mean by careful with the benchmarking. Taking the results
> >>> from an athlon XP and comparing it to an altix is silly for example.
> >>
> >> I'm going to cautiously disagree. If the CPU needed was scaled so it
> >> represented a fixed number of cycles (operations, work units) then the
> >> effect of faster CPU would be shown. And the total power of all attached
> >> CPUs should be taken into account, using HT or SMP does have an effect
> >> of feel.
> >
> > That is rather hard to do because each architecture's interpretation of
> > fixed number of cycles is different and this doesn't represent their
> > speed in the real world. The calculation when interbench is first run to
> > see how many "loops per ms" took quite a bit of effort to find just how
> > many loops each different cpu would do per ms and then find a way to make
> > that not change through compiler optimised code. The "loops per ms"
> > parameter did not end up being proportional to cpu Mhz except on the same
> > cpu type.
>
> right, but the amount of cpu required to do a specific task will also vary
> significantly between CPU families for the same task as well. as long as
> the loops don't get optimized away by the compiler I think you can setup
> some loops to do the same work on each CPU, even if they take
> significantly different amounts of time (as an off-the-wall, obviously
> untested example you could make your 'loop' be a calculation of Pi and for
> the 'audio' test you compute the first 100 digits, for the video test you
> compute the first 1000 digits, and for the X test you compute a random
> number of digits between 10 and 10000)

Once again I don't disagree, and the current system of loops_per_ms does 
exactly that and can be simply used as a fixed number of loops already. My 
point was there'd be argument about what sort of "loop" or load should be 
used as each cpu type would do different "loops" faster and they won't 
necessarily represent video, audio or X in the real world. Currently the loop 
in interbench is simply:
	for (i = 0 ; i < loops ; i++)
	     asm volatile("" : : : "memory");

and if noone argues i can use that for fixed workload.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 3489 bytes --]

On Thu, 14 Jul 2005 10:46, Con Kolivas wrote:
> On Thu, 14 Jul 2005 10:31, David Lang wrote:
> > On Thu, 14 Jul 2005, Con Kolivas wrote:
> > > On Thu, 14 Jul 2005 03:54, Bill Davidsen wrote:
> > >> Con Kolivas wrote:
> > >>> On Tue, 12 Jul 2005 21:57, David Lang wrote:
> > >>>> for audio and video this would seem to be a fairly simple scaleing
> > >>>> factor (or just doing a fixed amount of work rather then a fixed
> > >>>> percentage of the CPU worth of work), however for X it is probably
> > >>>> much more complicated (is the X load really linearly random in how
> > >>>> much work it does, or is it weighted towards small amounts with
> > >>>> occasional large amounts hitting? I would guess that at least beyond
> > >>>> a certin point the liklyhood of that much work being needed would be
> > >>>> lower)
> > >>>
> > >>> Actually I don't disagree. What I mean by hardware changes is more
> > >>> along the lines of changing the hard disk type in the same setup.
> > >>> That's what I mean by careful with the benchmarking. Taking the
> > >>> results from an athlon XP and comparing it to an altix is silly for
> > >>> example.
> > >>
> > >> I'm going to cautiously disagree. If the CPU needed was scaled so it
> > >> represented a fixed number of cycles (operations, work units) then the
> > >> effect of faster CPU would be shown. And the total power of all
> > >> attached CPUs should be taken into account, using HT or SMP does have
> > >> an effect of feel.
> > >
> > > That is rather hard to do because each architecture's interpretation of
> > > fixed number of cycles is different and this doesn't represent their
> > > speed in the real world. The calculation when interbench is first run
> > > to see how many "loops per ms" took quite a bit of effort to find just
> > > how many loops each different cpu would do per ms and then find a way
> > > to make that not change through compiler optimised code. The "loops per
> > > ms" parameter did not end up being proportional to cpu Mhz except on
> > > the same cpu type.
> >
> > right, but the amount of cpu required to do a specific task will also
> > vary significantly between CPU families for the same task as well. as
> > long as the loops don't get optimized away by the compiler I think you
> > can setup some loops to do the same work on each CPU, even if they take
> > significantly different amounts of time (as an off-the-wall, obviously
> > untested example you could make your 'loop' be a calculation of Pi and
> > for the 'audio' test you compute the first 100 digits, for the video test
> > you compute the first 1000 digits, and for the X test you compute a
> > random number of digits between 10 and 10000)
>
> Once again I don't disagree, and the current system of loops_per_ms does
> exactly that and can be simply used as a fixed number of loops already. My
> point was there'd be argument about what sort of "loop" or load should be
> used as each cpu type would do different "loops" faster and they won't
> necessarily represent video, audio or X in the real world. Currently the
> loop in interbench is simply:
> 	for (i = 0 ; i < loops ; i++)
> 	     asm volatile("" : : : "memory");
>
> and if noone argues i can use that for fixed workload.

What I mean is if you take the loops_per_ms from one machine and plug it in 
using the -l option on another you can do it now without any modification to 
the interbench code.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Bill Davidsen]

Con Kolivas wrote:

>On Thu, 14 Jul 2005 03:54, Bill Davidsen wrote:
>  
>
>>Con Kolivas wrote:
>>    
>>
>>>On Tue, 12 Jul 2005 21:57, David Lang wrote:
>>>      
>>>
>>>>for audio and video this would seem to be a fairly simple scaleing factor
>>>>(or just doing a fixed amount of work rather then a fixed percentage of
>>>>the CPU worth of work), however for X it is probably much more
>>>>complicated (is the X load really linearly random in how much work it
>>>>does, or is it weighted towards small amounts with occasional large
>>>>amounts hitting? I would guess that at least beyond a certin point the
>>>>liklyhood of that much work being needed would be lower)
>>>>        
>>>>
>>>Actually I don't disagree. What I mean by hardware changes is more along
>>>the lines of changing the hard disk type in the same setup. That's what I
>>>mean by careful with the benchmarking. Taking the results from an athlon
>>>XP and comparing it to an altix is silly for example.
>>>      
>>>
>>I'm going to cautiously disagree. If the CPU needed was scaled so it
>>represented a fixed number of cycles (operations, work units) then the
>>effect of faster CPU would be shown. And the total power of all attached
>>CPUs should be taken into account, using HT or SMP does have an effect
>>of feel.
>>    
>>
>
>That is rather hard to do because each architecture's interpretation of fixed 
>number of cycles is different and this doesn't represent their speed in the 
>real world. The calculation when interbench is first run to see how many 
>"loops per ms" took quite a bit of effort to find just how many loops each 
>different cpu would do per ms and then find a way to make that not change 
>through compiler optimised code. The "loops per ms" parameter did not end up 
>being proportional to cpu Mhz except on the same cpu type.
>
>  
>
>>Disk tests should be at a fixed rate, not all you can do. That's NOT
>>realistic.
>>    
>>
>
>Not true; what you suggest is another thing to check entirely, and that would 
>be a valid benchmark too. What I'm interested in is what happens if you read 
>or write a DVD ISO image for example to your hard disk and what this does to 
>interactivity. This sort of reading or writing is not throttled in real life.
>

Of course it is. At least the read. It's limited to the speed needed to 
either play (watch) the image or to burn it.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[kernel]

Quoting Bill Davidsen <davidsen@tmr.com>:

> Con Kolivas wrote:
> 
> >On Thu, 14 Jul 2005 03:54, Bill Davidsen wrote:
> >  
> >
> >>Con Kolivas wrote:
> >>    
> >>
> >>>On Tue, 12 Jul 2005 21:57, David Lang wrote:
> >>>      
> >>>
> >>>>for audio and video this would seem to be a fairly simple scaleing
> factor
> >>>>(or just doing a fixed amount of work rather then a fixed percentage of
> >>>>the CPU worth of work), however for X it is probably much more
> >>>>complicated (is the X load really linearly random in how much work it
> >>>>does, or is it weighted towards small amounts with occasional large
> >>>>amounts hitting? I would guess that at least beyond a certin point the
> >>>>liklyhood of that much work being needed would be lower)
> >>>>        
> >>>>
> >>>Actually I don't disagree. What I mean by hardware changes is more along
> >>>the lines of changing the hard disk type in the same setup. That's what I
> >>>mean by careful with the benchmarking. Taking the results from an athlon
> >>>XP and comparing it to an altix is silly for example.
> >>>      
> >>>
> >>I'm going to cautiously disagree. If the CPU needed was scaled so it
> >>represented a fixed number of cycles (operations, work units) then the
> >>effect of faster CPU would be shown. And the total power of all attached
> >>CPUs should be taken into account, using HT or SMP does have an effect
> >>of feel.
> >>    
> >>
> >
> >That is rather hard to do because each architecture's interpretation of
> fixed 
> >number of cycles is different and this doesn't represent their speed in the
> 
> >real world. The calculation when interbench is first run to see how many 
> >"loops per ms" took quite a bit of effort to find just how many loops each 
> >different cpu would do per ms and then find a way to make that not change 
> >through compiler optimised code. The "loops per ms" parameter did not end up
> 
> >being proportional to cpu Mhz except on the same cpu type.
> >
> >  
> >
> >>Disk tests should be at a fixed rate, not all you can do. That's NOT
> >>realistic.
> >>    
> >>
> >
> >Not true; what you suggest is another thing to check entirely, and that
> would 
> >be a valid benchmark too. What I'm interested in is what happens if you read
> 
> >or write a DVD ISO image for example to your hard disk and what this does to
> 
> >interactivity. This sort of reading or writing is not throttled in real
> life.
> >
> 
> Of course it is. At least the read. It's limited to the speed needed to 
> either play (watch) the image or to burn it.

Ok we'll call it hair splitting. We do both. You read the file and I copy it.
Both happen in real life, and I plan to emulate both.

Cheers,
Con

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Bill Davidsen]

kernel@kolivas.org wrote:

>Quoting Bill Davidsen <davidsen@tmr.com>:
>
>  
>

>>>>Disk tests should be at a fixed rate, not all you can do. That's NOT
>>>>realistic.
>>>>   
>>>>
>>>>        
>>>>
>>>Not true; what you suggest is another thing to check entirely, and that
>>>      
>>>
>>would 
>>    
>>
>>>be a valid benchmark too. What I'm interested in is what happens if you read
>>>      
>>>
>>>or write a DVD ISO image for example to your hard disk and what this does to
>>>      
>>>
>>>interactivity. This sort of reading or writing is not throttled in real
>>>      
>>>
>>life.
>>    
>>
>>Of course it is. At least the read. It's limited to the speed needed to 
>>either play (watch) the image or to burn it.
>>    
>>
>
>Ok we'll call it hair splitting. We do both. You read the file and I copy it.
>Both happen in real life, and I plan to emulate both.
>

And that sounds exactly correct.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

RE: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[szonyi calin]

--- Con Kolivas <kernel@kolivas.org> a écrit :

> 	Interbench - The Linux Interactivity Benchmark v0.20
> 
> http://interbench.kolivas.org
> 
> direct download link:
> http://ck.kolivas.org/apps/interbench/interbench-0.20.tar.bz2
> 
>
[snip]
 
> Audio:
> Audio is simulated as a thread that tries to run at 50ms
> intervals that then
> requires 5% cpu. This behaviour ignores any caching that would
> normally be 
> done by well designed audio applications, but has been seen as
> the interval 
> used to write to audio cards by a popular linux audio player.
> It also ignores 
> any of the effects of different audio drivers and audio cards.
> Audio can also 
> be run as a real time SCHED_FIFO task.
> 

I have the following problem with audio:
Xmms is running with threads for audio and spectrum
analyzer(OpenGL).
The audio eats 5% cpu, the spectrum analyzer about 80 %. The
problem is that sometimes the spectrum analyzer is eating all of
the cpu while the audio is skipping. Xmms is version 1.2.10
kernel is vanilla, latest "stable" version 2.6.12, suid root.

Does your benchmark simultes this kind of behaviour ? 
 

> 
> Cheers,
> Con Kolivas
> 

I'll give it a try
Thanks
Calin


--
A mouse is a device used to point at 
the xterm you want to type in.
Kim Alm on a.s.r.


	

	
		
___________________________________________________________________________ 
Appel audio GRATUIT partout dans le monde avec le nouveau Yahoo! Messenger 
Téléchargez cette version sur http://fr.messenger.yahoo.com

RE: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Lee Revell]

On Wed, 2005-07-13 at 13:27 +0200, szonyi calin wrote:
> I have the following problem with audio:
> Xmms is running with threads for audio and spectrum
> analyzer(OpenGL).
> The audio eats 5% cpu, the spectrum analyzer about 80 %. The
> problem is that sometimes the spectrum analyzer is eating all of
> the cpu while the audio is skipping. Xmms is version 1.2.10
> kernel is vanilla, latest "stable" version 2.6.12, suid root.
> 
> Does your benchmark simultes this kind of behaviour ? 

That's just a broken app, the kernel can't do anything about it.  XMMS
should not be running the spectrum analyzer thread at such a high
priority as to interfere with the audio thread.

Lee

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 802 bytes --]

On Thu, 14 Jul 2005 03:34, Lee Revell wrote:
> On Wed, 2005-07-13 at 13:27 +0200, szonyi calin wrote:
> > I have the following problem with audio:
> > Xmms is running with threads for audio and spectrum
> > analyzer(OpenGL).
> > The audio eats 5% cpu, the spectrum analyzer about 80 %. The
> > problem is that sometimes the spectrum analyzer is eating all of
> > the cpu while the audio is skipping. Xmms is version 1.2.10
> > kernel is vanilla, latest "stable" version 2.6.12, suid root.
> >
> > Does your benchmark simultes this kind of behaviour ?
>
> That's just a broken app, the kernel can't do anything about it.  XMMS
> should not be running the spectrum analyzer thread at such a high
> priority as to interfere with the audio thread.

I agree; optimising for this is just silly.

Cheers,
Con

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [ANNOUNCE] Interbench v0.20 - Interactivity benchmark

[Lee Revell]

On Thu, 2005-07-14 at 09:57 +1000, Con Kolivas wrote:
> On Thu, 14 Jul 2005 03:34, Lee Revell wrote:
> > On Wed, 2005-07-13 at 13:27 +0200, szonyi calin wrote:
> > > I have the following problem with audio:
> > > Xmms is running with threads for audio and spectrum
> > > analyzer(OpenGL).
> > > The audio eats 5% cpu, the spectrum analyzer about 80 %. The
> > > problem is that sometimes the spectrum analyzer is eating all of
> > > the cpu while the audio is skipping. Xmms is version 1.2.10
> > > kernel is vanilla, latest "stable" version 2.6.12, suid root.
> > >
> > > Does your benchmark simultes this kind of behaviour ?
> >
> > That's just a broken app, the kernel can't do anything about it.  XMMS
> > should not be running the spectrum analyzer thread at such a high
> > priority as to interfere with the audio thread.
> 
> I agree; optimising for this is just silly.

I took a closer look and it's indeed quite broken.  If XMMS is run in
realtime mode, and multithreading is enabled, it runs everything
SCHED_RR - GUI, audio, spectrum analyzer.  (And since it relies on new
threads inheriting RT scheduling when pthread_create()'d, the threads
will fail to get SCHED_RR on buggy NPTL versions like the one in Debian
Sarge).

This untested patch should enable RT scheduling for only the (ALSA)
audio thread.  Make sure NOT to run in realtime mode.

Lee

--- ../xmms-1.2.10+cvs20050209.orig/Output/alsa/audio.c	2005-01-31 17:45:08.000000000 -0500
+++ Output/alsa/audio.c	2005-07-16 16:06:06.000000000 -0400
@@ -962,6 +962,7 @@
 /* open callback */
 int alsa_open(AFormat fmt, int rate, int nch)
 {
+	struct sched_param sparam;
 	debug("Opening device");
 	inputf = snd_format_from_xmms(fmt, rate, nch);
 	effectf = snd_format_from_xmms(fmt, rate, nch);
@@ -1010,6 +1011,9 @@
 		flush_request = -1;
 
 		pthread_create(&audio_thread, NULL, alsa_loop, NULL);
+		sparam.sched_priority = sched_get_priority_max(SCHED_FIFO);
+		if (pthread_setschedparam (audio_thread, SCHED_FIFO, &sparam) )
+			fprintf(stderr, "Error %s getting SCHED_FIFO for audio thread\n", strerror (errno));
 	}
 
 	return 1;