Update on Timer Frequencies

Update on Timer Frequencies

[AndyLiebman]

>Your system looks pretty much interrupt driven, and I would assume  that
>all these are of same importance.  So I would actually  recommend the
>vanilla kernel, with HZ=100 and preemption turned  off.

Hi Steve, 

I thought I would update you (and those on the  list who care) about my real 
world experience comparing the 2.6.14 kernel  compiled with the timer 
frequency set to 100 versus 1000. 
 
If you recall, I'm running a video server that is taking in and sending out  
a large number of streams of video and audio clips that range in data rates 
from  3.8 to 28 MB/sec. 
 
We have a benchmark test that we developed that simulates multiple  
non-linear video editing workstations playing back sequences of video each from  their 
own timeline. We developed this test mostly to tune our storage subsystem,  so 
it's not a perfect test to evaluate timer frequencies (because it doesn't  
involve any network traffic). 
 
Each test is really a sequence of scripts that a) run dd on a  trio of one 
video and two audio files and b) move on to the next script  until all scripts 
have been executed, each time reading data from the RAIDS and  writing the data 
to /dev/null until 21 clips worth of data have been read. We do  this in 
parallel 5, 6, 10, 15 times to simulate that same number of workstations  calling 
for data simultaneously. 
 
The test is an approximate simulation of video  editing workstations playing 
back unique sequences of 21 clips that each  range in length from 3 to 5 
seconds and that run at a rate of 18 MB/sec video  (which is roughly 8-bit 
uncompressed standard definition). 
 
The test puts extreme stress on our RAID subsystem -- forcing drives to  seek 
all over the place to play back these random clips (think of each sequence  
as a little 90 second story with 21 separate shots). 
 
After conducting many tests over the weekend, we concluded that the tests  
consistently took 6 percent longer to complete when running under a kernel  
compiled with the 100 hz timer versus an identical kernel compiled with a 1000  hz 
timer. 
 
Again, I realize this isn't a perfect simulation of what our server does in  
real life; we weren't moving data over a network -- thus there were no  NIC 
card interrupts involved, no samba or netatalk work taking place, and  instead 
all of our scripts and dd commands are running on the server. 
 
Still, the results are interesting. That plus the fact that our "user  
management applications" open and run excruciatingly slowly when the timer  
frequency is 100 versus 1000 (3 x longer to open the application) is making us  stick 
with a timer frequency of 1000 for now. 
 
Thanks again for your help. I'm sorry if I offended anyone on this list by  
writing from an AOL address. 

Re: Update on Timer Frequencies

[Steven Rostedt]

On Mon, 2005-11-07 at 22:26 -0500, AndyLiebman@aol.com wrote:
> >Your system looks pretty much interrupt driven, and I would assume  that
> >all these are of same importance.  So I would actually  recommend the
> >vanilla kernel, with HZ=100 and preemption turned  off.
> 
> Hi Steve, 
> 
> I thought I would update you (and those on the  list who care) about my real 
> world experience comparing the 2.6.14 kernel  compiled with the timer 
> frequency set to 100 versus 1000. 
>  
> If you recall, I'm running a video server that is taking in and sending out  
> a large number of streams of video and audio clips that range in data rates 
> from  3.8 to 28 MB/sec. 
>  
> We have a benchmark test that we developed that simulates multiple  
> non-linear video editing workstations playing back sequences of video each from  their 
> own timeline. We developed this test mostly to tune our storage subsystem,  so 
> it's not a perfect test to evaluate timer frequencies (because it doesn't  
> involve any network traffic). 

So the server would just be handling data for other machines on a
network?  There may be clashes between the scripts and the services.
Could you run scripts on another machine that communicates to the server
and do your benchmarks that way?

>  
> Each test is really a sequence of scripts that a) run dd on a  trio of one 
> video and two audio files and b) move on to the next script  until all scripts 
> have been executed, each time reading data from the RAIDS and  writing the data 
> to /dev/null until 21 clips worth of data have been read. We do  this in 
> parallel 5, 6, 10, 15 times to simulate that same number of workstations  calling 
> for data simultaneously. 

I wonder what the over head is of these.  Could you post numbers running
these scripts under the time command?  So something like "time dd
if=audio_whatever of=/dev/null"


>  
> The test is an approximate simulation of video  editing workstations playing 
> back unique sequences of 21 clips that each  range in length from 3 to 5 
> seconds and that run at a rate of 18 MB/sec video  (which is roughly 8-bit 
> uncompressed standard definition). 
>  
> The test puts extreme stress on our RAID subsystem -- forcing drives to  seek 
> all over the place to play back these random clips (think of each sequence  
> as a little 90 second story with 21 separate shots). 
>  
> After conducting many tests over the weekend, we concluded that the tests  
> consistently took 6 percent longer to complete when running under a kernel  
> compiled with the 100 hz timer versus an identical kernel compiled with a 1000  hz 
> timer. 
>  
> Again, I realize this isn't a perfect simulation of what our server does in  
> real life; we weren't moving data over a network -- thus there were no  NIC 
> card interrupts involved, no samba or netatalk work taking place, and  instead 
> all of our scripts and dd commands are running on the server. 

I still wouldn't trust those tests.  Try something like netcat (nc) from
another machine and see how it compares.  Although that would still be a
crude test.

>  
> Still, the results are interesting. That plus the fact that our "user  
> management applications" open and run excruciatingly slowly when the timer  
> frequency is 100 versus 1000 (3 x longer to open the application) is making us  stick 
> with a timer frequency of 1000 for now. 

Are there other services running when you open these applications?  If
not, then this would not make sense.  But if other services where not
active when these applications are being opened, I'd be surprised at
these numbers.

The time slice algorithm in the scheduler is pretty complex. Not the
implementation, but the assigning of time slices and the dynamic nature
of them.  I wonder if a lower HZ would cause more schedules?  I'll have
to test this out tomorrow and see what I get in scheduling frequencies
between the two.

>  
> Thanks again for your help. I'm sorry if I offended anyone on this list by  
> writing from an AOL address. 

I'm sure they'll get over it ;-)

-- Steve

Re: Update on Timer Frequencies

[Steven Rostedt]

On Mon, 2005-11-07 at 22:58 -0500, Steven Rostedt wrote:
> On Mon, 2005-11-07 at 22:26 -0500, AndyLiebman@aol.com wrote:
> >
> 
> >  
> > Still, the results are interesting. That plus the fact that our "user  
> > management applications" open and run excruciatingly slowly when the timer  
> > frequency is 100 versus 1000 (3 x longer to open the application) is making us  stick 
> > with a timer frequency of 1000 for now. 
> 
> Are there other services running when you open these applications?  If
> not, then this would not make sense.  But if other services where not
> active when these applications are being opened, I'd be surprised at
> these numbers.
> 
> The time slice algorithm in the scheduler is pretty complex. Not the
> implementation, but the assigning of time slices and the dynamic nature
> of them.  I wonder if a lower HZ would cause more schedules?  I'll have
> to test this out tomorrow and see what I get in scheduling frequencies
> between the two.

Hi Andy,

I did a few tests and couldn't see 100HZ being slower than 1000HZ for
server type work*.  I ran the following script, which kicks off 10
processes doing 'find /' so it should hit the hard drive a bit.  Grant
it, each process will probably read off another's buffer cache.  But if
you have a better test to try on a not so big machine. Feel free to pass
it on :-)

(*) well maybe not really server type work!

testme:
#!/bin/sh

date
echo 1 > /proc/logdev/switch
for i in `seq 1 10`; do
  find / &> /dev/null &
done
wait `jobs -p`
echo 0 > /proc/logdev/switch
date


I ran this on 2.6.14 with my logdev patch (that is like relayfs, but I
wrote this before relayfs was available so I still use it ;-) and it
does nice recording of context switches.

http://www.kihontech.com/logdev/patch-2.6.14-logdev1.patch

The /proc/logdev/switch turns on recording of context switches. 

I ran this test once on the following machine:

# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 8
model name      : Pentium III (Coppermine)
stepping        : 3
cpu MHz         : 368.002
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr sse
bogomips        : 737.16

The test was run like:

# time testme

At 100HZ I got:

----
Wed Nov  9 11:30:24 EST 2005
Wed Nov  9 11:37:55 EST 2005

real    7m31.789s
user    0m45.320s
sys     4m36.930s
----

At 1000HZ I got:

----
Wed Nov  9 11:54:05 EST 2005
Wed Nov  9 12:01:54 EST 2005

real    7m49.644s
user    0m46.430s
sys     4m41.145s
----

Running my logdev tools from:
http://www.kihontech.com/logdev/logdev_tools_0.3.0.tar.bz2

./logread /dev/logdev > 1000HZ.out  # with 1000HZ
./logread /dev/logdev > 100HZ.out # with 100HZ

These files can be found at:
http://www.kihontech.com/tests/hz_times/

These show all the times that a context switch took place. This is a
ring buffer, so it only captured the last 30 some seconds of the test.
But that should be good enough.

with my analyze.pl script (also supplied at the website) I ran:

./analyze.pl 1000HZ.out > 1000HZ.txt
and again for 100HZ.

This calculates the times between the context switches and at the end
produces an average.

(all times are in seconds)

For 100HZ:

[54543.530810] CPU:0 (testme:2539) -->> (find:2550)
  diff: 0.000213
[54543.546730] CPU:0 (find:2550) -->> (testme:2539)
  diff: 0.015920
count: 28974  total: 38.384180
average: 0.001325


For 1000HZ:

[  629.034303] CPU:0 (testme:2090) -->> (find:2097)
  diff: 0.000126
[  629.035776] CPU:0 (find:2097) -->> (testme:2090)
  diff: 0.001473
count: 28887  total: 35.322831
average: 0.001223


Conclusion:

It works as I would expect it to.  The 1000HZ actually doesn't let
processes run for as long as the 100HZ does, and 100HZ seems to be
faster (for just the test).

Oh I did have preemption on.  I guess I should turn it off and run it
again.  Tomorrow I'll give that a try and I'll only report back if I
find something interesting.

The config of the 100HZ is also on the site and the only change between
the two runs was changing 1000HZ to 100HZ.

Do what you want with this info. ;-)

-- Steve

Re: Update on Timer Frequencies

[Steven Rostedt]

On Wed, 2005-11-09 at 23:35 -0500, Steven Rostedt wrote:
> On Mon, 2005-11-07 at 22:58 -0500, Steven Rostedt wrote:
> > 
> 
> Running my logdev tools from:
> http://www.kihontech.com/logdev/logdev_tools_0.3.0.tar.bz2
> 
> ./logread /dev/logdev > 1000HZ.out  # with 1000HZ
> ./logread /dev/logdev > 100HZ.out # with 100HZ
> 
> These files can be found at:
> http://www.kihontech.com/tests/hz_times/
> 
> These show all the times that a context switch took place. This is a
> ring buffer, so it only captured the last 30 some seconds of the test.
> But that should be good enough.
> 
> with my analyze.pl script (also supplied at the website) I ran:
> 
> ./analyze.pl 1000HZ.out > 1000HZ.txt
> and again for 100HZ.
> 
> This calculates the times between the context switches and at the end
> produces an average.
> 
> (all times are in seconds)
> 
> For 100HZ:
> 
> [54543.530810] CPU:0 (testme:2539) -->> (find:2550)
>   diff: 0.000213
> [54543.546730] CPU:0 (find:2550) -->> (testme:2539)
>   diff: 0.015920
> count: 28974  total: 38.384180
> average: 0.001325

The above 100HZ test is invalid.  As shown in the times, it was run
after 54000 some seconds, which means that the file system was probably
already cached.  So I did fresh reboots and ran the test shortly after
bootup. 

The test results are again at http://www.kihontech.com/tests/hz_times/
but they are all with a *_2.* in the name.

Here are the times that were run for each test (100HZ vs 1000HZ and
preempt vs nopreempt).

1000HZ preempt:
Thu Nov 10 18:05:11 EST 2005
Thu Nov 10 18:13:01 EST 2005

real    7m49.741s
user    0m46.464s
sys     4m41.524s


1000HZ nopreempt:
Thu Nov 10 22:17:33 EST 2005
Thu Nov 10 22:25:15 EST 2005

real    7m42.339s
user    0m47.156s
sys     4m33.205s


100HZ preempt:
# time ./testme
Thu Nov 10 17:40:29 EST 2005
Thu Nov 10 17:48:12 EST 2005

real    7m42.418s
user    0m46.190s
sys     4m40.350s


100HZ nopreempt:
Thu Nov 10 17:52:15 EST 2005
Thu Nov 10 17:59:56 EST 2005

real    7m40.976s
user    0m44.840s
sys     4m34.510s

This seems to show that 100HZ with no preemption was the fastest, but
the times are too close, so it is of a difference of ~0.2% which is well
in the margin of error.  So this test really doesn't show much benefit
between the settings.

Tomorrow, I'll see if I can make a test that serves up web pages, and
see if that will show the benefits for servers and the settings for HZ
and preemption.

-- Steve