Re: Premptible Kernels and Timer Frequencies

Re: Premptible Kernels and Timer Frequencies

[AndyLiebman]

Hello Steve, 

I'm going to add my message  at the top of this post, because it's not a 
direct response to your post.  

Thanks for the good start at explaining some of the new kernel compiling  
options. I really appreciate the time and effort you made at explaining some of  
these new choices. And I took your suggestion and tried out the 2.6.14-rt1  
kernel with total success (at least getting it to run). 

However,  ultimately you don't really answer my fundamental question about 
what options  are best for what kind of application. Maybe someone like Linus or 
Andrew Morton  could chime in here. Put a little write up on the kernel.org 
site??? 

You  guys must have a good idea about WHY one would choose a timer frequenzy 
of 100  versus 1000 -- YOU put these options in the kernel. Please give us 
some examples  about where the different sets of options are appropriate -- what 
options should  be used together and which one could contradict each other. 

To recap MY  particular case, I have a file server that is sending out -- and 
taking in --  video and audio files from 5 to 30 workstations. The data rates 
typically range  form 3.8 MB/sec per workstation up to 60 MB/sec per 
workstation. All of the  transactions are handled by TCP/IP. In some cases, we are 
using a Chelsio 10  Gigabit Ethernet card with TCP/IP offload. But generally, we 
have a bunch of  Intel Pro 1000 MT server adapters feeding one or more 
workstations each. Or we  are bonding a group of NICS and sending all the data to a 
managed switch.  

So, we have:

a lot of TCP/IP transactions
NIC drivers  running
3ware Hardware RAID -5 drivers running
Sometimes Linux Software  RAID -0 striping two 3ware cards together
Samba
Netatalk
Occasional use  of a Twisted Web Server
Occasional use of TightVNC
A mostly inactive KDE  session
A UPS program

That's pretty much what's running. Only rarely  do any applications "run" on 
the server (once in a while, an administrator will  do something to "manage" 
the system -- but that's maybe 1 or 2 minutes out of  every 24 hours. We are 
running on both 32-bit and 64-bit systems. 

So, my  question was, what kernel compiling options are appropriate for the 
best  performance?  (for ME, best performance = lowest latency in delivering  
random data to -- and taking data from -- a group of video editing 
workstations,  AND best throughput -- goals which may be mutually exclusive). For MY  
application, how should I set: 

Preemptible Kernel  Model
No Preemption
Voluntary  Preemption
Premptible Kernel

Prempt The Big Kernel  Lock (Y/N)

Timer Frequency
100  hz
250 hz
1000 hz

While my  case might be different from sombody else's case, I'm sure it would 
be useful to  give three or four examples of different server uses that at 
least "in theory"  would benefit from a particular group of settings. I'm not 
sure that "desktop"  versus "server" is the most helpful distinction. Are ALL 
servers (including  mine) best off with: 
No premption
Saying No to Preempt The Big Kernel Lock
Timer Frequence  of 100hz

Somehow, I doubt it. 

Yeah, I'm ready to experiment. I've  been doing that with Linux for three 
years and getting great results. But I'd  love to hear the theory of what SHOULD 
be best. It's about seeing the forest  through the trees, getting the "big 
picture" after looking at all the minute  details. 

Thanks in advance for more information... Linux rocks!  

Andy  Liebman

-------------------------------------------------------


In  a message dated 10/28/2005 7:06:50 P.M. Eastern Standard Time,  
rostedt@goodmis.org writes:
On Fri, 2005-10-28 at 12:36 -0400,  AndyLiebman@aol.com wrote:
> Thanks to all responsible for getting the  2.6.14  kernel out quickly. 
There 
> are many important bugfixes and  features in this  kernel that are relevant 
to 
> my work. I look  forward to using it starting today!  
> 
> I have a request.  Would it be possible for a knowledgeable person to  
write 
> up a  little description of some of the following kernel options and what  
they  
> REALLY mean and how to use them? I think it would be great to put a  brief  
> "white paper" on the kernel.org site. 

I'll respond  a little here, but this is far from being a white paper.
Although, if I have  time, I'll write one.  My involvement with Linux
today is mostly with  Ingo's RT patch that really brings down the
latencies of the  kernel.

> 
> In specific, I am referring to the  following:  
> 
> Premptible Kernel Model
> No   Premption

Simple.  If a process is in the kernel (via a system call)  it will not
be preempted by another process.  There are points in the  kernel where a
process will need to wait for an event (like memory swapping  in) and
will call schedule to let other processes run, instead of a busy  loop.

The problem with this is that when an interrupt comes in and wakes  up a
higher priority process. That high priority process needs to wait  till
the lower priority process either explicitly calls schedule or  returns
to user land.

> Voluntary Premption

There are  several points in the kernel that "might schedule".  Like
allocating  more memory, the kernel may need to swap out memory to get
more available  memory, so this may or may not schedule.

With Voluntary Preemption, these  points always check to see if it should
schedule because another higher  process is waiting to run.  So
basically, if a higher priority process  wants to run on a CPU that is
currently running a lower priority process that  happens to be in the
kernel. The "need_resched" flag is set on the lower  priority process and
when one of these "voluntary preemption" points are hit,  it will call
schedule.  The high priority process still needs to wait  till one of
these points are hit, or the process actually calls schedule, or  it goes
back to user space.

> Premptible Kernel (Low-latency  Desktop)

This is currently the lowest latency selection in the vanilla  kernel.
When this is turned on, a high priority process that is woken up,  will
preempt the currently running lower priority process even if it is  in
the kernel.

The trick here, is that there are SMP protected areas  in the kernel
protected by spin_locks.  These usually are only for  multiprocessor
machines, but when you have a fully preemptible kernel, these  areas must
also be protected.  So, on a uni-processor machine without  "Preempt
Kernel", these spin_locks are optimized out (no-ops).  But with  "Preempt
Kernel", these areas prevent preemption from occurring. So now the  high
priority process can preempt the lower priority process anytime it  is
not in one of theses areas protected by a spin_lock.  If the  lower
priority process is in one of these areas, the higher priority  process
only needs to wait for that process to leave this area, which is  usually
a very short time, since on SMP this would cause a CPU to spin in a  busy
loop, and Linux tries to make these areas small.

This explanation  is a little simple but gets the gist of what is
happening.

As for  which one is best for you.  It matters what type of latencies  you
want.  That is, do you want better reaction times for higher  priority
processes.  Without the user caring about priorities, this  usually means
the most interactive tasks, where servers that serve web pages  and such
probably don't care about one process over the other. But a gamer  might
be more interested in his game having higher priority than  some
background job, and having a lower latency in shooting the  monster.

> 
> Prempt The Big Kernel  Lock

The Big  Kernel Lock is an old relic that I wish would go away.   Before
spin_locks, everything was protected with this behemoth.  One  lock to
rule them all!  So basically, any time you needed to protect  some data,
you would grab this lock.  So on SMP machines, CPUs were  waiting around
a lot to handle some data because some other CPU was working  on some
totally separate data.  This obviously was very  inefficient.  Well,
fortunately, the kernel developers implemented a  fine grain locking
(spin_locks) and got rid of most of the Big Kernel  Lock.  But
unfortunately, it is still used in some pretty scary parts of  the
kernel.  Scary, because it will be difficult to remove this lock for  in
favor of smaller variants.

This lock is different than the  spin_locks by the fact that you can call
schedule holding this lock.   Schedule will simply let go of this lock
for the scheduling process.   When the process is scheduled again, it
will try to retake the lock.   Also, a process may grab this lock as many
times as it wants while holding  it, as long as it releases it the same
amount of times.

To preempt  while this lock is held, takes some special care, since you
don't want to let  go of this lock when it is being preempted.  So work
is done to prevent  the scheduler from releasing this lock when it calls
schedule because of  preemption and not because of the process
voluntarily scheduling.  Also  care is taken when retaking the lock when
woken up in the scheduler since the  process might not get it because a
process it just preempted has  it.

So this is more complex, and the developers decided to let people  turn
it off if they are nervous about the complexity on critical  servers.

> 
> Timer Frequency
> 100 Hz
> 250  Hz
> 1000 Hz

Again, this has to do with response times and  latencies.  Processes are
given time slices that they can run in. These  are broken up into
"jiffies".  A jiffy is defined in the kernel as 1/HZ  second.  So at
100Hz, a jiffy is 10ms, 250Hz 4ms and 1000Hz  1ms.

Also the resolution of sleeps and timers are done at the jiffy  level
(with current vanilla Linux).  So at 100Hz a timer set to go off  in
100ms is most likely to go off in 110ms where as with the 1000Hz it  has
a better probability of going off in 101ms.  Note, this is not a  good
example, because latencies usually cause timers and such to not  be
predictable in how close to the actual time it goes off.  It only  needs
to guarantee that it doesn't go off early.

Also note, the better  the latency, the higher the over head.  This is
pretty much true in all  cases.  For HZ = 1000, the timer interrupt needs
to go off 1000 times a  second, which will cause 900 more interrupts to
go off in that second than  for HZ = 100.

> 
> 
> I think it would be helpful  for  the whole Linux community to get better 
> hints about the  theoretical optimal  settings for various types of 
systems. Saying 
>  that one option is good for  "desktops" and another is good for "servers"  
is 
> a start, but I'm sure that with  just a few more words, you  could be much 
more 
> helpful to users who are trying to  decide what  options to select for 
> compiling. 

I'm not a server maintainer or  much of a gamer, so someone else can
explain what is best for them.  I'm  more of the embedded or special
purpose tool guy.  But I do understand  what these things actually mean.

> 
> For instance, I have  a  number of file servers that send video files out 
> simultaneously  to multiple  video editing systems. Each of 10-20 video 
editing 
>  systems could be reading  files from the server (or writing files to the  
server) 
> at data rates ranging  from 3.5 MB/sec up to 30,40 or 50  MB/sec. 
> 
> The server ITSELF is rarely  running any  applications other than Samba, 
> Netatalk, Software RAID 0, NIC   drivers, and fairly low-CPU usage things 
like the 
> UPS monitor, 3ware  Drivers for  Hardware RAID 5, and KDE (which is hardly 
ever 
>  getting any input from users).  
> 
> We get really great  performance with the 2.6.12 and 2.6.10 kernels. And  
> typically,  CPU usage ranges from 10-20 perecent with many client systems 
going.   
> However, we are wondering if it is worth it to try the Preemptible  Kernels 
for  
> greater responsiveness and perhaps support for more  simultaneous streams, 
> which  MUST arrive at the video editing  system on time or we get "dropped 
> frames".  

Have you taken  a look at Ingo Molnar's  PREEMPT_RT?

http://people.redhat.com/mingo/realtime-preempt/

This  has the best in low latency than any of the above options.  But I
must  also warn you that the lower the latency, the performance will
usually take a  slight hit as well. But we try to keep that at a minimum.

> 
>  I'm sure it's a similar situtation to Video On Demand servers only our   
data 
> rates are much higher per client, so we can't support that many  clients as 
 a 
> VOD system. 
> 
> We have tested  compiling the 2.6.12 kernel with both  voluntary preemption 
> and  low-latency preemption. Both seem to work fine, but we  haven't 
>  stress-tested them yet because we were waiting for the 2.6.14 kernel to   
come out. 
> 
> Could we get into any danger with the preemptible  kernels if  the system 
load 
> gets heavy? 

Preemptible  kernels are suppose to work _better_ on higher load.
Performance goes  slightly down, but predictability goes way up.

> 
> And now that  the Timer frequency option has  been added, what would be 
> optimal  for such a video server that is not itself  running any taxing  
applications? 

I'm not sure if that would effect it. If it is interrupt  driven, the
timer frequency is of no use. If the application is setting up  its own
frequency, then that would be a different story.

> 
>  Hope to hear from one of the "big guys"  on this. 

I'm not a "big  guy" ;-) but I'm pretty active with Ingo's RT patch.  I
just submit bug  fixes for the vanilla guys.

> 
> Thanks again for all your great  work. 

Hope this explains things a bit better.  I've been meaning to  write up
something about Ingo's patch to explain things in his kernel  options
just like this.  When I get some time, maybe I'll do that  too.

-- Steve  

Re: Premptible Kernels and Timer Frequencies

[AndyLiebman]

Thanks again, Steve. That was really well explained. It would be great to  
hear from Alan too if he has a moment. And it would be great to post all of this 
 on the kernel.org site. I'm sure lots of people are scratching their heads 
about  these new options and information such as this will help people set off 
in the  right direction. 

Andy

--------------------------------------------

In a message dated 11/3/2005 7:18:20 P.M. Eastern Standard Time,  
rostedt@goodmis.org writes:
On Thu, 2005-11-03 at 18:50 -0500,  AndyLiebman@aol.com wrote:
> Hello Steve, 

Hello  Andy

>  
> I'm going to add my message at the top of this  post, because it's not
> a direct response to your post. 
>   
> Thanks for the good start at explaining some of the new kernel
>  compiling options. I really appreciate the time and effort you made at
>  explaining some of these new choices. And I took your suggestion and
>  tried out the 2.6.14-rt1 kernel with total success (at least getting
> it  to run). 

The -rt branch is a fast moving target, and is always getting  better.
It's still in development, but is getting pretty stable. As I  write
this, the current version is at 2.6.14-rt5.  I'd recommend  downloading
that one. Although for what you are doing, vanilla linux should  be good
enough (see below).

>  
> However, ultimately  you don't really answer my fundamental question
> about what options are  best for what kind of application. Maybe
> someone like Linus or Andrew  Morton could chime in here. Put a little
> write up on the kernel.org  site??? 

Actually Alan Cox might be a good person to chime in, since he  does a
lot in networking.

>  
> You guys must have a  good idea about WHY one would choose a timer
> frequenzy of 100 versus  1000 -- YOU put these options in the kernel.
> Please give us some  examples about where the different sets of options
> are appropriate --  what options should be used together and which one
> could contradict each  other. 
>  
> To recap MY particular case, I have a file server  that is sending out
> -- and taking in -- video and audio files from 5 to  30 workstations.
> The data rates typically range form 3.8 MB/sec per  workstation up
> to 60 MB/sec per workstation. All of the transactions are  handled by
> TCP/IP. In some cases, we are using a Chelsio 10 Gigabit  Ethernet card
> with TCP/IP offload. But generally, we have a bunch of  Intel Pro 1000
> MT server adapters feeding one or more workstations each.  Or we are
> bonding a group of NICS and sending all the data to a managed  switch. 
>  
> So, we have:
>  
> a lot of  TCP/IP transactions
> NIC drivers running
> 3ware Hardware RAID -5  drivers running
> Sometimes Linux Software RAID -0 striping two 3ware  cards together
> Samba
> Netatalk
> Occasional use of a  Twisted Web Server
> Occasional use of TightVNC
> A mostly inactive  KDE session
> A UPS program
>  
> That's pretty much  what's running. Only rarely do any applications
> "run" on the server  (once in a while, an administrator will do
> something to "manage" the  system -- but that's maybe 1 or 2 minutes
> out of every 24 hours. We are  running on both 32-bit and 64-bit
> systems. 

Your system looks  pretty much interrupt driven, and I would assume that
all these are of same  importance.  So I would actually recommend the
vanilla kernel, with  HZ=100 and preemption turned off.

Why?

Well, every time you get  preempted, or happen to do a task switch, there
is overhead.  So if all  processes or actions are of the same importance,
whatever is running now,  doesn't need to be preempted by something that
just woke up.  And since  this is mainly work that is in the kernel, with
only the user apps applying  services.  I wouldn't think you really need
the preemption.

The  HZ value is what makes up time slices for applications, and since
what you  explained, is interrupt driven, you want a low value for HZ.
This will keep  the timer interrupt down from preempting whatever is
going  on.


>  
> So, my question was, what kernel compiling  options are appropriate for
> the best performance?  (for ME, best  performance = lowest latency in
> delivering random data to -- and taking  data from -- a group of video
> editing workstations, AND best throughput  -- goals which may be
> mutually exclusive). For MY application, how  should I set: 
>  
> Preemptible Kernel  Model
>     No  Preemption
>     Voluntary  Preemption
>     Premptible Kernel
>   
> Prempt The Big Kernel Lock (Y/N)
>  
> Timer  Frequency
>     100 hz
>      250 hz
>     1000 hz
>  
> While my  case might be different from sombody else's case, I'm sure it
> would be  useful to give three or four examples of different server
> uses that at  least "in theory" would benefit from a particular group
> of settings. I'm  not sure that "desktop" versus "server" is the most
> helpful distinction.  Are ALL servers (including mine) best off with: 
>      No premption
>     Saying No to Preempt The Big Kernel  Lock
>     Timer Frequence of 100hz
>   
> Somehow, I doubt it. 

As I explained up above.  Yes!

>  
> Yeah, I'm ready to experiment. I've been doing  that with Linux for
> three years and getting great results. But I'd love  to hear the theory
> of what SHOULD be best. It's about seeing the forest  through the
> trees, getting the "big picture" after looking at all the  minute
> details. 
>  
> Thanks in advance for more  information... Linux rocks! 
>  
> 

-- Steve
 

Re: Premptible Kernels and Timer Frequencies

[Felix Oxley]

AndyLiebman@aol.com wrote:
> Thanks again, Steve. That was really well explained. It would be great to  
> hear from Alan too if he has a moment. And it would be great to post all of this 
>  on the kernel.org site. I'm sure lots of people are scratching their heads 
> about  these new options and information such as this will help people set off 
> in the  right direction. 
> 
> Andy

Andy,

I know it would be cool to get a mail from Linux or AM or AC, but you 
really have no right to expect an answer from anybody in particular, 
especially with a reasonably straight forward question of a general nature.

After all if you phone Microsoft to discuss whether you should configure 
Windows to optimize for foreground or background tasks you don't expect 
to speak to Bill Gates or Steve Balmer do you?

In my view it is bad form and shows you up when you do this.

BTW, you are top posting again.

(Finally, this may be a cheap shot, but it is possible that posting to 
LKML with an AOL email address carries some stigma :-)


regards,
Felix

---
I lurk therefore I am

Re: Premptible Kernels and Timer Frequencies

[Steven Rostedt]

On Thu, 2005-11-03 at 18:50 -0500, AndyLiebman@aol.com wrote:
> Hello Steve, 

Hello Andy

>  
> I'm going to add my message at the top of this post, because it's not
> a direct response to your post. 
>  
> Thanks for the good start at explaining some of the new kernel
> compiling options. I really appreciate the time and effort you made at
> explaining some of these new choices. And I took your suggestion and
> tried out the 2.6.14-rt1 kernel with total success (at least getting
> it to run). 

The -rt branch is a fast moving target, and is always getting better.
It's still in development, but is getting pretty stable. As I write
this, the current version is at 2.6.14-rt5.  I'd recommend downloading
that one. Although for what you are doing, vanilla linux should be good
enough (see below).

>  
> However, ultimately you don't really answer my fundamental question
> about what options are best for what kind of application. Maybe
> someone like Linus or Andrew Morton could chime in here. Put a little
> write up on the kernel.org site??? 

Actually Alan Cox might be a good person to chime in, since he does a
lot in networking.

>  
> You guys must have a good idea about WHY one would choose a timer
> frequenzy of 100 versus 1000 -- YOU put these options in the kernel.
> Please give us some examples about where the different sets of options
> are appropriate -- what options should be used together and which one
> could contradict each other. 
>  
> To recap MY particular case, I have a file server that is sending out
> -- and taking in -- video and audio files from 5 to 30 workstations.
> The data rates typically range form 3.8 MB/sec per workstation up
> to 60 MB/sec per workstation. All of the transactions are handled by
> TCP/IP. In some cases, we are using a Chelsio 10 Gigabit Ethernet card
> with TCP/IP offload. But generally, we have a bunch of Intel Pro 1000
> MT server adapters feeding one or more workstations each. Or we are
> bonding a group of NICS and sending all the data to a managed switch. 
>  
> So, we have:
>  
> a lot of TCP/IP transactions
> NIC drivers running
> 3ware Hardware RAID -5 drivers running
> Sometimes Linux Software RAID -0 striping two 3ware cards together
> Samba
> Netatalk
> Occasional use of a Twisted Web Server
> Occasional use of TightVNC
> A mostly inactive KDE session
> A UPS program
>  
> That's pretty much what's running. Only rarely do any applications
> "run" on the server (once in a while, an administrator will do
> something to "manage" the system -- but that's maybe 1 or 2 minutes
> out of every 24 hours. We are running on both 32-bit and 64-bit
> systems. 

Your system looks pretty much interrupt driven, and I would assume that
all these are of same importance.  So I would actually recommend the
vanilla kernel, with HZ=100 and preemption turned off.

Why?

Well, every time you get preempted, or happen to do a task switch, there
is overhead.  So if all processes or actions are of the same importance,
whatever is running now, doesn't need to be preempted by something that
just woke up.  And since this is mainly work that is in the kernel, with
only the user apps applying services.  I wouldn't think you really need
the preemption.

The HZ value is what makes up time slices for applications, and since
what you explained, is interrupt driven, you want a low value for HZ.
This will keep the timer interrupt down from preempting whatever is
going on.


>  
> So, my question was, what kernel compiling options are appropriate for
> the best performance?  (for ME, best performance = lowest latency in
> delivering random data to -- and taking data from -- a group of video
> editing workstations, AND best throughput -- goals which may be
> mutually exclusive). For MY application, how should I set: 
>  
> Preemptible Kernel Model
>     No Preemption
>     Voluntary Preemption
>     Premptible Kernel
>  
> Prempt The Big Kernel Lock (Y/N)
>  
> Timer Frequency
>     100 hz
>     250 hz
>     1000 hz
>  
> While my case might be different from sombody else's case, I'm sure it
> would be useful to give three or four examples of different server
> uses that at least "in theory" would benefit from a particular group
> of settings. I'm not sure that "desktop" versus "server" is the most
> helpful distinction. Are ALL servers (including mine) best off with: 
>     No premption
>     Saying No to Preempt The Big Kernel Lock
>     Timer Frequence of 100hz
>  
> Somehow, I doubt it. 

As I explained up above. Yes!

>  
> Yeah, I'm ready to experiment. I've been doing that with Linux for
> three years and getting great results. But I'd love to hear the theory
> of what SHOULD be best. It's about seeing the forest through the
> trees, getting the "big picture" after looking at all the minute
> details. 
>  
> Thanks in advance for more information... Linux rocks! 
>  
> 

-- Steve

Premptible Kernels and Timer Frequencies

[AndyLiebman]

Thanks to all responsible for getting the 2.6.14  kernel out quickly. There 
are many important bugfixes and features in this  kernel that are relevant to 
my work. I look forward to using it starting today!  

I have a request. Would it be possible for a knowledgeable person to  write 
up a little description of some of the following kernel options and what  they 
REALLY mean and how to use them? I think it would be great to put a brief  
"white paper" on the kernel.org site. 

In specific, I am referring to the  following: 

Premptible Kernel Model
No  Premption
Voluntary Premption
Premptible Kernel (Low-latency Desktop)

Prempt The Big Kernel  Lock

Timer Frequency
100 Hz
250 Hz
1000 Hz


I think it would be helpful for  the whole Linux community to get better 
hints about the theoretical optimal  settings for various types of systems. Saying 
that one option is good for  "desktops" and another is good for "servers" is 
a start, but I'm sure that with  just a few more words, you could be much more 
helpful to users who are trying to  decide what options to select for 
compiling. 

For instance, I have a  number of file servers that send video files out 
simultaneously to multiple  video editing systems. Each of 10-20 video editing 
systems could be reading  files from the server (or writing files to the server) 
at data rates ranging  from 3.5 MB/sec up to 30,40 or 50 MB/sec. 

The server ITSELF is rarely  running any applications other than Samba, 
Netatalk, Software RAID 0, NIC  drivers, and fairly low-CPU usage things like the 
UPS monitor, 3ware Drivers for  Hardware RAID 5, and KDE (which is hardly ever 
getting any input from users).  

We get really great performance with the 2.6.12 and 2.6.10 kernels. And  
typically, CPU usage ranges from 10-20 perecent with many client systems going.  
However, we are wondering if it is worth it to try the Preemptible Kernels for  
greater responsiveness and perhaps support for more simultaneous streams, 
which  MUST arrive at the video editing system on time or we get "dropped 
frames".  

I'm sure it's a similar situtation to Video On Demand servers only our  data 
rates are much higher per client, so we can't support that many clients as  a 
VOD system. 

We have tested compiling the 2.6.12 kernel with both  voluntary preemption 
and low-latency preemption. Both seem to work fine, but we  haven't 
stress-tested them yet because we were waiting for the 2.6.14 kernel to  come out. 

Could we get into any danger with the preemptible kernels if  the system load 
gets heavy? 

And now that the Timer frequency option has  been added, what would be 
optimal for such a video server that is not itself  running any taxing applications? 

Hope to hear from one of the "big guys"  on this. 

Thanks again for all your great work. 

Andy Liebman  

Re: Premptible Kernels and Timer Frequencies

[Steven Rostedt]

On Fri, 2005-10-28 at 12:36 -0400, AndyLiebman@aol.com wrote:
> Thanks to all responsible for getting the 2.6.14  kernel out quickly. There 
> are many important bugfixes and features in this  kernel that are relevant to 
> my work. I look forward to using it starting today!  
> 
> I have a request. Would it be possible for a knowledgeable person to  write 
> up a little description of some of the following kernel options and what  they 
> REALLY mean and how to use them? I think it would be great to put a brief  
> "white paper" on the kernel.org site. 

I'll respond a little here, but this is far from being a white paper.
Although, if I have time, I'll write one.  My involvement with Linux
today is mostly with Ingo's RT patch that really brings down the
latencies of the kernel.

> 
> In specific, I am referring to the  following: 
> 
> Premptible Kernel Model
> No  Premption

Simple.  If a process is in the kernel (via a system call) it will not
be preempted by another process.  There are points in the kernel where a
process will need to wait for an event (like memory swapping in) and
will call schedule to let other processes run, instead of a busy loop.

The problem with this is that when an interrupt comes in and wakes up a
higher priority process. That high priority process needs to wait till
the lower priority process either explicitly calls schedule or returns
to user land.

> Voluntary Premption

There are several points in the kernel that "might schedule".  Like
allocating more memory, the kernel may need to swap out memory to get
more available memory, so this may or may not schedule.

With Voluntary Preemption, these points always check to see if it should
schedule because another higher process is waiting to run.  So
basically, if a higher priority process wants to run on a CPU that is
currently running a lower priority process that happens to be in the
kernel. The "need_resched" flag is set on the lower priority process and
when one of these "voluntary preemption" points are hit, it will call
schedule.  The high priority process still needs to wait till one of
these points are hit, or the process actually calls schedule, or it goes
back to user space.

> Premptible Kernel (Low-latency Desktop)

This is currently the lowest latency selection in the vanilla kernel.
When this is turned on, a high priority process that is woken up, will
preempt the currently running lower priority process even if it is in
the kernel.

The trick here, is that there are SMP protected areas in the kernel
protected by spin_locks.  These usually are only for multiprocessor
machines, but when you have a fully preemptible kernel, these areas must
also be protected.  So, on a uni-processor machine without "Preempt
Kernel", these spin_locks are optimized out (no-ops).  But with "Preempt
Kernel", these areas prevent preemption from occurring. So now the high
priority process can preempt the lower priority process anytime it is
not in one of theses areas protected by a spin_lock.  If the lower
priority process is in one of these areas, the higher priority process
only needs to wait for that process to leave this area, which is usually
a very short time, since on SMP this would cause a CPU to spin in a busy
loop, and Linux tries to make these areas small.

This explanation is a little simple but gets the gist of what is
happening.

As for which one is best for you.  It matters what type of latencies you
want.  That is, do you want better reaction times for higher priority
processes.  Without the user caring about priorities, this usually means
the most interactive tasks, where servers that serve web pages and such
probably don't care about one process over the other. But a gamer might
be more interested in his game having higher priority than some
background job, and having a lower latency in shooting the monster.
 
> 
> Prempt The Big Kernel  Lock

The Big Kernel Lock is an old relic that I wish would go away.  Before
spin_locks, everything was protected with this behemoth.  One lock to
rule them all!  So basically, any time you needed to protect some data,
you would grab this lock.  So on SMP machines, CPUs were waiting around
a lot to handle some data because some other CPU was working on some
totally separate data.  This obviously was very inefficient.  Well,
fortunately, the kernel developers implemented a fine grain locking
(spin_locks) and got rid of most of the Big Kernel Lock.  But
unfortunately, it is still used in some pretty scary parts of the
kernel.  Scary, because it will be difficult to remove this lock for in
favor of smaller variants.

This lock is different than the spin_locks by the fact that you can call
schedule holding this lock.  Schedule will simply let go of this lock
for the scheduling process.  When the process is scheduled again, it
will try to retake the lock.  Also, a process may grab this lock as many
times as it wants while holding it, as long as it releases it the same
amount of times.

To preempt while this lock is held, takes some special care, since you
don't want to let go of this lock when it is being preempted.  So work
is done to prevent the scheduler from releasing this lock when it calls
schedule because of preemption and not because of the process
voluntarily scheduling.  Also care is taken when retaking the lock when
woken up in the scheduler since the process might not get it because a
process it just preempted has it.

So this is more complex, and the developers decided to let people turn
it off if they are nervous about the complexity on critical servers.

> 
> Timer Frequency
> 100 Hz
> 250 Hz
> 1000 Hz

Again, this has to do with response times and latencies.  Processes are
given time slices that they can run in. These are broken up into
"jiffies".  A jiffy is defined in the kernel as 1/HZ second.  So at
100Hz, a jiffy is 10ms, 250Hz 4ms and 1000Hz 1ms.

Also the resolution of sleeps and timers are done at the jiffy level
(with current vanilla Linux).  So at 100Hz a timer set to go off in
100ms is most likely to go off in 110ms where as with the 1000Hz it has
a better probability of going off in 101ms.  Note, this is not a good
example, because latencies usually cause timers and such to not be
predictable in how close to the actual time it goes off.  It only needs
to guarantee that it doesn't go off early.

Also note, the better the latency, the higher the over head.  This is
pretty much true in all cases.  For HZ = 1000, the timer interrupt needs
to go off 1000 times a second, which will cause 900 more interrupts to
go off in that second than for HZ = 100.

> 
> 
> I think it would be helpful for  the whole Linux community to get better 
> hints about the theoretical optimal  settings for various types of systems. Saying 
> that one option is good for  "desktops" and another is good for "servers" is 
> a start, but I'm sure that with  just a few more words, you could be much more 
> helpful to users who are trying to  decide what options to select for 
> compiling. 

I'm not a server maintainer or much of a gamer, so someone else can
explain what is best for them.  I'm more of the embedded or special
purpose tool guy.  But I do understand what these things actually mean.

> 
> For instance, I have a  number of file servers that send video files out 
> simultaneously to multiple  video editing systems. Each of 10-20 video editing 
> systems could be reading  files from the server (or writing files to the server) 
> at data rates ranging  from 3.5 MB/sec up to 30,40 or 50 MB/sec. 
> 
> The server ITSELF is rarely  running any applications other than Samba, 
> Netatalk, Software RAID 0, NIC  drivers, and fairly low-CPU usage things like the 
> UPS monitor, 3ware Drivers for  Hardware RAID 5, and KDE (which is hardly ever 
> getting any input from users).  
> 
> We get really great performance with the 2.6.12 and 2.6.10 kernels. And  
> typically, CPU usage ranges from 10-20 perecent with many client systems going.  
> However, we are wondering if it is worth it to try the Preemptible Kernels for  
> greater responsiveness and perhaps support for more simultaneous streams, 
> which  MUST arrive at the video editing system on time or we get "dropped 
> frames".  

Have you taken a look at Ingo Molnar's PREEMPT_RT?

http://people.redhat.com/mingo/realtime-preempt/

This has the best in low latency than any of the above options.  But I
must also warn you that the lower the latency, the performance will
usually take a slight hit as well. But we try to keep that at a minimum.

> 
> I'm sure it's a similar situtation to Video On Demand servers only our  data 
> rates are much higher per client, so we can't support that many clients as  a 
> VOD system. 
> 
> We have tested compiling the 2.6.12 kernel with both  voluntary preemption 
> and low-latency preemption. Both seem to work fine, but we  haven't 
> stress-tested them yet because we were waiting for the 2.6.14 kernel to  come out. 
> 
> Could we get into any danger with the preemptible kernels if  the system load 
> gets heavy? 

Preemptible kernels are suppose to work _better_ on higher load.
Performance goes slightly down, but predictability goes way up.

> 
> And now that the Timer frequency option has  been added, what would be 
> optimal for such a video server that is not itself  running any taxing applications? 

I'm not sure if that would effect it. If it is interrupt driven, the
timer frequency is of no use. If the application is setting up its own
frequency, then that would be a different story.

> 
> Hope to hear from one of the "big guys"  on this. 

I'm not a "big guy" ;-) but I'm pretty active with Ingo's RT patch.  I
just submit bug fixes for the vanilla guys.

> 
> Thanks again for all your great work. 

Hope this explains things a bit better.  I've been meaning to write up
something about Ingo's patch to explain things in his kernel options
just like this.  When I get some time, maybe I'll do that too.

-- Steve