Gurus, a silly question for preemptive behavior

Gurus, a silly question for preemptive behavior

[jesse]

 
As i know, in linux, user space application is
preemptive at any time. however, linux kernel is NOT
preemptive, that means, even some event is finished,
Linux kernel scheduler itself still can't have
opportunity to interrupt the current user application
and switch it out. it is called scheduler latency.
normally , the latency is about 88us in mean , maximum
: 200ms. Thus, the short latency shouldn't impact user
applications too much and is not a problem. It is an
issue in those embedded voice processing systems by
introducing jitters, thus smart people came up with
two kernel schedule patch: preemptive patch and low
latency patch. 

My application which has nice value as 10 of low
priority, however, it holds the CPU and excludes other
applciations that have higher priority (nice 0) to
run, my application causes CPU pegging. 

Thus, I am wondering: why  user space application
can't be effectively interrupted? why there is CPU
pegging?   Could you please educate me 
on this particular issue and shed me some light to
address it? 

my system: 
[root@sa-c2-7 proc]# uname  -a 
Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22
00:18:24 EDT 2004 i686 i686 i386 GNU/Linux 

thanks!

Re: Gurus, a silly question for preemptive behavior

[Con Kolivas]

jesse writes:

>  
> As i know, in linux, user space application is
> preemptive at any time. however, linux kernel is NOT
> preemptive, that means, even some event is finished,
> Linux kernel scheduler itself still can't have
> opportunity to interrupt the current user application
> and switch it out. it is called scheduler latency.

The kernel is preemptible if you enable the preempt option in the 
configuration. There are some code paths that are not preemptible despite 
this, but they are gradually being improved over time.

> normally , the latency is about 88us in mean , maximum
> : 200ms. Thus, the short latency shouldn't impact user
> applications too much and is not a problem. It is an
> issue in those embedded voice processing systems by
> introducing jitters, thus smart people came up with
> two kernel schedule patch: preemptive patch and low
> latency patch. 

You're thinking about the 2.4 kernel. 2.6 is effectively both of those 
patches inclusive.

> my system: 
> [root@sa-c2-7 proc]# uname  -a 
> Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22
> 00:18:24 EDT 2004 i686 i686 i386 GNU/Linux 

If you want lower latency on a 2.4 kernel you need further patches. You are 
most likely to benefit from a move to a 2.6 kernel and enabling preempt.

Cheers,
Con

Re: Gurus, a silly question for preemptive behavior

[jesse]

Con:

   thank you for your prompt reply in the holiday
season. 

   My point is: Even kernel 2.4 is not 
preemptive, the latence should be very
minimal.(<300ms)
why user space application with low nice priority
can't be effectively interrupted and holds the CPU
resource since all user space application is
preemptive?

   Merry Xmas.

jesse

  

--- Con Kolivas <kernel@kolivas.org> wrote:

> jesse writes:
> 
> >  
> > As i know, in linux, user space application is
> > preemptive at any time. however, linux kernel is
> NOT
> > preemptive, that means, even some event is
> finished,
> > Linux kernel scheduler itself still can't have
> > opportunity to interrupt the current user
> application
> > and switch it out. it is called scheduler latency.
> 
> The kernel is preemptible if you enable the preempt
> option in the 
> configuration. There are some code paths that are
> not preemptible despite 
> this, but they are gradually being improved over
> time.
> 
> > normally , the latency is about 88us in mean ,
> maximum
> > : 200ms. Thus, the short latency shouldn't impact
> user
> > applications too much and is not a problem. It is
> an
> > issue in those embedded voice processing systems
> by
> > introducing jitters, thus smart people came up
> with
> > two kernel schedule patch: preemptive patch and
> low
> > latency patch. 
> 
> You're thinking about the 2.4 kernel. 2.6 is
> effectively both of those 
> patches inclusive.
> 
> > my system: 
> > [root@sa-c2-7 proc]# uname  -a 
> > Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22
> > 00:18:24 EDT 2004 i686 i686 i386 GNU/Linux 
> 
> If you want lower latency on a 2.4 kernel you need
> further patches. You are 
> most likely to benefit from a move to a 2.6 kernel
> and enabling preempt.
> 
> Cheers,
> Con
> 
> 

Re: Gurus, a silly question for preemptive behavior

[Paulo Marques]

jesse wrote:
> Con:
> 
>    thank you for your prompt reply in the holiday
> season. 
> 
>    My point is: Even kernel 2.4 is not 
> preemptive, the latence should be very
> minimal.(<300ms)
> why user space application with low nice priority
> can't be effectively interrupted and holds the CPU
> resource since all user space application is
> preemptive?

If your process has got work to do and has a higher priority than other 
processes, it gets to run. If you don't want this behavior, don't give 
it such a high priority.

If you want low latency to do some quick high priority task, just do it 
quickly and relinquish the processor, instead of hogging it.

What are you trying to accomplish, anyway?

-- 
Paulo Marques - www.grupopie.com

"A journey of a thousand miles begins with a single step."
Lao-tzu, The Way of Lao-tzu

Re: Gurus, a silly question for preemptive behavior

[jesse]

Paulo:
 
   I already said in the messsage that my user space
application has a low nice priority, i set it to 10.
since my application has low priority compared to
other user space applications, it is supposed to be
interrupted. but it is not.

 regards!

jesse


--- Paulo Marques <pmarques@grupopie.com> wrote:

> jesse wrote:
> > Con:
> > 
> >    thank you for your prompt reply in the holiday
> > season. 
> > 
> >    My point is: Even kernel 2.4 is not 
> > preemptive, the latence should be very
> > minimal.(<300ms)
> > why user space application with low nice priority
> > can't be effectively interrupted and holds the CPU
> > resource since all user space application is
> > preemptive?
> 
> If your process has got work to do and has a higher
> priority than other 
> processes, it gets to run. If you don't want this
> behavior, don't give 
> it such a high priority.
> 
> If you want low latency to do some quick high
> priority task, just do it 
> quickly and relinquish the processor, instead of
> hogging it.
> 
> What are you trying to accomplish, anyway?
> 
> -- 
> Paulo Marques - www.grupopie.com
> 
> "A journey of a thousand miles begins with a single
> step."
> Lao-tzu, The Way of Lao-tzu
> 

Re: Gurus, a silly question for preemptive behavior

[Paulo Marques]

jesse wrote:
> Paulo:
>  
>    I already said in the messsage that my user space
> application has a low nice priority, i set it to 10.
> since my application has low priority compared to
> other user space applications, it is supposed to be
> interrupted. but it is not.

The confusion comes from "low nice priority". The lower the nice value 
the higher the priority.

Anyway, you still haven't give enough data to analyze. What does your 
application do? Is it I/O intensive? If it is, it could be that the 
kernel itself is hogging the CPU doing I/O on behalf of a low prio 
process (priority, specially in 2.4, only affects CPU distribution and 
not I/O). How do you know it's not being preempted? What is your 
.config? What patches do you have applied? And finally, why don't you 
upgrade to a 2.6 kernel :) ?

-- 
Paulo Marques - www.grupopie.com

"A journey of a thousand miles begins with a single step."
Lao-tzu, The Way of Lao-tzu

Re: Gurus, a silly question for preemptive behavior

[Con Kolivas]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

jesse wrote:
> Paulo:
>  
>    I already said in the messsage that my user space
> application has a low nice priority, i set it to 10.
> since my application has low priority compared to
> other user space applications, it is supposed to be
> interrupted. but it is not.

If your task is better priority the scheduler will make it preempt the 
worse priority task. It sounds to me like you are complaining that the 
worse priority task is still getting cpu? If so, you misunderstand 
priority - it orders tasks according to priority giving lower latency 
and preemptive behaviour to the better task, and gives _more_ cpu but 
not all the cpu. The cpu must still be shared, but with more cpu 
distributed to the better priority task. If you want your better 
priority task to get _all_ the cpu you have to use real time scheduling.

Cheers,
Con

P.S. Please dont top post when replying, it's hard to follow an email 
thread to see what you're replying to if you do that. These threads can 
get very long and confusing.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: Gurus, a silly question for preemptive behavior

[jesse]

--- Con Kolivas <kernel@kolivas.org> wrote:

> jesse wrote:
> > Paulo:
> >  
> >    I already said in the messsage that my user
> space
> > application has a low nice priority, i set it to
> 10.
> > since my application has low priority compared to
> > other user space applications, it is supposed to
> be
> > interrupted. but it is not.
> 
> If your task is better priority the scheduler will
> make it preempt the 
> worse priority task. It sounds to me like you are
> complaining that the 
> worse priority task is still getting cpu? If so, you
> misunderstand 
> priority - it orders tasks according to priority
> giving lower latency 
> and preemptive behaviour to the better task, and
> gives _more_ cpu but 
> not all the cpu. The cpu must still be shared, but
> with more cpu 
> distributed to the better priority task. If you want
> your better 
> priority task to get _all_ the cpu you have to use
> real time scheduling.
> 
> Cheers,
> Con
> 

ok, Con, your explaining makes some sense to me , but
still not very well.

   suppose I have five high process: A1, A2, A3, A4,
A5 all have nice = 0. and I also have a low priority
process B with nice = 10.

    a) when process B is scheduled to run, it is given
a short time slot based on its priority, for example 5
secs. because at that point, A1/2/3/4/5 are not
started yet. B will get CPU and run at full speed. 
    b) at the end of time slot(5 secs), scheduler
finds higher priority A1/A2/A3/A4/A5 are ready,
scheduler will interrupt process B and starts to pick 
a process from group A, even though B still needs CPU
cycle.
    c)unfortunately, process A1/2/3/4/5 are so active,
thus process B should never get opportunity to run
again, in consequence, CPU Usage% of Process B should
be very Low.
    
   However, The above theretic assumption is in
contrary to what i observed. in my LAB, the low
priority process B seems to hold the CPU forever and
Top command always shows Process B with a 90% CPU
usage.
  
  If _more_ cpu but not _all_ the cpu are given to
Process A1/2/3/4/5, Process B shouldn't have a 90% CPU
usage.   

  Thus, i can't help asking why low priority process B
gets most CPU cycle.

  thanks in advance. 

jesse

 

Re: Gurus, a silly question for preemptive behavior

[Con Kolivas]

jesse writes:

> 
> --- Con Kolivas <kernel@kolivas.org> wrote:
> 
>> jesse wrote:
>> > Paulo:
>> >  
>> >    I already said in the messsage that my user
>> space
>> > application has a low nice priority, i set it to
>> 10.
>> > since my application has low priority compared to
>> > other user space applications, it is supposed to
>> be
>> > interrupted. but it is not.
>> 
>> If your task is better priority the scheduler will
>> make it preempt the 
>> worse priority task. It sounds to me like you are
>> complaining that the 
>> worse priority task is still getting cpu? If so, you
>> misunderstand 
>> priority - it orders tasks according to priority
>> giving lower latency 
>> and preemptive behaviour to the better task, and
>> gives _more_ cpu but 
>> not all the cpu. The cpu must still be shared, but
>> with more cpu 
>> distributed to the better priority task. If you want
>> your better 
>> priority task to get _all_ the cpu you have to use
>> real time scheduling.
>> 
>> Cheers,
>> Con
>> 
> 
> ok, Con, your explaining makes some sense to me , but
> still not very well.
> 
>    suppose I have five high process: A1, A2, A3, A4,
> A5 all have nice = 0. and I also have a low priority
> process B with nice = 10.
> 
>     a) when process B is scheduled to run, it is given
> a short time slot based on its priority, for example 5
> secs. because at that point, A1/2/3/4/5 are not
> started yet. B will get CPU and run at full speed. 
>     b) at the end of time slot(5 secs), scheduler
> finds higher priority A1/A2/A3/A4/A5 are ready,
> scheduler will interrupt process B and starts to pick 
> a process from group A, even though B still needs CPU
> cycle.
>     c)unfortunately, process A1/2/3/4/5 are so active,
> thus process B should never get opportunity to run
> again, in consequence, CPU Usage% of Process B should
> be very Low.
>     
>    However, The above theretic assumption is in
> contrary to what i observed. in my LAB, the low
> priority process B seems to hold the CPU forever and
> Top command always shows Process B with a 90% CPU
> usage.
>   
>   If _more_ cpu but not _all_ the cpu are given to
> Process A1/2/3/4/5, Process B shouldn't have a 90% CPU
> usage.   
> 
>   Thus, i can't help asking why low priority process B
> gets most CPU cycle.

What you are describing is completely wrong behaviour. Please post output of 
top running during this workload to demonstrate/prove this is happening. 
Easiest thing to do is get your workload running and do 'top -b -n 1 > 
top.log' and post top.log please.

Cheers,
Con

Re: Gurus, a silly question for preemptive behavior

[K.R. Foley]

jesse wrote:
> --- Con Kolivas <kernel@kolivas.org> wrote:
> 
> 
>>jesse wrote:
>>
>>>Paulo:
>>> 
>>>   I already said in the messsage that my user
>>
>>space
>>
>>>application has a low nice priority, i set it to
>>
>>10.
>>
>>>since my application has low priority compared to
>>>other user space applications, it is supposed to
>>
>>be
>>
>>>interrupted. but it is not.
>>
>>If your task is better priority the scheduler will
>>make it preempt the 
>>worse priority task. It sounds to me like you are
>>complaining that the 
>>worse priority task is still getting cpu? If so, you
>>misunderstand 
>>priority - it orders tasks according to priority
>>giving lower latency 
>>and preemptive behaviour to the better task, and
>>gives _more_ cpu but 
>>not all the cpu. The cpu must still be shared, but
>>with more cpu 
>>distributed to the better priority task. If you want
>>your better 
>>priority task to get _all_ the cpu you have to use
>>real time scheduling.
>>
>>Cheers,
>>Con
>>
> 
> 
> ok, Con, your explaining makes some sense to me , but
> still not very well.
> 
>    suppose I have five high process: A1, A2, A3, A4,
> A5 all have nice = 0. and I also have a low priority
> process B with nice = 10.
> 
>     a) when process B is scheduled to run, it is given
> a short time slot based on its priority, for example 5
> secs. because at that point, A1/2/3/4/5 are not
> started yet. B will get CPU and run at full speed. 
>     b) at the end of time slot(5 secs), scheduler
> finds higher priority A1/A2/A3/A4/A5 are ready,
> scheduler will interrupt process B and starts to pick 
> a process from group A, even though B still needs CPU
> cycle.
>     c)unfortunately, process A1/2/3/4/5 are so active,
> thus process B should never get opportunity to run
> again, in consequence, CPU Usage% of Process B should
> be very Low.
>     
>    However, The above theretic assumption is in
> contrary to what i observed. in my LAB, the low
> priority process B seems to hold the CPU forever and
> Top command always shows Process B with a 90% CPU
> usage.
>   
>   If _more_ cpu but not _all_ the cpu are given to
> Process A1/2/3/4/5, Process B shouldn't have a 90% CPU
> usage. 

This statement is not exactly true. If process B is always ready to run 
(CPU bound, not I/O bound) it could easily consume more CPU than all the 
other processes combined. It all depends on what A1/2/3/4/5 are doing. 
Just because A1 has a higher priority doesn't mean it is ready to run. 
If processes A1/2/3/4/5 spend most of their time waiting for I/O or 
sleeping (not ready to run) and B does a lot of computation or just busy 
spins, B could easily consume more CPU than the others.

> 
>   Thus, i can't help asking why low priority process B
> gets most CPU cycle.
> 
>   thanks in advance. 
> 
> jesse
> 
>  
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Gurus, a silly question for preemptive behavior

[Stephen Satchell]

jesse wrote:
>  
> As i know, in linux, user space application is
> preemptive at any time. however, linux kernel is NOT
> preemptive, that means, even some event is finished,
> Linux kernel scheduler itself still can't have
> opportunity to interrupt the current user application
> and switch it out. it is called scheduler latency.
> normally , the latency is about 88us in mean , maximum
> : 200ms. Thus, the short latency shouldn't impact user
> applications too much and is not a problem. It is an
> issue in those embedded voice processing systems by
> introducing jitters, thus smart people came up with
> two kernel schedule patch: preemptive patch and low
> latency patch. 
> 
> My application which has nice value as 10 of low
> priority, however, it holds the CPU and excludes other
> applciations that have higher priority (nice 0) to
> run, my application causes CPU pegging. 
> 
> Thus, I am wondering: why  user space application
> can't be effectively interrupted? why there is CPU
> pegging?   Could you please educate me 
> on this particular issue and shed me some light to
> address it? 
> 
> my system: 
> [root@sa-c2-7 proc]# uname  -a 
> Linux sa-c2-7 2.4.21-15.ELsmp #1 SMP Thu Apr 22
> 00:18:24 EDT 2004 i686 i686 i386 GNU/Linux 

First, a pre-emption happens when an external event occurs that makes a 
higher priority task change state from blocked to ready-to-run, and the 
scheduling algorithm says that the new task should be run in place of 
the old task -- and the complexity of the time-slice schedule is such 
that your assumption doesn't always hold true.

If you have a low-priority task that needs to make sure that 
higher-priority functions *always* run, and you can't move up to the 2.6 
kernel (or even after moving up you find yourself with problems) the 
userland solution is a simple one:

   if (1) {
    struct timeval delay = {0,20000};
    select(0, NULL, NULL, NULL, &delay);
    }

(DON'T declare the variable delay as constant, because select modifies 
the contents of this variable.)

A number of real-time operating systems provide for a give-up-control 
system call, which says "if you have something better to do than to run 
me, then do it" for just this sort of thing.  The legacy of the Linux 
schedules being from a classic time-sharing system, which trys to 
implement "fairness" in allocating CPU resources,  there is no 
equivalent call that I'm aware of.

So you have to make your program block in order to guarantee that you 
give up the CPU.  The delay of 20 ms in my example call assumes your HZ 
value is 100, and that you really don't mind having a few idle CPU 
cycles if your higher-priority program isn't ready to run when you make 
the select call.  In my Web servers, I use 100 ms delays to ensure that 
I/O for other processes is allowed to take place; Webalizer stats, for 
example, have hogged disk access to the point that it impacts the entire 
system, so I've inserted the Perl equivalent of the above code in key 
places so that background tasks really do run in the background.

A delay value of zero to select is, I believe, an effective no-operation 
which may or may not trigger a scheduling event.  (I looked a long time 
ago at this -- many, MANY kernels ago -- and it's as though you didn't 
make the call at all, absent the CPU cycles expended to make the call 
and for the kernel to get back to the userland task.)  A value of 10 ms 
(on a system with HZ set to 100) can also be taken as though you coded 
zero in some cases -- at least some code I wrote a couple of years ago 
suggests this.

Why select?  I write portable code, and this is in my cookbook.  There 
is a nanosleep system call that is part of POSIX, and that looks to be a 
preferable way to deal with things.  You would need to look at the Linux 
source code to see what a nanosleep with a value of one nanosecond does, 
and whether that's guaranteed to block your code for one scheduling cycle.

How frequently you put scheduling breaks into your code is up to you.  I 
typicall profile code, and try to put in scheduling breaks at 1/4 of the 
desired system response latency.  For a 100 ms latency, that means that 
I do scheduling breaks no less frequently than 25 ms., and in many cases 
more often than that.  Don't forget that I/O provides breaks, too, so 
the number of places where you need to put in explicit scheduling breaks 
may be small, indeed.

For Webalizer, I run a scheduling break every 100 lines read from the 
log files.  That count, by experimentation, has shown to provide a good 
balance between throughout of the background process and system reaction 
latency.  That's stuff running on a 2.4 kernel, by the way, because I'm 
slow to adopt new, bleeding-edge stuff in my production environments.

A couple of thoughts from a different place in the trenches.

Stephen Satchell