SCHED_FIFO & system()

SCHED_FIFO & system()

[linux]

Hello,
I have some strange behavior in one of my systems.

I have a real-time kernel thread under SCHED_FIFO which is running every
10ms.
It is blocking on a semaphore and released by a timer interrupt every 10ms.
Generally this works really well.

However, there is a module in the system that makes a / system() / call
from c-code in user-space;

  system("run_my_script");

By calling and running a bash script. Independent of how the actual
script looks like the real time kernel thread does not get scheduled under
the time of 80ms -- the time it takes for the system() call to finish.

I can see when running a LTT session that the wake_up event occurs for
the real time thread 10ms into the system call but nevertheless the real
time kernel thread does not get scheduled.

The thread that calls system("run_my_script") is configured as SCHED_OTHER.

The Kernel is 2.6.21.


Anybody who recognize this or similar situations?



Cheers // Matias

Re: SCHED_FIFO & system()

[Matias Sundman]

Hi,
I forgot to add one important detail - the Kernel is configured as - 
Preemptible Kernel (Low-Latency Desktop).

Cheers // Matias

linux@sundmangroup.com skrev:
> Hello,
> I have some strange behavior in one of my systems.
>
> I have a real-time kernel thread under SCHED_FIFO which is running every
> 10ms.
> It is blocking on a semaphore and released by a timer interrupt every 10ms.
> Generally this works really well.
>
> However, there is a module in the system that makes a / system() / call
> from c-code in user-space;
>
>   system("run_my_script");
>
> By calling and running a bash script. Independent of how the actual
> script looks like the real time kernel thread does not get scheduled under
> the time of 80ms -- the time it takes for the system() call to finish.
>
> I can see when running a LTT session that the wake_up event occurs for
> the real time thread 10ms into the system call but nevertheless the real
> time kernel thread does not get scheduled.
>
> The thread that calls system("run_my_script") is configured as SCHED_OTHER.
>
> The Kernel is 2.6.21.
>
>
> Anybody who recognize this or similar situations?
>
>
>
> Cheers // Matias
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>   

Re: SCHED_FIFO & system()

[Gregory Haskins]

linux@sundmangroup.com wrote:
> Hello,
> I have some strange behavior in one of my systems.
> 
> I have a real-time kernel thread under SCHED_FIFO which is running every
> 10ms.
> It is blocking on a semaphore and released by a timer interrupt every 10ms.
> Generally this works really well.
> 
> However, there is a module in the system that makes a / system() / call
> from c-code in user-space;
> 
>   system("run_my_script");
> 
> By calling and running a bash script. Independent of how the actual
> script looks like the real time kernel thread does not get scheduled under
> the time of 80ms -- the time it takes for the system() call to finish.
> 
> I can see when running a LTT session that the wake_up event occurs for
> the real time thread 10ms into the system call but nevertheless the real
> time kernel thread does not get scheduled.
> 
> The thread that calls system("run_my_script") is configured as SCHED_OTHER.
> 
> The Kernel is 2.6.21.
> 
> 
> Anybody who recognize this or similar situations?


Hi,
   I am not sure if this is whats happening to you (*), but there is a 
known problem in the mainline tree related to the system allowing lower 
priority threads to stay on the CPU under certain circumstances.

You can see the problem happening with the following test:

http://rt.wiki.kernel.org/index.php/Preemption_Test

and I know the problem goes back to at least 2.6.16.

Steven Rostedt and I submitted some patches to address the issue, and 
they have been accepted into Ingo's sched-devel tree.  I would suggest 
trying your code against Ingo's tree and see if the problem persists. 
You can get it from:

git://git.kernel.org/pub/scm/linux/kernel/git/mingo/linux-2.6-sched-devel.git

In the event that this does fix your issue, I will defer to Ingo for 
if/when he plans to push these upstream.

(*) I have no information on whether the futex-plist implemetation was 
pulled from the tree to cause your regression.  It is possible that the 
changes between 22 and 23 are just tickling your environment enough to 
bring out this RT-preempt issue.

HTH!

Regards,
-Greg

Re: SCHED_FIFO & system()

[Gregory Haskins]

Gregory Haskins wrote:

> 
> (*) I have no information on whether the futex-plist implemetation was 
> pulled from the tree to cause your regression.  It is possible that the 
> changes between 22 and 23 are just tickling your environment enough to 
> bring out this RT-preempt issue.
> 


Hmm...seems I should not send out email before my morning coffee ;)

I think I was answering

http://lkml.org/lkml/2008/1/18/43

having read them back to back.  Sorry for the confusion.


There's a chance that this is what is causing your issue, Matias, but 
you should also confirm that what this module is doing doesn't disable 
preemption.  That would also cause the issue.

HTH
-Greg

Re: SCHED_FIFO & system()

[Steven Rostedt]

On Thu, 17 Jan 2008 linux@sundmangroup.com wrote:
> time kernel thread does not get scheduled.
>
> The thread that calls system("run_my_script") is configured as SCHED_OTHER.
>
> The Kernel is 2.6.21.

Could you try the latest kernel, and if that still gives you problems, try
out Ingo's sched-devel (from what Gregory Haskins posted).

Let us know how it turns out.

Thanks,

-- Steve


>
>
> Anybody who recognize this or similar situations?
>
>

Re: SCHED_FIFO & System()

[Arnon Kaufman(Work)]

[-- Attachment #1: Type: text/plain, Size: 2161 bytes --]

> Hello,
> I have some strange behavior in one of my systems.
>
> I have a real-time thread under SCHED_FIFO which is running every 10ms.
> It is blocking on a semaphore and released by a timer interrupt every
> 10ms.
> Generally this works really well.
>

I've seen this kind of behavior. i may guess that you configure the kernel
with PREEMPT, assuming it will solve the problem for you.

I'll split the problem into two:
1. program loading stage is finished (+ reschedule)
2. thread execution time (delay)

when executing a new application it's involved with plenty of kernel
activity such as fetching the application from storage, loader
relocation,allocating and remapping pages, resolving dynamic linking etc...,
all the activity occurs under the hood of the kernel, suspending most of the
kernel activities, due to many spin locks on the way. that delays other
kernel threads as well, and even workqueues.

trying to bypass the symptom  - re-running the exact external application ,
should not be involved with a second delay, as it was cached. so may be
running the external application in advance may be good enough for you.

what you actually need is a better kernel preemption such as  using Ingo
Molnar 's  PREEMPT_RT patch. (it may reduce performance a little).

the second part is that the scheduler delays the thread execution as it
think other kernel's entities has a higher priority such as softirqs,
workqueue, etc...

the PREEMPT_RT may solve this problem as well as it bring a better priority
control, with the addition of priority inheritance.


> However, there is a module in the system that makes a / system() / call
> from c-code ;
>
>  system("run_my_script");
>
> By calling and running a bash script. Independent of how the actual
> script looks like the real time thread does not get scheduled under 80ms
> -- the time it takes
> for the system() call to finish.
>
> I can see when running a LTT session that the wake_up event occurs for
> the real time thread 10ms into the system call but nevertheless the real
> time thread does not get scheduled.
>
> Anybody who recognize this or similar situations?
>
>
>
> Cheers // Matias
>

[-- Attachment #2: Type: text/html, Size: 2639 bytes --]

SCHED_FIFO & System()

[Matias Sundman]

Hello,
I have some strange behavior in one of my systems.

I have a real-time thread under SCHED_FIFO which is running every 10ms. 
It is blocking on a semaphore and released by a timer interrupt every 10ms.
Generally this works really well.

However, there is a module in the system that makes a / system() / call 
from c-code ;

  system("run_my_script");

By calling and running a bash script. Independent of how the actual 
script looks like the real time thread does not get scheduled under 80ms 
-- the time it takes
for the system() call to finish.

I can see when running a LTT session that the wake_up event occurs for 
the real time thread 10ms into the system call but nevertheless the real 
time thread does not get scheduled.

Anybody who recognize this or similar situations?



Cheers // Matias

Re: SCHED_FIFO & System()

[Scott Wood]

Matias Sundman wrote:
> Hello,
> I have some strange behavior in one of my systems.
> 
> I have a real-time thread under SCHED_FIFO which is running every 10ms. 
> It is blocking on a semaphore and released by a timer interrupt every 10ms.
> Generally this works really well.
> 
> However, there is a module in the system that makes a / system() / call 
> from c-code ;
> 
>   system("run_my_script");
> 
> By calling and running a bash script. Independent of how the actual 
> script looks like the real time thread does not get scheduled under 80ms 
> -- the time it takes
> for the system() call to finish.

Are you running a preemptible kernel?

-Scott

Re: SCHED_FIFO & System()

[Matias Sundman]

Hello,

/>>Are you running a preemptible kernel? [Scott]/
 
Yes the Kernel is configured as - Preemptible Kernel (Low-Latency Desktop)

/>>//what you actually need is a better kernel preemption such as  using 
Ingo Molnar 's  PREEMPT_RT patch. (it may reduce performance a little) 
[Arnon]
/
Ok - we can try to enable the PREEMPT_RT patches.

-------------------------------

I also tried with a sched_yield() before the call to system() in order 
to see if the behavior changed somewhat but it didn't.


Cheers // Matias :-)



Scott Wood skrev:
> Matias Sundman wrote:
>   
>> Hello,
>> I have some strange behavior in one of my systems.
>>
>> I have a real-time thread under SCHED_FIFO which is running every 10ms. 
>> It is blocking on a semaphore and released by a timer interrupt every 10ms.
>> Generally this works really well.
>>
>> However, there is a module in the system that makes a / system() / call 
>> from c-code ;
>>
>>   system("run_my_script");
>>
>> By calling and running a bash script. Independent of how the actual 
>> script looks like the real time thread does not get scheduled under 80ms 
>> -- the time it takes
>> for the system() call to finish.
>>     
>
> Are you running a preemptible kernel?
>
> -Scott
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded
>   
[From : Arnon Kaufman(Work) <arnon.work@gmail.com>]

I've seen this kind of behavior. i may guess that you configure the 
kernel with PREEMPT, assuming it will solve the problem for you.

I'll split the problem into two:
1. program loading stage is finished (+ reschedule)
2. thread execution time (delay)

when executing a new application it's involved with plenty of kernel 
activity such as fetching the application from storage, loader 
relocation,allocating and remapping pages, resolving dynamic linking 
etc..., all the activity occurs under the hood of the kernel, suspending 
most of the kernel activities, due to many spin locks on the way. that 
delays other kernel threads as well, and even workqueues.

trying to bypass the symptom  - re-running the exact external 
application ,  should not be involved with a second delay, as it was 
cached. so may be running the external application in advance may be 
good enough for you.

what you actually need is a better kernel preemption such as  using Ingo 
Molnar 's  PREEMPT_RT patch. (it may reduce performance a little).

the second part is that the scheduler delays the thread execution as it 
think other kernel's entities has a higher priority such as softirqs, 
workqueue, etc...

the PREEMPT_RT may solve this problem as well as it bring a better 
priority control, with the addition of priority inheritance.