[Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[Li Yi (Adam)]

[-- Attachment #1: Type: text/plain, Size: 2788 bytes --]

 Hi,

When a real-time thread is running, the Linux kernel will lose Timer Tick
(and lose timer interrupts), is this the expected behavior?
If yes, in this case,  no interrupts will be passed to Linux domain? This
may cause some services depending on timing in Linux domain do not work
correctly. In another thread I read:

"
I do not know if it is what you mean, but when running Xenomai, your
tasks are supposed to suspend sufficiently frequently to leave Linux
tick normally. This means that if you configured CONFIG_HZ=100, your
real-time activity (including kernel-space tasks and user-space tasks in
primary mode) is supposed to take a break at least once every 1/100
second.
--
                                           Gilles Chanteperdrix.
"
Is this the best solution? Since there may be possibility that a real-time
thread keep running for severl hundred ms (e.g, processing large block of
data).


I tried to run a test on Xenomai-2.1.0 on Blackfin. This test looks like:

In a user space RT task (created with rt_task_create()), it does:

"
    read_proc(); // Read "/proc/interrupts" and "/proc/uptime"

    for(;;)
    {

        fprintf(stderr, "+++++++++ start test +++++++++++++++\n");
        read_proc();
        fprintf(stderr, "-------- sleep for 2 sec-----\n");
        sleep(2);
        fprintf(stderr, "-------- start spin ---------\n");
        start_tsc = rt_timer_tsc();

        for (cnt = 0; cnt < 6000; cnt++)
                rt_timer_spin(1000000); // in ns

        diff = rt_timer_tsc() - start_tsc;
        fprintf(stderr, "spin for %d msec\n", rt_timer_tsc2ns(diff) /
1000000);
        read_proc();
        fprintf(stderr, "+++++++++ end test +++++++++++++++++\n");

        getchar();
    }
"

The thread calls rt_timer_spin(), which calls instruction "R0 = 0 (X)"
repeatedly until the specified time is reached. In this case, the thread
spins for 6 seconds.

Here is the test result:

+++++++++ start test +++++++++++++++
-----------------
14:          0   rtc
21:          0   BFIN_UART_RX
22:       5182   BFIN_UART_TX
23:      33531   BFIN Timer Tick
27:       1165   eth0
30:          1   timer_latency
Err:          0

335.41 0.31
-----------------
-------- sleep for 2 sec-----
-------- start spin ---------
spin for 6025 msec
-----------------
14:          0   rtc
21:          0   BFIN_UART_RX
22:       5385   BFIN_UART_TX
23:      33732   BFIN Timer Tick
27:       1165   eth0
30:          1   timer_latency
Err:          0

337.42 0.31
-----------------
+++++++++ end test +++++++++++++++++

>From the test result, " 23:  BFIN Timer Tick", changed from "33531" to
"33732", that is "201 * 10 ＝ 2010 ms". And the "up time" is changed
from "335.41 to 337.42", although in fact that "2 + 6 = 8 sec" has
passed.


Thanks,

-Li Yi (Adam)

[-- Attachment #2: Type: text/html, Size: 5471 bytes --]

Re: [Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[Philippe Gerum]

Li Yi (Adam) wrote:
> Hi,
>  
> When a real-time thread is running, the Linux kernel will lose Timer 
> Tick (and lose timer interrupts), is this the expected behavior?
>  

No, it should not lose any tick, regardless of the Xenomai port. You are
likely observing a pecularity of the Blackfin port where Adeos moves
Linux IRQ handlers (top-halves) in thread context to work-around a high
scheduling latency source which is Blackfin specific.

> If yes, in this case,  no interrupts will be passed to Linux domain? 
> This may cause some services depending on timing in Linux domain do not 
> work correctly. In another thread I read:
>  
> "
> I do not know if it is what you mean, but when running Xenomai, your
> tasks are supposed to suspend sufficiently frequently to leave Linux
> tick normally. This means that if you configured CONFIG_HZ=100, your
> real-time activity (including kernel-space tasks and user-space tasks in
> primary mode) is supposed to take a break at least once every 1/100
> second.
> --
>                                            Gilles Chanteperdrix.  
> "
> Is this the best solution? Since there may be possibility that a 
> real-time thread keep running for severl hundred ms (e.g, processing 
> large block of data).
>  

Keep in mind that real-time using a co-scheduling infrastructure in
Linux is not a matter of pure RTOS design, but rather an issue involving
 a GPOS+RTOS combination. For this reason, you just should not starve
the GPOS from the critical resources it needs, and time ticks are some
of these. If you do so though, what will happen is that the ticks will
be logged at Adeos level in the various pipeline logs, and played when
Linux eventually gets back in control as the active Adeos domain. If the
starvation did not last beyond Linux's robustness to such event, then
the kernel will catch up with the pending ticks and the system will
continue normally. IOW, it's not impossible to do lengthy real-time
processing that might delay/unsynchronize Linux ticks, and this might
work properly if some breathing time is left to the kernel, but you
cannot ask the real-time system to drop the determinism just for the
sake of allowing Linux to tick regularly. It's up to the application
designer to properly evaluate the real-time load which is going to be
bearable by the underlying architecture & kernel combo.

In the particular case of multi-ms processing, I would likely suggest to
move it to a thread running in secondary mode without interrupt
shielding, so that Linux asynchronous activities such as interrupt
handling would still be possible, at the expense of a lesser execution
time predictability of such processing though.

>  
> I tried to run a test on Xenomai-2.1.0 on Blackfin. This test looks like:
>  

<snip>


> +++++++++ end test +++++++++++++++++
> 
>  From the test result, " 23:  BFIN Timer Tick", changed from "33531" to
> "33732", that is "201 * 10 ＝ 2010 ms". And the "up time" is changed
> from "335.41 to 337.42", although in fact that "2 + 6 = 8 sec" has
> passed.
>

You may want to make your measurement thread sleep() for a while before
looking at the final tick counter; this would make sure that the IRQ
threads have been given enough time to catch up and process all pending
ticks they have in their log. Btw, did you activate CONFIG_PREEMPT?

-- 

Philippe.

Re: [Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[Philippe Gerum]

Philippe Gerum wrote:
> Li Yi (Adam) wrote:
> 
>>Hi,
>> 
>>When a real-time thread is running, the Linux kernel will lose Timer 
>>Tick (and lose timer interrupts), is this the expected behavior?
>> 
> 
> 
> No, it should not lose any tick, regardless of the Xenomai port. You are
> likely observing a pecularity of the Blackfin port where Adeos moves
> Linux IRQ handlers (top-halves) in thread context to work-around a high
> scheduling latency source which is Blackfin specific.
> 
> 
>>If yes, in this case,  no interrupts will be passed to Linux domain? 
>>This may cause some services depending on timing in Linux domain do not 
>>work correctly. In another thread I read:
>> 
>>"
>>I do not know if it is what you mean, but when running Xenomai, your
>>tasks are supposed to suspend sufficiently frequently to leave Linux
>>tick normally. This means that if you configured CONFIG_HZ=100, your
>>real-time activity (including kernel-space tasks and user-space tasks in
>>primary mode) is supposed to take a break at least once every 1/100
>>second.
>>--
>>                                           Gilles Chanteperdrix.  
>>"
>>Is this the best solution? Since there may be possibility that a 
>>real-time thread keep running for severl hundred ms (e.g, processing 
>>large block of data).
>> 
> 
> 
> Keep in mind that real-time using a co-scheduling infrastructure in
> Linux is not a matter of pure RTOS design, but rather an issue involving
>  a GPOS+RTOS combination. For this reason, you just should not starve
> the GPOS from the critical resources it needs, and time ticks are some
> of these. If you do so though, what will happen is that the ticks will
> be logged at Adeos level in the various pipeline logs, and played when
> Linux eventually gets back in control as the active Adeos domain. If the
> starvation did not last beyond Linux's robustness to such event, then
> the kernel will catch up with the pending ticks and the system will
> continue normally. IOW, it's not impossible to do lengthy real-time
> processing that might delay/unsynchronize Linux ticks, and this might
> work properly if some breathing time is left to the kernel, but you
> cannot ask the real-time system to drop the determinism just for the
> sake of allowing Linux to tick regularly. It's up to the application
> designer to properly evaluate the real-time load which is going to be
> bearable by the underlying architecture & kernel combo.
> 
> In the particular case of multi-ms processing, I would likely suggest to
> move it to a thread running in secondary mode without interrupt
> shielding, so that Linux asynchronous activities such as interrupt
> handling would still be possible, at the expense of a lesser execution
> time predictability of such processing though.
> 
> 
>> 
>>I tried to run a test on Xenomai-2.1.0 on Blackfin. This test looks like:
>> 
> 
> 
> <snip>
> 
> 
>>+++++++++ end test +++++++++++++++++
>>
>> From the test result, " 23:  BFIN Timer Tick", changed from "33531" to
>>"33732", that is "201 * 10 ＝ 2010 ms". And the "up time" is changed
>>from "335.41 to 337.42", although in fact that "2 + 6 = 8 sec" has
>>passed.
>>
> 
> 
> You may want to make your measurement thread sleep() for a while before
> looking at the final tick counter; this would make sure that the IRQ
> threads have been given enough time to catch up and process all pending
> ticks they have in their log. Btw, did you activate CONFIG_PREEMPT?
> 

Er, sorry. There's no CONFIG_PREEMPT on Blackfin yet.

-- 

Philippe.

Re: [Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[Li Yi]

Thanks for the clarification.
 
I added a sleep(1) before read the "/proc/interrupts", and the timer
interrupts for Linux kernel really got "Replayed". The system time
catches up with the wall clock.

But still some question:

On Thu, 2006-04-20 at 10:19 +0200, Philippe Gerum wrote:

> > 
> > In the particular case of multi-ms processing, I would likely suggest to
> > move it to a thread running in secondary mode without interrupt
> > shielding, so that Linux asynchronous activities such as interrupt
> > handling would still be possible, at the expense of a lesser execution
> > time predictability of such processing though.
> > 
> >

In my test, I am using rt_timer_spin() to simulate the real-time
workload. And "/proc/xenomai/stat" shows "MSW" as "1/1", and in one test
loop, there is no change for "MSW". So I concluded my test case is
running in secondary mode. And my "Interrupt shield support"
configuration is _not_ select. But it looks the Linux kernel does _not_
handle timer interrupt while the real-time task is running. Did I miss
anything? 


> > 
> > You may want to make your measurement thread sleep() for a while before
> > looking at the final tick counter; this would make sure that the IRQ
> > threads have been given enough time to catch up and process all pending
> > ticks they have in their log. Btw, did you activate CONFIG_PREEMPT?
> > 
> 
> Er, sorry. There's no CONFIG_PREEMPT on Blackfin yet.


Thanks,

-Li Yi (Adam)

Re: [Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[Philippe Gerum]

Li Yi wrote:
> Thanks for the clarification.
>  
> I added a sleep(1) before read the "/proc/interrupts", and the timer
> interrupts for Linux kernel really got "Replayed". The system time
> catches up with the wall clock.
> 
> But still some question:
> 
> On Thu, 2006-04-20 at 10:19 +0200, Philippe Gerum wrote:
> 
> 
>>>In the particular case of multi-ms processing, I would likely suggest to
>>>move it to a thread running in secondary mode without interrupt
>>>shielding, so that Linux asynchronous activities such as interrupt
>>>handling would still be possible, at the expense of a lesser execution
>>>time predictability of such processing though.
>>>
>>>
> 
> 
> In my test, I am using rt_timer_spin() to simulate the real-time
> workload. And "/proc/xenomai/stat" shows "MSW" as "1/1", and in one test
> loop, there is no change for "MSW". So I concluded my test case is
> running in secondary mode. And my "Interrupt shield support"
> configuration is _not_ select. But it looks the Linux kernel does _not_
> handle timer interrupt while the real-time task is running. Did I miss
> anything? 
>

rt_timer_spin() is a busy-waiting service, and your measurement task 
never relinquishes the CPU while processing. I would bet that you set up 
a very high (maximum?) SCHED_FIFO priority to the Xenomai task, which 
did not allow the IRQ thread to run.

i.e. On the Blackfin architecture, Adeos scales the IRQ thread priority 
from SCHED_FIFO(50) to SCHED_FIFO(50 + (IVG13-IVG7)), which would have a 
lesser priority than anything above SCHED_FIFO(56).

> 
> 
>>>You may want to make your measurement thread sleep() for a while before
>>>looking at the final tick counter; this would make sure that the IRQ
>>>threads have been given enough time to catch up and process all pending
>>>ticks they have in their log. Btw, did you activate CONFIG_PREEMPT?
>>>
>>
>>Er, sorry. There's no CONFIG_PREEMPT on Blackfin yet.
> 
> 
> 
> Thanks,
> 
> -Li Yi (Adam)
> 


-- 

Philippe.

Re: [Xenomai-help] Will Linux kernel lose timer ticks when RT thread is running

[adam li]

On Thu, 2006-04-20 at 14:46 +0200, Philippe Gerum wrote:
> Li Yi wrote:
> > Thanks for the clarification.
> >  
> > I added a sleep(1) before read the "/proc/interrupts", and the timer
> > interrupts for Linux kernel really got "Replayed". The system time
> > catches up with the wall clock.
> > 
> > But still some question:
> > 
> > On Thu, 2006-04-20 at 10:19 +0200, Philippe Gerum wrote:
> > 
> > 
> >>>In the particular case of multi-ms processing, I would likely suggest to
> >>>move it to a thread running in secondary mode without interrupt
> >>>shielding, so that Linux asynchronous activities such as interrupt
> >>>handling would still be possible, at the expense of a lesser execution
> >>>time predictability of such processing though.
> >>>
> >>>
> > 
> > 
> > In my test, I am using rt_timer_spin() to simulate the real-time
> > workload. And "/proc/xenomai/stat" shows "MSW" as "1/1", and in one test
> > loop, there is no change for "MSW". So I concluded my test case is
> > running in secondary mode. And my "Interrupt shield support"
> > configuration is _not_ select. But it looks the Linux kernel does _not_
> > handle timer interrupt while the real-time task is running. Did I miss
> > anything? 
> >
> 
> rt_timer_spin() is a busy-waiting service, and your measurement task 
> never relinquishes the CPU while processing. I would bet that you set up 
> a very high (maximum?) SCHED_FIFO priority to the Xenomai task, which 
> did not allow the IRQ thread to run.
> 
> i.e. On the Blackfin architecture, Adeos scales the IRQ thread priority 
> from SCHED_FIFO(50) to SCHED_FIFO(50 + (IVG13-IVG7)), which would have a 
> lesser priority than anything above SCHED_FIFO(56).
> 

Yes. I set the the priority to be 99 - the highest. I tried to lower the
priority of the thread to 40, and the Linux kernel handles timer
ticks.  

Thanks for the answer.