[Xenomai-help] no-brainer issue found, but not solved

[Xenomai-help] no-brainer issue found, but not solved

[Steven Seeger]

I have found what seems to cause my problem.

I have two threads, t1 and t2.

t1 has a priority level of 30, and t2 has a priority level of 5. I am using
the native skin, so t1 has the higher priority.

t2 is flashing an LED on my board every 40 ms, and t1 does nothing until I
hit a key on the keyboard. When t1 activates, it does for(i=0; i<100000000;
i++) which locks up my board for about 3 seconds. During this time, I expect
my LED to stop flashing. However, the LED will flash a few times just once,
at a rate of about 1 second. (Coincidentally, 1 second is the timing of
another unrelated thread in the system.)

At the start of my t1 thread, I use syslog (a system call) to log that the
thread is created. This call is made once before the thread goes into its
loop, waiting for a global variable to be set otherwise just calling
rt_task_wait_period(). If I comment out my call to syslog, which again I
state is NOT in the thread loop, this behavior goes away. It seems that once
a syscall is made in a shadow thread, xenomai fails to have it behave in a
true hard realtime fashion. It's as if the thread stays relaxed.

Does this shed any light on the issue I have?

Steven

Re: [Xenomai-help] no-brainer issue found, but not solved

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1545 bytes --]

Steven Seeger wrote:
> I have found what seems to cause my problem.
> 
> I have two threads, t1 and t2.
> 
> t1 has a priority level of 30, and t2 has a priority level of 5. I am using
> the native skin, so t1 has the higher priority.
> 
> t2 is flashing an LED on my board every 40 ms, and t1 does nothing until I
> hit a key on the keyboard. When t1 activates, it does for(i=0; i<100000000;
> i++) which locks up my board for about 3 seconds. During this time, I expect
> my LED to stop flashing. However, the LED will flash a few times just once,
> at a rate of about 1 second. (Coincidentally, 1 second is the timing of
> another unrelated thread in the system.)
> 
> At the start of my t1 thread, I use syslog (a system call) to log that the
> thread is created. This call is made once before the thread goes into its
> loop, waiting for a global variable to be set otherwise just calling
> rt_task_wait_period(). If I comment out my call to syslog, which again I
> state is NOT in the thread loop, this behavior goes away. It seems that once
> a syscall is made in a shadow thread, xenomai fails to have it behave in a
> true hard realtime fashion. It's as if the thread stays relaxed.

It does stay relaxed until the next blocking RT function is invoked.
When unsure if this happens (I am from your description, maybe you
should rather post a code snippet), try rt_task_set_mode(0, T_PRIMARY,
NULL) before entering the loop.

> 
> Does this shed any light on the issue I have?
> 
> Steven
> 
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] no-brainer issue found, but not solved

[Steven Seeger]

Here are the routines in question:

low priority:

for(;;) {
    /* regular rf watchdogs */
    outb(0x45, 0x????);
    outb(0x6a, 0x????);
    outb(0x7e, 0x????);
               
    outb(0, 0x????); //patient watchdog
    outb(0, 0x????); //fault led

    flip_reg_bits(WRFPC, 0x????); //flash yellow led
                
    ????_flip_spi_bits(HV_STATUS_CONTROL, HV_WDOG_STROBE_MONITOR);
                
    rt_task_wait_period();
}

high priority:

    for(;;) {
        if(motor_move) {
            syslog(LOG_INFO, "about to lock up");
            for(int i=0; i<100000000; i++);
            motor_move=0;
        }

        rt_task_wait_period();
    }

All function calls contain only inb or outb, no system calls. If I comment
out the syslog call in the high priority thread, the yellow LED stops
flashing completely. Otherwise, it flashes every second, with the
periodicity of the higher priority thread.

I have another problem, too:

Using the same yellow LED flashing thread, if I start up a LOWER priority
task like this, I can see jitter on my LED strobe line on the scope once a
second. Why would the lower priority thread making a system call affect the
higher priority one?

for(;;) {
    syslog(LOG_INFO, "sleeping 1");
    sleep(1);
}

Steven

Re: [Xenomai-help] no-brainer issue found, but not solved

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2341 bytes --]

Steven Seeger wrote:
> Here are the routines in question:
> 
> low priority:
> 
> for(;;) {
>     /* regular rf watchdogs */
>     outb(0x45, 0x????);
>     outb(0x6a, 0x????);
>     outb(0x7e, 0x????);
>                
>     outb(0, 0x????); //patient watchdog
>     outb(0, 0x????); //fault led
> 
>     flip_reg_bits(WRFPC, 0x????); //flash yellow led
>                 
>     ????_flip_spi_bits(HV_STATUS_CONTROL, HV_WDOG_STROBE_MONITOR);
>                 
>     rt_task_wait_period();
> }
> 
> high priority:
> 
>     for(;;) {
>         if(motor_move) {
>             syslog(LOG_INFO, "about to lock up");
>             for(int i=0; i<100000000; i++);

This loop is always executed in secondary mode. This means it /may/ run
at the original high prio of that thread if syslog() did not trigger a
Linux reschedule, and it will run at prio 0 if syslog did so. This
depends on the arrival of Linux IRQs or the need of syslog to block on
some other Linux process.

Again, if you want RT for sure after a Linux call, use explicit mode
switching.

>             motor_move=0;
>         }
> 
>         rt_task_wait_period();
>     }
> 
> All function calls contain only inb or outb, no system calls. If I comment
> out the syslog call in the high priority thread, the yellow LED stops
> flashing completely. Otherwise, it flashes every second, with the
> periodicity of the higher priority thread.
> 
> I have another problem, too:
> 
> Using the same yellow LED flashing thread, if I start up a LOWER priority
> task like this, I can see jitter on my LED strobe line on the scope once a

Also as rt-thread?

> second. Why would the lower priority thread making a system call affect the
> higher priority one?
> 
> for(;;) {
>     syslog(LOG_INFO, "sleeping 1");
>     sleep(1);
> }

Something seems to be broken, but it is hard to analyse remotely if in
the test setup or the system.

Please replace your LED thread in this scenario with the latency test
tool, running at the same frequency. Collect the worst-case data. If
they show quirks as well, consider applying the tracer patch and catch
the abnormal jitters via "latency -f". You can tune the number if
backtrace_points in /proc/ipipe/tracer until you get a significant time
interval into the trace.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] no-brainer issue found, but not solved

[Jeroen Van den Keybus]

[-- Attachment #1: Type: text/plain, Size: 414 bytes --]

It would be helpful if a complete code could be posted. That means,
including the main() function, in which the task dispatching is done as
well.

Furthermore, be sure to declare counting variables inside waiting loops with
the 'volatile' specifier. The compiler might optimize it away otherwise.

Another, maybe dumb suggestion: how is inb() / outb() actually implemented
on your platform ?


Jeroen.

[-- Attachment #2: Type: text/html, Size: 551 bytes --]

Re: [Xenomai-help] no-brainer issue found, but not solved

[Steven Seeger]

[-- Attachment #1: Type: text/plain, Size: 999 bytes --]

There is a awful amount of code in this project, and I should point out that
this same code did not exhibit these problems under a fusion cvs from
October. I can see about posting the full code, but my supervisors don¹t
want anything having our hardware addresses in it going out.

This is an x86 board, so inb/outb are just instructions.

I know about the volatile thing but that isn¹t my problem. I¹m having
realtime preemption issues.

Steven

On 2/21/06 1:20 PM, "Jeroen Van den Keybus" <jeroen.vandenkeybus@domain.hid>
wrote:

> It would be helpful if a complete code could be posted. That means, including
> the main() function, in which the task dispatching is done as well.
>  
> Furthermore, be sure to declare counting variables inside waiting loops with
> the 'volatile' specifier. The compiler might optimize it away otherwise.
>  
> Another, maybe dumb suggestion: how is inb() / outb() actually implemented on
> your platform ?
>  
>  
> Jeroen.
>  
> 



[-- Attachment #2: Type: text/html, Size: 1513 bytes --]

Re: [Xenomai-help] no-brainer issue found, but not solved

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]

Steven Seeger wrote:
> There is a awful amount of code in this project, and I should point out that
> this same code did not exhibit these problems under a fusion cvs from

You code contains a wrong assumption about the execution context. Older
Xenomai/fusion versions may have concealed this due to different
internal design.

> October. I can see about posting the full code, but my supervisors don¹t
> want anything having our hardware addresses in it going out.
> 
> This is an x86 board, so inb/outb are just instructions.

Then please shrink down your scenario so that

a) no proprietary code or addresses are contained AND
b) only a minimum of surrounding code is used AND
c) the problem (timing issues) still show up.

This first of all means to replace the specific detection of deadline
misses (flickering LEDs) with a generic mechanism - e.g. the latency
tool. Then we can try to reproduce your problem or already find the
issue in the test code. As I pointed out: mixing non-RT with RT code is
feasible but requires careful design.

> 
> I know about the volatile thing but that isn¹t my problem. I¹m having
> realtime preemption issues.
> 
> Steven
> 
> On 2/21/06 1:20 PM, "Jeroen Van den Keybus" <jeroen.vandenkeybus@domain.hid.com>
> wrote:
> 
>> It would be helpful if a complete code could be posted. That means, including
>> the main() function, in which the task dispatching is done as well.
>>  
>> Furthermore, be sure to declare counting variables inside waiting loops with
>> the 'volatile' specifier. The compiler might optimize it away otherwise.
>>  
>> Another, maybe dumb suggestion: how is inb() / outb() actually implemented on
>> your platform ?
>>  
>>  
>> Jeroen.
>>  
>>
> 
> 
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] no-brainer issue found, but not solved

[Philippe Gerum]

Jan Kiszka wrote:
> Steven Seeger wrote:
> 
>>Here are the routines in question:
>>
>>low priority:
>>
>>for(;;) {
>>    /* regular rf watchdogs */
>>    outb(0x45, 0x????);
>>    outb(0x6a, 0x????);
>>    outb(0x7e, 0x????);
>>               
>>    outb(0, 0x????); //patient watchdog
>>    outb(0, 0x????); //fault led
>>
>>    flip_reg_bits(WRFPC, 0x????); //flash yellow led
>>                
>>    ????_flip_spi_bits(HV_STATUS_CONTROL, HV_WDOG_STROBE_MONITOR);
>>                
>>    rt_task_wait_period();
>>}
>>
>>high priority:
>>
>>    for(;;) {
>>        if(motor_move) {
>>            syslog(LOG_INFO, "about to lock up");
>>            for(int i=0; i<100000000; i++);
> 
> 
> This loop is always executed in secondary mode. This means it /may/ run
> at the original high prio of that thread if syslog() did not trigger a
> Linux reschedule, and it will run at prio 0 if syslog did so.

To be precise, the Linux kernel as a whole will inherit the RT priority of the 
syslog caller until the latter suspends Linux-wise, which is quite possible due to 
the way syslog works.

  This
> depends on the arrival of Linux IRQs or the need of syslog to block on
> some other Linux process.
> 
> Again, if you want RT for sure after a Linux call, use explicit mode
> switching.
> 
> 
>>            motor_move=0;
>>        }
>>
>>        rt_task_wait_period();
>>    }
>>
>>All function calls contain only inb or outb, no system calls. If I comment
>>out the syslog call in the high priority thread, the yellow LED stops
>>flashing completely. Otherwise, it flashes every second, with the
>>periodicity of the higher priority thread.
>>
>>I have another problem, too:
>>
>>Using the same yellow LED flashing thread, if I start up a LOWER priority
>>task like this, I can see jitter on my LED strobe line on the scope once a
> 
> 
> Also as rt-thread?
> 
> 
>>second. Why would the lower priority thread making a system call affect the
>>higher priority one?
>>
>>for(;;) {
>>    syslog(LOG_INFO, "sleeping 1");
>>    sleep(1);
>>}
> 
> 
> Something seems to be broken, but it is hard to analyse remotely if in
> the test setup or the system.
> 
> Please replace your LED thread in this scenario with the latency test
> tool, running at the same frequency. Collect the worst-case data. If
> they show quirks as well, consider applying the tracer patch and catch
> the abnormal jitters via "latency -f". You can tune the number if
> backtrace_points in /proc/ipipe/tracer until you get a significant time
> interval into the trace.
> 
> Jan
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help


-- 

Philippe.

Re: [Xenomai-help] no-brainer issue found, but not solved

[Steven Seeger]

I didn't get Jan's original response to my messages. I haven't seen what he
wrote until Philippe's response.

> 
> To be precise, the Linux kernel as a whole will inherit the RT priority of the
> syslog caller until the latter suspends Linux-wise, which is quite possible
> due to 
> the way syslog works.

Hmm, I thought threads in the secondary domain ran at the same priority as
Linux but I didn't know that Linux was elevated also. This is good to know,
thanks.

Since my program is working correctly on the new board, my boss doesn't want
to bother with the geode for now. Again, something is amiss if this program
that uses 100% of the cpu on the geode uses hardly any on another board.

Steven