[Xenomai-help] high latency every 8 seconds

[Xenomai-help] high latency every 8 seconds

[Theo Veenker]

Hi all,

I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
other/older systems, but on the new systems there is something going
on every 8 seconds. See output of latency test below. What could
this be? Is this an SMI effect?

== Sampling period: 100 us
== Test mode: periodic user-mode task
== All results in microseconds
warming up...
RTT|  00:00:01  (periodic user-mode task, 100 us period, priority 99)
RTH|-----lat min|-----lat avg|-----lat max|-overrun|----lat best|---lat worst
RTD|      -0.234|      -0.063|       7.845|       0|      -0.234|       7.845
RTD|      -0.308|      -0.015|       8.496|       0|      -0.308|       8.496
RTD|      -0.313|      -0.014|       7.847|       0|      -0.313|       8.496
RTD|      -0.308|      -0.017|       8.502|       0|      -0.313|       8.502
RTD|      -0.424|      -0.012|       8.431|       0|      -0.424|       8.502
RTD|      -0.308|      -0.030|       8.329|       0|      -0.424|       8.502
RTD|      -0.311|      -0.012|       8.459|       0|      -0.424|       8.502
RTD|      -0.308|       0.696|    7169.625|      71|      -0.424|    7169.625
RTD|      -0.308|      -0.016|       8.351|      71|      -0.424|    7169.625
RTD|      -0.309|      -0.017|       8.421|      71|      -0.424|    7169.625
RTD|      -0.313|      -0.016|       8.340|      71|      -0.424|    7169.625
RTD|      -0.308|      -0.015|       8.395|      71|      -0.424|    7169.625
RTD|      -0.311|      -0.010|       8.607|      71|      -0.424|    7169.625
RTD|      -0.308|      -0.021|       8.450|      71|      -0.424|    7169.625
RTD|      -0.308|      -0.017|       8.401|      71|      -0.424|    7169.625
RTD|      -0.308|       0.704|    7161.139|     142|      -0.424|    7169.625
RTD|      -0.311|      -0.017|       8.349|     142|      -0.424|    7169.625
RTD|      -0.312|       0.042|       8.435|     142|      -0.424|    7169.625
RTD|      -0.318|       0.020|       8.467|     142|      -0.424|    7169.625
RTD|      -0.394|       0.335|       8.820|     142|      -0.424|    7169.625
RTD|      -0.363|       1.034|       5.484|     142|      -0.424|    7169.625
RTD|      -0.410|       0.185|       8.428|     142|      -0.424|    7169.625
RTD|      -0.312|       0.024|       8.478|     142|      -0.424|    7169.625
RTD|      -0.308|       0.728|    7060.879|     212|      -0.424|    7169.625
RTD|      -0.282|       0.016|       8.424|     212|      -0.424|    7169.625
RTD|      -0.499|       0.011|       8.490|     212|      -0.499|    7169.625

and so on...

Regards,
Theo

[Xenomai-help] Re: high latency every 8 seconds

[Jan Kiszka]

Theo Veenker wrote:
> Hi all,
> 
> I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of

When starting new, pick something new, i.e. 2.3 right now. Only if you
have good reasons, staying at older versions is recommended.

> systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
> other/older systems, but on the new systems there is something going
> on every 8 seconds. See output of latency test below. What could
> this be? Is this an SMI effect?

Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel output
tell you about if it's working? Do you have an ICH7 chipset on your
board? Would be another reason to update Xenomai (2.3 takes care of it,
2.2.x doesn't)... :)

If this all doesn't help, have a look at this tool:

http://www.xenomai.org/index.php/I-pipe:Tracer

In any case, please give a feedback on the outcome.

Jan

Re: [Xenomai-help] high latency every 8 seconds

[Gilles Chanteperdrix]

Theo Veenker wrote:
> Hi all,
> 
> I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
> systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
> other/older systems, but on the new systems there is something going
> on every 8 seconds. See output of latency test below. What could
> this be? Is this an SMI effect?

Yes, typically, enable the SMI workaround and you will see.


-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: high latency every 8 seconds

[Theo Veenker]

Jan Kiszka wrote:
> Theo Veenker wrote:
>> Hi all,
>>
>> I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
> 
> When starting new, pick something new, i.e. 2.3 right now. Only if you
> have good reasons, staying at older versions is recommended.

I would love to, but I couldn't get my software running with 2.3.
I had to convert (a month ago) my software from RTLinux to Xenomai
having only a few days for it. I have a kernel module and an app
that talks to it. I use the posix skin. But with 2.3 apparently
I had to load posixio and something (forgot) manually, but I just
couldn't get these loaded. Since time was running out I decided
to stay with 2.2.5 for the time being (unfortunately).

> 
>> systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
>> other/older systems, but on the new systems there is something going
>> on every 8 seconds. See output of latency test below. What could
>> this be? Is this an SMI effect?
> 
> Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel output
> tell you about if it's working? Do you have an ICH7 chipset on your
> board? Would be another reason to update Xenomai (2.3 takes care of it,
> 2.2.x doesn't)... :)
> 
> If this all doesn't help, have a look at this tool:
> 
> http://www.xenomai.org/index.php/I-pipe:Tracer
> 
> In any case, please give a feedback on the outcome.

Thanks. I will.

Theo

[Xenomai-help] Re: high latency every 8 seconds

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1656 bytes --]

Theo Veenker wrote:
> Jan Kiszka wrote:
>> Theo Veenker wrote:
>>> Hi all,
>>>
>>> I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
>>
>> When starting new, pick something new, i.e. 2.3 right now. Only if you
>> have good reasons, staying at older versions is recommended.
> 
> I would love to, but I couldn't get my software running with 2.3.
> I had to convert (a month ago) my software from RTLinux to Xenomai
> having only a few days for it. I have a kernel module and an app
> that talks to it. I use the posix skin. But with 2.3 apparently
> I had to load posixio and something (forgot) manually, but I just
> couldn't get these loaded. Since time was running out I decided
> to stay with 2.2.5 for the time being (unfortunately).

I do understand. But once your time constraints relaxed a bit, please
don't forget to post remaining issues here. Maybe they reveal some real bug.

> 
>>
>>> systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
>>> other/older systems, but on the new systems there is something going
>>> on every 8 seconds. See output of latency test below. What could
>>> this be? Is this an SMI effect?
>>
>> Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel output
>> tell you about if it's working? Do you have an ICH7 chipset on your
>> board? Would be another reason to update Xenomai (2.3 takes care of it,
>> 2.2.x doesn't)... :)
>>
>> If this all doesn't help, have a look at this tool:
>>
>> http://www.xenomai.org/index.php/I-pipe:Tracer
>>
>> In any case, please give a feedback on the outcome.
> 
> Thanks. I will.
> 
> Theo
> 

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Re: high latency every 8 seconds

[Gilles Chanteperdrix]

Theo Veenker wrote:
 > Jan Kiszka wrote:
 > > Theo Veenker wrote:
 > >> Hi all,
 > >>
 > >> I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
 > > 
 > > When starting new, pick something new, i.e. 2.3 right now. Only if you
 > > have good reasons, staying at older versions is recommended.
 > 
 > I would love to, but I couldn't get my software running with 2.3.
 > I had to convert (a month ago) my software from RTLinux to Xenomai
 > having only a few days for it. I have a kernel module and an app
 > that talks to it. I use the posix skin. But with 2.3 apparently
 > I had to load posixio and something (forgot) manually, but I just
 > couldn't get these loaded. Since time was running out I decided
 > to stay with 2.2.5 for the time being (unfortunately).

The only thing I see in 2.3 that could be a problem is the fact that
user space mapping of posix shared memory is no longer enabled by
default, was it your problem, are you using posix shared memory for
communication between kernel-space and user-space ?

-- 


					    Gilles Chanteperdrix.

Re: [Xenomai-help] Re: high latency every 8 seconds

[Theo Veenker]

Gilles Chanteperdrix wrote:
> Theo Veenker wrote:
>  > Jan Kiszka wrote:
>  > > Theo Veenker wrote:
>  > >> Hi all,
>  > >>
>  > >> I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
>  > > 
>  > > When starting new, pick something new, i.e. 2.3 right now. Only if you
>  > > have good reasons, staying at older versions is recommended.
>  > 
>  > I would love to, but I couldn't get my software running with 2.3.
>  > I had to convert (a month ago) my software from RTLinux to Xenomai
>  > having only a few days for it. I have a kernel module and an app
>  > that talks to it. I use the posix skin. But with 2.3 apparently
>  > I had to load posixio and something (forgot) manually, but I just
>  > couldn't get these loaded. Since time was running out I decided
>  > to stay with 2.2.5 for the time being (unfortunately).
> 
> The only thing I see in 2.3 that could be a problem is the fact that
> user space mapping of posix shared memory is no longer enabled by
> default, was it your problem, are you using posix shared memory for
> communication between kernel-space and user-space ?
> 

That could have been the problem. I'm using shared memory. As time
permits I will try again and report what happens.

Theo

Re: [Xenomai-help] Re: high latency every 8 seconds

[Gilles Chanteperdrix]

Theo Veenker wrote:
> Gilles Chanteperdrix wrote:
>>The only thing I see in 2.3 that could be a problem is the fact that
>>user space mapping of posix shared memory is no longer enabled by
>>default, was it your problem, are you using posix shared memory for
>>communication between kernel-space and user-space ?
>>
> 
> 
> That could have been the problem. I'm using shared memory. As time
> permits I will try again and report what happens.

Ok, if this is the issue, simply enable "Shared memory" under the menu
Real-time sub-system/Interfaces/POSIX API of the kernel configuration menu.

-- 
                                                 Gilles Chanteperdrix

[Xenomai-help] Re: high latency every 8 seconds

[Theo Veenker]

Jan Kiszka wrote:
> Theo Veenker wrote:
> 
>>Jan Kiszka wrote:
>>
>>>Theo Veenker wrote:
>>>
>>>>Hi all,
>>>>
>>>>I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
>>>
>>>When starting new, pick something new, i.e. 2.3 right now. Only if
> 
> you
> 
>>>have good reasons, staying at older versions is recommended.
>>
>>I would love to, but I couldn't get my software running with 2.3.
>>I had to convert (a month ago) my software from RTLinux to Xenomai
>>having only a few days for it. I have a kernel module and an app
>>that talks to it. I use the posix skin. But with 2.3 apparently
>>I had to load posixio and something (forgot) manually, but I just
>>couldn't get these loaded. Since time was running out I decided
>>to stay with 2.2.5 for the time being (unfortunately).
> 
> 
> I do understand. But once your time constraints relaxed a bit, please
> don't forget to post remaining issues here. Maybe they reveal some real
> bug.
> 
> 
>>>>systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
>>>>other/older systems, but on the new systems there is something going
>>>>on every 8 seconds. See output of latency test below. What could
>>>>this be? Is this an SMI effect?
>>>
>>>Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel output
>>>tell you about if it's working? Do you have an ICH7 chipset on your
>>>board? Would be another reason to update Xenomai (2.3 takes care of
>>>it, 2.2.x doesn't)... :)

OK. I finally installed my clients. It took a while because although all
systems are identical they didn't all behave the same. Apparently some
MSI 945P boards have problems when installing an ubuntu 686 kernel image.
Taking a 386 image as a starting point worked.

Recompiled my kernel 2.6.17.14 with xenomai 2.2.5 and SMI WORKAROUND
selected, but it didn't have any effect. I don't see any xenomai
output in the kernel logs. Same thing without the workaround. So it
doesn't even detect SMI. The mobo has a ICH7 chipset. I understand
from your message I would have to upgrade to xenomai 2.3 to have
support for this. Does this mean that with 2.3 my SMI problems will
magically go away, or does it mean I can use the SMI workaround then?

BTW, can turning off SMI (completely or partly) cause any damage to
the system? I mean can it effect cpu cooling or voltages?

>>>
>>>If this all doesn't help, have a look at this tool:
>>>
>>>http://www.xenomai.org/index.php/I-pipe:Tracer
>>>
>>>In any case, please give a feedback on the outcome.
>>
>>Thanks. I will.
>>
>>Theo
>>
> 
> 
> Jan
> 

I will now try with 2.3 again to find out what is was that troubled me
a while ago.

Theo

Re: [Xenomai-help] Re: high latency every 8 seconds

[Ulrich Schwab]

On Tuesday 06 March 2007 11:32, Theo Veenker wrote:
> Jan Kiszka wrote:
> > Theo Veenker wrote:
> >>Jan Kiszka wrote:
> >>>Theo Veenker wrote:
> >>>>Hi all,
> >>>>
> >>>>I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
> >>>
> >>>When starting new, pick something new, i.e. 2.3 right now. Only if
> >
> > you
> >
> >>>have good reasons, staying at older versions is recommended.
> >>
> >>I would love to, but I couldn't get my software running with 2.3.
> >>I had to convert (a month ago) my software from RTLinux to Xenomai
> >>having only a few days for it. I have a kernel module and an app
> >>that talks to it. I use the posix skin. But with 2.3 apparently
> >>I had to load posixio and something (forgot) manually, but I just
> >>couldn't get these loaded. Since time was running out I decided
> >>to stay with 2.2.5 for the time being (unfortunately).
> >
> > I do understand. But once your time constraints relaxed a bit, please
> > don't forget to post remaining issues here. Maybe they reveal some real
> > bug.
> >
> >>>>systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
> >>>>other/older systems, but on the new systems there is something going
> >>>>on every 8 seconds. See output of latency test below. What could
> >>>>this be? Is this an SMI effect?
> >>>
> >>>Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel output
> >>>tell you about if it's working? Do you have an ICH7 chipset on your
> >>>board? Would be another reason to update Xenomai (2.3 takes care of
> >>>it, 2.2.x doesn't)... :)
>
> OK. I finally installed my clients. It took a while because although all
> systems are identical they didn't all behave the same. Apparently some
> MSI 945P boards have problems when installing an ubuntu 686 kernel image.
> Taking a 386 image as a starting point worked.
>
> Recompiled my kernel 2.6.17.14 with xenomai 2.2.5 and SMI WORKAROUND
> selected, but it didn't have any effect. I don't see any xenomai
> output in the kernel logs. Same thing without the workaround. So it
> doesn't even detect SMI. The mobo has a ICH7 chipset.
ICH7 is not in the list of SMI chipsets  in xenomai 2.2.5
compare the file  ksrc/arch/i386/smi.c between versions 2.3 and 2.2.5
This is the only change needed for the SMI detection and workaround
to work for this chipset in 2.2.5, if You decide NOT to switch to 2.3
You will need to enanble the SMI workaround  in the configuration, of course.

Ulrich

Re: [Xenomai-help] Re: high latency every 8 seconds

[Theo Veenker]

Ulrich Schwab wrote:
> On Tuesday 06 March 2007 11:32, Theo Veenker wrote:
> 
>>Jan Kiszka wrote:
>>
>>>Theo Veenker wrote:
>>>
>>>>Jan Kiszka wrote:
>>>>
>>>>>Theo Veenker wrote:
>>>>>
>>>>>>Hi all,
>>>>>>
>>>>>>I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
>>>>>
>>>>>When starting new, pick something new, i.e. 2.3 right now. Only if
>>>
>>>you
>>>
>>>
>>>>>have good reasons, staying at older versions is recommended.
>>>>
>>>>I would love to, but I couldn't get my software running with 2.3.
>>>>I had to convert (a month ago) my software from RTLinux to Xenomai
>>>>having only a few days for it. I have a kernel module and an app
>>>>that talks to it. I use the posix skin. But with 2.3 apparently
>>>>I had to load posixio and something (forgot) manually, but I just
>>>>couldn't get these loaded. Since time was running out I decided
>>>>to stay with 2.2.5 for the time being (unfortunately).
>>>
>>>I do understand. But once your time constraints relaxed a bit, please
>>>don't forget to post remaining issues here. Maybe they reveal some real
>>>bug.
>>>
>>>
>>>>>>systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
>>>>>>other/older systems, but on the new systems there is something going
>>>>>>on every 8 seconds. See output of latency test below. What could
>>>>>>this be? Is this an SMI effect?
>>>>>
>>>>>Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel output
>>>>>tell you about if it's working? Do you have an ICH7 chipset on your
>>>>>board? Would be another reason to update Xenomai (2.3 takes care of
>>>>>it, 2.2.x doesn't)... :)
>>
>>OK. I finally installed my clients. It took a while because although all
>>systems are identical they didn't all behave the same. Apparently some
>>MSI 945P boards have problems when installing an ubuntu 686 kernel image.
>>Taking a 386 image as a starting point worked.
>>
>>Recompiled my kernel 2.6.17.14 with xenomai 2.2.5 and SMI WORKAROUND
>>selected, but it didn't have any effect. I don't see any xenomai
>>output in the kernel logs. Same thing without the workaround. So it
>>doesn't even detect SMI. The mobo has a ICH7 chipset.
> 
> ICH7 is not in the list of SMI chipsets  in xenomai 2.2.5
> compare the file  ksrc/arch/i386/smi.c between versions 2.3 and 2.2.5
> This is the only change needed for the SMI detection and workaround
> to work for this chipset in 2.2.5, if You decide NOT to switch to 2.3
> You will need to enanble the SMI workaround  in the configuration, of course.

Thanks! I'll use that until I manage to use 2.3 for my app.

Theo

[Xenomai-help] Re: high latency every 8 seconds

[Theo Veenker]

Jan Kiszka wrote:
> Theo Veenker wrote:
> 
>>Jan Kiszka wrote:
>>
>>>Theo Veenker wrote:
>>>
>>>>Hi all,
>>>>
>>>>I'm installing xenomai 2.2.5 (kernel 2.6.17.14) on a number of
>>>
>>>When starting new, pick something new, i.e. 2.3 right now. Only if
> 
> you
> 
>>>have good reasons, staying at older versions is recommended.
>>
>>I would love to, but I couldn't get my software running with 2.3.
>>I had to convert (a month ago) my software from RTLinux to Xenomai
>>having only a few days for it. I have a kernel module and an app
>>that talks to it. I use the posix skin. But with 2.3 apparently
>>I had to load posixio and something (forgot) manually, but I just
>>couldn't get these loaded. Since time was running out I decided
>>to stay with 2.2.5 for the time being (unfortunately).
> 
> 
> I do understand. But once your time constraints relaxed a bit, please
> don't forget to post remaining issues here. Maybe they reveal some real
> bug.
> 
> 
>>>>systems (MSI 945P, Intel P4 3Ghz). My kernel runs fine on several
>>>>other/older systems, but on the new systems there is something going
>>>>on every 8 seconds. See output of latency test below. What could
>>>>this be? Is this an SMI effect?
>>>
>>>Is CONFIG_XENO_HW_SMI_WORKAROUND selected? What does the kernel
> 
> output
> 
>>>tell you about if it's working? Do you have an ICH7 chipset on your
>>>board? Would be another reason to update Xenomai (2.3 takes care of
> 
> it,
> 
>>>2.2.x doesn't)... :)
>>>
>>>If this all doesn't help, have a look at this tool:
>>>
>>>http://www.xenomai.org/index.php/I-pipe:Tracer
>>>
>>>In any case, please give a feedback on the outcome.
>>
>>Thanks. I will.


My problems have been solved. First I did Ulich's fix to enable SMI
detect/workaround for ICH7 on 2.2.5. That worked fine. All devices
continue to work and no more SMI induced latencies.

Then I tried Xenomai 2.3.0 again to find out why I couldn't get that
going some time ago. Gilles reported that if I user shared memory then
I should enable shared memory for the posix api. My first thought was,
"duh of course I have enabled that, I'm not stupid". Guess what? I AM
stupid! Now everything works on all systems.

Thanks everybody.

Theo