* [Xenomai] Context switching kernel tracing
@ 2013-05-30 7:03 Stéphane ANCELOT
2013-05-30 11:38 ` Gilles Chanteperdrix
0 siblings, 1 reply; 6+ messages in thread
From: Stéphane ANCELOT @ 2013-05-30 7:03 UTC (permalink / raw)
To: xenomai
Hi,
I have got some problems with an architecture, and 2 realtime tasks.
The realtime is not always respected.
This is a very strange problem in this architecture, since it happens
statically almost every three reboots...
It looks like there is something in the kernel / or setted up by bios
that is happening and locks the task switching context.
I implemented LTTNG 2.0 tracing in my target, I would need to know which
xenomai functions/events are interesting to watch in my kernel traces ?
Regards,
S.Ancelot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing
2013-05-30 7:03 [Xenomai] Context switching kernel tracing Stéphane ANCELOT
@ 2013-05-30 11:38 ` Gilles Chanteperdrix
2013-05-30 13:12 ` Stéphane ANCELOT
0 siblings, 1 reply; 6+ messages in thread
From: Gilles Chanteperdrix @ 2013-05-30 11:38 UTC (permalink / raw)
To: Stéphane ANCELOT; +Cc: xenomai
On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote:
> Hi,
>
> I have got some problems with an architecture, and 2 realtime tasks.
>
> The realtime is not always respected.
Hi,
a few things to check:
Would there be any involuntary mode switches? See rt_task_set_mode or
pthread_set_mode_np documentation to enable debug.
Are you able to reproduce the problem with the latency test? You can
launch several instances in parallel, if you absolutely need several
task. If yes, then probably the easiest solution is to enable the I-pipe
tracer, then run the latency test with the -f argument.
>
> This is a very strange problem in this architecture, since it happens
> statically almost every three reboots...
>
> It looks like there is something in the kernel / or setted up by bios
> that is happening and locks the task switching context.
Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch
and Xenomai are you using? Can not it be an SMI issue, have you tried
the SMI workaround?
http://www.xenomai.org/documentation/xenomai-2.6/html/TROUBLESHOOTING/#SMI
Regards.
--
Gilles.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing
2013-05-30 11:38 ` Gilles Chanteperdrix
@ 2013-05-30 13:12 ` Stéphane ANCELOT
2013-05-30 18:19 ` Gilles Chanteperdrix
0 siblings, 1 reply; 6+ messages in thread
From: Stéphane ANCELOT @ 2013-05-30 13:12 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: xenomai
On 30/05/2013 13:38, Gilles Chanteperdrix wrote:
> On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote:
>
>> Hi,
>>
>> I have got some problems with an architecture, and 2 realtime tasks.
>>
>> The realtime is not always respected.
>
> Hi,
>
> a few things to check:
>
> Would there be any involuntary mode switches? See rt_task_set_mode or
> pthread_set_mode_np documentation to enable debug.
There is not any involuntary mode switches in the xenomai application.
The buggy architecture is celeron M:
Intel® 910GMLE / ICH6M chipsets
the same disk application, if setted up in the following architecture,
has no problem:
Intel® 852GM and ICH4 chipset
Are you able to reproduce the problem with the latency test? You can
launch several instances in parallel, if you absolutely need several
task. If yes, then probably the easiest solution is to enable the I-pipe
tracer, then run the latency test with the -f argument.
I have read more documentation in kernel tracing, and it seems I won't
need lttng, but it looks like only kernel tracing would be enough ?
I have not managed to reproduce it with a single instance of latency
test. (and I have no problem using a single rt task...) . I will try 2
instances.
One more thing that is surprising : I monitored the tasks delay using
clock_gettime() , the software does not watch any big lag !!! ...but it
is visible in the scopemeter using com port signals (up to 300us
possible !!!!) ....
I also was thinking about a problem with the high res. timers, I tried
the HPET timers, but this has not helped.
>> This is a very strange problem in this architecture, since it happens
>> statically almost every three reboots...
>>
>> It looks like there is something in the kernel / or setted up by bios
>> that is happening and locks the task switching context.
>
> Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch
> and Xenomai are you using? Can not it be an SMI issue, have you tried
> the SMI workaround?
yes, I have tried disabling SMI, but in anyway it fails as follow:
Xenomai: SMI-enabled chipset found
Xenomai: SMI workaround failed!
I have the same problem using either
kernel 2.6.34 and xenomai 2.5.6
or
kernel 3.5.7 and xenomai 2.6.2.1
Further informations on Monday.
Regards,
Steph
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing
2013-05-30 13:12 ` Stéphane ANCELOT
@ 2013-05-30 18:19 ` Gilles Chanteperdrix
2013-06-03 13:33 ` Stéphane ANCELOT
0 siblings, 1 reply; 6+ messages in thread
From: Gilles Chanteperdrix @ 2013-05-30 18:19 UTC (permalink / raw)
To: Stéphane ANCELOT; +Cc: xenomai
On 05/30/2013 03:12 PM, Stéphane ANCELOT wrote:
> On 30/05/2013 13:38, Gilles Chanteperdrix wrote:
>> On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote:
>>
>>> Hi,
>>>
>>> I have got some problems with an architecture, and 2 realtime tasks.
>>>
>>> The realtime is not always respected.
>>
>> Hi,
>>
>> a few things to check:
>>
>> Would there be any involuntary mode switches? See rt_task_set_mode or
>> pthread_set_mode_np documentation to enable debug.
> There is not any involuntary mode switches in the xenomai application.
>
> The buggy architecture is celeron M:
>
> Intel® 910GMLE / ICH6M chipsets
>
> the same disk application, if setted up in the following architecture,
> has no problem:
> Intel® 852GM and ICH4 chipset
>
> Are you able to reproduce the problem with the latency test? You can
> launch several instances in parallel, if you absolutely need several
> task. If yes, then probably the easiest solution is to enable the I-pipe
> tracer, then run the latency test with the -f argument.
>
> I have read more documentation in kernel tracing, and it seems I won't
> need lttng, but it looks like only kernel tracing would be enough ?
>
> I have not managed to reproduce it with a single instance of latency
> test. (and I have no problem using a single rt task...) . I will try 2
> instances.
>
> One more thing that is surprising : I monitored the tasks delay using
> clock_gettime() , the software does not watch any big lag !!! ...but it
> is visible in the scopemeter using com port signals (up to 300us
> possible !!!!) ....
I would say it means that the tsc (and APIC) stop during some idle
states of the system. I suppose you have CONFIG_CPU_IDLE turned off?
Could you try booting with nohlt or idle=poll ?
>
> I also was thinking about a problem with the high res. timers, I tried
> the HPET timers, but this has not helped.
I do not believe Xenomai is able to use the HPET timers with Linux 2.6.34.
>
>>> This is a very strange problem in this architecture, since it happens
>>> statically almost every three reboots...
>>>
>>> It looks like there is something in the kernel / or setted up by bios
>>> that is happening and locks the task switching context.
>>
>> Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch
>> and Xenomai are you using? Can not it be an SMI issue, have you tried
>> the SMI workaround?
>
>
> yes, I have tried disabling SMI, but in anyway it fails as follow:
>
> Xenomai: SMI-enabled chipset found
> Xenomai: SMI workaround failed!
Ok then, your BIOS vendor does not want you to disable the SMI global
bit, have you tried other bits?
--
Gilles.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing
2013-05-30 18:19 ` Gilles Chanteperdrix
@ 2013-06-03 13:33 ` Stéphane ANCELOT
2013-06-03 18:07 ` Gilles Chanteperdrix
0 siblings, 1 reply; 6+ messages in thread
From: Stéphane ANCELOT @ 2013-06-03 13:33 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: xenomai
Hi,
The SMI is the source of my problems. the global SMI bit in the ICH6 can
not be disabled .
However, I used a 32 bits mask trying to disable any bits and managed to
reduce the latency (worst : 15us )
refer to paragraph 10.8.3.12 for ICH6 chipset :
http://www.intel.com/content/www/us/en/io/io-controller-hub-6-datasheet.html
Regards,
Steph
On 30/05/2013 20:19, Gilles Chanteperdrix wrote:
> On 05/30/2013 03:12 PM, Stéphane ANCELOT wrote:
>
>> On 30/05/2013 13:38, Gilles Chanteperdrix wrote:
>>> On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote:
>>>
>>>> Hi,
>>>>
>>>> I have got some problems with an architecture, and 2 realtime tasks.
>>>>
>>>> The realtime is not always respected.
>>> Hi,
>>>
>>> a few things to check:
>>>
>>> Would there be any involuntary mode switches? See rt_task_set_mode or
>>> pthread_set_mode_np documentation to enable debug.
>> There is not any involuntary mode switches in the xenomai application.
>>
>> The buggy architecture is celeron M:
>>
>> Intel® 910GMLE / ICH6M chipsets
>>
>> the same disk application, if setted up in the following architecture,
>> has no problem:
>> Intel® 852GM and ICH4 chipset
>>
>> Are you able to reproduce the problem with the latency test? You can
>> launch several instances in parallel, if you absolutely need several
>> task. If yes, then probably the easiest solution is to enable the I-pipe
>> tracer, then run the latency test with the -f argument.
>>
>> I have read more documentation in kernel tracing, and it seems I won't
>> need lttng, but it looks like only kernel tracing would be enough ?
>>
>> I have not managed to reproduce it with a single instance of latency
>> test. (and I have no problem using a single rt task...) . I will try 2
>> instances.
>>
>> One more thing that is surprising : I monitored the tasks delay using
>> clock_gettime() , the software does not watch any big lag !!! ...but it
>> is visible in the scopemeter using com port signals (up to 300us
>> possible !!!!) ....
>
> I would say it means that the tsc (and APIC) stop during some idle
> states of the system. I suppose you have CONFIG_CPU_IDLE turned off?
>
> Could you try booting with nohlt or idle=poll ?
>
>> I also was thinking about a problem with the high res. timers, I tried
>> the HPET timers, but this has not helped.
>
> I do not believe Xenomai is able to use the HPET timers with Linux 2.6.34.
>
>>>> This is a very strange problem in this architecture, since it happens
>>>> statically almost every three reboots...
>>>>
>>>> It looks like there is something in the kernel / or setted up by bios
>>>> that is happening and locks the task switching context.
>>> Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch
>>> and Xenomai are you using? Can not it be an SMI issue, have you tried
>>> the SMI workaround?
>>
>> yes, I have tried disabling SMI, but in anyway it fails as follow:
>>
>> Xenomai: SMI-enabled chipset found
>> Xenomai: SMI workaround failed!
>
> Ok then, your BIOS vendor does not want you to disable the SMI global
> bit, have you tried other bits?
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing
2013-06-03 13:33 ` Stéphane ANCELOT
@ 2013-06-03 18:07 ` Gilles Chanteperdrix
0 siblings, 0 replies; 6+ messages in thread
From: Gilles Chanteperdrix @ 2013-06-03 18:07 UTC (permalink / raw)
To: Stéphane ANCELOT; +Cc: xenomai
On 06/03/2013 03:33 PM, Stéphane ANCELOT wrote:
> Hi,
>
>
> The SMI is the source of my problems. the global SMI bit in the ICH6 can
> not be disabled .
I read the URL you posted (which I had already read when writing the smi
module when Xenomai was still part of the RTAI project), and it does not
say that. It says that the global bit can only be used if the SMI_LOCK
bit is not set. So there was a time when the smi module checked the
SMI_LOCK bit, but after some time, we found that on some machines, even
when the SMI_LOCK bit was not armed, it was not possible to mask the
SMI_EN bit, so we replaced the check with the current check (re-reading
the bits, to check if the masked bit are actually masked).
Anyway, the only reason why the SMI_LOCK bit is armed, is if some code,
so, presumably the BIOS code, has armed it. Hence my original answer to
your query:
>> Ok then, your BIOS vendor does not want you to disable the SMI global
>> bit, have you tried other bits?
Got it?
--
Gilles.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-06-03 18:07 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-05-30 7:03 [Xenomai] Context switching kernel tracing Stéphane ANCELOT
2013-05-30 11:38 ` Gilles Chanteperdrix
2013-05-30 13:12 ` Stéphane ANCELOT
2013-05-30 18:19 ` Gilles Chanteperdrix
2013-06-03 13:33 ` Stéphane ANCELOT
2013-06-03 18:07 ` Gilles Chanteperdrix
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.