* [Xenomai] Context switching kernel tracing @ 2013-05-30 7:03 Stéphane ANCELOT 2013-05-30 11:38 ` Gilles Chanteperdrix 0 siblings, 1 reply; 6+ messages in thread From: Stéphane ANCELOT @ 2013-05-30 7:03 UTC (permalink / raw) To: xenomai Hi, I have got some problems with an architecture, and 2 realtime tasks. The realtime is not always respected. This is a very strange problem in this architecture, since it happens statically almost every three reboots... It looks like there is something in the kernel / or setted up by bios that is happening and locks the task switching context. I implemented LTTNG 2.0 tracing in my target, I would need to know which xenomai functions/events are interesting to watch in my kernel traces ? Regards, S.Ancelot ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing 2013-05-30 7:03 [Xenomai] Context switching kernel tracing Stéphane ANCELOT @ 2013-05-30 11:38 ` Gilles Chanteperdrix 2013-05-30 13:12 ` Stéphane ANCELOT 0 siblings, 1 reply; 6+ messages in thread From: Gilles Chanteperdrix @ 2013-05-30 11:38 UTC (permalink / raw) To: Stéphane ANCELOT; +Cc: xenomai On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote: > Hi, > > I have got some problems with an architecture, and 2 realtime tasks. > > The realtime is not always respected. Hi, a few things to check: Would there be any involuntary mode switches? See rt_task_set_mode or pthread_set_mode_np documentation to enable debug. Are you able to reproduce the problem with the latency test? You can launch several instances in parallel, if you absolutely need several task. If yes, then probably the easiest solution is to enable the I-pipe tracer, then run the latency test with the -f argument. > > This is a very strange problem in this architecture, since it happens > statically almost every three reboots... > > It looks like there is something in the kernel / or setted up by bios > that is happening and locks the task switching context. Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch and Xenomai are you using? Can not it be an SMI issue, have you tried the SMI workaround? http://www.xenomai.org/documentation/xenomai-2.6/html/TROUBLESHOOTING/#SMI Regards. -- Gilles. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing 2013-05-30 11:38 ` Gilles Chanteperdrix @ 2013-05-30 13:12 ` Stéphane ANCELOT 2013-05-30 18:19 ` Gilles Chanteperdrix 0 siblings, 1 reply; 6+ messages in thread From: Stéphane ANCELOT @ 2013-05-30 13:12 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai On 30/05/2013 13:38, Gilles Chanteperdrix wrote: > On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote: > >> Hi, >> >> I have got some problems with an architecture, and 2 realtime tasks. >> >> The realtime is not always respected. > > Hi, > > a few things to check: > > Would there be any involuntary mode switches? See rt_task_set_mode or > pthread_set_mode_np documentation to enable debug. There is not any involuntary mode switches in the xenomai application. The buggy architecture is celeron M: Intel® 910GMLE / ICH6M chipsets the same disk application, if setted up in the following architecture, has no problem: Intel® 852GM and ICH4 chipset Are you able to reproduce the problem with the latency test? You can launch several instances in parallel, if you absolutely need several task. If yes, then probably the easiest solution is to enable the I-pipe tracer, then run the latency test with the -f argument. I have read more documentation in kernel tracing, and it seems I won't need lttng, but it looks like only kernel tracing would be enough ? I have not managed to reproduce it with a single instance of latency test. (and I have no problem using a single rt task...) . I will try 2 instances. One more thing that is surprising : I monitored the tasks delay using clock_gettime() , the software does not watch any big lag !!! ...but it is visible in the scopemeter using com port signals (up to 300us possible !!!!) .... I also was thinking about a problem with the high res. timers, I tried the HPET timers, but this has not helped. >> This is a very strange problem in this architecture, since it happens >> statically almost every three reboots... >> >> It looks like there is something in the kernel / or setted up by bios >> that is happening and locks the task switching context. > > Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch > and Xenomai are you using? Can not it be an SMI issue, have you tried > the SMI workaround? yes, I have tried disabling SMI, but in anyway it fails as follow: Xenomai: SMI-enabled chipset found Xenomai: SMI workaround failed! I have the same problem using either kernel 2.6.34 and xenomai 2.5.6 or kernel 3.5.7 and xenomai 2.6.2.1 Further informations on Monday. Regards, Steph ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing 2013-05-30 13:12 ` Stéphane ANCELOT @ 2013-05-30 18:19 ` Gilles Chanteperdrix 2013-06-03 13:33 ` Stéphane ANCELOT 0 siblings, 1 reply; 6+ messages in thread From: Gilles Chanteperdrix @ 2013-05-30 18:19 UTC (permalink / raw) To: Stéphane ANCELOT; +Cc: xenomai On 05/30/2013 03:12 PM, Stéphane ANCELOT wrote: > On 30/05/2013 13:38, Gilles Chanteperdrix wrote: >> On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote: >> >>> Hi, >>> >>> I have got some problems with an architecture, and 2 realtime tasks. >>> >>> The realtime is not always respected. >> >> Hi, >> >> a few things to check: >> >> Would there be any involuntary mode switches? See rt_task_set_mode or >> pthread_set_mode_np documentation to enable debug. > There is not any involuntary mode switches in the xenomai application. > > The buggy architecture is celeron M: > > Intel® 910GMLE / ICH6M chipsets > > the same disk application, if setted up in the following architecture, > has no problem: > Intel® 852GM and ICH4 chipset > > Are you able to reproduce the problem with the latency test? You can > launch several instances in parallel, if you absolutely need several > task. If yes, then probably the easiest solution is to enable the I-pipe > tracer, then run the latency test with the -f argument. > > I have read more documentation in kernel tracing, and it seems I won't > need lttng, but it looks like only kernel tracing would be enough ? > > I have not managed to reproduce it with a single instance of latency > test. (and I have no problem using a single rt task...) . I will try 2 > instances. > > One more thing that is surprising : I monitored the tasks delay using > clock_gettime() , the software does not watch any big lag !!! ...but it > is visible in the scopemeter using com port signals (up to 300us > possible !!!!) .... I would say it means that the tsc (and APIC) stop during some idle states of the system. I suppose you have CONFIG_CPU_IDLE turned off? Could you try booting with nohlt or idle=poll ? > > I also was thinking about a problem with the high res. timers, I tried > the HPET timers, but this has not helped. I do not believe Xenomai is able to use the HPET timers with Linux 2.6.34. > >>> This is a very strange problem in this architecture, since it happens >>> statically almost every three reboots... >>> >>> It looks like there is something in the kernel / or setted up by bios >>> that is happening and locks the task switching context. >> >> Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch >> and Xenomai are you using? Can not it be an SMI issue, have you tried >> the SMI workaround? > > > yes, I have tried disabling SMI, but in anyway it fails as follow: > > Xenomai: SMI-enabled chipset found > Xenomai: SMI workaround failed! Ok then, your BIOS vendor does not want you to disable the SMI global bit, have you tried other bits? -- Gilles. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing 2013-05-30 18:19 ` Gilles Chanteperdrix @ 2013-06-03 13:33 ` Stéphane ANCELOT 2013-06-03 18:07 ` Gilles Chanteperdrix 0 siblings, 1 reply; 6+ messages in thread From: Stéphane ANCELOT @ 2013-06-03 13:33 UTC (permalink / raw) To: Gilles Chanteperdrix; +Cc: xenomai Hi, The SMI is the source of my problems. the global SMI bit in the ICH6 can not be disabled . However, I used a 32 bits mask trying to disable any bits and managed to reduce the latency (worst : 15us ) refer to paragraph 10.8.3.12 for ICH6 chipset : http://www.intel.com/content/www/us/en/io/io-controller-hub-6-datasheet.html Regards, Steph On 30/05/2013 20:19, Gilles Chanteperdrix wrote: > On 05/30/2013 03:12 PM, Stéphane ANCELOT wrote: > >> On 30/05/2013 13:38, Gilles Chanteperdrix wrote: >>> On 05/30/2013 09:03 AM, Stéphane ANCELOT wrote: >>> >>>> Hi, >>>> >>>> I have got some problems with an architecture, and 2 realtime tasks. >>>> >>>> The realtime is not always respected. >>> Hi, >>> >>> a few things to check: >>> >>> Would there be any involuntary mode switches? See rt_task_set_mode or >>> pthread_set_mode_np documentation to enable debug. >> There is not any involuntary mode switches in the xenomai application. >> >> The buggy architecture is celeron M: >> >> Intel® 910GMLE / ICH6M chipsets >> >> the same disk application, if setted up in the following architecture, >> has no problem: >> Intel® 852GM and ICH4 chipset >> >> Are you able to reproduce the problem with the latency test? You can >> launch several instances in parallel, if you absolutely need several >> task. If yes, then probably the easiest solution is to enable the I-pipe >> tracer, then run the latency test with the -f argument. >> >> I have read more documentation in kernel tracing, and it seems I won't >> need lttng, but it looks like only kernel tracing would be enough ? >> >> I have not managed to reproduce it with a single instance of latency >> test. (and I have no problem using a single rt task...) . I will try 2 >> instances. >> >> One more thing that is surprising : I monitored the tasks delay using >> clock_gettime() , the software does not watch any big lag !!! ...but it >> is visible in the scopemeter using com port signals (up to 300us >> possible !!!!) .... > > I would say it means that the tsc (and APIC) stop during some idle > states of the system. I suppose you have CONFIG_CPU_IDLE turned off? > > Could you try booting with nohlt or idle=poll ? > >> I also was thinking about a problem with the high res. timers, I tried >> the HPET timers, but this has not helped. > > I do not believe Xenomai is able to use the HPET timers with Linux 2.6.34. > >>>> This is a very strange problem in this architecture, since it happens >>>> statically almost every three reboots... >>>> >>>> It looks like there is something in the kernel / or setted up by bios >>>> that is happening and locks the task switching context. >>> Ok, so if it has a bios, it is an x86. Which version of the I-pipe patch >>> and Xenomai are you using? Can not it be an SMI issue, have you tried >>> the SMI workaround? >> >> yes, I have tried disabling SMI, but in anyway it fails as follow: >> >> Xenomai: SMI-enabled chipset found >> Xenomai: SMI workaround failed! > > Ok then, your BIOS vendor does not want you to disable the SMI global > bit, have you tried other bits? > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai] Context switching kernel tracing 2013-06-03 13:33 ` Stéphane ANCELOT @ 2013-06-03 18:07 ` Gilles Chanteperdrix 0 siblings, 0 replies; 6+ messages in thread From: Gilles Chanteperdrix @ 2013-06-03 18:07 UTC (permalink / raw) To: Stéphane ANCELOT; +Cc: xenomai On 06/03/2013 03:33 PM, Stéphane ANCELOT wrote: > Hi, > > > The SMI is the source of my problems. the global SMI bit in the ICH6 can > not be disabled . I read the URL you posted (which I had already read when writing the smi module when Xenomai was still part of the RTAI project), and it does not say that. It says that the global bit can only be used if the SMI_LOCK bit is not set. So there was a time when the smi module checked the SMI_LOCK bit, but after some time, we found that on some machines, even when the SMI_LOCK bit was not armed, it was not possible to mask the SMI_EN bit, so we replaced the check with the current check (re-reading the bits, to check if the masked bit are actually masked). Anyway, the only reason why the SMI_LOCK bit is armed, is if some code, so, presumably the BIOS code, has armed it. Hence my original answer to your query: >> Ok then, your BIOS vendor does not want you to disable the SMI global >> bit, have you tried other bits? Got it? -- Gilles. ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-06-03 18:07 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-05-30 7:03 [Xenomai] Context switching kernel tracing Stéphane ANCELOT 2013-05-30 11:38 ` Gilles Chanteperdrix 2013-05-30 13:12 ` Stéphane ANCELOT 2013-05-30 18:19 ` Gilles Chanteperdrix 2013-06-03 13:33 ` Stéphane ANCELOT 2013-06-03 18:07 ` Gilles Chanteperdrix
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.