* [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6
@ 2011-08-26 20:42 George Broz
2011-08-26 21:18 ` Gilles Chanteperdrix
2011-08-29 10:18 ` Jan Kiszka
0 siblings, 2 replies; 6+ messages in thread
From: George Broz @ 2011-08-26 20:42 UTC (permalink / raw)
To: xenomai
[-- Attachment #1.1: Type: text/plain, Size: 2605 bytes --]
Hello,
I'm experiencing very large overruns in the neighborhood of 10 to 50
*milli-seconds* as reported by the Xenomai latency tool in user-task mode
whenever the Linux OS is heavily loaded. Under lighter loads these are in
the 30 to 70 micro-second (normal) range.
My system is:
- Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
- Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
- Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz
- Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
- Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
- gcc version 4.4.5
I've followed all of the recommendations for kernel configuration and have
disabled CPU sleep features in the BIOS such as SpeedStep and C-States.
Legacy USB has been disabled in the BIOS. Hyperthreading has also been
disabled, although I don't think it matters.
The heavy load was to compile a new Linux kernel while the latency test
ran.
I've captured several traces with the I-pipe tracer - here are some of the
milli-second overrun excerpts:
snippet:
Type User Val. Time Delay Function (Parent)
:| +begin 0x80000001 -14384 0.551 __ipipe_handle_exception+0x3d3
(error_code+0x5f)
:| +end 0x80000001 -14383! 14344.649 __ipipe_handle_exception+0x2d9
(error_code+0x5f)
:| +begin 0xffffff02 -39 0.995 reschedule_interrupt+0x30
(<40116e89>)
snippet:
: +func -17242 0.545 _raw_spin_lock+0x3 (ipc_lock+0x2e)
: +func -17241! 17188.986 ipcperms+0x10
(sys_semtimedop+0x2f8)
:| +begin 0xffffff09 -52 0.797 ipipe_ipi3+0x30
(sys_semtimedop+0x396)
:| +func -51 0.701 __ipipe_handle_irq+0xa
(ipipe_ipi3+0x37)
snippet:
: #func -39426 0.605 scheduler_tick+0xa
(update_process_times+0x5d)
: #func -39425 0.797 ktime_get+0xa
(sched_clock_tick+0x5a)
: #func -39424! 39377.690 acpi_pm_read+0x3 (ktime_get+0x6a)
:| #begin 0xffffff09 -47 0.923 ipipe_ipi3+0x30 (acpi_pm_read+0xf)
:| #func -46 0.707 __ipipe_handle_irq+0xa
(ipipe_ipi3+0x37)
Attached below is a zipped text file with a more complete dump of these
traces, plus more system info, and the complete .config file used to build
this kernel.
If I run the exact same kernel and load on a single-core (Atom N450)
version of this development board, these large overruns do not occur.
Would much appreciate any insight in figuring out why these overruns are
occurring and what I can try to get rid of them.
Thanks in advance!
---
George Broz
Moog, Inc. Industrial Group
[-- Attachment #1.2: Type: text/html, Size: 4641 bytes --]
[-- Attachment #2: HL-ovrruns-xen2_5_6-AtomD510-txt.zip --]
[-- Type: application/zip, Size: 26065 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6
2011-08-26 20:42 [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6 George Broz
@ 2011-08-26 21:18 ` Gilles Chanteperdrix
2011-08-26 21:49 ` George Broz
2011-08-29 10:18 ` Jan Kiszka
1 sibling, 1 reply; 6+ messages in thread
From: Gilles Chanteperdrix @ 2011-08-26 21:18 UTC (permalink / raw)
To: George Broz; +Cc: xenomai
On 08/26/2011 10:42 PM, George Broz wrote:
> Hello,
>
> I'm experiencing very large overruns in the neighborhood of 10 to 50
> *milli-seconds* as reported by the Xenomai latency tool in user-task mode
> whenever the Linux OS is heavily loaded. Under lighter loads these are in
> the 30 to 70 micro-second (normal) range.
>
> My system is:
> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz
> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> - gcc version 4.4.5
>
> I've followed all of the recommendations for kernel configuration and have
> disabled CPU sleep features in the BIOS such as SpeedStep and C-States.
> Legacy USB has been disabled in the BIOS. Hyperthreading has also been
> disabled, although I don't think it matters.
Have you tried to enable the SMI workaround?
--
Gilles.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6
2011-08-26 21:18 ` Gilles Chanteperdrix
@ 2011-08-26 21:49 ` George Broz
2011-08-29 7:13 ` Gilles Chanteperdrix
0 siblings, 1 reply; 6+ messages in thread
From: George Broz @ 2011-08-26 21:49 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: xenomai
>Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on
08/26/2011 02:18:05 PM:
> On 08/26/2011 10:42 PM, George Broz wrote:
> > Hello,
> >
> > I'm experiencing very large overruns in the neighborhood of 10 to 50
> > *milli-seconds* as reported by the Xenomai latency tool in user-task
mode
> > whenever the Linux OS is heavily loaded. Under lighter loads these are
in
> > the 30 to 70 micro-second (normal) range.
> >
> > My system is:
> > - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> > - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> > - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz
> > - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> > - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> > - gcc version 4.4.5
> >
> > I've followed all of the recommendations for kernel configuration and
have
> > disabled CPU sleep features in the BIOS such as SpeedStep and
C-States.
> > Legacy USB has been disabled in the BIOS. Hyperthreading has also been
> > disabled, although I don't think it matters.
>
> Have you tried to enable the SMI workaround?
>
> --
> Gilles.
>
Yes - SMI workaround was enabled when the overruns were observed.
# CONFIG_XENO_HW_SMI_DETECT_DISABLE is not set
CONFIG_XENO_HW_SMI_DETECT=y
CONFIG_XENO_HW_SMI_WORKAROUND=y
CONFIG_XENO_HW_SMI_ALL=y
Aug 25 11:32:33 nxtgenhd kernel: [ 2.492709] I-pipe: Domain Xenomai
registered.
:
Aug 25 11:32:33 nxtgenhd kernel: [ 11.579015] Xenomai: SMI-enabled
chipset found
Aug 25 11:32:33 nxtgenhd kernel: [ 11.579044] Xenomai: SMI workaround
enabled
--George
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6
2011-08-26 21:49 ` George Broz
@ 2011-08-29 7:13 ` Gilles Chanteperdrix
2011-08-30 2:45 ` George Broz
0 siblings, 1 reply; 6+ messages in thread
From: Gilles Chanteperdrix @ 2011-08-29 7:13 UTC (permalink / raw)
To: George Broz; +Cc: xenomai
On 08/26/2011 11:49 PM, George Broz wrote:
>> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on
> 08/26/2011 02:18:05 PM:
>> On 08/26/2011 10:42 PM, George Broz wrote:
>>> Hello,
>>>
>>> I'm experiencing very large overruns in the neighborhood of 10 to 50
>>> *milli-seconds* as reported by the Xenomai latency tool in user-task
> mode
>>> whenever the Linux OS is heavily loaded. Under lighter loads these are
> in
>>> the 30 to 70 micro-second (normal) range.
>>>
>>> My system is:
>>> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
>>> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
>>> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz
>>> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
>>> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
>>> - gcc version 4.4.5
>>>
>>> I've followed all of the recommendations for kernel configuration and
> have
>>> disabled CPU sleep features in the BIOS such as SpeedStep and
> C-States.
>>> Legacy USB has been disabled in the BIOS. Hyperthreading has also been
>
>>> disabled, although I don't think it matters.
>>
>> Have you tried to enable the SMI workaround?
>>
>> --
>> Gilles.
>>
>
> Yes - SMI workaround was enabled when the overruns were observed.
Could you try disabling CONFIG_SMP to see if you get the same behaviour,
and take a few traces?
If you processor supports 64 bits mode, could you try running a 64 bits
kernel?
--
Gilles.
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6
2011-08-29 7:13 ` Gilles Chanteperdrix
@ 2011-08-30 2:45 ` George Broz
0 siblings, 0 replies; 6+ messages in thread
From: George Broz @ 2011-08-30 2:45 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: xenomai
[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]
Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on
08/29/2011 12:13:48 AM:
> On 08/26/2011 11:49 PM, George Broz wrote:
> >> Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org> wrote on
> > 08/26/2011 02:18:05 PM:
> >> On 08/26/2011 10:42 PM, George Broz wrote:
> >>> Hello,
> >>>
> >>> I'm experiencing very large overruns in the neighborhood of 10 to 50
> >>> *milli-seconds* as reported by the Xenomai latency tool in user-task
> > mode
> >>> whenever the Linux OS is heavily loaded. Under lighter loads these
are
> > in
> >>> the 30 to 70 micro-second (normal) range.
> >>>
> >>> My system is:
> >>> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> >>> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> >>> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz
> >>> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> >>> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> >>> - gcc version 4.4.5
> >>>
> >>> I've followed all of the recommendations for kernel configuration
and
> > have
> >>> disabled CPU sleep features in the BIOS such as SpeedStep and
> > C-States.
> >>> Legacy USB has been disabled in the BIOS. Hyperthreading has also
been
> >
> >>> disabled, although I don't think it matters.
> >>
> >> Have you tried to enable the SMI workaround?
> >>
> >> --
> >> Gilles.
> >>
> >
> > Yes - SMI workaround was enabled when the overruns were observed.
>
> Could you try disabling CONFIG_SMP to see if you get the same behaviour,
> and take a few traces?
> If you processor supports 64 bits mode, could you try running a 64 bits
> kernel?
>
>
> --
> Gilles.
>
I disabled CONFIG_SMP and did not get the same behaviour - latencies were
normal (albeit now slightly increased). The processor supports 64 bit
mode, but my initial attempt to get that working failed (for unrelated
reasons).
I then started looking at the hardware some more after Jan's comment, and
found another CPU frequency throttling mechanism that exists in the Atom
N450 and D510. This is the "thermal monitor" (TM1) as described in this
Intel document:
http://download.intel.com/embedded/processors/thermalguide/323439.pdf
Even though my dev. board had a fan, adding even more cooling
significantly reduced the occurrence of the overruns (but they would still
occur occasionally). I then disabled the thermal monitor (TM1) feature
from the BIOS, and since then have not seen any more happen (after about 2
hours of testing). I will test more tomorrow, and post if I see the
problem still exists.
Thanks very much for your (and Jan's) help.
--George
[-- Attachment #2: Type: text/html, Size: 3894 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6
2011-08-26 20:42 [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6 George Broz
2011-08-26 21:18 ` Gilles Chanteperdrix
@ 2011-08-29 10:18 ` Jan Kiszka
1 sibling, 0 replies; 6+ messages in thread
From: Jan Kiszka @ 2011-08-29 10:18 UTC (permalink / raw)
To: George Broz; +Cc: xenomai
On 2011-08-26 22:42, George Broz wrote:
> Hello,
>
> I'm experiencing very large overruns in the neighborhood of 10 to 50
> *milli-seconds* as reported by the Xenomai latency tool in user-task mode
> whenever the Linux OS is heavily loaded. Under lighter loads these are in
> the 30 to 70 micro-second (normal) range.
>
> My system is:
> - Linux 2.6.37.6 (32-bit), Ubuntu 10.10 distribution
> - Xenomai 2.5.6 using I-pipe patch adeos-ipipe-2.6.37.6-x86-2.9-02
> - Intel x86 Atom (dual-core) CPU D510 @ 1.66GHz
> - Emb. dev board from Intel (ICH8M ctrlr, Ethernet h/w for e1000e)
> - Xenomai configured w&w/o --enable-smp --enable-x86-tsc options
> - gcc version 4.4.5
>
> I've followed all of the recommendations for kernel configuration and have
> disabled CPU sleep features in the BIOS such as SpeedStep and C-States.
> Legacy USB has been disabled in the BIOS. Hyperthreading has also been
> disabled, although I don't think it matters.
>
> The heavy load was to compile a new Linux kernel while the latency test
> ran.
>
> I've captured several traces with the I-pipe tracer - here are some of the
> milli-second overrun excerpts:
>
> snippet:
> Type User Val. Time Delay Function (Parent)
> :| +begin 0x80000001 -14384 0.551 __ipipe_handle_exception+0x3d3
> (error_code+0x5f)
> :| +end 0x80000001 -14383! 14344.649 __ipipe_handle_exception+0x2d9
> (error_code+0x5f)
> :| +begin 0xffffff02 -39 0.995 reschedule_interrupt+0x30
> (<40116e89>)
>
> snippet:
> : +func -17242 0.545 _raw_spin_lock+0x3 (ipc_lock+0x2e)
> : +func -17241! 17188.986 ipcperms+0x10
> (sys_semtimedop+0x2f8)
> :| +begin 0xffffff09 -52 0.797 ipipe_ipi3+0x30
> (sys_semtimedop+0x396)
> :| +func -51 0.701 __ipipe_handle_irq+0xa
> (ipipe_ipi3+0x37)
>
> snippet:
> : #func -39426 0.605 scheduler_tick+0xa
> (update_process_times+0x5d)
> : #func -39425 0.797 ktime_get+0xa
> (sched_clock_tick+0x5a)
> : #func -39424! 39377.690 acpi_pm_read+0x3 (ktime_get+0x6a)
> :| #begin 0xffffff09 -47 0.923 ipipe_ipi3+0x30 (acpi_pm_read+0xf)
> :| #func -46 0.707 __ipipe_handle_irq+0xa
> (ipipe_ipi3+0x37)
Specifically this last snippet suggests that your hardware has some
serious latency problems. There is nothing in that code path that allows
a software explanation of those 39 ms.
Jan
--
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2011-08-30 2:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-08-26 20:42 [Xenomai-help] milli-second overruns under heavy load on x86, running Xenomai 2.5.6 George Broz
2011-08-26 21:18 ` Gilles Chanteperdrix
2011-08-26 21:49 ` George Broz
2011-08-29 7:13 ` Gilles Chanteperdrix
2011-08-30 2:45 ` George Broz
2011-08-29 10:18 ` Jan Kiszka
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.