[Xenomai-help] Unexpected preemption of highest priority thread

[Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 1890 bytes --]

Hi,

I have some code (simplified example below) which uses
'pthread_wait_np(&Overrun)' within a while loop, using a period of
1,000,000ns (1ms) which has been set using 'pthread_make_periodic_np()'.
This wakes the thread up less than 2,000ns late in general, sometimes
20,000ns late, and at most less than 45,000ns late. To ensure that I perform
the tasks within the thread from a consistent 'millisecond tick' I set an
expected time value 45,000ns later than the earliest wake time each period,
and then use a busy loop until the current time equals this expected time.
This busy loop takes < 600ns per cycle, and so the latency from my expected
time should be a maximum of 600ns. The latencies I am actually seeing are as
high as 13,000ns, and these occur during this busy loop (I have timed it),
implying that this loop is being preempted.

while(1)
{
   pthread_wait_np(&Overrun);

   WakeTime = GetTime();
   while(WakeTime < ExpectedTime)
   { WakeTime = GetTime(); } // Preempted here

   DoWork();

   ExpectedTime += Period;
}

This is a Xenomai POSIX thread with a priority of 99, and is the only thread
within my program with a priority of 99 or higher. There are no context
switches occuring within this thread (confirmed using pthread_set_mode_np(0,
PTHREAD_WARNSW)).

I have had a brief experimentation with kernel settings, for example
applying the SMI workaround, but this produces no difference to the
preemption that I measure. I have also re-read all the documentation I could
find about Xenomai and preemption and I cannot see any way of fixing this
without trial and error with the kernel settings.

Do you have any other ideas for avenues that I could explore, or could you
recommend documentation about parts of the kernel that are more likely to
have an effect on preemption to speed up this process?

I am running Ubuntu on:
Kernel: 2.6.32.8
Xenomai: 2.5.3

Steve

[-- Attachment #2: Type: text/html, Size: 2024 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> Hi,
> 
> I have some code (simplified example below) which uses
> 'pthread_wait_np(&Overrun)' within a while loop, using a period of
> 1,000,000ns (1ms) which has been set using 'pthread_make_periodic_np()'.
> This wakes the thread up less than 2,000ns late in general, sometimes
> 20,000ns late, and at most less than 45,000ns late.

Since you do not tell us, I guess you are running on x86. Depending on
the x86 you are running, 45 us may be high.


 To ensure that I perform
> the tasks within the thread from a consistent 'millisecond tick' I set an
> expected time value 45,000ns later than the earliest wake time each period,
> and then use a busy loop until the current time equals this expected time.
> This busy loop takes < 600ns per cycle, and so the latency from my expected
> time should be a maximum of 600ns.

This reasoning is flawed: you may be preempted by interrupts any time.
Including just about at the time where you were going to exit the busy
loop. Unless you spin irqs off, and only reenable irqs when the job the
thread is supposed to be doing is done. And if you have an SMI issue,
even turning off the interrupts will not help.


 The latencies I am actually seeing are as
> high as 13,000ns, and these occur during this busy loop (I have timed it),
> implying that this loop is being preempted.
> 
> while(1)
> {
>    pthread_wait_np(&Overrun);
> 
>    WakeTime = GetTime();
>    while(WakeTime < ExpectedTime)
>    { WakeTime = GetTime(); } // Preempted here
> 
>    DoWork();
> 
>    ExpectedTime += Period;
> }
> 
> This is a Xenomai POSIX thread with a priority of 99, and is the only thread
> within my program with a priority of 99 or higher. There are no context
> switches occuring within this thread (confirmed using pthread_set_mode_np(0,
> PTHREAD_WARNSW)).
> 
> I have had a brief experimentation with kernel settings, for example
> applying the SMI workaround, but this produces no difference to the
> preemption that I measure.

The question is: is the SMI workaround working for your chipset? If that
is the case, you should get messages in the kernel logs.

-- 
					    Gilles.

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 2993 bytes --]

On 21 July 2010 12:06, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Stephen Bryant wrote:
> > Hi,
> >
> > I have some code (simplified example below) which uses
> > 'pthread_wait_np(&Overrun)' within a while loop, using a period of
> > 1,000,000ns (1ms) which has been set using 'pthread_make_periodic_np()'.
> > This wakes the thread up less than 2,000ns late in general, sometimes
> > 20,000ns late, and at most less than 45,000ns late.
>
> Since you do not tell us, I guess you are running on x86. Depending on
> the x86 you are running, 45 us may be high.
>
>
Yes, it is x86, sorry for the omission. What sort of latencies would you
expect me to see?


>  To ensure that I perform
> > the tasks within the thread from a consistent 'millisecond tick' I set an
> > expected time value 45,000ns later than the earliest wake time each
> period,
> > and then use a busy loop until the current time equals this expected
> time.
> > This busy loop takes < 600ns per cycle, and so the latency from my
> expected
> > time should be a maximum of 600ns.
>
> This reasoning is flawed: you may be preempted by interrupts any time.
> Including just about at the time where you were going to exit the busy
> loop. Unless you spin irqs off, and only reenable irqs when the job the
> thread is supposed to be doing is done. And if you have an SMI issue,
> even turning off the interrupts will not help.
>
>
I was under the (apparently flawed) impression that Adeos was used to
prevent interrupts preempting a running high priority task in primary mode,
only forwarding the interrupts to Linux once the task was waiting /
finished. I guess that it only prevents a subset of all interrupts, and the
rest must be specifically disabled then re-enabled by the program, or maybe
it does not intercept any interrupts?


>
>  The latencies I am actually seeing are as
> > high as 13,000ns, and these occur during this busy loop (I have timed
> it),
> > implying that this loop is being preempted.
> >
> > while(1)
> > {
> >    pthread_wait_np(&Overrun);
> >
> >    WakeTime = GetTime();
> >    while(WakeTime < ExpectedTime)
> >    { WakeTime = GetTime(); } // Preempted here
> >
> >    DoWork();
> >
> >    ExpectedTime += Period;
> > }
> >
> > This is a Xenomai POSIX thread with a priority of 99, and is the only
> thread
> > within my program with a priority of 99 or higher. There are no context
> > switches occuring within this thread (confirmed using
> pthread_set_mode_np(0,
> > PTHREAD_WARNSW)).
> >
> > I have had a brief experimentation with kernel settings, for example
> > applying the SMI workaround, but this produces no difference to the
> > preemption that I measure.
>
> The question is: is the SMI workaround working for your chipset? If that
> is the case, you should get messages in the kernel logs.
>
>
I've had a look at dmesg, but cannot find any references to SMI - what
should I be looking for?


> --
>                                             Gilles.
>

Steve

[-- Attachment #2: Type: text/html, Size: 4198 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 3278 bytes --]

On 21 July 2010 13:05, Stephen Bryant <sbxenomai@domain.hid> wrote:

> On 21 July 2010 12:06, Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org> wrote:
>
>> Stephen Bryant wrote:
>> > Hi,
>> >
>> > I have some code (simplified example below) which uses
>> > 'pthread_wait_np(&Overrun)' within a while loop, using a period of
>> > 1,000,000ns (1ms) which has been set using 'pthread_make_periodic_np()'.
>> > This wakes the thread up less than 2,000ns late in general, sometimes
>> > 20,000ns late, and at most less than 45,000ns late.
>>
>> Since you do not tell us, I guess you are running on x86. Depending on
>> the x86 you are running, 45 us may be high.
>>
>>
> Yes, it is x86, sorry for the omission. What sort of latencies would you
> expect me to see?
>

Sorry, I should have given more info. It is an Intel Core 2 Duo, running a
kernel with SMP disabled.


>
>>  To ensure that I perform
>> > the tasks within the thread from a consistent 'millisecond tick' I set
>> an
>> > expected time value 45,000ns later than the earliest wake time each
>> period,
>> > and then use a busy loop until the current time equals this expected
>> time.
>> > This busy loop takes < 600ns per cycle, and so the latency from my
>> expected
>> > time should be a maximum of 600ns.
>>
>> This reasoning is flawed: you may be preempted by interrupts any time.
>> Including just about at the time where you were going to exit the busy
>> loop. Unless you spin irqs off, and only reenable irqs when the job the
>> thread is supposed to be doing is done. And if you have an SMI issue,
>> even turning off the interrupts will not help.
>>
>>
> I was under the (apparently flawed) impression that Adeos was used to
> prevent interrupts preempting a running high priority task in primary mode,
> only forwarding the interrupts to Linux once the task was waiting /
> finished. I guess that it only prevents a subset of all interrupts, and the
> rest must be specifically disabled then re-enabled by the program, or maybe
> it does not intercept any interrupts?
>
>
>>
>>  The latencies I am actually seeing are as
>> > high as 13,000ns, and these occur during this busy loop (I have timed
>> it),
>> > implying that this loop is being preempted.
>> >
>> > while(1)
>> > {
>> >    pthread_wait_np(&Overrun);
>> >
>> >    WakeTime = GetTime();
>> >    while(WakeTime < ExpectedTime)
>> >    { WakeTime = GetTime(); } // Preempted here
>> >
>> >    DoWork();
>> >
>> >    ExpectedTime += Period;
>> > }
>> >
>> > This is a Xenomai POSIX thread with a priority of 99, and is the only
>> thread
>> > within my program with a priority of 99 or higher. There are no context
>> > switches occuring within this thread (confirmed using
>> pthread_set_mode_np(0,
>> > PTHREAD_WARNSW)).
>> >
>> > I have had a brief experimentation with kernel settings, for example
>> > applying the SMI workaround, but this produces no difference to the
>> > preemption that I measure.
>>
>> The question is: is the SMI workaround working for your chipset? If that
>> is the case, you should get messages in the kernel logs.
>>
>>
> I've had a look at dmesg, but cannot find any references to SMI - what
> should I be looking for?
>
>
>> --
>>                                             Gilles.
>>
>
> Steve
>
Steve

[-- Attachment #2: Type: text/html, Size: 4885 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> On 21 July 2010 12:06, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org
> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> 
>     Stephen Bryant wrote:
>     > Hi,
>     >
>     > I have some code (simplified example below) which uses
>     > 'pthread_wait_np(&Overrun)' within a while loop, using a period of
>     > 1,000,000ns (1ms) which has been set using
>     'pthread_make_periodic_np()'.
>     > This wakes the thread up less than 2,000ns late in general, sometimes
>     > 20,000ns late, and at most less than 45,000ns late.
> 
>     Since you do not tell us, I guess you are running on x86. Depending on
>     the x86 you are running, 45 us may be high.
> 
> 
> Yes, it is x86, sorry for the omission. What sort of latencies would you
> expect me to see?

If it is for instance, a core 2 duo, it should be around, say, 10 or
20us. If on the other hand, it is a geode, then 45us may be OK. The
actual results also depend on the system load, an unloaded core 2 will
have average latencies under 10us.

>     This reasoning is flawed: you may be preempted by interrupts any time.
>     Including just about at the time where you were going to exit the busy
>     loop. Unless you spin irqs off, and only reenable irqs when the job the
>     thread is supposed to be doing is done. And if you have an SMI issue,
>     even turning off the interrupts will not help.
> 
> 
> I was under the (apparently flawed) impression that Adeos was used to
> prevent interrupts preempting a running high priority task in primary
> mode, only forwarding the interrupts to Linux once the task was waiting
> / finished. I guess that it only prevents a subset of all interrupts,
> and the rest must be specifically disabled then re-enabled by the
> program, or maybe it does not intercept any interrupts?

It prevents secondary domain interrupts from being executed, but the
primary domain interrupt such as the timer interrupt are still executed,
and the secondary domain interrupts are still ticking and cause a
context switch, even if masked immediately.

>     The question is: is the SMI workaround working for your chipset? If that
>     is the case, you should get messages in the kernel logs.
> 
> 
> I've had a look at dmesg, but cannot find any references to SMI - what
> should I be looking for?

What is described in the TROUBLESHOOTING file.

-- 
					    Gilles.

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 3109 bytes --]

On 21 July 2010 14:06, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Stephen Bryant wrote:
> > On 21 July 2010 12:06, Gilles Chanteperdrix
> > <gilles.chanteperdrix@xenomai.org
> > <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> >
> >     Stephen Bryant wrote:
> >     > Hi,
> >     >
> >     > I have some code (simplified example below) which uses
> >     > 'pthread_wait_np(&Overrun)' within a while loop, using a period of
> >     > 1,000,000ns (1ms) which has been set using
> >     'pthread_make_periodic_np()'.
> >     > This wakes the thread up less than 2,000ns late in general,
> sometimes
> >     > 20,000ns late, and at most less than 45,000ns late.
> >
> >     Since you do not tell us, I guess you are running on x86. Depending
> on
> >     the x86 you are running, 45 us may be high.
> >
> >
> > Yes, it is x86, sorry for the omission. What sort of latencies would you
> > expect me to see?
>
> If it is for instance, a core 2 duo, it should be around, say, 10 or
> 20us. If on the other hand, it is a geode, then 45us may be OK. The
> actual results also depend on the system load, an unloaded core 2 will
> have average latencies under 10us.
>
> >     This reasoning is flawed: you may be preempted by interrupts any
> time.
> >     Including just about at the time where you were going to exit the
> busy
> >     loop. Unless you spin irqs off, and only reenable irqs when the job
> the
> >     thread is supposed to be doing is done. And if you have an SMI issue,
> >     even turning off the interrupts will not help.
> >
> >
> > I was under the (apparently flawed) impression that Adeos was used to
> > prevent interrupts preempting a running high priority task in primary
> > mode, only forwarding the interrupts to Linux once the task was waiting
> > / finished. I guess that it only prevents a subset of all interrupts,
> > and the rest must be specifically disabled then re-enabled by the
> > program, or maybe it does not intercept any interrupts?
>
> It prevents secondary domain interrupts from being executed, but the
> primary domain interrupt such as the timer interrupt are still executed,
> and the secondary domain interrupts are still ticking and cause a
> context switch, even if masked immediately.
>
> >     The question is: is the SMI workaround working for your chipset? If
> that
> >     is the case, you should get messages in the kernel logs.
> >
> >
> > I've had a look at dmesg, but cannot find any references to SMI - what
> > should I be looking for?
>
> What is described in the TROUBLESHOOTING file.
>
>
I have tried disabling the SMI workaround, ensuring that SMI detection is
not disabled, and logging the kernel output (setting the kernel log level to
8 in grub and sending the output to the serial port to be picked up by
another machine). The troubleshooting file states that I should see
"Xenomai: Intel chipset found and SMI workaround not
enabled, you may encounter high interrupt latencies." in this output,
however this does not occur - I am using an Intel Core 2 Duo but with SMP
disabled, if this has any relevance.

Steve

[-- Attachment #2: Type: text/html, Size: 3928 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> On 21 July 2010 14:06, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org
> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> 
>     Stephen Bryant wrote:
>     > I've had a look at dmesg, but cannot find any references to SMI - what
>     > should I be looking for?
> 
>     What is described in the TROUBLESHOOTING file.
> 
> 
> I have tried disabling the SMI workaround, ensuring that SMI detection
> is not disabled, and logging the kernel output (setting the kernel log
> level to 8 in grub and sending the output to the serial port to be
> picked up by another machine). The troubleshooting file states that I
> should see "Xenomai: Intel chipset found and SMI workaround not
> enabled, you may encounter high interrupt latencies." in this output,
> however this does not occur - I am using an Intel Core 2 Duo but with
> SMP disabled, if this has any relevance.

The SMI detection/workaround is based on the chipset you have. If your
chipset is an ICH10, Stefan posted a patch a few days ago. If it is
another one, then please send us the result of lspci -vv on your target.

-- 
					    Gilles.

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1.1: Type: text/plain, Size: 1305 bytes --]

On 22 July 2010 08:54, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Stephen Bryant wrote:
> > On 21 July 2010 14:06, Gilles Chanteperdrix
> > <gilles.chanteperdrix@xenomai.org
> > <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> >
> >     Stephen Bryant wrote:
> >     > I've had a look at dmesg, but cannot find any references to SMI -
> what
> >     > should I be looking for?
> >
> >     What is described in the TROUBLESHOOTING file.
> >
> >
> > I have tried disabling the SMI workaround, ensuring that SMI detection
> > is not disabled, and logging the kernel output (setting the kernel log
> > level to 8 in grub and sending the output to the serial port to be
> > picked up by another machine). The troubleshooting file states that I
> > should see "Xenomai: Intel chipset found and SMI workaround not
> > enabled, you may encounter high interrupt latencies." in this output,
> > however this does not occur - I am using an Intel Core 2 Duo but with
> > SMP disabled, if this has any relevance.
>
> The SMI detection/workaround is based on the chipset you have. If your
> chipset is an ICH10, Stefan posted a patch a few days ago. If it is
> another one, then please send us the result of lspci -vv on your target.
>

It seems to be ICH8, so the result is attached,

Steve

[-- Attachment #1.2: Type: text/html, Size: 1954 bytes --]

[-- Attachment #2: lspci-vv.txt --]
[-- Type: text/plain, Size: 9135 bytes --]

00:00.0 Host bridge: Intel Corporation 82Q963/Q965 Memory Controller Hub (rev 02)
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ >SERR- <PERR-
	Latency: 0
	Capabilities: <access denied>

00:02.0 VGA compatible controller: Intel Corporation 82Q963/Q965 Integrated Graphics Controller (rev 02) (prog-if 00 [VGA controller])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at f0400000 (32-bit, non-prefetchable) [size=1M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at 1200 [size=8]
	Capabilities: <access denied>

00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 20
	Region 4: I/O ports at 1100 [size=32]

00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 21
	Region 4: I/O ports at 1120 [size=32]

00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 02) (prog-if 20 [EHCI])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin C routed to IRQ 22
	Region 0: Memory at f0504000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>

00:1b.0 Audio device: Intel Corporation 82801H (ICH8 Family) HD Audio Controller (rev 02)
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 5
	Region 0: Memory at f0500000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>

00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=20, subordinate=20, sec-latency=0
	I/O behind bridge: 00002000-00002fff
	Memory behind bridge: f0900000-f0bfffff
	Prefetchable memory behind bridge: 0000000080000000-00000000801fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: <access denied>

00:1c.1 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 2 (rev 02) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=3f, subordinate=3f, sec-latency=0
	I/O behind bridge: 00003000-00003fff
	Memory behind bridge: f0600000-f08fffff
	Prefetchable memory behind bridge: 0000000080200000-00000000803fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: <access denied>

00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 20
	Region 4: I/O ports at 1140 [size=32]

00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 02) (prog-if 00 [UHCI])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 21
	Region 4: I/O ports at 1160 [size=32]

00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 02) (prog-if 20 [EHCI])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin A routed to IRQ 20
	Region 0: Memory at f0504400 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>

00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2) (prog-if 01 [Subtractive decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Bus: primary=00, secondary=07, subordinate=07, sec-latency=32
	Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: <access denied>

00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controller (rev 02)
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Capabilities: <access denied>

00:1f.2 IDE interface: Intel Corporation 82801H (ICH8 Family) 4 port SATA IDE Controller (rev 02) (prog-if 8a [Master SecP PriP])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 18
	Region 0: I/O ports at 01f0 [size=8]
	Region 1: I/O ports at 03f4 [size=1]
	Region 2: I/O ports at 0170 [size=8]
	Region 3: I/O ports at 0374 [size=1]
	Region 4: I/O ports at 11c0 [size=16]
	Region 5: I/O ports at 11d0 [size=16]
	Capabilities: <access denied>

00:1f.5 IDE interface: Intel Corporation 82801H (ICH8 Family) 2 port SATA IDE Controller (rev 02) (prog-if 85 [Master SecO PriO])
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0
	Interrupt: pin B routed to IRQ 18
	Region 0: I/O ports at 1218 [size=8]
	Region 1: I/O ports at 1230 [size=4]
	Region 2: I/O ports at 1220 [size=8]
	Region 3: I/O ports at 1234 [size=4]
	Region 4: I/O ports at 11e0 [size=16]
	Region 5: I/O ports at 11f0 [size=16]
	Capabilities: <access denied>

20:00.0 Communication controller: Unknown device 163f:0018 (rev 01)
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 11
	Region 0: Memory at f0900000 (32-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>

3f:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5755 Gigabit Ethernet PCI Express (rev 02)
	Subsystem: Hewlett-Packard Company Unknown device 2808
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at f0800000 (64-bit, non-prefetchable) [size=64K]
	Expansion ROM at <ignored> [disabled]
	Capabilities: <access denied>

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> On 22 July 2010 08:54, Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org> wrote:
> 
>> Stephen Bryant wrote:
>>> On 21 July 2010 14:06, Gilles Chanteperdrix
>>> <gilles.chanteperdrix@xenomai.org
>>> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
>>>
>>>     Stephen Bryant wrote:
>>>     > I've had a look at dmesg, but cannot find any references to SMI -
>> what
>>>     > should I be looking for?
>>>
>>>     What is described in the TROUBLESHOOTING file.
>>>
>>>
>>> I have tried disabling the SMI workaround, ensuring that SMI detection
>>> is not disabled, and logging the kernel output (setting the kernel log
>>> level to 8 in grub and sending the output to the serial port to be
>>> picked up by another machine). The troubleshooting file states that I
>>> should see "Xenomai: Intel chipset found and SMI workaround not
>>> enabled, you may encounter high interrupt latencies." in this output,
>>> however this does not occur - I am using an Intel Core 2 Duo but with
>>> SMP disabled, if this has any relevance.
>> The SMI detection/workaround is based on the chipset you have. If your
>> chipset is an ICH10, Stefan posted a patch a few days ago. If it is
>> another one, then please send us the result of lspci -vv on your target.
>>
> 
> It seems to be ICH8, so the result is attached,

Please try with the following patch:

diff --git a/ksrc/arch/x86/smi.c b/ksrc/arch/x86/smi.c
index 2116899..a547e88 100644
--- a/ksrc/arch/x86/smi.c
+++ b/ksrc/arch/x86/smi.c
@@ -48,6 +48,7 @@ static struct pci_device_id rthal_smi_pci_tbl[] = {
        {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH6_2)},
        {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH7_0)},
        {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH7_1)},
+       {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH8_0)},
        {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ICH8_4)},
        {0,},
 };


-- 
					    Gilles.

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 1879 bytes --]

On 22 July 2010 11:56, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Stephen Bryant wrote:
> > On 22 July 2010 08:54, Gilles Chanteperdrix <
> > gilles.chanteperdrix@xenomai.org> wrote:
> >
> >> Stephen Bryant wrote:
> >>> On 21 July 2010 14:06, Gilles Chanteperdrix
> >>> <gilles.chanteperdrix@xenomai.org
> >>> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> >>>
> >>>     Stephen Bryant wrote:
> >>>     > I've had a look at dmesg, but cannot find any references to SMI -
> >> what
> >>>     > should I be looking for?
> >>>
> >>>     What is described in the TROUBLESHOOTING file.
> >>>
> >>>
> >>> I have tried disabling the SMI workaround, ensuring that SMI detection
> >>> is not disabled, and logging the kernel output (setting the kernel log
> >>> level to 8 in grub and sending the output to the serial port to be
> >>> picked up by another machine). The troubleshooting file states that I
> >>> should see "Xenomai: Intel chipset found and SMI workaround not
> >>> enabled, you may encounter high interrupt latencies." in this output,
> >>> however this does not occur - I am using an Intel Core 2 Duo but with
> >>> SMP disabled, if this has any relevance.
> >> The SMI detection/workaround is based on the chipset you have. If your
> >> chipset is an ICH10, Stefan posted a patch a few days ago. If it is
> >> another one, then please send us the result of lspci -vv on your target.
> >>
> >
> > It seems to be ICH8, so the result is attached,
>
> Please try with the following patch:
>
>
> --
>                                             Gilles.
>

With the patch applied and SMI Workaround not used I get the kernel message
as expected: "...SMI workaround not enabled...".

However, with the SMI Workaround enabled and Globally disabling SMI I get
the messages:
"Xenomai: SMI-enabled chipset found
Xenomai: SMI workaround failed!"

Steve

[-- Attachment #2: Type: text/html, Size: 2790 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> On 22 July 2010 11:56, Gilles Chanteperdrix <
> gilles.chanteperdrix@xenomai.org> wrote:
> 
>> Stephen Bryant wrote:
>>> On 22 July 2010 08:54, Gilles Chanteperdrix <
>>> gilles.chanteperdrix@xenomai.org> wrote:
>>>
>>>> Stephen Bryant wrote:
>>>>> On 21 July 2010 14:06, Gilles Chanteperdrix
>>>>> <gilles.chanteperdrix@xenomai.org
>>>>> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
>>>>>
>>>>>     Stephen Bryant wrote:
>>>>>     > I've had a look at dmesg, but cannot find any references to SMI -
>>>> what
>>>>>     > should I be looking for?
>>>>>
>>>>>     What is described in the TROUBLESHOOTING file.
>>>>>
>>>>>
>>>>> I have tried disabling the SMI workaround, ensuring that SMI detection
>>>>> is not disabled, and logging the kernel output (setting the kernel log
>>>>> level to 8 in grub and sending the output to the serial port to be
>>>>> picked up by another machine). The troubleshooting file states that I
>>>>> should see "Xenomai: Intel chipset found and SMI workaround not
>>>>> enabled, you may encounter high interrupt latencies." in this output,
>>>>> however this does not occur - I am using an Intel Core 2 Duo but with
>>>>> SMP disabled, if this has any relevance.
>>>> The SMI detection/workaround is based on the chipset you have. If your
>>>> chipset is an ICH10, Stefan posted a patch a few days ago. If it is
>>>> another one, then please send us the result of lspci -vv on your target.
>>>>
>>> It seems to be ICH8, so the result is attached,
>> Please try with the following patch:
>>
>>
>> --
>>                                             Gilles.
>>
> 
> With the patch applied and SMI Workaround not used I get the kernel message
> as expected: "...SMI workaround not enabled...".
> 
> However, with the SMI Workaround enabled and Globally disabling SMI I get
> the messages:
> "Xenomai: SMI-enabled chipset found
> Xenomai: SMI workaround failed!"

It probably means that your board manufacturer has decided, at BIOS
level, to prevent you from disabling the SMIs. You may want to try
disabling the SMI bits one by one.

-- 
					    Gilles.

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 2096 bytes --]

On 26 July 2010 13:11, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> >>>>>
> >>>>>     Stephen Bryant wrote:
> >>>>>     > I've had a look at dmesg, but cannot find any references to SMI
> -
> >>>> what
> >>>>>     > should I be looking for?
> >>>>>
> >>>>>     What is described in the TROUBLESHOOTING file.
> >>>>>
> >>>>>
> >>>>> I have tried disabling the SMI workaround, ensuring that SMI
> detection
> >>>>> is not disabled, and logging the kernel output (setting the kernel
> log
> >>>>> level to 8 in grub and sending the output to the serial port to be
> >>>>> picked up by another machine). The troubleshooting file states that I
> >>>>> should see "Xenomai: Intel chipset found and SMI workaround not
> >>>>> enabled, you may encounter high interrupt latencies." in this output,
> >>>>> however this does not occur - I am using an Intel Core 2 Duo but with
> >>>>> SMP disabled, if this has any relevance.
> >>>> The SMI detection/workaround is based on the chipset you have. If your
> >>>> chipset is an ICH10, Stefan posted a patch a few days ago. If it is
> >>>> another one, then please send us the result of lspci -vv on your
> target.
> >>>>
> >>> It seems to be ICH8, so the result is attached,
> >> Please try with the following patch:
> >>
> >>
> >> --
> >>                                             Gilles.
> >>
> >
> > With the patch applied and SMI Workaround not used I get the kernel
> message
> > as expected: "...SMI workaround not enabled...".
> >
> > However, with the SMI Workaround enabled and Globally disabling SMI I get
> > the messages:
> > "Xenomai: SMI-enabled chipset found
> > Xenomai: SMI workaround failed!"
>
> It probably means that your board manufacturer has decided, at BIOS
> level, to prevent you from disabling the SMIs. You may want to try
> disabling the SMI bits one by one.
>
> --
>                                             Gilles.
>

Do you mean enabling / disabling the individual options in the SMI
Workaround section of the kernel config file, or are you suggesting
something lower level / something else?

Steve

[-- Attachment #2: Type: text/html, Size: 2983 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> Do you mean enabling / disabling the individual options in the SMI
> Workaround section of the kernel config file, or are you suggesting
> something lower level / something else?

The options in the SMI workaround section of the kernel config file
actually disable some bits in some SMI registers, so they are already
low level. Anyway, yes, it is what I ask you to do.


-- 
					    Gilles.

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Stephen Bryant]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

On 26 July 2010 14:45, Gilles Chanteperdrix <
gilles.chanteperdrix@xenomai.org> wrote:

> Stephen Bryant wrote:
> > Do you mean enabling / disabling the individual options in the SMI
> > Workaround section of the kernel config file, or are you suggesting
> > something lower level / something else?
>
> The options in the SMI workaround section of the kernel config file
> actually disable some bits in some SMI registers, so they are already
> low level. Anyway, yes, it is what I ask you to do.
>
>
> --
>                                             Gilles.
>

I've been looking into this when I've had the time and the results so far
are that:
- Globally disabling SMI fails.
- Disabling all 8 'non-global' SMI options succeeds, but does not change my
latency results.

>From this I conclude that 'Globally disable SMI' really does try to disable
every SMI signal, which is different from just deselecting each of the 8
'Enable ... SMI' options presented, which I had not realised. Also, my
problem may still be SMI related, and I can only test this if I can convince
the BIOS to allow global SMI disabling.

So my next steps are to:
1 - Look for ways to change or circumvent the BIOS (if possible).
2 - Try a few other kernel settings (for examples the timers available) to
find out if my issue stems from something other than SMIs.
3 - Progressively change the Xenomai SMI disabling code to test SMIs not
covered by the 8 options presented.

I'll be checking the literature for help with these, but if anyone has any
pointers that would be much appreciated,

Steve

[-- Attachment #2: Type: text/html, Size: 2015 bytes --]

Re: [Xenomai-help] Unexpected preemption of highest priority thread

[Gilles Chanteperdrix]

Stephen Bryant wrote:
> On 26 July 2010 14:45, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org
> <mailto:gilles.chanteperdrix@xenomai.org>> wrote:
> 
>     Stephen Bryant wrote:
>     > Do you mean enabling / disabling the individual options in the SMI
>     > Workaround section of the kernel config file, or are you suggesting
>     > something lower level / something else?
> 
>     The options in the SMI workaround section of the kernel config file
>     actually disable some bits in some SMI registers, so they are already
>     low level. Anyway, yes, it is what I ask you to do.
> 
> 
>     --
>                                                Gilles.
> 
> 
> I've been looking into this when I've had the time and the results so
> far are that:
> - Globally disabling SMI fails.
> - Disabling all 8 'non-global' SMI options succeeds, but does not change
> my latency results.
> 
> From this I conclude that 'Globally disable SMI' really does try to
> disable every SMI signal, which is different from just deselecting each
> of the 8 'Enable ... SMI' options presented, which I had not realised.

The "Global disabling" bit, disable all SMI sources without even having
to list them. As such, as explained in the TROUBLESHOOTING file, it is
way more reliable than having to know all the SMI sources and disable
them one by one.

> Also, my problem may still be SMI related, and I can only test this if I
> can convince the BIOS to allow global SMI disabling.
> 
> So my next steps are to:
> 1 - Look for ways to change or circumvent the BIOS (if possible).

There is an "SMI lock" bit, which once enabled, can not be disabled. So,
if the BIOS you use uses this bit, there is nothing you can do, except
changing the BIOS. You can ask the board manufacturer to give you
another BIOS where this bit is not locked, so that you can test whether
SMI are the source of your problem.

> 2 - Try a few other kernel settings (for examples the timers available)
> to find out if my issue stems from something other than SMIs.
> 3 - Progressively change the Xenomai SMI disabling code to test SMIs not
> covered by the 8 options presented.
> 
> I'll be checking the literature for help with these, but if anyone has
> any pointers that would be much appreciated,

What you need is the datasheet of your chipset. I had looked at a
datasheet when doing this code, and found other sources, but did not
implement their disabling, you will find references in the
ksrc/arch/x86/smi.c file, and they are probably incomplete for your
particular chipset, which is why you should look for the datasheet of
your exact chipset.

-- 
					    Gilles.