[Xenomai-core] Isolated CPU, SMI problems an thoughts

[Xenomai-core] Isolated CPU, SMI problems an thoughts

[John Schipper]

Hello,
I'm new to this list but have in the past used RTAI in a single 
processor off the shelf solution. I'm looking to switch to native 
Xenomai api but have a general problem...
  The problem is SMI on new systems, and other latency killers that 
sometimes are not controllable by software always popping up when trying 
to migrate to a newer platform.  Can a dual core processor using 
isolcpus, preemp-rt and xenomai effectively future proof agains smi/chip 
set issues (specificly AMD or Intel dual core solutions) by isolating a 
cpu for exclusively xenomai/realtime use?

Some background information: 
  The realtime software I've developed in the past (with RTAI/adeos in 
user space) is a simple high speed serializer driver (mmap) to 
communicate with outside hardware and is responsible for syncronizing 
(with a semaphore/mutex) a linux process (soft realtime) at ~60Hz.  The 
realtime process is periodic at 1.2Khz or 2.4Khz and calculates/filters 
the data before sending commands back down the serializer interface and 
to the linux process for soft realtime network access.

  Generally we like to use "off the shelf" business PC's (Dell 170's and 
Dell 270's, HP 5000 with 1Gig memory) and find that 20-30us latency is 
achievable.  We use "off the shelf" hardware because availability 
(recieve within a week) and low cost are desired.   Whenever looking for 
an alernative solution either availablity or cost becomes a show 
stopper.  I'm open to suggestions and invite anyones thoughts on the 
subject.

Thanks for your time !

JKS

Re: [Xenomai-core] Isolated CPU, SMI problems an thoughts

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 2566 bytes --]

John Schipper wrote:
> Hello,
> I'm new to this list but have in the past used RTAI in a single
> processor off the shelf solution. I'm looking to switch to native
> Xenomai api but have a general problem...
>  The problem is SMI on new systems, and other latency killers that

What do you mean with "other" precisely?

> sometimes are not controllable by software always popping up when trying
> to migrate to a newer platform.  Can a dual core processor using
> isolcpus, preemp-rt and xenomai effectively future proof agains smi/chip
> set issues (specificly AMD or Intel dual core solutions) by isolating a
> cpu for exclusively xenomai/realtime use?

SMI can only be addressed with CPU isolation if you are able to redirect
the related event only to one CPU. Don't know if this is possible / default.

There are tricks to disable SMI on modern Intel chipsets. Xenomai
implements this, you can select the workaround during system
configuration. Don't this work for your particular systems? Then please
report details.

"Other", more subtle latency issues can only be addressed when the
mechanisms behind them are understood. Depends on the chipset
manufacturer's documentation. So, no general answer is possible.

> 
> Some background information:  The realtime software I've developed in
> the past (with RTAI/adeos in user space) is a simple high speed
> serializer driver (mmap) to communicate with outside hardware and is
> responsible for syncronizing (with a semaphore/mutex) a linux process
> (soft realtime) at ~60Hz.  The realtime process is periodic at 1.2Khz or
> 2.4Khz and calculates/filters the data before sending commands back down
> the serializer interface and to the linux process for soft realtime
> network access.
> 
>  Generally we like to use "off the shelf" business PC's (Dell 170's and
> Dell 270's, HP 5000 with 1Gig memory) and find that 20-30us latency is
> achievable.  We use "off the shelf" hardware because availability
> (recieve within a week) and low cost are desired.   Whenever looking for
> an alernative solution either availablity or cost becomes a show
> stopper.  I'm open to suggestions and invite anyones thoughts on the
> subject.

Using off the shelf standard systems is always risky. I've heard of a
larger German automation company ordering standard PC hardware for
industrial control purposes only when initial latency tests on a
specific charge were successful. Ok, they are ordering large enough
quantities, so they can dictate certain conditions...

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Isolated CPU, SMI problems an thoughts

[John Schipper]

Jan Kiszka wrote:

> John Schipper wrote:
>  
>
>> Hello,
>> I'm new to this list but have in the past used RTAI in a single
>> processor off the shelf solution. I'm looking to switch to native
>> Xenomai api but have a general problem...
>> The problem is SMI on new systems, and other latency killers that
>>   
>
>
> What do you mean with "other" precisely?
>  
>
I have seen on systems the need to disable USB Legacy/emulation? and 
only use a PS/2 keyboard.  The USB interface (tied to a usb keyboard and 
mouse) are standard on a Dell GX280 without a PS/2 interface.  I believe 
and have seen real-time having problems.  This is due to to SMM mode, I 
believe, being a latency killer (not sure if this is SMI related).

In regard to SMI I poked around the intel chips registers (sorry can't 
remember what the GX280 chipset was but I could get it if important!  ) 
and I found that their are lock bits that do not allow disabling global 
SMI or the watchdog capability.  This was 6 months ago at least so I 
have not tested the latest smi workaround module (in RTAI at least 
because I have not use xenomai yet but plan too :) )

>  
>
>> sometimes are not controllable by software always popping up when trying
>> to migrate to a newer platform.  Can a dual core processor using
>> isolcpus, preemp-rt and xenomai effectively future proof agains smi/chip
>> set issues (specificly AMD or Intel dual core solutions) by isolating a
>> cpu for exclusively xenomai/realtime use?
>>   
>
>
> SMI can only be addressed with CPU isolation if you are able to redirect
> the related event only to one CPU. Don't know if this is possible / 
> default.
>
>  
>
This redirection has occured to me also.   A couple months ago I 
inquired and did not get any confirmation either way from Dell about 
doing this and maybe they do not know and I need to ask 
Intel/Via/AMD?......  Its not clear (if even possible ) to me if this 
would be a bios feature or only controlled by using a special chipset 
kernel patch or module?

> There are tricks to disable SMI on modern Intel chipsets. Xenomai
> implements this, you can select the workaround during system
> configuration. Don't this work for your particular systems? Then please
> report details.
>
>  
>
 SMI tricks had not worked on a Dell GX280 system but I plan to start 
some more testing again with xenomai.  The fact that it did not work 
prompted me to dig into the SMI/SMM registers and is where I discovered 
that I could not get past the lock bit capability of the SMI/SMM chipset 
that was being set by the bios (not positive this is where it was 
happening ) and after some failed attempts I just continued with the 
Dell 170/270 solution.  If there is an advantage or maybe even a hope :) 
that an isolated cpu will help in regards to SMI/SMM then I plan to go 
ahead and order some systems to test otherwise I may continue to try to 
testing single core solutions.

> "Other", more subtle latency issues can only be addressed when the
> mechanisms behind them are understood. Depends on the chipset
> manufacturer's documentation. So, no general answer is possible.
>
>  
>
>> Some background information:  The realtime software I've developed in
>> the past (with RTAI/adeos in user space) is a simple high speed
>> serializer driver (mmap) to communicate with outside hardware and is
>> responsible for syncronizing (with a semaphore/mutex) a linux process
>> (soft realtime) at ~60Hz.  The realtime process is periodic at 1.2Khz or
>> 2.4Khz and calculates/filters the data before sending commands back down
>> the serializer interface and to the linux process for soft realtime
>> network access.
>>
>> Generally we like to use "off the shelf" business PC's (Dell 170's and
>> Dell 270's, HP 5000 with 1Gig memory) and find that 20-30us latency is
>> achievable.  We use "off the shelf" hardware because availability
>> (recieve within a week) and low cost are desired.   Whenever looking for
>> an alernative solution either availablity or cost becomes a show
>> stopper.  I'm open to suggestions and invite anyones thoughts on the
>> subject.
>>   
>
>
> Using off the shelf standard systems is always risky. I've heard of a
> larger German automation company ordering standard PC hardware for
> industrial control purposes only when initial latency tests on a
> specific charge were successful. Ok, they are ordering large enough
> quantities, so they can dictate certain conditions...
>
>  
>

 Testing the system is done to verify latency and expected real-time 
isolation from the normal linux tasks. Its done on a standard system and 
once verified used until the system is no longer available which seems 
to be happening at an increasing rate.  PCI express may be a reason, 
maybe money.. but systems don't last much longer then a year and a 
half.  Either way the motherboards themselves don't stay around to 
long.  Our quantity is not there for the needed leverage with providers 
:( .  The system is used for an industrial purpose which is NOT for a 
life saving/risking situation.  Its used for simulation or simulator 
purposes.

> Jan
>
>  
>
John

Re: [Xenomai-core] Isolated CPU, SMI problems an thoughts

[Max Krasnyansky]

John Schipper wrote:
> Jan Kiszka wrote:
> 
>> John Schipper wrote:
>>  
>>
>>> Hello,
>>> I'm new to this list but have in the past used RTAI in a single
>>> processor off the shelf solution. I'm looking to switch to native
>>> Xenomai api but have a general problem...
>>> The problem is SMI on new systems, and other latency killers that
>>>   
>> What do you mean with "other" precisely?
>>  
>>
> I have seen on systems the need to disable USB Legacy/emulation? and 
> only use a PS/2 keyboard.  The USB interface (tied to a usb keyboard and 
> mouse) are standard on a Dell GX280 without a PS/2 interface.  I believe 
> and have seen real-time having problems.  This is due to to SMM mode, I 
> believe, being a latency killer (not sure if this is SMI related).
> 
> In regard to SMI I poked around the intel chips registers (sorry can't 
> remember what the GX280 chipset was but I could get it if important!  ) 
> and I found that their are lock bits that do not allow disabling global 
> SMI or the watchdog capability.  This was 6 months ago at least so I 
> have not tested the latest smi workaround module (in RTAI at least 
> because I have not use xenomai yet but plan too :) )

I'd say if you do the following:
	- Disable any kind of power management (BIOS, etc)
	- Enabled ACPI but _disable_ ACPI processor module
	- Enable and load USB hci drivers
	- Use "idle=poll"

You should see pretty good worst case latencies on your Dell machines.
Last thing (although not very friendly to power consumption) is pretty important
because what I found out (the hard way) is that ACPI based and MONITOR/MWAIT based
idle loops trigger SMM and introduce horrible latencies with some Intel chipsets
(I've seen it on IBM Z-Pro and HP Intel based lines).

>>> sometimes are not controllable by software always popping up when trying
>>> to migrate to a newer platform.  Can a dual core processor using
>>> isolcpus, preemp-rt and xenomai effectively future proof agains smi/chip
>>> set issues (specificly AMD or Intel dual core solutions) by isolating a
>>> cpu for exclusively xenomai/realtime use?
>>>   
>> SMI can only be addressed with CPU isolation if you are able to redirect
>> the related event only to one CPU. Don't know if this is possible / 
>> default.
Yeah I don't think it's possible. I chatted with HP BIOS folks a bit and they
didn't see any good solutions for that.

>> Using off the shelf standard systems is always risky. I've heard of a
>> larger German automation company ordering standard PC hardware for
>> industrial control purposes only when initial latency tests on a
>> specific charge were successful. Ok, they are ordering large enough
>> quantities, so they can dictate certain conditions...
> 
> Testing the system is done to verify latency and expected real-time 
> isolation from the normal linux tasks. Its done on a standard system and 
> once verified used until the system is no longer available which seems 
> to be happening at an increasing rate.  PCI express may be a reason, 
> maybe money.. but systems don't last much longer then a year and a 
> half.  Either way the motherboards themselves don't stay around to 
> long.  Our quantity is not there for the needed leverage with providers 
> :( .  The system is used for an industrial purpose which is NOT for a 
> life saving/risking situation.  Its used for simulation or simulator 
> purposes.

btw For what it's worth I can recommend HP9300 boxes. I worked with their
BIOS folks to resolve interrupt routing issues and stuff and now it's my
dream machine for RT stuff that requires loads of CPU compute power. The
box is has nice isolation features for example it has 1 PCIe slot that is
dedicated to CPU1 (ie on a separate bus). It's NUMA and so it has dedicated
memory banks per CPU. No shared interrupts, etc. You won't be able to run
Xenomai on it in native 64bit mode but 32bit mode should work.

Max