Re: [Xenomai] Problems with running Xenomai on Core i5

Re: [Xenomai] Problems with running Xenomai on Core i5

[Tobias Luksch]

Hello Gilles,

thank you for your help. I disabled ACPI in the kernel as another desperate try while hunting down my problem, now I will enable it again. 

I mainly opted for kernel version 3.2.21 as we need a 3.2 kernel because of some other software we have to use in the project and the adeos patch for 3.2.21 came with the xenomai 2.6.2.1 tarball. Could you suggest another 3.2 kernel version, or would you discourage me from using a 3.2.x kernel at all?

Regards,
Tobias

-----Ursprüngliche Nachricht-----
Von: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org] 
Gesendet: Mittwoch, 7. August 2013 20:01
An: Tobias Luksch
Cc: xenomai@xenomai.org
Betreff: Re: [Xenomai] Problems with running Xenomai on Core i5

On 08/07/2013 03:38 PM, Tobias Luksch wrote:
> Hello,
> 
> (Some key data to begin with: kernel v3.2.21, xenomai v2.6.2.1, 
> ipipe-core-3.2.21-x86-5.patch as within the 2.6.2.1 xenomai release, 
> all on Ubuntu 12.04 with gcc 4.6.3)
> 
> 
> currently I am trying to use Xenomai on an embedded PC with an Intel 
> Core i5 CPU (M520 @ 2.40GHz). While debugging processing times of our 
> application, I encountered strange non-determinism, i.e. parts of the 
> program doing only computations (w/o any mutexes etc.) sometimes took 
> more than twice the time to compute than usual.
> 
> So I wrote a simple test program (attached) based on 
> examples/native/trivial-periodic creating a 1000Hz loop. In each 
> cycle, a few
> 1000 sine functions are computed, with measuring the time it takes to 
> do so. I would expect a rather constant computation time, but I see 
> significant jitter (average duration ~0.09ms, jitter up to +-40%).
> 
> What's more, when I generate heavy CPU load on the linux side, the 
> computation time for the sine functions drops to about 0.06ms and 
> shows no more significant jitter at all. As soon as I stop producing 
> CPU load, computation time and jitter go up again.
> 
> Running the same test program on a laptop with an Intel Core2 Duo and 
> same kernel version, kernel settings, and xenomai version runs as 
> expected: constant computation time, very low jitter, no influence by 
> adding load on the linux side.
> 
> To me it looks like some power management feature of the CPU is 
> kicking in. I have disabled (hopefully) all relevant features in the 
> BIOS (like EISS, TurboBoost, Idle Modes, ...) and in the kernel (CPU 
> frequency scaling, ..., config file of me latest try is attached), 
> /proc/cpuinfo always shows the same frequency. Still I get the behavior described above.

The config file lacks CONFIG_ACPI, you should enable ACPI, only CONFIG_ACPI_PROCESSOR should be disabled.

Other than that, you should not use 3.2.21, this version is known to have issues with xenomai. Also, since you are using a Debian based system, you may try to add:
deb http://www.xenomai.org/debian squeeze main

to /etc/apt/sources.list, then do

apt-get update
apt-get install xenomai-keyring
apt-get update
apt-get install linux-image-3.5.7-xenomai-2.6.2.1

-- 
                                                                Gilles.

Re: [Xenomai] Problems with running Xenomai on Core i5

[Gilles Chanteperdrix]

On 08/09/2013 02:48 PM, Tobias Luksch wrote:
> Hello Gilles,
> 
> thank you for your help. I disabled ACPI in the kernel as another
> desperate try while hunting down my problem, now I will enable it
> again.
> 
> I mainly opted for kernel version 3.2.21 as we need a 3.2 kernel
> because of some other software we have to use in the project and the
> adeos patch for 3.2.21 came with the xenomai 2.6.2.1 tarball. Could
> you suggest another 3.2 kernel version, or would you discourage me
> from using a 3.2.x kernel at all?

If you want to use the 3.2 kernel, you have to backport some patches
from the 3.4 or 3.5 branch.

-- 
                                                                Gilles.

[Xenomai] Problems with running Xenomai on Core i5

[Tobias Luksch]

Hello,

(Some key data to begin with: kernel v3.2.21, xenomai v2.6.2.1, 
ipipe-core-3.2.21-x86-5.patch as within the 2.6.2.1 xenomai release, all on 
Ubuntu 12.04 with gcc 4.6.3)


currently I am trying to use Xenomai on an embedded PC with an Intel Core i5 
CPU (M520 @ 2.40GHz). While debugging processing times of our application, I 
encountered strange non-determinism, i.e. parts of the program doing only 
computations (w/o any mutexes etc.) sometimes took more than twice the time to 
compute than usual.

So I wrote a simple test program (attached) based on 
examples/native/trivial-periodic creating a 1000Hz loop. In each cycle, a few 
1000 sine functions are computed, with measuring the time it takes to do so. I 
would expect a rather constant computation time, but I see significant jitter 
(average duration ~0.09ms, jitter up to +-40%). 

What's more, when I generate heavy CPU load on the linux side, the computation 
time for the sine functions drops to about 0.06ms and shows no more significant 
jitter at all. As soon as I stop producing CPU load, computation time and 
jitter go up again.

Running the same test program on a laptop with an Intel Core2 Duo and same 
kernel version, kernel settings, and xenomai version runs as expected: constant 
computation time, very low jitter, no influence by adding load on the linux 
side.

To me it looks like some power management feature of the CPU is kicking in. I 
have disabled (hopefully) all relevant features in the BIOS (like EISS, 
TurboBoost, Idle Modes, ...) and in the kernel (CPU frequency scaling, ..., 
config file of me latest try is attached), /proc/cpuinfo always shows the same 
frequency. Still I get the behavior described above.

Any help and feedback would be appreciated.

Thanks,
Tobias

-------------- next part --------------
A non-text attachment was scrubbed...
Name: config-3.2.21-xenomai-2.6.2.1
Type: application/octet-stream
Size: 143553 bytes
Desc: config-3.2.21-xenomai-2.6.2.1
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20130807/f2a77ea9/attachment.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: trivial-periodic.c
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20130807/f2a77ea9/attachment.c>

Re: [Xenomai] Problems with running Xenomai on Core i5

[Matthew Fornero]

On Wed, Aug 7, 2013 at 9:38 AM, Tobias Luksch
<Tobias.Luksch@itk-engineering.de> wrote:
> To me it looks like some power management feature of the CPU is kicking in. I
> have disabled (hopefully) all relevant features in the BIOS (like EISS,
> TurboBoost, Idle Modes, ...) and in the kernel (CPU frequency scaling, ...,
> config file of me latest try is attached), /proc/cpuinfo always shows the same
> frequency. Still I get the behavior described above.

Hi Tobias,

What you're describing sounds exactly like C1E kicking in-- there
seems to be a latency penalty when cores exit C1E. This is usually not
exposed in the BIOS power settings (we had to add it to our boards).
Fortunately it can be controlled via MSR and takes effect immediately.
There's a bit in MSR 0x1FC (MSR_IA32_POWER_CTL) that needs to be
cleared. I *believe* it's bit 1, but can't remember. Note that I
believe this MSR is defined per-core, so you'll need to toggle the MSR
on all the cores that you want to disable C1E on (usually all of
them).

A google search for MSR_IA32_POWER_CTL and C1E should be able to find
some sort of documentation.

You should be able to tell if it's working using the built-in xenomai
latency test. Run it on an idle (and otherwise well configured) system
for 5-10 minutes and you should see a very noticable difference in
scheduler latency with C1E on and off.

-Matt

Re: [Xenomai] Problems with running Xenomai on Core i5

[Gilles Chanteperdrix]

On 08/07/2013 05:13 PM, Matthew Fornero wrote:
> On Wed, Aug 7, 2013 at 9:38 AM, Tobias Luksch
> <Tobias.Luksch@itk-engineering.de> wrote:
>> To me it looks like some power management feature of the CPU is kicking in. I
>> have disabled (hopefully) all relevant features in the BIOS (like EISS,
>> TurboBoost, Idle Modes, ...) and in the kernel (CPU frequency scaling, ...,
>> config file of me latest try is attached), /proc/cpuinfo always shows the same
>> frequency. Still I get the behavior described above.
> 
> Hi Tobias,
> 
> What you're describing sounds exactly like C1E kicking in-- there
> seems to be a latency penalty when cores exit C1E. This is usually not
> exposed in the BIOS power settings (we had to add it to our boards).
> Fortunately it can be controlled via MSR and takes effect immediately.
> There's a bit in MSR 0x1FC (MSR_IA32_POWER_CTL) that needs to be
> cleared. I *believe* it's bit 1, but can't remember. Note that I
> believe this MSR is defined per-core, so you'll need to toggle the MSR
> on all the cores that you want to disable C1E on (usually all of
> them).
> 
> A google search for MSR_IA32_POWER_CTL and C1E should be able to find
> some sort of documentation.
> 
> You should be able to tell if it's working using the built-in xenomai
> latency test. Run it on an idle (and otherwise well configured) system
> for 5-10 minutes and you should see a very noticable difference in
> scheduler latency with C1E on and off.

IIRC, the C1E can be enabled/disabled with Intel BIOS testing utilities:
http://biosbits.org/

-- 
                                                                Gilles.

Re: [Xenomai] Problems with running Xenomai on Core i5

[Matthew Fornero]

On Fri, Aug 9, 2013 at 8:41 AM, Tobias Luksch
<Tobias.Luksch@itk-engineering.de> wrote:
> Hello Matt,
>
> thank you very much for your suggestion. Clearing the second bit of the MSR_IA32_POWER_CTL register did in fact do the trick. The Xenomai latency test program as well as my little test app now behave much better. No more change in computation time, no more jitter when adding CPU load. Perfect!

Glad to hear it helped.

CC'ing the list in case anyone else encounters a similar issue.

-Matt

Re: [Xenomai] Problems with running Xenomai on Core i5

[Gilles Chanteperdrix]

On 08/07/2013 03:38 PM, Tobias Luksch wrote:
> Hello,
> 
> (Some key data to begin with: kernel v3.2.21, xenomai v2.6.2.1, 
> ipipe-core-3.2.21-x86-5.patch as within the 2.6.2.1 xenomai release, all on 
> Ubuntu 12.04 with gcc 4.6.3)
> 
> 
> currently I am trying to use Xenomai on an embedded PC with an Intel Core i5 
> CPU (M520 @ 2.40GHz). While debugging processing times of our application, I 
> encountered strange non-determinism, i.e. parts of the program doing only 
> computations (w/o any mutexes etc.) sometimes took more than twice the time to 
> compute than usual.
> 
> So I wrote a simple test program (attached) based on 
> examples/native/trivial-periodic creating a 1000Hz loop. In each cycle, a few 
> 1000 sine functions are computed, with measuring the time it takes to do so. I 
> would expect a rather constant computation time, but I see significant jitter 
> (average duration ~0.09ms, jitter up to +-40%). 
> 
> What's more, when I generate heavy CPU load on the linux side, the computation 
> time for the sine functions drops to about 0.06ms and shows no more significant 
> jitter at all. As soon as I stop producing CPU load, computation time and 
> jitter go up again.
> 
> Running the same test program on a laptop with an Intel Core2 Duo and same 
> kernel version, kernel settings, and xenomai version runs as expected: constant 
> computation time, very low jitter, no influence by adding load on the linux 
> side.
> 
> To me it looks like some power management feature of the CPU is kicking in. I 
> have disabled (hopefully) all relevant features in the BIOS (like EISS, 
> TurboBoost, Idle Modes, ...) and in the kernel (CPU frequency scaling, ..., 
> config file of me latest try is attached), /proc/cpuinfo always shows the same 
> frequency. Still I get the behavior described above.

The config file lacks CONFIG_ACPI, you should enable ACPI, only
CONFIG_ACPI_PROCESSOR should be disabled.

Other than that, you should not use 3.2.21, this version is known to
have issues with xenomai. Also, since you are using a Debian based
system, you may try to add:
deb http://www.xenomai.org/debian squeeze main

to /etc/apt/sources.list, then do

apt-get update
apt-get install xenomai-keyring
apt-get update
apt-get install linux-image-3.5.7-xenomai-2.6.2.1

-- 
                                                                Gilles.