[Xenomai-help] Running real time task on one of the cores of a Dual Core CPU

[Xenomai-help] Running real time task on one of the cores of a Dual Core CPU

[M. Koehrer]

Hi all,

I have some results and some questions on running Xenomai on a Dual Core CPU.
At my PC I have a Pentium D CPU (3.2 GHz), kernel 2.6.19.2, Xenomai 2.3 + some COW patches.
In the kernel I have enabled SMP.

The first interesting thing I noticed is that this kernel seems to run all standard processes
on core 0. The kernel does not automatically distribute processes on two cores.
I think I saw a different behaviour with an earlier kernel version (2.6.17 or even 2.4.33).
I do not have to use the isolcpus kernel parameter to force running all user space tasks on core 0.
This is exactly the thing I wanted to have. All "standard" linux tasks are running
on core 0.
Does anybody know if there was a change in the kernel?

Now I placed all real time tasks from Xenomai on core 1. This works fine.
My application calculates periodically a fairly complex simulation model. The interesting thing
is now, that the computation time needed for this model is much more constant in opposite to the
UP mode. On an UP system I had computation times between 111 and 170 microseconds.
With the SMP system I have now computation times between 111 and 115 microseconds.
Great! The worst case execution time is very close to the best case execution time!
I think this is caused by a more deterministic usage of the cache for the real time system.
In the UP system the cache cannot be reserved for the real time application, i.e. the real time
application has to re-read more often directly from memory.

Now my questions:
In a setup that uses core 0 for standard linux stuff and core 1 for Xenomai stuff the
Xenomai scheduler is running in SMP mode. However, I am not sure if this required in this setup.
As all real time code is bound to one core there does not have to be any multi processing
save synchronisation for the real time code.
My proposal is to support three different scheduling modes:
1 - UP as now
2 - SMP as now
3 - UP on a certain core: All real time code runs on one (runtime selectable) core even
if the linux kernel is fully SMP. This should increase the performance as inter CPU
protection is not required for the real time code.
Any comments on this idea are welcome!

I also use rtnet. Unfortunately I am not able to force rtnet (RTDM based) to run on CPU core 1
without patching the RTDM code.
A couple of months ago, I already asked for this issue.
Jan, can you tell me the latest news regarding this issue?

Thanks for all feedback on the topics

Regards

Mathias 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Running real time task on one of the cores of a Dual Core CPU

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 4222 bytes --]

M. Koehrer wrote:
> Hi all,
> 
> I have some results and some questions on running Xenomai on a Dual Core CPU.
> At my PC I have a Pentium D CPU (3.2 GHz), kernel 2.6.19.2, Xenomai 2.3 + some COW patches.
> In the kernel I have enabled SMP.
> 
> The first interesting thing I noticed is that this kernel seems to run all standard processes
> on core 0. The kernel does not automatically distribute processes on two cores.
> I think I saw a different behaviour with an earlier kernel version (2.6.17 or even 2.4.33).
> I do not have to use the isolcpus kernel parameter to force running all user space tasks on core 0.
> This is exactly the thing I wanted to have. All "standard" linux tasks are running
> on core 0.
> Does anybody know if there was a change in the kernel?
> 
> Now I placed all real time tasks from Xenomai on core 1. This works fine.
> My application calculates periodically a fairly complex simulation model. The interesting thing
> is now, that the computation time needed for this model is much more constant in opposite to the
> UP mode. On an UP system I had computation times between 111 and 170 microseconds.
> With the SMP system I have now computation times between 111 and 115 microseconds.
> Great! The worst case execution time is very close to the best case execution time!
> I think this is caused by a more deterministic usage of the cache for the real time system.
> In the UP system the cache cannot be reserved for the real time application, i.e. the real time
> application has to re-read more often directly from memory.

Yep, that makes sense.

> 
> Now my questions:
> In a setup that uses core 0 for standard linux stuff and core 1 for Xenomai stuff the
> Xenomai scheduler is running in SMP mode. However, I am not sure if this required in this setup.

With the current Xenomai code, this is required because you cannot
exclude that some application accidentally or intentionally forces some
threads or IRQs on CPU 0. So you would otherwise leave a fragile system
behind.

> As all real time code is bound to one core there does not have to be any multi processing
> save synchronisation for the real time code.
> My proposal is to support three different scheduling modes:
> 1 - UP as now
> 2 - SMP as now
> 3 - UP on a certain core: All real time code runs on one (runtime selectable) core even
> if the linux kernel is fully SMP. This should increase the performance as inter CPU
> protection is not required for the real time code.
> Any comments on this idea are welcome!

Even more generic ideas in this direction are haunting in my head for
quite some time, but no concrete design nor some implementation roadmap
exist yet.

The approach is basically to revamp SMP support so that you are able to
declare whatever core object (threads, IPC objects, devices, etc.)
CPU-local, thus avoiding the need to use SMP locking for it, but also
install safety belts that prevents you from accidentally accessing such
objects from remote CPUs. In other words: instead of one global
cross-CPU nucleus lock, per-CPU nklocks (which means IRQ disable) +
classic spinlocks for those objects that are still used across CPUs.

One may think about an intermediate step: create a Xenomai variant that
builds like UP over an SMP kernel and binds only to one CPU. But I
haven't looked at details and cannot asses right now if the effort would
be worth it or if we should better direct resources to the more advanced
(IMHO) design as I sketched above. Granted, the SMP revamping will be
not be trivial work (and will probably require some $funding), but UP
over SMP will waste CPUs for RT usage once we are no longer just talking
about *dual* cores...

> I also use rtnet. Unfortunately I am not able to force rtnet (RTDM based) to run on CPU core 1
> without patching the RTDM code.
> A couple of months ago, I already asked for this issue.
> Jan, can you tell me the latest news regarding this issue?

The global affinity mask is now part of Xenomai 2.3 (though I haven't
tested it again after it was merged). What part of RTnet precisely is
resisting to move to CPU 1, or what do you currently have to patch?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Re:: [Xenomai-help] Running real time task on one of the cores of a Dual

[M. Koehrer]

Hi Jan,

>> I also use rtnet. Unfortunately I am not able to force rtnet (RTDM based) to run on CPU core 1
>> without patching the RTDM code.
>> A couple of months ago, I already asked for this issue.
>> Jan, can you tell me the latest news regarding this issue?
>
>The global affinity mask is now part of Xenomai 2.3 (though I haven't
>tested it again after it was merged). 

Ah, I didn't know that! How can I set the affinity mask?
Perhaps this is the problem I have...

>What part of RTnet precisely is
>resisting to move to CPU 1, or what do you currently have to patch?

Currently I see the rtnet tasks "rtnet-stack" and "rtnet-rtpc" to be assigned to CPU 0.
I have patched xenomai/ksrc/skins/rtdm/drvlib.c in the call
to xnpod_start_thread(). I replace XNPOD_ALL_CPUS by cpumask_of_cpu(1).

Regards

Mathias

 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: : [Xenomai-help] Running real time task on one of the cores of a Dual

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

M. Koehrer wrote:
> Hi Jan,
> 
>>> I also use rtnet. Unfortunately I am not able to force rtnet (RTDM based) to run on CPU core 1
>>> without patching the RTDM code.
>>> A couple of months ago, I already asked for this issue.
>>> Jan, can you tell me the latest news regarding this issue?
>> The global affinity mask is now part of Xenomai 2.3 (though I haven't
>> tested it again after it was merged). 
> 
> Ah, I didn't know that! How can I set the affinity mask?

/proc/xenomai/affinity

> Perhaps this is the problem I have...
> 
>> What part of RTnet precisely is
>> resisting to move to CPU 1, or what do you currently have to patch?
> 
> Currently I see the rtnet tasks "rtnet-stack" and "rtnet-rtpc" to be assigned to CPU 0.
> I have patched xenomai/ksrc/skins/rtdm/drvlib.c in the call
> to xnpod_start_thread(). I replace XNPOD_ALL_CPUS by cpumask_of_cpu(1).

That should be obsolete when issuing

echo 2 > /proc/xenomai/affinity

before you load RTnet.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]