[Xenomai] Dedicated Core?

[Xenomai] Dedicated Core?

[Charles Steinkuehler]

Is it possible to (easily) dedicate a CPU core to a single Xenomai
user-space thread?

Background:
I am running software based step/direction generation logic for motion
control as part of LinuxCNC.  On the x86 platform, latency numbers are
very good, and it's possible to run fast enough as a Xenomai user-space
thread.

ARM systems have significantly worse latency numbers, but on the
BeagleBone I have migrated the step/direction code to the PRU on-board
co-processors so everything works fine (actually better than on most x86
systems).

But most available ARM CPUs do not have the PRU co-processors found on
the BeagleBone.  However, there are a number of systems that have
multiple cores, and I'm particularly looking at the quad-core iMX6 parts.

If it would be possible to simply keep a thread running full-time on one
core, I could get really good step/direction performance without having
to worry about interrupt latency, but I don't know if there's an (easy)
way to do this using Xenomai, or if the 'dedicated' core would still be
subject to hardware interrupts and if my running thread might be
occasionally migrated to other physical cores.

-- 
Charles Steinkuehler
charles@steinkuehler.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140330/39738b01/attachment.sig>

Re: [Xenomai] Dedicated Core?

[Gilles Chanteperdrix]

On 03/30/2014 10:49 PM, Charles Steinkuehler wrote:
> Is it possible to (easily) dedicate a CPU core to a single Xenomai
> user-space thread?
> 
> Background:
> I am running software based step/direction generation logic for motion
> control as part of LinuxCNC.  On the x86 platform, latency numbers are
> very good, and it's possible to run fast enough as a Xenomai user-space
> thread.
> 
> ARM systems have significantly worse latency numbers, but on the
> BeagleBone I have migrated the step/direction code to the PRU on-board
> co-processors so everything works fine (actually better than on most x86
> systems).
> 
> But most available ARM CPUs do not have the PRU co-processors found on
> the BeagleBone.  However, there are a number of systems that have
> multiple cores, and I'm particularly looking at the quad-core iMX6 parts.

First, the imx6 is reported to have high latencies, and it is mostly due
to cache effects, so isolating the cpus will not help.

Second, I would say relying on low user-space latencies to do in
user-space things that the kernel should do is a bad design. You would
have better success implementing the step motor control low level
details as a kernel driver, and only move the high level stuff in
user-space.

> 
> If it would be possible to simply keep a thread running full-time on one
> core, I could get really good step/direction performance without having
> to worry about interrupt latency, but I don't know if there's an (easy)
> way to do this using Xenomai, or if the 'dedicated' core would still be
> subject to hardware interrupts and if my running thread might be
> occasionally migrated to other physical cores.
> 

The latency you get in user-spâce are not interrupt latency, but
user-space scheduling latencies, which are significantly higher. See the
difference between latency -t 0 and latency -t 2 on your plaforms. If
you want to only depend on interrupt latencies, you should implement a
driver, and get user-space to interact with the driver.

As you may have already read on this list, Xenomai never migrates task
automatically, and you can change the task affinity to define on which
core it should run. That said, you probably want the processor to remain
interrupt triggered, and the real-time task to suspend from time to time
to let linux do its duty (like handling Linux domain timer interrupts).


-- 
                                                                Gilles.

Re: [Xenomai] Dedicated Core?

[Charles Steinkuehler]

On 3/30/2014 4:41 PM, Gilles Chanteperdrix wrote:
> 
> First, the imx6 is reported to have high latencies, and it is mostly due
> to cache effects, so isolating the cpus will not help.
> 
> Second, I would say relying on low user-space latencies to do in
> user-space things that the kernel should do is a bad design. You would
> have better success implementing the step motor control low level
> details as a kernel driver, and only move the high level stuff in
> user-space.

I am not looking to migrate something that should be a kernel service,
instead I'm trying to replace having dedicated hardware.

>> If it would be possible to simply keep a thread running full-time on one
>> core, I could get really good step/direction performance without having
>> to worry about interrupt latency, but I don't know if there's an (easy)
>> way to do this using Xenomai, or if the 'dedicated' core would still be
>> subject to hardware interrupts and if my running thread might be
>> occasionally migrated to other physical cores.
> 
> The latency you get in user-spâce are not interrupt latency, but
> user-space scheduling latencies, which are significantly higher. See the
> difference between latency -t 0 and latency -t 2 on your plaforms. If
> you want to only depend on interrupt latencies, you should implement a
> driver, and get user-space to interact with the driver.
> 
> As you may have already read on this list, Xenomai never migrates task
> automatically, and you can change the task affinity to define on which
> core it should run. That said, you probably want the processor to remain
> interrupt triggered, and the real-time task to suspend from time to time
> to let linux do its duty (like handling Linux domain timer interrupts).

We already have hooks to set processor affinity.  What I am interested
in is if I dedicate an entire core to running a tight code loop, can I
avoid Xenomai and/or Linux "pulling the core out from under me" as a
result of various hardware interrupts or general task scheduling?

I can live with the code not running for the duration of the Xenomai
interrupt handler (that's happening now anyway), I just want to know if
it is (easy) to keep Xenomai and Linux from grabbing the core running my
tight loop after the interrupt event.  It's similar to having one core
running "bare metal" code, but I actually need Linux and Xenomai to
manage setting up the shared memory regions used for IPC prior to
launching my code.

-- 
Charles Steinkuehler
charles@steinkuehler.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20140330/b55d427c/attachment.sig>

Re: [Xenomai] Dedicated Core?

[Lennart Sorensen]

On Sun, Mar 30, 2014 at 05:04:03PM -0500, Charles Steinkuehler wrote:
> I am not looking to migrate something that should be a kernel service,
> instead I'm trying to replace having dedicated hardware.

Those PRUs sure can be handy.

> We already have hooks to set processor affinity.  What I am interested
> in is if I dedicate an entire core to running a tight code loop, can I
> avoid Xenomai and/or Linux "pulling the core out from under me" as a
> result of various hardware interrupts or general task scheduling?
> 
> I can live with the code not running for the duration of the Xenomai
> interrupt handler (that's happening now anyway), I just want to know if
> it is (easy) to keep Xenomai and Linux from grabbing the core running my
> tight loop after the interrupt event.  It's similar to having one core
> running "bare metal" code, but I actually need Linux and Xenomai to
> manage setting up the shared memory regions used for IPC prior to
> launching my code.

There are people running the i.MX6 in AMP mode with linux one one core
and a seperate RTOS on another core.

For example https://www.youtube.com/watch?v=1uzrX-YZBnQ mentions doing
exactly that.

Xenomai is good at letting you use all your resources for both real time
and non real time, but if you are willing to dedicate some hardware
entirely to real time, then there are other options for doing that.
I don't think it is at all what Xenomai is aimed at.

-- 
Len Sorensen

Re: [Xenomai] Dedicated Core?

[Gilles Chanteperdrix]

On 03/31/2014 12:04 AM, Charles Steinkuehler wrote:
> On 3/30/2014 4:41 PM, Gilles Chanteperdrix wrote:
>> 
>> First, the imx6 is reported to have high latencies, and it is
>> mostly due to cache effects, so isolating the cpus will not
>> help.
>> 
>> Second, I would say relying on low user-space latencies to do in 
>> user-space things that the kernel should do is a bad design. You
>> would have better success implementing the step motor control low
>> level details as a kernel driver, and only move the high level
>> stuff in user-space.
> 
> I am not looking to migrate something that should be a kernel
> service, instead I'm trying to replace having dedicated hardware.

This is better done in kernel-space than in user-space. At least, you
should give a try to a kernel driver before you decide to resort to
what you want to do.

Besides, notably the L2 cache issues on imx6, make it impossible for
the activity on one core to not influence the activity on other cores.

> 
>>> If it would be possible to simply keep a thread running
>>> full-time on one core, I could get really good step/direction
>>> performance without having to worry about interrupt latency,
>>> but I don't know if there's an (easy) way to do this using
>>> Xenomai, or if the 'dedicated' core would still be subject to
>>> hardware interrupts and if my running thread might be 
>>> occasionally migrated to other physical cores.
>> 
>> The latency you get in user-spâce are not interrupt latency, but 
>> user-space scheduling latencies, which are significantly higher.
>> See the difference between latency -t 0 and latency -t 2 on your
>> plaforms. If you want to only depend on interrupt latencies, you
>> should implement a driver, and get user-space to interact with
>> the driver.
>> 
>> As you may have already read on this list, Xenomai never migrates
>> task automatically, and you can change the task affinity to
>> define on which core it should run. That said, you probably want
>> the processor to remain interrupt triggered, and the real-time
>> task to suspend from time to time to let linux do its duty (like
>> handling Linux domain timer interrupts).
> 
> We already have hooks to set processor affinity.  What I am
> interested in is if I dedicate an entire core to running a tight
> code loop, can I avoid Xenomai and/or Linux "pulling the core out
> from under me" as a result of various hardware interrupts or
> general task scheduling?
> 
> I can live with the code not running for the duration of the
> Xenomai interrupt handler (that's happening now anyway), I just
> want to know if it is (easy) to keep Xenomai and Linux from
> grabbing the core running my tight loop after the interrupt event.
> It's similar to having one core running "bare metal" code, but I
> actually need Linux and Xenomai to manage setting up the shared
> memory regions used for IPC prior to launching my code.
> 

As I said, the answer is no, Linux needs to answer to some events on
all the cores that are not turned off, and Xenomai can not run on a
turned off core.



-- 
                                                                Gilles.

Re: [Xenomai] Dedicated Core?

[Gilles Chanteperdrix]

On 03/31/2014 12:04 AM, Charles Steinkuehler wrote:
> On 3/30/2014 4:41 PM, Gilles Chanteperdrix wrote:
>>
>> First, the imx6 is reported to have high latencies, and it is mostly due
>> to cache effects, so isolating the cpus will not help.
>>
>> Second, I would say relying on low user-space latencies to do in
>> user-space things that the kernel should do is a bad design. You would
>> have better success implementing the step motor control low level
>> details as a kernel driver, and only move the high level stuff in
>> user-space.
> 
> I am not looking to migrate something that should be a kernel service,
> instead I'm trying to replace having dedicated hardware.

Actually, this reminds me of a discussion we had some time ago, about 
generating PWM with omap3. You can find it here:
http://veter-project.blogspot.se/2011/09/real-time-enough-about-pwms-and-shaky.html
http://veter-project.blogspot.se/2012/04/precise-pwms-with-gpio-using-xenomai.html
http://comments.gmane.org/gmane.linux.real-time.xenomai.users/14026

In the end, Andrey succeeded to have precise control by using timers,
in kernel-space, instead of threads in user-space, and a little trick:
waking up early and spinning waiting for the target time.

-- 
                                                                Gilles.

Re: [Xenomai] Dedicated Core?

[Andy Pugh]

On 31 March 2014 12:29, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:

> In the end, Andrey succeeded to have precise control by using timers,
> in kernel-space, instead of threads in user-space, and a little trick:
> waking up early and spinning waiting for the target time.

LinuxCNC is rather tightly wedded to the thread model. The PWM /
Stepgen / Encoder modules already exist and are in use on many
machines.
It is possible to run LinuxCNC with Xenomai in both kernel mode and
user mode, but for reasons that I am not clear on the conclusion has
been that user mode is preferable.
(This may not be the same distinction as you are making).

-- 
atp

Re: [Xenomai] Dedicated Core?

[Gilles Chanteperdrix]

On 03/31/2014 02:19 PM, Andy Pugh wrote:
> On 31 March 2014 12:29, Gilles Chanteperdrix
> <gilles.chanteperdrix@xenomai.org> wrote:
>
>> In the end, Andrey succeeded to have precise control by using timers,
>> in kernel-space, instead of threads in user-space, and a little trick:
>> waking up early and spinning waiting for the target time.
>
> LinuxCNC is rather tightly wedded to the thread model. The PWM /
> Stepgen / Encoder modules already exist and are in use on many
> machines.
> It is possible to run LinuxCNC with Xenomai in both kernel mode and
> user mode, but for reasons that I am not clear on the conclusion has
> been that user mode is preferable.
> (This may not be the same distinction as you are making).
>
I do not advocate putting everything in kernel-space, like in the old 
days. What I mean is that the user/kernel interface, should act as an 
"hardware abstraction layer". So, for instance, for a PWM, the interface 
between driver and application should be ioctl(s) to pass period and 
duty cycle. The driver should take care of the low-level details of 
toggling the GPIO for the given period and duty cycle, and the 
application should only call the ioctls when it wants to change the 
period or duty cycle.

-- 
					    Gilles.