[Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Alex Plits]

Hi,

We have a requirement to schedule N-RT linux kernel specific work  (e.g. workQ) on a dedicated core from an RTDM driver running
In Xenomai "context".
The current RTDM API does in fact provide an API to schedule a N-RT workQ but this does not meet our needs as we need the N-RT job to run on
A dedicated CPU. Current API implementation will run this special workQ on the CPU where the RT scheduling call occurred, so I have 2 queries we need help with -

1.       Is there an API/example/configuration that will achieve our needs?

2.       In case there isn't what would be the best approach to implement such a mechanism - I can think of solutions but it seems excessive:

a.       Run another Xenomai thread that has CPU affinity on the target cpu Core which will call a dedicated RTDM blocking call (e.g. IOCTL with semTake) and then the scheduling RTDM driver will release that semaphore - which will re-schedule the thread on the core we need and now it can call a N-RT IOCTL for example.

b.      Initialize a new WorkQ on I-Pipe init which will be binded on the target cpu Core only then add new RTDM api to schedule work on that WorkQ - although I'm not sure this will work as it will use the VIRQ  mechanism which will somehow need to change cores.

Thanks,
Alex P.

Re: [Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Gilles Chanteperdrix]

On Sun, Mar 20, 2016 at 07:15:42AM +0000, Alex Plits wrote:
> Hi,
> 
> We have a requirement to schedule N-RT linux kernel specific work  (e.g. workQ) on a dedicated core from an RTDM driver running
> In Xenomai "context".
> The current RTDM API does in fact provide an API to schedule a N-RT workQ but this does not meet our needs as we need the N-RT job to run on
> A dedicated CPU. Current API implementation will run this special workQ on the CPU where the RT scheduling call occurred, so I have 2 queries we need help with -
> 
> 1.       Is there an API/example/configuration that will achieve our needs?
> 
> 2.       In case there isn't what would be the best approach to implement such a mechanism - I can think of solutions but it seems excessive:
> 
> a.       Run another Xenomai thread that has CPU affinity on the target cpu Core which will call a dedicated RTDM blocking call (e.g. IOCTL with semTake) and then the scheduling RTDM driver will release that semaphore - which will re-schedule the thread on the core we need and now it can call a N-RT IOCTL for example.
> 
> b.      Initialize a new WorkQ on I-Pipe init which will be binded on the target cpu Core only then add new RTDM api to schedule work on that WorkQ - although I'm not sure this will work as it will use the VIRQ  mechanism which will somehow need to change cores.

If the Linux kernel provides a function to do this, we could
implement a function mirroring that function in the RTDM API. Looks
simple. Am I missing something ?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Alex Plits]

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Sunday, March 20, 2016 9:25 AM
> To: Alex Plits
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai] Suggested approach to schedule N-RT work in kernel
> on a specific CPU core
> 
> On Sun, Mar 20, 2016 at 07:15:42AM +0000, Alex Plits wrote:
> > Hi,
> >
> > We have a requirement to schedule N-RT linux kernel specific work
> > (e.g. workQ) on a dedicated core from an RTDM driver running In Xenomai
> "context".
> > The current RTDM API does in fact provide an API to schedule a N-RT
> > workQ but this does not meet our needs as we need the N-RT job to run
> > on A dedicated CPU. Current API implementation will run this special
> > workQ on the CPU where the RT scheduling call occurred, so I have 2
> > queries we need help with -
> >
> > 1.       Is there an API/example/configuration that will achieve our needs?
> >
> > 2.       In case there isn't what would be the best approach to implement
> such a mechanism - I can think of solutions but it seems excessive:
> >
> > a.       Run another Xenomai thread that has CPU affinity on the target cpu
> Core which will call a dedicated RTDM blocking call (e.g. IOCTL with semTake)
> and then the scheduling RTDM driver will release that semaphore - which will
> re-schedule the thread on the core we need and now it can call a N-RT IOCTL
> for example.
> >
> > b.      Initialize a new WorkQ on I-Pipe init which will be binded on the target
> cpu Core only then add new RTDM api to schedule work on that WorkQ -
> although I'm not sure this will work as it will use the VIRQ  mechanism which
> will somehow need to change cores.
> 
> If the Linux kernel provides a function to do this, we could implement a
> function mirroring that function in the RTDM API. Looks simple. Am I missing
> something ?
> 

[Alex Plits] 
The Linux provides an option to configure the workQ on specific CPU's on it's initialization stage of the WorkQ. I'm unsure
Of how/whether it actually schedules it's execution when schedule_work is called from a different CPU core that it was configured up-on. I'm guessing that if it were trivial the same mechanism would also be used in the current API for N-RT or xenomai printk mechanism (both use dedicated workQs which are different than Linux WorkQs awaken via VIRQs).


> --
> 					    Gilles.
> https://click-hack.org

Re: [Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Gilles Chanteperdrix]

On Sun, Mar 20, 2016 at 07:49:16AM +0000, Alex Plits wrote:
> 
> 
> > -----Original Message-----
> > From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> > If the Linux kernel provides a function to do this, we could implement a
> > function mirroring that function in the RTDM API. Looks simple. Am I missing
> > something ?
> > 
> 
> [Alex Plits] The Linux provides an option to configure the workQ
> on specific CPU's on it's initialization stage of the WorkQ. I'm
> unsure Of how/whether it actually schedules it's execution when
> schedule_work is called from a different CPU core that it was
> configured up-on.

I doubt very much this does not work. Have you tried it and observed
that it does not work?



> I'm guessing that if it were trivial the same
> mechanism would also be used in the current API for N-RT or
> xenomai printk mechanism (both use dedicated workQs which are
> different than Linux WorkQs awaken via VIRQs).

There is no xenomai printk mechanism. printk is implemented by the
I-pipe patch, a layer below Xenomai, so can not use an RTDM service.
Besides, the I-pipe patch printk implementation predates
ipipe_post_work_root.

The way RTDM work works is with the ipipe_post_work_root service,
so it does not use virqs, and since the schedule_work gets invoked
in a plain linux context, anything which works with schedule_work
(like triggering a work queue on a different cpu) should work with
the RTDM service.

So, again, have you tried running the RTDM service with a work queue
pinned on a different cpu and observed that it did not work?

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Alex Plits]

> -----Original Message-----
> From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> Sent: Sunday, March 20, 2016 10:00 AM
> To: Alex Plits
> Cc: xenomai@xenomai.org
> Subject: Re: [Xenomai] Suggested approach to schedule N-RT work in kernel
> on a specific CPU core
> 
> On Sun, Mar 20, 2016 at 07:49:16AM +0000, Alex Plits wrote:
> >
> >
> > > -----Original Message-----
> > > From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> > > If the Linux kernel provides a function to do this, we could
> > > implement a function mirroring that function in the RTDM API. Looks
> > > simple. Am I missing something ?
> > >
> >
> > [Alex Plits] The Linux provides an option to configure the workQ on
> > specific CPU's on it's initialization stage of the WorkQ. I'm unsure
> > Of how/whether it actually schedules it's execution when schedule_work
> > is called from a different CPU core that it was configured up-on.
> 
> I doubt very much this does not work. Have you tried it and observed that it
> does not work?
> 
> 
> 
> > I'm guessing that if it were trivial the same mechanism would also be
> > used in the current API for N-RT or xenomai printk mechanism (both use
> > dedicated workQs which are different than Linux WorkQs awaken via
> > VIRQs).
> 
> There is no xenomai printk mechanism. printk is implemented by the I-pipe
> patch, a layer below Xenomai, so can not use an RTDM service.
> Besides, the I-pipe patch printk implementation predates
> ipipe_post_work_root.
> 
> The way RTDM work works is with the ipipe_post_work_root service, so it
> does not use virqs, and since the schedule_work gets invoked in a plain linux
> context, anything which works with schedule_work (like triggering a work
> queue on a different cpu) should work with the RTDM service.
> 
> So, again, have you tried running the RTDM service with a work queue
> pinned on a different cpu and observed that it did not work?
> 
[Alex Plits] 
First off, thanks for such quick replies!
Secondly no I have not tried since I have not seen such an example.
So basically what you're saying - 
Run schedule_work in handler which is scheduled by rtdm_schedule_nrt_work and use some linux defined workQ with cpu "affinity" - right?

Thanks,
AlexP.
> --
> 					    Gilles.
> https://click-hack.org

Re: [Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Gilles Chanteperdrix]

On Sun, Mar 20, 2016 at 08:20:36AM +0000, Alex Plits wrote:
> 
> 
> > -----Original Message-----
> > From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> > Sent: Sunday, March 20, 2016 10:00 AM
> > To: Alex Plits
> > Cc: xenomai@xenomai.org
> > Subject: Re: [Xenomai] Suggested approach to schedule N-RT work in kernel
> > on a specific CPU core
> > 
> > On Sun, Mar 20, 2016 at 07:49:16AM +0000, Alex Plits wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> > > > If the Linux kernel provides a function to do this, we could
> > > > implement a function mirroring that function in the RTDM API. Looks
> > > > simple. Am I missing something ?
> > > >
> > >
> > > [Alex Plits] The Linux provides an option to configure the workQ on
> > > specific CPU's on it's initialization stage of the WorkQ. I'm unsure
> > > Of how/whether it actually schedules it's execution when schedule_work
> > > is called from a different CPU core that it was configured up-on.
> > 
> > I doubt very much this does not work. Have you tried it and observed that it
> > does not work?
> > 
> > 
> > 
> > > I'm guessing that if it were trivial the same mechanism would also be
> > > used in the current API for N-RT or xenomai printk mechanism (both use
> > > dedicated workQs which are different than Linux WorkQs awaken via
> > > VIRQs).
> > 
> > There is no xenomai printk mechanism. printk is implemented by the I-pipe
> > patch, a layer below Xenomai, so can not use an RTDM service.
> > Besides, the I-pipe patch printk implementation predates
> > ipipe_post_work_root.
> > 
> > The way RTDM work works is with the ipipe_post_work_root service, so it
> > does not use virqs, and since the schedule_work gets invoked in a plain linux
> > context, anything which works with schedule_work (like triggering a work
> > queue on a different cpu) should work with the RTDM service.
> > 
> > So, again, have you tried running the RTDM service with a work queue
> > pinned on a different cpu and observed that it did not work?
> > 
> [Alex Plits] 
> First off, thanks for such quick replies!
> Secondly no I have not tried since I have not seen such an example.
> So basically what you're saying - 
> Run schedule_work in handler which is scheduled by rtdm_schedule_nrt_work and use some linux defined workQ with cpu "affinity" - right?

I meant simply use rtdm_schedule_nrt_work with a work queue that has
been pinned on a given cpu. rtdm_schedule_nrt_work simply gets
schedule_work invoked in a plain linux context. So, there should not
be much difference between using rtdm_schedule_nrt_work and
schedule_work. The only difference being that rtdm_schedule_nrt_work
can be invoked in head domain.

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] Suggested approach to schedule N-RT work in kernel on a specific CPU core

[Alex Plits]

> -----Original Message-----
> From: Alex Plits
> Sent: Sunday, March 20, 2016 10:21 AM
> To: 'Gilles Chanteperdrix'
> Cc: xenomai@xenomai.org
> Subject: RE: [Xenomai] Suggested approach to schedule N-RT work in kernel
> on a specific CPU core
> 
> 
> 
> > -----Original Message-----
> > From: Gilles Chanteperdrix [mailto:gilles.chanteperdrix@xenomai.org]
> > Sent: Sunday, March 20, 2016 10:00 AM
> > To: Alex Plits
> > Cc: xenomai@xenomai.org
> > Subject: Re: [Xenomai] Suggested approach to schedule N-RT work in
> > kernel on a specific CPU core
> >
> > On Sun, Mar 20, 2016 at 07:49:16AM +0000, Alex Plits wrote:
> > >
> > >
> > > > -----Original Message-----
> > > > From: Gilles Chanteperdrix
> > > > [mailto:gilles.chanteperdrix@xenomai.org]
> > > > If the Linux kernel provides a function to do this, we could
> > > > implement a function mirroring that function in the RTDM API.
> > > > Looks simple. Am I missing something ?
> > > >
> > >
> > > [Alex Plits] The Linux provides an option to configure the workQ on
> > > specific CPU's on it's initialization stage of the WorkQ. I'm unsure
> > > Of how/whether it actually schedules it's execution when
> > > schedule_work is called from a different CPU core that it was configured
> up-on.
> >
> > I doubt very much this does not work. Have you tried it and observed
> > that it does not work?
> >
> >
> >
> > > I'm guessing that if it were trivial the same mechanism would also
> > > be used in the current API for N-RT or xenomai printk mechanism
> > > (both use dedicated workQs which are different than Linux WorkQs
> > > awaken via VIRQs).
> >
> > There is no xenomai printk mechanism. printk is implemented by the
> > I-pipe patch, a layer below Xenomai, so can not use an RTDM service.
> > Besides, the I-pipe patch printk implementation predates
> > ipipe_post_work_root.
> >
> > The way RTDM work works is with the ipipe_post_work_root service, so
> > it does not use virqs, and since the schedule_work gets invoked in a
> > plain linux context, anything which works with schedule_work (like
> > triggering a work queue on a different cpu) should work with the RTDM
> service.
> >
> > So, again, have you tried running the RTDM service with a work queue
> > pinned on a different cpu and observed that it did not work?
> >
> [Alex Plits]
> First off, thanks for such quick replies!
> Secondly no I have not tried since I have not seen such an example.
> So basically what you're saying -
> Run schedule_work in handler which is scheduled by
> rtdm_schedule_nrt_work and use some linux defined workQ with cpu
> "affinity" - right?
> 
> Thanks,
> AlexP.

[Alex Plits] 
Umm, No  - either my interpretation is wrong or the suggested method is not what we're looking for.
We cannot involve the core which schedules the N-RT work in N-RT jobs like linux kernel workQ re-scheduling.


> > --
> > 					    Gilles.
> > https://click-hack.org