[PATCH] Allow changing CPU affinity of RT kernel tasks

[PATCH] Allow changing CPU affinity of RT kernel tasks

[Richard Weinberger]

Currently, calling sched_affinity() on an RT kernel task has no effect
on the Xenomai side, even though the shadow task reflects the desired
affinity. This discrepancy causes confusion because the failure is not
immediately visible.
A cryptic WARN_ON() triggers only when CONFIG_XENO_OPT_DEBUG_COBALT is
enabled.

This change addresses the issue by implementing an approach similar to
that used for userspace tasks: forcing a relaxation and hardening cycle
to apply the new affinity upon receiving SIGSHADOW.

A new helper, rtdm_task_test_sigshadow(), checks whether SIGSHADOW is
pending for the shadow task. If so, the signal is consumed, and a
relax/harden cycle is performed. This works because signal_pending()
only checks a flag and does not cause a domain switch. The new helper
is integrated into rtdm_task_should_stop(), so existing RT tasks utilize
it automatically.

RFC Discussion: Sending SIGSHADOW to a kernel task causes blocking
functions to return -EINTR. Currently, almost all RT kernel tasks treat
non-zero returns from functions like rtdm_event_wait() as fatal errors
and terminate. With this change, blocking functions that return -EINTR
require restarting (continuing the thread loop).

An alternative approach would be implementing restart logic in the rtdm_*
wrapper functions. If XNBREAK is set and it is a kernel task, the wrapper
could run rtdm_task_test_sigshadow() and restart the call internally
instead of returning -EINTR.

Signed-off-by: Richard Weinberger <richard@nod.at>
---
 include/cobalt/kernel/rtdm/driver.h | 17 +++++++++++++++++
 kernel/cobalt/thread.c              |  7 +++++--
 2 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/include/cobalt/kernel/rtdm/driver.h b/include/cobalt/kernel/rtdm/driver.h
index 604d54b2b..d06e1126c 100644
--- a/include/cobalt/kernel/rtdm/driver.h
+++ b/include/cobalt/kernel/rtdm/driver.h
@@ -34,6 +34,7 @@
 #include <linux/cdev.h>
 #include <linux/wait.h>
 #include <linux/notifier.h>
+#include <linux/sched/signal.h>
 #include <pipeline/lock.h>
 #include <xenomai/version.h>
 #include <cobalt/kernel/heap.h>
@@ -1043,8 +1044,24 @@ static inline void rtdm_task_destroy(rtdm_task_t *task)
 	xnthread_join(task, true);
 }
 
+static inline int __rtdm_task_should_stop(void)
+{
+	return xnthread_test_info(xnthread_current(), XNCANCELD);
+}
+
+static inline void rtdm_task_test_sigshadow(void)
+{
+	if (signal_pending(current)) {
+		xnthread_relax(0, 0);
+		XENO_WARN_ON(COBALT, kernel_dequeue_signal() != SIGSHADOW);
+		xnthread_harden();
+	}
+}
+
 static inline int rtdm_task_should_stop(void)
 {
+	rtdm_task_test_sigshadow();
+
 	return xnthread_test_info(xnthread_current(), XNCANCELD);
 }
 
diff --git a/kernel/cobalt/thread.c b/kernel/cobalt/thread.c
index 607f115a7..e9baf38e1 100644
--- a/kernel/cobalt/thread.c
+++ b/kernel/cobalt/thread.c
@@ -204,6 +204,8 @@ static int kthread_trampoline(void *arg)
 		return ret;
 	}
 
+	allow_signal(SIGSHADOW);
+
 	trace_cobalt_shadow_entry(thread);
 
 	thread->entry(thread->cookie);
@@ -2373,8 +2375,9 @@ void __xnthread_signal(struct xnthread *thread, int sig, int arg)
 	struct lostage_signal *sigwork;
 	int slot;
 
-	if (XENO_WARN_ON(COBALT, !xnthread_test_state(thread, XNUSER)))
-		return;
+	if (!xnthread_test_state(thread, XNUSER))
+		if (XENO_WARN_ON(COBALT, sig != SIGSHADOW))
+			return;
 
 	slot = get_slot_index_from_sig(sig, arg);
 	if (WARN_ON_ONCE(slot < 0))
-- 
2.51.0

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Jan Kiszka]

On 26.01.26 13:57, Richard Weinberger wrote:
> Currently, calling sched_affinity() on an RT kernel task has no effect
> on the Xenomai side, even though the shadow task reflects the desired
> affinity. This discrepancy causes confusion because the failure is not
> immediately visible.
> A cryptic WARN_ON() triggers only when CONFIG_XENO_OPT_DEBUG_COBALT is
> enabled.
> 
> This change addresses the issue by implementing an approach similar to
> that used for userspace tasks: forcing a relaxation and hardening cycle
> to apply the new affinity upon receiving SIGSHADOW.
> 
> A new helper, rtdm_task_test_sigshadow(), checks whether SIGSHADOW is
> pending for the shadow task. If so, the signal is consumed, and a
> relax/harden cycle is performed. This works because signal_pending()
> only checks a flag and does not cause a domain switch. The new helper
> is integrated into rtdm_task_should_stop(), so existing RT tasks utilize
> it automatically.
> 
> RFC Discussion: Sending SIGSHADOW to a kernel task causes blocking
> functions to return -EINTR. Currently, almost all RT kernel tasks treat
> non-zero returns from functions like rtdm_event_wait() as fatal errors
> and terminate. With this change, blocking functions that return -EINTR
> require restarting (continuing the thread loop).
> 
> An alternative approach would be implementing restart logic in the rtdm_*
> wrapper functions. If XNBREAK is set and it is a kernel task, the wrapper
> could run rtdm_task_test_sigshadow() and restart the call internally
> instead of returning -EINTR.

This is one issue with setting the affinity "from outside",
asynchronously. The other is that it will always imply a migration of
the kernel task, thus a phase of non-RT. If someone forgets about this
and sets the affinity at the wrong time, it can ruin RT workloads.

This tuning gap for RTDM task is rather old. I was already considering
to add an explicit affinity API to RTDM to set this from the driver.
That would allow to set the affinity early, right after task creation
and before the driver signals it is ready. Risk of such an interface is
that people start to hard-code the affinity into the drivers, rather
than making it configurable to userspace.

Jan

> 
> Signed-off-by: Richard Weinberger <richard@nod.at>
> ---
>  include/cobalt/kernel/rtdm/driver.h | 17 +++++++++++++++++
>  kernel/cobalt/thread.c              |  7 +++++--
>  2 files changed, 22 insertions(+), 2 deletions(-)
> 
> diff --git a/include/cobalt/kernel/rtdm/driver.h b/include/cobalt/kernel/rtdm/driver.h
> index 604d54b2b..d06e1126c 100644
> --- a/include/cobalt/kernel/rtdm/driver.h
> +++ b/include/cobalt/kernel/rtdm/driver.h
> @@ -34,6 +34,7 @@
>  #include <linux/cdev.h>
>  #include <linux/wait.h>
>  #include <linux/notifier.h>
> +#include <linux/sched/signal.h>
>  #include <pipeline/lock.h>
>  #include <xenomai/version.h>
>  #include <cobalt/kernel/heap.h>
> @@ -1043,8 +1044,24 @@ static inline void rtdm_task_destroy(rtdm_task_t *task)
>  	xnthread_join(task, true);
>  }
>  
> +static inline int __rtdm_task_should_stop(void)
> +{
> +	return xnthread_test_info(xnthread_current(), XNCANCELD);
> +}
> +
> +static inline void rtdm_task_test_sigshadow(void)
> +{
> +	if (signal_pending(current)) {
> +		xnthread_relax(0, 0);
> +		XENO_WARN_ON(COBALT, kernel_dequeue_signal() != SIGSHADOW);
> +		xnthread_harden();
> +	}
> +}
> +
>  static inline int rtdm_task_should_stop(void)
>  {
> +	rtdm_task_test_sigshadow();
> +
>  	return xnthread_test_info(xnthread_current(), XNCANCELD);
>  }
>  
> diff --git a/kernel/cobalt/thread.c b/kernel/cobalt/thread.c
> index 607f115a7..e9baf38e1 100644
> --- a/kernel/cobalt/thread.c
> +++ b/kernel/cobalt/thread.c
> @@ -204,6 +204,8 @@ static int kthread_trampoline(void *arg)
>  		return ret;
>  	}
>  
> +	allow_signal(SIGSHADOW);
> +
>  	trace_cobalt_shadow_entry(thread);
>  
>  	thread->entry(thread->cookie);
> @@ -2373,8 +2375,9 @@ void __xnthread_signal(struct xnthread *thread, int sig, int arg)
>  	struct lostage_signal *sigwork;
>  	int slot;
>  
> -	if (XENO_WARN_ON(COBALT, !xnthread_test_state(thread, XNUSER)))
> -		return;
> +	if (!xnthread_test_state(thread, XNUSER))
> +		if (XENO_WARN_ON(COBALT, sig != SIGSHADOW))
> +			return;
>  
>  	slot = get_slot_index_from_sig(sig, arg);
>  	if (WARN_ON_ONCE(slot < 0))

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Richard Weinberger]

On Montag, 26. Jänner 2026 15:28 'Jan Kiszka' via upstream wrote:
> This is one issue with setting the affinity "from outside",
> asynchronously. The other is that it will always imply a migration of
> the kernel task, thus a phase of non-RT. If someone forgets about this
> and sets the affinity at the wrong time, it can ruin RT workloads.

I assumed this is known and expected since changing the CPU affinity of userspace
tasks as the same implications.

> This tuning gap for RTDM task is rather old. I was already considering
> to add an explicit affinity API to RTDM to set this from the driver.
> That would allow to set the affinity early, right after task creation
> and before the driver signals it is ready. Risk of such an interface is
> that people start to hard-code the affinity into the drivers, rather
> than making it configurable to userspace.

Well, the goal of my patch is improving the situation to match userspace
semantics.
People expect that sched_setaffinity() kinda works.

Thanks,
//richard

-- 
​​​​​sigma star gmbh | Eduard-Bodem-Gasse 6, 6020 Innsbruck, AUT UID/VAT Nr:
ATU 66964118 | FN: 374287y

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Jan Kiszka]

On 26.01.26 16:11, Richard Weinberger wrote:
> On Montag, 26. Jänner 2026 15:28 'Jan Kiszka' via upstream wrote:
>> This is one issue with setting the affinity "from outside",
>> asynchronously. The other is that it will always imply a migration of
>> the kernel task, thus a phase of non-RT. If someone forgets about this
>> and sets the affinity at the wrong time, it can ruin RT workloads.
> 
> I assumed this is known and expected since changing the CPU affinity of userspace
> tasks as the same implications.
> 

Likely true. EINTR can be a topic with some calls there as well, and
some applications may just bail out (or because of trapping SIGDEBUG).

>> This tuning gap for RTDM task is rather old. I was already considering
>> to add an explicit affinity API to RTDM to set this from the driver.
>> That would allow to set the affinity early, right after task creation
>> and before the driver signals it is ready. Risk of such an interface is
>> that people start to hard-code the affinity into the drivers, rather
>> than making it configurable to userspace.
> 
> Well, the goal of my patch is improving the situation to match userspace
> semantics.
> People expect that sched_setaffinity() kinda works.

At least "taskset" (script-based tuning) - we do not have an API for
userspace to retrieve the PID of a RTDM task. Yeah, there is some truth
in this.

So we have the following issues still to consider when adding this
capability:

 - Driver supporting external tuning need to handle EINTR from
   relevant blocking calls - others will fail, and that is fine.

 - Affinity tuning by userspace should be done while the kernel thread
   is (still) "idle" - identifying this remains driver-specific. Maybe
   document this in rtdm_task_init?

 - Should we also provide a RTDM API that maps an RTDM task on
   set_cpus_allowed* or at least officially provide the task_struct for
   some rtdm_task_t? rtdm_task_t is opaque, xnthread_host_task is
   internal from RTDM-perspective.

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Richard Weinberger]

On Montag, 26. Jänner 2026 17:26 Jan Kiszka wrote:
> On 26.01.26 16:11, Richard Weinberger wrote:
> > On Montag, 26. Jänner 2026 15:28 'Jan Kiszka' via upstream wrote:
> >> This is one issue with setting the affinity "from outside",
> >> asynchronously. The other is that it will always imply a migration of
> >> the kernel task, thus a phase of non-RT. If someone forgets about this
> >> and sets the affinity at the wrong time, it can ruin RT workloads.
> > 
> > I assumed this is known and expected since changing the CPU affinity of userspace
> > tasks as the same implications.
> > 
> 
> Likely true. EINTR can be a topic with some calls there as well, and
> some applications may just bail out (or because of trapping SIGDEBUG).

Yes. I did more investigation. sched_setaffinity() causes always two domain
switches but cobalt syscalls get restarted automatically due to SA_RESTART
of the SIGSHADOW handler.

> >> This tuning gap for RTDM task is rather old. I was already considering
> >> to add an explicit affinity API to RTDM to set this from the driver.
> >> That would allow to set the affinity early, right after task creation
> >> and before the driver signals it is ready. Risk of such an interface is
> >> that people start to hard-code the affinity into the drivers, rather
> >> than making it configurable to userspace.
> > 
> > Well, the goal of my patch is improving the situation to match userspace
> > semantics.
> > People expect that sched_setaffinity() kinda works.
> 
> At least "taskset" (script-based tuning) - we do not have an API for
> userspace to retrieve the PID of a RTDM task. Yeah, there is some truth
> in this.

For scripts /proc/xenomai/sched/stat should work just fine.
In my tests I'm always using it together with taskset.

Do you have a new cobalt syscall in mind to query the RTDM pid?
 
> So we have the following issues still to consider when adding this
> capability:
> 
>  - Driver supporting external tuning need to handle EINTR from
>    relevant blocking calls - others will fail, and that is fine.

The in-tree drivers needs only minimal changes from what I can tell.

>  - Affinity tuning by userspace should be done while the kernel thread
>    is (still) "idle" - identifying this remains driver-specific. Maybe
>    document this in rtdm_task_init?

Yes, I'll happily document this.

>  - Should we also provide a RTDM API that maps an RTDM task on
>    set_cpus_allowed* or at least officially provide the task_struct for
>    some rtdm_task_t? rtdm_task_t is opaque, xnthread_host_task is
>    internal from RTDM-perspective.

What use cases do you have in mind?

Beside of that, I have also a patch in the pipline which adds a new parameter
to rtdm_task_init() to specify the target CPU instead of CPU_MASK_ALL.

Thanks,
//richard

-- 
​​​​​sigma star gmbh | Eduard-Bodem-Gasse 6, 6020 Innsbruck, AUT UID/VAT Nr:
ATU 66964118 | FN: 374287y

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Jan Kiszka]

On 26.01.26 23:05, Richard Weinberger wrote:
> On Montag, 26. Jänner 2026 17:26 Jan Kiszka wrote:
>> On 26.01.26 16:11, Richard Weinberger wrote:
>>> On Montag, 26. Jänner 2026 15:28 'Jan Kiszka' via upstream wrote:
>>>> This is one issue with setting the affinity "from outside",
>>>> asynchronously. The other is that it will always imply a migration of
>>>> the kernel task, thus a phase of non-RT. If someone forgets about this
>>>> and sets the affinity at the wrong time, it can ruin RT workloads.
>>>
>>> I assumed this is known and expected since changing the CPU affinity of userspace
>>> tasks as the same implications.
>>>
>>
>> Likely true. EINTR can be a topic with some calls there as well, and
>> some applications may just bail out (or because of trapping SIGDEBUG).
> 
> Yes. I did more investigation. sched_setaffinity() causes always two domain
> switches but cobalt syscalls get restarted automatically due to SA_RESTART
> of the SIGSHADOW handler.
> 
>>>> This tuning gap for RTDM task is rather old. I was already considering
>>>> to add an explicit affinity API to RTDM to set this from the driver.
>>>> That would allow to set the affinity early, right after task creation
>>>> and before the driver signals it is ready. Risk of such an interface is
>>>> that people start to hard-code the affinity into the drivers, rather
>>>> than making it configurable to userspace.
>>>
>>> Well, the goal of my patch is improving the situation to match userspace
>>> semantics.
>>> People expect that sched_setaffinity() kinda works.
>>
>> At least "taskset" (script-based tuning) - we do not have an API for
>> userspace to retrieve the PID of a RTDM task. Yeah, there is some truth
>> in this.
> 
> For scripts /proc/xenomai/sched/stat should work just fine.
> In my tests I'm always using it together with taskset.
> 
> Do you have a new cobalt syscall in mind to query the RTDM pid?
>  

RTDM tasks are no generic objects for userspace, thus such a syscall
cannot exist with the given APIs.

>> So we have the following issues still to consider when adding this
>> capability:
>>
>>  - Driver supporting external tuning need to handle EINTR from
>>    relevant blocking calls - others will fail, and that is fine.
> 
> The in-tree drivers needs only minimal changes from what I can tell.
> 
>>  - Affinity tuning by userspace should be done while the kernel thread
>>    is (still) "idle" - identifying this remains driver-specific. Maybe
>>    document this in rtdm_task_init?
> 
> Yes, I'll happily document this.
> 
>>  - Should we also provide a RTDM API that maps an RTDM task on
>>    set_cpus_allowed* or at least officially provide the task_struct for
>>    some rtdm_task_t? rtdm_task_t is opaque, xnthread_host_task is
>>    internal from RTDM-perspective.
> 
> What use cases do you have in mind?

If a driver creates a thread associated to specific IRQs and it already
uses rtdm_irq_set_affinity, based on userspace provided masks, then it
could self-adjust the thread affinity.

> 
> Beside of that, I have also a patch in the pipline which adds a new parameter
> to rtdm_task_init() to specify the target CPU instead of CPU_MASK_ALL.

...or that. I guess that should cover the above case (dynamic thread
creation on behalf of userspace with upfront known affinity).

Jan

-- 
Siemens AG, Foundational Technologies
Linux Expert Center

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Florian Bezdeka]

On Mon, 2026-01-26 at 13:57 +0100, Richard Weinberger wrote:

General note: xenomai@xenomai.org is the old address. Not sure if those
mails make it to all the inboxes. 

New: xenomai@lists.linux.dev

> [snip] 
>  
> +static inline int __rtdm_task_should_stop(void)
> +{
> +	return xnthread_test_info(xnthread_current(), XNCANCELD);
> +}

^ Is that used somewhere? Haven't looked into more details yet.

> +
> +static inline void rtdm_task_test_sigshadow(void)
> +{
> +	if (signal_pending(current)) {
> +		xnthread_relax(0, 0);
> +		XENO_WARN_ON(COBALT, kernel_dequeue_signal() != SIGSHADOW);
> +		xnthread_harden();
> +	}
> +}
> +
>  static inline int rtdm_task_should_stop(void)
>  {
> +	rtdm_task_test_sigshadow();
> +
>  	return xnthread_test_info(xnthread_current(), XNCANCELD);
>  }
>  
> diff --git a/kernel/cobalt/thread.c b/kernel/cobalt/thread.c
> index 607f115a7..e9baf38e1 100644
> --- a/kernel/cobalt/thread.c
> +++ b/kernel/cobalt/thread.c
> @@ -204,6 +204,8 @@ static int kthread_trampoline(void *arg)
>  		return ret;
>  	}
>  
> +	allow_signal(SIGSHADOW);
> +
>  	trace_cobalt_shadow_entry(thread);
>  
>  	thread->entry(thread->cookie);
> @@ -2373,8 +2375,9 @@ void __xnthread_signal(struct xnthread *thread, int sig, int arg)
>  	struct lostage_signal *sigwork;
>  	int slot;
>  
> -	if (XENO_WARN_ON(COBALT, !xnthread_test_state(thread, XNUSER)))
> -		return;
> +	if (!xnthread_test_state(thread, XNUSER))
> +		if (XENO_WARN_ON(COBALT, sig != SIGSHADOW))
> +			return;
>  
>  	slot = get_slot_index_from_sig(sig, arg);
>  	if (WARN_ON_ONCE(slot < 0))
> -- 
> 2.51.0

Re: [PATCH] Allow changing CPU affinity of RT kernel tasks

[Richard Weinberger]

On Montag, 26. Jänner 2026 16:50 'Florian Bezdeka' via upstream wrote:
> On Mon, 2026-01-26 at 13:57 +0100, Richard Weinberger wrote:
> 
> General note: xenomai@xenomai.org is the old address. Not sure if those
> mails make it to all the inboxes. 
> 
> New: xenomai@lists.linux.dev

rw@foxxylove:~/xenomai (next)> git grep xenomai@lists.linux.dev | wc -l
0

:-(

> 
> > [snip] 
> >  
> > +static inline int __rtdm_task_should_stop(void)
> > +{
> > +	return xnthread_test_info(xnthread_current(), XNCANCELD);
> > +}
> 
> ^ Is that used somewhere? Haven't looked into more details yet.

No. It's a helper for RTDM threads that know what they're doing
and want to call rtdm_task_test_sigshadow() at different places.

Thanks,
//richard


-- 
​​​​​sigma star gmbh | Eduard-Bodem-Gasse 6, 6020 Innsbruck, AUT UID/VAT Nr:
ATU 66964118 | FN: 374287y