[Xenomai-help] Limiting Xenomai tasks to one certain core in a dual core system

[Xenomai-help] Limiting Xenomai tasks to one certain core in a dual core system

[M. Koehrer]

Hi all,

I want to run Xenomai and rtnet (using RTDM) on a dual core CPU.
Using the kernel parameter isolcpus I can force the standard Linux tasks
to run on one of the cores.
The other core should be used exclusively by all real time tasks (including rtnet).
I can - of course - pass a parameter on each rt_task_create() call.
However how can I force rtnet (using RTDM) to run on this CPU core?
Is there a possibility to mask the CPUs to be used globally?
This could also avoid to pass the CPU number with each rt_task_create() call.

Thanks for any feedback on this.

Regards

Mathias

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

[Xenomai-core] Re: [Xenomai-help] Limiting Xenomai tasks to one certain core in a dual core system

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1301 bytes --]

M. Koehrer wrote:
> Hi all,
> 
> I want to run Xenomai and rtnet (using RTDM) on a dual core CPU.
> Using the kernel parameter isolcpus I can force the standard Linux tasks
> to run on one of the cores.
> The other core should be used exclusively by all real time tasks (including rtnet).
> I can - of course - pass a parameter on each rt_task_create() call.
> However how can I force rtnet (using RTDM) to run on this CPU core?

You hit a weak point, see the Task Market in the wiki: "CPU affinity for
RTDM threads." We both need an API for RTDM and its application on
RTnet. Not complex work, just work...

> Is there a possibility to mask the CPUs to be used globally?
> This could also avoid to pass the CPU number with each rt_task_create() call.

Well, that would actually be an additional way. Yeah, maybe some global
CPU mask that controls the affinity on future thread creation. We could
export it via /proc so that you can set it right before starting a
specific group of applications and drivers.

On the long term, a control interface to the Xenomai scheduler would be
good so that things like policy, priority, time-slice, or CPU affinity
could be manipulated with some command line tool or via /proc after
thread creation.

Further feedback welcome!

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [Xenomai-core] Re: [Xenomai-help] Limiting Xenomai tasks to one certain core in a dual core system

[Jan Kiszka]

[-- Attachment #1.1: Type: text/plain, Size: 661 bytes --]

Jan Kiszka wrote:
> M. Koehrer wrote:
>> Is there a possibility to mask the CPUs to be used globally?
>> This could also avoid to pass the CPU number with each rt_task_create() call.
> 
> Well, that would actually be an additional way. Yeah, maybe some global
> CPU mask that controls the affinity on future thread creation. We could
> export it via /proc so that you can set it right before starting a
> specific group of applications and drivers.

Some food for thoughts:

Here is an experimental implementation of such a global mask. Seems to
work, at least I was able to prevent any task creation on a UP box by
setting the mask to 0.

Jan

[-- Attachment #1.2: global-cpu-affinity.patch --]
[-- Type: text/plain, Size: 3717 bytes --]

---
 include/nucleus/pod.h |    2 +
 ksrc/nucleus/module.c |   57 ++++++++++++++++++++++++++++++++++++++++++++++++++
 ksrc/nucleus/pod.c    |    3 ++
 3 files changed, 62 insertions(+)

Index: xenomai/include/nucleus/pod.h
===================================================================
--- xenomai.orig/include/nucleus/pod.h
+++ xenomai/include/nucleus/pod.h
@@ -253,6 +253,8 @@ extern u_long nktickdef;
 
 extern char *nkmsgbuf;
 
+extern xnarch_cpumask_t nkaffinity;
+
 #define xnprintf(fmt,args...)  xnarch_printf(fmt , ##args)
 #define xnloginfo(fmt,args...) xnarch_loginfo(fmt , ##args)
 #define xnlogwarn(fmt,args...) xnarch_logwarn(fmt , ##args)
Index: xenomai/ksrc/nucleus/module.c
===================================================================
--- xenomai.orig/ksrc/nucleus/module.c
+++ xenomai/ksrc/nucleus/module.c
@@ -688,6 +688,59 @@ static int heap_read_proc(char *page,
 	return len;
 }
 
+static int affinity_read_proc(char *page,
+			      char **start,
+			      off_t off, int count, int *eof, void *data)
+{
+	unsigned long val = 0;
+	int len, cpu;
+
+	for (cpu = 0; cpu < sizeof(val) * 8; cpu++)
+		if (xnarch_cpu_isset(cpu, nkaffinity))
+			val |= (1 << cpu);
+
+	len = sprintf(page, "0x%08lx\n", val);
+	len -= off;
+	if (len <= off + count)
+		*eof = 1;
+	*start = page + off;
+	if (len > count)
+		len = count;
+	if (len < 0)
+		len = 0;
+
+	return len;
+}
+
+static int affinity_write_proc(struct file *file,
+			       const char __user * buffer,
+			       unsigned long count, void *data)
+{
+	char *end, buf[16];
+	unsigned long val;
+	xnarch_cpumask_t new_affinity;
+	int n, cpu;
+
+	n = count > sizeof(buf) - 1 ? sizeof(buf) - 1 : count;
+
+	if (copy_from_user(buf, buffer, n))
+		return -EFAULT;
+
+	buf[n] = '\0';
+	val = simple_strtol(buf, &end, 0);
+
+	if (*end != '\0' && !isspace(*end))
+		return -EINVAL;
+
+	xnarch_cpus_clear(new_affinity);
+	for (cpu = 0; cpu < sizeof(val) * 8; cpu++, val >>= 1)
+		if (val & 1)
+			xnarch_cpu_set(cpu, new_affinity);
+	nkaffinity = new_affinity;
+
+	return count;
+}
+
 static struct proc_dir_entry *add_proc_leaf(const char *name,
 					    read_proc_t rdproc,
 					    write_proc_t wrproc,
@@ -760,6 +813,9 @@ void xnpod_init_proc(void)
 
 	add_proc_leaf("heap", &heap_read_proc, NULL, NULL, rthal_proc_root);
 
+	add_proc_leaf("affinity", &affinity_read_proc, &affinity_write_proc,
+		      NULL, rthal_proc_root);
+
 #ifdef CONFIG_XENO_OPT_PERVASIVE
 	iface_proc_root =
 	    create_proc_entry("interfaces", S_IFDIR, rthal_proc_root);
@@ -778,6 +834,7 @@ void xnpod_delete_proc(void)
 
 	remove_proc_entry("interfaces", rthal_proc_root);
 #endif /* CONFIG_XENO_OPT_PERVASIVE */
+	remove_proc_entry("affinity", rthal_proc_root);
 	remove_proc_entry("heap", rthal_proc_root);
 	remove_proc_entry("irq", rthal_proc_root);
 	remove_proc_entry("timer", rthal_proc_root);
Index: xenomai/ksrc/nucleus/pod.c
===================================================================
--- xenomai.orig/ksrc/nucleus/pod.c
+++ xenomai/ksrc/nucleus/pod.c
@@ -70,6 +70,8 @@ MODULE_PARM_DESC(tick_arg, "Fixed clock 
 
 char *nkmsgbuf = NULL;
 
+xnarch_cpumask_t nkaffinity = XNPOD_ALL_CPUS;
+
 const char *xnpod_fatal_helper(const char *format, ...)
 {
 	const unsigned nr_cpus = xnarch_num_online_cpus();
@@ -915,6 +917,7 @@ int xnpod_start_thread(xnthread_t *threa
 	xnlock_get_irqsave(&nklock, s);
 
 	thread->affinity = xnarch_cpu_online_map;
+	xnarch_cpus_and(thread->affinity, nkaffinity, thread->affinity);
 	xnarch_cpus_and(thread->affinity, affinity, thread->affinity);
 
 	if (xnarch_cpus_empty(thread->affinity)) {

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Limiting Xenomai tasks to one certain core in a dual core system

[Jan Kiszka]

[-- Attachment #1.1: Type: text/plain, Size: 2156 bytes --]

Jan Kiszka wrote:
> Jan Kiszka wrote:
>> M. Koehrer wrote:
>>> Is there a possibility to mask the CPUs to be used globally?
>>> This could also avoid to pass the CPU number with each rt_task_create() call.
>> Well, that would actually be an additional way. Yeah, maybe some global
>> CPU mask that controls the affinity on future thread creation. We could
>> export it via /proc so that you can set it right before starting a
>> specific group of applications and drivers.
> 
> Some food for thoughts:
> 
> Here is an experimental implementation of such a global mask. Seems to
> work, at least I was able to prevent any task creation on a UP box by
> setting the mask to 0.

Even more brain exercises:

This is version 2 of my patch, consequently applying the global mask
idea also on IRQ affinity. I think this simple mechanism can solve quite
a few basic SMP setup issue by forcing all user threads, driver threads,
and IRQs that are created/attached during a certain period to a specific
CPU set.

That's the theory. In practice, things look not that clear ATM. I
noticed some oddity on a 4-way qemu box running the latency test and
hacked the attached enhancement for /proc/xenomai/sched. This is what I
get on that box:

> root@domain.hid :/root# cat /proc/xenomai/sched 
> CPU  AFFINITY PID    PRI      PERIOD   TIMEOUT    STAT       NAME
>   0  00000001 0       -1      0        0          R          ROOT/0
>   1  00000002 0       -1      0        0          R          ROOT/1
>   2  00000004 0       -1      0        0          R          ROOT/2
>   3  00000008 0       -1      0        0          R          ROOT/3
>   0  0000000c 917      0      0        0          W          display-916
>   2  0000000c 918     99      5000000  1          D          sampling-916
> root@domain.hid :/root# cat /proc/xenomai/affinity 
> 0x0000000c

Philippe, anyone else: This scheduling of display-916 is not like as it
should be, is it?

Jan


PS: /proc output of IRQ affinities would be nice as well, but we are
lacking the require abstraction by ipipe ATM, something to read out the
currently set affinity.

[-- Attachment #1.2: global-cpu-affinity-v2.patch --]
[-- Type: text/plain, Size: 4204 bytes --]

---
 include/nucleus/pod.h |    2 +
 ksrc/nucleus/intr.c   |    3 ++
 ksrc/nucleus/module.c |   57 ++++++++++++++++++++++++++++++++++++++++++++++++++
 ksrc/nucleus/pod.c    |    3 ++
 4 files changed, 65 insertions(+)

Index: xenomai/include/nucleus/pod.h
===================================================================
--- xenomai.orig/include/nucleus/pod.h
+++ xenomai/include/nucleus/pod.h
@@ -253,6 +253,8 @@ extern u_long nktickdef;
 
 extern char *nkmsgbuf;
 
+extern xnarch_cpumask_t nkaffinity;
+
 #define xnprintf(fmt,args...)  xnarch_printf(fmt , ##args)
 #define xnloginfo(fmt,args...) xnarch_loginfo(fmt , ##args)
 #define xnlogwarn(fmt,args...) xnarch_logwarn(fmt , ##args)
Index: xenomai/ksrc/nucleus/intr.c
===================================================================
--- xenomai.orig/ksrc/nucleus/intr.c
+++ xenomai/ksrc/nucleus/intr.c
@@ -626,6 +626,9 @@ int xnintr_attach(xnintr_t *intr, void *
 
 	xnlock_get_irqsave(&intrlock, s);
 
+#ifdef CONFIG_SMP
+	xnarch_set_irq_affinity(intr->irq, nkaffinity);
+#endif /* CONFIG_SMP */
 	err = xnintr_irq_attach(intr);
 
 	xnlock_put_irqrestore(&intrlock, s);
Index: xenomai/ksrc/nucleus/module.c
===================================================================
--- xenomai.orig/ksrc/nucleus/module.c
+++ xenomai/ksrc/nucleus/module.c
@@ -688,6 +688,59 @@ static int heap_read_proc(char *page,
 	return len;
 }
 
+static int affinity_read_proc(char *page,
+			      char **start,
+			      off_t off, int count, int *eof, void *data)
+{
+	unsigned long val = 0;
+	int len, cpu;
+
+	for (cpu = 0; cpu < sizeof(val) * 8; cpu++)
+		if (xnarch_cpu_isset(cpu, nkaffinity))
+			val |= (1 << cpu);
+
+	len = sprintf(page, "%08lx\n", val);
+	len -= off;
+	if (len <= off + count)
+		*eof = 1;
+	*start = page + off;
+	if (len > count)
+		len = count;
+	if (len < 0)
+		len = 0;
+
+	return len;
+}
+
+static int affinity_write_proc(struct file *file,
+			       const char __user * buffer,
+			       unsigned long count, void *data)
+{
+	char *end, buf[16];
+	unsigned long val;
+	xnarch_cpumask_t new_affinity;
+	int n, cpu;
+
+	n = count > sizeof(buf) - 1 ? sizeof(buf) - 1 : count;
+
+	if (copy_from_user(buf, buffer, n))
+		return -EFAULT;
+
+	buf[n] = '\0';
+	val = simple_strtol(buf, &end, 0);
+
+	if (*end != '\0' && !isspace(*end))
+		return -EINVAL;
+
+	xnarch_cpus_clear(new_affinity);
+	for (cpu = 0; cpu < sizeof(val) * 8; cpu++, val >>= 1)
+		if (val & 1)
+			xnarch_cpu_set(cpu, new_affinity);
+	nkaffinity = new_affinity;
+
+	return count;
+}
+
 static struct proc_dir_entry *add_proc_leaf(const char *name,
 					    read_proc_t rdproc,
 					    write_proc_t wrproc,
@@ -760,6 +813,9 @@ void xnpod_init_proc(void)
 
 	add_proc_leaf("heap", &heap_read_proc, NULL, NULL, rthal_proc_root);
 
+	add_proc_leaf("affinity", &affinity_read_proc, &affinity_write_proc,
+		      NULL, rthal_proc_root);
+
 #ifdef CONFIG_XENO_OPT_PERVASIVE
 	iface_proc_root =
 	    create_proc_entry("interfaces", S_IFDIR, rthal_proc_root);
@@ -778,6 +834,7 @@ void xnpod_delete_proc(void)
 
 	remove_proc_entry("interfaces", rthal_proc_root);
 #endif /* CONFIG_XENO_OPT_PERVASIVE */
+	remove_proc_entry("affinity", rthal_proc_root);
 	remove_proc_entry("heap", rthal_proc_root);
 	remove_proc_entry("irq", rthal_proc_root);
 	remove_proc_entry("timer", rthal_proc_root);
Index: xenomai/ksrc/nucleus/pod.c
===================================================================
--- xenomai.orig/ksrc/nucleus/pod.c
+++ xenomai/ksrc/nucleus/pod.c
@@ -70,6 +70,8 @@ MODULE_PARM_DESC(tick_arg, "Fixed clock 
 
 char *nkmsgbuf = NULL;
 
+xnarch_cpumask_t nkaffinity = XNPOD_ALL_CPUS;
+
 const char *xnpod_fatal_helper(const char *format, ...)
 {
 	const unsigned nr_cpus = xnarch_num_online_cpus();
@@ -915,6 +917,7 @@ int xnpod_start_thread(xnthread_t *threa
 	xnlock_get_irqsave(&nklock, s);
 
 	thread->affinity = xnarch_cpu_online_map;
+	xnarch_cpus_and(thread->affinity, nkaffinity, thread->affinity);
 	xnarch_cpus_and(thread->affinity, affinity, thread->affinity);
 
 	if (xnarch_cpus_empty(thread->affinity)) {

[-- Attachment #1.3: print-thread-affinity.patch --]
[-- Type: text/plain, Size: 1631 bytes --]

---
 ksrc/nucleus/module.c |   10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

Index: xenomai/ksrc/nucleus/module.c
===================================================================
--- xenomai.orig/ksrc/nucleus/module.c
+++ xenomai/ksrc/nucleus/module.c
@@ -103,6 +103,7 @@ struct sched_seq_iterator {
 		xnticks_t period;
 		xnticks_t timeout;
 		xnflags_t status;
+		xnarch_cpumask_t affinity;
 	} sched_info[1];
 };
 
@@ -145,8 +146,9 @@ static int sched_seq_show(struct seq_fil
 	char sbuf[64], pbuf[16];
 
 	if (v == SEQ_START_TOKEN)
-		seq_printf(seq, "%-3s  %-6s %-8s %-8s %-10s %-10s %s\n",
-			   "CPU", "PID", "PRI", "PERIOD", "TIMEOUT", "STAT", "NAME");
+		seq_printf(seq, "%-3s  %-8s %-6s %-8s %-8s %-10s %-10s %s\n",
+			   "CPU", "AFFINITY", "PID", "PRI", "PERIOD", "TIMEOUT",
+			   "STAT", "NAME");
 	else {
 		struct sched_seq_info *p = (struct sched_seq_info *)v;
 
@@ -156,8 +158,9 @@ static int sched_seq_show(struct seq_fil
 		else
 			snprintf(pbuf, sizeof(pbuf), "%3d", p->cprio);
 
-		seq_printf(seq, "%3u  %-6d %-8s %-8Lu %-10Lu %-10s %s\n",
+		seq_printf(seq, "%3u  %08x %-6d %-8s %-8Lu %-10Lu %-10s %s\n",
 			   p->cpu,
+			   p->affinity,
 			   p->pid,
 			   pbuf,
 			   p->period,
@@ -237,6 +240,7 @@ static int sched_seq_open(struct inode *
 		iter->sched_info[n].period = xnthread_get_period(thread);
 		iter->sched_info[n].timeout = xnthread_get_timeout(thread, iter->start_time);
 		iter->sched_info[n].status = thread->status;
+		iter->sched_info[n].affinity = thread->affinity;
 
 		holder = nextq(&nkpod->threadq, holder);
 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Limiting Xenomai tasks to one certain core in a dual core system

[Jan Kiszka]

[-- Attachment #1.1: Type: text/plain, Size: 2015 bytes --]

Jan Kiszka wrote:
> This is version 2 of my patch, consequently applying the global mask
> idea also on IRQ affinity. I think this simple mechanism can solve quite
> a few basic SMP setup issue by forcing all user threads, driver threads,
> and IRQs that are created/attached during a certain period to a specific
> CPU set.
> 
> That's the theory. In practice, things look not that clear ATM. I
> noticed some oddity on a 4-way qemu box running the latency test and
> hacked the attached enhancement for /proc/xenomai/sched. This is what I
> get on that box:
> 
>> root@domain.hid :/root# cat /proc/xenomai/sched 
>> CPU  AFFINITY PID    PRI      PERIOD   TIMEOUT    STAT       NAME
>>   0  00000001 0       -1      0        0          R          ROOT/0
>>   1  00000002 0       -1      0        0          R          ROOT/1
>>   2  00000004 0       -1      0        0          R          ROOT/2
>>   3  00000008 0       -1      0        0          R          ROOT/3
>>   0  0000000c 917      0      0        0          W          display-916
>>   2  0000000c 918     99      5000000  1          D          sampling-916
>> root@domain.hid :/root# cat /proc/xenomai/affinity 
>> 0x0000000c
> 

[I should really stop believing heavy issues can be solved with
light-weight patches...]

OK, after reading a bit more in what I was hacking on, I'm starting to
understand the CPU selection mechanisms of shadow threads. Here comes
version 3 of the patch. Now it actually forces all threads to the
desired CPUs.

The reason why I failed with previous approaches could even be a SMP bug
of the original code. The Linux affinity of a shadow thread was only set
if no u_completion has been passed to xnshadow_map - which affected
native threads e.g. So I reordered the code, and it works.

I also changed the policy of /proc/xenoami/affinity. It will now only
override the mapping if the user didn't provide an explicit mask. I
think this provides more flexibility.

Comment?

Jan

[-- Attachment #1.2: global-cpu-affinity-v3.patch --]
[-- Type: text/plain, Size: 5329 bytes --]

---
 include/nucleus/pod.h |    2 +
 ksrc/nucleus/intr.c   |    3 ++
 ksrc/nucleus/module.c |   57 ++++++++++++++++++++++++++++++++++++++++++++++++++
 ksrc/nucleus/pod.c    |    4 ++-
 ksrc/nucleus/shadow.c |   15 +++++--------
 5 files changed, 71 insertions(+), 10 deletions(-)

Index: xenomai/include/nucleus/pod.h
===================================================================
--- xenomai.orig/include/nucleus/pod.h
+++ xenomai/include/nucleus/pod.h
@@ -253,6 +253,8 @@ extern u_long nktickdef;
 
 extern char *nkmsgbuf;
 
+extern xnarch_cpumask_t nkaffinity;
+
 #define xnprintf(fmt,args...)  xnarch_printf(fmt , ##args)
 #define xnloginfo(fmt,args...) xnarch_loginfo(fmt , ##args)
 #define xnlogwarn(fmt,args...) xnarch_logwarn(fmt , ##args)
Index: xenomai/ksrc/nucleus/intr.c
===================================================================
--- xenomai.orig/ksrc/nucleus/intr.c
+++ xenomai/ksrc/nucleus/intr.c
@@ -626,6 +626,9 @@ int xnintr_attach(xnintr_t *intr, void *
 
 	xnlock_get_irqsave(&intrlock, s);
 
+#ifdef CONFIG_SMP
+	xnarch_set_irq_affinity(intr->irq, nkaffinity);
+#endif /* CONFIG_SMP */
 	err = xnintr_irq_attach(intr);
 
 	xnlock_put_irqrestore(&intrlock, s);
Index: xenomai/ksrc/nucleus/module.c
===================================================================
--- xenomai.orig/ksrc/nucleus/module.c
+++ xenomai/ksrc/nucleus/module.c
@@ -688,6 +688,59 @@ static int heap_read_proc(char *page,
 	return len;
 }
 
+static int affinity_read_proc(char *page,
+			      char **start,
+			      off_t off, int count, int *eof, void *data)
+{
+	unsigned long val = 0;
+	int len, cpu;
+
+	for (cpu = 0; cpu < sizeof(val) * 8; cpu++)
+		if (xnarch_cpu_isset(cpu, nkaffinity))
+			val |= (1 << cpu);
+
+	len = sprintf(page, "%08lx\n", val);
+	len -= off;
+	if (len <= off + count)
+		*eof = 1;
+	*start = page + off;
+	if (len > count)
+		len = count;
+	if (len < 0)
+		len = 0;
+
+	return len;
+}
+
+static int affinity_write_proc(struct file *file,
+			       const char __user * buffer,
+			       unsigned long count, void *data)
+{
+	char *end, buf[16];
+	unsigned long val;
+	xnarch_cpumask_t new_affinity;
+	int n, cpu;
+
+	n = count > sizeof(buf) - 1 ? sizeof(buf) - 1 : count;
+
+	if (copy_from_user(buf, buffer, n))
+		return -EFAULT;
+
+	buf[n] = '\0';
+	val = simple_strtol(buf, &end, 0);
+
+	if (*end != '\0' && !isspace(*end))
+		return -EINVAL;
+
+	xnarch_cpus_clear(new_affinity);
+	for (cpu = 0; cpu < sizeof(val) * 8; cpu++, val >>= 1)
+		if (val & 1)
+			xnarch_cpu_set(cpu, new_affinity);
+	nkaffinity = new_affinity;
+
+	return count;
+}
+
 static struct proc_dir_entry *add_proc_leaf(const char *name,
 					    read_proc_t rdproc,
 					    write_proc_t wrproc,
@@ -760,6 +813,9 @@ void xnpod_init_proc(void)
 
 	add_proc_leaf("heap", &heap_read_proc, NULL, NULL, rthal_proc_root);
 
+	add_proc_leaf("affinity", &affinity_read_proc, &affinity_write_proc,
+		      NULL, rthal_proc_root);
+
 #ifdef CONFIG_XENO_OPT_PERVASIVE
 	iface_proc_root =
 	    create_proc_entry("interfaces", S_IFDIR, rthal_proc_root);
@@ -778,6 +834,7 @@ void xnpod_delete_proc(void)
 
 	remove_proc_entry("interfaces", rthal_proc_root);
 #endif /* CONFIG_XENO_OPT_PERVASIVE */
+	remove_proc_entry("affinity", rthal_proc_root);
 	remove_proc_entry("heap", rthal_proc_root);
 	remove_proc_entry("irq", rthal_proc_root);
 	remove_proc_entry("timer", rthal_proc_root);
Index: xenomai/ksrc/nucleus/pod.c
===================================================================
--- xenomai.orig/ksrc/nucleus/pod.c
+++ xenomai/ksrc/nucleus/pod.c
@@ -70,6 +70,8 @@ MODULE_PARM_DESC(tick_arg, "Fixed clock 
 
 char *nkmsgbuf = NULL;
 
+xnarch_cpumask_t nkaffinity = XNPOD_ALL_CPUS;
+
 const char *xnpod_fatal_helper(const char *format, ...)
 {
 	const unsigned nr_cpus = xnarch_num_online_cpus();
@@ -910,7 +912,7 @@ int xnpod_start_thread(xnthread_t *threa
 		return -EBUSY;
 
 	if (xnarch_cpus_empty(affinity))
-		affinity = XNARCH_CPU_MASK_ALL;
+		affinity = nkaffinity;
 
 	xnlock_get_irqsave(&nklock, s);
 
Index: xenomai/ksrc/nucleus/shadow.c
===================================================================
--- xenomai.orig/ksrc/nucleus/shadow.c
+++ xenomai/ksrc/nucleus/shadow.c
@@ -853,20 +853,17 @@ int xnshadow_map(xnthread_t *thread, xnc
 	xnshadow_thrptd(current) = thread;
 	xnpod_suspend_thread(thread, XNRELAX, XN_INFINITE, NULL);
 
+	affinity = thread->affinity;
+	if (xnarch_cpus_empty(affinity))
+		affinity = nkaffinity;
+	set_cpus_allowed(current, affinity);
+
 	if (u_completion) {
 		xnshadow_signal_completion(u_completion, 0);
 		return 0;
 	}
 
-	/* Nobody waits for us, so we may start the shadow immediately
-	   after having forced the CPU affinity to the current
-	   processor. Note that we don't use smp_processor_id() to prevent
-	   kernel debug stuff to yell at us for calling it in a preemptible
-	   section of code. */
-
-	affinity = xnarch_cpumask_of_cpu(rthal_processor_id());
-	set_cpus_allowed(current, affinity);
-
+	/* Nobody waits for us, so we may start the shadow immediately. */
 	mode = thread->rrperiod != XN_INFINITE ? XNRRB : 0;
 	xnpod_start_thread(thread, mode, 0, affinity, NULL, NULL);
 

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-core] Limiting Xenomai tasks to one certain core in a dual core system

[Philippe Gerum]

On Sat, 2006-11-25 at 16:12 +0100, Jan Kiszka wrote:

[...]

> OK, after reading a bit more in what I was hacking on, I'm starting to
> understand the CPU selection mechanisms of shadow threads. Here comes
> version 3 of the patch. Now it actually forces all threads to the
> desired CPUs.
> 
> The reason why I failed with previous approaches could even be a SMP bug
> of the original code. The Linux affinity of a shadow thread was only set
> if no u_completion has been passed to xnshadow_map - which affected
> native threads e.g. So I reordered the code, and it works.
> 

The root issue is why LO_START_REQ which normally sets the affinity at
the end of the create -> start protocol did not get a chance to run.
Likely that the newly mapped thread was still in a runnable state when
tested from xnshadow_start(), which might be due to a kernel preemption
artefact.

> I also changed the policy of /proc/xenoami/affinity. It will now only
> override the mapping if the user didn't provide an explicit mask. I
> think this provides more flexibility.
> 
> Comment?

[...]

>  	const unsigned nr_cpus = xnarch_num_online_cpus();
> @@ -910,7 +912,7 @@ int xnpod_start_thread(xnthread_t *threa
>  		return -EBUSY;
>  
>  	if (xnarch_cpus_empty(affinity))
> -		affinity = XNARCH_CPU_MASK_ALL;
> +		affinity = nkaffinity;

This is fairly significant API change that should be loudly documented,
especially because its behaviour now depends on a new variable.

>  
>  	xnlock_get_irqsave(&nklock, s);
>  
> Index: xenomai/ksrc/nucleus/shadow.c
> ===================================================================
> --- xenomai.orig/ksrc/nucleus/shadow.c
> +++ xenomai/ksrc/nucleus/shadow.c
> @@ -853,20 +853,17 @@ int xnshadow_map(xnthread_t *thread, xnc
>  	xnshadow_thrptd(current) = thread;
>  	xnpod_suspend_thread(thread, XNRELAX, XN_INFINITE, NULL);
>  
> +	affinity = thread->affinity;
> +	if (xnarch_cpus_empty(affinity))
> +		affinity = nkaffinity;

This won't work as expected. The thread->affinity info should not be
used in the xnshadow_map() context, because it has not been set yet
(xnshadow_map() should be seen as the equivalent to xnpod_init_thread()
for shadow threads). This information is set by the nucleus by a call to
xnpod_start_thread(), so one step later in the task spawn protocol.
Luckily, it seems to be zeroed in your test, but it has not been
initialized yet.

> +	set_cpus_allowed(current, affinity);
> +

This should be ok to pin the newly mapped task to a CPU early on, even
in the auto-start configuration, but that case, we should kill the now
redundant set_cpus_allowed() from LO_START_REQ. Actually, we would be
better of killing LO_START_REQ entirely, since it would not make sense
anymore, compared to a regular WAKEUP request.

-- 
Philippe.