[Xenomai-help] Xenomai and SMP load balancing

[Xenomai-help] Xenomai and SMP load balancing

[Arnout Vandecappelle]

 [Please keep me in CC]

 Hoi all,

 I have a Xenomai application that I run on an SMP processor (i3 or i5).  
It takes about 90% load on one of the cores (cpu0).  The rest of the system 
is usually idle, except for occasional runs of some compute-intensive batch 
processes (that run in the non-RT Linux domain).

 The problem is that the Linux load balancer doesn't know about the
Xenomai thread.  It therefore tends to select cpu0 as well for the
batch processes.  That means that the batch processes take much longer
then needed.

 I can solve this by reserving one complete core for the Xenomai thread
(by setting /proc/xenomai/affinity to 1 and giving the isolcpus=0 at boot
time).  But this makes it difficult to move to a non-SMT platform, or
to run several Xenomai applications on different cores.

 Is there a better way to let the Linux load balancer take into account
the load of the Xenomai threads?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

Re: [Xenomai-help] Xenomai and SMP load balancing

[Arnout Vandecappelle]

On Wednesday 21 December 2011 15:26:08 Arnout Vandecappelle wrote:
>  [Please keep me in CC]
> 
>  Hoi all,
> 
>  I have a Xenomai application that I run on an SMP processor (i3 or i5).  
> It takes about 90% load on one of the cores (cpu0).  The rest of the system 
> is usually idle, except for occasional runs of some compute-intensive batch 
> processes (that run in the non-RT Linux domain).
> 
>  The problem is that the Linux load balancer doesn't know about the
> Xenomai thread.  It therefore tends to select cpu0 as well for the
> batch processes.  That means that the batch processes take much longer
> then needed.
> 
>  I can solve this by reserving one complete core for the Xenomai thread
> (by setting /proc/xenomai/affinity to 1 and giving the isolcpus=0 at boot
> time).  But this makes it difficult to move to a non-SMT platform, or
> to run several Xenomai applications on different cores.
> 
>  Is there a better way to let the Linux load balancer take into account
> the load of the Xenomai threads?

 Ping?

 Also, for the benefit of others, I now have a more flexible way to assign
separate cpus to Xenomai and Linux.  Below is a scriptlet that implements
it.

 I have also seen that something like this is really needed to get reasonable
predictability on x86 processors.  Without it, the jitter on latency is much
higher if there is even the slightest amount of work to be done in the Linux
domain (in my case, just sending the output of the latency benchmark over ssh 
once per second increases the jitter on latency tenfold).  I guess this is due
to cache effects when a Linux thread gets scheduled between runs of a Xenomai
task.

 Also note that this does not remove all Linux work from the CPU that is
reserved for Xenomai: I found that the following kernel work may still run
on it.

- Some kernel threads are created per CPU.  These cannot be (re)moved.  They
may execute due to e.g. interrupts.

- All kernel threads that are created later on will be schedulable on any
CPU, because the CPU mask is not inherited for kernel threads.  The only
work-around I found is to reassign the kernel thread back to the root cpuset
and again to the other cpuset after it is created.  E.g., after mounting a
journalling filesystem I move the kjournald thread back to / and again to
/other.  Hugely annoying, that.

- It looks like some per-cpu administration has to execute per cpu as well.
As a result, when a CPU is completely occupied by Xenomai tasks (which means
no Linux code ever executes on it), the entire Linux domain will freeze.
Of course, occupying a CPU 100% with Xenomai work is usually not a good idea
anyway.


 Hopefully this information will be useful for other people using Xenomai on
x86 SMP processors.

 Regards,
 Arnout



---------------------------------------------------------------------------
# Setup processor allocation
# We define two CPU sets: 'xenomai' and 'other'.
# For a single model, the 'xenomai' CPU set contains a single cpu
# All existing tasks are migrated to the 'other' CPU set
cpulist=$(echo /sys/devices/system/cpu/cpu[0-9]* | sed 's%/sys/devices/system/cpu/cpu%%g' | tr ' ' ,)
last_cpu=$(echo $cpulist | sed s/.*,//)
# Skip the whole cpuset thing if there is only one cpu
if [ $last_cpu != 0 ]; then
	echo "Preparing cpu sets for Xenomai on cpu $last_cpu"
	mkdir -p /dev/cpuset
	mount -t cgroup -ocpuset cgroup /dev/cpuset

	# other: all except last_cpu, mem is same as root
	mkdir /dev/cpuset/other
	/bin/echo $cpulist | sed s/,$last_cpu// > /dev/cpuset/other/cpuset.cpus
	cat /dev/cpuset/cpuset.mems > /dev/cpuset/other/cpuset.mems

	# Move all existing tasks to the 'other' cgroup
	for task in `cat /dev/cpuset/tasks`; do
		/bin/echo $task > /dev/cpuset/other/tasks || echo "Failed to move PID $task"
	done

	# Reset the Xenomai CPU to make sure that all kernel threads are moved off it
	# *** This must be done before xenomai modules are loaded ***
	# I'm not sure if this is needed at all.
	echo "Resetting cpu $last_cpu"
	echo 0 > /sys/devices/system/cpu/cpu${last_cpu}/online
	echo 1 > /sys/devices/system/cpu/cpu${last_cpu}/online

	# xenomai: only last_cpu
	# This must be done after resetting, because offline cpus are removed from cpusets
	mkdir /dev/cpuset/xenomai
	/bin/echo $last_cpu > /dev/cpuset/xenomai/cpuset.cpus
	/bin/echo 1 > /dev/cpuset/xenomai/cpuset.cpu_exclusive
	cat /dev/cpuset/cpuset.mems > /dev/cpuset/xenomai/cpuset.mems
	# FIXME disable the thread siblings of the Xenomai cores /sys/devices/cpu/cpu$last_cpu/topology/thread_siblings

	# Move self to the 'xenomai' cgroup
	/bin/echo $$ > /dev/cpuset/xenomai/tasks || echo "Failed to move self (PID $$)"
fi

# load xenomai kernel modules (=realtime environment)
modprobe xeno_nucleus
modprobe xeno_native
modprobe xeno_rtdm
modprobe xeno_posix

# Set Xenomai affinity
let 'last_cpu_mask = 1 << last_cpu'
echo $last_cpu_mask > /proc/xenomai/affinity
echo -n "Set affinity to "; cat /proc/xenomai/affinity

# Start Xenomai applications
...

# Move back to other cpuset
if [ -d /dev/cpuset/other ]; then
	/bin/echo $$ > /dev/cpuset/other/tasks || echo "Failed to move self (PID $$)"
fi


-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F