[Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Stoidner, Christoph]

Hi at all,

I need some sort of real time behaviour in secondary domain, so I am trying to merge Preempt RT patch with I-Pipe/Xenomai. Since Preempt RT as well as I-Pipe doing some specific changes to kernel's spinlock handling (mostly in  "include/linux/spinlock.h") this becomes a bit of tricky. The challenge seems to combine Preemp RT's "rt_spin_lock" with I-Pipe's "ipipe_spinlock".  

My first approach was to instrument I-Pipe's "PICK_SPINLOCK" macros with Preempt RT's "rt_spin_lock" calls. In detail that means I have added calls of rt_spin_lock to all PICK_SPINLOCK macros for each spinlock type  that is not an I-Pipe spinlock. However I am not sure if that is the best solution and if it is correct for all situations. 

Has someone experiences how to merge these spinlocks or what to consider at all when using Preempt RT with I-Pipe/Xenomai?

I am grateful for each hint according to that theme.

For Information: Currently I am working with kernel Kernel 3.10.18 and Xenomai 2.6.3

Thanks and regards,
Christoph

Re: [Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Philippe Gerum]

On 05/25/2014 09:59 PM, Stoidner, Christoph wrote:
> Hi at all,
>
> I need some sort of real time behaviour in secondary domain, so I am trying to merge Preempt RT patch with I-Pipe/Xenomai. Since Preempt RT as well as I-Pipe doing some specific changes to kernel's spinlock handling (mostly in  "include/linux/spinlock.h") this becomes a bit of tricky. The challenge seems to combine Preemp RT's "rt_spin_lock" with I-Pipe's "ipipe_spinlock".
>
> My first approach was to instrument I-Pipe's "PICK_SPINLOCK" macros with Preempt RT's "rt_spin_lock" calls. In detail that means I have added calls of rt_spin_lock to all PICK_SPINLOCK macros for each spinlock type  that is not an I-Pipe spinlock. However I am not sure if that is the best solution and if it is correct for all situations.
>
> Has someone experiences how to merge these spinlocks or what to consider at all when using Preempt RT with I-Pipe/Xenomai?
>

Fixing up ipipe_lock.h to switch the regular locks (i.e. neither 
ipipe/raw) to the sleeping rt_ spinlock variant is pretty much all you 
should have to do these days.

My last experience merging I-pipe+preempt-rt dates back to kernel 2.6.x, 
so a few more conflicts might have popped up since then though. This 
said, I did not see any showstopper in recent kernels that would prevent 
such combo from working.

e.g.:

+#ifdef PREEMPT_RT
+ #define __regular_spin_lock_irqsave(lock, flags)		\
+	do { rt_spin_lock(std_spinlock(lock)); flags = 0; } while (0) /* can't 
discard flags */
+#else
+#define __regular_spin_lock_irqsave(lock, flags)		\
+	__real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags)
+#endif

#define PICK_SPINLOCK_IRQSAVE(lock, flags)				\
	do {								\
		if (ipipe_spinlock_p(lock))				\
			(flags) = __ipipe_spin_lock_irqsave(ipipe_spinlock(lock)); \
		else if (std_spinlock_raw_p(lock))				\
			__real_raw_spin_lock_irqsave(std_spinlock_raw(lock), flags); \
		else if (std_spinlock_p(lock))				\
-			__real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \
+			__regular_spin_lock_irqsave(lock, flags); \
		else __bad_lock_type();					\
	} while (0)

and so on for all other helpers defining a preempt-rt specific variant.

-- 
Philippe.

Re: [Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Stoidner, Christoph]

Hi Philippe,

thanks for the fast response! 

I will try to solve the spinlock conflicts according to your description and post my results here later on.

Regards,
Christoph

________________________________________
Von: Philippe Gerum <rpm@xenomai.org>
Gesendet: Montag, 26. Mai 2014 13:08
An: Stoidner, Christoph; xenomai@xenomai.org
Betreff: Re: [Xenomai] Using Preempt RT with I-Pipe/Xenomai

On 05/25/2014 09:59 PM, Stoidner, Christoph wrote:
> Hi at all,
>
> I need some sort of real time behaviour in secondary domain, so I am trying to merge Preempt RT patch with I-Pipe/Xenomai. Since Preempt RT as well as I-Pipe doing some specific changes to kernel's spinlock handling (mostly in  "include/linux/spinlock.h") this becomes a bit of tricky. The challenge seems to combine Preemp RT's "rt_spin_lock" with I-Pipe's "ipipe_spinlock".
>
> My first approach was to instrument I-Pipe's "PICK_SPINLOCK" macros with Preempt RT's "rt_spin_lock" calls. In detail that means I have added calls of rt_spin_lock to all PICK_SPINLOCK macros for each spinlock type  that is not an I-Pipe spinlock. However I am not sure if that is the best solution and if it is correct for all situations.
>
> Has someone experiences how to merge these spinlocks or what to consider at all when using Preempt RT with I-Pipe/Xenomai?
>

Fixing up ipipe_lock.h to switch the regular locks (i.e. neither
ipipe/raw) to the sleeping rt_ spinlock variant is pretty much all you
should have to do these days.

My last experience merging I-pipe+preempt-rt dates back to kernel 2.6.x,
so a few more conflicts might have popped up since then though. This
said, I did not see any showstopper in recent kernels that would prevent
such combo from working.

e.g.:

+#ifdef PREEMPT_RT
+ #define __regular_spin_lock_irqsave(lock, flags)              \
+       do { rt_spin_lock(std_spinlock(lock)); flags = 0; } while (0) /* can't
discard flags */
+#else
+#define __regular_spin_lock_irqsave(lock, flags)               \
+       __real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags)
+#endif

#define PICK_SPINLOCK_IRQSAVE(lock, flags)                              \
        do {                                                            \
                if (ipipe_spinlock_p(lock))                             \
                        (flags) = __ipipe_spin_lock_irqsave(ipipe_spinlock(lock)); \
                else if (std_spinlock_raw_p(lock))                              \
                        __real_raw_spin_lock_irqsave(std_spinlock_raw(lock), flags); \
                else if (std_spinlock_p(lock))                          \
-                       __real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \
+                       __regular_spin_lock_irqsave(lock, flags); \
                else __bad_lock_type();                                 \
        } while (0)

and so on for all other helpers defining a preempt-rt specific variant.

--
Philippe.

Re: [Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Vasudev Kamath]

Hi Philippe,

Philippe Gerum <rpm <at> xenomai.org> writes:

> 
> Fixing up ipipe_lock.h to switch the regular locks (i.e. neither 
> ipipe/raw) to the sleeping rt_ spinlock variant is pretty much all you 
> should have to do these days.
> 
> My last experience merging I-pipe+preempt-rt dates back to kernel 2.6.x, 
> so a few more conflicts might have popped up since then though. This 
> said, I did not see any showstopper in recent kernels that would prevent 
> such combo from working.
> 
> e.g.:
> 
> +#ifdef PREEMPT_RT
> + #define __regular_spin_lock_irqsave(lock, flags)		\
> +	do { rt_spin_lock(std_spinlock(lock)); flags = 0; } while (0) /* can't 
> discard flags */
> +#else
> +#define __regular_spin_lock_irqsave(lock, flags)		\
> +	__real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags)
> +#endif
> 
> #define PICK_SPINLOCK_IRQSAVE(lock, flags)				\
> 	do {								\
> 		if (ipipe_spinlock_p(lock))				\
> 			(flags) = __ipipe_spin_lock_irqsave(ipipe_spinlock(lock)); \
> 		else if (std_spinlock_raw_p(lock))				\
> 			__real_raw_spin_lock_irqsave(std_spinlock_raw(lock), flags); \
> 		else if (std_spinlock_p(lock))				\
> -			__real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \
> +			__regular_spin_lock_irqsave(lock, flags); \
> 		else __bad_lock_type();					\
> 	} while (0)
> 
> and so on for all other helpers defining a preempt-rt specific variant.

I'm also trying to use both Xenomai IPipe with RT_PREEMPT patch and stumbled
across this thread. I should say I'm novice in RT world so I just followed
your hint above and did this patch to ipipe_lock.h. I'm using Kernel 3.14.25
with Xenomai 2.6 from Git (snapshot 546d1fd)

[PS: Wanted to attach the patch but Gmane didn't provide a option so pasting
it here]
diff --git a/include/linux/ipipe_lock.h b/include/linux/ipipe_lock.h
index f71d2f1..7959a65 100644
--- a/include/linux/ipipe_lock.h
+++ b/include/linux/ipipe_lock.h
@@ -42,6 +42,20 @@ typedef struct {
 #define std_spinlock_raw(lock)	((raw_spinlock_t *)(lock))
 #define std_spinlock(lock)	((spinlock_t *)(lock))
 
+/* Thanks to Philippe Gerrum for this snippet
+ * https://www.mail-archive.com/xenomai@xenomai.org/msg05815.html
+ */
+#ifdef CONFIG_PREEMPT_RT_FULL
+#define __regular_spin_lock_irqsave(lock, flags)                       \
+	do { rt_spin_lock(std_spinlock(lock)); flags = 0; } while (0) /* Can't
+								       * discard
+								       * flags
+								       * */
+#else
+#define __regular_spin_lock_irqsave(lock, flags)                       \
+	__real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags)
+#endif
+
 #define PICK_SPINLOCK_IRQSAVE(lock, flags)				\
 	do {								\
 		if (ipipe_spinlock_p(lock))				\
@@ -49,7 +63,7 @@ typedef struct {
 		else if (std_spinlock_raw_p(lock))				\
 			__real_raw_spin_lock_irqsave(std_spinlock_raw(lock), flags); \
 		else if (std_spinlock_p(lock))				\
-			__real_raw_spin_lock_irqsave(&std_spinlock(lock)->rlock, flags); \
+			__regular_spin_lock_irqsave(lock, flags);       \
 		else __bad_lock_type();					\
 	} while (0)
 
@@ -79,6 +93,14 @@ typedef struct {
 		__ret__;						\
 	 })
 
+#ifdef CONFIG_PREEMPT_RT_FULL
+#define __regular_spin_unlock_irqrestore(lock, flags)                   \
+	do { rt_spin_unlock(std_spinlock(lock)); flags = 0; } while (0)
+#else
+#define __regular_spin_unlock_irqrestore(lock, flags)                   \
+	__real_raw_spin_unlock_irqrestore(&std_spinlock(lock)->rlock, flags);
+#endif
+
 #define PICK_SPINUNLOCK_IRQRESTORE(lock, flags)				\
 	do {								\
 		if (ipipe_spinlock_p(lock))				\
@@ -88,10 +110,18 @@ typedef struct {
 			if (std_spinlock_raw_p(lock))			\
 				__real_raw_spin_unlock_irqrestore(std_spinlock_raw(lock), flags); \
 			else if (std_spinlock_p(lock))			\
-				__real_raw_spin_unlock_irqrestore(&std_spinlock(lock)->rlock, flags); \
+				__regular_spin_unlock_irqrestore(lock, flags); \
 		}							\
 	} while (0)
 
+#ifdef CONFIG_PREEMPT_RT_FULL
+#define __regular_spin(op, lock)	\
+	rt_spin_lock(std_spinlock(lock))
+#else
+#define __regular_spin(op, lock)					\
+	__real_raw_spin##op(&std_spinlock(lock)->rlock)
+#endif
+
 #define PICK_SPINOP(op, lock)						\
 	({								\
 		if (ipipe_spinlock_p(lock))				\
@@ -99,7 +129,7 @@ typedef struct {
 		else if (std_spinlock_raw_p(lock))			\
 			__real_raw_spin##op(std_spinlock_raw(lock));	\
 		else if (std_spinlock_p(lock))				\
-			__real_raw_spin##op(&std_spinlock(lock)->rlock); \
+			__regular_spin(op, lock);               	\
 		else __bad_lock_type();					\
 		(void)0;						\
 	})

After the patch Kernel did compile, though I've not tested the compiled
kernel yet. Since I've no idea if what I've done is correct or not I would
be happy to get some review on this patch from you or fellow Xenomai folks.

Also after the patching I did see several kernel compile warning similar to
the below

kernel/ipipe/core.c: In function 'ipipe_alloc_virq':
kernel/ipipe/core.c:825:2: warning: passing argument 1 of 'rt_spin_lock'
from incompatible pointer type [enabled by default]
In file included from include/linux/spinlock.h:310:0,
                 from include/linux/seqlock.h:35,
                 from include/linux/time.h:5,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from kernel/ipipe/core.c:24:
include/linux/spinlock_rt.h:21:56: note: expected 'struct spinlock_t *' but
argument is of type 'struct __ipipe_spinlock_t *'
kernel/ipipe/core.c:833:2: warning: passing argument 1 of 'rt_spin_unlock'
from incompatible pointer type [enabled by default]
In file included from include/linux/spinlock.h:310:0,
                 from include/linux/seqlock.h:35,
                 from include/linux/time.h:5,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from kernel/ipipe/core.c:24:
include/linux/spinlock_rt.h:24:56: note: expected 'struct spinlock_t *' but
argument is of type 'struct __ipipe_spinlock_t *'
kernel/ipipe/core.c: In function 'ipipe_request_irq':
kernel/ipipe/core.c:863:2: warning: passing argument 1 of 'rt_spin_lock'
from incompatible pointer type [enabled by default]
In file included from include/linux/spinlock.h:310:0,
                 from include/linux/seqlock.h:35,
                 from include/linux/time.h:5,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from kernel/ipipe/core.c:24:
include/linux/spinlock_rt.h:21:56: note: expected 'struct spinlock_t *' but
argument is of type 'struct __ipipe_spinlock_t *'
kernel/ipipe/core.c:881:2: warning: passing argument 1 of 'rt_spin_unlock'
from incompatible pointer type [enabled by default]
In file included from include/linux/spinlock.h:310:0,
                 from include/linux/seqlock.h:35,
                 from include/linux/time.h:5,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from kernel/ipipe/core.c:24:
include/linux/spinlock_rt.h:24:56: note: expected 'struct spinlock_t *' but
argument is of type 'struct __ipipe_spinlock_t *'
kernel/ipipe/core.c: In function 'ipipe_free_irq':
kernel/ipipe/core.c:896:2: warning: passing argument 1 of 'rt_spin_lock'
from incompatible pointer type [enabled by default]
In file included from include/linux/spinlock.h:310:0,
                 from include/linux/seqlock.h:35,
                 from include/linux/time.h:5,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from kernel/ipipe/core.c:24:
include/linux/spinlock_rt.h:21:56: note: expected 'struct spinlock_t *' but
argument is of type 'struct __ipipe_spinlock_t *'
kernel/ipipe/core.c:909:2: warning: passing argument 1 of 'rt_spin_unlock'
from incompatible pointer type [enabled by default]
In file included from include/linux/spinlock.h:310:0,
                 from include/linux/seqlock.h:35,
                 from include/linux/time.h:5,
                 from include/linux/stat.h:18,
                 from include/linux/module.h:10,
                 from kernel/ipipe/core.c:24:
include/linux/spinlock_rt.h:24:56: note: expected 'struct spinlock_t *' but
argument is of type 'struct __ipipe_spinlock_t *'

Again I couldn't figure out the reasons for warning, and will be happy to
find some hints on fixing this.

Thanks in Advance,

Re: [Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Vasudev Kamath]

Vasudev Kamath <vasudev@copyninja.info> writes:

> After the patch Kernel did compile, though I've not tested the compiled
> kernel yet. Since I've no idea if what I've done is correct or not I would
> be happy to get some review on this patch from you or fellow Xenomai folks.
>

Err correction, it looks like there are still errors but compilation
didn't stop on errors. Currently I saw following error.

kernel/locking/rtmutex.c: In function _rt_mutex_adjust_prio_chain_:
kernel/locking/rtmutex.c:459:969: error: _spinlock_t_ has no member named _rlock_
kernel/locking/rtmutex.c: In function _rt_spin_lock_slowunlock_hirq_:
kernel/locking/rtmutex.c:1081:971: error: _spinlock_t_ has no member named _rlock_
kernel/locking/rtmutex.c: In function _rt_mutex_slowtrylock_:
kernel/locking/rtmutex.c:1473:969: error: _spinlock_t_ has no member named _rlock_

Will be looking at code now. But it looks like fixing this might be
beyond my expertise. I wonder if Christoph managed to compile kernel
from his side.

Cheers,

Re: [Xenomai] Using Preempt RT with I-Pipe/Xenomai

[Vasudev Kamath]

Vasudev Kamath <vasudev@copyninja.info> writes:
>
> Err correction, it looks like there are still errors but compilation
> didn't stop on errors. Currently I saw following error.
>
> kernel/locking/rtmutex.c: In function _rt_mutex_adjust_prio_chain_:
> kernel/locking/rtmutex.c:459:969: error: _spinlock_t_ has no member named _rlock_
> kernel/locking/rtmutex.c: In function _rt_spin_lock_slowunlock_hirq_:
> kernel/locking/rtmutex.c:1081:971: error: _spinlock_t_ has no member named _rlock_
> kernel/locking/rtmutex.c: In function _rt_mutex_slowtrylock_:
> kernel/locking/rtmutex.c:1473:969: error: _spinlock_t_ has no member named _rlock_
>
> Will be looking at code now. But it looks like fixing this might be
> beyond my expertise. I wonder if Christoph managed to compile kernel
> from his side.

OK I did manage to get the compilation fixed, of course with my naive
approach. I'm attaching updated patch with this email.

I tried booting into resulting kernel, but ended up having kernel panic,
attaching screen shot for reference.

So it clearly means I've done something wrong when patching
ipipe_lock.h. I would be thankful to any one who can help me with
getting this patch working.

Thanks in Advance,

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ipipe_lock_rtpreempt.patch
Type: text/x-diff
Size: 4343 bytes
Desc: ipipe_lock patch to work with rt_preempt
URL: <http://xenomai.org/pipermail/xenomai/attachments/20151027/632cb155/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 20151027_001.jpg
Type: image/jpeg
Size: 917100 bytes
Desc: kernel panic with rt_preempt xenomai patch
URL: <http://xenomai.org/pipermail/xenomai/attachments/20151027/632cb155/attachment.jpg>