[Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Marc Strämke]

Hello everyone,

during migration to xenomai 3 we found in testing that enabling the kernel 
config option CONFIG_CC_STACKPROTECTOR_REGULAR leads to hangs or reboots 
(depends on the target platform we are testing on). 

This post: http://www.xenomai.org/pipermail/xenomai/2012-July/000511.html 
shows that the old xenomai 2.6 triggered an error when the stack protector was 
enabled.

I think this warning should be readded to the newer source or at least a 
warning added to the kernel compile documentation.

Attached see my kernel config which crashed (based on debian 8 config) 

Marc

-- 
Dipl.-Ing. Marc Strämke
Geschäftsführer / CEO

ELTROPULS Anlagenbau GmbH
Arnold-Sommerfeld-Ring 3
52499 Baesweiler
Germany

www.eltropuls.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config_crash
Type: text/x-mpsub
Size: 134194 bytes
Desc: not available
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/9e4a5a6e/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/9e4a5a6e/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 04:33:40PM +0200, Marc Strämke wrote:
> Hello everyone,
> 
> during migration to xenomai 3 we found in testing that enabling the kernel 
> config option CONFIG_CC_STACKPROTECTOR_REGULAR leads to hangs or reboots 
> (depends on the target platform we are testing on). 
> 
> This post: http://www.xenomai.org/pipermail/xenomai/2012-July/000511.html 
> shows that the old xenomai 2.6 triggered an error when the stack protector was 
> enabled.
> 
> I think this warning should be readded to the newer source or at least a 
> warning added to the kernel compile documentation.

No. At least ARM and x86 architectures are supposed to support the
stack protector. Maybe you could tell us on what architecture you
have this issue?

-- 
					    Gilles.

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Marc Strämke]

Am Donnerstag, 30. April 2015, 16:41:05 schrieb Gilles Chanteperdrix:
> No. At least ARM and x86 architectures are supposed to support the
> stack protector. Maybe you could tell us on what architecture you
> have this issue?

The platform is 32bit x86. GCC version is 4.9.2. I can reliably get the 
behaviour by switching just this single config option on three different 
machines. 

Anything else I can do to help troubleshooting? The kernel actually does not 
print any message to the console even with all debugging information (ipipe 
and xenomai) switched on. Most of the time it just reboots, which seems 
strange to me as is (the TCO watchdog etc is switched off and reboot on panic 
also..). It did hang on some trys also though ... (not fully reproducible)

-- 
Dipl.-Ing. Marc Strämke
Geschäftsführer / CEO

ELTROPULS Anlagenbau GmbH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/e1a124f2/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 04:44:46PM +0200, Marc Strämke wrote:
> Am Donnerstag, 30. April 2015, 16:41:05 schrieb Gilles Chanteperdrix:
> > No. At least ARM and x86 architectures are supposed to support the
> > stack protector. Maybe you could tell us on what architecture you
> > have this issue?
> 
> The platform is 32bit x86. GCC version is 4.9.2. I can reliably get the 
> behaviour by switching just this single config option on three different 
> machines. 

I have no doubt about that. I am just telling you that disabling the
stack protector is not an option.

> 
> Anything else I can do to help troubleshooting? The kernel actually does not 
> print any message to the console even with all debugging information (ipipe 
> and xenomai) switched on. Most of the time it just reboots, which seems 
> strange to me as is (the TCO watchdog etc is switched off and reboot on panic 
> also..). It did hang on some trys also though ... (not fully reproducible)

Is CONFIG_CC_STACKPROTECTOR enabled when you enable the *_REGULAR
option ? The first thing to check is that the xnarch_switch_to
function contains the necessary bits in the kernel ELF image
disassembly (function vmlinux at the top of the kernel build
directory). You should compare this code to the contents of
kernel/cobalt/arch/x86/thread.c in xenomai sources (where you see
some code with #ifdef CONFIG_CC_STACKPROTECTOR).

-- 
					    Gilles.

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Marc Strämke]

Am Donnerstag, 30. April 2015, 16:53:52 schrieb Gilles Chanteperdrix:
> Is CONFIG_CC_STACKPROTECTOR enabled when you enable the *_REGULAR
> option ? The first thing to check is that the xnarch_switch_to

I can confirm that CONFIG_CC_STACKPROTECTOR gets set together with the 
*_REGULAR option. I am inspecting the disassembly now...

-- 
Dipl.-Ing. Marc Strämke
Geschäftsführer / CEO

ELTROPULS Anlagenbau GmbH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/b995ec22/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Marc Strämke]

Am Donnerstag, 30. April 2015, 17:03:54 schrieb Marc Strämke:
> Am Donnerstag, 30. April 2015, 16:53:52 schrieb Gilles Chanteperdrix:
> > Is CONFIG_CC_STACKPROTECTOR enabled when you enable the *_REGULAR
> > option ? The first thing to check is that the xnarch_switch_to
> 
> I can confirm that CONFIG_CC_STACKPROTECTOR gets set together with the
> *_REGULAR option. I am inspecting the disassembly now...

I compared the disassembly of xnarch_switch_to with and without  the option 
set. That does like fine to me (even though i do not fully understand each 
line..). The CANARY_SWITCH gets inserted correctly in to the do_switch_threads 
inline function.

Marc
-- 
Dipl.-Ing. Marc Strämke
Geschäftsführer / CEO

ELTROPULS Anlagenbau GmbH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/a003c92d/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Gilles Chanteperdrix]

On Thu, Apr 30, 2015 at 05:12:09PM +0200, Marc Strämke wrote:
> Am Donnerstag, 30. April 2015, 17:03:54 schrieb Marc Strämke:
> > Am Donnerstag, 30. April 2015, 16:53:52 schrieb Gilles Chanteperdrix:
> > > Is CONFIG_CC_STACKPROTECTOR enabled when you enable the *_REGULAR
> > > option ? The first thing to check is that the xnarch_switch_to
> > 
> > I can confirm that CONFIG_CC_STACKPROTECTOR gets set together with the
> > *_REGULAR option. I am inspecting the disassembly now...
> 
> I compared the disassembly of xnarch_switch_to with and without  the option 
> set. That does like fine to me (even though i do not fully understand each 
> line..). The CANARY_SWITCH gets inserted correctly in to the do_switch_threads 
> inline function.

I guess the next step is to look at what this *_REGULAR option adds
to CONFIG_CC_STACKPROTECTOR in the kernel code, and see if it does
not add something more to switch_to that we should add to
xnarch_switch_to.

-- 
					    Gilles.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 811 bytes
Desc: not available
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/fd9e7886/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Marc Strämke]

Am Donnerstag, 30. April 2015, 17:14:42 schrieb Gilles Chanteperdrix:
> > I compared the disassembly of xnarch_switch_to with and without  the
> > option
> > set. That does like fine to me (even though i do not fully understand each
> > line..). The CANARY_SWITCH gets inserted correctly in to the
> > do_switch_threads inline function.
> 
> I guess the next step is to look at what this *_REGULAR option adds
> to CONFIG_CC_STACKPROTECTOR in the kernel code, and see if it does
> not add something more to switch_to that we should add to
> xnarch_switch_to.

This will probably take a bit of time on my side... 

Thank you for your help Gilles, I might have something more to say or ask next 
week...

Marc

-- 
Dipl.-Ing. Marc Strämke
Geschäftsführer / CEO

ELTROPULS Anlagenbau GmbH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150430/b2642240/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Jan Kiszka]

On 2015-04-30 17:17, Marc Strämke wrote:
> Am Donnerstag, 30. April 2015, 17:14:42 schrieb Gilles Chanteperdrix:
>>> I compared the disassembly of xnarch_switch_to with and without  the
>>> option
>>> set. That does like fine to me (even though i do not fully understand each
>>> line..). The CANARY_SWITCH gets inserted correctly in to the
>>> do_switch_threads inline function.
>>
>> I guess the next step is to look at what this *_REGULAR option adds
>> to CONFIG_CC_STACKPROTECTOR in the kernel code, and see if it does
>> not add something more to switch_to that we should add to
>> xnarch_switch_to.
> 
> This will probably take a bit of time on my side... 
> 
> Thank you for your help Gilles, I might have something more to say or ask next 
> week...

Reproduced and quickly debugged in KVM: The switch code was just broken
regarding argument numbering when the protector is on. This fixes it
(will push a patch soon):

diff --git a/kernel/cobalt/arch/x86/thread.c b/kernel/cobalt/arch/x86/thread.c
index 1b73b0d..2c2574a 100644
--- a/kernel/cobalt/arch/x86/thread.c
+++ b/kernel/cobalt/arch/x86/thread.c
@@ -57,12 +57,12 @@ static inline void do_switch_threads(struct xnarchtcb *out_tcb,
 
 	__asm__ __volatile__("pushfl\n\t"
 			     "pushl %%ebp\n\t"
-			     "movl %6,%%ecx\n\t"
+			     "movl %[spp_out_ptr],%%ecx\n\t"
 			     "movl %%esp,(%%ecx)\n\t"
-			     "movl %7,%%ecx\n\t"
+			     "movl %[ipp_out_ptr],%%ecx\n\t"
 			     "movl $1f,(%%ecx)\n\t"
-			     "movl %8,%%ecx\n\t"
-			     "movl %9,%%edi\n\t"
+			     "movl %[spp_in_ptr],%%ecx\n\t"
+			     "movl %[ipp_in_ptr],%%edi\n\t"
 			     "movl (%%ecx),%%esp\n\t"
 			     "pushl (%%edi)\n\t"
 			     __CANARY_SWITCH
@@ -76,10 +76,10 @@ static inline void do_switch_threads(struct xnarchtcb *out_tcb,
 			       "+a"(outproc),
 			       "+d"(inproc)
 			       __CANARY_OUTPUT
-			     : "m"(out_tcb->spp),
-			       "m"(out_tcb->ipp),
-			       "m"(in_tcb->spp),
-			       "m"(in_tcb->ipp)
+			     : [spp_out_ptr] "m"(out_tcb->spp),
+			       [ipp_out_ptr] "m"(out_tcb->ipp),
+			       [spp_in_ptr] "m"(in_tcb->spp),
+			       [ipp_in_ptr] "m"(in_tcb->ipp)
 			       __CANARY_INPUT
 			     : "memory");
 }

Jan

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: OpenPGP digital signature
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150501/64ff7520/attachment.sig>

Re: [Xenomai] Hang/Reboot when using CONFIG_CC_STACKPROTECTOR_REGULAR

[Marc Strämke]

Am Freitag, 1. Mai 2015, 10:15:31 schrieb Jan Kiszka:
> Reproduced and quickly debugged in KVM: The switch code was just broken
> regarding argument numbering when the protector is on. This fixes it
> (will push a patch soon):
> 

this fixes the problem on my targets also. Thanks for your help.

Marc

-- 
Dipl.-Ing. Marc Strämke
Geschäftsführer / CEO

ELTROPULS Anlagenbau GmbH
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150502/de4701b6/attachment.sig>