[Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Andreas Glatz]

Hi,

I consistently get a Kernel oops when I do a 'cat /proc/xenomai/stats' after killing the Xenomai user-space application which registers for several interrupts.

It's definitely Xenomai related as we are trying to allocate -784bytes of memory (~4GBytes) in nucleus/module.c:stat_seq_open() and we just hv 512MBytes :)

It seems that xnintr_count gets decremented twice when killing the application. The application itself doesn't delete the RT_INTR. We rely on the cleanup which
is done by the xenomai library (?) before exiting the application.

Best regards,
Andreas


That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below the oops):

HUHU count=1 xnintr_count=-12
HUHU stat_seq_open: size=-784 count=-11
------------[ cut here ]------------
Badness at c007f28c [verbose debug info unavailable]
NIP: c007f28c LR: c007ef4c CTR: 00000014
REGS: db309c70 TRAP: 0700   Not tainted  (2.6.32-trunk-8360e)
MSR: 00029032 <EE,ME,CE,IR,DR>  CR: 24002444  XER: 00000000
TASK = db3cc860[3414] 'cat' THREAD: db308000
GPR00: 00000001 db309d20 db3cc860 00000000 00000001 00000000 db309ca8 00000041 
GPR08: 00000041 c0440000 00019ce4 00000000 24002448 1001cb7c 100042bc 100df49c 
GPR16: 00000002 c0380000 c0438434 c0440000 10014b50 db3b8380 000240d0 c0438d2c 
GPR24: 00000000 00000014 00000000 00000000 00000001 00000010 c0438d28 000040d0 
NIP [c007f28c] __alloc_pages_nodemask+0x418/0x4ac
LR [c007ef4c] __alloc_pages_nodemask+0xd8/0x4ac
Call Trace:
[db309d20] [c007ef4c] __alloc_pages_nodemask+0xd8/0x4ac (unreliable)
[db309d90] [c007f33c] __get_free_pages+0x1c/0x4c
[db309da0] [c00a57bc] __kmalloc+0xfc/0x11c
[db309dc0] [c0068d6c] stat_seq_open+0xf8/0x448
[db309e20] [c00ebf14] proc_reg_open+0x74/0x100
[db309e40] [c00a7e24] __dentry_open+0xd0/0x280
[db309e60] [c00b65dc] do_filp_open+0x250/0x874
[db309f10] [c00a7c60] do_sys_open+0x64/0x130
[db309f40] [c0013eec] ret_from_syscall+0x0/0x38
--- Exception: c01 at 0xff508a8
    LR = 0xff50848
Instruction dump:
419dfdb8 8011000c 540b016f 40a2fdac 61d30050 4bfffda4 73e00200 4082fd00 
3d20c044 8009261c 7c000034 5400d97e <0f000000> 2f800000 419efce4 38000001 

Message from syslogd@domain.hid at Wed Feb 10 13:02:26 2010 ...
ruggedcom kernel: ------------[ cut here ]------------
cat: /proc/xenomai/stat: Cannot allocate memory



diff -ruN   ../../../linux-2.6.30-8RR11/kernel/xenomai/nucleus/module.c nucleus/module.c
--- ../../../linux-2.6.30-8RR11/kernel/xenomai/nucleus/module.c	2009-12-08 18:30:05.000000000 -0500
+++ nucleus/module.c	2010-02-10 12:58:23.000000000 -0500
@@ -348,7 +348,7 @@
 	struct seq_file *seq;
 	xnholder_t *holder;
 	struct stat_seq_info *stat_info;
-	int err, count, thrq_rev, intr_rev, irq;
+	int err, count, thrq_rev, intr_rev, irq, size;
 	spl_t s;
 
 	if (!xnpod_active_p())
@@ -373,15 +373,20 @@
 	 * fact later.  Should xnintr_list_rev change later,
 	 * xnintr_query() will trigger an appropriate error below. */
 
+	printk("HUHU count=%d xnintr_count=%d\n", count, xnintr_count);
+
 	intr_rev = xnintr_list_rev;
 	xnarch_memory_barrier();
 	count += xnintr_count * RTHAL_NR_CPUS;
 
 	if (iter)
 		kfree(iter);
-	iter = kmalloc(sizeof(*iter)
-		       + (count - 1) * sizeof(struct stat_seq_info),
-		       GFP_KERNEL);
+
+	size = sizeof(*iter) + (count - 1) * sizeof(struct stat_seq_info);
+	printk("HUHU stat_seq_open: size=%d count=%d\n", size, count);
+
+	iter = kmalloc(size, GFP_KERNEL);
+
 	if (!iter)
 		return -ENOMEM;

Re: [Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Jan Kiszka]

Andreas Glatz wrote:
> Hi,
> 
> I consistently get a Kernel oops when I do a 'cat /proc/xenomai/stats' after killing the Xenomai user-space application which registers for several interrupts.
> 
> It's definitely Xenomai related as we are trying to allocate -784bytes of memory (~4GBytes) in nucleus/module.c:stat_seq_open() and we just hv 512MBytes :)
> 
> It seems that xnintr_count gets decremented twice when killing the application. The application itself doesn't delete the RT_INTR. We rely on the cleanup which
> is done by the xenomai library (?) before exiting the application.
> 
> Best regards,
> Andreas
> 
> 
> That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below the oops):
> 
> HUHU count=1 xnintr_count=-12

That's not good. What is your Xenomai version?

Does everything work fine if you application terminates normally and
deletes the rt_intr objects explicitly?

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Andreas Glatz]

Hi,


>>
>> That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below the oops):
>>
>> HUHU count=1 xnintr_count=-12
>
>That's not good. What is your Xenomai version?

Hardware=MPC8360, Linux=2.6.32, Xenomai=2.4.10, ipipe=2.8-00

>
>Does everything work fine if you application terminates normally and
>deletes the rt_intr objects explicitly?

Checking...


Tnx,
Andreas

Re: [Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Jan Kiszka]

Andreas Glatz wrote:
> Hi,
> 
> 
>>> That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below the oops):
>>>
>>> HUHU count=1 xnintr_count=-12
>> That's not good. What is your Xenomai version?
> 
> Hardware=MPC8360, Linux=2.6.32, Xenomai=2.4.10, ipipe=2.8-00
> 
>> Does everything work fine if you application terminates normally and
>> deletes the rt_intr objects explicitly?
> 
> Checking...
> 

And then you may want to track this down at the site that decrements
xnintr_count, ie. rt_intr_delete -> xnintr_detach.

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Andreas Glatz]

Hi,

>>>> That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below the oops):
>>>>
>>>> HUHU count=1 xnintr_count=-12
>>> That's not good. What is your Xenomai version?
>>
>> Hardware=MPC8360, Linux=2.6.32, Xenomai=2.4.10, ipipe=2.8-00
>>
>>> Does everything work fine if you application terminates normally and
>>> deletes the rt_intr objects explicitly?
>>
>> Checking...
>>
>
> And then you may want to track this down at the site that decrements
> xnintr_count, ie. rt_intr_delete -> xnintr_detach.

This Email contains three patches.

The problem was in rt_intr_delete(): 

First xnintr_detach() was called followed by xnintr_destroy() which in
turn called xnintr_detach() once more. I'm assuming xnintr_destroy()
is the right one to call.

--- A/kernel/xenomai/skins/native/intr.c	2009-12-08 18:30:05.000000000 -0500
+++ B/kernel/xenomai/skins/native/intr.c	2010-02-10 17:30:41.000000000 -0500
@@ -373,8 +373,7 @@
 
 	xnlock_put_irqrestore(&nklock, s);
 
-	err = xnintr_detach(&intr->intr_base);
-	xnintr_destroy(&intr->intr_base);
+	err = xnintr_destroy(&intr->intr_base);
 
 	if (rc == XNSYNCH_RESCHED)
 		/* Some task has been woken up as a result of the deletion:

This change makes xnintr_destroy() return the return value of 
xnintr_detach() as described in the comment. 

--- A/kernel/xenomai/nucleus/intr.c	2009-12-08 18:30:05.000000000 -0500
+++ B/kernel/xenomai/nucleus/intr.c	2010-02-10 17:25:47.000000000 -0500
@@ -638,7 +640,7 @@
  * @param intr The descriptor address of the interrupt object to
  * destroy.
  *
- * @return 0 is returned on success. Otherwise, -EBUSY is returned if
+ * @return 0 is returned on success. Otherwise, -EINVAL is returned if
  * an error occurred while detaching the interrupt (see
  * xnintr_detach()).
  *
@@ -654,8 +656,7 @@
 
 int xnintr_destroy(xnintr_t *intr)
 {
-	xnintr_detach(intr);
-	return 0;
+	return xnintr_detach(intr);
 }
 
 /*!

I also found a minor issue. I think noone ever noticed that because
the second parameter passed to xeno_flush_rq() is always named rq.

--- A/include/xenomai/native/ppd.h	2009-12-08 18:30:05.000000000 -0500
+++ B/include/xenomai/native/ppd.h	2010-02-10 17:36:18.000000000 -0500
@@ -104,7 +104,7 @@
 			if (unlikely(err)) {				\
 				if ((__rq) != &__native_global_rholder.__name##q) { \
 					xnlock_get_irqsave(&nklock, s);	\
-					nholder = popq((rq), holder);	\
+					nholder = popq((__rq), holder);	\
 					appendq(&__native_global_rholder.__name##q, holder); \
 					obj->rqueue = &__native_global_rholder.__name##q; \
 				}


Andreas

Re: [Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Jan Kiszka]

Andreas Glatz wrote:
> Hi,
> 
>>>>> That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below the oops):
>>>>>
>>>>> HUHU count=1 xnintr_count=-12
>>>> That's not good. What is your Xenomai version?
>>> Hardware=MPC8360, Linux=2.6.32, Xenomai=2.4.10, ipipe=2.8-00
>>>
>>>> Does everything work fine if you application terminates normally and
>>>> deletes the rt_intr objects explicitly?
>>> Checking...
>>>
>> And then you may want to track this down at the site that decrements
>> xnintr_count, ie. rt_intr_delete -> xnintr_detach.
> 
> This Email contains three patches.

Good catches! But please post them as separated patches following the
well-know standards (proper subject, a short description, signed-off,
inlined patch against xenomai.git).

> 
> The problem was in rt_intr_delete(): 
> 
> First xnintr_detach() was called followed by xnintr_destroy() which in
> turn called xnintr_detach() once more. I'm assuming xnintr_destroy()
> is the right one to call.
> 
> --- A/kernel/xenomai/skins/native/intr.c	2009-12-08 18:30:05.000000000 -0500
> +++ B/kernel/xenomai/skins/native/intr.c	2010-02-10 17:30:41.000000000 -0500
> @@ -373,8 +373,7 @@
>  
>  	xnlock_put_irqrestore(&nklock, s);
>  
> -	err = xnintr_detach(&intr->intr_base);
> -	xnintr_destroy(&intr->intr_base);
> +	err = xnintr_destroy(&intr->intr_base);

Yep, that detach used to be redundant for a fairly long time.

But the actual bug is in xnintr_irq_attach/detach for
!CONFIG_XENO_OPT_SHIRQ: Those functions fail to maintain and evaluate
XN_ISR_ATTACHED like their big brothers do. So, if you are already at
it... :)

>  
>  	if (rc == XNSYNCH_RESCHED)
>  		/* Some task has been woken up as a result of the deletion:
> 
> This change makes xnintr_destroy() return the return value of 
> xnintr_detach() as described in the comment. 
> 
> --- A/kernel/xenomai/nucleus/intr.c	2009-12-08 18:30:05.000000000 -0500
> +++ B/kernel/xenomai/nucleus/intr.c	2010-02-10 17:25:47.000000000 -0500
> @@ -638,7 +640,7 @@
>   * @param intr The descriptor address of the interrupt object to
>   * destroy.
>   *
> - * @return 0 is returned on success. Otherwise, -EBUSY is returned if
> + * @return 0 is returned on success. Otherwise, -EINVAL is returned if
>   * an error occurred while detaching the interrupt (see
>   * xnintr_detach()).
>   *
> @@ -654,8 +656,7 @@
>  
>  int xnintr_destroy(xnintr_t *intr)
>  {
> -	xnintr_detach(intr);
> -	return 0;
> +	return xnintr_detach(intr);
>  }
>  
>  /*!
> 
> I also found a minor issue. I think noone ever noticed that because
> the second parameter passed to xeno_flush_rq() is always named rq.
> 
> --- A/include/xenomai/native/ppd.h	2009-12-08 18:30:05.000000000 -0500
> +++ B/include/xenomai/native/ppd.h	2010-02-10 17:36:18.000000000 -0500
> @@ -104,7 +104,7 @@
>  			if (unlikely(err)) {				\
>  				if ((__rq) != &__native_global_rholder.__name##q) { \
>  					xnlock_get_irqsave(&nklock, s);	\
> -					nholder = popq((rq), holder);	\
> +					nholder = popq((__rq), holder);	\
>  					appendq(&__native_global_rholder.__name##q, holder); \
>  					obj->rqueue = &__native_global_rholder.__name##q; \
>  				}
> 
> 
> Andreas

Jan

-- 
Siemens AG, Corporate Technology, CT T DE IT 1
Corporate Competence Center Embedded Linux

Re: [Xenomai-help] Kernel oops when reading /proc/xenomai/stats

[Andreas Glatz]

>> Hi,
>>
>>>>>> That's the kernel oops with additional debugging info (starts with 'HUHU' - see code below
>+the oops):
>>>>>>
>>>>>> HUHU count=1 xnintr_count=-12
>>>>> That's not good. What is your Xenomai version?
>>>> Hardware=MPC8360, Linux=2.6.32, Xenomai=2.4.10, ipipe=2.8-00
>>>>
>>>>> Does everything work fine if you application terminates normally and
>>>>> deletes the rt_intr objects explicitly?
>>>> Checking...
>>>>
>>> And then you may want to track this down at the site that decrements
>>> xnintr_count, ie. rt_intr_delete -> xnintr_detach.
>>
>> This Email contains three patches.
>
>Good catches! But please post them as separated patches following the
>well-know standards (proper subject, a short description, signed-off,
>inlined patch against xenomai.git).

Hope the format is ok now.

Andreas