[Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Andreas Glatz]

[-- Attachment #1: Type: text/plain, Size: 3156 bytes --]

Hi,

I got a kernel crash because inside xnheap_test_and_free a 
invalid pointer contained in variable 'nextpage' is dereferenced:

<snip>
free_pages:
  /* Mark the released pages as free in the extent's page map. */
  for (pagecont = 0; pagecont < npages; pagecont++)
       extent->pagemap[pagenum + pagecont].type = XNHEAP_PFREE;

  /* Return the sub-list to the free page list, keeping
     an increasing address order to favor coalescence. */

  for (nextpage = extent->freelist, lastpage = NULL;
       nextpage != NULL && nextpage < (caddr_t) block;
       lastpage = nextpage, 
////////////////////////
/* PROBLEM IS HERE => */  nextpage = *((caddr_t *) nextpage))
////////////////////////
  ; /* Loop */
</snip>


This error occurs when running the test application on our 
PowerPC target as well as when running it on the x86 host 
with the newest version of Xenomai (2.4.7).

Target setup:
- Xenomai 2.4.4
- Linux 2.6.26
- PowerPC

Host setup:
- Xenomai 2.4.7
- Linux 2.6.26
- i686

You should be able to confirm my problem with 'rtpipetest', a 
small application of which the source code is attached to this 
Email.

I got the kernel crash after the following sequence of commands (and
the kernel doesn't crash if I DON'T do the 'echo f> /dev/rtp0'): 

rr10:~# ./rtpipetest &
[1] 2568
rr10:~# Info: rt_pipe_stream is full (ret=0)
cat /dev/rtp0
ddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
^C
rr10:~# echo f> /dev/rtp0
rr10:~# kill -s SIGINT 2568
rr10:~# 


This is the error report from our target. The error report
on the host also tells me that the kernel crashed in 
'xnheap_test_and_free'...

Unable to handle kernel paging request for data at address 0x64646464
Faulting instruction address: 0xc0054324
Oops: Kernel access of bad area, sig: 11 [#1]
RC8360 CM
Modules linked in: lm75 max6369_wdt rtc_ds1307
NIP: c0054324 LR: c006e4e4 CTR: 00000000
REGS: df13fd80 TRAP: 0300   Not tainted  (2.6.26-1-8360e)
MSR: 00001032 <ME,IR,DR>  CR: 24002488  XER: 00000000
DAR: 64646464, DSISR: 20000000
TASK = df899ce0[2568] 'main' THREAD: df13e000
GPR00: 00000000 df13fe30 df899ce0 e100e9f8 00000009 00000000 c9b26c9b
00000000 
GPR08: df052240 64646464 00000002 64646464 84004028 1001a6f0 df13ff50
c0392f80 
GPR16: c0375eac ffffffff fffeffff 00000040 00000010 c0360000 00000400
00000001 
GPR24: 00000004 0000000a 00000000 e100e9f8 c0360000 df052240 df052040
df052000 
NIP [c0054324] xnheap_test_and_free+0x2c4/0x3cc
LR [c006e4e4] rt_pipe_delete+0xf0/0x158
Call Trace:
[df13fe30] [c005dbb8] xntimer_start_aperiodic+0x2dc/0x2e4 (unreliable)
[df13fe70] [c006e4e4] rt_pipe_delete+0xf0/0x158
[df13fe90] [c0068d00] __rt_pipe_delete+0x74/0xac
[df13feb0] [c0060c00] hisyscall_event+0x1cc/0x2c4
[df13fee0] [c0051a38] __ipipe_dispatch_event+0x110/0x21c
[df13ff30] [c0009694] __ipipe_syscall_root+0x40/0xe8
[df13ff40] [c0010f44] DoSyscall+0x20/0x5c
--- Exception: c01 at 0xff7ecdc
    LR = 0xff7ecb4
Instruction dump:
5529103a 7d3f4a14 98090004 4200ffe8 813f0010 2f890000 419e0040 7f89f040 
41bc000c 48000034 40980018 7d2b4b78 <81290000> 2f890000 7f09f040
409effec 
---[ end trace 90e6f47d0e66c1c4 ]---

[-- Attachment #2: rtpipetest.c --]
[-- Type: text/x-csrc, Size: 1789 bytes --]

#include <rtdk.h>
#include <native/pipe.h>
#include <native/task.h>
#include <errno.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdlib.h>

static RT_TASK	m_task;
static RT_PIPE	m_pipe;

#define rc_error(fn, ret)	rt_printf("Error: " fn ":%s (%d) %s\n", strerror(-ret), -ret)

int main(void)
{
	int			err;
	const char*	out_str		= "d";
	int			out_str_len	= 1;
	int			in_str_len	= 32;
	char		in_str[in_str_len];

	// Lock pages in memory
	mlockall(MCL_CURRENT|MCL_FUTURE);

	// Init rtdk framework for rt_printf
	rt_print_auto_init(1);

	// Add rt shadow
	err	= rt_task_shadow(&m_task, "main", 22, 0);
	if(err) {
		rc_error("rt_task_shadow", err);
		return err;
	}

	// Create pipe
	err	= rt_pipe_create(&m_pipe, "rtp0", 0, 2048);
	if(err) {
		rc_error("rt_pipe_create", err);
		goto cleanup;
	}

	// Deliberately fill pipe without a reader on
	// the other side...
	while(1) {
		err	= rt_pipe_stream(&m_pipe, out_str, out_str_len);
		// Check if there was an error
		if(err < 0) {
			rc_error("rt_pipe_stream", err);
			goto cleanup;
		}
		// Check if all bytes where written to the pipe
		if(err != out_str_len) {
			rt_printf("Info: rt_pipe_stream is full (ret=%d)\n", err);
			break;
		}
	}

	// Wait for the user to connect to the pipe and 
	// loop until we are able to read a byte
	while(1) {
		err	= rt_pipe_read(&m_pipe, in_str, in_str_len, TM_NONBLOCK);
		if(err < 0 && err != -EAGAIN) {
			rc_error("rt_pipe_read", err);
			goto cleanup;
		}
		// Check if we received something	
		if(err > 0) {
			rt_printf("Received: %s\n", in_str);
			break;
		}

		// Wait 1ms
		rt_task_sleep(1000000);
	}

cleanup:
	rt_pipe_delete(&m_pipe);
	rt_task_delete(&m_task);
	return err;
}

Re: [Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Andreas Glatz]

[-- Attachment #1: Type: text/plain, Size: 137 bytes --]


The last source code I sent is outdated.

The newest version with a sighandler for SIGINT is attached here.

Sorry for that,

Andreas



[-- Attachment #2: rtpipetest.c --]
[-- Type: text/x-csrc, Size: 1958 bytes --]

#include <rtdk.h>
#include <native/pipe.h>
#include <native/task.h>
#include <errno.h>
#include <string.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdlib.h>
#include <signal.h>

static RT_TASK			m_task;
static RT_PIPE			m_pipe;
static volatile int		cont	= 1;

void sig_hdl(int sig) 
{
	cont	= 0;
}


#define rc_error(fn, ret)	rt_printf("Error: " fn ":%s (%d) %s\n", strerror(-ret), -ret)

int main(void)
{
	int			err;
	const char*	out_str		= "d";
	int			out_str_len	= 1;
	int			in_str_len	= 32;
	char		in_str[in_str_len];

	// Lock pages in memory
	mlockall(MCL_CURRENT|MCL_FUTURE);

	// Init rtdk framework for rt_printf
	rt_print_auto_init(1);

	// Catch SIG_INT
	signal(SIGINT, sig_hdl);

	// Add rt shadow
	err	= rt_task_shadow(&m_task, "main", 22, 0);
	if(err) {
		rc_error("rt_task_shadow", err);
		return err;
	}

	// Create pipe
	err	= rt_pipe_create(&m_pipe, "rtp0", 0, 2048);
	if(err) {
		rc_error("rt_pipe_create", err);
		goto cleanup;
	}

	// Deliberately fill pipe without a reader on
	// the other side...
	while(1) {
		err	= rt_pipe_stream(&m_pipe, out_str, out_str_len);
		// Check if there was an error
		if(err < 0) {
			rc_error("rt_pipe_stream", err);
			goto cleanup;
		}
		// Check if all bytes where written to the pipe
		if(err != out_str_len) {
			rt_printf("Info: rt_pipe_stream is full (ret=%d)\n", err);
			break;
		}
	}

	// Wait for the user to connect to the pipe and 
	// loop until we are able to read a byte
	while(cont) {
		err	= rt_pipe_read(&m_pipe, in_str, in_str_len, TM_NONBLOCK);
		if(err < 0 && err != -EAGAIN) {
			rc_error("rt_pipe_read", err);
			goto cleanup;
		}
		// Check if we received something	
		if(err >= 0) {
			rt_printf("Received: %d bytes\n", err);
			break;
		}

		// Wait 1ms
		rt_task_sleep(1000000);
	}
	err	= 0;

cleanup:
	rt_pipe_delete(&m_pipe);
	rt_task_delete(&m_task);
	return err;
}

Re: [Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Philippe Gerum]

Andreas Glatz wrote:
> The last source code I sent is outdated.
> 
> The newest version with a sighandler for SIGINT is attached here.
>

Confirmed, it's perfectly reproducible here as well. Will dig this. Thanks for 
reporting.

> Sorry for that,
> 
> Andreas
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core


-- 
Philippe.

Re: [Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Philippe Gerum]

Andreas Glatz wrote:
> Hi,
> 
> I got a kernel crash because inside xnheap_test_and_free a 
> invalid pointer contained in variable 'nextpage' is dereferenced:
>

<snip>

This turned out to be caused by an out-of-bound write triggered by the streaming 
output service.

The patch below fixes the issue; it has been committed to both the maintenance 
(v2.4.x) and development branches.

Sidenote: your test scenario involves echoing some data to /dev/rtp0 for 
triggering the issue; this will now work, but you won't get that input available 
to rt_pipe_read(). In case you wonder why, the reason is that 'echo' will exit 
immediately after sending the bytes, which will cause the user-space side of the 
channel to be closed, and the input queue (the one that goes user -> kernel) to 
be flushed from any pending data.

--- ksrc/skins/native/pipe.c	(revision 4712)
+++ ksrc/skins/native/pipe.c	(working copy)
@@ -110,6 +110,7 @@
  		/* Reset the streaming buffer. */
  		xnlock_get_irqsave(&nklock, s);
  		pipe->fillsz = 0;
+		xnpipe_m_size(pipe->buffer) = 0;
  		__clear_bit(P_SYNCWAIT, &pipe->status);
  		__clear_bit(P_ATOMIC, &pipe->status);
  		xnlock_put_irqrestore(&nklock, s);
@@ -284,8 +285,8 @@
  				       NULL);
  		return -ENOMEM;
  	}
-	inith(&pipe->buffer->link);
-	pipe->buffer->size = streamsz - sizeof(RT_PIPE_MSG);
+	inith(xnpipe_m_link(pipe->buffer));
+	xnpipe_m_size(pipe->buffer) = streamsz - sizeof(RT_PIPE_MSG);
  #endif /* CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ > 0 */

  	ops.output = NULL;
@@ -881,8 +882,10 @@
  		goto unlock_and_exit;
  	}

-	if (size > CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ - pipe->fillsz)
-		outbytes = CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ - pipe->fillsz;
+	if (size > CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ
+	    - sizeof(RT_PIPE_MSG) - pipe->fillsz)
+		outbytes = CONFIG_XENO_OPT_NATIVE_PIPE_BUFSZ
+			- sizeof(RT_PIPE_MSG) - pipe->fillsz;
  	else
  		outbytes = size;


-- 
Philippe.

Re: [Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Philippe Gerum]

Philippe Gerum wrote:
> Andreas Glatz wrote:
>> Hi,
>>
>> I got a kernel crash because inside xnheap_test_and_free a 
>> invalid pointer contained in variable 'nextpage' is dereferenced:
>>
> 
> <snip>
> 
> This turned out to be caused by an out-of-bound write triggered by the streaming 
> output service.
> 
> The patch below fixes the issue; it has been committed to both the maintenance 
> (v2.4.x) and development branches.
> 
> Sidenote: your test scenario involves echoing some data to /dev/rtp0 for 
> triggering the issue; this will now work, but you won't get that input available 
> to rt_pipe_read(). In case you wonder why, the reason is that 'echo' will exit 
> immediately after sending the bytes, which will cause the user-space side of the 
> channel to be closed, and the input queue (the one that goes user -> kernel) to 
> be flushed from any pending data.
> 

...unless your polling RT read loop wakes up at the right time and manages to 
preempt the Linux kernel shortly after the echo sent the bytes, in which case 
you will receive the data, but that is obviously not the most frequent situation.

-- 
Philippe.

Re: [Xenomai-core] Kernel crash in xnheap_test_and_free (native/heap.c)

[Andreas Glatz]

On Wed, 2009-03-18 at 23:01 +0100, Philippe Gerum wrote:
> Philippe Gerum wrote:
> > Andreas Glatz wrote:
> >> Hi,
> >>
> >> I got a kernel crash because inside xnheap_test_and_free a 
> >> invalid pointer contained in variable 'nextpage' is dereferenced:
> >>
> > 
> > <snip>
> > 
> > This turned out to be caused by an out-of-bound write triggered by the streaming 
> > output service.
> > 
> > The patch below fixes the issue; it has been committed to both the maintenance 
> > (v2.4.x) and development branches.

Great! This fixes the bug! Many thanks!


> > 
> > Sidenote: your test scenario involves echoing some data to /dev/rtp0 for 
> > triggering the issue; this will now work, but you won't get that input available 
> > to rt_pipe_read(). In case you wonder why, the reason is that 'echo' will exit 
> > immediately after sending the bytes, which will cause the user-space side of the 
> > channel to be closed, and the input queue (the one that goes user -> kernel) to 
> > be flushed from any pending data.
> > 
> 
> ...unless your polling RT read loop wakes up at the right time and manages to 
> preempt the Linux kernel shortly after the echo sent the bytes, in which case 
> you will receive the data, but that is obviously not the most frequent situation.
> 

Actually we don't use echo or cat to write to/read from the pipe. I just
used it to describe the failure. We are using a patched version of
minicom to read from/write to the pipe. Probably you know that you can
use minicom to connect to a unix-socket, I just went ahead and patched
it so that you can connect to a named pipe.

Anyways,

Thanks a lot!

Andreas