[Xenomai-help] Kernel Oops calling rt_dev_close

[Xenomai-help] Kernel Oops calling rt_dev_close

[Steven Kauffmann]

Hi all,

I'm getting Kernel oops when calling rt_dev_close in my application
[*]. The application creates a new real-time task and in this task
rt_dev_recvfrom is waiting for incoming CAN messages.

In the close function of the application I first call rt_dev_close and
after that I kill the thread with rt_task_delete. I also wait for the
termination of this thread with rt_task_join. But when rt_dev_close is
called, the kernel oops occurs.

I'm using the xenomai 2.4.x branch revision 4163 and Linux kernel 2.6.26.3.

Any ideas?

Regards

Steven

[*] Kernel Oops

Xenomai: watchdog triggered -- killing runaway thread 'Read can rtcan0'
Clocksource tsc unstable (delta = 3150076657 ns)
BUG: unable to handle kernel paging request at 00100104
IP: [<f89d159b>] :xeno_can:rtcan_socket_cleanup+0x8e/0xbb
*pde = 00000000
Oops: 0002 [#1] PREEMPT
Modules linked in: xeno_native ext2 dm_mod loop xeno_can_peak_pci
xeno_can_sja1000 xeno_can
intel_agp agpgart xeno_rtdm  ta_piix libata scsi_mod ehci_hcd uhci_hcd
usbcore e1000 ide_cd_mod cdrom

Pid: 3880, comm: unit_test_rtcan Not tainted (2.6.26.3-ipipe-2.0-09 #1)
EIP: 0060:[<f89d159b>] EFLAGS: 00010046 CPU: 0
EIP is at rtcan_socket_cleanup+0x8e/0xbb [xeno_can]
EAX: 00200200 EBX: f7720e28 ECX: 00000000 EDX: 00100100
ESI: 00000001 EDI: 00000000 EBP: 00000000 ESP: f6ecff2c
 DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068
Process unit_test_rtcan (pid: 3880, ti=f6ece000 task=f78163a0
task.ti=f6ece000)<0>
I-pipe domain Linux
Stack: f7720e28 00000000 f7720e10 f89d214a f7720e10 00000001 00000000 f88fc1c3
       f78163a0 00000050 f6ecffb8 00000000 00000018 c0139cfc 00000001 00000000
       00000002 c02e6f04 c02e6f00 c012f638 f6ecffb8 00000021 c02e6f00 c0139c6b
Call Trace:
 [<f89d214a>] rtcan_raw_close+0x78/0x7e [xeno_can]
 [<f88fc1c3>] __rt_dev_close+0x8a/0x154 [xeno_rtdm]
 [<c0139cfc>] losyscall_event+0x91/0x13b
 [<c012f638>] __ipipe_dispatch_event+0x96/0x14e
 [<c0139c6b>] losyscall_event+0x0/0x13b
 [<c0109dc3>] __ipipe_syscall_root+0x6f/0xe7
 [<c0102a59>] system_call+0x29/0x4a
 =======================
Code: 75 c7 85 f6 74 9d 8d 83 24 04 00 00 ba 02 00 00 00 e8 92 c3 f2
ff fa 0f ba 2d c0 08 2a c0 00
19 c0 89 c1 83 e1 01 03 00 01 10 00 c7 43 04 00 02 20 00 a1 c0 08
EIP: [<f89d159b>] rtcan_socket_cleanup+0x8e/0xbb [xeno_can] SS:ESP 0068:f6ecff2c
---[ end trace 6b394a1e918d7480 ]---

Re: [Xenomai-help] Kernel Oops calling rt_dev_close

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

Steven Kauffmann wrote:
> Hi all,
> 
> I'm getting Kernel oops when calling rt_dev_close in my application
> [*]. The application creates a new real-time task and in this task
> rt_dev_recvfrom is waiting for incoming CAN messages.
> 
> In the close function of the application I first call rt_dev_close and
> after that I kill the thread with rt_task_delete. I also wait for the
> termination of this thread with rt_task_join. But when rt_dev_close is
> called, the kernel oops occurs.

Do you have a (small) test case for this issue? Does it work with a
virtual CAN device as well? That would be the fastest way to track down
the issue.

Otherwise, some disassembly of drivers/xenomai/can/rtcan_socket.o would
be nice (objdump -drS). Also, enabling CONFIG_IPIPE_TRACE_PANIC would
enrich the backtrace with some longer function call history. But, again,
local reproducibility would be best.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-help] Kernel Oops calling rt_dev_close

[Steven Kauffmann]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Tue, Sep 2, 2008 at 12:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
> Steven Kauffmann wrote:
>> Hi all,
>>
>> I'm getting Kernel oops when calling rt_dev_close in my application
>> [*]. The application creates a new real-time task and in this task
>> rt_dev_recvfrom is waiting for incoming CAN messages.
>>
>> In the close function of the application I first call rt_dev_close and
>> after that I kill the thread with rt_task_delete. I also wait for the
>> termination of this thread with rt_task_join. But when rt_dev_close is
>> called, the kernel oops occurs.
>
> Do you have a (small) test case for this issue? Does it work with a
> virtual CAN device as well? That would be the fastest way to track down
> the issue.

The test case (see attached file) reproduces the kernel oops.

The kernel oops also occurs when using the virtual CAN device.

> Otherwise, some disassembly of drivers/xenomai/can/rtcan_socket.o would
> be nice (objdump -drS). Also, enabling CONFIG_IPIPE_TRACE_PANIC would
> enrich the backtrace with some longer function call history. But, again,
> local reproducibility would be best.
>
> Jan
>
>

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: rtcan.c --]
[-- Type: text/x-csrc; name=rtcan.c, Size: 1886 bytes --]

#include <rtdm/rtcan.h>
#include <native/task.h>

#include <sys/mman.h>

#define TASK_PRIO  99 /* Highest RT priority */
#define TASK_MODE  T_JOINABLE /* Flags */

RT_TASK task;
RT_TASK read_can_task;

int socket_id;
struct ifreq ifr;
struct sockaddr_can addr;

void read_can() 
{
  struct can_frame frame;
  struct sockaddr_can recvaddr;
  socklen_t addrlen = sizeof(recvaddr);
  while(1)
     rt_dev_recvfrom(socket_id, (void *)&frame, sizeof(can_frame_t), 0,
			       (struct sockaddr *)&recvaddr, &addrlen);
}


int main() {
  int ret;
  
  mlockall(MCL_CURRENT|MCL_FUTURE);

  rt_task_shadow(&task, 0, TASK_PRIO, TASK_MODE);

  socket_id = rt_dev_socket(PF_CAN, SOCK_RAW, CAN_RAW);
  if (socket_id < 0) {
    printf("Cannot open socket, error %d\n", socket_id);
    return -1;
  }

  strcpy(ifr.ifr_name, "rtcan0");

  ret = rt_dev_ioctl(socket_id, SIOCGIFINDEX, &ifr);
  if(ret) {
    printf("rt_dev_ioctl error %d\n", ret);
    return -1;
  }
  
  addr.can_family = AF_CAN;  
  addr.can_ifindex = ifr.ifr_ifindex;
  
  ret = rt_dev_bind(socket_id, (struct sockaddr *)&addr, sizeof(addr));
  if(ret) {
    printf("cannot bind, error %d\n", ret);
    return -1;
  }

  ret = rt_task_create(&read_can_task, "Read can rtcan0", 0, TASK_PRIO, T_JOINABLE);
  if(ret) {
    printf("cannot create task, error %d\n", ret);
    return -1;
  }
  
  ret = rt_task_start(&read_can_task, &read_can, NULL);
  if(ret) {
    printf("Cannot start task, error %d\n", ret);
    return -1;
  }
  
  sleep(2);
  
  ret = rt_dev_close(socket_id);
  if(ret) {
    printf("Cannot close socket, error %d\n", ret);
    return -1;
  }

  ret = rt_task_delete(&read_can_task);
  if(ret) {
    printf("Error while deleting the RT task %d\n", ret);
    return -1;
  }

  ret = rt_task_join(&read_can_task);
  if(ret) {
    printf("Error while joining the RT task %d\n", ret);
    return -1;
  }

  return 0;
}

Re: [Xenomai-help] Kernel Oops calling rt_dev_close

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

Steven Kauffmann wrote:
> On Tue, Sep 2, 2008 at 12:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>> Steven Kauffmann wrote:
>>> Hi all,
>>>
>>> I'm getting Kernel oops when calling rt_dev_close in my application
>>> [*]. The application creates a new real-time task and in this task
>>> rt_dev_recvfrom is waiting for incoming CAN messages.
>>>
>>> In the close function of the application I first call rt_dev_close and
>>> after that I kill the thread with rt_task_delete. I also wait for the
>>> termination of this thread with rt_task_join. But when rt_dev_close is
>>> called, the kernel oops occurs.
>> Do you have a (small) test case for this issue? Does it work with a
>> virtual CAN device as well? That would be the fastest way to track down
>> the issue.
> 
> The test case (see attached file) reproduces the kernel oops.

So it's socket closing while some other tasks blocks in recv.

> 
> The kernel oops also occurs when using the virtual CAN device.

Great, will look into this later today.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Xenomai-help] Kernel Oops calling rt_dev_close

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

Steven Kauffmann wrote:
> On Tue, Sep 2, 2008 at 12:22 PM, Jan Kiszka <jan.kiszka@domain.hid> wrote:
>> Steven Kauffmann wrote:
>>> Hi all,
>>>
>>> I'm getting Kernel oops when calling rt_dev_close in my application
>>> [*]. The application creates a new real-time task and in this task
>>> rt_dev_recvfrom is waiting for incoming CAN messages.
>>>
>>> In the close function of the application I first call rt_dev_close and
>>> after that I kill the thread with rt_task_delete. I also wait for the
>>> termination of this thread with rt_task_join. But when rt_dev_close is
>>> called, the kernel oops occurs.
>> Do you have a (small) test case for this issue? Does it work with a
>> virtual CAN device as well? That would be the fastest way to track down
>> the issue.
> 
> The test case (see attached file) reproduces the kernel oops.
> 
> The kernel oops also occurs when using the virtual CAN device.

Fixed both in SVN HEAD and 2.4.x. I also happen to fix a nasty 64-bit
issue of RTDM at this chance.

But you now have to fix your test case as well: read_can() will enter a
tight endless loop when you close the socket underneath it and don't
check for the error rt_dev_recvfrom returns.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]