From: Jesper Christensen <jbc@domain.hid>
To: xenomai@xenomai.org
Subject: Re: [Xenomai-core] kernel threads crash
Date: Tue, 12 Apr 2011 15:31:59 +0200 [thread overview]
Message-ID: <4DA4544F.7020300@domain.hid> (raw)
In-Reply-To: <4DA31EB6.4040909@domain.hid>
I have managed to print the stack of a faulting thread:
Xenomai: suspending kernel thread b911f518 ('rtnet-rtpc') at
nip=0xb911f940, lr=0xb911f940, r1=0xaf2c4580 after exception #1792
Xenomai: dumping stack at af2c4600
Xenomai: 0xaf2c45ec - 0xaf2c45fc: 00000000 af2c4600 8009a334
00000000 00000000
Xenomai: 0xaf2c45d8 - 0xaf2c45e8: 00000000 0 00000000 00000000
00000000
Xenomai: 0xaf2c45c4 - 0xaf2c45d4: b911f518 0 00000000 af2c45f0
8009a364
Xenomai: 0xaf2c45b0 - 0xaf2c45c0: 00000000 0 00000000 00000000
00000000
Xenomai: 0xaf2c459c - 0xaf2c45ac: 00000000 0 00000000 b911f518
8009a334
Xenomai: 0xaf2c4588 - 0xaf2c4598: 00000000 b911f4e0 af2c45d0
b911649c 00000000
Xenomai: 0xaf2c4574 - 0xaf2c4584: 805e3988 8000000 805a89f0
00000001 b911f940
Xenomai: 0xaf2c4560 - 0xaf2c4570: b911f940 20000000 22000022
b911ebb8 00000700
Xenomai: 0xaf2c454c - 0xaf2c455c: 805a89f0 b911f940 00029000
ffffffff 8009b2e4
Xenomai: 0xaf2c4538 - 0xaf2c4548: 00000000 b911ebb8 805e50c4
805e3988 805a89f0
Xenomai: 0xaf2c4524 - 0xaf2c4534: 00100100 ffffffff 00000000
af2c4580 8000bf48
Xenomai: 0xaf2c4510 - 0xaf2c4520: 00000000 0 00000000 00000000
00200200
Xenomai: 0xaf2c44fc - 0xaf2c450c: 805e3988 22000022 00000000
00000000 00000000
Manually decoded link register words:
---------------------------------------------------------------------------------------------------------
8009a334:
$ powerpc-linux-gnu-addr2line -e vmlinux 0x8009a334
linux-2.6.29.6/arch/powerpc/include/asm/xenomai/bits/pod.h:168
8009a364:
$ powerpc-linux-gnu-addr2line -e vmlinux 0x8009a364
linux-2.6.29.6/arch/powerpc/include/asm/xenomai/bits/pod.h:172
b911649c:
$ powerpc-linux-gnu-addr2line -e
../3rd_party/XM-Linux/rtnet_build/stack/rtnet.ko 0x249c
rtnet_build/stack/rtnet_rtpc.c:201
8000bf48:
$ powerpc-linux-gnu-addr2line -e vmlinux 0x8000bf48
linux-2.6.29.6/arch/powerpc/kernel/ipipe.c:429
(ipipe_trigger_irq(unsigned irq) at local_irq_restore_hw(flags);)
---------------------------------------------------------------------------------------------------------
Notice the "r1" register in the first line i assume should point to a
back chain word, but the value is 00000001 and the "link register" word
immediately after is b911f940 which points to:
# grep b911f940 /proc/kallsyms
b911f940 b pending_calls_lock [rtnet]
I'm not sure of the significance of the stack frame after that one.
/Jesper
On 2011-04-11 17:31, Jesper Christensen wrote:
> hmm...
>
> Xenomai: suspending kernel thread b911f518 ('rtnet-rtpc') at
> nip=0x1088860, lr=0x1088862 after exception #1025
>
> LR points to nowhere...Maybe i should do a hexdump of the stack and
> manually decode it.
>
> /Jesper
>
>
> On 2011-04-11 16:49, Jesper Christensen wrote:
>
>> I'll just give them a run and see, thanks!
>>
>> /Jesper
>>
>>
>> On 2011-04-11 16:39, Philippe Gerum wrote:
>>
>>
>>> On Mon, 2011-04-11 at 16:32 +0200, Jesper Christensen wrote:
>>>
>>>
>>>
>>>> How do i see that?
>>>>
>>>>
>>>>
>>>>
>>> diff --git a/include/asm-powerpc/system.h b/include/asm-powerpc/system.h
>>> index 5cc4a23..8dbc537 100644
>>> --- a/include/asm-powerpc/system.h
>>> +++ b/include/asm-powerpc/system.h
>>> @@ -104,7 +104,7 @@ typedef struct xnarch_fltinfo {
>>> #define xnarch_fault_trap(fi) ((unsigned int)(fi)->regs->trap)
>>> #define xnarch_fault_code(fi) ((fi)->regs->dar)
>>> #define xnarch_fault_pc(fi) ((fi)->regs->nip)
>>> -#define xnarch_fault_pc(fi) ((fi)->regs->nip)
>>> +#define xnarch_fault_lr(fi) ((fi)->regs->link)
>>> /* FIXME: FPU faults ignored by the nanokernel on PPC. */
>>> #define xnarch_fault_fpu_p(fi) (0)
>>> /* The following predicates are only usable over a regular Linux stack
>>> diff --git a/ksrc/nucleus/pod.c b/ksrc/nucleus/pod.c
>>> index b5ddbaa..c1722e7 100644
>>> --- a/ksrc/nucleus/pod.c
>>> +++ b/ksrc/nucleus/pod.c
>>> @@ -2591,8 +2591,8 @@ int xnpod_trap_fault(xnarch_fltinfo_t *fltinfo)
>>>
>>> if (!xnpod_userspace_p()) {
>>> xnprintf
>>> - ("suspending kernel thread %p ('%s') at 0x%lx after exception #%u\n",
>>> - thread, thread->name, xnarch_fault_pc(fltinfo),
>>> + ("suspending kernel thread %p ('%s') at nip=0x%lx, lr=0x%lx after exception #%u\n",
>>> + thread, thread->name, xnarch_fault_pc(fltinfo), xnarch_fault_lr(fltinfo),
>>> xnarch_fault_trap(fltinfo));
>>>
>>> xnpod_suspend_thread(thread, XNSUSP, XN_INFINITE, XN_RELATIVE, NULL);
>>>
>>>
>>>
>>>> /Jesper
>>>>
>>>>
>>>> On 2011-04-11 16:27, Philippe Gerum wrote:
>>>>
>>>>
>>>>
>>>>> On Mon, 2011-04-11 at 16:20 +0200, Jesper Christensen wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Problem is the NIP in question is the address of the thread structure as
>>>>>> seen in the error message.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> LR?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> /Jesper
>>>>>>
>>>>>>
>>>>>> On 2011-04-11 16:18, Philippe Gerum wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Mon, 2011-04-11 at 16:13 +0200, Jesper Christensen wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> I have updated to xenomai 2.5.6, but i'm still seeing exceptions
>>>>>>>> (considerably less often though):
>>>>>>>>
>>>>>>>> Xenomai: suspending kernel thread b92a39d0 ('tt_upgw_0') at 0xb92a39d0
>>>>>>>> after exception #1792
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> You should build your code statically into the kernel, not as a module,
>>>>>>> and find out which code raises the MCE.
>>>>>>>
>>>>>>> CONFIG_DEBUG_INFO=y, then objdump -dl vmlinux, looking for the NIP
>>>>>>> mentioned.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> /Jesper
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2011-04-08 15:12, Philippe Gerum wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Fri, 2011-04-08 at 14:58 +0200, Jesper Christensen wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> I'm trying to implement some gateway functionality in the kernel on a
>>>>>>>>>> emerson CPCI6200 board, but have run into some strange errors. The
>>>>>>>>>> kernel module is made up of two threads that run every 1 ms. I have also
>>>>>>>>>> made use of the rtpc dispatcher in rtnet to dispatch control messages
>>>>>>>>>> from a netlink socket to the RT part of my kernel module.
>>>>>>>>>>
>>>>>>>>>> The problem is that when loaded the threads get suspended due to exceptions:
>>>>>>>>>>
>>>>>>>>>> Xenomai: suspending kernel thread b929cbc0 ('tt_upgw_0') at 0xb929cbc0
>>>>>>>>>> after exception #1792
>>>>>>>>>>
>>>>>>>>>> or
>>>>>>>>>>
>>>>>>>>>> Xenomai: suspending kernel thread b929cbc0 ('tt_upgw_0') at 0x0 after
>>>>>>>>>> exception #1025
>>>>>>>>>>
>>>>>>>>>> or
>>>>>>>>>>
>>>>>>>>>> Xenomai: suspending kernel thread b911f518 ('rtnet-rtpc') at 0xb911f940
>>>>>>>>>> after exception #1792
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have ported the "gianfar" driver from linux to rtnet.
>>>>>>>>>>
>>>>>>>>>> The versions and hardware are listed below. The errors are most likely
>>>>>>>>>> due to faulty software on my part, but i would like to ask if there are
>>>>>>>>>> any known issues with the versions or hardware i'm using. I would also
>>>>>>>>>> like to ask if there are any ways of further debugging the errors as i
>>>>>>>>>> am not getting very far with the above messages.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> A severe bug at kthread init was fixed in the 2.5.5.2 - 2.5.6 timeframe,
>>>>>>>>> which would cause exactly the kind of weird behavior you are seeing
>>>>>>>>> right now. The bug triggered random code execution due to stack memory
>>>>>>>>> pollution at init on powerpc for Xenomai kthreads:
>>>>>>>>> http://git.xenomai.org/?p=xenomai-rpm.git;a=commit;h=90699565cbce41f2cec193d57857bb5817efc19a
>>>>>>>>> http://git.xenomai.org/?p=xenomai-rpm.git;a=commit;h=da20c20d4b4d892d40c657ad1d32ddb6d0ceb47c
>>>>>>>>> http://git.xenomai.org/?p=xenomai-rpm.git;a=commit;h=a5886b354dc18f054b187b58cfbacfb60bccaf47
>>>>>>>>>
>>>>>>>>> You need at the very least those three patches (from the top of my
>>>>>>>>> head), but it would be much better to upgrade to 2.5.6.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> System info:
>>>>>>>>>>
>>>>>>>>>> Linux kernel: 2.6.29.6
>>>>>>>>>> i-pipe version: 2.7-04
>>>>>>>>>> processor: powerpc mpc8572
>>>>>>>>>> xenomai version: 2.5.3
>>>>>>>>>> rtnet version: 0.9.12
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Xenomai-core mailing list
>>>>>>>> Xenomai-core@domain.hid
>>>>>>>> https://mail.gna.org/listinfo/xenomai-core
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>> _______________________________________________
>> Xenomai-core mailing list
>> Xenomai-core@domain.hid
>> https://mail.gna.org/listinfo/xenomai-core
>>
>>
>>
>
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
>
>
next prev parent reply other threads:[~2011-04-12 13:31 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-04-08 12:58 [Xenomai-core] kernel threads crash Jesper Christensen
2011-04-08 13:12 ` Philippe Gerum
2011-04-08 13:20 ` Jesper Christensen
2011-04-08 13:39 ` Philippe Gerum
2011-04-08 13:47 ` Jesper Christensen
2011-04-08 14:33 ` Jesper Christensen
2011-04-11 14:13 ` Jesper Christensen
2011-04-11 14:18 ` Philippe Gerum
2011-04-11 14:20 ` Jesper Christensen
2011-04-11 14:27 ` Philippe Gerum
2011-04-11 14:32 ` Jesper Christensen
2011-04-11 14:39 ` Philippe Gerum
2011-04-11 14:49 ` Jesper Christensen
2011-04-11 15:31 ` Jesper Christensen
2011-04-12 13:31 ` Jesper Christensen [this message]
2011-04-12 13:39 ` Gilles Chanteperdrix
2011-04-12 13:40 ` Jesper Christensen
2011-04-12 13:40 ` Jan Kiszka
2011-04-12 13:45 ` Jesper Christensen
2011-04-12 14:09 ` Jesper Christensen
2011-04-12 14:14 ` Jan Kiszka
2011-04-12 14:21 ` Jesper Christensen
2011-04-12 15:24 ` Jan Kiszka
2011-04-12 15:50 ` Jesper Christensen
2011-04-19 7:26 ` Jesper Christensen
2011-04-19 7:39 ` Philippe Gerum
2011-04-19 7:58 ` Jesper Christensen
2011-04-19 8:02 ` Philippe Gerum
2011-04-19 8:42 ` Gilles Chanteperdrix
2011-04-19 9:29 ` Philippe Gerum
2011-04-19 9:30 ` Philippe Gerum
2011-04-19 9:34 ` Jesper Christensen
2011-04-11 14:34 ` Philippe Gerum
2011-04-11 14:35 ` Jesper Christensen
2011-04-11 14:23 ` Philippe Gerum
2011-04-11 14:36 ` Gilles Chanteperdrix
2011-04-08 19:15 ` Richard Cochran
2011-04-11 6:52 ` Jesper Christensen
2011-04-11 6:55 ` Richard Cochran
2011-04-11 6:59 ` Jesper Christensen
2011-04-11 9:18 ` Jan Kiszka
2011-04-11 9:26 ` Jesper Christensen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4DA4544F.7020300@domain.hid \
--to=jbc@domain.hid \
--cc=xenomai@xenomai.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.