[Xenomai-core] xeno task debigging

[Xenomai-core] xeno task debigging

[s.a.]

Hi,
I have got the following kernel message in a process that crashes sometimes
the action to obtain it is console switching with ctrl alt  Fx
the two processes are xeno tasks

any idea about the problem ?

how to debug this in xenomai ?
is there a workaround ?

general protection fault: 0000 [#1]
PREEMPT
Modules linked in: nls_iso8859_1 nls_cp437 i915 rtcan_sja1000 e100
usb_storage eepro100 mii rtc unix
CPU:    0
EIP:    0060:[<c0103112>]    Not tainted VLI
EFLAGS: 00010246   (2.6.16.19-ipipe #1)
EIP is at sysenter_exit+0xf/0x11
eax: 00000010   ebx: 00000000   ecx: b7d1223c   edx: ffffe410
esi: b7d122cc   edi: 00000000   ebp: 00000000   esp: dc1a7fbc
ds: 007b   es: 007b   ss: 0068
Process EnvoiPDO0 (pid: 1879, threadinfo=dc1a6000 task=de35d030)
Stack: <0>00000000 b7d122cc 00000000 b7d122cc 00000000 b7d12268 00000010
0000007b
       0000007b 0802022b ffffe410 00000073 00000206 b7d1223c 0000007b
00000000
       00000000
Call Trace:
Code: 44 24 18 e8 81 ca 03 00 fb 8b 4d 08 66 f7 c1 ff fe 0f 85 4e 01 00
00 e8 ed ea 00 00 8b 44 24 18 8b 54 24 28 8b 4c 24 34 31 ed fb <0f> 35
50 fc 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e da 8e
 <0>general protection fault: 0000 [#2]
PREEMPT
Modules linked in: nls_iso8859_1 nls_cp437 i915 rtcan_sja1000 e100
usb_storage eepro100 mii rtc unix
CPU:    0
EIP:    0060:[<c0103112>]    Not tainted VLI
EFLAGS: 00010246   (2.6.16.19-ipipe #1)
EIP is at sysenter_exit+0xf/0x11
eax: 00000000   ebx: 00000000   ecx: b7ce6a2c   edx: ffffe410
esi: 00669627   edi: 00000000   ebp: 00000000   esp: d704ffbc
ds: 007b   es: 007b   ss: 0068
Process Automat (pid: 1882, threadinfo=d704e000 task=d701b030)
Stack: <0>00000000 00000001 b7d8a660 00669627 00000000 b7ce6a58 00000000
2801007b
       0000007b 0801022b ffffe410 00000073 00000206 b7ce6a2c 0000007b
00000000
       00000000
Call Trace:
Code: 44 24 18 e8 81 ca 03 00 fb 8b 4d 08 66 f7 c1 ff fe 0f 85 4e 01 00
00 e8 ed ea 00 00 8b 44 24 18 8b 54 24 28 8b 4c 24 34 31 ed fb <0f> 35
50 fc 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e da 8e

Re: [Xenomai-core] xeno task debigging

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

s.a. wrote:
> Hi,
> I have got the following kernel message in a process that crashes sometimes
> the action to obtain it is console switching with ctrl alt  Fx
> the two processes are xeno tasks
> 
> any idea about the problem ?

Please tell us your Xenomai and I-ipipe patch version. Also your kernel
.config my be helpful. Do you use any "special" Linux drivers, e.g. some
binary-only nvidia or ati stuff?

> 
> how to debug this in xenomai ?
> is there a workaround ?

Can you derive a simple demo code from your application so that we can
give it a try as well? I have an emulator here (qemu) where I can easily
step into such issues (when they trigger on my box too...).

If you want to dig deeper on your own: there is a kgdb add-on patch
available for 2.6.16 (not 2.6.17 as falsely claimed by the readme) [1].

Jan


[1] http://download.gna.org/adeos/patches/v2.6/i386/kgdb/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 249 bytes --]

Re: [Xenomai-core] xeno task debigging

[Dmitry Adamushko]

Hi,

do you mean that your rt tasks work fine until you press Ctrl-Alt-FN
to switch virtual terminals?

well, it's pretty enough information at least to point out the code
that caused a problem in question :)

it seems to be "sysexit" in entry.S, namely the following block

/* if something modifies registers it must also disable sysexit */
       EMULATE_ROOT_IRET(sysenter_exit)
        movl EIP(%esp), %edx
        movl OLDESP(%esp), %ecx
        xorl %ebp,%ebp
        sti
->     sysexit

Looking at dumped register values :

edx == ffffe410 that stands for some code inside "vsyscall page" -
[fffe000, ffff0000]. I suppose the next instruction after "sysenter".

ecx - a pointer to user-space stack of the task that issued a syscall
through "vsyscall page". it's 0xb7d1223c and at least we may say it
belongs to user-space (but can point to something invalid).

this is exactly what "sysexit" expects to find in edx and ecx, namely
EIP and ESP.

Then let's look at the "code":

Code: 44 24 18 e8 81 ca 03 00 fb 8b 4d 08 66 f7 c1 ff fe 0f 85 4e 01 00
00 e8 ed ea 00 00 8b 44 24 18 8b 54 24 28 8b 4c 24 34 31 ed fb <0f> 35
50 fc 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e da 8e

<0f> 35 seems to be the current instruction and this is an opcode for
"sysexit" on e.g. AMD aa64 (google helps :)

The following also confirms that:
EIP is at sysenter_exit+0xf/0x11

EIP points to last 2 bytes of sysenter_exit and that's "sysexit".

I tried to find what exactly "sysexit" does and when/why it may fault
but failed so far. e.g. if ecx is wrong (points to non-existent vma).

It's also interesting that 2 tasks get GPF, both returning from some syscall.

Maybe Philippe who (I suppose) introduced sysenter/exit support for
Adeos/ipipe can shed some light.


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-core] xeno task debigging

[s.a.]

Yes, it is.
However, this kind of problem happens easily with a Pentium III @ 700Mhz
and never with a celeron @1.3Ghz !



Dmitry Adamushko wrote:
> Hi,
>
> do you mean that your rt tasks work fine until you press Ctrl-Alt-FN
> to switch virtual terminals?
>
> well, it's pretty enough information at least to point out the code
> that caused a problem in question :)
>
> it seems to be "sysexit" in entry.S, namely the following block
>
> /* if something modifies registers it must also disable sysexit */
>       EMULATE_ROOT_IRET(sysenter_exit)
>        movl EIP(%esp), %edx
>        movl OLDESP(%esp), %ecx
>        xorl %ebp,%ebp
>        sti
> ->     sysexit
>
> Looking at dumped register values :
>
> edx == ffffe410 that stands for some code inside "vsyscall page" -
> [fffe000, ffff0000]. I suppose the next instruction after "sysenter".
>
> ecx - a pointer to user-space stack of the task that issued a syscall
> through "vsyscall page". it's 0xb7d1223c and at least we may say it
> belongs to user-space (but can point to something invalid).
>
> this is exactly what "sysexit" expects to find in edx and ecx, namely
> EIP and ESP.
>
> Then let's look at the "code":
>
> Code: 44 24 18 e8 81 ca 03 00 fb 8b 4d 08 66 f7 c1 ff fe 0f 85 4e 01 00
> 00 e8 ed ea 00 00 8b 44 24 18 8b 54 24 28 8b 4c 24 34 31 ed fb <0f> 35
> 50 fc 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e da 8e
>
> <0f> 35 seems to be the current instruction and this is an opcode for
> "sysexit" on e.g. AMD aa64 (google helps :)
>
> The following also confirms that:
> EIP is at sysenter_exit+0xf/0x11
>
> EIP points to last 2 bytes of sysenter_exit and that's "sysexit".
>
> I tried to find what exactly "sysexit" does and when/why it may fault
> but failed so far. e.g. if ecx is wrong (points to non-existent vma).
>
> It's also interesting that 2 tasks get GPF, both returning from some
> syscall.
>
> Maybe Philippe who (I suppose) introduced sysenter/exit support for
> Adeos/ipipe can shed some light.
>
>

Re: [Xenomai-core] xeno task debigging

[Philippe Gerum]

On Fri, 2006-06-30 at 12:32 +0000, s.a. wrote:
> Hi,
> I have got the following kernel message in a process that crashes sometimes
> the action to obtain it is console switching with ctrl alt  Fx
> the two processes are xeno tasks
> 
> any idea about the problem ?
> 
> how to debug this in xenomai ?

The first step is always to check if the same issue occurs using the
standard Xenomai testsuite, e.g. running the latency and switch tests.
Nobody knows what your application is doing and the way it does it,
including the kernel module you seem to be using for that purpose; so
let's properly segregate issues if need be. Try to reproduce the crash
with a freshly patched kernel, without running neither extra modules nor
applications.

> is there a workaround ?
> 
> general protection fault: 0000 [#1]
> PREEMPT
> Modules linked in: nls_iso8859_1 nls_cp437 i915 rtcan_sja1000 e100
> usb_storage eepro100 mii rtc unix
> CPU:    0
> EIP:    0060:[<c0103112>]    Not tainted VLI
> EFLAGS: 00010246   (2.6.16.19-ipipe #1)
> EIP is at sysenter_exit+0xf/0x11
> eax: 00000010   ebx: 00000000   ecx: b7d1223c   edx: ffffe410
> esi: b7d122cc   edi: 00000000   ebp: 00000000   esp: dc1a7fbc
> ds: 007b   es: 007b   ss: 0068
> Process EnvoiPDO0 (pid: 1879, threadinfo=dc1a6000 task=de35d030)
> Stack: <0>00000000 b7d122cc 00000000 b7d122cc 00000000 b7d12268 00000010
> 0000007b
>        0000007b 0802022b ffffe410 00000073 00000206 b7d1223c 0000007b
> 00000000
>        00000000
> Call Trace:
> Code: 44 24 18 e8 81 ca 03 00 fb 8b 4d 08 66 f7 c1 ff fe 0f 85 4e 01 00
> 00 e8 ed ea 00 00 8b 44 24 18 8b 54 24 28 8b 4c 24 34 31 ed fb <0f> 35
> 50 fc 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e da 8e
>  <0>general protection fault: 0000 [#2]
> PREEMPT
> Modules linked in: nls_iso8859_1 nls_cp437 i915 rtcan_sja1000 e100
> usb_storage eepro100 mii rtc unix
> CPU:    0
> EIP:    0060:[<c0103112>]    Not tainted VLI
> EFLAGS: 00010246   (2.6.16.19-ipipe #1)
> EIP is at sysenter_exit+0xf/0x11
> eax: 00000000   ebx: 00000000   ecx: b7ce6a2c   edx: ffffe410
> esi: 00669627   edi: 00000000   ebp: 00000000   esp: d704ffbc
> ds: 007b   es: 007b   ss: 0068
> Process Automat (pid: 1882, threadinfo=d704e000 task=d701b030)
> Stack: <0>00000000 00000001 b7d8a660 00669627 00000000 b7ce6a58 00000000
> 2801007b
>        0000007b 0801022b ffffe410 00000073 00000206 b7ce6a2c 0000007b
> 00000000
>        00000000
> Call Trace:
> Code: 44 24 18 e8 81 ca 03 00 fb 8b 4d 08 66 f7 c1 ff fe 0f 85 4e 01 00
> 00 e8 ed ea 00 00 8b 44 24 18 8b 54 24 28 8b 4c 24 34 31 ed fb <0f> 35
> 50 fc 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e da 8e
> 
> _______________________________________________
> Xenomai-core mailing list
> Xenomai-core@domain.hid
> https://mail.gna.org/listinfo/xenomai-core
-- 
Philippe.