Re: [Xenomai-help] Debugging in Xenomai

[Jan Kiszka <jan.kiszka@domain.hid>]

[-- Attachment #1: Type: text/plain, Size: 3270 bytes --]

Daniel Schnell wrote:
> Hi,
>  
> our application deadlocks from time to time. 
>  
> To isolate which of the threads are deadlocked, I look into
> /proc/xenomai/stat. Here I can see if there are context switches
> happening or not. Further I can see which threads have which PID and
> which task state they are into. One of the task states that is not clear
> to me btw is the state "X" or "relaxed shadow". Anybody ?

Thread is pending in secondary mode on some Linux resource.

>  
> So after gathering that I have some basic idea which of the threads is
> having a deadlock and now the question is, how to continue ?
>  
> One of the next steps would be finding out which actual function back
> trace the suspicious thread has. So I execute gdb and try to attach to
> the appropriate process, which works. Problem: sending Ctrl-C doesn't
> work, independant of if gdb is executed via ssh or serial console. So I
> cannot stop the actual program beeing debugged, rendering the gdb
> approach useless. Also sending SIGINT to the GDB process doesn't work.
> It seems to be simply ignored. As I understand CTRL-C is effectively
> sending SIGINT and is sent to GDB itself and not to the underlying appl.

I would say there is a bug, either in gdb or in Xenomai. Does debugging
in general work on your box, i.e. without Xenomai? Does it fail with any
trivial Xenomai application? What version? On PPC? If there is a bug
/wrt Xenomai, it has to be fixed.

>  
> Of course I always can insert logger output strings. I even wrote a
> logger that doesn't force to switch the process into secondary mode. I
> just thought we might have something that gives a developer a more
> effective way of fastly finding out which task has which actual back
> trace.

Already tried strace? It may help to understand more if your thread
hangs in secondary mode.

Beyond that, we are all desperately waiting for LTTng... :)

>  
> Under VxWorks there is a very elegant way of debugging with the help of
> the userLib. Of course this is possible here because VxWorks makes no
> distinction between user and kernel space and the shell itself is
> running as a task having access to symbol tables etc. You can say "i"
> which gives a similar output as /proc/xenomai/stat but with much more
> informations like stack size, actual stack consumption, actual executed
> function pointer, etc. "b" gives you a global breakpoint for any
> function. One can get stack traces without the help of gdb via "tt",
> etc. etc.
>  
> In Xenomai I suppose the actual stack execution pointer, stack size,
> etc. of a task has to be available to the kernel as well, so
> principially it should be possible to provide more informations than
> what /proc/xenomai/stat or /proc/xenomai/sched provide. Those migrating
> from VxWorks (and many others probably as well) would certainly like
> that.
>  
> How do the Xenomai veterans debug their application in case of deadlocks
> / synchronization errors, crashes etc. ?

Log message instrumentation (we also have such a RT-safe mechanism),
meditating on the code (better: let someone else meditate that looks at
it from a different angle), gdb (at least on x86 we had no problems so far).

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]