[Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Kolja Waschk]

Hi,

[On Blackfin uClinux 2.6.30 w/v2.5-rc2 with Philippe's patch from
yesterday, trying to run C or C++ FDPIC executables using the POSIX
skin]

When starting the example I recently posted (and a completely different
app, or even just an empty main(){return 0;} too) via "strace" or from
gdbserver, it segfaults almost immediately.

Unfortunately, I'm not able to make gdb break anywhere before this
happens. Therefore, I assume it happens before main().  The bfin trace
output shows that PC when it happened was at
libpthread/linuxthreads.old/signals.c:127 (uClibc __pthread_sigaction?)

These are the last instructions executed before the trap (last first)

    Source : <0xffa00740> { _trap + 0x1c } IF !CC JUMP
10 Target : <0xffa00724> { _trap + 0x0 }
    Source : <0x00457846> [ /lib/libpthread.so.0 + 0x7846 ] 0x9153
11 Target : <0x0045783a> [ /lib/libpthread.so.0 + 0x783a ]
    Source : <0x004578a2> [ /lib/libpthread.so.0 + 0x78a2 ] JUMP.S
12 Target : <0x0045789a> [ /lib/libpthread.so.0 + 0x789a ]
    Source : <0x00457838> [ /lib/libpthread.so.0 + 0x7838 ] IF !CC JUMP
       (libpthread/linuxthreads.old/signals.c:125)
13 Target : <0x004577fc> [ /lib/libpthread.so.0 + 0x77fc ]
       (libpthread/linuxthreads.old/internals.h:410)
    Source : <0xffa00cd6> { __common_int_entry + 0xde } RTI
14 Target : <0xffa00c74> { __common_int_entry + 0x7c }
    Source : <0xffa01070> { _evt_system_call + 0x64 } JUMP.S
15 Target : <0xffa01070> { _evt_system_call + 0x64 }
    Source : <0xffa00944> { _system_call + 0xe8 } RTS

As said, it happens with a source file consisting of only int main()
{return 0;} and a compiler invocation as simple as

  bfin-linux-uclibc-gcc -L/usr/xenomai/lib -lpthread_rt -pthread
     -Wl,@/usr/xenomai/lib/posix.wrappers main.c -o main

Any idea?

Thanks!
Kolja

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Philippe Gerum]

On Fri, 2009-07-24 at 18:13 +0200, Kolja Waschk wrote:
> Hi,
> 
> [On Blackfin uClinux 2.6.30 w/v2.5-rc2 with Philippe's patch from
> yesterday, trying to run C or C++ FDPIC executables using the POSIX
> skin]
> 
> When starting the example I recently posted (and a completely different
> app, or even just an empty main(){return 0;} too) via "strace" or from
> gdbserver, it segfaults almost immediately.
> 
> Unfortunately, I'm not able to make gdb break anywhere before this
> happens. Therefore, I assume it happens before main().  The bfin trace
> output shows that PC when it happened was at
> libpthread/linuxthreads.old/signals.c:127 (uClibc __pthread_sigaction?)
> 
> These are the last instructions executed before the trap (last first)
> 
>     Source : <0xffa00740> { _trap + 0x1c } IF !CC JUMP
> 10 Target : <0xffa00724> { _trap + 0x0 }
>     Source : <0x00457846> [ /lib/libpthread.so.0 + 0x7846 ] 0x9153
> 11 Target : <0x0045783a> [ /lib/libpthread.so.0 + 0x783a ]
>     Source : <0x004578a2> [ /lib/libpthread.so.0 + 0x78a2 ] JUMP.S
> 12 Target : <0x0045789a> [ /lib/libpthread.so.0 + 0x789a ]
>     Source : <0x00457838> [ /lib/libpthread.so.0 + 0x7838 ] IF !CC JUMP
>        (libpthread/linuxthreads.old/signals.c:125)
> 13 Target : <0x004577fc> [ /lib/libpthread.so.0 + 0x77fc ]
>        (libpthread/linuxthreads.old/internals.h:410)
>     Source : <0xffa00cd6> { __common_int_entry + 0xde } RTI
> 14 Target : <0xffa00c74> { __common_int_entry + 0x7c }
>     Source : <0xffa01070> { _evt_system_call + 0x64 } JUMP.S
> 15 Target : <0xffa01070> { _evt_system_call + 0x64 }
>     Source : <0xffa00944> { _system_call + 0xe8 } RTS

Might be a stack related issue, bit no proof of it so far.

> 
> As said, it happens with a source file consisting of only int main()
> {return 0;} and a compiler invocation as simple as
> 
>   bfin-linux-uclibc-gcc -L/usr/xenomai/lib -lpthread_rt -pthread
>      -Wl,@/usr/xenomai/lib/posix.wrappers main.c -o main
> 
> Any idea?

To eliminate toolchain/dist issues, here is the combo that works for me
(commit numbers from the relevant git trees):

blackfin toolchain: ce79f3d5449c78877e66965584c55a132d73d4e1
uclinux-dist: 2137ff7920b51643c5c84536fe99fe7c5904498a
blackfin kernel: 015153eee3851d2c485a85814888f031f3794cf4

How far are you from this setup?

> 
> Thanks!
> Kolja
> 
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Philippe Gerum]

On Fri, 2009-07-24 at 18:30 +0200, Philippe Gerum wrote:
> On Fri, 2009-07-24 at 18:13 +0200, Kolja Waschk wrote:
> > Hi,
> > 
> > [On Blackfin uClinux 2.6.30 w/v2.5-rc2 with Philippe's patch from
> > yesterday, trying to run C or C++ FDPIC executables using the POSIX
> > skin]
> > 
> > When starting the example I recently posted (and a completely different
> > app, or even just an empty main(){return 0;} too) via "strace" or from
> > gdbserver, it segfaults almost immediately.
> > 
> > Unfortunately, I'm not able to make gdb break anywhere before this
> > happens. Therefore, I assume it happens before main().  The bfin trace
> > output shows that PC when it happened was at
> > libpthread/linuxthreads.old/signals.c:127 (uClibc __pthread_sigaction?)
> > 
> > These are the last instructions executed before the trap (last first)
> > 
> >     Source : <0xffa00740> { _trap + 0x1c } IF !CC JUMP
> > 10 Target : <0xffa00724> { _trap + 0x0 }
> >     Source : <0x00457846> [ /lib/libpthread.so.0 + 0x7846 ] 0x9153
> > 11 Target : <0x0045783a> [ /lib/libpthread.so.0 + 0x783a ]
> >     Source : <0x004578a2> [ /lib/libpthread.so.0 + 0x78a2 ] JUMP.S
> > 12 Target : <0x0045789a> [ /lib/libpthread.so.0 + 0x789a ]
> >     Source : <0x00457838> [ /lib/libpthread.so.0 + 0x7838 ] IF !CC JUMP
> >        (libpthread/linuxthreads.old/signals.c:125)
> > 13 Target : <0x004577fc> [ /lib/libpthread.so.0 + 0x77fc ]
> >        (libpthread/linuxthreads.old/internals.h:410)
> >     Source : <0xffa00cd6> { __common_int_entry + 0xde } RTI
> > 14 Target : <0xffa00c74> { __common_int_entry + 0x7c }
> >     Source : <0xffa01070> { _evt_system_call + 0x64 } JUMP.S
> > 15 Target : <0xffa01070> { _evt_system_call + 0x64 }
> >     Source : <0xffa00944> { _system_call + 0xe8 } RTS
> 
> Might be a stack related issue, bit no proof of it so far.
> 
> > 
> > As said, it happens with a source file consisting of only int main()
> > {return 0;} and a compiler invocation as simple as
> > 
> >   bfin-linux-uclibc-gcc -L/usr/xenomai/lib -lpthread_rt -pthread
> >      -Wl,@/usr/xenomai/lib/posix.wrappers main.c -o main
> > 
> > Any idea?
> 
> To eliminate toolchain/dist issues, here is the combo that works for me
> (commit numbers from the relevant git trees):
> 
> blackfin toolchain: ce79f3d5449c78877e66965584c55a132d73d4e1
> uclinux-dist: 2137ff7920b51643c5c84536fe99fe7c5904498a
> blackfin kernel: 015153eee3851d2c485a85814888f031f3794cf4
> 

Forgot to mention:

http://download.gna.org/adeos/patches/v2.6/blackfin/adeos-ipipe-2.6.30.1-blackfin.git-ed2e9479-1.11-00.patch
on top of the blackfin kernel, and
git://xenomai.org/xenomai-head.git
(.../xenomai-2.4.git would do as well, though).

> How far are you from this setup?
> 
> > 
> > Thanks!
> > Kolja
> > 
> > 
> > _______________________________________________
> > Xenomai-help mailing list
> > Xenomai-help@domain.hid
> > https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Philippe Gerum]

On Fri, 2009-07-24 at 18:33 +0200, Philippe Gerum wrote:
> On Fri, 2009-07-24 at 18:30 +0200, Philippe Gerum wrote:
> > On Fri, 2009-07-24 at 18:13 +0200, Kolja Waschk wrote:
> > > Hi,
> > > 
> > > [On Blackfin uClinux 2.6.30 w/v2.5-rc2 with Philippe's patch from
> > > yesterday, trying to run C or C++ FDPIC executables using the POSIX
> > > skin]
> > > 
> > > When starting the example I recently posted (and a completely different
> > > app, or even just an empty main(){return 0;} too) via "strace" or from
> > > gdbserver, it segfaults almost immediately.
> > > 
> > > Unfortunately, I'm not able to make gdb break anywhere before this
> > > happens. Therefore, I assume it happens before main().  The bfin trace
> > > output shows that PC when it happened was at
> > > libpthread/linuxthreads.old/signals.c:127 (uClibc __pthread_sigaction?)
> > > 
> > > These are the last instructions executed before the trap (last first)
> > > 
> > >     Source : <0xffa00740> { _trap + 0x1c } IF !CC JUMP
> > > 10 Target : <0xffa00724> { _trap + 0x0 }
> > >     Source : <0x00457846> [ /lib/libpthread.so.0 + 0x7846 ] 0x9153
> > > 11 Target : <0x0045783a> [ /lib/libpthread.so.0 + 0x783a ]
> > >     Source : <0x004578a2> [ /lib/libpthread.so.0 + 0x78a2 ] JUMP.S
> > > 12 Target : <0x0045789a> [ /lib/libpthread.so.0 + 0x789a ]
> > >     Source : <0x00457838> [ /lib/libpthread.so.0 + 0x7838 ] IF !CC JUMP
> > >        (libpthread/linuxthreads.old/signals.c:125)
> > > 13 Target : <0x004577fc> [ /lib/libpthread.so.0 + 0x77fc ]
> > >        (libpthread/linuxthreads.old/internals.h:410)
> > >     Source : <0xffa00cd6> { __common_int_entry + 0xde } RTI
> > > 14 Target : <0xffa00c74> { __common_int_entry + 0x7c }
> > >     Source : <0xffa01070> { _evt_system_call + 0x64 } JUMP.S
> > > 15 Target : <0xffa01070> { _evt_system_call + 0x64 }
> > >     Source : <0xffa00944> { _system_call + 0xe8 } RTS
> > 
> > Might be a stack related issue, bit no proof of it so far.
> > 
> > > 
> > > As said, it happens with a source file consisting of only int main()
> > > {return 0;} and a compiler invocation as simple as
> > > 
> > >   bfin-linux-uclibc-gcc -L/usr/xenomai/lib -lpthread_rt -pthread
> > >      -Wl,@/usr/xenomai/lib/posix.wrappers main.c -o main
> > > 
> > > Any idea?
> > 
> > To eliminate toolchain/dist issues, here is the combo that works for me
> > (commit numbers from the relevant git trees):
> > 
> > blackfin toolchain: ce79f3d5449c78877e66965584c55a132d73d4e1

Yet another detail: -pg is utterly broken in this release, so you won't
get neither ftrace or the I-pipe tracer working, but the rest looks
fine, and this should be enough for testing your particular issue. I did
not check whether the patch fixing -pg was eventually committed to the
toolchain tree, but it does exist for sure, so there is hope.

> > uclinux-dist: 2137ff7920b51643c5c84536fe99fe7c5904498a
> > blackfin kernel: 015153eee3851d2c485a85814888f031f3794cf4
> > 
> 
> Forgot to mention:
> 
> http://download.gna.org/adeos/patches/v2.6/blackfin/adeos-ipipe-2.6.30.1-blackfin.git-ed2e9479-1.11-00.patch
> on top of the blackfin kernel, and
> git://xenomai.org/xenomai-head.git
> (.../xenomai-2.4.git would do as well, though).
> 
> > How far are you from this setup?
> > 
> > > 
> > > Thanks!
> > > Kolja
> > > 
> > > 
> > > _______________________________________________
> > > Xenomai-help mailing list
> > > Xenomai-help@domain.hid
> > > https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Waschk,Kolja]

Hi,

> blackfin toolchain: ce79f3d5449c78877e66965584c55a132d73d4e1
> uclinux-dist: 2137ff7920b51643c5c84536fe99fe7c5904498a
> blackfin kernel: 015153eee3851d2c485a85814888f031f3794cf4

while I was able to locate that toolchain commit at blackfin.uclinux.org (Jun
14?), I do not seem to know the correct sources for dist and kernel... I
mereley downloaded an uclinux-dist SVN snapshot (rev 8572) and a precompiled
toolchain, probably rev 3515 (20090721) from their Files section, and also just
used the kernel, kernel patch and Xenomai as they came with that snapshot.
Where should I be able to locate those commit numbers? Sorry for asking... the
information about which repository is currently tracked by whom and where is a
little complicated to trace, at least for a newbie in regarding bfin+xenomai.

Kolja

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Philippe Gerum]

On Fri, 2009-07-24 at 19:33 +0200, Waschk,Kolja wrote:
> Hi,
> 
> > blackfin toolchain: ce79f3d5449c78877e66965584c55a132d73d4e1
> > uclinux-dist: 2137ff7920b51643c5c84536fe99fe7c5904498a
> > blackfin kernel: 015153eee3851d2c485a85814888f031f3794cf4
> 
> while I was able to locate that toolchain commit at blackfin.uclinux.org (Jun
> 14?), I do not seem to know the correct sources for dist and kernel... I
> mereley downloaded an uclinux-dist SVN snapshot (rev 8572) and a precompiled
> toolchain, probably rev 3515 (20090721) from their Files section, and also just
> used the kernel, kernel patch and Xenomai as they came with that snapshot.

This is what I used to do, until non backward compatible changes were
introduced circa 2.6.29-2.6.30 in the bootloader and kernel support
which ended up requiring a more recent toolchain and uclinux-dist tree. 

> Where should I be able to locate those commit numbers? Sorry for asking... the
> information about which repository is currently tracked by whom and where is a
> little complicated to trace, at least for a newbie in regarding bfin+xenomai.

You will find the kernel commit mentioned in that tree:
git://git.kernel.org/pub/scm/linux/kernel/git/vapier/blackfin.git

This tree is maintained by Mike Frysinger; this is the source of what is
submitted upstream for inclusion blackfin-wise.

I pull the uclinux-dist code from a git tree mirroring the project's SVN
repo, the related commit number applies there:
git://sources.blackfin.uclinux.org/git/readonly-mirrors/uclinux-dist.git

> 
> Kolja
> 
-- 
Philippe.

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Waschk,Kolja]

Hi,

> git://git.kernel.org/pub/scm/linux/kernel/git/vapier/blackfin.git
> git://sources.blackfin.uclinux.org/git/readonly-mirrors/uclinux-dist.git

Okay, that's where I looked but both returned 403 Unknown (on the web
interface) when I tried the numbers you quoted, i.e.

http://git.kernel.org/?p=linux/kernel/git/vapier/blackfin.git;a=commit;h=015153eee3851d2c485a85814888f031f3794cf4

or

http://blackfin.uclinux.org/git/?p=readonly-mirrors/uclinux-dist.git;a=commit;h=2137ff7920b51643c5c84536fe99fe7c5904498a

but for the tools the number seems valid

https://blackfin.uclinux.org/git/?p=readonly-mirrors/toolchain.git;a=commit;h=ce79f3d5449c78877e66965584c55a132d73d4e1

I'll try again with git locally.

Thanks for all the information

Kolja

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Philippe Gerum]

On Fri, 2009-07-24 at 23:09 +0200, Waschk,Kolja wrote:
> Hi,
> 
> > git://git.kernel.org/pub/scm/linux/kernel/git/vapier/blackfin.git
> > git://sources.blackfin.uclinux.org/git/readonly-mirrors/uclinux-dist.git
> 
> Okay, that's where I looked but both returned 403 Unknown (on the web
> interface) when I tried the numbers you quoted, i.e.
> 
> http://git.kernel.org/?p=linux/kernel/git/vapier/blackfin.git;a=commit;h=015153eee3851d2c485a85814888f031f3794cf4
> 
> or
> 
> http://blackfin.uclinux.org/git/?p=readonly-mirrors/uclinux-dist.git;a=commit;h=2137ff7920b51643c5c84536fe99fe7c5904498a
> 

Mmf, try this one instead, I mistakenly sent you the top commit # from
my local dev branch, the following tags a commit in trunk/ as expected
(June 20) :
2e17eba8a5b0b2ce3cbd57b68bbb04e028f4260a

> but for the tools the number seems valid
> 
> https://blackfin.uclinux.org/git/?p=readonly-mirrors/toolchain.git;a=commit;h=ce79f3d5449c78877e66965584c55a132d73d4e1
> 
> I'll try again with git locally.
> 
> Thanks for all the information
> 
> Kolja
> 
-- 
Philippe.

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Waschk,Kolja]

Hi,

I've finally found a configuration of releases that works for me, good enough
to continue with the actual project development for now. It consists of
Blackfin tools 2009R1-RC10, dist 2009R1-RC2 including kernel and nucleus 2.4.7,
and external Xenomai 2.4.8 in userspace.

It's still not doing all what I expect, but the remaining problems probably are
caused by incorrect memory access in my application and/or driver. The
application runs with POSIX calls and critical threads controlled using the
native API (using rt_task_shadow), or all threads controlled via POSIX skin,
and it is well debuggable. Only when I tried to use the native skin for
everything, I ran into problems this time.

When there's time I'll again take a look at newer versions.

Thanks for the help so far!

Kolja

Re: [Xenomai-help] POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing

[Philippe Gerum]

On Tue, 2009-07-28 at 19:12 +0200, Waschk,Kolja wrote:
> Hi,
> 
> I've finally found a configuration of releases that works for me, good enough
> to continue with the actual project development for now. It consists of
> Blackfin tools 2009R1-RC10, dist 2009R1-RC2 including kernel and nucleus 2.4.7,
> and external Xenomai 2.4.8 in userspace.

Albeit this combo will work since no ABI change may occur between
releases of the stable branch, this does not make much sense to combine
an older kernel support with newer userland libs during the development
stage. You may want to upgrade the Xenomai kernel space support of
2009R1-RC2 to 2.4.8 as well. This must work, or your current combo would
only work by luck and there would very likely still be a problem hiding
somewhere.

> 
> It's still not doing all what I expect, but the remaining problems probably are
> caused by incorrect memory access in my application and/or driver. The
> application runs with POSIX calls and critical threads controlled using the
> native API (using rt_task_shadow), or all threads controlled via POSIX skin,
> and it is well debuggable. Only when I tried to use the native skin for
> everything, I ran into problems this time.
> 

Any details about this? What did not work with GDB precisely (btw, did
you check the stack sizes used to create your native tasks)?

> When there's time I'll again take a look at newer versions.
> 
> Thanks for the help so far!
> 
> Kolja
> 
> 
> 
-- 
Philippe.

[Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Kolja Waschk]

Hi,

Ok, it now seems to run on 2.4.7 (matched kernel + userland) just as
well. I'll try to keep it that way because that's as it comes in the
2009R1 release branch.

>> The application runs with POSIX calls and critical threads controlled
>> using the native API (using rt_task_shadow), or all threads
>> controlled via POSIX skin, and it is well debuggable. Only when I
>> tried to use the native skin for everything, I ran into problems this
>> time.
>
> Any details about this? What did not work with GDB precisely (btw, did
> you check the stack sizes used to create your native tasks)?

Yes, the problems are very probably caused by wrong configuration of various
memory areas, including the task stack(s). I'm not sure how the stacks
and heaps of main, NRT and RT tasks are related and where the memory is
allocated from.

The main thread stack can be set with linker options (-Wl,--defsym,__stacksize=x). 
Is the same memory used if I create a rt task using rt_task_shadow in main?

Where is the memory for rt task stacks taken from?  Does it differ depending on the skin?

Does the Xenomai memory configuration in the kernel limit the rt task stack size?

One of the problems appears when the following code is run via
gdbserver, a breakpoint is hit at main() and I "step" over
rt_task_shadow:

> int main (int argc, char *argv[])
> {
>     RT_TASK maintask;
>     rt_task_shadow(&maintask, "maintask", 50, 0);
>     return 0;
> }

The kernel, with Xenomai debugging enabled, then emits the following:

> Xenomai: fatal: xnshadow_relax() failed for thread maintask[222]
>  CPU  PID    PRI      TIMEOUT  STAT      NAME
>    0  0       50      0        00400088  ROOT
> >  0  222     50      0        00300380  maintask
> Master time base: clock=264885399009

When started without gdb or "cont" without breakpoints set, this doesn't
happen, main returns successfully.

Kolja

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Kolja Waschk]

Hi,

> One of the problems appears when the following code is run via
> gdbserver, a breakpoint is hit at main() and I "step" over
> rt_task_shadow:
>
>> Xenomai: fatal: xnshadow_relax() failed for thread maintask[222]

I just observed the same without gdb in play, with main extended a bit,
creating a second (RT) thread from main (after rt_task_shadow in main).

Xenomai: fatal: xnshadow_relax() failed for thread logger[219]
  CPU  PID    PRI      TIMEOUT  STAT      NAME
    0  0       50      0        00400088  ROOT
    0  217     50      0        00300380  maintask
>  0  219     50      0        00300380  logger
Master time base: clock=74044591006

However, it doesn't happen with the simple example I posted last week.
Maybe I can find some evolutional stage between that example and my
larger application that just causes the failure.

Kolja

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Wed, 2009-07-29 at 13:40 +0200, Kolja Waschk wrote:
> Hi,
> 
> > One of the problems appears when the following code is run via
> > gdbserver, a breakpoint is hit at main() and I "step" over
> > rt_task_shadow:
> >
> >> Xenomai: fatal: xnshadow_relax() failed for thread maintask[222]

This may be an I-pipe issue. As I understand, you are currently running
2009R1-RC2 unmodified, could you summarize where the various Xenomai
related components come from so I could try looking at the issue?

e.g. I-pipe/Adeos patch from uclinux-dist's bfin_patch/adeos_patches/ ?
     Xenomai 2.4.7 support from xenomai.org?

There are two different sources for the I-pipe support for Blackfin, so
it is important to know which one comes into play here.

> 
> I just observed the same without gdb in play, with main extended a bit,
> creating a second (RT) thread from main (after rt_task_shadow in main).
> 
> Xenomai: fatal: xnshadow_relax() failed for thread logger[219]
>   CPU  PID    PRI      TIMEOUT  STAT      NAME
>     0  0       50      0        00400088  ROOT
>     0  217     50      0        00300380  maintask
> >  0  219     50      0        00300380  logger
> Master time base: clock=74044591006
> 
> However, it doesn't happen with the simple example I posted last week.
> Maybe I can find some evolutional stage between that example and my
> larger application that just causes the failure.
> 
> Kolja
-- 
Philippe.

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Waschk,Kolja]

Hi,

> This may be an I-pipe issue. As I understand, you are currently running
> 2009R1-RC2 unmodified, could you summarize where the various Xenomai
> related components come from so I could try looking at the issue?

I'll try: I haven't put in any extra sources, and it seems that
user/xenomai/Makefile indeed refers to ADEOS =
$(ROOTDIR)/bfin_patch/adeos_patch/adeos-bfin.patch

The latest SVN log entry says "update to
http://download.gna.org/adeos/patches/v2.6/blackfin/adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch"
however adeos-bfin.patch in there is not exactly that file.

Also, I use and didn't touch anything inside the dist's
user/xenomai/xenomai-2.4.7 directory which seems to contain
the almost unmodified 2.4.7 sources. Only minor changes regarding an
installation prefix check and asm-blackfin/calibration.h are present. And it
modifies the link order for the test programs. I noticed that last "trick" only
just now; I'll try(*) tomorrow whether linking to (e.g) libnative.la changes
anything over linking with -lnative. Specifying an extra -static anyway didn't
help.  The uclinux-dist build puts the build Xenomai libraries into
uClinux-dist/staging/usr/lib, and that's where my app links to.

*) Because there's one more (strange, to me) observation: If I do not try
"next" over the rt_task_shadow (which is the first line in main()), but "step",
it seems to go directly into "__map_heap_memory" with heap=0 and php=argv[0]
(i.e. the arguments to main()). So it seems it's not in main() but in in
__map_heap_memory?!  In another case, I was able to make the code run further
until a rt_task_create. A breakpoint there and one step, and all of the sudden
gdb says I'm was actually in "rt_heap_create" but with the arguments that were
passed to rt_task_create. It happens that both take similar type and count of
arguments. This may be gdb displaying wrong info or a badly linked FDPIC elf?!

BTW, the toolchain is ADI's 2009R1-RC10 (binary download),
bfin-linux-uclibc-gcc reports version 4.1.2.

Kolja

-- 
Mr. K.Waschk - http://www.ixo.de - http://urjtag.org - Hamburg, DE

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Wed, 2009-07-29 at 22:55 +0200, Waschk,Kolja wrote:
> Hi,
> 
> > This may be an I-pipe issue. As I understand, you are currently running
> > 2009R1-RC2 unmodified, could you summarize where the various Xenomai
> > related components come from so I could try looking at the issue?
> 
> I'll try: I haven't put in any extra sources, and it seems that
> user/xenomai/Makefile indeed refers to ADEOS =
> $(ROOTDIR)/bfin_patch/adeos_patch/adeos-bfin.patch
> 
> The latest SVN log entry says "update to
> http://download.gna.org/adeos/patches/v2.6/blackfin/adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch"
> however adeos-bfin.patch in there is not exactly that file.
> 
> Also, I use and didn't touch anything inside the dist's
> user/xenomai/xenomai-2.4.7 directory which seems to contain
> the almost unmodified 2.4.7 sources. Only minor changes regarding an
> installation prefix check and asm-blackfin/calibration.h are present. And it
> modifies the link order for the test programs. I noticed that last "trick" only
> just now; I'll try(*) tomorrow whether linking to (e.g) libnative.la changes
> anything over linking with -lnative. Specifying an extra -static anyway didn't
> help.  The uclinux-dist build puts the build Xenomai libraries into
> uClinux-dist/staging/usr/lib, and that's where my app links to.
> 
> *) Because there's one more (strange, to me) observation: If I do not try
> "next" over the rt_task_shadow (which is the first line in main()), but "step",
> it seems to go directly into "__map_heap_memory" with heap=0 and php=argv[0]
> (i.e. the arguments to main()). So it seems it's not in main() but in in
> __map_heap_memory?!  In another case, I was able to make the code run further
> until a rt_task_create. A breakpoint there and one step, and all of the sudden
> gdb says I'm was actually in "rt_heap_create" but with the arguments that were
> passed to rt_task_create. It happens that both take similar type and count of
> arguments. This may be gdb displaying wrong info or a badly linked FDPIC elf?!
> 

You seem to be using flat binaries, not ELF, at least if you did not
change anything to the stock Blackfin Makefiles for 2009R1-RC2.

> BTW, the toolchain is ADI's 2009R1-RC10 (binary download),
> bfin-linux-uclibc-gcc reports version 4.1.2.

I have just rebuilt your exact setup, i.e. everything from a stock
2009R1-RC2 which includes Xenomai 2.4.7, with the 09r1-rc10 toolchain.
So far, everything looks fine when running the testsuite.

It seems that you made some changes to the kernel configuration compared
to the default one, since you did activate the Xenomai debug option
which is off by default (this is the one that triggers the
xnshadow_relax() assertion). I would suggest to try the following, in
sequence:

- rebuild your full uclinux-dist stuff based on default settings, still
using the Xenomai code that ships with it; I would not try to tweak
anything before getting a working default system though.

- override the shipped Xenomai release with a stock one:
http://download.gna.org/xenomai/stable/xenomai-2.4.7.tar.bz2
You will have to rebuild the kernel and Xenomai libs/programs aside of
the standard uclinux-dist process, the following should do the trick
(untested though):

$ cd $uclinux-dist-root/linux-2.6.x
$ ~/xenomai-stock/scripts/prepare-kernel.sh --arch=blackfin --linux=.
--forcelink

$ cd /tmp/somewhere
$ ~/xenomai-stock/configure --build=i686-pc-linux-gnu
--host=blackfin-unknown-linux-gnu CC=bfin-linux-uclibc-gcc
CXX=bfin-linux-uclibc-gcc AR=bfin-linux-uclibc-ar
LD=bfin-linux-uclibc-ld --prefix=/usr
$ make DESTDIR=$uclinux-dist-root/staging install
$ cd $uclinux-dist-root && make linux image
$ cp images/uImage /tftpboot/somewhere/

The Xenomai libs/programs will be produced in ELF format, not flat. I
understand this may be an issue for some users, but this is how the
Xenomai project tests them anyway. In any case, you may want to check
whether building for ELF without converting to FLT makes a difference
with your GDB issue.

PS: looking at the toolchain code (09r1-rc10), the mcount bug is still
there in the gcc config frag for blackfin, so don't even bother trying
ftrace or the I-pipe tracer; they would break your kernel during early
boot up.

> 
> Kolja
> 
-- 
Philippe.

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Wed, 2009-07-29 at 22:55 +0200, Waschk,Kolja wrote:
> Hi,
> 
> > This may be an I-pipe issue. As I understand, you are currently running
> > 2009R1-RC2 unmodified, could you summarize where the various Xenomai
> > related components come from so I could try looking at the issue?
> 
> I'll try: I haven't put in any extra sources, and it seems that
> user/xenomai/Makefile indeed refers to ADEOS =
> $(ROOTDIR)/bfin_patch/adeos_patch/adeos-bfin.patch
> 
> The latest SVN log entry says "update to
> http://download.gna.org/adeos/patches/v2.6/blackfin/adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch"
> however adeos-bfin.patch in there is not exactly that file.
> 
> Also, I use and didn't touch anything inside the dist's
> user/xenomai/xenomai-2.4.7 directory which seems to contain
> the almost unmodified 2.4.7 sources. Only minor changes regarding an
> installation prefix check and asm-blackfin/calibration.h are present. And it
> modifies the link order for the test programs. I noticed that last "trick" only
> just now; I'll try(*) tomorrow whether linking to (e.g) libnative.la changes
> anything over linking with -lnative. Specifying an extra -static anyway didn't
> help.  The uclinux-dist build puts the build Xenomai libraries into
> uClinux-dist/staging/usr/lib, and that's where my app links to.
> 
> *) Because there's one more (strange, to me) observation: If I do not try
> "next" over the rt_task_shadow (which is the first line in main()), but "step",
> it seems to go directly into "__map_heap_memory" with heap=0 and php=argv[0]
> (i.e. the arguments to main()). So it seems it's not in main() but in in
> __map_heap_memory?!  In another case, I was able to make the code run further
> until a rt_task_create. A breakpoint there and one step, and all of the sudden
> gdb says I'm was actually in "rt_heap_create" but with the arguments that were
> passed to rt_task_create. It happens that both take similar type and count of
> arguments. This may be gdb displaying wrong info or a badly linked FDPIC elf?!
> 

You seem to be using flat binaries, not ELF, at least if you did not
change anything to the stock Blackfin Makefiles for 2009R1-RC2.

> BTW, the toolchain is ADI's 2009R1-RC10 (binary download),
> bfin-linux-uclibc-gcc reports version 4.1.2.

I have just rebuilt your exact setup, i.e. everything from a stock
2009R1-RC2 which includes Xenomai 2.4.7, with the 09r1-rc10 toolchain.
So far, everything looks fine when running the testsuite.

It seems that you made some changes to the kernel configuration compared
to the default one, since you did activate the Xenomai debug option
which is off by default (this is the one that triggers the
xnshadow_relax() assertion). I would suggest to try the following, in
sequence:

- rebuild your full uclinux-dist stuff based on default settings, still
using the Xenomai code that ships with it; I would not try to tweak
anything before getting a working default system though. If that works,
try to enable the Xenomai nucleus debug option and only this one, to
check whether the xnshadow_relax() issue still triggers.

- if the first option does not work, override the shipped Xenomai
release with a stock one for testing:
http://download.gna.org/xenomai/stable/xenomai-2.4.7.tar.bz2
You will have to rebuild the kernel and Xenomai libs/programs aside of
the standard uclinux-dist process, the following should do the trick
(untested though):

$ cd $uclinux-dist-root/linux-2.6.x
$ ~/xenomai-stock/scripts/prepare-kernel.sh --arch=blackfin --linux=.
--forcelink

$ cd /tmp/somewhere
$ ~/xenomai-stock/configure --build=i686-pc-linux-gnu
--host=blackfin-unknown-linux-gnu CC=bfin-linux-uclibc-gcc
CXX=bfin-linux-uclibc-gcc AR=bfin-linux-uclibc-ar
LD=bfin-linux-uclibc-ld --prefix=/usr
$ make DESTDIR=$uclinux-dist-root/staging install
$ cd $uclinux-dist-root && make linux image
$ cp images/uImage /tftpboot/somewhere/

The Xenomai libs/programs will be produced in ELF format, not flat. I
understand this may be an issue for some users, but this is how the
Xenomai project tests them anyway. In any case, you may want to check
whether building for ELF without converting to FLT makes a difference
with your GDB issue.

PS: looking at the toolchain code (09r1-rc10), the mcount bug is still
there in the gcc config frag for blackfin, so don't even bother trying
ftrace or the I-pipe tracer; they would break your kernel during early
boot up.

> 
> Kolja
> 
-- 
Philippe.

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Kolja Waschk]

> You seem to be using flat binaries, not ELF, at least if you did not
> change anything to the stock Blackfin Makefiles for 2009R1-RC2.

I tried both settings for userland in parallel, each time building 
in an absolutely clean (just untarred) uclinux-dist directory.

> I have just rebuilt your exact setup, i.e. everything from a stock
> 2009R1-RC2 which includes Xenomai 2.4.7, with the 09r1-rc10 toolchain.
> So far, everything looks fine when running the testsuite.

The smallest example that I can make that exhibits at least some of the
strange behaviour also doesn't make any problems when it is just run.
Let me describe the steps to reproduce, leaving as much at default as
possible. I first untar the uClinux-dist-2009R1-RC2.tar.bz2 and make
AnalogDevices/BF537-STAMP_default. When finished, I add Xenomai, using
"make menuconfig" and [X]ing Xenomai under Miscellaneous Applications,
and rebuild. All questions regarding Xenomai configuration when building
the kernel are answered with [Enter] and thus left at their default
value. Finally, I build an example and try to debug it.

The example is built from this source try.c:

#include <stdio.h>
#include <sys/mman.h>
#include <native/task.h>

int main ()
{
   int r;
   RT_TASK maintask;
   mlockall(MCL_CURRENT|MCL_FUTURE);
   r = rt_task_shadow(&maintask, "maintask", 3, 0);
   printf("main:shadow:%d\n", r);
   return 0;
}

Using this command:

bfin-uclinux-gcc -I/opt/uClinux/uClinux-dist/staging/usr/include
-D_GNU_SOURCE -D_REENTRANT -D__XENO__ -o try try.c -Wl,-elf2flt
-L/opt/uClinux/uClinux-dist.flat/staging/usr/lib -lnative -lpthread -lrt

I start it on the BF537-STAMP using "gdbserver localhost:2222 /mnt/try"
and debug from my PC with bfin-uclinux-gdb and this .gdbinit:

file try.gdb
target remote bfin:2222
break main
cont
next

The result is  (on the PC)

> 0x008e0044 in _stext ()
> Breakpoint 1 at 0x8e0188
> [New Thread 252]
> [Switching to Thread 252]
> 
> Breakpoint 1, 0x008e0188 in main ()
> Single stepping until exit from function main, 
> which has no line number information.
>
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x008e71ae in __libc_write (fd=<value optimized out>, buf=0x8f4edc,
> count=20) at libc/sysdeps/linux/common/write.c:15
> 15	libc/sysdeps/linux/common/write.c: No such file or directory.
> 	in libc/sysdeps/linux/common/write.c

And on the BF537-STAMP the displayed result value is incorrect:

> Process /mnt/try created; pid = 252
> ...
> main:shadow:9343576
> Child exited with retcode = 0

Now, I enable Debug support for Xenomai in the Kernel, with Nucleus
Debugging support, but everything else (including Watchdog) disabled.
Without changes in try.c or .gdbinit, the same steps now produce the
error:

Xenomai: fatal: xnshadow_relax() failed for thread maintask[219]
  CPU  PID    PRI      TIMEOUT  STAT      NAME
    0  0        3      0        00400088  ROOT
>  0  219      3      0        00300380  maintask


Next I'll try with stock 2.4.7 Xenomai as you described. Thanks for the
concise guideline!

Greetings from Hamburg
Kolja

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Kolja Waschk]

Hi,

> - if the first option does not work, override the shipped Xenomai
> release with a stock one for testing:

I've now managed to use the prepare-kernel.sh (with
adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch as it comes with
xenomai-2.4.8) on the 2.6.28.10 kernel that comes as "linux-2.6.x" with
uClinux-dist 2009R1-RC2.  Unfortunately, the three files ipipe.h,
ipipe.c and ints-priority.c therein already contain the changes from the
1.10 patch, so to apply that patch, they had to be reverted beforehand.

In userland and for my application I used stock xenomai-2.4.8; only the
xenomai-2.4.x.patch from uClinux-dist/user/xenomai had to be applied
so that /usr can be specified as a destination during configure. But
that patch doesn't change anything in the code.

Now, well, what should I say - the xnshadow call still fails. Or
whatever the real cause for this message is...

Kolja

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Mon, 2009-08-03 at 19:29 +0200, Kolja Waschk wrote:
> Hi,
> 
> > - if the first option does not work, override the shipped Xenomai
> > release with a stock one for testing:
> 
> I've now managed to use the prepare-kernel.sh (with
> adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch as it comes with
> xenomai-2.4.8) on the 2.6.28.10 kernel that comes as "linux-2.6.x" with
> uClinux-dist 2009R1-RC2.  Unfortunately, the three files ipipe.h,
> ipipe.c and ints-priority.c therein already contain the changes from the
> 1.10 patch, so to apply that patch, they had to be reverted beforehand.
> 
> In userland and for my application I used stock xenomai-2.4.8; only the
> xenomai-2.4.x.patch from uClinux-dist/user/xenomai had to be applied
> so that /usr can be specified as a destination during configure. But
> that patch doesn't change anything in the code.
> 
> Now, well, what should I say - the xnshadow call still fails. Or
> whatever the real cause for this message is...

Could you send me your test code? TIA,

> 
> Kolja
> 
-- 
Philippe.

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Mon, 2009-08-03 at 19:34 +0200, Philippe Gerum wrote:
> On Mon, 2009-08-03 at 19:29 +0200, Kolja Waschk wrote:
> > Hi,
> > 
> > > - if the first option does not work, override the shipped Xenomai
> > > release with a stock one for testing:
> > 
> > I've now managed to use the prepare-kernel.sh (with
> > adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch as it comes with
> > xenomai-2.4.8) on the 2.6.28.10 kernel that comes as "linux-2.6.x" with
> > uClinux-dist 2009R1-RC2.  Unfortunately, the three files ipipe.h,
> > ipipe.c and ints-priority.c therein already contain the changes from the
> > 1.10 patch, so to apply that patch, they had to be reverted beforehand.
> > 
> > In userland and for my application I used stock xenomai-2.4.8; only the
> > xenomai-2.4.x.patch from uClinux-dist/user/xenomai had to be applied
> > so that /usr can be specified as a destination during configure. But
> > that patch doesn't change anything in the code.
> > 
> > Now, well, what should I say - the xnshadow call still fails. Or
> > whatever the real cause for this message is...
> 
> Could you send me your test code? TIA,

And the kernel .config file that goes with your latest test triggering
the shadow_relax issue would help as well.

Which bfin board are you running your app on btw?

> 
> > 
> > Kolja
> > 
-- 
Philippe.

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Mon, 2009-08-03 at 19:29 +0200, Kolja Waschk wrote:
> Hi,
> 
> > - if the first option does not work, override the shipped Xenomai
> > release with a stock one for testing:
> 
> I've now managed to use the prepare-kernel.sh (with
> adeos-ipipe-2.6.28.10-blackfin.git-1.10-00.patch as it comes with
> xenomai-2.4.8) on the 2.6.28.10 kernel that comes as "linux-2.6.x" with
> uClinux-dist 2009R1-RC2.  Unfortunately, the three files ipipe.h,
> ipipe.c and ints-priority.c therein already contain the changes from the
> 1.10 patch, so to apply that patch, they had to be reverted beforehand.
> 
> In userland and for my application I used stock xenomai-2.4.8; only the
> xenomai-2.4.x.patch from uClinux-dist/user/xenomai had to be applied
> so that /usr can be specified as a destination during configure. But
> that patch doesn't change anything in the code.
> 
> Now, well, what should I say - the xnshadow call still fails. Or
> whatever the real cause for this message is...
> 

Does this help?

diff --git a/ksrc/arch/blackfin/switch.S b/ksrc/arch/blackfin/switch.S
index 906e5f5..6696a2f 100644
--- a/ksrc/arch/blackfin/switch.S
+++ b/ksrc/arch/blackfin/switch.S
@@ -85,15 +85,19 @@ ENTRY(_rthal_defer_switch_p)
 	r1 = EVT_IVG15 (z);
         r0 = r0 & r1;
 	cc = r0 == 0;
-	if !cc jump 1f; 
+	if !cc jump 1f;
 #endif /* CONFIG_XENO_OPT_PERVASIVE */
 	p2.l = lo(IPEND);
         p2.h = hi(IPEND);
 	csync;
         r2 = [p2];
-        r1 = 1;
-        r1 = r2 - r1;
+        r1 = LO(~0x13) (Z);
         r0 = r2 & r1;
+        cc = r0 == 0;
+        if cc jump 1f;
+        r1 = 1;
+        r1 = r0 - r1;
+        r0 = r0 & r1;
 1:
 	rts
 
@@ -108,4 +112,4 @@ ENTRY(_rthal_nmi_handler)
 	RESTORE_ALL_SYS
 	rtn; 
 		
-#endif /* CONFIG_XENO_HW_NMI_DEBUG_LATENCY_MAX */
+#endif /* CONFIG_XENO_HW_NMI_DEBUG_LATENCY */

> Kolja
> 
-- 
Philippe.

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Waschk,Kolja]

Hi,

> Does this help?
> diff --git a/ksrc/arch/blackfin/switch.S b/ksrc/arch/blackfin/switch.S

Yes! Great - it doesn't just help to step 'next' over the small example, it
also allows me to actually use GDB with my larger app in use cases that
previously always ended with the xnshadow_relax failure. Is it a bugfix or
workaround and how did you actually find it out? - Kolja

Re: [Xenomai-help] Stack sizes etc. (Re: POSIX skin/Blackfin: SIGSEGV when stracing or gdb'ing)

[Philippe Gerum]

On Wed, 2009-08-05 at 22:37 +0200, Waschk,Kolja wrote:
> Hi,
> 
> > Does this help?
> > diff --git a/ksrc/arch/blackfin/switch.S b/ksrc/arch/blackfin/switch.S
> 
> Yes! Great - it doesn't just help to step 'next' over the small example, it
> also allows me to actually use GDB with my larger app in use cases that
> previously always ended with the xnshadow_relax failure. Is it a bugfix or
> workaround

It is an official bug fix, for a blackfin-specific issue.

>  and how did you actually find it out?

Your test case helped reproducing it in a very reliable manner, thanks
for this. The rest was about remembering a wacky aspect of the blackfin
architecture when it comes to implement rescheduling among supervisor
mode tasks: specifically, you just can't do context switching anytime
your OS sees fit, but only when the CPU is not nesting interrupts. This
issue implies a deferral of the Xenomai rescheduling procedure in some
circumstances (uClinux suffers the same pain), and the related piece of
my assembly code ended up being as wacky as the original blackfin issue.
This is what this patch fixes.

>  - Kolja
> 
-- 
Philippe.