[Buildroot] How to get symbolic debugging to work?

[Buildroot] How to get symbolic debugging to work?

[Grant Edwards]

I've given up temporarily on getting thread support to work in
gdbserver.  Now I'm just trying (and failing) to get basic
symbolic debugging to work.

The first problem is how to get enable debug/symbol info in the
libraries (particularly uclibc) on the development host while
still putting stripped ones on the target.  If you enable debug
symbols for uclibc, then I get .so files that are too big for
my target.  I'm working around that problem with a shell script
that strips libuClibc-*.so and libm-*.so and re-builds the
target filesystem images before copying them to the tftpserver.
It works but it's a PITA.

The second problem is that even when I point gdb to uClibc
with debug symbols, gdb acts clueless.  For example, I have a
small C program:

   #include <stdio.h>
   
   char buffer[1024];
   int main(void)
   {
     while (1)
       {
         gets(buffer);
         puts(buffer);
       }
     return 0;
   }

I can debug the program find as long as it's executing in
user-application code.  I can "next" through the code, set
breakpoints, etc.  

However, if I let it run and hit ctrl-C to stop it, it's
blocked in a read() call inside uclibc and gdb acts completely
clueless:

   GNU gdb 6.8
   [...]
   This GDB was configured as "--host=i386-pc-linux-gnu --target=arm-linux-uclibc".
   [New Thread 815]
   0x40001350 in _start () at ldso/ldso/arm/elfinterp.c:332
   332     }
   (gdb) c
   Continuing.
   ^C
   Program received signal SIGINT, Interrupt.
   0x40031914 in ?? ()
   (gdb) bt
   #0  0x40031914 in ?? ()

If I look at the smaps file for the process, it's clearly in
uclibc.so:

   40025000-4008a000 r-xp 00000000 01:00 637         /lib/libuClibc-0.9.30.1.so
   Size:                404 kB
   Rss:                  84 kB
   Pss:                  10 kB
   Shared_Clean:         84 kB
   Shared_Dirty:          0 kB
   Private_Clean:         0 kB
   Private_Dirty:         0 kB
   Referenced:           84 kB
   Swap:                  0 kB

40031914 is offset 0xc914 in /lib/libuClibc-0.9.30.1.so, and
looking at the symbols in that file shows that's in __libc_read()

   0000c5e8 T __libc_pselect
   0000c6d8 T ptrace
   0000c858 T quotactl
   0000c8e0 T __libc_read
==>  0xc914
   0000c968 T readlink
   0000c9f4 T __libc_readv
   0000ca78 T reboot

Why can't gdb figure out that the process is blocked in
__libc_read() and print a proper backtrace?


-- 
Grant Edwards                   grante             Yow! INSIDE, I have the
                                  at               same personality disorder
                               visi.com            as LUCY RICARDO!!

[Buildroot] How to get symbolic debugging to work?

[Grant Edwards]

On 2009-06-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:

> I've given up temporarily on getting thread support to work in
> gdbserver.  Now I'm just trying (and failing) to get basic
> symbolic debugging to work.

Gdb 6.8 just plain doesn't work for ARM targets.

I switched to 6.7.1, and it works fine (including debugging
multiple threaded apps).

-- 
Grant Edwards                   grante             Yow! My polyvinyl cowboy
                                  at               wallet was made in Hong
                               visi.com            Kong by Montgomery Clift!

[Buildroot] How to get symbolic debugging to work?

[Sven Neumann]

Hi,

On Tue, 2009-06-23 at 18:18 +0000, Grant Edwards wrote:
> On 2009-06-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> 
> > I've given up temporarily on getting thread support to work in
> > gdbserver.  Now I'm just trying (and failing) to get basic
> > symbolic debugging to work.
> 
> Gdb 6.8 just plain doesn't work for ARM targets.
> 
> I switched to 6.7.1, and it works fine (including debugging
> multiple threaded apps).

We also have problems with gdb 6.8 on the ARM. Or in other words, it
just plain does not work. I will now follow your advice and switch to
version 6.7.1. If that actually helps, then buildroot should probably
select that version when building for an ARM target (similar as what is
done currently for AVR32).


Sven

[Buildroot] How to get symbolic debugging to work?

[Bernhard Reutner-Fischer]

On Wed, Jun 24, 2009 at 09:59:03AM +0200, Sven Neumann wrote:
>Hi,
>
>On Tue, 2009-06-23 at 18:18 +0000, Grant Edwards wrote:
>> On 2009-06-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>> 
>> > I've given up temporarily on getting thread support to work in
>> > gdbserver.  Now I'm just trying (and failing) to get basic
>> > symbolic debugging to work.
>> 
>> Gdb 6.8 just plain doesn't work for ARM targets.
>> 
>> I switched to 6.7.1, and it works fine (including debugging
>> multiple threaded apps).
>
>We also have problems with gdb 6.8 on the ARM. Or in other words, it
>just plain does not work. I will now follow your advice and switch to
>version 6.7.1. If that actually helps, then buildroot should probably
>select that version when building for an ARM target (similar as what is
>done currently for AVR32).

Alternatively you could check if it works OK with the current gdb and
file a bug either against 6.8 with or CVS

PS: I see that you don't have a required fix for m68k for 6.8, just as a sidenote

[Buildroot] How to get symbolic debugging to work?

[Sven Neumann]

On Tue, 2009-06-23 at 18:18 +0000, Grant Edwards wrote:
> On 2009-06-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> 
> > I've given up temporarily on getting thread support to work in
> > gdbserver.  Now I'm just trying (and failing) to get basic
> > symbolic debugging to work.
> 
> Gdb 6.8 just plain doesn't work for ARM targets.
> 
> I switched to 6.7.1, and it works fine (including debugging
> multiple threaded apps).

I've now tried this change as well, but it doesn't help with our
problem. Whatever I try, gdb always immidiately receives SIGCHLD and
dies:

gdb /usr/bin/image-test 
GNU gdb 6.7.1
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show
copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-uclibcgnueabi"...
Using host libthread_db library "/lib/libthread_db.so.1".
(gdb) r
Starting program: /usr/bin/image-test 
warning: Can not parse XML target description; XML support was disabled
at compile time

[1]+  Stopped(SIGCHLD)        gdb /usr/bin/image-test


That's the same result that I also got with gdb 6.8, except that I
didn't get the warning about the XML target description with that
version.

Our platform is armv5tel with EABI


Sven

[Buildroot] How to get symbolic debugging to work?

[Peter Korsgaard]

>>>>> "Sven" == Sven Neumann <s.neumann@raumfeld.com> writes:

 Sven> We also have problems with gdb 6.8 on the ARM. Or in other
 Sven> words, it just plain does not work. I will now follow your
 Sven> advice and switch to version 6.7.1. If that actually helps,
 Sven> then buildroot should probably select that version when
 Sven> building for an ARM target (similar as what is done currently
 Sven> for AVR32).

Or fix 6.8. I'll take a look at the fedora patches the coming weekend.

-- 
Bye, Peter Korsgaard

[Buildroot] How to get symbolic debugging to work?

[Grant Edwards]

On 2009-06-24, Sven Neumann <s.neumann@raumfeld.com> wrote:
> On Tue, 2009-06-23 at 18:18 +0000, Grant Edwards wrote:
>> On 2009-06-22, Grant Edwards <grant.b.edwards@gmail.com> wrote:
>> 
>>> I've given up temporarily on getting thread support to work in
>>> gdbserver.  Now I'm just trying (and failing) to get basic
>>> symbolic debugging to work.
>> 
>> Gdb 6.8 just plain doesn't work for ARM targets.
>> 
>> I switched to 6.7.1, and it works fine (including debugging
>> multiple threaded apps).
>
> I've now tried this change as well, but it doesn't help with
> our problem. Whatever I try, gdb always immidiately receives
> SIGCHLD and dies:
>
> gdb /usr/bin/image-test 
> GNU gdb 6.7.1
> [...]
> This GDB was configured as "arm-linux-uclibcgnueabi"...

I'm afraid I don't have enough memory to use a target-native
gdb -- I have to use gdbserver on the target and a cross-gdb on
another development host.  IIRC, I'm also using OABI and not
EABI.

> Using host libthread_db library "/lib/libthread_db.so.1".
> (gdb) r
> Starting program: /usr/bin/image-test 
> warning: Can not parse XML target description; XML support was disabled
> at compile time
>
> [1]+  Stopped(SIGCHLD)        gdb /usr/bin/image-test
>
> That's the same result that I also got with gdb 6.8, except that I
> didn't get the warning about the XML target description with that
> version.
>
> Our platform is armv5tel with EABI

You could try using gdbserver, but my _guess_ is it's an
EABI/OABI issue.

-- 
Grant Edwards                   grante             Yow! We're going to a
                                  at               new disco!
                               visi.com            

[Buildroot] How to get symbolic debugging to work?

[Peter Korsgaard]

>>>>> "Bernhard" == Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> writes:

Hi,

 Bernhard> PS: I see that you don't have a required fix for m68k for
 Bernhard> 6.8, just as a sidenote

Which probably means we have very few m68k users. In fact I marked 68k
as broken back around christmas:

config BR2_m68k
        bool "m68k"
        depends on BROKEN # ice in uclibc / inet_ntoa_r

Patches are welcome if someone is interested in supporting m68k.

-- 
Bye, Peter Korsgaard

[Buildroot] How to get symbolic debugging to work?

[Grant Edwards]

On 2009-06-24, Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> wrote:
> On Wed, Jun 24, 2009 at 09:59:03AM +0200, Sven Neumann wrote:

>>> Gdb 6.8 just plain doesn't work for ARM targets.
>>> 
>>> I switched to 6.7.1, and it works fine (including debugging
>>> multiple threaded apps).
>>
>>We also have problems with gdb 6.8 on the ARM. Or in other
>>words, it just plain does not work. I will now follow your
>>advice and switch to version 6.7.1. If that actually helps,
>>then buildroot should probably select that version when
>>building for an ARM target (similar as what is done currently
>>for AVR32).
>
> Alternatively you could check if it works OK with the current
> gdb and file a bug either against 6.8 with or CVS

I'll probably not get around to trying the CVS gdb, so I filed
a bug against 6.8:

  http://sourceware.org/bugzilla/show_bug.cgi?id=10328

If somebody does try out CVS, please add the results to the bug
above.  It might also be useful to know if the same problems
occur when using glibc instead of uclibc, but this is probably
the wrong place to ask for data from glibc users. :)

-- 
Grant Edwards                   grante             Yow! Am I SHOPLIFTING?
                                  at               
                               visi.com            

[Buildroot] How to get symbolic debugging to work?

[Grant Edwards]

On 2009-06-24, Peter Korsgaard <jacmet@uclibc.org> wrote:
>>>>>> "Sven" == Sven Neumann <s.neumann@raumfeld.com> writes:
>
> Sven> We also have problems with gdb 6.8 on the ARM. Or in other
> Sven> words, it just plain does not work. I will now follow your
> Sven> advice and switch to version 6.7.1. If that actually helps,
> Sven> then buildroot should probably select that version when
> Sven> building for an ARM target (similar as what is done currently
> Sven> for AVR32).
>
> Or fix 6.8. I'll take a look at the fedora patches the coming weekend.

I think the underlying problem for my backtrace and SIG32
problems is that gdb (on the dev-host) is failing to figure out
what library files are mmap'ed.  The only one that shows up in
the output of "info files" is ld-uClibc.so.  If you look at
/proc/<pid>/maps on the target, you can see the other library
files (e.g. libgcc_s, libuClibc, libpthread, etc.) that gdb is
unaware of. There's an example of this shown here:

http://news.gmane.org/find-root.php?message_id=%3ch1po1r%24457%241%40ger.gmane.org%3e

-- 
Grant Edwards                   grante             Yow! Mr and Mrs PED, can I
                                  at               borrow 26.7% of the RAYON
                               visi.com            TEXTILE production of the
                                                   INDONESIAN archipelago?

[Buildroot] How to get symbolic debugging to work?

[Bernhard Reutner-Fischer]

On Wed, Jun 24, 2009 at 04:46:32PM +0200, Peter Korsgaard wrote:
>>>>>> "Bernhard" == Bernhard Reutner-Fischer <rep.dot.nop@gmail.com> writes:
>
>Hi,
>
> Bernhard> PS: I see that you don't have a required fix for m68k for
> Bernhard> 6.8, just as a sidenote
>
>Which probably means we have very few m68k users. In fact I marked 68k
>as broken back around christmas:
>
>config BR2_m68k
>        bool "m68k"
>        depends on BROKEN # ice in uclibc / inet_ntoa_r
>
>Patches are welcome if someone is interested in supporting m68k.

worked in my tree, i point any interrested party to factor out the
needed bits and submit it to here..

HTH,

[Buildroot] How to get symbolic debugging to work?

[Sven Neumann]

Hi,

On Wed, 2009-06-24 at 14:45 +0000, Grant Edwards wrote:

> > Our platform is armv5tel with EABI
> 
> You could try using gdbserver, but my _guess_ is it's an
> EABI/OABI issue.

I did try gdbserver, same problem :(  Thanks for your help anyway.


Sven