[Buildroot] thread debugging with target gdb

[Buildroot] thread debugging with target gdb

[Thierry Bultel]

Hi,
I am using buildroot 2012.08 , configured for ARM cortex A9, crosstools
and glibc-2.11.
The glibc has been built with 2.6.38.8 headers, my running kernel is
3.2.21, in qemu for versatile express.

The issue is that thread debugging does not work, first because
/lib/libthread_db.so.1 was not pushed on the 'target' directory
I 'solved' this by adding it:

CTNG_LIBS_LIB := ld*.so libc.so libcrypt.so libdl.so libgcc_s.so
libm.so    \
                 libnsl.so libpthread.so libresolv.so librt.so
libutil.so    \
                 libthread_db.so

... in crosstools-ng.mk

but is is maybe not the right way.

I have activated some debug log in gdb, and I am falling into those issues.

Reading symbols from /root/ihm...done.
(gdb) r
Starting program: /root/ihm
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
thread_db_load_search returning 0
Trying host libthread_db library: libthread_db.so.1.
Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
Trying host libthread_db library: /lib/libthread_db.so.1.
td_ta_new failed: application not linked with libthread
thread_db_load_search returning 0
warning: Unable to find libthread_db matching inferior's thread library,
thread debugging will not be available.
Error: Can't open display:
[Inferior 1 (process 612) exited with code 01]

Thus the "td_ta_new" symbol is found but the call fails

Any clues or hints ?
Regards
Thierry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120919/517f2ca4/attachment.html>

[Buildroot] thread debugging with target gdb

[Arnout Vandecappelle]

On 09/19/12 17:44, Thierry Bultel wrote:
> Hi,
> I am using buildroot 2012.08 , configured for ARM cortex A9, crosstools and glibc-2.11.
> The glibc has been built with 2.6.38.8 headers, my running kernel is 3.2.21, in qemu for versatile express.
>
> The issue is that thread debugging does not work, first because /lib/libthread_db.so.1 was not pushed on the 'target'
> directory
> I 'solved' this by adding it:
>
> CTNG_LIBS_LIB := ld*.so libc.so libcrypt.so libdl.so libgcc_s.so libm.so \
> libnsl.so libpthread.so libresolv.so librt.so libutil.so \
> libthread_db.so

  It should probably be:

ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
CTNG_LIBS_LIB += libthread_db.so
endif

  But indeed, it looks like this is missing in crosstools-ng.mk.

  Yann?

>
> ... in crosstools-ng.mk
>
> but is is maybe not the right way.
>
> I have activated some debug log in gdb, and I am falling into those issues.
>
> Reading symbols from /root/ihm...done.
> (gdb) r
> Starting program: /root/ihm
> Trying host libthread_db library: libthread_db.so.1.
> Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.

  Are you running full gdb on the target?  You may have more luck with
gdbserver on the target and output/host/usr/bin/arm-*-gdb on the host.
Don't forget to do 'set sysroot .../output/staging' in gdb.

[snip]

  Regards,
  Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] thread debugging with target gdb

[Thierry Bultel]

Actually I am not currently -debugging- a multithreaded process, but building and configuring a platform
for other developers that will require full gdb.
?
On the web, I have found some contradictory information about the libpthread and libthread_db stripping. 
And moreover:

On my host (Ubuntu) libthread_db is stripped and libpthread is not
In my Buildroot,  libthread_db is not stripped, and libpthread is !!

By reading at the sources of GDB, it does not seem that it uses debug information of 
libthread_db, but only performs dlopen on it, dlsym, and calls "td_ta_new"  
?
I have also seeked into the nptl-db sources, leading to the conclusion that the symbol
containing the NPTL version was not found in libpthread, and it does not get any further.

I really wonder if the crosstools logic for stripping these libraries is in the right way.
I will try to do the opposite, as soon as I get more familiar with the crosstools build.

Of course, if you have any ideas, that would save me a lot of time



Trying gdbserver is my next step, but I feel pessimistic / my conclusions above.

Cheers
Thierry


-----Original message-----
From: Arnout Vandecappelle?<arnout@mind.be>
Sent: Wed 19-Sep-2012 23:04
To: Thierry Bultel <thierry.bultel@basystemes.fr>
Cc: buildroot at busybox.net
Subject: Re: [Buildroot] thread debugging with target gdb


On 09/19/12 17:44, Thierry Bultel wrote:
> Hi,
> I am using buildroot 2012.08 , configured for ARM cortex A9, crosstools and glibc-2.11.
> The glibc has been built with 2.6.38.8 headers, my running kernel is 3.2.21, in qemu for versatile express.
>
> The issue is that thread debugging does not work, first because /lib/libthread_db.so.1 was not pushed on the 'target'
> directory
> I 'solved' this by adding it:
>
> CTNG_LIBS_LIB := ld*.so libc.so libcrypt.so libdl.so libgcc_s.so libm.so \
> libnsl.so libpthread.so libresolv.so librt.so libutil.so \
> libthread_db.so

  It should probably be:

ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
CTNG_LIBS_LIB += libthread_db.so
endif



  But indeed, it looks like this is missing in crosstools-ng.mk.

  Yann?

>
> ... in crosstools-ng.mk
>
> but is is maybe not the right way.
>
> I have activated some debug log in gdb, and I am falling into those issues.
>
> Reading symbols from /root/ihm...done.
> (gdb) r
> Starting program: /root/ihm
> Trying host libthread_db library: libthread_db.so.1.
> Host libthread_db.so.1 resolved to: /lib/libthread_db.so.1.

  Are you running full gdb on the target?  You may have more luck with
gdbserver on the target and output/host/usr/bin/arm-*-gdb on the host.
Don't forget to do 'set sysroot .../output/staging' in gdb.

[snip]

  Regards,
  Arnout

-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be <http://www.mind.be> 
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle <http://www.linkedin.com/in/arnoutvandecappelle> 
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120919/d8b8774b/attachment.html>

[Buildroot] thread debugging with target gdb

[Arnout Vandecappelle]

On 09/19/12 23:27, Thierry Bultel wrote:
>
> Trying gdbserver is my next step, but I feel pessimistic / my conclusions above.

  Most BR developers use gdbserver only, so that's better supported than full gdb on
the target.

  Regards,
  Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] thread debugging with target gdb

[Thierry Bultel]

I finnally solved my problem, and threads debugging now works fine.

The fix consists in 2 things

1) adding libpthreads_db.so on target
Just like you said, this can be achieved in croostools-ng.mk like this:

ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
CTNG_LIBS_LIB += libthread_db.so
endif

But I would also add:

ifeq ($(BR2_PACKAGE_GDB),y)
CTNG_LIBS_LIB += libthread_db.so
endif

since full gdb definitively needs it. Whatever ORing mecanism should maybe be done, not to add the lib twice,
as options are not exclusive.

2) Not stripping libpthread.so

In the top Makefile:

ifeq ($(BR2_PACKAGE_GDB),y)
BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
endif

Thanks for your help, is there a chance that those fixes get integrated in the mainline ?

Cheers 
Thierry



-----Original message-----
From: Arnout Vandecappelle?<arnout@mind.be>
Sent: Wed 19-Sep-2012 23:30
To: Thierry Bultel <thierry.bultel@basystemes.fr>
Cc: buildroot at busybox.net
Subject: Re: [Buildroot] thread debugging with target gdb


On 09/19/12 23:27, Thierry Bultel wrote:
>
> Trying gdbserver is my next step, but I feel pessimistic / my conclusions above.

  Most BR developers use gdbserver only, so that's better supported than full gdb on
the target.

  Regards,
  Arnout
-- 
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be <http://www.mind.be> 
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle <http://www.linkedin.com/in/arnoutvandecappelle> 
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F ifeq ($(BR2_PACKAGE_GDB),y)
> $(warning "THIERRY")
> BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
> endif
 ifeq ($(BR2_PACKAGE_GDB),y)
> $(warning "THIERRY")
> BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
> endif
> ifeq ($(BR2_PACKAGE_GDB),y)
> $(warning "THIERRY")
> BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
> endif


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20120920/bb972315/attachment.html>

[Buildroot] thread debugging with target gdb

[Thomas Petazzoni]

Dear Thierry Bultel,

On Thu, 20 Sep 2012 13:38:55 +0200, Thierry Bultel wrote:

> Thanks for your help, is there a chance that those fixes get integrated in the mainline ?

Yes, sure. Could you send a proper patch? This way we can keep
attribution, and credit you for the investigation of this problem.

Thanks!

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH] Enables thread debugging with full gdb on target

[Thierry Bultel]

The libthread_db was not copied on target
GDB needs libpthread to be not stripped

Signed-off-by: Thierry Bultel <thierry.bultel@wanadoo.fr>
diff -Naur buildroot-2012.08.orig/Makefile buildroot-2012.08/Makefile
--- buildroot-2012.08.orig/Makefile	2012-08-31 10:49:41.000000000 +0200
+++ buildroot-2012.08/Makefile	2012-09-20 16:42:11.938021804 +0200
@@ -416,6 +416,10 @@
  
  $(TARGET_DIR): $(BUILD_DIR)/.root
  
+ifeq ($(BR2_PACKAGE_GDB),y)
+BR2_STRIP_EXCLUDE_FILES += libpthread*.so*
+endif
+
  STRIP_FIND_CMD = find $(TARGET_DIR)
  ifneq (,$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS)))
  STRIP_FIND_CMD += \( $(call finddirclauses,$(TARGET_DIR),$(call qstrip,$(BR2_STRIP_EXCLUDE_DIRS))) \) -prune -o
diff -Naur buildroot-2012.08.orig/toolchain/toolchain-crosstool-ng/crosstool-ng.mk buildroot-2012.08/toolchain/toolchain-crosstool-ng/crosstool-ng.mk
--- buildroot-2012.08.orig/toolchain/toolchain-crosstool-ng/crosstool-ng.mk	2012-08-31 10:49:41.000000000 +0200
+++ buildroot-2012.08/toolchain/toolchain-crosstool-ng/crosstool-ng.mk	2012-09-20 16:41:46.741771907 +0200
@@ -41,6 +41,18 @@
  CTNG_LIBS_LIB := ld*.so libc.so libcrypt.so libdl.so libgcc_s.so libm.so    \
                   libnsl.so libpthread.so libresolv.so librt.so libutil.so
  
+ifeq ($(BR2_PACKAGE_GDB_SERVER),y)
+CTNG_LIBS_LIB_THREAD_DB_NEEDED := y
+endif
+
+ifeq ($(BR2_PACKAGE_GDB),y)
+CTNG_LIBS_LIB_THREAD_DB_NEEDED := y
+endif
+
+ifeq ($(CTNG_LIBS_LIB_THREAD_DB_NEEDED),y)
+CTNG_LIBS_LIB += libthread_db.so
+endif
+
  #--------------
  # The libc-specific system libraries (in /lib)
  # Note: it may be needed to tweak the NSS libs in the glibc and eglibc cases...