[PATCH 0/1] GDB Enhancements for device threaded core support

[PATCH 0/1] GDB Enhancements for device threaded core support

[blloyd]

core file generation can create a broken core file when threading is used in the application.  This change provides updates to the gdb package so multi-threaded applications debugging from failures works properly from end devices that include gdb.

The following changes since commit 523f2a9ea970713fb775bc48f84b67420b1106a0:

  cross-canadian: Improve comment (2013-10-30 18:01:34 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib blloyd/gdb_assist
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=blloyd/gdb_assist

blloyd (1):
  Enhancements to allow core file debugging to work properly from
    devices

 meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
 meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
 2 files changed, 15 insertions(+), 6 deletions(-)

-- 
1.8.1.4

[PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[blloyd]

gdb includes a script to make generating core files trivial.  Add a package to allow inclusion of this script in images.  GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump.  Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.

Signed-off-by: blloyd <blloyd@familyhonor.net>
---
 meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
 meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
 2 files changed, 15 insertions(+), 6 deletions(-)

diff --git a/meta/recipes-devtools/gdb/gdb-common.inc b/meta/recipes-devtools/gdb/gdb-common.inc
index 6954954..02eef85 100644
--- a/meta/recipes-devtools/gdb/gdb-common.inc
+++ b/meta/recipes-devtools/gdb/gdb-common.inc
@@ -61,8 +61,10 @@ do_install_append() {
 	rm -rf ${D}${datadir}/locale
 }
 
-RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
-RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
-
+# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
+# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
+RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
+RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
+RDEPENDS_gcore_append = " gdb "
diff --git a/meta/recipes-devtools/gdb/gdb.inc b/meta/recipes-devtools/gdb/gdb.inc
index 3321a24..9bf0e86 100644
--- a/meta/recipes-devtools/gdb/gdb.inc
+++ b/meta/recipes-devtools/gdb/gdb.inc
@@ -10,5 +10,12 @@ SRC_URI += "file://kill_arm_map_symbols.patch \
 #export CFLAGS_append=" -L${STAGING_LIBDIR}"
 
 # cross-canadian must not see this
-PACKAGES =+ "gdbserver"
+PACKAGES =+ "gdbserver gcore"
 FILES_gdbserver = "${bindir}/gdbserver"
+FILES_gcore = "${bindir}/gcore"
+
+do_install_append() {
+  # When trying to find what went wrong with a program, a core dump can be very helpful
+  # Go ahead and make it easier to get one when we have gdb on the system
+  install -m 0755 ${S}/gdb/gdb_gcore.sh ${D}${bindir}/gcore
+}
-- 
1.8.1.4

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[Mark Hatle]

On 10/31/13, 1:54 AM, blloyd wrote:
> gdb includes a script to make generating core files trivial.  Add a package to allow inclusion of this script in images.  GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump.  Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.
>
> Signed-off-by: blloyd <blloyd@familyhonor.net>
> ---
>   meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
>   meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
>   2 files changed, 15 insertions(+), 6 deletions(-)
>
> diff --git a/meta/recipes-devtools/gdb/gdb-common.inc b/meta/recipes-devtools/gdb/gdb-common.inc
> index 6954954..02eef85 100644
> --- a/meta/recipes-devtools/gdb/gdb-common.inc
> +++ b/meta/recipes-devtools/gdb/gdb-common.inc
> @@ -61,8 +61,10 @@ do_install_append() {
>   	rm -rf ${D}${datadir}/locale
>   }
>
> -RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
> -RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "
> -RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
> -RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
> -
> +# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
> +# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
> +RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
> +RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
> +RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
> +RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
> +RDEPENDS_gcore_append = " gdb "

The debug symbols should be in the -dbg package, not the -dev package.  The -dev 
package should simply be symlinks and special development only components, while 
the -dbg is the split symbols.

With the eglibc-dbg installed onto the gdb environment, it should be possible to 
have all necessary symbols to do whatever processing is required.  If this 
doesn't work, it likely needs to be investigated.  (If something in -dev is 
required, likely there is another bug somewhere else that needs to be fixed in 
the packaging of eglibc.)

Do you have steps on using this code (a test command perhaps?).  I've recently 
been asked to investigate some issues with debugging when using libpthread, and 
I suspect the problem may be over zealous stripping of the libpthread.so on the 
target.  But I still haven't finished my evaluation yet.

--Mark

> diff --git a/meta/recipes-devtools/gdb/gdb.inc b/meta/recipes-devtools/gdb/gdb.inc
> index 3321a24..9bf0e86 100644
> --- a/meta/recipes-devtools/gdb/gdb.inc
> +++ b/meta/recipes-devtools/gdb/gdb.inc
> @@ -10,5 +10,12 @@ SRC_URI += "file://kill_arm_map_symbols.patch \
>   #export CFLAGS_append=" -L${STAGING_LIBDIR}"
>
>   # cross-canadian must not see this
> -PACKAGES =+ "gdbserver"
> +PACKAGES =+ "gdbserver gcore"
>   FILES_gdbserver = "${bindir}/gdbserver"
> +FILES_gcore = "${bindir}/gcore"
> +
> +do_install_append() {
> +  # When trying to find what went wrong with a program, a core dump can be very helpful
> +  # Go ahead and make it easier to get one when we have gdb on the system
> +  install -m 0755 ${S}/gdb/gdb_gcore.sh ${D}${bindir}/gcore
> +}
>

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[Brian Lloyd]

[-- Attachment #1: Type: text/plain, Size: 5280 bytes --]

The patch adds gcore when gcore package is added to the install packages
list for an image.  That gives a simple core generation wrapper around
gcc, which might simplify your testing.
It appears -dev includes -dbg, and the file I want is in -dbg, so I can
submit a new patch limiting to only -dbg just fine.  The -dev for
glib-thread-db isn't very big so I didn't catch the extra files not
really needed.  I have used this in an image that was still 50mbish in
size.  Since before I made the patch, I made a hybrid system by manually
placing the one debug symbols file onto the disk, and once the symbols
file was in place cores generated became readable, and before that the
cores were unreadable, I have strong evidence only the debug symbols are
needed.

gcore PID
is the syntax to create a core dump using the helper script.  You can
also use gdb directly to debug or generate a core dump.  If the program
is multi-threaded, without the above patch a core loaded into a system
that has full -dev config (either an SDK or larger system image for
development) could not properly read the core dump.  With the above
patch or adding the debug symbol file manually, the core could be read
by the debugger.  Since I am building a i686 based system, I am able to
run my images on a PC in a sysroot, which made my testing easier.  I
also tried via a standard end system type disk on a device and a
development environment loaded on the device, and got the same results.


Brian


On Thu, 2013-10-31 at 08:54 -0500, Mark Hatle wrote:

> On 10/31/13, 1:54 AM, blloyd wrote:
> > gdb includes a script to make generating core files trivial.  Add a package to allow inclusion of this script in images.  GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump.  Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.
> >
> > Signed-off-by: blloyd <blloyd@familyhonor.net>
> > ---
> >   meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
> >   meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
> >   2 files changed, 15 insertions(+), 6 deletions(-)
> >
> > diff --git a/meta/recipes-devtools/gdb/gdb-common.inc b/meta/recipes-devtools/gdb/gdb-common.inc
> > index 6954954..02eef85 100644
> > --- a/meta/recipes-devtools/gdb/gdb-common.inc
> > +++ b/meta/recipes-devtools/gdb/gdb-common.inc
> > @@ -61,8 +61,10 @@ do_install_append() {
> >   	rm -rf ${D}${datadir}/locale
> >   }
> >
> > -RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
> > -RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "
> > -RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
> > -RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
> > -
> > +# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
> > +# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
> > +RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
> > +RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
> > +RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
> > +RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
> > +RDEPENDS_gcore_append = " gdb "
> 
> The debug symbols should be in the -dbg package, not the -dev package.  The -dev 
> package should simply be symlinks and special development only components, while 
> the -dbg is the split symbols.
> 
> With the eglibc-dbg installed onto the gdb environment, it should be possible to 
> have all necessary symbols to do whatever processing is required.  If this 
> doesn't work, it likely needs to be investigated.  (If something in -dev is 
> required, likely there is another bug somewhere else that needs to be fixed in 
> the packaging of eglibc.)
> 
> Do you have steps on using this code (a test command perhaps?).  I've recently 
> been asked to investigate some issues with debugging when using libpthread, and 
> I suspect the problem may be over zealous stripping of the libpthread.so on the 
> target.  But I still haven't finished my evaluation yet.
> 
> --Mark
> 
> > diff --git a/meta/recipes-devtools/gdb/gdb.inc b/meta/recipes-devtools/gdb/gdb.inc
> > index 3321a24..9bf0e86 100644
> > --- a/meta/recipes-devtools/gdb/gdb.inc
> > +++ b/meta/recipes-devtools/gdb/gdb.inc
> > @@ -10,5 +10,12 @@ SRC_URI += "file://kill_arm_map_symbols.patch \
> >   #export CFLAGS_append=" -L${STAGING_LIBDIR}"
> >
> >   # cross-canadian must not see this
> > -PACKAGES =+ "gdbserver"
> > +PACKAGES =+ "gdbserver gcore"
> >   FILES_gdbserver = "${bindir}/gdbserver"
> > +FILES_gcore = "${bindir}/gcore"
> > +
> > +do_install_append() {
> > +  # When trying to find what went wrong with a program, a core dump can be very helpful
> > +  # Go ahead and make it easier to get one when we have gdb on the system
> > +  install -m 0755 ${S}/gdb/gdb_gcore.sh ${D}${bindir}/gcore
> > +}
> >
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> 



[-- Attachment #2: Type: text/html, Size: 7381 bytes --]

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[Burton, Ross]

On 31 October 2013 06:54, blloyd <blloyd@familyhonor.net> wrote:
> +FILES_gcore = "${bindir}/gcore"

Why can't this script just be added to the gdb package?

Ross

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 609 bytes --]

On Friday, November 1, 2013, Burton, Ross wrote:

> On 31 October 2013 06:54, blloyd <blloyd@familyhonor.net <javascript:;>>
> wrote:
> > +FILES_gcore = "${bindir}/gcore"
>
> Why can't this script just be added to the gdb package?


actually if we have it as a separate package then all those new rrcommends
baggage can be added to gcore package instead of gdb proper


>
> Ross
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org <javascript:;>
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 1194 bytes --]

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[Brian Lloyd]

[-- Attachment #1: Type: text/plain, Size: 3675 bytes --]

Thank you for the interest.

Personally, I wouldn't mind just adding gcore to the base package, as it
isn't very large.  However, it really is a support script that isn't
necessary for gdb or gdb-server.  It does make testing the other part of
the patch easier, though.  But as it isn't strictly necessary, I opted
to let people opt in or out of having it.
I am hoping in the future there will be a way to view available packages
so end users can browse a list and add things without having to have
pre-knowledge that it exists to go looking for it.

I do not think the -dbg symbols should be a requirement of the gcore
package.  It wasn't while using gcore that I found the missing
dependency, but actually trying to analyze crash dumps created with gdb
installed on the device and a segfaulting multi-threaded program.  All
core files produced would have no more than one usable thread's data in
it, and it never seemed to be from the crashing thread.  OS level crash
dumps and gdb debugging of the application both were not properly
functional until those symbols were placed on the device, after which
both worked.  Therefore, it seems appropriate to place the dependency
against the gdb package as it fixes a defect with the program installed
by that package.  While I haven't been using gdb-server on the device,
the research I did pointed to that program also needing those symbols,
so the dependency was added there too.

As I have never used a system that didn't have gdb installed, I am not
sure whether core dumps can be generated without the debugger installed.
If they can, perhaps the eglibc package itself should be made NO_STRIP
or else eglibc made to recommend it's own debug symbols.  I don't really
like the no strip idea, as stripped binaries are smaller and thus faster
to load and these symbols are not needed very often, so why force them
to be loaded all the time, but it is an option.



As far as testing this, I could make a small multi-threaded application
that intentionally dereferences a null pointer in one of 4 threads
spawned.  That can show the broken nature of gdb core generation without
the thread library symbols.  If this is needed, I will put one together
and just reply here that it can be pulled from the poky-contrib
repository.  I found and resolved the problem while working on a project
my company is not yet ready to release.

However, if you just want to see that gdb doesn't handle threads
properly, you can just pick a multi-threaded program, do a core dump
(the gcore script makes this trivial), and then load the core file in a
debugger on the development system.  Even though the development system
has the symbols, that core won't be readable except for one thread.

 
I will do another patch fixing the commit message and incorporating
improvements we identify during this discussion.

Brian


On Fri, 2013-11-01 at 09:32 -0700, Khem Raj wrote:

> 
> 
> On Friday, November 1, 2013, Burton, Ross wrote:
> 
>         On 31 October 2013 06:54, blloyd <blloyd@familyhonor.net>
>         wrote:
>         > +FILES_gcore = "${bindir}/gcore"
>         
>         Why can't this script just be added to the gdb package?
>         
> 
> 
> actually if we have it as a separate package then all those new
> rrcommends baggage can be added to gcore package instead of gdb proper
>  
>         
>         
>         Ross
>         _______________________________________________
>         Openembedded-core mailing list
>         Openembedded-core@lists.openembedded.org
>         http://lists.openembedded.org/mailman/listinfo/openembedded-core



[-- Attachment #2: Type: text/html, Size: 4476 bytes --]

Re: [PATCH 1/1] Enhancements to allow core file debugging to work properly from devices

[Richard Purdie]

On Thu, 2013-10-31 at 01:54 -0500, blloyd wrote:
> gdb includes a script to make generating core files trivial.  Add a package to allow inclusion of this script in images.  GDB also requires symbols that are usually stripped from the threads library to properly generate a core dump.  Therefore, ensure when gdb is installed the necessary debug file is also present so gdb is fully functional.
> 
> Signed-off-by: blloyd <blloyd@familyhonor.net>
> ---
>  meta/recipes-devtools/gdb/gdb-common.inc | 12 +++++++-----
>  meta/recipes-devtools/gdb/gdb.inc        |  9 ++++++++-
>  2 files changed, 15 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/recipes-devtools/gdb/gdb-common.inc b/meta/recipes-devtools/gdb/gdb-common.inc
> index 6954954..02eef85 100644
> --- a/meta/recipes-devtools/gdb/gdb-common.inc
> +++ b/meta/recipes-devtools/gdb/gdb-common.inc
> @@ -61,8 +61,10 @@ do_install_append() {
>  	rm -rf ${D}${datadir}/locale
>  }
>  
> -RRECOMMENDS_gdb_append_linux = " glibc-thread-db "
> -RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db "
> -RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db "
> -RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db "
> -
> +# Unfortunately, for gdb multi-threading to function reliably it must have the eglibc debug symbols
> +# which are a part of the eglibc-dev package.  So we have added a dependency on that here.
> +RRECOMMENDS_gdb_append_linux = " glibc-thread-db eglibc-dev "
> +RRECOMMENDS_gdb_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
> +RRECOMMENDS_gdbserver_append_linux = " glibc-thread-db eglibc-dev "
> +RRECOMMENDS_gdbserver_append_linux-gnueabi = " glibc-thread-db eglibc-dev "
> +RDEPENDS_gcore_append = " gdb "

Aren't they part of the -dbg package, not -dev?

You also need to put "gdb: " at the start of the commit message summary
as per the commit message guidelines. It may be worth splitting this
change into two commits too.

Cheers,

Richard