Debugging using sysroots and GDB

Debugging using sysroots and GDB

[Pascal Bach]

Hi Everybody

Currently debugging using sysroots seems to work as long as the work folder containing the original source is available.
Once this work dir is gone the debugger is no longer able to find the source code. This is especially confusing if some packages are taken from sstate and others are built in the current projects, because for some libraries the sources can be found and for others not. It is hard at first to figure out the reason why one works and the other not.

In order to make it easier to debug using sysroots I propose to modify the how the sysroots is built to do somethins similar to rootfs:

1. Changing the source reference in the debug symbols to point to the source under /usr/src/debug
2. Copy the sources into /usr/src/debug under sysroots similar to how it is done for a debug rootfs.
3. Document that the user needs to add the following to the .gdbinit to make it work:
    ```
    set sysroot /project/oe/build/tmp/sysroots/<target_sysrtoot>
    set substitute-path /usr/src/debug /project/oe/build/tmp/sysroots/<target_sysrtoot>/usr/src/debug
    ```

This would also allow the sysroot to be relocatable as it is kind of standalone and everything required to debug is included.

What do you think? Does this proposal make sense, or did I miss something and this is completly unnecessary and there is an easier way already working?

Regards
Pascal

Re: Debugging using sysroots and GDB

[Pascal Bach]

ping!
No comment on this topic?

Am 23.02.2016 um 15:09 schrieb Pascal Bach:
> Hi Everybody
>
> Currently debugging using sysroots seems to work as long as the work folder containing the original source is available.
> Once this work dir is gone the debugger is no longer able to find the source code. This is especially confusing if some packages are taken from sstate and others are built in the current projects, because for some libraries the sources can be found and for others not. It is hard at first to figure out the reason why one works and the other not.
>
> In order to make it easier to debug using sysroots I propose to modify the how the sysroots is built to do somethins similar to rootfs:
>
> 1. Changing the source reference in the debug symbols to point to the source under /usr/src/debug
> 2. Copy the sources into /usr/src/debug under sysroots similar to how it is done for a debug rootfs.
> 3. Document that the user needs to add the following to the .gdbinit to make it work:
>     ```
>     set sysroot /project/oe/build/tmp/sysroots/<target_sysrtoot>
>     set substitute-path /usr/src/debug /project/oe/build/tmp/sysroots/<target_sysrtoot>/usr/src/debug
>     ```
>
> This would also allow the sysroot to be relocatable as it is kind of standalone and everything required to debug is included.
>
> What do you think? Does this proposal make sense, or did I miss something and this is completly unnecessary and there is an easier way already working?
>
> Regards
> Pascal

Re: Debugging using sysroots and GDB

[Richard Purdie]

On Tue, 2016-02-23 at 15:09 +0100, Pascal Bach wrote:
> Currently debugging using sysroots seems to work as long as the work
> folder containing the original source is available.
> Once this work dir is gone the debugger is no longer able to find the
> source code. This is especially confusing if some packages are taken
> from sstate and others are built in the current projects, because for
> some libraries the sources can be found and for others not. It is
> hard at first to figure out the reason why one works and the other
> not.
> 
> In order to make it easier to debug using sysroots I propose to
> modify the how the sysroots is built to do somethins similar to
> rootfs:
> 
> 1. Changing the source reference in the debug symbols to point to the
> source under /usr/src/debug
> 2. Copy the sources into /usr/src/debug under sysroots similar to how
> it is done for a debug rootfs.
> 3. Document that the user needs to add the following to the .gdbinit
> to make it work:
>     ```
>     set sysroot /project/oe/build/tmp/sysroots/<target_sysrtoot>
>     set substitute-path /usr/src/debug
> /project/oe/build/tmp/sysroots/<target_sysrtoot>/usr/src/debug
>     ```
> 
> This would also allow the sysroot to be relocatable as it is kind of
> standalone and everything required to debug is included.
> 
> What do you think? Does this proposal make sense, or did I miss
> something and this is completly unnecessary and there is an easier
> way already working?

It really depends on your view of what the sysroot is used for. We
intentionally strip a variety of things out of it as things work today
basically for size/performance reasons. If we start putting debugging
source in there, it will become huge and this will image the size of
things like eSDK and affect the speed of operations like building from
sstate.

We don't really support debugging from the sysroot. We could talk about
changing that but I suspect people wouldn't like the performance
penalty.

You might ask "ok, can we make it optional?". The trouble with that is
more code paths which people will complain we're not testing and
consequently an increased test matrix and even slower patch merging
cycles.

So the question is, is this something we want to support this way?

Cheers,

Richard

Re: Debugging using sysroots and GDB

[Pascal Bach]

> It really depends on your view of what the sysroot is used for. We
> intentionally strip a variety of things out of it as things work today
> basically for size/performance reasons. If we start putting debugging
> source in there, it will become huge and this will image the size of
> things like eSDK and affect the speed of operations like building from
> sstate.

From my point of view debugging from sysroots is not mandatory if I can find a suitable alternative.
One thing I want to avoid is to install -dbg packages to my image.

What I currently do is the following:

1. Enable Debugfs generation using: `IMAGE_GEN_DEBUGFS = "1"`
2. Switch debug split style to: `PACKAGE_DEBUG_SPLIT_STYLE = "debug-file-directory"`

After building the image I end up with a rootfs and a rootfs-dbg

3. Extract both rootfs and rootfs-dbg somewhere (/home/projects)
4. Setup gdb with the following .gdbinit:

```
set sysroot /home/projects/rootfs
set debug-file-directory /home/projects/rootfs-dbg/usr/lib/debug
set substitute-path /usr/src/debug /home/projects/rootfs-dbg/usr/src/debug
```

With this setup I'm able to debug trough all libraries on the system.
So so far so good.

However for this to work I need to change the `PACKAGE_DEBUG_SPLIT_STYLE` to a non default and this made me wonder if I missed something.
- Is there a way to get this setup working with the default ".debug" style of OE?
- If not why is the .debug style the default instead of "debug-file-directory"?
- Is there any documentation? I was unable to find anything more than [1] in the Yocto Mega manual.
- Is there some best practice I missed?

[1] http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#platdev-gdb-remotedebug-setup


> We don't really support debugging from the sysroot. We could talk about
> changing that but I suspect people wouldn't like the performance
> penalty.
Ok if this is not supported is there any documentation on whats the recommended debugging method?
>
> You might ask "ok, can we make it optional?". The trouble with that is
> more code paths which people will complain we're not testing and
> consequently an increased test matrix and even slower patch merging
> cycles.
I agree, there should only be one recommended and supported way to do debugging and this way should be well documented.
>
> So the question is, is this something we want to support this way?
>
> Cheers,
>
> Richard
Pascal

Re: Debugging using sysroots and GDB

[Richard Purdie]

On Fri, 2016-03-04 at 15:06 +0100, Pascal Bach wrote:
> > It really depends on your view of what the sysroot is used for. We
> > intentionally strip a variety of things out of it as things work
> > today
> > basically for size/performance reasons. If we start putting
> > debugging
> > source in there, it will become huge and this will image the size
> > of
> > things like eSDK and affect the speed of operations like building
> > from
> > sstate.
> 
> From my point of view debugging from sysroots is not mandatory if I
> can find a suitable alternative.
> One thing I want to avoid is to install -dbg packages to my image.
> 
> What I currently do is the following:
> 
> 1. Enable Debugfs generation using: `IMAGE_GEN_DEBUGFS = "1"`
> 2. Switch debug split style to: `PACKAGE_DEBUG_SPLIT_STYLE = "debug
> -file-directory"`
> 
> After building the image I end up with a rootfs and a rootfs-dbg
> 
> 3. Extract both rootfs and rootfs-dbg somewhere (/home/projects)
> 4. Setup gdb with the following .gdbinit:
> 
> ```
> set sysroot /home/projects/rootfs
> set debug-file-directory /home/projects/rootfs-dbg/usr/lib/debug
> set substitute-path /usr/src/debug /home/projects/rootfs
> -dbg/usr/src/debug
> ```
> 
> With this setup I'm able to debug trough all libraries on the system.
> So so far so good.
> 
> However for this to work I need to change the
> `PACKAGE_DEBUG_SPLIT_STYLE` to a non default and this made me wonder
> if I missed something.
> - Is there a way to get this setup working with the default ".debug"
> style of OE?
> - If not why is the .debug style the default instead of "debug-file
> -directory"?
> - Is there any documentation? I was unable to find anything more than
> [1] in the Yocto Mega manual.
> - Is there some best practice I missed?

Using this method is a supported approach. I'm a little surprised
.debug directories don't work. Have you tried just setting:

set sysroot /home/projects/rootfs-dbg
set substitute-path /usr/src/debug /home/projects/rootfs
-bbg/usr/src/debug

as I believe it should work with IMAGE_GEN_DEBUGFS=1 "out the box". If
it doesn't that is a bug and we should try and look into fixing that.

Cheers,

Richard

Re: Debugging using sysroots and GDB

[Pascal Bach]

Hi Richard
> Using this method is a supported approach. I'm a little surprised
> .debug directories don't work. Have you tried just setting:
>
> set sysroot /home/projects/rootfs-dbg
> set substitute-path /usr/src/debug /home/projects/rootfs
> -bbg/usr/src/debug
This doesn't seem to work (tried only with dizzy).
The only way I could get this to work is to extract both the rootfs and the rootfs-dbg into the same place and then set sysroot to that location.
Is this the way it is supposed to be?

>
> as I believe it should work with IMAGE_GEN_DEBUGFS=1 "out the box". If
> it doesn't that is a bug and we should try and look into fixing that.
I'm currently using dizzy. I will give it another try with jethro next week maybe it is working there.

Pascal

Re: Debugging using sysroots and GDB

[Andreas Müller]

On Fri, Mar 4, 2016 at 12:20 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2016-02-23 at 15:09 +0100, Pascal Bach wrote:
>> Currently debugging using sysroots seems to work as long as the work
>> folder containing the original source is available.

I am using sysroot for debugging too. I have no

* INHERIT += "rm_work"

and added

INHIBIT_SYSROOT_STRIP = "1"

to my local.conf

I use this approach for very long time now and from my point of view
it has one BIG advantage: I can use these settings for productive
images and start remote debug within seconds without further
rebuilding.
OK - default compiler optimization sometimes causes headaches but it
works for solving >80% of my cases. In case I want to debug software
that I have developed setting optimization recipe wise helps or I
debug that on PC or target itself.

How much time is really saved by stripping sysroot?

Andreas

Re: Debugging using sysroots and GDB

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 354 bytes --]

On 8 March 2016 at 10:01, Andreas Müller <schnitzeltony@googlemail.com>
wrote:

> How much time is really saved by stripping sysroot?
>

It was more about reducing the size of the sstate than build performance.
It's a toggle for precisely this reason: some users may not care about the
space saving but want to use the sysroot for gdb.

Ross

[-- Attachment #2: Type: text/html, Size: 760 bytes --]

Re: Debugging using sysroots and GDB

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1095 bytes --]

On Mar 8, 2016 3:54 PM, "Pascal Bach" <pascal.bach@siemens.com> wrote:
>
> Hi Richard
> > Using this method is a supported approach. I'm a little surprised
> > .debug directories don't work. Have you tried just setting:
> >
> > set sysroot /home/projects/rootfs-dbg
> > set substitute-path /usr/src/debug /home/projects/rootfs
> > -bbg/usr/src/debug
> This doesn't seem to work (tried only with dizzy).
> The only way I could get this to work is to extract both the rootfs and
the rootfs-dbg into the same place and then set sysroot to that location.
> Is this the way it is supposed to be?

Yes

>
> >
> > as I believe it should work with IMAGE_GEN_DEBUGFS=1 "out the box". If
> > it doesn't that is a bug and we should try and look into fixing that.
> I'm currently using dizzy. I will give it another try with jethro next
week maybe it is working there.
>
> Pascal
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

[-- Attachment #2: Type: text/html, Size: 1568 bytes --]

Re: Debugging using sysroots and GDB

[Pascal Bach]

Hi Andreas
> I am using sysroot for debugging too. I have no
>
> * INHERIT += "rm_work"
>
> and added
>
> INHIBIT_SYSROOT_STRIP = "1"
>
> to my local.conf
>
> I use this approach for very long time now and from my point of view
> it has one BIG advantage: I can use these settings for productive
> images and start remote debug within seconds without further
> rebuilding.
> OK - default compiler optimization sometimes causes headaches but it
> works for solving >80% of my cases. In case I want to debug software
> that I have developed setting optimization recipe wise helps or I
> debug that on PC or target itself.
How do you deal with the case where there is no source in your work dir because the package was taken from sstate?
The only solution I found so far is to build without sstate :(

Pascal

Re: Debugging using sysroots and GDB

[Pascal Bach]

> > The only way I could get this to work is to extract both the rootfs and the rootfs-dbg into the same place and then set sysroot to that location.
> > Is this the way it is supposed to be?
>
> Yes
>
Ok I was hoping to keep them seperate.
This because I'm using the rootfs for NFS and by having the debug symbols in a seperate rootfs-dbg directory they would not be visible from the target.
By setting `PACKAGE_DEBUG_SPLIT_STYLE = "debug-file-directory"` I got this working.

Re: Debugging using sysroots and GDB

[Andreas Müller]

On Tue, Mar 8, 2016 at 3:06 PM, Pascal Bach <pascal.bach@siemens.com> wrote:
> Hi Andreas
>> I am using sysroot for debugging too. I have no
>>
>> * INHERIT += "rm_work"
>>
>> and added
>>
>> INHIBIT_SYSROOT_STRIP = "1"
>>
>> to my local.conf
>>
>> I use this approach for very long time now and from my point of view
>> it has one BIG advantage: I can use these settings for productive
>> images and start remote debug within seconds without further
>> rebuilding.
>> OK - default compiler optimization sometimes causes headaches but it
>> works for solving >80% of my cases. In case I want to debug software
>> that I have developed setting optimization recipe wise helps or I
>> debug that on PC or target itself.
> How do you deal with the case where there is no source in your work dir because the package was taken from sstate?
> The only solution I found so far is to build without sstate :(
>
To be honest: I never had that - maybe because I am keeping a complete
tmp for each machine I am working with.

Andreas