Not finding symbol information

Not finding symbol information

[Vuille, Martin (Martin)]

We are using 'perf record' to collect information on an embedded ARM board (the target)
and then using 'perf report' on a PC (the host) to interpret the data.

The host has a copy of the rootfs of the target that also includes debug info. We pass the
'--symfs' option to 'perf report' to point to the rootfs.

Whenever an event in the data corresponds to a DSO that exists on the host, 'perf report'
complains that the debug info is not available, even though that info is in fact available in
the rootfs.

What seems to be happening is that 'perf report' "sees" the host's copy of the DSO first,
"remembers" this DSO's buildid (which is incorrect), then "sees" the debug info in the
rootfs but rejects it because it doesn't match the buildid "remembered" earlier, and then
concludes that there is no debug info available.

Using /bin/bash as an example, selected output from 'perf report -v':

...
build id event received for /bin/bash: a35856957f6f165a1ca4ea98afbd1a986dfa785c
...
symsrc__init: build id mismatch for sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash.
symsrc__init: build id mismatch for sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash.
symsrc__init: build id mismatch for sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/bash.
/bin/bash with build id 04eca96c5bf3e9a300952a29ef3218f00487d37b not found, continuing without symbols
...

$ file /bin/bash
/bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=04eca96c5bf3e9a300952a29ef3218f00487d37b, stripped

$ file sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash
sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=a35856957f6f165a1ca4ea98afbd1a986dfa785c, stripped

$ file sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/bash
sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter *empty*, for GNU/Linux 3.2.0, BuildID[sha1]=a35856957f6f165a1ca4ea98afbd1a986dfa785c, not stripped

We are using Linux 4.4-11 on the target and 4.4.0-109-generic (Ubuntu 16.04) on the host.

Is this a known issue? Has it been addressed in a more recent kernel?

Thanks in advance,
MV

Re: Not finding symbol information

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 3771 bytes --]

On Friday, February 2, 2018 3:06:32 PM CET Vuille, Martin (Martin) wrote:
> We are using 'perf record' to collect information on an embedded ARM board
> (the target) and then using 'perf report' on a PC (the host) to interpret
> the data.
> 
> The host has a copy of the rootfs of the target that also includes debug
> info. We pass the '--symfs' option to 'perf report' to point to the rootfs.
> 
> Whenever an event in the data corresponds to a DSO that exists on the host,
> 'perf report' complains that the debug info is not available, even though
> that info is in fact available in the rootfs.
> 
> What seems to be happening is that 'perf report' "sees" the host's copy of
> the DSO first, "remembers" this DSO's buildid (which is incorrect), then
> "sees" the debug info in the rootfs but rejects it because it doesn't match
> the buildid "remembered" earlier, and then concludes that there is no debug
> info available.

`perf record` (not `perf report`) is "remembering" build-ids in ~/.debug.

> Using /bin/bash as an example, selected output from 'perf report -v':
> 
> ...
> build id event received for /bin/bash:
> a35856957f6f165a1ca4ea98afbd1a986dfa785c ...
> symsrc__init: build id mismatch for
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash.
> symsrc__init: build id mismatch for
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash.
> symsrc__init: build id mismatch for
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/
> bash. /bin/bash with build id 04eca96c5bf3e9a300952a29ef3218f00487d37b not
> found, continuing without symbols ...
> 
> $ file /bin/bash
> /bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically
> linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32,
> BuildID[sha1]=04eca96c5bf3e9a300952a29ef3218f00487d37b, stripped
> 
> $ file
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash:
> ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked,
> interpreter /lib/ld-linux.so.3, for GNU/Linux 3.2.0,
> BuildID[sha1]=a35856957f6f165a1ca4ea98afbd1a986dfa785c, stripped
> 
> $ file
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/
> bash
> sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/
> bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically
> linked, interpreter *empty*, for GNU/Linux 3.2.0,
> BuildID[sha1]=a35856957f6f165a1ca4ea98afbd1a986dfa785c, not stripped

This indeed looks like a bug in the build id matching algorithm. I suggest you 
debug it to figure out why it thinks the build id mismatches. I.e. expand the 
debug output and write the build id it has read and compare that to what file 
sees and what the perf file expects.

> We are using Linux 4.4-11 on the target and 4.4.0-109-generic (Ubuntu 16.04)
> on the host.
> 
> Is this a known issue? Has it been addressed in a more recent kernel?

This should not be part of the kernel, but rather part of the userland perf 
tools. You can easily build the latest versions of those yourself, without 
having to build a new kernel. So on your host, try this:

git clone git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git -b 
perf/core
cd tools/perf
make # make sure you have all the required dependencies, most notably 
elfutils, libiberty, ...
/perf report -i <path to perf.data>

Hope that helps

PS: You could also try to use our hotspot GUI:
https://github.com/KDAB/hotspot
-- 
Milian Wolff | milian.wolff@kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 3826 bytes --]

Re: Not finding symbol information

[Kim Phillips]

On Fri, 2 Feb 2018 15:31:08 +0100
Milian Wolff <milian.wolff@kdab.com> wrote:

> On Friday, February 2, 2018 3:06:32 PM CET Vuille, Martin (Martin) wrote:
> > We are using 'perf record' to collect information on an embedded ARM board
> > (the target) and then using 'perf report' on a PC (the host) to interpret
> > the data.
> > 
> > The host has a copy of the rootfs of the target that also includes debug
> > info. We pass the '--symfs' option to 'perf report' to point to the rootfs.
> > 
> > Whenever an event in the data corresponds to a DSO that exists on the host,
> > 'perf report' complains that the debug info is not available, even though
> > that info is in fact available in the rootfs.
> > 
> > What seems to be happening is that 'perf report' "sees" the host's copy of
> > the DSO first, "remembers" this DSO's buildid (which is incorrect), then
> > "sees" the debug info in the rootfs but rejects it because it doesn't match
> > the buildid "remembered" earlier, and then concludes that there is no debug
> > info available.
> 
> `perf record` (not `perf report`) is "remembering" build-ids in ~/.debug.

I've experienced bugs in the area that come about when using different
versions of perf writing to the common cache area, and at one point
started rm -fr ~/.debug; mkdir -p ~/.debug, and re-extracting the 'perf
archive' result, with tar jxf perf.data.tar.bz2 -C ~/.debug, just for
sanity's sake.

> > Using /bin/bash as an example, selected output from 'perf report -v':
> > 
> > ...
> > build id event received for /bin/bash:
> > a35856957f6f165a1ca4ea98afbd1a986dfa785c ...
> > symsrc__init: build id mismatch for
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash.
> > symsrc__init: build id mismatch for
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash.
> > symsrc__init: build id mismatch for
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/
> > bash. /bin/bash with build id 04eca96c5bf3e9a300952a29ef3218f00487d37b not
> > found, continuing without symbols ...
> > 
> > $ file /bin/bash
> > /bin/bash: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically
> > linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32,
> > BuildID[sha1]=04eca96c5bf3e9a300952a29ef3218f00487d37b, stripped
> > 
> > $ file
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/bash:
> > ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked,
> > interpreter /lib/ld-linux.so.3, for GNU/Linux 3.2.0,
> > BuildID[sha1]=a35856957f6f165a1ca4ea98afbd1a986dfa785c, stripped
> > 
> > $ file
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/
> > bash
> > sip96x1_prod/cp/lib/dvf99/sysroots/armv5te-dspg-linux-gnueabi///bin/.debug/
> > bash: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically
> > linked, interpreter *empty*, for GNU/Linux 3.2.0,
> > BuildID[sha1]=a35856957f6f165a1ca4ea98afbd1a986dfa785c, not stripped
> 
> This indeed looks like a bug in the build id matching algorithm. I suggest you 
> debug it to figure out why it thinks the build id mismatches. I.e. expand the 
> debug output and write the build id it has read and compare that to what file 
> sees and what the perf file expects.
> 
> > We are using Linux 4.4-11 on the target and 4.4.0-109-generic (Ubuntu 16.04)
> > on the host.
> > 
> > Is this a known issue? Has it been addressed in a more recent kernel?
> 
> This should not be part of the kernel, but rather part of the userland perf 
> tools. You can easily build the latest versions of those yourself, without 
> having to build a new kernel. So on your host, try this:
> 
> git clone git://git.kernel.org/pub/scm/linux/kernel/git/acme/linux.git -b 
> perf/core
> cd tools/perf
> make # make sure you have all the required dependencies, most notably 
> elfutils, libiberty, ...
> /perf report -i <path to perf.data>

FWIW, I just tried acme's tip against some old arm32 perf.data capture
& archive, and the problem still seems to exist:

$ file ~/.debug/bin/dd/0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb 
/home/kim/.debug/bin/dd/0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32, BuildID[sha1]=0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb, stripped

$ perf report -vv
...
build id event received for /bin/dd: 0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb
...
symsrc__init: build id mismatch for /bin/dd.
no symbols found in /bin/dd, maybe install a debug package?
...

Although the executable didn't have debug data, so the symbols weren't
going to be resolved anyway...

It might also be worth trying running the report on the target, if
possible.  I'll try and take a look next week...I need to revive my
arm32 boxes...

Kim

RE: Not finding symbol information

[Vuille, Martin (Martin)]

> > > What seems to be happening is that 'perf report' "sees" the host's
> > > copy of the DSO first, "remembers" this DSO's buildid (which is
> > > incorrect), then "sees" the debug info in the rootfs but rejects it
> > > because it doesn't match the buildid "remembered" earlier, and then
> > > concludes that there is no debug info available.
> >
> > `perf record` (not `perf report`) is "remembering" build-ids in ~/.debug.
> 
> I've experienced bugs in the area that come about when using different
> versions of perf writing to the common cache area, and at one point started
> rm -fr ~/.debug; mkdir -p ~/.debug, and re-extracting the 'perf archive'
> result, with tar jxf perf.data.tar.bz2 -C ~/.debug, just for sanity's sake.
> 

I wasn't referring to the buildid cache, but to the internal buildid processing
logic of 'perf report'.

If I understand what it does correctly, 'perf archive' would be of no use to me,
as there is no debug info to preserve on the target. I use a custom script to prepare
the buildid cache on the host based on the output of 'perf buildid-list'.

BTW, I found some odd behaviour around that too. 'perf report --symfs PATH' is
looking for the buildid cache in 'PATH/$HOME/.debug', not '$HOME/.debug'.
Not sure whether that's design intent or a bug.

> FWIW, I just tried acme's tip against some old arm32 perf.data capture &
> archive, and the problem still seems to exist:
> 
> $ file ~/.debug/bin/dd/0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb
> /home/kim/.debug/bin/dd/0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb:
> ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked,
> interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32,
> BuildID[sha1]=0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb, stripped
> 
> $ perf report -vv
> ...
> build id event received for /bin/dd:
> 0ff7b46a4aec92df5ccdb83ab5eef7f1888315fb
> ...
> symsrc__init: build id mismatch for /bin/dd.
> no symbols found in /bin/dd, maybe install a debug package?
> ...
> 
> Although the executable didn't have debug data, so the symbols weren't
> going to be resolved anyway...

Thanks.

> 
> It might also be worth trying running the report on the target, if possible.  I'll
> try and take a look next week...I need to revive my
> arm32 boxes...

Unfortunately the stripped binaries just barely fit on the target, so getting the
debug info on there is out of the question.

Regards,
MV

Re: Not finding symbol information

[Kim Phillips]

On Sat, 3 Feb 2018 03:30:02 +0000
"Vuille, Martin (Martin)" <vmartin@avaya.com> wrote:

> BTW, I found some odd behaviour around that too. 'perf report --symfs PATH' is
> looking for the buildid cache in 'PATH/$HOME/.debug', not '$HOME/.debug'.
> Not sure whether that's design intent or a bug.

Yeah, that doesn't sound right: the manpage just says "Look for files
with symbols relative to this directory."

> > It might also be worth trying running the report on the target, if possible.  I'll
> > try and take a look next week...I need to revive my
> > arm32 boxes...
> 
> Unfortunately the stripped binaries just barely fit on the target, so getting the
> debug info on there is out of the question.

OK, in that case can you try cherry-picking commit 9b200653518e "perf
symbols: Take into account symfs setting when reading file build ID"?
It looks highly relevant but I haven't been able to make sure yet.

Thanks,

Kim

RE: Not finding symbol information

[Vuille, Martin (Martin)]

> 
> OK, in that case can you try cherry-picking commit 9b200653518e "perf
> symbols: Take into account symfs setting when reading file build ID"?
> It looks highly relevant but I haven't been able to make sure yet.
> 

I just build from the latest on the perf/core branch (which includes that commit)
and the problem seems to be fixed. Will test further tomorrow.

On a slightly related note, should the "[vectors]" mapping be handled?
I see the perf code handles "[vdso]" but there doesn't seem to be anything
for "[vectors]".

MV

RE: Not finding symbol information

[Vuille, Martin (Martin)]

> > BTW, I found some odd behaviour around that too. 'perf report --symfs
> > PATH' is looking for the buildid cache in 'PATH/$HOME/.debug', not
> '$HOME/.debug'.
> > Not sure whether that's design intent or a bug.
> 
> Yeah, that doesn't sound right: the manpage just says "Look for files with
> symbols relative to this directory."

In latest perf/core, this is now changed to PATH/.debug, which makes more sense.

> OK, in that case can you try cherry-picking commit 9b200653518e "perf
> symbols: Take into account symfs setting when reading file build ID"?
> It looks highly relevant but I haven't been able to make sure yet.

That's the fix. Thank you very much for the tip

MV

RE: Not finding symbol information

[Vuille, Martin (Martin)]

> In latest perf/core, this is now changed to PATH/.debug, which makes more
> sense.

Actually, not quite. Now it is looking in PATH/PATH/.debug. Argh.

I think this may be  side-effect of the fix for 9b200653518e, which adds the symfs
path to the DSO name.

Is this the right place to report such issues?
MV