perf trace: Problems initializing symbol libraries!

perf trace: Problems initializing symbol libraries!

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

Hey all,

I'm getting the following error when trying to run `perf trace` built from 
acme's perf/core branch on an Ubuntu 16.04 machine:

$ perf trace
Problems initializing symbol libraries!

strace shows this at the end:

...
lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
lstat("/proc/kallsyms", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
write(2, "Problems initializing symbol lib"..., 40Problems initializing symbol 
libraries!
) = 40
exit_group(244) 

Is something wrong with my /proc/kallsyms? Can someone tell me what the 
problems are that are being encountered?

Thanks
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: perf trace: Problems initializing symbol libraries!

[Arnaldo Carvalho de Melo]

Em Thu, May 12, 2016 at 02:05:35PM +0200, Milian Wolff escreveu:
> I'm getting the following error when trying to run `perf trace` built from 
> acme's perf/core branch on an Ubuntu 16.04 machine:
 
> $ perf trace
> Problems initializing symbol libraries!
 
> strace shows this at the end:
 
> ...
> lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
> lstat("/proc/kallsyms", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> write(2, "Problems initializing symbol lib"..., 40Problems initializing symbol 
> libraries!
> ) = 40
> exit_group(244) 
 
> Is something wrong with my /proc/kallsyms? Can someone tell me what the 
> problems are that are being encountered?

Does it work as root?

What is the value of /proc/sys/kernel/kptr_restrict?

Need to improve that message :-\

- Arnaldo

Re: perf trace: Problems initializing symbol libraries!

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 1435 bytes --]

On Donnerstag, 12. Mai 2016 15:10:44 CEST Arnaldo Carvalho de Melo wrote:
> Em Thu, May 12, 2016 at 02:05:35PM +0200, Milian Wolff escreveu:
> > I'm getting the following error when trying to run `perf trace` built from
> > acme's perf/core branch on an Ubuntu 16.04 machine:
> > 
> > $ perf trace
> > Problems initializing symbol libraries!
> > 
> > strace shows this at the end:
> > 
> > ...
> > lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
> > lstat("/proc/kallsyms", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> > write(2, "Problems initializing symbol lib"..., 40Problems initializing
> > symbol libraries!
> > ) = 40
> > exit_group(244)
> > 
> > Is something wrong with my /proc/kallsyms? Can someone tell me what the
> > problems are that are being encountered?
> 
> Does it work as root?

It does work as root. Note that on my other machines, I can run these commands 
to make it work as a non-root user:

sudo mount -o remount,mode=755 /sys/kernel/debug
sudo mount -o remount,mode=755 /sys/kernel/debug/tracing
echo "-1" | sudo tee /proc/sys/kernel/perf_event_paranoid

This does not work on Ubuntu 16.04 anymore.

> 
> What is the value of /proc/sys/kernel/kptr_restrict?
> 

This file does not exist.

> Need to improve that message :-\

Yes please :)

-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: perf trace: Problems initializing symbol libraries!

[Arnaldo Carvalho de Melo]

Em Mon, May 16, 2016 at 01:31:48PM +0200, Milian Wolff escreveu:
> On Donnerstag, 12. Mai 2016 15:10:44 CEST Arnaldo Carvalho de Melo wrote:
> > Em Thu, May 12, 2016 at 02:05:35PM +0200, Milian Wolff escreveu:
> > > I'm getting the following error when trying to run `perf trace` built from
> > > acme's perf/core branch on an Ubuntu 16.04 machine:
> > > 
> > > $ perf trace
> > > Problems initializing symbol libraries!
> > > 
> > > strace shows this at the end:
> > > 
> > > ...
> > > lstat("/proc", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
> > > lstat("/proc/kallsyms", {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
> > > write(2, "Problems initializing symbol lib"..., 40Problems initializing
> > > symbol libraries!
> > > ) = 40
> > > exit_group(244)
> > > 
> > > Is something wrong with my /proc/kallsyms? Can someone tell me what the
> > > problems are that are being encountered?
> > 
> > Does it work as root?
> 
> It does work as root. Note that on my other machines, I can run these commands 
> to make it work as a non-root user:
> 
> sudo mount -o remount,mode=755 /sys/kernel/debug
> sudo mount -o remount,mode=755 /sys/kernel/debug/tracing
> echo "-1" | sudo tee /proc/sys/kernel/perf_event_paranoid
> 
> This does not work on Ubuntu 16.04 anymore.

Ok, downloading 16.04 workstation to install on a guinea pig as I can't
possibly repro this on a docker image, the issue most likely is with the
kernel.
 
> > What is the value of /proc/sys/kernel/kptr_restrict?
> > 
> 
> This file does not exist.
> 
> > Need to improve that message :-\
> 
> Yes please :)

Will try, as soon as I install u16.04 on bare metal.

- Arnaldo

Re: perf trace: Problems initializing symbol libraries!

[Arnaldo Carvalho de Melo]

Em Mon, May 16, 2016 at 01:31:48PM +0200, Milian Wolff escreveu:
> On Donnerstag, 12. Mai 2016 15:10:44 CEST Arnaldo Carvalho de Melo wrote:
> > Em Thu, May 12, 2016 at 02:05:35PM +0200, Milian Wolff escreveu:
> > > I'm getting the following error when trying to run `perf trace` built from
> > > acme's perf/core branch on an Ubuntu 16.04 machine:
> > > 
> > > $ perf trace
> > > Problems initializing symbol libraries!

Ok, so with the perf tool that comes with ubuntu 16.04 I get:

  acme@ubuntu:~$ perf record ls
  WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
  check /proc/sys/kernel/kptr_restrict.

  Samples in kernel functions may not be resolved if a suitable vmlinux
  file is not found in the buildid cache or in the vmlinux path.

  Samples in kernel modules won't be resolved at all.

  If some relocation was applied (e.g. kexec) symbols may be misresolved
  even with a suitable vmlinux or kallsyms file.

  Cannot read kernel map
  Couldn't record kernel reference relocation symbol
  Symbol resolution may be skewed if relocation was used (e.g. kexec).
  Check /proc/kallsyms permission or run as root.
  git  perf.data
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.002 MB perf.data (14 samples) ]
  acme@ubuntu:~$

Which is a better message than:

  acme@ubuntu:~$ perf trace
  Error:	No permissions to read /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit)
  Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/debug/tracing'

  acme@ubuntu:~$ sudo mount -o remount,mode=755 /sys/kernel/debug/tracing
  acme@ubuntu:~$ perf trace
  Error:	No permissions to read /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit)
  Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/debug/tracing'

  acme@ubuntu:~$ sudo mount -o remount,mode=755 /sys/kernel/debug
  acme@ubuntu:~$ perf trace
  Problems initializing symbol libraries!
  acme@ubuntu:~$

So, if I follow the instructions given in the 'perf record ls' case:

  acme@ubuntu:~$ sudo sh -c "echo 0 > /proc/sys/kernel/kptr_restrict"
  acme@ubuntu:~$ perf trace
  Error:	Operation not permitted.
  Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
  Hint:	For system wide tracing it needs to be set to -1.
  Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
  Hint:	The current value is 1.
  acme@ubuntu:~$ 

It gets a bit better, then, one more step...

  acme@ubuntu:~$ perf trace --ev sched:sched_switch -e nanosleep sleep 1
       0.938 (   0.025 ms): sleep/7461 nanosleep(rqtp: 0x7ffc77ca6310                                        ) ...
       0.938 (           ): sched:sched_switch:prev_comm=sleep prev_pid=7461 prev_prio=120 prev_state=S ==> next_comm=swapper/1 next_pid=0 next_prio=120)
    1001.076 (1000.163 ms): sleep/7461  ... [continued]: nanosleep()) = 0
  acme@ubuntu:~$ 

So, the problem is not with what is in my perf/core branch, its that ubuntu
16.04's kernel comes with kptr_restrict defaulting to 1, I have to get that
warning emitted by 'perf record' into 'perf trace'.

- Arnaldo

Re: perf trace: Problems initializing symbol libraries!

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 3622 bytes --]

On Monday, May 16, 2016 5:13:59 PM CEST Arnaldo Carvalho de Melo wrote:
> Em Mon, May 16, 2016 at 01:31:48PM +0200, Milian Wolff escreveu:
> > On Donnerstag, 12. Mai 2016 15:10:44 CEST Arnaldo Carvalho de Melo wrote:
> > > Em Thu, May 12, 2016 at 02:05:35PM +0200, Milian Wolff escreveu:
> > > > I'm getting the following error when trying to run `perf trace` built
> > > > from
> > > > acme's perf/core branch on an Ubuntu 16.04 machine:
> > > > 
> > > > $ perf trace
> > > > Problems initializing symbol libraries!
> 
> Ok, so with the perf tool that comes with ubuntu 16.04 I get:
> 
>   acme@ubuntu:~$ perf record ls
>   WARNING: Kernel address maps (/proc/{kallsyms,modules}) are restricted,
>   check /proc/sys/kernel/kptr_restrict.
> 
>   Samples in kernel functions may not be resolved if a suitable vmlinux
>   file is not found in the buildid cache or in the vmlinux path.
> 
>   Samples in kernel modules won't be resolved at all.
> 
>   If some relocation was applied (e.g. kexec) symbols may be misresolved
>   even with a suitable vmlinux or kallsyms file.
> 
>   Cannot read kernel map
>   Couldn't record kernel reference relocation symbol
>   Symbol resolution may be skewed if relocation was used (e.g. kexec).
>   Check /proc/kallsyms permission or run as root.
>   git  perf.data
>   [ perf record: Woken up 1 times to write data ]
>   [ perf record: Captured and wrote 0.002 MB perf.data (14 samples) ]
>   acme@ubuntu:~$
> 
> Which is a better message than:
> 
>   acme@ubuntu:~$ perf trace
>   Error:	No permissions to read
> /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit) Hint:	Try
> 'sudo mount -o remount,mode=755 /sys/kernel/debug/tracing'
> 
>   acme@ubuntu:~$ sudo mount -o remount,mode=755 /sys/kernel/debug/tracing
>   acme@ubuntu:~$ perf trace
>   Error:	No permissions to read
> /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit) Hint:	Try
> 'sudo mount -o remount,mode=755 /sys/kernel/debug/tracing'
> 
>   acme@ubuntu:~$ sudo mount -o remount,mode=755 /sys/kernel/debug
>   acme@ubuntu:~$ perf trace
>   Problems initializing symbol libraries!
>   acme@ubuntu:~$
> 
> So, if I follow the instructions given in the 'perf record ls' case:
> 
>   acme@ubuntu:~$ sudo sh -c "echo 0 > /proc/sys/kernel/kptr_restrict"
>   acme@ubuntu:~$ perf trace
>   Error:	Operation not permitted.
>   Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
>   Hint:	For system wide tracing it needs to be set to -1.
>   Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/
perf_event_paranoid"'
>   Hint:	The current value is 1.
>   acme@ubuntu:~$
> 
> It gets a bit better, then, one more step...
> 
>   acme@ubuntu:~$ perf trace --ev sched:sched_switch -e nanosleep sleep 1
>        0.938 (   0.025 ms): sleep/7461 nanosleep(rqtp: 0x7ffc77ca6310       
>                                 ) ... 0.938 (           ):
> sched:sched_switch:prev_comm=sleep prev_pid=7461 prev_prio=120 prev_state=S
> ==> next_comm=swapper/1 next_pid=0 next_prio=120) 1001.076 (1000.163 ms):
> sleep/7461  ... [continued]: nanosleep()) = 0 acme@ubuntu:~$
> 
> So, the problem is not with what is in my perf/core branch, its that ubuntu
> 16.04's kernel comes with kptr_restrict defaulting to 1, I have to get that
> warning emitted by 'perf record' into 'perf trace'.

Thanks a lot Arnaldo,

that makes it work for me. I'll include the line to set kptr_restrict to 0 
into my "perf init" script that elevates the restrictions.

Thanks again

-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: perf trace: Problems initializing symbol libraries!

[Arnaldo Carvalho de Melo]

Em Tue, May 17, 2016 at 09:51:44AM +0200, Milian Wolff escreveu:
> On Monday, May 16, 2016 5:13:59 PM CEST Arnaldo Carvalho de Melo wrote:
> > So, the problem is not with what is in my perf/core branch, its that ubuntu
> > 16.04's kernel comes with kptr_restrict defaulting to 1, I have to get that
> > warning emitted by 'perf record' into 'perf trace'.
 
> that makes it work for me. I'll include the line to set kptr_restrict to 0 
> into my "perf init" script that elevates the restrictions.

That will not be strictly needed with what I have now on my perf/core
branch, where it will just warn the user if/when symbol resolution is
needed and kptr_restrict=1.

Thanks for the report, tested on fedora 23 with kptr_restrict set to 1
and then again on ubuntu 16.04, default kernel sysctl values.

- Arnaldo

Re: perf trace: Problems initializing symbol libraries!

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 1138 bytes --]

On Tuesday, May 17, 2016 12:06:17 PM CEST Arnaldo Carvalho de Melo wrote:
> Em Tue, May 17, 2016 at 09:51:44AM +0200, Milian Wolff escreveu:
> > On Monday, May 16, 2016 5:13:59 PM CEST Arnaldo Carvalho de Melo wrote:
> > > So, the problem is not with what is in my perf/core branch, its that
> > > ubuntu
> > > 16.04's kernel comes with kptr_restrict defaulting to 1, I have to get
> > > that
> > > warning emitted by 'perf record' into 'perf trace'.
> > 
> > that makes it work for me. I'll include the line to set kptr_restrict to 0
> > into my "perf init" script that elevates the restrictions.
> 
> That will not be strictly needed with what I have now on my perf/core
> branch, where it will just warn the user if/when symbol resolution is
> needed and kptr_restrict=1.
> 
> Thanks for the report, tested on fedora 23 with kptr_restrict set to 1
> and then again on ubuntu 16.04, default kernel sysctl values.

Much appreciated, thanks a lot Arnaldo!

-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]