perf trace & vfs_getname

perf trace & vfs_getname

[Milian Wolff]

[-- Attachment #1: Type: text/plain, Size: 2033 bytes --]

Hey Arnaldo,

I just wanted to try out your ongoing work to get file names printed in perf 
trace:

~~~~~~~~~~~~~~
$ perf trace -e open ls builtin-trace.c
     0.007 ( 0.007 ms): ls/26364 open(filename: 0x2a1b0d20, flags: CLOEXEC                             
) = -1 ENOENT No such file or directorybuiltin-trace.c

     0.022 ( 0.002 ms): ls/26364 open(filename: 0x2a1b0d20, flags: CLOEXEC                             
) = -1 ENOENT No such file or directory
...
~~~~~~~~~~~~~~

So apparently I still have to manually create the probe for vfs_getname. Could 
this be simplified for the user? E.g. by adding a static tracepoint for that 
purpose, so one doesn't have to run `perf probe` manually? Also, note that 
`vfs_getname` is only mentioned in `man perf trace` for its `--tool_stats` 
parameter, but the actual line to create the probe is nowhere to be found. 

Grepping the git log I found the following invocation, which errors out on my 
Arch machine:

~~~~~~~~~~~~~~~
$ perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
The /home/milian/.debug/.build-id/1a/03b857e3611b7a14fbf60d67cad7415706a933 
file has no debug information.
Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
  Error: Failed to add events.
~~~~~~~~~~~~~~~

On a different Arch machine it's:

~~~~~~~~~~~~~~~
$ perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
The /lib/modules/4.5.4-1-ARCH/build/vmlinux file has no debug information.
Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
  Error: Failed to add events.
~~~~~~~~~~~~~~~

On an Ubuntu 16.04 I'm getting this:

~~~~~~~~~~~~~~~
$ perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
Failed to find the path for kernel: No such file or directory
  Error: Failed to add events.
~~~~~~~~~~~~~~~

So does this really mean I'm out of luck?
-- 
Milian Wolff | milian.wolff@kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5903 bytes --]

Re: perf trace & vfs_getname

[Arnaldo Carvalho de Melo]

Em Tue, May 24, 2016 at 10:42:12PM +0200, Milian Wolff escreveu:
> Hey Arnaldo,
> 
> I just wanted to try out your ongoing work to get file names printed in perf 
> trace:
> 
> ~~~~~~~~~~~~~~
> $ perf trace -e open ls builtin-trace.c
>      0.007 ( 0.007 ms): ls/26364 open(filename: 0x2a1b0d20, flags: CLOEXEC                             
> ) = -1 ENOENT No such file or directorybuiltin-trace.c
> 
>      0.022 ( 0.002 ms): ls/26364 open(filename: 0x2a1b0d20, flags: CLOEXEC                             
> ) = -1 ENOENT No such file or directory
> ...
> ~~~~~~~~~~~~~~
> 
> So apparently I still have to manually create the probe for vfs_getname. Could 
> this be simplified for the user? E.g. by adding a static tracepoint for that 

Yeah, this has to be simplified for the user.

> purpose, so one doesn't have to run `perf probe` manually? Also, note that 

That would be good, someone needs to try to convince upstream that one
more such tracepoint is warranted.

> `vfs_getname` is only mentioned in `man perf trace` for its `--tool_stats` 
> parameter, but the actual line to create the probe is nowhere to be found. 
> 
> Grepping the git log I found the following invocation, which errors out on my 
> Arch machine:
> 
> ~~~~~~~~~~~~~~~
> $ perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
> The /home/milian/.debug/.build-id/1a/03b857e3611b7a14fbf60d67cad7415706a933 
> file has no debug information.
> Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
>   Error: Failed to add events.
> ~~~~~~~~~~~~~~~
> 
> On a different Arch machine it's:
> 
> ~~~~~~~~~~~~~~~
> $ perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
> The /lib/modules/4.5.4-1-ARCH/build/vmlinux file has no debug information.
> Rebuild with CONFIG_DEBUG_INFO=y, or install an appropriate debuginfo package.
>   Error: Failed to add events.
> ~~~~~~~~~~~~~~~
> 
> On an Ubuntu 16.04 I'm getting this:
> 
> ~~~~~~~~~~~~~~~
> $ perf probe 'vfs_getname=getname_flags:72 pathname=filename:string'
> Failed to find the path for kernel: No such file or directory
>   Error: Failed to add events.
> ~~~~~~~~~~~~~~~
> 
> So does this really mean I'm out of luck?

For 'perf trace' to be a superset of strace it needs to pretty print
syscall args other than integers, and for that we need things like the
vfs_getname, or to go and look at several places where those syscalls
are copied to the kernel (move_addr_to_kernel, gename_flags), and back
(move_addr_to_user), etc.

I prototyped that with vfs_getname (if there is one such "wannabe
tracepoint" in place, set up with, say, perf probe, it will be used) but
never got around to actually try to push for a tracepoint for that, now
I'm briefly trying to deal with move_addr_to_kernel, to print the peer
addr in socket fds, but then I have to try to use BPF, as the
infrastructure that 'perf probe' uses in the kernel has no support for
memcpy, say, a struct (sockaddr_storage), just strings, which is enough
for "vfs_getname".

So, till we get there, the best you can do is to go from:

# perf probe -L getname_flags

to:

[root@jouet ~]# perf probe -L getname_flags:72
<getname_flags@/home/acme/git/linux/fs/namei.c:72>
     72         result->uptr = filename;
     73         result->aname = NULL;
                audit_getname(result);
     75         return result;
     76  }
         
         struct filename *
         getname(const char __user * filename)

[root@jouet ~]# 

And try adding the probe there.

Finding the right place without having debug info would be an option, we
would have to have a instruction decoder and find the right place,
tricky, brute forcish :-\

In a recent discussion (pvt) with Masami, we agreed that Ubuntu is even
worse in this regard, as we couldn't find the source code for the
running kernel easily, AFAIK it is in the linux-sources, but just as a
.bz2 file that you would have to uncompress somewhere :-\

Arch? I never tried it, would have to google for instructions on how to
use, say, systemtap (which is an older tool, so may have some tutorials
for Arch), and then set up the debuginfo/equivalent packages.

Yeah, the solution is to have tracepoints were such arguments are copied
to/from kernel/userspace, then no extra packages would be needed, etc.

- Arnaldo

Re: perf trace & vfs_getname

[Frank Ch. Eigler]

acme wrote:

> [...]
>> I just wanted to try out your ongoing work to get file names printed in perf 
>> trace:
>> 
>> $ perf trace -e open ls builtin-trace.c
>>      0.007 ( 0.007 ms): ls/26364 open(filename: 0x2a1b0d20, flags: CLOEXEC
>> ) = -1 ENOENT No such file or directorybuiltin-trace.c
>> [...]

> For 'perf trace' to be a superset of strace it needs to pretty print
> syscall args other than integers, and for that we need things like the
> vfs_getname, or to go and look at several places where those syscalls
> are copied to the kernel (move_addr_to_kernel, gename_flags), and back
> (move_addr_to_user), etc.

In systemtap's case, we copy from userspace & pretty-print right from
the event where we detect the system call, not via a secondary
tracepoint like vfs_getname that would need to have its values
saved/coordinated.


> [...]  Arch? I never tried it, would have to google for instructions
> on how to use, say, systemtap [...]

FYI, Arch is (in)famous for dropping debuginfo completely.  Systemtap
can work out-of-the-box based on symbol table info only, but many
users end up recompiling their kernel with CONFIG_DEBUG_INFO=1 to get
the full capabilities.

Just as demo, this may work for you (including file names):

# stap -e '
probe nd_syscall.* { printf("%s/%d %s(%s) ", execname(), tid(), name, argstr) }
probe nd_syscall.*.return { printf("-> %s\n", retstr) }
'

... which is the core of what

     /usr/share/doc/systemtap*/examples/*/strace.stp

does, sans process-filtering and other goodies.


- FChE