perf: relative path to source for perf probe?

perf: relative path to source for perf probe?

[Chase Douglas]

I'm trying to play with perf probe to insert trace events into a
running kernel, but I haven't found a way to specify relative
pathnames. For example, our build machines for Ubuntu build the kernel
inside /build/buildd/linux-2.6.34. If I want to use perf probe, it
seems I need to ensure the source exists in exactly the same location
on my machine.

Compare this to gdb, where I can put the source at
/home/cndougla/perf/build/buildd/linux-2.6.34 and specify a relative
path for sources of /home/cndougla/perf using the dir command. I have
read the perf code and I have not found any way to do this. I would be
interested in adding support for this as a cmd line option, but I'm
not very familiar with the rest of perf. Is this functionality
specific to perf-probe? Should this be in some logic in util/ or in
builtin-probe.c?

Thanks,

-- Chase

Re: perf: relative path to source for perf probe?

[Arnaldo Carvalho de Melo]

Em Thu, May 13, 2010 at 04:13:41PM +0200, Chase Douglas escreveu:
> I'm trying to play with perf probe to insert trace events into a
> running kernel, but I haven't found a way to specify relative
> pathnames. For example, our build machines for Ubuntu build the kernel
> inside /build/buildd/linux-2.6.34. If I want to use perf probe, it
> seems I need to ensure the source exists in exactly the same location
> on my machine.
> 
> Compare this to gdb, where I can put the source at
> /home/cndougla/perf/build/buildd/linux-2.6.34 and specify a relative
> path for sources of /home/cndougla/perf using the dir command. I have
> read the perf code and I have not found any way to do this. I would be
> interested in adding support for this as a cmd line option, but I'm
> not very familiar with the rest of perf. Is this functionality
> specific to perf-probe? Should this be in some logic in util/ or in
> builtin-probe.c?

Look in tools/perf/util/symbol.c, these variables are the ones tools use
to govern how the symbol system work wrt finding vmlinux:

symbol_conf.use_vmlinux_path
symbol_conf.vmlinux_name

In addition to this it will use what is in ~/.debug/ if it has a
build-id in the perf.data header.

The changes for support kvm also touched this and allow for some
prefixing to look for guest symbols, generalizing that to make guest
kernel vmlinux + modules relative location be reused to look for
relative location for host kernel vmlinux + modules seems the way to go.

Probably you won't find this if you look at the old 2.6.34 codebase
(yay, for tools/perf it is already hundreds of patches behind :-)), so
I recomend you look at what we have in linux-2.6-tip, branch
tip/perf/core, what is in 2.6.34 is in tip/perf/urgent and right now,
IIRC, just a one patch that is not in 2.6.34 proper, a cherry-pick from
tip/perf/core, no less :-)

- Arnaldo

Re: perf: relative path to source for perf probe?

[Chase Douglas]

On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
<acme@ghostprotocols.net> wrote:
> Probably you won't find this if you look at the old 2.6.34 codebase
> (yay, for tools/perf it is already hundreds of patches behind :-)), so
> I recomend you look at what we have in linux-2.6-tip, branch
> tip/perf/core, what is in 2.6.34 is in tip/perf/urgent and right now,
> IIRC, just a one patch that is not in 2.6.34 proper, a cherry-pick from
> tip/perf/core, no less :-)

Where are these branches? I've looked at:

http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=summary

but I don't see these branches listed.

Thanks for the pointers!

-- Chase

Re: perf: relative path to source for perf probe?

[Arnaldo Carvalho de Melo]

Em Thu, May 13, 2010 at 05:51:45PM +0200, Chase Douglas escreveu:
> On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
> <acme@ghostprotocols.net> wrote:
> > Probably you won't find this if you look at the old 2.6.34 codebase
> > (yay, for tools/perf it is already hundreds of patches behind :-)), so
> > I recomend you look at what we have in linux-2.6-tip, branch
> > tip/perf/core, what is in 2.6.34 is in tip/perf/urgent and right now,
> > IIRC, just a one patch that is not in 2.6.34 proper, a cherry-pick from
> > tip/perf/core, no less :-)
> 
> Where are these branches? I've looked at:
> 
> http://git.kernel.org/?p=linux/kernel/git/acme/linux-2.6.git;a=summary
> 
> but I don't see these branches listed.

Well, if you decide to go from my tree, its all in the "perf" branch,
but I'd recommend you follow Ingo, my git tree, at the moment, is just a
conduit to get things out to him, which happens without noticeable delay
once I send a pull request.
 
> Thanks for the pointers!

You're welcome, please let us know about any other thing we may be of
help to make you of help to us ;-)

> -- Chase

Re: perf: relative path to source for perf probe?

[Chase Douglas]

On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
<acme@ghostprotocols.net> wrote:
> Look in tools/perf/util/symbol.c, these variables are the ones tools use
> to govern how the symbol system work wrt finding vmlinux:
>
> symbol_conf.use_vmlinux_path
> symbol_conf.vmlinux_name
>
> In addition to this it will use what is in ~/.debug/ if it has a
> build-id in the perf.data header.
>
> The changes for support kvm also touched this and allow for some
> prefixing to look for guest symbols, generalizing that to make guest
> kernel vmlinux + modules relative location be reused to look for
> relative location for host kernel vmlinux + modules seems the way to go.

After reading this some more, I think we are talking about two
different things. I think your notes above are referring to locating
the vmlinux image and other debug symbols. This seems to work fine for
me using the -k flag. However, I'm encountering an issue where the
source code location isn't found. I think this is fairly specific to
the probe command of perf, which is fairly new, so maybe it's missing
the same level of support that the symbol finding code has?

Thanks,

-- Chase

Re: perf: relative path to source for perf probe?

[Arnaldo Carvalho de Melo]

Em Thu, May 13, 2010 at 06:01:10PM +0200, Chase Douglas escreveu:
> On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
> <acme@ghostprotocols.net> wrote:
> > Look in tools/perf/util/symbol.c, these variables are the ones tools use
> > to govern how the symbol system work wrt finding vmlinux:
> >
> > symbol_conf.use_vmlinux_path
> > symbol_conf.vmlinux_name
> >
> > In addition to this it will use what is in ~/.debug/ if it has a
> > build-id in the perf.data header.
> >
> > The changes for support kvm also touched this and allow for some
> > prefixing to look for guest symbols, generalizing that to make guest
> > kernel vmlinux + modules relative location be reused to look for
> > relative location for host kernel vmlinux + modules seems the way to go.
> 
> After reading this some more, I think we are talking about two
> different things. I think your notes above are referring to locating
> the vmlinux image and other debug symbols. This seems to work fine for
> me using the -k flag.

What about finding the associated modules?

> However, I'm encountering an issue where the source code location
> isn't found. I think this is fairly specific to the probe command of
> perf, which is fairly new, so maybe it's missing the same level of
> support that the symbol finding code has?

Annotation needs it, in top, report, too.

But probably (haven't looked at the sources) the way to specify it will
be different, since we rely on objdump to do the disassembly, we need to
inform it where the sources are, but look at the source of all this, the
relevant DWARF tags:

 <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
    < c>   DW_AT_stmt_list   : 0x0      
    <10>   DW_AT_ranges      : 0x0      
    <14>   DW_AT_name        : /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S       
    <57>   DW_AT_comp_dir    : /home/acme_unencrypted/git/build/v2.6.34-rc6-

- Arnaldo

Re: perf: relative path to source for perf probe?

[Masami Hiramatsu]

Arnaldo Carvalho de Melo wrote:
> Em Thu, May 13, 2010 at 06:01:10PM +0200, Chase Douglas escreveu:
>> On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
>> <acme@ghostprotocols.net> wrote:
>>> Look in tools/perf/util/symbol.c, these variables are the ones tools use
>>> to govern how the symbol system work wrt finding vmlinux:
>>>
>>> symbol_conf.use_vmlinux_path
>>> symbol_conf.vmlinux_name
>>>
>>> In addition to this it will use what is in ~/.debug/ if it has a
>>> build-id in the perf.data header.
>>>
>>> The changes for support kvm also touched this and allow for some
>>> prefixing to look for guest symbols, generalizing that to make guest
>>> kernel vmlinux + modules relative location be reused to look for
>>> relative location for host kernel vmlinux + modules seems the way to go.
>>
>> After reading this some more, I think we are talking about two
>> different things. I think your notes above are referring to locating
>> the vmlinux image and other debug symbols. This seems to work fine for
>> me using the -k flag.
> 
> What about finding the associated modules?
> 
>> However, I'm encountering an issue where the source code location
>> isn't found. I think this is fairly specific to the probe command of
>> perf, which is fairly new, so maybe it's missing the same level of
>> support that the symbol finding code has?
> 
> Annotation needs it, in top, report, too.
> 
> But probably (haven't looked at the sources) the way to specify it will
> be different, since we rely on objdump to do the disassembly, we need to
> inform it where the sources are, but look at the source of all this, the
> relevant DWARF tags:
> 
>  <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>     < c>   DW_AT_stmt_list   : 0x0      
>     <10>   DW_AT_ranges      : 0x0      
>     <14>   DW_AT_name        : /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S       
>     <57>   DW_AT_comp_dir    : /home/acme_unencrypted/git/build/v2.6.34-rc6-

Actually, these doesn't help us so much. DW_AT_name shows the absolute path
of source file (and perf probe already have it), and DW_AT_comp_dir just 
shows where the object code has been stored (or where the make command ran).

In this case (after compiling kernel, user moved its source into another
directory), we have to find the top directory of this source code.

Imagine that, if user compiled code under /home/user/git-ksrc/, and 
it was moved under  /usr/src/2.6.x/, perf probe will be given an option
'-s /usr/src/2.6.x/'.

If we know the top directory is /home/user/git-ksrc/, we can replace
it with /usr/src/2.6.x/. However, dwarf doesn't provide this information.
So we need to assume that where is the top directory in absolute path.

I think, we can do it by brute force (I'm not sure how gdb does it).

Replace the root directory with given directory and try to open it.

 fopen("/usr/src/2.6.x/" "home/user/git-ksrc/kernel/kprobes.c"); -> NG

If there is no file exist, remove next directory and add given directory
and try to open it again.

 fopen("/usr/src/2.6.x/" "user/git-ksrc/kernel/kprobes.c"); -> NG

And repeat it until the file exist.

 fopen("/usr/src/2.6.x/" "git-ksrc/kernel/kprobes.c"); -> NG

 fopen("/usr/src/2.6.x/" "kernel/kprobes.c"); -> OK!


Thank you,

-- 
Masami Hiramatsu
e-mail: mhiramat@redhat.com

Re: perf: relative path to source for perf probe?

[Arnaldo Carvalho de Melo]

Em Thu, May 13, 2010 at 05:23:14PM -0400, Masami Hiramatsu escreveu:
> Arnaldo Carvalho de Melo wrote:
> > Em Thu, May 13, 2010 at 06:01:10PM +0200, Chase Douglas escreveu:
> >> On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
> >> <acme@ghostprotocols.net> wrote:
> >>> Look in tools/perf/util/symbol.c, these variables are the ones tools use
> >>> to govern how the symbol system work wrt finding vmlinux:
> >>>
> >>> symbol_conf.use_vmlinux_path
> >>> symbol_conf.vmlinux_name
> >>>
> >>> In addition to this it will use what is in ~/.debug/ if it has a
> >>> build-id in the perf.data header.
> >>>
> >>> The changes for support kvm also touched this and allow for some
> >>> prefixing to look for guest symbols, generalizing that to make guest
> >>> kernel vmlinux + modules relative location be reused to look for
> >>> relative location for host kernel vmlinux + modules seems the way to go.
> >>
> >> After reading this some more, I think we are talking about two
> >> different things. I think your notes above are referring to locating
> >> the vmlinux image and other debug symbols. This seems to work fine for
> >> me using the -k flag.
> > 
> > What about finding the associated modules?
> > 
> >> However, I'm encountering an issue where the source code location
> >> isn't found. I think this is fairly specific to the probe command of
> >> perf, which is fairly new, so maybe it's missing the same level of
> >> support that the symbol finding code has?
> > 
> > Annotation needs it, in top, report, too.
> > 
> > But probably (haven't looked at the sources) the way to specify it will
> > be different, since we rely on objdump to do the disassembly, we need to
> > inform it where the sources are, but look at the source of all this, the
> > relevant DWARF tags:
> > 
> >  <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
> >     < c>   DW_AT_stmt_list   : 0x0      
> >     <10>   DW_AT_ranges      : 0x0      
> >     <14>   DW_AT_name        : /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S       
> >     <57>   DW_AT_comp_dir    : /home/acme_unencrypted/git/build/v2.6.34-rc6-
> 
> Actually, these doesn't help us so much. DW_AT_name shows the absolute path
> of source file (and perf probe already have it), and DW_AT_comp_dir just 
> shows where the object code has been stored (or where the make command ran).

IOW:

real_path = user_provided_path + (DW_AT_name - DW_AT_comp_dir)

+ == concat

- remove initial string

if the project is build in place, i.e. in the kernel case, when one
doesn't use 'make O='

But then, given user_provided_path what we can do is to find a match
from the end of DW_AT_name, thus infering the missing (non-existent in
DWARF) DW_AT_source_dir tag :-)
 
> In this case (after compiling kernel, user moved its source into another
> directory), we have to find the top directory of this source code.
> 
> Imagine that, if user compiled code under /home/user/git-ksrc/, and 
> it was moved under  /usr/src/2.6.x/, perf probe will be given an option
> '-s /usr/src/2.6.x/'.

So, for the DW_AT_name example above, we would have these files:

At build time (DW_AT_name):

/home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S

And at analysis time (using the path where the user moved all the source
tree):

/usr/src/2.6.x/arch/x86/kernel/head_64.S

Putting then side by side:

/home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S
                          /usr/src/2.6.x/arch/x86/kernel/head_64.S

So know we know what DW_AT_source_dir is.

Implementation is left to the reader 8-)

- Arnaldo

Re: perf: relative path to source for perf probe?

[Chase Douglas]

On Thu, May 13, 2010 at 11:46 PM, Arnaldo Carvalho de Melo
<acme@ghostprotocols.net> wrote:
> IOW:
>
> real_path = user_provided_path + (DW_AT_name - DW_AT_comp_dir)
>
> + == concat
>
> - remove initial string
>
> if the project is build in place, i.e. in the kernel case, when one
> doesn't use 'make O='
>
> But then, given user_provided_path what we can do is to find a match
> from the end of DW_AT_name, thus infering the missing (non-existent in
> DWARF) DW_AT_source_dir tag :-)
>
>> In this case (after compiling kernel, user moved its source into another
>> directory), we have to find the top directory of this source code.
>>
>> Imagine that, if user compiled code under /home/user/git-ksrc/, and
>> it was moved under  /usr/src/2.6.x/, perf probe will be given an option
>> '-s /usr/src/2.6.x/'.
>
> So, for the DW_AT_name example above, we would have these files:
>
> At build time (DW_AT_name):
>
> /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S
>
> And at analysis time (using the path where the user moved all the source
> tree):
>
> /usr/src/2.6.x/arch/x86/kernel/head_64.S
>
> Putting then side by side:
>
> /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S
>                          /usr/src/2.6.x/arch/x86/kernel/head_64.S
>
> So know we know what DW_AT_source_dir is.
>
> Implementation is left to the reader 8-)

Thank you both for this discussion! I think I have enough information
to try to create an implementation. Now I just need to find the time
to do it :).

-- Chase

Re: perf: relative path to source for perf probe?

[Masami Hiramatsu]

Arnaldo Carvalho de Melo wrote:
> Em Thu, May 13, 2010 at 05:23:14PM -0400, Masami Hiramatsu escreveu:
>> Arnaldo Carvalho de Melo wrote:
>>> Em Thu, May 13, 2010 at 06:01:10PM +0200, Chase Douglas escreveu:
>>>> On Thu, May 13, 2010 at 4:58 PM, Arnaldo Carvalho de Melo
>>>> <acme@ghostprotocols.net> wrote:
>>>>> Look in tools/perf/util/symbol.c, these variables are the ones tools use
>>>>> to govern how the symbol system work wrt finding vmlinux:
>>>>>
>>>>> symbol_conf.use_vmlinux_path
>>>>> symbol_conf.vmlinux_name
>>>>>
>>>>> In addition to this it will use what is in ~/.debug/ if it has a
>>>>> build-id in the perf.data header.
>>>>>
>>>>> The changes for support kvm also touched this and allow for some
>>>>> prefixing to look for guest symbols, generalizing that to make guest
>>>>> kernel vmlinux + modules relative location be reused to look for
>>>>> relative location for host kernel vmlinux + modules seems the way to go.
>>>>
>>>> After reading this some more, I think we are talking about two
>>>> different things. I think your notes above are referring to locating
>>>> the vmlinux image and other debug symbols. This seems to work fine for
>>>> me using the -k flag.
>>>
>>> What about finding the associated modules?
>>>
>>>> However, I'm encountering an issue where the source code location
>>>> isn't found. I think this is fairly specific to the probe command of
>>>> perf, which is fairly new, so maybe it's missing the same level of
>>>> support that the symbol finding code has?
>>>
>>> Annotation needs it, in top, report, too.
>>>
>>> But probably (haven't looked at the sources) the way to specify it will
>>> be different, since we rely on objdump to do the disassembly, we need to
>>> inform it where the sources are, but look at the source of all this, the
>>> relevant DWARF tags:
>>>
>>>  <0><b>: Abbrev Number: 1 (DW_TAG_compile_unit)
>>>     < c>   DW_AT_stmt_list   : 0x0      
>>>     <10>   DW_AT_ranges      : 0x0      
>>>     <14>   DW_AT_name        : /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S       
>>>     <57>   DW_AT_comp_dir    : /home/acme_unencrypted/git/build/v2.6.34-rc6-
>>
>> Actually, these doesn't help us so much. DW_AT_name shows the absolute path
>> of source file (and perf probe already have it), and DW_AT_comp_dir just 
>> shows where the object code has been stored (or where the make command ran).
> 
> IOW:
> 
> real_path = user_provided_path + (DW_AT_name - DW_AT_comp_dir)
> 
> + == concat
> 
> - remove initial string
> 
> if the project is build in place, i.e. in the kernel case, when one
> doesn't use 'make O='
> 
> But then, given user_provided_path what we can do is to find a match
> from the end of DW_AT_name, thus infering the missing (non-existent in
> DWARF) DW_AT_source_dir tag :-)

I recommend you to find a match from the top of DW_AT_name, because
we have many same-name files under kernel/ and arch/XXX/kernel/ :P


>> In this case (after compiling kernel, user moved its source into another
>> directory), we have to find the top directory of this source code.
>>
>> Imagine that, if user compiled code under /home/user/git-ksrc/, and 
>> it was moved under  /usr/src/2.6.x/, perf probe will be given an option
>> '-s /usr/src/2.6.x/'.
> 
> So, for the DW_AT_name example above, we would have these files:
> 
> At build time (DW_AT_name):
> 
> /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S
> 
> And at analysis time (using the path where the user moved all the source
> tree):
> 
> /usr/src/2.6.x/arch/x86/kernel/head_64.S
> 
> Putting then side by side:
> 
> /home/acme_unencrypted/git/linux-2.6-tip/arch/x86/kernel/head_64.S
>                           /usr/src/2.6.x/arch/x86/kernel/head_64.S
> 
> So know we know what DW_AT_source_dir is.

Sure :)

> 
> Implementation is left to the reader 8-)
> 
> - Arnaldo

Thank you,

-- 
Masami Hiramatsu
e-mail: mhiramat@redhat.com

Re: perf: relative path to source for perf probe?

[Masami Hiramatsu]

Chase Douglas wrote:
> I'm trying to play with perf probe to insert trace events into a
> running kernel, but I haven't found a way to specify relative
> pathnames. For example, our build machines for Ubuntu build the kernel
> inside /build/buildd/linux-2.6.34. If I want to use perf probe, it
> seems I need to ensure the source exists in exactly the same location
> on my machine.

Right, perf-probe currently doesn't support relative *source* path option.
It just uses a source code path information in dwarf. I need to dig it more
into dwarf, especially kernel debuginfo has those information.

That could be a good improvement idea for perf probe (adding -s option?).

> Compare this to gdb, where I can put the source at
> /home/cndougla/perf/build/buildd/linux-2.6.34 and specify a relative
> path for sources of /home/cndougla/perf using the dir command. I have
> read the perf code and I have not found any way to do this. I would be
> interested in adding support for this as a cmd line option, but I'm
> not very familiar with the rest of perf. Is this functionality
> specific to perf-probe? Should this be in some logic in util/ or in
> builtin-probe.c?

Both of it. Adding -s option handling code to builtin-probe.c and
actual path modifying code in show_line_range()@util/probe-event.c.
(I assume you'd like to use -L option)

Thank you,

-- 
Masami Hiramatsu
e-mail: mhiramat@redhat.com