Re: malloc() tracing in perf?

Re: malloc() tracing in perf?

[Pekka Enberg]

Peter Zijlstra wrote:
> On Thu, 2009-08-06 at 10:48 +0300, Pekka Enberg wrote:
>> Hi,
>>
>> It's me again :-).
>>
>> I have a little user-space application that is pretty memory hungry and 
>> I want to understand why. I started to google around for a memory 
>> profiler or a malloc() tracer but didn't seem to find anything really 
>> useful.
>>
>> But then it hit me, why can't I have kmemtrace + perf but for 
>> user-space? Something like the "Malloc Trace" shown here:
>>
>> http://developer.apple.com/documentation/developertools/conceptual/SharkUserGuide/OtherProfilingandTracingTechniques/OtherProfilingandTracingTechniques.html#//apple_ref/doc/uid/TP40005233-CH6-SW17
>>
>> Does this sound like something that could/should be part of "perf"? How 
>> would all this work anyway? Can we intercept malloc() and free() 
>> somehow? Where would the data be pushed? Am I just going perf-crazy and 
>> trying to turn it into a swiss army knife because it's so easy to use?-)
> 
> OK, you just trod into a wasp's nest :-)
> 
> 
> That sounds like uprobes, the equivalent of kprobes but for userspace.
> 
> I seem to have heard people are working on such a thing, but I can't
> seem to find a single LKML post with 'uprobe' in the subject in the past
> two years or something (except for MTUprobe) -- so I guess its not
> really going anywhere any fast.
> 
> Now doing probes on userspace is hard because you need to know more
> about the userspace bits than a kernel really ought to be interested in.
> 
> Then again, the only way to extract bits from userspace is to stop it --
> now one could do that using [pu]trace and have some monitoring app prod
> at it like any other debugger would, and I think this is the approach
> suggested by some (hch iirc).
> 
> Others seem to think we ought to stuff all this into the kernel, I can
> only imagine the pain that will cause, since you need to teach the
> kernel about these instrumentation sites' context, so I can only imagine
> it'd be through a kernel module interface much like system-tap does
> (they would be doing the in-kernel bit).
> 
> Then there are the tracer folks who also want to collect userspace
> traces. Some have proposed a sys_trace() call, others want to play silly
> games with mmap() and then there is the uprobe idea. Others (tglx and
> me) proposed letting userspace log itself and post-merge all the various
> trace buffers to get a complete picture.
> 
> 
> Anyway, like you say, it has uses (potentially very powerful ones),
> Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite
> agreed on how to do it :-)
> 
> 
> And here I see LKML isn't on the CC list, perhaps we should?

Oh sure, lets do that, half of the people seem to be on the CC anyway 
now :-).

			Pekka

Re: malloc() tracing in perf?

[Pekka Enberg]

Pekka Enberg wrote:
> Hi Peter,
> 
> On Thu, 2009-08-06 at 10:48 +0300, Pekka Enberg wrote:
>>> Hi,
>>>
>>> It's me again :-).
>>>
>>> I have a little user-space application that is pretty memory hungry 
>>> and I want to understand why. I started to google around for a memory 
>>> profiler or a malloc() tracer but didn't seem to find anything really 
>>> useful.
>>>
>>> But then it hit me, why can't I have kmemtrace + perf but for 
>>> user-space? Something like the "Malloc Trace" shown here:
>>>
>>> http://developer.apple.com/documentation/developertools/conceptual/SharkUserGuide/OtherProfilingandTracingTechniques/OtherProfilingandTracingTechniques.html#//apple_ref/doc/uid/TP40005233-CH6-SW17 
>>>
>>>
>>> Does this sound like something that could/should be part of "perf"? 
>>> How would all this work anyway? Can we intercept malloc() and free() 
>>> somehow? Where would the data be pushed? Am I just going perf-crazy 
>>> and trying to turn it into a swiss army knife because it's so easy to 
>>> use?-)
> 
> Peter Zijlstra wrote:
>> OK, you just trod into a wasp's nest :-)
>>
>> That sounds like uprobes, the equivalent of kprobes but for userspace.
>>
>> I seem to have heard people are working on such a thing, but I can't
>> seem to find a single LKML post with 'uprobe' in the subject in the past
>> two years or something (except for MTUprobe) -- so I guess its not
>> really going anywhere any fast.
> 
> [snip]
> 
> Right. Are dynamic tracepoints what we want for malloc() and free() 
> interception, though? I guess we can just do generic "userspace function 
> called" events and record the passed parameters. Then "perf memreport" 
> can just go find all the malloc() and free() calls and construct a 
> memory profile out of it?
> 
> That said, I'd also be interested in wiring my userspace VM garbage 
> collector to emit similar alloc and free events as well (because perf 
> already has a JIT symbol map) which makes me think we want generic 
> "userspace allocated memory" events.

Argh, I dropped LKML from the CC. Sorry for the duplicate email!

			Pekka

Re: malloc() tracing in perf?

[Peter Zijlstra]

On Thu, 2009-08-06 at 12:19 +0300, Pekka Enberg wrote:
> Pekka Enberg wrote:
> > Hi Peter,
> > 
> > On Thu, 2009-08-06 at 10:48 +0300, Pekka Enberg wrote:
> >>> Hi,
> >>>
> >>> It's me again :-).
> >>>
> >>> I have a little user-space application that is pretty memory hungry 
> >>> and I want to understand why. I started to google around for a memory 
> >>> profiler or a malloc() tracer but didn't seem to find anything really 
> >>> useful.
> >>>
> >>> But then it hit me, why can't I have kmemtrace + perf but for 
> >>> user-space? Something like the "Malloc Trace" shown here:
> >>>
> >>> http://developer.apple.com/documentation/developertools/conceptual/SharkUserGuide/OtherProfilingandTracingTechniques/OtherProfilingandTracingTechniques.html#//apple_ref/doc/uid/TP40005233-CH6-SW17 
> >>>
> >>>
> >>> Does this sound like something that could/should be part of "perf"? 
> >>> How would all this work anyway? Can we intercept malloc() and free() 
> >>> somehow? Where would the data be pushed? Am I just going perf-crazy 
> >>> and trying to turn it into a swiss army knife because it's so easy to 
> >>> use?-)
> > 
> > Peter Zijlstra wrote:
> >> OK, you just trod into a wasp's nest :-)
> >>
> >> That sounds like uprobes, the equivalent of kprobes but for userspace.
> >>
> >> I seem to have heard people are working on such a thing, but I can't
> >> seem to find a single LKML post with 'uprobe' in the subject in the past
> >> two years or something (except for MTUprobe) -- so I guess its not
> >> really going anywhere any fast.
> > 
> > [snip]
> > 
> > Right. Are dynamic tracepoints what we want for malloc() and free() 
> > interception, though? I guess we can just do generic "userspace function 
> > called" events and record the passed parameters. Then "perf memreport" 
> > can just go find all the malloc() and free() calls and construct a 
> > memory profile out of it?
> > 
> > That said, I'd also be interested in wiring my userspace VM garbage 
> > collector to emit similar alloc and free events as well (because perf 
> > already has a JIT symbol map) which makes me think we want generic 
> > "userspace allocated memory" events.

Right, there is the distinction between static and dynamic tracepoints,
and I'm not sure how that'll work out. Most people who've looked at the
subject matter are on CC, so I'm hoping someone will expand.

That said, we could simply LD_PRELOAD malloc and free (new and delete)
like any other memory debugger out there already does, that's not rocket
science (in fact, I've done so on many an occasion).

I remember mpatrol being a hackable code base, but I just looked it up
and its LGPL-v3 so I'm not sure we should be borrowing code from there.
Maybe there is another such creature out there we can borrow code from.

I've got similar ideas for your JIT stuff, you now open-code the tmp
file writing, ideally I'd like to push that into a library and have you
link against something with weak stubs and let perf LD_PRELOAD real code
which will do whatever is needed. That'd rid you of having to know about
where and what to write.

Anyway, malloc/free are easy, but like you say, you might be interested
in other functions which aren't as easy.

Re: malloc() tracing in perf?

[Frank Ch. Eigler]

Hi -

On Thu, Aug 06, 2009 at 10:16:50AM +0200, Peter Zijlstra wrote:
> [...]
> > But then it hit me, why can't I have kmemtrace + perf but for 
> > user-space? Something like the "Malloc Trace" shown here:
> > [...]

> I seem to have heard people are working on such a thing, but I can't
> seem to find a single LKML post with 'uprobe' in the subject in the
> past two years [...]

That work is ongoing, and being discussed on utrace-devel@redhat.com,
since it is a prerequisite.  One of the widgets that is being proposed
for it is an ftrace engine frontend for uprobes, paralleling the one
mhiramat wrote for kprobes.


> Now doing probes on userspace is hard because you need to know more
> about the userspace bits than a kernel really ought to be interested
> in.  [...]  Anyway, like you say, it has uses (potentially very
> powerful ones), Sun/Apple do it with Dtrace, Linux wants it but I
> don't think we quite agreed on how to do it :-)

While these deliberations are ongoing, you can use systemtap.  Probing
random places in userspace is about as casual as probing the kernel:


# stap -e '
   probe process("/lib/libc.so.6").function("malloc").return,
         process("/lib/libc.so.6").function("free").return
   { println(probfunc()," ",$$parms," ",$$return) }
' -c 'ls'   # -c 'CMD ARGS'

__libc_malloc bytes=0x238  return=0x1524010 
__libc_malloc bytes=0x238  return=0x1524010 
__libc_malloc bytes=0x78  return=0x1524250 
__libc_malloc bytes=0xa  return=0x15242d0 
__libc_free mem=0x1524250  
__libc_free mem=0x1524010  
__libc_malloc bytes=0x64  return=0x1524010 
__libc_malloc bytes=0x16  return=0x1524080 
__libc_free mem=0x1524010  
__libc_malloc bytes=0x64  return=0x1524010 
[...]
__libc_free mem=0x1524490  
__libc_free mem=0x1524500  
__libc_free mem=0x15245e0  
__libc_free mem=0x1524620  
__libc_free mem=0x1524430  
__libc_free mem=0x1524570  
__libc_free mem=0x1524660  
__libc_free mem=0x15246d0  
__libc_free mem=0x15242d0  

You can google some other success stories or ask for more help at
systemtap@sourceware.org.


- FChE

Re: malloc() tracing in perf?

[Peter Zijlstra]

On Thu, 2009-08-06 at 07:20 -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Aug 06, 2009 at 10:16:50AM +0200, Peter Zijlstra wrote:
> > [...]
> > > But then it hit me, why can't I have kmemtrace + perf but for 
> > > user-space? Something like the "Malloc Trace" shown here:
> > > [...]
> 
> > I seem to have heard people are working on such a thing, but I can't
> > seem to find a single LKML post with 'uprobe' in the subject in the
> > past two years [...]
> 
> That work is ongoing, and being discussed on utrace-devel@redhat.com,
> since it is a prerequisite. 

Still hiding the discussion and the design never helped anybody.

You might think it ready and then post it to LKML only to have it ripped
apart -- have the discussion openly please.

> > Now doing probes on userspace is hard because you need to know more
> > about the userspace bits than a kernel really ought to be interested
> > in.  [...]  Anyway, like you say, it has uses (potentially very
> > powerful ones), Sun/Apple do it with Dtrace, Linux wants it but I
> > don't think we quite agreed on how to do it :-)
> 
> While these deliberations are ongoing, you can use systemtap.  Probing
> random places in userspace is about as casual as probing the kernel:

Right, but that still doesn't tell us anything on how you're doing that,
does it?

Re: malloc() tracing in perf?

[Frank Ch. Eigler]

Hi -

On Thu, Aug 06, 2009 at 01:28:14PM +0200, Peter Zijlstra wrote:
> [...]
> > That work is ongoing, and being discussed on utrace-devel@redhat.com,
> > since it is a prerequisite. 
> 
> Still hiding the discussion and the design never helped anybody.

I guess "hiding" is a matter of opinion.


> > While these deliberations are ongoing, you can use systemtap.  Probing
> > random places in userspace is about as casual as probing the kernel:
> 
> Right, but that still doesn't tell us anything on how you're doing that,
> does it?

Since you asked...  the probe process("/lib64/libc.so.6") points
systemtap to a shared library file, whose symbol table & debug data
gives us information about what functions & parameters are available.
Among other things, we record a shared-library base-relative address
for the function.

At run time, we monitor the entire system (or just a given process if
-x PID/-c CMD was specified) to see when that shared library gets
loaded.  (This in turn is done with a utrace-based hook of the mmap
syscall - see "task_finder" in our sources).  When it's loaded, we can
finally compute a run-time address for the probed malloc/free
functions.  At this point we insert a uprobe (using the uprobes module
API, which also relies on utrace.)  Should the shared library become
unmapped, or the application exit, the uprobe will be removed.

When/if the uprobe eventually hits, we run the compiled probe handler.
As usual, it can do lots of stuff beside just simple tracing, but in
the case of that statement (println ($$parms ...))  that's all we do.
$$parms was expanded at compilation time to the list of variables, and
instructions about how to fetch their values at run time.  So now
those instructions are run, their values formatted, and the tracing
message sent to our buffer.

This overview skims over issues related to return probes, tracing
buffer manipulations, and much other stuff.


All the code for this is hidden in plain sight in every systemtap
release, so please feel free to refer to that and/or ask more detailed
questions.


- FChE

Re: malloc() tracing in perf?

[Peter Zijlstra]

On Thu, 2009-08-06 at 07:55 -0400, Frank Ch. Eigler wrote:
> Hi -
> 
> On Thu, Aug 06, 2009 at 01:28:14PM +0200, Peter Zijlstra wrote:
> > [...]
> > > That work is ongoing, and being discussed on utrace-devel@redhat.com,
> > > since it is a prerequisite. 
> > 
> > Still hiding the discussion and the design never helped anybody.
> 
> I guess "hiding" is a matter of opinion.

Yeah in plain sight where nobody looks, really I thought the idea was to
get utrace upstream, this means posting to linux-kernel. Going on LKML
posts I'd say the utrace project was dead. Same goes for uprobe.

I've told this before and I'll say it again, post to LKML.

> > > While these deliberations are ongoing, you can use systemtap.  Probing
> > > random places in userspace is about as casual as probing the kernel:
> > 
> > Right, but that still doesn't tell us anything on how you're doing that,
> > does it?
> 
> Since you asked...  the probe process("/lib64/libc.so.6") points
> systemtap to a shared library file, whose symbol table & debug data
> gives us information about what functions & parameters are available.
> Among other things, we record a shared-library base-relative address
> for the function.
> 
> At run time, we monitor the entire system (or just a given process if
> -x PID/-c CMD was specified) to see when that shared library gets
> loaded.  (This in turn is done with a utrace-based hook of the mmap
> syscall - see "task_finder" in our sources). 

Does this also iterate the already existing tasks to find if it was
already mmap()ed?

> This overview skims over issues related to return probes, tracing
> buffer manipulations, and much other stuff.

Right, so you basically read the (dwarf2) debug info for a particular
lib/symbol and generate a kernel module that knows about that and then
insert that into the kernel to act as a uprobe handler?

Uprobe will then do the code rewrite on mmap() time to insert a trap
much like kprobe does? What if its a JITed code section and the JIT
rewrites it? Will uprobes detect that?

> All the code for this is hidden in plain sight in every systemtap
> release, so please feel free to refer to that and/or ask more detailed
> questions.

I thought the goal was to get stuff upstream, but if you want to
continue living in la-la land and not bother with upstream Linux then so
be it I guess :-(

Re: malloc() tracing in perf?

[Mathieu Desnoyers]

* Peter Zijlstra (a.p.zijlstra@chello.nl) wrote:
> On Thu, 2009-08-06 at 10:48 +0300, Pekka Enberg wrote:
> > Hi,
> > 
> > It's me again :-).
> > 
> > I have a little user-space application that is pretty memory hungry and 
> > I want to understand why. I started to google around for a memory 
> > profiler or a malloc() tracer but didn't seem to find anything really 
> > useful.
> > 
> > But then it hit me, why can't I have kmemtrace + perf but for 
> > user-space? Something like the "Malloc Trace" shown here:
> > 
> > http://developer.apple.com/documentation/developertools/conceptual/SharkUserGuide/OtherProfilingandTracingTechniques/OtherProfilingandTracingTechniques.html#//apple_ref/doc/uid/TP40005233-CH6-SW17
> > 
> > Does this sound like something that could/should be part of "perf"? How 
> > would all this work anyway? Can we intercept malloc() and free() 
> > somehow? Where would the data be pushed? Am I just going perf-crazy and 
> > trying to turn it into a swiss army knife because it's so easy to use?-)
> 
> OK, you just trod into a wasp's nest :-)
> 
> 
> That sounds like uprobes, the equivalent of kprobes but for userspace.
> 
> I seem to have heard people are working on such a thing, but I can't
> seem to find a single LKML post with 'uprobe' in the subject in the past
> two years or something (except for MTUprobe) -- so I guess its not
> really going anywhere any fast.
> 
> Now doing probes on userspace is hard because you need to know more
> about the userspace bits than a kernel really ought to be interested in.
> 
> Then again, the only way to extract bits from userspace is to stop it --
> now one could do that using [pu]trace and have some monitoring app prod
> at it like any other debugger would, and I think this is the approach
> suggested by some (hch iirc).
> 
> Others seem to think we ought to stuff all this into the kernel, I can
> only imagine the pain that will cause, since you need to teach the
> kernel about these instrumentation sites' context, so I can only imagine
> it'd be through a kernel module interface much like system-tap does
> (they would be doing the in-kernel bit).
> 
> Then there are the tracer folks who also want to collect userspace
> traces. Some have proposed a sys_trace() call, others want to play silly
> games with mmap() and then there is the uprobe idea. Others (tglx and
> me) proposed letting userspace log itself and post-merge all the various
> trace buffers to get a complete picture.
> 

Hi !

Since we are comparing solutions, here is the path we are taking in
LTTng. We choose to let userspace log itself and merge the
kernel/userspace trace streams at post-processing. Past winter,
Pierre-Marc has been working on a port of the LTTng kernel tracer to
userspace. We named it "ust" library (for userspace tracing). Although
it still requires some polishing, it should be a generally usable state
(currently only on x86 with synchronized TSCs).

The plan is to gradually instrument userspace libraries and applications
with tracepoints, just like we are currently doing in the kernel.

The main difference between our approach and Dtrace is that Dtrace
requires a trap when userspace instrumentation is hit (yeah, even for
static instrumentation, isn't it a shame ?), adding a huge performance
overhead (especially on x86). Using tracepoints and staying in userspace
to log data helps keeping the performance overhead low.

For more info on lib ust :

Combined Tracing of the Kernel and Applications with LTTng at Linux
Symposium 2009 (presentation)
http://lttng.org/sites/default/files/presentation.pdf

Access to the git tree:

http://git.dorsal.polymtl.ca/?p=ust.git
(please note that we are about 99.9% done in asking permissions or
rewriting code to distribute under LGPL. I think we only wait for 1-2
bits of code to be rewritten. Pierre-Marc could tell more about it.
Until then, it's temporarily distributed under GPLv2.)

Eventually, parts of this ust library could be integrated to the libc,
so its usage should become more transparent that it currently is.

Eventually, the only dependency ust would have on the kernel is to
require a userspace trace clock mapped in a vDSO, synchronized with the
kernel trace clock.

Feedback is welcome,

Thanks,

Mathieu

> 
> Anyway, like you say, it has uses (potentially very powerful ones),
> Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite
> agreed on how to do it :-)
> 
> 
> And here I see LKML isn't on the CC list, perhaps we should?
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: malloc() tracing in perf?

[Steven Rostedt]

On Thu, 6 Aug 2009, Peter Zijlstra wrote:

> On Thu, 2009-08-06 at 07:55 -0400, Frank Ch. Eigler wrote:
> > Hi -
> > 
> > On Thu, Aug 06, 2009 at 01:28:14PM +0200, Peter Zijlstra wrote:
> > > [...]
> > > > That work is ongoing, and being discussed on utrace-devel@redhat.com,
> > > > since it is a prerequisite. 
> > > 
> > > Still hiding the discussion and the design never helped anybody.
> > 
> > I guess "hiding" is a matter of opinion.
> 
> Yeah in plain sight where nobody looks, really I thought the idea was to
> get utrace upstream, this means posting to linux-kernel. Going on LKML
> posts I'd say the utrace project was dead. Same goes for uprobe.
> 
> I've told this before and I'll say it again, post to LKML.

I second this. _Any_ kernel development that is new must be discussed on 
LKML. Sure, you can do the initial prototypes on a smaller mailing list, 
but once you have that prototype ready, it should go to LKML. This was the 
problem with Xen. The -rt patch was developed on LKML and even though 
there's a lot more to get in, a lot already has. The reason is that all 
development was done on the kernel mailing list.

-- Steve

Re: malloc() tracing in perf?

[Frank Ch. Eigler]

Hi -

On Thu, Aug 06, 2009 at 02:59:34PM +0200, Peter Zijlstra wrote:

> [...]  Yeah in plain sight where nobody looks, really I thought the
> idea was to get utrace upstream, this means posting to
> linux-kernel. Going on LKML posts [...]

I don't wish to get into arguing about how much lkml posting is
appropriate, or to editorialize about what happened when some of this
code did actually get posted.  I'm simply discussing the technology.
It's never been hard to find.


> [...]
> Does this also iterate the already existing tasks to find if it was
> already mmap()ed?

Yes.

> > This overview skims over issues related to return probes, tracing
> > buffer manipulations, and much other stuff.
> 
> Right, so you basically read the (dwarf2) debug info for a particular
> lib/symbol and generate a kernel module that knows about that and then
> insert that into the kernel to act as a uprobe handler?

Yes.

> Uprobe will then do the code rewrite on mmap() time to insert a trap
> much like kprobe does? What if its a JITed code section and the JIT
> rewrites it? Will uprobes detect that?

You mean if the uprobe is made to refer to code that was created by a
JIT?  That does not happen in the preceding scenario, since the
uprobes addresses were based on prior generated debugging data.

Putting uprobes into JIT code such as LLVM or JVM code is something
for the future.


> > All the code for this is hidden in plain sight in every systemtap
> > release, so please feel free to refer to that and/or ask more detailed
> > questions.
> 
> I thought the goal was to get stuff upstream, but if you want to
> continue living in la-la land and not bother with upstream Linux then so
> be it I guess :-(

It's not that simple, and you know it.


- FChE