Re: malloc() tracing in perf?

[Peter Zijlstra <a.p.zijlstra@chello.nl>]

On Thu, 2009-08-06 at 12:19 +0300, Pekka Enberg wrote:
> Pekka Enberg wrote:
> > Hi Peter,
> > 
> > On Thu, 2009-08-06 at 10:48 +0300, Pekka Enberg wrote:
> >>> Hi,
> >>>
> >>> It's me again :-).
> >>>
> >>> I have a little user-space application that is pretty memory hungry 
> >>> and I want to understand why. I started to google around for a memory 
> >>> profiler or a malloc() tracer but didn't seem to find anything really 
> >>> useful.
> >>>
> >>> But then it hit me, why can't I have kmemtrace + perf but for 
> >>> user-space? Something like the "Malloc Trace" shown here:
> >>>
> >>> http://developer.apple.com/documentation/developertools/conceptual/SharkUserGuide/OtherProfilingandTracingTechniques/OtherProfilingandTracingTechniques.html#//apple_ref/doc/uid/TP40005233-CH6-SW17 
> >>>
> >>>
> >>> Does this sound like something that could/should be part of "perf"? 
> >>> How would all this work anyway? Can we intercept malloc() and free() 
> >>> somehow? Where would the data be pushed? Am I just going perf-crazy 
> >>> and trying to turn it into a swiss army knife because it's so easy to 
> >>> use?-)
> > 
> > Peter Zijlstra wrote:
> >> OK, you just trod into a wasp's nest :-)
> >>
> >> That sounds like uprobes, the equivalent of kprobes but for userspace.
> >>
> >> I seem to have heard people are working on such a thing, but I can't
> >> seem to find a single LKML post with 'uprobe' in the subject in the past
> >> two years or something (except for MTUprobe) -- so I guess its not
> >> really going anywhere any fast.
> > 
> > [snip]
> > 
> > Right. Are dynamic tracepoints what we want for malloc() and free() 
> > interception, though? I guess we can just do generic "userspace function 
> > called" events and record the passed parameters. Then "perf memreport" 
> > can just go find all the malloc() and free() calls and construct a 
> > memory profile out of it?
> > 
> > That said, I'd also be interested in wiring my userspace VM garbage 
> > collector to emit similar alloc and free events as well (because perf 
> > already has a JIT symbol map) which makes me think we want generic 
> > "userspace allocated memory" events.

Right, there is the distinction between static and dynamic tracepoints,
and I'm not sure how that'll work out. Most people who've looked at the
subject matter are on CC, so I'm hoping someone will expand.

That said, we could simply LD_PRELOAD malloc and free (new and delete)
like any other memory debugger out there already does, that's not rocket
science (in fact, I've done so on many an occasion).

I remember mpatrol being a hackable code base, but I just looked it up
and its LGPL-v3 so I'm not sure we should be borrowing code from there.
Maybe there is another such creature out there we can borrow code from.

I've got similar ideas for your JIT stuff, you now open-code the tmp
file writing, ideally I'd like to push that into a library and have you
link against something with weak stubs and let perf LD_PRELOAD real code
which will do whatever is needed. That'd rid you of having to know about
where and what to write.

Anyway, malloc/free are easy, but like you say, you might be interested
in other functions which aren't as easy.