Re: Perf support for interpreted and Just-In-Time translated olanguages

[Carl Love <cel@us.ibm.com>]

On Tue, 2014-12-09 at 16:38 -0800, Andi Kleen wrote:
> Arnaldo Carvalho de Melo <acme@kernel.org> writes:
> 
> > When we use mmap(addr, len, PROT_EXEC) the kernel has a meta event
> > called PERF_RECORD_MMAP that will record addr range, symtab DSO path,
> > and we ask it to be timestamped, how to do that for the equivalent part
> > in the JVM?
> 
> http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html
> 
> You just need a way to push it from the agent to perf.
> 

This may be more complex then needed.  We really just need to track the
symbol.  From talking with the Java experts, they say the reused memory
is almost always from methods that are loaded/used early when
initializing the program.  Then the memory gets reused by a method that
runs for the bulk of the time.  So by getting a timestamp for when the
symbol was loaded you can then determine the length of time each symbol
was in the anonymous memory.  Then just associate the sample with the
symbol that was present in the anonymous memory the longest.  You can
also tag the symbol names to say that they reside in the same memory,
ie, the memory was reused. To warn the user that there maybe some
ambiguity in the sample to symbol association.  


> >
> > My initial thought was to find that using perf probe and insert there a
> > probe point, but I think that there may be already an existing
> > tracepoint in the jvm for that, one that, from what I've read so far,
> > is _not_ being used by this java perf agent, right?
> 
> The agent interface works by linking an agent with a special library
> I believe. So the agent can do whatever it wants.

> > From what I understood, how would it insert that event into the
> > perf.data event stream? Only if it necessarily involved a new mmap, via
> > the kernel, etc.
> 
> You basically need a way to trigger a perf event from user space.
> One simple way would be to just write a perf.data too into /tmp,
> and let perf collect that.
> 
> But it's more than that. You also need to transfer the executable
> code somewhere, and the line numbers.
> 

The source file and line number information is optionally available from
the jvm via the jvmti complied method load callback.  With this
information on the symbol, line number and source file, perf could then
persist the data in an ELF formatted file. 


> >  
> >> It just needs a better interface from the agent to perf, to pass all
> >> needed information, including symbols, line numbers, executable code
> >> (for PT decoding and for showing diassembler), and ordering it by time
> >> so that no hacks are needed.
> >> 
> >> BTW other JITs (LLVM, Mono, V8, ...) have similar interfaces.
> >> 
> >> Longer term as the kernel gets more JITed (eBPF etc.) it likely needs 
> >> some kind of JIT interface too.
> >
> > Right, this is something we need to have, no questions about it, its
> > just a matter of cooking up some prototype implementation...
> >
> > If, for instance, the java agent would put on some file those events,
> > timestamped, then when in perf report we would just insert them into the
> > event stream as synthesized PERF_RECORD_MMAPs, probably that would be
> > enough.
> 
> Really to be useful you need at least line numbers, better code.
> 
> In theory the agent could write ELF files with dwarf, but that may get
> really ugly.  Probably better to have something more light weight.

Perf would need to do this so the data collection and ELF file
conversion will be done only when pref is doing the profiling of the
JITed code. Specifically, if perf is passed the pid of a running JVM the
agent library would not know when profiling starts and stops.

         Carl Love