Re: Perf support for interpreted and Just-In-Time translated olanguages

[Arnaldo Carvalho de Melo <acme@kernel.org>]

Em Tue, Dec 09, 2014 at 04:38:30PM -0800, Andi Kleen escreveu:
> Arnaldo Carvalho de Melo <acme@kernel.org> writes:
> 
> > When we use mmap(addr, len, PROT_EXEC) the kernel has a meta event
> > called PERF_RECORD_MMAP that will record addr range, symtab DSO path,
> > and we ask it to be timestamped, how to do that for the equivalent part
> > in the JVM?
> 
> http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html
> 
> You just need a way to push it from the agent to perf.

Yeah, things like the CompiledMethodLoad with a machinepc location
format, etc:

http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html#CompiledMethodLoad

http://docs.oracle.com/javase/7/docs/platform/jvmti/jvmti.html#GetJLocationFormat

JVMTI_JLOCATION_MACHINEPC       2       jlocation values represent native
                                        machine program counter values.

I haven't had a look at any agent implementation, but it would have to
use this kind of info to figure out if it would have to generate a new
PERF_RECORD_MMAP with a new syhthesized DSO map, i.e. if a method load
would put some method in a different place, not reusing its previous
addr range, etc, interesting.
 
> > My initial thought was to find that using perf probe and insert there a
> > probe point, but I think that there may be already an existing
> > tracepoint in the jvm for that, one that, from what I've read so far,
> > is _not_ being used by this java perf agent, right?
 
> The agent interface works by linking an agent with a special library
> I believe. So the agent can do whatever it wants.

From the quick look we receive some kind of event (CompiledMethodLoad)
with addr ranges for JITted methods, etc.
 
> > From what I understood, how would it insert that event into the
> > perf.data event stream? Only if it necessarily involved a new mmap, via
> > the kernel, etc.
> 
> You basically need a way to trigger a perf event from user space.

Yes if you want to reuse the existing tooling somehow, but we could have
a perf data merger that would pick multiple perf.data files and, having
a common clocksource, get a perf.data file generated by the agent with
the relevant PERF_RECORD_MMAP records.

From what I saw we could event generate PERF_RECORD_FORK/EXIT for native
Java threads and use that mostly like native OS threads :-)

> One simple way would be to just write a perf.data too into /tmp,
> and let perf collect that.
> 
> But it's more than that. You also need to transfer the executable
> code somewhere, and the line numbers.

Right, if the most easy for the existing perf tooling would be to
generate multiple ELF files with DWARF info that would be accessed by
build-id inserted into a PERF_RECORD_MMAP3 record.
 
> >> It just needs a better interface from the agent to perf, to pass all
> >> needed information, including symbols, line numbers, executable code
> >> (for PT decoding and for showing diassembler), and ordering it by time
> >> so that no hacks are needed.
> >> 
> >> BTW other JITs (LLVM, Mono, V8, ...) have similar interfaces.
> >> 
> >> Longer term as the kernel gets more JITed (eBPF etc.) it likely needs 
> >> some kind of JIT interface too.
> >
> > Right, this is something we need to have, no questions about it, its
> > just a matter of cooking up some prototype implementation...
> >
> > If, for instance, the java agent would put on some file those events,
> > timestamped, then when in perf report we would just insert them into the
> > event stream as synthesized PERF_RECORD_MMAPs, probably that would be
> > enough.
> 
> Really to be useful you need at least line numbers, better code.
> 
> In theory the agent could write ELF files with dwarf, but that may get
> really ugly.  Probably better to have something more light weight.

For reducing the changes needed to tools/perf, that would be optimal,
for the agent implementers... I guess not ;-)

- Arnaldo