Re: Perf support for interpreted and Just-In-Time translated olanguages

[Arnaldo Carvalho de Melo <acme@kernel.org>]

Em Wed, Dec 10, 2014 at 11:27:18AM -0700, David Ahern escreveu:
> On 12/10/14 11:05 AM, Andi Kleen wrote:
> >On Wed, Dec 10, 2014 at 10:39:53AM -0700, David Ahern wrote:
> >>On 12/10/14 10:32 AM, Andi Kleen wrote:
> >>>> From what I understood, how would it insert that event into the
> >>>>perf.data event stream? Only if it necessarily involved a new mmap, via
> >>>>the kernel, etc.

> >>>That's the new interface to be defined.

> >>>Just write a perf.data?

> >>Pawel Moll's new ioctl -- assumming it gets committed.

> >We can't push large volume data -- like line numbers and executable code -
> >through a ioctl.
 
> If you write a separate perf.data file then it has to be merged with the
> file generated by perf which brings in the perf_clock timestamp problem
> since timestamps are needed to merge the data sets.

yeap, but what he is saying is kinda like the problem with long running
perf sessions where a library may be updated while keeping the same
pathname, i.e. we samples taken up to the time the symtab backing
storage was replaced would be unreliable.

I think this should be solved in the same way, i.e. content based keys,
that we call build-ids, every mapping, when put in place generates an
event, say PERF_RECORD_MMAP3, that comes with a key that can later be
used to retrieve the matching ELF file with DWARF info for annotation,
symbol resolution, unwinding, etc.

Modern distros have this and that is why we store in the perf.data file
just the build ids, not the full ELF files at the time of the recording
session:

[root@ssdandy ~]# perf record -a sleep 1
^[[A^[[A[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.475 MB perf.data (~20748 samples) ]
[root@ssdandy ~]# perf buildid-list
2aae14c5b9aef2a3ebce0f168814ca7391c7ea6b /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64/vmlinux
e62a248621f017c39fbdeff22e3fb68ad3e0be77 /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/ata/libahci.ko
8acd5b3b420f0a37af9d661da5d6f88a86908c52 /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko
37bed793bb0736d341002165538659862dcc9e4f /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/scsi/sd_mod.ko
8f0c72d8c938fdeecef941e47a586614ce1014e7 /lib/modules/3.10.0-123.el7.x86_64/kernel/fs/xfs/xfs.ko
9bb557cc045eb57af2f83b315389dd8287fb7f60 /lib/modules/3.10.0-123.el7.x86_64/kernel/net/ipv4/netfilter/ip_tables.ko
cff370844d00ea5451d7add439646a93c64d48a5 /usr/lib64/libc-2.17.so
18562ee0363bc9bd7101610bd86469aa426d0c44 /usr/lib64/libpthread-2.17.so
ebd9fbf2265129ceab3866d40c826c9629f08cd0 [vdso]
a8cf24d1279557a3f2e563c21a858dcd8784b665 /usr/lib64/libcrypto.so.1.0.1e
148a981dc96876372c6f5c06bbb3efd0a1668432 /usr/sbin/sshd
ce3715b450fc4015a6763b210c868f930d2cffa6 /usr/bin/find
[root@ssdandy ~]# 

[root@ssdandy ~]# ls -la perf.data
-rw-------. 1 root root 501828 Dec 10 16:26 perf.data
[root@ssdandy ~]# ls -la /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64/vmlinux
-rwxr-xr-x. 2 root root 146606780 May  5  2014 /usr/lib/debug/lib/modules/3.10.0-123.el7.x86_64/vmlinux
[root@ssdandy ~]# ls -la /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/ata/libahci.ko
-rw-r--r--. 2 root root 51561 May  5  2014 /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/ata/libahci.ko
[root@ssdandy ~]# ls -la /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko
-rw-r--r--. 2 root root 400273 May  5  2014 /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko
[root@ssdandy ~]# /lib/modules/3.10.0-123.el7.x86_64/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko


And by the build-id we can get to the right file, that may even not be
installed on the target machine at the time of profiling.

[root@ssdandy ~]# perf buildid-list -i /usr/lib64/libc-2.17.so 
cff370844d00ea5451d7add439646a93c64d48a5
[root@ssdandy ~]# ls -la /usr/lib/debug/.build-id/cf/f370844d00ea5451d7add439646a93c64d48a5
lrwxrwxrwx. 1 root root 33 Dec 10 16:32 /usr/lib/debug/.build-id/cf/f370844d00ea5451d7add439646a93c64d48a5 -> ../../../../../lib64/libc-2.17.so
[root@ssdandy ~]# rpm -qf /usr/lib/debug/.build-id/cf/f370844d00ea5451d7add439646a93c64d48a5
glibc-debuginfo-2.17-55.el7_0.1.x86_64
[root@ssdandy ~]# rpm -q glibc
glibc-2.17-55.el7_0.1.x86_64
[root@ssdandy ~]#

The agent would need to, using PERF_RECORD_MMAP3 + build-ids plus intercepting the
CompiledMethodLoad synthesize the right ELF files + build-ids, which
could be just one per workload, or multiple, if those methods gets
recompiled frequently and move to different locations.

And we may want even to have a different one per version of the compiled
method, so that developers could figure out which compiled method
generation is best :-)

- Arnaldo