Re: [RFC 4/5] x86, perf: implements lwp-perf-integration (rc1)

Ingo Molnar
2011-12-19T11:42:38Z

* Avi Kivity <avi@redhat.com> wrote:

> On 12/19/2011 12:54 PM, Ingo Molnar wrote:
> > * Robert Richter <robert.richter@amd.com> wrote:
> >
> > > On 19.12.11 00:43:10, Ingo Molnar wrote:
> > >
> > > > So the question becomes, how well is it integrated: can perf
> > > > 'record -a + perf report', or 'perf top' use LWP, to do
> > > > system-wide precise [user-space] profiling and such?
> > >
> > > There is only self-monitoring of a process possible, no
> > > kernel and system-wide profiling. This is because we can
> > > not allocate memory regions in the kernel for a thread
> > > other than the current. This would require a complete
> > > rework of mm code.
> >
> > Hm, i don't think a rework is needed: check the
> > vmalloc_to_page() code in kernel/events/ring_buffer.c. Right
> > now CONFIG_PERF_USE_VMALLOC is an ARM, MIPS, SH and Sparc
> > specific feature, on x86 it turns on if
> > CONFIG_DEBUG_PERF_USE_VMALLOC=y.
> >
> > That should be good enough for prototyping the kernel/user
> > shared buffering approach.
>
> LWP wants user memory, vmalloc is insufficient. You need
> do_mmap() with a different mm.

Take a look at PERF_USE_VMALLOC, it allows in-kernel allocated
memory to be mmap()ed to user-space. It is basically a
shared/dual user/kernel mode vmalloc implementation.

So all the conceptual pieces are there.

> You could let a workqueue call use_mm() and then do_mmap().
> Even then it is subject to disruption by the monitored thread
> (and may disrupt the monitored thread by playing with its
> address space). [...]

Injecting this into another thread's context is indeed advanced
stuff:

> [...] This is for thread monitoring only, I don't think
> system-wide monitoring is possible with LWP.

That should be possible too, via two methods:

1) the easy hack: a (per cpu) vmalloc()ed buffer is made ring 3
accessible (by clearing the system bit in the ptes) - and
thus accessible to all user-space.

This is obviously globally writable/readable memory so only a
debugging/prototyping hack - but would be a great first step
to prove the concept and see some nice perf top and perf
record results ...

2) the proper solution: creating a 'user-space vmalloc()' that
is per mm and that gets inherited transparently, across
fork() and exec(), and which lies outside the regular vma
spaces. On 64-bit this should be straightforward.

These vmas are not actually 'known' to user-space normally -
the kernel PMU code knows about it and does what we do with
PEBS: flushes it when necessary and puts it into the
regular perf event channels.

This solves the inherited perf record workflow immediately:
the parent task just creates the buffer, which gets inherited
across exec() and fork(), into every portion of the workload.

System-wide profiling is a small additional variant of this:
creating such a user-vmalloc() area for all tasks in the
system so that the PMU code has them ready in the
context-switch code.

Solution #2 has the additional advantage that we could migrate
PEBS to it and could allow interested user-space access to the
'raw' PEBS buffer as well. (currently the PEBS buffer is only
visible to kernel-space.)

I'd suggest the easy hack first, to get things going - we can
then help out with the proper solution.

Thanks,

Ingo