Re: [RFC 4/5] x86, perf: implements lwp-perf-integration (rc1)

Ingo Molnar
2011-12-20T09:17:14Z

* Avi Kivity <avi@redhat.com> wrote:

> On 12/19/2011 01:40 PM, Ingo Molnar wrote:
> >
> > 2) the proper solution: creating a 'user-space vmalloc()' that
> > is per mm and that gets inherited transparently, across
> > fork() and exec(), and which lies outside the regular vma
> > spaces. On 64-bit this should be straightforward.
>
> That probably has uses outside perf too, but I can see mm nacks piling up.

This can be done in arch/x86/ code if it's too x86 specific -
the platform controls the VM layout and can (and does) use
special per CPU VM areas.

> > These vmas are not actually 'known' to user-space
> > normally - the kernel PMU code knows about it and does
> > what we do with PEBS: flushes it when necessary and puts
> > it into the regular perf event channels.
> >
> > This solves the inherited perf record workflow
> > immediately: the parent task just creates the buffer,
> > which gets inherited across exec() and fork(), into every
> > portion of the workload.
>
> The buffer still needs to be managed. [...]

Of course, like we manage the DS buffer for PEBS.

> [...] While you may be able to juggle different threads on
> the same cpu using different events, threads on other cpus
> need to use separate LWP contexts and buffers.

Yes, like different threads on different CPUs have different DS
buffers, *here and today*.

Try this on (most) modern Intel CPUs:

perf top -e cycles:pp

That will activate that exact mechanism.

The LWPCB and the LWP ring-buffer are really just an extension
of that concept: per task buffers which are ring 3 visible.

Note that user-space does not actually have to know about any of
these LWP addresses (but can access them if it wants to - no
strong feelings about that) - in the correctly implemented model
it's fully kernel managed.

In fact the PEBS case had one more complication: there's the BTS
branch-tracing feature which we support as well, and which
overlaps PEBS use of the DS.

All these PMU hardware limitations can be supported, as long as
the instrumentation *capability* adds value to the system in one
way or another.

> > System-wide profiling is a small additional variant of
> > this: creating such a user-vmalloc() area for all tasks
> > in the system so that the PMU code has them ready in the
> > context-switch code.
>
> What about security? Do we want to allow any userspace
> process to mess up the buffers? It can even reprogram the LWP
> block, so you're counting different things, or at higher
> frequencies, or into other processes ordinary vmas?

In most usecases it's the application messing up its own
profiling - don't do that if it hurts.

I'd argue that future LWP versions should allow kernel-protected
LWP pages, as long as the LWPCB is privileged as well as well.
That would be useful for another purpose as well: LWP could be
allowed to sample kernel-space execution as well, an obviously
useful feature that was left out from LWP for barely explicable
reasons.

Granted, LWP was mis-designed to quite a degree, those AMD chip
engineers should have talked to people who understand how modern
PMU abstractions are added to the OS kernel properly. But this
mis-design does not keep us from utilizing this piece of
hardware intelligently. PEBS/DS/BTS wasnt a beauty either.

> You could rebuild the LWP block on every context switch I
> guess, but you need to prevent access to other cpus' LWP
> blocks (since they may be running other processes). I think
> this calls for per-cpu cr3, even for threads in the same
> process.

Why would we want to rebuild the LWPCB? Just keep one per task
and do a lightweight switch to it during switch_to() - like we
do it with the PEBS hardware-ring-buffer. It can be in the same
single block of memory with the ring-buffer itself. (PEBS has
similar characteristics)

Thanks,

Ingo