[RFC] perf/perf_events: misleading number of samples due to mmap()

[RFC] perf/perf_events: misleading number of samples due to mmap()

[Stephane Eranian]

Hi,

I was using perf record to run various tests and I
realized perf output was rather misleading.

If you sample a noploop program wich runs for 10s:

$ perf record -F 1000  noploop 10

You expect a number of samples around: 10,000.

Now if you divide the rate by 4:

$ perf record -F 250  noploop 10
You expect around: 2500 samples

Well, it turns out the printed count depends on
the state of the whole system, not just noploop.

The reason is that perf reports an estimate based on the
number of bytes written to the buffer divided by the minimal
sample size of 24 bytes.

I think this is very confusing. It certainly got me.

I understand that perf does not parse the samples it gets from
the mmap'ed sampling buffer. Thus, it is not possible to get an
accurate average sample size nor actual number of samples.

What skews the estimate is the MMAP events (for the most part).
The sampling buffer records *all* mmap()s happening in the system
and this even if you are monitoring in per-thread mode. On a single-user
workstation that may be fine, but on a loaded server you get lots
of mmap events. And you don't care about most of them.

This leads me to another point. For per-thread sampling, why
do we need to record mmap() events happening *outside* of
the process? I can understand the exception of kernel modules.

Couldn't we restrict the event to those happening to the PID
the event is attached to?

Re: [RFC] perf/perf_events: misleading number of samples due to mmap()

[Peter Zijlstra]

On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
> This leads me to another point. For per-thread sampling, why
> do we need to record mmap() events happening *outside* of
> the process? I can understand the exception of kernel modules. 

How does that happen? The per-thread events should be on the per-task
context, so another task's mmap() events should never end up there.

Re: [RFC] perf/perf_events: misleading number of samples due to mmap()

[stephane eranian]

On Wed, Jun 16, 2010 at 4:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
>> This leads me to another point. For per-thread sampling, why
>> do we need to record mmap() events happening *outside* of
>> the process? I can understand the exception of kernel modules.
>
> How does that happen? The per-thread events should be on the per-task
> context, so another task's mmap() events should never end up there.
>

I don't see the test that says the vma does not belong to the current task.
I also don't see anything in perf_event_mmap_match().

It does seem to work as you said in recent kernels, though. So I am certainly
missing something here.

Re: [RFC] perf/perf_events: misleading number of samples due to mmap()

[stephane eranian]

On Wed, Jun 16, 2010 at 6:41 PM, stephane eranian
<eranian@googlemail.com> wrote:
> On Wed, Jun 16, 2010 at 4:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
>>> This leads me to another point. For per-thread sampling, why
>>> do we need to record mmap() events happening *outside* of
>>> the process? I can understand the exception of kernel modules.
>>
>> How does that happen? The per-thread events should be on the per-task
>> context, so another task's mmap() events should never end up there.
>>
>
> I don't see the test that says the vma does not belong to the current task.
> I also don't see anything in perf_event_mmap_match().
>
> It does seem to work as you said in recent kernels, though. So I am certainly
> missing something here.
>
I think I got it now. It's all based on the current->perf_event_ctxp.
So you only
look at the events (and thus sampling buffers) attached to current which issued
the mmap().

I found out that my problem is coming from perf creating synthetic
mmap() entries
for everything it finds in /proc/PID.

Re: [RFC] perf/perf_events: misleading number of samples due to mmap()

[Peter Zijlstra]

On Wed, 2010-06-16 at 18:41 +0200, stephane eranian wrote:
> On Wed, Jun 16, 2010 at 4:52 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> > On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
> >> This leads me to another point. For per-thread sampling, why
> >> do we need to record mmap() events happening *outside* of
> >> the process? I can understand the exception of kernel modules.
> >
> > How does that happen? The per-thread events should be on the per-task
> > context, so another task's mmap() events should never end up there.
> >
> 
> I don't see the test that says the vma does not belong to the current task.
> I also don't see anything in perf_event_mmap_match().
> 
> It does seem to work as you said in recent kernels, though. So I am certainly
> missing something here.

vma's are always part of the current task, its impossible to call mmap()
on another process's address space.

Look at the tail of perf_event_mmap_event(), it does:

        rcu_read_lock();
        cpuctx = &get_cpu_var(perf_cpu_context);
        perf_event_mmap_ctx(&cpuctx->ctx, mmap_event, vma->vm_flags & VM_EXEC);
        ctx = rcu_dereference(current->perf_event_ctxp);
        if (ctx)
                perf_event_mmap_ctx(ctx, mmap_event, vma->vm_flags & VM_EXEC);
        put_cpu_var(perf_cpu_context);
        rcu_read_unlock();

There it traverses the per-cpu context and the per-task context.

Re: [RFC] perf/perf_events: misleading number of samples due to mmap()

[Peter Zijlstra]

On Wed, 2010-06-16 at 16:40 +0200, Stephane Eranian wrote:
> The reason is that perf reports an estimate based on the
> number of bytes written to the buffer divided by the minimal
> sample size of 24 bytes. 

Right, we should change that based on the PERF_SAMPLE flags used. It
will remain an estimate due to the out of band events, but it should be
closer than assuming the minimal sample size.