Re: [patch 0/3] [Announcement] Performance Counters for Linux

Ingo Molnar
2008-12-05T12:08:17Z

* Paul Mackerras <paulus@samba.org> wrote:

> Ingo Molnar writes:
>
> > * Paul Mackerras <paulus@samba.org> wrote:
> [...]
> > > Isn't it two separate read() calls to read the two counters? If so,
> > > the only way the two values are actually going to correspond to the
> > > same point in time is if the task being monitored is stopped. In which
> > > case the monitoring task needs to use ptrace or something similar in
> > > order to make sure that the monitored task is actually stopped.
> >
> > It doesnt matter in practice.
>
> Can I ask - and this is a real question, I'm not being sarcastic - is
> that statement made with substantial serious experience in performance
> analysis behind it, or is it just an intuition?
>
> I will happily admit that I am not a great expert on performance
> analysis with years of experience. But I have taken a bit of a look at
> what people with that sort of experience do, and I don't think they
> would agree with your "doesn't matter" statement.

A stream of read()s possibly slightly being off is an order of magnitude
smaller of an effect to precision. Look at the numbers: on the testbox i
have a read() syscall takes 0.2 microseconds, while a context-switch
takes 2 microseconds on the local CPU and about 5-10 microseconds
cross-CPU (or more, if the cache pattern is unlucky/unaffine). That's
10-25-50 times more expensive. You can do 9-24-49 reads and still be
cheaper. Compound syscalls are almost never worth the complexity.

So as a scheduler person i cannot really take the perfmon "ptrace
approach" seriously, and i explained that in great detail already. It
clearly came from HPC workload quarters where tasks are persistent
entities running alone on a single CPU that just use up CPU time there
and dont interact with each other too much. That's a good and important
profiling target for sure - but by no means the only workload target to
design a core kernel facility for. It's an absolutely horrible approach
for a number of more common workloads for sure.

> > Such kind of 'group system call facility' has been suggested several
> > times in the past - but ... never got anywhere because system calls
> > are cheap enough, it really does not count in practice.
> >
> > It could be implemented, and note that because our code uses a proper
> > Linux file descriptor abstraction, such a sys_read_fds() facility
> > would help _other_ applications as well, not just performance
> > counters.
> >
> > But it brings complications: demultiplexing of error conditions on
> > individual counters is a real pain with any compound abstraction. We
> > very consciously went with the 'one fd, one object, one counter'
> > design.
>
> And I think that is the fundamental flaw. On the machines I am
> familiar with, the performance counters as not separate things that can
> individually and independently be assigned to count one thing or
> another.

Today we've implemented virtual counter scheduling in our to-be-v2 code:

3 files changed, 36 insertions(+), 1 deletion(-)

hello.c gives:

counter[0 cycles ]: 10121258163 , delta: 844256826 events
counter[1 instructions ]: 4160893621 , delta: 347054666 events
counter[2 cache-refs ]: 2297 , delta: 179 events
counter[3 cache-misses ]: 3 , delta: 0 events
counter[4 branch-instructions ]: 799422166 , delta: 66551572 events
counter[5 branch-misses ]: 7286 , delta: 775 events

All we need to get that array of information from 6 sw counters is a
_single_ hardware counter. I'm not sure where you read "you must map sw
counters to hw counters directly" or "hw counters must be independent of
each other" into our design - it's not part of it, emphatically.

And i dont see your (fully correct!) statement above about counter
constraints to be in any sort of conflict with what we are doing.

Intel hardware is just as constrained as powerpc hardware: there are
counter inter-dependencies and many CPUs have just two performance
counters. We very much took this into account while designing this code.

[ Obviously, you _can_ do higher quality profiling if you have more
hardware resources that help it. Nothing will change that fact. ]

> Rather, what the hardware provides is ONE performance monitor unit,
> which the OS can context-switch between tasks. The performance monitor
> unit has several counters that can be assigned (within limits) to count
> various aspects of the performance of the code being executed. That is
> why, for instance, if you ask for the counters to be frozen when one of
> them overflows, they all get frozen at that point.

i dont see this as an issue at all - it's a feature of powerpc over x86
that the core perfcounter code can support just fine. The overflow IRQ
handler is arch specific. The overflow IRQ handler, if it triggers,
updates the sw counters, creates any event records if needed, wakes up
the monitor task if needed, and continues the task and performance
measurement without having scheduled out. Demultiplexing of hw counters
is arch-specific.

Ingo