round-robining per-cpu counters

round-robining per-cpu counters

[Paul Mackerras]

It used to be, and as far as I can see still is, the case that per-cpu
counters take priority over per-task counters by virtue of being
scheduled in first.  That is, if you have N hardware counters and >= N
per-cpu counters, then no per-task counters will ever get scheduled
onto the PMU.

That being the case, I don't see what the point of having the
perf_reserved_percpu variable is.  It doesn't do anything except set
cpuctx->max_pertask, which isn't actually used anywhere.  In any case
with the current counter scheduling system there's no need to
"reserve" hardware counters for use by per-cpu counters since any new
per-cpu counters will just bump existing per-task counters off - if
not immediately then the next time that perf_counter_task_tick gets
called.

What was the intended meaning of perf_reserved_percpu?  I presume it
was that there would always be that many hardware counters available
for per-cpu counters regardless of how many per-task counters there
are.  But that doesn't answer the complementary question - how many
hardware counters can we rely on being available for per-task
counters?  At the moment the answer is 0, but I don't think that is a
good answer.

Does anyone have any good ideas about what the scheduling policy
should be?

Paul.

Re: round-robining per-cpu counters

[Ingo Molnar]

* Paul Mackerras <paulus@samba.org> wrote:

> It used to be, and as far as I can see still is, the case that 
> per-cpu counters take priority over per-task counters by virtue of 
> being scheduled in first.  That is, if you have N hardware 
> counters and >= N per-cpu counters, then no per-task counters will 
> ever get scheduled onto the PMU.
> 
> That being the case, I don't see what the point of having the 
> perf_reserved_percpu variable is.  It doesn't do anything except 
> set cpuctx->max_pertask, which isn't actually used anywhere.  In 
> any case with the current counter scheduling system there's no 
> need to "reserve" hardware counters for use by per-cpu counters 
> since any new per-cpu counters will just bump existing per-task 
> counters off - if not immediately then the next time that 
> perf_counter_task_tick gets called.
> 
> What was the intended meaning of perf_reserved_percpu?  I presume 
> it was that there would always be that many hardware counters 
> available for per-cpu counters regardless of how many per-task 
> counters there are.  But that doesn't answer the complementary 
> question - how many hardware counters can we rely on being 
> available for per-task counters?  At the moment the answer is 0, 
> but I don't think that is a good answer.
> 
> Does anyone have any good ideas about what the scheduling policy 
> should be?

The reservation mechanism really suffered from not being used by 
anything or anyone, and it thus bit-rotted across 300 follow-on 
commits.

What would be the primary usecase? Allow admin to set aside (and 
guarantee) space for task counters? Allow admin to 'force' 
exclusivity of counter ownership?

I think a better general solution would be to have a single 
round-robin list for all currently active counters (both percpu and 
task counters) - and fairly round-robin all of them. The scaling 
information makes it obvious when this is happening.

If admin wants stronger ownership of counters then the 
pinned/exclusive attribute can be used.

We really want to keep the counter-scheduler simple, and we also 
want to make the default to be as permissive as possible.

	Ingo

Re: round-robining per-cpu counters

[Paul Mackerras]

Ingo Molnar writes:

> The reservation mechanism really suffered from not being used by 
> anything or anyone, and it thus bit-rotted across 300 follow-on 
> commits.

Yeah.

> What would be the primary usecase? Allow admin to set aside (and 
> guarantee) space for task counters? Allow admin to 'force' 
> exclusivity of counter ownership?
> 
> I think a better general solution would be to have a single 
> round-robin list for all currently active counters (both percpu and 
> task counters) - and fairly round-robin all of them. The scaling 
> information makes it obvious when this is happening.

Sounds good, but I'm having trouble visualizing how we combine a
fairly static set of per-cpu counters with several sets of per-task
counters that come and go as their tasks get scheduled in and out.
Is there some clever scheduling technique that sorts all that out?

> If admin wants stronger ownership of counters then the 
> pinned/exclusive attribute can be used.

I agree.

> We really want to keep the counter-scheduler simple, and we also 
> want to make the default to be as permissive as possible.

Definitely.

Paul.

Re: round-robining per-cpu counters

[Ingo Molnar]

* Paul Mackerras <paulus@samba.org> wrote:

> > I think a better general solution would be to have a single 
> > round-robin list for all currently active counters (both percpu 
> > and task counters) - and fairly round-robin all of them. The 
> > scaling information makes it obvious when this is happening.
> 
> Sounds good, but I'm having trouble visualizing how we combine a 
> fairly static set of per-cpu counters with several sets of 
> per-task counters that come and go as their tasks get scheduled in 
> and out. Is there some clever scheduling technique that sorts all 
> that out?

Fair scheduling techniques would import the complexity of 
kernel/sched*.c into kernel/perf_counter.c - not sure we want that 
(at this stage at least).

So i'd go for the second best option: the simplest one.

One simple option would be to 'interleave' the two lists on sched-in 
(take a counter from one list, then from the other, then from the 
first, etc.), combined with the alternate rotation of both lists.

Another, perhaps even simpler option would be to do the alternation 
on a per list basis: rotate which of the two lists go first on 
sched-in, from the task tick.

Can you see any showstopper artifacts in this scheme?

Later on, if anyone finds this method lacking it can be improved 
gradually either by making the accounting/statistics smarter, or by 
increasing the frequency of rotation events and coupling it not to 
the scheduler tick but to a counter perhaps.

	Ingo