Re: [profile] amortize atomic hit count increments

[William Lee Irwin III <wli@holomorphy.com>]

On Mon, Sep 13, 2004 at 10:05:07PM -0700, David S. Miller wrote:
> William, any reason not to fully per-cpu the profile buffer
> and then only traverse the array when the user attempts to
> capture the counters?
> Then we can undo the atomics altogether, as well as the cacheline
> traffic, for the extremely common case.
> Are there space concerns?

This was my original approach (modulo eliminating the global buffer
and the atomic operations), but space concerns stymied it, as the
profile buffer can be several megabytes large. It would likely perform
better in general if admissible, for whatever value performance is
considered to have.

There is also an unusual facet to this; the TLB overhead of a loop like:
	for (i = 0; i < prof_len; ++i) {
		for_each_online_cpu(cpu)
			global_buf[i] += per_cpu(cpu_prof_buffer, cpu)[i];
	}
is very large and caused "effective nontermination", otherwise known as
"exhausting the user's patience", on SGI's systems after about half an
hour. So some TLB overhead amortization is necessary for this to be
feasible. I suspect iterating over pages of the profile buffer and
storing intermediate results for a page full of profile buffer hits
in a buffer page may suffice though I've not tried it.


-- wli