Re: [PATCH] Cpuset: rcu optimization of page alloc hook

Eric Dumazet
2005-12-13T16:32:20Z

Paul Jackson a écrit :

>
>
> Hmmm ... I suspect one possible downside.
>
> I would think we would want to spread the hot spots out, to reduce the
> chance of getting two hot spots in the same cache line, and starting a
> bidding war for that line.
>
> So my intuition is:
> If read alot but seldom written, mark "__read_mostly".
> If seldom read or written, leave unmarked.
>

Your analysis is very good but not complete :)

There are different kind of hot cache lines, depending if they are :
- Mostly read
- read/written

Say you move to read mostly most of struct kmem_cache *, they are guaranteed
to stay in 'mostly read'.

Mixing for example filp_cachep and dcache_lock in the same cache line is not a
good thing. And this is what happening on typical kernel :

c04f15f0 B dcache_lock
c04f15f4 B names_cachep
c04f15f8 B filp_cachep
c04f15fc b rename_lock

I do think we should have defined a special section for very hot (and written)
spots. It's more easy to locate thos hot spots than 'mostly read and shared by
all cpus without cache ping pongs' places...

> so as to leave plenty of the rarely used (neither read nor written on
> kernel hot path code) as "cannon fodder" to fill the rest of the cache
> lines favored by the hot data.
>
> This leads me to ask, of any item marked "__read_mostly":
>
> Is it accessed (for read, presumably) frequently, on a hot path?
>
> If not, then I'd favor (absent actual measurements to the contrary) not
> marking it.
>
> By this criteria:
>
> 1) I would -not- mark "struct kmem_cache *cpuset" __read_mostly, as it
> is rarely accessed on -any- code path, much less a hot one. It is
> ideal cannon fodder.
>
> 2) I -would- (following a private email suggestion of Christoph Lameter)
> mark my recently added "int number_of_cpusets" __read_mostly,
> because it is accessed for every zone considered in the loops
> within^Wbeneath __alloc_pages().
>
> Disclaimer -- none of the above speculation is tempered by the heat of any
> actual performance measurements. Hence, it is worth about as much as my
> legal advice.
>