Re: [PATCH] Cpuset: rcu optimization of page alloc hook

[Eric Dumazet <dada1@cosmosbay.com>]

Paul Jackson a écrit :

> 
> 
> Hmmm ... I suspect one possible downside.
> 
> I would think we would want to spread the hot spots out, to reduce the
> chance of getting two hot spots in the same cache line, and starting a
> bidding war for that line.
> 
> So my intuition is:
>   If read alot but seldom written, mark "__read_mostly".
>   If seldom read or written, leave unmarked.
> 

Your analysis is very good but not complete :)

There are different kind of hot cache lines, depending if they are :
- Mostly read
- read/written

Say you move to read mostly most of struct kmem_cache *, they are guaranteed 
to stay in 'mostly read'.

Mixing for example filp_cachep and dcache_lock in the same cache line is not a 
good thing. And this is what happening on typical kernel :

c04f15f0 B dcache_lock
c04f15f4 B names_cachep
c04f15f8 B filp_cachep
c04f15fc b rename_lock

I do think we should have defined a special section for very hot (and written) 
spots. It's more easy to locate thos hot spots than 'mostly read and shared by 
all cpus without cache ping pongs' places...




> so as to leave plenty of the rarely used (neither read nor written on
> kernel hot path code) as "cannon fodder" to fill the rest of the cache
> lines favored by the hot data.
> 
> This leads me to ask, of any item marked "__read_mostly":
> 
>   Is it accessed (for read, presumably) frequently, on a hot path?
> 
> If not, then I'd favor (absent actual measurements to the contrary) not
> marking it.
> 
> By this criteria:
> 
>   1) I would -not- mark "struct kmem_cache *cpuset" __read_mostly, as it
>      is rarely accessed on -any- code path, much less a hot one.  It is
>      ideal cannon fodder.
> 
>   2) I -would- (following a private email suggestion of Christoph Lameter)
>      mark my recently added "int number_of_cpusets" __read_mostly,
>      because it is accessed for every zone considered in the loops
>      within^Wbeneath __alloc_pages().
> 
> Disclaimer -- none of the above speculation is tempered by the heat of any
> actual performance measurements.  Hence, it is worth about as much as my
> legal advice.
>