Re: [PATCH 1/5] cpuset memory spread basic implementation

[Andi Kleen <ak@suse.de>]

On Monday 06 February 2006 09:22, Ingo Molnar wrote:
> 
> * Paul Jackson <pj@sgi.com> wrote:
> 
> > Ingo wrote:
> > > Could you perhaps outline two actual use-cases 
> > > that would need two cpusets with different policies,
> > > on the same box?
> > 
> > We normally run with different policies, in the same box, on different 
> > cpusets at the same time.  But this might be because some cpusets 
> > -need- the memory spreading, and the others that don't are left to the 
> > default policy.
> 
> so in practice, the memory spreading is in fact a global setting, used
> by all cpusets that matter? That seems to support Andrew's observation
> that our assumptions / defaults are bad, pretty much independently of
> the workload.

Yes in general page cache and other global caches should be spread 
around all nodes by default. There was a patch to do this for the page 
cache in SLES9 for a long time, but d/icache and possibly other slab
caches have of course the same problem. For doing IO caching it doesn't matter much if 
the cache is local or not because memory access in read/write is not that
critical. But it's a big problem when one node fills up with 
IO (or dentry or inode) caches because an IO intensive process ran
on it on some time and later a process running there cannot get local
memory.

Of course there might be some corner cases where using local
memory for caching is still better (like mmap file IO), but my 
guess is that it isn't a good default. 

-Andi