Re: x86 patches was Re: -mm merge plans for 2.6.24

[Lee Schermerhorn <Lee.Schermerhorn@hp.com>]

On Tue, 2007-10-02 at 00:43 -0700, Andrew Morton wrote:
> On Tue, 2 Oct 2007 16:36:24 +0900 KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com> wrote:
> 
> > On Tue, 2 Oct 2007 00:18:09 -0700
> > Andrew Morton <akpm@linux-foundation.org> wrote:
> > > > 
> > > > > How come?  Memoryless node can and do occur in real-world machines.  Kernel
> > > > > should support that?
> > > > 
> > > > But a node is just defined by its memory? 
> > > 
> > > Don't think so.  A node is a lump of circuitry which can have zero or more
> > > CPUs, IO and memory.
> > > 
> > > It may initially have been conceived as a memory-only concept in the Linux
> > > kernel, but that doesn't fully map onto reality (does it?)
> > > 
> > > There was a real-world need for this, I think from the Fujitsu guys.  That
> > > should be spelled out in the changelog but isn't.
> > 
> > Yes, Fujitsu and HP guys really need this memory-less-node support. 
> > 
> 
> For what reason, please?

For the HP platforms, we can configure each cell with from 0% to 100%
"cell local memory".  When we configure with <100% CLM, the "missing
percentages" are interleaved by hardware on a cache-line granularity to
improve bandwidth at the expense of latency for numa-challenged
applications [and OSes, but not our problem ;-)].  When we boot Linux on
such a config, all of the real nodes have no memory--it all resides in a
single interleaved pseudo-node.  

When we boot Linux on a 100% CLM configuration [== NUMA], we still have
the interleaved pseudo-node.  It contains a few hundred MB stolen from
the real nodes to contain the DMA zone.  [Interleaved memory resides at
phys addr 0].  The memoryless-nodes patches, along with the zoneorder
patches, support this config as well.

Also, when we boot a NUMA config with the "mem=" command line,
specifying less memory than actually exists, Linux takes the excluded
memory "off the top" rather than distributing it across the nodes.  This
can result in memoryless nodes, as well.

Lee