Re: Whining about NUMA. :) [Was whining about 2.5...]

[Rob Landley <landley@trommello.org>]

On Thursday 04 October 2001 16:53, Alan Cox wrote:
> > Is there really a NUMA machine out there where you can DMA out of another
> > node's 16 bit ISA space?  So far the differences in the zones seem to be
>
> DMA engines are tied to the node the device is tied to not to the processor
> in question in most NUMA systems.

Oh good.  I'd sort of guessed that part, but wasn't quite sure.  (I've seen 
hardware people do some amazingly odd things before.  Luckily not recently, 
though...)

So would a workable (if naieve) attempt to use Andrea's 
memory-zones-grouped-into-classes approach on NUMA just involve making a 
class/zone list for each node?  (Okay, you've got to identify nodes, and 
group together processors, bridges, DMAable devices, etc, but it seems like 
that has to be done anyway, class/zone or not.)  How does what people want to 
do for NUMA improve on that?

Is a major goal of NUMA figuring out how to borrow resources from adjacent 
nodes (and less-adjacent nodes) in a "second choice, third choice, twelfth 
choice" kind of way?  Or is a "this resource set is local to this node, and 
allocating beyond this group is some variant of swapping behavior" approach 
an acceptable first approximation?

If class/zone is so evil for NUMA, what's the alternative that's being 
considered?  (Pointer to paper?)

I'm wondering how the class/zone approach is more evil than the alternative 
of having lots and lots of little zones which have different properties for 
each processor and DMAable device on the system, and then trying to figure 
out what to do from there at allocation time or during each attempt to 
inflict I/O upon buffers.

Rob

P.S.  Rik pointed out in email (replying to my "master of stupid questions" 
signoff) that I am indeed confused about a great many things, but didn't 
elaborate.  Of course I agree with this, but I do try to make it up on volume 
:).