Re: numa=on broken

[Ryan Harper <ryanh@us.ibm.com>]

* Keir Fraser <Keir.Fraser@cl.cam.ac.uk> [2007-03-31 04:09]:
> On 30/3/07 20:39, "Ryan Harper" <ryanh@us.ibm.com> wrote:
> 
> >> Turning on by default is pointless because guests are not restricted to
> >> running on specific nodes by default. Since manual intervention is required
> >> to achieve that (right now at least) requiring numa=on is not much of a
> >> hardship.
> > 
> > I'm getting ready to re-submit patches to export the topology information
> > so the userspace tools can use that info to make intelligent selections.
> > This was available back in October, but was never picked up, or even
> > commented upon.
> 
> But can tools make sane automatic decisions on domain creation? And if tools

I don't think the tools would do any worse than what an admin would do:
keep the domains within a node.

> decide not to use NUMA-ness of the system, should the Xen allocator still
> hoover up all the memory of the node that vcpu0 happens to start on?

I'm not sure I understand what you mean by decide to not use the
NUMA-ness.

> 
> My primary concern is simply whether enabling NUMA by default can hurt
> performance, or cause problems by hitting certain memory nodes or memory
> zones harder than others, for the great majority of users who will not use
> NUMA (even if they have a small NUMA system like AMD K8).

Folks without NUMA hardware see the same path through the allocator
today whether they pass numa=on or not.  Last summer, I did [1]overhead
testing specifically on small NUMA systems to address this concern.  I
assumed that those numbers were satisfactory or the patches would not
have been picked up in the first place.

On systems with NUMA, when the domains are kept within a NUMA node, we
see significant performance [2]boost.  I don't have any data to to say
how well a NUMA system would perform with a mixed load of on and off
node access, but when presented with the option of running a domain
entirely within a node on a NUMA system, we should.

What amount of testing is enough for you to consider enabling numa=on by
default post 3.0.5?  I think we ought to cook numa=on by default through
another development cycle so we have time to address any performance
issues that may arise.

1. http://lists.xensource.com/archives/html/xen-devel/2006-07/msg01057.html
2. http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00958.html

-- 
Ryan Harper
Software Engineer; Linux Technology Center
IBM Corp., Austin, Tx
(512) 838-9253   T/L: 678-9253
ryanh@us.ibm.com