From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ryan Harper Subject: Re: numa=on broken Date: Sun, 1 Apr 2007 00:20:18 -0500 Message-ID: <20070401052018.GA28736@us.ibm.com> References: <20070330193958.GZ28736@us.ibm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Keir Fraser Cc: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org * Keir Fraser [2007-03-31 04:09]: > On 30/3/07 20:39, "Ryan Harper" wrote: > > >> Turning on by default is pointless because guests are not restricted to > >> running on specific nodes by default. Since manual intervention is required > >> to achieve that (right now at least) requiring numa=on is not much of a > >> hardship. > > > > I'm getting ready to re-submit patches to export the topology information > > so the userspace tools can use that info to make intelligent selections. > > This was available back in October, but was never picked up, or even > > commented upon. > > But can tools make sane automatic decisions on domain creation? And if tools I don't think the tools would do any worse than what an admin would do: keep the domains within a node. > decide not to use NUMA-ness of the system, should the Xen allocator still > hoover up all the memory of the node that vcpu0 happens to start on? I'm not sure I understand what you mean by decide to not use the NUMA-ness. > > My primary concern is simply whether enabling NUMA by default can hurt > performance, or cause problems by hitting certain memory nodes or memory > zones harder than others, for the great majority of users who will not use > NUMA (even if they have a small NUMA system like AMD K8). Folks without NUMA hardware see the same path through the allocator today whether they pass numa=on or not. Last summer, I did [1]overhead testing specifically on small NUMA systems to address this concern. I assumed that those numbers were satisfactory or the patches would not have been picked up in the first place. On systems with NUMA, when the domains are kept within a NUMA node, we see significant performance [2]boost. I don't have any data to to say how well a NUMA system would perform with a mixed load of on and off node access, but when presented with the option of running a domain entirely within a node on a NUMA system, we should. What amount of testing is enough for you to consider enabling numa=on by default post 3.0.5? I think we ought to cook numa=on by default through another development cycle so we have time to address any performance issues that may arise. 1. http://lists.xensource.com/archives/html/xen-devel/2006-07/msg01057.html 2. http://lists.xensource.com/archives/html/xen-devel/2006-09/msg00958.html -- Ryan Harper Software Engineer; Linux Technology Center IBM Corp., Austin, Tx (512) 838-9253 T/L: 678-9253 ryanh@us.ibm.com