From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andre Przywara Subject: Re: RFC: automatic NUMA placement Date: Mon, 27 Sep 2010 23:46:59 +0200 Message-ID: <4CA110D3.5050000@amd.com> References: <4C921DDF.6020809@ts.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: quoted-printable Return-path: In-Reply-To: <4C921DDF.6020809@ts.fujitsu.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Juergen Gross Cc: "xen-devel@lists.xensource.com" List-Id: xen-devel@lists.xenproject.org Juergen Gross wrote: > Hi, >=20 > I just stumbled upon the automatic pinning of vcpus on domain creation = in > case of NUMA. > This behaviour is questionable IMO, as it breaks correct handling of > scheduling weights on NUMA machines. > I would suggest to switch this feature off per default and make it a > configuration option of xend. It would make sense, however, to change c= pu pool > processor allocation to be NUMA-aware. > Switching NUMA off via boot option would remove NUMA-optimized memory > allocation, which would be sub-optimal :-) Hi J=FCrgen, stumbled over your mail just now, so sorry for the delay. First: Don't turn off automatic NUMA placement ;-) In my tests it helped a lot to preserve performance on NUMA machines. I was just browsing through the ML archive to find your original CPU=20 pools description from April, and it seems to fit the requirements in=20 NUMA machines quite well. I haven't done any experiments with Cpupools nor haven't looked at the=20 code yet, but just a quick idea: What about if we marry static NUMA placement and Cpupools? I'd suggest to introduce static NUMA pools, one for each node. The CPUs=20 assigned to each pool are fixed and cannot be removed nor added (because=20 the NUMA topology is fixed). Is that possible? Can we assign one physical CPUs to multiple pools (to=20 Pool-0 and to NUMA-0?) Or are they exclusive or hierarchical like the=20 Linux' cpusets? We could introduce magic names for each NUMA pool, so that people just=20 say cpupool=3D"NUMA-2" and get their domain pinned to that pool. Without=20 any explicit assignment the system would pick a NUMA node (like it does=20 today) and would just use the respective Cpupool. I think that is very=20 similar to what it does today, only that the pinning nature is more=20 evident to the user (as it uses the Cpupool name space). Also it would allow for users to override the pinning by specifying a=20 different Cpupool explicitly (like Pool-0). Just tell me what you think about this and whether I am wrong with my=20 thinking ;-) Regards, Andre. --=20 Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 448-3567-12