From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Can't always start 32 bit domains after 64 bit domains Date: Tue, 22 Nov 2016 19:37:20 +0100 Message-ID: <1479839840.2712.31.camel@citrix.com> References: <3423fa13-18bd-ff7b-f44a-af015eda2eb7@prgmr.com> <5832BC4F0200007800120396@prv-mh.provo.novell.com> <5832D50602000078001203F7@prv-mh.provo.novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3978871076847559379==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Sarah Newman , Andrew Cooper , Jan Beulich Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============3978871076847559379== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-015tkYuQD9WAJkevmF61" --=-015tkYuQD9WAJkevmF61 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2016-11-21 at 11:37 -0800, Sarah Newman wrote: > On 11/21/2016 05:21 AM, Andrew Cooper wrote: > > I suspect that libxl's preference towards NUMA allocation of > > domains > > interferes with this, by adding a NUMA constraints to memory > > allocations > > for 64bit PV guests. >=20 > I ran xl info -n (which I didn't know about before) and that shows > the problem much more clearly. >=20 > If that's the reason not all the higher memory is being used first: > is a potential workaround to pin 64 bit domains to the second > physical core on > boot, and 32 bit domains to the first physical core on boot, and then > change the allowed cores with 'xl vcpu-pin' after the domain is > loaded? >=20 If you're looking for a way to disable libxl's NUMA-aware domain placement --which does indeed interfere whit what memory (as in, the memory of what NUMA node) is going to be used for the domains-- you can "just" specify this, in all the domains' config files: cpus=3D"all" This will leave all the vcpus free to run everywhere, and stop libxl to pass down to Xen any hint on memory allocation. Using cpus=3D"foo" and/or cpus_soft=3D"bar" would allow to tweak things more. And the same is true for creating cpupools, with specific cpus from specific NUMA nodes in them, and creating the domain direcly inside those various pools. That being said, what values are the best for your use case, I'm not really sure... But maybe have a look at this. Some more info: https://wiki.xen.org/wiki/Xen_on_NUMA_Machines https://wiki.xen.org/wiki/Xen_4.3_NUMA_Aware_Scheduling https://wiki.xen.org/wiki/Tuning_Xen_for_Performance#vCPU_Soft_Affinity_for= _guests Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-015tkYuQD9WAJkevmF61 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJYNJBgAAoJEBZCeImluHPuAawQALdsNHGUxXHq0QQ9J2V+kBxX 0nbruFsmtKcFp7h7voJRWW3osz/bRu9CLF98rUZNxAXPeUGMg5P59FMQIl9G/2ai QIDiSQP3sx5rKzjfxayBk2Crv2ht018iRH7vfa/rcDAb9DF5rUtHxyv+hToRbTh5 mi73ZppftL0/EKo2s02nBkS7SDFOEBkgDmggWas+wPtouw9xkum56OyhVF0JSPdF Dnxl0K3EyFkj1dhif3PcMvDV/5rWKycWFo7u0HkYRTI9SLjEl2iA2UntiM8Gn9C3 y8CFXHJLGyHMbBavGtkAQfpI1Y428PcdNtsgLGMhmVxwKOAcm8KLDwQ5jwb+6/df ngJUQhHe0iOl2cgBhyoUgY6tzlnXFY4Vhv6Igr5u8+Mu9wvcTxiBacBiTlqNUDIA WRUAJKPSjlgxmTfxwb8K4hYJg8lqhs5Aq0BWZxuzfStTr6X91+TYiJlRGYlpERTu M3TG8JuHHlOEk6r0YVx1te0YrR9ckYIpALbvF+SR37+sBppROceU2gTAyEcYDYJn YP5Mu2qPIfZE2ztFwEi9LU/DEyY7YPHdR17pQOdQrML0Iacd0mQZTM83EfJ6sIOA XyfBh59Drh7nBEJ9VClJV7z85gZ/ulk5NkWOpmEM9hkEy/Fkf9eKHrtjgRoDjtr+ 7ETzqABw3GaQiLJMjn/F =a+0X -----END PGP SIGNATURE----- --=-015tkYuQD9WAJkevmF61-- --===============3978871076847559379== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============3978871076847559379==--