From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Can't always start 32 bit domains after 64 bit domains Date: Tue, 22 Nov 2016 22:40:19 +0100 Message-ID: <1479850819.2712.53.camel@citrix.com> References: <3423fa13-18bd-ff7b-f44a-af015eda2eb7@prgmr.com> <5832BC4F0200007800120396@prv-mh.provo.novell.com> <5832D50602000078001203F7@prv-mh.provo.novell.com> <1479840405.2712.36.camel@citrix.com> <6a400e0e-9150-5652-706d-95dc4b660f8d@prgmr.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2734563485764385398==" Return-path: In-Reply-To: <6a400e0e-9150-5652-706d-95dc4b660f8d@prgmr.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Sarah Newman , Andrew Cooper , Jan Beulich , LarsKurth Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============2734563485764385398== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-QJRu0c477X0yGyiGpKO0" --=-QJRu0c477X0yGyiGpKO0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, 2016-11-22 at 11:37 -0800, Sarah Newman wrote: > On 11/22/2016 10:46 AM, Dario Faggioli wrote: > > It's documented in here (although, the text can be improved a bit, > > I think)... look for 'cpus' and 'cpus_soft' within the page:=C2=A0 > > https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html > >=20 > > A more clear mention to using "all" can perhaps be added to the > > wiki pages I listed in my other email. > >=20 >=20 > Testing shows that memory allocation is still alternated between the > two physical nodes, even after setting cpus=3D'all' or removing > cpus=3D... from the > configuration file entirely. >=20 Not sure I'm getting. So, libxl default is to try to place a new domain on a NUMA node (or on the smallest possible set of NUMA nodes). If you don't specify neither cpus=3D nor cpus_soft=3D such default applies, and you should _not_ get memory spread on more than one node (unless placement fails for some reason, or the domain does not fit there). If you specify cpus=3D"node:0" or cpus_soft=3D"node:0", libxl automatic NUMA placement won't be execute, and you'll get memory only from NUMA node 0. If you specify cpus=3D"all", libxl automatic NUMA placement won't be executed, and you get memory from all the NUMA nodes, which AFAICT, is what Xen does, if not instructed otherwise by the toolstack. Are you seeing something different from this? > If you're saying not specifying "cpus=3D..." will keep libxl from > interfering with the Xens default allocation policy, then Xens > default allocation > policy no longer starts from the top of memory for 64 bit PV domains, > at least for 4.6.3. Maybe it's starting from the top of memory per > node. >=20 No, I'm saying that _not_ specifying cpus=3D will trigger libxl interference. _Do_ specifying it, will either limit or disable it, depending on how you specify it. That's all I'm saying. As far as I can remember, Xen's always been striping the memory among all the available nodes, if not told otherwise. This has at least been the case for all my test and experiments, on any NUMA box I've touched. I have to admit I've not played much with 32 bit guests, though. And let me add that, if we figure out what and where the issue is, and come up with a sensible plan, I'm happy to think and help improving xl and libxl NUMA placement to take these constraints into account. Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-QJRu0c477X0yGyiGpKO0 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJYNLtDAAoJEBZCeImluHPup1cQANas7eE5hQOgfZVfS70avjvv HBta86ikuewujIl7bdvyVbqaweIuoDraPbUCziuCOWlzlGgpLlzqLkNLb2BOHybu U5rLeNW7daXWibRhOQBYOsl0obbU3M/XYgbCdtigSdrsiS9i4JHbBQmRCCl2ADuc 0z7n8inxjz5lNWfkHbnILpn67TG/Rea48ESGxVrGo0avMX5VfU4AfQkoTaUJGujv lgxqNCk8227TN5cukJ/p/GWFQCYnQuUWfpLgaOLK4Ko//x9RQQ9MrGEEJX1JpOOY KcQD+omSotKWgzW+2xchAY49s6QbX2X+YgNW8FnSre3ie/dOUZIH1jCeFY0SvqDl hsgrc49T4XqP9vRZw76Lycrk92fM0Oen34+3OyvtZoFMDLxhenEd3x7bjoyyr6w0 sXecpj+P6IGU/epsRuskd2StmfKz4TvdmLZibIHPWR7/936EOO+8OBTv1zpRrwek 4GY2AwI59wNXHuzYQhW00erbPja7Li554ZBPIQegj+2oGt6JlLiodHwTcQ2SpQt/ 1NqvKcpcWhvlaobhf+B2JPaRpsHd3Tft33aw/8VtbGiuCL6rlaItVace4hGsIUOS 3qeV8HGKa7mrYjGFkbj/jW3yVxQ6+EjynRBivlTateXe852ghb48b3D1922rDpWN ioZhqmyAnQ1nX840n3PS =6gj8 -----END PGP SIGNATURE----- --=-QJRu0c477X0yGyiGpKO0-- --===============2734563485764385398== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============2734563485764385398==--