From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Can't always start 32 bit domains after 64 bit domains Date: Tue, 22 Nov 2016 19:46:45 +0100 Message-ID: <1479840405.2712.36.camel@citrix.com> References: <3423fa13-18bd-ff7b-f44a-af015eda2eb7@prgmr.com> <5832BC4F0200007800120396@prv-mh.provo.novell.com> <5832D50602000078001203F7@prv-mh.provo.novell.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3139034624977875934==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Sarah Newman , Andrew Cooper , Jan Beulich , LarsKurth Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============3139034624977875934== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-0nQpaEMeeZiMMZPTGxN2" --=-0nQpaEMeeZiMMZPTGxN2 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2016-11-21 at 13:06 -0800, Sarah Newman wrote: > On 11/21/2016 11:37 AM, Sarah Newman wrote: > >=C2=A0 > > If that's the reason not all the higher memory is being used first: > > is a potential workaround to pin 64 bit domains to the second > > physical core on > > boot, and 32 bit domains to the first physical core on boot, and > > then change the allowed cores with 'xl vcpu-pin' after the domain > > is loaded? >=20 > Free memory on a test server with no domains: > node:=C2=A0=C2=A0=C2=A0=C2=A0memsize=C2=A0=C2=A0=C2=A0=C2=A0memfree=C2=A0= =C2=A0=C2=A0=C2=A0distances > =C2=A0=C2=A0=C2=A00:=C2=A0=C2=A0=C2=A0=C2=A0148480=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0142983=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A010,21 > =C2=A0=C2=A0=C2=A01:=C2=A0=C2=A0=C2=A0=C2=A0147456=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0144645=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A021,10 >=20 > Free memory booting 116 256M 64-bit domains, limited to cpus=3D'all,^0- > 1' on boot: > node:=C2=A0=C2=A0=C2=A0=C2=A0memsize=C2=A0=C2=A0=C2=A0=C2=A0memfree=C2=A0= =C2=A0=C2=A0=C2=A0distances > =C2=A0=C2=A0=C2=A00:=C2=A0=C2=A0=C2=A0=C2=A0148480=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0128416=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A010,21 > =C2=A0=C2=A0=C2=A01:=C2=A0=C2=A0=C2=A0=C2=A0147456=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0129669=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A021,10 >=20 > Free memory booting 116 256M 64-bit domains, limited to cpus=3D'12-23' > on boot: > node:=C2=A0=C2=A0=C2=A0=C2=A0memsize=C2=A0=C2=A0=C2=A0=C2=A0memfree=C2=A0= =C2=A0=C2=A0=C2=A0distances > =C2=A0=C2=A0=C2=A00:=C2=A0=C2=A0=C2=A0=C2=A0148480=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0143397=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A010,21 > =C2=A0=C2=A0=C2=A01:=C2=A0=C2=A0=C2=A0=C2=A0147456=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0114693=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A021,10 >=20 > This looks like a viable workaround. Where should I document it? >=20 It's documented in here (although, the text can be improved a bit, I think)... look for 'cpus' and 'cpus_soft' within the page: https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html A more clear mention to using "all" can perhaps be added to the wiki pages I listed in my other email. However, what I think is totally missing, is any documentation about the fact that, in use cases like yours, domain creation should be done in a certain order, for what reasons, which order is that, and the fact that NUMA placement may interfere. I'm not sure where and how to properly document all this [adding Lars], but I'd say it probably deserves a dedicated wiki page. Thoughts? Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-0nQpaEMeeZiMMZPTGxN2 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJYNJKVAAoJEBZCeImluHPuyooP/jQTqSdx2ydT8V8ebccV4GLj sFkSUDdosnDUAvAuiUjb6n+PqZgGF6RWETZ0Tga+7vdAJyYJJUYK4FAA4AS5lPCz 0kaIJ2Tdet1d3AXJbl7nYu5c4JKVnm6svx55fdwbR8OPA0EuGs8zQZ7psZ6FS/VG JC/9R0X64AQ3NethfyPftniY/E4EZ4Yc3tbUDe03wE4ENCTHq3VYhc7oNAek9sWo xBpF5VfJXsNDgs10d8oN/dqnRv1aijJ/TEvATkqhd3zGQJoBfJisFoADqxkffCc2 3oaOEx0t3Hlp/VwBn+1+kBc71tOwkzCsH+R2Hdu041b9mpzxqTJa1eGeNgxn45GU YPRE8UknOkTk6LgJnYo/uF0zJLOowqN3P5c5zAgtz4KzxOrOs5iVO6tyhkiuZQl0 syb9P18KUFU2Z7zD5srgTngIPdKsDTjO6xjPX7mpeICa9/EcMDFhChOhGY3bDgvf eHfg4UCaJ0EvOSnkqOZp0nSVGRZabTknxFeG9UYsHR+ylcdd5pscWh8+2/YS/KUn V+TwfHlVEdAwLU9cQunQTschrAp78GS9wauiZ0B90+2Wxt+X6GsCdVc6nzJYDEXB uNcljiZilydDH5Zl1xtvQlb0ZYhV2EmeWjXOxDJiZ6geiBvgtuXTu/jAjZaBUl1K o9a8hPD2Qf7XESjJiGaG =5ICW -----END PGP SIGNATURE----- --=-0nQpaEMeeZiMMZPTGxN2-- --===============3139034624977875934== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============3139034624977875934==--