From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: arm64: Approach for DT based NUMA and issues Date: Sun, 27 Nov 2016 21:51:32 +0100 Message-ID: <1480279892.2712.261.camel@citrix.com> References: <1480208512.2712.241.camel@citrix.com> <6dd921fa-eef2-08ba-565e-e0dfa1f0ab4e@arm.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============7228504176442841805==" Return-path: Received: from mail6.bemta6.messagelabs.com ([193.109.254.103]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cB6QO-0001FS-EC for xen-devel@lists.xenproject.org; Sun, 27 Nov 2016 20:51:48 +0000 In-Reply-To: <6dd921fa-eef2-08ba-565e-e0dfa1f0ab4e@arm.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Julien Grall , Vijay Kilari , Stefano Stabellini , Andre Przywara Cc: xen-devel , "prasun.kapoor" List-Id: xen-devel@lists.xenproject.org --===============7228504176442841805== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-Tn37Q21pLLx4PuobsEJW" --=-Tn37Q21pLLx4PuobsEJW Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Sun, 2016-11-27 at 12:23 +0000, Julien Grall wrote: > Hi Dario, >=20 Hi, > On 27/11/2016 01:01, Dario Faggioli wrote: > > On Sat, 2016-11-26 at 12:29 +0530, Vijay Kilari wrote: > > I agree that we need to support vNUMA for Dom0 sooner rather than > > later, and I agree that Dom0 is a bit special, so some tricks may > > be > > necessary. But until we don't implement vNUMA for Dom0, Dom0 is > > just a > > non-NUMA virtual machine, and the kernel running inside that should > > just behave like it behaves on a non-NUMA box. > >=20 > > Again, I don't know much about ARM, but I think that, until we > > don't > > have vNUMA for Dom0, that devm_zalloc() thing will just see 1 and > > only > > 1 NUMA node from which to allocate memory. >=20 > I would rather divide the NUMA work for ARM in 2 distinct tasks: > - Make Xen NUMA-aware > - Make DOM0 NUMA-aware >=20 That makes perfect sense to me, and FWIW, is also what I'd do. In fact, the whole point of what I was saying was not to confuse Xen NUMA support and Dom0 NUMA support; if we want to do both of them, the latter right after the former, fine, but they're separate things indeed. > Vijay, if I understood correctly what Dario said, on x86 DOM0 is not > yet=C2=A0 > NUMA-aware.=20 > You did. It is not. > > > Ex: SMMU driver of device on node 1 tries to allocate memory > > > on node 1. > > >=20 > > > ISSUE: > > > =C2=A0- Dom0's memory should be split across all the available memory > > > nodes > > > =C2=A0=C2=A0=C2=A0of the system and memory nodes should be generated > > > accordingly. > > >=20 > > ...This is the default behavior, at least on x86. >=20 > Are you speaking about the command line parameter dom0_nodes? >=20 Not exactly. As said, Dom0 is not NUMA aware and does not have any virtual NUMA layout. This means that, by default, Dom0 memory is indeed spread among various existing nodes. Eg., on my NUMA test box here at home, here's how things are for Dom0: (XEN) [ 970.100116] NODE0 start->1720320 size->1572864 free->0 (XEN) [ 970.100122] NODE1 start->0 size->1720320 free->460155 (XEN) [ 970.100130] CPU0...7 -> NODE0 (XEN) [ 970.100136] CPU8...15 -> NODE1 (XEN) [ 970.100140] Memory location of each domain: (XEN) [ 970.100149] Domain 0 (total: 258512): (XEN) [ 970.102268] Node 0: 159254 (XEN) [ 970.102273] Node 1: 99258 dom0_nodes=3Dx is a way to tell Xen to (try as hard as it can) to only allocate the memory for dom0 only from NUMA node x but, even if more than one node is specified, that does not include giving to him a virtual NUMA topology, nor making it aware of the underline NUMA topology of the host in any way. > > Generating DT nodes for Dom0 is exactly what I mean when I say > > "implementing / enabling vNUMA for Dom0" (in this case on ARM). > >=20 > > So, yes, let's do it, but let's discuss how to do it properly > > (e.g., if > > there's anything that can be common between archs, such as some > > bits of > > the interface). >=20 > I would expect vNUMA for Dom0 to be common between x86 and ARM. >=20 As much as possible, indeed. Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-Tn37Q21pLLx4PuobsEJW Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJYO0dWAAoJEBZCeImluHPu44cP/iFNYWN7/VO6pVMvlNYBrFtg r3Azkk9Ke2UpAmoaxIyevtD5sm/zr4GSBuqkaZpYFWISNyhrv4rw0zuIVLW0gNcs fC6OO5rZ7akmjIA6aBN0cfs95QgUaqYQ9Q+Ts95dV8qX6Ua10+fS9XDH7sd4sPSQ A7+Apgk9R/E5tYwY7E4I+/nQE7ZocmSycZfdGw5Kfj5X8mslW1VewnaXB/RZBNEl w/ywoZhzaIr9L9UW5IXDoHcMNYFJUg9NoIN86o0ZUYeMGj8NaLk58uHXl62NpTK8 55IZshnjFzsOwYVm6RrmPUlD5WgX1w7N1ndu/YiMdGgnkyhEGTg2YYuqS44siRFT 71Ch06N4c8K7P2hNEZmRPVZe+0lIbBjb9+2KAtTquHs1fNV/9Loc86HMBlNryldh j7K8GvOZJ5Cpz6S39pXWGkqYnpqfQ8cAISqs04Y8BeY5pT09hf/Qsn8Byxq5lJ+d aPQiPKJoySPpUyByHTjtMTf3wXo4jwRwVRBEPZwOWjIzULxPXNYoC21hdG3YrS03 6wrx/jS8r2uF5jat8EER7doIpsODJyQ6q+UEz0+8W+HOvI5JifkpGqBWp34COhvr 4yDgoqtctaQ4P18FLbmW+PZDmq5CIAODooMI8cB7Cj9vz3n9/NmeCidi69kTTAYZ dDUDAtI00E2HtP0LraGi =Fd3y -----END PGP SIGNATURE----- --=-Tn37Q21pLLx4PuobsEJW-- --===============7228504176442841805== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============7228504176442841805==--