From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Xen NUMA memory allocation policy Date: Wed, 18 Dec 2013 02:38:29 +0100 Message-ID: <1387330709.3880.52.camel@Solace> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1501571387969690849==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Saurabh Mishra Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org --===============1501571387969690849== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-KpkHNVzCzwEaJTkNSBeG" --=-KpkHNVzCzwEaJTkNSBeG Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mar, 2013-12-17 at 11:41 -0800, Saurabh Mishra wrote: > Hi -- > Hi, > We are using Xen 4.2.2_06 on SLES SP3 Updates and wanted to know if > there is a simple way to gather information about physical pages > allocated for a HVM guest.=20 > In general, no, here are no simple ways to retrieve such information. Actually, putting something together that would allow one to get much more info on the memory layout of a guest (wrt NUMA) is something that is on my TODO list for quite some time, but I haven't got there yet... I'll get to there eventually, and any help is appreciated! :-) > We are trying to figure whether XL is better off in allocating > contiguous huge/large pages for a guest or XM. I guess it does not > matter since Xen's hypervisor would be implementing page allocation > polices. >=20 Indeed. What changes between xl and xm/xend, is whether and how they build up a vcpu-to-pcpu pinning mask, when the domain is created. In fact, as of now, that is all that matters, as far as allocating pages on nodes (happening in the hypervisor) is concerned. In both cases, if you specify a vcpu-to-pcpu pinning mask in the domain config file, that is passed directly to the hypervisor, which would then allocate memory striping the pages on the NUMA nodes to which the pcpus in the mask belong. Also, in case no pinning is specified in the config file, both toolstack tries to come up with a best possible placement of the new guest on the host NUMA nodes, and build up a suitable vcpu-to-pcpu pinning mask, pass it to the hypervisor, and... See above. :-) What differs between xl and xm is the algorithm used to come up with such automatic placement (i.e., both algorithms are based on some heuristics, but those heuristics are different). I'd say that the xl's algorithm is better, but that's a very biased opinion, as I'm the one who wrote it! :-P However, since xl is the default toolstack, while xm is already deprecated and won't be even built by default very soon, I'm definitely saying, try xl, and, if there is anything that doesn't work or seems wrong, please report it here (putting me in Cc). Hope this clarifies things a bit for you... > With xl debug-key u, we know how much memory was allocated from each > NUMA node, but we would also like to know whether how much of them > were huge pages and were they contiguous or not.=20 > I'm not aware of any tool giving this sort of information. > Basically we need to retrieve machine pfn and VM's pfn to do some > comparison. >=20 Well, at some point, for debugging an understanding purposes, I wrote something called xen-mfnup, which is in tree: http://xenbits.xen.org/gitweb/?p=3Dxen.git;a=3Dcommit;h=3Dae763e4224304983a= 1cde2fbb3d6e0c4d60b2688 It does allow you to get some insights about pfn-s and mfn-s, but not as much as you need, I'm afraid (not to mention that I did it mostly with PV guests in mind, and tested mostly on them). > (XEN) Memory location of each domain: > (XEN) Domain 0 (total: 603765): > (XEN) Node 0: 363652 > (XEN) Node 1: 240113 > (XEN) Domain 1 (total: 2096119): > (XEN) Node 0: 1047804 > (XEN) Node 1: 1048315 > (XEN) Domain 2 (total: 25164798): > (XEN) Node 0: 12582143 > (XEN) Node 1: 12582655 >=20 >=20 Mmm... BTW, if I can ask, what's the config file for these domains? Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-KpkHNVzCzwEaJTkNSBeG Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKw/JUACgkQk4XaBE3IOsQHeQCfW+XsK3jRSf50E8//bMF18Ez1 620AnjoQpMl6Gi0QeGQ4mp5HCSUrHJ3R =69b+ -----END PGP SIGNATURE----- --=-KpkHNVzCzwEaJTkNSBeG-- --===============1501571387969690849== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1501571387969690849==--