From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH v2 16/16] libxl: automatic NUMA placement affects soft affinity Date: Thu, 14 Nov 2013 17:48:40 +0100 Message-ID: <1384447720.29902.180.camel@Abyss> References: <20131113190852.18086.5437.stgit@Solace> <20131113191334.18086.72275.stgit@Solace> <5284F435.7080106@eu.citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5146968989922841898==" Return-path: In-Reply-To: <5284F435.7080106@eu.citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: George Dunlap Cc: Marcus Granado , Keir Fraser , Ian Campbell , Li Yechen , Andrew Cooper , Juergen Gross , Ian Jackson , xen-devel@lists.xen.org, Jan Beulich , Justin Weaver , Matt Wilson , Elena Ufimtseva List-Id: xen-devel@lists.xenproject.org --===============5146968989922841898== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-sTovKnZR3QFZXwmkpKcN" --=-sTovKnZR3QFZXwmkpKcN Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On gio, 2013-11-14 at 16:03 +0000, George Dunlap wrote: > On 13/11/13 19:13, Dario Faggioli wrote: > > vCPU soft affinity and NUMA-aware scheduling does not have > > to be related. However, soft affinity is how NUMA-aware > > scheduling is actually implemented, and therefore, by default, > > the results of automatic NUMA placement (at VM creation time) > > are also used to set the soft affinity of all the vCPUs of > > the domain. > > > > Of course, this only happens if automatic NUMA placement is > > enabled and actually takes place (for instance, if the user > > does not specify any hard and soft affiniy in the xl config > > file). > > > > This also takes care of the vice-versa, i.e., don't trigger > > automatic placement if the config file specifies either an > > hard (the check for which was already there) or a soft (the > > check for which is introduced by this commit) affinity. >=20 > It looks like with this patch you set *both* hard and soft affinities = =20 > when doing auto-numa placement. Would it make more sense to change it= =20 > to setting only the soft affinity, and leaving the hard affinity to "any"= ? > Nope, it indeed sets only soft affinity after automatic placement, hard affinity is left untouched. > (My brain is running low, so forgive me if I've mis-read it...) >=20 :-) This is the spot: > > Signed-off-by: Dario Faggioli > > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > > @@ -222,21 +222,39 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domi= d, > > * some weird error manifests) the subsequent call to > > * libxl_domain_set_nodeaffinity() will do the actual placement, > > * whatever that turns out to be. > > + * > > + * As far as scheduling is concerned, we achieve NUMA-aware schedu= ling > > + * by having the results of placement affect the soft affinity of = all > > + * the vcpus of the domain. Of course, we want that iff placement = is > > + * enabled and actually happens, so we only change info->cpumap_so= ft to > > + * reflect the placement result if that is the case > > */ > > if (libxl_defbool_val(info->numa_placement)) { > > =20 > > - if (!libxl_bitmap_is_full(&info->cpumap)) { > > + /* We require both hard and soft affinity not to be set */ > > + if (!libxl_bitmap_is_full(&info->cpumap) || > > + !libxl_bitmap_is_full(&info->cpumap_soft)) { > > LOG(ERROR, "Can run NUMA placement only if no vcpu " > > - "affinity is specified"); > > + "(hard or soft) affinity is specified"); > > return ERROR_INVAL; > > } > > =20 > > rc =3D numa_place_domain(gc, domid, info); > > if (rc) > > return rc; > > + > > + /* > > + * We change the soft affinity in domain_build_info here, of c= ourse > > + * after converting the result of placement from nodes to cpus= . the > > + * following call to libxl_set_vcpuaffinity_all_soft() will do= the > > + * actual updating of the domain's vcpus' soft affinity. > > + */ > > + libxl_nodemap_to_cpumap(ctx, &info->nodemap, &info->cpumap_sof= t); > ^ | Here: -----------------------------------------------------------/ I only copy the result of placement into info->cpumap_soft, without touching info->cpumap, which is "all" (or we won't be at this point) and stays that way. > > } > > libxl_domain_set_nodeaffinity(ctx, domid, &info->nodemap); > > libxl_set_vcpuaffinity_all(ctx, domid, info->max_vcpus, &info->cp= umap); > > + libxl_set_vcpuaffinity_all_soft(ctx, domid, info->max_vcpus, > > + &info->cpumap_soft); > > =20 Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-sTovKnZR3QFZXwmkpKcN Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlKE/ugACgkQk4XaBE3IOsSocgCeJ5r7LsdE8w0/eoHnwSeCKo2L exsAni0oKQM3pgLKb2mbdvWgy7mlsCQA =REl4 -----END PGP SIGNATURE----- --=-sTovKnZR3QFZXwmkpKcN-- --===============5146968989922841898== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============5146968989922841898==--