From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH RESEND 05/12] xen: numa-sched: make space for per-vcpu node-affinity Date: Tue, 5 Nov 2013 23:54:18 +0100 Message-ID: <1383692058.15457.47.camel@Abyss> References: <20131105142844.30446.78671.stgit@Solace> <20131105143500.30446.9976.stgit@Solace> <5279143702000078000FFB15@nat28.tlf.novell.com> <527908B2.5090208@eu.citrix.com> <52790A93.4020903@eu.citrix.com> <52791B8702000078000FFBC4@nat28.tlf.novell.com> <5279114B.9080405@eu.citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2585348628231896003==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VdpVx-0001f8-W9 for xen-devel@lists.xenproject.org; Tue, 05 Nov 2013 22:54:26 +0000 In-Reply-To: <5279114B.9080405@eu.citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: George Dunlap Cc: MarcusGranado , Justin Weaver , Jan Beulich , Li Yechen , Andrew Cooper , Juergen Gross , Ian Jackson , Matt Wilson , xen-devel , Daniel De Graaf , KeirFraser , Elena Ufimtseva , Ian Campbell List-Id: xen-devel@lists.xenproject.org --===============2585348628231896003== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-tb5/6XvdwUimlaU8GHYV" --=-tb5/6XvdwUimlaU8GHYV Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mar, 2013-11-05 at 15:39 +0000, George Dunlap wrote: > On 11/05/2013 03:23 PM, Jan Beulich wrote: > >>>> On 05.11.13 at 16:11, George Dunlap wr= ote: > >> Or, we could internally change the names to "cpu_hard_affinity" and > >> "cpu_soft_affinity", since that's effectively what the scheduler will > >> do. It's possible someone might want to set soft affinities for some > >> other reason besides NUMA performance. > > > > I like that. >=20 I like that too. > A potential problem with that is the "auto domain numa" thing. In this= =20 > patch, if the domain numa affinity is not set but vcpu numa affinity is,= =20 > the domain numa affinity (which will be used to allocate memory for the= =20 > domain) will be set based on the vcpu numa affinity. > That would indeed be an issue, but, hopefully, even if we want to go for the soft!=3DNUMA way, that would mean that, at the toolstack level, two calls instead than one are required to properly place a guest on one/some NUMA node: one for vcpus (setting the soft affinity) and one for memory (setting the node affinity, which would then affect mem only). Really, let's see if I succeed in coding it up tomorrow morning, but I don't expect too serious issues. The only thing that puzzles me is what the 'default behaviour' should be. I mean, if in a VM config file I find the following: cpus =3D "2,9" And nothing else, related to hard, soft or memory affinity, what do I do? I'd say, even if only for consistency with what we're doing right now (even without this patch), we should go all the way through both the calls above, and set both hard affinity and memory node affinity. If not, people doing this in 4.4 (or 4.5, if we miss the train) would experience different behaviour wrt current one, which would be bad! Thoughts? Similar question, provided I implement something like libxl_set_vcpu_hardaffinity(), libxl_set_vcpu_softaffinity() and libxl_set_mem_nodeaffinity(), what should a call to the existent libxl_set_vcpuaffinity() do? As above, right now it affects both vcpu-pinning and node-affinity, so I'd say it should call both libxl_set_vcpu_hardaffinity() and libxl_set_mem_nodeaffinity(). Is that ok? BTW, does the interface sketched above look reasonable? Someone else up for suggesting more fancy names? :-) > That seems like a=20 > useful feature (though perhaps it's starting to violate the "policy=20 > should be in the tools" principle). =20 > The current situation being that d->node_affinity is set based on vcpu-pinning, having it retain the same meaning and behavior, apart from being set based on vcpu node-affinity looked like the natural extension of it to me. However... > If we change this to just "hard=20 > affinity" and "soft affinity", we'll lose the natural logical connection= =20 > there. > ...I think I see what you mean and I agree that it wasn't that much of a violation if we stick to soft=3D=3DNUMA (as implemented in the series). If we go for your proposal (which, again, I like), it'd indeed become hard to decide whether node affinity should derive from hard or soft affinity. :-) > It might have impacts on how we end up doing vNUMA as well. So=20 > I'm a bit torn ATM. >=20 > Dario, any thoughts? >=20 As said above, I like the idea of separating soft affinities and NUMA-ness... I'll give it a try and either submit an updated series implementing such behaviour, or follow-up to this conversation with any issue I find while doing so. :-) Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-tb5/6XvdwUimlaU8GHYV Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlJ5dxoACgkQk4XaBE3IOsS5RwCePogSLZVs6gvt/Wvce1l9smmE YskAoIxuy0NSWPn+TEvd8PDzsgui4Sns =s6Y7 -----END PGP SIGNATURE----- --=-tb5/6XvdwUimlaU8GHYV-- --===============2585348628231896003== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2585348628231896003==--