From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH RESEND 05/12] xen: numa-sched: make space for per-vcpu node-affinity Date: Wed, 6 Nov 2013 17:48:31 +0100 Message-ID: <1383756511.9207.165.camel@Solace> References: <20131105142844.30446.78671.stgit@Solace> <20131105143500.30446.9976.stgit@Solace> <5279143702000078000FFB15@nat28.tlf.novell.com> <527908B2.5090208@eu.citrix.com> <52790A93.4020903@eu.citrix.com> <52791B8702000078000FFBC4@nat28.tlf.novell.com> <5279114B.9080405@eu.citrix.com> <52792326.4050206@eu.citrix.com> <527927E3.3000004@eu.citrix.com> <1383730770.9207.93.camel@Solace> <527A1DFC0200007800100033@nat28.tlf.novell.com> <1383732000.9207.99.camel@Solace> <527A2BA8.9060601@eu.citrix.com> <1383747963.9207.134.camel@Solace> <527A5890.90709@eu.citrix.com> <527A6AF2020000780010033E@nat28.tlf.novell.com> <527A6A6A.6030604@eu.citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2571635145667662071==" Return-path: Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Ve6HW-0003uZ-Rx for xen-devel@lists.xenproject.org; Wed, 06 Nov 2013 16:48:39 +0000 In-Reply-To: <527A6A6A.6030604@eu.citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: George Dunlap Cc: MarcusGranado , Justin Weaver , IanCampbell , Li Yechen , Andrew Cooper , JuergenGross , Ian Jackson , Jan Beulich , xen-devel , DanielDe Graaf , KeirFraser , MattWilson , Elena Ufimtseva List-Id: xen-devel@lists.xenproject.org --===============2571635145667662071== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-2MjO5PP8frSo1jlvqmnH" --=-2MjO5PP8frSo1jlvqmnH Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mer, 2013-11-06 at 16:12 +0000, George Dunlap wrote: > We have two "affinities" at the moment: > * per-vcpu cpu affinity. This is this is a scheduling construct, and is= =20 > used to restrict vcpus to run on specific pcpus. This is what we would= =20 > call hard affinity -- "hard" because vcpus are *not allowed* to run on= =20 > cpus not in this mask. > * Domain NUMA affinity. As of 4.3 this has two functions: both a memory= =20 > allocation function, and a scheduling function. For the scheduling=20 > function, it acts as a "soft affinity", but it's domain-wide, rather=20 > than being per-vcpu. It's "soft" because the scheduler will *try* to=20 > run it on pcpus in that mask, but if it cannot, it *will allow* them to= =20 > run on pcpus not in that mask. >=20 > At the moment you can set this manually, or if it's not set, then Xen=20 > will set it automatically based on the vcpu hard affinities (and I think= =20 > the cpupool that it's in). >=20 > What we're proposing is to remove the domain-wide soft affinity and=20 > replace it with a per-vcpu soft affinity. That way each vcpu has some= =20 > pcpus that it prefers (in the soft affinity *and* the hard affinity),=20 > some that it doesn't prefer but is OK with (hard affinity but not soft= =20 > affinity), and some that it cannot run on at all (not in the hard affinit= y). >=20 Great explanation... A bit longer but *much* better than mine. :-) > The question Dario has is this: given that we now have per-vcpu hard and= =20 > soft scheduling affinity, how should we automatically construct the=20 > per-domain memory allocation affinity, if at all? Should we construct= =20 > it from the "hard" scheduling affinities, or from the "soft" scheduling= =20 > affinities? >=20 Exactly. > I said that I thought we should use the soft affinity; but I really=20 > meant the "effective soft affinity" -- i.e., the union of soft, hard,=20 > and cpupools. >=20 Aha... I see it now... Only one more thing: union or intersection? Union would be nice, especially because there are very few risks of it being empty, but it may be too broad (if one of the tree is 'all cpus/nodes', so will the memory affinity). I'd go for the intersection, with some extra care to avoid degeneration into the empty set. Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-2MjO5PP8frSo1jlvqmnH Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlJ6ct8ACgkQk4XaBE3IOsS6vQCeMo13Ck0RFsfYb6dLKe3C/Eli 8OUAoIYA7OfnIPNhfYYPsFUThOXcJkVq =kL3I -----END PGP SIGNATURE----- --=-2MjO5PP8frSo1jlvqmnH-- --===============2571635145667662071== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2571635145667662071==--