From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH RESEND 05/12] xen: numa-sched: make space for per-vcpu node-affinity Date: Wed, 6 Nov 2013 15:26:03 +0100 Message-ID: <1383747963.9207.134.camel@Solace> References: <20131105142844.30446.78671.stgit@Solace> <20131105143500.30446.9976.stgit@Solace> <5279143702000078000FFB15@nat28.tlf.novell.com> <527908B2.5090208@eu.citrix.com> <52790A93.4020903@eu.citrix.com> <52791B8702000078000FFBC4@nat28.tlf.novell.com> <5279114B.9080405@eu.citrix.com> <52792326.4050206@eu.citrix.com> <527927E3.3000004@eu.citrix.com> <1383730770.9207.93.camel@Solace> <527A1DFC0200007800100033@nat28.tlf.novell.com> <1383732000.9207.99.camel@Solace> <527A2BA8.9060601@eu.citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6628790040099730423==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1Ve43o-0002NE-HM for xen-devel@lists.xenproject.org; Wed, 06 Nov 2013 14:26:20 +0000 In-Reply-To: <527A2BA8.9060601@eu.citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: George Dunlap Cc: MarcusGranado , Justin Weaver , IanCampbell , Li Yechen , Andrew Cooper , JuergenGross , Ian Jackson , Jan Beulich , xen-devel , Daniel De Graaf , KeirFraser , Matt Wilson , Elena Ufimtseva List-Id: xen-devel@lists.xenproject.org --===============6628790040099730423== Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="=-3gZHLvGsOPTZMWaAhp6Y" --=-3gZHLvGsOPTZMWaAhp6Y Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On mer, 2013-11-06 at 11:44 +0000, George Dunlap wrote: > On 06/11/13 10:00, Dario Faggioli wrote: > > I see, and that sounds sensible to me... It's mostly a matter a matter > > of deciding whether o not we want something like that, and, if yes, > > whether we want it based on hard of soft. > > > > Personally, I think I agree with you on having it based on hard > > affinities by default. > > > > Let's see if George get to say something before I get to that part of > > the (re)implementation. :-) >=20 > I would probably have it based on soft affinities, since that's where we= =20 > expect to have the domain's vcpus actually running most if the time;=20 > True. However, doing that would rule out cpupool and vcpu-pinning. That is to say, if you create a domain in a pool or with its vcpu pinned (by specifying "pool=3D" or "cpus=3D" in the config file), it'll get its memory striped over all the nodes. In fact, in both these cases, there really is no soft affinity involved. This is bad, because people may be already used to create a domain with "cpus=3D", and have the memory allocated from the relevant nodes. OTOH, there is no similar thing (i.e., a user interface/API) for soft affinities, and the way I was using it at the xl and libxl level for the sake of NUMA performance, I can easily implement there on top of soft affinities. So, in summary, I'd have liked to have it based on soft affinity too, but I think that would break more things than having it based on hard ones. > but=20 > it's really a bike-shed issue, and something we can change / adjust in= =20 > the future. >=20 That is also true, indeed. > (Although I suppose ideal behavior would be for the allocator to have=20 > three levels of preference instead of just two: allocate from soft=20 > affinity first; if that's not available, allocate from hard affinity;=20 > and finally allocate wherever you can find ram. But that's probably=20 > more work than it's worth at this point.) >=20 I like this too, but that's definitely something longer term than this or next week. > So what's the plan now Dario? You're going to re-spin the patch series= =20 > to just do hard and soft affinities at the HV level, plumbing the=20 > results through the toolstack? >=20 Yes, that was what I had in mind, and what I already started working on (see the other mail I sent before lunch about (re)naming). I should be able to craft and send something by ether today or tomorrow. > I think for now I might advise putting off doing a NUMA interface at the= =20 > libxl level, and do a full vNUMA interface in another series (perhaps=20 > for 4.5, depending on the timing). >=20 Well, I agree that all this is of very little use without vNUMA, but at the same time, it's not necessarily only useful for it. Also, whatever it is vcpu-node-affinity or soft-affinity, if it is not wired up properly up to the higher layers, there's very few point of having the HV part only... So my idea was to redo and resend everything, including libxl and xl bits. Of course, that doesn't mean we must necessarily have this for 4.4 (although I think it would still be feasible), just that we either check-in or wait for both the implementation and the interface. Again, how's the updated release schedule? Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-3gZHLvGsOPTZMWaAhp6Y Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.15 (GNU/Linux) iEYEABECAAYFAlJ6UXsACgkQk4XaBE3IOsS80gCeJ7PcFvEHdI9hhAI65YSD7HxU xioAoKNA4GNaPtJBKn4gVFwdbwa3eJ+U =yjwe -----END PGP SIGNATURE----- --=-3gZHLvGsOPTZMWaAhp6Y-- --===============6628790040099730423== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============6628790040099730423==--