From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: About vcpu wakeup and runq tickling in credit Date: Fri, 16 Nov 2012 13:00:52 +0100 Message-ID: <1353067252.5351.124.camel@Solace> References: <1350999260.5064.56.camel@Solace> <5086B4DF.6060701@eu.citrix.com> <1352981447.5351.51.camel@Solace> <50A4DD95.5020107@eu.citrix.com> <1353063234.5351.107.camel@Solace> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2383207931543302055==" Return-path: In-Reply-To: <1353063234.5351.107.camel@Solace> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: George Dunlap Cc: Keir Fraser , David Vrabel , Jan Beulich , xen-devel List-Id: xen-devel@lists.xenproject.org --===============2383207931543302055== Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-591mSKpjXB/GL/EzDm3n" --=-591mSKpjXB/GL/EzDm3n Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, 2012-11-16 at 11:53 +0100, Dario Faggioli wrote: > On Thu, 2012-11-15 at 12:18 +0000, George Dunlap wrote: > > Maybe what we should do is do the wake-up based on who is likely to run= =20 > > on the current cpu: i.e., if "current" is likely to be pre-empted, look= =20 > > at idlers based on "current"'s mask; if "new" is likely to be put on th= e=20 > > queue, look at idlers based on "new"'s mask. > >=20 > Ok, find attached the two (trivial) patches that I produced and am > testing in these days. Unfortunately, early results shows that I/we > might be missing something. >=20 I'm just came to thinking that this approach, although more, say, correct, could have a bad impact on caches and locality in general. In fact, suppose a new vcpu N wakes up on pcpu #x where another vcpu C is running, with prio(N)>prio(C). What upstream does is asking to #x and to all the idlers that can execute N to reschedule. Doing both is, I think, wrong, as there's the chance of ending up with N being scheduled on #x and C being runnable but not running (in #x's runqueue) even if there are idle cpus that could run it, as they're not poked (as already and repeatedly said). What the patches do, in this case (remember (prio(N)>prio(C)), is asking #x and all the idlers that can run C to reschedule, the effect being that N will likely run on #x, after a context switch, and C will run somewhere else, after a migration, potentially wasting its cache-hotness (it is running after all!). It looks like we can do better... Something like the below: + if there are no idlers where N can run, ask #x and the idlers where=20 C can run to reschedule (exactly what the patches do, although, they=20 do that _unconditionally_), as there isn't anything else we can do to try to make sure they both will run; + if *there*are* idlers where N can run, _do_not_ ask #x to reschedule=20 and only poke them to come pick N up. In fact, in this case, it is=20 not necessary to send C away for having both the vcpus ruunning, and=20 it seems better to have N experience the migration as, since it's=20 waking-up, it's more likely for him than for C to be cache-cold. I'll run the benchmarks with this variant as soon as the one that I'm running right now finish... In the meanwhile, any thoughts? Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://retis.sssup.it/people/faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-591mSKpjXB/GL/EzDm3n Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iEYEABECAAYFAlCmKvQACgkQk4XaBE3IOsQCFQCfQMQYQ8D9jUKKn5WWzgUkACZD PbsAnAuKcE8iHBoS/VzadHjedt6RFtPD =i2k7 -----END PGP SIGNATURE----- --=-591mSKpjXB/GL/EzDm3n-- --===============2383207931543302055== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============2383207931543302055==--