From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: [PATCH 3/6] xen: credit1: increase efficiency and scalability of load balancing. Date: Thu, 6 Apr 2017 09:37:58 +0200 Message-ID: <1491464278.18721.10.camel@citrix.com> References: <148844531279.23452.17528540110704914171.stgit@Solace.fritz.box> <148845109955.23452.14312315410693510946.stgit@Solace.fritz.box> <57001cc1-de50-eea3-da2c-737b27981c1b@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============2001359895971407900==" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cw1zg-0004Yn-Sd for xen-devel@lists.xenproject.org; Thu, 06 Apr 2017 07:38:12 +0000 In-Reply-To: <57001cc1-de50-eea3-da2c-737b27981c1b@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Andrew Cooper , xen-devel@lists.xenproject.org Cc: George Dunlap List-Id: xen-devel@lists.xenproject.org --===============2001359895971407900== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-s2q0A5r8GOz2Wvkb2bpP" --=-s2q0A5r8GOz2Wvkb2bpP Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, 2017-03-02 at 11:06 +0000, Andrew Cooper wrote: > On 02/03/17 10:38, Dario Faggioli wrote: > >=20 > > To mitigate this, we introduce here the concept of > > overloaded runqueues, and a cpumask where to record > > what pCPUs are in such state. > >=20 > > An overloaded runqueue has at least runnable 2 vCPUs > > (plus the idle one, which is always there). Typically, > > this means 1 vCPU is running, and 1 is sitting in=C2=A0=C2=A0the > > runqueue, and can hence be stolen. > >=20 > > Then, in=C2=A0=C2=A0csched_balance_load(), it is enough to go > > over the overloaded pCPUs, instead than all non-idle > > pCPUs, which is better. > >=20 > Malcolm=E2=80=99s solution to this problem is https://github.com/xenserve= r/xe > n-4.7.pg/commit/0f830b9f229fa6472accc9630ad16cfa42258966=C2=A0 This has > been in 2 releases of XenServer now, and has a very visible > improvement for aggregate multi-queue multi-vm intrahost network > performance (although I can't find the numbers right now). >=20 > The root of the performance problems is that pcpu_schedule_trylock() > is expensive even for the local case, while cross-cpu locking is much > worse.=C2=A0 Locking every single pcpu in turn is terribly expensive, in > terms of hot cacheline pingpong, and the lock is frequently > contended. >=20 BTW, both my patch in this series, and the patch linked above are _wrong_ in using __runq_insert() and __runq_remove() for counting the runnable vCPUs. In fact, in Credit1, during the main scheduling function (csched_schedule()), we call runqueue insert for temporarily putting the running vCPU. This increments the counter, making all the other pCPUs think that there is a vCPU available for stealing in there, while that: 1) may not be true, if we end up choosing to run the same vCPU again 2) even if true, they'll always fail on the trylock, unless until we're out of csched_schedule(), as it holds the runqueue lock itself. So, yeah, it's not really a matter of correctness, but there's more overhead cut. In v2 of this series, that I'm about to send, I've "fixed" this (i.e., I'm only modifying the counter when really necessary). > As a first opinion of this patch, you are adding another cpumask > which is going to play hot cacheline pingpong. >=20 Yeah, well, despite liking the cpumask based approach, I agree it's overkill in this case. In v2, I got rid of it, and I am doing something even more similar to Malcolm's patch above. Thanks and Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-s2q0A5r8GOz2Wvkb2bpP Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJY5fBXAAoJEBZCeImluHPuCAEQAKE4DzE+EuDXVyM/wGU3y3QG nq9Np394mMxShQ/jTwGuAWO4PxdjAQibLmUyrKL5LUNTn3zp1aTyJ/OfUZlg80Az An0zrLLxMzfRgdgdG+aXi5quxXWVe8JGsK3kyKbx10LtK1UhjK+c0T47a+TP14zU acLu/7dhTYGlfRy1X2ZmBk54Mhj8ct3BV45DqZTLUBGayAUmQv4Z2U7Wh+xmiO4+ 2wSAQQlBBwN4z/qavsKvKwMwLjeDtthLd3635nOmQ2URjTWRHYnMgjFMFVb90SQv SwNyyH2BF9sE86JnaDB2rNIoN9U7eSW8r6FVgnmKDYWuO1iQDwut4fdWYseplVeL m22qHAH5G2wHbSQkulPZgA0Nt7I5FWnJXIsbAAMCcTbwkP755Iy6HhOH77RRzID3 eP5Pv6qpQqYm4I/kXNRxZXzwCpWKFTq6zFyzYDwTk9HIoyiHSdOho7wlvanERpzA gA+yTHMZMDjX/SDdrrC5EJWIwkAVfYDTos9kag+L783TInQAFYh4JFdnLcv4mRyL jQMbIEqaz5funSrOSnCbtQah58G6I66SIMlhD8+nOAhXI3GMzqggs8dAVVVYnLxo +mUG+d1xZr2rfgaAlLfMWGqJhWYR7/baSNKSEd4/EmuhIroaj+Y62j2y/zOft3Sp 6sr3fUB5AoIagULeCOaj =MKuO -----END PGP SIGNATURE----- --=-s2q0A5r8GOz2Wvkb2bpP-- --===============2001359895971407900== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============2001359895971407900==--