From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Notes on stubdoms and latency on ARM Date: Thu, 20 Jul 2017 11:10:57 +0200 Message-ID: <1500541857.20438.6.camel@citrix.com> References: <8c63069d-c909-e82c-ecba-5451f822a5cc@citrix.com> <1497953518.7405.21.camel@citrix.com> <1499445690.3620.8.camel@citrix.com> <1499840091.7756.12.camel@citrix.com> <3121c88c-fbda-a494-ce91-b06fa0fc10f3@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5576010704402204847==" Return-path: In-Reply-To: <3121c88c-fbda-a494-ce91-b06fa0fc10f3@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: George Dunlap , Julien Grall , Stefano Stabellini , Volodymyr Babchuk Cc: Artem_Mygaiev@epam.com, xen-devel@lists.xensource.com, Andrii Anisov List-Id: xen-devel@lists.xenproject.org --===============5576010704402204847== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-59sImAYE2FIGj7ovc7lY" --=-59sImAYE2FIGj7ovc7lY Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2017-07-17 at 12:28 +0100, George Dunlap wrote: > Most schedulers have one runqueue per logical cpu.=C2=A0=C2=A0Credit2 has= the > option of having one runqueue per logical cpu, one per core (i.e., > hyperthreads share a runqueue), one runqueue per socket (i.e., all > cores > on the same socket share a runqueue), or one socket across the whole > system.=C2=A0=C2=A0 > You mean "or one runqueue across the whole system", I guess? :-) > I *think* we made one socket per core the default a while back > to deal with multithreading, but I may not be remembering correctly. >=20 We've have per-core runqueue as default, to deal with hyperthreading for some time. Nowadays, handling hyperthreading is done independently by runqueue arrangement, and so the current default is one runqueue per-socket. > In any case, if you don't have threads, then reporting each logical > cpu as its own core is the right thing to do. >=20 Yep. > If you're mis-reporting sockets, then the scheduler will be unable to > take that into account.=C2=A0=C2=A0 > And if this means that each logical CPU is also reported as being its own socket, then you have one runqueue per logical CPU. > But that's not usually going to be a major > issue, mainly because the scheduler is not actually in a position to > determine, most of the time, which is the optimal configuration.=C2=A0=C2= =A0If > two > vcpus are communicating a lot, then the optimal configuration is to > put > them on different cores of the same socket (so they can share an L3 > cache); if two vcpus are computing independently, then the optimal > configuration is to put them on different sockets, so they can each > have > their own L3 cache.=C2=A0 > This is all very true. However, if two CPUs share one runqueue, vCPUs will seamlessly move between the two CPUs, without having to wait for the load balancing logic to kick in. This is a rather cheap way of achieving good fairness and load balancing, but is only effective if this movement is also cheap, which, e.g., is probably the case if the CPUs share some level of cache. So, figuring out what the best runqueue arrangement is, is rather hard to do automatically, as it depends both on the workload and on the hardware characteristics of the platform, but having at last some degree of runqueue sharing, among the CPUs that have some cache levels in common, would be, IMO, our best bet. And we do need topology information to try to do that. (We would also need, in Credit2 code, to take more into account cache and memory hierarchy information, rather than "just" CPU topology. We're already working, for instance, of changing CSCHED2_MIGRATE_RESIST from being constant, to vary depending on the amount of cache-sharing between two CPUs.) > All that to say: It shouldn't be a major issue if you are mis- > reporting > sockets. :-) >=20 Maybe yes, maybe not. It may actually be even better on some combination of platforms and workloads, indeed... but it also means that the Credit2 load balancer is being invoked a lot, which may be unideal. Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-59sImAYE2FIGj7ovc7lY Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJZcHOjAAoJEBZCeImluHPuhAcP/0X6Jg+MRZmcIymJrOts+WYp ZUDuBz9gDZS+yHUVyzxNBy74IvaK0FDmXaVLfyPWgEJEXxm4PmIuqNMJrQQIojVp zNSJsdmLIyrvCPutMeUrQ+97YTBt6KMupK1wHIVQI9sclK/s2kCPrrLVC6xgRW5N m692UqfPuG3ATik5Sghd+WaLiPMFrvDJH49pFynpidLJyQyXznq9beWsrgAqFE+K ZIBsTK+eM2JEzMaUswLWrJNUBqFLn+J8gueoBxv3gt9N6QqUKu6SfdJWwjI7eDz4 othyhMqQx8zHREUSt/mnz3+jRxzgp29+R3TUT90qFHTdwkMjFC/hG8fy+zJWMmSu zmhKTQf0w1//lSsTSCFpgZWOPzW6f6KbLnoIxPAmqhwMDSnI9FnV+hnDTURNkBkg kZnWHL54khgRAo7Ib2wXdEfp4To0Xo94st3uLX5l1RfCmB92jC0HcAW96UQtJapI rgccxv/rFJ4qywvCwEnJQUt3ls2EKFS4Ibq+19h8J7GYMqZRYjmQAbd9MEb0ME+4 EKZXEuoxevAl01C2HRndLZeX8cmSuLjrOXc6v5Gg8ZPcj4/s+rIb08exqXD+AIum 5Uqh9tFtKSNgLT0fTEG4FJV+61pMpaEke+zEYKijYxvrx2uJhZZr1SlqoNrUgdN1 hcbV+MHzOofXMWLSJBtA =dzKE -----END PGP SIGNATURE----- --=-59sImAYE2FIGj7ovc7lY-- --===============5576010704402204847== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============5576010704402204847==--