From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Xen on ARM IRQ latency and scheduler overhead Date: Fri, 17 Feb 2017 19:40:45 +0100 Message-ID: <1487356845.6732.100.camel@citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1205066604057535936==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Stefano Stabellini , xen-devel@lists.xen.org Cc: george.dunlap@eu.citrix.com, edgar.iglesias@xilinx.com, julien.grall@arm.com List-Id: xen-devel@lists.xenproject.org --===============1205066604057535936== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-zXTZpmTNUA6OktQdDQLj" --=-zXTZpmTNUA6OktQdDQLj Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Thu, 2017-02-09 at 16:54 -0800, Stefano Stabellini wrote: > These are the results, in nanosec: >=20 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 AVG=C2= =A0=C2=A0=C2=A0=C2=A0 MIN=C2=A0=C2=A0=C2=A0=C2=A0 MAX=C2=A0=C2=A0=C2=A0=C2= =A0 WARM MAX >=20 > NODEBUG no WFI=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 1890= =C2=A0=C2=A0=C2=A0 1800=C2=A0=C2=A0=C2=A0 3170=C2=A0=C2=A0=C2=A0 2070 > NODEBUG WFI=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 4850=C2=A0=C2=A0=C2=A0 4810=C2=A0=C2=A0=C2=A0 7030=C2=A0=C2=A0= =C2=A0 4980 > NODEBUG no WFI credit2=C2=A0 2217=C2=A0=C2=A0=C2=A0 2090=C2=A0=C2=A0=C2= =A0 3420=C2=A0=C2=A0=C2=A0 2650 > NODEBUG WFI credit2=C2=A0=C2=A0=C2=A0=C2=A0 8080=C2=A0=C2=A0=C2=A0 7890= =C2=A0=C2=A0=C2=A0 10320=C2=A0=C2=A0 8300 >=20 > DEBUG no WFI=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 2252=C2=A0=C2=A0=C2=A0 2080=C2=A0=C2=A0=C2=A0 3320=C2=A0=C2=A0=C2=A0= 2650 > DEBUG WFI=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 6500=C2=A0=C2=A0=C2=A0 6140=C2=A0=C2=A0=C2=A0 8520=C2= =A0=C2=A0=C2=A0 8130 > DEBUG WFI, credit2=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 8050=C2=A0=C2=A0=C2=A0 7= 870=C2=A0=C2=A0=C2=A0 10680=C2=A0=C2=A0 8450 >=20 > As you can see, depending on whether the guest issues a WFI or not > while > waiting for interrupts, the results change significantly. > Interestingly, > credit2 does worse than credit1 in this area. >=20 I did some measuring myself, on x86, with different tools. So, cyclictest is basically something very very similar to the app Stefano's app. I've run it both within Dom0, and inside a guest. I also run a Xen build (in this case, only inside of the guest). > We are down to 2000-3000ns. Then, I started investigating the > scheduler. > I measured how long it takes to run "vcpu_unblock": 1050ns, which is > significant. I don't know what is causing the remaining 1000-2000ns, > but > I bet on another scheduler function. Do you have any suggestions on > which one? >=20 So, vcpu_unblock() calls vcpu_wake(), which then invokes the scheduler's wakeup related functions. If you time vcpu_unblock(), from beginning to end of the function, you actually capture quite a few things. E.g., the scheduler lock is taken inside vcpu_wake(), so you're basically including time spent waited on the lock in the estimation. That is probably ok (as in, lock contention definitely is something relevant to latency), but it is expected for things to be rather different between Credit1 and Credit2. I've, OTOH, tried to time, SCHED_OP(wake) and SCHED_OP(do_schedule), and here's the result. Numbers are in cycles (I've used RDTSC) and, for making sure to obtain consistent and comparable numbers, I've set the frequency scaling governor to performance. Dom0, [performance] =09 cyclictest 1us cyclictest 1ms cyclictest 100ms =09 (cycles) Credit1 Credit2 Credit1 Credit2 Credit1 Credit2 =09 wakeup-avg 2429 2035 1980 1633 2535 1979 =09 wakeup-max 14577 113682 15153 203136 12285 115164 =09 sched-avg 1716 1860 2527 1651 2286 1670 =09 sched-max 16059 15000 12297 101760 15831 13122 =09 =09 VM, [performance] =09 cyclictest 1us cyclictest 1ms cyclictest 100ms make -j xen=09 (cycles) Credit1 Credit2 Credit1 Credit2 Credit1 Credit2 Credit1 Credit2 wakeup-avg 2213 2128 1944 2342 2374 2213 2429 1618 wakeup-max 9990 10104 11262 9927 10290 10218 14430 15108 sched-avg 2437 2472 1620 1594 2498 1759 2449 1809 sched-max 14100 14634 10071 9984 10878 8748 16476 14220 Actually, TSC on this box should be stable and invariant, so I guess I can try with the default governor. Will do that on Monday. Does ARM have frequency scaling (I did remember something on xen-devel, but I am not sure whether it landed upstream)? But anyway. You're seeing big differences between Credit1 and Credit2, while I, at least as far as the actual schedulers' code is concerned, don't. Credit2 shows higher wakeup-max values, but only in cases where the workload runs in dom0. But it also shows better (lower) averages, in both the two kind of workload considered and both in the dom0 and VM case. I therefore wonder what is actually responsible for the huge differences between the two scheduler that you are seeing... could be lock contention, but with only 4 pCPUs and 2 active vCPUs, I honestly doubt it... Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-zXTZpmTNUA6OktQdDQLj Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJYp0OtAAoJEBZCeImluHPujw8QAJJtDU5uzzZNyuQUlZeSMSz1 TVPhzWHO4L/b+jYCuVffRIikpZjrCJct2hfOyWXLmgS7ACz4tEe1pmK+EIjxXJgo TKcD5ZSGTEMKAejovi9CQG2XtOLASEgzAu2RT3gpyxJgyWkIF+thN8G9mIR2zOkD Xc2IHQFdjol6f0d+aKxxY4LwE+jZzrsLEQIwtbRgaDVAvu0OsTpExmrGsJ56mZcQ X468wVyxaTvH0qSXK3j2P4IFI0HZn6EQKCz1fw8HLpLEXrSjYXLqMB+oLlRIe0ds KcWDLiHzAlYfPnx5cyPq4rdSdV3a0XB0uEj2j4tDAo9b801LB/Th12dKD7iEQBDq 5XL9tBNhVoeH65aoPMrzD9mFKnNdSFieAMvMYPUa6By2pSRlEJv3YVfjm/IDTTy6 nFe9YmChDvAh0pMeSIcwSWRWTAe9Nm6+rx0iqrXTfaA1det8VVQgXxj8U3RijeRh pD/RIaR6wMgIbRJ8LDqxK/ONYh88jCKNZ19Zt/UfVYrhlv1RrkedJXFRGjLHqg2t 7+vDlhQAAKLHj3JCwd2Y6T2AUWuMT9sYrcAANzrP2/gWDyLdl6kq5t5H1Bm+RYpV oSB4+o2PMv32pdE7WLnkXyZdWAiHXUsSQLCepjENVnIsnZUoPuKFQA08AQboo0DA 5DvdIiak/fH0B/FLOY3H =7Vc+ -----END PGP SIGNATURE----- --=-zXTZpmTNUA6OktQdDQLj-- --===============1205066604057535936== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============1205066604057535936==--