From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Notes on stubdoms and latency on ARM Date: Fri, 7 Jul 2017 18:41:30 +0200 Message-ID: <1499445690.3620.8.camel@citrix.com> References: <8c63069d-c909-e82c-ecba-5451f822a5cc@citrix.com> <1497953518.7405.21.camel@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============4569927228358419152==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Volodymyr Babchuk Cc: Artem_Mygaiev@epam.com, xen-devel@lists.xensource.com, Andrii Anisov , George Dunlap , Julien Grall , Stefano Stabellini List-Id: xen-devel@lists.xenproject.org --===============4569927228358419152== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-BvoNjWhsylpa8lKMommL" --=-BvoNjWhsylpa8lKMommL Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Fri, 2017-07-07 at 18:02 +0300, Volodymyr Babchuk wrote: > Hello Dario, >=20 Hi! > On 20 June 2017 at 13:11, Dario Faggioli > wrote: > > On Mon, 2017-06-19 at 11:36 -0700, Volodymyr Babchuk wrote: > > >=20 > > > Thanks. Actually, we discussed this topic internally today. Main > > > concern today is not a SMCs and OP-TEE (I will be happy to do > > > this > > > right in XEN), but vcopros and GPU virtualization. Because of > > > legal > > > issues, we can't put this in XEN. And because of vcpu framework > > > nature > > > we will need multiple calls to vgpu driver per one vcpu context > > > switch. > > > I'm going to create worst case scenario, where multiple vcpu are > > > active and there are no free pcpu, to see how credit or credit2 > > > scheduler will call my stubdom. > > >=20 > >=20 > > Well, that would be interesting and useful, thanks for offering > > doing > > that. >=20 > Yeah, so I did that.=20 > Ok, great! Thanks for doing and reporting about this. :-D > And I have get some puzzling results. I don't know why, > but when I have 4 (or less) active vcpus on 4 pcpus, my test=C2=A0=C2=A0t= akes > about 1 second to execute. > But if there are 5 (or mode) active vcpus on 4 pcpus, it executes > from > 80 to 110 seconds. >=20 I see. So, I've got just a handful of minutes right now, to only quickly look at the result and ask a couple of questions. Will think about this more in the coming days... > There will be the details, but first let me remind you my setup. > =C2=A0I'm testing on ARM64 machine with 4 Cortex A57 cores. I wrote > special test driver for linux, that calls SMC instruction 100 000 > times. > Also I hacked miniOS to act as monitor for DomU. This means that > XEN traps SMC invocation and asks MiniOS to handle this. > Ok. > So, every SMC is handled in this way: >=20 > DomU->XEN->MiniOS->XEN->DomU. >=20 Right. Nice work again. > Now, let's get back to results. >=20 > ** Case 1: > - Dom0 has 4 vcpus and is idle > - DomU has 4 vcpus and is idle > - Minios has 1 vcpu and is not idle, because it's scheduler does > not calls WFI. > I run test in DomU: >=20 > root@salvator-x-h3-xt:~# time -p cat /proc/smc_bench > Will call SMC 100000 time(s) > So, given what you said above, this means that the vCPU that is running this will frequently block (when calling SMC) and resume (when SMC is handled) quite frequently, right? Also, are you sure (e.g., because of how the Linux driver is done) that this always happen on one vCPU? > Done! > real 1.10 > user 0.00 > sys 1.10 > ** Case 2: > - Dom0 has 4 vcpus. They all are executing endless loop with sh > oneliner: > # while : ; do : ; done & > - DomU has 4 vcpus and is idle > - Minios has 1 vcpu and is not idle, because it's scheduler does not > calls WFI. > Ah, I see. This is unideal IMO. It's fine for this POC, of course, but I guess you've got plans to change this (if we decide to go the stubdom route)? > - In total there are 6 vcpus active >=20 > I run test in DomU: > real 113.08 > user 0.00 > sys 113.04 >=20 Ok, so there's contention for pCPUs. Dom0's vCPUs are CPU hogs, while, if my assumption above is correct, the "SMC vCPU" of the DomU is I/O bound, in the sense that it blocks on an operation --which turns out to be SMC call to MiniOS-- then resumes and block again almost immediately. Since you are using Credit, can you try to disable context switch rate limiting? Something like: # xl sched-credit -s -r 0 should work. This looks to me like one of those typical scenario where rate limiting is counterproductive. In fact, every time that your SMC vCPU is woken up, despite being boosted, it finds all the pCPUs busy, and it can't preempt any of the vCPUs that are running there, until rate limiting expires. That means it has to wait an interval of time that varies between 0 and 1ms. This happens 100000 times, and 1ms*100000 is 100 seconds... Which is roughly how the test takes, in the overcommitted case. > * Case 7 > - Dom0 has 4 vcpus and is idle. > - DomU has 4 vcpus. Two of them are executing endless loop with sh > oneliner: > # while : ; do : ; done & > - Minios have 1 vcpu and is not idle, because it's scheduler does not > calls WFI. > - *Minios is running on separate cpu pool with 1 pcpu*: > Name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0CPUs=C2=A0=C2=A0=C2=A0Sched=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0Active=C2=A0=C2=A0=C2=A0Domain count > Pool-0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A03=C2=A0=C2=A0=C2=A0=C2=A0credit=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A02 > minios=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A01=C2=A0=C2=A0=C2=A0=C2=A0credit=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0y=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A01 >=20 > I run test in DomU: > real 1.11 > user 0.00 > sys 1.10 >=20 > * Case 8 > - Dom0 has 4 vcpus and is idle. > - DomU has 4 vcpus. Three of them are executing endless loop with sh > oneliner: > # while : ; do : ; done & > - Minios have 1 vcpu and is not idle, because it's scheduler does not > calls WFI. > - Minios is running on separate cpu pool with 1 pcpu: >=20 > I run test in DomU: > real 100.12 > user 0.00 > sys 100.11 >=20 >=20 > As you can see, I tried to move minios to separate cpu pool. But it > didn't helped a lot. >=20 Yes, but it again makes sense. In fact, now there are 3 CPUs in Pool-0,=20 and all are kept always busy by the the 3 DomU vCPUs running endless loops. So, when the DomU's SMC vCPU wakes up, has again to wait for the rate limit to expire on one of them. > I expected that it would be 20% to 50% slower, when there are more > vCPUs than pCPUs. But it is 100 times slower and I can't explain > this. > Probably, something is very broken in my XEN. But I used 4.9 with > some > hacks to make minios work. I didn't touched scheduler at all. >=20 If you can, try with rate limiting off and let me know. :-D Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-BvoNjWhsylpa8lKMommL Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJZX7m7AAoJEBZCeImluHPuAl0QAN/iNIeIjW6/1E2WoB4LrGX+ 8azZum/AWwnREjxoKqoAOH9LWPiosfDf6Uy79VR8lm/rbd7Py0t01kYqjwzqWtq6 3TUhHcTuyjJk1vATm495vJzvD8ICYpr5rO+olHiYUMGVExrDTilMtn9cpjvctc5j 1U7Giv3kxjBPhtADjQ37hAzrOjbk8yTy62Ii43yIMc8PZFdnDxmKaZnl9vVuCS3u arBhACNHUiV85CvGP0h4lvhe1D/LE3iWQ10KNZsWRIiCcR5BHw9tbs6Y06lS2Pxd O5u7wHTrBXND6D2oSi98n6Xq2HIdnvIxrBFIIlYAsKzHcqD/Ue2aGZV+KgasPRXC SiYnn8psgXbaPrNwr0DayGBPyFv43OioPeeZfrp2kjRA97raQfN/59JoVjZII8+7 X1MwB5WlJ8TmHDCmLNz35xAu2sVOiX7Fuh8JZS8zlZvnuz/xn+GsHx3fTW3DrXW4 zHZeYd6ZgQIWKkXi3x97MPKdUrVVGdwozyIjp9Aqk3iq/U0HfLAY7M1Wnw8op+Uw 6IZm99k44rJjuih/fksiah9Cq2ePmLvJjgrFD0od5635cBS7I/UVaQvwzA+/MV7p HXjbtXiF4K0l/W7WINagHaoxd8tT1QKAj7g1FMp+OnaBoX8YiHV7p3g4ZcWKFyWr ZQ46vUwchqhyqyoQ7H1U =TwrZ -----END PGP SIGNATURE----- --=-BvoNjWhsylpa8lKMommL-- --===============4569927228358419152== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============4569927228358419152==--