From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dario Faggioli Subject: Re: Enabling VT-d PI by default Date: Tue, 16 May 2017 13:52:38 +0200 Message-ID: <1494935558.7393.37.camel@citrix.com> References: <20170411005929.GA34726@skl-2s3.sh.intel.com> <58ECAE13020000780014FB6C@prv-mh.provo.novell.com> <20170416201354.GA23350@skl-2s3.sh.intel.com> <671e935a-f832-13fe-016d-674f837b2a4e@citrix.com> <5901B5010200007800154A27@prv-mh.provo.novell.com> <794aac62-4d0c-f13c-916a-b219fcba852e@citrix.com> <95d77405-3f01-4d50-ddd7-e2cfd35ddc63@citrix.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============6964217624020623702==" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: George Dunlap , Andrew Cooper Cc: Kevin Tian , "xen-devel@lists.xen.org" , Jan Beulich , Chao Gao List-Id: xen-devel@lists.xenproject.org --===============6964217624020623702== Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="=-rI0wAnFxJZ5BXBMqgkUF" --=-rI0wAnFxJZ5BXBMqgkUF Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2017-05-15 at 15:32 +0100, George Dunlap wrote: > On Mon, May 15, 2017 at 2:35 PM, Andrew Cooper > wrote: > > On systems with this number of in-flight interrupts, trying to > > track > > "who got what interrupt" for priority > > boosting purposes is a waste of > > time, as we spend ages taking vmexits to process interrupt > > notifications > > for out-of-context vcpus. > >=20 > > The way the PI architects envisaged the technology being used is > > that > > Suppress Notification is set at all points other than executing in > > non-root mode for the vcpu in question (there is a small race > > window > > around clearing SN on vmentry), and that the scheduler uses > > Outstanding > > Notification on each of the PI blocks when it rebalances credit to > > see > > which vcpus have had interrupts in the last 30ms. >=20 > It sounds like they may have made the mistake that the Credit1 > designers made, in analyzing only a system that was overloaded; and > one where all workloads were identical, as opposed to analyzing a > system that was at least sometimes partially loaded, and where > workloads were very different. >=20 Totally agree. Also, I'm not sure I follow why PI architects would be basing hardware design on specific characteristics of a particular Xen scheduler. E.g., in Linux --which I'd think they also had in mind when envisioning uses of the technology-- there is no such thing as 30ms timeslice, nor credits redistribution. And AFAICU what you seem to suggest, not notifying an interrupt/not waking up anyone, at the time at which it happens, means there must be some kind of list_for_each_vcpu() anyway, for checking which vCPUs have pending notifications. Hence the problem we're discussing here, would just be moved between subsystems, rather than going away. And, finally, I don't get what you mean when you say that we're trying to use PI "for priority boosting purposes". I don't think we do that. FTR, I've quickly checked how this is done in Linux, and the solution pushed there looks really similar to the one that has been pushed to Xen as well. E.g., the also there, the handler scans the blocked vCPUs list: http://elixir.free-electrons.com/linux/latest/source/arch/x86/kvm/vmx.c#L64= 64 > In both cases, waiting 30ms to see if we should wake somebody up is > far too long. >=20 Absoluely! Regards, Dario --=20 <> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) --=-rI0wAnFxJZ5BXBMqgkUF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCAAGBQJZGugHAAoJEBZCeImluHPueo4QAOlCDp5+rSuKw5pDvgDiWVGg Mlc35+jiNXDPplP73ExIQo5AFWC+M0j4UehX1cP8oBddu0O3gq3Dd69dE556JfPJ U0J/SYClZT4W3EeobClEnw4nDiurF0+yKUHGQ0S28ImRve1QmM3Mj4+jkMA3XP0a 46hSiijqqnmQAL11qGpRdY+ktKVdczmmHsHFOaUofaeIB1XJH9FZhY4HJYmme8Tm j3rhsz599ojvwA6AT1kNKg1uwaeXVyfSBAW02itUSiloXsXNvI05dFQbIuR9CGrb CUdVVxpbEh3vX2E4lcGvWuuyYhf1m3//8ICRrabFIw4K3nJeqDWmy8gQDwrCUfb6 Tun9Pqyee5y/sh+peYC/p/E8UDAAqx9AynSlf5SvLDC+c/lLRuLqHxuraXydW4nq sTnsSd9OGnfakkTF8/BRQ7HXovqBrMRsIOWueCFkv8VAFXVzMKWYtHUR3EmABNRt bHoR7zIrIvlLJ7ojHzncVnBjiH8Kz3ZAoakSprHbWHlOc9Vl5xKklZy8euKv6bXS seDXqRN6KFHw861s9+KVd9XiGE7TXikZaMmrxNw/NNIiSvo5vDnoMTTG1uiqjX5V 0HzfkcD1SlgmhWmRQeGmSwptHjMQKi7uiQsMjFw3WAUD/8ZT5UqO2Vg6YSFeWhh6 NTFLG89nsn7ExLt7tazG =3bYN -----END PGP SIGNATURE----- --=-rI0wAnFxJZ5BXBMqgkUF-- --===============6964217624020623702== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --===============6964217624020623702==--