From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: [Xenhackthon] Virtualized APIC registers - virtual interrupt delivery. Date: Tue, 28 May 2013 09:22:38 -0400 Message-ID: <20130528132238.GB724@phenom.dumpdata.com> References: <20130522162132.GA8859@phenom.dumpdata.com> <519DE63E02000078000D85C5@nat28.tlf.novell.com> <20130524143142.GK3900@phenom.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhang, Yang Z" Cc: "Nakajima, Jun" , Stefano Stabellini , xen-devel , "keir.xen@gmail.com" , Jan Beulich , "Zhang, Xiantao" List-Id: xen-devel@lists.xenproject.org On Tue, May 28, 2013 at 10:51:21AM +0000, Zhang, Yang Z wrote: > Stefano Stabellini wrote on 2013-05-27: > > On Mon, 27 May 2013, Zhang, Yang Z wrote: > >> Konrad Rzeszutek Wilk wrote on 2013-05-24: > >>> On Thu, May 23, 2013 at 08:25:06AM +0000, Zhang, Yang Z wrote: > >>>> Jan Beulich wrote on 2013-05-23: > >>>>>>>> On 22.05.13 at 18:21, Konrad Rzeszutek Wilk > > > >>>>> wrote: > >>>>>> Which means that if this is set to be higher than the hypervisor > >>>>>> timer or IPI callback the guest can run unbounded. Also it would > >>>>>> seem that this value has to be often reset when migrating a guest > >>>>>> between the pCPUs. And it would appear that this value is static. > >>>>>> Meaning the guest only sets these vectors once and the hypervisor > >>>>>> is responsible for managing the priority of that guest and other > >>>>>> guests (say dom0) on the CPU. > >>>>>> > >>>>>> For example, we have a guest with a 10gB NIC and the guest has > >>>>>> decided to use vector 0x80 for it (assume a UP guest). Dom0 has an > >>>>>> SAS controller and is using event number 30, 31, 32, and 33 (there > >>>>>> are only 4 PCPUS). The hypervisor maps them to be 0x58, 0x68, 0x78 > >>>>>> and 0x88 and spreads those vectors on each pCPU. The guest is running > >>>>>> on pCPU1 and there are two vectors - 0x80 and 0x58. The one assigned > >>>>>> to the guest wins and dom0 SAS controller is preempted. > >>>>>> > >>>>>> The solution for that seems to have some interaction with the > >>>>>> guest when it allocates the vectors so that they are always below > >>>>>> the dom0 priority vectors. Or hypervisor has to dynamically shuffle its > >>>>>> own vectors to be higher priority. > >>>>>> > >>>>>> Or is there an guest vector <-> hypervisor vector lookup table that > >>>>>> the CPU can use? So the hypervisor can say: the vector 0x80 in the > >>>>>> guest actually maps to vector 0x48 in the hypervisor? > >>>>> > >>>>> It is my understanding that the vector spaces are separate, and > >>>>> hence guest interrupts can't block host ones (like the timer). Iirc > >>>> Right. virtual interrupt delivery only for delivering guest virtual > > interrupt(from > >>> emulation device and assigned device.) which is located in guest's > >>> vector space. It has nothing to do with other guest. > > > > I think you mean "It has nothing to do with _the hypervisor_"? > Yes. Both hypervisor and guest have separated vector space. > > > > >>> OK, in which case Linux ~v2.6.32 (when the event callback mechanism was > >>> introduced for HVM guests) will _not_ take advantage of this, right? > >> Yes, event mechanism cannot benefit from it. > > > > I think that Konrad was referring to the vector callback mechanism: > You are right. What I want to say is vector callback mechanism. > > > > > linux side drivers/xen/events.c:xen_callback_vector > > xen side xen/arch/x86/hvm/irq.c:hvm_set_callback_via > > > > Also see: > > > > commit e5fd1f6505c43440bc2450253c79c80174b693bc > > Author: Keir Fraser > > Date: Tue May 25 11:28:58 2010 +0100 > > > > x86 hvm: implement vector callback for evtchn delivery > > > > Signed-off-by: Sheng Yang > > Signed-off-by: Stefano Stabellini > > Signed-off-by: Keir Fraser > > > > From the guest point of view it looks like a normal vector callback > > (similar to an IPI). > > > > > >>> Is there a way to solve this so that they _will_ take advantage of this. > >> Perhaps not. virtual interrupt delivery relies on EOI logic to inject the pending > > interrupt. But event channel doesn't have such mechanism. > > > > It's true that we don't do any EOIs with the vector callback mechanism, > > the same way the operating system doesn't do any EOIs when it receives > > an IPI. > IPI also need EOI. Can we fix this to use the HVM (which would use the Virtual APIC and EOI) mechanism for: - passthrough devices - IPIs And for everything else use the vector callback mechanism? Which baremetal mechanism is needed? The x2APIC one? > > > Can IPIs take advantage of virtual interrupt delivery? >>From my reading - yes. Yang, that is correct right? > > Best regards, > Yang > >