Re: [Xenhackthon] Virtualized APIC registers - virtual interrupt delivery.

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, May 28, 2013 at 10:51:21AM +0000, Zhang, Yang Z wrote:
> Stefano Stabellini wrote on 2013-05-27:
> > On Mon, 27 May 2013, Zhang, Yang Z wrote:
> >> Konrad Rzeszutek Wilk wrote on 2013-05-24:
> >>> On Thu, May 23, 2013 at 08:25:06AM +0000, Zhang, Yang Z wrote:
> >>>> Jan Beulich wrote on 2013-05-23:
> >>>>>>>> On 22.05.13 at 18:21, Konrad Rzeszutek Wilk
> > <konrad.wilk@oracle.com>
> >>>>> wrote:
> >>>>>> Which means that if this is set to be higher than the hypervisor
> >>>>>> timer or IPI callback the guest can run unbounded. Also it would
> >>>>>> seem that this value has to be often reset when migrating a guest
> >>>>>> between the pCPUs. And it would appear that this value is static.
> >>>>>> Meaning the guest only sets these vectors once and the hypervisor
> >>>>>> is responsible for managing the priority of that guest and other
> >>>>>> guests (say dom0) on the CPU.
> >>>>>> 
> >>>>>> For example, we have a guest with a 10gB NIC and the guest has
> >>>>>> decided to use vector 0x80 for it (assume a UP guest). Dom0 has an
> >>>>>> SAS controller and is using event number 30, 31, 32, and 33 (there
> >>>>>> are only 4 PCPUS). The hypervisor maps them to be 0x58, 0x68, 0x78
> >>>>>> and 0x88 and spreads those vectors on each pCPU. The guest is running
> >>>>>> on pCPU1 and there are two vectors - 0x80 and 0x58. The one assigned
> >>>>>> to the guest wins and dom0 SAS controller is preempted.
> >>>>>> 
> >>>>>> The solution for that seems to have some interaction with the
> >>>>>> guest when it allocates the vectors so that they are always below
> >>>>>> the dom0 priority vectors. Or hypervisor has to dynamically shuffle its
> >>>>>> own vectors to be higher priority.
> >>>>>> 
> >>>>>> Or is there an guest vector <-> hypervisor vector lookup table that
> >>>>>> the CPU can use? So the hypervisor can say: the vector 0x80 in the
> >>>>>> guest actually maps to vector 0x48 in the hypervisor?
> >>>>> 
> >>>>> It is my understanding that the vector spaces are separate, and
> >>>>> hence guest interrupts can't block host ones (like the timer). Iirc
> >>>> Right. virtual interrupt delivery only for delivering guest virtual
> > interrupt(from
> >>> emulation device and assigned device.) which is located in guest's
> >>> vector space. It has nothing to do with other guest.
> > 
> > I think you mean "It has nothing to do with _the hypervisor_"?
> Yes. Both hypervisor and guest have separated vector space.
> 
> > 
> >>> OK, in which case Linux ~v2.6.32 (when the event callback mechanism was
> >>> introduced for HVM guests) will _not_ take advantage of this, right?
> >> Yes, event mechanism cannot benefit from it.
> > 
> > I think that Konrad was referring to the vector callback mechanism:
> You are right. What I want to say is vector callback mechanism.
> 
> > 
> > linux side  drivers/xen/events.c:xen_callback_vector
> > xen side    xen/arch/x86/hvm/irq.c:hvm_set_callback_via
> > 
> > Also see:
> > 
> > commit e5fd1f6505c43440bc2450253c79c80174b693bc
> > Author: Keir Fraser <keir.fraser@citrix.com>
> > Date:   Tue May 25 11:28:58 2010 +0100
> > 
> >     x86 hvm: implement vector callback for evtchn delivery
> >     
> >     Signed-off-by: Sheng Yang <sheng@linux.intel.com>
> >     Signed-off-by: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> >     Signed-off-by: Keir Fraser <keir.fraser@citrix.com>
> > 
> > From the guest point of view it looks like a normal vector callback
> > (similar to an IPI).
> >
> > 
> >>> Is there a way to solve this so that they _will_ take advantage of this.
> >> Perhaps not. virtual interrupt delivery relies on EOI logic to inject the pending
> > interrupt. But event channel doesn't have such mechanism.
> > 
> > It's true that we don't do any EOIs with the vector callback mechanism,
> > the same way the operating system doesn't do any EOIs when it receives
> > an IPI.
> IPI also need EOI.

Can we fix this to use the HVM (which would use the Virtual APIC and
EOI) mechanism for:
 - passthrough devices 
 - IPIs

And for everything else use the vector callback mechanism?

Which baremetal mechanism is needed? The x2APIC one?

> 
> > Can IPIs take advantage of virtual interrupt delivery?

>From my reading - yes.  Yang, that is correct right?
> 
> Best regards,
> Yang
> 
>