Re: Interrupt to CPU routing in HVM domains - again

[Steve Ofsthun <sofsthun@virtualiron.com>]

John Levon wrote:
> On Fri, Sep 05, 2008 at 11:15:43AM -0400, Steve Ofsthun wrote:
> 
>> While the event channel delivery code "binds" HVM event channel interrupts 
>> to VCPU0, the interrupt is delivered via the emulated IOAPIC.  The guest OS 
>> may program this "hardware" to deliver the interrupt to other VCPUs.  For 
>> linux, this gets done by the irqbalance code among others.  Xen overrides 
>> this routing for the timer 0 interrupt path in vioapic.c under the #define 
>> IRQ0_SPECIAL_ROUTING.  We hacked our version of Xen to piggyback on this 
>> code to force all event channel interrupts for HVM guests to also avoid any 
>> guest rerouting:
>>
>> #ifdef IRQ0_SPECIAL_ROUTING
>>    /* Force round-robin to pick VCPU 0 */
>>    if ( ((irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled()) ||
>>         is_hvm_callback_irq(vioapic, irq) )
>>        deliver_bitmask = (uint32_t)1;
>> #endif
> 
> Yes, please - Solaris 10 PV drivers are buggy in that they use the
> current VCPUs vcpu_info. I just found this bug, and it's getting fixed,
> but if this makes sense anyway, it'd be good.

I can submit a patch for this, but we feel this is something of a hack.  We'd like to provide a more general mechanism for allowing event channel binding to "work" for HVM guests.  But to do this, we are trying to address conflicting goals.  Either we honor the event channel binding by circumventing the IOAPIC emulation, or we faithfully emulate the IOAPIC and circumvent the event channel binding.

Our driver writers would like to see support for multiple callback IRQs.  Then particular event channel interrupts could be bound to particular IRQs.  This would allow PV device interrupts to be distributed intelligently.  It would also allow net and block interrupts to be disentangled for Windows PV drivers.

We deal pretty much exclusively with HVM guests, do SMP PV environments selectively bind device interrupts to different VCPUs?

Steve

>> This routing override provides a significant performance boost [or rather 
>> avoids the performance penalty] for SMP PV drivers up until the time that 
>> VCPU0 is saturated with interrupts.  You can probably achieve the same 
> 
> Of course there's no requirement that the evtchn is actually dealt with
> on the same CPU, just the callback IRQ and the evtchn "ack" (clearing
> evtchn_upcall_pending).
> 
> regards
> john