Re: Interrupt to CPU routing in HVM domains - again

[Steve Ofsthun <sofsthun@virtualiron.com>]

John Levon wrote:
> On Fri, Sep 05, 2008 at 01:11:41PM -0400, Steve Ofsthun wrote:
> 
>>>> #ifdef IRQ0_SPECIAL_ROUTING
>>>>   /* Force round-robin to pick VCPU 0 */
>>>>   if ( ((irq == hvm_isa_irq_to_gsi(0)) && pit_channel0_enabled()) ||
>>>>        is_hvm_callback_irq(vioapic, irq) )
>>>>       deliver_bitmask = (uint32_t)1;
>>>> #endif
>>> Yes, please - Solaris 10 PV drivers are buggy in that they use the
>>> current VCPUs vcpu_info. I just found this bug, and it's getting fixed,
>>> but if this makes sense anyway, it'd be good.
>> I can submit a patch for this, but we feel this is something of a hack.  
> 
> Yep.

OK, I'll throw a patch together.

>> We'd like to provide a more general mechanism for allowing event channel 
>> binding to "work" for HVM guests.  But to do this, we are trying to address 
>> conflicting goals.  Either we honor the event channel binding by 
>> circumventing the IOAPIC emulation, or we faithfully emulate the IOAPIC and 
>> circumvent the event channel binding.
> 
> Well, this doesn't really make sense anyway as is: the IRQ binding has little
> to do with where the evtchns are handled (I don't think there's any
> requirement that they both happen on the same CPU).

Yes, there is no requirement, but there is a significant latency penalty for redirecting an event channel interrupt through an IRQ routed to a different VCPU.  Think 10s of milliseconds delay minimum due to Xen scheduling on a busy node (the current VCPU will not yield unless it is idle).  Add to this the fact that almost any significant I/O load on an HVM Windows guest becomes cpu bound quickly (so your scheduling priority is reduced).

>> Our driver writers would like to see support for multiple callback IRQs.  
>> Then particular event channel interrupts could be bound to particular IRQs. 
>> This would allow PV device interrupts to be distributed intelligently.  It 
>> would also allow net and block interrupts to be disentangled for Windows PV 
>> drivers.
> 
> You could do a bunch of that just by distributing them from the single
> callback IRQ. But I suppose it would be nice to move to a
> one-IRQ-per-evtchn model. You'd need to keep the ABI of course, so you'd
> need a feature flag or something.

Distributing from a single IRQ works OK for Linux, but doesn't work very well for older versions of Windows.  For block handling you want to deliver the real interrupts in SCSI miniport context.  The network can deal with the interrupt redirect.  But the network easily generates the highest interrupt rates and is sensitive to additional latency.  So you end up slowing SCSI down with "extra" network interrupts, and slowing the network down with increased interrupt latency.  Delivering net and block interrupts independently would avoid these issues.  Delivering interrupts to a bus driver and forwarding these to virtual device drivers directly is only an option on newer versions of Windows.

>> We deal pretty much exclusively with HVM guests, do SMP PV environments 
>> selectively bind device interrupts to different VCPUs?
> 
> For true PV you can bind evtchns at will.
> 
> regards
> john