Re: [PROPOSAL] Event channel for SMP-VMs: per-vCPU or per-OS?

["Roger Pau Monné" <roger.pau@citrix.com>]

On 28/10/13 16:26, Luwei Cheng wrote:
> This following idea was first discussed with George Dunlap, David Vrabel 
> and Wei Liu in XenDevSummit13. Many thanks for their encouragement to 
> post this idea to the community for a wider discussion.
> 
> [Current Design]
> Each event channel is associated with only “one” notified vCPU: one-to-one.
> 
> [Problem]
> Some events are per-vCPU (such as local timer interrupts) while some others 
> are per-OS (such as I/O interrupts: network and disk). 
> For SMP-VMs, it is possible that when one vCPU is waiting in the scheduling 
> queue, another vCPU is running. So, if the I/O events can be dynamically 
> routed to the running vCPU, the events can be processed quickly, without 
> suffering from VM scheduling delays (tens of milliseconds). On the other 
> hand, no reschedule operations are introduced.
> 
> Though users can set IRQ affinity in the guest OS, the current 
> implementation forces to bind the IRQ to the first vCPU of the 
> affinity mask [events.c: set_affinity_irq].
> If the hypervisor delivers the event to a different vCPU, the event 
> will get lost because the guest OS has masked out this event in all 
> non-notified vCPUs [events.c: bind_evtchn_to_cpu].
> 
> [New Design]
> For per-OS event channel, add “vCPU affinity” support: one-to-many.
> The “affinity” should be consistent with the ‘/proc/irq/#/smp_affinity’
> of the 
> guest OS and users can change the mapping at runtime. But by default, 
> all vCPUs should be enabled to serve I/O.
> 
> When such flexibility is enabled, I/O balancing among vCPUs can be 
> offloaded to the hypervisor. “irqbalance” is designed for physical 
> SMP systems, not virtual SMP systems.
> 
> Any comments are welcome!

Hello,

Looks interesting, but if IO events can indeed fire on any CPU, isn't
this going to introduce locking (and contention) on the event channel
upcall handler in order to prevent two (or more) CPUs from firing the
same event concurrently?

Roger.