Re: VT-d Posted-interrupt (PI) design for XEN

[Andrew Cooper <andrew.cooper3@citrix.com>]

On 09/03/15 10:33, Tim Deegan wrote:
> At 02:03 +0000 on 09 Mar (1425863009), Wu, Feng wrote:
>>
>>> -----Original Message-----
>>> From: Tim Deegan [mailto:tim@xen.org]
>>> Sent: Friday, March 06, 2015 5:44 PM
>>> To: Wu, Feng
>>> Cc: Jan Beulich; Zhang, Yang Z; Tian, Kevin; xen-devel@lists.xen.org
>>> Subject: Re: [Xen-devel] VT-d Posted-interrupt (PI) design for XEN
>>>
>>> At 02:07 +0000 on 06 Mar (1425604054), Wu, Feng wrote:
>>>>> From: Tim Deegan [mailto:tim@xen.org]
>>>>> But I don't understand why we would need a new global vector for
>>>>> RUNSTATE_blocked rather than suppressing the posted interrupts as you
>>>>> suggest for RUNSTATE_runnable.  (Or conversely why not use the new
>>>>> global vector for RUNSTATE_runnable too?)
>>>> If we suppress the posted-interrupts when vCPU is blocked, it cannot
>>>> be unblocked by the external interrupts, this is not correct.
>>> OK, I don't understand at all now. :)  When the posted interrupt is
>>> suppressed, what happens to the interrupt? 
>> When the posted interrupt is suppressed, VT-d engine will not issue
>> notification events.
>>
>>> If it's just dropped, then we can't use that for _any_ cases. 
>> We can suppress the posted-interrupt when vCPU is waiting in the runqueue
>> (vCPU is in RUNSTATE_runnable state), it is not needed to send notification
>> event when vCPU is in this state, since when interrupt happens, the interrupt
>> information are not _dropped_, instead, they are stored in PIR, and this will
>> be synced to vIRR before VM-Entry.
> So you think you can use the same system for RUNSTATE_runnable as
> RUNSTATE_blocked?  That seems like a good idea. 
>
> I'll leave the details (e.g. single global vector + queue vs any other
> way to wake the vcpu) to people who know the x86 irq code better than
> I do. :)

>From my reading the relevant section in the VT-d spec, to the best of my
understanding:

We only need the second vector if Xen wishes to be informed that an
interrupt has been queued for a vcpu.  The spec suggests that, for one
usecase, this information should affect scheduling decisions.

If we do not wish to make scheduling alterations based on interrupt
delivery, the extra vector can be ignored.

If we do wish to make scheduling alterations, we will need to be able to
uniquely identify a vcpu from a vector, which will involve allocating
one vector per vcpu.


If my understanding is correct, I would suggest that Xen opt for not
getting notifications.  Interrupting one guest to indicate that another
vcpu has been interrupted scales progressively worse with the number of
running VMs, and there are existing usecases which have already
exhausted the x86 vector space completely.

It might be sensible to have the option available as a per-domain opt-in
option.  A usecase such as device driver domain could easily want to
deal with its interrupts ahead of running the domains it is servicing.

~Andrew