AMD IOMMU intremap tables and IOAPICs

AMD IOMMU intremap tables and IOAPICs

[George Dunlap]

Wei,

Quick question:  Am I reading the code correctly, that even with
per-device interrupt remap tables, that GSIs are accounted to the
intremap table of the corresponding IOAPIC, presumably because the
IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
that case, then we need all devices sharing the same IOAPIC must not
have any vector collisions.  Is that correct?

 -George

Re: AMD IOMMU intremap tables and IOAPICs

[Andrew Cooper]

On 06/09/11 16:47, George Dunlap wrote:
> Wei,
>
> Quick question:  Am I reading the code correctly, that even with
> per-device interrupt remap tables, that GSIs are accounted to the
> intremap table of the corresponding IOAPIC, presumably because the
> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
> that case, then we need all devices sharing the same IOAPIC must not
> have any vector collisions.  Is that correct?

Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
cant have any two IRQs across any IO-APICs sharing a vector,
irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
only takes account of vector and not destination)

If we were to disable the auto EOI broadcast and do manual EOI'ing (only
available on newer versions of the local apic) then we could reduce that
restriction to "no two IRQs in the same IO-APIC may share a vector".

~Andrew

>  -George
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: AMD IOMMU intremap tables and IOAPICs

[George Dunlap]

On Tue, Sep 6, 2011 at 4:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> On 06/09/11 16:47, George Dunlap wrote:
>> Wei,
>>
>> Quick question:  Am I reading the code correctly, that even with
>> per-device interrupt remap tables, that GSIs are accounted to the
>> intremap table of the corresponding IOAPIC, presumably because the
>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
>> that case, then we need all devices sharing the same IOAPIC must not
>> have any vector collisions.  Is that correct?
>
> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
> cant have any two IRQs across any IO-APICs sharing a vector,
> irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
> only takes account of vector and not destination)
>
> If we were to disable the auto EOI broadcast and do manual EOI'ing (only
> available on newer versions of the local apic) then we could reduce that
> restriction to "no two IRQs in the same IO-APIC may share a vector".

Hmm, so it sounds like enforcing non-sharing of vectors within a
single IOAPIC is something we probably want to do even when we're not
using an AMD IOMMU?

 -George

Re: AMD IOMMU intremap tables and IOAPICs

[Andrew Cooper]

On 06/09/11 16:57, George Dunlap wrote:
> On Tue, Sep 6, 2011 at 4:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> On 06/09/11 16:47, George Dunlap wrote:
>>> Wei,
>>>
>>> Quick question:  Am I reading the code correctly, that even with
>>> per-device interrupt remap tables, that GSIs are accounted to the
>>> intremap table of the corresponding IOAPIC, presumably because the
>>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
>>> that case, then we need all devices sharing the same IOAPIC must not
>>> have any vector collisions.  Is that correct?
>> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
>> cant have any two IRQs across any IO-APICs sharing a vector,
>> irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
>> only takes account of vector and not destination)
>>
>> If we were to disable the auto EOI broadcast and do manual EOI'ing (only
>> available on newer versions of the local apic) then we could reduce that
>> restriction to "no two IRQs in the same IO-APIC may share a vector".
> Hmm, so it sounds like enforcing non-sharing of vectors within a
> single IOAPIC is something we probably want to do even when we're not
> using an AMD IOMMU?
>
>  -George

Currently there is no code to disable auto EOI and do manual EOI
instead.  As a result, we should enforce non-sharing of vectors across
all IO-APICs.  It was on my irq cleanup list but seems to be a specific
problem for you at the moment.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: AMD IOMMU intremap tables and IOAPICs

[George Dunlap]

On Tue, 2011-09-06 at 17:03 +0100, Andrew Cooper wrote:
> On 06/09/11 16:57, George Dunlap wrote:
> > On Tue, Sep 6, 2011 at 4:54 PM, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> >> On 06/09/11 16:47, George Dunlap wrote:
> >>> Wei,
> >>>
> >>> Quick question:  Am I reading the code correctly, that even with
> >>> per-device interrupt remap tables, that GSIs are accounted to the
> >>> intremap table of the corresponding IOAPIC, presumably because the
> >>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
> >>> that case, then we need all devices sharing the same IOAPIC must not
> >>> have any vector collisions.  Is that correct?
> >> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
> >> cant have any two IRQs across any IO-APICs sharing a vector,
> >> irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
> >> only takes account of vector and not destination)
> >>
> >> If we were to disable the auto EOI broadcast and do manual EOI'ing (only
> >> available on newer versions of the local apic) then we could reduce that
> >> restriction to "no two IRQs in the same IO-APIC may share a vector".
> > Hmm, so it sounds like enforcing non-sharing of vectors within a
> > single IOAPIC is something we probably want to do even when we're not
> > using an AMD IOMMU?
> >
> >  -George
> 
> Currently there is no code to disable auto EOI and do manual EOI
> instead.  As a result, we should enforce non-sharing of vectors across
> all IO-APICs.  It was on my irq cleanup list but seems to be a specific
> problem for you at the moment.

My problem doesn't have anything to do with EOIs, but with the AMD IOMMU
interrupt remapping table, and a device driver which simply dies if it
misses any interrupts.  If we can kill two birds with one stone, all the
better.

 -George

Re: AMD IOMMU intremap tables and IOAPICs

[Tim Deegan]

At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote:
> On 06/09/11 16:47, George Dunlap wrote:
> > Wei,
> >
> > Quick question:  Am I reading the code correctly, that even with
> > per-device interrupt remap tables, that GSIs are accounted to the
> > intremap table of the corresponding IOAPIC, presumably because the
> > IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
> > that case, then we need all devices sharing the same IOAPIC must not
> > have any vector collisions.  Is that correct?
> 
> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
> cant have any two IRQs across any IO-APICs sharing a vector,
> irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
> only takes account of vector and not destination)

If this is the case, is there any point in having per-CPU IDTs?  
Or per-device remapping tables?

Tim.

Re: AMD IOMMU intremap tables and IOAPICs

[George Dunlap]

On Tue, 2011-09-06 at 17:06 +0100, Tim Deegan wrote:
> At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote:
> > On 06/09/11 16:47, George Dunlap wrote:
> > > Wei,
> > >
> > > Quick question:  Am I reading the code correctly, that even with
> > > per-device interrupt remap tables, that GSIs are accounted to the
> > > intremap table of the corresponding IOAPIC, presumably because the
> > > IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
> > > that case, then we need all devices sharing the same IOAPIC must not
> > > have any vector collisions.  Is that correct?
> > 
> > Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
> > cant have any two IRQs across any IO-APICs sharing a vector,
> > irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
> > only takes account of vector and not destination)
> 
> If this is the case, is there any point in having per-CPU IDTs?  
> Or per-device remapping tables?

Yes, for devices that use MSIs, at least.

 -George

Re: AMD IOMMU intremap tables and IOAPICs

[Keir Fraser]

On 06/09/2011 17:06, "Tim Deegan" <tim@xen.org> wrote:

> At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote:
>> On 06/09/11 16:47, George Dunlap wrote:
>>> Wei,
>>> 
>>> Quick question:  Am I reading the code correctly, that even with
>>> per-device interrupt remap tables, that GSIs are accounted to the
>>> intremap table of the corresponding IOAPIC, presumably because the
>>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
>>> that case, then we need all devices sharing the same IOAPIC must not
>>> have any vector collisions.  Is that correct?
>> 
>> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
>> cant have any two IRQs across any IO-APICs sharing a vector,
>> irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
>> only takes account of vector and not destination)
> 
> If this is the case, is there any point in having per-CPU IDTs?
> Or per-device remapping tables?

It still makes sense for MSIs, which are the most common interrupt type
these days.

 -- Keir

> Tim.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: AMD IOMMU intremap tables and IOAPICs

[Andrew Cooper]

On 06/09/11 17:06, Tim Deegan wrote:
> At 16:54 +0100 on 06 Sep (1315328056), Andrew Cooper wrote:
>> On 06/09/11 16:47, George Dunlap wrote:
>>> Wei,
>>>
>>> Quick question:  Am I reading the code correctly, that even with
>>> per-device interrupt remap tables, that GSIs are accounted to the
>>> intremap table of the corresponding IOAPIC, presumably because the
>>> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
>>> that case, then we need all devices sharing the same IOAPIC must not
>>> have any vector collisions.  Is that correct?
>> Based on the ICH10 IO-APIC documentation with respect to auto EOIs, we
>> cant have any two IRQs across any IO-APICs sharing a vector,
>> irrespective of IOMMU or not.  (Because the EOI'ing an IO-APIC entry
>> only takes account of vector and not destination)
> If this is the case, is there any point in having per-CPU IDTs?  
> Or per-device remapping tables?
>
> Tim.

My understanding is:

GSIs should always have unique vectors (due to the auto EOI broadcast)
AMD IOMMU with per-device remapping tables means that devices cant ever
share interrupts when migrating.
AMD IOMMU with a global remapping table negates the benefit of per-CPU IDTs

Therefore per-CPU IDTs are useful, but there is juggling required when
choosing a new vector on migrate (With the possibility of having free
vectors, but unable to assign any because of other restrictions)

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: AMD IOMMU intremap tables and IOAPICs

[Wei Wang2]

On Tuesday 06 September 2011 17:47:59 George Dunlap wrote:
> Wei,
>
> Quick question:  Am I reading the code correctly, that even with
> per-device interrupt remap tables, that GSIs are accounted to the
> intremap table of the corresponding IOAPIC, presumably because the
> IOMMU sees interrupts generated as GSIs as coming from the IOAPIC?  In
> that case, then we need all devices sharing the same IOAPIC must not
> have any vector collisions.  Is that correct?
>
>  -George

That is true.  All legacy devices have to send interrupts to IOMMU via IOAPIC. 
So even we use per-device table. All devices attached to the same IOAPIC will 
use the same interrupt table. There should be no vector collisions. 
Per-device table only makes sense for MSI devices or systems with multiple 
IOAPICs.

Thanks
Wei