Re: [PATCH] VT-d: improve RMRR validity checking

[Alex Williamson <alex.williamson@hp.com>]

On Wed, 2010-03-10 at 12:25 +0800, Weidong Han wrote:
> Alex Williamson wrote:
> > On Wed, 2010-03-10 at 11:28 +0800, Weidong Han wrote:
> >   
> >> Alex Williamson wrote:
> >>     
> >>> On Wed, 2010-03-10 at 10:40 +0800, Weidong Han wrote:
> >>>   
> >>>       
> >>>> Alex Williamson wrote:
> >>>>     
> >>>>         
> >>>>> I have a system with what I consider to be a valid DRHD that's getting
> >>>>> tripped up on this patch.  The problem is that the DRHD includes an
> >>>>> IOAPIC scope, where the IOAPIC is not materialized on the PCI bus.  I
> >>>>> think Xen is being overzealous in it's validity checking and that this
> >>>>> is a valid configuration.  What do others think?  Are IOAPICs a
> >>>>> special case that we can allow to be non-existent on the PCI bus?
> >>>>>   
> >>>>>       
> >>>>>           
> >>>> Yes, IOAPIC can be not pci-discoverable. IOAPICs are only reported in 
> >>>> the "Include_all" DRHD, and our patch won't check if the device is 
> >>>> pci-discoverable or not for the "Include_all" DRHD. So I think the patch 
> >>>> is no problem unless IOAPIC is not included in the "Include_all" DRHD. 
> >>>> Can you post your boot logs?
> >>>>     
> >>>>         
> >>> Weidong,
> >>>
> >>> That's a very subtle restriction, and I'm not sure how it works in
> >>> practice.  If I have a multi-IOH system, each with VT-d hardware, each
> >>> supporting interrupt remapping, each with one or more IOAPICs below
> >>> them, how can interrupt remapping work if we can only associate an
> >>> IOAPIC with the "include all" DRHD?  I'm confused.  Thanks,
> >>>   
> >>>       
> >> Each IOH will have one "include all" DRHD which reports IOAPICs for each 
> >> IOH.
> >>     
> >
> > Wouldn't that imply multiple PCI segments?  The configuration I'm
> > looking at has multiple IOHs, all on the same PCI segment.  By my
> > reading of the spec, we're only allowed to declare INCLUDE_PCI_ALL for
> > one DRHD within the segment.  Am I incorrect?  Thanks,
> >   
> 
> Currently multiple PCI segments are not supported in Xen yet. So you 
> encounter issue on multiple PCI segment system. We will support it after 
> xen 4.0.


Which is exactly why we have multiple IOHs on the *same* PCI segment on
this system.  How is it possible to support multiple IOHs, all on the
same PCI segment, each with VT-d hardware with interrupt remapping
support, each with one or more IOAPICs below them given the current
code?  We cannot list the IOAPICs only under the INCLUDE_PCI_ALL DRHD
because that wouldn't provide the right information in the right place
for interrupt remapping on the other DRHDs.  We cannot specify
INCLUDE_PCI_ALL on all of the DRHDs because the spec indicates we can
only have one INCLUDE_PCI_ALL DRHD per PCI segment (besides, we can't
have PCI sub-hierarchy scopes specified on INCLUDE_PCI_ALL DRHDs, which
means we'd have no way to associate PCI devices to a specific DRHD if
they all set this flag).

I'm inclined to believe the hardware actually works correctly if we
associate an IOAPIC to a non-INCLUDE_PCI_ALL DRHD, but this validity
checking code prevents Xen from even trying to use it.

Alex