Re: [PATCH] VT-d: improve RMRR validity checking

[Weidong Han <weidong.han@intel.com>]

Alex Williamson wrote:
> On Wed, 2010-03-10 at 12:25 +0800, Weidong Han wrote:
>   
>> Alex Williamson wrote:
>>     
>>> On Wed, 2010-03-10 at 11:28 +0800, Weidong Han wrote:
>>>   
>>>       
>>>> Alex Williamson wrote:
>>>>     
>>>>         
>>>>> On Wed, 2010-03-10 at 10:40 +0800, Weidong Han wrote:
>>>>>   
>>>>>       
>>>>>           
>>>>>> Alex Williamson wrote:
>>>>>>     
>>>>>>         
>>>>>>             
>>>>>>> I have a system with what I consider to be a valid DRHD that's getting
>>>>>>> tripped up on this patch.  The problem is that the DRHD includes an
>>>>>>> IOAPIC scope, where the IOAPIC is not materialized on the PCI bus.  I
>>>>>>> think Xen is being overzealous in it's validity checking and that this
>>>>>>> is a valid configuration.  What do others think?  Are IOAPICs a
>>>>>>> special case that we can allow to be non-existent on the PCI bus?
>>>>>>>   
>>>>>>>       
>>>>>>>           
>>>>>>>               
>>>>>> Yes, IOAPIC can be not pci-discoverable. IOAPICs are only reported in 
>>>>>> the "Include_all" DRHD, and our patch won't check if the device is 
>>>>>> pci-discoverable or not for the "Include_all" DRHD. So I think the patch 
>>>>>> is no problem unless IOAPIC is not included in the "Include_all" DRHD. 
>>>>>> Can you post your boot logs?
>>>>>>     
>>>>>>         
>>>>>>             
>>>>> Weidong,
>>>>>
>>>>> That's a very subtle restriction, and I'm not sure how it works in
>>>>> practice.  If I have a multi-IOH system, each with VT-d hardware, each
>>>>> supporting interrupt remapping, each with one or more IOAPICs below
>>>>> them, how can interrupt remapping work if we can only associate an
>>>>> IOAPIC with the "include all" DRHD?  I'm confused.  Thanks,
>>>>>   
>>>>>       
>>>>>           
>>>> Each IOH will have one "include all" DRHD which reports IOAPICs for each 
>>>> IOH.
>>>>     
>>>>         
>>> Wouldn't that imply multiple PCI segments?  The configuration I'm
>>> looking at has multiple IOHs, all on the same PCI segment.  By my
>>> reading of the spec, we're only allowed to declare INCLUDE_PCI_ALL for
>>> one DRHD within the segment.  Am I incorrect?  Thanks,
>>>   
>>>       
>> Currently multiple PCI segments are not supported in Xen yet. So you 
>> encounter issue on multiple PCI segment system. We will support it after 
>> xen 4.0.
>>     
>
>
> Which is exactly why we have multiple IOHs on the *same* PCI segment on
> this system.  How is it possible to support multiple IOHs, all on the
> same PCI segment, each with VT-d hardware with interrupt remapping
> support, each with one or more IOAPICs below them given the current
> code?  We cannot list the IOAPICs only under the INCLUDE_PCI_ALL DRHD
> because that wouldn't provide the right information in the right place
> for interrupt remapping on the other DRHDs.  We cannot specify
> INCLUDE_PCI_ALL on all of the DRHDs because the spec indicates we can
> only have one INCLUDE_PCI_ALL DRHD per PCI segment (besides, we can't
> have PCI sub-hierarchy scopes specified on INCLUDE_PCI_ALL DRHDs, which
> means we'd have no way to associate PCI devices to a specific DRHD if
> they all set this flag).
>
> I'm inclined to believe the hardware actually works correctly if we
> associate an IOAPIC to a non-INCLUDE_PCI_ALL DRHD, but this validity
> checking code prevents Xen from even trying to use it.
>
> Alex
>   

This patch is no problem on our platform which has two IOHs, two 
IOAPICs. But there is only one DRHD, which is also INCLUDE_PCI_ALL DRHD. 
Can you post your Xen boot logs?

Regards,
Weidong