Re: [PATCH v2] VTD/Intremap: Disable Intremap on Chipset 5500/5520/X58 due to errata

[Malcolm Crossley <malcolm.crossley@citrix.com>]

On 13/03/13 08:48, Jan Beulich wrote:
>>>> On 12.03.13 at 22:55, "Zhang, Xiantao" <xiantao.zhang@intel.com> wrote:
>>>   
>>>>> And - do you really need to iterate over all buses on segment 0?
>>>>> The X58 data sheet says at the top of section 17.1: "All devices
>>>>> on the IOH reside on bus 0". I wonder whether you wouldn't
>>>>> instead need to do this over all segments, on each bus 0.
>>>> The chipsets do not support multi-segment systems, and we have a
>>>> multi-socket affected systems with multiple of these chipsets, with none
>>>> of the IOH's on bus 0.
>>> That's contrary to the spec then, and will need clarification.
>>> Don, Xiantao?
>> For x58,  it is true because it is a UP platform, but it is not true for
>> 5500/5520, it depends on how many IOHs are in the platforms, and also depends
>> on how it is configured by platform, so the spec may only say x58's IOH
>> resides in bus0, but not for applicable for all.
> In which case (considering that neither the most recent [and only]
> data sheet rev nor the spec update say so) the question is whether
> there's any restriction which buses to look at, or whether the loop
> really needs to go over all buses on all segments (the latter of
> which is a problem on its own, as Xen may not be able to access the
> config space of devices on segments other than 0, as the approval
> to use MMCFG may need to come from Dom0). For Andrew's
> statement above regarding these chipsets not supporting multi-
> segment setups I haven't found confirmation anywhere, and
> (considering the potential of extra gluing hardware) may be hard
> to be had.
Section 7.3.1 of the 5520-5500 datasheet describes how the PCI bus 
numbers are setup for each 55xx chipset.
The datasheet confusing refers to segments but it is refering to subsets 
of the 256 buses in segment 0.
This is confirmed by the Bus subset control registers LCFGBUS and 
GCFGBUS having no support for a segment ID
and only support for Bus ranges.

I have read the Intel 7500 series datasheet volume 2 section 4 and Intel 
5500/7500 chipset series datasheet volume 2
section 7 and come to the understanding that PCI segments are 
implemented at a processor level and not at the chipset level.

This means the processor needs an additional MMCONFIG address decoder in 
order know which address ranges go to which PCI segments.
The Intel 7500 series has that additional MMCONFIG address decoder 
capability (Intel 7500 series datasheet volume 2 section 4.2.4.3 )
but the Intel 5500/5600 series processors only has 1 MMCONFIG decoder 
(SAD_PCIEXBAR). It would be good for Intel to confirm that
Intel 5500/5600 series processors can't support multiple PCI segments 
though.

My impression is that the Intel 5500/5520/X58 series chipsets and Intel 
5500/5600 series processors are targeted at two socket maximum
and so don't support large scale features like x2apic or multiple PCI 
segments.

The Intel EX series processors like the 7500 series, E7-88xx series and 
Intel 7500 series chipset go to 8 sockets or more and so support x2apic
and multiple PCI segments, luckily the Intel 7500 series chipset does 
not suffer from this errata.

Regards

Malcolm