Re: [Patch v2] Make PCI extended config space (MMCONFIG) a driver opt-in

[Loic Prylli <loic@myri.com>]

On 12/28/2007 1:06 AM, Benjamin Herrenschmidt wrote:
> On Thu, 2007-12-27 at 21:37 -0800, Linus Torvalds wrote:
>   
>> On Fri, 28 Dec 2007, Benjamin Herrenschmidt wrote:
>>     
>>> I have embedded boards where proper CRS operations is critical since the
>>> kernel brings the PCIe link up itself, and thus is likely to hit devices
>>> still in the middle of CRS.
>>>       
>> .. but that's perfectly fine. A PCI-E bridge will certainly retry it in 
>> hardware (or it isn't a PCI-E bridge!).
>>     
>
> Only a handful of times in many bridges I've seen.
>   




Yes retry implementation are not universal. But bridges with
CRS-visibility might be even less common (both are
optional/implementation-specific features).


To make some choices for our own devices, we experimented at Myricom
with CRS on a few chipsets to see what they implement:

broadcom/serverworks 2000/1000: no-retry/ no-visibility
nvidia CK804/IO804: no-retry/no-visibility
nvidia MPC55/IO55: no-retry/no-visibility
intel-7520: does-retry/no-visibility
intel-975X:  does-retry/no-visibility
intel-5000series: does-retry/no-visibility
intel X38: does-retry/no-visibility
AMD/ATI RD790: no-retry/no-visibility


[ in all no-retry cases, when a CRS is somehow returned to the
pcie-root, the requester will get a 0xfffffffff, depending on chipsets,
other error bits (malformed-tlp, completion-timeout) might be triggered ]


>From the data reported on this mailing-list, it seems like the ATI tries
to implement CRS-visibility, but it is obviously buggy.


Not knowing whether there is any chipset with the visibility feature,
but without the retry capability, and given that CRS is irrelevant for
most Linux platforms (it only matters just after power-on, long before
Linux is started in the common case), it seems fair that
CRS-visibility-enabling should only be added in the specific code of the
specific embedded platforms (with no BIOS/firmware of any kind) that
might need it.



>  
>   
>> So I'm going to disable that thing. If there is some _other_ PCI-E bridge 
>> that is simply buggy, and cannot handle the hw retry itself or is just 
>> otherwise dodgy, we can have a white-list for cases where it really needs 
>> to be done, but the current code is just bogus.
>>     
>
> If you disable it, then isn't there also a problem with PCIE->PCI-X
> bridge which will stop issuing CRS when they should ? (not sure here, I
> may be a bit confused).
>   


This is mostly independant, allowing a PCIE->PCI-X bridge to generate
CRS is a different bit (bit 15 of pcie->devctl on the bridge). FWIW,
Linux does not seem to touch it, and it defaults to zero, so it does not
seem like most current PCIE->PCI-X bridge will never generates a CRS
(some BIOSes might do it, but not the couple of platforms I looked at).
Again the choice of setting here seems something better left to the
specific BIOS/embedded-code for a given platform.


Loic