Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Robert Hancock]

Tony Camuso wrote:
> Greg KH wrote:
>>
>> Sure, I realize this, but it solves the problem in one way for broken
>> hardware, such that it at least allows it to work, right?  It also
>> provides a better incentive for the manufacturer to fix their bios,
>> which as you are on-site at HP, it would seem odd that they would just
>> not do that instead of trying to work around this in the kernel...
>>
>> thanks,
>>
>> greg k-h
> 
> I don't think that many OEMs have that much control over the BIOS in
> their "value lines".
> :)
> 
> And the MMCONFIG problem with enterprise systems and workstations, where
> we do control the BIOS (for the most part), is due to known bugs in
> certain versions of certain chipsets, HT1000, AMD8132, among them, not
> the BIOS.
> 
> Anyway, we are devising better ways to deal with these anomalies
> than blacklists and telling customers to use "pci=nommconf"
> 
> And we're bringing them to the community for discussion, improvement,
> and, we hope, acceptance.

First off, I would like to see confirmation from the horses's mouths 
here (namely AMD, ServerWorks/Broadcom, and whoever else) that there is 
no other way to get around this problem than disabling MMCONFIG for 
accesses behind those chips.

The case of the device built into the K8 northbridge that's unreachable 
by MMCONFIG kind of makes sense, since the northbridge is what's 
translating the MMCONFIG memory access into config accesses. It seems 
bizarre to me that a bridge chip could possibly have such a problem. The 
MMCONFIG access should get translated into a configuration space access 
in the northbridge and from that point on there's no difference between 
an MMCONFIG and type1 access.

Look at MSI for another example, we recently had a patch from NVIDIA 
posted to enable Hypertransport MSI mapping bits on some chipsets so 
that MSI would function, because the BIOS failed to set them up 
properly. Are we sure there's not a similar BIOS configuration issue 
that could ideally be fixed in the BIOS, or else fixed up in the kernel?

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Tony Camuso]

Robert Hancock wrote:

> First off, I would like to see confirmation from the horses's mouths 
> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there is 
> no other way to get around this problem than disabling MMCONFIG for 
> accesses behind those chips.
> 

I happen to have this one stored in my desktop.

 From AMD-8132TM HyperTransportTM
      PCI-X®2.0 Tunnel
       Revision Guide

http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/30801.pdf

79 AMD-8132TM Tunnel Lacks Extended Configuration
         Space Memory-Mapped I/O Base Address Register

Description

Current AMD processors do not natively support PCI-defined extended configuration space. A memory
mapped I/O base address register (MMIO BAR) is required in chipset devices to support extended
configuration space. The AMD-8132 does not have this MMIO BAR.
Potential Effect On System

The AMD-8132 is a PCI-X® Mode 2 capable device and requires the MMIO BAR to support extended
configuration space. Using a device which does have this MMIO BAR and an AMD-8132 on the same
HyperTransportTM link of the processor may cause firmware/software problems.

The base configuration space of the AMD-8132 and PCI(-X) devices attached to it are accessible using only
the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices attached to the AMD-8132 in the
extended configuration space are not accessible. The AMD-8132 has no registers in the extended
configuration space.

Suggested Workaround

It is strongly recommended that system designers do not connect the AMD-8132 and devices that use extended
configuration space MMIO BARs (ex: HyperTransport-to-PCI Express® bridges) to the same processor
HyperTransport link.

Fix Planned
No

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Tony Camuso]

> Robert Hancock wrote:
> 
> First off, I would like to see confirmation from the horses's mouths 
> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there 
> is no other way to get around this problem than disabling MMCONFIG for 
> accesses behind those chips.
>
> 

And here are the excerpts from that page of the spec which are salient
to the present discussion:

----------------------------------------------------------------------

The base configuration space of the AMD-8132 and PCI(-X) devices attached to it are accessible using only
the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices attached to the AMD-8132 in the
extended configuration space are not accessible. The AMD-8132 has no registers in the extended
configuration space.

Fix Planned
No

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Bhavana Nagendra]

Tony Camuso wrote:
>> Robert Hancock wrote:
>>
>> First off, I would like to see confirmation from the horses's mouths 
>> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there 
>> is no other way to get around this problem than disabling MMCONFIG 
>> for accesses behind those chips.
>>
>>
>
> And here are the excerpts from that page of the spec which are salient
> to the present discussion:
>
> ----------------------------------------------------------------------
>
> The base configuration space of the AMD-8132 and PCI(-X) devices 
> attached to it are accessible using only
> the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices 
> attached to the AMD-8132 in the
> extended configuration space are not accessible. The AMD-8132 has no 
> registers in the extended
> configuration space.
>
> Fix Planned
> No
>
>
Prarit, Tony et al, this is as official as it gets.   It's pretty 
clearly stated.

Bhavana

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Robert Hancock]

Tony Camuso wrote:
> Robert Hancock wrote:
> 
>> First off, I would like to see confirmation from the horses's mouths 
>> here (namely AMD, ServerWorks/Broadcom, and whoever else) that there 
>> is no other way to get around this problem than disabling MMCONFIG for 
>> accesses behind those chips.
>>
> 
> I happen to have this one stored in my desktop.
> 
>  From AMD-8132TM HyperTransportTM
>      PCI-X®2.0 Tunnel
>       Revision Guide
> 
> http://www.amd.com/us-en/assets/content_type/white_papers_and_tech_docs/30801.pdf 
> 
> 
> 79 AMD-8132TM Tunnel Lacks Extended Configuration
>         Space Memory-Mapped I/O Base Address Register
> 
> Description
> 
> Current AMD processors do not natively support PCI-defined extended 
> configuration space. A memory
> mapped I/O base address register (MMIO BAR) is required in chipset 
> devices to support extended
> configuration space. The AMD-8132 does not have this MMIO BAR.
> Potential Effect On System
> 
> The AMD-8132 is a PCI-X® Mode 2 capable device and requires the MMIO BAR 
> to support extended
> configuration space. Using a device which does have this MMIO BAR and an 
> AMD-8132 on the same
> HyperTransportTM link of the processor may cause firmware/software 
> problems.
> 
> The base configuration space of the AMD-8132 and PCI(-X) devices 
> attached to it are accessible using only
> the mechanism defined in PCI 2.3. Registers of PCI-X Mode 2 devices 
> attached to the AMD-8132 in the
> extended configuration space are not accessible. The AMD-8132 has no 
> registers in the extended
> configuration space.
> 
> Suggested Workaround
> 
> It is strongly recommended that system designers do not connect the 
> AMD-8132 and devices that use extended
> configuration space MMIO BARs (ex: HyperTransport-to-PCI Express® 
> bridges) to the same processor
> HyperTransport link.
> 
> Fix Planned
> No

That does sound fairly definitive. I have to wonder why certain system 
designers then didn't follow their strong recommendation..

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Tony Camuso]

Robert Hancock wrote:

> I have to wonder why certain system 
> designers then didn't follow their strong recommendation..
> 
I don't think I want to go there.

I used to be a hardware/firmware guy.
:D :D

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Loic Prylli]

On 12/20/2007 9:15 PM, Robert Hancock wrote:
>>
>> Suggested Workaround
>>
>> It is strongly recommended that system designers do not connect the
>> AMD-8132 and devices that use extended
>> configuration space MMIO BARs (ex: HyperTransport-to-PCI Express®
>> bridges) to the same processor
>> HyperTransport link.
>>
>> Fix Planned
>> No
>
> That does sound fairly definitive. I have to wonder why certain system
> designers then didn't follow their strong recommendation..



Just curious, do you know of any system where that recommendation was
not followed? On all motherboards where I have seen a AMD-8131 or a
AMD-8132, they were alone on their hypertransport link, and other
"northbridges" (more precisely hypertransport to pci-express or
pci-whatever, often nvidia) with a "MMCONFIG BAR" where on one of the
other available hypertransport links in the system.


Loic

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Tony Camuso]

Loic Prylli wrote:
> 
> Just curious, do you know of any system where that recommendation was
> not followed? On all motherboards where I have seen a AMD-8131 or a
> AMD-8132, they were alone on their hypertransport link, and other
> "northbridges" (more precisely hypertransport to pci-express or
> pci-whatever, often nvidia) with a "MMCONFIG BAR" where on one of the
> other available hypertransport links in the system.
> 
> 
> Loic
> 

Here is the PCI configuration of the HP DL585G2.

You can see two nVidia CK804 PCIE root ports at bus 0 and bus 0x40.
Each of them has an 8132 connected as a subordinate bridge.

[root@hp-dl585g2-01 ~]# lspci -vt
-+-[0000:40]-+-00.0  nVidia Corporation CK804 Memory Controller
  |           +-01.0  nVidia Corporation CK804 Memory Controller
  |           +-0b.0-[0000:4f-51]--
  |           +-0c.0-[0000:4c-4e]--
  |           +-0d.0-[0000:49-4b]--
  |           +-0e.0-[0000:46-48]--
  |           +-10.0-[0000:41]--+-01.0  Broadcom Corporation NetXtreme II BCM5706 Gigabit Ethernet
  |           |                 \-02.0  Broadcom Corporation NetXtreme II BCM5706 Gigabit Ethernet
  |           +-10.1  Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
  |           +-11.0-[0000:42-45]--
  |           \-11.1  Advanced Micro Devices [AMD] AMD-8132 PCI-X IOAPIC
  \-[0000:00]-+-00.0  nVidia Corporation CK804 Memory Controller
              +-01.0  nVidia Corporation CK804 ISA Bridge
              +-02.0  nVidia Corporation CK804 USB Controller
              +-02.1  nVidia Corporation CK804 USB Controller
              +-06.0  nVidia Corporation CK804 IDE
              +-09.0-[0000:01]--+-03.0  ATI Technologies Inc ES1000
              |                 +-04.0  Compaq Computer Corporation Integrated Lights Out Controller
              |                 +-04.2  Compaq Computer Corporation Integrated Lights Out  Processor
              |                 +-04.4  Hewlett-Packard Company Proliant iLO2 virtual USB controller
              |                 \-04.6  Hewlett-Packard Company Proliant iLO2 virtual UART
              +-0c.0-[0000:08-0a]----00.0  Hewlett-Packard Company Smart Array Controller
              +-0d.0-[0000:05-07]--
              +-0e.0-[0000:02-04]--
              +-18.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
              +-18.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
              +-18.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
              +-18.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
              +-19.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
              +-19.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
              +-19.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
              +-19.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
              +-1a.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
              +-1a.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
              +-1a.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
              +-1a.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
              +-1b.0  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] HyperTransport Technology Configuration
              +-1b.1  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Address Map
              +-1b.2  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] DRAM Controller
              \-1b.3  Advanced Micro Devices [AMD] K8 [Athlon64/Opteron] Miscellaneous Control
[root@hp-dl585g2-01 ~]#

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Tony Camuso]

Robert Hancock wrote:

> The case of the device built into the K8 northbridge that's unreachable 
> by MMCONFIG kind of makes sense, since the northbridge is what's 
> translating the MMCONFIG memory access into config accesses. It seems 
> bizarre to me that a bridge chip could possibly have such a problem. The 
> MMCONFIG access should get translated into a configuration space access 
> in the northbridge and from that point on there's no difference between 
> an MMCONFIG and type1 access.
> 
Robert's point is well taken.

Only northbridge chips can give us this kind of trouble, and the only
chips mentioned in the present discussion as not being mmconf-compliant
are northbridges (8132, ht1000).

The patch is aware of this, so once a root bus has been programmed for
legacy pci config access, all descendent buses automatically inherit
this access mechanism and are therefore not probed by the patch.

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Andi Kleen]

Robert Hancock <hancockr@shaw.ca> writes:

> First off, I would like to see confirmation from the horses's mouths
> here (namely AMD,

AMD publicly releases errata sheets/data sheets for their PCI bridges
(check their website).  I haven't checked the 8132 errata for this
though. Not sure it implements MMCONFIG at all.

However the PCI Express chipsets typically do implement 
their own MMCONFIG aperture.

> The case of the device built into the K8 northbridge that's
> unreachable by MMCONFIG kind of makes sense, 

The internal northbridge devices on K8 are not reachable through
mmconfig. While BIOS are supposed to express this in MCFG by excluding
that bus many don't. That was the original reason I added the
type1<->mcfg sanity check. It catches the K8 case fine.

> since the northbridge is
> what's translating the MMCONFIG memory access into config accesses. 

The way it works on K8 systems is that the CPU internal northbridge
knows nothing about MMCONFIG, but that the external chipsets
implement an MMCONFIG aperture on their own outside the northbridge.

If you have multiple bridges like some SLI K8 setups that could
be multiple ones.

This has changed on the Quad Core Fam10h CPUs BTW -- there the
NB can deliver an single mmconfig aperture that is translated
to appropiate transactions on the Hypertransport link.

What might happen with K8 and 8132 (I'm speculating here) is that 
they got a PCI Express chipset that implements an MMCONFIG 
aperture for its devices, but the system also has a PCI-X 8132
bridge and the MMCONFIG aperture inside the chipset doesn't
support talking to the 8132 which might be "upstream" in the HT
topology. And the BIOS' MCFG doesn't tell Linux that.

-Andi

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Robert Hancock]

Loic Prylli wrote:
> On 12/20/2007 6:21 PM, Tony Camuso wrote:
>> And the MMCONFIG problem with enterprise systems and workstations, where
>> we do control the BIOS (for the most part), is due to known bugs in
>> certain versions of certain chipsets, HT1000, AMD8132, among them, not
>> the BIOS.
> 
> 
> 
> The lack of MMCONFIG support is indeed because some hypertransport
> chipsets lack that support. But there are some BIOSes out there that are
> advertising support for all busses in their MCFG acpi attribute (even
> the busses managed by some amd8131 in a mixed nvidia-ck804/amd8131
> motherboard), and the BIOS seems at least faulty for advertising a
> capability that does not exist.

This didn't really occur to me before for some reason. But yes, the MCFG 
table lists the buses to which each MMCONFIG region is applicable. If 
there are entire buses which MMCONFIG cannot access, it should not be 
indicating they are accessible via MMCONFIG in the ACPI MCFG table. If 
it is, then it's truly a BIOS bug.

Unless of course Linux isn't handling what the MCFG table is indicating 
properly. Then it's our bug. It would be good to verify this on one of 
the systems involved..

One of the things this patch (currently in -mm) does is dump out the 
segment and starting/ending buses for each MCFG configuration listed. 
The dmesg from this patch applied on such a system would tell you which 
is the case:

http://git.kernel.org/?p=linux/kernel/git/x86/linux-2.6-x86.git;a=commit;h=e18c985289ee356f06dbc953281a3c140a02fbb3

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: [Fwd: Re: [PATCH 4/5]PCI: x86 MMCONFIG] introduce pcibios_fix_bus_scan_quirk()

[Tony Camuso]

I need to resubmit this patch.

Resubmission will be in my next email.

It was a minor problem whereby the patch would advise the user

	If a device isn't working, try "pci=nommconf".

even when "pci=mmconf" was used at the command line.

One additional line of logic fixed it, but I would like to resubmit
the patch rather than patch the patch.

Re: [Fwd: Re: [PATCH 0/5]PCI: x86 MMCONFIG]

[Robert Hancock]

Loic Prylli wrote:
> I just realized one thing: the bar sizing code in pci_read_bases() (that
> writes 0xffffffff in the bars) does not seem to disable the
> PCI_COMMAND_MEM/PCI_COMMAND_IO bits in the cmd register before
> manipulating the BARs. And it seems nobody else ensures they are
> disabled at this point either (or am I missing something?).

No you're not missing anything. This problem causes many machines to 
break horribly when MMCONFIG is enabled. There's a patch in -mm to fix 
this. (It special-cases the case of host bridges and doesn't disable the 
  decode bits for those, since some are known to do crazy things if you 
do that.)

http://www.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.24-rc6/2.6.24-rc6-mm1/broken-out/pci-disable-decoding-during-sizing-of-bars.patch

> 
> Touching the bars while they are enabled would be buggy behaviour from
> our part, and something trivial to fix. And it might well fix that
> particular problem (it's fair play from the machine to crash if we
> create a decoding conflict, simply disabling the cmd bits in
> pci_read_bases() should remove that conflict).
> 
> FWIW, to partially answer your last question, Windows does disable
> mem-space and/or IO-space when sizing the bars of a device (I have some
> traces of configuration-space-access taken on a window machine for one
> of the PCI busses).

Good to know. There was some speculation that it did not.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/