Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Suravee Suthikulanit <suravee.suthikulpanit@amd.com>]

On 5/22/2014 9:54 PM, Bjorn Helgaas wrote:
> I've been poking around for recent dmesg logs that contain "PCI: Using
> configuration type 1 for extended access", and there are quite a few.
> In most cases there*is*  an MCFG table, but apparently we decide not
> to use it for some reason (unfortunately we don't print the specific
> reason).  One example is at
> https://bugzilla.kernel.org/show_bug.cgi?id=68591  .
>
> I'm going to go out on a limb and guess that Windows does not enable
> ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
> of MCFG is broken in some way, and we probably*could*  use ECAM in all
> these cases I'm seeing.
>
> It would probably be prudent to figure out why Linux is rejecting
> these MCFG tables.  We'll probably see similar tables on Fam17h
> systems, and if we continue rejecting them, and we don't turn on ECS,
> we won't be able to access extended config space.
>
> I opened a bugzilla for this issue:
> https://bugzilla.kernel.org/show_bug.cgi?id=76771
>
> I'm wavering on whether it's a good idea to put this patch in before
> understanding the issue.  As much as I'd like to stop fiddling with
> ECS, we'd likely end up with a v3.15 where extended config space
> doesn't work on some Fam17h systems.

So, I have located a system which presents issue with MMCONFIG. Here is 
my investigation:

DEBUG: pci_io_ecs_init: pci_probe = 4000f
ACPI: bus type PCI registered
DEBUG: -----> pci_mmcfg_early_init
DEBUG: pci_parse_mcfg
PCI: MMCONFIG for domain 0000 [bus 00-01] at [mem 0xe0000000-0xe01fffff] 
(base 0xe0000000)
DEBUG: pci_mmcfg_check_reserved
DEBUG: is_mmconf_reserved: method = E820
PCI: not using MMCONFIG
DEBUG: pci_direct_init
PCI: Using configuration type 1 for base access
PCI: Using configuration type 1 for extended access
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
[Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored
\_SB_:_OSC invalid UUID
_OSC request data:1 1f
ACPI: Interpreter enabled
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] 
(20140214/hwxface-580)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] 
(20140214/hwxface-580)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] 
(20140214/hwxface-580)
ACPI: (supports S0 S3 S5)
ACPI: Using IOAPIC for interrupt routing
DEBUG: ----> pci_mmcfg_late_init
DEBUG: pci_parse_mcfg
PCI: MMCONFIG for domain 0000 [bus 00-01] at [mem 0xe0000000-0xe01fffff] 
(base 0xe0000000)
DEBUG: pci_mmcfg_check_reserved
DEBUG: is_mmconf_reserved: method = ACPI motherboard resources
PCI: MMCONFIG at [mem 0xe0000000-0xe01fffff] reserved in ACPI 
motherboard resources

During pci_mmcfg_early_init(), the MMCONFIG failed because the range 
0xe0000000 is not showing as reserved in the E820 mapping.  Here is the 
snippet of E820 mapping from the system:
........
BIOS-e820: [mem 0x00000000c7eb0000-0x00000000c7ec0fff] ACPI data
BIOS-e820: [mem 0x00000000c7ec1000-0x00000000c7ec2fff] ACPI NVS
BIOS-e820: [mem 0x00000000c7ec3000-0x00000000c7efefff] reserved
BIOS-e820: [mem 0x00000000c7f00000-0x00000000c7ffffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved

However, during pci_mmcfg_late_init(), the area is reserved in the "ACPI 
motherboard resources", and the pci_mmcfg_check_reserved() does not fail 
here.  But this is too late since we already setup the "raw_pci_ext_ops" 
in the "arch/x86/pci/direct.c: pci_direct_init()" (during to use the IO_ECS.

Suravee