PCI IO ECS access is no longer possible for AMD family 17h

PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

Hi all,

I am currently working on a custom AMD Ryzen™ Embedded R2000 (AMD
Family 17h) device and have discovered that PCI IO Extended
Configuration Space (ECS) access is no longer possible.

Consider the following functions: amd_bus_cpu_online() and
pci_enable_pci_io_ecs(). These functions are part of the
amd_postcore_init() initcall and are responsible for enabling PCI IO
ECS. Both functions modify the CoreMasterAccessCtrl (EnableCf8ExtCfg)
value via the PCI device function or the MSR register directly (see
the "BIOS and Kernel Developer’s Guide (BKDG) for AMD Family 15h",
Section 2.7). However, neither the MSR register nor the PCI function
at the specified address (D18F3x8C) exists for AMD Family 17h. The
CoreMasterAccessCtrl register still exists but is now located at a
different address (see the "Processor Programming Reference (PPR) for
AMD Family 17h", Section 2.1.8).

I would be happy to submit a patch to fix this issue. However, since
the most recent change affecting this functionality appears to be 14
years old, I would like to confirm whether this is still relevant or
if the kernel should always be built with CONFIG_PCI_MMCONFIG when ECS
access is required.

Linux Kernel info:

root@qt5222:~# uname -a
Linux qt5222 6.6.49-2447-qtec-standard--gef032148967a #1 SMP Fri Nov
22 09:25:55 UTC 2024 x86_64 GNU/Linux

Best regards,
Ros

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Borislav Petkov]

+ Yazen.

On Mon, Dec 16, 2024 at 02:15:26PM +0100, Rostyslav Khudolii wrote:
> Hi all,
> 
> I am currently working on a custom AMD Ryzen™ Embedded R2000 (AMD
> Family 17h) device and have discovered that PCI IO Extended
> Configuration Space (ECS) access is no longer possible.

Perhaps that particular type of system of yours needs some special handling.

Things to try:

* use the latest upstream kernel

* add some debug printks to the paths you mention to see where they fail

Looking at the relevant chapter in the PPR - 2.1.7 or 2.1.8 - that
should still work.

Leaving in the rest for reference.

> Consider the following functions: amd_bus_cpu_online() and
> pci_enable_pci_io_ecs(). These functions are part of the
> amd_postcore_init() initcall and are responsible for enabling PCI IO
> ECS. Both functions modify the CoreMasterAccessCtrl (EnableCf8ExtCfg)
> value via the PCI device function or the MSR register directly (see
> the "BIOS and Kernel Developer’s Guide (BKDG) for AMD Family 15h",
> Section 2.7). However, neither the MSR register nor the PCI function
> at the specified address (D18F3x8C) exists for AMD Family 17h. The
> CoreMasterAccessCtrl register still exists but is now located at a
> different address (see the "Processor Programming Reference (PPR) for
> AMD Family 17h", Section 2.1.8).
> 
> I would be happy to submit a patch to fix this issue. However, since
> the most recent change affecting this functionality appears to be 14
> years old, I would like to confirm whether this is still relevant or
> if the kernel should always be built with CONFIG_PCI_MMCONFIG when ECS
> access is required.
> 
> Linux Kernel info:
> 
> root@qt5222:~# uname -a
> Linux qt5222 6.6.49-2447-qtec-standard--gef032148967a #1 SMP Fri Nov
> 22 09:25:55 UTC 2024 x86_64 GNU/Linux
> 
> Best regards,
> Ros

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Yazen Ghannam]

On Thu, Dec 19, 2024 at 12:23:37PM +0100, Borislav Petkov wrote:
> + Yazen.
> 
> On Mon, Dec 16, 2024 at 02:15:26PM +0100, Rostyslav Khudolii wrote:
> > Hi all,
> > 
> > I am currently working on a custom AMD Ryzen™ Embedded R2000 (AMD
> > Family 17h) device and have discovered that PCI IO Extended
> > Configuration Space (ECS) access is no longer possible.
> 
> Perhaps that particular type of system of yours needs some special handling.
> 
> Things to try:
> 
> * use the latest upstream kernel
> 
> * add some debug printks to the paths you mention to see where they fail
> 
> Looking at the relevant chapter in the PPR - 2.1.7 or 2.1.8 - that
> should still work.
> 
> Leaving in the rest for reference.
> 
> > Consider the following functions: amd_bus_cpu_online() and
> > pci_enable_pci_io_ecs(). These functions are part of the
> > amd_postcore_init() initcall and are responsible for enabling PCI IO
> > ECS. Both functions modify the CoreMasterAccessCtrl (EnableCf8ExtCfg)
> > value via the PCI device function or the MSR register directly (see
> > the "BIOS and Kernel Developer’s Guide (BKDG) for AMD Family 15h",
> > Section 2.7). However, neither the MSR register nor the PCI function
> > at the specified address (D18F3x8C) exists for AMD Family 17h. The
> > CoreMasterAccessCtrl register still exists but is now located at a
> > different address (see the "Processor Programming Reference (PPR) for
> > AMD Family 17h", Section 2.1.8).
> > 
> > I would be happy to submit a patch to fix this issue. However, since
> > the most recent change affecting this functionality appears to be 14
> > years old, I would like to confirm whether this is still relevant or
> > if the kernel should always be built with CONFIG_PCI_MMCONFIG when ECS
> > access is required.
> > 

Hi Ros,

I expect you would want CONFIG_ACPI_MCFG and the "MCFG" table should be
provided through ACPI.

Can you please confirm if this config option is enabled, and that the
system provides MCFG?

> > Linux Kernel info:
> > 
> > root@qt5222:~# uname -a
> > Linux qt5222 6.6.49-2447-qtec-standard--gef032148967a #1 SMP Fri Nov
> > 22 09:25:55 UTC 2024 x86_64 GNU/Linux
> > 

The ECS space is expected to be enabled by BIOS and advertised through
ACPI.

The oldest public doc I found mentions this. This is the BKDG for Family
10h dated from 2013, and the product was launched in 2007.
https://www.amd.com/content/dam/amd/en/documents/archived-tech-docs/programmer-references/31116.pdf

"The BIOS may use either configuration space access mechanism during
boot. Before booting the OS, BIOS must disable IO access to ECS, enable
MMIO configuration and build an ACPI defined MCFG table. BIOS ACPI code
must use MMIO to access configuration space. "

This paragraph remains unchanged through the docs for the latest
products.

The PCI Firmware space v3.0, dated from 2005, has this in an
implementation note:

"If this platform also needs to support legacy operating systems or x86
BIOS Option ROMs, the CF8/CFCh access mechanism (which is PCI Segment
Group unaware and only can support up to 256 bus numbers) must also be
supported."

My understanding, based on the above info, is that ACPI should be used.
The direct register enablement is still possible for backwards
compatibility, if needed.

I think your observation proves a good point. The registers were moved
starting in Zen. But this is not an issue on modern OSes since ACPI is
used by default.

For your specific issue, I think we should determine if there is a
configuration or a firmware problem.

There's also a difference between "early" and "late" mmcfg init. But I
think we need to have more details on the issue.

Thanks,
Yazen

P.S. Thanks for reporting this! It's a good question, and I find doing
"archaeological" research enjoyable. :)

P.P.S. For the maintainers: should we consider cleaning up legacy code
such as this?

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

Hi Yazen, Borislav

Thank you for the elaborate reply.

> Things to try:
>
> * use the latest upstream kernel
>
> * add some debug printks to the paths you mention to see where they fail
>
> Looking at the relevant chapter in the PPR - 2.1.7 or 2.1.8 - that
> should still work.

As far as I can see the behavior should be the same with the latest upstream
kernel and the same configuration (see below).

> I expect you would want CONFIG_ACPI_MCFG and the "MCFG" table should be
> provided through ACPI.
>
> Can you please confirm if this config option is enabled, and that the
> system provides MCFG?
>

Neither CONFIG_ACPI_MCFG nor CONFIG_PCI_MMCONFIG were originally enabled
in my kernel, when I discovered this issue.

> My understanding, based on the above info, is that ACPI should be used.
> The direct register enablement is still possible for backwards
> compatibility, if needed.
>
> I think your observation proves a good point. The registers were moved
> starting in Zen. But this is not an issue on modern OSes since ACPI is
> used by default.

This is my understanding too. However, what is the desired behavior on
Zen if the CONFIG_ACPI_MCFG
and CONFIG_PCI_MMCONFIG are both disabled? ECS should not be possible
since the registers were
moved, right?
If that's the case then, at the very least, it would be great to have
a warning message.

> For your specific issue, I think we should determine if there is a
> configuration or a firmware problem.

To give a bit more context: I am porting the kernel which works on the
AMD Ryzen™ Embedded V1000-based
device. On that system, it seems like the firmware doesn't disable IO
access to ECS (which is wrong), hence we
have never experienced this issue before. Now, the R2000-based
device's firmware disables IO access to ECS
(correctly) and that's when the issue starts to happen.

Thanks,
Ros

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Ingo Molnar]

* Rostyslav Khudolii <ros@qtec.com> wrote:

> > My understanding, based on the above info, is that ACPI should be 
> > used. The direct register enablement is still possible for 
> > backwards compatibility, if needed.
> >
> > I think your observation proves a good point. The registers were 
> > moved starting in Zen. But this is not an issue on modern OSes 
> > since ACPI is used by default.
> 
> This is my understanding too. However, what is the desired behavior 
> on Zen if the CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are both 
> disabled? ECS should not be possible since the registers were moved, 
> right? If that's the case then, at the very least, it would be great 
> to have a warning message.
>
> > For your specific issue, I think we should determine if there is a 
> > configuration or a firmware problem.
> 
> To give a bit more context: I am porting the kernel which works on 
> the AMD Ryzen™ Embedded V1000-based device. On that system, it seems 
> like the firmware doesn't disable IO access to ECS (which is wrong), 
> hence we have never experienced this issue before. Now, the 
> R2000-based device's firmware disables IO access to ECS (correctly) 
> and that's when the issue starts to happen.

[ Sorry about the late reply. ]

So what is the practical impact here? Do things start breaking 
unexpectedly if CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled? 
Then I'd suggest fixing that in the Kconfig space, either by adding a 
dependency on ACPI_MCFG && PCI_MMCONFIG, or by selecting those 
must-have pieces of infrastructure.

Thanks,

	Ingo

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

Hi,

> Rostyslav, I would like to ask you, do you have patches / updates for
> enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
> adjust my lspci changes for new machines.

Pali, sorry for the late reply. Do I understand correctly, that even
though you have access to the ECS via
the MMCFG you still want the legacy (direct IO) to work for the
debugging purposes? I can prepare a
simple patch that will allow you to do so if that's the case.

>
> So what is the practical impact here? Do things start breaking
> unexpectedly if CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled?
> Then I'd suggest fixing that in the Kconfig space, either by adding a
> dependency on ACPI_MCFG && PCI_MMCONFIG, or by selecting those
> must-have pieces of infrastructure.
>

Ingo, thank you for the reply.

The way I understand the access to the PCI ECS (via raw_pci_ext_ops)
works, is the following:
1. If CONFIG_ACPI_MCFG or CONFIG_PCI_MMCONFIG are enabled - set the
raw_pci_ext_ops to use
    MMCFG to access ECS. See pci_mmcfg_early_init() / pci_mmcfg_late_init();
2. If CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled - set the
raw_pci_ext_ops to use
    the 'direct' access to ECS. See pci_direct_init(). The direct
access is conditional on the PCI_HAS_IO_ECS
    flag being set.

On AMD, the kernel enables the ECS IO access via the
amd_bus_cpu_online() and pci_enable_pci_io_ecs().
Except those functions have no desired effect on the AMD 17h+ family
because the register (EnableCf8ExtCfg),
they access, has been moved. What is important though, is that the
PCI_HAS_IO_ECS flag is set unconditionally.
See pci_io_ecs_init() in amd_bus.c

Therefore I was wondering whether we should add support for the 17h+
family in those functions to have
the direct access work for those families as well.

Regarding your suggestion to address this in the Kconfig space - I'm
not quite sure I follow, since right now the kernel
will use raw_pci_ext_ops whenever access beyond the first 256 bytes is
requested. Say we want to make that
conditional on CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG, does it also
mean then we want to drop support
for the 'direct' PCI IO ECS access altogether?

Best regards,
Rostyslav

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Ingo Molnar]

* Rostyslav Khudolii <ros@qtec.com> wrote:

> Hi,
> 
> > Rostyslav, I would like to ask you, do you have patches / updates for
> > enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
> > adjust my lspci changes for new machines.
> 
> Pali, sorry for the late reply. Do I understand correctly, that even
> though you have access to the ECS via
> the MMCFG you still want the legacy (direct IO) to work for the
> debugging purposes? I can prepare a
> simple patch that will allow you to do so if that's the case.
> 
> >
> > So what is the practical impact here? Do things start breaking
> > unexpectedly if CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled?
> > Then I'd suggest fixing that in the Kconfig space, either by adding a
> > dependency on ACPI_MCFG && PCI_MMCONFIG, or by selecting those
> > must-have pieces of infrastructure.
> >
> 
> Ingo, thank you for the reply.
> 
> The way I understand the access to the PCI ECS (via raw_pci_ext_ops)
> works, is the following:
> 1. If CONFIG_ACPI_MCFG or CONFIG_PCI_MMCONFIG are enabled - set the
> raw_pci_ext_ops to use
>     MMCFG to access ECS. See pci_mmcfg_early_init() / pci_mmcfg_late_init();
> 2. If CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled - set the
> raw_pci_ext_ops to use
>     the 'direct' access to ECS. See pci_direct_init(). The direct
> access is conditional on the PCI_HAS_IO_ECS
>     flag being set.
> 
> On AMD, the kernel enables the ECS IO access via the
> amd_bus_cpu_online() and pci_enable_pci_io_ecs().
> Except those functions have no desired effect on the AMD 17h+ family
> because the register (EnableCf8ExtCfg),
> they access, has been moved. What is important though, is that the
> PCI_HAS_IO_ECS flag is set unconditionally.
> See pci_io_ecs_init() in amd_bus.c
> 
> Therefore I was wondering whether we should add support for the 17h+
> family in those functions to have
> the direct access work for those families as well.

Yeah, I think so, but I'm really just guessing:

> Regarding your suggestion to address this in the Kconfig space - I'm 
> not quite sure I follow, since right now the kernel will use 
> raw_pci_ext_ops whenever access beyond the first 256 bytes is 
> requested. Say we want to make that conditional on CONFIG_ACPI_MCFG 
> and CONFIG_PCI_MMCONFIG, does it also mean then we want to drop 
> support for the 'direct' PCI IO ECS access altogether?

I thought that enabling CONFIG_ACPI_MCFG would solve the problem, and 
other architectures are selecting it pretty broadly:

 arch/arm64/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
 arch/loongarch/Kconfig: select ACPI_MCFG if ACPI
 arch/riscv/Kconfig:     select ACPI_MCFG if (ACPI && PCI)

While x86 allows it to be user-configured, which may result in 
misconfiguration, given that PCI_HAS_IO_ECS is being followed 
unconditionally if a platform provides it?

Thanks,

	Ingo

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

* Ingo Molnar <mingo@kernel.org> wrote:
>
>
> * Rostyslav Khudolii <ros@qtec.com> wrote:
>
> > Hi,
> >
> > > Rostyslav, I would like to ask you, do you have patches / updates for
> > > enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
> > > adjust my lspci changes for new machines.
> >
> > Pali, sorry for the late reply. Do I understand correctly, that even
> > though you have access to the ECS via
> > the MMCFG you still want the legacy (direct IO) to work for the
> > debugging purposes? I can prepare a
> > simple patch that will allow you to do so if that's the case.
> >
> > >
> > > So what is the practical impact here? Do things start breaking
> > > unexpectedly if CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled?
> > > Then I'd suggest fixing that in the Kconfig space, either by adding a
> > > dependency on ACPI_MCFG && PCI_MMCONFIG, or by selecting those
> > > must-have pieces of infrastructure.
> > >
> >
> > Ingo, thank you for the reply.
> >
> > The way I understand the access to the PCI ECS (via raw_pci_ext_ops)
> > works, is the following:
> > 1. If CONFIG_ACPI_MCFG or CONFIG_PCI_MMCONFIG are enabled - set the
> > raw_pci_ext_ops to use
> >     MMCFG to access ECS. See pci_mmcfg_early_init() / pci_mmcfg_late_init();
> > 2. If CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled - set the
> > raw_pci_ext_ops to use
> >     the 'direct' access to ECS. See pci_direct_init(). The direct
> > access is conditional on the PCI_HAS_IO_ECS
> >     flag being set.
> >
> > On AMD, the kernel enables the ECS IO access via the
> > amd_bus_cpu_online() and pci_enable_pci_io_ecs().
> > Except those functions have no desired effect on the AMD 17h+ family
> > because the register (EnableCf8ExtCfg),
> > they access, has been moved. What is important though, is that the
> > PCI_HAS_IO_ECS flag is set unconditionally.
> > See pci_io_ecs_init() in amd_bus.c
> >
> > Therefore I was wondering whether we should add support for the 17h+
> > family in those functions to have
> > the direct access work for those families as well.
>
> Yeah, I think so, but I'm really just guessing:
>
> > Regarding your suggestion to address this in the Kconfig space - I'm
> > not quite sure I follow, since right now the kernel will use
> > raw_pci_ext_ops whenever access beyond the first 256 bytes is
> > requested. Say we want to make that conditional on CONFIG_ACPI_MCFG
> > and CONFIG_PCI_MMCONFIG, does it also mean then we want to drop
> > support for the 'direct' PCI IO ECS access altogether?
>
> I thought that enabling CONFIG_ACPI_MCFG would solve the problem, and
> other architectures are selecting it pretty broadly:
>
>  arch/arm64/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
>  arch/loongarch/Kconfig: select ACPI_MCFG if ACPI
>  arch/riscv/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
>
> While x86 allows it to be user-configured, which may result in
> misconfiguration, given that PCI_HAS_IO_ECS is being followed
> unconditionally if a platform provides it?
>
> Thanks,
>
>         Ingo

So it seems that the config option that needs to be enabled is
CONFIG_PCI_MMCONFIG.
The MCFG kernel support depends on it (aka not compiled when disabled).
With that said - the IO ECS question remains. Do we want to support it
for AMD 17h+ or
it's not required anymore?

Regards,
Rostyslav

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Yazen Ghannam]

On Mon, Mar 03, 2025 at 01:20:21PM +0100, Rostyslav Khudolii wrote:
> * Ingo Molnar <mingo@kernel.org> wrote:
> >
> >
> > * Rostyslav Khudolii <ros@qtec.com> wrote:
> >
> > > Hi,
> > >
> > > > Rostyslav, I would like to ask you, do you have patches / updates for
> > > > enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
> > > > adjust my lspci changes for new machines.
> > >
> > > Pali, sorry for the late reply. Do I understand correctly, that even
> > > though you have access to the ECS via
> > > the MMCFG you still want the legacy (direct IO) to work for the
> > > debugging purposes? I can prepare a
> > > simple patch that will allow you to do so if that's the case.
> > >
> > > >
> > > > So what is the practical impact here? Do things start breaking
> > > > unexpectedly if CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled?
> > > > Then I'd suggest fixing that in the Kconfig space, either by adding a
> > > > dependency on ACPI_MCFG && PCI_MMCONFIG, or by selecting those
> > > > must-have pieces of infrastructure.
> > > >
> > >
> > > Ingo, thank you for the reply.
> > >
> > > The way I understand the access to the PCI ECS (via raw_pci_ext_ops)
> > > works, is the following:
> > > 1. If CONFIG_ACPI_MCFG or CONFIG_PCI_MMCONFIG are enabled - set the
> > > raw_pci_ext_ops to use
> > >     MMCFG to access ECS. See pci_mmcfg_early_init() / pci_mmcfg_late_init();
> > > 2. If CONFIG_ACPI_MCFG and CONFIG_PCI_MMCONFIG are disabled - set the
> > > raw_pci_ext_ops to use
> > >     the 'direct' access to ECS. See pci_direct_init(). The direct
> > > access is conditional on the PCI_HAS_IO_ECS
> > >     flag being set.
> > >
> > > On AMD, the kernel enables the ECS IO access via the
> > > amd_bus_cpu_online() and pci_enable_pci_io_ecs().
> > > Except those functions have no desired effect on the AMD 17h+ family
> > > because the register (EnableCf8ExtCfg),
> > > they access, has been moved. What is important though, is that the
> > > PCI_HAS_IO_ECS flag is set unconditionally.
> > > See pci_io_ecs_init() in amd_bus.c
> > >
> > > Therefore I was wondering whether we should add support for the 17h+
> > > family in those functions to have
> > > the direct access work for those families as well.
> >
> > Yeah, I think so, but I'm really just guessing:
> >
> > > Regarding your suggestion to address this in the Kconfig space - I'm
> > > not quite sure I follow, since right now the kernel will use
> > > raw_pci_ext_ops whenever access beyond the first 256 bytes is
> > > requested. Say we want to make that conditional on CONFIG_ACPI_MCFG
> > > and CONFIG_PCI_MMCONFIG, does it also mean then we want to drop
> > > support for the 'direct' PCI IO ECS access altogether?
> >
> > I thought that enabling CONFIG_ACPI_MCFG would solve the problem, and
> > other architectures are selecting it pretty broadly:
> >
> >  arch/arm64/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
> >  arch/loongarch/Kconfig: select ACPI_MCFG if ACPI
> >  arch/riscv/Kconfig:     select ACPI_MCFG if (ACPI && PCI)
> >
> > While x86 allows it to be user-configured, which may result in
> > misconfiguration, given that PCI_HAS_IO_ECS is being followed
> > unconditionally if a platform provides it?
> >
> > Thanks,
> >
> >         Ingo
> 
> So it seems that the config option that needs to be enabled is
> CONFIG_PCI_MMCONFIG.
> The MCFG kernel support depends on it (aka not compiled when disabled).
> With that said - the IO ECS question remains. Do we want to support it
> for AMD 17h+ or
> it's not required anymore?
> 

Hi all,

I'd say it is not required (nor encouraged) anymore based on my reading
of the AMD docs and PCI Firmware spec.

The possibility for direct access is still in hardware for backwards
compatibility. But this doesn't mean that it *must* be used. And it
seems that it shouldn't be used, if possible.

I'm for the option to address this issue through Kconfig.

Ros,
Is/was there a reason why you didn't have the MCFG/MMCONFIG options
enabled in your kernel?

Was this a side effect of trying to build a minimal kernel or similar?

Thanks,
Yazen

P.S. Sorry for the late reply. My mailbox is missing Ros's reply to me.

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

Hi Yazen,

>
> Ros,
> Is/was there a reason why you didn't have the MCFG/MMCONFIG options
> enabled in your kernel?
>
> Was this a side effect of trying to build a minimal kernel or similar?
>
> Thanks,
> Yazen
>
> P.S. Sorry for the late reply. My mailbox is missing Ros's reply to me.

I inherited a kernel config used here from a previous custom,
V1000-based solution. Most likely originally
it came from the x86_64_defconfig. Maybe MMCONFIG wasn't enabled by
default at that time.
We never had problems with this because our BIOS keeps the IO ECS
enabled (that's against the spec recommendation).
The kernel default for PCI_MMCONFIG is 'y' now, so it's unlikely that
people will run into this issue unless
they explicitly disable it.

Regards,
Rostyslav

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Yazen Ghannam]

On Tue, Mar 04, 2025 at 08:39:29AM +0100, Rostyslav Khudolii wrote:
> Hi Yazen,
> 
> >
> > Ros,
> > Is/was there a reason why you didn't have the MCFG/MMCONFIG options
> > enabled in your kernel?
> >
> > Was this a side effect of trying to build a minimal kernel or similar?
> >
> > Thanks,
> > Yazen
> >
> > P.S. Sorry for the late reply. My mailbox is missing Ros's reply to me.
> 
> I inherited a kernel config used here from a previous custom,
> V1000-based solution. Most likely originally
> it came from the x86_64_defconfig. Maybe MMCONFIG wasn't enabled by
> default at that time.
> We never had problems with this because our BIOS keeps the IO ECS
> enabled (that's against the spec recommendation).
> The kernel default for PCI_MMCONFIG is 'y' now, so it's unlikely that
> people will run into this issue unless
> they explicitly disable it.
> 

Thanks Rostyslav for confirming.

This seems to be the commit:
55027a7772b1 ("x86: Align x86_64 PCI_MMCONFIG with 32-bit variant")

Before this, PCI_MMCONFIG was default 'y' on 32-bit and no default on
64-bit.

It has been default 'y' on 32-bit since the beginning of history.
1da177e4c3f4 ("Linux-2.6.12-rc2")

Thanks,
Yazen

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Pali Rohár]

Hello Rostyslav, you sent an email addressed to me, but you forgot to
put my email address into recipients. So I have not received it.
But luckily, today I found your email by a chance in lkml archive:
https://lore.kernel.org/lkml/CAJDH93uE+foFfRAXVJ48-PYvEUsbpEu_-BVoG-5HsDG66yY7AQ@mail.gmail.com/

On Monday 03 March 2025 10:21:25 Rostyslav Khudolii wrote:
> Hi,
> 
> > Rostyslav, I would like to ask you, do you have patches / updates for
> > enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
> > adjust my lspci changes for new machines.
> 
> Pali, sorry for the late reply. Do I understand correctly, that even
> though you have access to the ECS via
> the MMCFG you still want the legacy (direct IO) to work for the
> debugging purposes? I can prepare a
> simple patch that will allow you to do so if that's the case.

Yes, I would like to get access to ECS via CF8/CFC direct IO for
debugging purposes, even though MMCFG is enabled and used.

If you can send me a simple patch or any sample / idea how to do it,
I would be very happy. I was not able to figure out how to enable ECS
over CF8/CFC.

Pali

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

>
> Yes, I would like to get access to ECS via CF8/CFC direct IO for
> debugging purposes, even though MMCFG is enabled and used.
>
> If you can send me a simple patch or any sample / idea how to do it,
> I would be very happy. I was not able to figure out how to enable ECS
> over CF8/CFC.
>

Hi Pali,
I'm sorry for the late reply.

If I understand your request correctly, all you need to do  is patch
the pci_enable_pci_io_ecs() function to use
the different register to set EnableCF8ExtCfg: D18F4x044.

Regards,
Ros

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Pali Rohár]

On Tuesday 15 July 2025 12:43:16 Rostyslav Khudolii wrote:
> >
> > Yes, I would like to get access to ECS via CF8/CFC direct IO for
> > debugging purposes, even though MMCFG is enabled and used.
> >
> > If you can send me a simple patch or any sample / idea how to do it,
> > I would be very happy. I was not able to figure out how to enable ECS
> > over CF8/CFC.
> >
> 
> Hi Pali,
> I'm sorry for the late reply.
> 
> If I understand your request correctly, all you need to do  is patch
> the pci_enable_pci_io_ecs() function to use
> the different register to set EnableCF8ExtCfg: D18F4x044.
> 
> Regards,
> Ros

Hello, thank you for information.

Just I would like to know, where did you find information that the
EnableCF8ExtCfg register was moved to D18F4x044? It is documented in
some AMD specification?

I did not find anything regarding this change.

Pali

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

> Hello, thank you for information.
>
> Just I would like to know, where did you find information that the
> EnableCF8ExtCfg register was moved to D18F4x044? It is documented in
> some AMD specification?
>
> I did not find anything regarding this change.

I mentioned the exact specification in my first message. It's under
NDA, unfortunately.

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Pali Rohár]

On Wednesday 16 July 2025 10:03:40 Rostyslav Khudolii wrote:
> > Hello, thank you for information.
> >
> > Just I would like to know, where did you find information that the
> > EnableCF8ExtCfg register was moved to D18F4x044? It is documented in
> > some AMD specification?
> >
> > I did not find anything regarding this change.
> 
> I mentioned the exact specification in my first message. It's under
> NDA, unfortunately.

Do you know if this applies only for AMD family 17? Or also for later
fam 18, 19 and 1a?

And for confirmation, which bit represent the EnableCF8ExtCfg in PCI
config space register D18F4x044? It is still 14th bit like in family 16h?


Just for explanation, in your first message you wrote:

  "register still exists but is now located at a different address (see
  the "Processor Programming Reference (PPR) for AMD Family 17h", Section 2.1.8)."

Document named "Processor Programming Reference (PPR) for AMD Family 17h
Models 01h,08h, Revision B2 Processors" with revision information
"54945 Rev 3.03 - Jun 14, 2019" is publicly available at AMD web:

https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/54945-ppr-family-17h-models-00h-0fh-processors.zip

I looked into that document into section 2.1.8 "PCI Configuration Legacy Access"
and there is not related to EnableCF8ExtCfg.

So I was somehow confused to which section / document you are referring.
And now if I understand correctly, you have NDA documentation with the
same title as the above public one, but with the different content, right?

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Pali Rohár]

Hello Rostyslav, I would like to remind the previous email.
I still do not know which bit in D18F4x044 represents the
EnableCF8ExtCfg config.

On Wednesday 16 July 2025 20:13:20 Pali Rohár wrote:
> On Wednesday 16 July 2025 10:03:40 Rostyslav Khudolii wrote:
> > > Hello, thank you for information.
> > >
> > > Just I would like to know, where did you find information that the
> > > EnableCF8ExtCfg register was moved to D18F4x044? It is documented in
> > > some AMD specification?
> > >
> > > I did not find anything regarding this change.
> > 
> > I mentioned the exact specification in my first message. It's under
> > NDA, unfortunately.
> 
> Do you know if this applies only for AMD family 17? Or also for later
> fam 18, 19 and 1a?
> 
> And for confirmation, which bit represent the EnableCF8ExtCfg in PCI
> config space register D18F4x044? It is still 14th bit like in family 16h?
> 
> 
> Just for explanation, in your first message you wrote:
> 
>   "register still exists but is now located at a different address (see
>   the "Processor Programming Reference (PPR) for AMD Family 17h", Section 2.1.8)."
> 
> Document named "Processor Programming Reference (PPR) for AMD Family 17h
> Models 01h,08h, Revision B2 Processors" with revision information
> "54945 Rev 3.03 - Jun 14, 2019" is publicly available at AMD web:
> 
> https://www.amd.com/content/dam/amd/en/documents/processor-tech-docs/programmer-references/54945-ppr-family-17h-models-00h-0fh-processors.zip
> 
> I looked into that document into section 2.1.8 "PCI Configuration Legacy Access"
> and there is not related to EnableCF8ExtCfg.
> 
> So I was somehow confused to which section / document you are referring.
> And now if I understand correctly, you have NDA documentation with the
> same title as the above public one, but with the different content, right?

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Rostyslav Khudolii]

>
> Hello Rostyslav, I would like to remind the previous email.
> I still do not know which bit in D18F4x044 represents the
> EnableCF8ExtCfg config.
>

Hi Pali,

Sorry for the late reply again. So in D18F4x044:
- bit 0. EnableCf8ExtCfg. Enable PCI extended configuration register
access for CFC/CF8
accesses;
- bit 1. DisPciCfgReg. Disable CFC/CF8 Accesses to IO space. When set
to 1, CFC/CF8 accesses are treated as
PCI IO space accesses and not PCI configuration accesses.

Unfortunately, I don't know if this also applies to later families.

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Pali Rohár]

On Tuesday 26 August 2025 17:07:38 Rostyslav Khudolii wrote:
> >
> > Hello Rostyslav, I would like to remind the previous email.
> > I still do not know which bit in D18F4x044 represents the
> > EnableCF8ExtCfg config.
> >
> 
> Hi Pali,
> 
> Sorry for the late reply again. So in D18F4x044:
> - bit 0. EnableCf8ExtCfg. Enable PCI extended configuration register
> access for CFC/CF8
> accesses;
> - bit 1. DisPciCfgReg. Disable CFC/CF8 Accesses to IO space. When set
> to 1, CFC/CF8 accesses are treated as
> PCI IO space accesses and not PCI configuration accesses.
> 
> Unfortunately, I don't know if this also applies to later families.

Thank you very much for the info. I was missing the information about
the bit 0 offset.

Re: PCI IO ECS access is no longer possible for AMD family 17h

[Pali Rohár]

On Monday 16 December 2024 14:15:26 Rostyslav Khudolii wrote:
> Hi all,
> 
> I am currently working on a custom AMD Ryzen™ Embedded R2000 (AMD
> Family 17h) device and have discovered that PCI IO Extended
> Configuration Space (ECS) access is no longer possible.
> 
> Consider the following functions: amd_bus_cpu_online() and
> pci_enable_pci_io_ecs(). These functions are part of the
> amd_postcore_init() initcall and are responsible for enabling PCI IO
> ECS. Both functions modify the CoreMasterAccessCtrl (EnableCf8ExtCfg)
> value via the PCI device function or the MSR register directly (see
> the "BIOS and Kernel Developer’s Guide (BKDG) for AMD Family 15h",
> Section 2.7). However, neither the MSR register nor the PCI function
> at the specified address (D18F3x8C) exists for AMD Family 17h. The
> CoreMasterAccessCtrl register still exists but is now located at a
> different address (see the "Processor Programming Reference (PPR) for
> AMD Family 17h", Section 2.1.8).
> 
> I would be happy to submit a patch to fix this issue. However, since
> the most recent change affecting this functionality appears to be 14
> years old, I would like to confirm whether this is still relevant or
> if the kernel should always be built with CONFIG_PCI_MMCONFIG when ECS
> access is required.
> 
> Linux Kernel info:
> 
> root@qt5222:~# uname -a
> Linux qt5222 6.6.49-2447-qtec-standard--gef032148967a #1 SMP Fri Nov
> 22 09:25:55 UTC 2024 x86_64 GNU/Linux
> 
> Best regards,
> Ros

Hello,

For a longer time I have patches for pciutils project which allows to
query PCIe devices via this AMD ECS I/O API. lspci provides more options
how to query PCI and PCIe devices and one option is raw cf8/cfc I/O
access (via ioport/iopl syscalls), which is useful for debugging
purposes. The disadvantage is that this method provides access only to
PCI registers. And that is why this AMD ECS I/O API can be useful as it
extends lspci raw cf8/cfc I/O to access also PCIe registers.

My implementation in those lspci patches follows the linux kernel logic,
but for unknown reason it do not want to work. Upper cf8 (ECS) bits are
ignored when accessing register value via cfc. Filip retested it today
on AMD 19h family machine, that it really does not work.

Now when I found and read this Rostyslav's email I understood why it
does not work.

Rostyslav, I would like to ask you, do you have patches / updates for
enabling the EnableCf8ExtCfg bit for AMD 17h+ family? I could try to
adjust my lspci changes for new machines.

Anyway, if you are interesting in lspci patches for my WIP AMD ECS, I
can send them.

Pali