Does dwc/pci-layerscape.c support AER?

Does dwc/pci-layerscape.c support AER?

[Bjorn Helgaas]

I see "aer" mentioned in layerscape DT 'interrupt-names':

  $ git grep "interrupt-names.*aer" Documentation/devicetree/bindings/pci/ arch
  Documentation/devicetree/bindings/pci/fsl,layerscape-pcie.yaml: interrupt-names = "aer";
  arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi: interrupt-names = "pme", "aer";
  arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer";
  arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer";
  ...

But I don't know whether or how these are connected to the AER driver
(drivers/pci/pcie/aer.c).

Does the AER driver actually work on these platforms?

Is there some magic that connects the 'interrupt-names'/'interrupts'
DT properties to the pcie_device.irq that aer_probe() requests and
hooks up with the aer_irq() handler?

The pcie_device.irq for AER was assigned by portdrv in this path:

  pcie_portdrv_probe
    pcie_port_device_register(pci_dev *dev)
      int irqs[PCIE_PORT_DEVICE_MAXSERVICES]
      pcie_init_service_irqs(dev, irqs, ...)
        # try MSI/MSI-X first:
        pcie_port_enable_irq_vec(dev, irqs, ...)
          pci_alloc_irq_vectors(PCI_IRQ_MSIX | PCI_IRQ_MSI)
          pcie_message_numbers(dev, mask, &pme, &aer, &dpc)
          irqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, aer)
        # fall back to INTx if no MSI/MSI-X
        pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_INTX);
      for (i = 0; ...; i++)
        pcie_device_init(pdev, irqs[i])
          pcie = kzalloc()            # struct pcie_device
          pcie->irq = irq             # <-- pcie_device.irq

but I only see attempts to use MSI/MSI-X/INTx, which we discover and
configure based on the MSI or MSI-X Capability or the INTx pin
advertised at PCI_INTERRUPT_PIN.

I don't see anything related to DT or platform IRQs that I can connect
with the DT 'interrupts' property.

Bjorn

Re: Does dwc/pci-layerscape.c support AER?

[Brian Norris]

Hi Bjorn,

I'm not so familiar with layerscape, but I'm familiar with parts of
this:

On Wed, Jul 02, 2025 at 05:38:41PM -0500, Bjorn Helgaas wrote:
> I see "aer" mentioned in layerscape DT 'interrupt-names':
> 
>   $ git grep "interrupt-names.*aer" Documentation/devicetree/bindings/pci/ arch
>   Documentation/devicetree/bindings/pci/fsl,layerscape-pcie.yaml: interrupt-names = "aer";
>   arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi: interrupt-names = "pme", "aer";
>   arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer";
>   arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer";
>   ...

NB: in fsl-ls1012a.dtsi at least, the &pcie1 node has 'msi-parent =
<&msi>' that points at a 'fsl,ls1012a-msi'-compatible irqchip. I expect
that's how MSIs are getting resolved, and so that's likely how portdrv
gets its PME IRQ to enable AER among other things.

I think the other SoCs you point at above do something similar, or with
GIC ITS.

> 
> But I don't know whether or how these are connected to the AER driver
> (drivers/pci/pcie/aer.c).
> 
> Does the AER driver actually work on these platforms?
> 
> Is there some magic that connects the 'interrupt-names'/'interrupts'
> DT properties to the pcie_device.irq that aer_probe() requests and
> hooks up with the aer_irq() handler?

I think you're right to notice the disconnect. I believe a typical DWC
implementation does not actually route the root port's PME/AER through
its MSI line, so many SoCs will have a dedicated platform IRQ for them
instead -- but I also don't know of any that actually hook those
platform IRQs up to the AER driver properly [1]. If they want AER, they
do something like commit ("9c4cd0aef259 arm64: dts: qcom: x1e80100:
enable GICv3 ITS for PCIe").

Brian

[1] Although I've seen some try to abuse interrupt-map to pretend that
    their platform-AER interrupt is actually an INTx source... But that
    breaks actually INTx support.

Re: Does dwc/pci-layerscape.c support AER?

[Frank Li]

On Wed, Jul 02, 2025 at 05:38:41PM -0500, Bjorn Helgaas wrote:
> I see "aer" mentioned in layerscape DT 'interrupt-names':
>
>   $ git grep "interrupt-names.*aer" Documentation/devicetree/bindings/pci/ arch
>   Documentation/devicetree/bindings/pci/fsl,layerscape-pcie.yaml: interrupt-names = "aer";
>   arch/arm64/boot/dts/freescale/fsl-ls1012a.dtsi: interrupt-names = "pme", "aer";
>   arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer";
>   arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi: interrupt-names = "pme", "aer";
>   ...
>
> But I don't know whether or how these are connected to the AER driver
> (drivers/pci/pcie/aer.c).
>
> Does the AER driver actually work on these platforms?
>
> Is there some magic that connects the 'interrupt-names'/'interrupts'
> DT properties to the pcie_device.irq that aer_probe() requests and
> hooks up with the aer_irq() handler?
>
> The pcie_device.irq for AER was assigned by portdrv in this path:
>
>   pcie_portdrv_probe
>     pcie_port_device_register(pci_dev *dev)
>       int irqs[PCIE_PORT_DEVICE_MAXSERVICES]
>       pcie_init_service_irqs(dev, irqs, ...)
>         # try MSI/MSI-X first:
>         pcie_port_enable_irq_vec(dev, irqs, ...)
>           pci_alloc_irq_vectors(PCI_IRQ_MSIX | PCI_IRQ_MSI)
>           pcie_message_numbers(dev, mask, &pme, &aer, &dpc)
>           irqs[PCIE_PORT_SERVICE_AER_SHIFT] = pci_irq_vector(dev, aer)
>         # fall back to INTx if no MSI/MSI-X
>         pci_alloc_irq_vectors(dev, 1, 1, PCI_IRQ_INTX);
>       for (i = 0; ...; i++)
>         pcie_device_init(pdev, irqs[i])
>           pcie = kzalloc()            # struct pcie_device
>           pcie->irq = irq             # <-- pcie_device.irq
>
> but I only see attempts to use MSI/MSI-X/INTx, which we discover and
> configure based on the MSI or MSI-X Capability or the INTx pin
> advertised at PCI_INTERRUPT_PIN.
>
> I don't see anything related to DT or platform IRQs that I can connect
> with the DT 'interrupts' property.

There are several attempts to upstream customer Aer irq support in past years.

For example:
  https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1161848.html

some change port drivers.

If you think it is valuable to support customer AER IRQ support, I can restart
this work.

Frank

>
> Bjorn

Re: Does dwc/pci-layerscape.c support AER?

[Brian Norris]

Hi Frank,

On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > Does the AER driver actually work on these platforms?
...
> There are several attempts to upstream customer Aer irq support in past years.
> 
> For example:
>   https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1161848.html
> 
> some change port drivers.
> 
> If you think it is valuable to support customer AER IRQ support, I can restart
> this work.

Interesting thread. I read through it, but I'm still not convinced about
one detail:

Are you sure that AER can't possibly work over MSI? Even today, the
Synopsys manuals say that their integrated MSI receiver "terminate[s]
inbound MSI requests (received on the RX wire)" and after terminating,
"an interrupt is signaled locally through the msi_ctrl_int output."

That means that their msi_ctrl_int signal only handles MSI requests from
downstream functions, and it implies that the default
drivers/pci/controller/dwc/pcie-designware-host.c
dw_pcie_msi_domain_info implementation will not actually see MSIs from
the root port (such as PME and AER). So yes, it *appears* that AER does
not work over MSI.

But crucially, it does *not* mean that the port will not generate valid
MSI requests, if you have some kind of logic that will receive it. So
for instance, I pointed out in another reply that some SoCs choose to
hook up GIC ITS:

 commit 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for
 PCIe")

"""
    Note that using the GIC ITS on x1e80100 will cause Advanced Error
    Reporting (AER) interrupts to be received on errors unlike when using
    the internal MSI controller. Consequently, notifications about
    (correctable) errors may now be logged for errors that previously went
    unnoticed.
"""

And in fact, your arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi seems
to be doing the same. I'd be surprised if these port MSIs still don't
work after that.

OTOH, I do also believe there are SoCs where DWC PCIe is available, but
there is no external MSI controller, and so that same problem still may
exist. I may even have such SoCs available...

Brian

Re: Does dwc/pci-layerscape.c support AER?

[Frank Li]

On Wed, Jul 02, 2025 at 04:44:48PM -0700, Brian Norris wrote:
> Hi Frank,
>
> On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > Does the AER driver actually work on these platforms?
> ...
> > There are several attempts to upstream customer Aer irq support in past years.
> >
> > For example:
> >   https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1161848.html
> >
> > some change port drivers.
> >
> > If you think it is valuable to support customer AER IRQ support, I can restart
> > this work.
>
> Interesting thread. I read through it, but I'm still not convinced about
> one detail:
>
> Are you sure that AER can't possibly work over MSI? Even today, the
> Synopsys manuals say that their integrated MSI receiver "terminate[s]
> inbound MSI requests (received on the RX wire)" and after terminating,
> "an interrupt is signaled locally through the msi_ctrl_int output."
>
> That means that their msi_ctrl_int signal only handles MSI requests from
> downstream functions, and it implies that the default
> drivers/pci/controller/dwc/pcie-designware-host.c
> dw_pcie_msi_domain_info implementation will not actually see MSIs from
> the root port (such as PME and AER). So yes, it *appears* that AER does
> not work over MSI.
>
> But crucially, it does *not* mean that the port will not generate valid
> MSI requests, if you have some kind of logic that will receive it. So
> for instance, I pointed out in another reply that some SoCs choose to
> hook up GIC ITS:
>
>  commit 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for
>  PCIe")
>
> """
>     Note that using the GIC ITS on x1e80100 will cause Advanced Error
>     Reporting (AER) interrupts to be received on errors unlike when using
>     the internal MSI controller. Consequently, notifications about
>     (correctable) errors may now be logged for errors that previously went
>     unnoticed.
> """
>
> And in fact, your arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi seems
> to be doing the same. I'd be surprised if these port MSIs still don't
> work after that.
>
> OTOH, I do also believe there are SoCs where DWC PCIe is available, but
> there is no external MSI controller, and so that same problem still may
> exist. I may even have such SoCs available...

I saw AER and PME irq registed. But I have not seen irq increased. I am not
sure how to inject an error to test it.

I suppose it go through AER irq line.

Frank

>
> Brian

Re: Does dwc/pci-layerscape.c support AER?

[Brian Norris]

Hi Frank,

Thanks for the response.

On Thu, Jul 03, 2025 at 10:22:35PM -0400, Frank Li wrote:
> I saw AER and PME irq registed. But I have not seen irq increased. I am not
> sure how to inject an error to test it.

I've tested AER-like conditions via one of two ways:

1. force asserting PERST#, and then try to read a config register. This
   should generate Complection Timeouts at least, and possibly other
   errors. This method may not necessarily yield AER logs, as it may
   also reset the error reporting registers that the Linux AER driver
   would expect to read. But it probably should still trigger an
   interrupt.
   This depends on having access to PERST#; many SoCs provide this as a
   GPIO which you could potentially control, although I don't see this
   in the layerscape driver at the moment.
2. asserting HOT RESET in the DWC controller. This is especially
   implementation specific, as it depends on how (if at all) the hot
   reset signal is connected into your SoC.

Not sure if any of that helps you for testing. And maybe you want to
wire up your platform IRQs anyway.

Brian

Re: Does dwc/pci-layerscape.c support AER?

[Manivannan Sadhasivam]

On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:

Sorry for jumping late into this thread. I missed it completely as I was not
CCed.

> Hi Frank,
> 
> On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > Does the AER driver actually work on these platforms?
> ...
> > There are several attempts to upstream customer Aer irq support in past years.
> > 
> > For example:
> >   https://www.mail-archive.com/linux-kernel@vger.kernel.org/msg1161848.html
> > 
> > some change port drivers.
> > 
> > If you think it is valuable to support customer AER IRQ support, I can restart
> > this work.
> 
> Interesting thread. I read through it, but I'm still not convinced about
> one detail:
> 
> Are you sure that AER can't possibly work over MSI? Even today, the
> Synopsys manuals say that their integrated MSI receiver "terminate[s]
> inbound MSI requests (received on the RX wire)" and after terminating,
> "an interrupt is signaled locally through the msi_ctrl_int output."
> 
> That means that their msi_ctrl_int signal only handles MSI requests from
> downstream functions, and it implies that the default
> drivers/pci/controller/dwc/pcie-designware-host.c
> dw_pcie_msi_domain_info implementation will not actually see MSIs from
> the root port (such as PME and AER). So yes, it *appears* that AER does
> not work over MSI.
> 
> But crucially, it does *not* mean that the port will not generate valid
> MSI requests, if you have some kind of logic that will receive it. So
> for instance, I pointed out in another reply that some SoCs choose to
> hook up GIC ITS:
> 
>  commit 9c4cd0aef259 ("arm64: dts: qcom: x1e80100: enable GICv3 ITS for
>  PCIe")
> 
> """
>     Note that using the GIC ITS on x1e80100 will cause Advanced Error
>     Reporting (AER) interrupts to be received on errors unlike when using
>     the internal MSI controller. Consequently, notifications about
>     (correctable) errors may now be logged for errors that previously went
>     unnoticed.
> """
> 
> And in fact, your arch/arm64/boot/dts/freescale/fsl-ls1028a.dtsi seems
> to be doing the same. I'd be surprised if these port MSIs still don't
> work after that.
> 
> OTOH, I do also believe there are SoCs where DWC PCIe is available, but
> there is no external MSI controller, and so that same problem still may
> exist. I may even have such SoCs available...
> 

Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation.
And the same should be true for other vendors also.

Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which
is only handled by the controller driver. This is similar to the 'aer' SPI
interrupt in layerscape platforms.

So I think there is an incentive in allowing the AER driver to work with vendor
specific IRQs.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: Does dwc/pci-layerscape.c support AER?

[Brian Norris]

On Wed, Jul 16, 2025 at 12:47:10PM +0530, Manivannan Sadhasivam wrote:
> On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
> > On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > OTOH, I do also believe there are SoCs where DWC PCIe is available, but
> > there is no external MSI controller, and so that same problem still may
> > exist. I may even have such SoCs available...
> > 
> 
> Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation.
> And the same should be true for other vendors also.
> 
> Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which
> is only handled by the controller driver. This is similar to the 'aer' SPI
> interrupt in layerscape platforms.

Yeah, I have some SoCs like this as well. But I also believe that I have
INTx available, and that even when MSI doesn't work for AER/PME, INTx
might.

Do Qcom SoCs route INTx?

> So I think there is an incentive in allowing the AER driver to work with vendor
> specific IRQs.

Yeah, I suppose even if my SoC (and Qcom, depending on the above answer)
might work with INTx, it really does seem like an arbitrary decision
about what SoC makers connected which DWC signals, so I suspect this is
true.

Brian

Re: Does dwc/pci-layerscape.c support AER?

[Manivannan Sadhasivam]

On Wed, Jul 16, 2025 at 08:20:38AM GMT, Brian Norris wrote:
> On Wed, Jul 16, 2025 at 12:47:10PM +0530, Manivannan Sadhasivam wrote:
> > On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
> > > On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > OTOH, I do also believe there are SoCs where DWC PCIe is available, but
> > > there is no external MSI controller, and so that same problem still may
> > > exist. I may even have such SoCs available...
> > > 
> > 
> > Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation.
> > And the same should be true for other vendors also.
> > 
> > Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which
> > is only handled by the controller driver. This is similar to the 'aer' SPI
> > interrupt in layerscape platforms.
> 
> Yeah, I have some SoCs like this as well. But I also believe that I have
> INTx available, and that even when MSI doesn't work for AER/PME, INTx
> might.
> 
> Do Qcom SoCs route INTx?
> 

Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi
cmdline parameter.

> > So I think there is an incentive in allowing the AER driver to work with vendor
> > specific IRQs.
> 
> Yeah, I suppose even if my SoC (and Qcom, depending on the above answer)
> might work with INTx, it really does seem like an arbitrary decision
> about what SoC makers connected which DWC signals, so I suspect this is
> true.
> 

Maybe we should be able to extend the dmi quirk in portdrv.c to allow Root Ports
or host bridge to use INT-X instead of forcing them to use cmdline params.

- Mani

-- 
மணிவண்ணன் சதாசிவம்

Re: Does dwc/pci-layerscape.c support AER?

[Brian Norris]

On Wed, Jul 16, 2025 at 09:35:38PM +0530, Manivannan Sadhasivam wrote:
> On Wed, Jul 16, 2025 at 08:20:38AM GMT, Brian Norris wrote:
> > On Wed, Jul 16, 2025 at 12:47:10PM +0530, Manivannan Sadhasivam wrote:
> > > On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
> > > > On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > > OTOH, I do also believe there are SoCs where DWC PCIe is available, but
> > > > there is no external MSI controller, and so that same problem still may
> > > > exist. I may even have such SoCs available...
> > > > 
> > > 
> > > Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from this limitation.
> > > And the same should be true for other vendors also.
> > > 
> > > Interestingly, the Qcom SoCs route the AER/PME via 'global' SPI interrupt, which
> > > is only handled by the controller driver. This is similar to the 'aer' SPI
> > > interrupt in layerscape platforms.
> > 
> > Yeah, I have some SoCs like this as well. But I also believe that I have
> > INTx available, and that even when MSI doesn't work for AER/PME, INTx
> > might.
> > 
> > Do Qcom SoCs route INTx?
> > 
> 
> Yes, they do. But currently, we can only use it by booting with pcie_pme=nomsi
> cmdline parameter.

Cool, so it sounds like you might be in a better spot than layerscape,
based on the explanations I've seen from them. They seem to require
multiplexing multiple platform-specific interrupts, which isn't as easy
to pretend is a proper INTx line.

> > > So I think there is an incentive in allowing the AER driver to work with vendor
> > > specific IRQs.
> > 
> > Yeah, I suppose even if my SoC (and Qcom, depending on the above answer)
> > might work with INTx, it really does seem like an arbitrary decision
> > about what SoC makers connected which DWC signals, so I suspect this is
> > true.
> > 
> 
> Maybe we should be able to extend the dmi quirk in portdrv.c to allow Root Ports
> or host bridge to use INT-X instead of forcing them to use cmdline params.

Yeah, it sounds like that would be sufficient for non-ITS Qualcomm PCIe,
and also for my SoCs. It's also necessary (but maybe not sufficient) for
Layerscape too.

One complexity: this is not exclusively a "host bridge" quirk. It's also
a quirk of the MSI provider. So for example, looking at
arch/arm64/boot/dts/qcom/x1e80100.dtsi, you have 2 controllers without
GIC ITS, and 2 with GIC ITS, all using the "same" type of bridge.
IIUC, only the former 2 would need to fall back to INTx.

Brian

Re: Does dwc/pci-layerscape.c support AER?

[Bjorn Helgaas]

On Wed, Jul 16, 2025 at 10:25:57AM -0700, Brian Norris wrote:
> On Wed, Jul 16, 2025 at 09:35:38PM +0530, Manivannan Sadhasivam wrote:
> > On Wed, Jul 16, 2025 at 08:20:38AM GMT, Brian Norris wrote:
> > > On Wed, Jul 16, 2025 at 12:47:10PM +0530, Manivannan Sadhasivam wrote:
> > > > On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
> > > > > On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > > > OTOH, I do also believe there are SoCs where DWC PCIe is
> > > > > available, but there is no external MSI controller, and so
> > > > > that same problem still may exist. I may even have such SoCs
> > > > > available...
> > > > 
> > > > Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from
> > > > this limitation.  And the same should be true for other
> > > > vendors also.
> > > > 
> > > > Interestingly, the Qcom SoCs route the AER/PME via 'global'
> > > > SPI interrupt, which is only handled by the controller driver.
> > > > This is similar to the 'aer' SPI interrupt in layerscape
> > > > platforms.
> > > 
> > > Yeah, I have some SoCs like this as well. But I also believe
> > > that I have INTx available, and that even when MSI doesn't work
> > > for AER/PME, INTx might.
> > > 
> > > Do Qcom SoCs route INTx?
> > 
> > Yes, they do. But currently, we can only use it by booting with
> > pcie_pme=nomsi cmdline parameter.

Ugh.  I think these controllers might be out of spec (or maybe we're
not configuring MSI/MSI-X correctly for them).  Per PCIe r7.0, sec
6.1.4.3:

  While enabled for MSI or MSI-X operation, a Function is prohibited
  from using INTx interrupts (if implemented) to request service (MSI,
  MSI-X, and INTx are mutually exclusive).

If the controller advertises MSI or MSI-X, I think we will enable it
and expect AER, PME, hotplug, etc. to use it.

Bjorn

Re: Does dwc/pci-layerscape.c support AER?

[Frank Li]

On Wed, Jul 16, 2025 at 03:42:07PM -0500, Bjorn Helgaas wrote:
> On Wed, Jul 16, 2025 at 10:25:57AM -0700, Brian Norris wrote:
> > On Wed, Jul 16, 2025 at 09:35:38PM +0530, Manivannan Sadhasivam wrote:
> > > On Wed, Jul 16, 2025 at 08:20:38AM GMT, Brian Norris wrote:
> > > > On Wed, Jul 16, 2025 at 12:47:10PM +0530, Manivannan Sadhasivam wrote:
> > > > > On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
> > > > > > On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > > > > OTOH, I do also believe there are SoCs where DWC PCIe is
> > > > > > available, but there is no external MSI controller, and so
> > > > > > that same problem still may exist. I may even have such SoCs
> > > > > > available...
> > > > >
> > > > > Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from
> > > > > this limitation.  And the same should be true for other
> > > > > vendors also.
> > > > >
> > > > > Interestingly, the Qcom SoCs route the AER/PME via 'global'
> > > > > SPI interrupt, which is only handled by the controller driver.
> > > > > This is similar to the 'aer' SPI interrupt in layerscape
> > > > > platforms.
> > > >
> > > > Yeah, I have some SoCs like this as well. But I also believe
> > > > that I have INTx available, and that even when MSI doesn't work
> > > > for AER/PME, INTx might.
> > > >
> > > > Do Qcom SoCs route INTx?
> > >
> > > Yes, they do. But currently, we can only use it by booting with
> > > pcie_pme=nomsi cmdline parameter.
>
> Ugh.  I think these controllers might be out of spec (or maybe we're
> not configuring MSI/MSI-X correctly for them).  Per PCIe r7.0, sec
> 6.1.4.3:
>
>   While enabled for MSI or MSI-X operation, a Function is prohibited
>   from using INTx interrupts (if implemented) to request service (MSI,
>   MSI-X, and INTx are mutually exclusive).
>
> If the controller advertises MSI or MSI-X, I think we will enable it
> and expect AER, PME, hotplug, etc. to use it.

I think it is controller implement problem, which route AER irq to INTx or
other irq line in SOC. Consider many users have similar issues, can you add
workaround or quirk for this type implementation.

Frank
>
> Bjorn

Re: Does dwc/pci-layerscape.c support AER?

[Bjorn Helgaas]

On Wed, Jul 16, 2025 at 05:22:58PM -0400, Frank Li wrote:
> On Wed, Jul 16, 2025 at 03:42:07PM -0500, Bjorn Helgaas wrote:
> > On Wed, Jul 16, 2025 at 10:25:57AM -0700, Brian Norris wrote:
> > > On Wed, Jul 16, 2025 at 09:35:38PM +0530, Manivannan Sadhasivam wrote:
> > > > On Wed, Jul 16, 2025 at 08:20:38AM GMT, Brian Norris wrote:
> > > > > On Wed, Jul 16, 2025 at 12:47:10PM +0530, Manivannan Sadhasivam wrote:
> > > > > > On Wed, Jul 02, 2025 at 04:44:48PM GMT, Brian Norris wrote:
> > > > > > > On Wed, Jul 02, 2025 at 07:09:42PM -0400, Frank Li wrote:
> > > > > > > OTOH, I do also believe there are SoCs where DWC PCIe is
> > > > > > > available, but there is no external MSI controller, and so
> > > > > > > that same problem still may exist. I may even have such SoCs
> > > > > > > available...
> > > > > >
> > > > > > Yes, pretty much all Qcom SoCs without GIC-v3 ITS suffer from
> > > > > > this limitation.  And the same should be true for other
> > > > > > vendors also.
> > > > > >
> > > > > > Interestingly, the Qcom SoCs route the AER/PME via 'global'
> > > > > > SPI interrupt, which is only handled by the controller driver.
> > > > > > This is similar to the 'aer' SPI interrupt in layerscape
> > > > > > platforms.
> > > > >
> > > > > Yeah, I have some SoCs like this as well. But I also believe
> > > > > that I have INTx available, and that even when MSI doesn't work
> > > > > for AER/PME, INTx might.
> > > > >
> > > > > Do Qcom SoCs route INTx?
> > > >
> > > > Yes, they do. But currently, we can only use it by booting with
> > > > pcie_pme=nomsi cmdline parameter.
> >
> > Ugh.  I think these controllers might be out of spec (or maybe we're
> > not configuring MSI/MSI-X correctly for them).  Per PCIe r7.0, sec
> > 6.1.4.3:
> >
> >   While enabled for MSI or MSI-X operation, a Function is prohibited
> >   from using INTx interrupts (if implemented) to request service (MSI,
> >   MSI-X, and INTx are mutually exclusive).
> >
> > If the controller advertises MSI or MSI-X, I think we will enable it
> > and expect AER, PME, hotplug, etc. to use it.
> 
> I think it is controller implement problem, which route AER irq to
> INTx or other irq line in SOC. Consider many users have similar
> issues, can you add workaround or quirk for this type
> implementation.

I think the spec allows the AER interrupt to be routed to INTx, but
only if the Root Port doesn't advertise MSI or MSI-X support or the OS
decides not to enable MSI/MSI-X.

Since there's no generic way to say "MSI/MSI-X works on this Root
Port, but only for certain interrupts, not including AER," there's no
way for Linux to figure out that it shouldn't enable MSI/MSI-X.

*Somebody* can add some kind of quirk for this; not me because I don't
have the hardware.  It needs to be generic so the same solution works
for all these devices.

Bjorn