Re: [RFC] PCI/MSI: Warning observed for NVMe with ACPI

[Thomas Gleixner <tglx@linutronix.de>]

On Fri, Dec 10 2021 at 11:39, Marc Zyngier wrote:
> On Fri, 10 Dec 2021 10:48:22 +0000,
> Jon Hunter <jonathanh@nvidia.com> wrote:
>> +       select PCI_MSI_ARCH_FALLBACKS if PCI_MSI
>>         select POWER_RESET
>>         select POWER_SUPPLY
>>         select SPARSE_IRQ
>
> +Thomas, as he's neck-deep in the MSI rework.
>
> No, this definitely is the wrong solution.

Correct.

> arm64 doesn't need any arch fallback (I actually went out of my way to
> kill them on this architecture), and requires the individual MSI
> controller drivers to do the right thing by using MSI domains.  Adding
> this config option makes the warning disappear, but the core issue is
> that you have a device that doesn't have a MSI domain associated with
> it.
>
> So either your device isn't MSI capable (odd), your host bridge
> doesn't make the link with the MSI controller to advertise the MSI
> domain (this should normally be dealt with via IORT), or there is a
> bug of a similar sort somewhere else.

What's even more odd is:

>> [    2.725479] WARNING: CPU: 0 PID: 94 at include/linux/msi.h:264 free_msi_irqs+0x84/0x188
>> [    2.825275]  free_msi_irqs+0x84/0x188
>> [    2.825288]  __pci_enable_msix_range+0x380/0x530
>> [    2.825299]  pci_alloc_irq_vectors_affinity+0x158/0x168

From __pci_enable_msix_range() there are two ways to reach free_msi_irqs():

1) pci_alloc_irq_vectors_affinity()
     __pci_enable_msix_range()
       __pci_enable_msix()
         msix_capability_init()
           msix_setup_entries()
             alloc_msi_entry(()      -> allocation fail

2) pci_alloc_irq_vectors_affinity()
     __pci_enable_msix_range()
       __pci_enable_msix()
         msix_capability_init()
           pci_msi_setup_msi_irqs(); -> any failure after this succeeded

#1 is unlikely

#2 is odd because if the irqdomain of the device is not hierarchical,
   then the same warning should trigger already in
   pci_msi_setup_msi_irqs() via arch_setup_msi_irqs().

Strange.

Thanks,

        tglx