Re: [PATCH v2] PCI: dwc: endpoint: Fix dw_pcie_ep_raise_msix_irq() alignment support

[Bjorn Helgaas <helgaas@kernel.org>]

On Tue, Nov 28, 2023 at 02:22:30PM +0100, Niklas Cassel wrote:
> From: Niklas Cassel <niklas.cassel@wdc.com>
> 
> Commit 6f5e193bfb55 ("PCI: dwc: Fix dw_pcie_ep_raise_msix_irq() to get
> correct MSI-X table address") modified dw_pcie_ep_raise_msix_irq() to
> support iATUs which require a specific alignment.
> 
> However, this support cannot have been properly tested.
> 
> The whole point is for the iATU to map an address that is aligned,
> using dw_pcie_ep_map_addr(), and then let the writel() write to
> ep->msi_mem + aligned_offset.
> 
> Thus, modify the address that is mapped such that it is aligned.
> With this change, dw_pcie_ep_raise_msix_irq() matches the logic in
> dw_pcie_ep_raise_msi_irq().

Was there a problem report for this?  Since 6f5e193bfb55 appeared in
v5.7 (May 31 2020), and this should affect imx6, keystone am654,
dw-pcie (platform), and keembay, it seems a little weird that this bug
persisted so long.  Maybe nobody really uses endpoint support yet?

But I assume you observed a failure and tested this fix somewhere.

And the failure is that we send the wrong MSI-X vector or something
and get an unexpected IRQ and a driver hang or something?  In other
words, how does the bug manifest to users?

> Cc: Kishon Vijay Abraham I <kishon@kernel.org>
> Fixes: 6f5e193bfb55 ("PCI: dwc: Fix dw_pcie_ep_raise_msix_irq() to get correct MSI-X table address")
> Signed-off-by: Niklas Cassel <niklas.cassel@wdc.com>
> ---
> Changes since v1:
> -Clarified commit message.
> -Add a working email for Kishon to CC.
> 
>  drivers/pci/controller/dwc/pcie-designware-ep.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
> index f6207989fc6a..bc94d7f39535 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-ep.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
> @@ -615,6 +615,7 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
>  	}
>  
>  	aligned_offset = msg_addr & (epc->mem->window.page_size - 1);
> +	msg_addr &= ~aligned_offset;
>  	ret = dw_pcie_ep_map_addr(epc, func_no, 0, ep->msi_mem_phys, msg_addr,
>  				  epc->mem->window.page_size);

Total tangent and I don't know enough to suggest any changes, but it's
a little strange as a reader that we want to write to ep->msi_mem, and
the ATU setup with dw_pcie_ep_map_addr() doesn't involve ep->msi_mem
at all.

I see that ep->msi_mem is allocated and ioremapped far away in
dw_pcie_ep_init().  It's just a little weird that there's no
connection *here* with ep->msi_mem.

I assume dw_pcie_ep_map_addr(), writel(), dw_pcie_ep_unmap_addr() have
to happen atomically so nobody else uses that piece of the ATU while
we're doing this?  There's no mutex here, so I guess we must know this
is atomic already because of something else?

Bjorn