mlx5 driver is broken when pci_msix_can_alloc_dyn() is false with v6.4-rc4

[Niklas Schnelle <schnelle@linux.ibm.com>]

Hi Saeed, Eli, Shay,

With v6.4-rc4 I'm getting a stream of RX and TX timeouts when trying to
use ConnectX-4 and ConnectX-6 VFs on s390. I've bisected this and found
the following commit to be broken:

commit 1da438c0ae02396dc5018b63237492cb5908608d
Author: Shay Drory <shayd@nvidia.com>
Date:   Mon Apr 17 10:57:50 2023 +0300

    net/mlx5: Fix indexing of mlx5_irq

    After the cited patch, mlx5_irq xarray index can be different then
    mlx5_irq MSIX table index.
    Fix it by storing both mlx5_irq xarray index and MSIX table index.

    Fixes: 3354822cde5a ("net/mlx5: Use dynamic msix vectors allocation")
    Signed-off-by: Shay Drory <shayd@nvidia.com>
    Reviewed-by: Eli Cohen <elic@nvidia.com>
    Signed-off-by: Saeed Mahameed <saeedm@nvidia.com>

The problem is that our IRQs currently still use a legacy mode instead
of a full fledged IRQ domain. One consequence of that is that
pci_msix_can_alloc_dyn(dev->pdev) returns false. That lands us in the
non dynamic case in mlx5_irq_alloc() where irq->map.index is set to 0.
Now prior to the above commit irq->map.index would later be set to i
(the irq number) but that was replaced with just setting irq-
>pool_index = i. For the dynamic case this is fine because
pci_msix_alloc_irq_at() sets it but for the non-dynamic case this leave
it unset. With the following diff the RX/TX timeouts go away:

diff --git a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
index db5687d9fec9..94dce3735204 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
@@ -237,7 +237,7 @@ struct mlx5_irq *mlx5_irq_alloc(struct
mlx5_irq_pool *pool, int i,
                 * vectors have also been allocated.
                 */
                irq->map.virq = pci_irq_vector(dev->pdev, i);
-               irq->map.index = 0;
+               irq->map.index = i;
        } else {
                irq->map = pci_msix_alloc_irq_at(dev->pdev,
MSI_ANY_INDEX, af_desc);
                if (!irq->map.virq) {



I'll sent a patch with the above shortly but wanted to give you a heads
up since I'd really like to get this fixed for -rc5 or at least -rc6.

Thanks,
Niklas