* mlx5_ib fails probe with old firmware @ 2026-10-09 18:20 Tony Battersby 2026-10-13 9:49 ` Moshe Shemesh 0 siblings, 1 reply; 4+ messages in thread From: Tony Battersby @ 2026-10-09 18:20 UTC (permalink / raw) To: Leon Romanovsky; +Cc: linux-rdma I have a number of Mellanox infiniband adapters out in the field with various firmware versions: lspci 01:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] Subsystem: Mellanox Technologies Device 0018 lspci -n 01:00.0 0207: 15b3:1011 Subsystem: 15b3:0018 Ones with old firmware don't work: mlx5_core 0000:01:00.0: mlx5_pci_vsc_init:66:(pid 731): Failed to get valid vendor specific ID mlx5_core 0000:01:00.0: firmware version: 10.10.5020 mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_ISSI(0x10a) op_mod(0x0) failed, status bad system state(0x4), syndrome (0x408b33), err(-5) mlx5_core 0000:01:00.0: mlx5_core_set_issi:866:(pid 731): Query ISSI is not supported by FW, ISSI is 0 mlx5_core 0000:01:00.0: mlx5_init_clock:1040:(pid 731): invalid device_frequency_khz, aborting HW clock init mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_HCA_VPORT_CONTEXT(0x762) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5), err(-22) infiniband mlx5_0: Couldn't create per-port data mlx5_ib.rdma: probe of mlx5_core.rdma.0 failed with error -12 Ones with new firmware do work: mlx5_core 0000:01:00.0: firmware version: 10.12.780 mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) mlx5_core 0000:01:00.0: mlx5_init_clock:1614:(pid 755): invalid device_frequency_khz, aborting HW clock init mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(1) mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link INIT mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link ACTIVE I can work around the problem with the following change: diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index a457647edacd..4fc903c2a0be 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -3611,7 +3611,7 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u32 port_num, err = mlx5_query_hca_vport_context(dev->mdev, 0, port_num, 0, &rep); - if (err) + if (0 && err) return err; } This code was added in kernel 4.19 by commit b02289b3d60f ("RDMA: Validate grh_required when handling AVs"). A similar problem was fixed in kernel 4.9 by commit f9c14e46748b ("net/mlx5: Fix query ISSI flow"), which is a source of some of the messages with the old firmware. Can anyone suggest a proper fix for this issue? I can upgrade the firmware if necessary, but that would be less convenient than having a fix for the driver. Thanks, Tony Battersby Cybernetics ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: mlx5_ib fails probe with old firmware 2026-10-09 18:20 mlx5_ib fails probe with old firmware Tony Battersby @ 2026-10-13 9:49 ` Moshe Shemesh 2026-10-13 14:07 ` Tony Battersby 0 siblings, 1 reply; 4+ messages in thread From: Moshe Shemesh @ 2026-10-13 9:49 UTC (permalink / raw) To: Tony Battersby, Leon Romanovsky; +Cc: linux-rdma On 10/9/2026 9:20 PM, Tony Battersby wrote: > I have a number of Mellanox infiniband adapters out in the field with > various firmware versions: > > lspci > 01:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] > Subsystem: Mellanox Technologies Device 0018 > > lspci -n > 01:00.0 0207: 15b3:1011 > Subsystem: 15b3:0018 > > Ones with old firmware don't work: > > mlx5_core 0000:01:00.0: mlx5_pci_vsc_init:66:(pid 731): Failed to get valid vendor specific ID > mlx5_core 0000:01:00.0: firmware version: 10.10.5020 > mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) > mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_ISSI(0x10a) op_mod(0x0) failed, status bad system state(0x4), syndrome (0x408b33), err(-5) > mlx5_core 0000:01:00.0: mlx5_core_set_issi:866:(pid 731): Query ISSI is not supported by FW, ISSI is 0 > mlx5_core 0000:01:00.0: mlx5_init_clock:1040:(pid 731): invalid device_frequency_khz, aborting HW clock init > mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_HCA_VPORT_CONTEXT(0x762) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5), err(-22) > infiniband mlx5_0: Couldn't create per-port data > mlx5_ib.rdma: probe of mlx5_core.rdma.0 failed with error -12 > > Ones with new firmware do work: > > mlx5_core 0000:01:00.0: firmware version: 10.12.780 > mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) > mlx5_core 0000:01:00.0: mlx5_init_clock:1614:(pid 755): invalid device_frequency_khz, aborting HW clock init > mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(1) > mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link INIT > mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link ACTIVE > > I can work around the problem with the following change: > > diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c > index a457647edacd..4fc903c2a0be 100644 > --- a/drivers/infiniband/hw/mlx5/main.c > +++ b/drivers/infiniband/hw/mlx5/main.c > @@ -3611,7 +3611,7 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u32 port_num, > > err = mlx5_query_hca_vport_context(dev->mdev, 0, port_num, 0, > &rep); > - if (err) > + if (0 && err) > return err; > } > > This code was added in kernel 4.19 by commit b02289b3d60f ("RDMA: > Validate grh_required when handling AVs"). > > A similar problem was fixed in kernel 4.9 by commit f9c14e46748b > ("net/mlx5: Fix query ISSI flow"), which is a source of some of the > messages with the old firmware. > > Can anyone suggest a proper fix for this issue? I can upgrade the > firmware if necessary, but that would be less convenient than having a > fix for the driver. Hi Tony, Thanks for your report. Please try the following change: diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index a457647edacd..632a45f663f6 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -3605,7 +3605,8 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u32 port_num, if (err) return err; - if (ll == IB_LINK_LAYER_INFINIBAND) { + if (ll == IB_LINK_LAYER_INFINIBAND && + MLX5_CAP_GEN(dev->mdev, ib_virt)) { if (ibdev->type == RDMA_DEVICE_TYPE_SMI) port_num = smi_to_native_portnum(dev, port_num); > > Thanks, > Tony Battersby > Cybernetics > > ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: mlx5_ib fails probe with old firmware 2026-10-13 9:49 ` Moshe Shemesh @ 2026-10-13 14:07 ` Tony Battersby 2026-10-13 15:10 ` Leon Romanovsky 0 siblings, 1 reply; 4+ messages in thread From: Tony Battersby @ 2026-10-13 14:07 UTC (permalink / raw) To: Moshe Shemesh, Leon Romanovsky; +Cc: linux-rdma On 10/13/26 05:49, Moshe Shemesh wrote: > > On 10/9/2026 9:20 PM, Tony Battersby wrote: >> I have a number of Mellanox infiniband adapters out in the field with >> various firmware versions: >> >> lspci >> 01:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] >> Subsystem: Mellanox Technologies Device 0018 >> >> lspci -n >> 01:00.0 0207: 15b3:1011 >> Subsystem: 15b3:0018 >> >> Ones with old firmware don't work: >> >> mlx5_core 0000:01:00.0: mlx5_pci_vsc_init:66:(pid 731): Failed to get valid vendor specific ID >> mlx5_core 0000:01:00.0: firmware version: 10.10.5020 >> mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) >> mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_ISSI(0x10a) op_mod(0x0) failed, status bad system state(0x4), syndrome (0x408b33), err(-5) >> mlx5_core 0000:01:00.0: mlx5_core_set_issi:866:(pid 731): Query ISSI is not supported by FW, ISSI is 0 >> mlx5_core 0000:01:00.0: mlx5_init_clock:1040:(pid 731): invalid device_frequency_khz, aborting HW clock init >> mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_HCA_VPORT_CONTEXT(0x762) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5), err(-22) >> infiniband mlx5_0: Couldn't create per-port data >> mlx5_ib.rdma: probe of mlx5_core.rdma.0 failed with error -12 >> >> Ones with new firmware do work: >> >> mlx5_core 0000:01:00.0: firmware version: 10.12.780 >> mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) >> mlx5_core 0000:01:00.0: mlx5_init_clock:1614:(pid 755): invalid device_frequency_khz, aborting HW clock init >> mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(1) >> mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link INIT >> mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link ACTIVE >> >> I can work around the problem with the following change: >> >> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c >> index a457647edacd..4fc903c2a0be 100644 >> --- a/drivers/infiniband/hw/mlx5/main.c >> +++ b/drivers/infiniband/hw/mlx5/main.c >> @@ -3611,7 +3611,7 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u32 port_num, >> >> err = mlx5_query_hca_vport_context(dev->mdev, 0, port_num, 0, >> &rep); >> - if (err) >> + if (0 && err) >> return err; >> } >> >> This code was added in kernel 4.19 by commit b02289b3d60f ("RDMA: >> Validate grh_required when handling AVs"). >> >> A similar problem was fixed in kernel 4.9 by commit f9c14e46748b >> ("net/mlx5: Fix query ISSI flow"), which is a source of some of the >> messages with the old firmware. >> >> Can anyone suggest a proper fix for this issue? I can upgrade the >> firmware if necessary, but that would be less convenient than having a >> fix for the driver. > Hi Tony, > Thanks for your report. > Please try the following change: > diff --git a/drivers/infiniband/hw/mlx5/main.c > b/drivers/infiniband/hw/mlx5/main.c > index a457647edacd..632a45f663f6 100644 > --- a/drivers/infiniband/hw/mlx5/main.c > +++ b/drivers/infiniband/hw/mlx5/main.c > @@ -3605,7 +3605,8 @@ static int mlx5_port_immutable(struct ib_device > *ibdev, u32 port_num, > if (err) > return err; > > - if (ll == IB_LINK_LAYER_INFINIBAND) { > + if (ll == IB_LINK_LAYER_INFINIBAND && > + MLX5_CAP_GEN(dev->mdev, ib_virt)) { > if (ibdev->type == RDMA_DEVICE_TYPE_SMI) > port_num = smi_to_native_portnum(dev, port_num); > > > That does fix the problem with the old firmware. However, the MLX5_CAP_GEN check triggers on both the old and new firmware (I added a printk to test). If that is intentional, then it looks good. Tested-by: Tony Battersby <tonyb@cybernetics.com> ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: mlx5_ib fails probe with old firmware 2026-10-13 14:07 ` Tony Battersby @ 2026-10-13 15:10 ` Leon Romanovsky 0 siblings, 0 replies; 4+ messages in thread From: Leon Romanovsky @ 2026-10-13 15:10 UTC (permalink / raw) To: Tony Battersby; +Cc: Moshe Shemesh, linux-rdma On Tue, Oct 13, 2026 at 10:07:17AM -0400, Tony Battersby wrote: > On 10/13/26 05:49, Moshe Shemesh wrote: > > > > On 10/9/2026 9:20 PM, Tony Battersby wrote: > >> I have a number of Mellanox infiniband adapters out in the field with > >> various firmware versions: > >> > >> lspci > >> 01:00.0 Infiniband controller: Mellanox Technologies MT27600 [Connect-IB] > >> Subsystem: Mellanox Technologies Device 0018 > >> > >> lspci -n > >> 01:00.0 0207: 15b3:1011 > >> Subsystem: 15b3:0018 > >> > >> Ones with old firmware don't work: > >> > >> mlx5_core 0000:01:00.0: mlx5_pci_vsc_init:66:(pid 731): Failed to get valid vendor specific ID > >> mlx5_core 0000:01:00.0: firmware version: 10.10.5020 > >> mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) > >> mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_ISSI(0x10a) op_mod(0x0) failed, status bad system state(0x4), syndrome (0x408b33), err(-5) > >> mlx5_core 0000:01:00.0: mlx5_core_set_issi:866:(pid 731): Query ISSI is not supported by FW, ISSI is 0 > >> mlx5_core 0000:01:00.0: mlx5_init_clock:1040:(pid 731): invalid device_frequency_khz, aborting HW clock init > >> mlx5_core 0000:01:00.0: mlx5_cmd_out_err:810:(pid 731): QUERY_HCA_VPORT_CONTEXT(0x762) op_mod(0x0) failed, status bad operation(0x2), syndrome (0x3590f5), err(-22) > >> infiniband mlx5_0: Couldn't create per-port data > >> mlx5_ib.rdma: probe of mlx5_core.rdma.0 failed with error -12 > >> > >> Ones with new firmware do work: > >> > >> mlx5_core 0000:01:00.0: firmware version: 10.12.780 > >> mlx5_core 0000:01:00.0: 63.008 Gb/s available PCIe bandwidth (8.0 GT/s PCIe x8 link) > >> mlx5_core 0000:01:00.0: mlx5_init_clock:1614:(pid 755): invalid device_frequency_khz, aborting HW clock init > >> mlx5_core 0000:01:00.0: Flow counters bulk query buffer size increased, bulk_query_len(1) > >> mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link INIT > >> mlx5_core 0000:01:00.0 mlx5_0: Port: 1 Link ACTIVE > >> > >> I can work around the problem with the following change: > >> > >> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c > >> index a457647edacd..4fc903c2a0be 100644 > >> --- a/drivers/infiniband/hw/mlx5/main.c > >> +++ b/drivers/infiniband/hw/mlx5/main.c > >> @@ -3611,7 +3611,7 @@ static int mlx5_port_immutable(struct ib_device *ibdev, u32 port_num, > >> > >> err = mlx5_query_hca_vport_context(dev->mdev, 0, port_num, 0, > >> &rep); > >> - if (err) > >> + if (0 && err) > >> return err; > >> } > >> > >> This code was added in kernel 4.19 by commit b02289b3d60f ("RDMA: > >> Validate grh_required when handling AVs"). > >> > >> A similar problem was fixed in kernel 4.9 by commit f9c14e46748b > >> ("net/mlx5: Fix query ISSI flow"), which is a source of some of the > >> messages with the old firmware. > >> > >> Can anyone suggest a proper fix for this issue? I can upgrade the > >> firmware if necessary, but that would be less convenient than having a > >> fix for the driver. > > Hi Tony, > > Thanks for your report. > > Please try the following change: > > diff --git a/drivers/infiniband/hw/mlx5/main.c > > b/drivers/infiniband/hw/mlx5/main.c > > index a457647edacd..632a45f663f6 100644 > > --- a/drivers/infiniband/hw/mlx5/main.c > > +++ b/drivers/infiniband/hw/mlx5/main.c > > @@ -3605,7 +3605,8 @@ static int mlx5_port_immutable(struct ib_device > > *ibdev, u32 port_num, > > if (err) > > return err; > > > > - if (ll == IB_LINK_LAYER_INFINIBAND) { > > + if (ll == IB_LINK_LAYER_INFINIBAND && > > + MLX5_CAP_GEN(dev->mdev, ib_virt)) { > > if (ibdev->type == RDMA_DEVICE_TYPE_SMI) > > port_num = smi_to_native_portnum(dev, port_num); > > > > > > > That does fix the problem with the old firmware. However, > the MLX5_CAP_GEN check triggers on both the old and new firmware (I > added a printk to test). If that is intentional, then it looks good. > > Tested-by: Tony Battersby <tonyb@cybernetics.com> Thanks, Moshe will submit official patch. > > ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2026-10-13 15:10 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-10-09 18:20 mlx5_ib fails probe with old firmware Tony Battersby 2026-10-13 9:49 ` Moshe Shemesh 2026-10-13 14:07 ` Tony Battersby 2026-10-13 15:10 ` Leon Romanovsky
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.