[BUG report] rdma-ndd: wrong NodeDescription for second device

[BUG report] rdma-ndd: wrong NodeDescription for second device

[Vladimir Chukov]

Description: rdma-ndd sets correct NodeDescription for device which
was initialised first; for the second device description in sysfs is
set correctly, but saquery  to that node  will return default
"NodeDescription.........MT4119 ConnectX5   Mellanox Technologies"

How to reproduce: initramfs-tools 0.130+ (default for buster) - loads
mlx5_core/mlx5_ib in initramfs, ports 1 and 2 are connected to
different fabrics (not to the same switch), as a result very close gap
between mlx5_0 and mlx5_1 netlink events. Then use saquery <lid of the
second port> -C mlx5_1

Workarounds: blacklist mlx5_core and mlx5_ib in initramfs OR connect
both ports to the same fabric OR restart OpenSM (will re-read sysfs)
Non-effective workarounds: systemd rdma-ndd.timer, systemd rdma-ndd
ExecStartPre sleep

Information about your system:
Linux distribution and version: Debian 10.1
Linux kernel and version: 4.19.118-2+deb10u1
InfiniBand hardware and firmware version: MT4119, 16.27.2008
OpenSM version: 3.3.20, 3.3.21

--
Vladimir

Re: [BUG report] rdma-ndd: wrong NodeDescription for second device

[Leon Romanovsky]

On Wed, Jun 17, 2020 at 01:04:15PM +0200, Vladimir Chukov wrote:
> Description: rdma-ndd sets correct NodeDescription for device which
> was initialised first; for the second device description in sysfs is
> set correctly, but saquery  to that node  will return default
> "NodeDescription.........MT4119 ConnectX5   Mellanox Technologies"

What do you see for the second device in NodeDescription?

Thanks

Re: [BUG report] rdma-ndd: wrong NodeDescription for second device

[Vladimir Chukov]

I'm seeing the default description:

sudo saquery 134 -C mlx5_1
NodeRecord dump:
lid.....................134
reserved................0x0
base_version............0x1
class_version...........0x1
node_type...............Channel Adapter
num_ports...............1
sys_guid................0x98039b03006c7912
node_guid...............0x98039b03006c7913
port_guid...............0x98039b03006c7913
partition_cap...........0x80
device_id...............0x1017
revision................0x0
port_num................1
vendor_id...............0x2C9
NodeDescription.........MT4119 ConnectX5   Mellanox Technologies

Thanks,

On Wed, Jun 17, 2020 at 1:17 PM Leon Romanovsky <leon@kernel.org> wrote:
>
> On Wed, Jun 17, 2020 at 01:04:15PM +0200, Vladimir Chukov wrote:
> > Description: rdma-ndd sets correct NodeDescription for device which
> > was initialised first; for the second device description in sysfs is
> > set correctly, but saquery  to that node  will return default
> > "NodeDescription.........MT4119 ConnectX5   Mellanox Technologies"
>
> What do you see for the second device in NodeDescription?
>
> Thanks