Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 3191 bytes --]

Hello Francesco,

[for the new-comers: This is about a regression in 6.11. Details
available at https://bugs.debian.org/1086520. The TL;DR; is that on
6.10.11 opensm works as expected, while it fails to start on 6.11.7.]

On Mon, Nov 18, 2024 at 08:06:16PM +0100, Francesco Poli wrote:
> On Mon, 18 Nov 2024 09:58:03 +0100 Uwe Kleine-König wrote:
> 
> [...]
> > On Wed, Nov 13, 2024 at 11:15:03PM +0100, Francesco Poli wrote:
> > > On Mon, 11 Nov 2024 11:22:26 +0100 Uwe Kleine-König wrote:
> [...]
> > > > I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
> > > > its contents look different on 6.10.x and 6.11.x?
> > > 
> > > I will look into this as soon as I can reboot the cluster head node.
> 
> I looked into this, while testing the new Debian Linux kernel that has
> just migrated to testing (which, once again, makes opensm fail to
> start, just like other 6.11.x versions).
> 
> With a working kernel:
> 
>   $ uname -v
>   #1 SMP PREEMPT_DYNAMIC Debian 6.10.11-1 (2024-09-22)
>   $ ls -altrF /sys/class/infiniband_mad/
>   total 0
>   lrwxrwxrwx  1 root root    0 Nov  4 15:58 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
>   lrwxrwxrwx  1 root root    0 Nov  4 15:58 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
>   lrwxrwxrwx  1 root root    0 Nov 11 15:54 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
>   lrwxrwxrwx  1 root root    0 Nov 11 15:54 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
>   drwxr-xr-x  2 root root    0 Nov 11 15:54 ./
>   drwxr-xr-x 72 root root    0 Nov 11 15:54 ../
>   -r--r--r--  1 root root 4096 Nov 11 15:54 abi_version
>   $ cat /sys/class/infiniband_mad/abi_version 
>   5
> 
> With a kernel that makes opensm fail to start:
> 
>   $ uname -v
>   #1 SMP PREEMPT_DYNAMIC Debian 6.11.7-1 (2024-11-09)
>   $ ls -altrF /sys/class/infiniband_mad/
>   total 0
>   drwxr-xr-x 73 root root    0 Nov 18 09:41 ../
>   -r--r--r--  1 root root 4096 Nov 18 09:41 abi_version
>   lrwxrwxrwx  1 root root    0 Nov 18 09:41 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
>   lrwxrwxrwx  1 root root    0 Nov 18 09:41 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
>   drwxr-xr-x  2 root root    0 Nov 18 09:43 ./
>   $ cat /sys/class/infiniband_mad/abi_version
>   5
> 
> As you can see, a couple of files (symlinks) are missing here...

It looks like the commit that is biting you is

https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d

So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.

I don't know about infiniband, but I'd say: Either your machine doesn't
have these issmX devices and opensm should cope with that, or these
issmX devices are available then
50660c5197f52b8137e223dc3ba8d43661179a1d is buggy.

> Does this ring a bell?

It doesn't for me, but maybe Mark Zhang or someone else among the new
recipients has an idea?

Best regards
Uwe


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Francesco Poli]

[-- Attachment #1: Type: text/plain, Size: 4025 bytes --]

On Thu, 21 Nov 2024 11:04:13 +0100 Uwe Kleine-König wrote:

[...]
> It looks like the commit that is biting you is
> 
> https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d
> 
> So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
> parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.

I started to bisect.

The first surprise is that 50660c5197f52b8137e223dc3ba8d43661179a1d is
good...   :-o

  $ git checkout 50660c5197f52b8137e223dc3ba8d43661179a1d
  $ make -j 12 my_defconfig bindeb-pkg

  [install and reboot with this kernel version]

  # ls /sys/class/infiniband_mad/ -altrF
  total 0
  drwxr-xr-x 70 root root    0 Nov 25 12:05 ../
  -r--r--r--  1 root root 4096 Nov 25 12:05 abi_version
  lrwxrwxrwx  1 root root    0 Nov 25 12:05 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Nov 25 12:05 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  lrwxrwxrwx  1 root root    0 Nov 25 12:08 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
  lrwxrwxrwx  1 root root    0 Nov 25 12:08 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
  drwxr-xr-x  2 root root    0 Nov 25 12:08 ./

  [InfiniBand works]

  $ git bisect start
  $ git bisect good
  $ git checkout v6.11
  $ make -j 12 my_defconfig bindeb-pkg

  [install and reboot with this kernel version]

  # ls /sys/class/infiniband_mad/ -altrF
  total 0
  drwxr-xr-x 70 root root    0 Nov 25 12:29 ../
  -r--r--r--  1 root root 4096 Nov 25 12:29 abi_version
  lrwxrwxrwx  1 root root    0 Nov 25 12:29 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Nov 25 12:29 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  drwxr-xr-x  2 root root    0 Nov 25 12:30 ./

  [InfiniBand fails, because OpenSM fails to start]

  $ git bisect bad
  Bisecting: 7036 revisions left to test after this (roughly 13 steps)
  [b3ce7a30847a54a7f96a35e609303d8afecd460b] Merge tag 'drm-next-2024-07-18' of https://gitlab.freedesktop.org/drm/kernel
  $ make -j 12 my_defconfig bindeb-pkg


Woooha, 13 steps are a lot...

I went on until 10 steps are left:

  [test b3ce7a30847a54a7f96a35e609303d8afecd460b]
  $ git bisect good
  Bisecting: 3385 revisions left to test after this (roughly 12 steps)
  [fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c] Merge tag 'mm-stable-2024-07-21-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
  
  [test fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c]
  $ git bisect bad
  Bisecting: 1763 revisions left to test after this (roughly 11 steps)
  [09ea8089abb5d851ce08a9b1a43706e42ef39db2] Merge tag 'staging-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging

  [test 09ea8089abb5d851ce08a9b1a43706e42ef39db2]
  $ git bisect bad
  Bisecting: 910 revisions left to test after this (roughly 10 steps)
  [4305ca0087dd99c3c3e0e2ac8a228b7e53a21c78] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi


Since I could not afford to keep the cluster out of service any longer
(each step takes at least 20 or 25 minutes: build + install + reboot +
check InfiniBand), I decided to return the cluster to service.

I will try to continue to bisect by testing the resulting kernels on a
compute node: there's no OpenSM there and it cannot run anyway, if
there's another OpenSM on the same InfiniBand network.
However, I can check whether those issm* symlinks are created in
/sys/class/infiniband_mad/ 
I really hope that this is enough to pinpoint the first bad
commit...

Any better ideas?


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Leon Romanovsky]

On Mon, Nov 25, 2024 at 07:54:43PM +0100, Francesco Poli wrote:
> On Thu, 21 Nov 2024 11:04:13 +0100 Uwe Kleine-König wrote:
> 
> [...]
> > It looks like the commit that is biting you is
> > 
> > https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d
> > 
> > So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
> > parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.
> 
> I started to bisect.
> 
> The first surprise is that 50660c5197f52b8137e223dc3ba8d43661179a1d is
> good...   :-o

It is good news, as I looked on it all that time from the day Uwe
reported it.

> 

<...>

> I will try to continue to bisect by testing the resulting kernels on a
> compute node: there's no OpenSM there and it cannot run anyway, if
> there's another OpenSM on the same InfiniBand network.
> However, I can check whether those issm* symlinks are created in
> /sys/class/infiniband_mad/ 
> I really hope that this is enough to pinpoint the first bad
> commit...

Yes, these symlinks should be there. Your test scenario is correct one.

> 
> Any better ideas?

I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
is the one which is causing to troubles, which leads me to suspect FW.

Thanks

> 
> 
> -- 
>  http://www.inventati.org/frx/
>  There's not a second to spare! To the laboratory!
> ..................................................... Francesco Poli .
>  GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Mark Zhang]

On 11/26/2024 3:38 AM, Leon Romanovsky wrote:
> On Mon, Nov 25, 2024 at 07:54:43PM +0100, Francesco Poli wrote:
>> On Thu, 21 Nov 2024 11:04:13 +0100 Uwe Kleine-König wrote:
>>
>> [...]
>>> It looks like the commit that is biting you is
>>>
>>> https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d
>>>
>>> So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
>>> parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.
>>
>> I started to bisect.
>>
>> The first surprise is that 50660c5197f52b8137e223dc3ba8d43661179a1d is
>> good...   :-o
> 
> It is good news, as I looked on it all that time from the day Uwe
> reported it.
> 
>>
> 
> <...>
> 
>> I will try to continue to bisect by testing the resulting kernels on a
>> compute node: there's no OpenSM there and it cannot run anyway, if
>> there's another OpenSM on the same InfiniBand network.
>> However, I can check whether those issm* symlinks are created in
>> /sys/class/infiniband_mad/
>> I really hope that this is enough to pinpoint the first bad
>> commit...
> 
> Yes, these symlinks should be there. Your test scenario is correct one.
> 
>>
>> Any better ideas?
> 
> I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
> is the one which is causing to troubles, which leads me to suspect FW.
> 

Yes looks like FW reports vport.num_plane > 0. What is your hw type and 
FW version ("ethtool -i <netdev_of_the_ibdev>")? I don't think it 
supports multiplane.

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Francesco Poli]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

On Tue, 26 Nov 2024 09:21:37 +0800 Mark Zhang wrote:

[...]
> Yes looks like FW reports vport.num_plane > 0. What is your hw type and 
> FW version ("ethtool -i <netdev_of_the_ibdev>")? I don't think it 
> supports multiplane.

  $ /sbin/ethtool -i ibp129s0f0
  driver: mlx5_core[ib_ipoib]
  version: 6.10.11-amd64
  firmware-version: 20.40.1000 (MT_0000000224)
  expansion-rom-version: 
  bus-info: 0000:81:00.0
  supports-statistics: yes
  supports-test: yes
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: yes

Please note that I determined <netdev_of_the_ibdev> by looking at
the output of 'ibv_devices': I hope this is a correct way to answer
your question.




-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Leon Romanovsky]

On Tue, Nov 26, 2024 at 08:18:24AM +0100, Francesco Poli wrote:
> On Tue, 26 Nov 2024 09:21:37 +0800 Mark Zhang wrote:
> 
> [...]
> > Yes looks like FW reports vport.num_plane > 0. What is your hw type and 
> > FW version ("ethtool -i <netdev_of_the_ibdev>")? I don't think it 
> > supports multiplane.
> 
>   $ /sbin/ethtool -i ibp129s0f0
>   driver: mlx5_core[ib_ipoib]
>   version: 6.10.11-amd64
>   firmware-version: 20.40.1000 (MT_0000000224)
>   expansion-rom-version: 
>   bus-info: 0000:81:00.0
>   supports-statistics: yes
>   supports-test: yes
>   supports-eeprom-access: no
>   supports-register-dump: no
>   supports-priv-flags: yes
> 
> Please note that I determined <netdev_of_the_ibdev> by looking at
> the output of 'ibv_devices': I hope this is a correct way to answer
> your question.

We forwarded this information to FW team and will update you on the
findings.

Thanks

> 
> 
> 
> 
> -- 
>  http://www.inventati.org/frx/
>  There's not a second to spare! To the laboratory!
> ..................................................... Francesco Poli .
>  GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Leon Romanovsky]

On Tue, Nov 26, 2024 at 10:38:59AM +0200, Leon Romanovsky wrote:
> On Tue, Nov 26, 2024 at 08:18:24AM +0100, Francesco Poli wrote:
> > On Tue, 26 Nov 2024 09:21:37 +0800 Mark Zhang wrote:
> > 
> > [...]
> > > Yes looks like FW reports vport.num_plane > 0. What is your hw type and 
> > > FW version ("ethtool -i <netdev_of_the_ibdev>")? I don't think it 
> > > supports multiplane.
> > 
> >   $ /sbin/ethtool -i ibp129s0f0
> >   driver: mlx5_core[ib_ipoib]
> >   version: 6.10.11-amd64
> >   firmware-version: 20.40.1000 (MT_0000000224)
> >   expansion-rom-version: 
> >   bus-info: 0000:81:00.0
> >   supports-statistics: yes
> >   supports-test: yes
> >   supports-eeprom-access: no
> >   supports-register-dump: no
> >   supports-priv-flags: yes
> > 
> > Please note that I determined <netdev_of_the_ibdev> by looking at
> > the output of 'ibv_devices': I hope this is a correct way to answer
> > your question.
> 
> We forwarded this information to FW team and will update you on the
> findings.

Francesco, 

Please update NICs FW to the latest version. In your FW version there
is a bug which causes to return vport.num_plane == 1 even if NIC doesn't
support multiplane mode.

We will continue to work internally to find a solution, which won't require
FW upgrade.

Thanks

> 
> Thanks
> 
> > 
> > 
> > 
> > 
> > -- 
> >  http://www.inventati.org/frx/
> >  There's not a second to spare! To the laboratory!
> > ..................................................... Francesco Poli .
> >  GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE
> 
> 
> 

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Francesco Poli]

[-- Attachment #1: Type: text/plain, Size: 5079 bytes --]

On Mon, 25 Nov 2024 21:38:37 +0200 Leon Romanovsky wrote:

> On Mon, Nov 25, 2024 at 07:54:43PM +0100, Francesco Poli wrote:
[...]
> > I will try to continue to bisect by testing the resulting kernels on a
> > compute node: there's no OpenSM there and it cannot run anyway, if
> > there's another OpenSM on the same InfiniBand network.
> > However, I can check whether those issm* symlinks are created in
> > /sys/class/infiniband_mad/ 
> > I really hope that this is enough to pinpoint the first bad
> > commit...
> 
> Yes, these symlinks should be there. Your test scenario is correct one.

OK, I have completed the bisect on a compute node without OpenSM, by
looking at the issm* symlinks, as I said.

See below.

> 
> > 
> > Any better ideas?
> 
> I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
> is the one which is causing to troubles, which leads me to suspect FW.
[...]

Thanks to your guess about the possibly troublesome commit, the bisect was completed in a few steps:

  $ git checkout 2a5db20fa532
  $ make -j 12 my_defconfig bindeb-pkg
  
  [install this version on a compute node test image and reboot
  one compute node with that image: the InfiniBand network was
  working for that node, that's no surprise, since OpenSM was running
  on the head node, but no issm* symlink was created; please note
  that, surprisingly, the Ethernet network was not working, I mean
  that the Ethernet interfaces were not found by the kernel...]
  
  root@node # ls -altrF /sys/class/infiniband_mad/
  total 0
  drwxr-xr-x 60 root root    0 Nov 26 17:06 ../
  lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
  -r--r--r--  1 root root 4096 Nov 26 17:06 abi_version
  lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
  drwxr-xr-x  2 root root    0 Nov 26 17:08 ./
  
  $ git bisect bad
  Bisecting: 0 revisions left to test after this (roughly 0 steps)
  [65528cfb21fdb68de8ae6dccae19af180d93e143] net/mlx5: mlx5_ifc update for multi-plane support
  $ make -j 12 my_defconfig bindeb-pkg
  
  [install this version on the compute node test image and reboot
  one compute node with that image: the InfiniBand network again
  working for that node, issm* symlinks were created;
  Ethernet network again not working for that node...]
  
  root@node # ls -altrF /sys/class/infiniband_mad/
  total 0
  drwxr-xr-x 60 root root    0 Nov 26 17:31 ../
  lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
  -r--r--r--  1 root root 4096 Nov 26 17:31 abi_version
  lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
  lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/issm1/
  lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/issm0/
  drwxr-xr-x  2 root root    0 Nov 26 17:36 ./
  
  $ git bisect good
  2a5db20fa532198639671713c6213f96ff285b85 is the first bad commit
  commit 2a5db20fa532198639671713c6213f96ff285b85
  Author: Mark Zhang <markzhang@nvidia.com>
  Date:   Sun Jun 16 19:08:35 2024 +0300
  
      RDMA/mlx5: Add support to multi-plane device and port
  
      When multi-plane is supported, a logical port, which is aggregation of
      multiple physical plane ports, is exposed for data transmission.
      Compared with a normal mlx5 IB port, this logical port supports all
      functionalities except Subnet Management.
  
      Signed-off-by: Mark Zhang <markzhang@nvidia.com>
      Link: https://lore.kernel.org/r/7e37c06c9cb243be9ac79930cd17053903785b95.1718553901.git.leon@kernel.org
      Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
  
   drivers/infiniband/hw/mlx5/main.c               | 60 +++++++++++++++++++++----
   drivers/infiniband/hw/mlx5/mlx5_ib.h            |  2 +
   drivers/net/ethernet/mellanox/mlx5/core/vport.c |  1 +
   include/linux/mlx5/driver.h                     |  1 +
   4 files changed, 55 insertions(+), 9 deletions(-)


In other words, bingo!, your guess looks correct, the first bad commit
is the one you mentioned.


Now, I will try to upgrade the firmware of the InfiniBand NICs, as you
suggested, and check whether this solves the issue with the recent
Linux kernel versions.

Please confirm that the procedure to be followed is the one described in
<https://docs.nvidia.com/networking/display/ubuntu2204/firmware+burning>

Thanks for your time and patience, and for all the help you are kindly
providing!   :-)


-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Leon Romanovsky]

On Wed, Nov 27, 2024 at 06:48:03PM +0100, Francesco Poli wrote:
> On Mon, 25 Nov 2024 21:38:37 +0200 Leon Romanovsky wrote:
> 
> > On Mon, Nov 25, 2024 at 07:54:43PM +0100, Francesco Poli wrote:
> [...]
> > > I will try to continue to bisect by testing the resulting kernels on a
> > > compute node: there's no OpenSM there and it cannot run anyway, if
> > > there's another OpenSM on the same InfiniBand network.
> > > However, I can check whether those issm* symlinks are created in
> > > /sys/class/infiniband_mad/ 
> > > I really hope that this is enough to pinpoint the first bad
> > > commit...
> > 
> > Yes, these symlinks should be there. Your test scenario is correct one.
> 
> OK, I have completed the bisect on a compute node without OpenSM, by
> looking at the issm* symlinks, as I said.
> 
> See below.
> 
> > 
> > > 
> > > Any better ideas?
> > 
> > I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
> > is the one which is causing to troubles, which leads me to suspect FW.
> [...]
> 
> Thanks to your guess about the possibly troublesome commit, the bisect was completed in a few steps:
> 
>   $ git checkout 2a5db20fa532
>   $ make -j 12 my_defconfig bindeb-pkg
>   
>   [install this version on a compute node test image and reboot
>   one compute node with that image: the InfiniBand network was
>   working for that node, that's no surprise, since OpenSM was running
>   on the head node, but no issm* symlink was created; please note
>   that, surprisingly, the Ethernet network was not working, I mean
>   that the Ethernet interfaces were not found by the kernel...]
>   
>   root@node # ls -altrF /sys/class/infiniband_mad/
>   total 0
>   drwxr-xr-x 60 root root    0 Nov 26 17:06 ../
>   lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
>   -r--r--r--  1 root root 4096 Nov 26 17:06 abi_version
>   lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
>   drwxr-xr-x  2 root root    0 Nov 26 17:08 ./
>   
>   $ git bisect bad
>   Bisecting: 0 revisions left to test after this (roughly 0 steps)
>   [65528cfb21fdb68de8ae6dccae19af180d93e143] net/mlx5: mlx5_ifc update for multi-plane support
>   $ make -j 12 my_defconfig bindeb-pkg
>   
>   [install this version on the compute node test image and reboot
>   one compute node with that image: the InfiniBand network again
>   working for that node, issm* symlinks were created;
>   Ethernet network again not working for that node...]
>   
>   root@node # ls -altrF /sys/class/infiniband_mad/
>   total 0
>   drwxr-xr-x 60 root root    0 Nov 26 17:31 ../
>   lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
>   -r--r--r--  1 root root 4096 Nov 26 17:31 abi_version
>   lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
>   lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/issm1/
>   lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/issm0/
>   drwxr-xr-x  2 root root    0 Nov 26 17:36 ./
>   
>   $ git bisect good
>   2a5db20fa532198639671713c6213f96ff285b85 is the first bad commit
>   commit 2a5db20fa532198639671713c6213f96ff285b85
>   Author: Mark Zhang <markzhang@nvidia.com>
>   Date:   Sun Jun 16 19:08:35 2024 +0300
>   
>       RDMA/mlx5: Add support to multi-plane device and port
>   
>       When multi-plane is supported, a logical port, which is aggregation of
>       multiple physical plane ports, is exposed for data transmission.
>       Compared with a normal mlx5 IB port, this logical port supports all
>       functionalities except Subnet Management.
>   
>       Signed-off-by: Mark Zhang <markzhang@nvidia.com>
>       Link: https://lore.kernel.org/r/7e37c06c9cb243be9ac79930cd17053903785b95.1718553901.git.leon@kernel.org
>       Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
>   
>    drivers/infiniband/hw/mlx5/main.c               | 60 +++++++++++++++++++++----
>    drivers/infiniband/hw/mlx5/mlx5_ib.h            |  2 +
>    drivers/net/ethernet/mellanox/mlx5/core/vport.c |  1 +
>    include/linux/mlx5/driver.h                     |  1 +
>    4 files changed, 55 insertions(+), 9 deletions(-)
> 
> 
> In other words, bingo!, your guess looks correct, the first bad commit
> is the one you mentioned.
> 
> 
> Now, I will try to upgrade the firmware of the InfiniBand NICs, as you
> suggested, and check whether this solves the issue with the recent
> Linux kernel versions.
> 
> Please confirm that the procedure to be followed is the one described in
> <https://docs.nvidia.com/networking/display/ubuntu2204/firmware+burning>

Yes, it looks correct procedure.
If you didn't upgrade FW, this diff will achieve same result for you:

diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index c2314797afc9..110ce177c305 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -2846,7 +2846,7 @@ static int mlx5_ib_get_plane_num(struct mlx5_core_dev *mdev, u8 *num_plane)
        if (err)
                return err;

-       *num_plane = vport_ctx.num_plane;
+       *num_plane = (vport_ctx.num_plane > 1) ? vport_ctx.num_plane : 0;
        return 0;
 }

The culprit of your issue that in some FW versions, the vport_ctx.num_plane
was 1 and not 0 for devices which don't support that mode, while for the driver
everything that is not 0 means supported.

Thanks

> 
> Thanks for your time and patience, and for all the help you are kindly
> providing!   :-)
> 
> 
> -- 
>  http://www.inventati.org/frx/
>  There's not a second to spare! To the laboratory!
> ..................................................... Francesco Poli .
>  GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Uwe Kleine-König]

[-- Attachment #1: Type: text/plain, Size: 6406 bytes --]

Hello Francesco,

On Wed, Nov 27, 2024 at 10:04:13PM +0200, Leon Romanovsky wrote:
> On Wed, Nov 27, 2024 at 06:48:03PM +0100, Francesco Poli wrote:
> > On Mon, 25 Nov 2024 21:38:37 +0200 Leon Romanovsky wrote:
> > 
> > > On Mon, Nov 25, 2024 at 07:54:43PM +0100, Francesco Poli wrote:
> > [...]
> > > > I will try to continue to bisect by testing the resulting kernels on a
> > > > compute node: there's no OpenSM there and it cannot run anyway, if
> > > > there's another OpenSM on the same InfiniBand network.
> > > > However, I can check whether those issm* symlinks are created in
> > > > /sys/class/infiniband_mad/ 
> > > > I really hope that this is enough to pinpoint the first bad
> > > > commit...
> > > 
> > > Yes, these symlinks should be there. Your test scenario is correct one.
> > 
> > OK, I have completed the bisect on a compute node without OpenSM, by
> > looking at the issm* symlinks, as I said.
> > 
> > See below.
> > 
> > > 
> > > > 
> > > > Any better ideas?
> > > 
> > > I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
> > > is the one which is causing to troubles, which leads me to suspect FW.
> > [...]
> > 
> > Thanks to your guess about the possibly troublesome commit, the bisect was completed in a few steps:
> > 
> >   $ git checkout 2a5db20fa532
> >   $ make -j 12 my_defconfig bindeb-pkg
> >   
> >   [install this version on a compute node test image and reboot
> >   one compute node with that image: the InfiniBand network was
> >   working for that node, that's no surprise, since OpenSM was running
> >   on the head node, but no issm* symlink was created; please note
> >   that, surprisingly, the Ethernet network was not working, I mean
> >   that the Ethernet interfaces were not found by the kernel...]
> >   
> >   root@node # ls -altrF /sys/class/infiniband_mad/
> >   total 0
> >   drwxr-xr-x 60 root root    0 Nov 26 17:06 ../
> >   lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
> >   -r--r--r--  1 root root 4096 Nov 26 17:06 abi_version
> >   lrwxrwxrwx  1 root root    0 Nov 26 17:06 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
> >   drwxr-xr-x  2 root root    0 Nov 26 17:08 ./
> >   
> >   $ git bisect bad
> >   Bisecting: 0 revisions left to test after this (roughly 0 steps)
> >   [65528cfb21fdb68de8ae6dccae19af180d93e143] net/mlx5: mlx5_ifc update for multi-plane support
> >   $ make -j 12 my_defconfig bindeb-pkg
> >   
> >   [install this version on the compute node test image and reboot
> >   one compute node with that image: the InfiniBand network again
> >   working for that node, issm* symlinks were created;
> >   Ethernet network again not working for that node...]
> >   
> >   root@node # ls -altrF /sys/class/infiniband_mad/
> >   total 0
> >   drwxr-xr-x 60 root root    0 Nov 26 17:31 ../
> >   lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
> >   -r--r--r--  1 root root 4096 Nov 26 17:31 abi_version
> >   lrwxrwxrwx  1 root root    0 Nov 26 17:31 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
> >   lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/issm1/
> >   lrwxrwxrwx  1 root root    0 Nov 26 17:36 issm0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/issm0/
> >   drwxr-xr-x  2 root root    0 Nov 26 17:36 ./
> >   
> >   $ git bisect good
> >   2a5db20fa532198639671713c6213f96ff285b85 is the first bad commit
> >   commit 2a5db20fa532198639671713c6213f96ff285b85
> >   Author: Mark Zhang <markzhang@nvidia.com>
> >   Date:   Sun Jun 16 19:08:35 2024 +0300
> >   
> >       RDMA/mlx5: Add support to multi-plane device and port
> >   
> >       When multi-plane is supported, a logical port, which is aggregation of
> >       multiple physical plane ports, is exposed for data transmission.
> >       Compared with a normal mlx5 IB port, this logical port supports all
> >       functionalities except Subnet Management.
> >   
> >       Signed-off-by: Mark Zhang <markzhang@nvidia.com>
> >       Link: https://lore.kernel.org/r/7e37c06c9cb243be9ac79930cd17053903785b95.1718553901.git.leon@kernel.org
> >       Signed-off-by: Leon Romanovsky <leonro@nvidia.com>
> >   
> >    drivers/infiniband/hw/mlx5/main.c               | 60 +++++++++++++++++++++----
> >    drivers/infiniband/hw/mlx5/mlx5_ib.h            |  2 +
> >    drivers/net/ethernet/mellanox/mlx5/core/vport.c |  1 +
> >    include/linux/mlx5/driver.h                     |  1 +
> >    4 files changed, 55 insertions(+), 9 deletions(-)
> > 
> > 
> > In other words, bingo!, your guess looks correct, the first bad commit
> > is the one you mentioned.
> > 
> > 
> > Now, I will try to upgrade the firmware of the InfiniBand NICs, as you
> > suggested, and check whether this solves the issue with the recent
> > Linux kernel versions.
> > 
> > Please confirm that the procedure to be followed is the one described in
> > <https://docs.nvidia.com/networking/display/ubuntu2204/firmware+burning>
> 
> Yes, it looks correct procedure.
> If you didn't upgrade FW, this diff will achieve same result for you:
> 
> diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
> index c2314797afc9..110ce177c305 100644
> --- a/drivers/infiniband/hw/mlx5/main.c
> +++ b/drivers/infiniband/hw/mlx5/main.c
> @@ -2846,7 +2846,7 @@ static int mlx5_ib_get_plane_num(struct mlx5_core_dev *mdev, u8 *num_plane)
>         if (err)
>                 return err;
> 
> -       *num_plane = vport_ctx.num_plane;
> +       *num_plane = (vport_ctx.num_plane > 1) ? vport_ctx.num_plane : 0;
>         return 0;
>  }
> 
> The culprit of your issue that in some FW versions, the vport_ctx.num_plane
> was 1 and not 0 for devices which don't support that mode, while for the driver
> everything that is not 0 means supported.

I wonder if you could test a firmware upgrade or the above patch. Would
be nice to know if there are still some things to do for us (= Debian
kernel team) here.

If everything is fine for you, I'd like to close this bug.

Best regards
Uwe

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Francesco Poli]

[-- Attachment #1: Type: text/plain, Size: 2569 bytes --]

On Wed, 4 Dec 2024 17:37:05 +0100 Uwe Kleine-König wrote:

> Hello Francesco,

Hello Uwe,

[...]
> I wonder if you could test a firmware upgrade or the above patch. Would
> be nice to know if there are still some things to do for us (= Debian
> kernel team) here.

Yes, I've finally got around to upgrading the firmware.

And today I had a time window, where I could reboot the cluster head
node.
After the reboot, the InfiniBand network works correctly:

  $ uname -v
  #1 SMP PREEMPT_DYNAMIC Debian 6.11.10-1 (2024-11-23)
  $ ls -altrF /sys/class/infiniband_mad/
  total 0
  lrwxrwxrwx  1 root root    0 Dec  4 10:15 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
  lrwxrwxrwx  1 root root    0 Dec  4 10:15 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
  drwxr-xr-x  2 root root    0 Dec  4 10:17 ./
  drwxr-xr-x 73 root root    0 Dec  4 10:17 ../
  -r--r--r--  1 root root 4096 Dec  4 10:17 abi_version
  lrwxrwxrwx  1 root root    0 Dec  4 18:08 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
  lrwxrwxrwx  1 root root    0 Dec  4 18:08 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
  # ethtool -i ibp129s0f0
  driver: mlx5_core[ib_ipoib]
  version: 6.11.10-amd64
  firmware-version: 20.43.1014 (MT_0000000224)
  expansion-rom-version:
  bus-info: 0000:81:00.0
  supports-statistics: yes
  supports-test: yes
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: yes
  # ethtool -i ibp129s0f1
  driver: mlx5_core[ib_ipoib]
  version: 6.11.10-amd64
  firmware-version: 20.43.1014 (MT_0000000224)
  expansion-rom-version:
  bus-info: 0000:81:00.1
  supports-statistics: yes
  supports-test: yes
  supports-eeprom-access: no
  supports-register-dump: no
  supports-priv-flags: yes
  $ ps aux | grep opens[m]
  root        1150  0.0  0.0 1560776 3636 ?        Ssl  10:15   0:00 /usr/sbin/opensm --guid 0x9c63c00300033240 --log_file /var/log/opensm.0x9c63c00300033240.log


> 
> If everything is fine for you, I'd like to close this bug.

I am closing the Debian bug report right now.
Thanks to everyone who has been involved for the great and kind help!

> 
> Best regards

Have a nice evening.   :-)

-- 
 http://www.inventati.org/frx/
 There's not a second to spare! To the laboratory!
..................................................... Francesco Poli .
 GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE

[-- Attachment #2: Type: application/pgp-signature, Size: 833 bytes --]

Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Leon Romanovsky]

On Wed, Dec 04, 2024 at 06:13:56PM +0100, Francesco Poli wrote:
> On Wed, 4 Dec 2024 17:37:05 +0100 Uwe Kleine-König wrote:
> 
> > Hello Francesco,
> 
> Hello Uwe,
> 
> [...]
> > I wonder if you could test a firmware upgrade or the above patch. Would
> > be nice to know if there are still some things to do for us (= Debian
> > kernel team) here.
> 
> Yes, I've finally got around to upgrading the firmware.
> 
> And today I had a time window, where I could reboot the cluster head
> node.
> After the reboot, the InfiniBand network works correctly:
> 
>   $ uname -v
>   #1 SMP PREEMPT_DYNAMIC Debian 6.11.10-1 (2024-11-23)
>   $ ls -altrF /sys/class/infiniband_mad/
>   total 0
>   lrwxrwxrwx  1 root root    0 Dec  4 10:15 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
>   lrwxrwxrwx  1 root root    0 Dec  4 10:15 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
>   drwxr-xr-x  2 root root    0 Dec  4 10:17 ./
>   drwxr-xr-x 73 root root    0 Dec  4 10:17 ../
>   -r--r--r--  1 root root 4096 Dec  4 10:17 abi_version
>   lrwxrwxrwx  1 root root    0 Dec  4 18:08 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
>   lrwxrwxrwx  1 root root    0 Dec  4 18:08 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
>   # ethtool -i ibp129s0f0
>   driver: mlx5_core[ib_ipoib]
>   version: 6.11.10-amd64
>   firmware-version: 20.43.1014 (MT_0000000224)
>   expansion-rom-version:
>   bus-info: 0000:81:00.0
>   supports-statistics: yes
>   supports-test: yes
>   supports-eeprom-access: no
>   supports-register-dump: no
>   supports-priv-flags: yes
>   # ethtool -i ibp129s0f1
>   driver: mlx5_core[ib_ipoib]
>   version: 6.11.10-amd64
>   firmware-version: 20.43.1014 (MT_0000000224)
>   expansion-rom-version:
>   bus-info: 0000:81:00.1
>   supports-statistics: yes
>   supports-test: yes
>   supports-eeprom-access: no
>   supports-register-dump: no
>   supports-priv-flags: yes
>   $ ps aux | grep opens[m]
>   root        1150  0.0  0.0 1560776 3636 ?        Ssl  10:15   0:00 /usr/sbin/opensm --guid 0x9c63c00300033240 --log_file /var/log/opensm.0x9c63c00300033240.log
> 
> 
> > 
> > If everything is fine for you, I'd like to close this bug.
> 
> I am closing the Debian bug report right now.
> Thanks to everyone who has been involved for the great and kind help!

Thanks a lot for your help. You helped a lot.

BTW, we have an official fix [1], but it wasn't sent yet as we want to
finish all various tests first (E2E, QA e.t.c).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/commit/?h=rdma-next&id=09754c1e5d0d204747928290cc8c6f4371fd4c6a

> 
> > 
> > Best regards
> 
> Have a nice evening.   :-)
> 
> -- 
>  http://www.inventati.org/frx/
>  There's not a second to spare! To the laboratory!
> ..................................................... Francesco Poli .
>  GnuPG key fpr == CA01 1147 9CD2 EFDF FB82  3925 3E1C 27E1 1F69 BFFE