Infiniband PKEY Regression 6.12.115

Infiniband PKEY Regression 6.12.115

[Ulf Markwardt]

[-- Attachment #1: Type: text/plain, Size: 3728 bytes --]

Dear Linux Developers,
we are running 6.12.x kernels with AlmaLinux 9.8 as OS on some HPC clusters.
 From 6.12.114 to 6.12.115 we experience this regression on a certain host type:

The InfiniBand P_Key tables do not get populated by the SM (opensm) when
booting kernel 6.12.115. All IPoIB child interfaces that require a specific P_Key
(e.g. 0x8001, 0x8002) fail with "P_Key not found". With kernel 6.12.114 the
P_Key tables get populated and all IPoIB subinterfaces come up at boot time.

   * Kernel config is identical between .114 and .115.
   * The problem is NOT driver-related (reproduces with both DOCA OFED and in-kernel
     mlx5 drivers).
   * On another host in the same IB fabric, with the same P_KEY setup, I can run .115
     without problems:
       o OK with node i7010: *only 1 HCA* ConnectX-6
       o Problem with node c11: *4 HCAs *ConnectX-7
   * The problem is NOT the SM:
       o no problem if I boot the node (c11) with kernel .114 instead of .115.
         (without any other changes)
       o partition policy is set with
             |port-regexp: ^c[0-9]{1,3}| ... and
             |port-regexp: ^i7[0-9]{3}| ...
   * This suggests the problem may be related to MULTI-HCA systems: when multiple
     mlx5 HCAs initialize simultaneously
   * With triggering a heavy sweel (|kill -HUP $(pidof opensm)|) the P_KEY tables get
     populated, and the node c11 with the .115 kernel can use the subinterfaces
     (until next boot).

Relevant dmesg lines (kernel .115, broken):

[   24.987281] mlx5_core 0000:23:00.0 mlx5_2: Port: 1 Link ACTIVE
[   25.465106] mlx5_core 0000:03:00.0 mlx5_3: Port: 1 Link ACTIVE
[   26.078531] mlx5_core 0000:c3:00.0 mlx5_4: Port: 1 Link ACTIVE
[   26.078576] mlx5_core 0000:a3:00.0 mlx5_5: Port: 1 Link ACTIVE

[   31.787242] ibp163s0.8002: P_Key 0x8002 is not found
[   31.803401] ibp163s0.0001: P_Key 0x8001 is not found
[   32.079644] ibp195s0.8002: P_Key 0x8002 is not found
[   32.097820] ibp195s0.0001: P_Key 0x8001 is not found
[   32.370439] ibp35s0.8002: P_Key 0x8002 is not found
[   32.386155] ibp35s0.0001: P_Key 0x8001 is not found
[   32.659225] ibp3s0.8002: P_Key 0x8002 is not found
[   32.674642] ibp3s0.0001: P_Key 0x8001 is not found

[   43.049397] ibp3s0.0001: P_Key 0x8001 is not found
[   45.486835] ibp3s0.8002: P_Key 0x8002 is not found
[   47.899294] ibp35s0.0001: P_Key 0x8001 is not found
[   50.282621] ibp35s0.8002: P_Key 0x8002 is not found
[   52.492136] ibp163s0.0001: P_Key 0x8001 is not found
[   54.716345] ibp163s0.8002: P_Key 0x8002 is not found
[   56.920143] ibp195s0.0001: P_Key 0x8001 is not found
[   59.157200] ibp195s0.8002: P_Key 0x8002 is not found
[   59.193759] ibp195s0.8002: P_Key 0x8002 is not found

Timeline analysis (done by AI):
- Ports Link ACTIVE at t=24.9-26.1s (SM assigns LIDs, port is up)
- IPoIB child interfaces created at t=31.8s - 6.8 seconds after port-up
- IPoIB retries P_Key lookup for 27.4 seconds (t=31.8s -> 59.2s, every ~2.4s)
- Interfaces give up at t=59.2s - stay DOWN permanently
- The SM never writes P_Key tables during this entire window
- Only a heavy sweep populates the P_KEY tables.

I can provide additional information (just tell me what to do).
We are in a production environment, but have some leeway for experiments.
I hope you can fix this problem with one of the next releases.

Thank you very much,
Ulf

-- 
___________________________________________________________________
Dr. Ulf Markwardt
Gruppenleiter HPC Services

Technische Universität Dresden
Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH)
D-01062  Dresden

Tel.:   +49 351 463-33640
E-Mail:ulf.markwardt@tu-dresden.de


[-- Attachment #2: Kryptografische S/MIME-Signatur --]
[-- Type: application/pkcs7-signature, Size: 5063 bytes --]

Re: Infiniband PKEY Regression 6.12.115

[Greg KH]

On Mon, Nov 02, 2026 at 07:38:39AM +0100, Ulf Markwardt wrote:
> Dear Linux Developers,
> we are running 6.12.x kernels with AlmaLinux 9.8 as OS on some HPC clusters.
> From 6.12.114 to 6.12.115 we experience this regression on a certain host type:
> 
> The InfiniBand P_Key tables do not get populated by the SM (opensm) when
> booting kernel 6.12.115. All IPoIB child interfaces that require a specific P_Key
> (e.g. 0x8001, 0x8002) fail with "P_Key not found". With kernel 6.12.114 the
> P_Key tables get populated and all IPoIB subinterfaces come up at boot time.

Any chance you can run 'git bisect' to find the offending commit?

And always cc: the relevant subsystem developers to please.

thanks,

greg k-h

Re: Infiniband PKEY Regression 6.12.115

[Ulf Markwardt]

[-- Attachment #1: Type: text/plain, Size: 1271 bytes --]

Dear all,
after 3 days of extensive testing we found the reason of the regression was an OpenSM problem.
We now have this kernel in production.
Thanks,
Ulf


Am 02.11.26 um 12:22 schrieb Greg KH:
> On Mon, Nov 02, 2026 at 07:38:39AM +0100, Ulf Markwardt wrote:
>> Dear Linux Developers,
>> we are running 6.12.x kernels with AlmaLinux 9.8 as OS on some HPC clusters.
>>  From 6.12.114 to 6.12.115 we experience this regression on a certain host type:
>>
>> The InfiniBand P_Key tables do not get populated by the SM (opensm) when
>> booting kernel 6.12.115. All IPoIB child interfaces that require a specific P_Key
>> (e.g. 0x8001, 0x8002) fail with "P_Key not found". With kernel 6.12.114 the
>> P_Key tables get populated and all IPoIB subinterfaces come up at boot time.
> 
> Any chance you can run 'git bisect' to find the offending commit?
> 
> And always cc: the relevant subsystem developers to please.
> 
> thanks,
> 
> greg k-h

-- 
___________________________________________________________________
Dr. Ulf Markwardt
Gruppenleiter HPC Services

Technische Universität Dresden
Zentrum für Informationsdienste und Hochleistungsrechnen (ZIH)
D-01062  Dresden

Tel.:   +49 351 463-33640
E-Mail: ulf.markwardt@tu-dresden.de


[-- Attachment #2: Kryptografische S/MIME-Signatur --]
[-- Type: application/pkcs7-signature, Size: 5063 bytes --]