Help with PCIe THP on ConnectX-7

Help with PCIe THP on ConnectX-7

[Bruce Merry]

Hello

I'm hoping someone from NVIDIA Networking can help with this; I wasn't
sure of the best way to get in contact with the engineers working on
mlx5 kernel code so thought I'd try here. I'm trying to write some
(userspace) code using ibv_reg_mr_ex to register a memory region using
TLP Processing Hints (THP) to improve performance on an Epyc Turin
(Zen 5) system. However, in the kernel mlx5_st_create is bailing out
because of this check:

       if (!MLX5_CAP_GEN(dev, mkey_pcie_tph))
                return NULL;

As far as I can tell, that is checking a capability bit returned by
the firmware.

Before I spend a lot more time debugging, can you tell me whether
ConnectX-7 supports this feature at all (I'm on the latest 28.48.1000
firmware)? I see it mentioned in release notes for ConnectX-8
firmware, but not in ConnectX-7 release notes.

If ConnectX-7 does support it, do you have any tips on what might
cause that capability bit to be false and how to determine what the
cause is? For example, could that happen if the motherboard BIOS
doesn't support TPH?

Thanks
Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/

Re: Help with PCIe THP on ConnectX-7

[Yishai Hadas]

On 22/04/2026 18:22, Bruce Merry wrote:
> Hello
> 
> I'm hoping someone from NVIDIA Networking can help with this; I wasn't
> sure of the best way to get in contact with the engineers working on
> mlx5 kernel code so thought I'd try here. I'm trying to write some
> (userspace) code using ibv_reg_mr_ex to register a memory region using
> TLP Processing Hints (THP) to improve performance on an Epyc Turin
> (Zen 5) system. However, in the kernel mlx5_st_create is bailing out
> because of this check:
> 
>         if (!MLX5_CAP_GEN(dev, mkey_pcie_tph))
>                  return NULL;
> 
> As far as I can tell, that is checking a capability bit returned by
> the firmware.

Right

> 
> Before I spend a lot more time debugging, can you tell me whether
> ConnectX-7 supports this feature at all (I'm on the latest 28.48.1000
> firmware)? I see it mentioned in release notes for ConnectX-8
> firmware, but not in ConnectX-7 release notes.
> 

It's supported only on CX8 for now.

Yishai

> If ConnectX-7 does support it, do you have any tips on what might
> cause that capability bit to be false and how to determine what the
> cause is? For example, could that happen if the motherboard BIOS
> doesn't support TPH?
> 
> Thanks
> Bruce

Re: Help with PCIe THP on ConnectX-7

[Bruce Merry]

On Thu, Apr 23, 2026 at 05:55:53PM +0300, Yishai Hadas wrote:
> On 22/04/2026 18:22, Bruce Merry wrote:
> > Hello
> > 
> > I'm hoping someone from NVIDIA Networking can help with this; I wasn't
> > sure of the best way to get in contact with the engineers working on
> > mlx5 kernel code so thought I'd try here. I'm trying to write some
> > (userspace) code using ibv_reg_mr_ex to register a memory region using
> > TLP Processing Hints (THP) to improve performance on an Epyc Turin

...

> It's supported only on CX8 for now.

Thanks. Does "for now" mean that you expect it to become available on
CX7 in future, and if so, are you able to say anything about timelines?

Thanks
Bruce
-- 
Dr Bruce Merry
bmerry <@> gmail <.> com
http://www.brucemerry.org.za/
http://blog.brucemerry.org.za/