Re: SRIOV virtual functions with Linux "inbox" opensm

[Leon Romanovsky <leon@kernel.org>]

On Wed, May 01, 2024 at 04:43:39AM +0000, Ewen Chan wrote:
> Leon:
> 
> I see in the link that you provided that the IB management packages are only available for Redhat.
> 
> Would it be safe to assume that there are no Debian equivalent to these packages?

Right, at the moment, there is no Debian equivalent to these packages.

Thanks

> 
> Your help is greatly appreciated.
> 
> Thank you.
> 
> 
> 
> 
> From: Leon Romanovsky <leon@kernel.org>
> Sent: April 30, 2024 8:03 AM
> To: Ewen Chan <alpha754293@hotmail.com>
> Cc: linux-rdma@vger.kernel.org <linux-rdma@vger.kernel.org>
> Subject: Re: SRIOV virtual functions with Linux "inbox" opensm
>  
> On Mon, Apr 22, 2024 at 10:09:15AM +0000, Ewen Chan wrote:
> > To Whom It May Concern:
> >
> > I am using a few Mellanox ConnectX-4 100 Gbps Infiniband NIC that's connected together via a Mellanox MSB7890 externally managed switch.
> >
> > I have a dual Xeon E5-2697A v4, Proxmox 7.4-17 (Debian 11) server that's running opensm, along with two AMD Ryzen 5950X compute nodes, that also have the ConnectX-4 in them, running Proxmox 7.4-17 as well.
> >
> > I have enabled SR-IOV on all three systems, and all three systems have 8 virtual functions for said ConnectX-4.
> >
> > I read in the Nvidia/Mellanox documentation that I would need to add the parameter "virt_enabled 2" to /etc/opensm/opensm.conf so that the OpenSM subnet manager will know that virtual functions are enabled, but it would appear that the opensm that ships with Debian 11/linux-rdma, either ignores that option or doesn't know what to do with it.
> >
> > I would prefer NOT to install the MLNX_OFED drivers for Debian (11) if I can avoid it.
> >
> > My two questions are how do I get the linux opensm to:
> >
> >     Recognise that I am using virtual functions (so that it would understand that there are multiple traffic streams coming over the wire, via one physical port)?
> >
> >     Automatically assign the Node GUID and Port GUID so that I don't have to set those manually.
> >
> >     (I've set the Node GUID and Port GUID on the my Ryzen compute node host already, and I can see the Node GUID and Port GUID inside my CentOS 7.7.1908 VM (which I've updated to use the 5.4.247 kernel), but it is still showing "Port 1, State: Down".)
> >
> >
> > Your help is greatly appreciated.
> 
> Linux opensm is not supporting ConnectX4+ SRIOV.
> 
> You can install SM RPM from NVIDIA Web to enable ConnectX4 SRIOV.
> https://network.nvidia.com/products/adapter-software/infiniband-management-and-monitoring-tools/
> 
> Thanks
> 
> >
> > Thank you.
> >