Re: [Question] Is address format "gid" supported by RDMACM with RoCE?

[Mark Zhang <markzhang@nvidia.com>]

On 3/7/2022 9:57 PM, Sylvain Didelot wrote:
> External email: Use caution opening links or attachments
> 
> 
> Hi,
> 
> I have configured one of my Mellanox network adapters for RoCE:
> ---
> CA 'roceP1p1s0f1'
>      CA type: MT4123
>      Number of ports: 1
>      Firmware version: 20.32.1010
>      Hardware version: 0
>      Node GUID: 0xb8cef603002d1707
>      System image GUID: 0xb8cef603002d1706
>      Port 1:
>          State: Active
>          Physical state: LinkUp
>          Rate: 100
>          Base lid: 0
>          LMC: 0
>          SM lid: 0
>          Capability mask: 0x00010000
>          Port GUID: 0xbacef6fffe2d1707
>          Link layer: Ethernet
> ---
> 
> The Infiniband stack was installed from the official Ubuntu repository
> (20.04.4 LTS):
> ---
> $ apt search ibverbs
> Sorting... Done
> Full Text Search... Done
> ibverbs-providers/focal,now 28.0-1ubuntu1 arm64 [installed]
>    User space provider drivers for libibverbs
> 
> ibverbs-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Examples for the libibverbs library
> 
> libibverbs-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Development files for the libibverbs library
> 
> libibverbs1/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Library for direct userspace use of RDMA (InfiniBand/iWARP)
> 
> librdmacm-dev/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Development files for the librdmacm library
> 
> librdmacm1/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Library for managing RDMA connections
> 
> rdmacm-utils/focal,now 28.0-1ubuntu1 arm64 [installed]
>    Examples for the librdmacm library
> ---
> 
> When I start the ucmatose server with the address format "gid", the
> tool fails binding with the error "No such device"
> 
> Here is an example of the output:
> ---
> $ cat /sys/class/infiniband/roceP1p1s0f1/ports/1/gids/0
> fe80:0000:0000:0000:bace:f6ff:fe2d:1707
> 
> $ ucmatose -b fe80:0000:0000:0000:bace:f6ff:fe2d:1707 -P ib -f gid
> cmatose: starting server
> cmatose: bind address failed: No such device
> test complete
> return status -1
> ---
> 
> Does rdmacm support connection establishment using GID with RoCE? Or
> Is it a known limitation for RoCE?
> FYI, the same experiment without RoCE (Link layer: Infiniband) works perfectly.
> 
> Thanks for your help and your feedback.
> 
> Sylvain Didelot

I think ucmatose doesn't support RoCE when using "-f gid", as in this 
case ai_family is set to AF_IB.