ENODEV in rdma_create_ep with loopback as address

ENODEV in rdma_create_ep with loopback as address

[Jun Han Ng]

When attempting to call rdma_create_ep in active mode with the
loopback address obtained by rdma_getaddrinfo, -1 is returned with
ENODEV errno(). This is reproducable by running the rdma_client.c
example in librdmacm/examples. The error is not appearing when the
host provided is a remote host from a different machine with a
different IP address.

Looks like the exact point of the errno being set is at the write
function in rdma_resolve_route inside librdmacm/cma.c. I am not able
to go deeper due to that write function being preloaded.


My operating system is Ubuntu 22.04.1 and my kernel is version 6.5.0.


Below is the output of my ibv_devinfo command locally; I appreciate your help.

hca_id:    rx0
    transport:            InfiniBand (0)
    fw_ver:                0.0.0
    node_guid:            2a6b:35ff:fe76:0e8a
    sys_image_guid:            2a6b:35ff:fe76:0e8a
    vendor_id:            0xffffff
    vendor_part_id:            0
    hw_ver:                0x0
    phys_port_cnt:            1
        port:    1
            state:            PORT_ACTIVE (4)
            max_mtu:        4096 (5)
            active_mtu:        1024 (3)
            sm_lid:            0
            port_lid:        0
            port_lmc:        0x00
            link_layer:        Ethernet


Thanks,

Jun

RE: ENODEV in rdma_create_ep with loopback as address

[Konstantin Taranov]

> When attempting to call rdma_create_ep in active mode with the loopback
> address obtained by rdma_getaddrinfo, -1 is returned with ENODEV errno().

What do you call a "loopback address"? 

Note, you cannot use 127.0.0.1 as there is no RDMA device behind this IP.
You can run "ibv_devinfo -v" and use any IP listed in the GID table for a loopback experiment.

- Konstantin

Re: ENODEV in rdma_create_ep with loopback as address

[Jun Han Ng]

I was indeed using 127.0.0.1, I changed the address to the interfaces
IP instead and it worked. Your advice was helpful!

On Thu, May 16, 2024 at 10:48 PM Konstantin Taranov
<kotaranov@microsoft.com> wrote:
>
> > When attempting to call rdma_create_ep in active mode with the loopback
> > address obtained by rdma_getaddrinfo, -1 is returned with ENODEV errno().
>
> What do you call a "loopback address"?
>
> Note, you cannot use 127.0.0.1 as there is no RDMA device behind this IP.
> You can run "ibv_devinfo -v" and use any IP listed in the GID table for a loopback experiment.
>
> - Konstantin

Re: ENODEV in rdma_create_ep with loopback as address

[Zhu Yanjun]

在 2024/5/16 16:48, Konstantin Taranov 写道:
>> When attempting to call rdma_create_ep in active mode with the loopback
>> address obtained by rdma_getaddrinfo, -1 is returned with ENODEV errno().
> 
> What do you call a "loopback address"?
> 
> Note, you cannot use 127.0.0.1 as there is no RDMA device behind this IP.
> You can run "ibv_devinfo -v" and use any IP listed in the GID table for a loopback experiment.

Exactly, 127.0.0.1 and 0.0.0.0 are used as private ip addresses in rdma 
cma. As such, it had better not to use the 2 ip addresses.

Zhu Yanjun

> 
> - Konstantin