Re: [ewg] rdma/cm: revert associating an RDMA device when binding to loopback

[Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org>]

On Tue, Feb 09, 2010 at 05:01:21PM -0500, Jeff Squyres wrote:

> 1. Is this now the recommended way to find all the IP interfaces that support RDMA:
> 
> - loop over all local IP addresses
> - if 127.0.0.1/8, skip
> - try to rdma_bind_addr()
> - if it succeeds and verbs ptr is != NULL, it's an RDMA device

RDMA is not special, it is just like any other IP service. RDMA is
supported on loopback.

To find the list of RMDA capable IPs you do the rdma_bind_addr test.

You then have to transform that list exactly as you would for TCP to
get a list of candidate addresses that could be used for remote
connection. This means removing loopback, doing someting about link
local addresses, and matching as necessary IPs to interfaces, to
networks, and to source IPs on the connecting side. It isn't trivial,
but it is exactly the same as for TCP. I suppose ideally OMPI would
use the same codes for both TCP and RDMACM - and it should have user
configurables!

It is worth reviewing what the OMPI TCP does and at least checking
that the RDMACM hits all the same points.

> 2. Before Sean backed out the localhost behavior, when you
> rdma_addr_bind(127.0.0.1), what did the id->verbs pointer
> correspond to?

One of the RDMA verbs devices in the system. The API does not define
which one the kernel will select.

I think the current patches simply picked the first one.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html