Re: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Jason Gunthorpe
2009-10-20T18:31:32Z

On Tue, Oct 20, 2009 at 11:08:58AM -0700, Sean Hefty wrote:
> >Looking on kernel cma.c format_hdr code it first branches on the address
> >family and next of the port space.
> >
> >Going with your proposed flow, I understand that an app call to
> >rdma_resolve_addr will be broken down to rdma_bind_addr, ACM resolution
> >of the destination GID and then rdma_set_ib_dest, so things should work
> >perfect for AF_INET / PS_TCP apps, correct?
>
> This is my current plan for the kernel:
>
> Export rdma_set_ib_paths to user space. Submit a patch. Get it accepted
> upstream. Eat ice cream to celebrate.
>
> Define AF_IB and struct sockaddr_ib (contains a gid and service id). Update
> rdma_bind_addr, rdma_resolve_addr, and rdma_connect to handle AF_IB.
>
> rdma_bind_addr fills in the service id according RDMA IP CM service
> annex.

Hm? If the sockaddr_ib contains the service ID why override it in the
kernel?

The AF_IB/PS_IB flow in the kernel side must just use straight service
IDs.

This is why I suggested rdma_getaddrinfo - if you specify an address
string that uses IP port notation but rdma_getaddrinfo decides to use
AF_GID/PS_IB then it can create the proper service ID for that port
number and build it into the sockaddr_ib.

Like in IP you'd use rdma_getaddrinfo with hints.flags = AI_PASSIVE to
construct a description for a listening RDMA connection, and it can
construct either a AF_GID with a service ID or AF_IP with the port.

> >> The ACM responds based on a configuration file. The ib_acme utility can
> >create that file using the active IP, pkey, port information of the system, but
> >the current ACM implementation does not adjust to dynamic changes or detect
> >misconfigurations or other made up words.
>
> >I see. Does the new flow of librdmacm is going to be under new API, eg
> >rdma_resolve_addr/route_ext or the same API optionally talking to ACM
> >through some IPC if the ACM daemon is running, or something else?
>
> I hope we can do this under the existing APIs with an optional ability to use
> ACM, but I like the idea of adding rdma_getaddrinfo.

IHMO, I would much rather see your ACM as part of a rdma_getaddrinfo
API. If you implement the nscd-like unix domain socket then ACM can
completely live outside the main stack as a bolt-on.

Jason