RE: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Sean Hefty
2009-10-20T18:08:58Z

>okay, I just wanted to make sure that the whole thing (ACM + modified
>librdmacm + modifed rdma-cm) is applicable AND inter-operable for
>AF_INET / PS_TCP applications.

I do not intend to have any changes that break anything.

>Looking on kernel cma.c format_hdr code it first branches on the address
>family and next of the port space.
>
>Going with your proposed flow, I understand that an app call to
>rdma_resolve_addr will be broken down to rdma_bind_addr, ACM resolution
>of the destination GID and then rdma_set_ib_dest, so things should work
>perfect for AF_INET / PS_TCP apps, correct?

This is my current plan for the kernel:

Export rdma_set_ib_paths to user space. Submit a patch. Get it accepted
upstream. Eat ice cream to celebrate.

Define AF_IB and struct sockaddr_ib (contains a gid and service id). Update
rdma_bind_addr, rdma_resolve_addr, and rdma_connect to handle AF_IB.

rdma_bind_addr fills in the service id according RDMA IP CM service annex.
rdma_resolve_addr just needs to save the GIDs.
rdma_connect will not modify the private data in the CM REQ for AF_IB.

For user space:

Initially, ACM support must be specifically built in and enabled using an
environment variable. If it's enabled, the librdmacm will convert AF_INET and
pass AF_IB to the kernel. The librdmacm will fall back to normal operation if
ACM fails.

With AF_IB, the flow would probably be something like: rdma_resolve_addr -> ACM
resolution, ucma_resolve_addr, then rdma_resolve_route -> ucma_set_ib_paths.

>The only missing piece here is the route lookup from user space for
>applications not specifying a source address in their rdma_resolve_addr
>invocation, do you still need help to implement that?

This is still missing, and help would be great.

>> The ACM responds based on a configuration file. The ib_acme utility can
>create that file using the active IP, pkey, port information of the system, but
>the current ACM implementation does not adjust to dynamic changes or detect
>misconfigurations or other made up words.

>I see. Does the new flow of librdmacm is going to be under new API, eg
>rdma_resolve_addr/route_ext or the same API optionally talking to ACM
>through some IPC if the ACM daemon is running, or something else?

I hope we can do this under the existing APIs with an optional ability to use
ACM, but I like the idea of adding rdma_getaddrinfo.

- Sean