RE: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Sean Hefty
2009-10-06T19:05:02Z

>If I understand correct your suggested changes are to optionally let an
>application to - instead of the following sequence of calls
>
>rdma_resolve_addr / addr resolved event
>rdma_create_qp
>rdma_resolve_route / route resolved event
>rdma_connect / cm events
>
>do
>
>rdma_set_ib_path
>rdma_create_qp
>rdma_connect / cm events

>From user space, the call sequence does not change. The user calls
rdma_resolve_addr, rdma_resolve_route, rdma_connect, etc. It is up to the
librdmacm to perform the resolution. Today, the resolution request is simply
passed down to the kernel, which restricts how the resolution can be performed.

>So in that respect, I am not sure how rdma_set_dest serves you. Further,
>rdma_resolve_addr does three resolutions
>
>1. the local device and source gid
>2. the PKEY (VLAN) to use
>3. the destination gid
>
>so in that respect, rdma_set_ib_path replaces both rdma_resolve_addr and
>rdma_resolve_route?

I kept resolving the address and route separate. rdma_set_ib_path, which has
always existed btw, simply sets the route/path. The new call,
rdma_set_ib_dest, sets the address mapping. To use rdma_set_ib_dest, the user
must have called rdma_bind_addr first, which covers steps 1 & 2 that you
mentioned above. The rdma_bind_addr call can be done internally to the
librdmacm as part of the rdma_resolve_addr implementation. If a user sets the
wrong address mapping or route, they should only affect themselves.

(FYI - I have not yet implemented the librdmacm to call rdma_bind_addr as part
of rdma_resolve_route on linux. I did not see an easy way to convert a
destination IP address to a source IP address. If anyone knows how, please let
me know.)

>I would prefer to have a solution where the app flow isn't touched,
>something like the kernel rdma-cm to communicate with the user space ACM
>daemon to get address and route resolutions. Does such a design makes
>sense to you?

Long term, this is exactly the type of flow that I envision. I'd like to have
real data to show that the ACM implementation scales first, which is part of my
problem. I do not have the ability to easily change kernel drivers on any
larger sized clusters. My approach is to allow user space to perform the
address and route resolution and pass the data to the kernel. This way, we have
the freedom to test multiple solutions, until we can settle on what works.

- Sean