Re: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Or Gerlitz
2009-10-07T22:23:13Z

Sean Hefty wrote:

> From user space, the call sequence does not change.  The user calls
> rdma_resolve_addr, rdma_resolve_route, rdma_connect, etc.  It is up to the
> librdmacm to perform the resolution.  Today, the resolution request is simply
> passed down to the kernel, which restricts how the resolution can be performed.

good, fair-enough

> I kept resolving the address and route separate.  rdma_set_ib_path, which has
> always existed btw, simply sets the route/path.   The new call,
> rdma_set_ib_dest, sets the address mapping.  To use rdma_set_ib_dest, the user
> must have called rdma_bind_addr first, which covers steps 1 & 2 that you
> mentioned above.  The rdma_bind_addr call can be done internally to the
> librdmacm as part of the rdma_resolve_addr implementation.

I understand that rdma_bind_address covers the local device and vlan
resolutions, but I we should also --keep-- supporting also
applications that use an explicit source address in rdma_resolve_addr
or that don't do bind, provide src=NULL to resolve_addr and rely on
the rdma-cm to use route lookup (as the rdma_resolve_addr man page
indicates) for the device/vlan resolution.

> If a user sets the wrong address mapping or route, they should only affect themselves

I wasn't sure to follow this comment, can you elaborate a bit more?

> (FYI - I have not yet implemented the librdmacm to call rdma_bind_addr as part
> of rdma_resolve_route on linux.  I did not see an easy way to convert a
> destination IP address to a source IP address.  If anyone knows how, please let
> me know.)

I assume you was referring rdma_resolve_addr, correct? there should be
a way to do that from user space and if not, you can go down to the
kernel, resolve the device/vlan and then call ACM to resolve the
destination. It seems that you must resolve the dev/vlan for issuing
the ACM ARP replacement...

> >I would prefer to have a solution where the app flow isn't touched,
> >something like the kernel rdma-cm to communicate with the user space ACM
> >daemon to get address and route resolutions.  Does such a design makes
> >sense to you?

> Long term, this is exactly the type of flow that I envision.  I'd like to have
> real data to show that the ACM implementation scales first, which is part of my
> problem.  I do not have the ability to easily change kernel drivers on any
> larger sized clusters.  My approach is to allow user space to perform the
> address and route resolution and pass the data to the kernel.  This way, we have
> the freedom to test multiple solutions, until we can settle on what works.

I am not sure to fully follow on the easily-change-kernel-drivers
claim, isn't some change to the kernel rdma-cm being a must for the
ACM + librdmacm solution to work? suppose you have a way to fully do
the addr+route resolutions from user space, will the kernel rdma-cm
state machine will be willing to issue

rdma_create_id
rdma_set_ib_path (you said this exists today?)
rdma_create_qp
rdma_connect

???

Or.