RE: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Sean Hefty
2009-10-07T23:42:13Z

>I understand that rdma_bind_address covers the local device and vlan
>resolutions, but I we should also --keep-- supporting also
>applications that use an explicit source address in rdma_resolve_addr
>or that don't do bind, provide src=NULL to resolve_addr and rely on
>the rdma-cm to use route lookup (as the rdma_resolve_addr man page
>indicates) for the device/vlan resolution.

My intent, which differs from Jason's, was to fully support the existing
librdmacm interfaces as they are defined.

Implementation wise, if the user of the librdmacm calls rdma_resolve_addr with a
src address, it's easy. Without the src address, it's hard, but I may just be
missing some easy interface for finding the src address.

Between the two patches, I'm fine with the change to set the PR and still
looking at ways to handle rdma_resolve_addr.

>> If a user sets the wrong address mapping or route, they should only affect
>themselves
>
>I wasn't sure to follow this comment, can you elaborate a bit more?

I meant that if some bogus app wants to specify an IP to GID mapping that's
invalid, the incorrect mapping should only affect connections for that app.
When used over IB, the IP address is little more than a qualifier contained
within the IB CM REQ private data.

>I assume you was referring rdma_resolve_addr, correct? there should be
>a way to do that from user space and if not, you can go down to the
>kernel, resolve the device/vlan and then call ACM to resolve the
>destination. It seems that you must resolve the dev/vlan for issuing
>the ACM ARP replacement...

Technically, rdma_resolve_addr could remain unchanged, in which case it will do
everything it does today, which may include sending an ARP. This is the
specific operation that I'd like to avoid.

>I am not sure to fully follow on the easily-change-kernel-drivers
>claim, isn't some change to the kernel rdma-cm being a must for the
>ACM + librdmacm solution to work? suppose you have a way to fully do
>the addr+route resolutions from user space, will the kernel rdma-cm
>state machine will be willing to issue

All of our larger clusters are for production use. Asking them to install and
use a supported kernel / OFED release is different than me asking them to apply
a patch to the kernel on all systems.

I can somewhat implement an ACM + librdmacm solution entirely in user space by
layering the librdmacm over libibcm. Because of the event reporting, it would
be limited in how it could be use, and is unlikely to be something that would
ever be supported.

- Sean