RE: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Sean Hefty
2009-10-07T19:16:56Z

>So, I'm just talking about the user space API, the others can be
>changed as necessary to align with it.
>
>This is open source, so choosing a technically better solution over a
>endlessly backwards compatible solution is done all the time and is
>normal, expected, etc. Cost of progress - that is the underlying
>rational of Documentation/stable_api_nonsense.txt, and it applies just
>as well to niche little user space libraries like these :)

stable_api_nonsense.txt only applies to the rdma_cm interface, not the ABI or a
user space library. I believe that any change to the library or ABI that forces
applications to change would be detrimental to OFA and the stack as a whole, and
I do not see a compelling reason to make such a change.

Discarding the existing librdmacm interface and ABI are not viable options in my
opinion.

>The rdma_ucm interface would have to be extended to be able to do 100%
>of the functionality of the ib cm interface using the rdma_cm_id
>abstraction. This is very useful in of itself and much better than
>adding an obscure option to override the ARP query. For instance,
>other MPIs could immediately provide their users an option to use GID
>addresess directly and cut out the ARP overhead instantly with
>little code change.

rdma_cm rdma_resolve_addr may result in issuing an ARP query - it depends on the
transport and device capabilities. I want to keep the other behavior of
librdmacm rdma_resolve_addr, and eliminate the ARP as unnecessary. Other
options I looked at were using fields inside the struct sockaddr_in6 (yuck) or
letting a timeout of 0 indicate that ARP should not be used. The latter leaves
the ABI intact. The drawback is that the DGID could still be unknown, which
would result in rdma_cm rdma_resolve_route failing. This may be acceptable.

An extension to the ABI is needed to allow user space to set the IB path. The
proposed 'set_option' ABI could support passing multiple PRs to the kernel. The
kernel implementation only handles one currently.

>From your other mails, it doesn't sound like you have an issue with an ABI
extension that allows setting the IB path record directly. Is this correct, and
do you have an issue with the proposed implementation of that?

You do seem to disagree with the changes to allow user space to specify the IP
to DGID mapping. Is there an alternative that you would agree with?

- Sean