Re: [PATCH] link-local address fix for rdma_resolve_addr

Or Gerlitz
2009-10-25T11:52:13Z

Jason Gunthorpe wrote:
> Mainly for RDMA what you get is more kernel flexability, IP like
> capabilities, better bonding, and better support of IB APM semantics:
> - bind() + listen() actually works properly if more than one
> interface is bound to the same IP - the cm_id returned by accept is
> bound to the hca and port that accepted the connection
> [ This is a L3 form of bonding Linux supports ]
>
> This is actually something of a mandatory notion to implement the
> full generality of the IB CM protocol which allows the CM REP to
> contain a port GUID of another port on the same node (multi-port
> APM is an IB feature). So you never know what port the accept()
> result will get bound to.
>
> BTW: I suppose ideally AF_IB would have a way to say 'CM accept REPs on
> any port on this node' Hmm, reserved GID prefix perhaps? Hmm.
>
> When used with bonding this would also afford the kernel with the
> ability to accept incoming connections across all the redundant
> RDMA devices - and still have correct bound-to-IP semantics.
>
> - rdma_resolve_addr more or less as the inverse of all the above
> * multiple interfaces with same IP case works, kernel and routing
> table can distribute outgoing connections
> * multi-port APM works, kernel and user space can choose primary
> and backup port for the IP addy
> * bonding works, kernel can balance outgoing connections across the
> bond slaves.
>
> These are all useful features.

Jason, Have you even looked into or tested any of the bonding
load-balancing modes with ipoib? some/most of them are not applicable to
IPoIB and I don't think that the ones which may be such were ever
tested. Next, multiple interfaces with the same ip address isn't
something I see very useful for production environment (but I'd be happy
to get educated what L3 bonding is and where it can play), next,
multi-port APM isn't something I ever heard to be required by customers
and more important, from comments made by Sean in the past, I don't
think it fits the rdma-cm spirit.

All in all, someone comes here and suggests some fixes to the rdma-cm
address resolution code to have IPv6 work. I don't think Dave should
carry on his back/patch all your proposed future enhancements. Let him
fix things and following that you can work on the patches to support all
these nice/nitch features starting with IPv4 and then IPv6.

Or