Re: [PATCH 04/11] IB/cm: Expose DGID in SIDR request events

[Jason Gunthorpe <jgunthorpe@obsidianresearch.com>]

On Tue, Jun 16, 2015 at 02:25:07PM +0300, Haggai Eran wrote:

> > But how is that going to work? How is the sender to know it should be
> > sending a GRH with the CM message?
> 
> If the admin wants to use SIDR with alias GIDs, they will need to
> configure the system to enable GRH for such GMPs. (This series doesn't
> include such a patch).

Gross.

> Regarding APM, currently the ib_cm code always sends the GMP to the
> primary path anyway, right? And in any case, one would expect the
> primary path's GID to have a valid net_device and local routing rules,
> so I think for the purpose of demuxing and validating the request using
> the primary path should be fine.

The current code works that way, but it is not what I'd expect
generally.

For instance, future APM support will be able to drive dual-rail and
policy will decide which rail is the current best rail for data
transfer. So the GMP may be directed to the IPoIB device with port 1,
but the data transfer may happen on the RDMA port 2. [Note, I already
have very rough patches that do this de-coupling]

> Why do you think the GMP's net_device should be used over the one of the
> future RDMA channel?

The code needs to match the incoming GMP with the logical netdev that
rx's *that GMP*. The fact that goes on to setup an RDMA channel is not
relevant, the nature of the future RDMA channel should not impact how
the GMP is recieved.

> So far we can work without GRH for CM requests, and also without GRH for
> SIDR requests if we rely on layer 3 for the interface resolution. I'm
> not against adding a LLADDR to the protocol somehow, but I don't think
> we should abandon all these use cases and the interoperability with
> existing software.

Well, there is a middle ground. Lets say we get the LLADDR in the GMP
somehow, then we get 100% correct operation when it is present.

For degraded operation we have the (device,port,pkey) and possibly
(device,port,pkey,gid) if there was a GRH. We also have the IP address
hack.

So, I'd say, search in this sequence:
 - If the LLADDR is present, just find the right netdev
 - Otherwise search for (device,port,pkey) / (device,port,pkey,gid)
   If there is only one match then that is unambiguously the correct
   device to use.
 - Repeat the above search, but add the IP address. This is the hack
   we perform for compatibility.

There is no reason to hackily look at the GMP path parameters if we are
relying on #3 above as the hack to save us in the legacy ambiguous case.

.. and to answer your question in the other email, I think we should
keep the hack clearly distinct from the proper operation (in fact,
perhaps it should be user configurable). So #3 should be a distinct
step taken when all else fails, not integrated into earlier steps.

So, this series as it stands just needs to do #2/#3 above and you guys
can figure out the LLADDR business later.

Jason