Re: [PATCH 2/2] rdma/cm: allow user to specify IP to DGID mapping

Jason Gunthorpe
2009-10-20T20:29:02Z

On Tue, Oct 20, 2009 at 01:05:15PM -0700, Sean Hefty wrote:
> >> I agree. But we can still support AF_IB/PS_TCP by simply assigning
> >> the service ID correctly. rdma_bind_addr only needs to fill in a
> >> service id if one is not given. This should enable
> >
> >'one is not given' == 0 service ID?
>
> yes
>
> I'm guessing PS_IB would need to assign from the IB CM assigned range starting
> at 0x020::0 to avoid conflicts.

Something like that..

Ok, this seems sensible to me. So to recap the main differences:

Private data:
- AF_IB/PS_TCP - the kernel munges the private data to be compatible
with AF_INET/PS_TCP, but otherwise is the same.
- AF_IB/PS_IB - the kernel doesn't touch the private data.

Service ID for bind or connect:
- AF_IB/PS_TCP - the service ID passed in through the sockaddr_ib
must be 0 or correctly formed as a IP RDMA CM service ID indicating
a port number. This is true in the bind or connect case.
- AF_IB/PS_IB - the service ID is just a service ID. No restrictions
or alteration are done. The bind'd service ID is ignored on connect

0 Service ID on bind:
- AF_IB/PS_TCP - a new random unused port is allocated and converted to a
service ID
- AF_IB/PS_IB - a new random unused service ID within the proper prefix is
allocated (usefull!)

What about the service_mask feature of IB CM?

How are the IP source and dest IPs going to be picked in for PS_TCP
mode?

I guess user space does that and passes it through to the kernel?
Another bit of binary blob data from rdma_getaddrinfo I guess. I
suppose rdma_getaddrinfo could return an array of option blobs that
must be copied to the kernel to setup the connection, or something
like that.

> >What functional difference are you imagining compared to listening on
> >AF_INET/PS_TCP?
>
> I hadn't thought about this, but making stuff up, I guess this could allow using
> the rdma_cm without ipoib running...? From IB's perspective, PS_TCP is just a
> specific subset of the service ID range.

Plus the IP addies in the REQ private data..

I think it would be fine to EINVAL on AF_IB/PS_TCP listen().

> >> Even once PS_IB is added, we need to make sure that it doesn't collide with
> >> PS_TCP at the IB service ID level, or IB will be in the same situation that
> >> iWarp is in with space collisions.
> >
> >I thought there was already a single service ID registration list that was
> >shared between RDMA CM and IB CM?
>
> Not really - the rdma cm constructs the service id, and the ib cm checks that
> it's usable, but not until listen is called.

Oh I see, well, a 'bind' api into the IB CM should take care of that?

Jason