Re: Is there a way to transfer a rdma connection between userspace processes?

["Stefan (metze) Metzmacher" <metze-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 2577 bytes --]

@Yann and Steve: thanks for the feedback!

>>> I'm currently researching how to implement SMBDirect [MS-SMBD]
>>> together with the multi channel feature of SMB 3.0 in Samba.
>>>
>>> As Samba currently uses one process per tcp connection
>>> and maintains a lot of in memory state within the process
>>> (e.g. for the SMB_VFS modules) it would require a lot of work
>>> to change Samba to coordinate two (or more) processes for one logical
>>> multi channel connection.
>>>
>>> My current plan tries to pass the socket fd of new connections
>>> (which join an existing multi channel session) via fd-passing to
>>> the existing process.
>>>
>>> Now I'm wondering if this would also be possible with
>>> a rdma connection (struct rdma_cm_i ).
>>>
>> RDMA / verbs ressources are tied to a process (especially Memory
>> Registration), but it's ending up in the HCA, which is probably unaware
>> of processes.
>>
>> Additionally, an RDMA_CM connection is not identified by a FD, so this
>> kind of Unix trick (FD passing through Unix socket: SCM_RIGHTS) is not
>> going to work.
>>
>> Forking might already be a challenge for a RDMA/verbs application, so I
>> don't think that sharing/moving an RDMA_CM connection across different
>> processes is supported.
>>
>> But other people on this list (especially Roland Dreier and Sean Hefty)
>> could find a solution.
>>
>> Regards.
>>
> 
> fork() support like you need is not there in Linux RDMA verbs. Another
> alternative is to fork() before you setup the RDMA connection.  IE if a
> regular TCP socket is first used to negotiate RDMA mode, then maybe you
> could fork() after negotiation but before setting up the RDMA connection
> and other resources?

For client connections that would work, but it can't work for a server
that uses fork() after listen().

Would it be possible to extend the librdmacm and libibverbs to:
- handle fork() after getting RDMA_CM_EVENT_CONNECT_REQUEST
  which means there have to be some functions to destroy contexts like
  struct rdma_cm_id, but without affecting the real connection.
  Similar to close() on an FD that is used shared between two processes.
- export/import an interprocess token attached to the hierarchy of objects
  similar to gss_export_sec_context/gss_import_sec_context.
  Or a function which exports the contexts into a FD (of a pipe),
  which could also transfer open FDs to the other end.
  Maybe this could pass the information through a rdma/ibverb event channel?

Are there any other possible solutions?

metze


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]