Re: Is there a way to transfer a rdma connection between userspace processes?

["Stefan (metze) Metzmacher" <metze-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 2881 bytes --]

Am 16.10.2012 16:25, schrieb Stefan (metze) Metzmacher:
> @Yann and Steve: thanks for the feedback!
> 
>>>> I'm currently researching how to implement SMBDirect [MS-SMBD]
>>>> together with the multi channel feature of SMB 3.0 in Samba.
>>>>
>>>> As Samba currently uses one process per tcp connection
>>>> and maintains a lot of in memory state within the process
>>>> (e.g. for the SMB_VFS modules) it would require a lot of work
>>>> to change Samba to coordinate two (or more) processes for one logical
>>>> multi channel connection.
>>>>
>>>> My current plan tries to pass the socket fd of new connections
>>>> (which join an existing multi channel session) via fd-passing to
>>>> the existing process.
>>>>
>>>> Now I'm wondering if this would also be possible with
>>>> a rdma connection (struct rdma_cm_i ).
>>>>
>>> RDMA / verbs ressources are tied to a process (especially Memory
>>> Registration), but it's ending up in the HCA, which is probably unaware
>>> of processes.
>>>
>>> Additionally, an RDMA_CM connection is not identified by a FD, so this
>>> kind of Unix trick (FD passing through Unix socket: SCM_RIGHTS) is not
>>> going to work.
>>>
>>> Forking might already be a challenge for a RDMA/verbs application, so I
>>> don't think that sharing/moving an RDMA_CM connection across different
>>> processes is supported.
>>>
>>> But other people on this list (especially Roland Dreier and Sean Hefty)
>>> could find a solution.
>>>
>>> Regards.
>>>
>>
>> fork() support like you need is not there in Linux RDMA verbs. Another
>> alternative is to fork() before you setup the RDMA connection.  IE if a
>> regular TCP socket is first used to negotiate RDMA mode, then maybe you
>> could fork() after negotiation but before setting up the RDMA connection
>> and other resources?
> 
> For client connections that would work, but it can't work for a server
> that uses fork() after listen().
> 
> Would it be possible to extend the librdmacm and libibverbs to:
> - handle fork() after getting RDMA_CM_EVENT_CONNECT_REQUEST
>   which means there have to be some functions to destroy contexts like
>   struct rdma_cm_id, but without affecting the real connection.
>   Similar to close() on an FD that is used shared between two processes.
> - export/import an interprocess token attached to the hierarchy of objects
>   similar to gss_export_sec_context/gss_import_sec_context.
>   Or a function which exports the contexts into a FD (of a pipe),
>   which could also transfer open FDs to the other end.
>   Maybe this could pass the information through a rdma/ibverb event channel?

It would be ok if there's a limitation, which means that the transfer to
a different process is only possible if there're:
- no posted work requests
- no queue pairs
- completion queues
active on the connection.

metze


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 259 bytes --]