ibv_fork_init() question

ibv_fork_init() question

[Steve Wise]

Hey Roland,

I'm experimenting with doing RDMA operations from a child process and I have a question.  My application calls 
ibv_fork_init() at startup (first thing in main())  and it succeeds, and I see, for instance, that libibverbs has 
initialized mm_root.  Next my application registers page-aligned memory that is 8KB in size.  The memory was allocated 
with memalign().  Then the app sets up a connection to an RDMA peer.  Next the application calls fork().  The parent 
does a wait() and the child does an rdma read operation into this registered memory from a valid mr of the peer.  The 
read succeeds from the rdma device perspective.  But when the child process tries to read the memory (to verify the rdma 
operation worked), I get a seg fault.

Q: Am I doing something incorrect here?  I can't find much documentation on how this all should work.

Thanks!

Steve.


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ibv_fork_init() question

[Roland Dreier]

On Mon, Feb 27, 2012 at 11:54 AM, Steve Wise
<swise-7bPotxP6k4+P2YhJcF5u+vpXobYPEAuW@public.gmane.org> wrote:
> I'm experimenting with doing RDMA operations from a child process and I have
> a question.  My application calls ibv_fork_init() at startup (first thing in
> main())  and it succeeds, and I see, for instance, that libibverbs has
> initialized mm_root.  Next my application registers page-aligned memory that
> is 8KB in size.  The memory was allocated with memalign().  Then the app
> sets up a connection to an RDMA peer.  Next the application calls fork().
>  The parent does a wait() and the child does an rdma read operation into
> this registered memory from a valid mr of the peer.  The read succeeds from
> the rdma device perspective.  But when the child process tries to read the
> memory (to verify the rdma operation worked), I get a seg fault.
>
> Q: Am I doing something incorrect here?  I can't find much documentation on
> how this all should work.

Yeah, unfortunately this isn't really intended to work and I don't have a
good suggestion on how to make it work.

Even with the libibverbs fork support, the expectation is that all RDMA
stuff will be done from the master process.  All that the ibv_fork_init()
call does is trigger tracking so that the address space that is used for
RDMA isn't mapped into the child process at all.

So yeah, what you see happening is exactly what I would expect.

 - R.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html