Fork safe clarification

Fork safe clarification

[Matthew Small]

I am trying to understand the behavior of the libibverbs after it has been
set into fork safe mode via a successful call to ibv_fork_init() or setting
the environmental variable IBV_FORK_SAFE.  For my purposes I would like to
know the following :

Are PDs, QPs and CQs created before a fork shared by the parent and child
after fork() has returned (ie. both can submit WRs, poll CQ, etc.)?


What about MRs registered before the fork?  Even though the child doesn't
have access to the parent's memory, can he sill submit WRs on a QP with an
MR created before the fork?


What if the MR pages in the above scenario are accessible in both parent and
child (shared memory)?  Are there complications with registering shared
memory?


In general, are pointers returned by libibverbs pointer to user/process
address space (as ibv_mr pointers must be) or kernel space (eg.  if an
unrelated process had another process's QP pointer, lkey, and a virtual
address could it post (almost certainly unsafely) a WR to the other
process's QP?


Sorry if the questions seem progressively more goofy and thanks in advance
for any clarification.

-Matt
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fork safe clarification

[Matthew Small]

I know many of the people here are very busy and these might be
questions that are not about your work at hand, but these intricacies
of libibverbs are hard to figure out without an intimate knowledge of
the drivers (from the user side).  I would really appreciate anyone
who would take the time to respond.

On Thu, Apr 1, 2010 at 10:09 AM, Matthew Small <matthewtsmall-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> I am trying to understand the behavior of the libibverbs after it has been
> set into fork safe mode via a successful call to ibv_fork_init() or setting
> the environmental variable IBV_FORK_SAFE.  For my purposes I would like to
> know the following :
>
> Are PDs, QPs and CQs created before a fork shared by the parent and child
> after fork() has returned (ie. both can submit WRs, poll CQ, etc.)?
>
>
> What about MRs registered before the fork?  Even though the child doesn't
> have access to the parent's memory, can he sill submit WRs on a QP with an
> MR created before the fork?
>
>
> What if the MR pages in the above scenario are accessible in both parent and
> child (shared memory)?  Are there complications with registering shared
> memory?
>
>
> In general, are pointers returned by libibverbs pointer to user/process
> address space (as ibv_mr pointers must be) or kernel space (eg.  if an
> unrelated process had another process's QP pointer, lkey, and a virtual
> address could it post (almost certainly unsafely) a WR to the other
> process's QP?
>
>
> Sorry if the questions seem progressively more goofy and thanks in advance
> for any clarification.
>
> -Matt
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Fork safe clarification

[Roland Dreier]

 > Are PDs, QPs and CQs created before a fork shared by the parent and child
 > after fork() has returned (ie. both can submit WRs, poll CQ, etc.)?

no, QPs and CQs are accessible only in the parent.  The child can still
use the uverbs file descriptor to do things, but libibverbs will
probably get very confused in this case.  More userspace development
would probably be required to make this really work.  Since the PD is
attached to the FD, it could be shared.

 > What about MRs registered before the fork?  Even though the child doesn't
 > have access to the parent's memory, can he sill submit WRs on a QP with an
 > MR created before the fork?

yes.

 > What if the MR pages in the above scenario are accessible in both parent and
 > child (shared memory)?  Are there complications with registering shared
 > memory?

shouldn't make a difference.

 > In general, are pointers returned by libibverbs pointer to user/process
 > address space (as ibv_mr pointers must be) or kernel space (eg.  if an
 > unrelated process had another process's QP pointer, lkey, and a virtual
 > address could it post (almost certainly unsafely) a WR to the other
 > process's QP?

Not sure I understand this.  All the pointers from libibverbs are of
course userspace pointers.  What could a userspace process do with a
kernel pointer?  Processes own all their resources and can't access
other resources.

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html