[PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[Hefty, Sean]

When an rsocket call is made before an application calls fork(),
the forked applications can hang.  This can be seen by running
netserver and two netperf clients simultaneously.  The second
netperf client will eventually stop performing data transfers.

LD_PRELOAD=librspreload.so netserver -D

LD_PRELOAD=librspreload.so netperf -v2 -c -C -H 192.168.0.101 -l30
LD_PRELOAD=librspreload.so netperf -v2 -c -C -H 192.168.0.101 -l30

It's not clear what the specific problem is.  The best guess is
that libibverbs or the provider library (e.g. libmlx4) perform
some initialization, such as mmap'ing device memory, which does not
work when fork is called.

As a work-around, avoid calling rsocket routines until immediately
before they are needed.  This allows the process to fork before
the libraries are initialized.

Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
---
 src/preload.c |   21 +++++++++++----------
 1 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/src/preload.c b/src/preload.c
index 0645f6d..4ba38f5 100644
--- a/src/preload.c
+++ b/src/preload.c
@@ -412,20 +412,21 @@ int socket(int domain, int type, int protocol)
 	if (index < 0)
 		return index;
 
+	if (fork_support && (domain == PF_INET || domain == PF_INET6) &&
+	    (type == SOCK_STREAM) && (!protocol || protocol == IPPROTO_TCP)) {
+		ret = real.socket(domain, type, protocol);
+		if (ret < 0)
+			return ret;
+		fd_store(index, ret, fd_normal, fd_fork);
+		return index;
+	}
+
 	recursive = 1;
 	ret = rsocket(domain, type, protocol);
 	recursive = 0;
 	if (ret >= 0) {
-		if (fork_support) {
-			rclose(ret);
-			ret = real.socket(domain, type, protocol);
-			if (ret < 0)
-				return ret;
-			fd_store(index, ret, fd_normal, fd_fork);
-		} else {
-			fd_store(index, ret, fd_rsocket, fd_ready);
-			set_rsocket_options(ret);
-		}
+		fd_store(index, ret, fd_rsocket, fd_ready);
+		set_rsocket_options(ret);
 		return index;
 	}
 	fd_close(index, &ret);


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[Or Gerlitz]

On 23/08/2012 21:51, Hefty, Sean wrote:
> When an rsocket call is made before an application calls fork(),
> the forked applications can hang.  This can be seen by running
> netserver and two netperf clients simultaneously.  The second
> netperf client will eventually stop performing data transfers.
>
> LD_PRELOAD=librspreload.so netserver -D
>
> LD_PRELOAD=librspreload.so netperf -v2 -c -C -H 192.168.0.101 -l30
> LD_PRELOAD=librspreload.so netperf -v2 -c -C -H 192.168.0.101 -l30
>
> It's not clear what the specific problem is.  The best guess is
> that libibverbs or the provider library (e.g. libmlx4) perform
> some initialization, such as mmap'ing device memory, which does not
> work when fork is called.

Yishai,

I think you pointed me to that problem the other day, do you have a 
patch to fix that?

Or.

>
> As a work-around, avoid calling rsocket routines until immediately
> before they are needed.  This allows the process to fork before
> the libraries are initialized.
>
> Signed-off-by: Sean Hefty <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
> ---
>   src/preload.c |   21 +++++++++++----------
>   1 files changed, 11 insertions(+), 10 deletions(-)
>
> diff --git a/src/preload.c b/src/preload.c
> index 0645f6d..4ba38f5 100644
> --- a/src/preload.c
> +++ b/src/preload.c
> @@ -412,20 +412,21 @@ int socket(int domain, int type, int protocol)
>   	if (index < 0)
>   		return index;
>   
> +	if (fork_support && (domain == PF_INET || domain == PF_INET6) &&
> +	    (type == SOCK_STREAM) && (!protocol || protocol == IPPROTO_TCP)) {
> +		ret = real.socket(domain, type, protocol);
> +		if (ret < 0)
> +			return ret;
> +		fd_store(index, ret, fd_normal, fd_fork);
> +		return index;
> +	}
> +
>   	recursive = 1;
>   	ret = rsocket(domain, type, protocol);
>   	recursive = 0;
>   	if (ret >= 0) {
> -		if (fork_support) {
> -			rclose(ret);
> -			ret = real.socket(domain, type, protocol);
> -			if (ret < 0)
> -				return ret;
> -			fd_store(index, ret, fd_normal, fd_fork);
> -		} else {
> -			fd_store(index, ret, fd_rsocket, fd_ready);
> -			set_rsocket_options(ret);
> -		}
> +		fd_store(index, ret, fd_rsocket, fd_ready);
> +		set_rsocket_options(ret);
>   		return index;
>   	}
>   	fd_close(index, &ret);
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[Or Gerlitz]

On 23/08/2012 21:51, Hefty, Sean wrote:
> It's not clear what the specific problem is.  The best guess is
> that libibverbs or the provider library (e.g. libmlx4) perform
> some initialization, such as mmap'ing device memory, which does not
> work when fork is called.

Are you calling from rsockets to ibv_fork_init or setting one of 
libibverb's  yyy_FORK_SAFE env vars?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[Hefty, Sean]

> Are you calling from rsockets to ibv_fork_init or setting one of
> libibverb's  yyy_FORK_SAFE env vars?

rsockets enables fork support only when RDMAV_FORK_SAFE has been set.  I do not call ibv_fork_init().

- Sean 
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[Or Gerlitz]

On Mon, Aug 27, 2012 at 8:54 PM, Hefty, Sean <sean.hefty-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> rsockets enables fork support only when RDMAV_FORK_SAFE has been set.
> I do not call ibv_fork_init().

understood, so you see the problem also when RDMAV_FORK_SAFE has been set?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] libdmacm/rspreload: Avoid rsocket calls until after fork

[Hefty, Sean]

> > rsockets enables fork support only when RDMAV_FORK_SAFE has been set.
> > I do not call ibv_fork_init().
> 
> understood, so you see the problem also when RDMAV_FORK_SAFE has been set?

yes
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html