nfsrdma broken on 2.6.34-rc1?

nfsrdma broken on 2.6.34-rc1?

[Steve Wise]

Hey Sean,

I'm trying NFSRDMA on net-next and the server side fails when 
registering the rdma transport.  I think its due to the INET6 support 
added to the rdma-cm.  I'm still debugging though.

In fs/nfsd/nfsctl.c:__write_ports_addxprt(), it tries to create a new 
svc transport for PF_INET, and PF_INET6 using the same port and the 
wildcard address.  If the INET6 fails with anything other than 
-EAFNOSUPPORT, then the entire transport registration fails (ie no 
RDMA/INET support is added). 

When I do echo "rdma 20049" > /proc/fs/nfsd/portlist  I see the PF_INET 
transport get created successfully, but the INET6 transport create fails 
with -EADDRNOTAVAIL. 

Does the rdma-cm allow concurrent binds to PF_INET, INADDR_ANY, port=X 
and PF_INET6, IN6ADDR_ANY_INIT, port=X ?  Apparently the native stack 
allows this (which makes sense seeing as how they are different protocol 
families).


Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Steve Wise]

Actually, this looks like a recent nfs change that now creates an INET 
and INET6 transport when you add one via the /proc/fs/nfsd/portlist 
file.  Looking at 2.6.30, which works, the nfsctl.c code is very 
different and only creates an INET transport...




Steve Wise wrote:
> Hey Sean,
>
> I'm trying NFSRDMA on net-next and the server side fails when 
> registering the rdma transport.  I think its due to the INET6 support 
> added to the rdma-cm.  I'm still debugging though.
>
> In fs/nfsd/nfsctl.c:__write_ports_addxprt(), it tries to create a new 
> svc transport for PF_INET, and PF_INET6 using the same port and the 
> wildcard address.  If the INET6 fails with anything other than 
> -EAFNOSUPPORT, then the entire transport registration fails (ie no 
> RDMA/INET support is added).
> When I do echo "rdma 20049" > /proc/fs/nfsd/portlist  I see the 
> PF_INET transport get created successfully, but the INET6 transport 
> create fails with -EADDRNOTAVAIL.
> Does the rdma-cm allow concurrent binds to PF_INET, INADDR_ANY, port=X 
> and PF_INET6, IN6ADDR_ANY_INIT, port=X ?  Apparently the native stack 
> allows this (which makes sense seeing as how they are different 
> protocol families).
>
>
> Steve.
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: nfsrdma broken on 2.6.34-rc1?

[Sean Hefty]

>Does the rdma-cm allow concurrent binds to PF_INET, INADDR_ANY, port=X
>and PF_INET6, IN6ADDR_ANY_INIT, port=X ?

No, since this shows up in the rdma_cm as using the same port space.

The rdma_cm might be able to support this if the port space were separated based
on the address family, depending on how PS IB ends up.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Steve Wise]

Sean Hefty wrote:
>
> The rdma_cm might be able to support this if the port space were separated based
> on the address family, depending on how PS IB ends up.

I think separate port spaces is the correct solution.

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Roland Dreier]

 > > The rdma_cm might be able to support this if the port space were separated based
 > > on the address family, depending on how PS IB ends up.
 > 
 > I think separate port spaces is the correct solution.

This gets a bit tricky -- for normal IP stuff, there's the "bindv6only"
sysctl (and the IPV6_V6ONLY socket option).  Without that, you can't
bind an IPv4 socket to the same port as an IPv6 socket, since the IPv6
socket will accept IPv4 connections via an v4->v6 mapped address.  (You
can look at inet_csk_bind_conflict() to see the full complexity of the
checking done when binding an IPv4 socket)

I wonder what the right way from the RDMA CM to stay close to Linux
sockets semantics without adding too much horror is.  (Adding Jason to
the CC list since he usually has an opinion about things like this :)

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Jason Gunthorpe]

On Mon, Mar 29, 2010 at 12:01:07PM -0700, Roland Dreier wrote:
>  > > The rdma_cm might be able to support this if the port space were separated based
>  > > on the address family, depending on how PS IB ends up.
>  > 
>  > I think separate port spaces is the correct solution.
> 
> This gets a bit tricky -- for normal IP stuff, there's the "bindv6only"
> sysctl (and the IPV6_V6ONLY socket option).  Without that, you can't
> bind an IPv4 socket to the same port as an IPv6 socket, since the IPv6
> socket will accept IPv4 connections via an v4->v6 mapped address.  (You
> can look at inet_csk_bind_conflict() to see the full complexity of the
> checking done when binding an IPv4 socket)

Yeah, exactly, it is very complex and there is a real need for
things pretending to be IP to capture all this subtlety. The details
can't just be skipped over, people will notice :(

Though, I'm also not entirely certain that NFS-RDMA is right to bind
to both AFs, generally speaking on Linux for a multi-protocol app you
only want to bind to v6 addresses.. Or is it using IPV6_V6ONLY or alike?

> I wonder what the right way from the RDMA CM to stay close to Linux
> sockets semantics without adding too much horror is.  (Adding Jason to
> the CC list since he usually has an opinion about things like this :)

Clearly the best way is to figure out some way to work with the
existing routines in the kernel. This stuff is complex and duplicating
all of it in rdma_cm would be annoying..

To match the semantics each CM ID would still register to one SID but
an incoming connection request on a v4 PF SID could be matched to a v6
SID, etc.

I don't think new port spaces in the API are desirable.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Steve Wise]

Jason Gunthorpe wrote:
> On Mon, Mar 29, 2010 at 12:01:07PM -0700, Roland Dreier wrote:
>   
>>  > > The rdma_cm might be able to support this if the port space were separated based
>>  > > on the address family, depending on how PS IB ends up.
>>  > 
>>  > I think separate port spaces is the correct solution.
>>
>> This gets a bit tricky -- for normal IP stuff, there's the "bindv6only"
>> sysctl (and the IPV6_V6ONLY socket option).  Without that, you can't
>> bind an IPv4 socket to the same port as an IPv6 socket, since the IPv6
>> socket will accept IPv4 connections via an v4->v6 mapped address.  (You
>> can look at inet_csk_bind_conflict() to see the full complexity of the
>> checking done when binding an IPv4 socket)
>>     
>
> Yeah, exactly, it is very complex and there is a real need for
> things pretending to be IP to capture all this subtlety. The details
> can't just be skipped over, people will notice :(
>
> Though, I'm also not entirely certain that NFS-RDMA is right to bind
> to both AFs, generally speaking on Linux for a multi-protocol app you
> only want to bind to v6 addresses.. Or is it using IPV6_V6ONLY or alike?
>
>   

This issue is really not in the NFS-RDMA code.  the nfsd code is doing 
the binding.  See commit:


37498292aa97658a5d0a9bb84699ce8c1016bb74
Author: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
Date:   Tue Jan 26 14:04:22 2010 -0500

    NFSD: Create PF_INET6 listener in write_ports


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Jason Gunthorpe]

On Mon, Mar 29, 2010 at 02:51:46PM -0500, Steve Wise wrote:

>> Yeah, exactly, it is very complex and there is a real need for
>> things pretending to be IP to capture all this subtlety. The details
>> can't just be skipped over, people will notice :(
>>
>> Though, I'm also not entirely certain that NFS-RDMA is right to bind
>> to both AFs, generally speaking on Linux for a multi-protocol app you
>> only want to bind to v6 addresses.. Or is it using IPV6_V6ONLY or alike?

> This issue is really not in the NFS-RDMA code.  the nfsd code is doing  
> the binding.  See commit:
>
> 37498292aa97658a5d0a9bb84699ce8c1016bb74
> Author: Chuck Lever <chuck.lever-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>
> Date:   Tue Jan 26 14:04:22 2010 -0500
>
>    NFSD: Create PF_INET6 listener in write_ports

Sure.. but it relies on the behavior of svcsock.c which does this:

        if (family == PF_INET6)
                kernel_setsockopt(sock, SOL_IPV6, IPV6_V6ONLY,
                                        (char *)&val, sizeof(val));

And the NFS-RDMA has no equivalent. Having the common code explicitly
rely on IPV6_V6ONLY is quite troublesome when you can't implement it
:)

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Roland Dreier]

 > This issue is really not in the NFS-RDMA code.  the nfsd code is doing
 > the binding.  See commit:

I think the really relevant thing is 7d21c0f9 ("SUNRPC: Set IPV6ONLY
flag on PF_INET6 RPC listener sockets") and followups.  NFS expects to
have one IPv6-only socket and one IPv4-only socket.

It seems RDMA CM should create a similar V6ONLY option for binding (and
probably default to the /proc/sys/net/ipv6/bindv6only sysctl value) to
handle this.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Tom Tucker]

Roland Dreier wrote:
>  > This issue is really not in the NFS-RDMA code.  the nfsd code is doing
>  > the binding.  See commit:
>
> I think the really relevant thing is 7d21c0f9 ("SUNRPC: Set IPV6ONLY
> flag on PF_INET6 RPC listener sockets") and followups.  NFS expects to
> have one IPv6-only socket and one IPv4-only socket.
>
>   

Yes, this new behavior is common to both sockets and rdma transports.

> It seems RDMA CM should create a similar V6ONLY option for binding (and
> probably default to the /proc/sys/net/ipv6/bindv6only sysctl value) to
> handle this.
>   
This makes sense to me.

Tom

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Steve Wise]

Roland Dreier wrote:
>  > This issue is really not in the NFS-RDMA code.  the nfsd code is doing
>  > the binding.  See commit:
>
> I think the really relevant thing is 7d21c0f9 ("SUNRPC: Set IPV6ONLY
> flag on PF_INET6 RPC listener sockets") and followups.  NFS expects to
> have one IPv6-only socket and one IPv4-only socket.
>
> It seems RDMA CM should create a similar V6ONLY option for binding (and
> probably default to the /proc/sys/net/ipv6/bindv6only sysctl value) to
> handle this.
>   

Sean, will you add this to the rdma_cm?


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: nfsrdma broken on 2.6.34-rc1?

[Sean Hefty]

>Sean, will you add this to the rdma_cm?

Not immediately because I lack the time to do it.

It would be really nice to share the kernel's port space code and remove the
port code in the rdma_cm.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Tom Tucker]

Sean Hefty wrote:
>> Sean, will you add this to the rdma_cm?
>>     
>
> Not immediately because I lack the time to do it.
>
> It would be really nice to share the kernel's port space code and remove the
> port code in the rdma_cm.
>
>   

LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?

> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Roland Dreier]

 > LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?

umm....

seriously is there anyway to un-screw this for 2.6.34?

hack around it in the NFS-RDMA server so it just skips the second bind
to the same port?

 - R.
-- 
Roland Dreier <rolandd-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Steve Wise]

Roland Dreier wrote:
>  > LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?
>
> umm....
>
> seriously is there anyway to un-screw this for 2.6.34?
>
> hack around it in the NFS-RDMA server so it just skips the second bind
> to the same port?
>
>   

If the nfsd code would ignore the error on the INET6 xprt create, then 
it works:

diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index 0f0e77f..d6abab9 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -1008,8 +1008,10 @@ static ssize_t __write_ports_addxprt(char *buf)
 
        err = svc_create_xprt(nfsd_serv, transport,
                                PF_INET6, port, SVC_SOCK_ANONYMOUS);
+#if 0
        if (err < 0 && err != -EAFNOSUPPORT)
                goto out_close;
+#endif
        return 0;
 out_close:
        xprt = svc_find_xprt(nfsd_serv, transport, PF_INET, port);

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: nfsrdma broken on 2.6.34-rc1?

[Tom Tucker]

Roland Dreier wrote:
>  > LOL. Yes...yes it would. There is of course a Dragon to be slain. Roland?
>
> umm....
>
> seriously is there anyway to un-screw this for 2.6.34?
>
> hack around it in the NFS-RDMA server so it just skips the second bind
> to the same port?
>
>   
Yes, I think all we have to do is have the nfsctl.c code ignore any 
transport error on the IPV6 call as opposed to _only_ NOTSUP

>  - R.
>   

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html