[RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[Tom Tucker]

Adds a call to the svc_makesock routine to create the RDMA transport. 
I think this should somehow be linked to the transport switch and the 
port mapper. 

Signed-off-by: Tom Tucker <tom@opengridcomputing.com>
---

 fs/nfsd/nfssvc.c |   11 +++++++++++
 1 files changed, 11 insertions(+), 0 deletions(-)

diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
index d7759ce..4a7982e 100644
--- a/fs/nfsd/nfssvc.c
+++ b/fs/nfsd/nfssvc.c
@@ -254,6 +254,17 @@ #ifdef CONFIG_NFSD_TCP
 	if (error < 0)
 		return error;
 #endif
+#ifdef CONFIG_NFSD_RDMA
+	/* KLUDGE: The rpc.nfsd utility passes down the
+	 * port number when it opens the nfsd file. We are
+	 * going to ignore it here and force the RDMA port number
+	 * to 2050. This will allow the TCP transport to coexist
+	 * with the RDMA transport.
+	 */
+	error = svc_makesock(nfsd_serv, IPPROTO_MAX, 2050 /* port */, 0);
+	if (error < 0)
+		return error;
+#endif
 	return 0;
 }
 


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[Neil Brown]

On Friday May 18, tom@opengridcomputing.com wrote:
> Adds a call to the svc_makesock routine to create the RDMA transport. 
> I think this should somehow be linked to the transport switch and the 
> port mapper. 
> 
> Signed-off-by: Tom Tucker <tom@opengridcomputing.com>
> ---
> 
>  fs/nfsd/nfssvc.c |   11 +++++++++++
>  1 files changed, 11 insertions(+), 0 deletions(-)
> 
> diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
> index d7759ce..4a7982e 100644
> --- a/fs/nfsd/nfssvc.c
> +++ b/fs/nfsd/nfssvc.c
> @@ -254,6 +254,17 @@ #ifdef CONFIG_NFSD_TCP
>  	if (error < 0)
>  		return error;
>  #endif
> +#ifdef CONFIG_NFSD_RDMA
> +	/* KLUDGE: The rpc.nfsd utility passes down the
> +	 * port number when it opens the nfsd file. We are
> +	 * going to ignore it here and force the RDMA port number
> +	 * to 2050. This will allow the TCP transport to coexist
> +	 * with the RDMA transport.
> +	 */
> +	error = svc_makesock(nfsd_serv, IPPROTO_MAX, 2050 /* port */, 0);
> +	if (error < 0)
> +		return error;
> +#endif
>  	return 0;
>  }
>  

As you say:  KLUDGE.

What is really going on here?  Why don't we have a protocol number,
and why can't we use port 2049?

I don't know a lot about RDMA (feel free to educate me) but it seems
that there is some connection to IP (is this IPoIB??) in that IP
address seem to be available for the end points.

Can you distinguish an RDMA connection from any other IP connection
simply based on the interface it gets routed through?

Can you actually get any sort of socket representing an RDMA
connection?  I appreciate that normal sendmsg/recvmsg calls on the
socket aren't going to work, but I'm dreaming that maybe a
socket-level 'accept' could establish the connection.  Then we look at
the connection, discover it is on a RDMA-capable device, and start
using the RDMA interface to move data around.
Does that make any sense at all, or am I totally off-beam?

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[Tom Tucker]

On Mon, 2007-05-21 at 17:04 +1000, Neil Brown wrote:
> On Friday May 18, tom@opengridcomputing.com wrote:
> > Adds a call to the svc_makesock routine to create the RDMA transport. 
> > I think this should somehow be linked to the transport switch and the 
> > port mapper. 
> > 
> > Signed-off-by: Tom Tucker <tom@opengridcomputing.com>
> > ---
> > 
> >  fs/nfsd/nfssvc.c |   11 +++++++++++
> >  1 files changed, 11 insertions(+), 0 deletions(-)
> > 
> > diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c
> > index d7759ce..4a7982e 100644
> > --- a/fs/nfsd/nfssvc.c
> > +++ b/fs/nfsd/nfssvc.c
> > @@ -254,6 +254,17 @@ #ifdef CONFIG_NFSD_TCP
> >  	if (error < 0)
> >  		return error;
> >  #endif
> > +#ifdef CONFIG_NFSD_RDMA
> > +	/* KLUDGE: The rpc.nfsd utility passes down the
> > +	 * port number when it opens the nfsd file. We are
> > +	 * going to ignore it here and force the RDMA port number
> > +	 * to 2050. This will allow the TCP transport to coexist
> > +	 * with the RDMA transport.
> > +	 */
> > +	error = svc_makesock(nfsd_serv, IPPROTO_MAX, 2050 /* port */, 0);
> > +	if (error < 0)
> > +		return error;
> > +#endif
> >  	return 0;
> >  }
> >  
> 
> As you say:  KLUDGE.
> 
> What is really going on here?  Why don't we have a protocol number,
> and why can't we use port 2049?
> 
> I don't know a lot about RDMA (feel free to educate me) but it seems
> that there is some connection to IP (is this IPoIB??) in that IP
> address seem to be available for the end points.

"RDMA" actually refers to two different transports: IB (not IPoIB), and
iWARP (RDDP on TCP). In the first case, IP is not the transport (so no
number), in the second case, TCP is actually the IP protocol, so there
is no RDMA-unique protocol number. I just stole IPPROTO_MAX as an ...
um ... workaround. 

What I think we need here is a "transport number". Transports that
register with the transport switch have a number that is passed in here
to create a listening endpoint of the appropriate type. This
infrastructure isn't there yet, but I believe Greg is noodling on a way
to do this.

> 
> Can you distinguish an RDMA connection from any other IP connection
> simply based on the interface it gets routed through?
> 

Unfortunately no for two reasons: iWARP interfaces support all of the
protocol types with the same IP address, therefore, the port number 2050
so we can distinguish between listening endpoints for TCP mounts vs.
iWARP mounts. Port number 2050 is being proposed in the IANA for the
NFS-RDMA service. 

> Can you actually get any sort of socket representing an RDMA
> connection?  I appreciate that normal sendmsg/recvmsg calls on the
> socket aren't going to work, but I'm dreaming that maybe a
> socket-level 'accept' could establish the connection.  Then we look at
> the connection, discover it is on a RDMA-capable device, and start
> using the RDMA interface to move data around.
> Does that make any sense at all, or am I totally off-beam?

No, you're right in the middle of it. The OFA API has a sockets-like
interface for connection management. The API has things like
rdma_create_id (socket), rdma_listen (listen), rdma_connect (connect),
etc... It is, however, not a socket. It is certainly possible to create
a sockets abstraction and in-fact it has been done, but the compromises
involved (no private data, no path record selection, etc...) were deemed
to onerous and they created there own.

Tom
> 
> NeilBrown


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[Neil Brown]

I must sat that I am hitting acronym-overload hear.
RDMA  IB iWARP OFA-API SQ-CQ  SQ-WR sge  max_qp_rd_atom ....

But to the topic of registering the RDMA listening point....

I now understand the point of port 2050 I think.  RDMA adds to the
protocol.  As well as all the bytes of the RPC request, there is
information about different ..uhm... regions (?) of the message.
This is like a scatter-gather list?
It lets you put the "write" data correctly aligned into a page,
so that we could eventually use the 'splice' technology to achieve
zero-copy write.

But we still have this concept of a different transport to handle
properly. 

A bit of an aside: You mention that with "IB", IP is not used, so there
is no number.  I assume you mean no IP address of the client?  In that
situation, how do we identify the client for authorisation purposes?

More on-topic, we need to consider how this interacts with 
   /proc/fs/nfsd/portlist 

This file can be written to and read from.
When writing, you write a decimal number of a file descriptor.
That fd should be a socket on which to expect incoming requests -
either a UDP socket or a TCP socket that is listening.
How can we extend that to RDMA?  What sort of handle does user-space
use for talking over one of these DDP interfaces?
We could arrange that writing e.g.
    RDMA TCP 2050
did what you want, but I would much rather avoid that sort of stuff.

When reading from a file you get one line per active transport:
ipv4 tcp 0.0.0.0 2049
ipv4 udp 0.0.0.0 2049

What would we read for RDMA?  You say that it uses TCP.  Can it use
UDP instead?  Might it make sense to listen on only one interface?
Is there an IPv6 version of RDMA??

It seems like a real pity that it couldn't get shoe-horned into a
socket interface.
It would seem that the msg_control part of sendmsg/recvmsg would be
ideal for managing the details of data placement.

NeilBrown

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[Tom Tucker]

On Tue, 2007-05-22 at 16:21 +1000, Neil Brown wrote:
> I must sat that I am hitting acronym-overload hear.
> RDMA  IB iWARP OFA-API SQ-CQ  SQ-WR sge  max_qp_rd_atom ....
> 

Yes, it's a bit daunting.

> But to the topic of registering the RDMA listening point....
> 
> I now understand the point of port 2050 I think.  RDMA adds to the
> protocol.  As well as all the bytes of the RPC request, there is
> information about different ..uhm... regions (?) of the message.

Yes, they call them "chunks" and the RPCRDMA header contains a
"chunk-list". The chunk-list (built by the client and used by the
server) tells the transport where in the XDR to place data.

> This is like a scatter-gather list?

Yes, three in fact: one for reading data from the client's memory and
placing it in the request (read-list), one for writing data into the
client's memory as part of the REPLY (write-list), one for the REPLY
header itself (reply-list).

> It lets you put the "write" data correctly aligned into a page,
> so that we could eventually use the 'splice' technology to achieve
> zero-copy write.
> 
Yes.

> But we still have this concept of a different transport to handle
> properly. 
> 
> A bit of an aside: You mention that with "IB", IP is not used, so there
> is no number.  I assume you mean no IP address of the client?  In that
> situation, how do we identify the client for authorisation purposes?

The RDMA CMA (connection management agent) has a "sockets-like" API for
transport independent connection management. This API uses IP addresses
and port numbers for both IB and iWARP. To implement this on IB, IPoIB
and IP addresses are used for connection management, but not for data.
So for the purposes of authorization, it all works the same for all
transports. 

For the purpose of transport selection, we just need a unique id. I was
simply pointing out that IP protocol numbers don't uniquely identify
transports for RDMA.

> More on-topic, we need to consider how this interacts with 
>    /proc/fs/nfsd/portlist 

> This file can be written to and read from.
> When writing, you write a decimal number of a file descriptor.
> That fd should be a socket on which to expect incoming requests -
> either a UDP socket or a TCP socket that is listening.
> How can we extend that to RDMA?  What sort of handle does user-space
> use for talking over one of these DDP interfaces?
> We could arrange that writing e.g.
>     RDMA TCP 2050
> did what you want, but I would much rather avoid that sort of stuff.
> 
> When reading from a file you get one line per active transport:
> ipv4 tcp 0.0.0.0 2049
> ipv4 udp 0.0.0.0 2049
> 
> What would we read for RDMA?

I think this makes sense:

ipv4 rdma 0.0.0.0 2050
ipv6 rdma 0.0.0.0 2050

>   You say that it uses TCP.

It currently uses TCP or IB, but could ultimately use SCTP, etc...

>   Can it use
> UDP instead? 

No.

>  Might it make sense to listen on only one interface?

This has been discussed, but it runs against this "converged NIC" or
"universal NIC" idea that there is one NIC, one interface and one IP
address that solves ALL your problems. They even eat acronyms :-)

> Is there an IPv6 version of RDMA??

Eventually.

> 
> It seems like a real pity that it couldn't get shoe-horned into a
> socket interface.

I think it is a matter of the application of sufficient force ;-)  In
fairness, the I/O model is very different and the buffer management is
very-very different. This coupled with the extreme performance pressures
put on these transports led to the creation of a new API. Just getting
enough abstraction to combine iWARP and IB into a single API took a lot
of convincing.

> It would seem that the msg_control part of sendmsg/recvmsg would be
> ideal for managing the details of data placement.
> 

Er, probably not enough (or too much) there, but connection management
-- absolutely.

> NeilBrown


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [RFC,PATCH 7/15] knfsd: create RDMA transport in nfssvc

[Talpey, Thomas]

Sorry, I've been travelling until this morning. Replies to port 2050 questions
and other transport issues below.

At 11:59 AM 5/22/2007, Tom Tucker wrote:
>On Tue, 2007-05-22 at 16:21 +1000, Neil Brown wrote:
>> I now understand the point of port 2050 I think.  RDMA adds to the
>> protocol.  As well as all the bytes of the RPC request, there is

I don't want to confuse things again, but I do want to add something about
the "protocol" question. As I mentioned before, the notion of an RPC transport
is actually two things, an API and a set of protocol semantics, e.g. sockets/TCP,
which we somewhat conflate into a single designator.

In the case of RDMA, the underlying protocol is actually not important. The
RDMA semantics are well-defined for any transport (you can read about
the RPC/RDMA protocol assumptions in the internet draft). Basically, the
RDMA abstraction hides the protocol almost completely.

[RDMA requirements are on page 3 ff]
<http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-rpcrdma-05.txt>

The exception, which turns out to be not much of an exception at all,
is the addressing. The reason it's not an issue is the Infiniband connection
manager, which magically allows IP addresses (IPv4 currently) to be passed
to connect. As a result, there's no need for an RDMA protocol selector,
the IP connect routing handles hardware selection and hides any notion
that IB or iWARP is handling the connection. This is a Good Thing, it makes
RPC totally RDMA-implementation agnostic.

As for why port 2050 exists, it's because the server needs to know whether
to expect RPC/TCP framing or RPC/RDMA on an incoming connection over
iWARP. Over Infiniband there's no ambiguity since connections are always
in RDMA mode, but over iWARP there is an optional "step-up" negotiation.
We need a new port to know whether the negotiation has been bypassed.
There is discussion of this in the nfsdirect Internet Draft.

[Port discussion on page 6 ff]
<http://www.ietf.org/internet-drafts/draft-ietf-nfsv4-nfsdirect-05.txt>

>For the purpose of transport selection, we just need a unique id. I was
>simply pointing out that IP protocol numbers don't uniquely identify
>transports for RDMA.

I'll pile on here. This is important, it goes to the heart of how we "name" RPC
transports. Currently, the client switch uses a protocol id (UDP=17, TCP=6),
and this number is the de facto name of the transport. The address family
of the server's address also comes into play.

For now, we've simply stolen either 255 or 256 to register the RDMA transport
(we changed the numbers at one point IIRC). This may be okay, or maybe not.
Personally, I'd prefer a string-based transport naming, but numbers are fine
too, as long as everyone agrees.

>> When reading from a file you get one line per active transport:
>> ipv4 tcp 0.0.0.0 2049
>> ipv4 udp 0.0.0.0 2049
>> 
>> What would we read for RDMA?
>
>I think this makes sense:
>
>ipv4 rdma 0.0.0.0 2050
>ipv6 rdma 0.0.0.0 2050

Like that - strings. The normal TCP transport would be "ipv4-tcp" for
instance.

For RDMA however, the "ipv4" is only significant to the RDMA connection
manager. It serves only to describe the format of the following address.
So while it makes sense, it sort of doesn't matter.

By the way, "ipv6 rdma 0.0.0.0 2050" doesn't make sense, because "0.0.0.0"
isn't an ipv6 address.


>>   You say that it uses TCP.
>
>It currently uses TCP or IB, but could ultimately use SCTP, etc...

It makes absolutely no difference whether TCP is used as the RDMA transport,
from the perspoective of the upper layer. The semantics would be identical
if SCTP were used. So, I would not include "tcp" in the rdma transport selector
at all.

Tom.


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs