NFS over RDMA with no TCP/UDP

NFS over RDMA with no TCP/UDP

[Adam Brenner]

Howdy,

On our HPC cluster, we are exploring the option of mounting a NFS
client over RDMA where no TCP/UDP connection will be available. Both
the client and the server will have RDMA configured with OFED.

Is this possible?


If so, what address would we specify on the NFS client side as the
source host? iWARP address + port number? How I find out the iWARP
address? Is that the same as the GID given by the command ibaddr? Or
is this the Node GUID from ibstat? Do we have to safe escape or quote
this address for both the mount option and /etc/exports?

thanks,
/adam

--
Adam Brenner
Computer Science, Undergraduate Student
Donald Bren School of Information and Computer Sciences

System Administrator, HPC Cluster
Office of Information Technology
http://hpc.oit.uci.edu/

University of California, Irvine
www.ics.uci.edu/~aebrenne/
aebrenne-sXc7qaQca9o@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NFS over RDMA with no TCP/UDP

[Caitlin Bestler]

The DDP draft alludes to the requirements for the generic transport layer that support iWARP.
You then have two concrete examples: TCP and SCTP.

One of the more critical requirements is that the LLP delivers all DDP Segments in order
or fails the DDP Connection/Association/Whatever.

DDP does not handle reliability or retries on its own. So if you wanted to build
“iWARP over raw Ethernet” you would have to build an intermediate reliability layer.

Now you might be able to define a new transport layer well tailored to your target
environment, or perhaps even better adapter to carrying DDP Segments, but it is
still a new transport layer.

That is not a trivial undertaking.

Attempting to map RPC over UDP directly to some sort of unreliable DDP transport
would have to address the safety guarantees built into iWARP that currently rely on
the reliable LLP layer. You do not want your NIC to provide access to your memory
to just any random packet. Unreliable RPC can accumulate and then authenticate
the entire command before executing it. NFS over RMDA is really RPC over RDMA,
but it utilizes RDMA transfers which transfer directly into user buffers without waiting
for the command to be validated as a whole. So not only are you recreating congestion
control and retransmission, but the authentication provided by a connection-oriented
session.

On Oct 6, 2014, at 12:11 PM, Adam Brenner <aebrenne-sXc7qaQca9o@public.gmane.org> wrote:

> Howdy,
> 
> On our HPC cluster, we are exploring the option of mounting a NFS
> client over RDMA where no TCP/UDP connection will be available. Both
> the client and the server will have RDMA configured with OFED.
> 
> Is this possible?
> 
> er
> If so, what address would we specify on the NFS client side as the
> source host? iWARP address + port number? How I find out the iWARP
> address? Is that the same as the GID given by the command ibaddr? Or
> is this the Node GUID from ibstat? Do we have to safe escape or quote
> this address for both the mount option and /etc/exports?
> 
> thanks,
> /adam
> 
> --
> Adam Brenner
> Computer Science, Undergraduate Student
> Donald Bren School of Information and Computer Sciences
> 
> System Administrator, HPC Cluster
> Office of Information Technology
> http://hpc.oit.uci.edu/
> 
> University of California, Irvine
> www.ics.uci.edu/~aebrenne/
> aebrenne-sXc7qaQca9o@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: NFS over RDMA with no TCP/UDP

[Steve Wise]

On 10/6/2014 2:11 PM, Adam Brenner wrote:
> Howdy,
>
> On our HPC cluster, we are exploring the option of mounting a NFS
> client over RDMA where no TCP/UDP connection will be available. Both
> the client and the server will have RDMA configured with OFED.
>
> Is this possible?

I'm not sure what you mean by "where no TCP/UDP connection will be 
available."?  What iWARP devices are you using?

>
> If so, what address would we specify on the NFS client side as the
> source host? iWARP address + port number? How I find out the iWARP
> address? Is that the same as the GID given by the command ibaddr? Or
> is this the Node GUID from ibstat? Do we have to safe escape or quote
> this address for both the mount option and /etc/exports?

Basically you configure the iWARP device's ethernet interface for normal 
NIC traffic, and use that IP address for the NFSRDMA mount. See 
Documentation/filesystems/nfs/nfs-rdma.txt for details.

Steve.


> thanks,
> /adam
>
> --
> Adam Brenner
> Computer Science, Undergraduate Student
> Donald Bren School of Information and Computer Sciences
>
> System Administrator, HPC Cluster
> Office of Information Technology
> http://hpc.oit.uci.edu/
>
> University of California, Irvine
> www.ics.uci.edu/~aebrenne/
> aebrenne-sXc7qaQca9o@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html