[PATCH] update NFS/RDMA documentation

[PATCH] update NFS/RDMA documentation

[James Lentini]

Bruce,

Below is an update to the NFS/RDMA documentation that clarifies how to 
run mount.nfs and addresses comments from you and Chuck. Could you 
please merge this for 2.6.26?

Signed-off-by: James Lentini <jlentini@netapp.com>

--- a/Documentation/filesystems/nfs-rdma.txt	2008-05-30 16:03:01.414533000 -0400
+++ a/Documentation/filesystems/nfs-rdma.txt	2008-06-02 15:26:51.138344000 -0400
@@ -5,7 +5,7 @@
 ################################################################################
 
  Author: NetApp and Open Grid Computing
- Date: April 15, 2008
+ Date: May 29, 2008
 
 Table of Contents
 ~~~~~~~~~~~~~~~~~
@@ -60,16 +60,18 @@ Installation
     The procedures described in this document have been tested with
     distributions from Red Hat's Fedora Project (http://fedora.redhat.com/).
 
-  - Install nfs-utils-1.1.1 or greater on the client
+  - Install nfs-utils-1.1.2 or greater on the client
 
-    An NFS/RDMA mount point can only be obtained by using the mount.nfs
-    command in nfs-utils-1.1.1 or greater. To see which version of mount.nfs
-    you are using, type:
+    An NFS/RDMA mount point can be obtained by using the mount.nfs command in
+    nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version
+    with support for NFS/RDMA mounts, but for various reasons we recommend using
+    nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are
+    using, type:
 
-    > /sbin/mount.nfs -V
+    $ /sbin/mount.nfs -V
 
-    If the version is less than 1.1.1 or the command does not exist,
-    then you will need to install the latest version of nfs-utils.
+    If the version is less than 1.1.2 or the command does not exist,
+    you should install the latest version of nfs-utils.
 
     Download the latest package from:
 
@@ -77,22 +79,32 @@ Installation
 
     Uncompress the package and follow the installation instructions.
 
-    If you will not be using GSS and NFSv4, the installation process
-    can be simplified by disabling these features when running configure:
+    If you will not need the idmapper and gssd executables (you do not need
+    these to create an NFS/RDMA enabled mount command), the installation
+    process can be simplified by disabling these features when running
+    configure:
 
-    > ./configure --disable-gss --disable-nfsv4
+    $ ./configure --disable-gss --disable-nfsv4
 
-    For more information on this see the package's README and INSTALL files.
+    To build nfs-utils you will need the tcp_wrappers package installed. For
+    more information on this see the package's README and INSTALL files.
 
     After building the nfs-utils package, there will be a mount.nfs binary in
     the utils/mount directory. This binary can be used to initiate NFS v2, v3,
     or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4.
     The standard technique is to create a symlink called mount.nfs4 to mount.nfs.
 
-    NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is only needed
+    This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
+
+    $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
+
+    In this location, mount.nfs will be invoked automatically for NFS mounts
+    by the system mount commmand.
+
+    NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed
     on the NFS client machine. You do not need this specific version of
     nfs-utils on the server. Furthermore, only the mount.nfs command from
-    nfs-utils-1.1.1 is needed on the client.
+    nfs-utils-1.1.2 is needed on the client.
 
   - Install a Linux kernel with NFS/RDMA
 
@@ -156,8 +168,8 @@ Check RDMA and NFS Setup
     this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel
     card:
 
-    > modprobe ib_mthca
-    > modprobe ib_ipoib
+    $ modprobe ib_mthca
+    $ modprobe ib_ipoib
 
     If you are using InfiniBand, make sure there is a Subnet Manager (SM)
     running on the network. If your IB switch has an embedded SM, you can
@@ -166,7 +178,7 @@ Check RDMA and NFS Setup
 
     If an SM is running on your network, you should see the following:
 
-    > cat /sys/class/infiniband/driverX/ports/1/state
+    $ cat /sys/class/infiniband/driverX/ports/1/state
     4: ACTIVE
 
     where driverX is mthca0, ipath5, ehca3, etc.
@@ -174,10 +186,10 @@ Check RDMA and NFS Setup
     To further test the InfiniBand software stack, use IPoIB (this
     assumes you have two IB hosts named host1 and host2):
 
-    host1> ifconfig ib0 a.b.c.x
-    host2> ifconfig ib0 a.b.c.y
-    host1> ping a.b.c.y
-    host2> ping a.b.c.x
+    host1$ ifconfig ib0 a.b.c.x
+    host2$ ifconfig ib0 a.b.c.y
+    host1$ ping a.b.c.y
+    host2$ ping a.b.c.x
 
     For other device types, follow the appropriate procedures.
 
@@ -214,9 +226,9 @@ NFS/RDMA Setup
 
     For InfiniBand using a Mellanox adapter:
 
-    > modprobe ib_mthca
-    > modprobe ib_ipoib
-    > ifconfig ib0 a.b.c.d
+    $ modprobe ib_mthca
+    $ modprobe ib_ipoib
+    $ ifconfig ib0 a.b.c.d
 
     NOTE: use unique addresses for the client and server
 
@@ -225,30 +237,31 @@ NFS/RDMA Setup
     If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
     load the RDMA transport module:
 
-    > modprobe svcrdma
+    $ modprobe svcrdma
 
     Regardless of how the server was built (module or built-in), start the server:
 
-    > /etc/init.d/nfs start
+    $ /etc/init.d/nfs start
 
     or
 
-    > service nfs start
+    $ service nfs start
 
     Instruct the server to listen on the RDMA transport:
 
-    > echo rdma 2050 > /proc/fs/nfsd/portlist
+    $ echo rdma 2050 > /proc/fs/nfsd/portlist
 
   - On the client system
 
     If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
     load the RDMA client module:
 
-    > modprobe xprtrdma.ko
+    $ modprobe xprtrdma.ko
 
-    Regardless of how the client was built (module or built-in), issue the mount.nfs command:
+    Regardless of how the client was built (module or built-in), use this command to 
+    mount the NFS/RDMA server:
 
-    > /path/to/your/mount.nfs <IPoIB-server-name-or-address>:/<export> /mnt -i -o rdma,port=2050
+    $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt
 
     To verify that the mount is using RDMA, run "cat /proc/mounts" and check the
     "proto" field for the given mount.

Re: [PATCH] update NFS/RDMA documentation

[J. Bruce Fields]

On Mon, Jun 02, 2008 at 03:33:59PM -0400, James Lentini wrote:
> 
> Bruce,
> 
> Below is an update to the NFS/RDMA documentation that clarifies how to 
> run mount.nfs and addresses comments from you and Chuck. Could you 
> please merge this for 2.6.26?

Yep, thanks.

Mind if we keep this wrapped at under 80 characters?

--b.

commit 80d7681a66c1d800fd6a8e03346219f5e8de185d
Author: J. Bruce Fields <bfields@citi.umich.edu>
Date:   Mon Jun 2 16:01:51 2008 -0400

    nfs: rewrap NFS/RDMA documentation to 80 lines
    
    Wrap long lines.
    
    Signed-off-by: J. Bruce Fields <bfields@citi.umich.edu>

diff --git a/Documentation/filesystems/nfs-rdma.txt b/Documentation/filesystems/nfs-rdma.txt
index 9ad453d..44bd766 100644
--- a/Documentation/filesystems/nfs-rdma.txt
+++ b/Documentation/filesystems/nfs-rdma.txt
@@ -63,10 +63,10 @@ Installation
   - Install nfs-utils-1.1.2 or greater on the client
 
     An NFS/RDMA mount point can be obtained by using the mount.nfs command in
-    nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version
-    with support for NFS/RDMA mounts, but for various reasons we recommend using
-    nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are
-    using, type:
+    nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils
+    version with support for NFS/RDMA mounts, but for various reasons we
+    recommend using nfs-utils-1.1.2 or greater). To see which version of
+    mount.nfs you are using, type:
 
     $ /sbin/mount.nfs -V
 
@@ -91,8 +91,9 @@ Installation
 
     After building the nfs-utils package, there will be a mount.nfs binary in
     the utils/mount directory. This binary can be used to initiate NFS v2, v3,
-    or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4.
-    The standard technique is to create a symlink called mount.nfs4 to mount.nfs.
+    or v4 mounts. To initiate a v4 mount, the binary must be called
+    mount.nfs4.  The standard technique is to create a symlink called
+    mount.nfs4 to mount.nfs.
 
     This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
 
@@ -214,11 +215,11 @@ NFS/RDMA Setup
     /vol0   192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)
     /vol0   192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
 
-    The IP address(es) is(are) the client's IPoIB address for an InfiniBand HCA or the
-    cleint's iWARP address(es) for an RNIC.
+    The IP address(es) is(are) the client's IPoIB address for an InfiniBand
+    HCA or the cleint's iWARP address(es) for an RNIC.
 
-    NOTE: The "insecure" option must be used because the NFS/RDMA client does not
-    use a reserved port.
+    NOTE: The "insecure" option must be used because the NFS/RDMA client does
+    not use a reserved port.
 
  Each time a machine boots:
 
@@ -234,12 +235,13 @@ NFS/RDMA Setup
 
   - Start the NFS server
 
-    If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
-    load the RDMA transport module:
+    If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
+    kernel config), load the RDMA transport module:
 
     $ modprobe svcrdma
 
-    Regardless of how the server was built (module or built-in), start the server:
+    Regardless of how the server was built (module or built-in), start the
+    server:
 
     $ /etc/init.d/nfs start
 
@@ -253,17 +255,17 @@ NFS/RDMA Setup
 
   - On the client system
 
-    If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
-    load the RDMA client module:
+    If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
+    kernel config), load the RDMA client module:
 
     $ modprobe xprtrdma.ko
 
-    Regardless of how the client was built (module or built-in), use this command to
-    mount the NFS/RDMA server:
+    Regardless of how the client was built (module or built-in), use this
+    command to mount the NFS/RDMA server:
 
     $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt
 
-    To verify that the mount is using RDMA, run "cat /proc/mounts" and check the
-    "proto" field for the given mount.
+    To verify that the mount is using RDMA, run "cat /proc/mounts" and check
+    the "proto" field for the given mount.
 
   Congratulations! You're using NFS/RDMA!

Re: [PATCH] update NFS/RDMA documentation

[James Lentini]

On Mon, 2 Jun 2008, J. Bruce Fields wrote:

> On Mon, Jun 02, 2008 at 03:33:59PM -0400, James Lentini wrote:
> > 
> > Bruce,
> > 
> > Below is an update to the NFS/RDMA documentation that clarifies how to 
> > run mount.nfs and addresses comments from you and Chuck. Could you 
> > please merge this for 2.6.26?
> 
> Yep, thanks.
> 
> Mind if we keep this wrapped at under 80 characters?

Sounds good.

> 
> --b.
> 
> commit 80d7681a66c1d800fd6a8e03346219f5e8de185d
> Author: J. Bruce Fields <bfields@citi.umich.edu>
> Date:   Mon Jun 2 16:01:51 2008 -0400
> 
>     nfs: rewrap NFS/RDMA documentation to 80 lines
>     
>     Wrap long lines.
>     
>     Signed-off-by: J. Bruce Fields <bfields@citi.umich.edu>
> 
> diff --git a/Documentation/filesystems/nfs-rdma.txt b/Documentation/filesystems/nfs-rdma.txt
> index 9ad453d..44bd766 100644
> --- a/Documentation/filesystems/nfs-rdma.txt
> +++ b/Documentation/filesystems/nfs-rdma.txt
> @@ -63,10 +63,10 @@ Installation
>    - Install nfs-utils-1.1.2 or greater on the client
>  
>      An NFS/RDMA mount point can be obtained by using the mount.nfs command in
> -    nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils version
> -    with support for NFS/RDMA mounts, but for various reasons we recommend using
> -    nfs-utils-1.1.2 or greater). To see which version of mount.nfs you are
> -    using, type:
> +    nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs-utils
> +    version with support for NFS/RDMA mounts, but for various reasons we
> +    recommend using nfs-utils-1.1.2 or greater). To see which version of
> +    mount.nfs you are using, type:
>  
>      $ /sbin/mount.nfs -V
>  
> @@ -91,8 +91,9 @@ Installation
>  
>      After building the nfs-utils package, there will be a mount.nfs binary in
>      the utils/mount directory. This binary can be used to initiate NFS v2, v3,
> -    or v4 mounts. To initiate a v4 mount, the binary must be called mount.nfs4.
> -    The standard technique is to create a symlink called mount.nfs4 to mount.nfs.
> +    or v4 mounts. To initiate a v4 mount, the binary must be called
> +    mount.nfs4.  The standard technique is to create a symlink called
> +    mount.nfs4 to mount.nfs.
>  
>      This mount.nfs binary should be installed at /sbin/mount.nfs as follows:
>  
> @@ -214,11 +215,11 @@ NFS/RDMA Setup
>      /vol0   192.168.0.47(fsid=0,rw,async,insecure,no_root_squash)
>      /vol0   192.168.0.0/255.255.255.0(fsid=0,rw,async,insecure,no_root_squash)
>  
> -    The IP address(es) is(are) the client's IPoIB address for an InfiniBand HCA or the
> -    cleint's iWARP address(es) for an RNIC.
> +    The IP address(es) is(are) the client's IPoIB address for an InfiniBand
> +    HCA or the cleint's iWARP address(es) for an RNIC.
>  
> -    NOTE: The "insecure" option must be used because the NFS/RDMA client does not
> -    use a reserved port.
> +    NOTE: The "insecure" option must be used because the NFS/RDMA client does
> +    not use a reserved port.
>  
>   Each time a machine boots:
>  
> @@ -234,12 +235,13 @@ NFS/RDMA Setup
>  
>    - Start the NFS server
>  
> -    If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
> -    load the RDMA transport module:
> +    If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
> +    kernel config), load the RDMA transport module:
>  
>      $ modprobe svcrdma
>  
> -    Regardless of how the server was built (module or built-in), start the server:
> +    Regardless of how the server was built (module or built-in), start the
> +    server:
>  
>      $ /etc/init.d/nfs start
>  
> @@ -253,17 +255,17 @@ NFS/RDMA Setup
>  
>    - On the client system
>  
> -    If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
> -    load the RDMA client module:
> +    If the NFS/RDMA client was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m in
> +    kernel config), load the RDMA client module:
>  
>      $ modprobe xprtrdma.ko
>  
> -    Regardless of how the client was built (module or built-in), use this command to
> -    mount the NFS/RDMA server:
> +    Regardless of how the client was built (module or built-in), use this
> +    command to mount the NFS/RDMA server:
>  
>      $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/<export> /mnt
>  
> -    To verify that the mount is using RDMA, run "cat /proc/mounts" and check the
> -    "proto" field for the given mount.
> +    To verify that the mount is using RDMA, run "cat /proc/mounts" and check
> +    the "proto" field for the given mount.
>  
>    Congratulations! You're using NFS/RDMA!
> 

Re: [PATCH] update NFS/RDMA documentation

[Chuck Lever]

On Jun 2, 2008, at 3:33 PM, James Lentini wrote:
>
> Bruce,
>
> Below is an update to the NFS/RDMA documentation that clarifies how to
> run mount.nfs and addresses comments from you and Chuck. Could you
> please merge this for 2.6.26?
>
> Signed-off-by: James Lentini <jlentini@netapp.com>
>
> --- a/Documentation/filesystems/nfs-rdma.txt	2008-05-30  
> 16:03:01.414533000 -0400
> +++ a/Documentation/filesystems/nfs-rdma.txt	2008-06-02  
> 15:26:51.138344000 -0400
> @@ -5,7 +5,7 @@
> ################################################################################
>
>  Author: NetApp and Open Grid Computing
> - Date: April 15, 2008
> + Date: May 29, 2008
>
> Table of Contents
> ~~~~~~~~~~~~~~~~~
> @@ -60,16 +60,18 @@ Installation
>     The procedures described in this document have been tested with
>     distributions from Red Hat's Fedora Project (http://fedora.redhat.com/ 
> ).
>
> -  - Install nfs-utils-1.1.1 or greater on the client
> +  - Install nfs-utils-1.1.2 or greater on the client
>
> -    An NFS/RDMA mount point can only be obtained by using the  
> mount.nfs
> -    command in nfs-utils-1.1.1 or greater. To see which version of  
> mount.nfs
> -    you are using, type:
> +    An NFS/RDMA mount point can be obtained by using the mount.nfs  
> command in
> +    nfs-utils-1.1.2 or greater (nfs-utils-1.1.1 was the first nfs- 
> utils version
> +    with support for NFS/RDMA mounts, but for various reasons we  
> recommend using
> +    nfs-utils-1.1.2 or greater). To see which version of mount.nfs  
> you are
> +    using, type:
>
> -    > /sbin/mount.nfs -V
> +    $ /sbin/mount.nfs -V
>
> -    If the version is less than 1.1.1 or the command does not exist,
> -    then you will need to install the latest version of nfs-utils.
> +    If the version is less than 1.1.2 or the command does not exist,
> +    you should install the latest version of nfs-utils.
>
>     Download the latest package from:
>
> @@ -77,22 +79,32 @@ Installation
>
>     Uncompress the package and follow the installation instructions.
>
> -    If you will not be using GSS and NFSv4, the installation process
> -    can be simplified by disabling these features when running  
> configure:
> +    If you will not need the idmapper and gssd executables (you do  
> not need
> +    these to create an NFS/RDMA enabled mount command), the  
> installation
> +    process can be simplified by disabling these features when  
> running
> +    configure:

This explanation still seems confusing.

idmapper *is* required for NFSv4 (and I assume you can mount an NFSv4  
server over RDMA), and gssd *is* required to support secure mounts,  
all of which may co-exist with the RDMA capability.

I think you would be much better off providing a recipe for building  
and installing mount.nfs instead of trying to "insert" bits into the  
instructions in the nfs-utils README.

> -    > ./configure --disable-gss --disable-nfsv4
> +    $ ./configure --disable-gss --disable-nfsv4
>
>
> -    For more information on this see the package's README and  
> INSTALL files.
> +    To build nfs-utils you will need the tcp_wrappers package  
> installed. For
> +    more information on this see the package's README and INSTALL  
> files.
>
>     After building the nfs-utils package, there will be a mount.nfs  
> binary in
>     the utils/mount directory. This binary can be used to initiate  
> NFS v2, v3,
>     or v4 mounts. To initiate a v4 mount, the binary must be called  
> mount.nfs4.
>     The standard technique is to create a symlink called mount.nfs4  
> to mount.nfs.
>
> -    NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is  
> only needed
> +    This mount.nfs binary should be installed at /sbin/mount.nfs as  
> follows:
> +
> +    $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
> +
> +    In this location, mount.nfs will be invoked automatically for  
> NFS mounts
> +    by the system mount commmand.
> +
> +    NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is  
> only needed
>     on the NFS client machine. You do not need this specific version  
> of
>     nfs-utils on the server. Furthermore, only the mount.nfs command  
> from
> -    nfs-utils-1.1.1 is needed on the client.
> +    nfs-utils-1.1.2 is needed on the client.
>
>   - Install a Linux kernel with NFS/RDMA
>
> @@ -156,8 +168,8 @@ Check RDMA and NFS Setup
>     this time. For example, if you are using a Mellanox Tavor/Sinai/ 
> Arbel
>     card:
>
> -    > modprobe ib_mthca
> -    > modprobe ib_ipoib
> +    $ modprobe ib_mthca
> +    $ modprobe ib_ipoib
>
>     If you are using InfiniBand, make sure there is a Subnet Manager  
> (SM)
>     running on the network. If your IB switch has an embedded SM,  
> you can
> @@ -166,7 +178,7 @@ Check RDMA and NFS Setup
>
>     If an SM is running on your network, you should see the following:
>
> -    > cat /sys/class/infiniband/driverX/ports/1/state
> +    $ cat /sys/class/infiniband/driverX/ports/1/state
>     4: ACTIVE
>
>     where driverX is mthca0, ipath5, ehca3, etc.
> @@ -174,10 +186,10 @@ Check RDMA and NFS Setup
>     To further test the InfiniBand software stack, use IPoIB (this
>     assumes you have two IB hosts named host1 and host2):
>
> -    host1> ifconfig ib0 a.b.c.x
> -    host2> ifconfig ib0 a.b.c.y
> -    host1> ping a.b.c.y
> -    host2> ping a.b.c.x
> +    host1$ ifconfig ib0 a.b.c.x
> +    host2$ ifconfig ib0 a.b.c.y
> +    host1$ ping a.b.c.y
> +    host2$ ping a.b.c.x
>
>     For other device types, follow the appropriate procedures.
>
> @@ -214,9 +226,9 @@ NFS/RDMA Setup
>
>     For InfiniBand using a Mellanox adapter:
>
> -    > modprobe ib_mthca
> -    > modprobe ib_ipoib
> -    > ifconfig ib0 a.b.c.d
> +    $ modprobe ib_mthca
> +    $ modprobe ib_ipoib
> +    $ ifconfig ib0 a.b.c.d
>
>     NOTE: use unique addresses for the client and server
>
> @@ -225,30 +237,31 @@ NFS/RDMA Setup
>     If the NFS/RDMA server was built as a module  
> (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
>     load the RDMA transport module:
>
> -    > modprobe svcrdma
> +    $ modprobe svcrdma
>
>     Regardless of how the server was built (module or built-in),  
> start the server:
>
> -    > /etc/init.d/nfs start
> +    $ /etc/init.d/nfs start
>
>     or
>
> -    > service nfs start
> +    $ service nfs start

I would expect that these commands, as well as the modprobe commands  
you mentioned above, require root privileges.  Since you specified  
"sudo" in your nfs-utils build instructions, you might want to use  
"sudo" here, or explicitly state that the administrator must be root  
to execute these commands successfully.

>      Instruct the server to listen on the RDMA transport:
>
> -    > echo rdma 2050 > /proc/fs/nfsd/portlist
> +    $ echo rdma 2050 > /proc/fs/nfsd/portlist
>
>   - On the client system
>
>     If the NFS/RDMA client was built as a module  
> (CONFIG_SUNRPC_XPRT_RDMA=m in kernel config),
>     load the RDMA client module:
>
> -    > modprobe xprtrdma.ko
> +    $ modprobe xprtrdma.ko
>
> -    Regardless of how the client was built (module or built-in),  
> issue the mount.nfs command:
> +    Regardless of how the client was built (module or built-in),  
> use this command to
> +    mount the NFS/RDMA server:
>
> -    > /path/to/your/mount.nfs <IPoIB-server-name-or-address>:/ 
> <export> /mnt -i -o rdma,port=2050
> +    $ mount -o rdma,port=2050 <IPoIB-server-name-or-address>:/ 
> <export> /mnt
>
>     To verify that the mount is using RDMA, run "cat /proc/mounts"  
> and check the
>     "proto" field for the given mount.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: [PATCH] update NFS/RDMA documentation

[James Lentini]

On Mon, 2 Jun 2008, Chuck Lever wrote:

> > @@ -77,22 +79,32 @@ Installation
> > 
> >    Uncompress the package and follow the installation instructions.
> > 
> > -    If you will not be using GSS and NFSv4, the installation process
> > -    can be simplified by disabling these features when running configure:
> > +    If you will not need the idmapper and gssd executables (you do not need
> > +    these to create an NFS/RDMA enabled mount command), the installation
> > +    process can be simplified by disabling these features when running
> > +    configure:
> 
> This explanation still seems confusing.
> 
> idmapper *is* required for NFSv4 (and I assume you can mount an 
> NFSv4 server over RDMA), 

Yes, NFSv4 works over RDMA.

> and gssd *is* required to support secure mounts, all of which may 
> co-exist with the RDMA capability.

I agree. The intent of the text is to inform the reader that the 
idmapper and gssd are not need if all they need is a 
NFS/RDMA-compatible (aka string mount API enabled) mount.nfs command.

> I think you would be much better off providing a recipe for building 
> and installing mount.nfs instead of trying to "insert" bits into the 
> instructions in the nfs-utils README.

I'm open to suggestions on how to improve the text. 

Can send an example of such a recipe?

> > -    > ./configure --disable-gss --disable-nfsv4
> > +    $ ./configure --disable-gss --disable-nfsv4
> > 
> > 
> > -    For more information on this see the package's README and INSTALL
> > files.
> > +    To build nfs-utils you will need the tcp_wrappers package installed.
> > For
> > +    more information on this see the package's README and INSTALL files.
> > 
> >    After building the nfs-utils package, there will be a mount.nfs binary in
> >    the utils/mount directory. This binary can be used to initiate NFS v2,
> > v3,
> >    or v4 mounts. To initiate a v4 mount, the binary must be called
> > mount.nfs4.
> >    The standard technique is to create a symlink called mount.nfs4 to
> > mount.nfs.
> > 
> > -    NOTE: mount.nfs and therefore nfs-utils-1.1.1 or greater is only needed
> > +    This mount.nfs binary should be installed at /sbin/mount.nfs as
> > follows:
> > +
> > +    $ sudo cp utils/mount/mount.nfs /sbin/mount.nfs
> > +
> > +    In this location, mount.nfs will be invoked automatically for NFS
> > mounts
> > +    by the system mount commmand.
> > +
> > +    NOTE: mount.nfs and therefore nfs-utils-1.1.2 or greater is only needed
> >    on the NFS client machine. You do not need this specific version of
> >    nfs-utils on the server. Furthermore, only the mount.nfs command from
> > -    nfs-utils-1.1.1 is needed on the client.
> > +    nfs-utils-1.1.2 is needed on the client.
> > 
> >  - Install a Linux kernel with NFS/RDMA
> > 
> > @@ -156,8 +168,8 @@ Check RDMA and NFS Setup
> >    this time. For example, if you are using a Mellanox Tavor/Sinai/Arbel
> >    card:
> > 
> > -    > modprobe ib_mthca
> > -    > modprobe ib_ipoib
> > +    $ modprobe ib_mthca
> > +    $ modprobe ib_ipoib
> > 
> >    If you are using InfiniBand, make sure there is a Subnet Manager (SM)
> >    running on the network. If your IB switch has an embedded SM, you can
> > @@ -166,7 +178,7 @@ Check RDMA and NFS Setup
> > 
> >    If an SM is running on your network, you should see the following:
> > 
> > -    > cat /sys/class/infiniband/driverX/ports/1/state
> > +    $ cat /sys/class/infiniband/driverX/ports/1/state
> >    4: ACTIVE
> > 
> >    where driverX is mthca0, ipath5, ehca3, etc.
> > @@ -174,10 +186,10 @@ Check RDMA and NFS Setup
> >    To further test the InfiniBand software stack, use IPoIB (this
> >    assumes you have two IB hosts named host1 and host2):
> > 
> > -    host1> ifconfig ib0 a.b.c.x
> > -    host2> ifconfig ib0 a.b.c.y
> > -    host1> ping a.b.c.y
> > -    host2> ping a.b.c.x
> > +    host1$ ifconfig ib0 a.b.c.x
> > +    host2$ ifconfig ib0 a.b.c.y
> > +    host1$ ping a.b.c.y
> > +    host2$ ping a.b.c.x
> > 
> >    For other device types, follow the appropriate procedures.
> > 
> > @@ -214,9 +226,9 @@ NFS/RDMA Setup
> > 
> >    For InfiniBand using a Mellanox adapter:
> > 
> > -    > modprobe ib_mthca
> > -    > modprobe ib_ipoib
> > -    > ifconfig ib0 a.b.c.d
> > +    $ modprobe ib_mthca
> > +    $ modprobe ib_ipoib
> > +    $ ifconfig ib0 a.b.c.d
> > 
> >    NOTE: use unique addresses for the client and server
> > 
> > @@ -225,30 +237,31 @@ NFS/RDMA Setup
> >    If the NFS/RDMA server was built as a module (CONFIG_SUNRPC_XPRT_RDMA=m
> > in kernel config),
> >    load the RDMA transport module:
> > 
> > -    > modprobe svcrdma
> > +    $ modprobe svcrdma
> > 
> >    Regardless of how the server was built (module or built-in), start the
> > server:
> > 
> > -    > /etc/init.d/nfs start
> > +    $ /etc/init.d/nfs start
> > 
> >    or
> > 
> > -    > service nfs start
> > +    $ service nfs start
> 
> I would expect that these commands, as well as the modprobe commands 
> you mentioned above, require root privileges.  Since you specified 
> "sudo" in your nfs-utils build instructions, you might want to use 
> "sudo" here, or explicitly state that the administrator must be root 
> to execute these commands successfully.

Ok.

Re: [PATCH] update NFS/RDMA documentation

[Chuck Lever]

On Mon, Jun 2, 2008 at 5:13 PM, James Lentini <jlentini@netapp.com> wrote:
> On Mon, 2 Jun 2008, Chuck Lever wrote:
>> > @@ -77,22 +79,32 @@ Installation
>> >
>> >    Uncompress the package and follow the installation instructions.
>> >
>> > -    If you will not be using GSS and NFSv4, the installation process
>> > -    can be simplified by disabling these features when running configure:
>> > +    If you will not need the idmapper and gssd executables (you do not need
>> > +    these to create an NFS/RDMA enabled mount command), the installation
>> > +    process can be simplified by disabling these features when running
>> > +    configure:
>>
>> This explanation still seems confusing.
>>
>> idmapper *is* required for NFSv4 (and I assume you can mount an
>> NFSv4 server over RDMA),
>
> Yes, NFSv4 works over RDMA.
>
>> and gssd *is* required to support secure mounts, all of which may
>> co-exist with the RDMA capability.
>
> I agree. The intent of the text is to inform the reader that the
> idmapper and gssd are not need if all they need is a
> NFS/RDMA-compatible (aka string mount API enabled) mount.nfs command.
>
>> I think you would be much better off providing a recipe for building
>> and installing mount.nfs instead of trying to "insert" bits into the
>> instructions in the nfs-utils README.
>
> I'm open to suggestions on how to improve the text.
>
> Can send an example of such a recipe?

Here's a stab at it.  It's a little wordy, but it spells out the necessary
steps, which someone can even copy-n-paste, recipe style, to build
and install the mount.nfs command, in most cases.  Part of the
problem is the nfs-utils README is a little sparse in some of these
areas.

Naturally you should test these instructions before publishing them :-)

  - Install nfs-utils-1.1.2 or later on the client

    To mount an NFS server over RDMA, you must have the
    mount.nfs subcommand from nfs-utils-1.1.2 or later installed on
    your client.  For the purposes of this discussion, we will assume
    that your distribution keeps the mount subcommands under /sbin.

    To see which version of mount.nfs is installed on your client,
    type:

    $ /sbin/mount.nfs -V
    mount.nfs (linux nfs-utils 1.1.2)
    $

    If the reported version is less than 1.1.2, or the command does
    not exist, then you will need to install the latest version of the
    NFS mount.nfs subcommand.

    Your distributor may provide an update that brings the nfs-utils
    package up to 1.1.2 or later, but if your client's packages are
    completely up-to-date and you still don't have nfs-utils 1.1.2,
    you will need to build nfs-utils from the latest upstream version.

    Download the latest nfs-utils package from:

        http://www.kernel.org/pub/linux/utils/nfs

    Uncompress the package into a convenient place.

    Building nfs-utils also requires that the tcp_wrapper-devel
    package is installed on your build system.  For example, on a
    typical Fedora system, you can use yum to install it:

    $ sudo yum install tcp_wrappers-devel

    On other systems, use an appropriate native package
    management facility to retrieve and install tcp_wrappers-devel.

    Now cd to the top level directory of the uncompressed nfs-utils
    tarball.

    $ cd /tmp/nfs-utils-1.1.2

    Set up the autotools scripts.  Since we only need to build the
    mount.nfs command, let's disable other components of nfs-utils
    that might depend on packages that are not typically available
    on most systems.

    $ ./configure --disable-gss --disable-nfsv4
    $ make

    If all has gone well, there should be a mount.nfs executable
    under the utils/mount directory when the build is finished.

    You can invoke the mount.nfs subcommand directly if desired,
    but the usual practice is to place the subcommand where your
    system's mount command can find it, and then it will be invoked
    automatically during NFS mount requests.  This way you can
    add NFS/RDMA mount points to your client's /etc/fstab or
    automounter maps.

    We haven't made any special changes here, so this version of
    the mount.nfs subcommand can safely replace your system's
    original mount.nfs subcommand.  When you are satisfied it is
    working as expected, install it.

    $ cd utils/mount
    $ sudo make install-exec

    Some features require that the mount.nfs subcommand is
    setuid root, but this is entirely optional.

    $ sudo chown root /sbin/mount.nfs
    $ sudo chmod u+s /sbin/mount.nfs

    You do not need to replace the mount.nfs command on your NFS
    server to support NFS over RDMA.

    More information on the mount.nfs command is available in the
    nfs-utils package's README and INSTALL files.

-- 
Chuck Lever
chuck[dot]lever[at]oracle[dot]com