Re: [Drbd-dev] [PATCH 1/3] User's Guide: add section on DRBD over InfiniBand

[Florian Haas <florian@hastexo.com>]

Hi Bart,

On 11/19/11 14:21, Bart Van Assche wrote:
> On Wed, Nov 16, 2011 at 6:38 PM, Florian Haas <florian@hastexo.com> wrote:
>> +To enable IPoIB, you need to enable the +ib_ipoib+ driver, typically
>> +available on any system stock kernel as a loadable module. Make sure
>> +that you configure the IPoIB driver to run in the Reliable Connected
>> +(RC) transport mode.
> 
> Maybe it's a good idea to mention here the impact of the IPoIB MTU on
> IPoIB bandwidth ?
> 
> On a typical setup these options can be configured as follows in
> /etc/infiniband/openib.conf:
> SET_IPOIB_CM=yes
> IPOIB_MTU=65520
> 
> By the way, that last setting is the default with recent versions of
> /etc/init.d/openibd.

Thanks for the suggestion; I've worked that in. I try not to go into too
much detail regarding configuration paths and init scripts as those may
very wildly by distro and release, but I've mentioned the relevant sysfs
files now. Diff is below.

Cheers,
Florian


diff --git a/users-guide/hp-transports.txt b/users-guide/hp-transports.txt
index 35fa5bd..dbd0186 100644
--- a/users-guide/hp-transports.txt
+++ b/users-guide/hp-transports.txt
@@ -23,14 +23,25 @@ resource configuration. DRBD then replicates using
TCP over IP --
 albeit not based on Ethernet as it normally would, but on InfiniBand.

 To enable IPoIB, you need to enable the +ib_ipoib+ driver, typically
-available on any system stock kernel as a loadable module. Make sure
-that you configure the IPoIB driver to run in the Reliable Connected
-(RC) transport mode.
+available on any system stock kernel as a loadable module.
+
+Make sure that you configure the IPoIB driver to run in the Reliable
+Connected (RC) transport mode. The transport mode is accessible via
+the +/sys/class/net/<interface>/mode+ virtual file. An example for
+setting the transport mode for the +ib0+ interface would be the
+following command:
+
+  echo connected > /sys/class/net/ib0/mode
+
+Distributions may, however, offer more convenient methods for setting
+the transport mode permanently.

 NOTE: The configuration of an IP address on an IPoIB is beyond the
 scope of this guide, and may vary between distributions. However, it
 is usually equivalent to the configuration of an Ethernet interface --
 simply replace +ethX+ with +ibX+ in the interface configuration.
+This also applies for the interface MTU, which you may set to as high
+as 65,520 bytes for IPoIB in connected mode.

 In the example below, +192.168.144.111+ and +192.168.144.112+ would be
 bound to an IPoIB interface -- for example, +ib0+.