[Drbd-dev] [PATCH 0/3] Document IB and Dolphin replication in User's Guide

[Drbd-dev] [PATCH 0/3] Document IB and Dolphin replication in User's Guide

[Florian Haas]

Phil/Lars,

please review the following additions to the User's Guide for
accuracy. They cover DRBD replication over IB (both IPoIB and SDP) and
Dolphin Express (SuperSockets). I'll be happy to make any required
modifications.

Thanks,
Florian

[Drbd-dev] [PATCH 1/3] User's Guide: add section on DRBD over InfiniBand

[Florian Haas]

---
 users-guide/hp-transports.txt |   89 +++++++++++++++++++++++++++++++++++++++++
 1 files changed, 89 insertions(+), 0 deletions(-)
 create mode 100644 users-guide/hp-transports.txt

diff --git a/users-guide/hp-transports.txt b/users-guide/hp-transports.txt
new file mode 100644
index 0000000..428e6ae
--- /dev/null
+++ b/users-guide/hp-transports.txt
@@ -0,0 +1,89 @@
+== High-performance replication transports
+
+This chapter outlines the configuration steps necessary to run DRBD
+over replication networks optimized for performance. DRBD supports two
+dedicated high-performance transports for replication:
+
+* *InfiniBand*, an industry standard implemented by several vendors,
+* *Dolphin Express*, a vendor specific interconnect available from
+  Dolphin Interconnect Solutions.
+
+=== Replication over InfiniBand networks
+
+DRBD supports replication with InfiniBand using two different
+approaches.
+
+==== IPoIB
+
+In *IP-over-InfiniBand* (IPoIB) an InfiniBand adapter is configured to
+host an IPv4 address, just like an Ethernet network card. For IPoIB,
+no specific changes to the DRBD configuration are necessary. You
+simply assign an IPv4 +address+ bound to an InfiniBand adapter in your
+resource configuration. DRBD then replicates using TCP over IP --
+albeit not based on Ethernet as it normally would, but on InfiniBand.
+
+To enable IPoIB, you need to enable the +ib_ipoib+ driver, typically
+available on any system stock kernel as a loadable module. Make sure
+that you configure the IPoIB driver to run in the Reliable Connected
+(RC) transport mode.
+
+NOTE: The configuration of an IP address on an IPoIB is beyond the
+scope of this guide, and may vary between distributions. However, it
+is usually equivalent to the configuration of an Ethernet interface --
+simply replace +ethX+ with +ibX+ in the interface configuration.
+
+In the example below, +192.168.144.111+ and +192.168.144.112+ would be
+bound to an IPoIB interface -- for example, +ib0+.
+
+  resource ib {
+  [...]
+    on alice {
+      address 192.168.144.111:7788;
+    }
+    on bob {
+      address 192.168.144.112:7788;
+    }
+  }
+
+==== SDP
+
+Sockets Direct Protocol (SDP) is an in-kernel BSD-style socket
+implementation for RDMA capable transports, such as InfiniBand. DRBD
+can use SDP for replication directly, without the intermediate TCP/IP
+layer that IPoIB requires. SDP uses IPoIB for an _addressing_
+purposes, however this does not imply that the _transport_ itself is
+IP (or TCP) based.
+
+SDP requires that
+
+* the +ib_sdp+ kernel module is present and loaded. Most distributions
+  ship this in a package separate from the standard stock kernel
+  image. A number of distributions ship SDP in a package named +ofed+
+  or similar.
+* an IPoIB address is configured on the InfiniBand adapter to be used
+  for SDP replication.
+
+NOTE: OFED is the Open Fabrics Enterprise Distribution, a stack of
+kernel drivers and userspace libraries for use with RDMA capable
+fabrics. It is maintained by the Open Fabrics Alliance (OFA).
+
+IMPORTANT: *Do not* use an OFED distribution prior to release 1.5 for
+DRBD-over-SDP replication. The SDP implementation in earlier OFED
+releases contains critical issues that make it unsuitable for DRBD
+replication.
+
+In the example below, +192.168.144.111+ and +192.168.144.112+ would be
+bound to an IPoIB interface -- for example, +ib0+. The +sdp+ keyword,
+prefixed to the replication address, configures DRBD to replicate over
+SDP.
+
+  resource ib {
+  [...]
+    on alice {
+      address sdp 192.168.144.111:7788;
+    }
+    on bob {
+      address sdp 192.168.144.112:7788;
+    }
+  }
+
-- 
1.7.5.4

Re: [Drbd-dev] [PATCH 1/3] User's Guide: add section on DRBD over InfiniBand

[Florian Haas]

Hi Bart,

On 11/19/11 14:21, Bart Van Assche wrote:
> On Wed, Nov 16, 2011 at 6:38 PM, Florian Haas <florian@hastexo.com> wrote:
>> +To enable IPoIB, you need to enable the +ib_ipoib+ driver, typically
>> +available on any system stock kernel as a loadable module. Make sure
>> +that you configure the IPoIB driver to run in the Reliable Connected
>> +(RC) transport mode.
> 
> Maybe it's a good idea to mention here the impact of the IPoIB MTU on
> IPoIB bandwidth ?
> 
> On a typical setup these options can be configured as follows in
> /etc/infiniband/openib.conf:
> SET_IPOIB_CM=yes
> IPOIB_MTU=65520
> 
> By the way, that last setting is the default with recent versions of
> /etc/init.d/openibd.

Thanks for the suggestion; I've worked that in. I try not to go into too
much detail regarding configuration paths and init scripts as those may
very wildly by distro and release, but I've mentioned the relevant sysfs
files now. Diff is below.

Cheers,
Florian


diff --git a/users-guide/hp-transports.txt b/users-guide/hp-transports.txt
index 35fa5bd..dbd0186 100644
--- a/users-guide/hp-transports.txt
+++ b/users-guide/hp-transports.txt
@@ -23,14 +23,25 @@ resource configuration. DRBD then replicates using
TCP over IP --
 albeit not based on Ethernet as it normally would, but on InfiniBand.

 To enable IPoIB, you need to enable the +ib_ipoib+ driver, typically
-available on any system stock kernel as a loadable module. Make sure
-that you configure the IPoIB driver to run in the Reliable Connected
-(RC) transport mode.
+available on any system stock kernel as a loadable module.
+
+Make sure that you configure the IPoIB driver to run in the Reliable
+Connected (RC) transport mode. The transport mode is accessible via
+the +/sys/class/net/<interface>/mode+ virtual file. An example for
+setting the transport mode for the +ib0+ interface would be the
+following command:
+
+  echo connected > /sys/class/net/ib0/mode
+
+Distributions may, however, offer more convenient methods for setting
+the transport mode permanently.

 NOTE: The configuration of an IP address on an IPoIB is beyond the
 scope of this guide, and may vary between distributions. However, it
 is usually equivalent to the configuration of an Ethernet interface --
 simply replace +ethX+ with +ibX+ in the interface configuration.
+This also applies for the interface MTU, which you may set to as high
+as 65,520 bytes for IPoIB in connected mode.

 In the example below, +192.168.144.111+ and +192.168.144.112+ would be
 bound to an IPoIB interface -- for example, +ib0+.

[Drbd-dev] [PATCH 2/3] User's Guide: add section about Dolphin SuperSockets replication

[Florian Haas]

---
 users-guide/hp-transports.txt |   40 ++++++++++++++++++++++++++++++++++++++++
 1 files changed, 40 insertions(+), 0 deletions(-)

diff --git a/users-guide/hp-transports.txt b/users-guide/hp-transports.txt
index 428e6ae..35fa5bd 100644
--- a/users-guide/hp-transports.txt
+++ b/users-guide/hp-transports.txt
@@ -87,3 +87,43 @@ SDP.
     }
   }
 
+=== Replication over Dolphin Express interconnects
+
+DRBD replicates over Dolphin Express interconnects using the
+SuperSockets protocol.
+
+In order to replicate over SuperSockets, DRBD requires that
+
+- you have a working Dolphin Express hardware interconnect between
+  nodes;
+- you have installed the SuperSockets driver and libraries from the
+  Self-Installing Archive (SIA) available from Dolphin;
+- you have configured your nodes as a Dolphin cluster with valid
+  +/etc/dis/dishosts+ and +/etc/dis/networkmanager.conf+ files;
+- the +dis_nodemgr+ service is running.
+
+NOTE: A convenient way to configure the Dolphin cluster is the
+graphical +dis_netconfig+ utility. For further information regarding
+the installation and initial configuration of Dolphin Express
+interconnects on Linux, please refer to the
+http://ww.dolphinics.no/download/IX_4_0_X_LINUX_DOC/index.html[Dolphin
+Express IX Installation and Reference Guide].
+
+Once SuperSockets are properly installed and configured, set the
++ssocks+ keyword on your resource configuration:
+
+  resource ssocks {
+  [...]
+    on alice {
+      address ssocks 192.168.144.111:7788;
+    }
+    on bob {
+      address ssocks 192.168.144.112:7788;
+    }
+  }
+
+IMPORTANT: SuperSockets provide transparent, seamless fallback to
+Ethernet in case the Dolphin Express interconnect is
+interrupted. Thus, the IPv4 address specified after the +address
+ssocks+ keywords _must_ be an address available and configured on an
+Ethernet interface.
-- 
1.7.5.4

[Drbd-dev] [PATCH 3/3] User's Guide: add chapter on high-performance replication

[Florian Haas]

Also, fix a whitespace issue in the preceding chapter.
---
 users-guide/drbd-users-guide.txt |    1 +
 users-guide/latency.txt          |    1 +
 2 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/users-guide/drbd-users-guide.txt b/users-guide/drbd-users-guide.txt
index f63d0e5..ddeaf13 100644
--- a/users-guide/drbd-users-guide.txt
+++ b/users-guide/drbd-users-guide.txt
@@ -37,6 +37,7 @@ include::xen.txt[]
 include::benchmark.txt[]
 include::throughput.txt[]
 include::latency.txt[]
+include::hp-transports.txt[]
 
 [[p-learn]]
 = Learning more about DRBD
diff --git a/users-guide/latency.txt b/users-guide/latency.txt
index 3f6fa11..deab8dc 100644
--- a/users-guide/latency.txt
+++ b/users-guide/latency.txt
@@ -176,3 +176,4 @@ You may also make a global I/O scheduler selection by passing the
 boot loader configuration (normally found in +/boot/grub/menu.lst+ if
 you are using the GRUB bootloader) and add +elevator=deadline+ to your
 list of kernel boot options.
+
-- 
1.7.5.4