[LARTC] How to check an inactive slave in a bond?

[LARTC] How to check an inactive slave in a bond?

[olivier arsac]

[-- Attachment #1.1: Type: text/plain, Size: 1471 bytes --]

I'm using bonding in active-fallback mode to guarantee maximum 
availability on some critical servers.
The mii mode is active so I can detect things like dead card and/or 
unplugged cable even on the inactive slave.
But how do I check that the inactive slave is properly 
configured/connected to the switch/vlan?
I ask this question because it has just bitten me in a part I'll keep 
undisclosed.

Scenario:
your bond0 is running fine. it uses eth0 as active slave and eth2 as 
inactive slave (different cards/ different driver to be safe)
some bozo reconfigures the switch port where your eth2 is plugged in and 
you don't notice it (the crucial point here)
later on, your eth0 dies (or is unplugged by the brother of the first bozo)
and bamm... your nice HA node is off-line.
note: eth2 is still plugged and fine at the mii level.
even an arp would return OK so going from mii to arp for the bond is not 
the right option.

So. How could I check that an IP packet send via eth2 would really reach 
its vlan?
I tried a probably naive thing:
ifenslave -d bond0 eth2
ifconfig eth2 $ip netmask 255.255.255.255
route add -host $target eth2
ping $target
(target is the gateway, ip is a reserved IP used only on this server to 
do that check)
but it does not work as I hopped. Sometimes the ping is OK (but goes 
thru bond0) sometimes it blocks...
The real question is How to do it properly (rather than how to fix my 
naive try).

Thank you for your help.

    Olivier







[-- Attachment #1.2: Type: text/html, Size: 1869 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] How to check an inactive slave in a bond?

[Jay Vosburgh]

olivier arsac <oliv@arsac.org> wrote:

[...]
>Scenario:
>your bond0 is running fine. it uses eth0 as active slave and eth2 as inactive
>slave (different cards/ different driver to be safe)
>some bozo reconfigures the switch port where your eth2 is plugged in and you
>don't notice it (the crucial point here)
>later on, your eth0 dies (or is unplugged by the brother of the first bozo)
>and bamm... your nice HA node is off-line.
>note: eth2 is still plugged and fine at the mii level.
>even an arp would return OK so going from mii to arp for the bond is not the
>right option.
>
>So. How could I check that an IP packet send via eth2 would really reach its
>vlan?

	One obvious remedy is better bozo control, but for purposes of
discussion, let's look at this as simply a question about assurance of
the inactive path.

	The short answer to you question is that you can't do what
you're trying to do.  The bonding driver itself, as you've noted, is
reasonably good at detecting link state, and connectivity to local
network peers (via the ARP monitor), but doesn't provide full end-to-end
path validation for all active and inactive slaves.

	The long answer is that end-to-end validation of the inactive
path is fairly complicated, and can be tricky to do correctly.  If an
inactive slave transmits something, it may cause updating of forwarding
tables on the network (either because ARP probes have this effect, or
because many switches snoop traffic to determine which destination is
reachable via which port), which is undesirable.  Inactive slaves, at
least in recent versions of bonding, also drop all incoming traffic to
prevent the same packet from being delivered multiple times (if, e.g., a
switch is flooding traffic to all ports of the bond for whatever
reason).

	The ideal case is to issue, e.g., a ping (ICMP Echo Request) to
some IP address on the desired destination.  An IP-level probe is better
in the grand scheme of things because an IP packet is routable and can
reach off the local network (which an ARP cannot).  If we move up to
IPv6, this becomes more complicated, as the "inactive" slave would have
to participated in the IPv6 stateless address autoconfiguration
independently from the master, which also causes headaches for IPv6
snooping switches.

	So, to achieve an actual end to end test from the "inactive"
slave to the peer of choice, it's necessary to isolate this traffic so
that it properly returns to the "inactive" slave (and isn't routed back
to the master).  This separate communication needs to take place on a
logically discrete network (which may also be physically discrete, as
appears to be the case in your situation).  If the "probe" network isn't
separate from the "real" network, then intermediate routers may send the
probes or replies over the wrong path, or improperly update forwarding
tables and so on.

	The bonding driver today doesn't support this type of
"independent" slave activity; all slaves are considered to be minions of
the master, and aren't allowed to operate independently.

	I will point out that the ARP monitor, for a subset of cases,
can come close to what you want.  The current versions of the bonding
driver support an "arp_validate" option, and can validate that the ARP
probes (which are broadcasts) sent out over the active slave actually
reach the inactive slave.  It doesn't validate the ARP replies, as those
are only received by the active slave, and it doesn't attempt to
transmit on the inactive slave.

>I tried a probably naive thing:
>ifenslave -d bond0 eth2
>ifconfig eth2 $ip netmask 255.255.255.255
>route add -host $target eth2
>ping $target
>(target is the gateway, ip is a reserved IP used only on this server to do that
>check)
>but it does not work as I hopped. Sometimes the ping is OK (but goes thru
>bond0) sometimes it blocks...
>The real question is How to do it properly (rather than how to fix my naive
>try).

	I would hazard to guess that your problem here is likely one of
routing.  Maybe on the sender end, maybe on the reply end, maybe both.
On the sender end, you can force ping to use a particular interface via
the "-I" option, which will assure you that you're using the eth2 for
the transmission.  The question is going to be which path the reply
packet takes to get back to the sender.

	The other problem, of course, is that you've removed your backup
link from the bond, so if the primary should fail while you've got it
running this ping test, you'll lose all connectivity.

	One perhaps cheap and easy, but not 100% reliable, method would
be for you to periodically manually fail over the bond to whichever link
is inactive (via ifenslave -c bond0 ethX, cycling X through your set of
slave interfaces).

	The up side of this is that you'll exercise both paths
regularly, so any bozo induced nonsense should become visible sooner
rather than later; the precise interval for "sooner" depending upon how
often the failover is induced.

	The down side is that a failover will probably lose at least a
few packets, and you'll have to arrange your script or whatever to stop
if you experience an actual failure.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] How to check an inactive slave in a bond?

[olivier arsac]

Thank you for this very complete answer.

- Assuming I really have to implement some sort of inactive slave-link 
check.
- Assuming it is acceptable to remove the inactive slave from the bound 
for the duration of the check.

Could you help me check my script? It works well for me but as I'm about 
to deploy it for production purpose I'd rather have a double check from 
you guys.
(Note: I'm not reliable when it comes to (among other things) routing 
and network related topics)

Thx.

       Olivier

---------------------------------------------------------------------------------------
#!/bin/bash

# Check all nics enslaved in a bond.
# This is a way to check that all nics (including inactive ones) are 
working properly.
#

# Authors:
#   OA: Olivier Arsac
# History:
#   19/04/2007: OA scratch
#   31/06/2007: OA better handling of "free" IPs used during test
# TODO:
#   remove all TODOs from the script

#set -x

# try to be robust -> exit if a variable is not set (probably something 
went wrong)
set -o nounset

trap clean INT TERM

PATH=/exploit/local/sbin:/exploit/local/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/exploit/unix/prod/bin

com=`basename "$0"`
fullcom="$*"

usage() {
    echo "Usage: $com [-q] [-i ip] [-t target] [bond]"
    echo "  Check all nics enslaved to a bond."
    echo "  This is a way to check that all nics (including inactive 
ones) are working properly. You should check that periodicaly to avoid 
nasty surprises when your active nic stops working and you have to 
fallback to your (unchecked) slave one."
    echo "  exit 0 if all is OK (or if no bond is present)."
    echo "  -q: quiet (no verbose message for human operator)."
    echo "  -i: ip to use during check of inactive slaves."
    echo "  -t: target ip to ping during checks."
    echo "eg: $com"
    echo "  check all nics from all bonds."
    echo "eg: $com -q bond0"
    echo "  check silently all nics from bond0."
}

quiet=0
ip=""
target=""
while getopts "qi:t:" option
do
  case $option in
    q) quiet=1;;
    i) ip=$OPTARG;;
    t) target=$OPTARG;;
    *) usage; exit 1;;
  esac
done

# drop what has been parsed by getopts
shift `expr $OPTIND - 1`

# get args
if [ "$#" -ne 0 ]
then
  bonds="$@"
  for bond in $bonds; do
    if [ ! -f /proc/net/bonding/$bond ]; then
      echoe "Error: $bond is not a valid bond."
      exit 6
    fi
  done
else
  bonds=`ls /proc/net/bonding/ 2>/dev/null`
fi

#match a MAC address
re_mac="([a-zA-Z0-9][a-zA-Z0-9]:){5}[a-zA-Z0-9][a-zA-Z0-9]"
re_ip="(([0-9]{3}[.]){3}[0-9]{3}"

function echoe(){
  echo "$@" >/dev/stderr
}

function echoq(){
  if [ $quiet -eq 0 ] ; then echo "$@"; fi
}

# set a valid mac to a nic
# (must get a mac from the slaves in bond but one that is not currently 
in use by the bond)
get_free_mac_ret=""
function get_free_mac(){
  bond=$1
  nic=$2
  free_mac=""
  macs=`grep "Permanent HW addr:" /proc/net/bonding/$bond | egrep -o 
$re_mac| tr 'a-z' 'A-Z'`
  bond_mac=`ifconfig $bond | grep HWaddr | egrep -o $re_mac`
  for mac in $macs; do
    if [ "$mac" != "$bond_mac" ]; then
      free_mac=$mac
    fi
  done
  get_free_mac_ret=$free_mac
}

# ping a target using a specified nic to test for IP connectivity
check_nic_ret=0
function check_nic(){
   target=$1
   if [ $# -ge 2 ]; then
     nic=$2
     ping -n -c 3 -I $nic $target 1>/dev/null 2>/dev/null
   else
     ping -n -c 3 $target 1>/dev/null 2>/dev/null
   fi
   if [ $? -ne 0 ]; then
     ma=""
     if [ $# -ge 3 ]; then
       ma="(using $3 as ip)"
     fi 
     echoq "      [ERROR]"
     echo "$nic interface on $host is not working properly! $ma" > 
/dev/stderr
     check_nic_ret=1
   else
     echoq "      [OK]"
     check_nic_ret=0
   fi
}

# arping a target using a specified nic to test for IP connectivity
function exercise_nic_arp(){
   target=$1
   nic=$2
   src_ip=$3
   arping -c 3 -s "$src_ip" -I "$nic" "$target" 1>/dev/null 2>/dev/null
}

# reset a properly configured bond if someone interrupts the script
clean_bond=""
clean_nic=""
function clean(){
  echoq
  echoq "Script interrupted, restoring bond."
  if [ ! -z $clean_bond ] && [ ! -z $clean_nic ]; then
    ifenslave $clean_bond $clean_nic 2>/dev/null
  fi
  exit 2
}

host=`hostname -s`
table 0

if [ ! -d /proc/net/bonding ]; then
  echoe "Warning: Module bonding not loaded. Obviously no bond to check."
  #trying to check a bond on a server where none is present is probably 
not realy an error -> exit 0 with a warning message
  exit 0
fi


if [ -z $target ]; then
  # no target given as parameter -> auto-detect
  # get the default gateway as a ping target
  target=`route -n | grep UG | awk '{print $2}'`
  if [ -z $target ]; then
    echoe "Error: Unable to auto-detect the target to use during test 
(use -t?)."
    exit 3
  fi
fi


if [ -z "$ip" ]; then
  # no ip given as parameter -> auto-detect
  ip_b1=`host "${host}-bond-t1" | grep -o "$re_ip"`
  ip_b2=`host "${host}-bond-t2" | grep -o "$re_ip"`
  if [ -z "$ip_b1" ] && [ -z "$ip_b2" ]; then
    echoe "Error: Unable to auto-detect an ip to use during test (use -i?)."
    exit 4
  fi
fi

error_nb=0
for bond in $bonds
do
  bond=`basename $bond`
  echoq "checking bond $bond"
  active=`grep  "Active Slave" /proc/net/bonding/$bond |cut -d':' -f2`
  echoq -n "  active slave   :$active"
  check_nic $target
  error_nb=$(($error_nb + $check_nic_ret))
  slaves=`grep "Slave Interface:" /proc/net/bonding/$bond |cut -d':' -f2`
  slave_nb=0
  for slave in $slaves
  do
    if [ $slave != $active ]; then
      # this nic is enslaved but not active. we want to check if it is 
ready to work (no cable or VPN trouble that will bite us only when the 
active slave will change)
      echoq -n "  inactive slave : $slave"
      # search for a free mac in this bond (ie a real phy MAC that is 
not the one used by the bond)
      get_free_mac $bond $slave
      free_mac=$get_free_mac_ret
      # store the bond/nic we are going to un-enslave (to be able de 
re-enslave it in case of interrupt)
      clean_bond=$bond
      clean_nic=$slave
      if [ -z "$ip" ]; then
        ip="$ip_b1" # TODO: use a clever way to match slave and free ip
      fi
      # free this nic from the bond
      ifenslave -d $bond $slave
      # set it up with a "free" mac
      ifconfig $slave hw ether $free_mac
      # set it up with a temp IP
      ifconfig $slave $ip netmask 255.255.255.255
      # it seems we need a small temporisation here or the rest may fail
      sleep 2
      exercise_nic_arp $target $slave $ip
      check_nic $target $slave $ip
      error_nb=$(($error_nb + $check_nic_ret))
      # clean this temporary ip/route
      ifconfig $slave down
      # re-enslave this nic to the bond
      ifenslave $bond $slave
      clean_bond=""; clean_nic=""
      slave_nb=$(($slave_nb + 1))
    fi
  done
  echoq -n "  bond           : $bond"
  check_nic $target $bond
  error_nb=$(($error_nb + $check_nic_ret))
  if [ $slave_nb -eq 0 ]; then
    echoe "Error: No inactive slave in $bond."
    exit 5
  fi
done

if [ $error_nb -ne 0 ]; then
  exit $((10 + $error_nb))
fi
exit 0



---------------------------------------------------------------------------------------






_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc