Bonding in active-backup mode with arp monitoring on Xen

Bonding in active-backup mode with arp monitoring on Xen

[Tsutomu Fujii]

[-- Attachment #1: Type: text/plain, Size: 1725 bytes --]

Hi.

I was trying to use bonding from a guest OS on xen in active-backup
mode with arp monitoring, but the virtual server can't communicate to
the peer server via bonding interface.

The network structure is the following.

+-------------------------------+
|        virtual I/F(eth0)      | <- have IP address.
+-------------------------------+
|      virtual bridge(xenbr0)   |
+-------------------------------+
|         bonding(bond0)        | <- don't have IP address
+--------------+-+--------------+
|phy I/F(peth0)| |phy I/F(peth1)|
+--------------+ +--------------+

The attached file describes how I configured the network.

In this case, when I run 'cat /proc/net/bonding/bond0', 'MII Status'
is shown as down.

This behavior only occurs in active-backup mode with arp monitoring.
It does not happen in active-backup mode with mii monitoring nor other
mode that can use arp monitoring.

>From the output of tcpdump, it seems that arp packet isn't transmitted.
This problem is caused because bond_activebackup_arp_mon(), which is the
function that sends arp packet in active-backup mode, checks whether
bonding master interface has an IP address(bond_has_ip()), before arp
packets are transmitted using bond_arp_send_all().
In the above setting, bonding master interface don't have IP address.

How about removing bond_has_ip() from the condition for calling
bond_arp_send_all() to use bonding with xen?
When I removed bond_has_ip(), the resulting kernel worked fine and
active-backup mode works with arp monitoring.

I confirmed this behavior in RHEL5(2.6.18-44.el5xen). And I confirmed
that the upstream kernel(2.6.23) also has the same condition before
calling bond_arp_send_all().

Thanks.

---
Tsutomu Fujii



[-- Attachment #2: set_bonding.txt --]
[-- Type: text/plain, Size: 1444 bytes --]

Setting procedure of bonding.

1) Make or modify the following config file on dom0.
   - /etc/sysconfig/network-scripts/ifcfg-bond0
     TYPE=Bonding
     DEVICE=bond0
     BOOTPROTO=none
     ONBOOT=no

   - /etc/sysconfig/network-scripts/ifcfg-eth1
     DEVICE=eth1
     BOOTPROTO=none
     ONBOOT=no

   - /etc/sysconfig/network-scripts/ifcfg-eth2
     DEVICE=eth2
     BOOTPROTO=none
     ONBOOT=no

   - /etc/modprobe.conf (Add to following two lines.)
     alias bond0 bonding
     options bonding mode=1 arp_interval=100 arp_ip_target=192.168.0.254

2) Run the following command on dom0.
   # brctl addbr xenbr0
   # ip link set xenbr0 arp off
   # ip link set xenbr0 multicast off

   # ip link set eth0 name peth0
   # ip link set eth1 name peth1
   # ip link set veth0 name eth0
   # ip link set veth1 name eth1

   # ifup bond0
   # ifenslave bond0 peth0 peth1
   # ifdown bond0

   # ip link set bond0 down
   # ip link set bond0 arp off
   # ip link set bond0 multicast off
   # ip link set bond0 addr fe:ff:ff:ff:ff:ff

   # ip link set vif0.0 down
   # ip link set vif0.0 arp off
   # ip link set vif0.0 multicast off
   # ip link set vif0.0 addr fe:ff:ff:ff:ff:ff

   # ip link set xenbr0 up
   # brctl addif xenbr0 bond0
   # ip link set bond0 up
   # brctl addif xenbr0 vif0.0
   # ip link set vif0.0 up

   # ip link set eth0 addr 00:16:3e:70:00:03
   # ip link set eth0 arp on
   # ifup eth0
   # ifconfig eth0 192.168.0.1

Re: Bonding in active-backup mode with arp monitoring on Xen

[Jay Vosburgh]

Tsutomu Fujii <t-fujii@nb.jp.nec.com> wrote:
[...]
>The network structure is the following.
>
>+-------------------------------+
>|        virtual I/F(eth0)      | <- have IP address.
>+-------------------------------+
>|      virtual bridge(xenbr0)   |
>+-------------------------------+
>|         bonding(bond0)        | <- don't have IP address
>+--------------+-+--------------+
>|phy I/F(peth0)| |phy I/F(peth1)|
>+--------------+ +--------------+
[...]
>How about removing bond_has_ip() from the condition for calling
>bond_arp_send_all() to use bonding with xen?
>When I removed bond_has_ip(), the resulting kernel worked fine and
>active-backup mode works with arp monitoring.

	I looked at this a bit.  I believe that when you remove the
bond_has_ip() test, the ARP probes sent out will be of the form "ARP
who-has A.B.C.D tell 0.0.0.0", which the linux kernel will interpret as
IPv4 Duplicate Address Detection probes (as a special case in
arp_process()), and generate some type of reply.  That reply appears to
be enough to keep the ARP monitor happy.

	I'm not sure if this is a solution that will work for any peer
(some peers may not reply to an ARP with an IP source of all zeros).  At
first glance, there doesn't seem to be much of a downside, but I'll have
to experiment with it a bit to see if the check should be optional or
simply removed entirely.

	-J

---
	-Jay Vosburgh, IBM Linux Technology Center, fubar@us.ibm.com

Re: Bonding in active-backup mode with arp monitoring on Xen

[Tsutomu Fujii]

Hi.

Jay Vosburgh wrote:
> 	I'm not sure if this is a solution that will work for any peer
> (some peers may not reply to an ARP with an IP source of all zeros).  At
> first glance, there doesn't seem to be much of a downside, but I'll have
> to experiment with it a bit to see if the check should be optional or
> simply removed entirely.

If there are no downside, I think it is better to remove the check.
For your information, other mode that can use arp monitoring send an
ARP with an IP source of all zero.

Thanks.

--
Tsutomu Fujii

Bonding in active-backup mode with arp monitoring on Xen

[Tsutomu Fujii]

Hi.

A little while ago I posted a patch to fix the communication problem
when bonding was used in active-backup mode with arp monitoring on Xen
guest OS.

http://marc.info/?l=linux-netdev&m=119362063924723&w=2

I have rebased the patch to the latest 2.6.25-rc7 kernel.

I've confirmed that active-backup mode works fine with arp monitoring
with this patch.

Thanks.

--
Tsutomu fujii

--- a/drivers/net/bonding/bond_main.c	2008-03-26 10:38:14.000000000 +0900
+++ b/drivers/net/bonding/bond_main.c	2008-03-27 19:39:06.000000000 +0900
@@ -3003,7 +3003,7 @@
 		/* the current slave must tx an arp to ensure backup slaves
 		 * rx traffic
 		 */
-		if (slave && bond_has_ip(bond)) {
+		if (slave && IS_UP(slave->dev)) {
 			bond_arp_send_all(bond, slave);
 		}
 	}