gratuitous arps lost during IB switch failure

gratuitous arps lost during IB switch failure

[Sumeet Lahorani]

Hi All,

We are using dual ported HCAs connected with each port connected to 2 
different IB switches so that we can tolerate the failure of any one of 
those switches and we are trying to cut down the amount of time it takes 
for traffic (TCP & RDS) to resume when there is an IB switch failure and 
the hosts failover from one port to the other.

We have the bonding driver configured in active-backup mode and setup to 
send out 100 gratuitous arps at intervals of 100ms whenever there is a 
failover. In most cases, traffic resumes within a few seconds after a 
failover because these gratuitous arps take care of updating all the 
nodes with the new IP:GID mapping.

The problem we are seeing is that sometimes, one or more of the nodes on 
the fabric do not receive even 1 of these gratuitous arps and 
re-establishing communication with these nodes takes a much longer time 
(over 40 seconds) as it depends on various arp cache timeouts. Does 
anyone know why all these gratuitous arps might be lost?

Besides the gratuitous arp settings, are there any other tunables to 
look at to minimize the time it takes for IPoIB traffic to resume?

- Sumeet

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: gratuitous arps lost during IB switch failure

[Sumeet]

It turns out that this problem was being caused because we had multiple IPs
configured on the bonded infiniband interface. It appears that grat. arps are
being sent out for only one of those IPs. 

For example, if we configure the bonded interfaces as below, and we trigger a
failover, we only see grat arps go out for the bond0:1 IP. Can the bonding
driver be fixed to send out grat arps for both these IPs? 

# cat ifcfg-bond0
DEVICE=bond0
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.50.118
NETMASK=255.255.254.0
NETWORK=192.168.50.0
BROADCAST=192.168.51.255
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"
MTU=65520

# cat ifcfg-bond0:1
DEVICE=bond0:1
USERCTL=no
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.169.50.118
NETMASK=255.255.254.0
NETWORK=192.169.50.0
BROADCAST=192.169.51.255
BONDING_OPTS="mode=active-backup miimon=100 downdelay=5000 updelay=5000"
MTU=65520

- Sumeet




--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: gratuitous arps lost during IB switch failure

[Or Gerlitz]

Sumeet <Sumeet.Lahorani-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> It turns out that this problem was being caused because we had multiple IPs
> configured on the bonded infiniband interface. It appears that grat. arps are
> being sent out for only one of those IPs. [...]  Can the bonding
> driver be fixed to send out grat arps for both these IPs?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: gratuitous arps lost during IB switch failure

[Or Gerlitz]

Sumeet <Sumeet.Lahorani-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
> It turns out that this problem was being caused because we had multiple IPs
> configured on the bonded infiniband interface. It appears that grat. arps are
> being sent out for only one of those IPs. [...]  Can the bonding
> driver be fixed to send out grat arps for both these IPs?

is there anything that makes you think this issue has something to do
with ipoib/bonding? did you check with ethernet? the bonding driver
isn't maintained over linux-rdma but rather over netdev.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html