[Bridge] Link fail over - bridge slow learning !?!

[Bridge] Link fail over - bridge slow learning !?!

[Alexander Indenbaum]

Hello,

I'm playing with dual-port NIC driver level link failover:
* Driver exposes single network interface to the OS and operates both
ports in active-passive failover mode.
* Upon Link down event on active port, driver switches active and
passive ports transparently for the OS.

I'm testing the driver using Linux bridge module: "failover" dual-port
NIC connected with two cables back-to-back to eth0 and eth1 which are
part of br0 bridge.

I simulate link fail with following scenario:
* At t0 both eth0 and eth1 port links are UP, traffic is accepted by
eth0 and forwarded to br0
* At t1 I manually unplug eth0 cable, causing link to go down.
"Failover" driver switches the traffic immediately from eth0 to eth1,
while using the same MAC address.
* From t1 till t1+12 secs packets are accepted by eth1 but dropped by
bridge and not forwarded to br0.
* At t1+12 secs bridge starts forwarding packets from eth1 to br0

Hmm... I would expect that eth0 link down event would flush from
bridge's table any MAC address associated with the port and that the
bridge would start forwarding packets from eth1 to br0 immediately.

Why does it take ~12 secs for bridge to learn that MAC address moved
from eth0 to eth1 in the described scenario?

Thank you for your help,
   Alexander Indenbaum

Re: [Bridge] Link fail over - bridge slow learning !?!

[Alex Zeffertt]

Alexander Indenbaum wrote:
> Hello,
> 
> I'm playing with dual-port NIC driver level link failover:
> * Driver exposes single network interface to the OS and operates both
> ports in active-passive failover mode.
> * Upon Link down event on active port, driver switches active and
> passive ports transparently for the OS.
> 
> I'm testing the driver using Linux bridge module: "failover" dual-port
> NIC connected with two cables back-to-back to eth0 and eth1 which are
> part of br0 bridge.
> 
> I simulate link fail with following scenario:
> * At t0 both eth0 and eth1 port links are UP, traffic is accepted by
> eth0 and forwarded to br0
> * At t1 I manually unplug eth0 cable, causing link to go down.
> "Failover" driver switches the traffic immediately from eth0 to eth1,
> while using the same MAC address.
> * From t1 till t1+12 secs packets are accepted by eth1 but dropped by
> bridge and not forwarded to br0.
> * At t1+12 secs bridge starts forwarding packets from eth1 to br0
> 
> Hmm... I would expect that eth0 link down event would flush from
> bridge's table any MAC address associated with the port and that the
> bridge would start forwarding packets from eth1 to br0 immediately.
> 
> Why does it take ~12 secs for bridge to learn that MAC address moved
> from eth0 to eth1 in the described scenario?
> 

It may be spanning tree, rather than MAC address learning that takes
so long.  Bridges spend a while just listening before forwarding, to
avoid becoming part of a bridging loop.

Experiment with:

   brctl showstp br0
   brctl stp br0 off
   brctl setfd br0 1

Alex

Re: [Bridge] Link fail over - bridge slow learning !?!

[Alexander Indenbaum]

Alex,

That was my first thought too, but STP is disabled.

On 10/11/06, Alex Zeffertt <ajz@cambridgebroadband.com> wrote:
> Alexander Indenbaum wrote:
> > Hello,
> >
> > I'm playing with dual-port NIC driver level link failover:
> > * Driver exposes single network interface to the OS and operates both
> > ports in active-passive failover mode.
> > * Upon Link down event on active port, driver switches active and
> > passive ports transparently for the OS.
> >
> > I'm testing the driver using Linux bridge module: "failover" dual-port
> > NIC connected with two cables back-to-back to eth0 and eth1 which are
> > part of br0 bridge.
> >
> > I simulate link fail with following scenario:
> > * At t0 both eth0 and eth1 port links are UP, traffic is accepted by
> > eth0 and forwarded to br0
> > * At t1 I manually unplug eth0 cable, causing link to go down.
> > "Failover" driver switches the traffic immediately from eth0 to eth1,
> > while using the same MAC address.
> > * From t1 till t1+12 secs packets are accepted by eth1 but dropped by
> > bridge and not forwarded to br0.
> > * At t1+12 secs bridge starts forwarding packets from eth1 to br0
> >
> > Hmm... I would expect that eth0 link down event would flush from
> > bridge's table any MAC address associated with the port and that the
> > bridge would start forwarding packets from eth1 to br0 immediately.
> >
> > Why does it take ~12 secs for bridge to learn that MAC address moved
> > from eth0 to eth1 in the described scenario?
> >
>
> It may be spanning tree, rather than MAC address learning that takes
> so long.  Bridges spend a while just listening before forwarding, to
> avoid becoming part of a bridging loop.
>
> Experiment with:
>
>    brctl showstp br0
>    brctl stp br0 off
>    brctl setfd br0 1
>
> Alex
>
>


-- 
Alexander Indenbaum

Re: [Bridge] Link fail over - bridge slow learning !?!

[Alex Zeffertt]

It still uses a forwarding delay.  Try "brctl setfd br0 1" to set it to
1 second.

Alexander Indenbaum wrote:
> Alex,
> 
> That was my first thought too, but STP is disabled.
> 
> On 10/11/06, Alex Zeffertt <ajz@cambridgebroadband.com> wrote:
>> Alexander Indenbaum wrote:
>> > Hello,
>> >
>> > I'm playing with dual-port NIC driver level link failover:
>> > * Driver exposes single network interface to the OS and operates both
>> > ports in active-passive failover mode.
>> > * Upon Link down event on active port, driver switches active and
>> > passive ports transparently for the OS.
>> >
>> > I'm testing the driver using Linux bridge module: "failover" dual-port
>> > NIC connected with two cables back-to-back to eth0 and eth1 which are
>> > part of br0 bridge.
>> >
>> > I simulate link fail with following scenario:
>> > * At t0 both eth0 and eth1 port links are UP, traffic is accepted by
>> > eth0 and forwarded to br0
>> > * At t1 I manually unplug eth0 cable, causing link to go down.
>> > "Failover" driver switches the traffic immediately from eth0 to eth1,
>> > while using the same MAC address.
>> > * From t1 till t1+12 secs packets are accepted by eth1 but dropped by
>> > bridge and not forwarded to br0.
>> > * At t1+12 secs bridge starts forwarding packets from eth1 to br0
>> >
>> > Hmm... I would expect that eth0 link down event would flush from
>> > bridge's table any MAC address associated with the port and that the
>> > bridge would start forwarding packets from eth1 to br0 immediately.
>> >
>> > Why does it take ~12 secs for bridge to learn that MAC address moved
>> > from eth0 to eth1 in the described scenario?
>> >
>>
>> It may be spanning tree, rather than MAC address learning that takes
>> so long.  Bridges spend a while just listening before forwarding, to
>> avoid becoming part of a bridging loop.
>>
>> Experiment with:
>>
>>    brctl showstp br0
>>    brctl stp br0 off
>>    brctl setfd br0 1
>>
>> Alex
>>
>>
> 
> 

Re: [Bridge] Link fail over - bridge slow learning !?!

[Stephen Hemminger]

Does this do what you want?

Flush the forwarding table when carrier is lost. This helps for
availability because new packets may come in on other links.

Signed-off-by: Stephen Hemminger <shemminger@osdl.org>
---
 net/bridge/br_fdb.c     |    9 ++++++++-
 net/bridge/br_if.c      |    4 ++--
 net/bridge/br_private.h |    2 +-
 net/bridge/br_stp_if.c  |    2 ++
 4 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/net/bridge/br_fdb.c b/net/bridge/br_fdb.c
index 3a73b8c..557789b 100644
--- a/net/bridge/br_fdb.c
+++ b/net/bridge/br_fdb.c
@@ -128,7 +128,10 @@ void br_fdb_cleanup(unsigned long _data)
 	mod_timer(&br->gc_timer, jiffies + HZ/10);
 }
 
-void br_fdb_delete_by_port(struct net_bridge *br, struct net_bridge_port *p)
+
+void br_fdb_delete_by_port(struct net_bridge *br, 
+			   const struct net_bridge_port *p,
+			   int do_all)
 {
 	int i;
 
@@ -142,6 +145,8 @@ void br_fdb_delete_by_port(struct net_br
 			if (f->dst != p) 
 				continue;
 
+			if (f->is_static & !do_all)
+				continue;
 			/*
 			 * if multiple ports all have the same device address
 			 * then when one port is deleted, assign
@@ -149,6 +154,7 @@ void br_fdb_delete_by_port(struct net_br
 			 */
 			if (f->is_local) {
 				struct net_bridge_port *op;
+
 				list_for_each_entry(op, &br->port_list, list) {
 					if (op != p && 
 					    !compare_ether_addr(op->dev->dev_addr,
@@ -166,6 +172,7 @@ void br_fdb_delete_by_port(struct net_br
 	spin_unlock_bh(&br->hash_lock);
 }
 
+
 /* No locking or refcounting, assumes caller has no preempt (rcu_read_lock) */
 struct net_bridge_fdb_entry *__br_fdb_get(struct net_bridge *br,
 					  const unsigned char *addr)
diff --git a/net/bridge/br_if.c b/net/bridge/br_if.c
index b1211d5..f753c40 100644
--- a/net/bridge/br_if.c
+++ b/net/bridge/br_if.c
@@ -163,7 +163,7 @@ static void del_nbp(struct net_bridge_po
 	br_stp_disable_port(p);
 	spin_unlock_bh(&br->lock);
 
-	br_fdb_delete_by_port(br, p);
+	br_fdb_delete_by_port(br, p, 1);
 
 	list_del_rcu(&p->list);
 
@@ -448,7 +448,7 @@ int br_add_if(struct net_bridge *br, str
 
 	return 0;
 err2:
-	br_fdb_delete_by_port(br, p);
+	br_fdb_delete_by_port(br, p, 1);
 err1:
 	kobject_del(&p->kobj);
 err0:
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index c491fb2..74258d8 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -143,7 +143,7 @@ extern void br_fdb_changeaddr(struct net
 			      const unsigned char *newaddr);
 extern void br_fdb_cleanup(unsigned long arg);
 extern void br_fdb_delete_by_port(struct net_bridge *br,
-			   struct net_bridge_port *p);
+				  const struct net_bridge_port *p, int do_all);
 extern struct net_bridge_fdb_entry *__br_fdb_get(struct net_bridge *br,
 						 const unsigned char *addr);
 extern struct net_bridge_fdb_entry *br_fdb_get(struct net_bridge *br,
diff --git a/net/bridge/br_stp_if.c b/net/bridge/br_stp_if.c
index 14cd025..d294224 100644
--- a/net/bridge/br_stp_if.c
+++ b/net/bridge/br_stp_if.c
@@ -113,6 +113,8 @@ void br_stp_disable_port(struct net_brid
 	del_timer(&p->forward_delay_timer);
 	del_timer(&p->hold_timer);
 
+	br_fdb_delete_by_port(br, p, 0);
+
 	br_configuration_update(br);
 
 	br_port_state_selection(br);
-- 
1.4.2.3

Re: [Bridge] Link fail over - bridge slow learning !?!

[Alexander Indenbaum]

On 10/11/06, Alex Zeffertt <ajz@cambridgebroadband.com> wrote:
> It still uses a forwarding delay.  Try "brctl setfd br0 1" to set it to
> 1 second.

Thanks Alex. It did the trick.
Could you explain why?

>
> Alexander Indenbaum wrote:
> > Alex,
> >
> > That was my first thought too, but STP is disabled.
> >
> > On 10/11/06, Alex Zeffertt <ajz@cambridgebroadband.com> wrote:
> >> Alexander Indenbaum wrote:
> >> > Hello,
> >> >
> >> > I'm playing with dual-port NIC driver level link failover:
> >> > * Driver exposes single network interface to the OS and operates both
> >> > ports in active-passive failover mode.
> >> > * Upon Link down event on active port, driver switches active and
> >> > passive ports transparently for the OS.
> >> >
> >> > I'm testing the driver using Linux bridge module: "failover" dual-port
> >> > NIC connected with two cables back-to-back to eth0 and eth1 which are
> >> > part of br0 bridge.
> >> >
> >> > I simulate link fail with following scenario:
> >> > * At t0 both eth0 and eth1 port links are UP, traffic is accepted by
> >> > eth0 and forwarded to br0
> >> > * At t1 I manually unplug eth0 cable, causing link to go down.
> >> > "Failover" driver switches the traffic immediately from eth0 to eth1,
> >> > while using the same MAC address.
> >> > * From t1 till t1+12 secs packets are accepted by eth1 but dropped by
> >> > bridge and not forwarded to br0.
> >> > * At t1+12 secs bridge starts forwarding packets from eth1 to br0
> >> >
> >> > Hmm... I would expect that eth0 link down event would flush from
> >> > bridge's table any MAC address associated with the port and that the
> >> > bridge would start forwarding packets from eth1 to br0 immediately.
> >> >
> >> > Why does it take ~12 secs for bridge to learn that MAC address moved
> >> > from eth0 to eth1 in the described scenario?
> >> >
> >>
> >> It may be spanning tree, rather than MAC address learning that takes
> >> so long.  Bridges spend a while just listening before forwarding, to
> >> avoid becoming part of a bridging loop.
> >>
> >> Experiment with:
> >>
> >>    brctl showstp br0
> >>    brctl stp br0 off
> >>    brctl setfd br0 1
> >>
> >> Alex
> >>
> >>
> >
> >
>
>


-- 
Alexander Indenbaum

Re: [Bridge] Link fail over - bridge slow learning !?!

[Alex Zeffertt]

Alex,

I don't fully understand STP.  You can look it up in the RFCs if you like.

What I know is this:

  * With STP on you get (by default) 15 seconds in LISTENING mode, then
    15 seconds in LEARNING mode, before entering FORWARDING mode.

  * With STP off you would think you'd just start FORWARDING as soon as
    br0 is if-upped, however you still get 15 seconds delay.  This can
    be reduced to 1 second though with "brctl setfd br0 1".  (I haven't
    tried 0 seconds!)

And that is all I know.  Perhaps somebody else on this list can tell you
why you still get a delay when spanning tree is disabled!

Alex

Alexander Indenbaum wrote:
> On 10/11/06, Alex Zeffertt <ajz@cambridgebroadband.com> wrote:
>> It still uses a forwarding delay.  Try "brctl setfd br0 1" to set it to
>> 1 second.
> 
> Thanks Alex. It did the trick.
> Could you explain why?
> 
>>
>> Alexander Indenbaum wrote:
>> > Alex,
>> >
>> > That was my first thought too, but STP is disabled.
>> >
>> > On 10/11/06, Alex Zeffertt <ajz@cambridgebroadband.com> wrote:
>> >> Alexander Indenbaum wrote:
>> >> > Hello,
>> >> >
>> >> > I'm playing with dual-port NIC driver level link failover:
>> >> > * Driver exposes single network interface to the OS and operates 
>> both
>> >> > ports in active-passive failover mode.
>> >> > * Upon Link down event on active port, driver switches active and
>> >> > passive ports transparently for the OS.
>> >> >
>> >> > I'm testing the driver using Linux bridge module: "failover" 
>> dual-port
>> >> > NIC connected with two cables back-to-back to eth0 and eth1 which 
>> are
>> >> > part of br0 bridge.
>> >> >
>> >> > I simulate link fail with following scenario:
>> >> > * At t0 both eth0 and eth1 port links are UP, traffic is accepted by
>> >> > eth0 and forwarded to br0
>> >> > * At t1 I manually unplug eth0 cable, causing link to go down.
>> >> > "Failover" driver switches the traffic immediately from eth0 to 
>> eth1,
>> >> > while using the same MAC address.
>> >> > * From t1 till t1+12 secs packets are accepted by eth1 but 
>> dropped by
>> >> > bridge and not forwarded to br0.
>> >> > * At t1+12 secs bridge starts forwarding packets from eth1 to br0
>> >> >
>> >> > Hmm... I would expect that eth0 link down event would flush from
>> >> > bridge's table any MAC address associated with the port and that the
>> >> > bridge would start forwarding packets from eth1 to br0 immediately.
>> >> >
>> >> > Why does it take ~12 secs for bridge to learn that MAC address moved
>> >> > from eth0 to eth1 in the described scenario?
>> >> >
>> >>
>> >> It may be spanning tree, rather than MAC address learning that takes
>> >> so long.  Bridges spend a while just listening before forwarding, to
>> >> avoid becoming part of a bridging loop.
>> >>
>> >> Experiment with:
>> >>
>> >>    brctl showstp br0
>> >>    brctl stp br0 off
>> >>    brctl setfd br0 1
>> >>
>> >> Alex
>> >>
>> >>
>> >
>> >
>>
>>
> 
>