[Bridge] bonding inside a bridge does not work when using arp monitoring

[Bridge] bonding inside a bridge does not work when using arp monitoring

[Leonardo Borda]

Hi Guys,

In case you're interested or you guys have some hints to add to this
thread, I've opened a bug in Ubuntu Launchpad regarding Bonding + Bridge
issues.

https://bugs.launchpad.net/bugs/736226

Please let me know your thoughts about it.

-- 
Leonardo Borda
Server Support Analyst
Canonical Canada

Re: [Bridge] [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Nicolas de Pesloüan]

Le 16/03/2011 20:15, Leonardo Borda a écrit :
> Hi Guys,
>
> In case you're interested or you guys have some hints to add to this
> thread, I've opened a bug in Ubuntu Launchpad regarding Bonding + Bridge
> issues.
>
> https://bugs.launchpad.net/bugs/736226
>
> Please let me know your thoughts about it.
>

Hi Leonardo,

I'm afraid I don't understand your setup.

See my comments below.

 > auto bond0
 > iface bond0 inet manual
 >     post-up ifconfig $IFACE up
 >     pre-down ifconfig $IFACE down
 >     bond-slaves none
 >     bond-mode active-backup
 >     bond_arp_ip_target 10.153.107.1
 >     bond_arp_interval 100

 > auto eth0
 > allow-bond0 eth0
 > iface eth0 inet manual
 >     bond-master bond0

 > auto eth1
 > allow-bond0 eth1
 > iface eth1 inet manual
 >     bond-master bond0

eth0----+
         |
         +----bond0
         |
eth1----+

'sounds good up to this point.

 > auto bond0.100
 > iface bond0.100 inet manual
 >     post-up ifconfig $IFACE up
 >     pre-down ifconfig $IFACE down
 >     vlan-raw-device bond0

 > auto bond0.200
 > iface bond0.200 inet manual
 >     post-up ifconfig $IFACE up
 >     pre-down ifconfig $IFACE down
 >     vlan-raw-device bond0

eth0----+             +----bond0.100
         |             |
         +----bond0----+
         |             |
eth1----+             +----bond0.200

So you split based on the VLAN ID.

 > auto br0
 > iface br0 inet static
 >     address 10.153.107.110
 >     netmask 255.255.255.0
 >     gateway 10.153.107.1
 >     bridge_ports bond0
 >     bridge_stp off
 >     bridge_fd 0
 >     bridge_maxwait 0

eth0----+             +----bond0.100
         |             |
         +----bond0----+----br0
         |             |
eth1----+             +----bond0.200

br0 clearly is built on bond0. Starting from this point, I'm not sure I understand your setup.

Because bond0.100, bond0.200 and br0 are all built on bond0, I can imagine bonding will take the 
frame before bridge have a chance to see it (because bonding handling happens before bridge handling 
in __netif_receive_skb(), in the kernel version you use).

 > auto br0-100
 > iface br0-100 inet manual
 >     post-up ifconfig $IFACE up
 >     pre-down ifconfig $IFACE down
 >     bridge_ports bond0.100
 >     bridge_stp off
 >     bridge_fd 0
 >     bridge_maxwait 0

 > auto br0-200
 > iface br0-200 inet manual
 >     post-up ifconfig $IFACE up
 >     pre-down ifconfig $IFACE down
 >     bridge_ports bond0.200
 >     bridge_stp off
 >     bridge_fd 0
 >     bridge_maxwait 0

eth0----+             +----bond0.100----br0-100
         |             |
         +----bond0----+----br0
         |             |
eth1----+             +----bond0.200----br0-200

br0-100 is built on bond0.100 and br0-200 is built on bond0.200.

So you end up with 3 bridges, all having a single port... this sounds useless, from my point of view.

Can you do some sort of ascii art to describe your expected setup?

The only reason I can imagine to stack a bridge on top of bonding is the following:

eth0----+
         |
         +----bond0----+
         |             |
eth1----+             |
                       +----br0
eth2----+             |
         |             |
         +----bond1----+
         |
eth3----+

That way, the bridge becomes 802.3ad capable, thanks to bonding.

And if you want to do vlan on top of that:

eth0----+
         |
         +----bond0----+
         |             |           +----br0.100
eth1----+             |           |
                       +----br0----+
eth2----+             |           |
         |             |           +----br0.200
         +----bond1----+
         |
eth3----+

The following (untested) setup should provide this configuration:

auto bond0
iface bond0 inet manual
     bond-slaves none
     bond-mode active-backup
     bond_arp_ip_target 10.153.107.1
     bond_arp_interval 100

auto eth0
allow-bond0 eth0
iface eth0 inet manual
     bond-master bond0

auto eth1
allow-bond0 eth1
iface eth1 inet manual
     bond-master bond0

auto eth2
allow-bond1 eth2
iface eth0 inet manual
     bond-master bond1

auto eth3
allow-bond1 eth3
iface eth1 inet manual
     bond-master bond1

auto br0
iface br0 inet static
     bridge_ports bond0 bond1
     bridge_stp off
     bridge_fd 0
     bridge_maxwait 0

auto br0.100
iface br0.100 inet manual
     vlan-raw-device br0
     address 10.153.107.110
     netmask 255.255.255.0
     gateway 10.153.107.1

auto br0.200
iface br0.200 inet manual
     vlan-raw-device br0

	Nicolas.

Re: [Bridge] [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Leonardo Borda]

Hi Nicolas,

Thank you for answering my question.
Actually this is what I want to achieve:

eth0----+               +----bond0.100----br0-100---{+virtual machines
          |             |
          +----bond0----+----br0---(LAN)
          |             |
eth1----+               +----bond0.200----br0-200---{+virtual machines


br0 --> br0 in my understanding is an untagged vlan therefore it
provides access to my LAN. So i am able to access that server from my
internal network.
br0-100 and br0-200 -> Vlans over a bridged interface will allow me to
have many virtual machines in the same vlan on each bridged interface.

I am misunderstanding concepts, maybe?
If you need to do further tests I have a test environment ready for use.

Leonardo


On Wed, 2011-03-16 at 21:47 +0100, Nicolas de Pesloüan wrote: 
> Le 16/03/2011 20:15, Leonardo Borda a écrit :
> > Hi Guys,
> >
> > In case you're interested or you guys have some hints to add to this
> > thread, I've opened a bug in Ubuntu Launchpad regarding Bonding + Bridge
> > issues.
> >
> > https://bugs.launchpad.net/bugs/736226
> >
> > Please let me know your thoughts about it.
> >
> 
> Hi Leonardo,
> 
> I'm afraid I don't understand your setup.
> 
> See my comments below.
> 
>  > auto bond0
>  > iface bond0 inet manual
>  >     post-up ifconfig $IFACE up
>  >     pre-down ifconfig $IFACE down
>  >     bond-slaves none
>  >     bond-mode active-backup
>  >     bond_arp_ip_target 10.153.107.1
>  >     bond_arp_interval 100
> 
>  > auto eth0
>  > allow-bond0 eth0
>  > iface eth0 inet manual
>  >     bond-master bond0
> 
>  > auto eth1
>  > allow-bond0 eth1
>  > iface eth1 inet manual
>  >     bond-master bond0
> 
> eth0----+
>          |
>          +----bond0
>          |
> eth1----+
> 
> 'sounds good up to this point.
> 
>  > auto bond0.100
>  > iface bond0.100 inet manual
>  >     post-up ifconfig $IFACE up
>  >     pre-down ifconfig $IFACE down
>  >     vlan-raw-device bond0
> 
>  > auto bond0.200
>  > iface bond0.200 inet manual
>  >     post-up ifconfig $IFACE up
>  >     pre-down ifconfig $IFACE down
>  >     vlan-raw-device bond0
> 
> eth0----+             +----bond0.100
>          |             |
>          +----bond0----+
>          |             |
> eth1----+             +----bond0.200
> 
> So you split based on the VLAN ID.
> 
>  > auto br0
>  > iface br0 inet static
>  >     address 10.153.107.110
>  >     netmask 255.255.255.0
>  >     gateway 10.153.107.1
>  >     bridge_ports bond0
>  >     bridge_stp off
>  >     bridge_fd 0
>  >     bridge_maxwait 0
> 
> eth0----+             +----bond0.100
>          |             |
>          +----bond0----+----br0
>          |             |
> eth1----+             +----bond0.200
> 
> br0 clearly is built on bond0. Starting from this point, I'm not sure I understand your setup.
> 
> Because bond0.100, bond0.200 and br0 are all built on bond0, I can imagine bonding will take the 
> frame before bridge have a chance to see it (because bonding handling happens before bridge handling 
> in __netif_receive_skb(), in the kernel version you use).
> 
>  > auto br0-100
>  > iface br0-100 inet manual
>  >     post-up ifconfig $IFACE up
>  >     pre-down ifconfig $IFACE down
>  >     bridge_ports bond0.100
>  >     bridge_stp off
>  >     bridge_fd 0
>  >     bridge_maxwait 0
> 
>  > auto br0-200
>  > iface br0-200 inet manual
>  >     post-up ifconfig $IFACE up
>  >     pre-down ifconfig $IFACE down
>  >     bridge_ports bond0.200
>  >     bridge_stp off
>  >     bridge_fd 0
>  >     bridge_maxwait 0
> 
> eth0----+             +----bond0.100----br0-100
>          |             |
>          +----bond0----+----br0
>          |             |
> eth1----+             +----bond0.200----br0-200
> 
> br0-100 is built on bond0.100 and br0-200 is built on bond0.200.
> 
> So you end up with 3 bridges, all having a single port... this sounds useless, from my point of view.
> 
> Can you do some sort of ascii art to describe your expected setup?
> 
> The only reason I can imagine to stack a bridge on top of bonding is the following:
> 
> eth0----+
>          |
>          +----bond0----+
>          |             |
> eth1----+             |
>                        +----br0
> eth2----+             |
>          |             |
>          +----bond1----+
>          |
> eth3----+
> 
> That way, the bridge becomes 802.3ad capable, thanks to bonding.
> 
> And if you want to do vlan on top of that:
> 
> eth0----+
>          |
>          +----bond0----+
>          |             |           +----br0.100
> eth1----+             |           |
>                        +----br0----+
> eth2----+             |           |
>          |             |           +----br0.200
>          +----bond1----+
>          |
> eth3----+
> 
> The following (untested) setup should provide this configuration:
> 
> auto bond0
> iface bond0 inet manual
>      bond-slaves none
>      bond-mode active-backup
>      bond_arp_ip_target 10.153.107.1
>      bond_arp_interval 100
> 
> auto eth0
> allow-bond0 eth0
> iface eth0 inet manual
>      bond-master bond0
> 
> auto eth1
> allow-bond0 eth1
> iface eth1 inet manual
>      bond-master bond0
> 
> auto eth2
> allow-bond1 eth2
> iface eth0 inet manual
>      bond-master bond1
> 
> auto eth3
> allow-bond1 eth3
> iface eth1 inet manual
>      bond-master bond1
> 
> auto br0
> iface br0 inet static
>      bridge_ports bond0 bond1
>      bridge_stp off
>      bridge_fd 0
>      bridge_maxwait 0
> 
> auto br0.100
> iface br0.100 inet manual
>      vlan-raw-device br0
>      address 10.153.107.110
>      netmask 255.255.255.0
>      gateway 10.153.107.1
> 
> auto br0.200
> iface br0.200 inet manual
>      vlan-raw-device br0
> 
> 	Nicolas.

-- 
Leonardo Borda
Server Support Analyst
Canonical Canada

Re: [Bridge] [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Nicolas de Pesloüan]

Le 23/03/2011 22:13, Leonardo Borda a écrit :
> Hi Nicolas,
>
> Thank you for answering my question.
> Actually this is what I want to achieve:
>
> eth0----+               +----bond0.100----br0-100---{+virtual machines
>            |             |
>            +----bond0----+----br0---(LAN)
>            |             |
> eth1----+               +----bond0.200----br0-200---{+virtual machines

Hi Leonardo,

I'm not sure recent kernels allow for a given interface to be a port for a bridge and the base 
interface for vlan interfaces at the same time. This might be particularly true for 2.6.38 or 
2.6.38+, because of the new rx_handler usage.

cc: netdev and Jiri Pirko, for advices. For the history of the thread, see:

http://sourceforge.net/mailarchive/forum.php?thread_name=1300914794.32252.68.camel%40bordalnx&
forum_name=bonding-devel

> br0 -->  br0 in my understanding is an untagged vlan therefore it
> provides access to my LAN. So i am able to access that server from my
> internal network.
> br0-100 and br0-200 ->  Vlans over a bridged interface will allow me to
> have many virtual machines in the same vlan on each bridged interface.
>
> I am misunderstanding concepts, maybe?
> If you need to do further tests I have a test environment ready for use.
>
> Leonardo

Re: [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Nicolas de Pesloüan]

Le 23/03/2011 22:13, Leonardo Borda a écrit :
> Hi Nicolas,
>
> Thank you for answering my question.
> Actually this is what I want to achieve:
>
> eth0----+               +----bond0.100----br0-100---{+virtual machines
>            |             |
>            +----bond0----+----br0---(LAN)
>            |             |
> eth1----+               +----bond0.200----br0-200---{+virtual machines

Hi Leonardo,

I'm not sure recent kernels allow for a given interface to be a port for a bridge and the base 
interface for vlan interfaces at the same time. This might be particularly true for 2.6.38 or 
2.6.38+, because of the new rx_handler usage.

cc: netdev and Jiri Pirko, for advices. For the history of the thread, see:

http://sourceforge.net/mailarchive/forum.php?thread_name=1300914794.32252.68.camel%40bordalnx&
forum_name=bonding-devel

> br0 -->  br0 in my understanding is an untagged vlan therefore it
> provides access to my LAN. So i am able to access that server from my
> internal network.
> br0-100 and br0-200 ->  Vlans over a bridged interface will allow me to
> have many virtual machines in the same vlan on each bridged interface.
>
> I am misunderstanding concepts, maybe?
> If you need to do further tests I have a test environment ready for use.
>
> Leonardo

Re: [Bridge] [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Jiri Pirko]

Sat, Mar 26, 2011 at 01:20:22PM CET, nicolas.2p.debian@gmail.com wrote:
>Le 23/03/2011 22:13, Leonardo Borda a écrit :
>>Hi Nicolas,
>>
>>Thank you for answering my question.
>>Actually this is what I want to achieve:
>>
>>eth0----+               +----bond0.100----br0-100---{+virtual machines
>>           |             |
>>           +----bond0----+----br0---(LAN)
>>           |             |
>>eth1----+               +----bond0.200----br0-200---{+virtual machines
>
>Hi Leonardo,
>
>I'm not sure recent kernels allow for a given interface to be a port
>for a bridge and the base interface for vlan interfaces at the same
>time. This might be particularly true for 2.6.38 or 2.6.38+, because
>of the new rx_handler usage.

This topology is not legit and should/will be prohibited.

Only consider that you have + br0.100 device on top of br0. Where should
the packet go?

I suggest to consider topology change.

>
>cc: netdev and Jiri Pirko, for advices. For the history of the thread, see:
>
>http://sourceforge.net/mailarchive/forum.php?thread_name=1300914794.32252.68.camel%40bordalnx&
>forum_name=bonding-devel
>
>>br0 -->  br0 in my understanding is an untagged vlan therefore it
>>provides access to my LAN. So i am able to access that server from my
>>internal network.
>>br0-100 and br0-200 ->  Vlans over a bridged interface will allow me to
>>have many virtual machines in the same vlan on each bridged interface.
>>
>>I am misunderstanding concepts, maybe?
>>If you need to do further tests I have a test environment ready for use.
>>
>>Leonardo

Re: [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Jiri Pirko]

Sat, Mar 26, 2011 at 01:20:22PM CET, nicolas.2p.debian@gmail.com wrote:
>Le 23/03/2011 22:13, Leonardo Borda a écrit :
>>Hi Nicolas,
>>
>>Thank you for answering my question.
>>Actually this is what I want to achieve:
>>
>>eth0----+               +----bond0.100----br0-100---{+virtual machines
>>           |             |
>>           +----bond0----+----br0---(LAN)
>>           |             |
>>eth1----+               +----bond0.200----br0-200---{+virtual machines
>
>Hi Leonardo,
>
>I'm not sure recent kernels allow for a given interface to be a port
>for a bridge and the base interface for vlan interfaces at the same
>time. This might be particularly true for 2.6.38 or 2.6.38+, because
>of the new rx_handler usage.

This topology is not legit and should/will be prohibited.

Only consider that you have + br0.100 device on top of br0. Where should
the packet go?

I suggest to consider topology change.

>
>cc: netdev and Jiri Pirko, for advices. For the history of the thread, see:
>
>http://sourceforge.net/mailarchive/forum.php?thread_name=1300914794.32252.68.camel%40bordalnx&
>forum_name=bonding-devel
>
>>br0 -->  br0 in my understanding is an untagged vlan therefore it
>>provides access to my LAN. So i am able to access that server from my
>>internal network.
>>br0-100 and br0-200 ->  Vlans over a bridged interface will allow me to
>>have many virtual machines in the same vlan on each bridged interface.
>>
>>I am misunderstanding concepts, maybe?
>>If you need to do further tests I have a test environment ready for use.
>>
>>Leonardo

Re: [Bridge] [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Michał Mirosław]

2011/3/26 Jiri Pirko <jpirko@redhat.com>:
> Sat, Mar 26, 2011 at 01:20:22PM CET, nicolas.2p.debian@gmail.com wrote:
>>Le 23/03/2011 22:13, Leonardo Borda a écrit :
>>>Thank you for answering my question.
>>>Actually this is what I want to achieve:
>>>
>>>eth0----+               +----bond0.100----br0-100---{+virtual machines
>>>           |             |
>>>           +----bond0----+----br0---(LAN)
>>>           |             |
>>>eth1----+               +----bond0.200----br0-200---{+virtual machines
>>
>>Hi Leonardo,
>>
>>I'm not sure recent kernels allow for a given interface to be a port
>>for a bridge and the base interface for vlan interfaces at the same
>>time. This might be particularly true for 2.6.38 or 2.6.38+, because
>>of the new rx_handler usage.
>
> This topology is not legit and should/will be prohibited.
>
> Only consider that you have + br0.100 device on top of br0. Where should
> the packet go?
>
> I suggest to consider topology change.

It should be possible to have bridge for untagged (or 802.1p only)
packets independent of 802.1q tagged packets. I wonder if tag 0
devices should be expanded to have a flag that will enable handling
untagged packets by it.

Best Regards,
Michał Mirosław

Re: [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Michał Mirosław]

2011/3/26 Jiri Pirko <jpirko@redhat.com>:
> Sat, Mar 26, 2011 at 01:20:22PM CET, nicolas.2p.debian@gmail.com wrote:
>>Le 23/03/2011 22:13, Leonardo Borda a écrit :
>>>Thank you for answering my question.
>>>Actually this is what I want to achieve:
>>>
>>>eth0----+               +----bond0.100----br0-100---{+virtual machines
>>>           |             |
>>>           +----bond0----+----br0---(LAN)
>>>           |             |
>>>eth1----+               +----bond0.200----br0-200---{+virtual machines
>>
>>Hi Leonardo,
>>
>>I'm not sure recent kernels allow for a given interface to be a port
>>for a bridge and the base interface for vlan interfaces at the same
>>time. This might be particularly true for 2.6.38 or 2.6.38+, because
>>of the new rx_handler usage.
>
> This topology is not legit and should/will be prohibited.
>
> Only consider that you have + br0.100 device on top of br0. Where should
> the packet go?
>
> I suggest to consider topology change.

It should be possible to have bridge for untagged (or 802.1p only)
packets independent of 802.1q tagged packets. I wonder if tag 0
devices should be expanded to have a flag that will enable handling
untagged packets by it.

Best Regards,
Michał Mirosław

Re: [Bridge] [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Nicolas de Pesloüan]

Le 26/03/2011 16:42, Michał Mirosław a écrit :
> 2011/3/26 Jiri Pirko<jpirko@redhat.com>:
>> Sat, Mar 26, 2011 at 01:20:22PM CET, nicolas.2p.debian@gmail.com wrote:
>>> Le 23/03/2011 22:13, Leonardo Borda a écrit :
>>>> Thank you for answering my question.
>>>> Actually this is what I want to achieve:
>>>>
>>>> eth0----+               +----bond0.100----br0-100---{+virtual machines
>>>>            |             |
>>>>            +----bond0----+----br0---(LAN)
>>>>            |             |
>>>> eth1----+               +----bond0.200----br0-200---{+virtual machines
>>>
>>> Hi Leonardo,
>>>
>>> I'm not sure recent kernels allow for a given interface to be a port
>>> for a bridge and the base interface for vlan interfaces at the same
>>> time. This might be particularly true for 2.6.38 or 2.6.38+, because
>>> of the new rx_handler usage.
>>
>> This topology is not legit and should/will be prohibited.
>>
>> Only consider that you have + br0.100 device on top of br0. Where should
>> the packet go?
>>
>> I suggest to consider topology change.
>
> It should be possible to have bridge for untagged (or 802.1p only)
> packets independent of 802.1q tagged packets. I wonder if tag 0
> devices should be expanded to have a flag that will enable handling
> untagged packets by it.

Isn't the BROUTING chain of the broute table of ebtables designed exactly for that?

I think DROPing in this chain should allow delivery to VLAN:

In br_input.c :

                 rhook = rcu_dereference(br_should_route_hook);
                 if (rhook) {
                         if ((*rhook)(skb)) {
                                 *pskb = skb;
                                 return RX_HANDLER_PASS;
                         }

RX_HANDLER_PASS causes the skb to be normally delivered in __netif_receive_skb.

Leonardo, would you please try to DROP vlan tagged packets in the BROUTING chain of the broute table 
of ebtables?

	Nicolas.

Re: [Bonding-devel] bonding inside a bridge does not work when using arp monitoring

[Nicolas de Pesloüan]

Le 26/03/2011 16:42, Michał Mirosław a écrit :
> 2011/3/26 Jiri Pirko<jpirko@redhat.com>:
>> Sat, Mar 26, 2011 at 01:20:22PM CET, nicolas.2p.debian@gmail.com wrote:
>>> Le 23/03/2011 22:13, Leonardo Borda a écrit :
>>>> Thank you for answering my question.
>>>> Actually this is what I want to achieve:
>>>>
>>>> eth0----+               +----bond0.100----br0-100---{+virtual machines
>>>>            |             |
>>>>            +----bond0----+----br0---(LAN)
>>>>            |             |
>>>> eth1----+               +----bond0.200----br0-200---{+virtual machines
>>>
>>> Hi Leonardo,
>>>
>>> I'm not sure recent kernels allow for a given interface to be a port
>>> for a bridge and the base interface for vlan interfaces at the same
>>> time. This might be particularly true for 2.6.38 or 2.6.38+, because
>>> of the new rx_handler usage.
>>
>> This topology is not legit and should/will be prohibited.
>>
>> Only consider that you have + br0.100 device on top of br0. Where should
>> the packet go?
>>
>> I suggest to consider topology change.
>
> It should be possible to have bridge for untagged (or 802.1p only)
> packets independent of 802.1q tagged packets. I wonder if tag 0
> devices should be expanded to have a flag that will enable handling
> untagged packets by it.

Isn't the BROUTING chain of the broute table of ebtables designed exactly for that?

I think DROPing in this chain should allow delivery to VLAN:

In br_input.c :

                 rhook = rcu_dereference(br_should_route_hook);
                 if (rhook) {
                         if ((*rhook)(skb)) {
                                 *pskb = skb;
                                 return RX_HANDLER_PASS;
                         }

RX_HANDLER_PASS causes the skb to be normally delivered in __netif_receive_skb.

Leonardo, would you please try to DROP vlan tagged packets in the BROUTING chain of the broute table 
of ebtables?

	Nicolas.