Re: igb + balance-rr + bridge + IPv6 = no go without promiscuous mode

[Chris Boot <bootc@bootc.net>]

On 23/12/2011 10:48, Nicolas de Pesloüan wrote:
> [ Forwarded to netdev, because two previous e-mail erroneously sent in 
> HTML ]
>
> Le 23/12/2011 11:15, Chris Boot a écrit :
>> On 23/12/2011 09:52, Nicolas de Pesloüan wrote:
>>>
>>>
>>> Le 23 déc. 2011 10:42, "Chris Boot" <bootc@bootc.net 
>>> <mailto:bootc@bootc.net>> a écrit :
>>> >
>>> > Hi folks,
>>> >
>>> > As per Eric Dumazet and Dave Miller, I'm opening up a separate 
>>> thread on this issue.
>>> >
>>> > I have two identical servers in a cluster for running KVM virtual 
>>> machines. They each have a
>>> single connection to the Internet (irrelevant for this) and two 
>>> gigabit connections between each
>>> other for cluster replication, etc... These two connections are in a 
>>> balance-rr bonded connection,
>>> which is itself member of a bridge that the VMs attach to. I'm 
>>> running v3.2-rc6-140-gb9e26df on
>>> Debian Wheezy.
>>> >
>>> > When the bridge is brought up, IPv4 works fine but IPv6 does not. 
>>> I can use neither the
>>> automatic link-local on the brid ge nor the static global address I 
>>> assign. Neither machine can
>>> perform neighbour discovery over the link until I put the bond 
>>> members (eth0 and eth1) into
>>> promiscuous mode.  I can do this either with tcpdump or 'ip link set 
>>> dev ethX promisc on' and this
>>> is enough to make the link spring to life.
>>>
>>> For as far as I remember, setting bond0 to promisc should set the 
>>> bonding member to promisc too.
>>> And inserting bond0 into br0 should set bond0 to promisc... So 
>>> everything should be in promisc
>>> mode anyway... but you shoudn't have to do it by hand.
>>>
>>
>> Sorry, I should have added that I tried this. Setting bond0 or br0 to 
>> promisc has no effect. I
>> discovered this by running tcpdump on br0 first, then bond0, then 
>> eventually each bond member in
>> turn. Only at the last stage did things jump to life.
>>
>>> >
>>> > This cluster is not currently live so I can easily test patches 
>>> and various configurations.
>>>
>>> Can you try to remove the bonding part, connecting eth0 and eth1 
>>> directly to br0 and see if it
>>> works better? (This is a test ony. I perfectly understand that you 
>>> would loose balance-rr in this
>>> setup.)
>>>
>>
>> Good call. Let's see.
>>
>> I took br0 and bond0 apart, took eth0 and eth1 out of enforced 
>> promisc mode, then manually built a
>> br0 with eth0 in only so I didn't cause a network loop. Adding eth0 
>> to br0 did not make it go into
>> promisc mode, but IPv6 does work over this setup. I also made sure ip 
>> -6 neigh was empty on both
>> machines before I started.
>>
>> I then decided to try the test with just the bond0 in balance-rr 
>> mode. Once again I took everything
>> down and ensured no promisc mode and no ip -6 neigh. I noticed bond0 
>> wasn't getting a link-local and
>> I found out for some reason 
>> /proc/sys/net/ipv6/conf/bond0/disable_ipv6 was set on both servers so I
>> set it to 0. That brought things to life.
>>
>> So then I put it all back together again and it didn't work. I once 
>> again noticed disable_ipv6 was
>> set on the bond0 interfaces, now part of the bridge. Toggling this on 
>> the _bond_ interface made
>> things work again.
>>
>> What's setting disable_ipv6? Should this be having an impact if the 
>> port is part of a bridge?

Hmm, as a further update... I brought up my VMs on the bridge with 
disable_ipv6 turned off. The VMs on one host couldn't see what was on 
the other side of the bridge (on the other server) until I turned 
promisc back on manually. So it's not entirely disable_ipv6's fault.

Chris

-- 
Chris Boot
bootc@bootc.net