bridge not learning from locally sent gratuitous ARP?

bridge not learning from locally sent gratuitous ARP?

[Patrick Schaaf]

Dear netdev,

I've got a peculiar issue, and hope for clarification / workarounds here.

Scenario:
- a bridge interface br0, over some ethernet base
- a macvlan interface br0-vrrp on top, set up by keepalived, with VRRP VMAC
- keepalived regularly sending gratuitous ARP with that VRRP VMAC
- (new) an additional tap interface in br0, for an openvpn link

In principle, everything is working fine. The base keepalived setup
has been in operation for a long time, running directly over a VLAN
interface. The conversion to a bridge interface is also working
without any issues by itself. The additional tap to openvpn, and the
VPN setup it realizes, is also working fine, in principle...

Issue:
- openvpn runs at 100% CPU ....
- because it it sent all packets destined to the VRRP VMAC
- because that VMAC is not in the br0 learned MAC address table (brctl showmacs)
- thus the (production webserver outbound...) traffic is flooded to
all br0 ports

Diagnosis I did so far:
- with tcpdump, verified that I can see the gratuitous ARPs on both the macvlan
  and bridge interface.
- verified that "brctl showmacs" does not contain the VRRP VMAC
- identical setup for a different VLAN with almost no traffic to the
VMAC, has openvpn running without the huge CPU consumption
- straced the openvpn daemon with the issue, seeing the packet rate
expected as tap reads / sends to the remote site

Kernel:
3.14.48 (vanilla)
keepalived 1.2.13 (with repeated gratuitous ARP support patched in)

Can anybody shed a light on how to cope with this issue?

best regards
  Patrick

Re: bridge not learning from locally sent gratuitous ARP?

[Toshiaki Makita]

On 2016/04/21 15:37, Patrick Schaaf wrote:
> Dear netdev,

Hi,

> 
> I've got a peculiar issue, and hope for clarification / workarounds here.
> 
> Scenario:
> - a bridge interface br0, over some ethernet base
> - a macvlan interface br0-vrrp on top, set up by keepalived, with VRRP VMAC
> - keepalived regularly sending gratuitous ARP with that VRRP VMAC
> - (new) an additional tap interface in br0, for an openvpn link
> 
> In principle, everything is working fine. The base keepalived setup
> has been in operation for a long time, running directly over a VLAN
> interface. The conversion to a bridge interface is also working
> without any issues by itself. The additional tap to openvpn, and the
> VPN setup it realizes, is also working fine, in principle...
> 
> Issue:
> - openvpn runs at 100% CPU ....
> - because it it sent all packets destined to the VRRP VMAC
> - because that VMAC is not in the br0 learned MAC address table (brctl showmacs)
> - thus the (production webserver outbound...) traffic is flooded to
> all br0 ports
> 
> Diagnosis I did so far:
> - with tcpdump, verified that I can see the gratuitous ARPs on both the macvlan
>   and bridge interface.
> - verified that "brctl showmacs" does not contain the VRRP VMAC
> - identical setup for a different VLAN with almost no traffic to the
> VMAC, has openvpn running without the huge CPU consumption
> - straced the openvpn daemon with the issue, seeing the packet rate
> expected as tap reads / sends to the remote site
> 
> Kernel:
> 3.14.48 (vanilla)
> keepalived 1.2.13 (with repeated gratuitous ARP support patched in)
> 
> Can anybody shed a light on how to cope with this issue?

(I understand the problem happens only if you use macvlan on the bridge
device. If wrong, correct me.)

Please use "bridge fdb show" instead of "brctl showmacs". brctl is an
old tool and has limitation in showing fdb entries.

I guess your bridge doesn't have the mac address of the macvlan device
in its fdb. If you can confirm that by bridge command, then add an entry
by hand.

# bridge fdb add <addr of macvlan> dev <brport> master
<brport> can be any device that belongs to the bridge.
This command (without "temp" option) installs an fdb entry that forwards
frames to the bridge device.


Some more analysis:
- bridge adds the fdb entry of br0 mac address permanently. bridge never
learns source mac addresses of frames sent from br0.
- bridge seems not to have an ability to propagate the mac address of
its upper device into fdb.
- vlan device has the same mac address as its real device by default, so
it does not cause the problem.

-- 
Toshiaki Makita

Re: bridge not learning from locally sent gratuitous ARP?

[Patrick Schaaf]

On Thu, Apr 21, 2016 at 12:31 PM, Toshiaki Makita
<makita.toshiaki@lab.ntt.co.jp> wrote:
> On 2016/04/21 15:37, Patrick Schaaf wrote:
> (I understand the problem happens only if you use macvlan on the bridge device. If wrong, correct me.)

That is my understanding, yes. That macvlan device is created by
keepalived, which I use for VRRP, via its use_vmac + vmac_xmit_base
settings.

> Please use "bridge fdb show" instead of "brctl showmacs". brctl is an
> old tool and has limitation in showing fdb entries.

Learn a new thing every day! Thanks, that's good to know.

> I guess your bridge doesn't have the mac address of the macvlan device
> in its fdb. If you can confirm that by bridge command, then add an entry
> by hand.
>
> # bridge fdb add <addr of macvlan> dev <brport> master
> <brport> can be any device that belongs to the bridge.
> This command (without "temp" option) installs an fdb entry that forwards
> frames to the bridge device.

The "bridge" command operates solely on bridge ports (not on the
bridge itself), as far as I can see. Checking the member interfaces,
indeed I cannot find an entry for the macvlan interface VMAC.

The macvlan interface itself is NOT a brport.

So I add that permanent entry to the underlying ethernet link of that bridge.

Seems to work - I can still ping the virtual IP address behind that
VMAC, so the macvlan interface continues to do what it should - and
running tcpdump on the tap brport to openvpn, I see these test pings
when the fdb entry has not been set, but I no longer see them after
the bridge fdb add.

So, assuming that proives stable, I have a workaround - but one that
needs knowlege of A) the unterlying interface (brport) to fondle, and
B) the macvlan MAC address that needs adding. keepalived couldn't
easily do that "in code" because it does not know what sits under the
bridge.

Naive question: shouldn't that work automatically, i.e. macvlan when
sitting on to of a bridge, doing that internally?

Anyway, thanks for the pointers to the workaround, much appreciated!

best regards
  Patrick

Re: bridge not learning from locally sent gratuitous ARP?

[Toshiaki Makita]

> So, assuming that proives stable, I have a workaround - but one that
> needs knowlege of A) the unterlying interface (brport) to fondle, and
> B) the macvlan MAC address that needs adding. keepalived couldn't
> easily do that "in code" because it does not know what sits under the
> bridge.
> 
> Naive question: shouldn't that work automatically, i.e. macvlan when
> sitting on to of a bridge, doing that internally?

I'll think of this, thank you.

Toshiaki Makita