[Bridge] Packet reflection breaks Linux bridge

[Bridge] Packet reflection breaks Linux bridge

[Thomas Glanzmann]

Hello,
I want to use OpenVPN to extend a layer 2 segment over a routed
connection. I was not able to ping a system on the same subnet, so I
tried to reproduce it with a different network and was _not_ able to: It
worked out of the box. It turned out that I had some Exchange sytem that
sends every packet it gets (including broadcast packets) out again.
After 3 hours of debugging with Michael Gernoth (on CC), it turned out
that the bridge was thinking that the MAC address of the device was on
eth1 instead of 'vpn' and the bridge in Linux was not forwarding any
packet. We we're able to fight the symptoms by issuing the following
command:

ebtables -t nat -A POSTROUTING -d ff:ff:ff:ff:ff:ff -o eth1 -j snat --to-source 00:50:56:98:0a:22

With this command we can access all systems using Layer 2, but not the
two Exchange systems which replicate every packet they're seeing.

This is what the Linux box sends:

(miniwheezy64) [~] tcpdump -e -n -i vpn ether host 00:50:56:98:12:ed
tcpdump: WARNING: vpn: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn, link-type EN10MB (Ethernet), capture size 65535 bytes
09:35:35.402501 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 35, length 64
09:35:36.410454 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 36, length 64
09:35:37.418526 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 37, length 64
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

This is how it looks, after the Microsoft Exchange settings saw the
package: As you can see, the packets are duplicated at least once,
sometimes twice. This also happens with arp who-has and is-at packets
which confused the Linux bridge because it thinks that the mac is now at
another port:

(miniwheezy64) [~] tcpdump -e -n -i br0 ether host 00:50:56:98:12:ed
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:35:42.458571 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64
09:35:42.458693 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64
09:35:42.459019 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
09:35:42.459220 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
09:35:42.459340 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
09:35:43.466531 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64
09:35:43.466758 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64
09:35:43.467162 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
09:35:43.467169 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
09:35:43.467366 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
09:35:44.474551 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64
09:35:44.474774 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64
09:35:44.475108 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
09:35:44.475150 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
09:35:44.475160 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
^C
15 packets captured
19 packets received by filter
0 packets dropped by kernel
(miniwheezy64) [~] brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.005056980a22       no              eth1
                                                        vpn

(miniwheezy64) [~] brctl showmacs br0 | grep -i 00:50:56:98:12:ed
  1     00:50:56:98:12:ed       no                 0.08

As you can see the mac address of the system is learned on port 1 but
should be learned on port 2.

This is the setup:

Exchange and other systems - VLAN 103 - eth1 - br0 - vpn - OpenVPN - vpn - br0 - eth1 - virtual switch - client (00:50:56:98:12:ed)

It seems that the Linux bridge behaves different than a physical switch.
Because with a physical switch it still forwards the packet. But the
Linux bridge gets offbeat by the reflected arp who-has and is-at packets
and thinks that the mac has moved to a different port even that it
hasn't. So I wonder why does the linux bridge gets confused by that, but
the physical switch not?

I found one other reference which confirms this problem.

http://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply

Also I wonder what else than source natting broadcast request we could
do to fight the symptoms? For example I would like to drop all packets
with the source mac address from 00:50:56:98:12:ed which are comming in
via eth1. I tried this using ebtables, arptables and iptables but did
not get the syntax right. Has someone, something I could try?

I'm running Debian wheezy with Linux minisqueeze 3.2.0-4-686-pae #1 SMP
Debian 3.2.51-1 i686 GNU/Linux.

If I set the MAC adress of a system static on the system behind the
layer 2 bridge, the system gets connectivity. If I remove OpenVPN from
the equation and use a layer 2 gre tunnel or just another physical
network card, I have the same behaviour.

Cheers,
        Thomas

Re: [Bridge] Packet reflection breaks Linux bridge

[Stephen Hemminger]

On Sat, 30 Nov 2013 09:54:40 +0100
Thomas Glanzmann <thomas@glanzmann.de> wrote:

> Hello,
> I want to use OpenVPN to extend a layer 2 segment over a routed
> connection. I was not able to ping a system on the same subnet, so I
> tried to reproduce it with a different network and was _not_ able to: It
> worked out of the box. It turned out that I had some Exchange sytem that
> sends every packet it gets (including broadcast packets) out again.
> After 3 hours of debugging with Michael Gernoth (on CC), it turned out
> that the bridge was thinking that the MAC address of the device was on
> eth1 instead of 'vpn' and the bridge in Linux was not forwarding any
> packet. We we're able to fight the symptoms by issuing the following
> command:
> 
> ebtables -t nat -A POSTROUTING -d ff:ff:ff:ff:ff:ff -o eth1 -j snat --to-source 00:50:56:98:0a:22
> 
> With this command we can access all systems using Layer 2, but not the
> two Exchange systems which replicate every packet they're seeing.
> 
> This is what the Linux box sends:
> 
> (miniwheezy64) [~] tcpdump -e -n -i vpn ether host 00:50:56:98:12:ed
> tcpdump: WARNING: vpn: no IPv4 address assigned
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on vpn, link-type EN10MB (Ethernet), capture size 65535 bytes
> 09:35:35.402501 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 35, length 64
> 09:35:36.410454 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 36, length 64
> 09:35:37.418526 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 37, length 64
> ^C
> 3 packets captured
> 3 packets received by filter
> 0 packets dropped by kernel
> 
> This is how it looks, after the Microsoft Exchange settings saw the
> package: As you can see, the packets are duplicated at least once,
> sometimes twice. This also happens with arp who-has and is-at packets
> which confused the Linux bridge because it thinks that the mac is now at
> another port:
> 
> (miniwheezy64) [~] tcpdump -e -n -i br0 ether host 00:50:56:98:12:ed
> tcpdump: WARNING: br0: no IPv4 address assigned
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 09:35:42.458571 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64
> 09:35:42.458693 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64
> 09:35:42.459019 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
> 09:35:42.459220 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
> 09:35:42.459340 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64
> 09:35:43.466531 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64
> 09:35:43.466758 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64
> 09:35:43.467162 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
> 09:35:43.467169 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
> 09:35:43.467366 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64
> 09:35:44.474551 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64
> 09:35:44.474774 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64
> 09:35:44.475108 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
> 09:35:44.475150 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
> 09:35:44.475160 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64
> ^C
> 15 packets captured
> 19 packets received by filter
> 0 packets dropped by kernel
> (miniwheezy64) [~] brctl show
> bridge name     bridge id               STP enabled     interfaces
> br0             8000.005056980a22       no              eth1
>                                                         vpn
> 
> (miniwheezy64) [~] brctl showmacs br0 | grep -i 00:50:56:98:12:ed
>   1     00:50:56:98:12:ed       no                 0.08
> 
> As you can see the mac address of the system is learned on port 1 but
> should be learned on port 2.
> 
> This is the setup:
> 
> Exchange and other systems - VLAN 103 - eth1 - br0 - vpn - OpenVPN - vpn - br0 - eth1 - virtual switch - client (00:50:56:98:12:ed)
> 
> It seems that the Linux bridge behaves different than a physical switch.
> Because with a physical switch it still forwards the packet. But the
> Linux bridge gets offbeat by the reflected arp who-has and is-at packets
> and thinks that the mac has moved to a different port even that it
> hasn't. So I wonder why does the linux bridge gets confused by that, but
> the physical switch not?
> 
> I found one other reference which confirms this problem.
> 
> http://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply
> 
> Also I wonder what else than source natting broadcast request we could
> do to fight the symptoms? For example I would like to drop all packets
> with the source mac address from 00:50:56:98:12:ed which are comming in
> via eth1. I tried this using ebtables, arptables and iptables but did
> not get the syntax right. Has someone, something I could try?
> 
> I'm running Debian wheezy with Linux minisqueeze 3.2.0-4-686-pae #1 SMP
> Debian 3.2.51-1 i686 GNU/Linux.
> 
> If I set the MAC adress of a system static on the system behind the
> layer 2 bridge, the system gets connectivity. If I remove OpenVPN from
> the equation and use a layer 2 gre tunnel or just another physical
> network card, I have the same behaviour.
> 
> Cheers,
>         Thomas


Bridging doesn't like loops, and you have created a loop.
If you are goin to mess around using ebtables, just write another rule
to drop the reflections.

Re: [Bridge] Packet reflection breaks Linux bridge

[Thomas Glanzmann]

Hello Stephan,

> Bridging doesn't like loops, and you have created a loop.

I agree. It was actually a Microsoft Load Balancing cluster that created
the loop. Michael told me that the physical switch works because it:

        a) broadcasts
        b) sends it to both ports.

> If you are goin to mess around using ebtables, just write another rule
> to drop the reflections.

Michael said that ebtables only after the bridge has seen the packets
and so is not applicable. Is that true? So should I use arptables. I
tried to block the looped packages by:

        - iptables
        - arptables
        - eptables

But somehow I never made it. Do you have an example for me or can tell
me which of the 3 tools should work, so that I can try again. The NATing
fought the symptoms of my problem.

Cheers,
        Thomas

Re: [Bridge] Packet reflection breaks Linux bridge

[Stephen Hemminger]

On Thu, 5 Dec 2013 18:27:51 +0100
Thomas Glanzmann <thomas@glanzmann.de> wrote:

> Hello Stephan,
> 
> > Bridging doesn't like loops, and you have created a loop.
> 
> I agree. It was actually a Microsoft Load Balancing cluster that created
> the loop. Michael told me that the physical switch works because it:
> 
>         a) broadcasts
>         b) sends it to both ports.
> 
> > If you are goin to mess around using ebtables, just write another rule
> > to drop the reflections.
> 
> Michael said that ebtables only after the bridge has seen the packets
> and so is not applicable. Is that true? So should I use arptables. I
> tried to block the looped packages by:
> 
>         - iptables
>         - arptables
>         - eptables
> 
> But somehow I never made it. Do you have an example for me or can tell
> me which of the 3 tools should work, so that I can try again. The NATing
> fought the symptoms of my problem.
> 
> Cheers,
>         Thomas

No example.

You can also define static fdb entries with lastest kernel/iproute
and that would problably pin the entry.

Re: [Bridge] Packet reflection breaks Linux bridge

[Bart De Schuymer]

Thomas Glanzmann schreef op 30/11/2013 9:54:
> I found one other reference which confirms this problem.
>
> http://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply

The author answered his own question stating it was an issue in his 
network :-)

> Also I wonder what else than source natting broadcast request we could
> do to fight the symptoms? For example I would like to drop all packets
> with the source mac address from 00:50:56:98:12:ed which are comming in
> via eth1. I tried this using ebtables, arptables and iptables but did
> not get the syntax right. Has someone, something I could try?
>

ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP

Use PREROUTING so that the Linux bridge fdb isn't updated. A better 
solution is probably to fix your network.


cheers,
Bart

Re: [Bridge] Packet reflection breaks Linux bridge

[Thomas Glanzmann]

Hello Bart,

> > http://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply

> The author answered his own question stating it was an issue in his
> network :-)

that is true. 

> ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP
> Use PREROUTING so that the Linux bridge fdb isn't updated.

Thank you a lot for this line. I just tried it and it works:

(miniwheezy64) [~]           ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP
(miniwheezy64) [~] tcpdump -e -n -i vpn ether host 00:50:56:98:12:ed
tcpdump: WARNING: vpn: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn, link-type EN10MB (Ethernet), capture size 65535 bytes
05:27:59.786228 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 14490, seq 29, length 64
05:27:59.786768 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 14490, seq 29, length 64
05:27:59.786787 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 14490, seq 29, length 64
05:27:59.787041 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 14490, seq 29, length 64
...

As you can see the system is totally broken. It even sends its unicast packets
three times back. A wonder that anything works in this broadcast domain. And
the MAC is learned on the rigth port:

(miniwheezy64) [~] brctl showmacs br0 | grep -i 00:50:56:98:12:ed
  2     00:50:56:98:12:ed       no                 0.28

> A better solution is probably to fix your network.

I agree. The problem is that it is _not_ _my_ network. I identified two
Microsoft Exchange servers which are part of two different load
balancing clusters running on Hyper-V as the culprit. They do not only
reflect all broadcast packets that they see, but also the unicast
packets. So when I do a unicast connection to them using ICMP or TCP,
all packets get reflected. So what I'm looking for now is a workaround
to carry out the migration. Because of the feedback provided by Michael,
Stephen and you, now I have three:

        - SNAT the broadcast packets that come from the far side of the
          bridge, and SNAT the unicast packets going to the systems
          which do reflect all packages that they can see.

                ebtables -t nat -F
                ebtables -t nat -A POSTROUTING -d ff:ff:ff:ff:ff:ff -o eth1 -j snat --to-source 00:50:56:98:0a:22
                ebtables -t nat -A POSTROUTING -d 03:bf:0a:40:67:80 -o eth1 -j snat --to-source 00:50:56:98:0a:22
                ebtables -t nat -A POSTROUTING -d 03:bf:0a:40:67:2a -o eth1 -j snat --to-source 00:50:56:98:0a:22

        - Use the newest code and pin the mac address of the far side
          machines to the right interface.

        - Use the ebtables PREROUTING statement to filter the packets
          out before they reach the bridge:

          ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP

Far side I call systems behind the OpenVPN bridge on the virtual
network.

I agree that the above is fighting the symptoms and not resolving the
real problem. I don't like that. However I'm not allowed to reconfigure
the Exchange Microsoft Cluster, so I work with that. I have informed the
people reasonable for it and the network administrators and hope that
they take care of the problem.

Cheers,
        Thomas