All of lore.kernel.org
 help / color / mirror / Atom feed
* [Bridge] linux bridge does not forward arp reply back packets in a vmware vm
@ 2017-12-15 13:37 Adrian P
  2017-12-15 15:55 ` Stephen Hemminger
  0 siblings, 1 reply; 15+ messages in thread
From: Adrian P @ 2017-12-15 13:37 UTC (permalink / raw)
  To: bridge

Hello,

I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.

From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.

Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:

# brctl show
bridge name     bridge id               STP enabled     interfaces
brq025a9a94-58          8000.005056a6b378       no              ens160
                                                        tap2eb4cad6-cd
   <----- neutron DHCP agent tap interface
                                                        tap6d31a191-9f
   <----- cirros VM instance tap interface

The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.

The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.

The problem is the following:

With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.

To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.

I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.

Any thoughts?

Many thanks in advance.

Best regards,
Adrian

^ permalink raw reply	[flat|nested] 15+ messages in thread
* [Bridge] linux bridge does not forward arp reply back packets in a vmware vm
@ 2019-08-09  5:56 Ben Shaw
  0 siblings, 0 replies; 15+ messages in thread
From: Ben Shaw @ 2019-08-09  5:56 UTC (permalink / raw)
  To: bridge

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

Hi,

there was a post on here with the same title in 2017 which seemed to
discuss the same issue I was seeing today on my ESXi environment with a
bridged Ubuntu VM. A host would not receive ARP replies because the Ubunut
VM between it and the ARP responder would drop the replies.

My issues seemed to reflect the previous poster's issue identically and
after some testing I can see that what is happening is that the ARP request
was being forwarded by the Ubuntu VM and then by the ESXi vswitch out one
of the two physical uplinks it had to the physical switched environment.
Because ESXi vswitches don't participate in STP hence do not block ports
that ARP request would be switched back to the same vSwitch via the second
uplink port and be forwarded back to the Ubuntu bridged VM.

This would cause the Ubuntu VM to see the host sending the ARP request to
be off the other interface so when the ARP reply was received the Ubuntu
host would drop the reply and not forward out the interface on the bridge
the intended recipient was actually on. This could be see in the brctl
showmacs output with the MAC address in question appearing on the wrong
bridge member port as shown below where the first entry should actually be
off port 2.

LAB-SOH01:~$ brctl showmacs br0 | grep no
port no mac addr                is local?       ageing timer
  1     00:0c:29:e7:e4:37       no                12.58
  1     00:a0:c9:0f:02:01       no                 2.00
LAB-SOH01:~$

A work around so far is to disable one of the physical uplinks to the
physical network. I will have to think if there is a better solution which
maintains redundancy. Hopefully this may be of help to some others who see
the same issue.

Thanks

[-- Attachment #2: Type: text/html, Size: 1955 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread
* Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm
@ 2019-10-02 12:00 Andreas Triller
  0 siblings, 0 replies; 15+ messages in thread
From: Andreas Triller @ 2019-10-02 12:00 UTC (permalink / raw)
  To: bridge@lists.linux-foundation.org, adrian27oradea@gmail.com

Dear Adrian,

Thanks a lot for posting this to the mailing list. I had almost the exact same setup and hit the same problem. Your post lead me to the solution.
In my case I use 2 Linux bridges connected with a VxLAN tunnel which then goes through an IPSec VPN over the internet. (Meant to stretch some VLANs on Layer 2 over a standard ISP line.)

[ESXi with productive servers and Linux bridge]--[VxLAN<->IPSec]--(Internet)--[IPSec<->VxLAN]--[Linux Bridge]--[Physical switch in remote office]

I also saw the ARP request 3 times in a capture, which as you wrote explains why the bridge fails in standard mode. I could also make it work by setting the setageing parameter to 0.
In the end I am quite sure I know what caused the multiplied ARP requests, in case you did not find out yourself.
My Linux bridges are VMs inside 2 different ESXi hosts. The tap device of the bridge was connected to a portgroup on a vswitch which allowed promiscuous mode and had VLAN 4095 assigned, which means "all VLANs".
I was quite surprised when I sniffed the traffic coming out of the far side bridge, since it contained complete IP conversations between other VMs living on the same ESXi host as the near end bridge! Not only broadcasts where transmitted, but everything the vswitch handled.
To mitigate, I moved the tap device of the Linux bridges to a new vswitch with only one port group (also VLAN 4095), connected to the physical switch by a dedicated uplink. This lets the physical switch filter all traffic that is not meant to enter the tunnel. This also stopped the multiplied ARP requests, so I could revert the bridges back to the normal mode with MAC ageing. I guess the reason for the multiplied ARP requests was the existence of the other port groups in the same vswitch as the tap device, maybe in combination with promiscuous mode.

Thanks a lot again for your input, you stopped me scratching my head.

Best regards

Andreas Triller

On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau
<https://lists.linuxfoundation.org/mailman/listinfo/bridge> wrote:
> On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
> <https://lists.linuxfoundation.org/mailman/listinfo/bridge> wrote:
>> On 2017/12/17 5:01, Adrian P wrote:
>> ...
>>> Further investigation reveals something strange: when the
>>> communication starts with an arp request (which happens almost all the
>>> time), the bridge wrongly assigns the eth0 mac address to port 1,
>>> instead of port 3.
>>>
>>> Flow again:
>>>
>>> default gw --- vmware --- [ ens160 bridge tap ] --- eth0
>>>
>>> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
>>> mac address is fa:16:3e:9a:04:95
>>>
>>> What I have found is that in the forwarding table, the bridge wrongly
>>> assigns the eth0 mac address to the port 1, which is ens160 interface,
>>> instead of assigning it to the port 3, which is the tap interface.
>>> This happens only if the arp table in the cirros VM instance does not
>>> contain the mac address of the destination I am pinging (default gw in
>>> this case), so the cirros VM sends an arp request. See below the eth0
>>> mac address wrongly  assigned in the forwarding table to the port 1:
>>>
>>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>>>   1     fa:16:3e:9a:04:95       no                 0.67
>>>
>>> However, if I manually add the mac address of the destination IP I am
>>> pining into the cirros VM instance arp table, and there is no arp
>>> request sent, just icmp packets going out, then the bridge correctly
>>> assigns the eth0 mac address to the port 3, which is the tap
>>> interface, and everything starts working fine. See below the eth0 mac
>>> address correctly assigned in the forwarding table to the port 3:


^ permalink raw reply	[flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-10-02 12:00 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-12-15 13:37 [Bridge] linux bridge does not forward arp reply back packets in a vmware vm Adrian P
2017-12-15 15:55 ` Stephen Hemminger
2017-12-15 16:29   ` Adrian P
2017-12-16  1:47     ` Stephen Hemminger
2017-12-16  7:12       ` Adrian P
2017-12-16 14:19         ` Adrian P
2017-12-16 17:35           ` Stephen Hemminger
2017-12-16 18:14             ` Adrian P
2017-12-16 20:01               ` Adrian P
2017-12-18  2:54                 ` Toshiaki Makita
2017-12-18  8:05                   ` Adrian Pascalau
2017-12-19 19:18                     ` Adrian Pascalau
2017-12-20  5:04                       ` Toshiaki Makita
  -- strict thread matches above, loose matches on Subject: below --
2019-08-09  5:56 Ben Shaw
2019-10-02 12:00 Andreas Triller

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.