[Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian P]

Hello,

I have a strange issue with a linux bridge created by
openstack-neutron (pike release). This linux bridge is hosted in a
vmware VM running latest CentOS 7, with a single network interface in
promiscuous mode.

From openstack neutron perspective, the networking setup is simple: a
single flat external provider network, with a single cirros VM
instance connected to it.

Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:

# brctl show
bridge name     bridge id               STP enabled     interfaces
brq025a9a94-58          8000.005056a6b378       no              ens160
                                                        tap2eb4cad6-cd
   <----- neutron DHCP agent tap interface
                                                        tap6d31a191-9f
   <----- cirros VM instance tap interface

The ens160 is the "physical" CentOS 7 host interface, that is in
promiscuous mode.

The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
and the tap6d31a191-9f tap interface is used by the cirros VM
instance.

The problem is the following:

With a tcpdump, I am able to see the arp request (ARP, Request who-has
10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
tap interface tap6d31a191-9f, and well as on the bridge itself
(brq025a9a94-58). However, the reply back to the arp request (Reply
10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
instance anymore. With tcpdump, I am able to see the arp reply back
packets in the bridge (brq025a9a94-58), however they do not show up
anymore on the cirros VM instance tap interface tap6d31a191-9f.

To me it seems that for whatever reason, the bridge does not forward
the arp reply back packets to the cirros VM tap interface, and I do
not understand why. The strange thing is that after a while, for
apparently no reason, a single arp reply back packet gets through the
bridge and the tap interface, and the arp table gets updated with
correct IP address in the cirros VM instance. However, if I clean up
the arp table in the cirros VM instance, it takes again 10 to 15
minutes of continuously sending arp requests, until a single arp reply
back packets gets through.

I was banging my head to the table for a few days with this issue, and
finally, for apparent no reason, I manually configured the bridge max
aging time to 0, to convert it in a hub, and from that moment
everything started to work without any issue. Still, I do no
understand why is this happening, and obviously I cannot manually set
up the bridge aging time to 0 all the time in all the bridges
openstack neutron creates automatically.

Any thoughts?

Many thanks in advance.

Best regards,
Adrian

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Stephen Hemminger]

On Fri, 15 Dec 2017 15:37:39 +0200
Adrian P <adrian27oradea@gmail.com> wrote:

> Hello,
> 
> I have a strange issue with a linux bridge created by
> openstack-neutron (pike release). This linux bridge is hosted in a
> vmware VM running latest CentOS 7, with a single network interface in
> promiscuous mode.
> 
> From openstack neutron perspective, the networking setup is simple: a
> single flat external provider network, with a single cirros VM
> instance connected to it.
> 
> Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:
> 
> # brctl show
> bridge name     bridge id               STP enabled     interfaces
> brq025a9a94-58          8000.005056a6b378       no              ens160
>                                                         tap2eb4cad6-cd
>    <----- neutron DHCP agent tap interface
>                                                         tap6d31a191-9f
>    <----- cirros VM instance tap interface
> 
> The ens160 is the "physical" CentOS 7 host interface, that is in
> promiscuous mode.
> 
> The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
> and the tap6d31a191-9f tap interface is used by the cirros VM
> instance.
> 
> The problem is the following:
> 
> With a tcpdump, I am able to see the arp request (ARP, Request who-has
> 10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
> tap interface tap6d31a191-9f, and well as on the bridge itself
> (brq025a9a94-58). However, the reply back to the arp request (Reply
> 10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
> instance anymore. With tcpdump, I am able to see the arp reply back
> packets in the bridge (brq025a9a94-58), however they do not show up
> anymore on the cirros VM instance tap interface tap6d31a191-9f.
> 
> To me it seems that for whatever reason, the bridge does not forward
> the arp reply back packets to the cirros VM tap interface, and I do
> not understand why. The strange thing is that after a while, for
> apparently no reason, a single arp reply back packet gets through the
> bridge and the tap interface, and the arp table gets updated with
> correct IP address in the cirros VM instance. However, if I clean up
> the arp table in the cirros VM instance, it takes again 10 to 15
> minutes of continuously sending arp requests, until a single arp reply
> back packets gets through.
> 
> I was banging my head to the table for a few days with this issue, and
> finally, for apparent no reason, I manually configured the bridge max
> aging time to 0, to convert it in a hub, and from that moment
> everything started to work without any issue. Still, I do no
> understand why is this happening, and obviously I cannot manually set
> up the bridge aging time to 0 all the time in all the bridges
> openstack neutron creates automatically.
> 
> Any thoughts?
> 
> Many thanks in advance.
> 
> Best regards,
> Adrian

Does each tap instance and the ens160 have a different and valid Ethernet
address?  Also make sure the these are in the bridge forwarding table.

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian P]

On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Fri, 15 Dec 2017 15:37:39 +0200
> Adrian P <adrian27oradea@gmail.com> wrote:
>
>> Hello,
>>
>> I have a strange issue with a linux bridge created by
>> openstack-neutron (pike release). This linux bridge is hosted in a
>> vmware VM running latest CentOS 7, with a single network interface in
>> promiscuous mode.
>>
>> From openstack neutron perspective, the networking setup is simple: a
>> single flat external provider network, with a single cirros VM
>> instance connected to it.
>>
>> Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:
>>
>> # brctl show
>> bridge name     bridge id               STP enabled     interfaces
>> brq025a9a94-58          8000.005056a6b378       no              ens160
>>                                                         tap2eb4cad6-cd
>>    <----- neutron DHCP agent tap interface
>>                                                         tap6d31a191-9f
>>    <----- cirros VM instance tap interface
>>
>> The ens160 is the "physical" CentOS 7 host interface, that is in
>> promiscuous mode.
>>
>> The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
>> and the tap6d31a191-9f tap interface is used by the cirros VM
>> instance.
>>
>> The problem is the following:
>>
>> With a tcpdump, I am able to see the arp request (ARP, Request who-has
>> 10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
>> tap interface tap6d31a191-9f, and well as on the bridge itself
>> (brq025a9a94-58). However, the reply back to the arp request (Reply
>> 10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
>> instance anymore. With tcpdump, I am able to see the arp reply back
>> packets in the bridge (brq025a9a94-58), however they do not show up
>> anymore on the cirros VM instance tap interface tap6d31a191-9f.
>>
>> To me it seems that for whatever reason, the bridge does not forward
>> the arp reply back packets to the cirros VM tap interface, and I do
>> not understand why. The strange thing is that after a while, for
>> apparently no reason, a single arp reply back packet gets through the
>> bridge and the tap interface, and the arp table gets updated with
>> correct IP address in the cirros VM instance. However, if I clean up
>> the arp table in the cirros VM instance, it takes again 10 to 15
>> minutes of continuously sending arp requests, until a single arp reply
>> back packets gets through.
>>
>> I was banging my head to the table for a few days with this issue, and
>> finally, for apparent no reason, I manually configured the bridge max
>> aging time to 0, to convert it in a hub, and from that moment
>> everything started to work without any issue. Still, I do no
>> understand why is this happening, and obviously I cannot manually set
>> up the bridge aging time to 0 all the time in all the bridges
>> openstack neutron creates automatically.
>>
>> Any thoughts?
>>
>> Many thanks in advance.
>>
>> Best regards,
>> Adrian
>
> Does each tap instance and the ens160 have a different and valid Ethernet
> address?  Also make sure the these are in the bridge forwarding table.

Yes, they have valid Ethernet addresses, and they do show up in the
forwarding table twice, see below:

# ip addr
<...>
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
brq025a9a94-58 state UP qlen 1000
    link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:fea6:b378/64 scope link
       valid_lft forever preferred_lft forever
4: tap2eb4cad6-cd@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
qdisc noqueue master brq025a9a94-58 state UP qlen 1000
    link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
    link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
    inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
       valid_lft forever preferred_lft forever
    inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
       valid_lft forever preferred_lft forever
6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
    link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fe9a:495/64 scope link
       valid_lft forever preferred_lft forever

# brctl showmacs brq025a9a94-58
port no mac addr                is local?       ageing timer
  1     00:50:56:a6:b3:78       yes                0.00
  1     00:50:56:a6:b3:78       yes                0.00
  2     8a:b2:15:4c:96:55       yes                0.00
  2     8a:b2:15:4c:96:55       yes                0.00
  3     fe:16:3e:9a:04:95       yes                0.00
  3     fe:16:3e:9a:04:95       yes                0.00

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Stephen Hemminger]

On Fri, 15 Dec 2017 18:29:58 +0200
Adrian P <adrian27oradea@gmail.com> wrote:

> On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> > On Fri, 15 Dec 2017 15:37:39 +0200
> > Adrian P <adrian27oradea@gmail.com> wrote:
> >  
> >> Hello,
> >>
> >> I have a strange issue with a linux bridge created by
> >> openstack-neutron (pike release). This linux bridge is hosted in a
> >> vmware VM running latest CentOS 7, with a single network interface in
> >> promiscuous mode.
> >>
> >> From openstack neutron perspective, the networking setup is simple: a
> >> single flat external provider network, with a single cirros VM
> >> instance connected to it.
> >>
> >> Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:
> >>
> >> # brctl show
> >> bridge name     bridge id               STP enabled     interfaces
> >> brq025a9a94-58          8000.005056a6b378       no              ens160
> >>                                                         tap2eb4cad6-cd
> >>    <----- neutron DHCP agent tap interface
> >>                                                         tap6d31a191-9f
> >>    <----- cirros VM instance tap interface
> >>
> >> The ens160 is the "physical" CentOS 7 host interface, that is in
> >> promiscuous mode.
> >>
> >> The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
> >> and the tap6d31a191-9f tap interface is used by the cirros VM
> >> instance.
> >>
> >> The problem is the following:
> >>
> >> With a tcpdump, I am able to see the arp request (ARP, Request who-has
> >> 10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
> >> tap interface tap6d31a191-9f, and well as on the bridge itself
> >> (brq025a9a94-58). However, the reply back to the arp request (Reply
> >> 10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
> >> instance anymore. With tcpdump, I am able to see the arp reply back
> >> packets in the bridge (brq025a9a94-58), however they do not show up
> >> anymore on the cirros VM instance tap interface tap6d31a191-9f.
> >>
> >> To me it seems that for whatever reason, the bridge does not forward
> >> the arp reply back packets to the cirros VM tap interface, and I do
> >> not understand why. The strange thing is that after a while, for
> >> apparently no reason, a single arp reply back packet gets through the
> >> bridge and the tap interface, and the arp table gets updated with
> >> correct IP address in the cirros VM instance. However, if I clean up
> >> the arp table in the cirros VM instance, it takes again 10 to 15
> >> minutes of continuously sending arp requests, until a single arp reply
> >> back packets gets through.
> >>
> >> I was banging my head to the table for a few days with this issue, and
> >> finally, for apparent no reason, I manually configured the bridge max
> >> aging time to 0, to convert it in a hub, and from that moment
> >> everything started to work without any issue. Still, I do no
> >> understand why is this happening, and obviously I cannot manually set
> >> up the bridge aging time to 0 all the time in all the bridges
> >> openstack neutron creates automatically.
> >>
> >> Any thoughts?
> >>
> >> Many thanks in advance.
> >>
> >> Best regards,
> >> Adrian  
> >
> > Does each tap instance and the ens160 have a different and valid Ethernet
> > address?  Also make sure the these are in the bridge forwarding table.  
> 
> Yes, they have valid Ethernet addresses, and they do show up in the
> forwarding table twice, see below:
> 
> # ip addr
> <...>
> 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
> brq025a9a94-58 state UP qlen 1000
>     link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::250:56ff:fea6:b378/64 scope link
>        valid_lft forever preferred_lft forever
> 4: tap2eb4cad6-cd@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
> qdisc noqueue master brq025a9a94-58 state UP qlen 1000
>     link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
> 5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP qlen 1000
>     link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
>     inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
>        valid_lft forever preferred_lft forever
>     inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
>        valid_lft forever preferred_lft forever
> 6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
>     link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::fc16:3eff:fe9a:495/64 scope link
>        valid_lft forever preferred_lft forever
> 
> # brctl showmacs brq025a9a94-58
> port no mac addr                is local?       ageing timer
>   1     00:50:56:a6:b3:78       yes                0.00
>   1     00:50:56:a6:b3:78       yes                0.00
>   2     8a:b2:15:4c:96:55       yes                0.00
>   2     8a:b2:15:4c:96:55       yes                0.00
>   3     fe:16:3e:9a:04:95       yes                0.00
>   3     fe:16:3e:9a:04:95       yes                0.00


Since there are multiple entries per port maybe you are also using VLANs?

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian P]

On Sat, Dec 16, 2017 at 3:47 AM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Fri, 15 Dec 2017 18:29:58 +0200
> Adrian P <adrian27oradea@gmail.com> wrote:
>
>> On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
>> <stephen@networkplumber.org> wrote:
>> > On Fri, 15 Dec 2017 15:37:39 +0200
>> > Adrian P <adrian27oradea@gmail.com> wrote:
>> >
>> >> Hello,
>> >>
>> >> I have a strange issue with a linux bridge created by
>> >> openstack-neutron (pike release). This linux bridge is hosted in a
>> >> vmware VM running latest CentOS 7, with a single network interface in
>> >> promiscuous mode.
>> >>
>> >> From openstack neutron perspective, the networking setup is simple: a
>> >> single flat external provider network, with a single cirros VM
>> >> instance connected to it.
>> >>
>> >> Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:
>> >>
>> >> # brctl show
>> >> bridge name     bridge id               STP enabled     interfaces
>> >> brq025a9a94-58          8000.005056a6b378       no              ens160
>> >>                                                         tap2eb4cad6-cd
>> >>    <----- neutron DHCP agent tap interface
>> >>                                                         tap6d31a191-9f
>> >>    <----- cirros VM instance tap interface
>> >>
>> >> The ens160 is the "physical" CentOS 7 host interface, that is in
>> >> promiscuous mode.
>> >>
>> >> The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
>> >> and the tap6d31a191-9f tap interface is used by the cirros VM
>> >> instance.
>> >>
>> >> The problem is the following:
>> >>
>> >> With a tcpdump, I am able to see the arp request (ARP, Request who-has
>> >> 10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
>> >> tap interface tap6d31a191-9f, and well as on the bridge itself
>> >> (brq025a9a94-58). However, the reply back to the arp request (Reply
>> >> 10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
>> >> instance anymore. With tcpdump, I am able to see the arp reply back
>> >> packets in the bridge (brq025a9a94-58), however they do not show up
>> >> anymore on the cirros VM instance tap interface tap6d31a191-9f.
>> >>
>> >> To me it seems that for whatever reason, the bridge does not forward
>> >> the arp reply back packets to the cirros VM tap interface, and I do
>> >> not understand why. The strange thing is that after a while, for
>> >> apparently no reason, a single arp reply back packet gets through the
>> >> bridge and the tap interface, and the arp table gets updated with
>> >> correct IP address in the cirros VM instance. However, if I clean up
>> >> the arp table in the cirros VM instance, it takes again 10 to 15
>> >> minutes of continuously sending arp requests, until a single arp reply
>> >> back packets gets through.
>> >>
>> >> I was banging my head to the table for a few days with this issue, and
>> >> finally, for apparent no reason, I manually configured the bridge max
>> >> aging time to 0, to convert it in a hub, and from that moment
>> >> everything started to work without any issue. Still, I do no
>> >> understand why is this happening, and obviously I cannot manually set
>> >> up the bridge aging time to 0 all the time in all the bridges
>> >> openstack neutron creates automatically.
>> >>
>> >> Any thoughts?
>> >>
>> >> Many thanks in advance.
>> >>
>> >> Best regards,
>> >> Adrian
>> >
>> > Does each tap instance and the ens160 have a different and valid Ethernet
>> > address?  Also make sure the these are in the bridge forwarding table.
>>
>> Yes, they have valid Ethernet addresses, and they do show up in the
>> forwarding table twice, see below:
>>
>> # ip addr
>> <...>
>> 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
>> brq025a9a94-58 state UP qlen 1000
>>     link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
>>     inet6 fe80::250:56ff:fea6:b378/64 scope link
>>        valid_lft forever preferred_lft forever
>> 4: tap2eb4cad6-cd@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>> qdisc noqueue master brq025a9a94-58 state UP qlen 1000
>>     link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>> 5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> noqueue state UP qlen 1000
>>     link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
>>     inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
>>        valid_lft forever preferred_lft forever
>>     inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
>>        valid_lft forever preferred_lft forever
>> 6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>> pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
>>     link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
>>     inet6 fe80::fc16:3eff:fe9a:495/64 scope link
>>        valid_lft forever preferred_lft forever
>>
>> # brctl showmacs brq025a9a94-58
>> port no mac addr                is local?       ageing timer
>>   1     00:50:56:a6:b3:78       yes                0.00
>>   1     00:50:56:a6:b3:78       yes                0.00
>>   2     8a:b2:15:4c:96:55       yes                0.00
>>   2     8a:b2:15:4c:96:55       yes                0.00
>>   3     fe:16:3e:9a:04:95       yes                0.00
>>   3     fe:16:3e:9a:04:95       yes                0.00
>
>
> Since there are multiple entries per port maybe you are also using VLANs?

So I have checked one more time: the host is a vmware centos7 VM, with
a single interface (ens160 that can be seen above) connected to a
vmware virtual switch port group configured in promiscuous mode, that
removes the VLAN tagging (similar with access mode port in a physical
switch).

I have another environment that behaves the same, where the host is a
physical server, and the interface is a bonded interface, with two
slaves, and I also have two entries for each local interface in the
forwarding table:

# ip addr
<...>
6: eno5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond2 portid 01000000000000000000004c4930394833 state UP qlen
1000
    link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
7: eno6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master bond2 portid 02000000000000000000004c4930394833 state UP qlen
1000
    link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
<...>
10: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
noqueue master brq75a55ef7-4a state UP qlen 1000
    link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fe15:b4ff:fe13:e6a3/64 scope link
       valid_lft forever preferred_lft forever
<...>
14: brq75a55ef7-4a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP qlen 1000
    link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
    inet 10.20.21.55/24 brd 10.20.21.255 scope global brq75a55ef7-4a
       valid_lft forever preferred_lft forever
15: tap44bc34bb-e2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast master brq75a55ef7-4a state UNKNOWN qlen 1000
    link/ether fe:16:3e:cc:dc:ec brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc16:3eff:fecc:dcec/64 scope link
       valid_lft forever preferred_lft forever

# brctl show
bridge name     bridge id               STP enabled     interfaces
brq75a55ef7-4a          8000.fc15b413e6a3       no              bond2
                                                        tap44bc34bb-e2

# brctl showmacs brq75a55ef7-4a | grep yes
  2     fc:15:b4:13:e6:a3       yes                0.00
  2     fc:15:b4:13:e6:a3       yes                0.00
  1     fe:16:3e:cc:dc:ec       yes                0.00
  1     fe:16:3e:cc:dc:ec       yes                0.00

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian P]

Investigating this further, I have noticed that the mac address of the
eth0 interface from the cirros VM instance does not appear in the
bridge forwarding table, and this explains why everything starts
working only when I set ageing time to 0, because in that case all
packets are flooded on all ports and the bridge behaves like a hub.

So now the question is: why the bridge does not learn the mac address
of the eth0 interface from the cirros VM instance? I am able to see
the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
going out from the cirros VM instance on tap interface, so the bridge
should learn the mac address and add it to the forwarding table.

The reply back to the arp request (Reply 10.20.21.1 is-at
00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
now I know why: there is no mac address in the forwarding table, so
the bridge does not know on which port to send the arp reply back.

This happens with tap interfaces only. I can see many mac addresses
associated with "physical" interface ens160 (that is interface number
1) in the forwarding table, but in case of the tap interfaces, there
are only two entries, and both entries shows the mac address of the
tap interfaces only:

# brctl showmacs brq025a9a94-58
port no mac addr                is local?       ageing timer
<...>
  1     00:50:56:a6:f8:ba       no                13.66
  1     00:60:16:6d:b3:f8       no                85.01
  1     00:60:16:6d:dd:b8       no               239.39
  2     52:db:7c:ac:4b:25       yes                0.00
  2     52:db:7c:ac:4b:25       yes                0.00
<...>
  1     fc:15:b4:13:b9:f0       no                 0.21
  1     fc:15:b4:13:b9:f1       no                19.40
  1     fc:15:b4:13:b9:f5       no                17.40
  3     fe:16:3e:9a:04:95       yes                0.00
  3     fe:16:3e:9a:04:95       yes                0.00


On Sat, Dec 16, 2017 at 9:12 AM, Adrian P <adrian27oradea@gmail.com> wrote:
> On Sat, Dec 16, 2017 at 3:47 AM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>> On Fri, 15 Dec 2017 18:29:58 +0200
>> Adrian P <adrian27oradea@gmail.com> wrote:
>>
>>> On Fri, Dec 15, 2017 at 5:55 PM, Stephen Hemminger
>>> <stephen@networkplumber.org> wrote:
>>> > On Fri, 15 Dec 2017 15:37:39 +0200
>>> > Adrian P <adrian27oradea@gmail.com> wrote:
>>> >
>>> >> Hello,
>>> >>
>>> >> I have a strange issue with a linux bridge created by
>>> >> openstack-neutron (pike release). This linux bridge is hosted in a
>>> >> vmware VM running latest CentOS 7, with a single network interface in
>>> >> promiscuous mode.
>>> >>
>>> >> From openstack neutron perspective, the networking setup is simple: a
>>> >> single flat external provider network, with a single cirros VM
>>> >> instance connected to it.
>>> >>
>>> >> Therefore, in the linux bridge running in the vmware host, I have 3 interfaces:
>>> >>
>>> >> # brctl show
>>> >> bridge name     bridge id               STP enabled     interfaces
>>> >> brq025a9a94-58          8000.005056a6b378       no              ens160
>>> >>                                                         tap2eb4cad6-cd
>>> >>    <----- neutron DHCP agent tap interface
>>> >>                                                         tap6d31a191-9f
>>> >>    <----- cirros VM instance tap interface
>>> >>
>>> >> The ens160 is the "physical" CentOS 7 host interface, that is in
>>> >> promiscuous mode.
>>> >>
>>> >> The  tap2eb4cad6-cd tap interface is the neutron DHCP agent interface,
>>> >> and the tap6d31a191-9f tap interface is used by the cirros VM
>>> >> instance.
>>> >>
>>> >> The problem is the following:
>>> >>
>>> >> With a tcpdump, I am able to see the arp request (ARP, Request who-has
>>> >> 10.20.21.1 tell 10.20.21.233) going out from the cirros VM instance on
>>> >> tap interface tap6d31a191-9f, and well as on the bridge itself
>>> >> (brq025a9a94-58). However, the reply back to the arp request (Reply
>>> >> 10.20.21.1 is-at 00:17:08:c4:52:80) does not reach the cirros VM
>>> >> instance anymore. With tcpdump, I am able to see the arp reply back
>>> >> packets in the bridge (brq025a9a94-58), however they do not show up
>>> >> anymore on the cirros VM instance tap interface tap6d31a191-9f.
>>> >>
>>> >> To me it seems that for whatever reason, the bridge does not forward
>>> >> the arp reply back packets to the cirros VM tap interface, and I do
>>> >> not understand why. The strange thing is that after a while, for
>>> >> apparently no reason, a single arp reply back packet gets through the
>>> >> bridge and the tap interface, and the arp table gets updated with
>>> >> correct IP address in the cirros VM instance. However, if I clean up
>>> >> the arp table in the cirros VM instance, it takes again 10 to 15
>>> >> minutes of continuously sending arp requests, until a single arp reply
>>> >> back packets gets through.
>>> >>
>>> >> I was banging my head to the table for a few days with this issue, and
>>> >> finally, for apparent no reason, I manually configured the bridge max
>>> >> aging time to 0, to convert it in a hub, and from that moment
>>> >> everything started to work without any issue. Still, I do no
>>> >> understand why is this happening, and obviously I cannot manually set
>>> >> up the bridge aging time to 0 all the time in all the bridges
>>> >> openstack neutron creates automatically.
>>> >>
>>> >> Any thoughts?
>>> >>
>>> >> Many thanks in advance.
>>> >>
>>> >> Best regards,
>>> >> Adrian
>>> >
>>> > Does each tap instance and the ens160 have a different and valid Ethernet
>>> > address?  Also make sure the these are in the bridge forwarding table.
>>>
>>> Yes, they have valid Ethernet addresses, and they do show up in the
>>> forwarding table twice, see below:
>>>
>>> # ip addr
>>> <...>
>>> 2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master
>>> brq025a9a94-58 state UP qlen 1000
>>>     link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
>>>     inet6 fe80::250:56ff:fea6:b378/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 4: tap2eb4cad6-cd@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
>>> qdisc noqueue master brq025a9a94-58 state UP qlen 1000
>>>     link/ether 8a:b2:15:4c:96:55 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>>> 5: brq025a9a94-58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>> noqueue state UP qlen 1000
>>>     link/ether 00:50:56:a6:b3:78 brd ff:ff:ff:ff:ff:ff
>>>     inet 10.20.21.249/24 brd 10.20.21.255 scope global brq025a9a94-58
>>>        valid_lft forever preferred_lft forever
>>>     inet6 fe80::803d:d0ff:fe2e:3ae4/64 scope link
>>>        valid_lft forever preferred_lft forever
>>> 6: tap6d31a191-9f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
>>> pfifo_fast master brq025a9a94-58 state UNKNOWN qlen 1000
>>>     link/ether fe:16:3e:9a:04:95 brd ff:ff:ff:ff:ff:ff
>>>     inet6 fe80::fc16:3eff:fe9a:495/64 scope link
>>>        valid_lft forever preferred_lft forever
>>>
>>> # brctl showmacs brq025a9a94-58
>>> port no mac addr                is local?       ageing timer
>>>   1     00:50:56:a6:b3:78       yes                0.00
>>>   1     00:50:56:a6:b3:78       yes                0.00
>>>   2     8a:b2:15:4c:96:55       yes                0.00
>>>   2     8a:b2:15:4c:96:55       yes                0.00
>>>   3     fe:16:3e:9a:04:95       yes                0.00
>>>   3     fe:16:3e:9a:04:95       yes                0.00
>>
>>
>> Since there are multiple entries per port maybe you are also using VLANs?
>
> So I have checked one more time: the host is a vmware centos7 VM, with
> a single interface (ens160 that can be seen above) connected to a
> vmware virtual switch port group configured in promiscuous mode, that
> removes the VLAN tagging (similar with access mode port in a physical
> switch).
>
> I have another environment that behaves the same, where the host is a
> physical server, and the interface is a bonded interface, with two
> slaves, and I also have two entries for each local interface in the
> forwarding table:
>
> # ip addr
> <...>
> 6: eno5: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond2 portid 01000000000000000000004c4930394833 state UP qlen
> 1000
>     link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
> 7: eno6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
> master bond2 portid 02000000000000000000004c4930394833 state UP qlen
> 1000
>     link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
> <...>
> 10: bond2: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc
> noqueue master brq75a55ef7-4a state UP qlen 1000
>     link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::fe15:b4ff:fe13:e6a3/64 scope link
>        valid_lft forever preferred_lft forever
> <...>
> 14: brq75a55ef7-4a: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> noqueue state UP qlen 1000
>     link/ether fc:15:b4:13:e6:a3 brd ff:ff:ff:ff:ff:ff
>     inet 10.20.21.55/24 brd 10.20.21.255 scope global brq75a55ef7-4a
>        valid_lft forever preferred_lft forever
> 15: tap44bc34bb-e2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
> pfifo_fast master brq75a55ef7-4a state UNKNOWN qlen 1000
>     link/ether fe:16:3e:cc:dc:ec brd ff:ff:ff:ff:ff:ff
>     inet6 fe80::fc16:3eff:fecc:dcec/64 scope link
>        valid_lft forever preferred_lft forever
>
> # brctl show
> bridge name     bridge id               STP enabled     interfaces
> brq75a55ef7-4a          8000.fc15b413e6a3       no              bond2
>                                                         tap44bc34bb-e2
>
> # brctl showmacs brq75a55ef7-4a | grep yes
>   2     fc:15:b4:13:e6:a3       yes                0.00
>   2     fc:15:b4:13:e6:a3       yes                0.00
>   1     fe:16:3e:cc:dc:ec       yes                0.00
>   1     fe:16:3e:cc:dc:ec       yes                0.00

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Stephen Hemminger]

On Sat, 16 Dec 2017 16:19:03 +0200
Adrian P <adrian27oradea@gmail.com> wrote:

> Investigating this further, I have noticed that the mac address of the
> eth0 interface from the cirros VM instance does not appear in the
> bridge forwarding table, and this explains why everything starts
> working only when I set ageing time to 0, because in that case all
> packets are flooded on all ports and the bridge behaves like a hub.
> 
> So now the question is: why the bridge does not learn the mac address
> of the eth0 interface from the cirros VM instance? I am able to see
> the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
> going out from the cirros VM instance on tap interface, so the bridge
> should learn the mac address and add it to the forwarding table.
> 
> The reply back to the arp request (Reply 10.20.21.1 is-at
> 00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
> now I know why: there is no mac address in the forwarding table, so
> the bridge does not know on which port to send the arp reply back.
> 
> This happens with tap interfaces only. I can see many mac addresses
> associated with "physical" interface ens160 (that is interface number
> 1) in the forwarding table, but in case of the tap interfaces, there
> are only two entries, and both entries shows the mac address of the
> tap interfaces only:

VMWare does ARP spoofing maybe it consumes the ARP

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian P]

On Sat, Dec 16, 2017 at 7:35 PM, Stephen Hemminger
<stephen@networkplumber.org> wrote:
> On Sat, 16 Dec 2017 16:19:03 +0200
> Adrian P <adrian27oradea@gmail.com> wrote:
>
>> Investigating this further, I have noticed that the mac address of the
>> eth0 interface from the cirros VM instance does not appear in the
>> bridge forwarding table, and this explains why everything starts
>> working only when I set ageing time to 0, because in that case all
>> packets are flooded on all ports and the bridge behaves like a hub.
>>
>> So now the question is: why the bridge does not learn the mac address
>> of the eth0 interface from the cirros VM instance? I am able to see
>> the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
>> going out from the cirros VM instance on tap interface, so the bridge
>> should learn the mac address and add it to the forwarding table.
>>
>> The reply back to the arp request (Reply 10.20.21.1 is-at
>> 00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
>> now I know why: there is no mac address in the forwarding table, so
>> the bridge does not know on which port to send the arp reply back.
>>
>> This happens with tap interfaces only. I can see many mac addresses
>> associated with "physical" interface ens160 (that is interface number
>> 1) in the forwarding table, but in case of the tap interfaces, there
>> are only two entries, and both entries shows the mac address of the
>> tap interfaces only:
>
> VMWare does ARP spoofing maybe it consumes the ARP

The flow is like this:

vmware <--> ens160 <--> bridge <--> tap if <--> eth0

So my understanding is that when the arp request goes out on the eth0,
through the tap interface, the bridge should learn the eth0 mac
address and add it to the forwarding table. But this does not happen,
so when the arp reply back comes back, the bridge does not know on
which interface to send the arp reply, and this is why nothing works.

I have another bare metal environment, without vmware, and I still
have this issue.

I suspect this is a bug related to the bridge and the tap interface,
since I see many mac address learned by the bridge on the ens160
interface... or maybe some tap / bridge setting? I have no clue. Where
should I report such issue?

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian P]

On Sat, Dec 16, 2017 at 8:14 PM, Adrian P <adrian27oradea@gmail.com> wrote:
> On Sat, Dec 16, 2017 at 7:35 PM, Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>> On Sat, 16 Dec 2017 16:19:03 +0200
>> Adrian P <adrian27oradea@gmail.com> wrote:
>>
>>> Investigating this further, I have noticed that the mac address of the
>>> eth0 interface from the cirros VM instance does not appear in the
>>> bridge forwarding table, and this explains why everything starts
>>> working only when I set ageing time to 0, because in that case all
>>> packets are flooded on all ports and the bridge behaves like a hub.
>>>
>>> So now the question is: why the bridge does not learn the mac address
>>> of the eth0 interface from the cirros VM instance? I am able to see
>>> the arp request (ARP, Request who-has 10.20.21.1 tell 10.20.21.233)
>>> going out from the cirros VM instance on tap interface, so the bridge
>>> should learn the mac address and add it to the forwarding table.
>>>
>>> The reply back to the arp request (Reply 10.20.21.1 is-at
>>> 00:17:08:c4:52:80) does not reach the cirros VM instance anymore, and
>>> now I know why: there is no mac address in the forwarding table, so
>>> the bridge does not know on which port to send the arp reply back.
>>>
>>> This happens with tap interfaces only. I can see many mac addresses
>>> associated with "physical" interface ens160 (that is interface number
>>> 1) in the forwarding table, but in case of the tap interfaces, there
>>> are only two entries, and both entries shows the mac address of the
>>> tap interfaces only:
>>
>> VMWare does ARP spoofing maybe it consumes the ARP
>
> The flow is like this:
>
> vmware <--> ens160 <--> bridge <--> tap if <--> eth0
>
> So my understanding is that when the arp request goes out on the eth0,
> through the tap interface, the bridge should learn the eth0 mac
> address and add it to the forwarding table. But this does not happen,
> so when the arp reply back comes back, the bridge does not know on
> which interface to send the arp reply, and this is why nothing works.
>
> I have another bare metal environment, without vmware, and I still
> have this issue.
>
> I suspect this is a bug related to the bridge and the tap interface,
> since I see many mac address learned by the bridge on the ens160
> interface... or maybe some tap / bridge setting? I have no clue. Where
> should I report such issue?

Further investigation reveals something strange: when the
communication starts with an arp request (which happens almost all the
time), the bridge wrongly assigns the eth0 mac address to port 1,
instead of port 3.

Flow again:

default gw --- vmware --- [ ens160 bridge tap ] --- eth0

On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
mac address is fa:16:3e:9a:04:95

What I have found is that in the forwarding table, the bridge wrongly
assigns the eth0 mac address to the port 1, which is ens160 interface,
instead of assigning it to the port 3, which is the tap interface.
This happens only if the arp table in the cirros VM instance does not
contain the mac address of the destination I am pinging (default gw in
this case), so the cirros VM sends an arp request. See below the eth0
mac address wrongly  assigned in the forwarding table to the port 1:

# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
  1     fa:16:3e:9a:04:95       no                 0.67

However, if I manually add the mac address of the destination IP I am
pining into the cirros VM instance arp table, and there is no arp
request sent, just icmp packets going out, then the bridge correctly
assigns the eth0 mac address to the port 3, which is the tap
interface, and everything starts working fine. See below the eth0 mac
address correctly assigned in the forwarding table to the port 3:

# brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
  3     fa:16:3e:9a:04:95       no                 9.26

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Toshiaki Makita]

On 2017/12/17 5:01, Adrian P wrote:
...
> Further investigation reveals something strange: when the
> communication starts with an arp request (which happens almost all the
> time), the bridge wrongly assigns the eth0 mac address to port 1,
> instead of port 3.
> 
> Flow again:
> 
> default gw --- vmware --- [ ens160 bridge tap ] --- eth0
> 
> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
> mac address is fa:16:3e:9a:04:95
> 
> What I have found is that in the forwarding table, the bridge wrongly
> assigns the eth0 mac address to the port 1, which is ens160 interface,
> instead of assigning it to the port 3, which is the tap interface.
> This happens only if the arp table in the cirros VM instance does not
> contain the mac address of the destination I am pinging (default gw in
> this case), so the cirros VM sends an arp request. See below the eth0
> mac address wrongly  assigned in the forwarding table to the port 1:
> 
> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>   1     fa:16:3e:9a:04:95       no                 0.67
> 
> However, if I manually add the mac address of the destination IP I am
> pining into the cirros VM instance arp table, and there is no arp
> request sent, just icmp packets going out, then the bridge correctly
> assigns the eth0 mac address to the port 3, which is the tap
> interface, and everything starts working fine. See below the eth0 mac
> address correctly assigned in the forwarding table to the port 3:
> 
> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>   3     fa:16:3e:9a:04:95       no                 9.26
> 

It looks like the broadcast arp request is coming back from gw or
vmware. Would you check it?

-- 
Toshiaki Makita

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian Pascalau]

On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
<makita.toshiaki@lab.ntt.co.jp> wrote:
> On 2017/12/17 5:01, Adrian P wrote:
> ...
>> Further investigation reveals something strange: when the
>> communication starts with an arp request (which happens almost all the
>> time), the bridge wrongly assigns the eth0 mac address to port 1,
>> instead of port 3.
>>
>> Flow again:
>>
>> default gw --- vmware --- [ ens160 bridge tap ] --- eth0
>>
>> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
>> mac address is fa:16:3e:9a:04:95
>>
>> What I have found is that in the forwarding table, the bridge wrongly
>> assigns the eth0 mac address to the port 1, which is ens160 interface,
>> instead of assigning it to the port 3, which is the tap interface.
>> This happens only if the arp table in the cirros VM instance does not
>> contain the mac address of the destination I am pinging (default gw in
>> this case), so the cirros VM sends an arp request. See below the eth0
>> mac address wrongly  assigned in the forwarding table to the port 1:
>>
>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>>   1     fa:16:3e:9a:04:95       no                 0.67
>>
>> However, if I manually add the mac address of the destination IP I am
>> pining into the cirros VM instance arp table, and there is no arp
>> request sent, just icmp packets going out, then the bridge correctly
>> assigns the eth0 mac address to the port 3, which is the tap
>> interface, and everything starts working fine. See below the eth0 mac
>> address correctly assigned in the forwarding table to the port 3:
>>
>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>>   3     fa:16:3e:9a:04:95       no                 9.26
>>
>
> It looks like the broadcast arp request is coming back from gw or
> vmware. Would you check it?

So if I look into the flow below, once more:

default gw --- physical switch ---  vmware --- [ ens160 bridge tap ] --- eth0

The arp request goes out on the eth0 interface, and enters the bridge
on the tap interface. The bridge assigns the eth0 mac address to the
tap interface, and sends the arp request out on the ens160 interface.
Now some deice on the left side of the bridge (either de vmware,
physical switch or the default gw), broadcasts that arp requests back,
therefore the same arp request enters back the bridge on the ens160
interface, and the bridge assigns the source mac address of the arp
request (which is still the eth0 mac address) to the ens160 port in
the forwarding table, which causes the behavior I have noticed...

This explains why I see multiple times the arp request when using
tcpdump. By why there are three? Maybe two devices broadcasts back the
arp request?

# tcpdump -n -i ens160 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28
01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46

What is strange in the above tcpdump is that the first arp request is
28 bytes in length (I suppose this is the one that enters the bridge
on the tap interface), and the 2 others is 46 bytes in length (I
suppose these are the ones that enters the bridge on the ens160
interface). If I open the capture in wireshark, I see some padding
added in the last two arp requests, which does not appear in the first
arp request:

Ethernet II, Src: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95), Dst:
Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
    Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95)
    Type: ARP (0x0806)
    Padding: 000000000000000000000000000000000000

Any thoughts how I can find out who duplicates the two additional arps?

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Adrian Pascalau]

On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau
<adrian27oradea@gmail.com> wrote:
> On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
> <makita.toshiaki@lab.ntt.co.jp> wrote:
>> On 2017/12/17 5:01, Adrian P wrote:
>> ...
>>> Further investigation reveals something strange: when the
>>> communication starts with an arp request (which happens almost all the
>>> time), the bridge wrongly assigns the eth0 mac address to port 1,
>>> instead of port 3.
>>>
>>> Flow again:
>>>
>>> default gw --- vmware --- [ ens160 bridge tap ] --- eth0
>>>
>>> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
>>> mac address is fa:16:3e:9a:04:95
>>>
>>> What I have found is that in the forwarding table, the bridge wrongly
>>> assigns the eth0 mac address to the port 1, which is ens160 interface,
>>> instead of assigning it to the port 3, which is the tap interface.
>>> This happens only if the arp table in the cirros VM instance does not
>>> contain the mac address of the destination I am pinging (default gw in
>>> this case), so the cirros VM sends an arp request. See below the eth0
>>> mac address wrongly  assigned in the forwarding table to the port 1:
>>>
>>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>>>   1     fa:16:3e:9a:04:95       no                 0.67
>>>
>>> However, if I manually add the mac address of the destination IP I am
>>> pining into the cirros VM instance arp table, and there is no arp
>>> request sent, just icmp packets going out, then the bridge correctly
>>> assigns the eth0 mac address to the port 3, which is the tap
>>> interface, and everything starts working fine. See below the eth0 mac
>>> address correctly assigned in the forwarding table to the port 3:
>>>
>>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>>>   3     fa:16:3e:9a:04:95       no                 9.26
>>>
>>
>> It looks like the broadcast arp request is coming back from gw or
>> vmware. Would you check it?
>
> So if I look into the flow below, once more:
>
> default gw --- physical switch ---  vmware --- [ ens160 bridge tap ] --- eth0
>
> The arp request goes out on the eth0 interface, and enters the bridge
> on the tap interface. The bridge assigns the eth0 mac address to the
> tap interface, and sends the arp request out on the ens160 interface.
> Now some deice on the left side of the bridge (either de vmware,
> physical switch or the default gw), broadcasts that arp requests back,
> therefore the same arp request enters back the bridge on the ens160
> interface, and the bridge assigns the source mac address of the arp
> request (which is still the eth0 mac address) to the ens160 port in
> the forwarding table, which causes the behavior I have noticed...
>
> This explains why I see multiple times the arp request when using
> tcpdump. By why there are three? Maybe two devices broadcasts back the
> arp request?
>
> # tcpdump -n -i ens160 arp
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
> listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
> 01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28
> 01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
> 01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
> 01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46
>
> What is strange in the above tcpdump is that the first arp request is
> 28 bytes in length (I suppose this is the one that enters the bridge
> on the tap interface), and the 2 others is 46 bytes in length (I
> suppose these are the ones that enters the bridge on the ens160
> interface). If I open the capture in wireshark, I see some padding
> added in the last two arp requests, which does not appear in the first
> arp request:
>
> Ethernet II, Src: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95), Dst:
> Broadcast (ff:ff:ff:ff:ff:ff)
>     Destination: Broadcast (ff:ff:ff:ff:ff:ff)
>     Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95)
>     Type: ARP (0x0806)
>     Padding: 000000000000000000000000000000000000
>
> Any thoughts how I can find out who duplicates the two additional arps?

Is there a way to trace the packet as it passes through the bridge?
Something like a debug mode?

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Toshiaki Makita]

On 2017/12/20 4:18, Adrian Pascalau wrote:
> On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau
> <adrian27oradea@gmail.com> wrote:
>> On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
>> <makita.toshiaki@lab.ntt.co.jp> wrote:
>>> It looks like the broadcast arp request is coming back from gw or
>>> vmware. Would you check it?
>>
>> So if I look into the flow below, once more:
>>
>> default gw --- physical switch ---  vmware --- [ ens160 bridge tap ] --- eth0
>>
>> The arp request goes out on the eth0 interface, and enters the bridge
>> on the tap interface. The bridge assigns the eth0 mac address to the
>> tap interface, and sends the arp request out on the ens160 interface.
>> Now some deice on the left side of the bridge (either de vmware,
>> physical switch or the default gw), broadcasts that arp requests back,
>> therefore the same arp request enters back the bridge on the ens160
>> interface, and the bridge assigns the source mac address of the arp
>> request (which is still the eth0 mac address) to the ens160 port in
>> the forwarding table, which causes the behavior I have noticed...
>>
>> This explains why I see multiple times the arp request when using
>> tcpdump. By why there are three? Maybe two devices broadcasts back the
>> arp request?
>>
>> # tcpdump -n -i ens160 arp
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
>> 01:38:29.008219 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 28
>> 01:38:29.008619 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
>> 01:38:29.008732 ARP, Request who-has 10.20.21.1 tell 10.20.21.233, length 46
>> 01:38:29.009000 ARP, Reply 10.20.21.1 is-at 00:17:08:c4:52:80, length 46
>>
>> What is strange in the above tcpdump is that the first arp request is
>> 28 bytes in length (I suppose this is the one that enters the bridge
>> on the tap interface), and the 2 others is 46 bytes in length (I
>> suppose these are the ones that enters the bridge on the ens160
>> interface). If I open the capture in wireshark, I see some padding
>> added in the last two arp requests, which does not appear in the first
>> arp request:
>>
>> Ethernet II, Src: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95), Dst:
>> Broadcast (ff:ff:ff:ff:ff:ff)
>>     Destination: Broadcast (ff:ff:ff:ff:ff:ff)
>>     Source: fa:16:3e:9a:04:95 (fa:16:3e:9a:04:95)
>>     Type: ARP (0x0806)
>>     Padding: 000000000000000000000000000000000000
>>
>> Any thoughts how I can find out who duplicates the two additional arps?
> 
> Is there a way to trace the packet as it passes through the bridge?
> Something like a debug mode?

FWIW you can use "bridge monitor" command to monitor fdb update.
tcpdump has --direction option so you can only capture either of "out"
packets or "in" packets.

-- 
Toshiaki Makita

[Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Ben Shaw]

[-- Attachment #1: Type: text/plain, Size: 1711 bytes --]

Hi,

there was a post on here with the same title in 2017 which seemed to
discuss the same issue I was seeing today on my ESXi environment with a
bridged Ubuntu VM. A host would not receive ARP replies because the Ubunut
VM between it and the ARP responder would drop the replies.

My issues seemed to reflect the previous poster's issue identically and
after some testing I can see that what is happening is that the ARP request
was being forwarded by the Ubuntu VM and then by the ESXi vswitch out one
of the two physical uplinks it had to the physical switched environment.
Because ESXi vswitches don't participate in STP hence do not block ports
that ARP request would be switched back to the same vSwitch via the second
uplink port and be forwarded back to the Ubuntu bridged VM.

This would cause the Ubuntu VM to see the host sending the ARP request to
be off the other interface so when the ARP reply was received the Ubuntu
host would drop the reply and not forward out the interface on the bridge
the intended recipient was actually on. This could be see in the brctl
showmacs output with the MAC address in question appearing on the wrong
bridge member port as shown below where the first entry should actually be
off port 2.

LAB-SOH01:~$ brctl showmacs br0 | grep no
port no mac addr                is local?       ageing timer
  1     00:0c:29:e7:e4:37       no                12.58
  1     00:a0:c9:0f:02:01       no                 2.00
LAB-SOH01:~$

A work around so far is to disable one of the physical uplinks to the
physical network. I will have to think if there is a better solution which
maintains redundancy. Hopefully this may be of help to some others who see
the same issue.

Thanks

[-- Attachment #2: Type: text/html, Size: 1955 bytes --]

Re: [Bridge] linux bridge does not forward arp reply back packets in a vmware vm

[Andreas Triller]

Dear Adrian,

Thanks a lot for posting this to the mailing list. I had almost the exact same setup and hit the same problem. Your post lead me to the solution.
In my case I use 2 Linux bridges connected with a VxLAN tunnel which then goes through an IPSec VPN over the internet. (Meant to stretch some VLANs on Layer 2 over a standard ISP line.)

[ESXi with productive servers and Linux bridge]--[VxLAN<->IPSec]--(Internet)--[IPSec<->VxLAN]--[Linux Bridge]--[Physical switch in remote office]

I also saw the ARP request 3 times in a capture, which as you wrote explains why the bridge fails in standard mode. I could also make it work by setting the setageing parameter to 0.
In the end I am quite sure I know what caused the multiplied ARP requests, in case you did not find out yourself.
My Linux bridges are VMs inside 2 different ESXi hosts. The tap device of the bridge was connected to a portgroup on a vswitch which allowed promiscuous mode and had VLAN 4095 assigned, which means "all VLANs".
I was quite surprised when I sniffed the traffic coming out of the far side bridge, since it contained complete IP conversations between other VMs living on the same ESXi host as the near end bridge! Not only broadcasts where transmitted, but everything the vswitch handled.
To mitigate, I moved the tap device of the Linux bridges to a new vswitch with only one port group (also VLAN 4095), connected to the physical switch by a dedicated uplink. This lets the physical switch filter all traffic that is not meant to enter the tunnel. This also stopped the multiplied ARP requests, so I could revert the bridges back to the normal mode with MAC ageing. I guess the reason for the multiplied ARP requests was the existence of the other port groups in the same vswitch as the tap device, maybe in combination with promiscuous mode.

Thanks a lot again for your input, you stopped me scratching my head.

Best regards

Andreas Triller

On Mon, Dec 18, 2017 at 10:05 AM, Adrian Pascalau
<https://lists.linuxfoundation.org/mailman/listinfo/bridge> wrote:
> On Mon, Dec 18, 2017 at 4:54 AM, Toshiaki Makita
> <https://lists.linuxfoundation.org/mailman/listinfo/bridge> wrote:
>> On 2017/12/17 5:01, Adrian P wrote:
>> ...
>>> Further investigation reveals something strange: when the
>>> communication starts with an arp request (which happens almost all the
>>> time), the bridge wrongly assigns the eth0 mac address to port 1,
>>> instead of port 3.
>>>
>>> Flow again:
>>>
>>> default gw --- vmware --- [ ens160 bridge tap ] --- eth0
>>>
>>> On my bridge, ens160 is port 1, and the tap interface is port 3. Eth0
>>> mac address is fa:16:3e:9a:04:95
>>>
>>> What I have found is that in the forwarding table, the bridge wrongly
>>> assigns the eth0 mac address to the port 1, which is ens160 interface,
>>> instead of assigning it to the port 3, which is the tap interface.
>>> This happens only if the arp table in the cirros VM instance does not
>>> contain the mac address of the destination I am pinging (default gw in
>>> this case), so the cirros VM sends an arp request. See below the eth0
>>> mac address wrongly  assigned in the forwarding table to the port 1:
>>>
>>> # brctl showmacs brq025a9a94-58 | grep fa:16:3e:9a:04:95
>>>   1     fa:16:3e:9a:04:95       no                 0.67
>>>
>>> However, if I manually add the mac address of the destination IP I am
>>> pining into the cirros VM instance arp table, and there is no arp
>>> request sent, just icmp packets going out, then the bridge correctly
>>> assigns the eth0 mac address to the port 3, which is the tap
>>> interface, and everything starts working fine. See below the eth0 mac
>>> address correctly assigned in the forwarding table to the port 3: