[Bridge] [PROBLEM] mac address learned on wrong port

[Bridge] [PROBLEM] mac address learned on wrong port

[Seweryn Niemiec]

Hi,

On one of my servers, the most basic thing doesn't work and after 2 days of 
unsuccessful digging in my severs and the Internet, I'm desperate enough to 
write to this list :)

Ubuntu server with kernel 3.19.0-33-generic #38~14.04.1-Ubuntu has p2p1.17 
physical interface attached to bri17 bridge. When I start some LXC container, it 
creates for ex. vethPQORXD interface and adds it to bri17 bridge. It looks like 
this:

# brctl show bri17
bridge name     bridge id               STP enabled     interfaces
bri17           8000.00163eaaaa30       no              p2p1.17
                                                         vethPQORXD

LXC container's eth0 interface (linked to vethPQORXD) has mac 00:16:3e:aa:aa:66. 
When I add IP do this eth0 and ping something to generate eth frame, that mac is 
learned on p2p1.17 port instead of vethPQORXD:

# brctl showmacs bri17
port no mac addr                is local?       ageing timer
   1     00:16:3e:aa:aa:66       no               112.50
[... some other macs ...]
   1     3c:fd:fe:01:ee:e0       yes                0.00
   1     3c:fd:fe:01:ee:e0       yes                0.00
   2     fe:56:74:59:a4:9e       yes                0.00
   2     fe:56:74:59:a4:9e       yes                0.00

Of course it makes communication of outside world with that container impossible 
(host<->container works).

Any ideas why this happens?

-- 
Best regards,
Seweryn Niemiec

Re: [Bridge] [PROBLEM] mac address learned on wrong port

[Stephen Hemminger]

[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]

This usualy happens when you either have a loop or packet leaks in from
other port.


On Tue, Apr 26, 2016 at 6:38 AM, Seweryn Niemiec <ser@man.szczecin.pl>
wrote:

> Hi,
>
> On one of my servers, the most basic thing doesn't work and after 2 days
> of unsuccessful digging in my severs and the Internet, I'm desperate enough
> to write to this list :)
>
> Ubuntu server with kernel 3.19.0-33-generic #38~14.04.1-Ubuntu has p2p1.17
> physical interface attached to bri17 bridge. When I start some LXC
> container, it creates for ex. vethPQORXD interface and adds it to bri17
> bridge. It looks like this:
>
> # brctl show bri17
> bridge name     bridge id               STP enabled     interfaces
> bri17           8000.00163eaaaa30       no              p2p1.17
>                                                         vethPQORXD
>
> LXC container's eth0 interface (linked to vethPQORXD) has mac
> 00:16:3e:aa:aa:66. When I add IP do this eth0 and ping something to
> generate eth frame, that mac is learned on p2p1.17 port instead of
> vethPQORXD:
>
> # brctl showmacs bri17
> port no mac addr                is local?       ageing timer
>   1     00:16:3e:aa:aa:66       no               112.50
> [... some other macs ...]
>   1     3c:fd:fe:01:ee:e0       yes                0.00
>   1     3c:fd:fe:01:ee:e0       yes                0.00
>   2     fe:56:74:59:a4:9e       yes                0.00
>   2     fe:56:74:59:a4:9e       yes                0.00
>
> Of course it makes communication of outside world with that container
> impossible (host<->container works).
>
> Any ideas why this happens?
>
> --
> Best regards,
> Seweryn Niemiec
>
>

[-- Attachment #2: Type: text/html, Size: 2263 bytes --]

Re: [Bridge] [PROBLEM] mac address learned on wrong port

[Seweryn Niemiec]

On 26.04.2016 18:49, Stephen Hemminger wrote:
> This usualy happens when you either have a loop or packet leaks in from other port.

Yeah, it looks like a loop, but p2p1 is connected to a production network with 
running MSTP, so I was sure there is no loop, but I have double checked it today 
and I'm still sure there is no loop in the net outside of the server. To make it 
super sure I have connected p2p1 to a dead end port (vlan with this one port 
only) on a switch and the effect was the same - container's mac is learned on p2p1.

Could you explain what you mean by packet leaking in from other port?

I have one more clue:

when I do:

# arping -c1 ip-of-gateway

inside the container, then on host I can observe:

tcpdump -en -i vethV05BAQ ether src host 00:16:3e:aa:aa:66

14:57:17.528848 00:16:3e:aa:aa:66 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), 
length 42: Request who-has 212.14.1.33 tell 212.14.1.42, length 28
14:57:17.528915 00:16:3e:aa:aa:66 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), 
length 60: Request who-has 212.14.1.33 tell 212.14.1.42, length 46

Two packets of same content, but different sizes. Difference is a padding of eth 
frame to 64 bytes.

Does it mean something?

-- 
Best regards,
Seweryn Niemiec

Re: [Bridge] [PROBLEM] mac address learned on wrong port

[Stephen Hemminger]

On Wed, 27 Apr 2016 15:17:01 +0200
Seweryn Niemiec <ser@man.szczecin.pl> wrote:

> On 26.04.2016 18:49, Stephen Hemminger wrote:
> > This usualy happens when you either have a loop or packet leaks in from other port.
> 
> Yeah, it looks like a loop, but p2p1 is connected to a production network with 
> running MSTP, so I was sure there is no loop, but I have double checked it today 
> and I'm still sure there is no loop in the net outside of the server. To make it 
> super sure I have connected p2p1 to a dead end port (vlan with this one port 
> only) on a switch and the effect was the same - container's mac is learned on p2p1.
> 

But you are not running Spanning Tree on your bridge.

Re: [Bridge] [PROBLEM] mac address learned on wrong port

[Seweryn Niemiec]

On 27.04.2016 21:37, Stephen Hemminger wrote:
> On Wed, 27 Apr 2016 15:17:01 +0200
> Seweryn Niemiec <ser@man.szczecin.pl> wrote:
>
>> On 26.04.2016 18:49, Stephen Hemminger wrote:
>>> This usualy happens when you either have a loop or packet leaks in from other port.
>>
>> Yeah, it looks like a loop, but p2p1 is connected to a production network with
>> running MSTP, so I was sure there is no loop, but I have double checked it today
>> and I'm still sure there is no loop in the net outside of the server. To make it
>> super sure I have connected p2p1 to a dead end port (vlan with this one port
>> only) on a switch and the effect was the same - container's mac is learned on p2p1.
>>
>
> But you are not running Spanning Tree on your bridge.

After two more days of tests :)

Now I'm running stp on my bridge too and it doesn't change anything. Booth ports 
are in forwarding state. This is freshly installed Ubuntu 14.04.3. There is only 
one bridge configured and two ports in the bridge. For me, it's impossible to 
have a loop there.

To make sure it's not a problem with the switch I have connected p2p1 port to 
powered on but unbootable server (just to keep link up) and the problem still 
exists.

The other test I have made was to remove p2p1 from the bridge and add some other 
physical port. The results was:
  - other port with same drive (p2p2) - problem exists
  - other port and other driver (em2) - problem does not exist

p2pX are Intel X710 with i40e driver
emX are Broadcom BCM5720 with tg3 driver

Any ideas where to look further?

-- 
Sew

Re: [Bridge] [PROBLEM] mac address learned on wrong port

[Seweryn Niemiec]

Hi

I've found a root of the problem. It is an unexpected "feature" of Intel's X710 
network adapter.
More here: https://communities.intel.com/thread/62234
-- 
Best regards,
Seweryn Niemiec