[Bridge] bridge, vlan and *no* stp/bpdu

[Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Hello list,

I've posted here about this before, but I realise that it may have been 
assumed that the bridged vlans simply put a switch port in a blocking 
state and left my question ignored.  So to recap.

I have two tg3 interfaces named 'in' and 'out' and a bridge named 'br0'

My vlan trunk is on the 'in' side of the network, and set as in.2, in.3 
...  The 'out' side goes straight to an ipv4 gateway on untagged plain 
ethernet.

Putting 'in.2' and 'out' on the bridge works quite well and is roughly 
what I've been using so far.

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.00e081342870       no              out
                                                        in.2

If I add in.3 to the bridge, trouble starts.  The bridge keeps 
forwarding packets just like it should, with the exception of ARP 
replies from the gateway to machines in vlan 2.  Machine that had ARPed 
the gateway prior to adding in.3 to the bridge keep working fine.

Here's the strange thing however.  Running a tcpdump on 'out' 'br0' or 
in.2 shows me the arp requests *and replies* for the machines that do 
not work, however, if I look on the wire leaving the 'in' interface 
itself (using a hub and another box), the arp replies are nowhere to be 
found.

So the arp replies get eaten *before* they make it onto the wire, but 
*after* tcpdump saw them on in.2.  It's driving me nuts...  I thought it 
might have to do with the tg3 hardware doing some funky vlan 
acceleration, but I've seen the same on plain dumb NICs too.

I'm willing to pay for a solution to this...  Or even for just someone 
knowledgeable enough with the code taking interest in the issue.

Thanks,

Jonathan

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Andy Gospodarek]

On Fri, Mar 7, 2008 at 2:47 PM, Jonathan Thibault <jonathan@navigue.com> wrote:
> Hello list,
>
>  I've posted here about this before, but I realise that it may have been
>  assumed that the bridged vlans simply put a switch port in a blocking
>  state and left my question ignored.  So to recap.

It might have been ignored because you didn't tell us what kernel were
using.  Things change so fast that a bug 2.6.23 might be fixed in
2.6.24 -- someone can easily tell you that if they know the version
info.

>  I have two tg3 interfaces named 'in' and 'out' and a bridge named 'br0'
>
>  My vlan trunk is on the 'in' side of the network, and set as in.2, in.3
>  ...  The 'out' side goes straight to an ipv4 gateway on untagged plain
>  ethernet.
>
>  Putting 'in.2' and 'out' on the bridge works quite well and is roughly
>  what I've been using so far.
>
>  # brctl show
>  bridge name     bridge id               STP enabled     interfaces
>  br0             8000.00e081342870       no              out
>                                                         in.2
>
>  If I add in.3 to the bridge, trouble starts.  The bridge keeps
>  forwarding packets just like it should, with the exception of ARP
>  replies from the gateway to machines in vlan 2.  Machine that had ARPed
>  the gateway prior to adding in.3 to the bridge keep working fine.
>

Are the ARPs from the 'different' machines on in.3 and in.2 using the
same MAC address (are they even the same machine but with different
VLANs)?

>  Here's the strange thing however.  Running a tcpdump on 'out' 'br0' or
>  in.2 shows me the arp requests *and replies* for the machines that do
>  not work, however, if I look on the wire leaving the 'in' interface
>  itself (using a hub and another box), the arp replies are nowhere to be
>  found.
>
>  So the arp replies get eaten *before* they make it onto the wire, but
>  *after* tcpdump saw them on in.2.  It's driving me nuts...  I thought it
>  might have to do with the tg3 hardware doing some funky vlan
>  acceleration, but I've seen the same on plain dumb NICs too.
>
>  I'm willing to pay for a solution to this...  Or even for just someone
>  knowledgeable enough with the code taking interest in the issue.
>
>  Thanks,
>
>  Jonathan
>  _______________________________________________
>  Bridge mailing list
>  Bridge@lists.linux-foundation.org
>  https://lists.linux-foundation.org/mailman/listinfo/bridge
>

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Andy Gospodarek wrote:
> Configurations like this can get tricky at times.  Though it seems to
> make perfect sense to put two interfaces like this in the same bridge
> the current setup of the linux networking stack makes it hard to get
> things working sometimes.  Let me make sure I can understand what you
> are trying to accomplish.  Do you want to prevent traffic from vlan 2
> and vlan 3 from ever talking while still allowing them to talk to the
> gateway?  Will you want vlan 2 and vlan3 to communicate with each
> other at all?
>   
Well, the plan is to let the vlans communicate together 'freely', for
now.  I'm essentially using the vlan tag as a way to 'figure out' where
the packet is coming from.  The network spans a little over 100km in a
large, flat /22 subnet.  I wish I could do this the usual way with
subnets and what not, but we don't really control the routing and
gateway side of things.  So basically, we want to assign a vlan to each
branch of the network and avoid having to add a new NIC to the bridge
each time we add a new branch.  Feeding it a trunk into it and adding
vlan interfaces as needed works better for us since bandwidth is not
really the limiting factor.  Well, it would be better if it worked :P

Now I fully understand that it's not how 90% of people would approach
this, including myself, but it's the only option we have given what's on
either side of the bridge, which we can't change right now.  The
alternative is having a switch split the trunk and then feed that into
real (as opposed to vlan) interfaces on the bridge, wich I haven't fully
tested yet but am sure would work fine.

I'd rather focus on solving the bridged vlan thing though, it just might
be useful to someone else someday and definitely falls into the 'should
work' category as far as I'm concerned.

Jonathan

P.S.:  Sorry, forgot to reply to the list.

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[richardvoigt]

On Sat, Mar 8, 2008 at 4:26 PM, Jonathan Thibault <jonathan@navigue.com> wrote:
> Andy Gospodarek wrote:
>  > Configurations like this can get tricky at times.  Though it seems to
>  > make perfect sense to put two interfaces like this in the same bridge
>  > the current setup of the linux networking stack makes it hard to get
>  > things working sometimes.  Let me make sure I can understand what you
>  > are trying to accomplish.  Do you want to prevent traffic from vlan 2
>  > and vlan 3 from ever talking while still allowing them to talk to the
>  > gateway?  Will you want vlan 2 and vlan3 to communicate with each
>  > other at all?
>  >
>  Well, the plan is to let the vlans communicate together 'freely', for
>  now.  I'm essentially using the vlan tag as a way to 'figure out' where
>  the packet is coming from.  The network spans a little over 100km in a
>  large, flat /22 subnet.  I wish I could do this the usual way with
>  subnets and what not, but we don't really control the routing and
>  gateway side of things.  So basically, we want to assign a vlan to each
>  branch of the network and avoid having to add a new NIC to the bridge
>  each time we add a new branch.  Feeding it a trunk into it and adding
>  vlan interfaces as needed works better for us since bandwidth is not
>  really the limiting factor.  Well, it would be better if it worked :P
>
>  Now I fully understand that it's not how 90% of people would approach
>  this, including myself, but it's the only option we have given what's on
>  either side of the bridge, which we can't change right now.  The
>  alternative is having a switch split the trunk and then feed that into
>  real (as opposed to vlan) interfaces on the bridge, wich I haven't fully
>  tested yet but am sure would work fine.
>
>  I'd rather focus on solving the bridged vlan thing though, it just might
>  be useful to someone else someday and definitely falls into the 'should
>  work' category as far as I'm concerned.

You still didn't tell us any version numbers, and I've got a similar
setup which "works for me".  The only real difference is that my box
routes between two logical bridges, and the bridged interfaces are
multiple vlans in the same trunk.  I can even protect individual vlans
from each other with netfilter rules.

What if you routed "out" as a new vlan on the same cable as "in"?

>
>  Jonathan
>
>  P.S.:  Sorry, forgot to reply to the list.
>
>
> _______________________________________________
>  Bridge mailing list
>  Bridge@lists.linux-foundation.org
>  https://lists.linux-foundation.org/mailman/listinfo/bridge
>

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Sorry, that's was in another mistakenly off-list reply.

Kernel is 2.6.24, been seeing this problem since 2.6.16 when I started 
the setup.

richardvoigt@gmail.com wrote:
>
> You still didn't tell us any version numbers, and I've got a similar
> setup which "works for me".  The only real difference is that my box
> routes between two logical bridges, and the bridged interfaces are
> multiple vlans in the same trunk.  I can even protect individual vlans
> from each other with netfilter rules.
>
> What if you routed "out" as a new vlan on the same cable as "in"?
>
>   
That's an interesting idea which I haven't tried yet.  An interesting 
tidbit is that there is a handful of machines in the lot which are 
affected right away (as soon as I add a second vlan interface to the 
bridge).  It may just be that they just have a very short arp timeout.  
I tried to find a pattern in their MAC addresses or hardware but there 
isn't really one.  I first assumed there was a problem with those 
machines but given that the ARP reply never gets to the trunk cable 
going their way, I concluded otherwise.

Thanks a lot for the help.  The fact that you have a setup that works 
gives me some confidence that I'm not just trying to do something insane ;)

Jonathan

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Leigh Sharpe]

Can you do a brctl showmacs br0, and see if the machines which are not
receiving an ARP response are being seen by the bridge as being on the
wrong VLAN?


Ie, I'm wondering if the bridge sees their MAC address on an interface
other than the one they are really connected to. If that's the case, the
bridge would send their response out of the wrong interface, which may
result in the symptoms you are describing. I think there may be some
ebtables rules which could help here, but my memory fails me at this
point.

Regards,
             Leigh
 
Leigh Sharpe
Network Systems Engineer
Pacific Wireless
Ph +61 3 9584 8966
Mob 0408 009 502
Helpdesk 1300 300 616
email lsharpe@pacificwireless.com.au
web www.pacificwireless.com.au
 

-----Original Message-----
From: bridge-bounces@lists.linux-foundation.org
[mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Jonathan
Thibault
Sent: Monday, 10 March 2008 2:53 AM
To: bridge@lists.osdl.org
Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu

Sorry, that's was in another mistakenly off-list reply.

Kernel is 2.6.24, been seeing this problem since 2.6.16 when I started 
the setup.

richardvoigt@gmail.com wrote:
>
> You still didn't tell us any version numbers, and I've got a similar
> setup which "works for me".  The only real difference is that my box
> routes between two logical bridges, and the bridged interfaces are
> multiple vlans in the same trunk.  I can even protect individual vlans
> from each other with netfilter rules.
>
> What if you routed "out" as a new vlan on the same cable as "in"?
>
>   
That's an interesting idea which I haven't tried yet.  An interesting 
tidbit is that there is a handful of machines in the lot which are 
affected right away (as soon as I add a second vlan interface to the 
bridge).  It may just be that they just have a very short arp timeout.  
I tried to find a pattern in their MAC addresses or hardware but there 
isn't really one.  I first assumed there was a problem with those 
machines but given that the ARP reply never gets to the trunk cable 
going their way, I concluded otherwise.

Thanks a lot for the help.  The fact that you have a setup that works 
gives me some confidence that I'm not just trying to do something insane
;)

Jonathan
_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Leigh Sharpe]

Can you do a brctl showmacs br0, and see if the machines which are not
receiving an ARP response are being seen by the bridge as being on the
wrong VLAN?


Ie, I'm wondering if the bridge sees their MAC address on an interface
other than the one they are really connected to. If that's the case, the
bridge would send their response out of the wrong interface, which may
result in the symptoms you are describing. I think there may be some
ebtables rules which could help here, but my memory fails me at this
point.

Regards,
             Leigh
 
Leigh Sharpe
Network Systems Engineer
Pacific Wireless
Ph +61 3 9584 8966
Mob 0408 009 502
Helpdesk 1300 300 616
email lsharpe@pacificwireless.com.au
web www.pacificwireless.com.au
 

-----Original Message-----
From: bridge-bounces@lists.linux-foundation.org
[mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Jonathan
Thibault
Sent: Monday, 10 March 2008 2:53 AM
To: bridge@lists.osdl.org
Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu

Sorry, that's was in another mistakenly off-list reply.

Kernel is 2.6.24, been seeing this problem since 2.6.16 when I started 
the setup.

richardvoigt@gmail.com wrote:
>
> You still didn't tell us any version numbers, and I've got a similar
> setup which "works for me".  The only real difference is that my box
> routes between two logical bridges, and the bridged interfaces are
> multiple vlans in the same trunk.  I can even protect individual vlans
> from each other with netfilter rules.
>
> What if you routed "out" as a new vlan on the same cable as "in"?
>
>   
That's an interesting idea which I haven't tried yet.  An interesting 
tidbit is that there is a handful of machines in the lot which are 
affected right away (as soon as I add a second vlan interface to the 
bridge).  It may just be that they just have a very short arp timeout.  
I tried to find a pattern in their MAC addresses or hardware but there 
isn't really one.  I first assumed there was a problem with those 
machines but given that the ARP reply never gets to the trunk cable 
going their way, I concluded otherwise.

Thanks a lot for the help.  The fact that you have a setup that works 
gives me some confidence that I'm not just trying to do something insane
;)

Jonathan
_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Hello Leigh,

This was one of the first things I looked at when I noticed that it 
didn't reach the client.  As far as I can tell, no, the mac isn't on the 
wrong port.  Come to think of it, I think the problem might have to do 
with the bridge not learning the affected machine's MAC at all until I 
removed in.3 from the bridge completely.  I'll do further tests and 
confirm this shortly, especially if you think there is something to it.

Also of note, since I monitor the trunk itself (and I only filter by mac 
when I do), I would have seen the reply if it went to the wrong vlan 
anyway, or is that logic wrong?

Anyway, if the problem is that the bridge stops learning a specific 
machine's MAC, is there a way for me to manually add the MAC to the 
bridge table and see if it solves the problem?  If such is the case, 
it's not really a solution, but it would certainly narrow down the scope 
of the problem.

Jonathan


Leigh Sharpe wrote:
> Can you do a brctl showmacs br0, and see if the machines which are not
> receiving an ARP response are being seen by the bridge as being on the
> wrong VLAN?
>
>
> Ie, I'm wondering if the bridge sees their MAC address on an interface
> other than the one they are really connected to. If that's the case, the
> bridge would send their response out of the wrong interface, which may
> result in the symptoms you are describing. I think there may be some
> ebtables rules which could help here, but my memory fails me at this
> point.
>
> Regards,
>              Leigh
>  
> Leigh Sharpe
> Network Systems Engineer
> Pacific Wireless
> Ph +61 3 9584 8966
> Mob 0408 009 502
> Helpdesk 1300 300 616
> email lsharpe@pacificwireless.com.au
> web www.pacificwireless.com.au
>  
>   

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Leigh Sharpe]

Hi Jonathan,

So, you're seeing the ARP reply come in, untagged, but not seeing it
exit the tagged interface? Check to see that the bridge has definitely
learned that MAC on the correct interface.

I am wondering if your are coming up against this issue:

---
(From http://ebtables.sourceforge.net/brnf-faq.html )

How do I let vlan-tagged traffic go through a vlan bridge port and the
other traffic through a non-vlan bridge port? 

Suppose eth0 and eth0.15 are ports of br0. Without countermeasures all
traffic, including traffic vlan-tagged with tag 15, entering the
physical device eth0 will go through the bridge port eth0. To make the
15-tagged traffic go through the eth0.15 bridge port, use the following
ebtables rule: 
ebtables -t broute -A BROUTING -i eth0 --vlan-id 15 -j DROP

With the above rule, 15-tagged traffic will enter the bridge on the
physical device eth0, will then be brouted and enter the bridge port
eth0.15, the vlan header will be stripped, after which the packet is
bridged. The packet thus enters the BROUTING chain twice, the first time
with input device eth0 and the second time with input device eth0.15.
The other chains are only traversed once. All other traffic will be
bridged with input device eth0. 
----

Somewhere I have seen a mailing list archive with a far better
explanation, but I just can't seem to find it right now.

Can you run TCPdump on 'in', as well as 'in.2' and see if the packets
are disappearing somewhere between the two?
Alternatively, put an ebtables rule on each interface, with a target of
CONTINUE, and then do a ebtables -L --Lc to see which rules are
matching, so you can get an ide of which interface the bridge is trying
to spit out the replies you are looking for.


Leigh.
 

-----Original Message-----
From: bridge-bounces@lists.linux-foundation.org
[mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Jonathan
Thibault
Sent: Wednesday, 12 March 2008 2:44 AM
To: bridge@lists.osdl.org
Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu

Hello Leigh,

This was one of the first things I looked at when I noticed that it 
didn't reach the client.  As far as I can tell, no, the mac isn't on the

wrong port.  Come to think of it, I think the problem might have to do 
with the bridge not learning the affected machine's MAC at all until I 
removed in.3 from the bridge completely.  I'll do further tests and 
confirm this shortly, especially if you think there is something to it.

Also of note, since I monitor the trunk itself (and I only filter by mac

when I do), I would have seen the reply if it went to the wrong vlan 
anyway, or is that logic wrong?

Anyway, if the problem is that the bridge stops learning a specific 
machine's MAC, is there a way for me to manually add the MAC to the 
bridge table and see if it solves the problem?  If such is the case, 
it's not really a solution, but it would certainly narrow down the scope

of the problem.

Jonathan


Leigh Sharpe wrote:
> Can you do a brctl showmacs br0, and see if the machines which are not
> receiving an ARP response are being seen by the bridge as being on the
> wrong VLAN?
>
>
> Ie, I'm wondering if the bridge sees their MAC address on an interface
> other than the one they are really connected to. If that's the case,
the
> bridge would send their response out of the wrong interface, which may
> result in the symptoms you are describing. I think there may be some
> ebtables rules which could help here, but my memory fails me at this
> point.
>
> Regards,
>              Leigh
>  
> Leigh Sharpe
> Network Systems Engineer
> Pacific Wireless
> Ph +61 3 9584 8966
> Mob 0408 009 502
> Helpdesk 1300 300 616
> email lsharpe@pacificwireless.com.au
> web www.pacificwireless.com.au
>  
>   

_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Hello Leigh,

I did a quick test this morning on a machine that always reliably 
exhibits the problem.  First and foremost, the machine was always in 
brctl showmacs so sorry, that was probably me imagining things.  Or 
rather, me assuming it from what I'll describe below.  It is always on 
interface 2, which is in.2.

Thankfully, this is a very quiet machine, so I only filtered for its mac 
address and did tcpdumps on in, in.2 and on another box connected 
directly to the trunk through a hub through eth1, which is dedicated to 
sniffing this traffic.  It's a test box, so sorry about the clock being 
wrong on that one, I know it looks bad ;)

Anyway.  This goes to show that this only seems to affects ARP.  Once 
the machine has learned the gateway's MAC, it will talk to the rest of 
the net just fine, even if in.3 is back on the bridge.

Could you be more specific as to the ebtables rules you'd need me to 
run?  Just match for arp replies to my 'test' host for both in.2 and 
in.3 and see which one hits?  Sounds a bit redundant given our tcpdump 
output now, but I'll give it a shot.

Thanks a bunch,

Jonathan

Here are the tcpdump outputs, they start with in.3 on the bridge and 
confirming that the machine's MAC is still on port 2:

 # tcpdump -n -i in ether host 08:10:17:0D:70:61
tcpdump: WARNING: in: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on in, link-type EN10MB (Ethernet), capture size 68 bytes
11:15:57.726790 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
11:15:57.726808 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
11:15:57.764616 arp who-has 207.134.8.1 tell 207.134.9.17
11:15:57.765116 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:15:58.727808 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 1, length 64
11:15:59.727964 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 2, length 64
11:16:00.729381 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 3, length 64
11:16:01.728932 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 4, length 64
11:16:02.730502 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 5, length 64
11:16:03.729910 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 6, length 64
11:16:10.246543 IP 24.64.96.74.30516 > 207.134.9.17.1026: UDP, length 484
11:16:10.246641 IP 24.64.96.74.30516 > 207.134.9.17.1027: UDP, length 484
11:16:10.247735 IP 24.64.96.74.30516 > 207.134.9.17.1028: UDP, length 484
------ removed in.3 11:16:42 ------
11:17:19.743994 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:20.402770 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 0, length 64
11:17:20.438702 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:21.403648 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 1, length 64
11:17:22.404685 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 2, length 64
11:17:23.404257 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 3, length 64
------ in.3 added 11:17:48, forwarding at 11:18:03 ------
11:18:31.522164 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
11:18:31.522182 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
11:18:32.522671 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 1, length 64
11:18:33.522717 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 2, length 64
11:18:41.372052 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:47.366774 arp who-has 207.134.9.90 tell 207.134.9.17
------ removed in.3 11:18:50 ------
11:18:56.050233 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 0, length 64
11:18:57.050744 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 1, length 64
11:18:58.050867 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 2, length 64
^C
28 packets captured
28 packets received by filter
0 packets dropped by kernel

# tcpdump -n -i in.2 ether host 08:10:17:0D:70:61
tcpdump: WARNING: in.2: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on in.2, link-type EN10MB (Ethernet), capture size 68 bytes
11:15:57.726800 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
11:15:57.764576 arp who-has 207.134.8.1 tell 207.134.9.17
11:15:57.765109 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:15:58.727798 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 1, length 64
11:15:59.727956 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 2, length 64
11:16:00.729372 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 3, length 64
11:16:01.728923 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 4, length 64
11:16:02.730493 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 5, length 64
11:16:03.729900 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 6, length 64
11:16:10.246534 IP 24.64.96.74.30516 > 207.134.9.17.1026: UDP, length 484
11:16:10.246636 IP 24.64.96.74.30516 > 207.134.9.17.1027: UDP, length 484
11:16:10.247729 IP 24.64.96.74.30516 > 207.134.9.17.1028: UDP, length 484
------ removed in.3 11:16:42 ------
11:17:19.743488 arp who-has 207.134.8.1 tell 207.134.9.17
11:17:19.743988 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:20.402761 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 0, length 64
11:17:20.437712 arp who-has 207.134.8.1 tell 207.134.9.17
11:17:20.438696 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:20.468842 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 0, length 64
11:17:21.403639 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 1, length 64
11:17:21.433356 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 1, length 64
11:17:22.404675 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 2, length 64
11:17:22.439420 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 2, length 64
11:17:23.404249 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 3, length 64
11:17:23.434231 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 3, length 64
------ in.3 added 11:17:48, forwarding at 11:18:03 ------
11:18:31.522176 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
11:18:31.559476 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 0, length 64
11:18:32.522663 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 1, length 64
11:18:32.554809 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 1, length 64
11:18:33.522708 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 2, length 64
11:18:33.558038 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 2, length 64
11:18:41.372035 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:41.393845 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
11:18:47.366759 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:47.386445 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
------ removed in.3 11:18:50 ------
11:18:53.374862 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:56.050224 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 0, length 64
11:18:56.090664 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 0, length 64
11:18:57.050736 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 1, length 64
11:18:57.081345 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 1, length 64
11:18:58.050859 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 2, length 64
11:18:58.081524 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 2, length 64
^C
41 packets captured
44 packets received by filter
0 packets dropped by kernel

# tcpdump -i eth1 -n ether host 08:10:17:0D:70:61
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 68 bytes
08:20:44.448485 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
08:20:44.448658 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
08:20:44.486159 arp who-has 207.134.8.1 tell 207.134.9.17
08:20:44.486261 arp who-has 207.134.8.1 tell 207.134.9.17
------ removed in.3 11:16:42 ------
08:22:06.430604 arp who-has 207.134.8.1 tell 207.134.9.17
08:22:06.431239 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
08:22:07.089746 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 0, length 64
08:22:07.124543 arp who-has 207.134.8.1 tell 207.134.9.17
08:22:07.125847 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
08:22:07.155719 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 0, length 64
08:22:08.090204 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 1, length 64
08:22:08.119830 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 1, length 64
08:22:09.091002 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 2, length 64
08:22:09.125404 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 2, length 64
08:22:10.089973 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 3, length 64
08:22:10.119863 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 3, length 64
------ in.3 added 11:17:48, forwarding at 11:18:03 ------
08:23:18.179485 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
08:23:18.179499 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
08:23:18.216403 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 0, length 64
08:23:19.179382 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 1, length 64
08:23:19.211428 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 1, length 64
08:23:20.179009 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 2, length 64
08:23:20.214195 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 2, length 64
08:23:28.024814 arp who-has 207.134.9.90 tell 207.134.9.17
08:23:28.025045 arp who-has 207.134.9.90 tell 207.134.9.17
08:23:28.046767 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
08:23:34.017173 arp who-has 207.134.9.90 tell 207.134.9.17   <--- 
Re-added in.3 Somewhere around here.
08:23:34.017252 arp who-has 207.134.9.90 tell 207.134.9.17
08:23:34.036851 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
08:23:40.022739 arp who-has 207.134.9.90 tell 207.134.9.17
------ removed in.3 11:18:50 ------
08:23:42.697280 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 0, length 64
08:23:42.737416 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 0, length 64
08:23:43.697339 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 1, length 64
08:23:43.727680 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 1, length 64
08:23:44.696936 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 2, length 64
08:23:44.727440 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 2, length 64

36 packets captured
40 packets received by filter
0 packets dropped by kernel

Leigh Sharpe wrote:
> Hi Jonathan,
>
> So, you're seeing the ARP reply come in, untagged, but not seeing it
> exit the tagged interface? Check to see that the bridge has definitely
> learned that MAC on the correct interface.
>
> I am wondering if your are coming up against this issue:
>
> ---
> (From http://ebtables.sourceforge.net/brnf-faq.html )
>
> How do I let vlan-tagged traffic go through a vlan bridge port and the
> other traffic through a non-vlan bridge port? 
>
> Suppose eth0 and eth0.15 are ports of br0. Without countermeasures all
> traffic, including traffic vlan-tagged with tag 15, entering the
> physical device eth0 will go through the bridge port eth0. To make the
> 15-tagged traffic go through the eth0.15 bridge port, use the following
> ebtables rule: 
> ebtables -t broute -A BROUTING -i eth0 --vlan-id 15 -j DROP
>
> With the above rule, 15-tagged traffic will enter the bridge on the
> physical device eth0, will then be brouted and enter the bridge port
> eth0.15, the vlan header will be stripped, after which the packet is
> bridged. The packet thus enters the BROUTING chain twice, the first time
> with input device eth0 and the second time with input device eth0.15.
> The other chains are only traversed once. All other traffic will be
> bridged with input device eth0. 
> ----
>
> Somewhere I have seen a mailing list archive with a far better
> explanation, but I just can't seem to find it right now.
>
> Can you run TCPdump on 'in', as well as 'in.2' and see if the packets
> are disappearing somewhere between the two?
> Alternatively, put an ebtables rule on each interface, with a target of
> CONTINUE, and then do a ebtables -L --Lc to see which rules are
> matching, so you can get an ide of which interface the bridge is trying
> to spit out the replies you are looking for.
>
>
> Leigh.
>  
>
> -----Original Message-----
> From: bridge-bounces@lists.linux-foundation.org
> [mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Jonathan
> Thibault
> Sent: Wednesday, 12 March 2008 2:44 AM
> To: bridge@lists.osdl.org
> Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu
>
> Hello Leigh,
>
> This was one of the first things I looked at when I noticed that it 
> didn't reach the client.  As far as I can tell, no, the mac isn't on the
>
> wrong port.  Come to think of it, I think the problem might have to do 
> with the bridge not learning the affected machine's MAC at all until I 
> removed in.3 from the bridge completely.  I'll do further tests and 
> confirm this shortly, especially if you think there is something to it.
>
> Also of note, since I monitor the trunk itself (and I only filter by mac
>
> when I do), I would have seen the reply if it went to the wrong vlan 
> anyway, or is that logic wrong?
>
> Anyway, if the problem is that the bridge stops learning a specific 
> machine's MAC, is there a way for me to manually add the MAC to the 
> bridge table and see if it solves the problem?  If such is the case, 
> it's not really a solution, but it would certainly narrow down the scope
>
> of the problem.
>
> Jonathan
>
>
> Leigh Sharpe wrote:
>   
>> Can you do a brctl showmacs br0, and see if the machines which are not
>> receiving an ARP response are being seen by the bridge as being on the
>> wrong VLAN?
>>
>>
>> Ie, I'm wondering if the bridge sees their MAC address on an interface
>> other than the one they are really connected to. If that's the case,
>>     
> the
>   
>> bridge would send their response out of the wrong interface, which may
>> result in the symptoms you are describing. I think there may be some
>> ebtables rules which could help here, but my memory fails me at this
>> point.
>>
>> Regards,
>>              Leigh
>>  
>> Leigh Sharpe
>> Network Systems Engineer
>> Pacific Wireless
>> Ph +61 3 9584 8966
>> Mob 0408 009 502
>> Helpdesk 1300 300 616
>> email lsharpe@pacificwireless.com.au
>> web www.pacificwireless.com.au
>>  
>>   
>>     
>
> _______________________________________________
> Bridge mailing list
> Bridge@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/bridge
>   

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Leigh Sharpe]

Hi Jonathan,
 Let me make sure I understand your situation properly:

You have an interface on which you have two vlans:

Vconfig add in 2
Vconfig add in 3

You have created a bridge and assigned interface out and interface in.2
to the bridge:

Brctl addbr br0
Brctl addif br0 out
Brctl addif br0 in.2
Ifconfig out up
Ifconfig in.2 up

Then, when you add in.3 to the bridge with this command,

Brctl addif br0 in.3

Things start to break, and devices connected to in.3 cannot talk to
devices on interface 'out'.

Does this look correct?


I'm presuming that the MAC of interest is 00:18:18:62:3e:80

I suspect that your bridge is seeing this mac coming in, VLAN tagged, on
interface 'in', and is learning it on that interface, before you add
in.3 to the bridge. Once that happens, any replies coming in on
interface 'out' will not be sent to in.3, as they should.

I recall reading somewhere that if you add an ebtables rule to 'in',
which drops all VLAN tagged packets, it will then be redirected to the
correct sub-interface (ie in.3). I envisage it would be something like 

Ebtables -t broute -A BROUTING -p 8021_Q -I in -J DROP

Have a good look at this post if you need further details:

 http://osdir.com/ml/linux.drivers.vlan/2006-07/msg00011.html


Hope this helps.
Regards,
             Leigh
 
Leigh Sharpe
Network Systems Engineer
Pacific Wireless
Ph +61 3 9584 8966
Mob 0408 009 502
Helpdesk 1300 300 616
email lsharpe@pacificwireless.com.au
web www.pacificwireless.com.au

 

-----Original Message-----
From: bridge-bounces@lists.linux-foundation.org
[mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Jonathan
Thibault
Sent: Thursday, 13 March 2008 4:44 AM
To: bridge@lists.osdl.org
Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu

Hello Leigh,

I did a quick test this morning on a machine that always reliably 
exhibits the problem.  First and foremost, the machine was always in 
brctl showmacs so sorry, that was probably me imagining things.  Or 
rather, me assuming it from what I'll describe below.  It is always on 
interface 2, which is in.2.

Thankfully, this is a very quiet machine, so I only filtered for its mac

address and did tcpdumps on in, in.2 and on another box connected 
directly to the trunk through a hub through eth1, which is dedicated to 
sniffing this traffic.  It's a test box, so sorry about the clock being 
wrong on that one, I know it looks bad ;)

Anyway.  This goes to show that this only seems to affects ARP.  Once 
the machine has learned the gateway's MAC, it will talk to the rest of 
the net just fine, even if in.3 is back on the bridge.

Could you be more specific as to the ebtables rules you'd need me to 
run?  Just match for arp replies to my 'test' host for both in.2 and 
in.3 and see which one hits?  Sounds a bit redundant given our tcpdump 
output now, but I'll give it a shot.

Thanks a bunch,

Jonathan

Here are the tcpdump outputs, they start with in.3 on the bridge and 
confirming that the machine's MAC is still on port 2:

 # tcpdump -n -i in ether host 08:10:17:0D:70:61
tcpdump: WARNING: in: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on in, link-type EN10MB (Ethernet), capture size 68 bytes
11:15:57.726790 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
11:15:57.726808 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
11:15:57.764616 arp who-has 207.134.8.1 tell 207.134.9.17
11:15:57.765116 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:15:58.727808 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 1, length 64
11:15:59.727964 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 2, length 64
11:16:00.729381 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 3, length 64
11:16:01.728932 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 4, length 64
11:16:02.730502 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 5, length 64
11:16:03.729910 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 6, length 64
11:16:10.246543 IP 24.64.96.74.30516 > 207.134.9.17.1026: UDP, length
484
11:16:10.246641 IP 24.64.96.74.30516 > 207.134.9.17.1027: UDP, length
484
11:16:10.247735 IP 24.64.96.74.30516 > 207.134.9.17.1028: UDP, length
484
------ removed in.3 11:16:42 ------
11:17:19.743994 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:20.402770 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 0, length 64
11:17:20.438702 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:21.403648 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 1, length 64
11:17:22.404685 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 2, length 64
11:17:23.404257 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 3, length 64
------ in.3 added 11:17:48, forwarding at 11:18:03 ------
11:18:31.522164 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
11:18:31.522182 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
11:18:32.522671 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 1, length 64
11:18:33.522717 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 2, length 64
11:18:41.372052 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:47.366774 arp who-has 207.134.9.90 tell 207.134.9.17
------ removed in.3 11:18:50 ------
11:18:56.050233 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 0, length 64
11:18:57.050744 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 1, length 64
11:18:58.050867 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 2, length 64
^C
28 packets captured
28 packets received by filter
0 packets dropped by kernel

# tcpdump -n -i in.2 ether host 08:10:17:0D:70:61
tcpdump: WARNING: in.2: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on in.2, link-type EN10MB (Ethernet), capture size 68 bytes
11:15:57.726800 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
11:15:57.764576 arp who-has 207.134.8.1 tell 207.134.9.17
11:15:57.765109 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:15:58.727798 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 1, length 64
11:15:59.727956 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 2, length 64
11:16:00.729372 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 3, length 64
11:16:01.728923 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 4, length 64
11:16:02.730493 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 5, length 64
11:16:03.729900 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 6, length 64
11:16:10.246534 IP 24.64.96.74.30516 > 207.134.9.17.1026: UDP, length
484
11:16:10.246636 IP 24.64.96.74.30516 > 207.134.9.17.1027: UDP, length
484
11:16:10.247729 IP 24.64.96.74.30516 > 207.134.9.17.1028: UDP, length
484
------ removed in.3 11:16:42 ------
11:17:19.743488 arp who-has 207.134.8.1 tell 207.134.9.17
11:17:19.743988 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:20.402761 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 0, length 64
11:17:20.437712 arp who-has 207.134.8.1 tell 207.134.9.17
11:17:20.438696 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
11:17:20.468842 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 0, length 64
11:17:21.403639 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 1, length 64
11:17:21.433356 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 1, length 64
11:17:22.404675 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 2, length 64
11:17:22.439420 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 2, length 64
11:17:23.404249 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 3, length 64
11:17:23.434231 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 3, length 64
------ in.3 added 11:17:48, forwarding at 11:18:03 ------
11:18:31.522176 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
11:18:31.559476 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 0, length 64
11:18:32.522663 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 1, length 64
11:18:32.554809 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 1, length 64
11:18:33.522708 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 2, length 64
11:18:33.558038 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 2, length 64
11:18:41.372035 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:41.393845 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
11:18:47.366759 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:47.386445 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
------ removed in.3 11:18:50 ------
11:18:53.374862 arp who-has 207.134.9.90 tell 207.134.9.17
11:18:56.050224 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 0, length 64
11:18:56.090664 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 0, length 64
11:18:57.050736 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 1, length 64
11:18:57.081345 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 1, length 64
11:18:58.050859 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 2, length 64
11:18:58.081524 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 2, length 64
^C
41 packets captured
44 packets received by filter
0 packets dropped by kernel

# tcpdump -i eth1 -n ether host 08:10:17:0D:70:61
tcpdump: WARNING: eth1: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol
decode
listening on eth1, link-type EN10MB (Ethernet), capture size 68 bytes
08:20:44.448485 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
08:20:44.448658 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
19986, seq 0, length 64
08:20:44.486159 arp who-has 207.134.8.1 tell 207.134.9.17
08:20:44.486261 arp who-has 207.134.8.1 tell 207.134.9.17
------ removed in.3 11:16:42 ------
08:22:06.430604 arp who-has 207.134.8.1 tell 207.134.9.17
08:22:06.431239 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
08:22:07.089746 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 0, length 64
08:22:07.124543 arp who-has 207.134.8.1 tell 207.134.9.17
08:22:07.125847 arp reply 207.134.8.1 is-at 00:18:18:62:3e:80
08:22:07.155719 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 0, length 64
08:22:08.090204 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 1, length 64
08:22:08.119830 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 1, length 64
08:22:09.091002 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 2, length 64
08:22:09.125404 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 2, length 64
08:22:10.089973 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
22802, seq 3, length 64
08:22:10.119863 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
22802, seq 3, length 64
------ in.3 added 11:17:48, forwarding at 11:18:03 ------
08:23:18.179485 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
08:23:18.179499 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 0, length 64
08:23:18.216403 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 0, length 64
08:23:19.179382 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 1, length 64
08:23:19.211428 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 1, length 64
08:23:20.179009 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24338, seq 2, length 64
08:23:20.214195 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24338, seq 2, length 64
08:23:28.024814 arp who-has 207.134.9.90 tell 207.134.9.17
08:23:28.025045 arp who-has 207.134.9.90 tell 207.134.9.17
08:23:28.046767 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
08:23:34.017173 arp who-has 207.134.9.90 tell 207.134.9.17   <--- 
Re-added in.3 Somewhere around here.
08:23:34.017252 arp who-has 207.134.9.90 tell 207.134.9.17
08:23:34.036851 arp reply 207.134.9.90 is-at 00:16:d4:dd:6f:4e
08:23:40.022739 arp who-has 207.134.9.90 tell 207.134.9.17
------ removed in.3 11:18:50 ------
08:23:42.697280 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 0, length 64
08:23:42.737416 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 0, length 64
08:23:43.697339 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 1, length 64
08:23:43.727680 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 1, length 64
08:23:44.696936 IP 66.38.210.117 > 207.134.9.17: ICMP echo request, id 
24594, seq 2, length 64
08:23:44.727440 IP 207.134.9.17 > 66.38.210.117: ICMP echo reply, id 
24594, seq 2, length 64

36 packets captured
40 packets received by filter
0 packets dropped by kernel

Leigh Sharpe wrote:
> Hi Jonathan,
>
> So, you're seeing the ARP reply come in, untagged, but not seeing it
> exit the tagged interface? Check to see that the bridge has definitely
> learned that MAC on the correct interface.
>
> I am wondering if your are coming up against this issue:
>
> ---
> (From http://ebtables.sourceforge.net/brnf-faq.html )
>
> How do I let vlan-tagged traffic go through a vlan bridge port and the
> other traffic through a non-vlan bridge port? 
>
> Suppose eth0 and eth0.15 are ports of br0. Without countermeasures all
> traffic, including traffic vlan-tagged with tag 15, entering the
> physical device eth0 will go through the bridge port eth0. To make the
> 15-tagged traffic go through the eth0.15 bridge port, use the
following
> ebtables rule: 
> ebtables -t broute -A BROUTING -i eth0 --vlan-id 15 -j DROP
>
> With the above rule, 15-tagged traffic will enter the bridge on the
> physical device eth0, will then be brouted and enter the bridge port
> eth0.15, the vlan header will be stripped, after which the packet is
> bridged. The packet thus enters the BROUTING chain twice, the first
time
> with input device eth0 and the second time with input device eth0.15.
> The other chains are only traversed once. All other traffic will be
> bridged with input device eth0. 
> ----
>
> Somewhere I have seen a mailing list archive with a far better
> explanation, but I just can't seem to find it right now.
>
> Can you run TCPdump on 'in', as well as 'in.2' and see if the packets
> are disappearing somewhere between the two?
> Alternatively, put an ebtables rule on each interface, with a target
of
> CONTINUE, and then do a ebtables -L --Lc to see which rules are
> matching, so you can get an ide of which interface the bridge is
trying
> to spit out the replies you are looking for.
>
>
> Leigh.
>  
>
> -----Original Message-----
> From: bridge-bounces@lists.linux-foundation.org
> [mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of
Jonathan
> Thibault
> Sent: Wednesday, 12 March 2008 2:44 AM
> To: bridge@lists.osdl.org
> Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu
>
> Hello Leigh,
>
> This was one of the first things I looked at when I noticed that it 
> didn't reach the client.  As far as I can tell, no, the mac isn't on
the
>
> wrong port.  Come to think of it, I think the problem might have to do

> with the bridge not learning the affected machine's MAC at all until I

> removed in.3 from the bridge completely.  I'll do further tests and 
> confirm this shortly, especially if you think there is something to
it.
>
> Also of note, since I monitor the trunk itself (and I only filter by
mac
>
> when I do), I would have seen the reply if it went to the wrong vlan 
> anyway, or is that logic wrong?
>
> Anyway, if the problem is that the bridge stops learning a specific 
> machine's MAC, is there a way for me to manually add the MAC to the 
> bridge table and see if it solves the problem?  If such is the case, 
> it's not really a solution, but it would certainly narrow down the
scope
>
> of the problem.
>
> Jonathan
>
>
> Leigh Sharpe wrote:
>   
>> Can you do a brctl showmacs br0, and see if the machines which are
not
>> receiving an ARP response are being seen by the bridge as being on
the
>> wrong VLAN?
>>
>>
>> Ie, I'm wondering if the bridge sees their MAC address on an
interface
>> other than the one they are really connected to. If that's the case,
>>     
> the
>   
>> bridge would send their response out of the wrong interface, which
may
>> result in the symptoms you are describing. I think there may be some
>> ebtables rules which could help here, but my memory fails me at this
>> point.
>>
>> Regards,
>>              Leigh
>>  
>> Leigh Sharpe
>> Network Systems Engineer
>> Pacific Wireless
>> Ph +61 3 9584 8966
>> Mob 0408 009 502
>> Helpdesk 1300 300 616
>> email lsharpe@pacificwireless.com.au
>> web www.pacificwireless.com.au
>>  
>>   
>>     
>
> _______________________________________________
> Bridge mailing list
> Bridge@lists.linux-foundation.org
> https://lists.linux-foundation.org/mailman/listinfo/bridge
>   

_______________________________________________
Bridge mailing list
Bridge@lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/bridge

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Malcolm Scott]

At 13:38 today, Leigh Sharpe wrote:

> Brctl addbr br0
> Brctl addif br0 out
> Brctl addif br0 in.2
> ...
> Brctl addif br0 in.3

Another thing to bear in mind: many commodity VLAN-aware switches cannot 
cope with another device bridging frames between two VLANs, as their address 
databases store {MAC address, VLAN, port} and do not permit duplicate MAC 
addresses.  In my experience (of 3Com managed switches of various models) 
this manifests itself in all manner of bizaare and unexpected ways.  So if 
your "in" interface is connected to a managed switch, expect unpleasantness.

This is not (AFAICT) a bug with the Linux bridging driver, but a design 
limitation of such switches.

-- 
Malcolm Scott
Research Assistant
University of Cambridge Computer Laboratory

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Leigh Sharpe wrote:
> Hi Jonathan,
>  Let me make sure I understand your situation properly:
>
> You have an interface on which you have two vlans:
>
> Vconfig add in 2
> Vconfig add in 3
>
> You have created a bridge and assigned interface out and interface in.2
> to the bridge:
>
> Brctl addbr br0
> Brctl addif br0 out
> Brctl addif br0 in.2
> Ifconfig out up
> Ifconfig in.2 up
>
> Then, when you add in.3 to the bridge with this command,
>
> Brctl addif br0 in.3
>
>   
Correct until now.
> Things start to break, and devices connected to in.3 cannot talk to
> devices on interface 'out'.
>
> Does this look correct?
>
>   
No.  There are no devices connected to in.3 at all.  Devices connected 
to in.2 stop being able to reach the gateway which is connected to 
'out'.  Actually, it's more a matter of being unable to ARP the gateway 
connected to 'out'.  The arp request reaches the gateway, but the reply 
never reaches the machine connected to in.2.  If the machine on in.2 
already has the gateway's mac in its arp table, then it can talk to it 
fine.  So far, the only thing that doesn't work with my setup is *arp 
replies*.
> I'm presuming that the MAC of interest is 00:18:18:62:3e:80
>
> I suspect that your bridge is seeing this mac coming in, VLAN tagged, on
> interface 'in', and is learning it on that interface, before you add
> in.3 to the bridge. Once that happens, any replies coming in on
> interface 'out' will not be sent to in.3, as they should.
>
>   
That's interesting.  Interface 'in' is not part of the bridge however, 
in.2 is (and at that point, in.3).  My assumption was that a 'vlan 
interface' spits out packets untagged rather than the raw (tagged) 
packets coming on the real physical interface, which would make logical 
sense.

Also of note is that I do see the reply show up on in.2 (you see it in 
my tcpdump -i in.2 log) as it should be with tcpdump, so I don't think 
the problem is that it goes to the wrong interface.  The problem is that 
it doesn't come *out* of the physical interface at all, tagged or not, 
onto the vlan trunk.  If it came out on vlan3 or untagged, I would still 
see it.

It shows up locally if I tcpdump -i in, but it never shows up on the 
wire (tcpdump -i eth1 on a box connected to the trunk right next to the 
bridge, no vlan switch involved, just a dumb hub acting as a repeater so 
I can sniff the traffic).  And that is before it gets to any vlan-aware 
switch.  So the problem at that point is not a matter of a switch not 
knowing how to deal with this peculiar situation.


              |   bridge   |       +-| 00:18:18:62:3e:80 & all other |
gateway-|(out)|out-br0-in.2|(in)|-hub
        |     |      +-in.3|    |  +-|(eth1)                         |
        | linux box with bridge |    | linux box used to sniff trunk |

The arp reply, from the gatway, shows up on 'in.2', it shows up on 'in', 
but it never shows up on 'eth1'.  I seriously doubt the hub is smart 
enough to notice that in.3 has been added to the bridge and suddently 
stop passing *only* arp replies as I see all other traffic just fine. I 
also see those arp replies when in.3 isn't part of the bridge.  So my 
fair assumption is that the arp reply packets never make it onto the hub 
at all.

As far as I can tell, they vanish between 'in' and the hub.

> I recall reading somewhere that if you add an ebtables rule to 'in',
> which drops all VLAN tagged packets, it will then be redirected to the
> correct sub-interface (ie in.3). I envisage it would be something like 
>
> Ebtables -t broute -A BROUTING -p 8021_Q -I in -J DROP
>
>   
I will give that a try for sure.  From reading the post you refer to, it 
makes a bit of sense, but I'm not entirely sure it covers the whole 
problem.  A problem such as what it refers to seems like it would affect 
all traffic instead of just arp replies.

Thanks a bunch, I will update you on my results with that ebtables rule 
shortly.

Jonathan
> Have a good look at this post if you need further details:
>
>  http://osdir.com/ml/linux.drivers.vlan/2006-07/msg00011.html
>
>
> Hope this helps.
> Regards,
>              Leigh
>  
> Leigh Sharpe
> Network Systems Engineer
> Pacific Wireless
> Ph +61 3 9584 8966
> Mob 0408 009 502
> Helpdesk 1300 300 616
> email lsharpe@pacificwireless.com.au
> web www.pacificwireless.com.au
>
>  
>
> -----Original Message-----
> From: bridge-bounces@lists.linux-foundation.org
> [mailto:bridge-bounces@lists.linux-foundation.org] On Behalf Of Jonathan
> Thibault
> Sent: Thursday, 13 March 2008 4:44 AM
> To: bridge@lists.osdl.org
> Subject: Re: [Bridge] bridge, vlan and *no* stp/bpdu
>
> Hello Leigh,
>
> I did a quick test this morning on a machine that always reliably 
> exhibits the problem.  First and foremost, the machine was always in 
> brctl showmacs so sorry, that was probably me imagining things.  Or 
> rather, me assuming it from what I'll describe below.  It is always on 
> interface 2, which is in.2.
>
> Thankfully, this is a very quiet machine, so I only filtered for its mac
>
> address and did tcpdumps on in, in.2 and on another box connected 
> directly to the trunk through a hub through eth1, which is dedicated to 
> sniffing this traffic.  It's a test box, so sorry about the clock being 
> wrong on that one, I know it looks bad ;)
>
> Anyway.  This goes to show that this only seems to affects ARP.  Once 
> the machine has learned the gateway's MAC, it will talk to the rest of 
> the net just fine, even if in.3 is back on the bridge.
>
> Could you be more specific as to the ebtables rules you'd need me to 
> run?  Just match for arp replies to my 'test' host for both in.2 and 
> in.3 and see which one hits?  Sounds a bit redundant given our tcpdump 
> output now, but I'll give it a shot.
>
> Thanks a bunch,
>
> Jonathan
>
> Here are the tcpdump outputs, they start with in.3 on the bridge and 
> confirming that the machine's MAC is still on port 2:
>
>  ...
>   

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Jonathan Thibault]

Hello Leigh and the lists,

I've had time to do some more tests on this today and here are my results.

First, I tried an ebtables rules as suggested by you and that link you 
offered:
  ebtables -t broute -A BROUTING -p 802_1Q -i in -j DROP

It did not seem to affect the issue at all, perhaps I did it wrong and 
would like some suggestions on that.

Now I figured I might try to find out exactly where the arp reply 
vanishes.  So I added this:

ebtables -t broute -A BROUTING -p arp --arp-opcode 2 --arp-mac-src 
00:18:18:62:3e:80 --arp-mac-dst 08:10:17:0D:70:61 --log

where 00:18:18:62:3e:80 is the gateway on the untagged (out) side of the 
bridge and 08:10:17:0D:70:61 is a host in vlan2 that never recieves its 
arp replies from the gateway as soon as I add in.3 to the bridge.

I added in.3 to the bridge, then I pinged 08:10:17:0D:70:61 from the 
outside (beyond the gateway) to have it arp the gateway.  Sure enough, I 
saw arp requests and replies on interface 'out' of the bridge but not in 
or in.2, as described in my previous post.

Here's what I got in the log:

Apr 11 11:56:18 bridgehost IN=out OUT= MAC source = 00:18:18:62:3e:80 
MAC dest = 08:10:17:0d:70:61 proto = 0x0806

Note the 'OUT=' field...  If things worked, I'd expect to see 'OUT=in.2' 
there.

So I put the bridge back in its 'working' state, removed 'in.3' so that 
I had only 'in.2' and 'out' as part of the bridge.

I ran another test and now the arp reply reached my host, it was able to 
reach the gateway and reply to my ping.  Time to check the logs...

Apr 11 11:59:03 ordralphabetix IN=out OUT= MAC source = 
00:18:18:62:3e:80 MAC dest = 08:10:17:0d:70:61 proto = 0x0806

Why don't I see 'OUT=in.2'.  If the arp reply reached my host, it 
obviously had to go through the bridge.  That is in the 'out' interface 
and out the 'in.2' interface.  It's pretty strange to me, I'd really 
like to know what's going on here.

Thanks a lot,

Jonathan
> Leigh Sharpe wrote:
>   
>> Hi Jonathan,
>>  Let me make sure I understand your situation properly:
>>
>> You have an interface on which you have two vlans:
>>
>> Vconfig add in 2
>> Vconfig add in 3
>>
>> You have created a bridge and assigned interface out and interface in.2
>> to the bridge:
>>
>> Brctl addbr br0
>> Brctl addif br0 out
>> Brctl addif br0 in.2
>> Ifconfig out up
>> Ifconfig in.2 up
>>
>> Then, when you add in.3 to the bridge with this command,
>>
>> Brctl addif br0 in.3
>>
>>   
>>     
> Correct until now.
>   
>> Things start to break, and devices connected to in.3 cannot talk to
>> devices on interface 'out'.
>>
>> Does this look correct?
>>
>>   
>>     
> No.  There are no devices connected to in.3 at all.  Devices connected 
> to in.2 stop being able to reach the gateway which is connected to 
> 'out'.  Actually, it's more a matter of being unable to ARP the gateway 
> connected to 'out'.  The arp request reaches the gateway, but the reply 
> never reaches the machine connected to in.2.  If the machine on in.2 
> already has the gateway's mac in its arp table, then it can talk to it 
> fine.  So far, the only thing that doesn't work with my setup is *arp 
> replies*.
>   
>> I'm presuming that the MAC of interest is 00:18:18:62:3e:80
>>
>> I suspect that your bridge is seeing this mac coming in, VLAN tagged, on
>> interface 'in', and is learning it on that interface, before you add
>> in.3 to the bridge. Once that happens, any replies coming in on
>> interface 'out' will not be sent to in.3, as they should.
>>
>>   
>>     
> That's interesting.  Interface 'in' is not part of the bridge however, 
> in.2 is (and at that point, in.3).  My assumption was that a 'vlan 
> interface' spits out packets untagged rather than the raw (tagged) 
> packets coming on the real physical interface, which would make logical 
> sense.
>
> Also of note is that I do see the reply show up on in.2 (you see it in 
> my tcpdump -i in.2 log) as it should be with tcpdump, so I don't think 
> the problem is that it goes to the wrong interface.  The problem is that 
> it doesn't come *out* of the physical interface at all, tagged or not, 
> onto the vlan trunk.  If it came out on vlan3 or untagged, I would still 
> see it.
>
> It shows up locally if I tcpdump -i in, but it never shows up on the 
> wire (tcpdump -i eth1 on a box connected to the trunk right next to the 
> bridge, no vlan switch involved, just a dumb hub acting as a repeater so 
> I can sniff the traffic).  And that is before it gets to any vlan-aware 
> switch.  So the problem at that point is not a matter of a switch not 
> knowing how to deal with this peculiar situation.
>
>
>               |   bridge   |       +-| 00:18:18:62:3e:80 & all other |
> gateway-|(out)|out-br0-in.2|(in)|-hub
>         |     |      +-in.3|    |  +-|(eth1)                         |
>         | linux box with bridge |    | linux box used to sniff trunk |
>
> The arp reply, from the gatway, shows up on 'in.2', it shows up on 'in', 
> but it never shows up on 'eth1'.  I seriously doubt the hub is smart 
> enough to notice that in.3 has been added to the bridge and suddently 
> stop passing *only* arp replies as I see all other traffic just fine. I 
> also see those arp replies when in.3 isn't part of the bridge.  So my 
> fair assumption is that the arp reply packets never make it onto the hub 
> at all.
>
> As far as I can tell, they vanish between 'in' and the hub.
>
>   
>> I recall reading somewhere that if you add an ebtables rule to 'in',
>> which drops all VLAN tagged packets, it will then be redirected to the
>> correct sub-interface (ie in.3). I envisage it would be something like 
>>
>> Ebtables -t broute -A BROUTING -p 8021_Q -I in -J DROP
>>
>>   
>>     
> I will give that a try for sure.  From reading the post you refer to, it 
> makes a bit of sense, but I'm not entirely sure it covers the whole 
> problem.  A problem such as what it refers to seems like it would affect 
> all traffic instead of just arp replies.
>
> Thanks a bunch, I will update you on my results with that ebtables rule 
> shortly.
>
> Jonathan
>   
>> Have a good look at this post if you need further details:
>>
>>  http://osdir.com/ml/linux.drivers.vlan/2006-07/msg00011.html
>>
>>
>> Hope this helps.
>> Regards,
>>              Leigh
>>  
>> Leigh Sharpe
>> Network Systems Engineer
>> Pacific Wireless
>> Ph +61 3 9584 8966
>> Mob 0408 009 502
>> Helpdesk 1300 300 616
>> email lsharpe@pacificwireless.com.au
>> web www.pacificwireless.com.au
>>
>>  

Re: [Bridge] bridge, vlan and *no* stp/bpdu

[Malcolm Scott]

On Fri, 11 Apr 2008, Jonathan Thibault wrote:

> Apr 11 11:59:03 ordralphabetix IN=out OUT= MAC source =
> 00:18:18:62:3e:80 MAC dest = 08:10:17:0d:70:61 proto = 0x0806
>
> Why don't I see 'OUT=in.2'.  If the arp reply reached my host, it
> obviously had to go through the bridge.  That is in the 'out' interface
> and out the 'in.2' interface.

A hunch (which I haven't had time to verify yet): empty OUT= probably means 
the frame will be flooded to all output interfaces.

Someone who knows the code better than I do will no doubt be able to confirm 
or deny :)

Malcolm

-- 
Malcolm Scott
Research Assistant
University of Cambridge Computer Laboratory