[LARTC] What I learned about Linux bridging

[LARTC] What I learned about Linux bridging

[Greg Scott]

Here are some notes I have about Linux bridging.  I'll try to separate
what I know I know from what I think I know.  

Let's say I want to bridge eth0, eth1, and eth2 together, all with an IP
Address of, say, 1.2.3.2.  This is how to do it:
 
echo "Setting up br0 to bridge eth0 with eth1 and eth2"
 
/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth0
/usr/sbin/brctl addif br0 eth1
/usr/sbin/brctl addif br0 eth2

/sbin/ip addr add 1.2.3.2/24 dev br0
/sbin/ip link set br0 up

Continuing with the above example, most of the writeups also say to
remove any IP Addresses from eth0, eth1, and eth2.  But I've found this
doesn't seem necessary - well, sort of.  

Let's say that eth0 is at IP Address 1.2.3.2, and now I bridge eth0,
eth1, and eth2 together and give bridge br0 the same  IP Address of
1.2.3.2.  Now I have a mess because both eth0 and br0 have the same IP
Address.  Doing this:

	ip addr del 1.2.3.2/24 dev eth0

cleans up the mess.  

But let's say that physical interface eth1 has IP Address 10.0.0.1.
From testing, it looks like other systems can ping 10.0.0.1 just fine,
assuming they have a route to it.  So I  **think**  I know that I can
assign an IP Address to a raw interface, as long as it's a different IP
Address than what I assigned to the overall bridge.  But I haven't seen
this capability documented anywhere.  

Let's say the bridge is up and working at IP Address 1.2.3.2.  I have a
system at IP Address 1.2.3.1 connected via eth0.  That system can ping
1.2.3.2 easily.  If I disconnect the Ethernet cable from eth0 and plug
into eth1 or eth2, after about 30 seconds, that bridged system begins
answering pings again.  As indicated in the writeups, that spare PC with
a bunch of NICs is now acting like a managed Ethernet switch.  Cool!

Filtering

iptables is a super-sophisticated toolset to filter IP packets.
ebtables is another toolset to filter at the OSI layer 2 (datalink)
layer.  iptables concerns itself (mostly) with routing across an IP
network, computer to computer.  ebtables concerns itself (mostly) with
filtering packets across physical NIC interfaces in the same computer.

Here is a great writeup on using ebtables and iptables together:
http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html
But - like everything I've been able to find so far, I don't think this
writeup is completely accurate.  

iptables has a module called physdev.  According to the writeup, I can
use the iptables physdev module to filter among the raw interfaces in a
bridge.  But a discussion in the netfilter list essentially says that
physdev is being removed because it creates all kinds of other problems.
At least, I think that's what it says.  The relevant discussion took
place in early July 2006.  Here is a pointer to the beginning of the
discussion:
https://lists.netfilter.org/pipermail/netfilter-devel/2006-July/024896.h
tml

So it looks like when filtering at the network layer, (IP in this case)
use iptables.  When filtering at the data link layer, use ebtables and
maybe arptables.  Avoid using -m physdev in iptables because it's going
away.  You can add IP Addresses to bridged eth-- interfaces as long as
they don't conflict with the bridge IP Address(es).  

Next up will be to try some filtering scenarios with ebtables and
iptables.  

- Greg Scott

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] What I learned about Linux bridging

[Grant Taylor]

On 6/6/2007 6:51 PM, Greg Scott wrote:
> Continuing with the above example, most of the write ups also say to 
> remove any IP Addresses from eth0, eth1, and eth2.  But I've found 
> this doesn't seem necessary - well, sort of.

No, you don't have to remove IP addresses from the bridge member 
interfaces, though it does make things (IMHO) cleaner and consistent.

You could put one IP address on each of the (physical) bridge member 
interfaces, or you could put the IP addresses (as aliases) on the bridge 
interface.

Although I'm not sure how well EBTables would be able to filter 
addresses on a raw interface verses on the bridge interface.

> Let's say that eth0 is at IP Address 1.2.3.2, and now I bridge eth0, 
> eth1, and eth2 together and give bridge br0 the same  IP Address of 
> 1.2.3.2.  Now I have a mess because both eth0 and br0 have the same 
> IP Address.  Doing this:

Yes, I would consider an IP address conflict a mess.

> But let's say that physical interface eth1 has IP Address 10.0.0.1. 
> From testing, it looks like other systems can ping 10.0.0.1 just 
> fine, assuming they have a route to it.  So I  **think**  I know that 
> I can assign an IP Address to a raw interface, as long as it's a 
> different IP Address than what I assigned to the overall bridge.  But 
> I haven't seen this capability documented anywhere.

Just because you can does not mean that you should.  Most people that 
are working (read:  using in production) bridging would not do so unless 
they had a need to do so.  If you do not need bridging, generally it is 
not used.  Thus if you can get by with putting the IP address(es) on the 
physical interfaces, then usually you will not be using bridging.  This 
does not mean that it can not be done.

> Let's say the bridge is up and working at IP Address 1.2.3.2.  I have 
> a system at IP Address 1.2.3.1 connected via eth0.  That system can 
> ping 1.2.3.2 easily.  If I disconnect the Ethernet cable from eth0 
> and plug into eth1 or eth2, after about 30 seconds, that bridged 
> system begins answering pings again.  As indicated in the write  ups, 
> that spare PC with a bunch of NICs is now acting like a managed
> Ethernet switch.  Cool!

In Spanning Tree Protocol, this is considered the listening / learning 
delay to prevent loops in the network.  This delay should be tunable 
(I'm not 100% certain of this fact).

> iptables is a super-sophisticated toolset to filter IP packets. 
> ebtables is another toolset to filter at the OSI layer 2 (datalink) 
> layer.  iptables concerns itself (mostly) with routing across an IP 
> network, computer to computer.  ebtables concerns itself (mostly) 
> with filtering packets across physical NIC interfaces in the same 
> computer.

Close, but not quite.

(I'll preface this paragraph by saying that by default, this is how 
IPTables behaves, as it can be changed.)

IPTables is a GREAT filtering framework that operates on OSI Layer 3 and 
higher (match extensions and what not).  EBTables is a VERY GOOD 
filtering framework that operates on OSI Layer 2.  I.e. one filters in 
the IP stack and the other filters in the bridging code (respectively). 
  Granted both IPTables and EBTables can filter on some things like 
source / destination IP address, IP protocol, TCP / UDP port, MAC 
address, and a few other things.  However IPTables has access to a LOT 
more information than EBTables does.

Now, for the non default config.  You can turn on "Bridge Netfilter" 
code in the kernel to allow the bridging code to use IPTables in 
addition to EBTables.  What this means is that you can use all of 
IPTables matches extensions to make OSI Layer 3 and higher decisions on 
OSI Layer 2.  I.e. you can have a bridging firewall do just about any 
thing that you want it to do.

> Here is a great writeup on using ebtables and iptables together: 
> http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html But - like 
> everything I've been able to find so far, I don't think this writeup 
> is completely accurate.

(It has been too long since I read that.  Please refresh my memory.) 
What about that write up do you think is in-accurate?  (I'll re-read the 
parts you point out.)  I'll see if I agree with you or the write up and 
if so try to help explain why.

> iptables has a module called physdev.  According to the writeup, I 
> can use the iptables physdev module to filter among the raw 
> interfaces in a bridge.  But a discussion in the netfilter list 
> essentially says that physdev is being removed because it creates all 
> kinds of other problems.  At least, I think that's what it says.  The 
> relevant discussion took place in early July 2006.  Here is a pointer 
> to the beginning of the discussion: 
> https://lists.netfilter.org/pipermail/netfilter-devel/2006-July/024896.html
> 
> So it looks like when filtering at the network layer, (IP in this 
> case) use iptables.  When filtering at the data link layer, use 
> ebtables and maybe arptables.  Avoid using -m physdev in iptables 
> because it's going away.  You can add IP Addresses to bridged eth-- 
> interfaces as long as they don't conflict with the bridge IP 
> Address(es).

With out reading too far in to it, it looks like the existing physdev 
match will be deprecated.  I however do not think this will be a 
permanent thing.  I believe that either a re-worked physdev, or 
something like it will re-appear in due time.

Also keep in mind that there is a LOT of re-working going on in the 
later 2.6.x kernels, including renaming variables and changing the 
internal workings.  (Too much so if you ask me for a .even release.)

> Next up will be to try some filtering scenarios with ebtables and 
> iptables.

...




Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] What I learned about Linux bridging

[Greg Scott]

> Although I'm not sure how well EBTables would be able 
> to filter addresses on a raw interface verses on the 
> bridge interface.

I ran some tests and it seemed to work.  

Here is a little test script:

#!/bin/sh

/firewall-scripts/allow-all

EBTABLES="/usr/local/sbin/ebtables"
IPTABLES="/sbin/iptables"

$EBTABLES -F

$EBTABLES -A INPUT -i eth0 -j DROP
$EBTABLES -A INPUT -i eth1 -j ACCEPT
$EBTABLES -A INPUT -i eth2 -j ACCEPT

#$IPTABLES -A FORWARD -i eth0 -j DROP
#$IPTABLES -A FORWARD -i eth1 -j ACCEPT
#$IPTABLES -A FORWARD -i eth2 -j ACCEPT

#$IPTABLES -A INPUT -i eth0 -j DROP
#$IPTABLES -A INPUT -i eth1 -j ACCEPT
#$IPTABLES -A INPUT -i eth2 -j ACCEPT

[root@IXFactor-fw gregs]# 

The IP Address for br0 is 1.2.3.2.  I setup a system at 1.2.3.1 and had
it ping 1.2.3.2.  Everything behaved as expected.  With bridging turned
on, none of the iptables rules made any difference, so I commented them
out.  With the cable plugged into eth0, none of the pings replied.
Tcpdump on the 1.2.3.2 box didn't even show anything coming into br0.
Connecting to eth1 or eth2, pings replied and tcpdump showed the ICMP
going back and forth on br0.  Setting br0 to 1.2.3.2 and eth1 to
10.0.0.1, my external host could ping both IP Addresses when I fudged in
appropriate routes on my test system.  

Here are a couple of my challenges.   

I want to block anything coming in from the Internet claiming to come
from 10/8, or 172.16/12 or 192.168/16.  But anything from these address
blocks from any trusted eth-- interface should be OK.  Easy to do with
pure routing, seems a little more challenging with bridging.  I think I
can whip up some rules, but they depend on ebtables filtering by
physical eth-- port.  

I have a public /24 block, call it 1.2.3.nnn.  Some of these are on
eth0, some on eth1, a few others on eth2.  Bridging seems to handle this
nicely.  eth0 faces the Internet.  I am a little nervous in packaging
all this.  Let's say something goes haywire in startup, before I run the
brctl stuff.  Or let's say something bad happens and bridging goes
haywire.  If eth0 has no IP Address, I have no remote access to the box.
So I'd like to at least give an IP Address to eth0.  This  seems to
work, but like I said, it's not documented to work.  It just hit me
what's bugging me - it bugs me to give an IP Address to a logical device
and cut off any kind of fallback access to the physical device.  Maybe
I'm looking for trouble that's not there and creating problems I don't
need just because bridging is so new in my little world.  I wonder what
others have done in this situation?

And then there's my PPTP clients.  I give these guys a 10.0.0.xxx IP
Address with a gateway of 10.0.0.1.  This was all  part of the eth1 LAN
when it was a pure router.  I suppose in this bridging setup, I could
make 10.0.0.1 an alias for br0 and leave eth1 with no IP Address.  This
just takes a little getting used to I guess.  

- Greg

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] What I learned about Linux bridging

[Grant Taylor]

On 6/7/2007 12:12 AM, Greg Scott wrote:
> The IP Address for br0 is 1.2.3.2.  I setup a system at 1.2.3.1 and 
> had it ping 1.2.3.2.  Everything behaved as expected.  With bridging 
> turned on, none of the iptables rules made any difference, so I 
> commented them out.  With the cable plugged into eth0, none of the 
> pings replied.  Tcpdump on the 1.2.3.2 box didn't even show anything 
> coming into br0.  Connecting to eth1 or eth2, pings replied and 
> tcpdump showed the ICMP going back and forth on br0.  Setting br0 to 
> 1.2.3.2 and eth1 to 10.0.0.1, my external host could ping both IP 
> Addresses when I fudged in appropriate routes on my test system.

Ok, good to know.

The only other thing that I can think of why you might bind your IP 
address(es) to the physical interface verses the bridge interface is if 
you were not wanting to bridge IP, but some other non routeable 
protocol, say NetBEUI.  (But why would you want to do that???)  In this 
scenario, it would be perfectly normal to use a bridging router to 
bridge any thing but IP and route IP.

> Here are a couple of my challenges.
> 
> I want to block anything coming in from the Internet claiming to come 
> from 10/8, or 172.16/12 or 192.168/16.  But anything from these 
> address blocks from any trusted eth-- interface should be OK.  Easy  
> to do with pure routing, seems a little more challenging with 
> bridging.  I think I can whip up some rules, but they depend on 
> ebtables filtering by physical eth-- port.

You should be able to easily do this with a few EBTables rules.  Just 
check the source IP of the packets as they come in eth0.

ebtables -t filter -N bogonsFromNet -P RETURN
ebtables -t filter -A bogonsFromNet -s 10.0.0.0/8 -j DROP
ebtables -t filter -A bogonsFromNet -s 172.16.0.0/12 -j DROP
ebtables -t filter -A bogonsFromNet -s 192.168.0.0/16 -j DROP

ebtables -t filter -A INPUT -j bogonsFromNet
ebtables -t filter -A FORWARD -j bogonsFromNet

(Note:  This is untested.)

If I have things correct, this should be a simple chain that checks the 
source IP and blocks drops them if they match.  The (new) chain its self 
has a default policy of RETURN to return back to the chain that jumped 
to it.

You might want to expand your bogons list a bit more to include some 
more forbidden networks.  I.e. Test Net 192.0.2.x/24...  RFC 3330 is 
your friend in this matter.

> I have a public /24 block, call it 1.2.3.nnn.  Some of these are on 
> eth0, some on eth1, a few others on eth2.  Bridging seems to handle 
> this nicely.  eth0 faces the Internet.  I am a little nervous in 
> packaging all this.  Let's say something goes haywire in startup, 
> before I run the brctl stuff.  Or let's say something bad happens and 
> bridging goes haywire.  If eth0 has no IP Address, I have no remote 
> access to the box.  So I'd like to at least give an IP Address to 
> eth0.  This  seems to work, but like I said, it's not documented to 
> work.  It just hit me what's bugging me - it bugs me to give an IP 
> Address to a logical device and cut off any kind of fallback access 
> to the physical device.  Maybe I'm looking for trouble that's not 
> there and creating problems I don't need just because bridging is so 
> new in my little world.  I wonder what others have done in this 
> situation?

I don't think you are being too paranoid, but I think you should be 
aware of something I have run in to in my own testing.

If I have eth0 up with an IP address w.x.y.z and I am connected to it 
(via ssh) and I enslave it to a bridge (bri0) (I like 3 letter / 1 
number device names) my connection goes dead.  I have to try 
reconnecting or log in to console and move the IP to the bridge.  Well, 
at least I think that's what I have to do, it's been too long sense I 
last did it and I don't remember the details.  You may want to play with 
this.  The reason that I bring it up is that I think an IP address on 
the ether interface is different than an IP address on the enslaved 
ether interface.  Your mileage may vary.  What I'm worried about is that 
if the bridging code is seeing the traffic before the system's IP stack 
sees it (supported by the fact that you can use EBTables to filter it), 
you are still passing through the bridging code even with the IP address 
on the enslaved ether interface.

I have personally used bridging extensively for the past 4+ years and 
I've been ABSOLUTELY THRILLED with it.  I have some systems with four 
interfaces bridged together with all IPs on the bri0 interface.  I have 
another system that is 802.1q trunking to a Layer 2 managed switch with 
25+ VLAN interfaces on the trunk.  I'm then bridging the 25+ VLANs plus 
the enslaved ether interface in one bridge with the IP bound to bri0.  I 
then have EBTables dividing up what access each VLAN has.  Specifically, 
each VLAN can communicate with the bri0 interface and the bri0 interface 
can communicate with all VLANs, but the VLANs can not communicate with 
each other.  I even threw in ARP tables and EBTables to masquerade MAC 
addresses b/c the managed switch that I was working with could only see 
any given MAC on one VLAN.  Let's just say this was fun to set up.  In 
the end, the system has been up and running with out problems at multi 
megabit speeds for 3+ years.

> And then there's my PPTP clients.  I give these guys a 10.0.0.xxx IP 
> Address with a gateway of 10.0.0.1.  This was all  part of the eth1 
> LAN when it was a pure router.  I suppose in this bridging setup, I 
> could make 10.0.0.1 an alias for br0 and leave eth1 with no IP 
> Address.  This just takes a little getting used to I guess.

I don't think you should be scared of bridging and / or EBTables.  With 
them you can do a LOT of things you could not do other wise.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] What I learned about Linux bridging

[Greg Scott]

>> I want to block anything coming in from the Internet claiming to come

>> from 10/8, or 172.16/12 or 192.168/16.  

> You should be able to easily do this with a few EBTables rules. 

Yup - I was off putting together something similar to what you did and
just now saw your reply.  It all tested good.  I'll paste it in below:

#
# ebtables rules for bridging
#
echo "ebtables rules"
echo "	Directing anything between the Internet and private IP Addresses
to the bogus_ip chain"

$EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-src 10.0.0.0/8 -j
bogus_ip
$EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-src 172.16.0.0/12 -j
bogus_ip
$EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-src 192.168.0.0/16 -j
bogus_ip
$EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-dst 10.0.0.0/8 -j
bogus_ip
$EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-dst 172.16.0.0/12 -j
bogus_ip
$EBTABLES -A FORWARD -i $INET_IFACE -p IPv4 --ip-dst 192.168.0.0/16 -j
bogus_ip

$EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-src 10.0.0.0/8 -j
bogus_ip
$EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-src 172.16.0.0/12 -j
bogus_ip
$EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-src 192.168.0.0/16 -j
bogus_ip
$EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-dst 10.0.0.0/8 -j
bogus_ip
$EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-dst 172.16.0.0/12 -j
bogus_ip
$EBTABLES -A INPUT -i $INET_IFACE -p IPv4 --ip-dst 192.168.0.0/16 -j
bogus_ip

#
# Set up the bogus_ip chain to log and drop packets to/from private IP
addresses
#

echo "Setting up the bogus_ip chain to LOG and DROP spoofed packets"

$EBTABLES -A bogus_ip --log-prefix " spoofed packet"
$EBTABLES -A bogus_ip -j DROP


I might see if I can do this with one set of rules in the PREROUTING
chain.


> I don't think you are being too paranoid, but 
> I think you should be aware of something I have 
> run in to in my own testing. . . .

Yup - I noticed similar behavior.  So this is how I'll handle it.  eth0
gets an IP Address of 1.2.3.6 during normal bootup.  And then when I do
the brctl stuff, br0 gets 1.2.3.2.  That way there's never a conflict
between the physical and logical.  All the physical interfaces have
unique addresses so I can route based on the address when it makes
sense, or bridge based on the interface when that makes sense.  I'm
feeling lots better about all this.  Hopefully this discussion can help
others out there.  

- Greg

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] What I learned about Linux bridging

[Grant Taylor]

On 06/07/07 01:55, Greg Scott wrote:
> I might see if I can do this with one set of rules in the PREROUTING
>  chain.

That is exactly why I put the rules in a new chain and jumped to the 
chain.  One place to maintain them.  With the ability to jump to the 
chain and then return to the chain that you jumped from, things act more 
like a sub-routine in programming.

  - Create new chains with a default policy of RETURN.
  - DROP on failures with in the new chain.
  - Default policy of RETURN will return packet flow to the calling chain.
  - Jump to the new chain from INPUT and return from it if everything is ok.
  - Jump to the new chain from FORWARD and return from it if everything 
is ok.

In fact, you could use the new chain as a (sub)chain from any chain with 
in the filter table.

> Yup - I noticed similar behavior.  So this is how I'll handle it. 
> eth0 gets an IP Address of 1.2.3.6 during normal bootup.  And then 
> when I do the brctl stuff, br0 gets 1.2.3.2.  That way there's never 
> a conflict between the physical and logical.  All the physical 
> interfaces have unique addresses so I can route based on the address 
> when it makes sense, or bridge based on the interface when that makes 
> sense.  I'm feeling lots better about all this.  Hopefully this 
> discussion can help others out there.

This is one way to do it.  Just be careful of race conditions while you 
are bringing things up.



Grant. . . .
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc