Bridge, DNAT, New Tables and ip rules

Bridge, DNAT, New Tables and ip rules

[semi linux]

I've got a rather elaborate setup here that doesn't seem to be working
for me and I was hoping someone here might be able to shed some light
on the problem.

My server (Jose) is a bridged link between two hosts (Jack and Dan).
Jack's messages are NATed to the local bridge interface and then the
local interface will talk to Dan.  The trick here is that I never know
Jack's IP - only Dan's and software on Jose acts as an intermediary.
This has worked fine for quite a while, without problem using the
following rules:

iptables --append PREROUTING -t nat -d $DANS_IP -p tcp --dport
$DEST_PORT -j DNAT --to-destination $BRIDGE_IP:$DEST_PORT

iptables --append PREROUTING -t nat -s $DANS_IP -p tcp --sport
$DEST_PORT -j DNAT --to-destination $BRIDGE_IP

It's an odd setup, I know, but it works.

The problem comes-in when adding a new network card to my server.
Depending on network address, if eth0 ever has a connection problem,
Jack talks to Jose, Jose to Dan and Dan to Jose, but Jose back to Jack
never works.  It always trys to use eth0 for that communication.  The
packets out of eth0 have SRC=Dan and DEST=Jack.  The weird thing is, I
don't see these packets on the Jose<->Dan link... only coming out of
eth0.  How can I route them out BR0 instead of eth0?

I've tried marking the packets using mangle, sending them to a
different routing table but it doesn't seem to have any effect.

In this setup, we want to use eth0 for _everything_ except the traffic
we DNAT.  If eth0 is brought down, br0 should be used as a backup.

Any suggestions, hint, tips, etc?

I've followed Chapter 11 of the LARTC Howto without any luck.

- Gareth

Re: Bridge, DNAT, New Tables and ip rules

[Grant Taylor]

On 6/11/2007 5:35 PM, semi linux wrote:
> I've got a rather elaborate setup here that doesn't seem to be 
> working for me and I was hoping someone here might be able to shed 
> some light on the problem.

Will you please clarify, has this set up worked for you in the past and 
something has now changed?  Or are you trying to make it work for the 
first time?

> My server (Jose) is a bridged link between two hosts (Jack and Dan). 
> Jack's messages are NATed to the local bridge interface and then the 
> local interface will talk to Dan.  The trick here is that I never 
> know Jack's IP - only Dan's and software on Jose acts as an 
> intermediary.  This has worked fine for quite a while, without 
> problem using the following rules:

It sounds like (I'm guessing) that eth0 and eth1 are bridged together 
and have an bridge interface of br0.  Correct?

Are you wanting to DNAT any traffic (originally) going to Dan to the 
bridge IP?  In other words any traffic that was going to Dan DNAT it to 
the bridge IP where software running on the bridge will talk to Dan as a 
proxy (of sorts)?  To pull this off you have to match based on Dan's 
destination IP correct?

Do you want to DNAT any other traffic flowing through the bridge?  Is 
there any other traffic flowing through the bridge?

> iptables --append PREROUTING -t nat -d $DANS_IP -p tcp --dport 
> $DEST_PORT -j DNAT --to-destination $BRIDGE_IP:$DEST_PORT
> 
> iptables --append PREROUTING -t nat -s $DANS_IP -p tcp --sport 
> $DEST_PORT -j DNAT --to-destination $BRIDGE_IP

Depending on how your application is set up, I'm worried that your last 
rule could accidentally match the traffic that the application is trying 
to send back to the world thus causing the traffic to loop back in to 
the application.  Will you please help clarify this?

> It's an odd setup, I know, but it works.

I can sort of see how you would be doing this.  I'm just not sure why. 
But the why is immaterial.

> The problem comes-in when adding a new network card to my server. 
> Depending on network address, if eth0 ever has a connection problem, 
> Jack talks to Jose, Jose to Dan and Dan to Jose, but Jose back to 
> Jack never works.  It always trys to use eth0 for that communication. 
> The packets out of eth0 have SRC=Dan and DEST=Jack.  The weird thing 
> is, I don't see these packets on the Jose<->Dan link... only coming 
> out of eth0.  How can I route them out BR0 instead of eth0?

I think this supports what I was worried above, but I'm not sure.

Is Jose's IP address bound to eth0 or br0?

> I've tried marking the packets using mangle, sending them to a 
> different routing table but it doesn't seem to have any effect.

I don't think marking is needed here, but I may be wrong.

> In this setup, we want to use eth0 for _everything_ except the 
> traffic we DNAT.  If eth0 is brought down, br0 should be used as a 
> backup.

What interfaces are in br0 again?  You are also talking about adding 
NIC(s) to (I think) Jose.  When you do add NIC(s) are your existing 
interfaces getting their assignment bumped down, or is the new NIC going 
on the end of the list?

> Any suggestions, hint, tips, etc?

I feel like (rapid) spanning tree is going to be your friend.  What are 
you wanting to add the NIC(s) for?  Are you wanting to add NICs eth2 and 
eth3 so that eth0 and eth2 face Jack and eth1 and eth3 face Dam?  (Note: 
  I just chose to pair eth0 / eth2 and eth1 / eth3 together so that eth0 
and eth1 face what they were facing previously, just add more on the 
top.)  If this is indeed correct, I would recommend that you use (R)STP 
to control which links of eth0 / eth2 and eth1 / eth3 are in a blocking 
and forwarding state.  If your IP is bound to br0 STP should put eth2 
and eth3 in to blocking state and use eth0 and eth1 to carry the 
traffic.  If for some reason either of the links goes down, (R)STP 
should  realize this and transition the corresponding link to the one 
that went down in to forwarding state thus allowing traffic to continue 
flowing.

If you do not want to do this with (R)STP, you probably should look in 
to bonding eth0 / eth2 and eth1 / eth3 in to two separate bond0 and 
bond1 interfaces.  Depending on how bonding is set up, you can use load 
balancing for increased bandwidth, or pure fail over.

> I've followed Chapter 11 of the LARTC Howto without any luck.

(Like above, I don't think marking has a lot to do with this.)



Grant. . . .

Re: Bridge, DNAT, New Tables and ip rules

[semi linux]

On 6/11/07, Grant Taylor, wrote:
> On 6/11/2007 5:35 PM, semi linux wrote:
> > I've got a rather elaborate setup here that doesn't seem to be
> > working for me and I was hoping someone here might be able to shed
> > some light on the problem.
>
> Will you please clarify, has this set up worked for you in the past and
> something has now changed?  Or are you trying to make it work for the
> first time?

Yes, I've had this setup running for quite a while but when adding a
new ethernet card (on the same or different networks) I get a problem.

> > My server (Jose) is a bridged link between two hosts (Jack and Dan).
> > Jack's messages are NATed to the local bridge interface and then the
> > local interface will talk to Dan.  The trick here is that I never
> > know Jack's IP - only Dan's and software on Jose acts as an
> > intermediary.  This has worked fine for quite a while, without
> > problem using the following rules:
>
> It sounds like (I'm guessing) that eth0 and eth1 are bridged together
> and have an bridge interface of br0.  Correct?

Actually, I've renamed two ports on a dual-port card to be eth50 and
eth51 (done using udev rules) and they have a bridge interface of br0.

> Are you wanting to DNAT any traffic (originally) going to Dan to the
> bridge IP?  In other words any traffic that was going to Dan DNAT it to
> the bridge IP where software running on the bridge will talk to Dan as a
> proxy (of sorts)?  To pull this off you have to match based on Dan's
> destination IP correct?

Spot-on.  My intermediary software is designed to traffic destined for
Dan on a given port.  That traffic hits the bridge and is NATed to the
local (br0) IP address and port.  It's a DNAT rewriting the
destination IP and port if the incoming message is from Jack.  If the
message is from Dan, it only rewrites the destination IP.

> Do you want to DNAT any other traffic flowing through the bridge?  Is
> there any other traffic flowing through the bridge?

All other traffic flows just like normal through the bridge.

> > iptables --append PREROUTING -t nat -d $DANS_IP -p tcp --dport
> > $DEST_PORT -j DNAT --to-destination $BRIDGE_IP:$DEST_PORT
> >
> > iptables --append PREROUTING -t nat -s $DANS_IP -p tcp --sport
> > $DEST_PORT -j DNAT --to-destination $BRIDGE_IP
>
> Depending on how your application is set up, I'm worried that your last
> rule could accidentally match the traffic that the application is trying
> to send back to the world thus causing the traffic to loop back in to
> the application.  Will you please help clarify this?

The second rule is in place just in case Dan initiates conversation,
instead of Jack.  When the source is local, wouldn't the outgoing
traffic be processed as follows?:

program -> routing decision -> mangle::output, nat::output,
filter::output, mangle::postrouting, nat::postrouting, interface,
wire.

Therefore it'd never hit the nat::prerouting (or _any_ ::prerouting
rules), right?

> > It's an odd setup, I know, but it works.
>
> I can sort of see how you would be doing this.  I'm just not sure why.
> But the why is immaterial.

Yeah, and trust me, you don't want to go there... if you think this is
odd, the reasons are doubly-so.

> > The problem comes-in when adding a new network card to my server.
> > Depending on network address, if eth0 ever has a connection problem,
> > Jack talks to Jose, Jose to Dan and Dan to Jose, but Jose back to
> > Jack never works.  It always trys to use eth0 for that communication.
> > The packets out of eth0 have SRC=Dan and DEST=Jack.  The weird thing
> > is, I don't see these packets on the Jose<->Dan link... only coming
> > out of eth0.  How can I route them out BR0 instead of eth0?
>
> I think this supports what I was worried above, but I'm not sure.
>
> Is Jose's IP address bound to eth0 or br0?

Jose has two IP address, eth0 and br0... they could be on the same
subnet or different subnets (depending on install details).

This is the crux of the problem, let me try to clarify... Jose does
talk to Jack, but it's through the wrong interface (eth0 instead of
br0 (eth50/eth51)).  The packets that are coming out of eth0 are the
proper responses, with Dan is listed as the source and Jack is the
destination.  The question is, w/o knowing Jack's IP how do I route
them through br0?

> > I've tried marking the packets using mangle, sending them to a
> > different routing table but it doesn't seem to have any effect.
>
> I don't think marking is needed here, but I may be wrong.

I was pointed in that direction by the good folks over on the Fedora
mailing list but I'm all ears to try anything here and have no problem
testing _sny_ suggestions.

> > In this setup, we want to use eth0 for _everything_ except the
> > traffic we DNAT.  If eth0 is brought down, br0 should be used as a
> > backup.
>
> What interfaces are in br0 again?  You are also talking about adding
> NIC(s) to (I think) Jose.  When you do add NIC(s) are your existing
> interfaces getting their assignment bumped down, or is the new NIC going
> on the end of the list?

br0 - eth50/51 - bridged.
eth0,1,2,3,etc... independent.
New NIC are brought-up in a typical fashion... added, with default gateway, etc.

> > Any suggestions, hint, tips, etc?
>
> I feel like (rapid) spanning tree is going to be your friend.  What are
> you wanting to add the NIC(s) for?  Are you wanting to add NICs eth2 and
> eth3 so that eth0 and eth2 face Jack and eth1 and eth3 face Dam?  (Note:
>   I just chose to pair eth0 / eth2 and eth1 / eth3 together so that eth0
> and eth1 face what they were facing previously, just add more on the
> top.)  If this is indeed correct, I would recommend that you use (R)STP
> to control which links of eth0 / eth2 and eth1 / eth3 are in a blocking
> and forwarding state.  If your IP is bound to br0 STP should put eth2
> and eth3 in to blocking state and use eth0 and eth1 to carry the
> traffic.  If for some reason either of the links goes down, (R)STP
> should  realize this and transition the corresponding link to the one
> that went down in to forwarding state thus allowing traffic to continue
> flowing.
>
> If you do not want to do this with (R)STP, you probably should look in
> to bonding eth0 / eth2 and eth1 / eth3 in to two separate bond0 and
> bond1 interfaces.  Depending on how bonding is set up, you can use load
> balancing for increased bandwidth, or pure fail over.
>
> > I've followed Chapter 11 of the LARTC Howto without any luck.

I'm guess with the information I've provided above, you're going to
suggest something different... I've already looked into bonding and
STP... even adding eth0 to the bridge, none of those solutions seem to
do the trick.  Let me know if I should reconsider some of these in
light of the above.

- Gareth

Re: Bridge, DNAT, New Tables and ip rules

[Grant Taylor]

On 6/12/2007 2:12 PM, semi linux wrote:
> Yes, I've had this setup running for quite a while but when adding a 
> new ethernet card (on the same or different networks) I get a 
> problem.

Ok, I just had to ask.

> Actually, I've renamed two ports on a dual-port card to be eth50 and 
> eth51 (done using udev rules) and they have a bridge interface of 
> br0.

Do you really have that many interfaces, or are you just skipping a 
bunch of interfaces?

> All other traffic flows just like normal through the bridge.

*nod*

> The second rule is in place just in case Dan initiates conversation, 
> instead of Jack.  When the source is local, wouldn't the outgoing 
> traffic be processed as follows?:

Does this rule ever match any packets?

> program -> routing decision -> mangle::output, nat::output, 
> filter::output, mangle::postrouting, nat::postrouting, interface, 
> wire.

Sorry, with my current state of mind, I can't respond to this.

> Therefore it'd never hit the nat::prerouting (or _any_ ::prerouting 
> rules), right?

(See above.)

> Jose has two IP address, eth0 and br0... they could be on the same 
> subnet or different subnets (depending on install details).

Hum.

> This is the crux of the problem, let me try to clarify... Jose does 
> talk to Jack, but it's through the wrong interface (eth0 instead of 
> br0 (eth50/eth51)).  The packets that are coming out of eth0 are the 
> proper responses, with Dan is listed as the source and Jack is the 
> destination.  The question is, w/o knowing Jack's IP how do I route 
> them through br0?

Baring in mind that (by default) Linux will (primarily) use one 
interface on a subnet unless you do something to alter it.  To this end 
I think you will need to match based on Dan's IP be it source or 
destination.

> I was pointed in that direction by the good folks over on the Fedora 
> mailing list but I'm all ears to try anything here and have no 
> problem testing _sny_ suggestions.

I'm still not convinced that you need to mark the packets.  In my 
opinion it is so much easier to match the source or destination IP.

> br0 - eth50/51 - bridged. eth0,1,2,3,etc... independent. New NIC are 
> brought-up in a typical fashion... added, with default gateway, etc.

Ok, I feel like I'm missing your config.  Will you please list out your 
interfaces (logical and physical) as well as subnets.  Granted the 
subnets can be a.b.c.x, d.e.f.x, g.h.i.x, etc.

> I'm guess with the information I've provided above, you're going to 
> suggest something different... I've already looked into bonding and 
> STP... even adding eth0 to the bridge, none of those solutions seem 
> to do the trick.  Let me know if I should reconsider some of these in 
> light of the above.

You will probably have to use custom routing tables including the tables 
including link addresses.



Grant. . . .

Bridge, DNAT, New Tables and ip rules

[semi linux]

Sorry, I sent this in HTML first by mistake... needed to switch to a
block font to do the ASCII art.

On 6/13/07, Grant Taylor <gtaylor@riverviewtech.net> wrote:
 > Do you really have that many interfaces, or are you just skipping a
> bunch of interfaces?

Oh, I'm skipping a bunch... 50 and 51 just denote the interfaces we
always use for the bridge.

> Does this rule ever match any packets?

 Not in my current setup, but I think it's for an alternate setup, in
any case, I'm not worried about it right now.

> Baring in mind that (by default) Linux will (primarily) use one
> interface on a subnet unless you do something to alter it.  To this end
> I think you will need to match based on Dan's IP be it source or
> destination.

Hrmm, that _might_ be it but I'm not convinced.  There are three
conditions which need to be satisfied:
 - bridge traffic is bridged w/o interruption
- all traffic from localhost uses eth0
- traffic from one port on the localhost uses br0 instead of eth0

> I'm still not convinced that you need to mark the packets.  In my
> opinion it is so much easier to match the source or destination IP.

 Ugg, I hate ASCII art, but here go my Picasso skills...

  ______             ______             _____
|      |-->--1-->--|      |-->--2-->--|     |
| Jack |           | Jose |           | Dan |
|______|           |______|--<--3--<--|_____|
                     |  |
                      |  |
                     5  4
                      |  |
                     V  V

This makes you appreciate whiteboards on a whole new level.

Assuming eth0 and br0 are on the same network, this is what I'm seeing:
 1 - packets from Jack flow in on the bridge (eth50).
2 - packets from localhost go out on the bridge (eth51).
3 - response is received from Dan on the bridge (eth51).
 4 - packets from localhost match both the eth0 and br0 networks, eth0
takes the cake and the packet is sent out the wrong interface.
5 - all other traffic.

 All packets from me, destined to that network should use eth0, with
the exception of packets like "4" (as described above).  Those packets
should use br0.  We've tried modifying our software to force an
interface but it looks like Java under Linux is incapable - the
packets seem to hit the routing rules anyway.  I only know the source
port of these packets; SRC IP is going to be one of mine, DST IP is
unknown (except to the software) and DST port is random (per
connection).

> Ok, I feel like I'm missing your config.  Will you please list out your
> interfaces (logical and physical) as well as subnets.  Granted the
> subnets can be a.b.c.x, d.e.f.x, g.h.i.x, etc.

 Sure -

JACK -
IP: 37.2.18.205 / 255.255.254.0
GW: 37.2.18.1 / 255.255.254.0
Sends 1, expects 4 in diagram above.

DAN -
IP: 72.2.2.80 / 255.255.254.0
GW: 72.2.2.1 /  255.255.254.0
Receives 2, Sends 3.

JOSE -
BR0: 72.2.2.92 / 255.255.254.0
GW: 72.2.2.1 /  255.255.254.0
Receives 1, Sends 2, Receives 3.

ETH0: 72.2.4.109  / 255.255.254.0
GW: 72.2.4.1 / 255.255.254.0
Sends 4

In this scenario, ETH0 in Jose is on a different subnet.  It is
feasible that it is on the same subnet as DAN/BR0.

Jose's IPTables:
[root@SES network-scripts]# iptables -L -t nat
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
DNAT       tcp  --  anywhere             72.2.2.80             tcp
dpt:5000 to: 72.2.2.92:2037
DNAT       tcp  --  72.2.2.80            anywhere              tcp
spt:5000 to:72.2.2.92

***** All other chains are empty *****


Jose's routing tables:
You'll find that these are pretty-much the default routing tables:

[root@SES network-scripts]# ip route list table local
local 72.2.4.109  dev eth0  proto kernel  scope host  src 72.2.4.109
local 72.2.2.92 dev br0  proto kernel  scope host  src 72.2.2.92
broadcast  127.255.255.255 dev lo  proto kernel  scope link  src 127.0.0.1
broadcast 72.2.4.0 dev eth0  proto kernel  scope link  src  72.2.4.109
broadcast 72.2.5.255 dev eth0  proto kernel  scope link  src 72.2.4.109
broadcast  127.0.0.0 dev lo  proto kernel  scope link  src 127.0.0.1
broadcast 72.2.2.0 dev br0  proto kernel  scope link  src 72.2.2.92
broadcast 72.2.3.255 dev br0  proto kernel  scope link  src 72.2.2.92
local 127.0.0.1 dev lo  proto kernel  scope host  src  127.0.0.1
local 127.0.0.0/8 dev lo  proto kernel  scope host  src 127.0.0.1

[root@JOSE network-scripts]# ip route list table main
72.2.2.80 via 72.2.2.92 dev br0
72.2.4.0/23 dev eth0  proto kernel  scope link  src 72.2.4.109
72.2.2.0/23 dev br0  proto kernel  scope link  src 72.2.2.92
169.254.0.0/16 dev eth0  scope link
default via  72.2.4.1 dev eth0


OR in the old-fashion style:

[root@JOSE network-scripts]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
72.2.4.0        *               255.255.254.0   U     0      0        0 eth0
72.2.2.0        *                255.255.254.0   U     0      0        0 br0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth0
default          72.2.4.1        0.0.0.0         UG    0      0        0 eth0


[root@JOSE network-scripts]# ip route list table default
[root@JOSE network-scripts]#


> You will probably have to use custom routing tables including the tables
> including link addresses.

 I'm not sure what this means...  Since I want all traffic to go to
eth0, except for the traffic with a given source port, how are the
routing tables going to help me?

I did notice one interesting thing last night...  My Cisco RV042
that's used as a router in the middle is sending a lot of Gratuitous
ARP packets for an interface which itself owns... I'm guessing this is
a firmware bug of some kind and probably wouldn't affect my setup
anyway so I'm going to ignore it.

- Gareth

Re: Bridge, DNAT, New Tables and ip rules

[Grant Taylor]

On 6/15/2007 2:04 PM, semi linux wrote:
> Hrmm, that _might_ be it but I'm not convinced.  There are three 
> conditions which need to be satisfied: 
> - bridge traffic is bridged w/o interruption
> - all traffic from localhost uses eth0
> - traffic from one port on the localhost uses br0 instead of eth0

Having a list of criteria makes it a lot easier to understand what you 
are wanting to do, or at least how to come up with something to fulfill 
your goal(s).

Just based on your above description, here is what I would try to do.
  - Enslave eth50 & eth51 to br0
  - Bind an IP to br0
  - Bind an IP to eth0
  - Bind IPs as necessary to any other interfaces.
  - Use dev eth0 and eth0s source IP to talk to the local network.
  - Use dev br0 and br0s source IP for the exception above.

I *THINK* this can be accomplished with ip rules.

Use the NATing or Redirection on the bridged traffic to bring the 
traffic you want to effect in to Jose.  Have Jose do what you want as 
far as communicating with Dan.  Have all traffic that Jose send out go 
out via eth0 based on your ip table entries.  Use an ip rule to match 
the specific traffic you want to send out br0 to use a different routing 
table that is set to use dev br0 with br0s source IP for the specific 
traffic.

> Ugg, I hate ASCII art, but here go my Picasso skills...
>  ______             ______             _____
> |      |-->--1-->--|      |-->--2-->--|     |
> | Jack |           | Jose |           | Dan |
> |______|           |______|--<--3--<--|_____|
>                      |  |
>                      |  |
>                      5  4
>                      |  |
>                      V  V
> 
> This makes you appreciate white boards on a whole new level.

(Not bad.)

> I'm not sure what this means...  Since I want all traffic to go to 
> eth0, except for the traffic with a given source port, how are the 
> routing tables going to help me?

Do some reading on how the LARTC guide(s) suggest you deal with multiple 
internet connections.  In short, you are wanting by default all traffic 
to use eth0 with only the exception traffic to use br0.  I think you 
will see the custom routing tables and how to write ip rules to tell the 
system to use them.

> I did notice one interesting thing last night...  My Cisco RV042 
> that's used as a router in the middle is sending a lot of Gratuitous 
> ARP packets for an interface which itself owns... I'm guessing this 
> is a firmware bug of some kind and probably wouldn't affect my setup 
> anyway so I'm going to ignore it.

Are your bridged ethernet ports, eth50 and eth51, both facing the RVO42? 
  If they are this could be a sign that the RVO42 is getting confused by 
seeing its own traffic coming back in to its self.  If the RVO42 is 
confused it may be GARPing to try to avoid ARP poisoning by preemptively 
using ARP poisoning to keep things running.



Grant. . . .