Fairly complex multi-ISP firewall/router problem

Fairly complex multi-ISP firewall/router problem

[Bill Davidsen]

I am trying to set up a single Linux router, RH9.0, for a non-profit I 
am supporting with some free consulting. They have two ISP lines, each 
of which has a three bit CIDR block, and an internal network.

Part one:

I want to have an IP for each of the services, mail and http, on each 
ISP, so that is DSL is down I can use cable, and vice-versa. I will do 
NAT in the firewall, and forward the packets to the actual server. 
Eventually the servers will move to a DMZ after the other stuff settles 
down.

The problem is that a packet can come from any IP outside, and when the 
reply packet is sent out, it may go out either NIC. And that's the root 
of the problem, getting the source IP to match the NIC. I've added rules 
to the mangle table to MARK the packets, that just doesn't seem to work 
reliably.

I want very much to do this without patching the kernel, I have two 
patches which seem to solve the problem on other systems, but 
maintaining a patched kernel long term is really undesirable, and makes 
it hard to turn over the job in the future.

All I want to do is send packets out the interface which matches the 
source IP, and I don't think there's any reasonable way to get there 
without patches or BSD.

Yes, I know about the lartc docs, nano.txt and several other things. The 
problem is that the marks don't reliably WORK, routing by destination IP 
is being used in some cases (but not all, which is really odd).

-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: Fairly complex multi-ISP firewall/router problem

[Antony Stone]

On Friday 02 April 2004 9:57 pm, Bill Davidsen wrote:

> I am trying to set up a single Linux router, RH9.0, for a non-profit I
> am supporting with some free consulting. They have two ISP lines, each
> of which has a three bit CIDR block, and an internal network.
>
> Part one:
>
> All I want to do is send packets out the interface which matches the
> source IP, and I don't think there's any reasonable way to get there
> without patches or BSD.

So why not use BSD?

BTW: What was Part two?

Antony.

-- 
You can spend the whole of your life trying to be popular,
but at the end of the day the size of the crowd at your funeral
will be largely dictated by the weather.

 - Frank Skinner

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Fairly complex multi-ISP firewall/router problem

[Bill Davidsen]

Antony Stone wrote:
> On Friday 02 April 2004 9:57 pm, Bill Davidsen wrote:
> 
> 
>>I am trying to set up a single Linux router, RH9.0, for a non-profit I
>>am supporting with some free consulting. They have two ISP lines, each
>>of which has a three bit CIDR block, and an internal network.
>>
>>Part one:
>>
>>All I want to do is send packets out the interface which matches the
>>source IP, and I don't think there's any reasonable way to get there
>>without patches or BSD.
> 
> 
> So why not use BSD?

That is what I'm asking myself. I guess the answer is that I like Linux 
better in other ways, but given the choice between maintaining a patch 
and using BSD... I may.
> 
> BTW: What was Part two?

Part two is the need to send to certain destinations using a known 
source IP (SNAT), which I would like to do on the firewall just to avoid 
having change on the server. That's easy, a rule for SNAT and a static 
route for the destination. It just adds cruft to the tables, that's been 
working for a while, trusted outgoing mail get SNAT now.

-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: Fairly complex multi-ISP firewall/router problem

[Cedric Blancher]

Le ven 02/04/2004 à 22:57, Bill Davidsen a écrit :
> I am trying to set up a single Linux router, RH9.0, for a non-profit I 
> am supporting with some free consulting. They have two ISP lines, each 
> of which has a three bit CIDR block, and an internal network.
> I want to have an IP for each of the services, mail and http, on each 
> ISP, so that is DSL is down I can use cable, and vice-versa.

So you have this kind of setup :

	ISP1		ISP2
	 |		 |
	 `----- RH9 -----'
		 |
		 |
		LAN ---- Server

And you DNAT both ISP1 and ISP2 IPs to internal server.

> The problem is that a packet can come from any IP outside, and when the 
> reply packet is sent out, it may go out either NIC. And that's the root 
> of the problem

Yes it is...

> getting the source IP to match the NIC. I've added rules 
> to the mangle table to MARK the packets, that just doesn't seem to work 
> reliably.

This won't work for a quite simple reason : MARK only works internally,
so you won't get any MARK back with the answer you need to route.
However, you can still use other means, such as the force.

> I want very much to do this without patching the kernel

Fine.

If I were you, I would double address the server within the LAN, one IP
affected to ISP1 DNAT and the other one to ISP2 DNAT. This way :

	iptables -t nat -A PREROUTING -d ISP1_IP -j DNAT --to LAN_IP1
	iptables -t nat -A PREROUTING -d ISP2_IP -j DNAT --to LAN_IP2

So, replies will come to the firewall with LAN_IP1 source if connection
has been initiated from ISP1, and LAN_IP2 for ISP2. And then, you can
use fairly simple source policy routing setting such as described in
LARTC :

	http://lartc.org/howto/lartc.rpdb.html#LARTC.RPDB.SIMPLE

# ip rule add from $LAN_IP1 table isp1
# ip route add default via $ISP1_GW dev $IFACE2ISP1 table isp1
# ip rule add from $LAN_IP2 table isp2
# ip route add default via $IPS2_GW dev $IFACE2ISP2 table isp2
# ip route cache flush

This way, packets should go out through the appropriate interface.
However, it implies some configuration stuff on the server, especially
for Web server on which you may want to set vhosts up to handle stuff
properly.

And finally, if you move your server to DMZ, double address your DMZ
with both ISP adresses ranges.

-- 
http://www.netexit.com/~sid/
PGP KeyID: 157E98EE FingerPrint: FA62226DA9E72FA8AECAA240008B480E157E98EE
>> Hi! I'm your friendly neighbourhood signature virus.
>> Copy me to your signature file and help me spread!

Re: Fairly complex multi-ISP firewall/router problem

[John A. Sullivan III]

On Fri, 2004-04-02 at 15:57, Bill Davidsen wrote:
> I am trying to set up a single Linux router, RH9.0, for a non-profit I 
> am supporting with some free consulting. They have two ISP lines, each 
> of which has a three bit CIDR block, and an internal network.
> 
> Part one:
> 
> I want to have an IP for each of the services, mail and http, on each 
> ISP, so that is DSL is down I can use cable, and vice-versa. I will do 
> NAT in the firewall, and forward the packets to the actual server. 
> Eventually the servers will move to a DMZ after the other stuff settles 
> down.
> 
> The problem is that a packet can come from any IP outside, and when the 
> reply packet is sent out, it may go out either NIC. And that's the root 
> of the problem, getting the source IP to match the NIC. I've added rules 
> to the mangle table to MARK the packets, that just doesn't seem to work 
> reliably.
> 
> I want very much to do this without patching the kernel, I have two 
> patches which seem to solve the problem on other systems, but 
> maintaining a patched kernel long term is really undesirable, and makes 
> it hard to turn over the job in the future.
> 
> All I want to do is send packets out the interface which matches the 
> source IP, and I don't think there's any reasonable way to get there 
> without patches or BSD.
> 
> Yes, I know about the lartc docs, nano.txt and several other things. The 
> problem is that the marks don't reliably WORK, routing by destination IP 
> is being used in some cases (but not all, which is really odd).

Hmmm . . . I admit to not having tried this and only giving it five
minute's thought but I'm not sure I see the problem.  Well, I see why
one can't be guaranteed to send the packet out the same interface but
I'm not sure why that is a problem.

If the packet comes in and is DNAT'd (I assume that is what you are
doing), wouldn't it be entered in conntrack so that the reply packets
will be SNAT'd to the same source address as the destination address of
the original packet? That should make the reply packet acceptable to the
originating device.  One cannot ensure which interface it goes out (I am
assuming that the source address alteration of the reply packet occurs
post routing and thus iproute2 is useless here) but it should make its
way back to the originating device via the magic of Internet routing
regardless of interface -- shouldn't it?

In the case of an interface or ISP failure, I assume you would disable
the interface which would eliminate the route.

So I would think the packet path would be from originating device to one
of the FW interfaces, DNAT to the private address, reply from the
private address, SNAT of the reply according to conntrack, out either of
the interfaces, back via Internet routing to the originating device.

Of course, I could be brain-cramping here as I am eager to return to my
current task from this quick e-mail break but I missed seeing the
problem - John
-- 
John A. Sullivan III
Chief Technology Officer
Nexus Management
+1 207-985-7880
john.sullivan@nexusmgmt.com

Re: Fairly complex multi-ISP firewall/router problem

[Antony Stone]

On Friday 02 April 2004 10:36 pm, John A. Sullivan III wrote:

> On Fri, 2004-04-02 at 15:57, Bill Davidsen wrote:
> >
> > All I want to do is send packets out the interface which matches the
> > source IP, and I don't think there's any reasonable way to get there
> > without patches or BSD.
>
> Hmmm . . . I admit to not having tried this and only giving it five
> minute's thought but I'm not sure I see the problem.  Well, I see why
> one can't be guaranteed to send the packet out the same interface but
> I'm not sure why that is a problem.

Some ISPs block packets with source addresses not matching their own network 
range, as a contribution to blocking spoofed packets.

> In the case of an interface or ISP failure, I assume you would disable
> the interface which would eliminate the route.

That's not necessarily a difficult task (bringing it back up again afterwards 
is not entirely trivial, however), but if the problem can be solved without 
sending all outbound traffic across a single connection, and leaving the 
other one largely idle, it would be a better solution.

Regards,

Antony.

-- 
The idea that Bill Gates appeared like a knight in shining armour to lead all 
customers out of a mire of technological chaos neatly ignores the fact that 
it was he who, by peddling second-rate technology, led them into it in the 
first place.

 - Douglas Adams in The Guardian, 25th August 1995

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Fairly complex multi-ISP firewall/router problem

[Joe Thompson]

What is the architecture between the rest of the world and your
firewall?  Is it possible to use BGP and only one of the public
subnets?  This would in effect move the redundancy of the public side to
the router(s) allowing you to use standard method's at the firewall.

We run a similar situation with one of our subnets, we have two circuits
from separate provider's who were both gracious enough to add the routes
in they're tables, if we lose one connection the rest of the world just
uses the other route in and we of course use the other route out.  The
downside is getting both upstream providers to cooperate in routing, the
upside is that you can utilize both links and keep things simple from an
addressing perspective.

Joe

On Fri, 2004-04-02 at 14:50, Antony Stone wrote:
> On Friday 02 April 2004 10:36 pm, John A. Sullivan III wrote:
> 
> > On Fri, 2004-04-02 at 15:57, Bill Davidsen wrote:
> > >
> > > All I want to do is send packets out the interface which matches the
> > > source IP, and I don't think there's any reasonable way to get there
> > > without patches or BSD.
> >
> > Hmmm . . . I admit to not having tried this and only giving it five
> > minute's thought but I'm not sure I see the problem.  Well, I see why
> > one can't be guaranteed to send the packet out the same interface but
> > I'm not sure why that is a problem.
> 
> Some ISPs block packets with source addresses not matching their own network 
> range, as a contribution to blocking spoofed packets.
> 
> > In the case of an interface or ISP failure, I assume you would disable
> > the interface which would eliminate the route.
> 
> That's not necessarily a difficult task (bringing it back up again afterwards 
> is not entirely trivial, however), but if the problem can be solved without 
> sending all outbound traffic across a single connection, and leaving the 
> other one largely idle, it would be a better solution.
> 
> Regards,
> 
> Antony.

Re: Fairly complex multi-ISP firewall/router problem

[Bill Davidsen]

Joe Thompson wrote:
> What is the architecture between the rest of the world and your
> firewall?  Is it possible to use BGP and only one of the public
> subnets?  This would in effect move the redundancy of the public side to
> the router(s) allowing you to use standard method's at the firewall.

All of the internal stuff is on a 192.168.x.x net. So that a packet 
comes in NIC1 and gets DNAT'd to the internat address, then the reply 
comes back to the firewall with the correct destination address, and the 
source address gets reset correctly. That's where the problem comes in, 
since now the packet can go back our through the NIC to either ISP. But 
if it goes out the *wrong* NIC, the ISP thinks I'm spoofing and drops 
the packet.
> 
> We run a similar situation with one of our subnets, we have two circuits
> from separate provider's who were both gracious enough to add the routes
> in they're tables, if we lose one connection the rest of the world just
> uses the other route in and we of course use the other route out.  The
> downside is getting both upstream providers to cooperate in routing, the
> upside is that you can utilize both links and keep things simple from an
> addressing perspective.

I actually left out a bit of the complexity, some of the outgoing 
connections *must* go through one NIC or the other, because the 
destination will only accept from a given source IP (and that IP will 
only go out through the appropriate ISP).

There are two ways to patch this, one by addint the IP to the ARP table 
with the MAC of the correct router, one by some code to ignore any NIC 
which doesn't have an alias for the source address. Both are patches I 
don't want to maintain.

-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979

Re: Fairly complex multi-ISP firewall/router problem

[Tarek W.]

On Fri, 2004-04-02 at 23:57, Bill Davidsen wrote: 
> I am trying to set up a single Linux router, RH9.0, for a non-profit I 
> am supporting with some free consulting. They have two ISP lines, each 
> of which has a three bit CIDR block, and an internal network.
> 
> Part one:
> 
> I want to have an IP for each of the services, mail and http, on each 
> ISP, so that is DSL is down I can use cable, and vice-versa. I will do 
> NAT in the firewall, and forward the packets to the actual server. 
> Eventually the servers will move to a DMZ after the other stuff settles 
> down.
> 
> The problem is that a packet can come from any IP outside, and when the 
> reply packet is sent out, it may go out either NIC. And that's the root 
> of the problem, getting the source IP to match the NIC. I've added rules 
> to the mangle table to MARK the packets, that just doesn't seem to work 
> reliably.
> 
> I want very much to do this without patching the kernel, I have two 
> patches which seem to solve the problem on other systems, but 
> maintaining a patched kernel long term is really undesirable, and makes 
> it hard to turn over the job in the future.
> 
> All I want to do is send packets out the interface which matches the 
> source IP, and I don't think there's any reasonable way to get there 
> without patches or BSD.
> 
> Yes, I know about the lartc docs, nano.txt and several other things. The 
> problem is that the marks don't reliably WORK, routing by destination IP 
> is being used in some cases (but not all, which is really odd).

first, the good news... u don't need to patch the kernel...

now, more good news... u actually have two solutions:

ur problem is a routing issue... since Linux works as expected, ur
solution is in modifying the routing tables and SNAT out...

SNAT should only be used to make sure outgoing packets leave an
interface with a proper address (the same address that is bound to the
interface).

u can match by almost anything with ip.

another solution would be to use the ROUTE target with the --continue
option. the target applies in the mangle table, with the --continue
option, the packet goes through nat also and gets SNAT'ed.

if u want some hand-holding, give us some more details...

RE: Fairly complex multi-ISP firewall/router problem

[Daniel Chemko]

Antony Stone wrote:
> On Friday 02 April 2004 10:36 pm, John A. Sullivan III wrote:
> 
>> On Fri, 2004-04-02 at 15:57, Bill Davidsen wrote:
>>> 
>>> All I want to do is send packets out the interface which matches the
>>> source IP, and I don't think there's any reasonable way to get there
>>> without patches or BSD.
>> 
>> Hmmm . . . I admit to not having tried this and only giving it five
>> minute's thought but I'm not sure I see the problem.  Well, I see why
>> one can't be guaranteed to send the packet out the same interface but
>> I'm not sure why that is a problem.
> 
> Some ISPs block packets with source addresses not matching their own
> network range, as a contribution to blocking spoofed packets.

This is a very real issue, especially when they're only consumer grade.

What I've used to fix the problem is to use the CONNMARK extension on
the PREROUTING step of mangle. Here, I can set the appropriate routes
and everything that uses CONNMARK will work fine.

Eg:
IPT_NOMARK="-m mark --mark 0"
IPT_MARKED="-m mark ! --mark 0"

${IPTABLES} -t mangle -A PREROUTING -j CONNMARK --restore-mark
# MARK packets that are inbound from INET3/INET4 to leave the same
interface
# You also get related traffic leaving the related session's route for
free
${IPTABLES} -t mangle -A PREROUTING ${IPT_NOMARK} -i ${IF_INET3} -j MARK
--set-mark ${RTABLE_INET3}
${IPTABLES} -t mangle -A PREROUTING ${IPT_NOMARK} -i ${IF_INET4} -j MARK
--set-mark ${RTABLE_INET4}
${IPTABLES} -t mangle -A PREROUTING -j CONNMARK --save-mark

You can setup something similar for outgoing sessions. I don't do it,
but there's nothing stopping you from it.


>> In the case of an interface or ISP failure, I assume you would
>> disable the interface which would eliminate the route.
> 
> That's not necessarily a difficult task (bringing it back up again
> afterwards is not entirely trivial, however), but if the problem can
> be solved without sending all outbound traffic across a single
> connection, and leaving the other one largely idle, it would be a
> better solution. 

As described above, you can use typical iptables matching to do policy
routing based on any layer supported by iptables.

This example would create equalized load balancing based on NEW sessions
${IPTABLES} -t mangle -A PREROUTING ${IPT_NOMARK} -i ${IF_INTERNAL} -m
nth --every 2 --packet 0 -j MARK --set-mark ${RTABLE_INET3}
${IPTABLES} -t mangle -A PREROUTING ${IPT_NOMARK} -i ${IF_INTERNAL} -m
nth --every 2 --packet 1 -j MARK --set-mark ${RTABLE_INET4}

My RTABLES have all the rules from my main table, with the exception of
the default route. This allows for overly broad iptables connmark rules
without breaking the routing of other interfaces.

This is my routing table builder:

function route_builder
{
   if [ "${OPT_SYS_ROUTER}" = "0" -o "${OPT_ROUTER_POLICYROUTING}" = "0"
]; then
      info "route_builder: Policy Routing is disabled. Skipping."
      return 1
   fi
   _table_id=${1}
   _table_gateway=${2}
   _table_source=${3}
   if [ "${1}" != "" -a "${2}" != "" ]; then
      ${IP} rule del fwmark ${_table_id} table ${_table_id}
      ${IP} rule add fwmark ${_table_id} table ${_table_id}
      ${IP} route flush table ${_table_id}
      ${IP} route show table main | grep -Ev ^default
\
      | while read ROUTE ; do
         ${IP} route add table ${_table_id} $ROUTE
      done
      ${IP} route add table ${_table_id} default via ${_table_gateway}
\
         src ${_table_source}
   else
      info "route_builder: Invalid arguments specified."
   fi
}

Re: Fairly complex multi-ISP firewall/router problem

[Bill Davidsen]

Daniel Chemko wrote:
> Antony Stone wrote:
> 
>>On Friday 02 April 2004 10:36 pm, John A. Sullivan III wrote:
>>
>>
>>>On Fri, 2004-04-02 at 15:57, Bill Davidsen wrote:
>>>
>>>>All I want to do is send packets out the interface which matches the
>>>>source IP, and I don't think there's any reasonable way to get there
>>>>without patches or BSD.
>>>
>>>Hmmm . . . I admit to not having tried this and only giving it five
>>>minute's thought but I'm not sure I see the problem.  Well, I see why
>>>one can't be guaranteed to send the packet out the same interface but
>>>I'm not sure why that is a problem.
>>
>>Some ISPs block packets with source addresses not matching their own
>>network range, as a contribution to blocking spoofed packets.
> 
> 
> This is a very real issue, especially when they're only consumer grade.
> 
> What I've used to fix the problem is to use the CONNMARK extension on
> the PREROUTING step of mangle. Here, I can set the appropriate routes
> and everything that uses CONNMARK will work fine.

Awesome! I have to read this for a bit and refresh my understanding of 
CONNMARK before I try it, but this may solve the whole problem.

Totally impressive technical part snipped for brevity but saved and printed!

-- 
bill davidsen <davidsen@tmr.com>
   CTO TMR Associates, Inc
   Doing interesting things with small computers since 1979