[LARTC] Multiple internet providers

[LARTC] Multiple internet providers

[Kim Jensen]

Hi all,

I'm having a problem with multiple internet providers. According to all the 
things I have found so far it should not be that uncomplicated to set it up - 
however, I'm having some problems getting it to work.

Firsth the scenario - I'm having 2 Internet providers, one fast with 32 static 
IP's and one slow with dynamic IP's. Internally we are using a private net 
which is set up for 32 ip's. Now, we have to map all 32 static ip's to the 
private IP's so all machines can be reached externally through the static 
net. The default route is suppose to be the slow ISP.

So far, I have found that I need to set up the routing using marking, and then 
set up the rules so it work accordingly. But this appears to just work with 
static routing, not when one of the providers is dynamic!

Does anyone have some suggestions for what to do?

/Kim

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Martin A. Brown]

Kim,

 : I'm having a problem with multiple internet providers. According to all
 : the things I have found so far it should not be that uncomplicated to
 : set it up - however, I'm having some problems getting it to work.

You are correct.  It is not uncomplicated.  It is also not difficult.  :)

 : Firsth the scenario - I'm having 2 Internet providers, one fast with 32
 : static IP's and one slow with dynamic IP's. Internally we are using a
 : private net which is set up for 32 ip's. Now, we have to map all 32
 : static ip's to the private IP's so all machines can be reached
 : externally through the static net. The default route is suppose to be
 : the slow ISP.
 :
 : So far, I have found that I need to set up the routing using marking,
 : and then set up the rules so it work accordingly.

That's one approach, documented here:

  http://linux-ip.net/html/adv-multi-internet.html

This only works if you are not trying to perform link load balancing.  If
that is your intended goal, you'll need to check out the Nano-HOWTO (Hey
Julian--this appears to be missing right now):

  http://www.linuxvirtualserver.org/~julian/nano.txt

 : But this appears to just work with static routing, not when one of the
 : providers is dynamic!

It doesn't matter that the IPs are dynamic, just that you can alter the
affected routing table when you get a new dynamic IP.  If you are using
rp-pppoe, you'll want to perform your additional routing table
manipulations in ip-up.local.

And also, unless you have a routing daemon, you are using static routing.
A dynamic IP address is simply that--your network-connected machine is
still performing static routing, though it has a dynamic address.

Since you have a dynamic IP on one interface, you may wish to use the -j
MASQUERADE target instead of the -j SNAT target.

 : Does anyone have some suggestions for what to do?

Do the above suggestions help?

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Julian Anastasov]

	Hello,

On Tue, 18 Mar 2003, Martin A. Brown wrote:

> that is your intended goal, you'll need to check out the Nano-HOWTO (Hey
> Julian--this appears to be missing right now):
>
>   http://www.linuxvirtualserver.org/~julian/nano.txt

	The source site is:

http://www.ssi.bg/~ja/

	The old URL is still maintained but as mirror, not under
my control, may be it is now back as redirect
(http://www.linuxvirtualserver.org/~julian/) to the primary site.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Kim Jensen]

On Tuesday 18 March 2003 17:47, Martin A. Brown wrote:
> That's one approach, documented here:
>
>   http://linux-ip.net/html/adv-multi-internet.html
>
Hi Martin,

Thanks for the help, I got started but have a strange problem. According to 
the adv-multi-internet.html doc, I should set up a couple of simple rules, 
first the IP routing, which seems to be running smoothely, secondly, I have 
to set up the iptables rules as well, this is also fairly straightforward.

Now, the interesting (or actually not!) things begin, my box gets packets in, 
which is suppose to be routed, I can DNAT them, but after that they simply 
disappear - they never reach the forward chain!!! Is there a reason why?

/Kim

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Martin A. Brown]

Hi there Kim,

 : Thanks for the help, I got started but have a strange problem.
 : According to the adv-multi-internet.html doc, I should set up a couple
 : of simple rules, first the IP routing, which seems to be running
 : smoothely, secondly, I have to set up the iptables rules as well, this
 : is also fairly straightforward.

Does that mean you have had success passing packets over multiple links?
Or perhaps no success yet?

 : Now, the interesting (or actually not!) things begin,

Sometimes, interesting is not good.  But, at least, interesting is the way
to understanding....

 : my box gets packets in, which is suppose to be routed, I can DNAT them,
 : but after that they simply disappear - they never reach the forward
 : chain!!! Is there a reason why?

Sounds like a routing problem.

Post "ip rule show", and "ip route show table $TABLEID" for your main
routing table, and the other routing tables, and if you can, a textual
description of the network, or a simple ASCII network map.

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Martin A. Brown]

Kim,

Well, I *think* the solution is a simple sysctl twiddle (see close to the
bottom), but I left my reasoning and research intact on the off-chance
that my method would benefit you or posterity reviewing this thread..

iptables mangle thoughts
- - - - - - - - - - - - -
I have line broken the iptables output, but I'm curious to know if you are
seeing any activity in these particular chains:

# iptables -t mangle -nvL FORWARD
<snip>
    0     0 MARK       all  --  *      eth1    0.0.0.0/0
       0.0.0.0/0          MARK set 0x1
    0     0 MARK       all  --  *      *       212.202.108.160/27   \
       0.0.0.0/0          MARK set 0x1
    0     0 MARK       all  --  *      *       0.0.0.0/0            \
       212.202.108.160/27 MARK set 0x1
</snip>

I expect that the counter on the first of these three chains is going up,
because this is the chain through which all of your packets leaving eth1
will pass.  This might lead you to believe that the chain was performing
some useful function--but since the routing decision has already been
made, your packet is getting marked after the routing decision has been
made.  I'm not sure about my diagnosis on this one, however....

I don't think the other two are doing anything.  I think you
will probably want to use 192.168.1.160/27 here instead of the public
network IPs.

Since your problem (for split access to the Internet via multiple
providers) is correctly routing the outbound packets, you really only need
to mark the outbound packets.  You don't need to distinguish the inbound
packets because connection tracking will take care of restoring the
internal IP address.

So, could you perform all of the marking in the PREROUTING chain?

  for internalnet in $ITX_NET $DMZ_NET ; do
    iptables -t mangle -I PREROUTING -s $internalnet \
      -j MARK --set-mark $FWMARK
  done

I don't see why you'd choose the FORWARD chain in the mangle table for
this marking.  I'd suggest the PREROUTING chain in the mangle table--at
least that way, you know the fwmark is available for ANY routing decision.

 : > Sounds like a routing problem.

Sounds like I was speculating without enough information.  :(  Everything
looks perfect in your routing tables and RPDB!

  <snipped your routing tables and RPDB>

 : I'm also attaching the firewall script, which both generates the
 : routing table and sets up the iptable rules.

Ah, the joys of digging through mountains of iptables rules.  At least
it's better than ipchains!  Here's what I think you are seeing

 - outbound packet from 192.168.1.161
 - marked and rewritten to appear to the Internet from 212.202.108.161
 - ACK returns, accepted on eth1
 - connection tracking rewrites destination IP to 192.168.1.161
 - you lose packet!

 : I can see packets running out, and getting an ACK back. From logging I
 : can see the packet getting into my DNAT, but then it stops!

Do we concur?

Question 1:  Why do you say that the packet is hitting the DNAT?
             Shouldn't the packet be handled by the connection tracking
             mechanism?
Question 2:  Did you try tcpdump'ing on eth2?  Are packets showing up on
             the wire with the wrong destination address?
Question 3:  Did you try defining the minimal required iptables rules to
             allow one single IP to reach the 'net via this split access
             technique?

After all of the above, I was reviewing your script one more time, and
discovered the rp_filter sysctl....

I think your rp_filter sysctl is disagreeing with your multiple uplink
paths.

What happens when you change nothing of the above, and change this?

    # IP Spoof protection
    if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then
        echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
    fi

To

    # IP Spoof protection
    if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then
        for i in $( ls -1 /proc/sys/net/ipv4/conf ) ; do
            echo 0 > /proc/sys/net/ipv4/conf/$i/rp_filter
        done
    fi

Reverse path filtering is very handy in some situations as a very basic
antispoof measure, but when you have multiple paths to the same network,
it's important to turn off this spoof protection.

For those following along on this thread, see the section on rp_filter in
the IP Sysctl tutorial:

  http://ipsysctl-tutorial.frozentux.net/
  http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html#AEN616

Oskar Andreasson's cautionary note mocks me gently after an hour puzzling
about this problem!  :)

 : I hope that you with this information can give me some hints. The net
 : setup is 2 internal and two external. The rules should appear from the
 : firewall script.

I think this is what your network looks like:

                    +------------------+   212.202.108.160/27   +--------+
 192.168.1.160/28 --+eth2          eth1+-- 192.168.0.0/24 ------+ router |
         $DMZ_NET   |                  |   $QSC_NET             +--------+
                    |      linux       |
                    |                  |                        +--------+
 192.168.1.176/28 --+eth3          eth0+-- 192.168.141.160/27 --+ router |
         $ITX_NET   +------------------+   (Arcor)              +--------+

Good luck, Kim,

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Kim Jensen]

Hi Martin,

First, just to clarify, all my packet destined for the internal net via our 
QSC provider was hitting the DNAT chain, and then lost. All packets going the 
other way was passing the FORWARD chain and then lost. So my problem seemed 
to be just after the incoming packet has arrived or before it leaves.

I have tracked packets using extensive logging (and multiple changes in my 
firewall script) as well as tcpdump on all relevant interfaces in my system. 
The packet looks great when it comes in, correct information, etc. But never 
reaches the destination on the box.

I have not played much with the firewall script after I got your reply, my 
ADSL died and I only just got it back up! But changing the rp_filter from one 
to zero seems to work, as I can track packets hitting my FORWARD chain now.

So, right now I seem to be on the right track - I think the general problem 
with setting up something like this is that all relevant information is 
usually scattered over many places with mostly only fragments put together. 
But now now, Thanks Martin - you have been a great help :-)

/Kim

On Saturday 22 March 2003 06:57, Martin A. Brown wrote:
> Kim,
>
> Well, I *think* the solution is a simple sysctl twiddle (see close to the
> bottom), but I left my reasoning and research intact on the off-chance
> that my method would benefit you or posterity reviewing this thread..
>
> iptables mangle thoughts
> - - - - - - - - - - - - -
> I have line broken the iptables output, but I'm curious to know if you are
> seeing any activity in these particular chains:
>
> # iptables -t mangle -nvL FORWARD
> <snip>
>     0     0 MARK       all  --  *      eth1    0.0.0.0/0
>        0.0.0.0/0          MARK set 0x1
>     0     0 MARK       all  --  *      *       212.202.108.160/27   \
>        0.0.0.0/0          MARK set 0x1
>     0     0 MARK       all  --  *      *       0.0.0.0/0            \
>        212.202.108.160/27 MARK set 0x1
> </snip>
>
> I expect that the counter on the first of these three chains is going up,
> because this is the chain through which all of your packets leaving eth1
> will pass.  This might lead you to believe that the chain was performing
> some useful function--but since the routing decision has already been
> made, your packet is getting marked after the routing decision has been
> made.  I'm not sure about my diagnosis on this one, however....
>
> I don't think the other two are doing anything.  I think you
> will probably want to use 192.168.1.160/27 here instead of the public
> network IPs.
>
> Since your problem (for split access to the Internet via multiple
> providers) is correctly routing the outbound packets, you really only need
> to mark the outbound packets.  You don't need to distinguish the inbound
> packets because connection tracking will take care of restoring the
> internal IP address.
>
> So, could you perform all of the marking in the PREROUTING chain?
>
>   for internalnet in $ITX_NET $DMZ_NET ; do
>     iptables -t mangle -I PREROUTING -s $internalnet \
>       -j MARK --set-mark $FWMARK
>   done
>
> I don't see why you'd choose the FORWARD chain in the mangle table for
> this marking.  I'd suggest the PREROUTING chain in the mangle table--at
> least that way, you know the fwmark is available for ANY routing decision.
>
>  : > Sounds like a routing problem.
>
> Sounds like I was speculating without enough information.  :(  Everything
> looks perfect in your routing tables and RPDB!
>
>   <snipped your routing tables and RPDB>
>
>  : I'm also attaching the firewall script, which both generates the
>  : routing table and sets up the iptable rules.
>
> Ah, the joys of digging through mountains of iptables rules.  At least
> it's better than ipchains!  Here's what I think you are seeing
>
>  - outbound packet from 192.168.1.161
>  - marked and rewritten to appear to the Internet from 212.202.108.161
>  - ACK returns, accepted on eth1
>  - connection tracking rewrites destination IP to 192.168.1.161
>  - you lose packet!
>
>  : I can see packets running out, and getting an ACK back. From logging I
>  : can see the packet getting into my DNAT, but then it stops!
>
> Do we concur?
>
> Question 1:  Why do you say that the packet is hitting the DNAT?
>              Shouldn't the packet be handled by the connection tracking
>              mechanism?
> Question 2:  Did you try tcpdump'ing on eth2?  Are packets showing up on
>              the wire with the wrong destination address?
> Question 3:  Did you try defining the minimal required iptables rules to
>              allow one single IP to reach the 'net via this split access
>              technique?
>
> After all of the above, I was reviewing your script one more time, and
> discovered the rp_filter sysctl....
>
> I think your rp_filter sysctl is disagreeing with your multiple uplink
> paths.
>
> What happens when you change nothing of the above, and change this?
>
>     # IP Spoof protection
>     if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then
>         echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
>     fi
>
> To
>
>     # IP Spoof protection
>     if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then
>         for i in $( ls -1 /proc/sys/net/ipv4/conf ) ; do
>             echo 0 > /proc/sys/net/ipv4/conf/$i/rp_filter
>         done
>     fi
>
> Reverse path filtering is very handy in some situations as a very basic
> antispoof measure, but when you have multiple paths to the same network,
> it's important to turn off this spoof protection.
>
> For those following along on this thread, see the section on rp_filter in
> the IP Sysctl tutorial:
>
>   http://ipsysctl-tutorial.frozentux.net/
>   http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html#AEN616
>
> Oskar Andreasson's cautionary note mocks me gently after an hour puzzling
> about this problem!  :)
>
>  : I hope that you with this information can give me some hints. The net
>  : setup is 2 internal and two external. The rules should appear from the
>  : firewall script.
>
> I think this is what your network looks like:
>
>                     +------------------+   212.202.108.160/27   +--------+
>  192.168.1.160/28 --+eth2          eth1+-- 192.168.0.0/24 ------+ router |
>          $DMZ_NET   |                  |   $QSC_NET             +--------+
>
>                     |      linux       |
>                     |
>                     |                  |                        +--------+
>
>  192.168.1.176/28 --+eth3          eth0+-- 192.168.141.160/27 --+ router |
>          $ITX_NET   +------------------+   (Arcor)              +--------+
>
> Good luck, Kim,
>
> -Martin

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Martin A. Brown]

 : But changing the rp_filter from one to zero seems to work, as I can
 : track packets hitting my FORWARD chain now.

Excellent.  I'm quite glad to hear it.

 : So, right now I seem to be on the right track - I think the general
 : problem with setting up something like this is that all relevant
 : information is usually scattered over many places with mostly only
 : fragments put together.  But now now, Thanks Martin - you have been a
 : great help :-)

Well, that's what my documentation attempts to remedy--but never can.
Such a dynamic target is very hard to adequately document, but that will
not prevent me from trying.  Regardless your experience with my section on
multiple Internet connections points out a now-obvious deficiency in the
section on multiple uplinks.  I will modify the section to include a
similar cautionary note about the rp_filter sysctl.

Thanks for letting me know it's working for you,

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Kim Jensen]

On Saturday 22 March 2003 17:33, you wrote:
>  : But changing the rp_filter from one to zero seems to work, as I can
>  : track packets hitting my FORWARD chain now.
>
> Excellent.  I'm quite glad to hear it.
>
>  : So, right now I seem to be on the right track - I think the general
>  : problem with setting up something like this is that all relevant
>  : information is usually scattered over many places with mostly only
>  : fragments put together.  But now now, Thanks Martin - you have been a
>  : great help :-)
>
> Well, that's what my documentation attempts to remedy--but never can.
> Such a dynamic target is very hard to adequately document, but that will
> not prevent me from trying.  Regardless your experience with my section on
> multiple Internet connections points out a now-obvious deficiency in the
> section on multiple uplinks.  I will modify the section to include a
> similar cautionary note about the rp_filter sysctl.
>
> Thanks for letting me know it's working for you,
>
Well, not so fast ... I'm still having some problems, but I fear that they 
will be very hard to solve!

We have 2 connections, our cheap (Arcor, dynamic IP) and our expensive (QSC) 
with 32 public addresses. Since my company is also linked up to a second 
company using a set of private addresses, we have to use the 192.168.1.160/27 
net internally, with a DHCP server. Most of our traffic must go via the Arcor 
connection, while all mail will go via the QSC together with SSH to specific 
machines. For incoming traffic, all addresses in our 32 public IP numbers, 
must be routed 1-1 for the internal net, so it is possible to make external 
connections to either mail, web, ssh, etc. on internal machines.

Now my question is, is it possible to make a destinction for packets coming 
from our internal nets, whether it is part of a connection from QSC or from 
Arcor? So far, I am lost when it comes to ideas and plans... It appears like 
SNAT is impossible as I don't know whether it will correctly translate the 
packet back, and MASQUERADE doesn't seem like the solution either!

Using MASQUERADING, it is possible to have traffic running normally via the 
Arcor net, but once we wish to include QSC in the calculation - I keep 
hitting my head on the wall.

/Kim

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Multiple internet providers

[Martin A. Brown]

Hello again Kim,

So, this is your network, correct?

                    +------------------+   212.202.108.160/27   +--------+
 192.168.1.160/28 --+eth2          eth1+-- 192.168.0.0/24 ------+ router |
         $DMZ_NET   |                  |   $QSC_NET             +--------+
                    |      linux       |
                    |                  |                        +--------+
 192.168.1.176/28 --+eth3          eth0+-- 192.168.141.160/27 --+ router |
         $ITX_NET   +------------------+   (Arcor)              +--------+


 : Well, not so fast ... I'm still having some problems, but I fear that
 : they will be very hard to solve!

Maybe...but maybe not....

 : We have 2 connections, our cheap (Arcor, dynamic IP) and our expensive
 : (QSC)  with 32 public addresses. Since my company is also linked up to
 : a second company using a set of private addresses, we have to use the
 : 192.168.1.160/27 net internally, with a DHCP server.

 : Most of our traffic must go via the Arcor connection, while

So, in this case, I'd make Arcor the default route in table main, as you
have done.

 : all mail will go via the QSC together with SSH to specific machines.

So, MAIL% (but could be anything):

  for internalnet in $ITX_NET $DMZ_NET ; do
    iptables -t mangle -I PREROUTING -p tcp -s $internalnet \
      --dport $MAIL -j MARK --set-mark $FWMARK
    for ssh_server in $SSH_HOSTS ; do
      iptables -t mangle -I PREROUTING -p tcp -s $internalnet \
        -d $ssh_server --dport 22 -j MARK --set-mark $FWMARK
    done   # -- end of ssh_server loop
  done     # -- end of internalnet loop

 : For incoming traffic, all addresses in our 32 public IP numbers, must
 : be routed 1-1 for the internal net, so it is possible to make external
 : connections to either mail, web, ssh, etc. on internal machines.

No problems...as long as the connections are initiated from the outside,
connection tracking will do the work of transforming addresses for you.
This is one of the big benefits of using netfilter--the connection
tracking mechanisms can make complex scenarios much easier.  You should be
able to use the MASQUERADE target on the outbound connections through
Arcor, which gets you connection tracking for anything initiated from
either of your inside networks.

I would, however, recommend adding the following to your RPDB (I assume
you will continue your convention--using an fwmark corresponding to the
table ID of the routing table):

# ip rule add from 212.202.108.160/27 table 1

This should ensure that traffic with an outgoing IP in this range is
transmitted via the QSC default route.

 : Now my question is, is it possible to make a destinction for packets
 : coming from our internal nets, whether it is part of a connection from
 : QSC or from Arcor? So far, I am lost when it comes to ideas and
 : plans... It appears like SNAT is impossible as I don't know whether it
 : will correctly translate the packet back, and MASQUERADE doesn't seem
 : like the solution either!
 :
 : Using MASQUERADING, it is possible to have traffic running normally via
 : the Arcor net, but once we wish to include QSC in the calculation - I
 : keep hitting my head on the wall.

Don't hit your head on the wall, Kim--this is linux!

Let us know if the above suggestion(s) help you get split access working
correctly.

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/