[LARTC] Split access, load balancing AND forwarding: HOW?

[LARTC] Split access, load balancing AND forwarding: HOW?

[Andrew McGill]

The LARTC howto correctly describes load balancing and split 
access for traffic from a machine with multiple ISP connections 
(http://www.lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS) -- 
*provided* the traffic originates from the machine itself (i.e. 
traffic regularly handled by the INPUT and OUTPUT chains of 
iptables).

When forwarding traffic from an attached local network, the 
following problems occur with traffic from the local network to 
internet hosts:

1. The ip rule add from x.x.x.x refers to local IP addresses
    before NAT, such as 192.168.0.44, rather than the public IP
    address after NAT (and certainly not both).  This is the
    fundamental problem that causes load balancing and split
    access to be unreliable.

2. Cached routes are dropped periodically from the route cache,
    even while in active use: this causes connection reset errors
    and strange timeouts.

3. To frustrate iptables based work-arounds, routing does not
    obey marks added with iptables -t mangle -A PREROUTING.  It
    seems that ip fwmark rules are not obeyed if the route is
    cached, and the cache hash does not include the firewall mark
    (or maybe it does, but it doesn't work ?!?).  (Interestingly,
    cached routing *does* obey the TOS bits, which makes creative
    work-arounds marginally possible.  There just aren't too many
    TOS values to play with.)

Is there a solution to these problems which works with the official
kernels?  If so, which versions?  If not, which patches resolve these
problems?

&:-)

-- 
Disclaimer: in the event of this disclaimer being incomplete
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Luciano Ruete]

On Wednesday 21 February 2007 11:10, Andrew McGill wrote:
> The LARTC howto correctly describes load balancing and split
> access for traffic from a machine with multiple ISP connections
> (http://www.lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS) --
> *provided* the traffic originates from the machine itself (i.e.
> traffic regularly handled by the INPUT and OUTPUT chains of
> iptables).
>
> When forwarding traffic from an attached local network, the
> following problems occur with traffic from the local network to
> internet hosts:
>
> 1. The ip rule add from x.x.x.x refers to local IP addresses
>     before NAT, such as 192.168.0.44, rather than the public IP
>     address after NAT (and certainly not both).  This is the
>     fundamental problem that causes load balancing and split
>     access to be unreliable.

the 'ip rule' is evaluated before the routing desition so, this is before 
FORWARD and before POSTROUTING(the place where NAT actualy happens) so far. 
Hence, refer to the local IP is correct.

> 2. Cached routes are dropped periodically from the route cache,
>     even while in active use: this causes connection reset errors
>     and strange timeouts.

this is true, and to increase the routes timeout does not help cause at the 
end you have all internet routes cached making your kernel/noswapable RAM to 
kicks out every single app on the host.

The solution is to use CONNTRACK from iptables, full example described in 
this[1] e-mail from the archive. No patches needed.

> 3. To frustrate iptables based work-arounds, routing does not
>     obey marks added with iptables -t mangle -A PREROUTING.  It
>     seems that ip fwmark rules are not obeyed if the route is
>     cached, 

this is not true, the rules are evaluated before the routing desition, so 
fwmark rules works as expected. You really think that this BUG can be in 
linux kernel since early 2.3/2.4 versions and be discovered today?

>     and the cache hash does not include the firewall mark 
>     (or maybe it does, but it doesn't work ?!?).  (Interestingly,
>     cached routing *does* obey the TOS bits, which makes creative
>     work-arounds marginally possible.  There just aren't too many
>     TOS values to play with.)

> Is there a solution to these problems which works with the official
> kernels?  If so, which versions?  If not, which patches resolve these
> problems?

Yes as i pointed out, there is a solution and no patches needed.

[1] http://mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html
-- 
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Ming-Ching Tiew]

From: "Luciano Ruete" <luciano@lugmen.org.ar>
> 
> The solution is to use CONNTRACK from iptables, full example described in 
> this[1] e-mail from the archive. No patches needed.
> 
> [1] http://mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html
>

I think you mean CONNMARK ( not CONNTRACK ) from iptables  ?

The ever popular routing command :-

>
> #route commands
> ip ro add default  nexthop via x.x.x.x dev eth1 weight 1 nexthop via y.y.y.y dev eth2
>

I personal view is that  ***NEVER*** use such a routing statement, or never let 
the system has a chance to use such a routing statement, especially when you are 
doing NAT. The email example above included this routing statement but it is 
not used because the  'ip rule' takes precedence. The multipath weighted cached
based routing is problematic.

I would say it would be better to re-order the the iptables command :-

#restore mark before ROUTING decision
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
#by-pass rules if it is already MARKed
iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
#1st packets(from a connection) will arrive here 
iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

ie restore-mark is moved to the top.

I strongly recommend that the LARTC documentation be updated, especially it 
encourages people to use multipath weighted routing instead of iptables 
based solution.

Cheers.



_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Ming-Ching Tiew]

From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
> 
> I would say it would be better to re-order the the iptables command :-
> 
> #restore mark before ROUTING decision
> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> #by-pass rules if it is already MARKed
> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
> #1st packets(from a connection) will arrive here 
> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
> 
> ie restore-mark is moved to the top.
> 

On more careful reading, I am wondering why it is using POSTROUTING ? 

Shouldn't it all be PREROUTING ?

Cheers.



_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Luciano Ruete]

On Wednesday 21 February 2007 23:58, Ming-Ching Tiew wrote:
> From: "Luciano Ruete" <luciano@lugmen.org.ar>
>
> > The solution is to use CONNTRACK from iptables, full example described in
> > this[1] e-mail from the archive. No patches needed.
> >
> > [1] http://mailman.ds9a.nl/pipermail/lartc/2006q2/018964.html
>
> I think you mean CONNMARK ( not CONNTRACK ) from iptables  ?

sory a brain-o, but in the email refered is well explained.

>
> The ever popular routing command :-
>
> > #route commands
> > ip ro add default  nexthop via x.x.x.x dev eth1 weight 1 nexthop via
> > y.y.y.y dev eth2
>
> I personal view is that  ***NEVER*** use such a routing statement, or never
> let the system has a chance to use such a routing statement, especially
> when you are doing NAT. 

You are ***WRONG*** here :-)

The multipath statement works really great, but is connection state-less 
without the iptables CONNMARK help.

> The email example above included this routing 
> statement but it is not used because the  'ip rule' takes precedence. 

WRONG, the first packet of a trackeable connection does get routed by the 
multipath routing statement. Once routed for one of the weighted gw, it is 
MARKEd and --saved by CONNMARK. The second(and all the rest) packet from that 
connection will use always the same gateway. 

So, 'ip ro nexthop' does the weighted gw selection and balancing, then i use 
CONNMARK to ensure that packets from the same flow keep always the same 
gateway. 

I got this working in production server in 3 ISPs, and belive me, it works 
like a swiss clock. 

> The 
> multipath weighted cached based routing is problematic.

if you do not use something that can track the connection yes, but hey, you 
have CONNMARK now, and before that you can do the same trick(and still can) 
with julian's anastasov patches.

> I would say it would be better to re-order the the iptables command :-
>
> #restore mark before ROUTING decision
> iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> #by-pass rules if it is already MARKed
> iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT
> #1st packets(from a connection) will arrive here
> iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
>
> ie restore-mark is moved to the top.

it produces the same result, i think is better to understand if the restore 
command goes at the end, cause first you talk about the mark, and at the end 
you talk about restore it. 
If you put the restore first, the newbie will ask "what the hell i am 
restoring???". But is a matter of taste. 

> I strongly recommend that the LARTC documentation be updated, especially it
> encourages people to use multipath weighted routing instead of iptables
> based solution.

The docs are outdated but tecnically ok, they where wrote by people who really 
know about the matters. It is more dangerous that you say things like the 
ones you wrote in this email (wich are enourmosly wrong) and google indexed 
them. 

-- 
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Luciano Ruete]

On Thursday 22 February 2007 01:57, Ming-Ching Tiew wrote:
> From: "Ming-Ching Tiew" <mingching.tiew@redtone.com>
>
> > I would say it would be better to re-order the the iptables command :-
> >
> > #restore mark before ROUTING decision
> > iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
> > #by-pass rules if it is already MARKed
> > iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT
> > #1st packets(from a connection) will arrive here
> > iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
> > iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
> > iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
> >
> > ie restore-mark is moved to the top.
>
> On more careful reading, I am wondering why it is using POSTROUTING ?
>
> Shouldn't it all be PREROUTING ?

_NO_, cause i need that 'multipath routing' makes the 'weighted routing 
decision' in the first packet of each new connection. Once it is routed, all 
the other packets from same flow are hacked in PREROUTING, they mark are 
resotred and ip rule garantize that they will go by the same gateway as the 
first packet.

This solution works in theory and in practice, so plz, get your hands dirty 
before you post your next great idea.

-- 
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Roy Orpiano]

[-- Attachment #1.1: Type: text/plain, Size: 338 bytes --]

Hello,

Can anyone tell me what is the effect, in terms of latency, if I have:
1.  1 thousand defined HTB classes which represents 1 thousand users.
     I filter the traffic based on source IP address.
2.  multiple class hierarchy inside a class.  Is there a limit on the depth
of hierarchy
     in HTB?

Thank you very much in advance.

[-- Attachment #1.2: Type: text/html, Size: 422 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Ming-Ching Tiew]

From: "Luciano Ruete" <luciano@lugmen.org.ar>
> 
> This solution works in theory and in practice, so plz, get your hands dirty 
> before you post your next great idea.
> 

I understand your explanation fully but believe me I also have got 
hand-on experience with using the alternative, ie

1. I don't use multipath weight routing. 
2. I use PREROUTING all the way, ie I don't use POSTROUTING.

Instead, I use iptables  'recent' and 'statistics'/'random' match to achieve
load sharing.

I have use this for many years already, believe me I am not theoretical.
It's just a matter of different ways to doing things. If you search the web
it will come upon many others using the same method I used.

Cheers


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Alex Samad]

[-- Attachment #1.1: Type: text/plain, Size: 1177 bytes --]

On Fri, Feb 23, 2007 at 03:23:42PM +0800, Ming-Ching Tiew wrote:
> From: "Luciano Ruete" <luciano@lugmen.org.ar>
> > 
> > This solution works in theory and in practice, so plz, get your hands dirty 
> > before you post your next great idea.
> > 
> 
> I understand your explanation fully but believe me I also have got 
> hand-on experience with using the alternative, ie
> 
> 1. I don't use multipath weight routing. 
> 2. I use PREROUTING all the way, ie I don't use POSTROUTING.
> 
> Instead, I use iptables  'recent' and 'statistics'/'random' match to achieve
> load sharing.

hi

sorry missed the previous bits of the thread, could you post the relevant info,
interested to see how this works and why you would pick it over the multipath
method


> 
> I have use this for many years already, believe me I am not theoretical.
> It's just a matter of different ways to doing things. If you search the web
> it will come upon many others using the same method I used.
> 
> Cheers
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 

[-- Attachment #1.2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Split access, load balancing AND forwarding: HOW?

[Ming-Ching Tiew]

From: "Alex Samad" <alex@samad.com.au>

> hi
>
> sorry missed the previous bits of the thread, could you post the relevant info,
> interested to see how this works and why you would pick it over the multipath
> method

Please note my checking of marked traffic is not ( according to the earlier posts )

> iptables -t mangle .... -m mark  --mark ! 0 -j ACCEPT

However, it is :-

> iptables -t mangle  .... -m mark  ! --mark  0 -j ACCEPT

I leave it to you guys to decide which is the correct syntax.

The code below is taken from part of my bigger code :-

Cheers.
---------------------code-------------------------------------------

LINK1_MARK=5
LINK2_MARK=7
OUTSIDE_DEV_WEIGHT=0.5
INSIDE_DEVICE=eth0
OUTSIDE_DEVICE=eth1
OUTSIDE_DEVICE2=eth2

SAVEMARK="-m mark ! --mark 0 -j CONNMARK --save-mark"
ACCEPTMARK="-m mark ! --mark 0 -j ACCEPT"
SETMARK1="-j MARK --set-mark ${LINK1_MARK}"
SETMARK2="-j MARK --set-mark ${LINK2_MARK}"
#
#first, restore and accept the mark if there is any
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING ${ACCEPTMARK}
#handle inbound for link1
iptables -t mangle -A PREROUTING -i ${OUTSIDE_DEVICE} ${SETMARK1}
iptables -t mangle -A PREROUTING -i ${OUTSIDE_DEVICE} ${SAVEMARK}
iptables -t mangle -A PREROUTING ${ACCEPTMARK}
#handle inbound for link2
iptables -t mangle -A PREROUTING -i ${OUTSIDE_DEVICE2} ${SETMARK2}
iptables -t mangle -A PREROUTING -i ${OUTSIDE_DEVICE2} ${SAVEMARK}
iptables -t mangle -A PREROUTING ${ACCEPTMARK}

# (other features implementation snipped )

#handle recent outbound
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} -m recent --name link1 \
 --update --second 300 ${SETMARK1}
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} -m recent --name link2 \
 --update --second 300 ${SETMARK2}
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} ${SAVEMARK}
iptables -t mangle -A PREROUTING ${ACCEPTMARK}
#
#non-recent outbound randomly allocated
#
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} \
  -m statistic --mode random --probability ${OUTSIDE_DEV2_WEIGHT} \
  -m recent --name link2 --set ${SETMARK2}
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} ${SAVEMARK}
iptables -t mangle -A PREROUTING ${ACCEPTMARK}
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} \
  -m recent --name link1 --set ${SETMARK1}
iptables -t mangle -A PREROUTING -i ${INSIDE_DEVICE} ${SAVEMARK}
iptables -t mangle -A PREROUTING ${ACCEPTMARK}




_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc