A question about routing cache (for load balancing).

A question about routing cache (for load balancing).

[Eliezer Croitoru]

If it's not the right place please help out.
I have seen that there was a commit in the linux kernel that removes or 
removes in a way the ipv4 route cache.
I Had couple questions to make sure I understood how it all chances from 
with cache to without cache.
and also is there any level of routing cache that is still in place and 
works??

Thanks,
Eliezer

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

Hi,

The cache control for load balance can be done with:

1. scheduled flush (to remove old entries)
/proc/sys/net/ipv4/route/gc_interval

2. Lifetime of each learned route
/proc/sys/net/ipv4/route/gc_timeout

If you want a aggressive load balance
echo 1 > /proc/sys/net/ipv4/route/gc_interval
echo 1 > /proc/sys/net/ipv4/route/gc_timeout

But the ideal is not to change the gc_timeout.
You certainly have problems with https.

I tend to set gc_interval with 1 and  a higher value for gc_timeout.


2013/11/4 Eliezer Croitoru <eliezer@ngtech.co.il>:
> If it's not the right place please help out.
> I have seen that there was a commit in the linux kernel that removes or
> removes in a way the ipv4 route cache.
> I Had couple questions to make sure I understood how it all chances from
> with cache to without cache.
> and also is there any level of routing cache that is still in place and
> works??
>
> Thanks,
> Eliezer
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: A question about routing cache (for load balancing).

[Eliezer Croitoru]

On 11/07/2013 12:26 PM, Humberto Jucá wrote:
> Hi,
>
> The cache control for load balance can be done with:
By load balance I am talking about Route load balance.

>
> 1. scheduled flush (to remove old entries)
> /proc/sys/net/ipv4/route/gc_interval
OK so every 60 secs remove the old entries?
>
> 2. Lifetime of each learned route
> /proc/sys/net/ipv4/route/gc_timeout
which I should see using "ip route show cache" ?
>
> If you want a aggressive load balance
> echo 1 > /proc/sys/net/ipv4/route/gc_interval
> echo 1 > /proc/sys/net/ipv4/route/gc_timeout
Agrressive load balance? how exactly?
we are talking about route which will have series issues with that.
>
> But the ideal is not to change the gc_timeout.
> You certainly have problems with https.
I will have.. but not just with https but with any tcp protocol that 
dont like the packets to be sent 1 to 1 host and 2 to 2 host and 3 to 
host 3.

>
> I tend to set gc_interval with 1 and  a higher value for gc_timeout.

I will try to ask about it more in the netdev later.

Thanks,
Eliezer

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

2013/11/7 Eliezer Croitoru <eliezer@ngtech.co.il>:
> On 11/07/2013 12:26 PM, Humberto Jucá wrote:
> By load balance I am talking about Route load balance.

Yes, i understand it too. I think! rs


> OK so every 60 secs remove the old entries?

Yes, but...

This also depends of gc_timeout value.
The route system will check/flush the route cache every 60 secs
(gc_interval) and "update" into lb algorithm (round robin, by
default), so:

ip route add default \
    nexthop via 10.1.1.1
    nexthop via 10.1.2.1

echo 1 > /proc/sys/net/ipv4/route/gc_interval

This is the same as (every 1s):
echo 1 > /proc/sys/net/ipv4/route/flush

ip route get 200
# This probe result in 10.1.1.1

# wait 1s and
ip route get 201
# This probe result in 10.1.2.1

# wait 1s and
ip route get 202
# This probe result 10.1.1.1

If you increase this value, all tests can result the same gateway in
gc_interval period.

Each learned path will be maintained by gc_timeout.
But this path will be *checked* only every gc_interval.

The result for 200 will be the same until gc_timeout.
This time expire after *300s of inactivity*.

ip route get 200
ip route get 200
ip route get 200

This probe will return the same path: only 10.1.1.1

But, if you change gc_timeout to 1
echo 1 > /proc/sys/net/ipv4/route/gc_timeout

ip route get 200 # wait 1 or 2s
ip route get 200 # wait 1 or 2s
ip route get 200 # wait 1 or 2s

The result will be balanced - i consider this a aggressive load balance.
This is not so complicated in TCP (because the protocol is
connection-oriented), but is very much in UDP.
I refer to the persistence of connection, not the application. You
will certainly have problems with https and email sessions.

So, a larger value for gc_timeout will allow you a greater connection
persistence.

Re: A question about routing cache (for load balancing).

[Eliezer Croitoru]

First thanks!

It helps to understand what was changed.
(notes in the email)

On 11/07/2013 10:59 PM, Humberto Jucá wrote:
> ip route get 200
> # This probe result in 10.1.1.1
>
> # wait 1s and
> ip route get 201
> # This probe result in 10.1.2.1
>
> # wait 1s and
> ip route get 202
> # This probe result 10.1.1.1

This is the part when I got confused while trying to understand the 
"ipv4 route cache removed".
ip route show cache will show "blank".
but when I run ip route get 201 I see the result with "cache".

>
> If you increase this value, all tests can result the same gateway in
> gc_interval period.

Are you talking about "before" the cache removal or "after"?

> Each learned path will be maintained by gc_timeout.
> But this path will be*checked*  only every gc_interval.
>
> The result for 200 will be the same until gc_timeout.
> This time expire after *300s of inactivity*.
>
> ip route get 200
> ip route get 200
> ip route get 200
>
> This probe will return the same path: only 10.1.1.1

Which is not happens after 3.6 kernel and the cache removal.
try a nice example:
$ watch -d --interval=0.1 ip route get 200
and see what I am talking about.

OK so the next scenario:
Client: 192.168.1.1/24

Lan Router: 192.168.1.254/24
Lan Router wan side:192.168.100.254/24

Wan Router 1:192.168.100.1/24
Wan Router 1-wanip: 3.3.3.3

Wan Router 1:192.168.100.2/24
Wan Router 1-wanip: 4.4.4.4

Simple HTTP\SMTP\SSH\TCP server: 6.6.6.6

Client -> SYN --> Lan ROUTER: --> *WAN-router1*(which does NAT) --> BIG 
INTERNET --> TCP server

TCP server SYN-ACK --> BIG INTERNET -> *WAN-router1*(NAT) --> Lan ROUTER 
--> Client

Client -> ACK --> Lan ROUTER: --> *WAN-router2*(which does NAT) --> BIG 
INTERNET --> TCP server

OK so now stop and feel the TCP server FW:
"Hmm what is this strange packet?? I think it's an invalid packet and 
the sentence for this one is *DROP*"
In the application level it will be almost the same:
"Hmm I do not recall any existing connection from this IP so 
*DROP*\ignore that"

For a simple router that handles internet traffic simple LoadBalancer 
router will not have any effect but in the case of TCP load balancing I 
am almost certain that IPTABLES will need do a thing or two about that.

Am I right about the direction of how it goes?

Thank,
Eliezer

>
> But, if you change gc_timeout to 1
> echo 1 > /proc/sys/net/ipv4/route/gc_timeout
>
> ip route get 200 # wait 1 or 2s
> ip route get 200 # wait 1 or 2s
> ip route get 200 # wait 1 or 2s
>
> The result will be balanced - i consider this a aggressive load balance.
> This is not so complicated in TCP (because the protocol is
> connection-oriented), but is very much in UDP.
> I refer to the persistence of connection, not the application. You
> will certainly have problems with https and email sessions.
>
> So, a larger value for gc_timeout will allow you a greater connection
> persistence.

Re: A question about routing cache (for load balancing).

[Neal Murphy]

On Thursday, November 07, 2013 05:03:06 PM Eliezer Croitoru wrote:

> OK so the next scenario:
> Client: 192.168.1.1/24
> 
> Lan Router: 192.168.1.254/24
> Lan Router wan side:192.168.100.254/24
> 
> Wan Router 1:192.168.100.1/24
> Wan Router 1-wanip: 3.3.3.3
> 
> Wan Router 1:192.168.100.2/24
> Wan Router 1-wanip: 4.4.4.4

This should be Wan Router 2?

> 
> Simple HTTP\SMTP\SSH\TCP server: 6.6.6.6
> 
> Client -> SYN --> Lan ROUTER: --> *WAN-router1*(which does NAT) --> BIG
> INTERNET --> TCP server
> 
> TCP server SYN-ACK --> BIG INTERNET -> *WAN-router1*(NAT) --> Lan ROUTER
> --> Client
> 
> Client -> ACK --> Lan ROUTER: --> *WAN-router2*(which does NAT) --> BIG
> INTERNET --> TCP server
> 
> OK so now stop and feel the TCP server FW:
> "Hmm what is this strange packet?? I think it's an invalid packet and
> the sentence for this one is *DROP*"
> In the application level it will be almost the same:
> "Hmm I do not recall any existing connection from this IP so
> *DROP*\ignore that"
> 

In order for you to do what you want to (balance the load--to arbitrary 
internet hosts--across two links), don't you need a single NAT with a single 
public address, and BGP (or equivalent) to inform the internet routers that 
your single IP can be reached via either route?

With the setup as described, can you only load balance at the conn level, not 
at the packet level?

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

2013/11/7 Eliezer Croitoru <eliezer@ngtech.co.il>:
> This is the part when I got confused while trying to understand the "ipv4
> route cache removed".
> ip route show cache will show "blank".
> but when I run ip route get 201 I see the result with "cache".

I was a little confused... rs
Are you working with a *disabled routing cache kernel*?

IMHO, as the Neal Murphy comment... the configuration that you showed
does not allow balance per packet.
You need the routing cache to avoid this problem. Just set gc_interval
and gc_timeout values according to what you need.

During 60sec (default) you get the same gateway (for all lookups).
But each learned path will be maintained by gc_timeout sec.
After this time, the system selects a new gateway and eliminates expired paths.

Just decrease these values and not set to 0.

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

And another doubt
How do you select the alternative paths... It is under the fib control
(only route rules) or using fwmark (or mixed)?

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

You're talking about this:
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=89aef8921bfbac22f00e04f8450f6e447db13e42

I will need to research about it too.

2013/11/7 Humberto Jucá <betolj@gmail.com>:
> And another doubt
> How do you select the alternative paths... It is under the fib control
> (only route rules) or using fwmark (or mixed)?

Re: A question about routing cache (for load balancing).

[Eliezer Croitoru]

On 11/08/2013 02:39 AM, Humberto Jucá wrote:
> You're talking about this:
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=89aef8921bfbac22f00e04f8450f6e447db13e42
Exactly!!!

> I will need to research about it too.

The main problem with LoadBalancing based on route cache is that it's 
not real loadbalancing but rather an exploit of the routing capabilities 
design.

A route cache is per *route*
means all traffic from this "src ip" that travels throw "this local ip" 
towards "this remote ip" will continue to be routed like this for the 
next 300 secs..
as a route cache entry states:
# ip route get 209.169.10.131
209.169.10.131 via 192.168.10.254 dev eth0  src 192.168.10.1
     cache  mtu 1500 advmss 1460 hoplimit 64
##end

So let say I am issuing 20 connections towards the same host the exact 
same gateway will be used as long as the garbage collection will not 
remove the entry.
In this time the nexthop\gateway could fall down and get up about 60++ 
times..

So from what I understood from the change in the kernel is that a 
routing system should use the FIB to calculate the right path (10Mpps is 
enough?) as expected from a dynamic routing system on the packet level 
instead of routing the packets based on a cache and a FIB lookup in a 
case needed(which means it can take two lookups one for the cache and 
second using the FIB).

In a case of LoadBalancer I would assume there is a need for iptables 
connection marking which has an option to really follow the TCP and UDP 
connections and not just routing based on cache.
In any case IPTABLES based loadbalancing of TCP level (not an 
application level) can take a bit more cycles and a bit more RAM but it 
still faster then any proxy application.

But an application that monitors the LB router and the services servers 
load constantly can change "static" routes to make sure that the load is 
distributed.
In this case there must be some connection tracking on the LB to make 
sure that the TCP connections will not just break to the clients in the 
middle of a connection which can lead to a "read error" for example.

How to handle these errors? I think it's another subject which I want to 
later on read more about.

Eliezer

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

I think I'll miss some scripts! rs
I dont belive in this!


Not tested, but might work:

Client: 192.168.1.1/24

Lan Router: 192.168.1.254/24
Lan Router wan side:192.168.100.254/24

Wan IF: eth1

Wan Router 1:192.168.100.1/24
Wan Router 1-wanip: 3.3.3.3

Wan Router 2:192.168.100.2/24
Wan Router 2-wanip: 4.4.4.4


Probably I would use VLAN (mark in vlan interface).
But I think you can resolve with realms (I do not know if it works).

ip route flush table 100
ip route add throw 10.0.0.0/8 table 100 realm 100
ip route add throw 192.168.1.0/24 realm 100
ip route add default via 192.168.100.1 realm 100

ip route flush table 101
ip route add throw 10.0.0.0/8 table 100 realm 101
ip route add throw 192.168.1.0/24 realm 101
ip route add default via 192.168.100.2 realm 101

ip route flush table 200
ip route add default \
    nexthop via 192.168.100.1 realm 100 \
    nexthop via 192.168.100.2 realm 101

ip rule add prio 10 table main
ip rule add prio 20 from 192.168.100.1 table 100
ip rule add prio 21 from 192.168.100.2 table 101
ip rule add prio 22 fwmark 0x100 table 100
ip rule add prio 23 fwmark 0x101 table 101
ip rule add prio 50 table 200

ip route del default

iptables -t mangle -A POSTROUTING -o eth1 -m realm --realm 100 -j
CONNMARK --set-mark 0x100
iptables -t mangle -A POSTROUTING -o eth1 -m realm --realm 101 -j
CONNMARK --set-mark 0x101

iptables -t mangle -A POSTROUTING -o eth1 -j CONNMARK --save-mark

iptables -t mangle -I PREROUTING -i eth1 -m mark ! --mark 0x0 -j
CONNMARK --restore-mark

2013/11/7 Eliezer Croitoru <eliezer@ngtech.co.il>:
> On 11/08/2013 02:39 AM, Humberto Jucá wrote:
>>
>> You're talking about this:
>>
>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=89aef8921bfbac22f00e04f8450f6e447db13e42
>
> Exactly!!!
>
>
>> I will need to research about it too.
>
>
> The main problem with LoadBalancing based on route cache is that it's not
> real loadbalancing but rather an exploit of the routing capabilities design.
>
> A route cache is per *route*
> means all traffic from this "src ip" that travels throw "this local ip"
> towards "this remote ip" will continue to be routed like this for the next
> 300 secs..
> as a route cache entry states:
> # ip route get 209.169.10.131
> 209.169.10.131 via 192.168.10.254 dev eth0  src 192.168.10.1
>     cache  mtu 1500 advmss 1460 hoplimit 64
> ##end
>
> So let say I am issuing 20 connections towards the same host the exact same
> gateway will be used as long as the garbage collection will not remove the
> entry.
> In this time the nexthop\gateway could fall down and get up about 60++
> times..
>
> So from what I understood from the change in the kernel is that a routing
> system should use the FIB to calculate the right path (10Mpps is enough?) as
> expected from a dynamic routing system on the packet level instead of
> routing the packets based on a cache and a FIB lookup in a case needed(which
> means it can take two lookups one for the cache and second using the FIB).
>
> In a case of LoadBalancer I would assume there is a need for iptables
> connection marking which has an option to really follow the TCP and UDP
> connections and not just routing based on cache.
> In any case IPTABLES based loadbalancing of TCP level (not an application
> level) can take a bit more cycles and a bit more RAM but it still faster
> then any proxy application.
>
> But an application that monitors the LB router and the services servers load
> constantly can change "static" routes to make sure that the load is
> distributed.
> In this case there must be some connection tracking on the LB to make sure
> that the TCP connections will not just break to the clients in the middle of
> a connection which can lead to a "read error" for example.
>
> How to handle these errors? I think it's another subject which I want to
> later on read more about.
>
> Eliezer
>
>
>
>

Re: A question about routing cache (for load balancing).

[Humberto Jucá]

# Remove the mark inversion
iptables -t mangle -I PREROUTING -i eth1 -m mark --mark 0x0 -j
CONNMARK --restore-mark

2013/11/7 Humberto Jucá <betolj@gmail.com>:
> I think I'll miss some scripts! rs
> I dont belive in this!
>
>
> Not tested, but might work:
>
> Client: 192.168.1.1/24
>
> Lan Router: 192.168.1.254/24
> Lan Router wan side:192.168.100.254/24
>
> Wan IF: eth1
>
> Wan Router 1:192.168.100.1/24
> Wan Router 1-wanip: 3.3.3.3
>
> Wan Router 2:192.168.100.2/24
> Wan Router 2-wanip: 4.4.4.4
>
>
> Probably I would use VLAN (mark in vlan interface).
> But I think you can resolve with realms (I do not know if it works).
>
> ip route flush table 100
> ip route add throw 10.0.0.0/8 table 100 realm 100
> ip route add throw 192.168.1.0/24 realm 100
> ip route add default via 192.168.100.1 realm 100
>
> ip route flush table 101
> ip route add throw 10.0.0.0/8 table 100 realm 101
> ip route add throw 192.168.1.0/24 realm 101
> ip route add default via 192.168.100.2 realm 101
>
> ip route flush table 200
> ip route add default \
>     nexthop via 192.168.100.1 realm 100 \
>     nexthop via 192.168.100.2 realm 101
>
> ip rule add prio 10 table main
> ip rule add prio 20 from 192.168.100.1 table 100
> ip rule add prio 21 from 192.168.100.2 table 101
> ip rule add prio 22 fwmark 0x100 table 100
> ip rule add prio 23 fwmark 0x101 table 101
> ip rule add prio 50 table 200
>
> ip route del default
>
> iptables -t mangle -A POSTROUTING -o eth1 -m realm --realm 100 -j
> CONNMARK --set-mark 0x100
> iptables -t mangle -A POSTROUTING -o eth1 -m realm --realm 101 -j
> CONNMARK --set-mark 0x101
>
> iptables -t mangle -A POSTROUTING -o eth1 -j CONNMARK --save-mark
>
> iptables -t mangle -I PREROUTING -i eth1 -m mark ! --mark 0x0 -j
> CONNMARK --restore-mark
>
> 2013/11/7 Eliezer Croitoru <eliezer@ngtech.co.il>:
>> On 11/08/2013 02:39 AM, Humberto Jucá wrote:
>>>
>>> You're talking about this:
>>>
>>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=89aef8921bfbac22f00e04f8450f6e447db13e42
>>
>> Exactly!!!
>>
>>
>>> I will need to research about it too.
>>
>>
>> The main problem with LoadBalancing based on route cache is that it's not
>> real loadbalancing but rather an exploit of the routing capabilities design.
>>
>> A route cache is per *route*
>> means all traffic from this "src ip" that travels throw "this local ip"
>> towards "this remote ip" will continue to be routed like this for the next
>> 300 secs..
>> as a route cache entry states:
>> # ip route get 209.169.10.131
>> 209.169.10.131 via 192.168.10.254 dev eth0  src 192.168.10.1
>>     cache  mtu 1500 advmss 1460 hoplimit 64
>> ##end
>>
>> So let say I am issuing 20 connections towards the same host the exact same
>> gateway will be used as long as the garbage collection will not remove the
>> entry.
>> In this time the nexthop\gateway could fall down and get up about 60++
>> times..
>>
>> So from what I understood from the change in the kernel is that a routing
>> system should use the FIB to calculate the right path (10Mpps is enough?) as
>> expected from a dynamic routing system on the packet level instead of
>> routing the packets based on a cache and a FIB lookup in a case needed(which
>> means it can take two lookups one for the cache and second using the FIB).
>>
>> In a case of LoadBalancer I would assume there is a need for iptables
>> connection marking which has an option to really follow the TCP and UDP
>> connections and not just routing based on cache.
>> In any case IPTABLES based loadbalancing of TCP level (not an application
>> level) can take a bit more cycles and a bit more RAM but it still faster
>> then any proxy application.
>>
>> But an application that monitors the LB router and the services servers load
>> constantly can change "static" routes to make sure that the load is
>> distributed.
>> In this case there must be some connection tracking on the LB to make sure
>> that the TCP connections will not just break to the clients in the middle of
>> a connection which can lead to a "read error" for example.
>>
>> How to handle these errors? I think it's another subject which I want to
>> later on read more about.
>>
>> Eliezer
>>
>>
>>
>>