[LARTC] Problems with Routing and Masquerading

[LARTC] Problems with Routing and Masquerading

[Vinod Chandran]

Hi,

I have a linux box which balances load between two interfaces ( say WAN1 
and WAN2). I have masquerading on for any request coming from LAN to the 
outside world.

The setup is in such a way that WAN1 drops packets with source ip 
belonging to WAN2's network and viceversa.
For some strange reason, I find that packet coming out from the WAN 
interface has source address of WAN2 and thereby getting dropped.

When I check the route cache , I find that for the same source and 
destination, I have two route cache entries

192.168.52.66   192.168.26.73   192.168.19.76         0      0        0 eth1
192.168.52.66   192.168.26.73   192.168.20.25   i     0      0       23 eth2

Here 192.168.19.76 is the WAN1 gateway and 192.168.20.25 is WAN2 
gateway, as we see the packets are going out through WAN2 , but the 
masquerading has happened to the WAN1 IP address.

It seems to me that the root of the problem is the creation of the two 
cache entries. Any idea why this happens, and how it can be avoided.

Thanks and Regards,
Vinod C

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Erik Slagter]

[-- Attachment #1.1: Type: text/plain, Size: 317 bytes --]

On Tue, 2006-05-30 at 20:02 +0530, Vinod Chandran wrote:
> It seems to me that the root of the problem is the creation of the two 
> cache entries. Any idea why this happens, and how it can be avoided.

What does "ip route get" say? I think this tool will be the key to the
solution, look at the "src address".

[-- Attachment #1.2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 2771 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Vinod Chandran]

Hi Erik,

Ip route get returns
192.168.26.73 via 192.168.19.76 dev eth1  src 192.168.19.29
    cache  mtu 1500 advmss 1460

Here too it gives me WAN1s IP address which is the same as the 
masqueraded one, but the packets are seen going out through WAN2.

Thanks and Regards,
Vinod C


Erik Slagter wrote:

>On Tue, 2006-05-30 at 20:02 +0530, Vinod Chandran wrote:
>  
>
>>It seems to me that the root of the problem is the creation of the two 
>>cache entries. Any idea why this happens, and how it can be avoided.
>>    
>>
>
>What does "ip route get" say? I think this tool will be the key to the
>solution, look at the "src address".
>  
>
>------------------------------------------------------------------------
>
>_______________________________________________
>LARTC mailing list
>LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>  
>

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problems with Routing and Masquerading

[Andrew Lyon]

>AFAICR using MASQUERADE with multipath is not recommended (but maybe this
is outdated, plz correct me if I'm wrong)

I have had problems using MASQUERADE with multipath on a 2.4.31 kernel box,
with two outbound default routes I got messages about "rustys brain broke"
and things like ICMP etc suffered packet loss.

Not recommended, SNAT works perfectly :)

Andy
Registered Office: J.O. Sims Ltd, Pudding Lane, Pinchbeck, Spalding, Lincs. PE11 3TJ
Company reg No: 2084187 Vat reg No: GB 437 4621 47
Tel: +44 (0) 1775 842100 Fax: +44 (0) 1775 842101 Web: www.josims.com
Email:enquiries@josims.com
The information contained in this e-mail is confidential and is intended for the addressee only. The contents of this e-mail must not be disclosed or copied without the sender's consent. If you are not the intended recipient of the message, please notify the sender immediately, and delete the message. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the company. No commitment may be inferred from the contents unless explicitly stated. The company does not take any responsibility for the personal views of the author. This message has been scanned for viruses before sending, but the company does not accept any responsibility for infection and recommends that you scan any attachments.JOSEDV001TAG
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Luciano Ruete]

On Tuesday 30 May 2006 11:32, Vinod Chandran wrote:
> Hi,
>
> I have a linux box which balances load between two interfaces ( say WAN1
> and WAN2). I have masquerading on for any request coming from LAN to the
> outside world.

If WAN ips are static, you can use 
iptables -t nat ... -o WAN1 -j SNAT --to x.x.x.x
iptables -t nat ... -o WAN2 -j SNAT --to x.x.x.x

AFAICR using MASQUERADE with multipath is not recommended (but maybe this is 
outdated, plz correct me if I'm wrong)

> The setup is in such a way that WAN1 drops packets with source ip
> belonging to WAN2's network and viceversa.
> For some strange reason, I find that packet coming out from the WAN
> interface has source address of WAN2 and thereby getting dropped.
>
> When I check the route cache , I find that for the same source and
> destination, I have two route cache entries

TOS is also computed to make a route cache entry, so it is possible and 
natural to 2 entries with same src and dst. Be aware that some ssh clients 
change the packet TOS after negotiation has been done, this could cause you 
get routed over a different link in the middle of your session, making 
impossible to do a simple ssh. 

> 192.168.52.66   192.168.26.73   192.168.19.76         0      0        0
> eth1 192.168.52.66   192.168.26.73   192.168.20.25   i     0      0      
> 23 eth2
>
> Here 192.168.19.76 is the WAN1 gateway and 192.168.20.25 is WAN2
> gateway, as we see the packets are going out through WAN2 , but the
> masquerading has happened to the WAN1 IP address.
>
> It seems to me that the root of the problem is the creation of the two
> cache entries. Any idea why this happens, and how it can be avoided.

i think the root of the problem is that you are using MASQUERADE and not SNAT. 
And somehow MASQUERADE has not an accurate method to guess the right ip 
address to do the NAT.

Besides that, you need to solve the problems that multipath will arise, like 
TOS situation described above or route cache expiration, that could made long 
term conns to be routed over a new iface.  The solutions  i know are 
CONNMARK(kernel>=2.6.12) and julian's  patches[1].  
Personally i prefer CONNMARK.

[1] http://www.ssi.bg/~ja/
-- 
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

RE: [LARTC] Problems with Routing and Masquerading

[Andrew Lyon]

 
My apologies for sending email disclaimer crap to the list, I forgot the
magic string to prevent it from being appended :(.

Oops.
Andy
JOSEDV001TAG
-----Original Message-----
From: Andrew Lyon [mailto:andrew.lyon@josims.com] 
Sent: 30 May 2006 17:13
To: 'lartc@mailman.ds9a.nl'
Subject: RE: [LARTC] Problems with Routing and Masquerading

>AFAICR using MASQUERADE with multipath is not recommended (but maybe 
>this
is outdated, plz correct me if I'm wrong)

I have had problems using MASQUERADE with multipath on a 2.4.31 kernel box,
with two outbound default routes I got messages about "rustys brain broke"
and things like ICMP etc suffered packet loss.

Not recommended, SNAT works perfectly :)

Andy
Registered Office: J.O. Sims Ltd, Pudding Lane, Pinchbeck, Spalding, Lincs.
PE11 3TJ Company reg No: 2084187 Vat reg No: GB 437 4621 47
Tel: +44 (0) 1775 842100 Fax: +44 (0) 1775 842101 Web: www.josims.com
Email:enquiries@josims.com The information contained in this e-mail is
confidential and is intended for the addressee only. The contents of this
e-mail must not be disclosed or copied without the sender's consent. If you
are not the intended recipient of the message, please notify the sender
immediately, and delete the message. The statements and opinions expressed
in this message are those of the author and do not necessarily reflect those
of the company. No commitment may be inferred from the contents unless
explicitly stated. The company does not take any responsibility for the
personal views of the author. This message has been scanned for viruses
before sending, but the company does not accept any responsibility for
infection and recommends that you scan any attachments.JOSEDV001TAG
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Jason Boxman]

Luciano Ruete wrote:
<snip>
> Besides that, you need to solve the problems that multipath will arise, like
> TOS situation described above or route cache expiration, that could made
> long
> term conns to be routed over a new iface.  The solutions  i know are
> CONNMARK(kernel>=2.6.12) and julian's  patches[1].
> Personally i prefer CONNMARK.

Could you elaborate a little more on the CONNMARK method?

Thanks.



_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Luciano Ruete]

On Tuesday 30 May 2006 13:23, Jason Boxman wrote:
> Luciano Ruete wrote:
> <snip>
>
> > Besides that, you need to solve the problems that multipath will arise,
> > like TOS situation described above or route cache expiration, that could
> > made long
> > term conns to be routed over a new iface.  The solutions  i know are
> > CONNMARK(kernel>=2.6.12) and julian's  patches[1].
> > Personally i prefer CONNMARK.
>
> Could you elaborate a little more on the CONNMARK method?

#by-pass rules if it is already MARKed
iptables -t mangle -A POSTROUTING -m mark  --mark ! 0 -j ACCEPT 
#1st packets(from a connection) will arrive here 
iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1
iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

#restore mark before ROUTING decision
iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark

#route commands
ip ro add default  nexthop via x.x.x.x dev eth1 weight 1 nexthop via
y.y.y.y dev eth2
ip route add default table provider1 via x.x.x.x dev eth1
ip route add default table provider2 via y.y.y.y dev eth2
# and most important
ip rule add fwmark 0x1 table provider1
ip rule add fwmark 0x2 table provider2


Some notes:
-The example uses 2 ifaces, but is scalable to any (i have it working with 5)
-FORWARD could be used instead of POSRTOUTING, it depends on your needs
-If you have a large network, think in change the default conntrack table size 
and hash-size
 ip_conntrack hashsize=xxx
 echo xxx  > /proc/sys/net/ipv4/ip_conntrack_max
 #lnstat is your friend, will help to find the magic numbers
 lnstat -f ip_conntrack -i 1 -c 1 
-Full discussion about this solution on this thread[1][2] (sorry spanish only)
-Credits to diego woitasen who point me out to this kind of solution

[1] http://www.lugmen.org.ar/pipermail/lug-list/2006-April/041078.html
[2] http://www.lugmen.org.ar/pipermail/lug-list/2006-May/041084.html
-- 
Luciano
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Raj Mathur]

>>>>> "Jason" = Jason Boxman <jasonb@edseek.com> writes:

    Jason> Luciano Ruete wrote: <snip>
    >> Besides that, you need to solve the problems that multipath
    >> will arise, like TOS situation described above or route cache
    >> expiration, that could made long term conns to be routed over a
    >> new iface.  The solutions i know are CONNMARK(kernel>=2.6.12)
    >> and julian's patches[1].  Personally i prefer CONNMARK.

    Jason> Could you elaborate a little more on the CONNMARK method?

I second that motion -- not too clear on the interaction between SNAT,
multiple interfaces, multiple default routes and CONNMARK </aol
mode="metoo">.  If someone could take out the time to make a complete
example with (say) 2 outgoing interfaces, I promise a small GPL script
in exchange which would automate the whole process.

Actually the script's already made, but it doesn't use CONNMARK and
suffers from the problems Jason describes and as documented in:

  http://mailman.ds9a.nl/pipermail/lartc/2006q1/018220.html

Regards,

-- Raju
-- 
Raj Mathur                raju@kandalaya.org      http://kandalaya.org/
       GPG: 78D4 FC67 367F 40E2 0DD5  0FEF C968 D0EF CC68 D17F
                      It is the mind that moves
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Vinod Chandran]

Hi,

Thanks Jason for the solution. With CONNMARK, I was able to route the 
packets properly.

Yeah, the problem was seen only for SSH sessions, I didnot see the 
problem with the Telnet and Ping sessions. TOS could be the answer to that.

The only change I had to do as far the CONNMARK solution was that in the 
PREROUTING chain, I had to add the rule with "-i eth0" where eth0 is my 
LAN, otherwise the return packets were not reaching the box in LAN.

Thanks and Regards,
Vinod C


Raj Mathur wrote:

>>>>>>"Jason" = Jason Boxman <jasonb@edseek.com> writes:
>>>>>>            
>>>>>>
>
>    Jason> Luciano Ruete wrote: <snip>
>    >> Besides that, you need to solve the problems that multipath
>    >> will arise, like TOS situation described above or route cache
>    >> expiration, that could made long term conns to be routed over a
>    >> new iface.  The solutions i know are CONNMARK(kernel>=2.6.12)
>    >> and julian's patches[1].  Personally i prefer CONNMARK.
>
>    Jason> Could you elaborate a little more on the CONNMARK method?
>
>I second that motion -- not too clear on the interaction between SNAT,
>multiple interfaces, multiple default routes and CONNMARK </aol
>mode="metoo">.  If someone could take out the time to make a complete
>example with (say) 2 outgoing interfaces, I promise a small GPL script
>in exchange which would automate the whole process.
>
>Actually the script's already made, but it doesn't use CONNMARK and
>suffers from the problems Jason describes and as documented in:
>
>  http://mailman.ds9a.nl/pipermail/lartc/2006q1/018220.html
>
>Regards,
>
>-- Raju
>  
>

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Jason Boxman]

Vinod Chandran wrote:
> Hi,
>
> Thanks Jason for the solution. With CONNMARK, I was able to route the
> packets properly.

Cool, but I don't think that was me.



_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Problems with Routing and Masquerading

[Vinod Chandran]

Oh yeah, my bad!
Thanks Luciano for the solution.

Regards,
Vinod C
Jason Boxman wrote:

>Vinod Chandran wrote:
>  
>
>>Hi,
>>
>>Thanks Jason for the solution. With CONNMARK, I was able to route the
>>packets properly.
>>    
>>
>
>Cool, but I don't think that was me.
>
>
>
>_______________________________________________
>LARTC mailing list
>LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
>  
>

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc