[LARTC] Masq/route based on port

[LARTC] Masq/route based on port

[Miron]

I have following setup:

- eth0 is an internal network
- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)

I would like to masquerade port 80 through eth2, but all other traffic 
should be masq'ed through eth1.

My routing configuration:

    (default route in main table is 1.1.1.1)

    ip rule add fwmark 2 pref 1002 table 666

    ip route flush table 666
    ip route add default via 2.2.2.1 dev eth3 proto static table 666
    ip route flush cache

My firewall configuration:
    iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
--set-mark 2
    iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
    iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128

Unfortunately, this does not work.  Outgoing packets are fine.  Incoming 
packets on port 80 are not de-masqueraded and do not reach the internal 
hosts.

Also, if I change the ip rule above to be based on the source address 
(instead of a mark), connections start working fine.

Here is the output of 'ip rule ls', to prove that I do have fwmark compiled:
    0:      from all lookup local
    1002:   from all fwmark        2 lookup http
    32766:  from all lookup main
    32767:  from all lookup 253

I am wondering if there is some kind of bug related to the interaction 
between fwmark and NAT. Any ideas?

Thanks,
Miron Cuperman



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

RE: [LARTC] Masq/route based on port

[Greg Scott]

This doesn't seem right:  

> My firewall configuration:
>   iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
> --set-mark 2
>   iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
>   iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128

I'm not the expert, but I don't think you want to SNAT.  

You're marking inbound packets with destination port 80 and then putting
rules in the POSTROUTING table to fudge in a different IP address for
outbound packets.  That doesn't seem right.  It doesn't redirect inbound
packets the way you want.  

I think you want to DNAT instead of SNAT and forget about marking packets.  
Set up some PREROUTING rules and DNAT all incoming port 80 stuff over to 
the interface you want.  That should be all you need to do because the 
connection tracking should take care of getting the reply packets from your
internal web server back to where they belong.  

Verify this with the experts before you do it, but I think I'm right on 
this one.

- Greg Scott



-----Original Message-----
From: Miron [mailto:miron@hyper.to]
Sent: Thursday, December 06, 2001 3:58 AM
To: lartc@mailman.ds9a.nl
Subject: [LARTC] Masq/route based on port


I have following setup:

- eth0 is an internal network
- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)

I would like to masquerade port 80 through eth2, but all other traffic 
should be masq'ed through eth1.

My routing configuration:

    (default route in main table is 1.1.1.1)

    ip rule add fwmark 2 pref 1002 table 666

    ip route flush table 666
    ip route add default via 2.2.2.1 dev eth3 proto static table 666
    ip route flush cache

My firewall configuration:
    iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
--set-mark 2
    iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
    iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128

Unfortunately, this does not work.  Outgoing packets are fine.  Incoming 
packets on port 80 are not de-masqueraded and do not reach the internal 
hosts.

Also, if I change the ip rule above to be based on the source address 
(instead of a mark), connections start working fine.

Here is the output of 'ip rule ls', to prove that I do have fwmark compiled:
    0:      from all lookup local
    1002:   from all fwmark        2 lookup http
    32766:  from all lookup main
    32767:  from all lookup 253

I am wondering if there is some kind of bug related to the interaction 
between fwmark and NAT. Any ideas?

Thanks,
Miron Cuperman



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO:
http://ds9a.nl/2.4Routing/

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] Masq/route based on port

[Miron]

This is a home setup, not a server setup.  We have no servers on our 
network.  The reason we want port 80 on eth2 is because eth2 has more 
download bandwidth.  For other protocols we want eth1, because it has 
more symmetric bandwidth.

Greg Scott wrote:

>This doesn't seem right:  
>
>>My firewall configuration:
>>  iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
>>--set-mark 2
>>  iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
>>  iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128
>>
>
>I'm not the expert, but I don't think you want to SNAT.  
>
>You're marking inbound packets with destination port 80 and then putting
>rules in the POSTROUTING table to fudge in a different IP address for
>outbound packets.  That doesn't seem right.  It doesn't redirect inbound
>packets the way you want.  
>
>I think you want to DNAT instead of SNAT and forget about marking packets.  
>Set up some PREROUTING rules and DNAT all incoming port 80 stuff over to 
>the interface you want.  That should be all you need to do because the 
>connection tracking should take care of getting the reply packets from your
>internal web server back to where they belong.  
>
>Verify this with the experts before you do it, but I think I'm right on 
>this one.
>
>- Greg Scott
>
>
>
>-----Original Message-----
>From: Miron [mailto:miron@hyper.to]
>Sent: Thursday, December 06, 2001 3:58 AM
>To: lartc@mailman.ds9a.nl
>Subject: [LARTC] Masq/route based on port
>
>
>I have following setup:
>
>- eth0 is an internal network
>- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
>- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)
>
>I would like to masquerade port 80 through eth2, but all other traffic 
>should be masq'ed through eth1.
>
>My routing configuration:
>
>    (default route in main table is 1.1.1.1)
>
>    ip rule add fwmark 2 pref 1002 table 666
>
>    ip route flush table 666
>    ip route add default via 2.2.2.1 dev eth3 proto static table 666
>    ip route flush cache
>
>My firewall configuration:
>    iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
>--set-mark 2
>    iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
>    iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128
>
>Unfortunately, this does not work.  Outgoing packets are fine.  Incoming 
>packets on port 80 are not de-masqueraded and do not reach the internal 
>hosts.
>
>Also, if I change the ip rule above to be based on the source address 
>(instead of a mark), connections start working fine.
>
>Here is the output of 'ip rule ls', to prove that I do have fwmark compiled:
>    0:      from all lookup local
>    1002:   from all fwmark        2 lookup http
>    32766:  from all lookup main
>    32767:  from all lookup 253
>
>I am wondering if there is some kind of bug related to the interaction 
>between fwmark and NAT. Any ideas?
>
>Thanks,
>Miron Cuperman
>
>
>
>_______________________________________________
>LARTC mailing list / LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO:
>http://ds9a.nl/2.4Routing/
>
>_______________________________________________
>LARTC mailing list / LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
>



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

RE: [LARTC] Masq/route based on port

[Greg Scott]

> This is a home setup, not a server setup.  We have no servers on our 
> network.  The reason we want port 80 on eth2 is because eth2 has more 
> download bandwidth.  For other protocols we want eth1, because it has 
> more symmetric bandwidth.


So anything that comes in from the Internet for port 80, no matter the 
source, you want the reply to go back out on ETH2.  And anything that
comes in from the Internet other than port 80, you want those replies to
go out ETH1.  So the web server process is inside your Linux box?  

Did I get that much right?  Or do I have it backwards?  

The Linux box is your internal LAN's default gateway and you want this 
box to decide which Internet interface to use, based on the destination 
port your internal client PCs choose?  

Hadn't thought about it that way before.

- Greg
 



-----Original Message-----
From: Miron [mailto:miron@hyper.to]
Sent: Thursday, December 06, 2001 1:03 PM
To: Greg Scott
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Masq/route based on port



>I have following setup:
>
>- eth0 is an internal network
>- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
>- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)
>
>I would like to masquerade port 80 through eth2, but all other traffic 
>should be masq'ed through eth1.
>
>My routing configuration:
>
>    (default route in main table is 1.1.1.1)
>
>    ip rule add fwmark 2 pref 1002 table 666
>
>    ip route flush table 666
>    ip route add default via 2.2.2.1 dev eth3 proto static table 666
>    ip route flush cache
>
>My firewall configuration:
>    iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
>--set-mark 2
>    iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
>    iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128
>
>Unfortunately, this does not work.  Outgoing packets are fine.  Incoming 
>packets on port 80 are not de-masqueraded and do not reach the internal 
>hosts.
>
>Also, if I change the ip rule above to be based on the source address 
>(instead of a mark), connections start working fine.
>
>Here is the output of 'ip rule ls', to prove that I do have fwmark
compiled:
>    0:      from all lookup local
>    1002:   from all fwmark        2 lookup http
>    32766:  from all lookup main
>    32767:  from all lookup 253
>
>I am wondering if there is some kind of bug related to the interaction 
>between fwmark and NAT. Any ideas?
>
>Thanks,
>Miron Cuperman

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] Masq/route based on port

[Miron]

I think we are on the same wavelength, except maybe exactly backwards. ;)

I'm trying to provide fast web browsing for clients that are on the 
internal network (talking to servers on the Internet).  Anything from 
the Internet with source port 80 should go through eth2.  Anything else 
should go through eth1.

I did some more investigation, and it seems that the Linux box is 
dropping all incoming packets arriving into eth2.  I can see packets 
with tcpdump, but the applications and NAT don't actually get it.  It 
must be getting dropped in the kernel.  It seems that this has something 
to do with the default route pointing to eth1.  I don't understand why 
the fact that the default route is not pointing to an interface should 
cause *incoming* packets to be dropped on that interface.

Sorry if my post was confusing.

Greg Scott wrote:

>>This is a home setup, not a server setup.  We have no servers on our 
>>network.  The reason we want port 80 on eth2 is because eth2 has more 
>>download bandwidth.  For other protocols we want eth1, because it has 
>>more symmetric bandwidth.
>>
>
>
>So anything that comes in from the Internet for port 80, no matter the 
>source, you want the reply to go back out on ETH2.  And anything that
>comes in from the Internet other than port 80, you want those replies to
>go out ETH1.  So the web server process is inside your Linux box?  
>
>Did I get that much right?  Or do I have it backwards?  
>
>The Linux box is your internal LAN's default gateway and you want this 
>box to decide which Internet interface to use, based on the destination 
>port your internal client PCs choose?  
>
>Hadn't thought about it that way before.
>
>- Greg
> 
>
>
>
>-----Original Message-----
>From: Miron [mailto:miron@hyper.to]
>Sent: Thursday, December 06, 2001 1:03 PM
>To: Greg Scott
>Cc: lartc@mailman.ds9a.nl
>Subject: Re: [LARTC] Masq/route based on port
>
>
>
>>I have following setup:
>>
>>- eth0 is an internal network
>>- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
>>- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)
>>
>>I would like to masquerade port 80 through eth2, but all other traffic 
>>should be masq'ed through eth1.
>>
>>My routing configuration:
>>
>>   (default route in main table is 1.1.1.1)
>>
>>   ip rule add fwmark 2 pref 1002 table 666
>>
>>   ip route flush table 666
>>   ip route add default via 2.2.2.1 dev eth3 proto static table 666
>>   ip route flush cache
>>
>>My firewall configuration:
>>   iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
>>--set-mark 2
>>   iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
>>   iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128
>>
>>Unfortunately, this does not work.  Outgoing packets are fine.  Incoming 
>>packets on port 80 are not de-masqueraded and do not reach the internal 
>>hosts.
>>
>>Also, if I change the ip rule above to be based on the source address 
>>(instead of a mark), connections start working fine.
>>
>>Here is the output of 'ip rule ls', to prove that I do have fwmark
>>
>compiled:
>
>>   0:      from all lookup local
>>   1002:   from all fwmark        2 lookup http
>>   32766:  from all lookup main
>>   32767:  from all lookup 253
>>
>>I am wondering if there is some kind of bug related to the interaction 
>>between fwmark and NAT. Any ideas?
>>
>>Thanks,
>>Miron Cuperman
>>
>
>_______________________________________________
>LARTC mailing list / LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
>



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/

Re: [LARTC] Masq/route based on port

[Miron]

Okay, I figured it out. The secret is to set rp_filter to zero on the 
non-default interface.  I guess the kernel thought that the packets were 
source-routed because of the fact that the interface received packets it 
could not route.

That was frustrating, yet fun... ;)

Miron wrote:

> I think we are on the same wavelength, except maybe exactly backwards. ;)
>
> I'm trying to provide fast web browsing for clients that are on the 
> internal network (talking to servers on the Internet).  Anything from 
> the Internet with source port 80 should go through eth2.  Anything 
> else should go through eth1.
>
> I did some more investigation, and it seems that the Linux box is 
> dropping all incoming packets arriving into eth2.  I can see packets 
> with tcpdump, but the applications and NAT don't actually get it.  It 
> must be getting dropped in the kernel.  It seems that this has 
> something to do with the default route pointing to eth1.  I don't 
> understand why the fact that the default route is not pointing to an 
> interface should cause *incoming* packets to be dropped on that 
> interface.
>
> Sorry if my post was confusing.
>
> Greg Scott wrote:
>
>>> This is a home setup, not a server setup.  We have no servers on our 
>>> network.  The reason we want port 80 on eth2 is because eth2 has 
>>> more download bandwidth.  For other protocols we want eth1, because 
>>> it has more symmetric bandwidth.
>>>
>>
>>
>> So anything that comes in from the Internet for port 80, no matter 
>> the source, you want the reply to go back out on ETH2.  And anything 
>> that
>> comes in from the Internet other than port 80, you want those replies to
>> go out ETH1.  So the web server process is inside your Linux box? 
>> Did I get that much right?  Or do I have it backwards? 
>> The Linux box is your internal LAN's default gateway and you want 
>> this box to decide which Internet interface to use, based on the 
>> destination port your internal client PCs choose? 
>> Hadn't thought about it that way before.
>>
>> - Greg
>>
>>
>>
>>
>> -----Original Message-----
>> From: Miron [mailto:miron@hyper.to]
>> Sent: Thursday, December 06, 2001 1:03 PM
>> To: Greg Scott
>> Cc: lartc@mailman.ds9a.nl
>> Subject: Re: [LARTC] Masq/route based on port
>>
>>
>>
>>> I have following setup:
>>>
>>> - eth0 is an internal network
>>> - eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
>>> - eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)
>>>
>>> I would like to masquerade port 80 through eth2, but all other 
>>> traffic should be masq'ed through eth1.
>>>
>>> My routing configuration:
>>>
>>>   (default route in main table is 1.1.1.1)
>>>
>>>   ip rule add fwmark 2 pref 1002 table 666
>>>
>>>   ip route flush table 666
>>>   ip route add default via 2.2.2.1 dev eth3 proto static table 666
>>>   ip route flush cache
>>>
>>> My firewall configuration:
>>>   iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK 
>>> --set-mark 2
>>>   iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
>>>   iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128
>>>
>>> Unfortunately, this does not work.  Outgoing packets are fine.  
>>> Incoming packets on port 80 are not de-masqueraded and do not reach 
>>> the internal hosts.
>>>
>>> Also, if I change the ip rule above to be based on the source 
>>> address (instead of a mark), connections start working fine.
>>>
>>> Here is the output of 'ip rule ls', to prove that I do have fwmark
>>>
>> compiled:
>>
>>>   0:      from all lookup local
>>>   1002:   from all fwmark        2 lookup http
>>>   32766:  from all lookup main
>>>   32767:  from all lookup 253
>>>
>>> I am wondering if there is some kind of bug related to the 
>>> interaction between fwmark and NAT. Any ideas?
>>>
>>> Thanks,
>>> Miron Cuperman
>>>



_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/