iptables NAT routing issues

iptables NAT routing issues

[Bas Verhoeven]

Hello,

We're having some problems with iptables, have been playing ~3 hours 
with this and I need some advice.

We want to 'forward' port 80 from one ip to another ip (other server) - 
mainly to protect the webserver. One could say that you could use basic 
portforwarding for this, but as far as i know this also breaks the 
source ip for apache logs, etc. So we decided that we needed NAT'ing.

On the 'outbound' server we're doing DNAT to the webserver, which seems 
to work fine (can see the packets in tcpdump on web), and on the 
webserver we do SNAT back to the 'original' (outboundserver) ip the user 
connected to.

The last part does not work tho, the webserver still uses its own 
outgoing ip, and not the IP we specified. If I telnet to some host 
however, using the webserver source ip - it DOES rewrite the source 
address (and fails to connect, of course).

Tried various iptables commands on the servers:

OUTBOUND SERVER:

iptables -t nat -A PREROUTING -p tcp --dport 80 -d <ext_web_ip> -j DNAT 
--to-destination <webserver_ip>:80

Note: We didn't touch this as it seems to works fine.

WEBSERVER:
 
Tried various, from the detailed:
    iptables -t nat -A POSTROUTING -s <webserver_ip>-p tcp --sport 80 -j 
SNAT --to-source <ext_web_ip>:80 -o eth0

To the 'should-be-catching-it-all':
    iptables -t nat -A POSTROUTING -s <webserver_ip> -j SNAT --to-source 
<ext_web_ip>

None of those seem to do the thing we want to do tho.

Some advice would be very welcome.

Cheers,
Bas Verhoeven

Re: iptables NAT routing issues

[Jan Engelhardt]

On May 8 2007 23:16, Bas Verhoeven wrote:
>
> OUTBOUND SERVER:
>
> iptables -t nat -A PREROUTING -p tcp --dport 80 -d <ext_web_ip> -j DNAT
> --to-destination <webserver_ip>:80
>
> Note: We didn't touch this as it seems to works fine.
>
> WEBSERVER:

You do not need any rules here. The outbound server will do the translation
in both directions.

> Tried various, from the detailed:
>   iptables -t nat -A POSTROUTING -s <webserver_ip>-p tcp --sport 80 -j SNAT
> --to-source <ext_web_ip>:80 -o eth0
>
> To the 'should-be-catching-it-all':
>   iptables -t nat -A POSTROUTING -s <webserver_ip> -j SNAT --to-source
> <ext_web_ip>
>
> None of those seem to do the thing we want to do tho.
>
> Some advice would be very welcome.
>
> Cheers,
> Bas Verhoeven
>

Jan
-- 

Re: iptables NAT routing issues

[Pascal Hambourg]

Hello,

Jan Engelhardt a écrit :
> On May 8 2007 23:16, Bas Verhoeven wrote:
> 
>>OUTBOUND SERVER:
>>
>>iptables -t nat -A PREROUTING -p tcp --dport 80 -d <ext_web_ip> -j DNAT
>>--to-destination <webserver_ip>:80
>>
>>Note: We didn't touch this as it seems to works fine.
>>
>>WEBSERVER:
> 
> You do not need any rules here. The outbound server will do the translation
> in both directions.

Assuming that the outbound server is the default gateway for the web 
server and receives the return traffic, of course.

Re: iptables NAT routing issues

[Maximilian Wilhelm]

Am Tuesday, den  8 May hub Bas Verhoeven folgendes in die Tasten:

Hi!

> We're having some problems with iptables, have been playing ~3 hours 
> with this and I need some advice.

> We want to 'forward' port 80 from one ip to another ip (other server) - 
> mainly to protect the webserver. One could say that you could use basic 
> portforwarding for this, but as far as i know this also breaks the 
> source ip for apache logs, etc. So we decided that we needed NAT'ing.
[...]

> OUTBOUND SERVER:

> iptables -t nat -A PREROUTING -p tcp --dport 80 -d <ext_web_ip> -j DNAT 
> --to-destination <webserver_ip>:80

That´s fine.
This will also make netfilter care of answer packages.

> Note: We didn't touch this as it seems to works fine.
> 
> WEBSERVER:
[...]
There are no NAT rules needed here.
All you have to accomplish is that the answer packages from WEBSERVER
to $client are routed via OUTBOUND SERVER.

I guess that´s just true by the network architecture.

e.g.

client network / inet  <--->  OUTBOUND SERVER  <->  WEBSERVER

HTH
Ciao
Max
-- 
	Follow the white penguin.

Re: iptables NAT routing issues

[Jan Engelhardt]

On May 9 2007 00:33, Pascal Hambourg wrote:
> Jan Engelhardt a écrit :
>> On May 8 2007 23:16, Bas Verhoeven wrote:
>> 
>> > OUTBOUND SERVER:
>> > 
>> > iptables -t nat -A PREROUTING -p tcp --dport 80 -d <ext_web_ip> -j DNAT
>> > --to-destination <webserver_ip>:80
>> > 
>> > Note: We didn't touch this as it seems to works fine.
>> > 
>> > WEBSERVER:
>> 
>> You do not need any rules here. The outbound server will do the
>> translation
>> in both directions.
>
> Assuming that the outbound server is the default gateway for the web server and
> receives the return traffic, of course.

You'd be kinda screwed if that was not the case. Because the 'nat' table
is only consulted for NEW connections afaics, the reply packets do _not_
get SNATed, since the _first_ packet was the TCP SYN, which makes the
connection ESTABLISHED when input processing is done.


Jan
-- 

Re: iptables NAT routing issues

[Bas Verhoeven]

Jan Engelhardt wrote:
>
>> Assuming that the outbound server is the default gateway for the web server and
>> receives the return traffic, of course.
>>     
>
> You'd be kinda screwed if that was not the case. Because the 'nat' table
> is only consulted for NEW connections afaics, the reply packets do _not_
> get SNATed, since the _first_ packet was the TCP SYN, which makes the
> connection ESTABLISHED when input processing is done.
>
>   
The webserver has its own gateway, and that's not the outer box. You 
seem to be describing the exact problem i'm seeing in tcpdump; new 
connections get snat'ed, 'forwarded' connections do not.

Is there really no solution for this?

Bas

Re: iptables NAT routing issues

[Maximilian Wilhelm]

Am Wednesday, den  9 May hub Bas Verhoeven folgendes in die Tasten:

> Jan Engelhardt wrote:
> >
> >>Assuming that the outbound server is the default gateway for the web 
> >>server and
> >>receives the return traffic, of course.
> >>    
> >
> >You'd be kinda screwed if that was not the case. Because the 'nat' table
> >is only consulted for NEW connections afaics, the reply packets do _not_
> >get SNATed, since the _first_ packet was the TCP SYN, which makes the
> >connection ESTABLISHED when input processing is done.

  
> The webserver has its own gateway, and that's not the outer box. You 
> seem to be describing the exact problem i'm seeing in tcpdump; new 
> connections get snat'ed, 'forwarded' connections do not.

> Is there really no solution for this?

The solution is to use OUTBOUND SERVER as default gateway or do the
DNATing on the gateway.

Or stop DNATing and use "normal" rules to only allow traffic to
WEBSERVER 80/tcp and reject the remaining traffic.

Ciao
Max
-- 
	Follow the white penguin.

Re: iptables NAT routing issues

[Pascal Hambourg]

Bas Verhoeven a écrit :
>>
>>> Assuming that the outbound server is the default gateway for the web 
>>> server and receives the return traffic, of course.
>>
>> You'd be kinda screwed if that was not the case.
> 
> The webserver has its own gateway, and that's not the outer box.

Don't look further. Here is why it does not work.

> Is there really no solution for this?

Use the outer box as a gateway, if it is in the same network. You do not 
have to use it as the default gateway for all traffic but at least for 
the HTTP return traffic. This could be done with iptables and advanced 
routing on the web server, for instance using the source port 80 to MARK 
packets or using CONNMARK/connmark.

You could try to use the NOTRACK target on the web server in order to 
prevent the incoming SYN packets from creating a new connection, so the 
replies could be SNATed. But I would not bet a euro-cent on such a dirty 
method. If someone asks who suggested it, don't say it's me. ;-)

Re: iptables NAT routing issues

[Bas Verhoeven]

Pascal Hambourg wrote:
> Use the outer box as a gateway, if it is in the same network. You do 
> not have to use it as the default gateway for all traffic but at least 
> for the HTTP return traffic. This could be done with iptables and 
> advanced routing on the web server, for instance using the source port 
> 80 to MARK packets or using CONNMARK/connmark.
I wasn't aware of that option, it seems to work fine ;-)

Thanks.
>
> You could try to use the NOTRACK target on the web server in order to 
> prevent the incoming SYN packets from creating a new connection, so 
> the replies could be SNATed. But I would not bet a euro-cent on such a 
> dirty method. If someone asks who suggested it, don't say it's me. ;-)
>

Re: iptables NAT routing issues

[Pascal Hambourg]

Bas Verhoeven a écrit :
> 
>> Use the outer box as a gateway, if it is in the same network. You do 
>> not have to use it as the default gateway for all traffic but at least 
>> for the HTTP return traffic. This could be done with iptables and 
>> advanced routing on the web server, for instance using the source port 
>> 80 to MARK packets or using CONNMARK/connmark.
> 
> I wasn't aware of that option, it seems to work fine ;-)

Which option did you choose ?

> Thanks.

You're welcome.

Re: iptables NAT routing issues

[Bas Verhoeven]

Pascal Hambourg wrote:
> Which option did you choose ?
We still have the outbound server DNAT-ing connections to the 
webserver's ip, that worked fine.
On the webserver we now mark all outgoing web packets:

# iptables -t mangle -A OUTPUT -s <webserver_inner_ip> -p tcp --sport 80 
-j MARK --set-mark 2

And we use iproute2 to forward them back to the outbound server:

# ip rule add fwmark 2 pref 10 table web.out
# ip route add default via <outbound_box_ip> dev eth0 table web.out

Couldn't test with CONNMARK, as the box doesn't ship with that, but MARK 
works great for now.

I did test your last option too, but that just didn't work and sounded 
very hacky-ish, not something we could rely on, even if it worked.

Re: iptables NAT routing issues

[Pascal Hambourg]

Bas Verhoeven a écrit :
> 
>> Which option did you choose ?
[...]
> On the webserver we now mark all outgoing web packets:
[...]
> And we use iproute2 to forward them back to the outbound server:
[...]

Ok. You seem to know how to use advanced routing, I am a bit surprised 
you said you "weren't aware of that option".

> Couldn't test with CONNMARK, as the box doesn't ship with that, but MARK 
> works great for now.

The 'CONNMARK' target and the 'connmark' match were included in the 
kernel 2.6.10. Earlier kernel versions had to be patched with the 
patch-o-matic-ng.

> I did test your last option too, but that just didn't work and sounded 
> very hacky-ish, not something we could rely on, even if it worked.

Huh, what last option ? I don't know what you're talking about. ;-)