Using DNAT and SNAT to do a local redirection does not work (want to do what rinetd does with iptables)

Using DNAT and SNAT to do a local redirection does not work (want to do what rinetd does with iptables)

[Stefan Mayr]

Hi,

assume the following clustering setup:

Loadbalancer              Server1:
eth0: 192.168.1.1/24  ->  eth0: 192.168.1.2/24
eth0: 192.168.1.101/32    lo0:  192.168.1.101/32 JBoss-Web bound to 8080
eth0: 192.168.1.102/32    lo0:  192.168.1.102/32 JBoss-Web bound to 8080
eth0: 192.168.1.103/32    lo0:  192.168.1.103/32 JBoss-Web bound to 8080
...                       ...

                          Server2:
                      ->  eth0: 192.168.1.3/24
                          lo0:  192.168.1.101/32 JBoss-Web bound to 8080
                          lo0:  192.168.1.102/32 JBoss-Web bound to 8080
                          lo0:  192.168.1.103/32 JBoss-Web bound to 8080
                          ...

all Linux machines (Balancer is Linux virtual server + keepalived with
direct routing setup)

Loadbalancing works so far:
1. Balancer receives packet and forwards it to Server1 or Server2 (L2
via MAC-address)
2. Server responds direct to the client (no backward traffic over the
loadbalancer)

And here comes the problem:
the loadbalancer should check whether the JBoss-Webservers are still
alive but internal check-utils can only connect to the ip eth0. The
different JBoss instances are not bound to this IP because they only the
first could bind to port 8080. But I can specify another port for each
health check. So my thought was:
Health check against Server1:
1. JBoss: check against 192.168.1.2:10001, NAT it to 192.168.1.101:8080
2. JBoss: check against 192.168.1.2:10002, NAT it to 192.168.1.102:8080
3. JBoss: check against 192.168.1.2:10003, NAT it to 192.168.1.103:8080
and so on for the next server.

I thought this would be easy to do with some simple iptables rules on
Server1/2. Maybe I am to stupid but I cannot get it to work.

I thought to use the following rules:
# Clear rules
iptables -t nat -F PREROUTING
iptables -t nat -F POSTROUTING
# Setup NAT to change destination
iptables -t nat -A PREROUTING  -p tcp -d 192.168.1.2  --dport 10001 -j
DNAT --to-destination 192.168.1.101:8080
iptables -t nat -A PREROUTING  -p tcp -d 192.168.1.2  --dport 10002 -j
DNAT --to-destination 192.168.1.102:8080
iptables -t nat -A PREROUTING  -p tcp -d 192.168.1.2  --dport 10003 -j
DNAT --to-destination 192.168.1.103:8080
# make my answers appear from the server-ip and change source back
iptables -t nat -A POSTROUTING -p tcp -s 192.168.1.101  --sport 8080
-j SNAT --to-source=192.168.1.2:10001
iptables -t nat -A POSTROUTING -p tcp -s 192.168.1.102  --sport 8080
-j SNAT --to-source=192.168.1.2:10002
iptables -t nat -A POSTROUTING -p tcp -s 192.168.1.103  --sport 8080
-j SNAT --to-source=192.168.1.2:10003

so far my idea - but it doesn't work. I added some logging to these
rules an found out the following:
1. when I open a connection to the server, e.g. 192.168.1.2:10001, the
DNAT works (at least I see the "SYN" in the nat-PREROUTING-LOG)
2. the server responds with "ACK RST" but from 192.168.1.101:8080
(filter-OUTPUT-LOG)

- Why does the response not go through nat-POSTROUTING?
- Why the "RST"? Or do I read the logs all wrong?

I hope somebody can help me with this.

What I want to achive is the functionality of rinetd with NAT to
addresses/ports on the same machine instead of a daemon that acts like a
proxy (only another process that could die ...)

Thanks in advance,

    Stefan

PS: The "big picture" with by JBoss-instances is only my example setup.
To exclude the JBoss as problem I tested it also with httpd and telnetd.
rinetd works - but I prefer having this done in the linux kernel - if it
can do it. Just to eliminate sources of failure.

Re: Using DNAT and SNAT to do a local redirection does not work (want to do what rinetd does with iptables)

[Grant Taylor]

On 6/7/2007 3:30 PM, Stefan Mayr wrote:
> And here comes the problem:  The loadbalancer should check whether 
> the JBoss-Webservers are still alive but internal check-utils can 
> only connect to the ip eth0. The different JBoss instances are not 
> bound to this IP because they only the first could bind to port 8080. 
> But I can specify another port for each health check. So my thought 
> was:

If you are using the loop back interface, this will not work.

> I thought this would be easy to do with some simple iptables rules on
> Server1/2. Maybe I am to stupid but I cannot get it to work.

No, you are not stupid.  At least I don't think you are.

> so far my idea - but it doesn't work. I added some logging to these
> rules an found out the following:
> 
> 1. when I open a connection to the server, e.g. 192.168.1.2:10001,
> the DNAT works (at least I see the "SYN" in the nat-PREROUTING-LOG)
> 2. the server responds with "ACK RST" but from 192.168.1.101:8080
> (filter-OUTPUT-LOG)
> 
> - Why does the response not go through nat-POSTROUTING?
> - Why the "RST"? Or do I read the logs all wrong?

You are using the loop back interface. Loop back is a very special 
network interface.  If I recall correctly, it will only allow its self 
to talk to it.  Thus you can not NAT traffic in to the loop back 
interface.  The kernel will block this.  I think this is why you are 
seeing the RST packets.

> I hope somebody can help me with this.

Try using a dummy network interface, or an ethernet interface that is 
not connected to any thing.

You could also probably bind the address to the main ethernet interface 
and use ARPTables to prevent each node from responding to ARP request by 
preventing it from ever seeing the ARP request.  The ARP issue (as I'm 
sure you are aware) is why you usually use other interfaces.



Grant. . . .

Re: Using DNAT and SNAT to do a local redirection does not work (want to do what rinetd does with iptables)

[Stefan Mayr]

Grant Taylor wrote:
> On 6/7/2007 3:30 PM, Stefan Mayr wrote:
> 
> If you are using the loop back interface, this will not work.

An answer I often read but nobody says what's wrong with loopback. I
thought it depended on the rules of the scenarios (obviously too much
thinking involved here).

> 
> You are using the loop back interface. Loop back is a very special
> network interface.  If I recall correctly, it will only allow its self
> to talk to it.  Thus you can not NAT traffic in to the loop back
> interface.  The kernel will block this.  I think this is why you are
> seeing the RST packets.

I really have to thank you for this enlightenment.

> Try using a dummy network interface, or an ethernet interface that is
> not connected to any thing.

I used dummy0 and now my iptables ruleset works.

> You could also probably bind the address to the main ethernet interface
> and use ARPTables to prevent each node from responding to ARP request by
> preventing it from ever seeing the ARP request.  The ARP issue (as I'm
> sure you are aware) is why you usually use other interfaces.

That is why I used the loopback-device and my /etc/sysctl.conf contains
the following lines:
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2

So arp-requests/announces are always answered/sent from the right interface.

Now the lesson is learned, setup is up and running.

Thanks,

	Stefan

Re: Using DNAT and SNAT to do a local redirection does not work (want to do what rinetd does with iptables)

[Grant Taylor]

On 6/9/2007 2:25 PM, Stefan Mayr wrote:
> An answer I often read but nobody says what's wrong with loopback. I 
> thought it depended on the rules of the scenarios (obviously too much 
> thinking involved here).

*nod*  There is nothing specifically wrong (per say) with loopback other 
than the kernel imposed security, which has been discussed elsewhere.  I 
guess this kernel imposed security is not in and of its self a bad thing 
so long as you are aware of it and have things like dummy to work around 
it.  ;)

> I really have to thank you for this enlightenment.

No problem.  I'm just glad that I was able to help.  I've all too often 
been working on a problem and not known the fact that was stopping me 
from making things work.  It is a way to either get gray hair or loose 
what little hair you may have left or worse yet both.

> I used dummy0 and now my iptables ruleset works.

Good.

> That is why I used the loopback-device and my /etc/sysctl.conf
> contains the following lines:
> 
> net.ipv4.conf.all.arp_ignore = 1
> net.ipv4.conf.all.arp_announce = 2

*nod*

> So arp-requests/announces are always answered/sent from the right 
> interface.

*nod*

> Now the lesson is learned, setup is up and running.

Good.

> Thanks,

You are welcome.  :)



Grant. . . .