Re: troublesome load balancing and iptables to the rescue...maybe

[Pascal Hambourg <pascal.mail@plouf.fr.eu.org>]

Hello,

Admin a écrit :
> 
> We have a load balanced set of web servers (behind an alteon AD3) which all 
> work fine - except when the real servers (the webserver instances on the load 
> balanced machines) try to access the virtual ip (vip) on the alteon.
[...]
> So the alteon substitutes the virtual server ip with the real server one and 
> back again for the load balancing to work.

This looks like good old DNAT.

> Now the problem, we need the real servers to be able to access the service 
> provided on the VIP also. The ports on the alteon are configured properly 
> (client and server enable and so on) but the problem seems to be a routing 
> one.
> 
> Here's the flow of traffic when the real server tries to access the vip
> 
> # client request
> (client) -> (vip) - sip=CC, dip=BB
> (vip) -> (real_server) - sip=CC, dip=CC
> # return traffic (real server on CC responds to client on CC - oops that's me)
> (real_server) -- sip=CC, dip=CC --> straight back to the real server, no dice.
> 
> Now this is a common problem

Yes, rather common when the client and the DNAT target are inside the 
same LAN.

If you can tolerate that each server send the request to itself, it's 
easy. Something like this will locally redirect connections to the VIP 
to the server's own address :

iptables -t nat -A OUTPUT -d <vip> -j DNAT --to <own_ip>

Else, if the alteon can do SNAT (source NAT), this is the usual 
workaround to this problem. You can masquerade the servers subnet with 
VIP. Written in iptables style :

iptables -t nat -j POSTROUTING -s <servers_subnet> -d <servers_subnet> \
   -j SNAT --to <vip>

But this method hides the original source server address on the 
destination server. If you need this information, you can do a static 
SNAT 1:1 mapping between each server address and a unique address chosen 
into an unused subnet which is distinct from the servers subnet. You 
won't have the real address in the logs but at least an address that 
uniquely identifies the source server. Again in iptables style :

iptables -t nat -j POSTROUTING -s <server1> -d <servers_subnet> \
   -j SNAT --to <alias1>
iptables -t nat -j POSTROUTING -s <server2> -d <servers_subnet> \
   -j SNAT --to <alias2>

and so on. Or, more concisely :

iptables -t nat -j POSTROUTING -s <servers_subnet> -d <servers_subnet> \
   -j NETMAP --to <alias_subnet>

> I have been trying to find a way to mangle the packets with iptables (on the 
> real servers) to make this work.
> 
> The obvious (well...kinda obvious) way is to change the sip (to something 
> outside the local network)  on the incoming packet somewhere in the 
> PREROUTING or INPUT phase, then modify the dip in the response packet in the 
> POSTROUTING phase back to that of the real server.
> 
> Unfortunately this is pretty much the reverse of what DNAT and SNAT are used 
> for.
>
> We want SNAT on the PREROUTING or INPUT phase and DNAT in the POSTROUTING (or 
> possibly OUTPUT, but I don't think so) one.

The reason why this is not possible is that routing would probably be 
disrupted.

> Or perhaps - is there any way to force traffic from one IP to the same IP to 
> go through a remote router?

That kind of traffic is for a local delivery, and the 'local' routing 
table has precedence over all other routing tables. Anyway why would 
there be traffic from one IP to the same IP ?

> Perhaps mark traffic from the real server to the 
> vip then act on that mark when the alteon directs it back?

A mark is a local information, and is lost as soon as the packet leaves 
the box. But you can alter return traffic this way (what you actually 
need). Remember, the wrong part is that replies go back directly to the 
client instead of going through the alteon. So iptables will first mark 
the packets we're interested in, assuming your servers listen only on 
TCP port 80 :

iptables -t mangle -A OUTPUT -d <servers_subnet> -p tcp --sport 80 \
   -m state --state ESTABLISHED -j MARK --set-mark <mark_value>

A routing rule will direct the marked packets to an alternate table :

ip rule add fwmark <mark_value> lookup <table_id>
ip route add default via <vip> table <table_id>

One alternative, although I personnaly find it very ugly : by-pass 
normal routing with the (experimental) ROUTE target, and force routing 
of the servers subnet via the alteon as a gateway.