troublesome load balancing and iptables to the rescue...maybe

troublesome load balancing and iptables to the rescue...maybe

[Admin]

Hi all

We have a load balanced set of web servers (behind an alteon AD3) which all 
work fine - except when the real servers (the webserver instances on the load 
balanced machines) try to access the virtual ip (vip) on the alteon.

Here's how it works

clientIP=AA
VIP=BB
RealServerIP=CC
SourceIP=sip
DestinationIP=dip

The vip and real_server ips are all on the same network.

# client request from the outside
(client) -> (vip) - sip=AA, dip=BB
(vip) -> (real_server) - sip=AA, dip=CC
# return traffic (real server on CC responds to client on AA)
(real_server) -- sip=CC, dip=AA --> (alteon) -- sip=BB, dip=AA --> client

So the alteon substitutes the virtual server ip with the real server one and 
back again for the load balancing to work.

Now the problem, we need the real servers to be able to access the service 
provided on the VIP also. The ports on the alteon are configured properly 
(client and server enable and so on) but the problem seems to be a routing 
one.

Here's the flow of traffic when the real server tries to access the vip

# client request
(client) -> (vip) - sip=CC, dip=BB
(vip) -> (real_server) - sip=CC, dip=CC
# return traffic (real server on CC responds to client on CC - oops that's me)
(real_server) -- sip=CC, dip=CC --> straight back to the real server, no dice.

Now this is a common problem, usually solved by using proxy-ips, however the 
alteon proxy-ips don't add X-Forwarded-For headers meaning our servers won't 
get the correct client IPs - which is unacceptable.

I have been trying to find a way to mangle the packets with iptables (on the 
real servers) to make this work.

The obvious (well...kinda obvious) way is to change the sip (to something 
outside the local network)  on the incoming packet somewhere in the 
PREROUTING or INPUT phase, then modify the dip in the response packet in the 
POSTROUTING phase back to that of the real server.

Unfortunately this is pretty much the reverse of what DNAT and SNAT are used 
for.

We want SNAT on the PREROUTING or INPUT phase and DNAT in the POSTROUTING (or 
possibly OUTPUT, but I don't think so) one.

Or perhaps - is there any way to force traffic from one IP to the same IP to 
go through a remote router? Perhaps mark traffic from the real server to the 
vip then act on that mark when the alteon directs it back?

Can anyone offer any ideas how to solve this issue?

Many thanks
Admin

Re: troublesome load balancing and iptables to the rescue...maybe

[Pascal Hambourg]

Hello,

Admin a écrit :
> 
> We have a load balanced set of web servers (behind an alteon AD3) which all 
> work fine - except when the real servers (the webserver instances on the load 
> balanced machines) try to access the virtual ip (vip) on the alteon.
[...]
> So the alteon substitutes the virtual server ip with the real server one and 
> back again for the load balancing to work.

This looks like good old DNAT.

> Now the problem, we need the real servers to be able to access the service 
> provided on the VIP also. The ports on the alteon are configured properly 
> (client and server enable and so on) but the problem seems to be a routing 
> one.
> 
> Here's the flow of traffic when the real server tries to access the vip
> 
> # client request
> (client) -> (vip) - sip=CC, dip=BB
> (vip) -> (real_server) - sip=CC, dip=CC
> # return traffic (real server on CC responds to client on CC - oops that's me)
> (real_server) -- sip=CC, dip=CC --> straight back to the real server, no dice.
> 
> Now this is a common problem

Yes, rather common when the client and the DNAT target are inside the 
same LAN.

If you can tolerate that each server send the request to itself, it's 
easy. Something like this will locally redirect connections to the VIP 
to the server's own address :

iptables -t nat -A OUTPUT -d <vip> -j DNAT --to <own_ip>

Else, if the alteon can do SNAT (source NAT), this is the usual 
workaround to this problem. You can masquerade the servers subnet with 
VIP. Written in iptables style :

iptables -t nat -j POSTROUTING -s <servers_subnet> -d <servers_subnet> \
   -j SNAT --to <vip>

But this method hides the original source server address on the 
destination server. If you need this information, you can do a static 
SNAT 1:1 mapping between each server address and a unique address chosen 
into an unused subnet which is distinct from the servers subnet. You 
won't have the real address in the logs but at least an address that 
uniquely identifies the source server. Again in iptables style :

iptables -t nat -j POSTROUTING -s <server1> -d <servers_subnet> \
   -j SNAT --to <alias1>
iptables -t nat -j POSTROUTING -s <server2> -d <servers_subnet> \
   -j SNAT --to <alias2>

and so on. Or, more concisely :

iptables -t nat -j POSTROUTING -s <servers_subnet> -d <servers_subnet> \
   -j NETMAP --to <alias_subnet>

> I have been trying to find a way to mangle the packets with iptables (on the 
> real servers) to make this work.
> 
> The obvious (well...kinda obvious) way is to change the sip (to something 
> outside the local network)  on the incoming packet somewhere in the 
> PREROUTING or INPUT phase, then modify the dip in the response packet in the 
> POSTROUTING phase back to that of the real server.
> 
> Unfortunately this is pretty much the reverse of what DNAT and SNAT are used 
> for.
>
> We want SNAT on the PREROUTING or INPUT phase and DNAT in the POSTROUTING (or 
> possibly OUTPUT, but I don't think so) one.

The reason why this is not possible is that routing would probably be 
disrupted.

> Or perhaps - is there any way to force traffic from one IP to the same IP to 
> go through a remote router?

That kind of traffic is for a local delivery, and the 'local' routing 
table has precedence over all other routing tables. Anyway why would 
there be traffic from one IP to the same IP ?

> Perhaps mark traffic from the real server to the 
> vip then act on that mark when the alteon directs it back?

A mark is a local information, and is lost as soon as the packet leaves 
the box. But you can alter return traffic this way (what you actually 
need). Remember, the wrong part is that replies go back directly to the 
client instead of going through the alteon. So iptables will first mark 
the packets we're interested in, assuming your servers listen only on 
TCP port 80 :

iptables -t mangle -A OUTPUT -d <servers_subnet> -p tcp --sport 80 \
   -m state --state ESTABLISHED -j MARK --set-mark <mark_value>

A routing rule will direct the marked packets to an alternate table :

ip rule add fwmark <mark_value> lookup <table_id>
ip route add default via <vip> table <table_id>

One alternative, although I personnaly find it very ugly : by-pass 
normal routing with the (experimental) ROUTE target, and force routing 
of the servers subnet via the alteon as a gateway.

Re: troublesome load balancing and iptables to the rescue...maybe

[Admin]

On Wednesday 28 June 2006 01:54, Pascal Hambourg wrote:
> Hello,
>
> Admin a écrit :
> > We have a load balanced set of web servers (behind an alteon AD3) which
> > all work fine - except when the real servers (the webserver instances on
> > the load balanced machines) try to access the virtual ip (vip) on the
> > alteon.
>
> [...]
>
> > So the alteon substitutes the virtual server ip with the real server one
> > and back again for the load balancing to work.
>
> This looks like good old DNAT.

yep.

> > Now the problem, we need the real servers to be able to access the
> > service provided on the VIP also. The ports on the alteon are configured
> > properly (client and server enable and so on) but the problem seems to be
> > a routing one.
> >
> > Here's the flow of traffic when the real server tries to access the vip
> >
> > # client request
> > (client) -> (vip) - sip=CC, dip=BB
> > (vip) -> (real_server) - sip=CC, dip=CC
> > # return traffic (real server on CC responds to client on CC - oops
> > that's me) (real_server) -- sip=CC, dip=CC --> straight back to the real
> > server, no dice.
> >
> > Now this is a common problem
>
> Yes, rather common when the client and the DNAT target are inside the
> same LAN.
>
> If you can tolerate that each server send the request to itself, it's
> easy. Something like this will locally redirect connections to the VIP
> to the server's own address :
>
> iptables -t nat -A OUTPUT -d <vip> -j DNAT --to <own_ip>

Unfortunately we also have ISD units (hardware SSL offloaders) attached to the 
alteon, so the web servers themselves don't do SSL. Thus if a real server 
tried to make a HTTPS connection this wouldn't work (since the traffic needs 
to be processed by the alteon rather than just redirected back to the real 
server.)

> Else, if the alteon can do SNAT (source NAT), this is the usual
> workaround to this problem. You can masquerade the servers subnet with
> VIP. Written in iptables style :
>
> iptables -t nat -j POSTROUTING -s <servers_subnet> -d <servers_subnet> \
>    -j SNAT --to <vip>
>
> But this method hides the original source server address on the
> destination server. If you need this information, you can do a static
> SNAT 1:1 mapping between each server address and a unique address chosen
> into an unused subnet which is distinct from the servers subnet. You
> won't have the real address in the logs but at least an address that
> uniquely identifies the source server. Again in iptables style :
>
> iptables -t nat -j POSTROUTING -s <server1> -d <servers_subnet> \
>    -j SNAT --to <alias1>
> iptables -t nat -j POSTROUTING -s <server2> -d <servers_subnet> \
>    -j SNAT --to <alias2>
>
> and so on. Or, more concisely :
>
> iptables -t nat -j POSTROUTING -s <servers_subnet> -d <servers_subnet> \
>    -j NETMAP --to <alias_subnet>

I'll try this one out when the alteon guru returns from his break.

> > I have been trying to find a way to mangle the packets with iptables (on
> > the real servers) to make this work.
> >
> > The obvious (well...kinda obvious) way is to change the sip (to something
> > outside the local network)  on the incoming packet somewhere in the
> > PREROUTING or INPUT phase, then modify the dip in the response packet in
> > the POSTROUTING phase back to that of the real server.
> >
> > Unfortunately this is pretty much the reverse of what DNAT and SNAT are
> > used for.
> >
> > We want SNAT on the PREROUTING or INPUT phase and DNAT in the POSTROUTING
> > (or possibly OUTPUT, but I don't think so) one.
>
> The reason why this is not possible is that routing would probably be
> disrupted.
>
> > Or perhaps - is there any way to force traffic from one IP to the same IP
> > to go through a remote router?
>
> That kind of traffic is for a local delivery, and the 'local' routing
> table has precedence over all other routing tables. Anyway why would
> there be traffic from one IP to the same IP ?
>
> > Perhaps mark traffic from the real server to the
> > vip then act on that mark when the alteon directs it back?
>
> A mark is a local information, and is lost as soon as the packet leaves
> the box. But you can alter return traffic this way (what you actually
> need). Remember, the wrong part is that replies go back directly to the
> client instead of going through the alteon. So iptables will first mark
> the packets we're interested in, assuming your servers listen only on
> TCP port 80 :
>
> iptables -t mangle -A OUTPUT -d <servers_subnet> -p tcp --sport 80 \
>    -m state --state ESTABLISHED -j MARK --set-mark <mark_value>
>
> A routing rule will direct the marked packets to an alternate table :
>
> ip rule add fwmark <mark_value> lookup <table_id>
> ip route add default via <vip> table <table_id>

So for example - assuming 192.168.1.0/24 as the servers_subnet (ignore that 
this is private address space)

realServerA=192.168.1.10
realServerB=192.168.1.11
VIP=192.168.1.1

the procedure would be:
o Compile the kernel with iptables support plus...
   * IP: advanced router
   * IP: policy routing
     * IP: use netfilter MARK value as routing key
echo '80 web' >> /etc/iproute2/rt_tables
iptables -t mangle -A OUTPUT -d 192.168.1.0/24 -p tcp --sport 80 \
-m state --state ESTABLISHED -j MARK --set-mark 200
ip rule add fwmark 200 lookup web
ip route add default via 192.168.1.1 table web

With that applied, then a server running on port 80 on 192.168.1.10 should be 
able to access the service on the vip on 192.168.1.1, which will get directed 
back to 192.168.1.10 or 192.168.1.11. That request should then be returned to 
the client on 192.168.1.10

It doesn't work yet (I'm still examining the packet dumps to try to work out 
why)

Added a mark log to check on the match
iptables -t mangle -A OUTPUT -m mark --mark 200 -j LOG \
--log-level DEBUG --log-prefix "fwmark 200: "

which show's the traffic from the health checks that come in from the alteon, 
and presumably the traffic from the real server to the vip when I try to make 
that connection.

> One alternative, although I personnaly find it very ugly : by-pass
> normal routing with the (experimental) ROUTE target, and force routing
> of the servers subnet via the alteon as a gateway.

Mmm...if I can't get the above working (which I haven't yet) then I'll give 
this a try.

Thanks for the info

Admin

Re: troublesome load balancing and iptables to the rescue...maybe

[Pascal Hambourg]

Admin a écrit :
>
> So for example - assuming 192.168.1.0/24 as the servers_subnet (ignore that 
> this is private address space)
> 
> realServerA=192.168.1.10
> realServerB=192.168.1.11
> VIP=192.168.1.1
> 
> the procedure would be:
> o Compile the kernel with iptables support plus...
>    * IP: advanced router
>    * IP: policy routing
>      * IP: use netfilter MARK value as routing key

Right, I always forget these functions are optional...

> echo '80 web' >> /etc/iproute2/rt_tables
> iptables -t mangle -A OUTPUT -d 192.168.1.0/24 -p tcp --sport 80 \
> -m state --state ESTABLISHED -j MARK --set-mark 200
> ip rule add fwmark 200 lookup web
> ip route add default via 192.168.1.1 table web
> 
> With that applied, then a server running on port 80 on 192.168.1.10 should be 
> able to access the service on the vip on 192.168.1.1, which will get directed 
> back to 192.168.1.10 or 192.168.1.11. That request should then be returned to 
> the client on 192.168.1.10

Oops, I just realized that for several reasons it won't work when the 
alteon redirects a connection to the same server.

> It doesn't work yet (I'm still examining the packet dumps to try to work out 
> why)

Ok. Check the packet MAC addresses on both servers.

> Added a mark log to check on the match
> iptables -t mangle -A OUTPUT -m mark --mark 200 -j LOG \
> --log-level DEBUG --log-prefix "fwmark 200: "
> 
> which show's the traffic from the health checks that come in from the alteon,

Huh ? This rule is in the OUTPUT chain, so it shouldn't log incoming 
packets from the alteon.

> and presumably the traffic from the real server to the vip when I try to make 
> that connection.

Rather the return traffic from the target server to the source server.