ip_conntrack_http?

ip_conntrack_http?

[Chris Shepherd]

Hi,
  I was attempting to perform some connection-level load balancing with 
NetFilter this past weekend, and I kept running into issues with non-static 
pages. Specifically PHP and/or ASP pages that utilise server-side Session 
variables. The problem stems from the fact that most, if not all browsers, open 
multiple connections to the same webserver. This problem may only occur in 
places where you are using frames. 

My Setup was as follows:

(10.0.0.1) FW (10.0.1.1) --------- WS (10.0.1.2)
                        \_________ WS (10.0.1.3)

Rules were as follows:
iptables -t nat -A PREROUTING -i eth0 -d 10.0.0.1 --dport 80 -j DNAT --to 
10.0.1.2-10.0.1.3
iptables -A FORWARD -i eth0 -o eth1 -d 10.0.1.0/24 --dport 80 -j ACCEPT
iptables -A FORWARD -o eth0 -s 10.0.1.0/24 -j ACCEPT
iptables -t nat -A POSTROUTING -o eth0 -s 10.0.1.0/24 -j SNAT --to 10.0.0.1

My problem was this:
I successfully login to my webdev site (triggering creation of session vars). 
At this point, it seems that the browser only initiated 1 connection. 
As soon as the login completes, it forwards to a php script containing a 
frameset.
This frameset has three frames. At this point, the browser initiates a second 
connection.
All my scripts check for the existance of a boolean session variable, and if it 
is not there, it forwards the user back to the login page. This session 
variable was not set on the second webserver for obvious reasons, so I get 
forwarded back to the login page. 
Now, the problem here is that the browser's second connection got Natted to 
10.0.1.3 instead of 10.0.1.2, and the webserver on 1.3 has no clue of the 
session variables on the other server. 

What I'm wondering is twofold:
1) Did I do something wrong with my configuration?
2) If not, and this is by design, is there any module/could a module be written 
that would track HTTP requests and forward ALL http requests from the same 
connection to the same IP?

What happens now:
CONN1a -> WS1
CONN1b -> WS2
CONN2a -> WS1
CONN2b -> WS2

What should happen:
CONN1a -> WS1
CONN1b -> WS1
CONN2a -> WS2
CONN2b -> WS2

Provided that CONN1[ab] are related connections, but unrelated to CONN2[ab].

Is this possible with the current NetFilter setup? Must a module be written?

I am very interested in knowing, because it could save myself and a few dozen 
webdevs I know a lot of money that we'd be spending on a hardware connection-
level load balancer.

-- 
Chris Shepherd




-------------------------------------------------
This email may contain confidential information. Use of any such information
is strictly prohibited without express written consent of the sender

Re: ip_conntrack_http?

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 2533 bytes --]

On Mon, 2002-12-09 at 17:16, Chris Shepherd wrote:

[big snip]
> What I'm wondering is twofold:
> 1) Did I do something wrong with my configuration?

No, this is how loadbalacing in iptables works, it balances the
individual connections.

> 2) If not, and this is by design, is there any module/could a module be written 
> that would track HTTP requests and forward ALL http requests from the same 
> connection to the same IP?

All http requests from the same connection? What do you mean? All
packets in a connection is forwarded to the same server. But not all
connections are forwarded to the same server.

> What happens now:
> CONN1a -> WS1
> CONN1b -> WS2
> CONN2a -> WS1
> CONN2b -> WS2
> 
> What should happen:
> CONN1a -> WS1
> CONN1b -> WS1
> CONN2a -> WS2
> CONN2b -> WS2
> 
> Provided that CONN1[ab] are related connections, but unrelated to CONN2[ab].

How do you relate http connections to each other?

> Is this possible with the current NetFilter setup? Must a module be written?

Depends on if it's possible to uniquely identify all connections that's
part of the same session or not. Only then it's possible to write a
module to do this. I don't know if that's possible.

One solution would be to forward all connections from a certain ip to a
certain server, this can be done with the SAME module if it's modified a
little (only permits SNAT at the moment IIRC).

I think I should remove this limit from SAME even if maybe you won't use
it, I'll go do that now. (Don't know why I put it in there in the first
place, guess I didn't think that it could be used for this).

Patch is attached, just patchyour kernel with the SAME patch from
patch-o-matic (after running './runme pending'). Then just patchit with
this patch and compile.

then it's just a matter of replacing -j SNAT with -j SAME and hope for
the best :)

(SAME has a --nodst option that makes it not include the destination
ipaddress in the calculation that decides which ip to redirect to,
probably doesn't matter in this situation)

If you try it, please report back and tell me if it works (it's
completely untested, but it should work :)

> I am very interested in knowing, because it could save myself and a few dozen 
> webdevs I know a lot of money that we'd be spending on a hardware connection-
> level load balancer.

Other options may be the LVS, Linux Virtual Server project. I believe
they have loadbalancers and stuff for http.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

[-- Attachment #2: ipt_SAME.c-prerouting.diff --]
[-- Type: text/plain, Size: 798 bytes --]

--- linux-2.4.20-rsn1/net/ipv4/netfilter/ipt_SAME.c.orig	2002-12-09 22:26:20.000000000 +0100
+++ linux-2.4.20-rsn1/net/ipv4/netfilter/ipt_SAME.c	2002-12-09 22:28:10.000000000 +0100
@@ -60,7 +60,7 @@
 		DEBUGP("same_check: size %u.\n", targinfosize);
 		return 0;
 	}
-	if (hook_mask & ~(1 << NF_IP_POST_ROUTING)) {
+	if (hook_mask & ~(1 << NF_IP_PRE_ROUTING | 1 << NF_IP_POST_ROUTING)) {
 		DEBUGP("same_check: bad hooks %x.\n", hook_mask);
 		return 0;
 	}
@@ -146,7 +146,8 @@
 	struct ip_nat_multi_range newrange;
 	const struct ip_conntrack_tuple *t;
 
-	IP_NF_ASSERT(hooknum == NF_IP_POST_ROUTING);
+	IP_NF_ASSERT(hooknum == NF_IP_PRE_ROUTING ||
+			hooknum == NF_IP_POST_ROUTING);
 	ct = ip_conntrack_get(*pskb, &ctinfo);
 
 	t = &ct->tuplehash[IP_CT_DIR_ORIGINAL].tuple;

Re: ip_conntrack_http?

[Chris Shepherd]

Quoting Martin Josefsson <gandalf@wlug.westbo.se>:

> On Mon, 2002-12-09 at 17:16, Chris Shepherd wrote:
> 
> [big snip]
> > What I'm wondering is twofold:
> > 1) Did I do something wrong with my configuration?
> 
> No, this is how loadbalacing in iptables works, it balances the
> individual connections.

That's what I thought.

> > 2) If not, and this is by design, is there any module/could a module be
> > written that would track HTTP requests and forward ALL http requests from 
> > the same connection to the same IP?
> 
> All http requests from the same connection? What do you mean? All
> packets in a connection is forwarded to the same server. But not all
> connections are forwarded to the same server.

Sorry, got my wording entirely mixed up. I see that you got it from my example.

> > What happens now:
> > CONN1a -> WS1
> > CONN1b -> WS2
> > CONN2a -> WS1
> > CONN2b -> WS2
> > 
> > What should happen:
> > CONN1a -> WS1
> > CONN1b -> WS1
> > CONN2a -> WS2
> > CONN2b -> WS2
> > 
> > Provided that CONN1[ab] are related connections, but unrelated to
> CONN2[ab].
> 
> How do you relate http connections to each other?

Well, I would think the TCP RELATED flag should be set for the second 
connection from the browser. Not being a Netfilter programmer myself, I'm not 
sure if there's any connection identifier that's passed along with the RELATED 
connection information... If there was, would it not be possible to just 
forward all related connection IDs to one IP? So at most there should only be 
two or three connections in a group, and this would sort out situations where 
people sharing a connection would otherwise all end up on the same webserver, 
and not be effectively load balanced.

> > Is this possible with the current NetFilter setup? Must a module be
> written?
> 
> Depends on if it's possible to uniquely identify all connections that's
> part of the same session or not. Only then it's possible to write a
> module to do this. I don't know if that's possible.

On the connection-level, is it possible to somehow see which connection a new 
connection is related to? If so, I would think it'd be logically easy, but not 
necessarily programmatically so.

> One solution would be to forward all connections from a certain ip to a
> certain server, this can be done with the SAME module if it's modified a
> little (only permits SNAT at the moment IIRC).
> 
> I think I should remove this limit from SAME even if maybe you won't use
> it, I'll go do that now. (Don't know why I put it in there in the first
> place, guess I didn't think that it could be used for this).
> 
> Patch is attached, just patchyour kernel with the SAME patch from
> patch-o-matic (after running './runme pending'). Then just patchit with
> this patch and compile.
> 
> then it's just a matter of replacing -j SNAT with -j SAME and hope for
> the best :)
> 
> (SAME has a --nodst option that makes it not include the destination
> ipaddress in the calculation that decides which ip to redirect to,
> probably doesn't matter in this situation)
> 
> If you try it, please report back and tell me if it works (it's
> completely untested, but it should work :)

I will let you know when I get a chance to test it. I have the sneaky feeling 
that if this works properly (which it should), a lot of developers might wanna 
know about it. :)
Thank you so much for your help, and being willing to make changes to the 
module for me!

> > I am very interested in knowing, because it could save myself and a few
> dozen 
> > webdevs I know a lot of money that we'd be spending on a hardware
> connection-
> > level load balancer.
> 
> Other options may be the LVS, Linux Virtual Server project. I believe
> they have loadbalancers and stuff for http.

From the documentation I've read, LVS does essentially the same thing as NF 
does: it forwards on a per connection basis. It too would succumb to this 
problem. 

-- 
Chris Shepherd

-------------------------------------------------
This email may contain confidential information. Use of any such information
is strictly prohibited without express written consent of the sender

Re: ip_conntrack_http?

[Martin Josefsson]

On Tue, 2002-12-10 at 16:29, Chris Shepherd wrote:
 
> > How do you relate http connections to each other?
> 
> Well, I would think the TCP RELATED flag should be set for the second 
> connection from the browser. Not being a Netfilter programmer myself, I'm not 
> sure if there's any connection identifier that's passed along with the RELATED 
> connection information... If there was, would it not be possible to just 
> forward all related connection IDs to one IP? So at most there should only be 
> two or three connections in a group, and this would sort out situations where 
> people sharing a connection would otherwise all end up on the same webserver, 
> and not be effectively load balanced.

The question is still, how do you determine that a new connection is
related to another? The new connection is just that, a new connection.
You need some way to relate it to an existing connection. I have no idea
how to do that.

No "RELATED connection information" is passed along anywhere. It's just
an internal state in the connectiontracking. See below to understand how
it's set.

> On the connection-level, is it possible to somehow see which connection a new 
> connection is related to? If so, I would think it'd be logically easy, but not 
> necessarily programmatically so.

Related connections are for example the ftp-data connections or irc dcc
connections. They are known prior to their initiation beacause we parsed
them out of the control-datastream earlier and set up an expectation for
them. But with http we don't have a control connection we can parse to
fint out which ports will be used. Then they become marked as RELATED to
the control-datastream in which the information about them was parsed
out.
 
> > If you try it, please report back and tell me if it works (it's
> > completely untested, but it should work :)
> 
> I will let you know when I get a chance to test it. I have the sneaky feeling 
> that if this works properly (which it should), a lot of developers might wanna 
> know about it. :)
> Thank you so much for your help, and being willing to make changes to the 
> module for me!

I made this module for about the same case but reversed... SNAT to
multiple addresses and iptables will loadbalance between them. Some
stuff expects clients to always make their connection from the same ip.
For example hotmail or banking via internet (many things that require
login seems to have this requirement, for a good reason).

So I wrote the module to calculate which sourceip clients should get
when they initiate connections. Now they always get the same for all
connections and everything is happy. Note that I still get a lot of
balancing, diffrent clients get diffrent ip's, but always the same ip
(unless the range of ipaddresses used for SNAT is changed, then the
calculation is changed)
 
> > Other options may be the LVS, Linux Virtual Server project. I believe
> > they have loadbalancers and stuff for http.
> 
> >From the documentation I've read, LVS does essentially the same thing as NF 
> does: it forwards on a per connection basis. It too would succumb to this 
> problem. 

Ok, I thought they had some kind of solution for this problem. Maybe I
should go read up on it a little bit.
 
-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

Re: ip_conntrack_http?

[Roberto Nibali]

Hello,

While I think you've got a problem with your software architecture I assure you 
that this can be handled with LVS.

>>No, this is how loadbalacing in iptables works, it balances the
>>individual connections.
> 
> That's what I thought.

Load balancing done on a netfilter level is just not extremely intelligent. Use 
LVS. It's the same story as with all the rather clumsy NF modules/enhancements, 
like for example ROUTE. Use the tools that are given to you and don't cludge NF 
with even more deisgns that are not complete. LVS does an excellent job at load 
balancing, iproute2 does an excellent job on routing (and RR balancing too 
actually).

>>>What happens now:
>>>CONN1a -> WS1
>>>CONN1b -> WS2
>>>CONN2a -> WS1
>>>CONN2b -> WS2
>>>
>>>What should happen:
>>>CONN1a -> WS1
>>>CONN1b -> WS1
>>>CONN2a -> WS2
>>>CONN2b -> WS2
>>>
>>>Provided that CONN1[ab] are related connections, but unrelated to
>>
>>CONN2[ab].
>>
>>How do you relate http connections to each other?

In LVS parlance you need persistency. The IP grouping can be achieved with 
fwmark based load balancing, but in your case you won't need it anyway.

> Well, I would think the TCP RELATED flag should be set for the second 
> connection from the browser. Not being a Netfilter programmer myself, I'm not 
> sure if there's any connection identifier that's passed along with the RELATED 
> connection information... If there was, would it not be possible to just 
> forward all related connection IDs to one IP? So at most there should only be 
> two or three connections in a group, and this would sort out situations where 
> people sharing a connection would otherwise all end up on the same webserver, 
> and not be effectively load balanced.

This is a problem, yes, but everyday reports on the LVS mailinglist do not 
mention a big problem with this. Especially since the Internet is very dynamic 
the load imbalance will eventually straighten out. So, don't worry about that.

> On the connection-level, is it possible to somehow see which connection a new 
> connection is related to? If so, I would think it'd be logically easy, but not 
> necessarily programmatically so.

It is relatively easy but not with netfilter. You can track connections with so 
called connection templates and depending on the scheduler the templates vary. 
It is much faster then netfilter.

> I will let you know when I get a chance to test it. I have the sneaky feeling 
> that if this works properly (which it should), a lot of developers might wanna 
> know about it. :)

I doubt this is a good solution. It may work but only solve a limited case of 
load balancing.

>>>I am very interested in knowing, because it could save myself and a few
>>
>>dozen 
>>
>>>webdevs I know a lot of money that we'd be spending on a hardware
>>
>>connection-
>>
>>>level load balancer.

Almost all the people on LVS save money by implementing LVS as a software load 
balancer. It's made for that besides other reasons of course.

>>Other options may be the LVS, Linux Virtual Server project. I believe
>>they have loadbalancers and stuff for http.

Our load balancer doesn't really care about the protocol as long as it is L4.

>>From the documentation I've read, LVS does essentially the same thing as NF 
> does: it forwards on a per connection basis. It too would succumb to this 
> problem. 

Please read the documentation again, you don't seem to have understood it. And 
if you're still not sure about how it works, please join our newcomer-friendly 
LVS mailinglist. Thanks.

Best regards,
Roberto Nibali, ratz
-- 
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc