[libnetfilter_queue] Load Balancing using multiple queues

[libnetfilter_queue] Load Balancing using multiple queues

[Vincent Arniego]

Hi Everyone,

We did a setup using libnetfilter_queue that examines the http headers of http packets.
In the setup, the firewall redirects packets in port 80 (source and destination) to an application listening on queue 0.
This box is acting as a router, so we setup the NFQUEUE rule in the forward chain

Somehow we were able to make it work after some adjustments in the kernel (sysctl net.core.rmem_max and rmem_default)

Assuming we are facing around 66 Mbps or around 11000 packets per second of traffic (from iptraf):
1. Is there a way to compute the correct optimized settings for net.core.rmem_max and rmem_default? Like a formula?
2.
Is there a way to automatically load balance the incoming packets to
multiple applications using multiple queues? This is assuming we cannot
segregate the packets by its source IP and/or destination IP.

Thanks!


      

Re: [libnetfilter_queue] Load Balancing using multiple queues

[Thomas Jacob]

On Thu, 2008-06-05 at 18:49 -0700, Vincent Arniego wrote:
> Hi Everyone,
> 
> We did a setup using libnetfilter_queue that examines the http headers of http packets.
> In the setup, the firewall redirects packets in port 80 (source and destination) to an application listening on queue 0.
> This box is acting as a router, so we setup the NFQUEUE rule in the forward chain
> 
> Somehow we were able to make it work after some adjustments in the kernel (sysctl net.core.rmem_max and rmem_default)
> 
> Assuming we are facing around 66 Mbps or around 11000 packets per second of traffic (from iptraf):
> 1. Is there a way to compute the correct optimized settings for net.core.rmem_max and rmem_default? Like a formula?
> 2.
> Is there a way to automatically load balance the incoming packets to
> multiple applications using multiple queues? This is assuming we cannot
> segregate the packets by its source IP and/or destination IP.

Why not use pound or some similar http proxy for that? 66mbps and 11.000
pps doesn't sound all that much (presumably this is the whole traffic,
not just ingress?), and pound is pretty fast:
http://www.apsis.ch/pound/index_html

This way, you also don't have to deal with the problem of where exactly
in the incoming packets you'll find your http headers. After all, what's
to stop a client from sending the http-request, for instance, in many
packets each containing only one character at a time.

Plus pound is very easy to use and presumably, given its size, easy
to hack.

Re: [libnetfilter_queue] Load Balancing using multiple queues

[Vincent Arniego]

Hi Thomas,

That's actually a good suggestion, Unfortunately we can't force teach our subscribers to use a proxy in their setup. They use a certain equipment that uses 3G and configuring it would be like rocket science to them.

11K packets per second, its ingress. We are  looking at around 114 Mbps total traffic at the worst case scenario. And yep, its doesn't look like much... yet.

The application's made already, and it looks at patterns in the payload to determine whether its a GET or http Response. Its working, but we would like to know if there's a way to spread the traffic out among multiple queues automatically.

I was thinking of something like

iptables -A FORWARD -p tcp --dport 80 -j NFQUEUE --queue-num 0:5

something like that. This doesn't work btw.


----- Original Message ----
> From: Thomas Jacob <jacob@internet24.de>
> To: Vincent Arniego <vincent_arniego@yahoo.com>
> Cc: netfilter@vger.kernel.org
> Sent: Friday, June 6, 2008 5:39:22 PM
> Subject: Re: [libnetfilter_queue] Load Balancing using multiple queues
> 
> On Thu, 2008-06-05 at 18:49 -0700, Vincent Arniego wrote:
> > Hi Everyone,
> > 
> > We did a setup using libnetfilter_queue that examines the http headers of http 
> packets.
> > In the setup, the firewall redirects packets in port 80 (source and 
> destination) to an application listening on queue 0.
> > This box is acting as a router, so we setup the NFQUEUE rule in the forward 
> chain
> > 
> > Somehow we were able to make it work after some adjustments in the kernel 
> (sysctl net.core.rmem_max and rmem_default)
> > 
> > Assuming we are facing around 66 Mbps or around 11000 packets per second of 
> traffic (from iptraf):
> > 1. Is there a way to compute the correct optimized settings for 
> net.core.rmem_max and rmem_default? Like a formula?
> > 2.
> > Is there a way to automatically load balance the incoming packets to
> > multiple applications using multiple queues? This is assuming we cannot
> > segregate the packets by its source IP and/or destination IP.
> 
> Why not use pound or some similar http proxy for that? 66mbps and 11.000
> pps doesn't sound all that much (presumably this is the whole traffic,
> not just ingress?), and pound is pretty fast:
> http://www.apsis.ch/pound/index_html
> 
> This way, you also don't have to deal with the problem of where exactly
> in the incoming packets you'll find your http headers. After all, what's
> to stop a client from sending the http-request, for instance, in many
> packets each containing only one character at a time.
> 
> Plus pound is very easy to use and presumably, given its size, easy
> to hack.



      

Re: [libnetfilter_queue] Load Balancing using multiple queues

[Eric Leblond]

[-- Attachment #1: Type: text/plain, Size: 1526 bytes --]

Hello,

On Friday, 2008 June  6 at  4:08:45 -0700, Vincent Arniego wrote:
> Hi Thomas,
> 
> That's actually a good suggestion, Unfortunately we can't force teach our subscribers to use a proxy in their setup. They use a certain equipment that uses 3G and configuring it would be like rocket science to them.
> 

Hmmm, most proxy can be configured to be used transparently.

> 11K packets per second, its ingress. We are  looking at around 114 Mbps total traffic at the worst case scenario. And yep, its doesn't look like much... yet.
> 
> The application's made already, and it looks at patterns in the payload to determine whether its a GET or http Response. Its working, but we would like to know if there's a way to spread the traffic out among multiple queues automatically.
> 
> I was thinking of something like
> 
> iptables -A FORWARD -p tcp --dport 80 -j NFQUEUE --queue-num 0:5
> 
> something like that. This doesn't work btw.

You can use statistic module to do so by using something like:
iptables -A FORWARD -p tcp --dport 80 -m statistic \
	--mode nth --every 6 --packet 0 -j NFQUEUE --queue-num 0
iptables -A FORWARD -p tcp --dport 80 -m statistic \
	--mode nth --every 6 --packet 1 -j NFQUEUE --queue-num 1
  ....
iptables -A FORWARD -p tcp --dport 80 -m statistic \
	--mode nth --every 6 --packet 5 -j NFQUEUE --queue-num 5

But, I don't think you will get much performance improvement by doing that.

BR,
-- 
Eric Leblond
INL: http://www.inl.fr/
NuFW: http://www.nufw.org/

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [libnetfilter_queue] Load Balancing using multiple queues

[Thomas Jacob]

On Fri, 2008-06-06 at 04:08 -0700, Vincent Arniego wrote:
> Hi Thomas,
> 
> That's actually a good suggestion, Unfortunately we can't force teach our subscribers to use a proxy in their setup. >
> They use a certain equipment that uses 3G and configuring it would be like rocket science to them.

So you're not load balancing requests to one website then... obviously I
didn't read your post thoroughly enough ;-)

A transparent proxy solution could still work (but not with pound).

> iptables -A FORWARD -p tcp --dport 80 -j NFQUEUE --queue-num 0:5
> 
> something like that. This doesn't work btw.

You could you statistic+connmark to do that, so that all packets
belonging to one connection are copied to the same queue:

iptables -N PORT80
iptables -N PORT80_NEW

iptables -A FORWARD -p tcp --dport 80 -j PORT80

iptables -A PORT80 -m connmark --mark 0 -j PORT80_NEW

iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 0 -m
connmark --mark 0 -j CONNMARK --set-mark 1
iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 1 -m
connmark --mark 0 -j CONNMARK --set-mark 2
iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 2 -m
connmark --mark 0 -j CONNMARK --set-mark 3
iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 3 -m
connmark --mark 0 -j CONNMARK --set-mark 4
iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 4 -m
connmark --mark 0 -j CONNMARK --set-mark 5


iptables -A PORT80 -m connmark --mark 1 -j NFQUEUE --queue-num 1
iptables -A PORT80 -m connmark --mark 2 -j NFQUEUE --queue-num 2
iptables -A PORT80 -m connmark --mark 3 -j NFQUEUE --queue-num 3
iptables -A PORT80 -m connmark --mark 4 -j NFQUEUE --queue-num 4
iptables -A PORT80 -m connmark --mark 5 -j NFQUEUE --queue-num 5

Re: [libnetfilter_queue] Load Balancing using multiple queues

[Grant Taylor]

On 06/06/08 06:08, Vincent Arniego wrote:
> That's actually a good suggestion, Unfortunately we can't force teach 
> our subscribers to use a proxy in their setup. They use a certain 
> equipment that uses 3G and configuring it would be like rocket 
> science to them.

Hum.  Are you dealing with clients going out to the world or the world 
coming in to your server(s) and / or service(s)?  Based on the fact that 
you are dealing with 3G clients (cellular?) and the fact that you are 
SNATing and DNATing traffic in to your application I'm thinking you are 
dealing with clients coming in to your server(s) and / or service(s).

Presuming this is the case, it is trivial to use a reverse proxy (a.k.a. 
web accelerator) to listen on the global address and redirect the 
queries internally as you would like, be it to one system or across 
multiple.

A note about Pound.  I've got a colleague that has been using Pound for 
a long time and I'm just starting to my self.  Pound is *extremely* 
small, with a striped binary at less than 90 kB with a control bin of 10 
kB (my *entire* Pound install is 128 kB, configs, boot scripts, and 
all).  Pound's config is pretty darn straight forward too.

ListenHTTP
	Address	<addy>
	Port	<port>
	Service
		BackEnd
			Address	<addy>
			Port	<port>
		End
		Emergency
			Address	<addy>
			Port	<port>
		End
	End
End

Repeat the BackEnd sections as many times as you need.  Emergency is 
optional and only used if no backends are available.

> 11K packets per second, its ingress. We are  looking at around 114 
> Mbps total traffic at the worst case scenario. And yep, its doesn't 
> look like much... yet.

It's that "yet" part that will get you.

> The application's made already, and it looks at patterns in the 
> payload to determine whether its a GET or http Response. Its working, 
> but we would like to know if there's a way to spread the traffic out 
> among multiple queues automatically.

Are you worried that the queues will get over loaded or be slowed down 
by something or that back end real servers will become over loaded and 
you have a fixed number of servers per queue and by increasing the 
number of queues you increase the number of back end servers?  Or are 
you thinking that having multiple queues on a single system will speed 
things up if the queue / application pare are the slow down?

> I was thinking of something like
> 
> iptables -A FORWARD -p tcp --dport 80 -j NFQUEUE --queue-num 0:5

*nod*

If only the perceived simple things actually were (simple) and existed 
and worked.

> something like that. This doesn't work btw.

*nod*

I guess I should ask, do you have to do the load balancing in NetFilter 
/ Queues or could you put something in between NetFilter and your Queues 
and have it do the load balancing for thus allowing you to concentrating 
on your queues?



Grant. . . .

Re: [libnetfilter_queue] Load Balancing using multiple queues

[Vincent Arniego]

----- Original Message ----
> From: Thomas Jacob <jacob@internet24.de>
> To: Vincent Arniego <vincent_arniego@yahoo.com>
> Cc: netfilter@vger.kernel.org
> Sent: Friday, June 6, 2008 10:46:12 PM
> Subject: Re: [libnetfilter_queue] Load Balancing using multiple queues
> 
> On Fri, 2008-06-06 at 04:08 -0700, Vincent Arniego wrote:
> > Hi Thomas,
> > 
> > That's actually a good suggestion, Unfortunately we can't force teach our 
> subscribers to use a proxy in their setup. >
> > They use a certain equipment that uses 3G and configuring it would be like 
> rocket science to them.
> 
> So you're not load balancing requests to one website then... obviously I
> didn't read your post thoroughly enough ;-)
> 
> A transparent proxy solution could still work (but not with pound).
> 
> > iptables -A FORWARD -p tcp --dport 80 -j NFQUEUE --queue-num 0:5
> > 
> > something like that. This doesn't work btw.
> 
> You could you statistic+connmark to do that, so that all packets
> belonging to one connection are copied to the same queue:
> 
> iptables -N PORT80
> iptables -N PORT80_NEW
> 
> iptables -A FORWARD -p tcp --dport 80 -j PORT80
> 
> iptables -A PORT80 -m connmark --mark 0 -j PORT80_NEW
> 
> iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 0 -m
> connmark --mark 0 -j CONNMARK --set-mark 1
> iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 1 -m
> connmark --mark 0 -j CONNMARK --set-mark 2
> iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 2 -m
> connmark --mark 0 -j CONNMARK --set-mark 3
> iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 3 -m
> connmark --mark 0 -j CONNMARK --set-mark 4
> iptables -A PORT80_NEW -m statistic --mode nth --every 5 --packet 4 -m
> connmark --mark 0 -j CONNMARK --set-mark 5
> 
> 
> iptables -A PORT80 -m connmark --mark 1 -j NFQUEUE --queue-num 1
> iptables -A PORT80 -m connmark --mark 2 -j NFQUEUE --queue-num 2
> iptables -A PORT80 -m connmark --mark 3 -j NFQUEUE --queue-num 3
> iptables -A PORT80 -m connmark --mark 4 -j NFQUEUE --queue-num 4
> iptables -A PORT80 -m connmark --mark 5 -j NFQUEUE --queue-num 5


Wow, this is the solution I'm looking for. Its not exactly like automatic load balancing, but spreading across multiple queues can do the trick.

Our application that is using libnetfilter_queue has some delays in processing the packets, around 200 ms max every so and so packets. by doing this, we could share the queue around. So that when a packet gets delayed somehow, the other packets can still move to the other queues.

Thanks a lot! Guys!