Help: Cycle through iptables rules

Help: Cycle through iptables rules

[Felipe W Damasio]

  Hi,

  I'm using squid on an ISP as a webcache.

  We have a very high load cache (6000 users with 300Mbps of web
access), and to solve the squid slowdown, the solution is to use
multiple http_port, using around 48 squid instances to serve the
users.

  It works fine when I put the machine (which is in bridge mode)
between the users and the final router (which does the NAT from
reserved IP addresses to the real IPs). I separated the rules like
this:

iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 192.168.1.0/24
--dport 80  -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3127
iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 192.168.2.0/24
--dport 80  -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3128
...


  Everything works fine when we use this scenario.

  But now it was decided (and I can't reverse the decision) that the
webcache machine was to be located between the final Router and the
internet....in this case, the cache machine is now only seeing a few
IP addresses....so I can't do this "-s <network or ip>" trick.

  So I'd like to know if I can cycle through all these rules based on
the number of connections.

  Something like "Forward the first 100 connections to port 3127, the
next 100 to 3128 ....at the end, forward the next 100 to port 3127
again", and so on.

  Is it possible?

  If it isn't currently, can this functionality be added? How can I help?

  Thanks for your advice.

   Cheers,

Felipe Damasio

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le mardi 25 mai 2010 à 21:31 -0300, Felipe W Damasio a écrit :
> Hi,
> 
>   I'm using squid on an ISP as a webcache.
> 
>   We have a very high load cache (6000 users with 300Mbps of web
> access), and to solve the squid slowdown, the solution is to use
> multiple http_port, using around 48 squid instances to serve the
> users.
> 
>   It works fine when I put the machine (which is in bridge mode)
> between the users and the final router (which does the NAT from
> reserved IP addresses to the real IPs). I separated the rules like
> this:
> 
> iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 192.168.1.0/24
> --dport 80  -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3127
> iptables -t mangle -A PREROUTING -i eth0 -p tcp -s 192.168.2.0/24
> --dport 80  -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3128
> ...
> 
> 
>   Everything works fine when we use this scenario.
> 
>   But now it was decided (and I can't reverse the decision) that the
> webcache machine was to be located between the final Router and the
> internet....in this case, the cache machine is now only seeing a few
> IP addresses....so I can't do this "-s <network or ip>" trick.
> 
>   So I'd like to know if I can cycle through all these rules based on
> the number of connections.
> 
>   Something like "Forward the first 100 connections to port 3127, the
> next 100 to 3128 ....at the end, forward the next 100 to port 3127
> again", and so on.
> 
>   Is it possible?
> 
>   If it isn't currently, can this functionality be added? How can I help?
> 

So all connections are coming from the NAT device, with a single IP, and
ports from say 30000 to 60000 (check your NAT device)

You could add a test on source port instead, using existing multiport ?

iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m multiport 
--source-ports 30000:31000 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3127

iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m multiport 
--source-ports 31000:32000 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3128

...

exact port ranges should be computed with (60000-30000/number of squid instances)

I bet a better scheme would be to hash (saddr, daddr, sport, dport) and be able to do this with one single rule

iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m multiport 
--source-ports 30000:31000 -j TPROXY --tproxy-mark 0x1/0x1 --on-ports 3127:3174

But it would need to extend TPROXY module



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Jan Engelhardt]

On Wednesday 2010-05-26 10:11, Eric Dumazet wrote:
>Le mardi 25 mai 2010 à 21:31 -0300, Felipe W Damasio a écrit :
>>
>>   So I'd like to know if I can cycle through all these rules based on
>> the number of connections.
>> 
>>   Is it possible?
>> 
>>   If it isn't currently, can this functionality be added? How can I help?
>
>So all connections are coming from the NAT device, with a single IP, and
>ports from say 30000 to 60000 (check your NAT device)
>
>iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m multiport 
>--source-ports 30000:31000 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3127

There may be a non-uniform distribution so it may not be advisable.
Instead perhaps,

-A PREROUTING -m conntrack --ctstate NEW -j extrachain
for (I = 0; I < N; ++I)
	-A extrachain -m statistic --mode nth --every I \
		-j CONNMARK --set-mark I
for (I = 0; I < N; ++I)
	-A PREROUTING -m connmark --mark I -j TPROXY \
		--tproxy-mark I/0xff --on-port I+3127
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] netfilter: xt_statistic: remove nth_lock spinlock

[Eric Dumazet]

Le mercredi 26 mai 2010 à 11:47 +0200, Jan Engelhardt a écrit :
> On Wednesday 2010-05-26 10:11, Eric Dumazet wrote:
> >Le mardi 25 mai 2010 à 21:31 -0300, Felipe W Damasio a écrit :
> >>
> >>   So I'd like to know if I can cycle through all these rules based on
> >> the number of connections.
> >> 
> >>   Is it possible?
> >> 
> >>   If it isn't currently, can this functionality be added? How can I help?
> >
> >So all connections are coming from the NAT device, with a single IP, and
> >ports from say 30000 to 60000 (check your NAT device)
> >
> >iptables -t mangle -A PREROUTING -i eth0 -p tcp --dport 80 -m multiport 
> >--source-ports 30000:31000 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 3127
> 
> There may be a non-uniform distribution so it may not be advisable.
> Instead perhaps,
> 
> -A PREROUTING -m conntrack --ctstate NEW -j extrachain
> for (I = 0; I < N; ++I)
> 	-A extrachain -m statistic --mode nth --every I \
> 		-j CONNMARK --set-mark I
> for (I = 0; I < N; ++I)
> 	-A PREROUTING -m connmark --mark I -j TPROXY \
> 		--tproxy-mark I/0xff --on-port I+3127

ok, you probably meant "-A extrachain -m statistic --mode nth --every
N ..."


packet for target "N" will take N time the nth_lock spinlock, but yes,
it should work.

This nth_lock should be removed BTW...

[PATCH] netfilter: xt_statistic: remove nth_lock spinlock

Use atomic_cmpxchg() to avoid dirtying a shared location.

xt_statistic_priv smp aligned to avoid sharing same cache line with
other stuff.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
diff --git a/net/netfilter/xt_statistic.c b/net/netfilter/xt_statistic.c
index 96e62b8..0114d2b 100644
--- a/net/netfilter/xt_statistic.c
+++ b/net/netfilter/xt_statistic.c
@@ -18,8 +18,8 @@
 #include <linux/netfilter/x_tables.h>
 
 struct xt_statistic_priv {
-	uint32_t count;
-};
+	atomic_t count;
+} ____cacheline_aligned_in_smp;
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Patrick McHardy <kaber@trash.net>");
@@ -27,13 +27,12 @@ MODULE_DESCRIPTION("Xtables: statistics-based matching (\"Nth\", random)");
 MODULE_ALIAS("ipt_statistic");
 MODULE_ALIAS("ip6t_statistic");
 
-static DEFINE_SPINLOCK(nth_lock);
-
 static bool
 statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
 {
 	const struct xt_statistic_info *info = par->matchinfo;
 	bool ret = info->flags & XT_STATISTIC_INVERT;
+	int nval, oval;
 
 	switch (info->mode) {
 	case XT_STATISTIC_MODE_RANDOM:
@@ -41,12 +40,14 @@ statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
 			ret = !ret;
 		break;
 	case XT_STATISTIC_MODE_NTH:
-		spin_lock_bh(&nth_lock);
-		if (info->master->count++ == info->u.nth.every) {
-			info->master->count = 0;
+		do {
+			oval = atomic_read(&info->master->count);
+			nval = oval + 1;
+			if (nval == info->u.nth.every)
+				nval = 0;
+		} while (atomic_cmpxchg(&info->master->count, oval, nval) != oval);
+		if (nval == 0)
 			ret = !ret;
-		}
-		spin_unlock_bh(&nth_lock);
 		break;
 	}
 
@@ -64,7 +65,7 @@ static int statistic_mt_check(const struct xt_mtchk_param *par)
 	info->master = kzalloc(sizeof(*info->master), GFP_KERNEL);
 	if (info->master == NULL)
 		return -ENOMEM;
-	info->master->count = info->u.nth.count;
+	atomic_set(&info->master->count, info->u.nth.count);
 
 	return 0;
 }


--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] netfilter: xt_statistic: remove nth_lock spinlock

[Changli Gao]

On Wed, May 26, 2010 at 7:54 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> ---
> diff --git a/net/netfilter/xt_statistic.c b/net/netfilter/xt_statistic.c
> index 96e62b8..0114d2b 100644
> --- a/net/netfilter/xt_statistic.c
> +++ b/net/netfilter/xt_statistic.c
> @@ -18,8 +18,8 @@
>  #include <linux/netfilter/x_tables.h>
>
>  struct xt_statistic_priv {
> -       uint32_t count;
> -};
> +       atomic_t count;
> +} ____cacheline_aligned_in_smp;
>
>  MODULE_LICENSE("GPL");
>  MODULE_AUTHOR("Patrick McHardy <kaber@trash.net>");
> @@ -27,13 +27,12 @@ MODULE_DESCRIPTION("Xtables: statistics-based matching (\"Nth\", random)");
>  MODULE_ALIAS("ipt_statistic");
>  MODULE_ALIAS("ip6t_statistic");
>
> -static DEFINE_SPINLOCK(nth_lock);
> -
>  static bool
>  statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
>  {
>        const struct xt_statistic_info *info = par->matchinfo;
>        bool ret = info->flags & XT_STATISTIC_INVERT;
> +       int nval, oval;
>
>        switch (info->mode) {
>        case XT_STATISTIC_MODE_RANDOM:
> @@ -41,12 +40,14 @@ statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
>                        ret = !ret;
>                break;
>        case XT_STATISTIC_MODE_NTH:
> -               spin_lock_bh(&nth_lock);
> -               if (info->master->count++ == info->u.nth.every) {
> -                       info->master->count = 0;
> +               do {
> +                       oval = atomic_read(&info->master->count);
> +                       nval = oval + 1;
> +                       if (nval == info->u.nth.every)
> +                               nval = 0;

these lines should be:
                           if (oval == info->u.nth.every)
                                    nval = 0;
                           else
                                    nval = oval + 1;


> +               } while (atomic_cmpxchg(&info->master->count, oval, nval) != oval);
> +               if (nval == 0)
>                        ret = !ret;
> -               }
> -               spin_unlock_bh(&nth_lock);
>                break;
>        }
>
> @@ -64,7 +65,7 @@ static int statistic_mt_check(const struct xt_mtchk_param *par)
>        info->master = kzalloc(sizeof(*info->master), GFP_KERNEL);
>        if (info->master == NULL)
>                return -ENOMEM;
> -       info->master->count = info->u.nth.count;
> +       atomic_set(&info->master->count, info->u.nth.count);
>
>        return 0;
>  }
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Regards，
Changli Gao(xiaosuo@gmail.com)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] netfilter: xt_statistic: remove nth_lock spinlock

[Eric Dumazet]

Le mercredi 26 mai 2010 à 20:07 +0800, Changli Gao a écrit :
> On Wed, May 26, 2010 at 7:54 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> >
> > Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
> > ---
> > diff --git a/net/netfilter/xt_statistic.c b/net/netfilter/xt_statistic.c
> > index 96e62b8..0114d2b 100644
> > --- a/net/netfilter/xt_statistic.c
> > +++ b/net/netfilter/xt_statistic.c
> > @@ -18,8 +18,8 @@
> >  #include <linux/netfilter/x_tables.h>
> >
> >  struct xt_statistic_priv {
> > -       uint32_t count;
> > -};
> > +       atomic_t count;
> > +} ____cacheline_aligned_in_smp;
> >
> >  MODULE_LICENSE("GPL");
> >  MODULE_AUTHOR("Patrick McHardy <kaber@trash.net>");
> > @@ -27,13 +27,12 @@ MODULE_DESCRIPTION("Xtables: statistics-based matching (\"Nth\", random)");
> >  MODULE_ALIAS("ipt_statistic");
> >  MODULE_ALIAS("ip6t_statistic");
> >
> > -static DEFINE_SPINLOCK(nth_lock);
> > -
> >  static bool
> >  statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
> >  {
> >        const struct xt_statistic_info *info = par->matchinfo;
> >        bool ret = info->flags & XT_STATISTIC_INVERT;
> > +       int nval, oval;
> >
> >        switch (info->mode) {
> >        case XT_STATISTIC_MODE_RANDOM:
> > @@ -41,12 +40,14 @@ statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
> >                        ret = !ret;
> >                break;
> >        case XT_STATISTIC_MODE_NTH:
> > -               spin_lock_bh(&nth_lock);
> > -               if (info->master->count++ == info->u.nth.every) {
> > -                       info->master->count = 0;
> > +               do {
> > +                       oval = atomic_read(&info->master->count);
> > +                       nval = oval + 1;
> > +                       if (nval == info->u.nth.every)
> > +                               nval = 0;
> 
> these lines should be:
>                            if (oval == info->u.nth.every)
>                                     nval = 0;
>                            else
>                                     nval = oval + 1;

Good catch, this "every" counter is off by one...

Thanks

[PATCH v2] netfilter: xt_statistic: remove nth_lock spinlock

Use atomic_cmpxchg() to avoid dirtying a shared location.

xt_statistic_priv smp aligned to avoid sharing same cache line with
other stuff.

Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
 net/netfilter/xt_statistic.c |   19 +++++++++----------
 1 file changed, 9 insertions(+), 10 deletions(-)

diff --git a/net/netfilter/xt_statistic.c b/net/netfilter/xt_statistic.c
index 96e62b8..42ecb71 100644
--- a/net/netfilter/xt_statistic.c
+++ b/net/netfilter/xt_statistic.c
@@ -18,8 +18,8 @@
 #include <linux/netfilter/x_tables.h>
 
 struct xt_statistic_priv {
-	uint32_t count;
-};
+	atomic_t count;
+} ____cacheline_aligned_in_smp;
 
 MODULE_LICENSE("GPL");
 MODULE_AUTHOR("Patrick McHardy <kaber@trash.net>");
@@ -27,13 +27,12 @@ MODULE_DESCRIPTION("Xtables: statistics-based matching (\"Nth\", random)");
 MODULE_ALIAS("ipt_statistic");
 MODULE_ALIAS("ip6t_statistic");
 
-static DEFINE_SPINLOCK(nth_lock);
-
 static bool
 statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
 {
 	const struct xt_statistic_info *info = par->matchinfo;
 	bool ret = info->flags & XT_STATISTIC_INVERT;
+	int nval, oval;
 
 	switch (info->mode) {
 	case XT_STATISTIC_MODE_RANDOM:
@@ -41,12 +40,12 @@ statistic_mt(const struct sk_buff *skb, struct xt_action_param *par)
 			ret = !ret;
 		break;
 	case XT_STATISTIC_MODE_NTH:
-		spin_lock_bh(&nth_lock);
-		if (info->master->count++ == info->u.nth.every) {
-			info->master->count = 0;
+		do {
+			oval = atomic_read(&info->master->count);
+			nval = (oval == info->u.nth.every) ? 0 : oval + 1;
+		} while (atomic_cmpxchg(&info->master->count, oval, nval) != oval);
+		if (nval == 0)
 			ret = !ret;
-		}
-		spin_unlock_bh(&nth_lock);
 		break;
 	}
 
@@ -64,7 +63,7 @@ static int statistic_mt_check(const struct xt_mtchk_param *par)
 	info->master = kzalloc(sizeof(*info->master), GFP_KERNEL);
 	if (info->master == NULL)
 		return -ENOMEM;
-	info->master->count = info->u.nth.count;
+	atomic_set(&info->master->count, info->u.nth.count);
 
 	return 0;
 }


--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] netfilter: xt_statistic: remove nth_lock spinlock

[Patrick McHardy]

Eric Dumazet wrote:
> [PATCH v2] netfilter: xt_statistic: remove nth_lock spinlock
> 
> Use atomic_cmpxchg() to avoid dirtying a shared location.
> 
> xt_statistic_priv smp aligned to avoid sharing same cache line with
> other stuff.
> 

Applied, thanks Eric.

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le mercredi 26 mai 2010 à 11:47 +0200, Jan Engelhardt a écrit :

> 
> There may be a non-uniform distribution so it may not be advisable.
> Instead perhaps,
> 
> -A PREROUTING -m conntrack --ctstate NEW -j extrachain
> for (I = 0; I < N; ++I)
> 	-A extrachain -m statistic --mode nth --every I \
> 		-j CONNMARK --set-mark I
> for (I = 0; I < N; ++I)
> 	-A PREROUTING -m connmark --mark I -j TPROXY \
> 		--tproxy-mark I/0xff --on-port I+3127

I am not sure it would work "all the time". 

Some packets could traverse the whole extrachain without hitting any
rule (extrachain is not an atomic group)

For high performance squid machine, it also adds yet another contention
points (after the conntrack lock of course...)

Some TPROXY extension would be necessary to handle RPS better on this
kind of machines.

Say you have 16 cpus, RPS is able to select a target cpu, then TPROXY
could use this CPU information to chose a destination port per cpu.

--on-port 3127+cpu



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Felipe W Damasio]

  Hi Mr. Engelhardt,

2010/5/26 Jan Engelhardt <jengelh@medozas.de>:
> -A PREROUTING -m conntrack --ctstate NEW -j extrachain
> for (I = 0; I < N; ++I)
>        -A extrachain -m statistic --mode nth --every I \
>                -j CONNMARK --set-mark I
> for (I = 0; I < N; ++I)
>        -A PREROUTING -m connmark --mark I -j TPROXY \
>                --tproxy-mark I/0xff --on-port I+3127

  You mean do this using:

N=48 (or whatever number of http_port we're using)

  So we create 48 rules using this setup?

  I can see why it'll work on the first 48 packets (one for each
rule), but what happens on the 49th new connection? It'll go on the
first rule again?

  Thanks,

Felipe Damasio
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Jan Engelhardt]

On Wednesday 2010-05-26 21:01, Felipe W Damasio wrote:
>
>2010/5/26 Jan Engelhardt <jengelh@medozas.de>:
>> -A PREROUTING -m conntrack --ctstate NEW -j extrachain
>> for (I = 0; I < N; ++I)
>>        -A extrachain -m statistic --mode nth --every I \
>>                -j CONNMARK --set-mark I
>> for (I = 0; I < N; ++I)
>>        -A PREROUTING -m connmark --mark I -j TPROXY \
>>                --tproxy-mark I/0xff --on-port I+3127
>
>  You mean do this using:
>
>N=48 (or whatever number of http_port we're using)
>
>  So we create 48 rules using this setup?

Since there are two loops to be done, it would be 96 rules in total.

>  I can see why it'll work on the first 48 packets (one for each
>rule), but what happens on the 49th new connection? It'll go on the
>first rule again?

Oh that reminds me of - "The Bible says we're supposed to work six days 
and rest on the seventh. But where is it written that we should start 
working again on the eighth?"

nth uses modulus, otherwise you can't get the "every Nth" semantic. :-)

(It should have been: --mode nth --every N --packet I)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le mercredi 26 mai 2010 à 22:18 +0200, Jan Engelhardt a écrit :
> On Wednesday 2010-05-26 21:01, Felipe W Damasio wrote:
> >
> >2010/5/26 Jan Engelhardt <jengelh@medozas.de>:
> >> -A PREROUTING -m conntrack --ctstate NEW -j extrachain
> >> for (I = 0; I < N; ++I)
> >>        -A extrachain -m statistic --mode nth --every I \
> >>                -j CONNMARK --set-mark I
> >> for (I = 0; I < N; ++I)
> >>        -A PREROUTING -m connmark --mark I -j TPROXY \
> >>                --tproxy-mark I/0xff --on-port I+3127
> >
> >  You mean do this using:
> >
> >N=48 (or whatever number of http_port we're using)
> >
> >  So we create 48 rules using this setup?
> 
> Since there are two loops to be done, it would be 96 rules in total.
> 
> >  I can see why it'll work on the first 48 packets (one for each
> >rule), but what happens on the 49th new connection? It'll go on the
> >first rule again?
> 
> Oh that reminds me of - "The Bible says we're supposed to work six days 
> and rest on the seventh. But where is it written that we should start 
> working again on the eighth?"
> 
> nth uses modulus, otherwise you can't get the "every Nth" semantic. :-)
> 
> (It should have been: --mode nth --every N --packet I)

not exactly :)

It should be --mode nth --every N-I --packet I

(first rule consume one packet out of 48, then second rule consume one
packet out of 47, ...


--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Felipe W Damasio]

  Hi Mr Dumazet and Mr Engelhardt,

2010/5/26 Eric Dumazet <eric.dumazet@gmail.com>:

>> nth uses modulus, otherwise you can't get the "every Nth" semantic. :-)
>>
>> (It should have been: --mode nth --every N --packet I)
>
> not exactly :)
>
> It should be --mode nth --every N-I --packet I
>
> (first rule consume one packet out of 48, then second rule consume one
> packet out of 47, ...

  Great, thanks. Just so I can understand it right, the 49th new
connection will be handled by the first rule, right?

  One last thing, mr. Engelhardt:

  Why did you suggest the "--tproxy-mark I/0xff" part on the first email?

  Why can't I let the old "--tproxy-mark 0x1/0x1"?

   Please correct me if I'm wrong, but I don't think the tproxy-mark
is needed for your script to work (since the packet was marked with $I
on the rule above with CONNMARK. Or is it? :-)

  Cheers,

Felipe Damasio

Re: Help: Cycle through iptables rules

[Jan Engelhardt]

On Wednesday 2010-05-26 23:49, Felipe W Damasio wrote:
>
>  Why did you suggest the "--tproxy-mark I/0xff" part on the first email?
>
>  Why can't I let the old "--tproxy-mark 0x1/0x1"?

I was not sure what you need the proxy mark for anyway,
but I just figured it was for local routing :>

Re: Help: Cycle through iptables rules

[Jan Engelhardt]

On Wednesday 2010-05-26 22:27, Eric Dumazet wrote:
>> >
>> >  So we create 48 rules using this setup?
>> 
>> Since there are two loops to be done, it would be 96 rules in total.
>> 
>> >  I can see why it'll work on the first 48 packets (one for each
>> >rule), but what happens on the 49th new connection? It'll go on the
>> >first rule again?
>> 
>> nth uses modulus, otherwise you can't get the "every Nth" semantic. :-)
>> 
>> (It should have been: --mode nth --every N --packet I)
>
>not exactly :)
>
>It should be --mode nth --every N-I --packet I
>
>(first rule consume one packet out of 48, then second rule consume one
>packet out of 47, ...

The second rule still consumes 48 packets, because CONNMARK is
non-terminating. Thus it's --every N.

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le jeudi 27 mai 2010 à 00:09 +0200, Jan Engelhardt a écrit :
> On Wednesday 2010-05-26 22:27, Eric Dumazet wrote:
> >> >
> >> >  So we create 48 rules using this setup?
> >> 
> >> Since there are two loops to be done, it would be 96 rules in total.
> >> 
> >> >  I can see why it'll work on the first 48 packets (one for each
> >> >rule), but what happens on the 49th new connection? It'll go on the
> >> >first rule again?
> >> 
> >> nth uses modulus, otherwise you can't get the "every Nth" semantic. :-)
> >> 
> >> (It should have been: --mode nth --every N --packet I)
> >
> >not exactly :)
> >
> >It should be --mode nth --every N-I --packet I
> >
> >(first rule consume one packet out of 48, then second rule consume one
> >packet out of 47, ...
> 
> The second rule still consumes 48 packets, because CONNMARK is
> non-terminating. Thus it's --every N.

Ouch, you are right, but this is very expensive then...



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Felipe W Damasio]

Hi Mr. Engelhardt,

I created a script using the instructions you gave me.

The result was:

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
extrachain  all  --  0.0.0.0/0            0.0.0.0/0           ctstate NEW
TPROXY     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
dpt:80 connmark match 0x0 TPROXY redirect 0.0.0.0:3127 mark 0x1/0x1
TPROXY     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
dpt:80 connmark match 0x1 TPROXY redirect 0.0.0.0:3128 mark 0x1/0x1
TPROXY     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
dpt:80 connmark match 0x2 TPROXY redirect 0.0.0.0:3129 mark 0x1/0x1

Chain extrachain (1 references)
target     prot opt source               destination
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 CONNMARK and 0x0
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 packet 1 CONNMARK xset 0x1/0xffffffff
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 packet 2 CONNMARK xset 0x2/0xffffffff

 This is for N=3 instances of squid.

 The TPROXY rules make sense to me (forwarding to a port based on the
mark that was done in the extrachain). I just don't get the extrachain
rules regarding that "--every 3" bit yet. :-)

 Are they right, though?

 I'm really trying to understand those statistic mode nth options :)

 Thanks.

 Cheers,

Felipe Damasio

2010/5/26 Jan Engelhardt <jengelh@medozas.de>:
> On Wednesday 2010-05-26 22:27, Eric Dumazet wrote:
>>> >
>>> >  So we create 48 rules using this setup?
>>>
>>> Since there are two loops to be done, it would be 96 rules in total.
>>>
>>> >  I can see why it'll work on the first 48 packets (one for each
>>> >rule), but what happens on the 49th new connection? It'll go on the
>>> >first rule again?
>>>
>>> nth uses modulus, otherwise you can't get the "every Nth" semantic. :-)
>>>
>>> (It should have been: --mode nth --every N --packet I)
>>
>>not exactly :)
>>
>>It should be --mode nth --every N-I --packet I
>>
>>(first rule consume one packet out of 48, then second rule consume one
>>packet out of 47, ...
>
> The second rule still consumes 48 packets, because CONNMARK is
> non-terminating. Thus it's --every N.
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le jeudi 27 mai 2010 à 17:29 -0300, Felipe W Damasio a écrit :
> Hi Mr. Engelhardt,
> 
> I created a script using the instructions you gave me.
> 
> The result was:
> 
> Chain PREROUTING (policy ACCEPT)
> target     prot opt source               destination
> extrachain  all  --  0.0.0.0/0            0.0.0.0/0           ctstate NEW
> TPROXY     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
> dpt:80 connmark match 0x0 TPROXY redirect 0.0.0.0:3127 mark 0x1/0x1
> TPROXY     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
> dpt:80 connmark match 0x1 TPROXY redirect 0.0.0.0:3128 mark 0x1/0x1
> TPROXY     tcp  --  0.0.0.0/0            0.0.0.0/0           tcp
> dpt:80 connmark match 0x2 TPROXY redirect 0.0.0.0:3129 mark 0x1/0x1
> 
> Chain extrachain (1 references)
> target     prot opt source               destination
> CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
> mode nth every 3 CONNMARK and 0x0
> CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
> mode nth every 3 packet 1 CONNMARK xset 0x1/0xffffffff
> CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
> mode nth every 3 packet 2 CONNMARK xset 0x2/0xffffffff
> 
>  This is for N=3 instances of squid.
> 
>  The TPROXY rules make sense to me (forwarding to a port based on the
> mark that was done in the extrachain). I just don't get the extrachain
> rules regarding that "--every 3" bit yet. :-)
> 
>  Are they right, though?

Its right only if you have a single NIC, and a single queue one.

Or else two SYN packets could flight in // in extrachain (using two
cpus), and the logic would be wrong.

I suggest using a first 'catch all' rule in extrachain, to make sure one
CONNMARK is done.

Or else, some connections would not go through any TPROXY rule.

Chain extrachain (1 references)
target     prot opt source               destination

CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           CONNMARK
xset 0x0/0xffffffff

CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 packet 1 CONNMARK xset 0x1/0xffffffff

CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 packet 2 CONNMARK xset 0x2/0xffffffff





--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Felipe W Damasio]

Hi Mr. Dumazet

2010/5/27 Eric Dumazet <eric.dumazet@gmail.com>:
> Its right only if you have a single NIC, and a single queue one.

  I have the machine in bridge mode with two NICs: one pointing the
users, and the other pointing the routers.

  So I guess this is the "single NIC" case, right?

> Or else two SYN packets could flight in // in extrachain (using two
> cpus), and the logic would be wrong.

  Oh I see.

  So your suggestion is to mark everything with 0x0 to make sure that
if something goes wrong the first port will get the packet, right?

  If everything goes right, it'll get marked using the other rules, right?

  Cheers,

Felipe Damasio

> I suggest using a first 'catch all' rule in extrachain, to make sure one
> CONNMARK is done.
>
> Or else, some connections would not go through any TPROXY rule.
>
> Chain extrachain (1 references)
> target     prot opt source               destination
>
> CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           CONNMARK
> xset 0x0/0xffffffff
>
> CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
> mode nth every 3 packet 1 CONNMARK xset 0x1/0xffffffff
>
> CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
> mode nth every 3 packet 2 CONNMARK xset 0x2/0xffffffff
>
>
>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Felipe W Damasio]

Hi again :)

2010/5/27 Felipe W Damasio <felipewd@gmail.com>:
>  So your suggestion is to mark everything with 0x0 to make sure that
> if something goes wrong the first port will get the packet, right?

 The resulting extrachain was:

Chain extrachain (1 references)
target     prot opt source               destination
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           CONNMARK and 0x0
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 CONNMARK and 0x0
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 packet 1 CONNMARK xset 0x1/0xffffffff
CONNMARK   all  --  0.0.0.0/0            0.0.0.0/0           statistic
mode nth every 3 packet 2 CONNMARK xset 0x2/0xffffffff

  Everything gets marked right up front, and later the statistic stuff
is used to evenly mark the packets to be forwarded to each squid port.

  Thanks for your help.

  Cheers,

Felipe Damasio
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Maciej Żenczykowski]

> not exactly :)
>
> It should be --mode nth --every N-I --packet I
>
> (first rule consume one packet out of 48, then second rule consume one
> packet out of 47, ...

I thought -j CONNMARK wasn't terminal?

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le jeudi 27 mai 2010 à 12:30 +0200, Maciej Żenczykowski a écrit :
> > not exactly :)
> >
> > It should be --mode nth --every N-I --packet I
> >
> > (first rule consume one packet out of 48, then second rule consume one
> > packet out of 47, ...
> 
> I thought -j CONNMARK wasn't terminal?

That what Jan said 12 hours ago :)



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Maciej Żenczykowski]

Yeah, I should learn to read through the entire thread before replying ;-)

2010/5/27 Eric Dumazet <eric.dumazet@gmail.com>:
> Le jeudi 27 mai 2010 à 12:30 +0200, Maciej Żenczykowski a écrit :
>> > not exactly :)
>> >
>> > It should be --mode nth --every N-I --packet I
>> >
>> > (first rule consume one packet out of 48, then second rule consume one
>> > packet out of 47, ...
>>
>> I thought -j CONNMARK wasn't terminal?
>
> That what Jan said 12 hours ago :)
>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Maciej Żenczykowski]

You could split it into a tree if you really really cared...

---

But, if you put the 48 rules in a chain which only deals with incoming
new connections then it only triggers on the initial syn and
connection tracking deals with the rest (at least if you use the
mangle table to mark, and nat table to REDIRECT -- don't know about
TPROXY).

Re: Help: Cycle through iptables rules

[Eric Dumazet]

Le jeudi 27 mai 2010 à 12:35 +0200, Maciej Żenczykowski a écrit :
> You could split it into a tree if you really really cared...
> 
> ---
> 
> But, if you put the 48 rules in a chain which only deals with incoming
> new connections then it only triggers on the initial syn and
> connection tracking deals with the rest (at least if you use the
> mangle table to mark, and nat table to REDIRECT -- don't know about
> TPROXY).

Somebody setting up 48 squid instances must care about performance, or
something is wrong...

I would expect maybe 10.000 new connections per second for such a
setup ?

I personnally would use RPS (Remote Packet Steering) to distribute the
load on all available cpus, and one squid per available cpu too.

TPROXY selection would then use a match on selected CPU

echo ff >/sys/class/net/eth0/queues/rx-0/rps_cpus

-A extrachain -m cpu 0 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3127
-A extrachain -m cpu 1 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3128
-A extrachain -m cpu 2 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3129
-A extrachain -m cpu 3 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3130
-A extrachain -m cpu 4 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3131
-A extrachain -m cpu 5 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3132
-A extrachain -m cpu 6 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3133
-A extrachain -m cpu 7 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3134



--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Help: Cycle through iptables rules

[Felipe W Damasio]

  Hi Mr. Dumazet

2010/5/27 Eric Dumazet <eric.dumazet@gmail.com>:
> Somebody setting up 48 squid instances must care about performance, or
> something is wrong...

  True, I do care, very much! :)

  We're actually using the cache machine on a 300Mbps as kind of
proof-of-concept: If it works well, we'll put on a 800Mbps network in
june.

> I would expect maybe 10.000 new connections per second for such a
> setup ?

  Yep, something like this.

> I personnally would use RPS (Remote Packet Steering) to distribute the
> load on all available cpus, and one squid per available cpu too.
>
> TPROXY selection would then use a match on selected CPU
>
> echo ff >/sys/class/net/eth0/queues/rx-0/rps_cpus
>
> -A extrachain -m cpu 0 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3127
> -A extrachain -m cpu 1 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3128
> -A extrachain -m cpu 2 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3129
> -A extrachain -m cpu 3 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3130
> -A extrachain -m cpu 4 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3131
> -A extrachain -m cpu 5 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3132
> -A extrachain -m cpu 6 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3133
> -A extrachain -m cpu 7 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3134

  That's a very good tip.

  Although the squid people told me that kernels > 2.6.32 don't play
nice with TPROXY.

  I personally haven't tested them yet, I plan to do so this weekend.
I'm currently using 2.6.29.6 on the production machine.

  But if some of the newer kernel work, I'll definitely use this, thanks.

  Cheers,

Felipe Damasio

Re: Help: Cycle through iptables rules

[Changli Gao]

2010/5/27 Eric Dumazet <eric.dumazet@gmail.com>:
> Le jeudi 27 mai 2010 à 12:35 +0200, Maciej Żenczykowski a écrit :
>> You could split it into a tree if you really really cared...
>>
>> ---
>>
>> But, if you put the 48 rules in a chain which only deals with incoming
>> new connections then it only triggers on the initial syn and
>> connection tracking deals with the rest (at least if you use the
>> mangle table to mark, and nat table to REDIRECT -- don't know about
>> TPROXY).
>
> Somebody setting up 48 squid instances must care about performance, or
> something is wrong...
>
> I would expect maybe 10.000 new connections per second for such a
> setup ?
>
> I personnally would use RPS (Remote Packet Steering) to distribute the
> load on all available cpus, and one squid per available cpu too.
>
> TPROXY selection would then use a match on selected CPU
>
> echo ff >/sys/class/net/eth0/queues/rx-0/rps_cpus
>
> -A extrachain -m cpu 0 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3127
> -A extrachain -m cpu 1 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3128
> -A extrachain -m cpu 2 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3129
> -A extrachain -m cpu 3 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3130
> -A extrachain -m cpu 4 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3131
> -A extrachain -m cpu 5 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3132
> -A extrachain -m cpu 6 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3133
> -A extrachain -m cpu 7 -j TPROXY --tproxy-mark 0x01/0xff --on-port 3134
>
>

It is much like my old idea about REDIRECT/DNAT. Anyway, an iptables
match cpu seems useful.

-- 
Regards，
Changli Gao(xiaosuo@gmail.com)
--
To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html