xt_statistic.c - the statistic match

xt_statistic.c - the statistic match

[Bryan Duff]

... gets out of sync in nth mode.  The count seems to be off somehow.  
At some point the count is off - in my case I have 3 rules that are 
consecutive:

//snip - iptables rules
iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 
10.10.11.0/24 -m statistic --mode nth --every 3 --packet 0 -j MARK 
--set-mark 1
iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 
10.10.11.0/24 -m statistic --mode nth --every 3 --packet 1 -j MARK 
--set-mark 2 iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 
-d 10.10.11.0/24 -m statistic --mode nth --every 3 --packet 2 -j MARK 
--set-mark 3
//end snip

Now when I accept those mark values, the packet counts which should be 
almost equal are off by large numbers (hundreds of thousands):

//snip - iptables -L
  978189 1210792980 ACCEPT     all  --  ethX   *       
10.10.10.0/24        10.10.11.0/24    MARK match 0x1
 2182885 2704995300 ACCEPT     all  --  ethX   *       
10.10.10.0/24        10.10.11.0/24    MARK match 0x2
 2289382 2862482240 ACCEPT     all  --  ethX   *       
10.10.10.0/24        10.10.11.0/24    MARK match 0x3
 1417708 1807169776 MARK       all  --  ethX   *       
10.10.10.0/24        10.10.11.0/24    MARK set 0x1
 1417708 1807169776 ACCEPT     all  --  ethX   *       
10.10.10.0/24        10.10.11.0/24    MARK match 0x1
//end snip

The mark target rule above should never see a packet.

//snip - printks show the goofiness (in this example packet 0, and 
packet 2 are on the same count)
...
Jan 9 22:35:32 localhost kernel: packet 0: count 1 of every 2
Jan 9 22:35:32 localhost kernel: packet 1: count 0 of every 2
Jan 9 22:35:32 localhost kernel: packet 2: count 1 of every 2
Jan 9 22:35:32 localhost kernel: packet 0: count 2 of every 2
Jan 9 22:35:32 localhost kernel: packet 0 to 0
Jan 9 22:35:32 localhost kernel: packet 1: count 1 of every 2
Jan 9 22:35:32 localhost kernel: packet 2: count 2 of every 2
Jan 9 22:35:32 localhost kernel: packet 2 to 0
...
//end snip


The last rule mark/accept should _never_ be hit (they should all be 
accepted by then). 

//snip (line 33 of xt_statistic.c - nth mode ...
        switch (info->mode) {
        case XT_STATISTIC_MODE_RANDOM:
                if ((net_random() & 0x7FFFFFFF) < 
info->u.random.probability)
                        ret = !ret;
                break;
        case XT_STATISTIC_MODE_NTH:
                info = info->master;
                spin_lock_bh(&nth_lock);
                if (info->u.nth.count++ == info->u.nth.every) {
                        info->u.nth.count = 0;
                        ret = !ret;
                }
                spin_unlock_bh(&nth_lock);
                break;
        }
//end snip

Why would the nth mode case screw up?  With 3 consecutive rules, I'll 
have problems after perhaps 500,000 packets (but it varies wildly). 

At some point of the 3 rules, 2 rules have the same packet count - so of 
3 packets: one is double counted, and 1 is missed.

Random mode has no problems, but it doesn't do any locking and keeps no 
state.  Even when I change nth from a switch statement to a if/else, I 
still have the same problem (I was thinking that maybe the compiler 
didn't like spin locks in a case statement).  I'm not sure if this is a 
locking issue, how if it is, how to resolve (spin_lock_irq?).

This module is very simple, and I see similar usage of 
spin_[un]lock_bh() all over the place.  I hope I'm missing something.

Thanks.

-Bryan

Re: xt_statistic.c - the statistic match

[Jan Engelhardt]

On Friday 2009-01-09 23:20, Bryan Duff wrote:

> ... gets out of sync in nth mode.  The count seems to be off somehow.  At some
> point the count is off - in my case I have 3 rules that are consecutive:
>
> //snip - iptables rules
> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m
> statistic --mode nth --every 3 --packet 0 -j MARK --set-mark 1
> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m
> statistic --mode nth --every 3 --packet 1 -j MARK --set-mark 2
> iptables -t
> mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m statistic
> --mode nth --every 3 --packet 2 -j MARK --set-mark 3
> //end snip
>
> Now when I accept those mark values, the packet counts which should be almost
> equal are off by large numbers (hundreds of thousands):

Works for me..

# iptables-save -c
[11253:5051887] -A PREROUTING -m statistic --mode nth --every 3 [--packet 0]
[11254:5117265] -A PREROUTING -m statistic --mode nth --every 3 --packet 1 

> //snip - iptables -L
> 978189 1210792980 ACCEPT     all  --  ethX   *       10.10.10.0/24
> 10.10.11.0/24    MARK match 0x1
> 2182885 2704995300 ACCEPT     all  --  ethX   *       10.10.10.0/24
> 10.10.11.0/24    MARK match 0x2
> 2289382 2862482240 ACCEPT     all  --  ethX   *       10.10.10.0/24
> 10.10.11.0/24    MARK match 0x3

These do not seem to be the same rules you posted above.
Where do all the mark matches come from?

> 1417708 1807169776 MARK       all  --  ethX   *       10.10.10.0/24
> 10.10.11.0/24    MARK set 0x1
> 1417708 1807169776 ACCEPT     all  --  ethX   *       10.10.10.0/24
> 10.10.11.0/24    MARK match 0x1
> //end snip

Re: xt_statistic.c - the statistic match

[Bryan Duff]

Jan Engelhardt wrote:
> On Friday 2009-01-09 23:20, Bryan Duff wrote:
>
>   
>> ... gets out of sync in nth mode.  The count seems to be off somehow.  At some
>> point the count is off - in my case I have 3 rules that are consecutive:
>>
>> //snip - iptables rules
>> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m
>> statistic --mode nth --every 3 --packet 0 -j MARK --set-mark 1
>> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m
>> statistic --mode nth --every 3 --packet 1 -j MARK --set-mark 2
>> iptables -t
>> mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m statistic
>> --mode nth --every 3 --packet 2 -j MARK --set-mark 3
>> //end snip
>>
>> Now when I accept those mark values, the packet counts which should be almost
>> equal are off by large numbers (hundreds of thousands):
>>     
>
> Works for me..
>
> # iptables-save -c
> [11253:5051887] -A PREROUTING -m statistic --mode nth --every 3 [--packet 0]
> [11254:5117265] -A PREROUTING -m statistic --mode nth --every 3 --packet 1 
>
>   
I have three rules.  Each rule marks one packet for every three that 
match - no packets matching that criteria should fall through.  After 
they are marked, I accept them.
>> //snip - iptables -L
>> 978189 1210792980 ACCEPT     all  --  ethX   *       10.10.10.0/24
>> 10.10.11.0/24    MARK match 0x1
>> 2182885 2704995300 ACCEPT     all  --  ethX   *       10.10.10.0/24
>> 10.10.11.0/24    MARK match 0x2
>> 2289382 2862482240 ACCEPT     all  --  ethX   *       10.10.10.0/24
>> 10.10.11.0/24    MARK match 0x3
>>     
>
> These do not seem to be the same rules you posted above.
> Where do all the mark matches come from?
>
>   
Those are the accept rules..., here are the match rules:

126489573 186243254796 MARK       all  --  eth0   *       
10.10.10.0/24        10.10.11.0/24    statistic mode nth every 3 MARK 
set 0x11
126489608 186238009472 MARK       all  --  eth0   *       
10.10.10.0/24        10.10.11.0/24    statistic mode nth every 3 packet 
1 MARK set 0x12
126489614 186238262872 MARK       all  --  eth0   *       
10.10.10.0/24        10.10.11.0/24    statistic mode nth every 3 packet 
2 MARK set 0x13
//the accept rules are right here...

I mark the packets (in this case a packet goes through 3 statistic match 
rules, and one should be marked).  And then I accept the marks - 
otherwise the are remarked at some point later (which I don't want).  
But the problem is that the 3 match rules get out of sync.  So instead 
of each rule matching on a different packet (and incrementing on every 
packet) - at some point 2 of the 3 rules are matching the same packet.

How could that happen?  I'm not accepting between the statistic match 
rules (which would definitely cause the rules to get out of sync).
>> 1417708 1807169776 MARK       all  --  ethX   *       10.10.10.0/24
>> 10.10.11.0/24    MARK set 0x1
>> 1417708 1807169776 ACCEPT     all  --  ethX   *       10.10.10.0/24
>> 10.10.11.0/24    MARK match 0x1
>> //end snip
>>     

Re: xt_statistic.c - the statistic match

[Patrick McHardy]

Bryan Duff wrote:
> Jan Engelhardt wrote:
>> On Friday 2009-01-09 23:20, Bryan Duff wrote:
>>
>>  
>>> ... gets out of sync in nth mode.  The count seems to be off 
>>> somehow.  At some
>>> point the count is off - in my case I have 3 rules that are consecutive:
>>>
>>> [...]
> How could that happen?  I'm not accepting between the statistic match 
> rules (which would definitely cause the rules to get out of sync).

Are you running on an SMP system? If so, try printing out the value
of info->master and verify that it always points to the same memory
(for each single match instance, so you should see exactly three
different values).

Re: xt_statistic.c - the statistic match

[Jan Engelhardt]

On Tuesday 2009-01-13 05:43, Patrick McHardy wrote:
>>>> [...]
>>
>> How could that happen?  I'm not accepting between the statistic
>> match rules (which would definitely cause the rules to get out of
>> sync).
>
> Are you running on an SMP system? If so, try printing out the value
> of info->master and verify that it always points to the same memory
> (for each single match instance, so you should see exactly three
> different values).

Since info->master is always on CPU0, it's easy:

	taskset -c 0 iptables -nvL

Re: xt_statistic.c - the statistic match

[Jan Engelhardt]

On Tuesday 2009-01-13 08:28, Jan Engelhardt wrote:
>On Tuesday 2009-01-13 05:43, Patrick McHardy wrote:
>>>>> [...]
>>>
>>> How could that happen?  I'm not accepting between the statistic
>>> match rules (which would definitely cause the rules to get out of
>>> sync).
>>
>> Are you running on an SMP system? If so, try printing out the value
>> of info->master and verify that it always points to the same memory
>> (for each single match instance, so you should see exactly three
>> different values).
>
>Since info->master is always on CPU0, it's easy:
>
>	taskset -c 0 iptables -nvL
>
Blip..

Ideally, for tracking, you can use the -m quota match (or alternatively,
-m quota2 with Xtables-addons, which supports counting upwards)
as an alternate counter to the per-counter rules. IOW

	iptables ... -m statistic ... -m quota --quota 20000000
or	iptables ... -m statistic ... -m quota2 --grow

	(should ideally be done with iptables-restore so that all rules
	take effect at the same time)

then, `taskset -c0 iptables -nvL` should return the same quota values
for all rules.

Re: xt_statistic.c - the statistic match

[James King]

On Mon, Jan 12, 2009 at 9:35 AM, Bryan Duff <bduff@astrocorp.com> wrote:
> Jan Engelhardt wrote:
>>
>> On Friday 2009-01-09 23:20, Bryan Duff wrote:
>>
>>
>>>
>>> ... gets out of sync in nth mode.  The count seems to be off somehow.  At
>>> some
>>> point the count is off - in my case I have 3 rules that are consecutive:
>>>
>>> //snip - iptables rules
>>> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d
>>> 10.10.11.0/24 -m
>>> statistic --mode nth --every 3 --packet 0 -j MARK --set-mark 1
>>> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d
>>> 10.10.11.0/24 -m
>>> statistic --mode nth --every 3 --packet 1 -j MARK --set-mark 2
>>> iptables -t
>>> mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m
>>> statistic
>>> --mode nth --every 3 --packet 2 -j MARK --set-mark 3
>>> //end snip
>>>
>>> Now when I accept those mark values, the packet counts which should be
>>> almost
>>> equal are off by large numbers (hundreds of thousands):
>
> I have three rules.  Each rule marks one packet for every three that match -
> no packets matching that criteria should fall through.  After they are
> marked, I accept them.
>>>
>>> //snip - iptables -L
>>> 978189 1210792980 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>> 10.10.11.0/24    MARK match 0x1
>>> 2182885 2704995300 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>> 10.10.11.0/24    MARK match 0x2
>>> 2289382 2862482240 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>> 10.10.11.0/24    MARK match 0x3
>>> 1417708 1807169776 MARK       all  --  ethX   *       10.10.10.0/24
>>>  10.10.11.0/24    MARK set 0x1
>>> 1417708 1807169776 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>  10.10.11.0/24    MARK match 0x1
> //end snip

I'm a bit curious about this.  I thought it was only possible to use
the MARK target in mangle, but you seem to be listing the filter
table.  I notice that mark_tg_reg[] revision 2 doesn't limit the table
to mangle like r0 and r1 do (anyone know if this a bug, or is r2
intended to be available everywhere?), but even so, when I try to
replicate rule #4 on 2.6.28+1.4.3-rc1, I just get a dmesg error.

> Those are the accept rules..., here are the match rules:
>
> 126489573 186243254796 MARK       all  --  eth0   *       10.10.10.0/24
>    10.10.11.0/24    statistic mode nth every 3 MARK set 0x11
> 126489608 186238009472 MARK       all  --  eth0   *       10.10.10.0/24
>    10.10.11.0/24    statistic mode nth every 3 packet 1 MARK set 0x12
> 126489614 186238262872 MARK       all  --  eth0   *       10.10.10.0/24
>    10.10.11.0/24    statistic mode nth every 3 packet 2 MARK set 0x13
> //the accept rules are right here...
>
> I mark the packets (in this case a packet goes through 3 statistic match
> rules, and one should be marked).  And then I accept the marks - otherwise
> the are remarked at some point later (which I don't want).  But the problem
> is that the 3 match rules get out of sync.  So instead of each rule matching
> on a different packet (and incrementing on every packet) - at some point 2
> of the 3 rules are matching the same packet.
>
> How could that happen?  I'm not accepting between the statistic match rules
> (which would definitely cause the rules to get out of sync).

Have you tried using iptables-restore?

If you're using iptables to commit these rules individually (as your
first message implies) and the system is under traffic already, it's
easy for them to get out of sync because each nth rule tracks its own
state individually and MARK is non-terminating.  Any packets processed
by the kernel between each rule addition will cause their packet
counters (and potentially nth count) to be out of sync.  There will be
a consistent delta between the packet counters for the mangle
PREROUTING rules, since they only mark the packet (or not) and
continue processing the rest of the chain.  However, if the nth count
isn't in sync across all the rules, the mark can be overwritten
further down the chain, and subsequently mess up the filter rules.
An example using your rules (assuming subnet, etc already match):

Rule #1 added
	Packet #1 processed
		Rule #1 (initial count = 2, every = 2) matches, set mark 0x11
	Packet #2 processed
		Rule #1 (count = 0, every = 2) no match
Rule #2 added
	Packet #3 processed
		Rule #1 (count = 1, every = 2) no match
		Rule #2 (initial count = 1, every = 2) no match
Rule #3 added
	Packet #4 processed
		Rule #1 (count = 2, every = 2) matches, set mark 0x11
		Rule #2 (count = 2, every = 2) matches, overwrite mark 0x12
		Rule #3 (initial count = 0, every = 2) no match
	Packet #5 processed
		Rule #1 (count = 0, every = 2) no match
		Rule #2 (count = 0, every = 2) no match
		Rule #3 (count = 1, every = 2) no match

Re: xt_statistic.c - the statistic match

[Jan Engelhardt]

On Thursday 2009-01-15 11:37, James King wrote:
>>>> //snip - iptables -L
>>>> 978189 1210792980 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>> 10.10.11.0/24    MARK match 0x1
>>>> 2182885 2704995300 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>> 10.10.11.0/24    MARK match 0x2
>>>> 2289382 2862482240 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>> 10.10.11.0/24    MARK match 0x3
>>>> 1417708 1807169776 MARK       all  --  ethX   *       10.10.10.0/24
>>>>  10.10.11.0/24    MARK set 0x1
>>>> 1417708 1807169776 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>>  10.10.11.0/24    MARK match 0x1
>> //end snip
>
>I'm a bit curious about this.  I thought it was only possible to use
>the MARK target in mangle, but you seem to be listing the filter
>table.  I notice that mark_tg_reg[] revision 2 doesn't limit the table
>to mangle like r0 and r1 do (anyone know if this a bug, or is r2
>intended to be available everywhere?)

Just posted moments ago:
http://marc.info/?l=netfilter-devel&m=123200677329507&w=2

>If you're using iptables to commit these rules individually (as your
>first message implies) and the system is under traffic already, it's
>easy for them to get out of sync because each nth rule tracks its own
>state individually and MARK is non-terminating.

And iptables -Z should take care of the counters if rules are added
one-by-one. Also noteworthy is that when iptables is run, the
ruleset (including counters) is downloaded from the kernel, and
later uploaded again - possible setting counters backwards.
(I do no think there are any workarounds to that in the kernel,
at least I have not seen any.)
But at least all of the counters are set to where they were.

Re: xt_statistic.c - the statistic match

[Bryan Duff]

Jan Engelhardt wrote:
> On Thursday 2009-01-15 11:37, James King wrote:
>   
>>>>> //snip - iptables -L
>>>>> 978189 1210792980 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>>> 10.10.11.0/24    MARK match 0x1
>>>>> 2182885 2704995300 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>>> 10.10.11.0/24    MARK match 0x2
>>>>> 2289382 2862482240 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>>> 10.10.11.0/24    MARK match 0x3
>>>>> 1417708 1807169776 MARK       all  --  ethX   *       10.10.10.0/24
>>>>>  10.10.11.0/24    MARK set 0x1
>>>>> 1417708 1807169776 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>>>  10.10.11.0/24    MARK match 0x1
>>>>>           
>>> //end snip
>>>       
>> I'm a bit curious about this.  I thought it was only possible to use
>> the MARK target in mangle, but you seem to be listing the filter
>> table.  I notice that mark_tg_reg[] revision 2 doesn't limit the table
>> to mangle like r0 and r1 do (anyone know if this a bug, or is r2
>> intended to be available everywhere?)
>>     
>
> Just posted moments ago:
> http://marc.info/?l=netfilter-devel&m=123200677329507&w=2
>
>   
>> If you're using iptables to commit these rules individually (as your
>> first message implies) and the system is under traffic already, it's
>> easy for them to get out of sync because each nth rule tracks its own
>> state individually and MARK is non-terminating.
>>     
>
> And iptables -Z should take care of the counters if rules are added
> one-by-one. Also noteworthy is that when iptables is run, the
> ruleset (including counters) is downloaded from the kernel, and
> later uploaded again - possible setting counters backwards.
> (I do no think there are any workarounds to that in the kernel,
> at least I have not seen any.)
> But at least all of the counters are set to where they were.
>   
Would iptables -Z fix the internal counter for the statistic nth match 
rule?  I don't see that it would.  Because that's the counter I really 
care about fixing.

A couple things - this problem occurs multiple times after adding the 
rules (as in it can correct itself by oops'ing again), the other amusing 
thing - if I use printk's I can make it happen faster, also if I'm doing 
more throughput it happens faster.

-Bryan

Re: xt_statistic.c - the statistic match

[Jan Engelhardt]

On Thursday 2009-01-15 16:46, Bryan Duff wrote:
>>
>> And iptables -Z should take care of the counters if rules are added
>> one-by-one. Also noteworthy is that when iptables is run, the
>> ruleset (including counters) is downloaded from the kernel, and
>> later uploaded again - possible setting counters backwards.
>> (I do no think there are any workarounds to that in the kernel,
>> at least I have not seen any.)
>> But at least all of the counters are set to where they were.
>
> Would iptables -Z fix the internal counter for the statistic nth match rule?  I
> don't see that it would.  Because that's the counter I really care about
> fixing.

It depends on the module and the implementation. As for -A/-I/-Z, all
private data will usually be retained. Only when the actual rule that
references a module is deleted, the private data of the module _may_
be removed too -- this obviously does not apply for modules that have
an information storage that can be referenced multiple times, such as
xt_recent, xt_condition or xt_quota2.

So if you want to have the nth state be zeroed too, it's best to use
iptables-restore to insert them all at once into the kernel.

> A couple things - this problem occurs multiple times after adding
> the rules (as in it can correct itself by oops'ing again), the
> other amusing thing - if I use printk's I can make it happen
> faster, also if I'm doing more throughput it happens faster.

Oopses, where?

Re: xt_statistic.c - the statistic match

[Bryan Duff]

Jan Engelhardt wrote:
> On Thursday 2009-01-15 16:46, Bryan Duff wrote:
>   
>>> And iptables -Z should take care of the counters if rules are added
>>> one-by-one. Also noteworthy is that when iptables is run, the
>>> ruleset (including counters) is downloaded from the kernel, and
>>> later uploaded again - possible setting counters backwards.
>>> (I do no think there are any workarounds to that in the kernel,
>>> at least I have not seen any.)
>>> But at least all of the counters are set to where they were.
>>>       
>> Would iptables -Z fix the internal counter for the statistic nth match rule?  I
>> don't see that it would.  Because that's the counter I really care about
>> fixing.
>>     
>
> It depends on the module and the implementation. As for -A/-I/-Z, all
> private data will usually be retained. Only when the actual rule that
> references a module is deleted, the private data of the module _may_
> be removed too -- this obviously does not apply for modules that have
> an information storage that can be referenced multiple times, such as
> xt_recent, xt_condition or xt_quota2.
>
> So if you want to have the nth state be zeroed too, it's best to use
> iptables-restore to insert them all at once into the kernel.
>
>   
>> A couple things - this problem occurs multiple times after adding
>> the rules (as in it can correct itself by oops'ing again), the
>> other amusing thing - if I use printk's I can make it happen
>> faster, also if I'm doing more throughput it happens faster.
>>     
>
> Oopses, where?
>   
I'm sorry - it doesn't kernel panic.  But the count will get out of sync 
continually (well after the rule has been put in place).  This problem 
has nothing to do with adding the rule.   Because it works just fine ... 
for a while.  Sending packets for an hour can cause this problem to 
happen a good half-dozen times at least.  So I say oopses in the sense 
that the module does not work as intended.

Once again the problem deals with the internal counter: 
"info->u.nth.count++".  If the problem was adding the rule it would only 
happen once.

-Bryan

Re: xt_statistic.c - the statistic match

[Bryan Duff]

James King wrote:
> On Mon, Jan 12, 2009 at 9:35 AM, Bryan Duff <bduff@astrocorp.com> wrote:
>   
>> Jan Engelhardt wrote:
>>     
>>> On Friday 2009-01-09 23:20, Bryan Duff wrote:
>>>
>>>
>>>       
>>>> ... gets out of sync in nth mode.  The count seems to be off somehow.  At
>>>> some
>>>> point the count is off - in my case I have 3 rules that are consecutive:
>>>>
>>>> //snip - iptables rules
>>>> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d
>>>> 10.10.11.0/24 -m
>>>> statistic --mode nth --every 3 --packet 0 -j MARK --set-mark 1
>>>> iptables -t mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d
>>>> 10.10.11.0/24 -m
>>>> statistic --mode nth --every 3 --packet 1 -j MARK --set-mark 2
>>>> iptables -t
>>>> mangle -A PREROUTING -i ethX -s 10.10.10.0/24 -d 10.10.11.0/24 -m
>>>> statistic
>>>> --mode nth --every 3 --packet 2 -j MARK --set-mark 3
>>>> //end snip
>>>>
>>>> Now when I accept those mark values, the packet counts which should be
>>>> almost
>>>> equal are off by large numbers (hundreds of thousands):
>>>>         
>> I have three rules.  Each rule marks one packet for every three that match -
>> no packets matching that criteria should fall through.  After they are
>> marked, I accept them.
>>     
>>>> //snip - iptables -L
>>>> 978189 1210792980 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>> 10.10.11.0/24    MARK match 0x1
>>>> 2182885 2704995300 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>> 10.10.11.0/24    MARK match 0x2
>>>> 2289382 2862482240 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>> 10.10.11.0/24    MARK match 0x3
>>>> 1417708 1807169776 MARK       all  --  ethX   *       10.10.10.0/24
>>>>  10.10.11.0/24    MARK set 0x1
>>>> 1417708 1807169776 ACCEPT     all  --  ethX   *       10.10.10.0/24
>>>>  10.10.11.0/24    MARK match 0x1
>>>>         
>> //end snip
>>     
>
> I'm a bit curious about this.  I thought it was only possible to use
> the MARK target in mangle, but you seem to be listing the filter
> table.  I notice that mark_tg_reg[] revision 2 doesn't limit the table
> to mangle like r0 and r1 do (anyone know if this a bug, or is r2
> intended to be available everywhere?), but even so, when I try to
> replicate rule #4 on 2.6.28+1.4.3-rc1, I just get a dmesg error.
>
>   
>> Those are the accept rules..., here are the match rules:
>>
>> 126489573 186243254796 MARK       all  --  eth0   *       10.10.10.0/24
>>    10.10.11.0/24    statistic mode nth every 3 MARK set 0x11
>> 126489608 186238009472 MARK       all  --  eth0   *       10.10.10.0/24
>>    10.10.11.0/24    statistic mode nth every 3 packet 1 MARK set 0x12
>> 126489614 186238262872 MARK       all  --  eth0   *       10.10.10.0/24
>>    10.10.11.0/24    statistic mode nth every 3 packet 2 MARK set 0x13
>> //the accept rules are right here...
>>
>> I mark the packets (in this case a packet goes through 3 statistic match
>> rules, and one should be marked).  And then I accept the marks - otherwise
>> the are remarked at some point later (which I don't want).  But the problem
>> is that the 3 match rules get out of sync.  So instead of each rule matching
>> on a different packet (and incrementing on every packet) - at some point 2
>> of the 3 rules are matching the same packet.
>>
>> How could that happen?  I'm not accepting between the statistic match rules
>> (which would definitely cause the rules to get out of sync).
>>     
>
> Have you tried using iptables-restore?
>
> If you're using iptables to commit these rules individually (as your
> first message implies) and the system is under traffic already, it's
> easy for them to get out of sync because each nth rule tracks its own
> state individually and MARK is non-terminating.  Any packets processed
> by the kernel between each rule addition will cause their packet
> counters (and potentially nth count) to be out of sync.  There will be
> a consistent delta between the packet counters for the mangle
> PREROUTING rules, since they only mark the packet (or not) and
> continue processing the rest of the chain.  However, if the nth count
> isn't in sync across all the rules, the mark can be overwritten
> further down the chain, and subsequently mess up the filter rules.
> An example using your rules (assuming subnet, etc already match):
>
>   
No, no.  That would imply my problem happens when I add these rules.  It 
doesn't.  The problem occurs some time much later (a few hundred 
thousand packets or more later).  So I don't see how the problem could 
be adding the rules, and letting a packet slip by while adding if that 
is _when_ the problem occurs.
> Rule #1 added
> 	Packet #1 processed
> 		Rule #1 (initial count = 2, every = 2) matches, set mark 0x11
> 	Packet #2 processed
> 		Rule #1 (count = 0, every = 2) no match
> Rule #2 added
> 	Packet #3 processed
> 		Rule #1 (count = 1, every = 2) no match
> 		Rule #2 (initial count = 1, every = 2) no match
> Rule #3 added
> 	Packet #4 processed
> 		Rule #1 (count = 2, every = 2) matches, set mark 0x11
> 		Rule #2 (count = 2, every = 2) matches, overwrite mark 0x12
> 		Rule #3 (initial count = 0, every = 2) no match
> 	Packet #5 processed
> 		Rule #1 (count = 0, every = 2) no match
> 		Rule #2 (count = 0, every = 2) no match
> 		Rule #3 (count = 1, every = 2) no match
>