limit module timer precision issue

limit module timer precision issue

[abirvalg]

Hi, it seems to me that limit module has issues with timer precision.
The only iptables rules i have are: 

iptables  -I OUTPUT 1 -m state --state NEW -m limit --limit 2000/sec --limit-burst 1  -j NFQUEUE --queue-num 11220
iptables -I OUTPUT 2 -m state --state NEW -j NFQUEUE --queue-num 11222

iptables  -I INPUT 1 -m state --state NEW -m limit --limit 2000/sec --limit-burst  1 -j NFQUEUE --queue-num 11221
iptables -I INPUT 2 -m state --state NEW -j NFQUEUE --queue-num 11222

(Both NFQUEUE 11220 and 11221 pass only NF_ACCEPT or NF_DROP verdicts.)

If I understand -m limit correctly, only if there is more than 2000 NEW connections going in or out, NFQUEUE 11222 will trigger.
When I seed a torrent, I hardly get 30 NEW connections per second and yet NFQUEUE 11222 triggers every now and then. I tried to lower the bar to --limit 100/sec and it still triggers ocasionally.
The way that I know that it triggers is that my app uses libnetfilter_queue and printf()s whenever it gets triggered, also watching
/proc/net/netfilter/nfnetlink_queue shows a steady growth for NFQ11222 in the column next to the last.

My hunch is that -m limit can't deal with such high precision timing. Could somebody please comment?

Re: limit module timer precision issue

[Jan Engelhardt]

On Thursday 2011-10-13 03:43, abirvalg@lavabit.com wrote:

>Hi, it seems to me that limit module has issues with timer precision.

Yes, such issues are known from earlier reports. Which is why there is 
the -m rateest/-j RATEEST pair that is supposed to be more accurate.

Re: limit module timer precision issue

[Michal Kubeček]

On Thursday 13 of October 2011, abirvalg@lavabit.com wrote:
> Hi, it seems to me that limit module has issues with timer precision.
> The only iptables rules i have are:
> 
> iptables  -I OUTPUT 1 -m state --state NEW -m limit --limit 2000/sec
> --limit-burst 1  -j NFQUEUE --queue-num 11220 iptables -I OUTPUT 2
> -m state --state NEW -j NFQUEUE --queue-num 11222
> 
> iptables  -I INPUT 1 -m state --state NEW -m limit --limit 2000/sec
> --limit-burst  1 -j NFQUEUE --queue-num 11221 iptables -I INPUT 2 -m
> state --state NEW -j NFQUEUE --queue-num 11222
> 
> (Both NFQUEUE 11220 and 11221 pass only NF_ACCEPT or NF_DROP
> verdicts.)
> 
> If I understand -m limit correctly, only if there is more than 2000
> NEW connections going in or out, NFQUEUE 11222 will trigger.

Not exactly. With "--limit-burst 1", no more than one packet per jiffy 
can pass the test. You may have only 30 NEW packets per second but it 
doesn't mean one packet every 1/30 s. And if there are two (or more) NEW 
packets within one jiffy, only the first passes the first rule.

Increasing the value of --limit-burst should help.

                                                         Michal Kubeèek

Re: limit module timer precision issue

[Jan Engelhardt]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset="windows-1252", Size: 1225 bytes --]

On Thursday 2011-10-13 09:10, Michal Kubeček wrote:

>On Thursday 13 of October 2011, abirvalg@lavabit.com wrote:
>> Hi, it seems to me that limit module has issues with timer precision.
>> The only iptables rules i have are:
>> 
>> iptables  -I OUTPUT 1 -m state --state NEW -m limit --limit 2000/sec
>> --limit-burst 1  -j NFQUEUE --queue-num 11220 iptables -I OUTPUT 2
>> -m state --state NEW -j NFQUEUE --queue-num 11222
>> 
>> iptables  -I INPUT 1 -m state --state NEW -m limit --limit 2000/sec
>> --limit-burst  1 -j NFQUEUE --queue-num 11221 iptables -I INPUT 2 -m
>> state --state NEW -j NFQUEUE --queue-num 11222
>> 
>> (Both NFQUEUE 11220 and 11221 pass only NF_ACCEPT or NF_DROP
>> verdicts.)
>> 
>> If I understand -m limit correctly, only if there is more than 2000
>> NEW connections going in or out, NFQUEUE 11222 will trigger.
>
>Not exactly. With "--limit-burst 1", no more than one packet per jiffy 
>can pass the test. You may have only 30 NEW packets per second but it 
>doesn't mean one packet every 1/30 s. And if there are two (or more) NEW 
>packets within one jiffy, only the first passes the first rule.

One packet per time quantum (which is 1 second here, not 1 jiffy).

Re: limit module timer precision issue

[Michal Kubeček]

On Thursday 13 of October 2011, Jan Engelhardt wrote:
> >Not exactly. With "--limit-burst 1", no more than one packet per
> >jiffy can pass the test. You may have only 30 NEW packets per
> >second but it doesn't mean one packet every 1/30 s. And if there
> >are two (or more) NEW packets within one jiffy, only the first
> >passes the first rule.
> 
> One packet per time quantum (which is 1 second here, not 1 jiffy).

I don't think so. With "--limit 2000/s --limit-burst 1" I can pass much 
more than one packet per second (but no more than 1000 per second with 
HZ=1000).

In fact, kernel doesn't even get this information, it works only with 
rate and it gets exactly the same value from iptables command with e.g. 
'3600/h', '60/m' and '1/s'.

                                                         Michal Kubeček

Re: limit module timer precision issue

[Jan Engelhardt]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: TEXT/PLAIN; charset="windows-1252", Size: 1387 bytes --]

On Thursday 2011-10-13 13:23, Michal Kubeček wrote:

>On Thursday 13 of October 2011, Jan Engelhardt wrote:
>> >Not exactly. With "--limit-burst 1", no more than one packet per
>> >jiffy can pass the test. You may have only 30 NEW packets per
>> >second but it doesn't mean one packet every 1/30 s. And if there
>> >are two (or more) NEW packets within one jiffy, only the first
>> >passes the first rule.
>> 
>> One packet per time quantum (which is 1 second here, not 1 jiffy).
>
>I don't think so. With "--limit 2000/s --limit-burst 1" I can pass much 
>more than one packet per second (but no more than 1000 per second with 
>HZ=1000).

Well, burst does not specify any rate, so claiming "per second" (or "per 
jiffy") does not make sense anyway. As per the manpage, it replenishes 
the available credit. It's just that the math is not very intuitive. In 
a way it specifies the window for calculation. --limit n/s --limit-burst 
1, IIRC, enforces 1 packet per 1/n seconds strictly, while --limit n/s 
--limit-burst n allows for spreading n over 1 second. (The two most 
common uses.)

--limit 2000/s should always give you 2000 packets somehow distributed 
over 1 second for any burst value <= 2000. The fact that it does not 
(you observe 1000/s) hints towards what I said earlier that high-rate 
matching in xt_limit and xt_hashlimit has inaccuracies.

Re: limit module timer precision issue

[Michal Kubeček]

On Thursday 13 of October 2011, Jan Engelhardt wrote:
> Well, burst does not specify any rate, so claiming "per second" (or
> "per jiffy") does not make sense anyway.

It does. The match works with jiffy granularity and if you look at the 
source, you'll see that no more than burst packets per jiffy can pass.

> --limit 2000/s should always give you 2000 packets somehow
> distributed over 1 second for any burst value <= 2000. The fact that
> it does not (you observe 1000/s) hints towards what I said earlier
> that high-rate matching in xt_limit and xt_hashlimit has
> inaccuracies.

This is not an inaccuracy, this is simple result of how the token bucket 
algorithm works. You can never get more than burst packets per jiffy 
simply because you don't get more than burst tokens. The rate you get is 
approximately min(avg, burst*HZ).

The same problem exists in TBF: no matter what rate do you specify, real 
rate is effectively limited by burst*HZ (and by mtu*HZ). Fort TBF, this 
is even mentioned in the documentation.

                                                          Michal Kubeček

Re: limit module timer precision issue

[fuzzy_4711]

-------- Original - Text --------

> Hi, it seems to me that limit module has issues with timer precision.
> The only iptables rules i have are: 
> 
> iptables  -I OUTPUT 1 -m state --state NEW -m limit --limit 2000/sec --limit-burst 1  -j NFQUEUE --queue-num 11220
> iptables -I OUTPUT 2 -m state --state NEW -j NFQUEUE --queue-num 11222
> 
> iptables  -I INPUT 1 -m state --state NEW -m limit --limit 2000/sec --limit-burst  1 -j NFQUEUE --queue-num 11221
> iptables -I INPUT 2 -m state --state NEW -j NFQUEUE --queue-num 11222
> 
> (Both NFQUEUE 11220 and 11221 pass only NF_ACCEPT or NF_DROP verdicts.)
> 
> If I understand -m limit correctly, only if there is more than 2000 NEW connections going in or out, NFQUEUE 11222 will trigger.

Are you sure? Couldn't it be that it triggers every time your *average*
(--limit 2000/second) of newly initiated connections is in the range of
up to 2000 per second based on the calculation that only 1 single packet
(--limit-burst 1) will make it through?

Take a look here and you will understand the dependencies between
--limit and --limit-burst since it is not intuitive at all:
http://www.josefassad.com/iptables_rate_limit_module

> When I seed a torrent, I hardly get 30 NEW connections per second and yet NFQUEUE 11222 triggers every now and then. I tried to lower the bar to --limit 100/sec and it still triggers ocasionally.
> The way that I know that it triggers is that my app uses libnetfilter_queue and printf()s whenever it gets triggered, also watching
> /proc/net/netfilter/nfnetlink_queue shows a steady growth for NFQ11222 in the column next to the last.
> 
> My hunch is that -m limit can't deal with such high precision timing. Could somebody please comment?
> 
--


Have a nice day.
-fuz