[iptables] Implement huge amount of iptables make system crash

[iptables] Implement huge amount of iptables make system crash

[Léo Cavaillé]

Hello everyone,

I recently tested a new environment to account some traffic on my
network with some iptables rules

1) I have got 2 rules (one for in and one for out stream) per IP I want
to account. Since I account some contiguous adresses I wondered if a
binary tree starting with the first IP of my /19 and then chaining
through different tables to reach a single IP leaf will result in a
logarithmic complexity rather than linear…
So far I am adding the whole set of rules with a perl script using
libiptc to commit only once the rules to netfilter.


2) These ~62k rules are not really significant for the load of my
firewall. But unfortunately, I had sometimes in my tests to manage some big 
failures, where an iptables command (-Z, -A or -L) is stuck on the system 
and then one of my cores is used at 100% and the load increases, and generally 
I get this kernel alert :
http://pastebin.com/F1DL7ZZT

Any advice, or explanation will be great for my new installation !

Thanks,

-- 
Léo Cavaillé

Re: [iptables] Implement huge amount of iptables make system crash

[Jan Engelhardt]

On Sunday 2011-12-04 14:34, Léo Cavaillé wrote:

>Hello everyone,
>
>I recently tested a new environment to account some traffic on my
>network with some iptables rules
>
>1) I have got 2 rules (one for in and one for out stream) per IP I want
>to account. Since I account some contiguous adresses I wondered if a
>binary tree starting with the first IP of my /19 and then chaining
>through different tables to reach a single IP leaf will result in a
>logarithmic complexity rather than linear…

Yes, Jesper Brouer wrote some Perl code that takes care of automatically 
producing a fanned tree like that.

>So far I am adding the whole set of rules with a perl script using
>libiptc to commit only once the rules to netfilter.
>
>
>2) These ~62k rules are not really significant for the load of my
>firewall. But unfortunately, I had sometimes in my tests to manage some big 
>failures, where an iptables command (-Z, -A or -L) is stuck on the system 
>and then one of my cores is used at 100% and the load increases, and generally 
>I get this kernel alert :
>http://pastebin.com/F1DL7ZZT

Keep in mind that the ruleset is replaced for each HW thread and thus 
puts big requirements on memory available; as such though, I would have 
expected an OOM message rather than a Unable To Handle Kernel Paging 
Request.

Re: [iptables] Implement huge amount of iptables make system crash

[Leo Cavaille]

Le Sun, Dec 04, 2011 at 10:09:07PM +0100, Jan Engelhardt a écrit :
> >2) These ~62k rules are not really significant for the load of my
> >firewall. But unfortunately, I had sometimes in my tests to manage some big 
> >failures, where an iptables command (-Z, -A or -L) is stuck on the system 
> >and then one of my cores is used at 100% and the load increases, and generally 
> >I get this kernel alert :
> >http://pastebin.com/F1DL7ZZT
> 
> Keep in mind that the ruleset is replaced for each HW thread and thus 
> puts big requirements on memory available; as such though, I would have 
> expected an OOM message rather than a Unable To Handle Kernel Paging 
> Request.

I had the chance to see a 'top' running when crashing and the memory (I
got plenty =16GB) was only used about 1/4.
But the CPUs are used at full capacity ! Do you know if it is SW or HW
issue ?

It could be a coincidence but my clock is running crazy. I have got some
huge offsets recently and I am trying to solve this other issue right
now. (AFAIK there is no reason it could be link to the iptables kernel
paging request) 

-- 
Leo Cavaille

Re: [iptables] Implement huge amount of iptables make system crash

[Jan Engelhardt]

On Monday 2011-12-05 01:07, Leo Cavaille wrote:

>Le Sun, Dec 04, 2011 at 10:09:07PM +0100, Jan Engelhardt a écrit :
>> >2) These ~62k rules are not really significant for the load of my
>> >firewall. But unfortunately, I had sometimes in my tests to manage some big 
>> >failures, where an iptables command (-Z, -A or -L) is stuck on the system 
>> >and then one of my cores is used at 100% and the load increases, and generally 
>> >I get this kernel alert :
>> >http://pastebin.com/F1DL7ZZT
>> 
>> Keep in mind that the ruleset is replaced for each HW thread and thus 
>> puts big requirements on memory available; as such though, I would have 
>> expected an OOM message rather than a Unable To Handle Kernel Paging 
>> Request.
>
>I had the chance to see a 'top' running when crashing and the memory (I
>got plenty =16GB) was only used about 1/4.

You won't see this in top, because the alloc-dealloc cycle is most 
likely faster than top's interval.

http://www.spinics.net/lists/netfilter/msg51895.html


>But the CPUs are used at full capacity ! Do you know if it is SW or HW
>issue ?

A layer-8 issue. To traverse 62000 rules just for a single packet 
requires time (which is why it's even more important to go the 
logarithmic way).


>It could be a coincidence but my clock is running crazy. I have got some
>huge offsets recently and I am trying to solve this other issue right
>now. (AFAIK there is no reason it could be link to the iptables kernel
>paging request) 

Wouldn't know, but report it to linux-kernel@. I did not keep track of 
timer issues (since I did not have any).