viewing rules and traffic while inserting/removing rules

viewing rules and traffic while inserting/removing rules

[lists+netfilter]

Hi there,
I have quite a large list of blacklisted networks in my iptables 
firewall, approx. 20,000.
Now I have a script that updates this blacklist according to my needs 
every hour. My problem now is that during the update period (which 
consists of several iptables [-I|-D] $CHAIN -s ... -j ... commands) my 
traffic accounting is going haywire.
I have narrowed it down to the problem that displaying the rules (from 
which I extract the traffic information via iptables -xvnL $CHAIN) 
intermittently shows 0s (zeroes) as paket and byte counters while the 
insert/delete commands are being issued.
Is there a locking problem? Should I maybe report this to the devel list?
Regards,
Marc

Re: viewing rules and traffic while inserting/removing rules

[Jan Engelhardt]

On Wednesday 2008-05-07 19:26, lists+netfilter@roth.lu wrote:
> Hi there,
> I have quite a large list of blacklisted networks in my iptables firewall,
> approx. 20,000.
> Now I have a script that updates this blacklist according to my needs every
> hour. My problem now is that during the update period (which consists of
> several iptables [-I|-D] $CHAIN -s ... -j ... commands) my traffic accounting
> is going haywire.
> I have narrowed it down to the problem that displaying the rules (from which I
> extract the traffic information via iptables -xvnL $CHAIN) intermittently shows
> 0s (zeroes) as paket and byte counters while the insert/delete commands are
> being issued.
> Is there a locking problem? Should I maybe report this to the devel list?

Each invocation of iptables retrieves and writes the rule table back
into the kernel, which is very antiperformant. You want to be using
iptables-restore here to minimize any delays.

RE: viewing rules and traffic while inserting/removing rules

[Steven Kath]

Marc wrote:
> 
> Hi there,
> I have quite a large list of blacklisted networks in my iptables
> firewall, approx. 20,000.
> Now I have a script that updates this blacklist according to my needs
> every hour. My problem now is that during the update period (which
> consists of several iptables [-I|-D] $CHAIN -s ... -j ... commands) my
> traffic accounting is going haywire.
> I have narrowed it down to the problem that displaying the rules (from
> which I extract the traffic information via iptables -xvnL $CHAIN)
> intermittently shows 0s (zeroes) as paket and byte counters while the
> insert/delete commands are being issued.
> Is there a locking problem? Should I maybe report this to the devel
list?

Have you considered using netfilter's ip sets for this?  Sounds like it
might be perfect for your needs. 

http://www.netfilter.org/projects/ipset/ 

Re: viewing rules and traffic while inserting/removing rules

[lists+netfilter]

Steven Kath wrote:
> Have you considered using netfilter's ip sets for this?  Sounds like it
> might be perfect for your needs. 
>
> http://www.netfilter.org/projects/ipset/ 
>   
Yes, if it was right in the kernel (without patch-o-matic) I agree it 
would be the way to go ;)

Jan Engelhardt wrote:
> Each invocation of iptables retrieves and writes the rule table back
> into the kernel, which is very antiperformant. You want to be using
> iptables-restore here to minimize any delays.
You mean building a file and then using 'iptables-restore -n < file' ? I 
will take a look at it. Thanks.

Re: viewing rules and traffic while inserting/removing rules

[Grant Taylor]

On 05/07/08 15:01, Steven Kath wrote:
> Have you considered using netfilter's ip sets for this?  Sounds like 
> it might be perfect for your needs.

With out knowing how or what the OP is accounting we don't know if s/he 
is looking at each individual network that is being filtered or if it is 
clients behind the system.  If it is networks that are being filtered I 
would think that the single ipset rule would make it much harder, if not 
impossible, to keep accounting data for a single rule.



Grant. . . .

Re: viewing rules and traffic while inserting/removing rules

[lists+netfilter]

Grant Taylor wrote:
> On 05/07/08 15:01, Steven Kath wrote:
>> Have you considered using netfilter's ip sets for this?  Sounds like 
>> it might be perfect for your needs.
>
> With out knowing how or what the OP is accounting we don't know if 
> s/he is looking at each individual network that is being filtered or 
> if it is clients behind the system.  If it is networks that are being 
> filtered I would think that the single ipset rule would make it much 
> harder, if not impossible, to keep accounting data for a single rule.
I do not need to keep individual accounting data for each network that 
is blocked.

Re: viewing rules and traffic while inserting/removing rules

[Grant Taylor]

On 05/08/08 12:17, lists+netfilter@roth.lu wrote:
> I do not need to keep individual accounting data for each network that 
> is blocked.

Ok.  In that case, I agree that an ipset would likely be a better way to 
go.  However you made a comment about it not being in your kernel.



Grant. . . .