RE: big table tweaks ?

RE: big table tweaks ?

[Baake, Matthias]

hi,

try to split up your input rules into some custom chains that the packets have 
not a really long way to traverse your input chain thats what i would do..

greets
matthias

> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org
> [mailto:netfilter-bounces@lists.netfilter.org]On Behalf Of 
> Jan Eidtmann
> Sent: Wednesday, September 14, 2005 2:43 PM
> To: netfilter@lists.netfilter.org
> Subject: big table tweaks ?
> 
> 
> hello,
> 
> i have one kind of big table with 80.000+ rules. its referenced in
> INPUT. lookup (when someone wants to connect) is kinda slow and system
> hangs noticable (desktop usage). i tried preemptive and non_preemptive
> kernels, its all the same. so, is there anything i could tweak to make
> this run smooth?
> 
> thx in advance,
> jan
> 
> 
> note: i am not subscribed...
> 
> 

Re: big table tweaks ?

[KOVACS Krisztian]

  Hi,

On Thursday 15 September 2005 14.09, Baake, Matthias wrote:
> try to split up your input rules into some custom chains that the
> packets have not a really long way to traverse your input chain thats
> what i would do..

  Or you could try using IPSet, maybe you can simplify your ruleset 
using the set match.

  http://ipset.netfilter.org

-- 
 Regards,
  Krisztian Kovacs

RE: big table tweaks ?

[John A. Sullivan III]

On Thu, 2005-09-15 at 14:09 +0200, Baake, Matthias wrote:
> hi,
> 
> try to split up your input rules into some custom chains that the packets have 
> not a really long way to traverse your input chain thats what i would do..
> 
> greets
> matthias
> 
> > -----Original Message-----
> > From: netfilter-bounces@lists.netfilter.org
> > [mailto:netfilter-bounces@lists.netfilter.org]On Behalf Of 
> > Jan Eidtmann
> > Sent: Wednesday, September 14, 2005 2:43 PM
> > To: netfilter@lists.netfilter.org
> > Subject: big table tweaks ?
> > 
> > 
> > hello,
> > 
> > i have one kind of big table with 80.000+ rules. its referenced in
> > INPUT. lookup (when someone wants to connect) is kinda slow and system
> > hangs noticable (desktop usage). i tried preemptive and non_preemptive
> > kernels, its all the same. so, is there anything i could tweak to make
> > this run smooth?
> ><snip>
If you're dealing with something that large, I would suggest you look at
how we handle rule sets in the ISCS network security management project
(http://iscs.sourceforge.net).  There is extensive documentation on how
it works.  You will especially want the document in the devel-docs
directory named PEPPacketFlow.  Since the code is still pre-alpha, we
have included the development docs in the tarball.

ISCS created rule sets are commonly the size you are referring to
because we granularly identify users and the individual resources to
which they need access (e.g., executive and financial staff only need
access to http and https only on the xyz server in New York).

The magic is done through custom chains but we take them a few steps
further.  We separate the evaluation of the question "who has access to
what," that is instead of one rule, we break each access into three
rule, first identify who they are, second find out what they have access
to, third grant or deny access.  This creates more overhead in small
rule sets but, in large rule sets, it gives you the advantage of
modularity and reusability.  In other words, with monolithic rules, I
need a rule to identify every possible combination of who and what.
With modular rules, I only need one rule for each "who" and one rule for
each "what" and then mix and match them as needed.  The result is the
same security with far fewer rules.

We then go further and arrange these rules in dual hierarchical
structures with inheritance, one for Accessors (our word for users) and
the other for Resources (the combination of a server and a service,
basically a socket).  This dramatically reduces the number of rules
required to implement highly granular security and is one of the major
reasons why the time to administer security with ISCS is reduced by
roughly 90% over the major commercial offerings like SolSoft.  Thus, one
can create a single rule which can apply to many different combinations
of users and resources.

We then take all of this and abstract the creation process away from
direct rule creation (who has time to write and maintain 80,000 rules in
a rapidly changing environment) into a small number of powerful, order
independent access control policies.

Even if you do not use ISCS to create your rules (it is still
pre-alpha), you may find understanding the rule creation paradigm very
helpful for your environment. Good luck - John

-- 
John A. Sullivan III
Open Source Development Corporation
+1 207-985-7880
jsullivan@opensourcedevel.com

Financially sustainable open source development
http://www.opensourcedevel.com

RE: big table tweaks ?

[Derick Anderson]

> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org 
> [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of 
> Jan Eidtmann
> Sent: Wednesday, September 14, 2005 8:43 AM
> To: netfilter@lists.netfilter.org
> Subject: big table tweaks ?
> 
> hello,
> 
> i have one kind of big table with 80.000+ rules. its 
> referenced in INPUT. lookup (when someone wants to connect) 
> is kinda slow and system hangs noticable (desktop usage). i 
> tried preemptive and non_preemptive kernels, its all the 
> same. so, is there anything i could tweak to make this run smooth?
> 
> thx in advance,
> jan
> 
> 
> note: i am not subscribed...

It would help knowing what you are trying to accomplish with those
80,000 rules. When I build a firewall, I know what each rule is for. I
would have a hard time keeping track of 800 rules, let alone 80,000.

You said this is referenced in INPUT, are you running services on this
host? Is it multi-homed? Are you blocking all the /24 networks of
everybody you don't like? Is -A INPUT -m state --state
RELATED,ESTABLISHED -j ACCEPT the first INPUT rule? Is this a Red Hat
box? (sorry, had to ask)

Derick Anderson

Re: big table tweaks ?

[Jan Eidtmann]

thanks for all your interest and replies! :)

well, i found a solution for my problem this morning. actually,
my setup was kind of stupid ( i was tired):

Chain INPUT (policy DROP)
target     prot opt source               destination
<...> # allow local stuff
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0           tcp flags:!0x16/0x02 state NEW
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
antip2p    all  --  0.0.0.0/0            0.0.0.0/0
<...> # allow services

NOTE: table "antip2p" drops all those networks i don't like or trust.
      see http://www.bluetack.co.uk/config/antip2p.txt

so almost every new incoming packet had to go through antip2p.

solution: only sent those packets to antip2p that actually hit a
service-port:

Chain INPUT (policy DROP)
target     prot opt source               destination
<...> # allow local stuff
DROP       tcp  --  0.0.0.0/0            0.0.0.0/0           tcp flags:!0x16/0x02 state NEW
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
antip2p    tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:21
antip2p    tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:22
antip2p    tcp  --  0.0.0.0/0            0.0.0.0/0           tcp dpt:80
<...> # allow services but walk through antip2p for each service (like above)

# iptables-show antip2p | tail -n2
87228     0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           source IP range 222.158.0.0-222.159.255.255
87229  6194  307K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0

this works smoother of course (as there are a lot less packets
going through antip2p).
not a perfect solution for busy hosts, but it works for me.

cheers,
jan

big table tweaks ?

[Jan Eidtmann]

hello,

i have one kind of big table with 80.000+ rules. its referenced in
INPUT. lookup (when someone wants to connect) is kinda slow and system
hangs noticable (desktop usage). i tried preemptive and non_preemptive
kernels, its all the same. so, is there anything i could tweak to make
this run smooth?

thx in advance,
jan


note: i am not subscribed...