Linux Netfilter discussions
 help / color / mirror / Atom feed
* how to block 10000's of addresses?
@ 2002-10-13 11:50 Phil Howard
  2002-10-13 12:10 ` Antony Stone
                   ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Phil Howard @ 2002-10-13 11:50 UTC (permalink / raw)
  To: netfilter

I would like to know how best to block 10000's of addresses using
netfilter.  Clearly I do not want to be placing 10000's of individual
filter table entries in.  Is there some kind of means to set up the
equivalent of a routing table like lookup structure (which can be
added to and removed from separately) which a single netfilter rule
would reference to apply matches?

I want to block _incoming_ packets.  Null routing these addresses is
not sufficient, as the lame SYNs will continue to eat up resources.

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 11:50 how to block 10000's of addresses? Phil Howard
@ 2002-10-13 12:10 ` Antony Stone
  2002-10-13 13:00   ` Phil Howard
  2002-10-13 17:25 ` Thomas Heinz
  2002-10-13 17:42 ` Thomas Heinz
  2 siblings, 1 reply; 15+ messages in thread
From: Antony Stone @ 2002-10-13 12:10 UTC (permalink / raw)
  To: netfilter

On Sunday 13 October 2002 12:50 pm, Phil Howard wrote:

> I would like to know how best to block 10000's of addresses using
> netfilter.  Clearly I do not want to be placing 10000's of individual
> filter table entries in.

Sounds like an incompatible set of requirements.   If you want to block 10000 
addresses (and assuming they don't fit into contiguous network ranges) then 
you need 10000 rules to be able to specify what you want to block.

Why don't you want 10000 rules on your netfilter box ?   Have you tried it 
and found it causes any problems ?

> Is there some kind of means to set up the
> equivalent of a routing table like lookup structure (which can be
> added to and removed from separately) which a single netfilter rule
> would reference to apply matches?

Set up source routing and send the packets to a separate netfilter box whose 
sole purpose is to eat packets ?

> I want to block _incoming_ packets.  Null routing these addresses is
> not sufficient, as the lame SYNs will continue to eat up resources.

I don't understand that last part.   If you null route packets, surely 
there's no destination for the SYNs, therefore no half-open connections get 
set up ?

Antony.

-- 

Never write it in Perl if you can do it in Awk.
Never do it in Awk if sed can handle it.
Never use sed when tr can do the job.
Never invoke tr when cat is sufficient.
Avoid using cat whenever possible.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 12:10 ` Antony Stone
@ 2002-10-13 13:00   ` Phil Howard
  2002-10-13 13:13     ` Thomas Lussnig
                       ` (2 more replies)
  0 siblings, 3 replies; 15+ messages in thread
From: Phil Howard @ 2002-10-13 13:00 UTC (permalink / raw)
  To: netfilter

On Sun, Oct 13, 2002 at 01:10:23PM +0100, Antony Stone wrote:

| On Sunday 13 October 2002 12:50 pm, Phil Howard wrote:
| 
| > I would like to know how best to block 10000's of addresses using
| > netfilter.  Clearly I do not want to be placing 10000's of individual
| > filter table entries in.
| 
| Sounds like an incompatible set of requirements.   If you want to block 10000 
| addresses (and assuming they don't fit into contiguous network ranges) then 
| you need 10000 rules to be able to specify what you want to block.

They in fact are 10000+ different netblocks.


| Why don't you want 10000 rules on your netfilter box ?   Have you tried it 
| and found it causes any problems ?

My understanding is they are tested sequentially.  Maybe this isn't true,
but I see no documentation to the contrary regarding netfilter being any
different than past table oriented access list style filtering which uses
sequential testing to implement the ordered logic usually involved.

One other goal I had not mentioned is being able to add/delete netblocks
as needed without replacing the whole ruleset.  But I don't think it would
be a big issue.


| > Is there some kind of means to set up the
| > equivalent of a routing table like lookup structure (which can be
| > added to and removed from separately) which a single netfilter rule
| > would reference to apply matches?
| 
| Set up source routing and send the packets to a separate netfilter box whose 
| sole purpose is to eat packets ?

I'll check into that.  If source routing uses the same kinds of hashed
route tables as regular routing should (but I never confirmed whether it
actually does or not in Linux, since I've never had more than about 6 or
so routes at one time), this would be the way to go.


| > I want to block _incoming_ packets.  Null routing these addresses is
| > not sufficient, as the lame SYNs will continue to eat up resources.
| 
| I don't understand that last part.   If you null route packets, surely 
| there's no destination for the SYNs, therefore no half-open connections get 
| set up ?

Null routing is the goal.  Deciding on the course/direction to pursue is
what I am doing at the moment.  It sounds like maybe source routing might
be more appropriate than netfilter in this case.  Before this posting,
everyone is telling me to use netfilter/iptables (though I doubted they
comprehended the scale).

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 13:00   ` Phil Howard
@ 2002-10-13 13:13     ` Thomas Lussnig
  2002-10-13 13:45       ` Phil Howard
  2002-10-13 13:47     ` Robert P. J. Day
  2002-10-13 13:53     ` Antony Stone
  2 siblings, 1 reply; 15+ messages in thread
From: Thomas Lussnig @ 2002-10-13 13:13 UTC (permalink / raw)
  To: Phil Howard; +Cc: netfilter

>
>
>| Why don't you want 10000 rules on your netfilter box ?   Have you tried it 
>| and found it causes any problems ?
>
>My understanding is they are tested sequentially.  Maybe this isn't true,
>but I see no documentation to the contrary regarding netfilter being any
>different than past table oriented access list style filtering which uses
>sequential testing to implement the ordered logic usually involved.
>
>One other goal I had not mentioned is being able to add/delete netblocks
>as needed without replacing the whole ruleset.  But I don't think it would
>be a big issue.
>
With the posibility of user defined tables you can create an BTREE that 
is much faster the linear search.
But i think that for this propose an mathing like "pool" should be the 
right. there was some ime ago an
discousion here. If pool should support sparse set of ip's to (rand 
spread). Maybe you can implement it
and the use the pool module.

Cu Thomas



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 13:13     ` Thomas Lussnig
@ 2002-10-13 13:45       ` Phil Howard
  0 siblings, 0 replies; 15+ messages in thread
From: Phil Howard @ 2002-10-13 13:45 UTC (permalink / raw)
  To: netfilter

On Sun, Oct 13, 2002 at 03:13:16PM +0200, Thomas Lussnig wrote:

| >
| >
| >| Why don't you want 10000 rules on your netfilter box ?   Have you tried it 
| >| and found it causes any problems ?
| >
| >My understanding is they are tested sequentially.  Maybe this isn't true,
| >but I see no documentation to the contrary regarding netfilter being any
| >different than past table oriented access list style filtering which uses
| >sequential testing to implement the ordered logic usually involved.
| >
| >One other goal I had not mentioned is being able to add/delete netblocks
| >as needed without replacing the whole ruleset.  But I don't think it would
| >be a big issue.
| >
| With the posibility of user defined tables you can create an BTREE that 
| is much faster the linear search.
| But i think that for this propose an mathing like "pool" should be the 
| right. there was some ime ago an
| discousion here. If pool should support sparse set of ip's to (rand 
| spread). Maybe you can implement it
| and the use the pool module.

Can you provide some references?  URLs?  I'm not really following what
you are saying (maybe language problem) but maybe I would understand it
better with technical background.  I know about BTREE.  I do not know
about "pool" in this context.

I certainly want to avoid linear search of 10000 rules.

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 13:00   ` Phil Howard
  2002-10-13 13:13     ` Thomas Lussnig
@ 2002-10-13 13:47     ` Robert P. J. Day
  2002-10-13 14:56       ` Phil Howard
  2002-10-13 13:53     ` Antony Stone
  2 siblings, 1 reply; 15+ messages in thread
From: Robert P. J. Day @ 2002-10-13 13:47 UTC (permalink / raw)
  To: netfilter mailing list

On Sun, 13 Oct 2002, Phil Howard wrote:

> On Sun, Oct 13, 2002 at 01:10:23PM +0100, Antony Stone wrote:
> 
> | On Sunday 13 October 2002 12:50 pm, Phil Howard wrote:
> | 
> | > I would like to know how best to block 10000's of addresses using
> | > netfilter.  Clearly I do not want to be placing 10000's of individual
> | > filter table entries in.
> | 
> | Sounds like an incompatible set of requirements.   If you want to block 10000 
> | addresses (and assuming they don't fit into contiguous network ranges) then 
> | you need 10000 rules to be able to specify what you want to block.
> 
> They in fact are 10000+ different netblocks.

can you perhaps explain just a bit of the rationale for what
you're doing?  it's not often that one has to block that many
independent, non-related IP addresses.  just curious.  i mean, 
if it were just for filtering SPAM, you could use other tools.

rday



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 13:00   ` Phil Howard
  2002-10-13 13:13     ` Thomas Lussnig
  2002-10-13 13:47     ` Robert P. J. Day
@ 2002-10-13 13:53     ` Antony Stone
  2002-10-13 15:10       ` Phil Howard
  2 siblings, 1 reply; 15+ messages in thread
From: Antony Stone @ 2002-10-13 13:53 UTC (permalink / raw)
  To: netfilter

On Sunday 13 October 2002 2:00 pm, Phil Howard wrote:

> On Sun, Oct 13, 2002 at 01:10:23PM +0100, Antony Stone wrote:

> | Sounds like an incompatible set of requirements.   If you want to block
> | 10000 addresses (and assuming they don't fit into contiguous network
> | ranges) then you need 10000 rules to be able to specify what you want to
> | block.
>
> They in fact are 10000+ different netblocks.

Doesn't matter.   You still need 10000+ rules to block 10000+ addresses, even 
if they're entire network addresses.

> | Why don't you want 10000 rules on your netfilter box ?   Have you tried
> | it and found it causes any problems ?
>
> My understanding is they are tested sequentially.  Maybe this isn't true,
> but I see no documentation to the contrary regarding netfilter being any
> different than past table oriented access list style filtering which uses
> sequential testing to implement the ordered logic usually involved.

Your understanding is correct.   Netfilter rules are tested sequentially.   
However, I think it would still be worth a test of setting up a few thousand 
rules and see whether you get acceptable bandwidth.   What speed is your 
external Internet connection ?

> One other goal I had not mentioned is being able to add/delete netblocks
> as needed without replacing the whole ruleset.  But I don't think it would
> be a big issue.

You can add and delete netfilter rules individually without having to replace 
the entire ruleset.   It's not particularly efficient at doing so, but you 
can either replace "-A" in your rule with "-D" to delete it, or you can 
insert & delete rules by number according to their sequence in the list.

> | > Is there some kind of means to set up the
> | > equivalent of a routing table like lookup structure (which can be
> | > added to and removed from separately) which a single netfilter rule
> | > would reference to apply matches?
> |
> | Set up source routing and send the packets to a separate netfilter box
> | whose sole purpose is to eat packets ?
>
> I'll check into that.  If source routing uses the same kinds of hashed
> route tables as regular routing should (but I never confirmed whether it
> actually does or not in Linux, since I've never had more than about 6 or
> so routes at one time), this would be the way to go.

Although I wrote this suggestion first, I consider it a second-best to the 
null-routing idea below:

> | > I want to block _incoming_ packets.  Null routing these addresses is
> | > not sufficient, as the lame SYNs will continue to eat up resources.
> |
> | I don't understand that last part.   If you null route packets, surely
> | there's no destination for the SYNs, therefore no half-open connections
> | get set up ?
>
> Null routing is the goal.  Deciding on the course/direction to pursue is
> what I am doing at the moment.  It sounds like maybe source routing might
> be more appropriate than netfilter in this case.

I think so.   Try using the standard routing table's abilities to block 
packets at the gateway (same way as 192.168.0.0 packets get blocked by 
routers), before actually sending them somewhere else to get eaten - the 
latter could just be a waste of time to set up.

Antony.

-- 

In science, one tries to tell people
in such a way as to be understood by everyone
something that no-one ever knew before.

In poetry, it is the exact opposite.

 - Paul Dirac


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 13:47     ` Robert P. J. Day
@ 2002-10-13 14:56       ` Phil Howard
  2002-10-13 16:25         ` Robert P. J. Day
  0 siblings, 1 reply; 15+ messages in thread
From: Phil Howard @ 2002-10-13 14:56 UTC (permalink / raw)
  To: netfilter mailing list

On Sun, Oct 13, 2002 at 09:47:56AM -0400, Robert P. J. Day wrote:

| On Sun, 13 Oct 2002, Phil Howard wrote:
| 
| > On Sun, Oct 13, 2002 at 01:10:23PM +0100, Antony Stone wrote:
| > 
| > | On Sunday 13 October 2002 12:50 pm, Phil Howard wrote:
| > | 
| > | > I would like to know how best to block 10000's of addresses using
| > | > netfilter.  Clearly I do not want to be placing 10000's of individual
| > | > filter table entries in.
| > | 
| > | Sounds like an incompatible set of requirements.   If you want to block 10000 
| > | addresses (and assuming they don't fit into contiguous network ranges) then 
| > | you need 10000 rules to be able to specify what you want to block.
| > 
| > They in fact are 10000+ different netblocks.
| 
| can you perhaps explain just a bit of the rationale for what
| you're doing?  it's not often that one has to block that many
| independent, non-related IP addresses.  just curious.  i mean, 
| if it were just for filtering SPAM, you could use other tools.

The initial inspiration is for filtering spam.  But I can see other
uses, and am looking at this for broader re-usable purposes which
would be initially deployed to filter spamming servers.

What other tools are you referring to?  Currently I use tools that
work in an SMTP daemon of my MTA to refuse mail.  But I am wanting
to go beyond that, especially considering some places just keep
pounding on SMTP to deliver spam despite getting permanent 5XX
rejections of months or even a couple years.  Are there other tools
you are thinking of besides this?

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 13:53     ` Antony Stone
@ 2002-10-13 15:10       ` Phil Howard
  2002-10-13 15:41         ` Antony Stone
  0 siblings, 1 reply; 15+ messages in thread
From: Phil Howard @ 2002-10-13 15:10 UTC (permalink / raw)
  To: netfilter

On Sun, Oct 13, 2002 at 02:53:46PM +0100, Antony Stone wrote:

| > | Why don't you want 10000 rules on your netfilter box ?   Have you tried
| > | it and found it causes any problems ?
| >
| > My understanding is they are tested sequentially.  Maybe this isn't true,
| > but I see no documentation to the contrary regarding netfilter being any
| > different than past table oriented access list style filtering which uses
| > sequential testing to implement the ordered logic usually involved.
| 
| Your understanding is correct.   Netfilter rules are tested sequentially.   
| However, I think it would still be worth a test of setting up a few thousand 
| rules and see whether you get acceptable bandwidth.   What speed is your 
| external Internet connection ?

The external speed is 45 mbps.  Connections come in at 20-30 per second
during certain peak times.  That works out to 200000-300000 tests per
second.  I think that's pushing the envelope a bit too much, even for a
route-only box.  It's these peaks (usually spam overloading an SMTP server
despite it will be rejecting the mail) that I'm wanting to reduce the impact
from.

What I was hoping for was a means to replace an address in a rule with some
kind of reference to a lookup table object that had multiple addresses and
scaled better than O(n).


| > | > I want to block _incoming_ packets.  Null routing these addresses is
| > | > not sufficient, as the lame SYNs will continue to eat up resources.
| > |
| > | I don't understand that last part.   If you null route packets, surely
| > | there's no destination for the SYNs, therefore no half-open connections
| > | get set up ?
| >
| > Null routing is the goal.  Deciding on the course/direction to pursue is
| > what I am doing at the moment.  It sounds like maybe source routing might
| > be more appropriate than netfilter in this case.
| 
| I think so.   Try using the standard routing table's abilities to block 
| packets at the gateway (same way as 192.168.0.0 packets get blocked by 
| routers), before actually sending them somewhere else to get eaten - the 
| latter could just be a waste of time to set up.

Standard routing uses the destination to look up what to do.  This will need
to be based on source address.  Apparently the policy routing has this
capability, but the documentation for that stuff is rather vague so far.

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 15:10       ` Phil Howard
@ 2002-10-13 15:41         ` Antony Stone
  2002-10-13 16:40           ` Thomas Lussnig
  0 siblings, 1 reply; 15+ messages in thread
From: Antony Stone @ 2002-10-13 15:41 UTC (permalink / raw)
  To: netfilter

On Sunday 13 October 2002 4:10 pm, Phil Howard wrote:

> | Your understanding is correct.   Netfilter rules are tested sequentially.
> | However, I think it would still be worth a test of setting up a few
> | thousand rules and see whether you get acceptable bandwidth.   What speed
> | is your external Internet connection ?
>
> The external speed is 45 mbps.  Connections come in at 20-30 per second
> during certain peak times.  That works out to 200000-300000 tests per
> second.  I think that's pushing the envelope a bit too much, even for a
> route-only box.  It's these peaks (usually spam overloading an SMTP server
> despite it will be rejecting the mail) that I'm wanting to reduce the
> impact from.

I really think that netfilter or routing are the wrong solutions to spam.   I 
agree with Robert that you should be doing spam filtering at the SMTP 
Application layer, not the TCP/IP Networking layer.   Remember that a mail 
server can quite happily filter which connections it accepts based on source 
address, as well as anything else in the headers etc.

> What I was hoping for was a means to replace an address in a rule with some
> kind of reference to a lookup table object that had multiple addresses and
> scaled better than O(n).

Well, you could clearly create your own nest of user-defined rules where a 
packet would have to traverse a maximum of 32 rules to get a definite "go / 
no go" result for any given IPv4 address.

Just test bit31 of the address, then bit30, then bit29, etc using as many 
rules as you need for the different addresses.   You'll still have around 
10000 rules, but each packet will only get checked against a maximum of 32 of 
them (and that would be for a specific IP address - a network range would 
need fewer tests).

A compromise between these two extremes (linear vs. logarithmic time) might 
be the best for maintainability:

Create perhaps 256 user-defined chains based on the first byte of the IP 
address (or the second byte, if the first doesn't give you an even enough 
distribution of rules at the next level), and then within each of the 256 
chains, use a linear set of rules to match the addresses you want to block.

If you can get the distribution fairly even, that means for 10000 addresses, 
each packet would need to traverse on average 256/2 + (10000/256)/2 = 128 + 
20 = 148 rules.   If you did the initial subdivision into 256 categories 
using a binary chop, you'd be down to 8 + 20 = 28 rules per packet.

> Standard routing uses the destination to look up what to do.  This will
> need to be based on source address.  Apparently the policy routing has this
> capability, but the documentation for that stuff is rather vague so far.

Yes, I was suggesting that you use policy routing, but I believe that still 
has the same sort of "reject" destination as the standard "route" command, so 
you don't need to actually send your unwanted packets on anywhere.

Antony.

-- 

What is this talk of software 'release' ?
Our software evolves and matures until it becomes capable of escape,
leaving a bloody trail of designers and quality assurance people in its wake.


^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 14:56       ` Phil Howard
@ 2002-10-13 16:25         ` Robert P. J. Day
  2002-10-13 22:05           ` Phil Howard
  0 siblings, 1 reply; 15+ messages in thread
From: Robert P. J. Day @ 2002-10-13 16:25 UTC (permalink / raw)
  To: netfilter mailing list

On Sun, 13 Oct 2002, Phil Howard wrote:

> The initial inspiration is for filtering spam.  But I can see other
> uses, and am looking at this for broader re-usable purposes which
> would be initially deployed to filter spamming servers.

for spam, what about things like spamassassin?  or other spam-filtering
tools?  of course, using iptables means you can stop that stuff much
sooner at the network level.  so here's a couple of thoughts:

1) if you really have this kind of filtering need, consider getting
   a separate hardware filter to do this kind of processing.  no
   matter how cleverly you set up your netfilter rules, it's going
   to cost you processing time.  what you're asking for might just
   be a little overwhelming to ask of iptables.  might be time to
   check into a custom box that can handle tens of thousands of rules
   so your system doesn't have to deal with it.

2) if you're really determined to use iptables, consider building a
   bunch of user-defined chains to create a tree structure for the
   rules to speed up processing.  as far as i can tell, you can do
   this:

   # iptables -N network1
   # iptables -N network2
   # iptables -N network3
   ...

   # iptables -A INPUT -s 1.0.0.0/8 -j network1
   # iptables -A INPUT -s 2.0.0.0/8 -j network2
   ...

   you'd then add to the user-defined chain network1 all rules for
   filtering from source addresses on the 1.0.0.0 class A network.
   (this is just a hypothetical example, of course.)  and just keep
   going.

   if you want to be really clever, write a script of some kind that
   reads, in a simple format, all the filtering you want to do, and
   dynamically builds a nicely-balanced B-tree or something so that
   you have the optimal tree structure, and rebuild the filtering
   rules whenever your filtering needs change.

anyway, i'm just getting the hang of iptables at the moment, so this
might all be just idiotic advice.  you get what you pay for. :-)

rday



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 15:41         ` Antony Stone
@ 2002-10-13 16:40           ` Thomas Lussnig
  0 siblings, 0 replies; 15+ messages in thread
From: Thomas Lussnig @ 2002-10-13 16:40 UTC (permalink / raw)
  To: Antony Stone; +Cc: netfilter

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

>
>
>>Standard routing uses the destination to look up what to do.  This will
>>need to be based on source address.  Apparently the policy routing has this
>>capability, but the documentation for that stuff is rather vague so far.
>>    
>>
>
>Yes, I was suggesting that you use policy routing, but I believe that still 
>has the same sort of "reject" destination as the standard "route" command, so 
>you don't need to actually send your unwanted packets on anywhere.
>
Nope, the IP-Route2 command is well explained. AND you can select 
between more than one filter.
unchreachable or blackhole
unreachable send's an command back (ICMP)
and blackhole redirect the packet to the fast null device *eg*

But to catch bursts and i think whats you talking about there are two 
other solutions.
- netfilter with limit/psd/recent
- ip/tc with bandwith management based on the syn flag

Cu Thomas Lußnig

p.s. It is not an good idee to drop the syn packets since if there come 
no connect the sender retry it for some times.
Bether solutions would be sending RST packets that finaly tell the 
sender that you are not willing to accept connection.
Even if the you wan't to block an FIXED set of IP's permanently for SMTP 
you can with DNS and view based give them
an wrong (127.0.0.1) DNS solution for the MX.

Cu Thomas Lußnig

[-- Attachment #2: Type: application/pgp-signature, Size: 252 bytes --]

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 11:50 how to block 10000's of addresses? Phil Howard
  2002-10-13 12:10 ` Antony Stone
@ 2002-10-13 17:25 ` Thomas Heinz
  2002-10-13 17:42 ` Thomas Heinz
  2 siblings, 0 replies; 15+ messages in thread
From: Thomas Heinz @ 2002-10-13 17:25 UTC (permalink / raw)
  To: Phil Howard, netfilter

Hi Phil

Phil Howard schrieb:
> I would like to know how best to block 10000's of addresses using
> netfilter.  Clearly I do not want to be placing 10000's of individual
> filter table entries in.

Do you know nf-hipac? Michael Bellion and I made an announcement
on the netfilter list some time ago.
It's a new framework which implements a packet filter that maintains
the semantics of linear lists of rules while the matching algorithm
is much more efficient than just matching rules sequentially.

nf-hipac is a registered sourceforge project: 
http://sourceforge.net/projects/nf-hipac/
You find some additional information (including a performance test)
on our homepage: http://www.hipac.org/
The missing documentation will be available soon (next week) but as
the userspace tool of nf-hipac uses the same syntax as iptables it
should not be a problem for you to cope with it. You don't even
have to recompile your kernel.

Finally, nf-hipac and iptables can be used at the same time. This might
be interesting for you if you need matches/targets which we do not yet
support. There is just one important thing to notice: the order in
which the packet filters are called. Currently,  the iptables filter
table is called before nf-hipac. If you need it the other way around
tell me and I'll send you a mini patch.


Regards,

Thomas



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 11:50 how to block 10000's of addresses? Phil Howard
  2002-10-13 12:10 ` Antony Stone
  2002-10-13 17:25 ` Thomas Heinz
@ 2002-10-13 17:42 ` Thomas Heinz
  2 siblings, 0 replies; 15+ messages in thread
From: Thomas Heinz @ 2002-10-13 17:42 UTC (permalink / raw)
  To: netfilter

Hi Phil

You wrote:
> I would like to know how best to block 10000's of addresses using
> netfilter.  Clearly I do not want to be placing 10000's of individual
> filter table entries in.

Do you know nf-hipac? Michael Bellion and I made an announcement
on the netfilter list some time ago.
It's a new framework which implements a packet filter that maintains
the semantics of linear lists of rules while the matching algorithm
is much more efficient than just matching rules sequentially.

nf-hipac is a registered sourceforge project: 
http://sourceforge.net/projects/nf-hipac/
You find some additional information (including a performance test)
on our homepage: http://www.hipac.org/
The missing documentation will be available soon (next week) but as
the userspace tool of nf-hipac uses the same syntax as iptables it
should not be a problem for you to cope with it. You don't even
have to recompile your kernel.

Finally, nf-hipac and iptables can be used at the same time. This might
be interesting for you if you need matches/targets which we do not yet
support. There is just one important thing to notice: the order in
which the packet filters are called. Currently,  the iptables filter
table is called before nf-hipac. If you need it the other way around
tell me and I'll send you a mini patch.


Regards,

Thomas



^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: how to block 10000's of addresses?
  2002-10-13 16:25         ` Robert P. J. Day
@ 2002-10-13 22:05           ` Phil Howard
  0 siblings, 0 replies; 15+ messages in thread
From: Phil Howard @ 2002-10-13 22:05 UTC (permalink / raw)
  To: netfilter mailing list

On Sun, Oct 13, 2002 at 12:25:50PM -0400, Robert P. J. Day wrote:

| On Sun, 13 Oct 2002, Phil Howard wrote:
| 
| > The initial inspiration is for filtering spam.  But I can see other
| > uses, and am looking at this for broader re-usable purposes which
| > would be initially deployed to filter spamming servers.
| 
| for spam, what about things like spamassassin?  or other spam-filtering
| tools?  of course, using iptables means you can stop that stuff much
| sooner at the network level.  so here's a couple of thoughts:

I already reject mail during the SMTP transaction.  Accepting it and
running through spamassassin after delivery would just be a greater
level of load.  I'm trying to reduce it.

| 
| 1) if you really have this kind of filtering need, consider getting
|    a separate hardware filter to do this kind of processing.  no
|    matter how cleverly you set up your netfilter rules, it's going
|    to cost you processing time.  what you're asking for might just
|    be a little overwhelming to ask of iptables.  might be time to
|    check into a custom box that can handle tens of thousands of rules
|    so your system doesn't have to deal with it.

If each packet did a lookup into a tree object of some sort based on the
source address, and possibly discarded, that should reduce the processing.
The challenge is finding the method.  Maybe the policy routing will do
this, but from its skimpy documentation (I don't know if I have found it
all, yet) it also looks like it is doing a linear search.


| 2) if you're really determined to use iptables, consider building a
|    bunch of user-defined chains to create a tree structure for the
|    rules to speed up processing.  as far as i can tell, you can do
|    this:
| 
|    # iptables -N network1
|    # iptables -N network2
|    # iptables -N network3
|    ...
| 
|    # iptables -A INPUT -s 1.0.0.0/8 -j network1
|    # iptables -A INPUT -s 2.0.0.0/8 -j network2
|    ...
| 
|    you'd then add to the user-defined chain network1 all rules for
|    filtering from source addresses on the 1.0.0.0 class A network.
|    (this is just a hypothetical example, of course.)  and just keep
|    going.
| 
|    if you want to be really clever, write a script of some kind that
|    reads, in a simple format, all the filtering you want to do, and
|    dynamically builds a nicely-balanced B-tree or something so that
|    you have the optimal tree structure, and rebuild the filtering
|    rules whenever your filtering needs change.

That certainly looks like it could do it with a lower level of processing.

-- 
-----------------------------------------------------------------
| Phil Howard - KA9WGN |   Dallas   | http://linuxhomepage.com/ |
| phil-nospam@ipal.net | Texas, USA | http://ka9wgn.ham.org/    |
-----------------------------------------------------------------


^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2002-10-13 22:05 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-10-13 11:50 how to block 10000's of addresses? Phil Howard
2002-10-13 12:10 ` Antony Stone
2002-10-13 13:00   ` Phil Howard
2002-10-13 13:13     ` Thomas Lussnig
2002-10-13 13:45       ` Phil Howard
2002-10-13 13:47     ` Robert P. J. Day
2002-10-13 14:56       ` Phil Howard
2002-10-13 16:25         ` Robert P. J. Day
2002-10-13 22:05           ` Phil Howard
2002-10-13 13:53     ` Antony Stone
2002-10-13 15:10       ` Phil Howard
2002-10-13 15:41         ` Antony Stone
2002-10-13 16:40           ` Thomas Lussnig
2002-10-13 17:25 ` Thomas Heinz
2002-10-13 17:42 ` Thomas Heinz

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox