Developing a user space library for filtering

Developing a user space library for filtering

[Darren.Reed]

Hi,

One of the core problems I see as people want to more and
more with firewall/NAT technology is integrate using it into
their application (whatever that may be.)  As time goes by,
this problem is becoming more and more acute and perhaps
is doing us (those who develop said technologies) a disservice
by making the "barrier to entry" too high.

Currently, to interact with filtering software inside the kernel
requires developers to build their application against whatever
specific version of the filtering software runs in the kernel.  For
application developers, this is a PITA.  What they want to see is
the equivalent of a libc for firewalls with functions that have a
similar stability to the likes of "fopen", "printf", etc.

And therein lies the problem.  Nothing currently exists, so if you
engage in developing for any one particular firewall/NAT product
then you wed yourself to using that product.  Not a great place
to be if you're the 3rd party.

This problem/situation is one that I believe needs to be improved.

Is anyone in the Linux netfilter community currently engaged
on working on anything to address this problem?

If there isn't, can I ask if anyone would be interested in being
involved in designing and implementing a new library where
the primary focus or target is going to be 3rd party developers
that want to build applications on top of netfilter, etc?  It is
important to understand that my focus isn't just on netfilter or
iptables, but also includes ipf/ipfw/pf and potentially others too
if people become interested.

The end goal isn't to build something that userland tools would
be rewritten to use (although ostensible they could), just to
provide what those on the outside need in order to build apps
that are both portable and functional.

Darren

Re: Developing a user space library for filtering

[Carl-Daniel Hailfinger]

Hi Darren,

On 22.05.2007 00:27, Darren.Reed@Sun.COM wrote:
> 
> One of the core problems I see as people want to more and
> more with firewall/NAT technology is integrate using it into
> their application (whatever that may be.)  As time goes by,
> this problem is becoming more and more acute and perhaps
> is doing us (those who develop said technologies) a disservice
> by making the "barrier to entry" too high.

Sorry if I'm being dense. Do you want to target firewall frontends
or applications which have the desire to punch holes into the
firewall?

> Currently, to interact with filtering software inside the kernel
> requires developers to build their application against whatever
> specific version of the filtering software runs in the kernel.  For
> application developers, this is a PITA.  What they want to see is
> the equivalent of a libc for firewalls with functions that have a
> similar stability to the likes of "fopen", "printf", etc.
> 
> And therein lies the problem.  Nothing currently exists, so if you
> engage in developing for any one particular firewall/NAT product
> then you wed yourself to using that product.  Not a great place
> to be if you're the 3rd party.

Maybe you're looking for the firewalling side of UPnP?

Regards,
Carl-Daniel

Re: Developing a user space library for filtering

[Darren.Reed]

Carl-Daniel Hailfinger wrote:

>Hi Darren,
>
>On 22.05.2007 00:27, Darren.Reed@Sun.COM wrote:
>  
>
>>One of the core problems I see as people want to more and
>>more with firewall/NAT technology is integrate using it into
>>their application (whatever that may be.)  As time goes by,
>>this problem is becoming more and more acute and perhaps
>>is doing us (those who develop said technologies) a disservice
>>by making the "barrier to entry" too high.
>>    
>>
>
>Sorry if I'm being dense. Do you want to target firewall frontends
>or applications which have the desire to punch holes into the
>firewall?
>  
>

Neither.  Applications that sit on top of firewall software.

As an example, using squid on your Linux firewall/router in
transparent proxying mode requires that squid has some code
in it that knows how to talk to Linux in order to discover the
original destination and change the outgoing connection
such that the original address is used again.  Doing that
requires specific knowledge of the API that netfilter/iptables
uses.

Another example might be IDS software running on your Linux
firewall/router.  If that detects an attack, it should be able to
talk to netfilter/iptables and do "something" to mitigate it.

In both cases I'd like to see something developed such that
the 3rd party applications don't need to know what NAT or
firewall technology is being used.

Darren

Re: Developing a user space library for filtering

[Jan Engelhardt]

On May 21 2007 15:52, Darren.Reed@Sun.COM wrote:
>> 
>> Sorry if I'm being dense. Do you want to target firewall frontends
>> or applications which have the desire to punch holes into the
>> firewall?
>
> Neither.  Applications that sit on top of firewall software.
>
> As an example, using squid on your Linux firewall/router in
> transparent proxying mode requires that squid has some code
> in it that knows how to talk to Linux in order to discover the
> original destination

As with most client-side transparent proxies, it uses the Host: header.

> and change the outgoing connection
> such that the original address is used again.  Doing that
> requires specific knowledge of the API that netfilter/iptables
> uses.

	Jan
-- 

Re: Developing a user space library for filtering

[Patrick Schaaf]

> > As an example, using squid on your Linux firewall/router in
> > transparent proxying mode requires that squid has some code
> > in it that knows how to talk to Linux in order to discover the
> > original destination
> 
> As with most client-side transparent proxies, it uses the Host: header.

And when the Host: header is missing, it uses the SO_GET_ORIGINAL_DESTINATION
getsockopt call, a speciality of netfilter, as described by the $OP above.

Anyway, regarding the original request, I don't think it is sensible to
expect from netfilter developers to invent such a library, especially
when the scope is desired to be abstracting from netfilter. As an analogy,
it was neither Microsoft nor Unix vendors that invented the Apache
Runtime Library, it was the Apache developers. Bringing back that
analogy, it would be a task for the developers of a vastly successful
high level firewall application running on all kinds of basic firewalls.

Personally, I think that it would be bound to fail anyway, because
different basic firewall structures are very different in what and
how they operate. But that's just the fast opinion of somebody who
has neither need nor vision for userlevel firewall applications
and systems other than Linux/netfilter.

best regards
  Patrick

Re: Developing a user space library for filtering

[Darren.Reed]

Patrick Schaaf wrote:

>...
>Anyway, regarding the original request, I don't think it is sensible to
>expect from netfilter developers to invent such a library, especially
>when the scope is desired to be abstracting from netfilter.
>

At this point in time, I was looking for people who might be interested
in helping design such an API.  In the end, what I'm hoping for is to
have a common API delivered as part of OpenSolaris as well as both
FreeBSD and NetBSD.  Given that it's still being drafted, I'm opening
the door and asking if there is anyone from Linux who's interested in
participating.  I should point out that I'm not interested in requesting
anyone here write code that isn't [L]GPL'd.

...

> Bringing back that
>analogy, it would be a task for the developers of a vastly successful
>high level firewall application running on all kinds of basic firewalls.
>  
>

Why should they have to do that?  That requires:
1) they understand the API of every firewall they support
2) to track the API changes of every firewall they support
3) recompile/redeliver their software every time that API changes

None of those 3 options are what I would call palatable.

Imagine if everytime a new glibc was delivered you needed to
recompile all of your programs, from ls all the way through to the
X server, or...

>Personally, I think that it would be bound to fail anyway, because
>different basic firewall structures are very different in what and
>how they operate. But that's just the fast opinion of somebody who
>has neither need nor vision for userlevel firewall applications
>and systems other than Linux/netfilter.
>  
>

The point of this is to understand that the low level/basic
structures of firewall software are quite different but the
"end goal" is the same.

For example, all firewalls let you say "block traffic between
X & Y", it is just how that is done which is different.  The
idea is to capture the high level goals (c.f. "block ..")
into an API that applications can use.

Darren

Re: Developing a user space library for filtering

[Phil Dibowitz]

[-- Attachment #1: Type: text/plain, Size: 2496 bytes --]

On Tue, May 22, 2007 at 01:50:20PM -0700, Darren.Reed@Sun.COM wrote:
> Patrick Schaaf wrote:
> 
> >...
> >Anyway, regarding the original request, I don't think it is sensible to
> >expect from netfilter developers to invent such a library, especially
> >when the scope is desired to be abstracting from netfilter.
> >
> 
> At this point in time, I was looking for people who might be interested
> in helping design such an API.  In the end, what I'm hoping for is to
> have a common API delivered as part of OpenSolaris as well as both
> FreeBSD and NetBSD.  Given that it's still being drafted, I'm opening
> the door and asking if there is anyone from Linux who's interested in
> participating.  I should point out that I'm not interested in requesting
> anyone here write code that isn't [L]GPL'd.

Actually the netfilter folks wrote an entire infrastructure for just this
purpose.

netfilter is a generic infrastructure for firewall software with a defined
kernel-user API and they're now writing many libraries on top of that.

My software, iptstate, uses libnetfilter-conntrack, which is built upon the
netfilter framework.

All this is not to be confused with iptables, which is simply an implmentation
of netfilter coincidentally written by the same people who write the netfilter
framework.

Or so I understand it.

> None of those 3 options are what I would call palatable.
> 
> Imagine if everytime a new glibc was delivered you needed to
> recompile all of your programs, from ls all the way through to the
> X server, or...

Darren, you're correct, this is definitely needed. If IPF and IPtables and
everyone else all used a common core kernel-userspace API, with a standard
library on top of it, that would be awesome.

Netfilter brings a lot of of this to the table, but the people involved in
writing the specs mostly worked on ipchains, and iptables, so they may have
made linux-specific assumptions without realizing it - but it was very much
purposed to be OS-agnostic.

-- 
Phil Dibowitz                             phil@ipom.com
Open Source software and tech docs        Insanity Palace of Metallica
http://www.phildev.net/                   http://www.ipom.com/

"Never write it in C if you can do it in 'awk';
 Never do it in 'awk' if 'sed' can handle it;
 Never use 'sed' when 'tr' can do the job;
 Never invoke 'tr' when 'cat' is sufficient;
 Avoid using 'cat' whenever possible" -- Taylor's Laws of Programming


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Developing a user space library for filtering

[Henrik Nordstrom]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

tis 2007-05-22 klockan 14:14 -0700 skrev Phil Dibowitz:

> Darren, you're correct, this is definitely needed. If IPF and IPtables and
> everyone else all used a common core kernel-userspace API, with a standard
> library on top of it, that would be awesome.

As things is today I am not sure how this would work in a general sense
for what is proposed. The way you structure rules is quite different in
the different firewall implementations.

But sure, for things like connection tracking and related events it's
surely doable. Also, writing a generic firewall ruleset "compiler" which
can translate to ipf, iptables and a few others is doable, but the
actual installed ruleset will need to be somewhat different in both
structure and syntax in the different implementations, not only syntax.

Having an API which says things like "Add a rule to accept this kind of
traffic" is only possible cross the different firewall implementations
if there also is a defined firewall ruleset structure requirement
defining suitable places in the ruleset where this API can make it's
modifications.

Regards
Henrik

[-- Attachment #2: Detta är en digitalt signerad meddelandedel --]
[-- Type: application/pgp-signature, Size: 307 bytes --]

Re: Developing a user space library for filtering

[Darren.Reed]

Henrik Nordstrom wrote:

>tis 2007-05-22 klockan 14:14 -0700 skrev Phil Dibowitz:
>
>  
>
>>Darren, you're correct, this is definitely needed. If IPF and IPtables and
>>everyone else all used a common core kernel-userspace API, with a standard
>>library on top of it, that would be awesome.
>>    
>>
>
>As things is today I am not sure how this would work in a general sense
>for what is proposed. The way you structure rules is quite different in
>the different firewall implementations.
>
>But sure, for things like connection tracking and related events it's
>surely doable. Also, writing a generic firewall ruleset "compiler" which
>can translate to ipf, iptables and a few others is doable, but the
>actual installed ruleset will need to be somewhat different in both
>structure and syntax in the different implementations, not only syntax.
>
>Having an API which says things like "Add a rule to accept this kind of
>traffic" is only possible cross the different firewall implementations
>if there also is a defined firewall ruleset structure requirement
>defining suitable places in the ruleset where this API can make it's
>modifications.
>  
>

Which means that coming up with a design that works won't
necessarily be a slam-dunk.

Darren

Re: Developing a user space library for filtering

[Philip Craig]

Darren.Reed@Sun.COM wrote:
> Henrik Nordstrom wrote:
>> Having an API which says things like "Add a rule to accept this kind of
>> traffic" is only possible cross the different firewall implementations
>> if there also is a defined firewall ruleset structure requirement
>> defining suitable places in the ruleset where this API can make it's
>> modifications.
>>  
>>
> 
> Which means that coming up with a design that works won't
> necessarily be a slam-dunk.

It means the API must an interface to a firewall policy, not to a kernel
mechanism.  For example, the API should enable you to tell shorewall
that you want to accept something, rather than directly generating
an iptables rule.  (Although a simple implementation for the case where
you have no firewall policy may directly generate an iptables rule.)

Re: Developing a user space library for filtering

[Henrik Nordstrom]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

tis 2007-05-22 klockan 16:55 -0700 skrev Darren.Reed@Sun.COM:

> Which means that coming up with a design that works won't
> necessarily be a slam-dunk.

More that you are aiming this API at the wrong level. This API needs to
be to the firewall (Smoothwall, Astaro, MARA, Cisco, Netgear,
Checkpoint, etc etc), not the firewall packet engine.

plain iptables might be one target, but only as a reference
implementation under well defined conditions.

To get going you need to first define a couple of target applications.
Trying to make this discussion without some well defined targets will
not gain good results.

Regards
Henrik

[-- Attachment #2: Detta är en digitalt signerad meddelandedel --]
[-- Type: application/pgp-signature, Size: 307 bytes --]

Re: Developing a user space library for filtering

[Allen Francom]

I worked a little bit with "Hogwash".
http://hogwash.sourceforge.net/docs/overview.html

This was a mangling of IDS/Snort into layer 2 for real-time filtering.

With a little help from my friends there I got an IPTables "target"
so that IPTables could direct traffic into the hogwash/snort.

I like it that you said "libc".

It would be nice to intercept traffic after stream reassembly which was 
always a problem.

In that case what you're asking makes maybe more sense to focus on "libc" 
itself and having an "intercept" there.

That way you could for example filter stream-reassembled data coming into 
a web server, or not, depending on the "rules engine" deciding to catch 
the traffic or not.

I always thought that would be the place to be, in the call to get data, 
but hookable, like a library with a hook that checks to see if it should 
be "scrubbing" incoming traffic on a particular port.

"Slick" is the word.

Word.

;)

-AEF


On Mon, 21 May 2007, Darren.Reed@Sun.COM wrote:

> Carl-Daniel Hailfinger wrote:
>
>> Hi Darren,
>> 
>> On 22.05.2007 00:27, Darren.Reed@Sun.COM wrote:
>> 
>>> One of the core problems I see as people want to more and
>>> more with firewall/NAT technology is integrate using it into
>>> their application (whatever that may be.)  As time goes by,
>>> this problem is becoming more and more acute and perhaps
>>> is doing us (those who develop said technologies) a disservice
>>> by making the "barrier to entry" too high.
>>> 
>> 
>> Sorry if I'm being dense. Do you want to target firewall frontends
>> or applications which have the desire to punch holes into the
>> firewall?
>> 
>
> Neither.  Applications that sit on top of firewall software.
>
> As an example, using squid on your Linux firewall/router in
> transparent proxying mode requires that squid has some code
> in it that knows how to talk to Linux in order to discover the
> original destination and change the outgoing connection
> such that the original address is used again.  Doing that
> requires specific knowledge of the API that netfilter/iptables
> uses.
>
> Another example might be IDS software running on your Linux
> firewall/router.  If that detects an attack, it should be able to
> talk to netfilter/iptables and do "something" to mitigate it.
>
> In both cases I'd like to see something developed such that
> the 3rd party applications don't need to know what NAT or
> firewall technology is being used.
>
> Darren
>
>
>