Re: POM Xtables???

Re: POM Xtables???

[Jozsef Kadlecsik]

Hi,

On Mon, 30 Jun 2008, Patrick McHardy wrote:

> History has repeatedly shown that out of tree patches are buggy
> and cause more problems than they solve, which is why there
> is no interest from the netfilter team in maintaining external
> patches (with the one exception of ipset, which is not considered
> ready for upstream yet by Jozsef, its author).

The reason why I haven't been considering ipset ready for kernel inclusion 
is actually simple: the current ipset framework is not flexible enough due 
to the rigid kernel<->userspace communication method and it does not 
support IPv6 at all. As I had expected to be ready with the next 
generation of ipset (codename nfset) more soon, I have been refraining to 
submit ipset for kernel inclusion. nfset is finally getting shape, by the 
end of summer it'll be released. So the reason not to submit ipset is 
stronger than ever.

In order to make life easier, as I apply the patches sent by Swen, 
there'll be a new, "standalone" version of ipset, i.e. it'll be possible 
to use it without patch-o-matic-ng.
 
Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: POM Xtables???

[Jan Engelhardt]

On Monday 2008-06-30 23:11, Jozsef Kadlecsik wrote:
>
>In order to make life easier, as I apply the patches sent by Swen, 
>there'll be a new, "standalone" version of ipset, i.e. it'll be possible 
>to use it without patch-o-matic-ng.

I have been tracking ipset a bit lately, it already seems
standing alone. Would you be willing to make it available
through Xtables-addons?
A test head for inspection can be found at
	git://dev.medozas.de/xtables-addons ipset

Re: POM Xtables???

[Jozsef Kadlecsik]

On Mon, 30 Jun 2008, Jan Engelhardt wrote:

> On Monday 2008-06-30 23:11, Jozsef Kadlecsik wrote:
> >
> >In order to make life easier, as I apply the patches sent by Swen, 
> >there'll be a new, "standalone" version of ipset, i.e. it'll be possible 
> >to use it without patch-o-matic-ng.
> 
> I have been tracking ipset a bit lately, it already seems
> standing alone. Would you be willing to make it available
> through Xtables-addons?
> A test head for inspection can be found at
> 	git://dev.medozas.de/xtables-addons ipset

The iptables match and target for ipset are in the iptables source tree, 
so that part is totally independent from pom-ng and supported by iptables, 
out of the box. As ipset is IPv4-only, there's no way to create an xtables
variant of the match and the target :-).

The source of the 'ipset' binary, which is totally independent from 
iptables or xtables, is currently in svn, but we can move it to git 
anytime. I don't really see what would be the benefit in adding ipset to 
xtables-addons, sorry. But thank you the offer!

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: POM Xtables???

[Jan Engelhardt]

On Tuesday 2008-07-01 12:00, Jozsef Kadlecsik wrote:
>On Mon, 30 Jun 2008, Jan Engelhardt wrote:
>> On Monday 2008-06-30 23:11, Jozsef Kadlecsik wrote:
>> >
>> >In order to make life easier, as I apply the patches sent by Swen, 
>> >there'll be a new, "standalone" version of ipset, i.e. it'll be possible 
>> >to use it without patch-o-matic-ng.
>> 
>> I have been tracking ipset a bit lately, it already seems
>> standing alone. Would you be willing to make it available
>> through Xtables-addons?
>> A test head for inspection can be found at
>> 	git://dev.medozas.de/xtables-addons ipset
>
>The iptables match and target for ipset are in the iptables source tree, 

Well that won't impact is positively nor negatively. It has always
been that way, and whether people get libipt_{SET,set} through
iptables or the ipset tarball - it would not matter.

>so that part is totally independent from pom-ng and supported by iptables, 
>out of the box. As ipset is IPv4-only, there's no way to create an xtables
>variant of the match and the target :-).

Just because something is limited to IPv4 does not mean it is
not 'x' ('x' as in xtables). It is all just names, and getting it
under one unified hut is a good thing. In fact, onnnnnnnnce (;-)
I get around to submit the next patchsets, we will have xt modules
that have neither IPv4 nor IPv6 components.

>The source of the 'ipset' binary, which is totally independent from 
>iptables or xtables, is currently in svn, but we can move it to git 
>anytime. I don't really see what would be the benefit in adding ipset to 
>xtables-addons, sorry. But thank you the offer!

Benefits?
- removing 73% of all #ifdef kludges from *.c (the kernel files)
- less tarballs for users to build
- autoconf for the binary :-P

Re: POM Xtables???

[Patrick McHardy]

Added netfilter-devel to CC.

Jan Engelhardt wrote:
> On Tuesday 2008-07-01 13:43, Patrick McHardy wrote:
>>> You only added the nfmark, and it is printed unconditionally at that,
>>> the latter of which I do not see as thrilling. Seems like it is time
>>> for a new revision? Also I think the two modules should be unified
>>> first.
>> It doesn't need a new revision, it doesn't affect the userspace
>> API in any way. As for the argument of parsers that might not
>> handle this: any parser needs to expect new things to be added
>> at the end of the line, otherwise its giving us no possibility
>> of ever extending the output, which is not reasonable.
>>
> That is why LOGMARK has been LOGMARK so far, where the (undocumented)
> requirement is that parsers have to split at whitespace and may not
> assume any particular order of fields. This allows to selectively log
> things by means of flags passed in from userspace and thus sparing
> syslog from being spammed with unwanted fields.

Spammed with unwanted fields in a huge exaggeration. And nobody
sane is using LOG for anything but debugging anyway due to the
unreliable nature of the ringbuffer and the huge slowdown this
might cause when using serial console.

> So the only way is to introduce a LOG2 target?

Only way for what? As I stated, you're free to add new fields
at the end.

Re: POM Xtables???

[Patrick McHardy]

Please don't trim CC lists.

Grant Taylor wrote:
> On 07/01/08 06:43, Patrick McHardy wrote:
>> As for the argument of parsers that might not handle this: any parser 
>> needs to expect new things to be added at the end of the line...
>
> *LOL*
>
> Now you are expecting people to do what they should verses what is easy.
>
> Unfortunately, what is considered best practice (at the time) is not 
> followed often enough.


We had multiple format extensions over time, you would hope
that people learn from their mistakes. Using scanf or a
lexer+scanner its probably impossible to even make this mistake,
assuming some perl parser someone would have to specifically
match on the (completely useless for parsing purposes) end of
line.

In any case, its unreasonable to expect us to never *extend*
(not change) the output to accomodate buggy parsers. This is
by the way the same way that is often used to extend binary
structures, even though someone stupid might use exact size
checks.

Re: POM Xtables???

[Patrick McHardy]

Again, please don't trim CC lists.

Grant Taylor wrote:
> On 07/01/08 09:10, Patrick McHardy wrote:
>> In any case, its unreasonable to expect us to never *extend* (not 
>> change) the output to accomodate buggy parsers. This is by the way 
>> the same way that is often used to extend binary structures, even 
>> though someone stupid might use exact size checks.
>
> *nod*
>
> Agreed.
>
> However as I sit here and think about it, it may be worth adding a new 
> field *as early as possible* (read closes to the start of line) in the 
> field list that indicates a version, which can be used to determine 
> the fields and their position there in.  This would make it very easy 
> for people to write strict parsers down the road.  A simple three 
> character hex field (4 bytes including the leading space) would allow 
> for 4k of strict layouts.  (Even more 0-9 and a-z or additionally 
> A-Z.) Just a thought.

I don't it would be very useful at this time since we already made
those changes over the time of multiple years, so basically the
damage (if any) is already done.

And we have ULOG and nfnetlink_log that should be used for anything
serious for the reasons I stated earlier (more reliable, doesn't block
when using serial consoles, ...).

Re: POM Xtables???

[Patrick McHardy]

Pablo Neira Ayuso wrote:
> James King wrote:
>> On Wed, Jul 23, 2008 at 4:21 PM, Patrick McHardy wrote:
>>
>>> Just send it to netfilter-devel. If its the thing with lots
>>> of hard-coded binary matches full of magic values I'm not
>>> interested :) I'd be more interested in a discussion what
>>> would be necessary to represent all those matches through
>>> the FSM textsearch match or something similar.
 >>
>> ipp2p is the one with hard coded magic values.
>>
>> What are your feelings on the kernel version of l7filter (regex
>> patterns loaded from the filesystem)?  Currently it requires a patch
>> to add a structure to nf_conn, but I've been meaning to rewrite it to
>> use ct_extend so that it could at least be included into xtables-addon
>> and used with a stock kernel, although if there's interest in having
>> it merged into mainline I'd be willing to focus on that.

That is definitely a much better way than to hardcode the
matches. I think there is some interest in having this in
the kernel, yes.

>> One thing
>> I'm not sure of is whether the license used by the Henry Spencer regex
>> library it depends on is acceptable by kernel standards (or whether
>> it's permissive enough to relicense under GPL, as IANAL).

I know of the regexp.old.zip library, which IIRC used a GPL
compatible license (and a non-POSIX conform interface).

> If we want to do this in-kernel I think that it's better if it must use
> the textsearch infrastructure. Probably it would require some patches to
> extend the existing infrastructure.

I'd also prefer that over adding a regex library to the kernel.
I think one of the bigger problems is that there are dependencies
in the match that can't be easily expressed, like "byte 4 has
skb->len - 10". At least ipp2p does something like that. But
maybe thats not necessary, since l7filter already uses regexes
there's apparently a different method for doing this.

It would be useful if someone could post an excerpt from the
l7filter expressions or simply the entire patch.

> The other choise is userspace by means NFQUEUE. If we use some
> heuristics, we may try to classify the traffic by means of the initial
> data packets and then mark the connection. Thus, the number of packets
> that go to userspace would be small and the classification logic is
> implemented in userspace using whatever regex
> engine/aho-corasick/bit-wise/boyer-moore/bayes whatsoever...

Yes, thats another possibilty (and a lot of people are doing
that), but it would still be nice to have a mechanism for
doing this in the kernel.

Re: POM Xtables???

[James King]

(Sorry for the spam Patrick, I just realized that I didn't reply-all
the first time around)

On Thu, Jul 24, 2008 at 2:43 AM, Patrick McHardy <kaber@trash.net> wrote:
>> James King wrote:
>>> One thing
>>> I'm not sure of is whether the license used by the Henry Spencer regex
>>> library it depends on is acceptable by kernel standards (or whether
>>> it's permissive enough to relicense under GPL, as IANAL).
>
> I know of the regexp.old.zip library, which IIRC used a GPL
> compatible license (and a non-POSIX conform interface).
>
>> If we want to do this in-kernel I think that it's better if it must use
>> the textsearch infrastructure. Probably it would require some patches to
>> extend the existing infrastructure.
>
> I'd also prefer that over adding a regex library to the kernel.
> I think one of the bigger problems is that there are dependencies
> in the match that can't be easily expressed, like "byte 4 has
> skb->len - 10". At least ipp2p does something like that. But
> maybe thats not necessary, since l7filter already uses regexes
> there's apparently a different method for doing this.
>
> It would be useful if someone could post an excerpt from the
> l7filter expressions or simply the entire patch.

ipp2p and l7filter both use different strategies for DPI
classification, each having their own pros and cons.

ipp2p has the potential to classify traffic faster since it uses magic
values/offsets before moving on to more detailed searches.  As you
noted above, the match can also be a bit "deeper" since it can express
more complex relationships.  It's also better for memory usage, since
it doesn't store packet data.  The downside is poor configurability
and maintainability.

l7filter adds a buffer (default size of 2048 bytes, kmalloc'd on the
first packet of a flow, and freed after classification is complete) to
the conntrack and appends inbound and outbound packets into this
buffer (in the order received), up to a configurable number of packets
before giving up on classification.  It runs each configured regex
rule against this buffer in sequence, which can get very expensive
especially when you have a lot of matches or your patterns include a
lot of branches, since a large amount of search work is potentially
being duplicated by each rule.  However, being able to setup these
matches at runtime is what makes l7filter much more convenient to use
overall.

After reviewing the l7filter code a bit, I've found that it does some
nasty things at present like holding a BH spinlock over the entire
match function, and allocating a per-conntrack character buffer
containing the string name of the pattern that we've matched (eg.
http-audio, bittorrent, battlefield2, etc) for the entire life of the
conntrack.  Also, the work necessary to convert the backend to use
textsearch and still maintain compatibility with the existing patterns
seems a bit prohibitive at the moment.

I think the best thing to do for now is to move it into xtables-addons
(providing Jan has no objections) for further review and cleanup.
I've had a bit of time to work on this during my vacation, but I was
wondering if I could get some clarification on a couple things:
-Under what circumstances does it become mandatory to implement the
ct_extend move callback?
-The description from Krzystof's recent accounting patch (which btw
doesn't appear to have been merged during the last window after all)
notes:

"...newly created connections get additional acct extend.  Old
connections are not changed as it is not possible to add a ct_extend
area to confirmed conntrack"

His patch adds a call to nf_conntrack_acct_init() inside of
conntrack_init() to add the private area.  For an out of tree module,
obviously this isn't a good solution (I'd like to use this module
against a stock kernel if possible).  I suppose I could include
something like this near the beginning of the match function:

layer7 = nf_ct_ext_find(ct, NF_CT_EXT_LAYER7);
if (!layer7 && !nf_ct_is_confirmed(ct))
	ret = nf_ct_ext_add(ct, NF_CT_EXT_LAYER7, GFP_ATOMIC);

...but this seems kludgy.  Is there a better way of doing this?  If
not, maybe we could add an init callback to ct_extend (and basically
clone nf_ct_ext_destroy() and __nf_ct_ext_destroy()), and call it from
conntrack_init(), so any number of extensions can be added without
having to patch conntrack_init() each time.

-In the same thread of avoiding a kernel recompile, it would be great
if there was a way to add a new extension type on the fly, or at least
had one enumeration reserved for use by out of tree modules (eg.
NF_CT_EXT_RESERVED).  Any thoughts here?

Re: POM Xtables???

[Brent Clark]

James King wrote:
> ipp2p and l7filter both use different strategies for DPI
> classification, each having their own pros and cons.
You know most people, groups etc look for the next best thing. Take a 
look at Firefox and apple ( *pod), they continuously announcing whats 
hip and new, what they doing etc, and looking at ways to keep a captive 
audience.

My question is what netfilters next best thing?

Having used and using Xtables, I thinking it FSCKING brilliant (excuse 
slander, hope I did not offend, but there was not other way to explain). 
I dont have to struggle and my turn around time is minutes.
I continuously thank Jan for the work his doing.

I suggest forget POM. its old and the process is slow and laborious (and 
thats hoping you can get it compiled in the kernel).

Getting back to iptables.

Its great to see others stepping forward and wanting to implement a 
Layer 7 filtering, and I say go for it and work on it, but in the mean 
time and to the netfilter team, my question is, how long will that take 
till its able to get off the ground to too hope that it gets accepted by 
the teams (netfilter and kernel).

To be constructive, and looking for a solid way forward (even if 
interim), would it not be better to implement l7 in xtables or better 
iptables.
Yes the L7 code may suck now or incorrectly thoughtout, but getting it 
working will help people. People understand that its not perfect or bug 
less, the fact they have option and it being worked on, helps.

Im of the opinion that Netfilter really needs to look and think out the 
box and realize ppl want *now*, troubleless (less not free), shiny and 
new (this goes hand in hand with promoting, marketing etc).
Google for pf vs iptables, and you will find a plethora of links 
promoting either / or. Netfilter needs that "shiny" that will set it 
apart from the rest that will and have the bells and whistles.

My aim it to not offend anyone, but let the powers that be know, that 
there is a demand for more. Ill probably get flamed, but I hope this 
email gets taken in the light of constructive criticism and for the 
greater of the user community that like quick install, all in one solution.

Kind Regards
Brent Clark

P.s. James, I hope you get your solution off the ground  and working.