[RFC PATCH IPTABLES 0/12]: matches/targets unification

[RFC PATCH IPTABLES 0/12]: matches/targets unification

[Yasuyuki KOZAKAI]

Hi, all,

This patch series enables to unify matches/targets of ip[6]tables.

I've changed the only parts to unify register_{match,target} and
moves duplicated functions to a file. Unfortunately, I had to change some
arguments of the traditional APIs for them, but they are to avoid
warning on compilation.

It might not be good time to submit this because of recent patch rush
and it's just before next release of iptables, so it's just RFC this time.

-- Yasuyuki Kozakai

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Patrick McHardy]

Yasuyuki KOZAKAI wrote:
> This patch series enables to unify matches/targets of ip[6]tables.
> 
> I've changed the only parts to unify register_{match,target} and
> moves duplicated functions to a file. Unfortunately, I had to change some
> arguments of the traditional APIs for them, but they are to avoid
> warning on compilation.
> 
> It might not be good time to submit this because of recent patch rush
> and it's just before next release of iptables, so it's just RFC this time.

Cool :) From a first look they all look good. I'll (try to) do a new
release as soon as I'm done with the nf_nat stuff so we can add this
quickly.

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Yasuyuki KOZAKAI]

[-- Attachment #1: Type: Text/Plain, Size: 800 bytes --]

From: Yasuyuki KOZAKAI <yasuyuki.kozakai@toshiba.co.jp>
Date: Thu, 30 Nov 2006 18:43:41 +0900 (JST)

> 
> Hi, all,
> 
> This patch series enables to unify matches/targets of ip[6]tables.
> 
> I've changed the only parts to unify register_{match,target} and
> moves duplicated functions to a file. Unfortunately, I had to change some
> arguments of the traditional APIs for them, but they are to avoid
> warning on compilation.
> 
> It might not be good time to submit this because of recent patch rush
> and it's just before next release of iptables, so it's just RFC this time.

The 8th and 9th patches seemed to be dropped due to too big message.
I re-send archived them.

[8/12]: replaces ip[6]t_entry_* with xt_entry_* in matches/targets
[9/12]: fixed warning on compilation

-- Yasuyuki Kozakai

[-- Attachment #2: xtables.tar.bz2 --]
[-- Type: Application/Octet-Stream, Size: 17445 bytes --]

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Patrick McHardy]

Yasuyuki KOZAKAI wrote:
> Hi, all,
> 
> This patch series enables to unify matches/targets of ip[6]tables.
> 
> I've changed the only parts to unify register_{match,target} and
> moves duplicated functions to a file. Unfortunately, I had to change some
> arguments of the traditional APIs for them, but they are to avoid
> warning on compilation.
> 
> It might not be good time to submit this because of recent patch rush
> and it's just before next release of iptables, so it's just RFC this time.

I just released 1.3.7, so I think we could add this now.

Since the next version will be more than just a bugfix release,
one thing I wanted to bring up for some time ..

We currently have lots of extensions in iptables for non-mainline
matches and targets, some obsolete, some already decided against
merging and some that might be merged in the future. For the
extensions belonging to a feature that won't be merged, the same
argument applies as to POM, they increase the maintenance burden
and should really be maintained along with the kernel patch (and
pom already supports patching iptables).

More problematic are extensions for stuff that might get merged.
In many cases so far they were included by distributors if
compiled by default, but when merging a feature we often noticed
that the ABI had problems like 32/64 bit issues and needed to be
changed before merging, which effectively broke compatibility.
I think it would be preferable to have people wait until their
distribution releases a new version to use a new feature than
have strange and unexplained errors.

So, what I'm proposing is to remove all extensions that are
neither part of 2.4 or 2.6 from iptables, pushing those for
externally maintained patches to the patch maintainers, those
for patches maintained in pom to pom, and delete the rest.

Comments?

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Yasuyuki KOZAKAI]

From: Patrick McHardy <kaber@trash.net>
Date: Wed, 06 Dec 2006 11:25:58 +0100

> I just released 1.3.7, so I think we could add this now.
> 
> Since the next version will be more than just a bugfix release,
> one thing I wanted to bring up for some time ..

I see, I'll keep rebase my tree.

> We currently have lots of extensions in iptables for non-mainline
> matches and targets, some obsolete, some already decided against
> merging and some that might be merged in the future. For the
> extensions belonging to a feature that won't be merged, the same
> argument applies as to POM, they increase the maintenance burden
> and should really be maintained along with the kernel patch (and
> pom already supports patching iptables).

Indeed, actually I had no way to do build test for some extensions with my
unification patches due to no kernel patches, then I pretended
I didn't see :)

> More problematic are extensions for stuff that might get merged.
> In many cases so far they were included by distributors if
> compiled by default, but when merging a feature we often noticed
> that the ABI had problems like 32/64 bit issues and needed to be
> changed before merging, which effectively broke compatibility.
>
> I think it would be preferable to have people wait until their
> distribution releases a new version to use a new feature than
> have strange and unexplained errors.
>
> So, what I'm proposing is to remove all extensions that are
> neither part of 2.4 or 2.6 from iptables, pushing those for
> externally maintained patches to the patch maintainers, those
> for patches maintained in pom to pom, and delete the rest.

I agree.

> Comments?

'make' should test whether kernel supports extensions or not.
Do distributors manually change -test[6] scripts and build them
these days ? Or you mean that distributors apply kernel patch to their
kernel as well ? In later case, that means that they definitely want to
support features. Then even if we moves them from source tree of iptables,
I think they will apply them and same things will happen.

But anyway, I think it's good idea to move extensions which is not
supported by mainline kernel.

-- Yasuyuki Kozakai

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Patrick McHardy]

Yasuyuki KOZAKAI wrote:
> From: Patrick McHardy <kaber@trash.net>
> Date: Wed, 06 Dec 2006 11:25:58 +0100
> 
>>So, what I'm proposing is to remove all extensions that are
>>neither part of 2.4 or 2.6 from iptables, pushing those for
>>externally maintained patches to the patch maintainers, those
>>for patches maintained in pom to pom, and delete the rest.
> 
> 
> I agree.
> 
> 
>>Comments?
> 
> 
> 'make' should test whether kernel supports extensions or not.

Yes, that would be an alternative (currently some do, some don't).
But the testing and maintenance problems would still persist.

> Do distributors manually change -test[6] scripts and build them
> these days ? Or you mean that distributors apply kernel patch to their
> kernel as well ? In later case, that means that they definitely want to
> support features. Then even if we moves them from source tree of iptables,
> I think they will apply them and same things will happen.

The later. In that case its out of our control of course, but currently
we are encouraging it by building unsupported extensions by default.

> But anyway, I think it's good idea to move extensions which is not
> supported by mainline kernel.

I'm going to go over them in the next days and make a list of those
to be deleted/pushed to patch maintainers.

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 1445 bytes --]

On Wed, 2006-12-06 at 11:25 +0100, Patrick McHardy wrote:

> We currently have lots of extensions in iptables for non-mainline
> matches and targets, some obsolete, some already decided against
> merging and some that might be merged in the future. For the
> extensions belonging to a feature that won't be merged, the same
> argument applies as to POM, they increase the maintenance burden
> and should really be maintained along with the kernel patch (and
> pom already supports patching iptables).
> 
> More problematic are extensions for stuff that might get merged.
> In many cases so far they were included by distributors if
> compiled by default, but when merging a feature we often noticed
> that the ABI had problems like 32/64 bit issues and needed to be
> changed before merging, which effectively broke compatibility.
> I think it would be preferable to have people wait until their
> distribution releases a new version to use a new feature than
> have strange and unexplained errors.
> 
> So, what I'm proposing is to remove all extensions that are
> neither part of 2.4 or 2.6 from iptables, pushing those for
> externally maintained patches to the patch maintainers, those
> for patches maintained in pom to pom, and delete the rest.
> 
> Comments?

Yes I think this is what we should do. Iirc this was loosely discussed
during a workshop, it's time to actually do it now. Go for it!

-- 
/Martin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [RFC PATCH IPTABLES 0/12]: matches/targets unification

[Patrick McHardy]

Martin Josefsson wrote:
> On Wed, 2006-12-06 at 11:25 +0100, Patrick McHardy wrote:
>  
>>[...]
>
> Yes I think this is what we should do. Iirc this was loosely discussed
> during a workshop, it's time to actually do it now. Go for it!

I can't really remember the discussion, but I certainly will :)

Time module included in the default Fedora

[Fred Trotter]

Hi, I tried this request on the devel list but I guess from the
silence it was off-topic.

        I was told to write here about getting the time module
included "upstream". I hope that someone here might be able  to
educate me on the process for getting these things done. I understand
that there are many iptables modules and that some are included by
default while others are not.
        The problem is that the iptables userspace project documents
the time modules as though it is included, but at least in Fedora it
is not by default. They hold that they will not include it until it is
included "upstream", they indicated that you would be able to tell me
how to get it included "upstream".
          I am confused as to why the man pages reference
functionality that seems to be not typically turned on by default. Why
is the man page "upstream" but not the module? I do not say this to
accuse, I want to understand so that I can help fix this. It seems to
me that either the man pages should not mention the module or the
module should typically be included.
         Does this problem impact other documented modules that are
not actually included "upstream"?

Here is the link to original discussion.
  https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233475

Regards,
-FT

-- 
Fred Trotter
http://www.fredtrotter.com

Re: Time module included in the default Fedora

[Jan Engelhardt]

Ccing Patrick and adding Reference-Hdr to [2] - questions to old 
questions :)

On Apr 11 2007 09:52, Fred Trotter wrote:
>
>       I was told to write here about getting the time module
> included "upstream". I hope that someone here might be able  to
> educate me on the process for getting these things done. I understand
> that there are many iptables modules and that some are included by
> default while others are not.

There are only a few cases why a maintainer decides against:
  (1) it's a hack (best example: ipt_ROUTE)
  (2) it's unmaintained (now that's strange)
  (3) violates coding style (happens often)
  (4) not enough interest on the users' sides (gotta change that)

Ask nicely (see [1],[2]), and maybe things get rolling (or not).

[1] http://archives.free.net.ph/message/20061206.204842.c1c8628a.en.html
[2] http://archives.free.net.ph/message/20061207.030828.0d81b372.en.html

Though that leaves me puzzled why connlimit has not gone in yet
(it all simplifies maintenance so much IMO). BTW, how about it?

>       The problem is that the iptables userspace project documents
> the time modules as though it is included, but at least in Fedora it
> is not by default. They hold that they will not include it until it is
> included "upstream", they indicated that you would be able to tell me
> how to get it included "upstream".

iptables is not the same as netfilter (= the kernel part). This is
perhaps the biggest mystery. Why include something in iptables when
it is not in the kernel.
But that's how things currently are:
 - source files are in iptables
 - some get compiled because they are listed in the
   "always-y"-style line in the Makefile, hence their manpage part
   will show up in `man iptables`, but you cannot effectively
   use them because the kernel module is missing.


Jan
-- 

Re: Time module included in the default Fedora

[Patrick McHardy]

Jan Engelhardt wrote:
> On Apr 11 2007 09:52, Fred Trotter wrote:
> 
>>      I was told to write here about getting the time module
>>included "upstream". I hope that someone here might be able  to
>>educate me on the process for getting these things done. I understand
>>that there are many iptables modules and that some are included by
>>default while others are not.


The question whether to merge the time module came up repeatedely
at netfilter workshops, but it was always decided against it so far,
mainly because it apparently can't deal with timezones and daylight
saving time. IIRC Harald had strong feelings about it, I personally
don't care much about this shortcoming as long as its documented.
I'm not even sure its correct since the kernel has sys_tz. So if
anyone finds out and submits a patch, I'll consider it.

> There are only a few cases why a maintainer decides against:
>   (1) it's a hack (best example: ipt_ROUTE)
>   (2) it's unmaintained (now that's strange)
>   (3) violates coding style (happens often)
>   (4) not enough interest on the users' sides (gotta change that)
> 
> Ask nicely (see [1],[2]), and maybe things get rolling (or not).
> 
> [1] http://archives.free.net.ph/message/20061206.204842.c1c8628a.en.html
> [2] http://archives.free.net.ph/message/20061207.030828.0d81b372.en.html
> 
> Though that leaves me puzzled why connlimit has not gone in yet
> (it all simplifies maintenance so much IMO). BTW, how about it?


As I stated multiple times, the reason why its not included is that
its horribly slow. But since I don't see any better way to do this
and I know quite a few people are using this, I would consider this
as well if someone sends me a patch, which has not happened so far.

>>      The problem is that the iptables userspace project documents
>>the time modules as though it is included, but at least in Fedora it
>>is not by default. They hold that they will not include it until it is
>>included "upstream", they indicated that you would be able to tell me
>>how to get it included "upstream".
> 
> 
> iptables is not the same as netfilter (= the kernel part). This is
> perhaps the biggest mystery. Why include something in iptables when
> it is not in the kernel.


It has been done in the past, but I deleted everything not included
upstream from iptables, mainly to avoid compatibilty breakage in
case we decide to merge something and the userspace API needs to
be changed (a lot of the old matches have 32/64 bit issues etc.)

Re: Time module included in the default Fedora

[Fred Trotter]

Thanks for the answers!

      A loong time ago, I might have had the time and energy to hack
this together myself. I think this is an important enough
functionality that barring a better solution I might look into doing
this myself. If I did decide to pursue contributing a patch, can
someone provide some links for me to make a stepwise process of what
needs to be done? if the answer is RTFM on this question thats fine,
but I would love to limit what I have to investigate to get this done.

       Alternatlively I am also willing to put my money where my mouth
is. If someone thinks this work is easy enough to be worth $300 I
would be glad to paypal that amount to someone in exchange for an
accepted patch. If someone can ligitimately explain why that is way
too little money, then I would be glad to start a bounty with that
amount on one of the several bounty sites and start yelling for more
money off-list. I would love some comments on what a "reasonable" cost
would be to get a good patch for this.

          Obviously the best patch would be something that takes
advantage of sys_tz (which I assume helps with daylight savings and
timezones) and that, if possible, would speed things up to be more
tolerable. But frankly, I have no idea what would be involved with
that, so my only real requirement would be "accepted". If Patrick
could give some hints about what would be required for acceptance that
would help. (Feel free to drop a link to a previous dicussion on this
subject...)

Thanks again,
Fred Trotter



On 4/11/07, Patrick McHardy <kaber@trash.net> wrote:
> Jan Engelhardt wrote:
> > On Apr 11 2007 09:52, Fred Trotter wrote:
> >
> >>      I was told to write here about getting the time module
> >>included "upstream". I hope that someone here might be able  to
> >>educate me on the process for getting these things done. I understand
> >>that there are many iptables modules and that some are included by
> >>default while others are not.
>
>
> The question whether to merge the time module came up repeatedely
> at netfilter workshops, but it was always decided against it so far,
> mainly because it apparently can't deal with timezones and daylight
> saving time. IIRC Harald had strong feelings about it, I personally
> don't care much about this shortcoming as long as its documented.
> I'm not even sure its correct since the kernel has sys_tz. So if
> anyone finds out and submits a patch, I'll consider it.
>
> > There are only a few cases why a maintainer decides against:
> >   (1) it's a hack (best example: ipt_ROUTE)
> >   (2) it's unmaintained (now that's strange)
> >   (3) violates coding style (happens often)
> >   (4) not enough interest on the users' sides (gotta change that)
> >
> > Ask nicely (see [1],[2]), and maybe things get rolling (or not).
> >
> > [1] http://archives.free.net.ph/message/20061206.204842.c1c8628a.en.html
> > [2] http://archives.free.net.ph/message/20061207.030828.0d81b372.en.html
> >
> > Though that leaves me puzzled why connlimit has not gone in yet
> > (it all simplifies maintenance so much IMO). BTW, how about it?
>
>
> As I stated multiple times, the reason why its not included is that
> its horribly slow. But since I don't see any better way to do this
> and I know quite a few people are using this, I would consider this
> as well if someone sends me a patch, which has not happened so far.
>
> >>      The problem is that the iptables userspace project documents
> >>the time modules as though it is included, but at least in Fedora it
> >>is not by default. They hold that they will not include it until it is
> >>included "upstream", they indicated that you would be able to tell me
> >>how to get it included "upstream".
> >
> >
> > iptables is not the same as netfilter (= the kernel part). This is
> > perhaps the biggest mystery. Why include something in iptables when
> > it is not in the kernel.
>
>
> It has been done in the past, but I deleted everything not included
> upstream from iptables, mainly to avoid compatibilty breakage in
> case we decide to merge something and the userspace API needs to
> be changed (a lot of the old matches have 32/64 bit issues etc.)
>


-- 
Fred Trotter
http://www.fredtrotter.com

Re: Time module included in the default Fedora

[Jan Engelhardt]

Hi,

On Apr 11 2007 18:33, Patrick McHardy wrote:
>
>The question whether to merge the time module came up repeatedely
>at netfilter workshops, but it was always decided against it so far,
>mainly because it apparently can't deal with timezones and daylight
>saving time.

Why, let iptables, or more precisely, ipt_time.c, handle timezones,
and pass an UTC value to the kernel -- that's what it is best at
dealing with.

>IIRC Harald had strong feelings about it, I personally
>don't care much about this shortcoming as long as its documented.
>I'm not even sure its correct since the kernel has sys_tz. So if
>anyone finds out and submits a patch, I'll consider it.
>
>> Though that leaves me puzzled why connlimit has not gone in yet
>> (it all simplifies maintenance so much IMO). BTW, how about it?
>
>As I stated multiple times, the reason why its not included is that
>its horribly slow. But since I don't see any better way to do this
>and I know quite a few people are using this, I would consider this
>as well if someone sends me a patch, which has not happened so far.

So it's just that I need to pull the pomng code and make a diff out
of it, is that all? (Plus any compilation and perhaps runtime fixes,
of course.)


Jan
-- 

Re: Time module included in the default Fedora

[Patrick McHardy]

Jan Engelhardt wrote:
> On Apr 11 2007 18:33, Patrick McHardy wrote:
> 
>>The question whether to merge the time module came up repeatedely
>>at netfilter workshops, but it was always decided against it so far,
>>mainly because it apparently can't deal with timezones and daylight
>>saving time.
> 
> 
> Why, let iptables, or more precisely, ipt_time.c, handle timezones,
> and pass an UTC value to the kernel -- that's what it is best at
> dealing with.


Than it wouldn't be able to deal with DST I guess. As I said, the
kernel already has knowledge about the timezone, so it should be
possible to do this properly quite easily, but I have no interest
in doing this myself.

>>IIRC Harald had strong feelings about it, I personally
>>don't care much about this shortcoming as long as its documented.
>>I'm not even sure its correct since the kernel has sys_tz. So if
>>anyone finds out and submits a patch, I'll consider it.
>>
>>
>>>Though that leaves me puzzled why connlimit has not gone in yet
>>>(it all simplifies maintenance so much IMO). BTW, how about it?
>>
>>As I stated multiple times, the reason why its not included is that
>>its horribly slow. But since I don't see any better way to do this
>>and I know quite a few people are using this, I would consider this
>>as well if someone sends me a patch, which has not happened so far.
> 
> 
> So it's just that I need to pull the pomng code and make a diff out
> of it, is that all? (Plus any compilation and perhaps runtime fixes,
> of course.)


From a quick look I'd say:

- move to x_tables
- remove ip_conntrack stuff
- clean up
- fix up for recent API changes and improvements
- fix 32/64 bit issues
- use proper list macros
- use kzalloc

And explain to me why it needs knowledge about TCP states. That should
be enough to get a discussion started.

Re: Time module included in the default Fedora

[Patrick McHardy]

Jan Engelhardt wrote:
> On Apr 11 2007 18:33, Patrick McHardy wrote:
> 
>>The question whether to merge the time module came up repeatedely
>>at netfilter workshops, but it was always decided against it so far,
>>mainly because it apparently can't deal with timezones and daylight
>>saving time.
> 
> 
> Why, let iptables, or more precisely, ipt_time.c, handle timezones,
> and pass an UTC value to the kernel -- that's what it is best at
> dealing with.


Than it wouldn't be able to deal with DST I guess. As I said, the
kernel already has knowledge about the timezone, so it should be
possible to do this properly quite easily, but I have no interest
in doing this myself.

>>IIRC Harald had strong feelings about it, I personally
>>don't care much about this shortcoming as long as its documented.
>>I'm not even sure its correct since the kernel has sys_tz. So if
>>anyone finds out and submits a patch, I'll consider it.
>>
>>
>>>Though that leaves me puzzled why connlimit has not gone in yet
>>>(it all simplifies maintenance so much IMO). BTW, how about it?
>>
>>As I stated multiple times, the reason why its not included is that
>>its horribly slow. But since I don't see any better way to do this
>>and I know quite a few people are using this, I would consider this
>>as well if someone sends me a patch, which has not happened so far.
> 
> 
> So it's just that I need to pull the pomng code and make a diff out
> of it, is that all? (Plus any compilation and perhaps runtime fixes,
> of course.)


>From a quick look I'd say:

- move to x_tables
- remove ip_conntrack stuff
- clean up
- fix up for recent API changes and improvements
- fix 32/64 bit issues
- use proper list macros
- use kzalloc

And explain to me why it needs knowledge about TCP states. That should
be enough to get a discussion started.

Re: Time module included in the default Fedora

[Pascal Hambourg]

Hello,

Jan Engelhardt a écrit :
> 
> iptables is not the same as netfilter (= the kernel part).

A part of iptables is in the kernel, in the Netfilter framework. 
Iptables is divided in two parts :
- a userspace tool (iptables) which allows to manage the rules ;
- a kernel part (ip_tables, iptable_filter...) which enforces the rules.