Use of Kernel Headers

Use of Kernel Headers

[Jim Gifford]

I have been asked this question a lot of times, and I think it should be 
asked here for an official answer.

It has been stated numerous times that userspace programs should not be 
compiled against raw kernel headers, but iptables does compile against 
userspace headers and breaks this rule. With the advent of the 
linux-libc-headers package, should iptables be compiled against the 
linux-libc-headers or the raw kernel headers since iptables is a user 
space program?

Should patch-o-matic update the headers in the proper location, 
/usr/include/linux/netfilter_ipv4 etc?

----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Antony Stone]

On Tuesday 17 August 2004 9:40 pm, Jim Gifford wrote:

> I have been asked this question a lot of times, and I think it should be
> asked here for an official answer.
>
> It has been stated numerous times that userspace programs should not be
> compiled against raw kernel headers, but iptables does compile against
> userspace headers and breaks this rule. With the advent of the
> linux-libc-headers package, should iptables be compiled against the
> linux-libc-headers or the raw kernel headers since iptables is a user
> space program?
>
> Should patch-o-matic update the headers in the proper location,
> /usr/include/linux/netfilter_ipv4 etc?

Patch-o-matic is not the right way of doing this - this would be dealt with in 
a new version release of iptables.

P-o-M is for adding new features and functions (which are optional and not 
used by most iptables users) to netfilter; it couldn't sensibly be used to 
change the structure of how it compiles.

As for the answer to your question, I think that is one for the developers' 
list.   Some of them do read this list, but cannot be guaranteed to see all 
posts or respond to them due to the volume.

Regards,

Antony.

-- 
People who use Microsoft software should be certified.

                                                     Please reply to the list;
                                                           please don't CC me.

Re: Use of Kernel Headers

[Jim Gifford]

Antony Stone wrote:

>On Tuesday 17 August 2004 9:40 pm, Jim Gifford wrote:
>
>  
>
>>I have been asked this question a lot of times, and I think it should be
>>asked here for an official answer.
>>
>>It has been stated numerous times that userspace programs should not be
>>compiled against raw kernel headers, but iptables does compile against
>>userspace headers and breaks this rule. With the advent of the
>>linux-libc-headers package, should iptables be compiled against the
>>linux-libc-headers or the raw kernel headers since iptables is a user
>>space program?
>>
>>Should patch-o-matic update the headers in the proper location,
>>/usr/include/linux/netfilter_ipv4 etc?
>>    
>>
>
>Patch-o-matic is not the right way of doing this - this would be dealt with in 
>a new version release of iptables.
>
>P-o-M is for adding new features and functions (which are optional and not 
>used by most iptables users) to netfilter; it couldn't sensibly be used to 
>change the structure of how it compiles.
>
>As for the answer to your question, I think that is one for the developers' 
>list.   Some of them do read this list, but cannot be guaranteed to see all 
>posts or respond to them due to the volume.
>
>Regards,
>
>Antony.
>
>  
>
Thanx Anthony, will do.

-- 
----
Jim Gifford
maillist@jg555.com

Use of Kernel Headers

[Jim Gifford]

I have been asked this question a lot of times, and I think it should be 
asked here for an official answer.

It has been stated numerous times that userspace programs should not be 
compiled against raw kernel headers, but iptables does compile against 
userspace headers and breaks this rule. With the advent of the 
linux-libc-headers package, should iptables be compiled against the 
linux-libc-headers or the raw kernel headers since iptables is a user 
space program?

Should patch-o-matic update the headers in the proper location, 
/usr/include/linux/netfilter_ipv4 etc?

-- 
----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Henrik Nordstrom]

On Tue, 17 Aug 2004, Jim Gifford wrote:

> It has been stated numerous times that userspace programs should not be 
> compiled against raw kernel headers, but iptables does compile against 
> userspace headers and breaks this rule. With the advent of the 
> linux-libc-headers package, should iptables be compiled against the 
> linux-libc-headers or the raw kernel headers since iptables is a user space 
> program?

This depends on if you build iptables for your custom patched kernel or a 
standard kernel.

For a standard kernel it should be sufficient with linux-libc-headers I 
think, but it is possible some required linux iptables headers is missing 
from the iptables package (include/linux/netfilter_ipv[46]/). If 
you find some missing please report here which files needs to be added 
from the kernel tree and maybe it can be cleaned up.

> Should patch-o-matic update the headers in the proper location, 
> /usr/include/linux/netfilter_ipv4 etc?

patch-o-matic should always update the kernel source tree and your 
iptables should then be built to this source tree. This to make sure 
the view of iptables and your kernel matches. But to be honest it should 
only be the include/linux/netfilter_ipv[46]/ directories which is required 
by iptables.

Regards
Henrik

Re: Use of Kernel Headers

[Jim Gifford]

Henrik Nordstrom wrote:

> On Tue, 17 Aug 2004, Jim Gifford wrote:
>
>> It has been stated numerous times that userspace programs should not 
>> be compiled against raw kernel headers, but iptables does compile 
>> against userspace headers and breaks this rule. With the advent of 
>> the linux-libc-headers package, should iptables be compiled against 
>> the linux-libc-headers or the raw kernel headers since iptables is a 
>> user space program?
>
>
> This depends on if you build iptables for your custom patched kernel 
> or a standard kernel.
>
> For a standard kernel it should be sufficient with linux-libc-headers 
> I think, but it is possible some required linux iptables headers is 
> missing from the iptables package (include/linux/netfilter_ipv[46]/). 
> If you find some missing please report here which files needs to be 
> added from the kernel tree and maybe it can be cleaned up.
>
>> Should patch-o-matic update the headers in the proper location, 
>> /usr/include/linux/netfilter_ipv4 etc?
>
>
> patch-o-matic should always update the kernel source tree and your 
> iptables should then be built to this source tree. This to make sure 
> the view of iptables and your kernel matches. But to be honest it 
> should only be the include/linux/netfilter_ipv[46]/ directories which 
> is required by iptables.
>
> Regards
> Henrik
>
Ok so we build iptables against the linux-libc-headers, then we then use 
patch-o-matic-ng to add new support for psd(insert you own example). But 
iptables is set to compile using the linux-libc-headers, won't the 
compile fail since it can't find ipt_psd.h in the linux-libc-headers.

If patch-o-matic changes a header, it should also check 
/usr/include/linux/netfilter{whatever} and patch that file also so they 
are insync with each other. Which presents a headache since the 
linux-libc-headers are only released when a new release is out. Unless 
you check the version.h file that comes with linux-libc-headers and 
verfiy the value of  LIBC_HEADERS_VERSION.

So it's a double edge sword the way I see it.

-- 
----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Henrik Nordstrom]

On Tue, 17 Aug 2004, Jim Gifford wrote:

> Ok so we build iptables against the linux-libc-headers, then we then use 
> patch-o-matic-ng to add new support for psd(insert you own example). But 
> iptables is set to compile using the linux-libc-headers, won't the compile 
> fail since it can't find ipt_psd.h in the linux-libc-headers.

Correct.

You then need to build your iptables to your modified Linux kernel 
headers, or copy include/linux/netfilter_ipv?/* to your iptables include 
directory.

> If patch-o-matic changes a header, it should also check 
> /usr/include/linux/netfilter{whatever} and patch that file also so they are 
> insync with each other.

No. This would not work as /usr/include/linux/ does not need to match the 
kernel you are currently patching.

What would be OK is for pom to modify the iptables include directory.

Regards
Henrik

Re: Use of Kernel Headers

[Jim Gifford]

Henrik Nordstrom wrote:

> On Tue, 17 Aug 2004, Jim Gifford wrote:
>
>> Ok so we build iptables against the linux-libc-headers, then we then 
>> use patch-o-matic-ng to add new support for psd(insert you own 
>> example). But iptables is set to compile using the 
>> linux-libc-headers, won't the compile fail since it can't find 
>> ipt_psd.h in the linux-libc-headers.
>
>
> Correct.
>
> You then need to build your iptables to your modified Linux kernel 
> headers, or copy include/linux/netfilter_ipv?/* to your iptables 
> include directory.
>
>> If patch-o-matic changes a header, it should also check 
>> /usr/include/linux/netfilter{whatever} and patch that file also so 
>> they are insync with each other.
>
>
> No. This would not work as /usr/include/linux/ does not need to match 
> the kernel you are currently patching.
>
> What would be OK is for pom to modify the iptables include directory.
>
> Regards
> Henrik
>
Thanx Henrik for the reply, there are a lot of people out there trying 
to figure out what do on this issue.

So would the proper course of action to be to remove the kernel 
directory from the iptables Makefile? or is there more to it.

-- 
----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Henrik Nordstrom]

On Wed, 18 Aug 2004, Jim Gifford wrote:

> So would the proper course of action to be to remove the kernel directory 
> from the iptables Makefile? or is there more to it.

Make it optional in the Makefile and make sure include/linux/netfilter* is 
up to date.

There is still value in allowing iptables to compile using the kernel 
headers, especially so during development of a new extensions but also to 
better match the kernel tree.

Regards
Henrik

Re: Use of Kernel Headers

[Jim Gifford]

Henrik Nordstrom wrote:

> On Wed, 18 Aug 2004, Jim Gifford wrote:
>
>> So would the proper course of action to be to remove the kernel 
>> directory from the iptables Makefile? or is there more to it.
>
>
> Make it optional in the Makefile and make sure 
> include/linux/netfilter* is up to date.
>
> There is still value in allowing iptables to compile using the kernel 
> headers, especially so during development of a new extensions but also 
> to better match the kernel tree.
>
> Regards
> Henrik
>
Doesn't patch-o-matic update the ones in iptables-version/include/linux?
    So the the kernel and the iptables are built from the same headers? 
(Is this desired)
   
Or would it benefit iptables to include all the updated headers in 
iptables-version/include/linux?

I don't mean to be pest on this, I just want the best solution possible 
for anyone who uses iptables.

-- 
----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Henrik Nordstrom]

On Wed, 18 Aug 2004, Jim Gifford wrote:

> Doesn't patch-o-matic update the ones in iptables-version/include/linux?

Nope.

> So the the kernel and the iptables are built from the same headers? (Is 
> this desired)

Today yes, and probably desired while building with patch-o-matic patches 
applied.

> Or would it benefit iptables to include all the updated headers in 
> iptables-version/include/linux?

This would make things more clean cut separation between kernel and 
userspace.

Regards
Henrik

Re: Use of Kernel Headers

[Jim Gifford]

Henrik Nordstrom wrote:

> On Wed, 18 Aug 2004, Jim Gifford wrote:
>
>> Doesn't patch-o-matic update the ones in iptables-version/include/linux?
>
>
> Nope.
>
>> So the the kernel and the iptables are built from the same headers? 
>> (Is this desired)
>
>
> Today yes, and probably desired while building with patch-o-matic 
> patches applied.
>
>> Or would it benefit iptables to include all the updated headers in 
>> iptables-version/include/linux?
>
>
> This would make things more clean cut separation between kernel and 
> userspace.
>
> Regards
> Henrik
>
So now what do we do to get these changes made, or is it going to be a 
distro specific thing?

-- 
----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Jozsef Kadlecsik]

On Wed, 18 Aug 2004, Henrik Nordstrom wrote:

> On Tue, 17 Aug 2004, Jim Gifford wrote:
>
> > It has been stated numerous times that userspace programs should not be
> > compiled against raw kernel headers, but iptables does compile against
> > userspace headers and breaks this rule. With the advent of the
> > linux-libc-headers package, should iptables be compiled against the
> > linux-libc-headers or the raw kernel headers since iptables is a user space
> > program?
>
> This depends on if you build iptables for your custom patched kernel or a
> standard kernel.
>
> For a standard kernel it should be sufficient with linux-libc-headers I
> think, but it is possible some required linux iptables headers is missing
> from the iptables package (include/linux/netfilter_ipv[46]/). If
> you find some missing please report here which files needs to be added
> from the kernel tree and maybe it can be cleaned up.

The include/linux tree in the iptables source exists for
forward-compatibility reasons only. The to-be-submitted new extensions are
added to the iptables tree, together with their header files and enabled
for default compilation in the Makefile. Thus when one donwloads/installs
the next kernel release, there is no need to touch the iptables binary
because it knows about the new extensions. The definite source of the
include files is the kernel tree for iptables, which overrides the
include directory in the userspace source.

> > Should patch-o-matic update the headers in the proper location,
> > /usr/include/linux/netfilter_ipv4 etc?
>
> patch-o-matic should always update the kernel source tree and your
> iptables should then be built to this source tree. This to make sure
> the view of iptables and your kernel matches. But to be honest it should
> only be the include/linux/netfilter_ipv[46]/ directories which is required
> by iptables.

pom should definitely not update /usr/include/linux because the kernel
version there can be (usually are) different than in the patched kernel
source. Also, /usr/include/linux is maintained by the given distribution.
Next update could simply overwrite patched files, checksumming would fail
in built-in IDS, etc.

My impression is that netfilter/iptables/pom does not really fit into
the linux-libc-headers schema. And iptables should compile cleanly on any
distro, including the ones which do not contain linux-libc-headers.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Use of Kernel Headers

[Jim Gifford]

Jozsef Kadlecsik wrote:

>On Wed, 18 Aug 2004, Henrik Nordstrom wrote:
>
>  
>
>>On Tue, 17 Aug 2004, Jim Gifford wrote:
>>
>>    
>>
>>>It has been stated numerous times that userspace programs should not be
>>>compiled against raw kernel headers, but iptables does compile against
>>>userspace headers and breaks this rule. With the advent of the
>>>linux-libc-headers package, should iptables be compiled against the
>>>linux-libc-headers or the raw kernel headers since iptables is a user space
>>>program?
>>>      
>>>
>>This depends on if you build iptables for your custom patched kernel or a
>>standard kernel.
>>
>>For a standard kernel it should be sufficient with linux-libc-headers I
>>think, but it is possible some required linux iptables headers is missing
>>from the iptables package (include/linux/netfilter_ipv[46]/). If
>>you find some missing please report here which files needs to be added
>>from the kernel tree and maybe it can be cleaned up.
>>    
>>
>
>The include/linux tree in the iptables source exists for
>forward-compatibility reasons only. The to-be-submitted new extensions are
>added to the iptables tree, together with their header files and enabled
>for default compilation in the Makefile. Thus when one donwloads/installs
>the next kernel release, there is no need to touch the iptables binary
>because it knows about the new extensions. The definite source of the
>include files is the kernel tree for iptables, which overrides the
>include directory in the userspace source.
>
>  
>
>>>Should patch-o-matic update the headers in the proper location,
>>>/usr/include/linux/netfilter_ipv4 etc?
>>>      
>>>
>>patch-o-matic should always update the kernel source tree and your
>>iptables should then be built to this source tree. This to make sure
>>the view of iptables and your kernel matches. But to be honest it should
>>only be the include/linux/netfilter_ipv[46]/ directories which is required
>>by iptables.
>>    
>>
>
>pom should definitely not update /usr/include/linux because the kernel
>version there can be (usually are) different than in the patched kernel
>source. Also, /usr/include/linux is maintained by the given distribution.
>Next update could simply overwrite patched files, checksumming would fail
>in built-in IDS, etc.
>
>My impression is that netfilter/iptables/pom does not really fit into
>the linux-libc-headers schema. And iptables should compile cleanly on any
>distro, including the ones which do not contain linux-libc-headers.
>
>Best regards,
>Jozsef
>-
>E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
>PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
>Address : KFKI Research Institute for Particle and Nuclear Physics
>          H-1525 Budapest 114, POB. 49, Hungary
>
>  
>
Jozsef,
    I understand you point, and have thought about it quite a bit before 
I replied. Most of the distro's who are distributing 2.6 are using the 
linux-libc-headers or making there own. Currently Linus has stated on 
numerous times that userspace programs should not use the kernel source 
headers.

This all goes back to having true separation between the kernel and 
userspace, I'm starting to think iptables is different in it's situation 
and true separation will never be possible


-- 
----
Jim Gifford
maillist@jg555.com

Re: Use of Kernel Headers

[Henrik Nordstrom]

On Fri, 20 Aug 2004, Jim Gifford wrote:

> This all goes back to having true separation between the kernel and 
> userspace, I'm starting to think iptables is different in it's situation and 
> true separation will never be possible

Separation is possible, it is mainly a matter of how the userspace headers 
should be maintained and how to ensure kernel compatibility to a 
reasonable manner (i.e. mostly an adminstrative issue). Today a lot of the 
kernel release compatibility depends on using the kernel headers. There is 
also a one or two extensions which have more complex dependencies but I 
think all of these is in pom extra.

Have you tried what I requested before? Copying the kernel netfilter_ipvX 
headers to the userspace include directory and testing how much 
of it compiles? Just specify the iptables userspace directory as kernel 
source after doing this to avoid having to fiddle with the makefiles etc..

Regards
Henrik

Re: Use of Kernel Headers

[Tobias DiPasquale]

On Sat, 21 Aug 2004 11:30:36 +0200 (CEST), Henrik Nordstrom
<hno@marasystems.com> wrote:
> 
> 
> On Fri, 20 Aug 2004, Jim Gifford wrote:
> 
> > This all goes back to having true separation between the kernel and
> > userspace, I'm starting to think iptables is different in it's situation and
> > true separation will never be possible
> 
> Separation is possible, it is mainly a matter of how the userspace headers
> should be maintained and how to ensure kernel compatibility to a
> reasonable manner (i.e. mostly an adminstrative issue). Today a lot of the
> kernel release compatibility depends on using the kernel headers. There is
> also a one or two extensions which have more complex dependencies but I
> think all of these is in pom extra.

Perhaps a special macro could be used to demarcate the sections of
kernel headers that are for userspace use (opposite of what __KERNEL__
does now). Then, when you wanted a set of headers for userspace apps
to use when they needed to interface with the kernel, a script could
be run to strip out all sections of the headers that were not
demarcated for use by userspace, and only those headers would be
copied into /usr/include/linux? Would that be a workable solution? Or
is it more complicated than that?

-- 
[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d

Re: Use of Kernel Headers

[Henrik Nordstrom]

On Sat, 21 Aug 2004, Tobias DiPasquale wrote:

> Perhaps a special macro could be used to demarcate the sections of
> kernel headers that are for userspace use (opposite of what __KERNEL__
> does now).

#ifndef __KERNEL__

works well for this purpose.

How the system user headers are composed is of no concern to iptables. 
What iptables need to worry about to solve this issue is to supply it's 
own required headers defining the current iptables kernel interface the 
iptables commands should use, allowing the userspace commands to be built 
without the kernel source.

Regards
Henrik