iptables: compiling with kernel headers

iptables: compiling with kernel headers

[josh lant]

Hi,

Apologies in advance for the long post… I wonder if someone could help
me understand the architecture of the iptables codebase, particularly
its use of kernel headers…

**Background**

I am trying to build for the Morello architecture, which uses
hardware-based capabilities for memory safety, effectively extending
pointer size to 128b, with 64b address and then added bounds/type
information etc in the upper 64b.

Because of this I have had to modify a number of the kernel uapi
headers. If you would like some more context of why I am having to do
this, please see the discussion in this thread:

https://op-lists.linaro.org/archives/list/linux-morello@op-lists.linaro.org/thread/ZUWKFSJDBB2EIR6UMX3QU63KRZFN7VTN/

TL;DR- The uapi structures used in iptables which hold kernel pointers
are not compatible with the ABI of Linux on the Morello architecture,
since currently kernel pointers are 64b, but in userspace a * declares
a capability of size 128b. This causes a discrepancy between what the
kernel expects and what is provided inside some of the netlink
messages, due to the alignment of structures now being 16B. As a
result I have had to modify any kernel pointer inside uapi structs to
be unsigned longs, casting them when used inside the kernel.

Does anyone have any opinion on this method of changing uapi structs
to not contain kernel pointers? Does simply changing them to unsigned
long seem sensible, or am I likely to come up against some horrible
problems I have not yet realised?

**Issue**

When I try to compile iptables using —with-kernel, or —with-ksource, I
get this error:

In file included from …/iptables-morello/extensions/libxt_TOS.c:16:
In file included from …/iptables-morello/extensions/tos_values.c:4:
In file included from …/kernel-source/include/uapi/linux/ip.h:22:
In file included from
…/usr/src/linux-headers-morello/include/asm/byteorder.h:23:
In file included from
…/kernel-source/include/uapi/linux/byteorder/little_endian.h:14:
…/kernel-source/include/uapi/linux/swab.h:48:15: error: unknown type
name '__attribute_const__'

I see that this error arises because when I set the —with-kernel flag
libxt_TOS.c is being compiled against ./include/uapi/linux/ip.h. But
when I compile without that flag, the -isystem flag value provides the
./include/linux/ip.h.

**Questions**

I see in the configure.ac script that setting this flag changes the
includes for the kernel, putting precedence on the uapi versions of
the headers. This was introduced in commit
59bbc59fd2fbbb7a51ed19945d82172890bc40f9 specifically in order to fix
the fact that —with-kernel was broken. However I read in the INSTALL
file:

 “prerequisites…  no kernel-source required “,
and
“--with-ksource= … Xtables does not depend on kernel headers anymore…
probably only useful for development.”

So I wonder, is this —with-kernel feature seldom used/tested and no
longer working in general? Or could my issue be due to the fact that
this __attribute_const__ is a GCC specific directive and I use clang,
and this is not being picked up properly when running configure?

What I thought might be a solution to compile with my modified headers
would be to simply copy over and replace the relevant headers which
are present in the ./include/linux/ directory of the iptables source
repo. However, even with unmodified kernel headers this throws up its
own issues, because I see that there are differences between some of
these headers in the iptables source and those in the kernel source
itself.

One example of these differences is in xt_connmark.h, leading to
errors with duplication of declarations when compiling
libxt_CONNMARK.c using the headers from the kernel source… In the
iptables source the libxt_CONNMARK.c file defines D_SHIFT_LEFT.
However, in the latest version of xt_connmark.h in the kernel, this
enum definition is in the header, so it needs to be removed from the
iptables libxt_CONNMARK.c file. The version of the header in the
iptables source has not been updated to correspond to the current
kernel header version.

commit for xt_connmark.h in kernel source:

commit 472a73e00757b971d613d796374d2727b2e4954d
Author: Jack Ma <jack.ma@alliedtelesis.co.nz>
Date:   Mon Mar 19 09:41:59 2018 +1300

+enum {
+       D_SHIFT_LEFT = 0,
+       D_SHIFT_RIGHT,
+};
+

commit for libxt_CONNMARK.c in iptables source:

commit db7b4e0de960c0ff86b10a3d303b4765dba13d6a
Author: Jack Ma <jack.ma@alliedtelesis.co.nz>
Date:   Tue Apr 24 14:58:57 2018 +1200

+enum {
+       D_SHIFT_LEFT = 0,
+       D_SHIFT_RIGHT,
+};
+

I suppose I am generally confused about why iptables uses its own
bespoke versions of kernel headers in its source, that do not marry up
with those actually in the kernel repo. Are the headers different for
backwards compatibility or portability or such?

Many thanks,

Josh

Re: iptables: compiling with kernel headers

[Florian Westphal]

josh lant <joshualant@googlemail.com> wrote:
> I am trying to build for the Morello architecture, which uses
> hardware-based capabilities for memory safety, effectively extending
> pointer size to 128b, with 64b address and then added bounds/type
> information etc in the upper 64b.
> 
> Because of this I have had to modify a number of the kernel uapi
> headers. If you would like some more context of why I am having to do
> this, please see the discussion in this thread:
> 
> https://op-lists.linaro.org/archives/list/linux-morello@op-lists.linaro.org/thread/ZUWKFSJDBB2EIR6UMX3QU63KRZFN7VTN/
> 
> TL;DR- The uapi structures used in iptables which hold kernel pointers
> are not compatible with the ABI of Linux on the Morello architecture,
> since currently kernel pointers are 64b, but in userspace a * declares
> a capability of size 128b.

Right, this will not work.

> This causes a discrepancy between what the
> kernel expects and what is provided inside some of the netlink
> messages, due to the alignment of structures now being 16B. As a
> result I have had to modify any kernel pointer inside uapi structs to
> be unsigned longs, casting them when used inside the kernel.

i.e. sizeof(unsigned long) == 16 on this architecture?

We cannot change any of these structures unless the layout doesn't
change on 32bit and 64 bit arches.

> Does anyone have any opinion on this method of changing uapi structs
> to not contain kernel pointers? Does simply changing them to unsigned
> long seem sensible, or am I likely to come up against some horrible
> problems I have not yet realised?

No idea, I don't know this architecture.
In iptables, userspace and kernel space exchange binary blobs via
get/setsockopt calls, these binary blobs consists of the relevant
ipt/ip6t/xt_entry structures, matches, targets etc.

Their layout must be the same in userspace and kernel.

If they are not, you lose and only "solution" is more crap added to
CONFIG_NETFILTER_XTABLES_COMPAT.
(The reason for this being a Kconfig option is because I want to remove it).

> When I try to compile iptables using —with-kernel, or —with-ksource, I
> get this error:
> 
> In file included from …/iptables-morello/extensions/libxt_TOS.c:16:
> In file included from …/iptables-morello/extensions/tos_values.c:4:
> In file included from …/kernel-source/include/uapi/linux/ip.h:22:
> In file included from
> …/usr/src/linux-headers-morello/include/asm/byteorder.h:23:
> In file included from
> …/kernel-source/include/uapi/linux/byteorder/little_endian.h:14:
> …/kernel-source/include/uapi/linux/swab.h:48:15: error: unknown type
> name '__attribute_const__'
> 
> I see that this error arises because when I set the —with-kernel flag
> libxt_TOS.c is being compiled against ./include/uapi/linux/ip.h. But
> when I compile without that flag, the -isystem flag value provides the
> ./include/linux/ip.h.

I doubt -—with-kernel is tested at all.

> **Questions**
> 
> I see in the configure.ac script that setting this flag changes the
> includes for the kernel, putting precedence on the uapi versions of
> the headers. This was introduced in commit
> 59bbc59fd2fbbb7a51ed19945d82172890bc40f9 specifically in order to fix
> the fact that —with-kernel was broken. However I read in the INSTALL
> file:
> 
>  “prerequisites…  no kernel-source required “,
> and
> “--with-ksource= … Xtables does not depend on kernel headers anymore…
> probably only useful for development.”
>
> So I wonder, is this —with-kernel feature seldom used/tested and no
> longer working in general?

Not tested, looks like it no longer works.

> Or could my issue be due to the fact that
> this __attribute_const__ is a GCC specific directive and I use clang,
> and this is not being picked up properly when running configure?

No idea, possible.

> What I thought might be a solution to compile with my modified headers
> would be to simply copy over and replace the relevant headers which
> are present in the ./include/linux/ directory of the iptables source
> repo. However, even with unmodified kernel headers this throws up its
> own issues, because I see that there are differences between some of
> these headers in the iptables source and those in the kernel source
> itself.

Yes, but this is unwanted.

> iptables libxt_CONNMARK.c file. The version of the header in the
> iptables source has not been updated to correspond to the current
> kernel header version.
> 
> commit for xt_connmark.h in kernel source:
> 
> commit 472a73e00757b971d613d796374d2727b2e4954d
> Author: Jack Ma <jack.ma@alliedtelesis.co.nz>
> Date:   Mon Mar 19 09:41:59 2018 +1300
> 
> +enum {
> +       D_SHIFT_LEFT = 0,
> +       D_SHIFT_RIGHT,
> +};
> +
> 
> commit for libxt_CONNMARK.c in iptables source:
> 
> commit db7b4e0de960c0ff86b10a3d303b4765dba13d6a
> Author: Jack Ma <jack.ma@alliedtelesis.co.nz>
> Date:   Tue Apr 24 14:58:57 2018 +1200
> 
> +enum {
> +       D_SHIFT_LEFT = 0,
> +       D_SHIFT_RIGHT,
> +};
> +
> 
> I suppose I am generally confused about why iptables uses its own
> bespoke versions of kernel headers in its source, that do not marry up
> with those actually in the kernel repo. Are the headers different for
> backwards compatibility or portability or such?

No, its just that noone has done a full resync in a long time.
The kernel headers are authoritative, but I fear that just replacing
them with recent upstream versions will result in more surprises just
like the ones you found, which need to be fixed up on userspace side.

Why are you interested in getting iptables to work?

It would be better to ensure that nftables is working properly; unlike
with xtables the kernel representation is hidden from userspace.

Re: iptables: compiling with kernel headers

[Phil Sutter]

Hi Josh,

On Wed, Aug 07, 2024 at 03:36:01PM +0100, josh lant wrote:
> Hi,
> 
> Apologies in advance for the long post… I wonder if someone could help
> me understand the architecture of the iptables codebase, particularly
> its use of kernel headers…
> 
> **Background**
> 
> I am trying to build for the Morello architecture, which uses
> hardware-based capabilities for memory safety, effectively extending
> pointer size to 128b, with 64b address and then added bounds/type
> information etc in the upper 64b.
> 
> Because of this I have had to modify a number of the kernel uapi
> headers. If you would like some more context of why I am having to do
> this, please see the discussion in this thread:
> 
> https://op-lists.linaro.org/archives/list/linux-morello@op-lists.linaro.org/thread/ZUWKFSJDBB2EIR6UMX3QU63KRZFN7VTN/
> 
> TL;DR- The uapi structures used in iptables which hold kernel pointers
> are not compatible with the ABI of Linux on the Morello architecture,
> since currently kernel pointers are 64b, but in userspace a * declares
> a capability of size 128b. This causes a discrepancy between what the
> kernel expects and what is provided inside some of the netlink
> messages, due to the alignment of structures now being 16B. As a
> result I have had to modify any kernel pointer inside uapi structs to
> be unsigned longs, casting them when used inside the kernel.
> 
> Does anyone have any opinion on this method of changing uapi structs
> to not contain kernel pointers? Does simply changing them to unsigned
> long seem sensible, or am I likely to come up against some horrible
> problems I have not yet realised?

I won't comment on the above, as it seems like a generic issue with
porting GNU/Linux to Morello, not an iptables-specific one.

> **Issue**
> 
> When I try to compile iptables using —with-kernel, or —with-ksource, I
> get this error:
> 
> In file included from …/iptables-morello/extensions/libxt_TOS.c:16:
> In file included from …/iptables-morello/extensions/tos_values.c:4:
> In file included from …/kernel-source/include/uapi/linux/ip.h:22:
> In file included from
> …/usr/src/linux-headers-morello/include/asm/byteorder.h:23:
> In file included from
> …/kernel-source/include/uapi/linux/byteorder/little_endian.h:14:
> …/kernel-source/include/uapi/linux/swab.h:48:15: error: unknown type
> name '__attribute_const__'

I can't reproduce this here.

> I see that this error arises because when I set the —with-kernel flag
> libxt_TOS.c is being compiled against ./include/uapi/linux/ip.h. But
> when I compile without that flag, the -isystem flag value provides the
> ./include/linux/ip.h.

What './include/linux/ip.h' is that? It's not in iptables.git. On my
system, /usr/include/linux/ip.h is basically identical to
include/uapi/linux/ip.h in my clone of linux.git.

> **Questions**
> 
> I see in the configure.ac script that setting this flag changes the
> includes for the kernel, putting precedence on the uapi versions of
> the headers. This was introduced in commit
> 59bbc59fd2fbbb7a51ed19945d82172890bc40f9 specifically in order to fix
> the fact that —with-kernel was broken. However I read in the INSTALL
> file:
> 
>  “prerequisites…  no kernel-source required “,
> and
> “--with-ksource= … Xtables does not depend on kernel headers anymore…
> probably only useful for development.”

Yeah, we cache needed kernel headers in iptables.git.

> So I wonder, is this —with-kernel feature seldom used/tested and no
> longer working in general? Or could my issue be due to the fact that
> this __attribute_const__ is a GCC specific directive and I use clang,
> and this is not being picked up properly when running configure?

Did you retry using gcc? I personally don't use --with-kernel/ksource,
so from my very own perspective, this feature is unused and untested. ;)

> What I thought might be a solution to compile with my modified headers
> would be to simply copy over and replace the relevant headers which
> are present in the ./include/linux/ directory of the iptables source
> repo. However, even with unmodified kernel headers this throws up its
> own issues, because I see that there are differences between some of
> these headers in the iptables source and those in the kernel source
> itself.

These are bugs IMO. Kernel headers are supposed to be compatible, so one
should not have to adjust user space for newer headers - the problem
with xt_connmark.h is an exception to the rule in my perspective.

> One example of these differences is in xt_connmark.h, leading to
> errors with duplication of declarations when compiling
> libxt_CONNMARK.c using the headers from the kernel source… In the
> iptables source the libxt_CONNMARK.c file defines D_SHIFT_LEFT.
> However, in the latest version of xt_connmark.h in the kernel, this
> enum definition is in the header, so it needs to be removed from the
> iptables libxt_CONNMARK.c file. The version of the header in the
> iptables source has not been updated to correspond to the current
> kernel header version.
> 
> commit for xt_connmark.h in kernel source:
> 
> commit 472a73e00757b971d613d796374d2727b2e4954d
> Author: Jack Ma <jack.ma@alliedtelesis.co.nz>
> Date:   Mon Mar 19 09:41:59 2018 +1300
> 
> +enum {
> +       D_SHIFT_LEFT = 0,
> +       D_SHIFT_RIGHT,
> +};
> +
> 
> commit for libxt_CONNMARK.c in iptables source:
> 
> commit db7b4e0de960c0ff86b10a3d303b4765dba13d6a
> Author: Jack Ma <jack.ma@alliedtelesis.co.nz>
> Date:   Tue Apr 24 14:58:57 2018 +1200
> 
> +enum {
> +       D_SHIFT_LEFT = 0,
> +       D_SHIFT_RIGHT,
> +};
> +

It's odd that Jack apparently preferred to copy that enum into the
source instead of updating the cached header. Oddly, he added the
definiton of struct xt_connmark_tginfo2 to the cached header in the same
commit.

> I suppose I am generally confused about why iptables uses its own
> bespoke versions of kernel headers in its source, that do not marry up
> with those actually in the kernel repo. Are the headers different for
> backwards compatibility or portability or such?

They are there for compatibility reasons, mostly to gain some
independence from host systems compiling the sources.

If it helps you, feel free to submit a patch updating the cached
xt_connmark.h and dropping said enum from libxt_CONNMARK.c. Same for
other issues you noticed. In doubt just send me a report and I'll see
how I can resolve things myself.

Thanks, Phil

Re: iptables: compiling with kernel headers

[Jan Engelhardt]

On Wednesday 2024-08-07 16:36, josh lant wrote:
>
>I am trying to build for the Morello architecture, which uses
>hardware-based capabilities for memory safety, effectively extending
>pointer size to 128b, with 64b address and then added bounds/type
>information etc in the upper 64b.
>
>TL;DR- The uapi structures used in iptables which hold kernel pointers
>are not compatible with the ABI of Linux on the Morello architecture,
>since currently kernel pointers are 64b, but in userspace a * declares
>a capability of size 128b. This causes a discrepancy between what the
>kernel expects and what is provided inside some of the netlink
>messages

I would think something like that would fall under the CONFIG_COMPAT
umbrella.

net/netfilter/xt_limit.c:       .compatsize       = sizeof(struct compat_xt_rateinfo),
net/netfilter/xt_limit.c:       .compat_from_user = limit_mt_compat_from_user,
net/netfilter/xt_limit.c:       .compat_to_user   = limit_mt_compat_to_user,

>I suppose I am generally confused about why iptables uses its own
>bespoke versions of kernel headers in its source

Because it is not guaranteed that the .h files exist anywhere else in the
system. The kernel may even have removed extensions, but there is the
concept that modern iptables ought to be able to run on old kernels with
those long-removed extensions.

Re: iptables: compiling with kernel headers

[Joshua Lant]

Hi Florian,

> Why are you interested in getting iptables to work?
> 
> It would be better to ensure that nftables is working properly; unlike
> with xtables the kernel representation is hidden from userspace.

Sorry I should have been clear initially, I am trying to compile using nftables.

> i.e. sizeof(unsigned long) == 16 on this architecture?
...
> No idea, I don't know this architecture.
> In iptables, userspace and kernel space exchange binary blobs via
> get/setsockopt calls, these binary blobs consists of the relevant
> ipt/ip6t/xt_entry structures, matches, targets etc.
>
> Their layout must be the same in userspace and kernel.

An unsigned long is 8B, which is why we want to use it instead of a "*", which
is a 128b capability in userspace, but is a standard 64b pointer in kernel space.
This is what I am trying to achieve; to maintain the layout between kernel and
userspace. The layout will be the same in kernel and userspace if unsigned long
is used since this is fixed size type to kernel and user.
Since these spaces in the structure i am modifying are reserved strictly for *kernel*
pointers, they should not be used by the userspace right? So I can safely have
userspace assume it is 64b and it will be okay since userspace will not try to
dereference what is stored within there?

> If they are not, you lose and only "solution" is more crap added to
> CONFIG_NETFILTER_XTABLES_COMPAT.
> (The reason for this being a Kconfig option is because I want to remove it).

I'm not sure I understand what you mean here. Although I have noticed the
compat functions in the code, I could not make sense of how/why and when
they are needed. Can you explain the use of this option? I guessed they were for
xtables/nftables compatibility, so they will not help or are not needed in
my instance?

> Not tested, looks like it no longer works.

Okay thanks that's good to know...

> > What I thought might be a solution to compile with my modified headers
> > would be to simply copy over and replace the relevant headers which
> > are present in the ./include/linux/ directory of the iptables source
> > repo. However, even with unmodified kernel headers this throws up its
> > own issues, because I see that there are differences between some of
> > these headers in the iptables source and those in the kernel source
> > itself.
> 
> Yes, but this is unwanted.

I guesss fixing up the --with-kernel option to use the specific headers from the
kernel repo (in my case my modified ones) would be a more sensible option where I
would solve my issue, but also be able to submit a patch that might be of use
to others?

> No, its just that noone has done a full resync in a long time.
> The kernel headers are authoritative, but I fear that just replacing
> them with recent upstream versions will result in more surprises just
> like the ones you found, which need to be fixed up on userspace side.

Okay. Well I suppose if I have to do this for my own work and I can fix any
of these surprises this would be useful?

Many thanks,

Josh

Re: iptables: compiling with kernel headers

[Joshua Lant]

Hi Phil,

> > When I try to compile iptables using —with-kernel, or —with-ksource, I
> > get this error:
> > 
> > In file included from …/iptables-morello/extensions/libxt_TOS.c:16:
> > In file included from …/iptables-morello/extensions/tos_values.c:4:
> > In file included from …/kernel-source/include/uapi/linux/ip.h:22:
> > In file included from
> > …/usr/src/linux-headers-morello/include/asm/byteorder.h:23:
> > In file included from
> > …/kernel-source/include/uapi/linux/byteorder/little_endian.h:14:
> > …/kernel-source/include/uapi/linux/swab.h:48:15: error: unknown type
> > name '__attribute_const__'
> 
> I can't reproduce this here.
>
> > I see that this error arises because when I set the —with-kernel flag
> > libxt_TOS.c is being compiled against ./include/uapi/linux/ip.h. But
> > when I compile without that flag, the -isystem flag value provides the
> > ./include/linux/ip.h.
> 
> What './include/linux/ip.h' is that? It's not in iptables.git. On my
> system, /usr/include/linux/ip.h is basically identical to
> include/uapi/linux/ip.h in my clone of linux.git.
> 

Both the ip.h are under the linux git clone, one in
<repo>/include/linux/ip.h and one in <repo>/include/uapi/linux/ip.h. I
am setting the --with-kernel or --with-ksource to be the root of the
linux git repo.

> 
> Did you retry using gcc? I personally don't use --with-kernel/ksource,
> so from my very own perspective, this feature is unused and untested. ;)
> 

GCC is not an option really. My whole build system is set up around
clang, and the status of the morello GCC compiler is not as mature I
don't think, so could cause more issues than it solves...

> 
> These are bugs IMO. Kernel headers are supposed to be compatible, so one
> should not have to adjust user space for newer headers - the problem
> with xt_connmark.h is an exception to the rule in my perspective.
> 
... 
> If it helps you, feel free to submit a patch updating the cached
> xt_connmark.h and dropping said enum from libxt_CONNMARK.c. Same for
> other issues you noticed. In doubt just send me a report and I'll see
> how I can resolve things myself.
> 

Okay, I'll start trying to update the headers and see how far I get through
the compilation. Thanks a lot for your input.

Cheers,

Josh

Re: iptables: compiling with kernel headers

[Florian Westphal]

Joshua Lant <joshualant@googlemail.com> wrote:
> > Why are you interested in getting iptables to work?
> > 
> > It would be better to ensure that nftables is working properly; unlike
> > with xtables the kernel representation is hidden from userspace.
> 
> Sorry I should have been clear initially, I am trying to compile using nftables.

No, I was talking about:
https://git.netfilter.org/nftables/

which doesn't use any of the old xtables structures and
is not supposed to have any 'binary blobs' passed between
kernel and userland.

iptables-nft still uses some parts of xtables, most of the
matches and targets are handled this way, and binary blob is
passed to kernel via netlink.  See net/netfilter/nft_compat.c

Admittingly, its less bad than the get/setsockopt format, but you've
already encountered things like include/uapi/linux/netfilter/xt_TEE.h

As for a way foward, there are several options:
1. "unsupported, use native nft binary"
2. what you did: force pointers to be sizeof(unsigned long), they
   aren't used by userland, they are placeholders for kernel only
3. Once https://patchwork.ozlabs.org/project/netfilter-devel/patch/20240731222703.22741-8-phil@nwl.cc/
   discussion is resolved, aggressively convert itpables-nft to
   prefer native nft expressions instead of the nft_compat proxy.

3) is definitely a lot more work than 2).
Furthermore iptables-nft cannot be made a full nft client because
iptables syntax lacks aequivalents for native nft constructs like
jump maps or sets, so users cannot mix nft and iptables-nft anyway.

Personally I think it would be better to let iptables move
to maintenance only mode and let it die rather than continue
to spending time on it, but this is the minority opinion so far.

Re: iptables: compiling with kernel headers

[Phil Sutter]

On Thu, Aug 08, 2024 at 11:08:09AM +0000, Joshua Lant wrote:
> Hi Phil,
> 
> > > When I try to compile iptables using —with-kernel, or —with-ksource, I
> > > get this error:
> > > 
> > > In file included from …/iptables-morello/extensions/libxt_TOS.c:16:
> > > In file included from …/iptables-morello/extensions/tos_values.c:4:
> > > In file included from …/kernel-source/include/uapi/linux/ip.h:22:
> > > In file included from
> > > …/usr/src/linux-headers-morello/include/asm/byteorder.h:23:
> > > In file included from
> > > …/kernel-source/include/uapi/linux/byteorder/little_endian.h:14:
> > > …/kernel-source/include/uapi/linux/swab.h:48:15: error: unknown type
> > > name '__attribute_const__'
> > 
> > I can't reproduce this here.
> >
> > > I see that this error arises because when I set the —with-kernel flag
> > > libxt_TOS.c is being compiled against ./include/uapi/linux/ip.h. But
> > > when I compile without that flag, the -isystem flag value provides the
> > > ./include/linux/ip.h.
> > 
> > What './include/linux/ip.h' is that? It's not in iptables.git. On my
> > system, /usr/include/linux/ip.h is basically identical to
> > include/uapi/linux/ip.h in my clone of linux.git.
> > 
> 
> Both the ip.h are under the linux git clone, one in
> <repo>/include/linux/ip.h and one in <repo>/include/uapi/linux/ip.h. I
> am setting the --with-kernel or --with-ksource to be the root of the
> linux git repo.

Ah! Well, include/linux/ip.h includes uapi/linux/ip.h so they are not
entirely incompatible. User space might choke on the former's extras,
though.

> > Did you retry using gcc? I personally don't use --with-kernel/ksource,
> > so from my very own perspective, this feature is unused and untested. ;)
> > 
> 
> GCC is not an option really. My whole build system is set up around
> clang, and the status of the morello GCC compiler is not as mature I
> don't think, so could cause more issues than it solves...

I assumed you see the problem with --with-kernel with x86_64, too.

> > These are bugs IMO. Kernel headers are supposed to be compatible, so one
> > should not have to adjust user space for newer headers - the problem
> > with xt_connmark.h is an exception to the rule in my perspective.
> > 
> ... 
> > If it helps you, feel free to submit a patch updating the cached
> > xt_connmark.h and dropping said enum from libxt_CONNMARK.c. Same for
> > other issues you noticed. In doubt just send me a report and I'll see
> > how I can resolve things myself.
> > 
> 
> Okay, I'll start trying to update the headers and see how far I get through
> the compilation. Thanks a lot for your input.

YW!

Cheers, Phil