Re: [ANN] net-next is OPEN

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
> > Going through the failing ksft-net series on
> > https://netdev.bots.linux.dev/status.html, all the tests I'm
> > responsible seem to be passing.  
> 
> Here's a more handy link filtered down to failures (clicking on 
> the test counts links here):
> 
> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> 
> I have been attributing the udpg[rs]o and timestamp tests to you,
> but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is
mapping to nftables on the executor. And either nftables doesn't
support the functionality the tests expect or we're missing configs :(
E.g. the TTL module.

Re: [ANN] net-next is OPEN

[Matthieu Baerts]

Hello,

24 Jan 2024 17:01:24 Jakub Kicinski <kuba@kernel.org>:

> On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
>>> Going through the failing ksft-net series on
>>> https://netdev.bots.linux.dev/status.html, all the tests I'm
>>> responsible seem to be passing. 
>>
>> Here's a more handy link filtered down to failures (clicking on
>> the test counts links here):
>>
>> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
>>
>> I have been attributing the udpg[rs]o and timestamp tests to you,
>> but I haven't actually checked.. are they not yours? :)
>
> Ah, BTW, a major source of failures seems to be that iptables is
> mapping to nftables on the executor. And either nftables doesn't
> support the functionality the tests expect or we're missing configs :(
> E.g. the TTL module.

I don't know if it is the same issue, but for MPTCP, we use
'iptables-legacy' if available.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

Cheers,
Matt

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 18:35:14 +0000 (GMT) Matthieu Baerts wrote:
> > Ah, BTW, a major source of failures seems to be that iptables is
> > mapping to nftables on the executor. And either nftables doesn't
> > support the functionality the tests expect or we're missing configs :(
> > E.g. the TTL module.  
> 
> I don't know if it is the same issue, but for MPTCP, we use
> 'iptables-legacy' if available.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

Great! Thanks for the pointer. I installed the packages now,
so folks should be able to fix up their scripts.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 09:01:23AM -0800, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
> > > Going through the failing ksft-net series on
> > > https://netdev.bots.linux.dev/status.html, all the tests I'm
> > > responsible seem to be passing.  
> > 
> > Here's a more handy link filtered down to failures (clicking on 
> > the test counts links here):
> > 
> > https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> > 
> > I have been attributing the udpg[rs]o and timestamp tests to you,
> > but I haven't actually checked.. are they not yours? :)
> 
> Ah, BTW, a major source of failures seems to be that iptables is
> mapping to nftables on the executor. And either nftables doesn't
> support the functionality the tests expect or we're missing configs :(
> E.g. the TTL module.

I could only find in the listing above this:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout

which shows:

 ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory

which seems like setup is broken, ie. it could not find libxt_conntrack.so

What is the issue?

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 06:35:14PM +0000, Matthieu Baerts wrote:
> Hello,
> 
> 24 Jan 2024 17:01:24 Jakub Kicinski <kuba@kernel.org>:
> 
> > On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
> >>> Going through the failing ksft-net series on
> >>> https://netdev.bots.linux.dev/status.html, all the tests I'm
> >>> responsible seem to be passing. 
> >>
> >> Here's a more handy link filtered down to failures (clicking on
> >> the test counts links here):
> >>
> >> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> >>
> >> I have been attributing the udpg[rs]o and timestamp tests to you,
> >> but I haven't actually checked.. are they not yours? :)
> >
> > Ah, BTW, a major source of failures seems to be that iptables is
> > mapping to nftables on the executor. And either nftables doesn't
> > support the functionality the tests expect or we're missing configs :(
> > E.g. the TTL module.
> 
> I don't know if it is the same issue, but for MPTCP, we use
> 'iptables-legacy' if available.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

I'd suggest you do the other way around, first check if iptables-nft
is available, otherwise fall back to iptables-nft

commit refers to 5.15 already have iptables-nft support, it should
work out of the box.

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 20:16:39 +0100 Pablo Neira Ayuso wrote:
> > Ah, BTW, a major source of failures seems to be that iptables is
> > mapping to nftables on the executor. And either nftables doesn't
> > support the functionality the tests expect or we're missing configs :(
> > E.g. the TTL module.  
> 
> I could only find in the listing above this:

Thanks for taking a look!

> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout
> 
> which shows:
> 
>  ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory
> 
> which seems like setup is broken, ie. it could not find libxt_conntrack.so

Hm, odd, it's there:

$ ls /lib64/xtables/libxt_conntrack.so
/lib64/xtables/libxt_conntrack.so

but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 
is in it (normal loaded always scans system paths)!

> What is the issue?

A lot of the tests print warning messages like the ones below.
Some of them pass some of them fail. Tweaking the kernel config
to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_*
are included seem to have made no difference, which I concluded was
because iptables CLI uses nf_tables here by default..

[435321]$ grep -nrI "Warning: Extension" .
./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./6-fib-tests-sh/stdout:308:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./6-fib-tests-sh/stdout:316:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./6-fib-tests-sh/stdout:319:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:12:# No GRO                                  Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:13:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:14:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:16:# GRO frag list                           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:17:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:18:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:19:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:21:# Warning: Extension DNAT revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:23:# GRO fwd                                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:24:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:38:# GRO frag list over UDP tunnel           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:39:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:40:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:41:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:43:# Warning: Extension DNAT revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:45:# GRO fwd over UDP tunnel                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:46:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:61:# No GRO                                  Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:62:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:63:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:65:# GRO frag list                           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:66:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:67:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:68:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:72:# GRO fwd                                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:73:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:88:# GRO frag list over UDP tunnel           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:89:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:90:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:91:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:95:# GRO fwd over UDP tunnel                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:96:# Warning: Extension udp revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:17:# Warning: Extension length revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:19:# Warning: Extension length revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:22:# Warning: Extension length revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:24:# Warning: Extension length revision 0 not supported, missing kernel module?
./56-xfrm-policy-sh/stdout:11:# Warning: Extension policy revision 0 not supported, missing kernel module?
./56-xfrm-policy-sh/stdout:13:# Warning: Extension policy revision 0 not supported, missing kernel module?
./54-amt-sh/stdout:94:# Warning: Extension TTL revision 0 not supported, missing kernel module?

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 11:40:57AM -0800, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 20:16:39 +0100 Pablo Neira Ayuso wrote:
> > > Ah, BTW, a major source of failures seems to be that iptables is
> > > mapping to nftables on the executor. And either nftables doesn't
> > > support the functionality the tests expect or we're missing configs :(
> > > E.g. the TTL module.  
> > 
> > I could only find in the listing above this:
> 
> Thanks for taking a look!
> 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout
> > 
> > which shows:
> > 
> >  ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory
> > 
> > which seems like setup is broken, ie. it could not find libxt_conntrack.so
> 
> Hm, odd, it's there:
> 
> $ ls /lib64/xtables/libxt_conntrack.so
> /lib64/xtables/libxt_conntrack.so
>
> but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 
> is in it (normal loaded always scans system paths)!

Could you also check your ./configure output for iptables? It shows
the directory where the .so file are search and found:

  ...
  Xtables extension directory:          /usr/lib/xtables

> > What is the issue?
> 
> A lot of the tests print warning messages like the ones below.
> Some of them pass some of them fail. Tweaking the kernel config
> to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_*
> are included seem to have made no difference, which I concluded was
> because iptables CLI uses nf_tables here by default..

Please, check if the symlink refers to -legacy or -nft via:

$ ls -la /usr/sbin/iptables

> [435321]$ grep -nrI "Warning: Extension" .
> ./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?

This could come from either legacy or nftables:

libxtables/xtables.c:                                   "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
iptables/nft.c:                         "Warning: Extension %s revision 0 not supported, missing kernel module?\n",

both have the same error.

if that is the nftables backend, it might be also that .config is
missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
options.

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 21:02:16 +0100 Pablo Neira Ayuso wrote:
> On Wed, Jan 24, 2024 at 11:40:57AM -0800, Jakub Kicinski wrote:
> > Hm, odd, it's there:
> > 
> > $ ls /lib64/xtables/libxt_conntrack.so
> > /lib64/xtables/libxt_conntrack.so
> >
> > but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 
> > is in it (normal loaded always scans system paths)!  
> 
> Could you also check your ./configure output for iptables? It shows
> the directory where the .so file are search and found:
> 
>   ...
>   Xtables extension directory:          /usr/lib/xtables

I'm using the OS iptables, I was hoping that for net/ tests
OS iptables should be good enough :S Is there a way to get
the info you're after?

$ iptables -V 
iptables v1.8.8 (nf_tables)

> > > What is the issue?  
> > 
> > A lot of the tests print warning messages like the ones below.
> > Some of them pass some of them fail. Tweaking the kernel config
> > to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_*
> > are included seem to have made no difference, which I concluded was
> > because iptables CLI uses nf_tables here by default..  
> 
> Please, check if the symlink refers to -legacy or -nft via:
> 
> $ ls -la /usr/sbin/iptables

Ah, neat:

$ ls -la /etc/alternatives/iptables
lrwxrwxrwx. 1 root root 22 Jan  7 22:10 /etc/alternatives/iptables -> /usr/sbin/iptables-nft
$ ls -la /usr/sbin/iptables
lrwxrwxrwx. 1 root root 26 Jan  7 22:10 /usr/sbin/iptables -> /etc/alternatives/iptables

> > [435321]$ grep -nrI "Warning: Extension" .
> > ./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?  
> 
> This could come from either legacy or nftables:
> 
> libxtables/xtables.c:                                   "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
> iptables/nft.c:                         "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
> 
> both have the same error.
> 
> if that is the nftables backend, it might be also that .config is
> missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> options.

FWIW full config:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config

CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
> > if that is the nftables backend, it might be also that .config is
> > missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> > options.  
> 
> FWIW full config:
> 
> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config
> 
> CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

NFT_COMPAT fixed a lot! One remaining warning comes from using 
-m length. Which NFT config do we need for that one?

Re: [netfilter-core] [ANN] net-next is OPEN

[Florian Westphal]

Jakub Kicinski <kuba@kernel.org> wrote:
> On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
> > > if that is the nftables backend, it might be also that .config is
> > > missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> > > options.  
> > 
> > FWIW full config:
> > 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config
> > 
> > CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.
> 
> NFT_COMPAT fixed a lot! One remaining warning comes from using 
> -m length. Which NFT config do we need for that one?

CONFIG_NETFILTER_XT_MATCH_LENGTH=m|y

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 09:07:24PM -0800, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
> > > if that is the nftables backend, it might be also that .config is
> > > missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> > > options.  
> > 
> > FWIW full config:
> > 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config
> > 
> > CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.
> 
> NFT_COMPAT fixed a lot! One remaining warning comes from using 
> -m length. Which NFT config do we need for that one?

May I have a look at the logs? How does the error look like?

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Thu, 25 Jan 2024 09:52:11 +0100 Florian Westphal wrote:
> > NFT_COMPAT fixed a lot! One remaining warning comes from using 
> > -m length. Which NFT config do we need for that one?  
> 
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m|y

Thank you, that seems to solve the remaining problem!

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Thu, 25 Jan 2024 10:29:41 +0100 Pablo Neira Ayuso wrote:
> > NFT_COMPAT fixed a lot! One remaining warning comes from using 
> > -m length. Which NFT config do we need for that one?  
> 
> May I have a look at the logs? How does the error look like?

With the config pointed out by Florian in addition to NFT_COMPAT
all the iptables errors in the logs are gone, without switching
to legacy. Thank you for the help!

Also LMK if you guys want us to try running netfilter tests.
I'm guessing we're unlikely to regress anything in net-next,
and you have running netfilter tests covered - but if it's 
not too hard we can try to hook them up to net-next, too!
(Or you can run them on our branch and report back :))

Re: [netfilter-core] [ANN] net-next is OPEN

[Matthieu Baerts]

Hi Pablo,

Thank you for your reply!

On 24/01/2024 20:18, Pablo Neira Ayuso wrote:
> On Wed, Jan 24, 2024 at 06:35:14PM +0000, Matthieu Baerts wrote:
>> Hello,
>>
>> 24 Jan 2024 17:01:24 Jakub Kicinski <kuba@kernel.org>:
>>
>>> On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
>>>>> Going through the failing ksft-net series on
>>>>> https://netdev.bots.linux.dev/status.html, all the tests I'm
>>>>> responsible seem to be passing. 
>>>>
>>>> Here's a more handy link filtered down to failures (clicking on
>>>> the test counts links here):
>>>>
>>>> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
>>>>
>>>> I have been attributing the udpg[rs]o and timestamp tests to you,
>>>> but I haven't actually checked.. are they not yours? :)
>>>
>>> Ah, BTW, a major source of failures seems to be that iptables is
>>> mapping to nftables on the executor. And either nftables doesn't
>>> support the functionality the tests expect or we're missing configs :(
>>> E.g. the TTL module.
>>
>> I don't know if it is the same issue, but for MPTCP, we use
>> 'iptables-legacy' if available.
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400
> 
> I'd suggest you do the other way around, first check if iptables-nft
> is available, otherwise fall back to iptables-nft
> 
> commit refers to 5.15 already have iptables-nft support, it should
> work out of the box.

Good point, I understand it sounds better to use 'iptables-nft' in new
kselftests. I should have added a bit of background and not just a link
to this commit: at that time (around ~v6.4), we didn't need to force
using 'iptables-legacy' on -net or net-next tree. But we needed that
when testing kernels <= v5.15.

When validating (old) stable kernels, the recommended practice is
apparently [1] to use the kselftests from the last stable version, e.g.
using the kselftests from v6.7.4 when validating kernel v5.15.148. The
kselftests are then supposed to support older kernels, e.g. by skipping
some parts if a feature is not available. I didn't know about that
before, and I don't know if all kselftests devs know about that.

I don't think that's easy to support old kernels, especially in the
networking area, where some features/behaviours are not directly exposed
to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
or use other (ugly?) workarounds [2] to predict what we are supposed to
have, depending on the kernel that is being used. But something has to
be done, not to have big kselftests, with many different subtests,
always marked as "failed" when validating new stable releases.

Back to the modification to use 'iptables-legacy', maybe a kernel config
was missing, but the same kselftest, with the same list of kconfig to
add, was not working with the v5.15 kernel, while everything was OK with
a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
will check if maybe an old kconfig option was not missing.

[1] https://lore.kernel.org/stable/ZAG8dla274kYfxoK@kroah.com/
[2]
https://lore.kernel.org/netdev/20230609-upstream-net-20230610-mptcp-selftests-support-old-kernels-part-3-v1-0-2896fe2ee8a3@tessares.net

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

Hi Matthieu,

On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
[...]
> Good point, I understand it sounds better to use 'iptables-nft' in new
> kselftests. I should have added a bit of background and not just a link
> to this commit: at that time (around ~v6.4), we didn't need to force
> using 'iptables-legacy' on -net or net-next tree. But we needed that
> when testing kernels <= v5.15.
> 
> When validating (old) stable kernels, the recommended practice is
> apparently [1] to use the kselftests from the last stable version, e.g.
> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
> kselftests are then supposed to support older kernels, e.g. by skipping
> some parts if a feature is not available. I didn't know about that
> before, and I don't know if all kselftests devs know about that.

We are sending backports to stable kernels, if one stable kernel
fails, then we have to fix it.

> I don't think that's easy to support old kernels, especially in the
> networking area, where some features/behaviours are not directly exposed
> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
> or use other (ugly?) workarounds [2] to predict what we are supposed to
> have, depending on the kernel that is being used. But something has to
> be done, not to have big kselftests, with many different subtests,
> always marked as "failed" when validating new stable releases.

iptables-nft is supported in all of the existing stable kernels.

> Back to the modification to use 'iptables-legacy', maybe a kernel config
> was missing, but the same kselftest, with the same list of kconfig to
> add, was not working with the v5.15 kernel, while everything was OK with
> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
> will check if maybe an old kconfig option was not missing.

I suspect this is most likely kernel config missing, as it happened to Jakub.

Thanks.

Re: [netfilter-core] [ANN] net-next is OPEN

[Matthieu Baerts]

Hi Pablo,

Thank you for your reply!

On 07/02/2024 10:49, Pablo Neira Ayuso wrote:
> Hi Matthieu,
> 
> On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
> [...]
>> Good point, I understand it sounds better to use 'iptables-nft' in new
>> kselftests. I should have added a bit of background and not just a link
>> to this commit: at that time (around ~v6.4), we didn't need to force
>> using 'iptables-legacy' on -net or net-next tree. But we needed that
>> when testing kernels <= v5.15.
>>
>> When validating (old) stable kernels, the recommended practice is
>> apparently [1] to use the kselftests from the last stable version, e.g.
>> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
>> kselftests are then supposed to support older kernels, e.g. by skipping
>> some parts if a feature is not available. I didn't know about that
>> before, and I don't know if all kselftests devs know about that.
> 
> We are sending backports to stable kernels, if one stable kernel
> fails, then we have to fix it.

Do you validate stable kernels by running the kselftests from the same
version (e.g. both from v5.15.x) or by using the kselftests from the
last stable one (e.g. kernel v5.15.148 validated using the kselftests
from v6.7.4)?

>> I don't think that's easy to support old kernels, especially in the
>> networking area, where some features/behaviours are not directly exposed
>> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
>> or use other (ugly?) workarounds [2] to predict what we are supposed to
>> have, depending on the kernel that is being used. But something has to
>> be done, not to have big kselftests, with many different subtests,
>> always marked as "failed" when validating new stable releases.
> 
> iptables-nft is supported in all of the existing stable kernels.

OK, then we should not have had the bug we had. I thought we were using
features that were not supported in v5.15.

>> Back to the modification to use 'iptables-legacy', maybe a kernel config
>> was missing, but the same kselftest, with the same list of kconfig to
>> add, was not working with the v5.15 kernel, while everything was OK with
>> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
>> will check if maybe an old kconfig option was not missing.
> 
> I suspect this is most likely kernel config missing, as it happened to Jakub.

Probably, yes. I just retried by testing a v5.15.148 kernel using the
kselftests from the net-next tree and forcing iptables-nft: I no longer
have the issue I had one year ago. Not sure why, we already had
NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and
similar, because we noticed some CI didn't have them?
Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

Hi,

Sorry for taking a while.

On Wed, Feb 07, 2024 at 12:33:44PM +0100, Matthieu Baerts wrote:
> Hi Pablo,
> 
> Thank you for your reply!
> 
> On 07/02/2024 10:49, Pablo Neira Ayuso wrote:
> > Hi Matthieu,
> > 
> > On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
> > [...]
> >> Good point, I understand it sounds better to use 'iptables-nft' in new
> >> kselftests. I should have added a bit of background and not just a link
> >> to this commit: at that time (around ~v6.4), we didn't need to force
> >> using 'iptables-legacy' on -net or net-next tree. But we needed that
> >> when testing kernels <= v5.15.
> >>
> >> When validating (old) stable kernels, the recommended practice is
> >> apparently [1] to use the kselftests from the last stable version, e.g.
> >> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
> >> kselftests are then supposed to support older kernels, e.g. by skipping
> >> some parts if a feature is not available. I didn't know about that
> >> before, and I don't know if all kselftests devs know about that.
> > 
> > We are sending backports to stable kernels, if one stable kernel
> > fails, then we have to fix it.
> 
> Do you validate stable kernels by running the kselftests from the same
> version (e.g. both from v5.15.x) or by using the kselftests from the
> last stable one (e.g. kernel v5.15.148 validated using the kselftests
> from v6.7.4)?

We have kselftests, but nftables/tests/shell probe for kernel
capabilities then it runs tests according to what the kernel
supports, this includes packet and control plane path tests. For
iptables, there are iptables-tests.py for the control plane path.

> >> I don't think that's easy to support old kernels, especially in the
> >> networking area, where some features/behaviours are not directly exposed
> >> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
> >> or use other (ugly?) workarounds [2] to predict what we are supposed to
> >> have, depending on the kernel that is being used. But something has to
> >> be done, not to have big kselftests, with many different subtests,
> >> always marked as "failed" when validating new stable releases.
> > 
> > iptables-nft is supported in all of the existing stable kernels.
> 
> OK, then we should not have had the bug we had. I thought we were using
> features that were not supported in v5.15.

I don't think so, iptables-nft supports the same features as
iptables-legacy.

> >> Back to the modification to use 'iptables-legacy', maybe a kernel config
> >> was missing, but the same kselftest, with the same list of kconfig to
> >> add, was not working with the v5.15 kernel, while everything was OK with
> >> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
> >> will check if maybe an old kconfig option was not missing.
> > 
> > I suspect this is most likely kernel config missing, as it happened to Jakub.
> 
> Probably, yes. I just retried by testing a v5.15.148 kernel using the
> kselftests from the net-next tree and forcing iptables-nft: I no longer
> have the issue I had one year ago. Not sure why, we already had
> NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and
> similar, because we noticed some CI didn't have them?
> Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Thanks. If you experience any issue, report back to netfilter-devel@

Re: [netfilter-core] [ANN] net-next is OPEN

[Matthieu Baerts]

Hi Pablo,

Thank you for your reply!

On 16/02/2024 16:38, Pablo Neira Ayuso wrote:
> Hi,
> 
> Sorry for taking a while.
> 
> On Wed, Feb 07, 2024 at 12:33:44PM +0100, Matthieu Baerts wrote:
>> Hi Pablo,
>>
>> Thank you for your reply!
>>
>> On 07/02/2024 10:49, Pablo Neira Ayuso wrote:
>>> Hi Matthieu,
>>>
>>> On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
>>> [...]
>>>> Good point, I understand it sounds better to use 'iptables-nft' in new
>>>> kselftests. I should have added a bit of background and not just a link
>>>> to this commit: at that time (around ~v6.4), we didn't need to force
>>>> using 'iptables-legacy' on -net or net-next tree. But we needed that
>>>> when testing kernels <= v5.15.
>>>>
>>>> When validating (old) stable kernels, the recommended practice is
>>>> apparently [1] to use the kselftests from the last stable version, e.g.
>>>> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
>>>> kselftests are then supposed to support older kernels, e.g. by skipping
>>>> some parts if a feature is not available. I didn't know about that
>>>> before, and I don't know if all kselftests devs know about that.
>>>
>>> We are sending backports to stable kernels, if one stable kernel
>>> fails, then we have to fix it.
>>
>> Do you validate stable kernels by running the kselftests from the same
>> version (e.g. both from v5.15.x) or by using the kselftests from the
>> last stable one (e.g. kernel v5.15.148 validated using the kselftests
>> from v6.7.4)?
> 
> We have kselftests, but nftables/tests/shell probe for kernel
> capabilities then it runs tests according to what the kernel
> supports, this includes packet and control plane path tests. For
> iptables, there are iptables-tests.py for the control plane path.

That's great! It is good to support all the different kernels.

>>>> I don't think that's easy to support old kernels, especially in the
>>>> networking area, where some features/behaviours are not directly exposed
>>>> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
>>>> or use other (ugly?) workarounds [2] to predict what we are supposed to
>>>> have, depending on the kernel that is being used. But something has to
>>>> be done, not to have big kselftests, with many different subtests,
>>>> always marked as "failed" when validating new stable releases.
>>>
>>> iptables-nft is supported in all of the existing stable kernels.
>>
>> OK, then we should not have had the bug we had. I thought we were using
>> features that were not supported in v5.15.
> 
> I don't think so, iptables-nft supports the same features as
> iptables-legacy.

We were probably unlucky and hit a kernel/userspace bug that has been
fixed in between, sorry for the noise!

>>>> Back to the modification to use 'iptables-legacy', maybe a kernel config
>>>> was missing, but the same kselftest, with the same list of kconfig to
>>>> add, was not working with the v5.15 kernel, while everything was OK with
>>>> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
>>>> will check if maybe an old kconfig option was not missing.
>>>
>>> I suspect this is most likely kernel config missing, as it happened to Jakub.
>>
>> Probably, yes. I just retried by testing a v5.15.148 kernel using the
>> kselftests from the net-next tree and forcing iptables-nft: I no longer
>> have the issue I had one year ago. Not sure why, we already had
>> NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and
>> similar, because we noticed some CI didn't have them?
>> Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!
> 
> Thanks. If you experience any issue, report back to netfilter-devel@

Will do, thank you!

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.