[ANN] net-next is OPEN

[ANN] net-next is OPEN

[Jakub Kicinski]

Hi,

net-next is open again, and accepting changes for v6.9.

Over the merge window I spent some time stringing together selftest
runner for netdev: https://netdev.bots.linux.dev/status.html
It is now connected to patchwork, meaning there should be a check
posted to each patch indicating whether selftests have passed or not.

If you authored any net or drivers/net selftests, please look around
and see if they are passing. If not - send patches or LMK what I need
to do to make them pass on the runner.. Make sure to scroll down to 
the "Not reporting to patchwork" section.

Re: [ANN] net-next is OPEN

[Simon Horman]

On Mon, Jan 22, 2024 at 09:16:12AM -0800, Jakub Kicinski wrote:
> Hi,
> 
> net-next is open again, and accepting changes for v6.9.
> 
> Over the merge window I spent some time stringing together selftest
> runner for netdev: https://netdev.bots.linux.dev/status.html
> It is now connected to patchwork, meaning there should be a check
> posted to each patch indicating whether selftests have passed or not.

Excellent, I'm really pleased to see this.

> If you authored any net or drivers/net selftests, please look around
> and see if they are passing. If not - send patches or LMK what I need
> to do to make them pass on the runner.. Make sure to scroll down to 
> the "Not reporting to patchwork" section.

Re: [ANN] net-next is OPEN

[Hangbin Liu]

On Mon, Jan 22, 2024 at 09:16:12AM -0800, Jakub Kicinski wrote:
> Hi,
> 
> net-next is open again, and accepting changes for v6.9.
> 
> Over the merge window I spent some time stringing together selftest
> runner for netdev: https://netdev.bots.linux.dev/status.html
> It is now connected to patchwork, meaning there should be a check
> posted to each patch indicating whether selftests have passed or not.

Cool! Does it group a couple of patches together and run the tests or
run for each patch separately?

> 
> If you authored any net or drivers/net selftests, please look around
> and see if they are passing. If not - send patches or LMK what I need
> to do to make them pass on the runner.. Make sure to scroll down to 
> the "Not reporting to patchwork" section.

Yes, this could push the selftest authors help checking all the failures.

[1] https://patchwork.kernel.org/project/netdevbpf/patch/20240122090544.1202880-1-liuhangbin@gmail.com/

Thanks
Hangbin

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Tue, 23 Jan 2024 16:45:55 +0800 Hangbin Liu wrote:
> > Over the merge window I spent some time stringing together selftest
> > runner for netdev: https://netdev.bots.linux.dev/status.html
> > It is now connected to patchwork, meaning there should be a check
> > posted to each patch indicating whether selftests have passed or not.  
> 
> Cool! Does it group a couple of patches together and run the tests or
> run for each patch separately?

It groups all patches outstanding in patchwork (which build cleanly).
I'm hoping we could also do HW testing using this setup, so batching
is a must. Not 100% sure it's the right direction for SW testing but
there's one way to find out :)

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/23/24 8:20 AM, Jakub Kicinski wrote:
> On Tue, 23 Jan 2024 16:45:55 +0800 Hangbin Liu wrote:
>>> Over the merge window I spent some time stringing together selftest
>>> runner for netdev: https://netdev.bots.linux.dev/status.html
>>> It is now connected to patchwork, meaning there should be a check
>>> posted to each patch indicating whether selftests have passed or not.  
>>
>> Cool! Does it group a couple of patches together and run the tests or
>> run for each patch separately?
> 
> It groups all patches outstanding in patchwork (which build cleanly).
> I'm hoping we could also do HW testing using this setup, so batching
> is a must. Not 100% sure it's the right direction for SW testing but
> there's one way to find out :)
> 

Really cool. Thanks for spending time to make this happen.

Scanning the tests I wrote, I think most of the failures are config
related. e.g., fib-nexthops.sh needs MPLS and those config settings are
enabled in tools/testing/selftests/net/config.

Another one, fcnal-test, needs nettest built. From
https://netdev-2.bots.linux.dev/vmksft-net/results/432660/36-fcnal-test-sh:

# which: no nettest in
(/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin)

It is in the Makefile, so how should that dependency be defined for this
new environment?

Finally, how can people replicate this setup to validate changes to get
everything work correctly?

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Tue, 23 Jan 2024 09:51:07 -0700 David Ahern wrote:
> Really cool. Thanks for spending time to make this happen.
> 
> Scanning the tests I wrote, I think most of the failures are config
> related. e.g., fib-nexthops.sh needs MPLS and those config settings are
> enabled in tools/testing/selftests/net/config.
> 
> Another one, fcnal-test, needs nettest built. From
> https://netdev-2.bots.linux.dev/vmksft-net/results/432660/36-fcnal-test-sh:
> 
> # which: no nettest in
> (/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin)
> 
> It is in the Makefile, so how should that dependency be defined for this
> new environment?

We run the tests with 

  make -C tools/testing/selftests TARGETS=net run_tests

the binary is there:

$ ls tools/testing/selftests/net/nettest
tools/testing/selftests/net/nettest

The script does:

if ! which nettest >/dev/null; then                                             
        PATH=$PWD:$PATH                                                         
        if ! which nettest >/dev/null; then                                     
                echo "'nettest' command not found; skipping tests"              
                exit $ksft_skip                                                 
        fi                                                                      
fi   

But that's not exactly the message that gets printed up top.
Is it coming from lib.sh? Not sure..

> Finally, how can people replicate this setup to validate changes to get
> everything work correctly?

You can run the exact scripts the CI runs, they are pretty standalone.
But they don't do any magic, really, I'd start by trying the steps I
outlined in
https://lore.kernel.org/all/20240123093834.23ea172a@kernel.org/
and if you can't repro the failure LMK, I'll share more instructions
on replicating exactly.

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/23/24 2:39 PM, Jakub Kicinski wrote:
> On Tue, 23 Jan 2024 09:51:07 -0700 David Ahern wrote:
>> Really cool. Thanks for spending time to make this happen.
>>
>> Scanning the tests I wrote, I think most of the failures are config
>> related. e.g., fib-nexthops.sh needs MPLS and those config settings are
>> enabled in tools/testing/selftests/net/config.
>>
>> Another one, fcnal-test, needs nettest built. From
>> https://netdev-2.bots.linux.dev/vmksft-net/results/432660/36-fcnal-test-sh:
>>
>> # which: no nettest in
>> (/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin)
>>
>> It is in the Makefile, so how should that dependency be defined for this
>> new environment?
> 
> We run the tests with 
> 
>   make -C tools/testing/selftests TARGETS=net run_tests

thanks for the tip to direclty run the tests.

> 
> the binary is there:
> 
> $ ls tools/testing/selftests/net/nettest
> tools/testing/selftests/net/nettest
> 
> The script does:
> 
> if ! which nettest >/dev/null; then                                             
>         PATH=$PWD:$PATH                                                         
>         if ! which nettest >/dev/null; then                                     
>                 echo "'nettest' command not found; skipping tests"              
>                 exit $ksft_skip                                                 
>         fi                                                                      
> fi   


This fixes the PATH problem:

diff --git a/tools/testing/selftests/net/fcnal-test.sh
b/tools/testing/selftests/net/fcnal-test.sh
index a8ad92850e63..0c5ac117155d 100755
--- a/tools/testing/selftests/net/fcnal-test.sh
+++ b/tools/testing/selftests/net/fcnal-test.sh
@@ -37,6 +37,8 @@
 #
 # server / client nomenclature relative to ns-A

+PATH=$PWD:$PWD/tools/testing/selftests/net:$PATH
+
 VERBOSE=0

 NSA_DEV=eth1

but given the permutations it does, the script needs lot more than 45
seconds. This does the trick, but not sure how to bump the timeout for a
specific test.

diff --git a/tools/testing/selftests/kselftest/runner.sh
b/tools/testing/selftests/kselftest/runner.sh
index cc9c846585f0..9e7dcb728249 100644
--- a/tools/testing/selftests/kselftest/runner.sh
+++ b/tools/testing/selftests/kselftest/runner.sh
@@ -9,7 +9,7 @@ export per_test_logging=

 # Defaults for "settings" file fields:
 # "timeout" how many seconds to let each test run before failing.
-export kselftest_default_timeout=45
+export kselftest_default_timeout=3600

 # There isn't a shell-agnostic way to find the path of a sourced file,
 # so we must rely on BASE_DIR being set to find other tools.

Re: [ANN] net-next is OPEN

[Paolo Abeni]

On Tue, 2024-01-23 at 22:20 -0700, David Ahern wrote:
[...]
> the script needs lot more than 45
> seconds. This does the trick, but not sure how to bump the timeout for a
> specific test.

You can set a test-group-specific timeout touching the 'settings' file
in the relevant directory. Note that for 'net' self-tests the timeout
is currently 3600 seconds (for each test).

AFAIK there is no way to set a single-test-specific timeout, without
running that specific test individually:

make install TARGETS=net
./kselftest_install/run_kselftest.sh -o <timeout in sec> -t net:fcnal-test.sh

Cheers,

Paolo

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 09:22:15 +0100 Paolo Abeni wrote:
> On Tue, 2024-01-23 at 22:20 -0700, David Ahern wrote:
> > thanks for the tip to direclty run the tests.

Actually, I simplified this one a little bit. What we do now is:

make -C tools/testing/selftests TARGETS={target} TEST_PROGS={prog} TEST_GEN_PROGS="" run_tests

This let's us run tests individually. This way we can put them in 
a FIFO and have multiple workers run them in parallel.

> > the script needs lot more than 45
> > seconds. This does the trick, but not sure how to bump the timeout for a
> > specific test.  
> 
> You can set a test-group-specific timeout touching the 'settings' file
> in the relevant directory. Note that for 'net' self-tests the timeout
> is currently 3600 seconds (for each test).
> 
> AFAIK there is no way to set a single-test-specific timeout, without
> running that specific test individually:
> 
> make install TARGETS=net
> ./kselftest_install/run_kselftest.sh -o <timeout in sec> -t net:fcnal-test.sh

Yet another thing run_kselftest.sh can do that make can't :(
We should probably bite the bullet and switch to it at some point.

David, I applied your diff locally, hopefully I did it in time for the
7am Pacific run to have it included :)

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 07:07:55 -0800 Jakub Kicinski wrote:
> David, I applied your diff locally, hopefully I did it in time for the
> 7am Pacific run to have it included :)

This is the latest run:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout

the nettest warning is indeed gone, but the failures are the same:

$ grep FAIL stdout 
# TEST: ping local, VRF bind - VRF IP                                           [FAIL]
# TEST: ping local, device bind - ns-A IP                                       [FAIL]
# TEST: ping local, VRF bind - VRF IP                                           [FAIL]
# TEST: ping local, device bind - ns-A IP                                       [FAIL]

:(

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/24/24 9:19 AM, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 07:07:55 -0800 Jakub Kicinski wrote:
>> David, I applied your diff locally, hopefully I did it in time for the
>> 7am Pacific run to have it included :)
> 
> This is the latest run:
> 
> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout
> 
> the nettest warning is indeed gone, but the failures are the same:

yep, I will send a formal patch. I see the timeout is high enough, so
good there.


> 
> $ grep FAIL stdout 
> # TEST: ping local, VRF bind - VRF IP                                           [FAIL]
> # TEST: ping local, device bind - ns-A IP                                       [FAIL]
> # TEST: ping local, VRF bind - VRF IP                                           [FAIL]
> # TEST: ping local, device bind - ns-A IP                                       [FAIL]
> 
> :(

known problems. I can disable the tests for now so we avoid regressions,
and add to the TO-DO list for someone with time.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 09:35:09 -0700 David Ahern wrote:
> > This is the latest run:
> > 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout
> > 
> > the nettest warning is indeed gone, but the failures are the same:  
> 
> yep, I will send a formal patch. I see the timeout is high enough, so
> good there.

Well, kinda, to be honest I did bump the time to 4000s locally.
The runtime of the entire net suite 1h 10min - that's pretty much
the runtime of this one test :) The VMs run the tests without
HW virtualization, so they are a bit slower, but it'd be nice
if no local hacks were necessary. 

I haven't sent a patch to bump it because it may make more sense
to split the test into multiple. But as a stop gap we can as well
bump the timeout.

> > $ grep FAIL stdout 
> > # TEST: ping local, VRF bind - VRF IP                 [FAIL]
> > # TEST: ping local, device bind - ns-A IP             [FAIL]
> > # TEST: ping local, VRF bind - VRF IP                 [FAIL]
> > # TEST: ping local, device bind - ns-A IP             [FAIL]
> > 
> > :(  
> 
> known problems. I can disable the tests for now so we avoid regressions,
> and add to the TO-DO list for someone with time.

Sounds good!

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/24/24 9:59 AM, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 09:35:09 -0700 David Ahern wrote:
>>> This is the latest run:
>>>
>>> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/1-fcnal-test-sh/stdout
>>>
>>> the nettest warning is indeed gone, but the failures are the same:  
>>
>> yep, I will send a formal patch. I see the timeout is high enough, so
>> good there.
> 
> Well, kinda, to be honest I did bump the time to 4000s locally.
> The runtime of the entire net suite 1h 10min - that's pretty much
> the runtime of this one test :) The VMs run the tests without

one *script* = 900+ tests. :-)


> HW virtualization, so they are a bit slower, but it'd be nice
> if no local hacks were necessary. 
> 
> I haven't sent a patch to bump it because it may make more sense
> to split the test into multiple. But as a stop gap we can as well
> bump the timeout.

The script has the tests in groups and each group can be run in parallel
(with Hangbin's recent namespace changes). I can look at splitting that
script into many (or write wrappers to run specific groups), but even
then most of those test groups need more than 45 seconds. There are lot
of permutations covered (with and without vrf, different address types,
different uapis, ...).

> 
>>> $ grep FAIL stdout 
>>> # TEST: ping local, VRF bind - VRF IP                 [FAIL]
>>> # TEST: ping local, device bind - ns-A IP             [FAIL]
>>> # TEST: ping local, VRF bind - VRF IP                 [FAIL]
>>> # TEST: ping local, device bind - ns-A IP             [FAIL]
>>>
>>> :(  
>>
>> known problems. I can disable the tests for now so we avoid regressions,
>> and add to the TO-DO list for someone with time.

Sent a PR to fix a few things. I did not have to disable any tests -
everything passes cleanly with the changes. If tests fail after those
are applied, let's compare OS environments - maybe some sysctl is
enabled or disabled (or a CONFIG) in your environment.

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/24/24 2:48 PM, David Ahern wrote:
>>
>>>> $ grep FAIL stdout 
>>>> # TEST: ping local, VRF bind - VRF IP                 [FAIL]
>>>> # TEST: ping local, device bind - ns-A IP             [FAIL]
>>>> # TEST: ping local, VRF bind - VRF IP                 [FAIL]
>>>> # TEST: ping local, device bind - ns-A IP             [FAIL]
>>>>
>>>> :(  

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout

still shows those 4 tests failing. since they pass on default Ubuntu
23.10, I need some information about the setup. What is the OS image in
use and any known changes to the sysctl settings?

Can I get `sysctl net > /tmp/sysctl.net` ? I will compare to Ubuntu and
see if I can figure out the difference and get those added to the script.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Fri, 26 Jan 2024 17:56:26 -0700 David Ahern wrote:
> On 1/24/24 2:48 PM, David Ahern wrote:
> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout
> 
> still shows those 4 tests failing. since they pass on default Ubuntu
> 23.10, I need some information about the setup. What is the OS image in
> use and any known changes to the sysctl settings?
> 
> Can I get `sysctl net > /tmp/sysctl.net` ? I will compare to Ubuntu and
> see if I can figure out the difference and get those added to the script.

Here's a boot and run of the command (not sure how to export the file
form the VM so I captured all of stdout):

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/sysctl-for-david

The OS is Amazon Linux, annoyingly.

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/26/24 6:13 PM, Jakub Kicinski wrote:
> On Fri, 26 Jan 2024 17:56:26 -0700 David Ahern wrote:
>> On 1/24/24 2:48 PM, David Ahern wrote:
>> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout
>>
>> still shows those 4 tests failing. since they pass on default Ubuntu
>> 23.10, I need some information about the setup. What is the OS image in
>> use and any known changes to the sysctl settings?
>>
>> Can I get `sysctl net > /tmp/sysctl.net` ? I will compare to Ubuntu and
>> see if I can figure out the difference and get those added to the script.
> 
> Here's a boot and run of the command (not sure how to export the file
> form the VM so I captured all of stdout):
> 
> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/sysctl-for-david
> 
> The OS is Amazon Linux, annoyingly.

It's a bug in that version of iputils ping. It sets the BINDTODEVICE and
then resets it because the source address is not set on the command line
(it should not be required).

There are a couple of workarounds - one which might not age well (ie.,
amazon linux moving forward to newer packages -I <addr> -I <vrf>) and
one that bypasses the purpose of the test (ip vrf exec)).

Re: [ANN] net-next is OPEN

[Paolo Abeni]

On Sat, 2024-01-27 at 21:26 -0700, David Ahern wrote:
> On 1/26/24 6:13 PM, Jakub Kicinski wrote:
> > On Fri, 26 Jan 2024 17:56:26 -0700 David Ahern wrote:
> > > On 1/24/24 2:48 PM, David Ahern wrote:
> > > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/438381/1-fcnal-test-sh/stdout
> > > 
> > > still shows those 4 tests failing. since they pass on default Ubuntu
> > > 23.10, I need some information about the setup. What is the OS image in
> > > use and any known changes to the sysctl settings?
> > > 
> > > Can I get `sysctl net > /tmp/sysctl.net` ? I will compare to Ubuntu and
> > > see if I can figure out the difference and get those added to the script.
> > 
> > Here's a boot and run of the command (not sure how to export the file
> > form the VM so I captured all of stdout):
> > 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/sysctl-for-david
> > 
> > The OS is Amazon Linux, annoyingly.
> 
> It's a bug in that version of iputils ping. It sets the BINDTODEVICE and
> then resets it because the source address is not set on the command line
> (it should not be required).
> 
> There are a couple of workarounds - one which might not age well (ie.,
> amazon linux moving forward to newer packages -I <addr> -I <vrf>) and
> one that bypasses the purpose of the test (ip vrf exec)).

Could the script validate the 'ping' command WRT the bad behavior/bug
and  eventually skip the related tests?

Cheers,

Paolo

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Mon, 29 Jan 2024 10:23:15 +0100 Paolo Abeni wrote:
> > It's a bug in that version of iputils ping. It sets the BINDTODEVICE and
> > then resets it because the source address is not set on the command line
> > (it should not be required).
> > 
> > There are a couple of workarounds - one which might not age well (ie.,
> > amazon linux moving forward to newer packages -I <addr> -I <vrf>) and
> > one that bypasses the purpose of the test (ip vrf exec)).  
> 
> Could the script validate the 'ping' command WRT the bad behavior/bug
> and  eventually skip the related tests?

Skipping if the package has a bug would be best, if we can figure that
out. The latest version is fixed, right? I can build it locally, just
like pretty much everything else..

Re: [ANN] net-next is OPEN

[David Ahern]

On 1/29/24 8:03 AM, Jakub Kicinski wrote:
> On Mon, 29 Jan 2024 10:23:15 +0100 Paolo Abeni wrote:
>>> It's a bug in that version of iputils ping. It sets the BINDTODEVICE and
>>> then resets it because the source address is not set on the command line
>>> (it should not be required).
>>>
>>> There are a couple of workarounds - one which might not age well (ie.,
>>> amazon linux moving forward to newer packages -I <addr> -I <vrf>) and
>>> one that bypasses the purpose of the test (ip vrf exec)).  
>>
>> Could the script validate the 'ping' command WRT the bad behavior/bug
>> and  eventually skip the related tests?
> 
> Skipping if the package has a bug would be best, if we can figure that
> out. The latest version is fixed, right? I can build it locally, just
> like pretty much everything else..

yes, that is why Ubuntu 23.10 passes. I downloaded iputils, built ping
locally at the Amazon release version and did a side by side comparison
of behavior to verify it is the ping command.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Mon, 29 Jan 2024 08:32:33 -0700 David Ahern wrote:
> >> Could the script validate the 'ping' command WRT the bad behavior/bug
> >> and  eventually skip the related tests?  
> > 
> > Skipping if the package has a bug would be best, if we can figure that
> > out. The latest version is fixed, right? I can build it locally, just
> > like pretty much everything else..  
> 
> yes, that is why Ubuntu 23.10 passes. I downloaded iputils, built ping
> locally at the Amazon release version and did a side by side comparison
> of behavior to verify it is the ping command.

Built iputils from git manually.
Could not figure out how to convince mason not to insert /local/ into
the install dir path so I copied the binaries manually :/ Seems to work,
tho.

Re: [ANN] net-next is OPEN

[Willem de Bruijn]

David Ahern wrote:
> On 1/23/24 8:20 AM, Jakub Kicinski wrote:
> > On Tue, 23 Jan 2024 16:45:55 +0800 Hangbin Liu wrote:
> >>> Over the merge window I spent some time stringing together selftest
> >>> runner for netdev: https://netdev.bots.linux.dev/status.html
> >>> It is now connected to patchwork, meaning there should be a check
> >>> posted to each patch indicating whether selftests have passed or not.  
> >>
> >> Cool! Does it group a couple of patches together and run the tests or
> >> run for each patch separately?
> > 
> > It groups all patches outstanding in patchwork (which build cleanly).
> > I'm hoping we could also do HW testing using this setup, so batching
> > is a must. Not 100% sure it's the right direction for SW testing but
> > there's one way to find out :)
> > 
> 
> Really cool. Thanks for spending time to make this happen.

Just to add to the choir: this is fantastic, thanks!

Hopefully it will encourage people to add kselftests, kunit tests or
other kinds that now get continuous coverage.

Going through the failing ksft-net series on
https://netdev.bots.linux.dev/status.html, all the tests I'm
responsible seem to be passing.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 10:59:36 -0500 Willem de Bruijn wrote:
> David Ahern wrote:
> > On 1/23/24 8:20 AM, Jakub Kicinski wrote:  
> > > It groups all patches outstanding in patchwork (which build cleanly).
> > > I'm hoping we could also do HW testing using this setup, so batching
> > > is a must. Not 100% sure it's the right direction for SW testing but
> > > there's one way to find out :)
> > >   
> > 
> > Really cool. Thanks for spending time to make this happen.  
> 
> Just to add to the choir: this is fantastic, thanks!
> 
> Hopefully it will encourage people to add kselftests, kunit tests or
> other kinds that now get continuous coverage.

Fingers crossed :)

> Going through the failing ksft-net series on
> https://netdev.bots.linux.dev/status.html, all the tests I'm
> responsible seem to be passing.

Here's a more handy link filtered down to failures (clicking on 
the test counts links here):

https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0

I have been attributing the udpg[rs]o and timestamp tests to you,
but I haven't actually checked.. are they not yours? :)

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
> > Going through the failing ksft-net series on
> > https://netdev.bots.linux.dev/status.html, all the tests I'm
> > responsible seem to be passing.  
> 
> Here's a more handy link filtered down to failures (clicking on 
> the test counts links here):
> 
> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> 
> I have been attributing the udpg[rs]o and timestamp tests to you,
> but I haven't actually checked.. are they not yours? :)

Ah, BTW, a major source of failures seems to be that iptables is
mapping to nftables on the executor. And either nftables doesn't
support the functionality the tests expect or we're missing configs :(
E.g. the TTL module.

Re: [ANN] net-next is OPEN

[Matthieu Baerts]

Hello,

24 Jan 2024 17:01:24 Jakub Kicinski <kuba@kernel.org>:

> On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
>>> Going through the failing ksft-net series on
>>> https://netdev.bots.linux.dev/status.html, all the tests I'm
>>> responsible seem to be passing. 
>>
>> Here's a more handy link filtered down to failures (clicking on
>> the test counts links here):
>>
>> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
>>
>> I have been attributing the udpg[rs]o and timestamp tests to you,
>> but I haven't actually checked.. are they not yours? :)
>
> Ah, BTW, a major source of failures seems to be that iptables is
> mapping to nftables on the executor. And either nftables doesn't
> support the functionality the tests expect or we're missing configs :(
> E.g. the TTL module.

I don't know if it is the same issue, but for MPTCP, we use
'iptables-legacy' if available.

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

Cheers,
Matt

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 18:35:14 +0000 (GMT) Matthieu Baerts wrote:
> > Ah, BTW, a major source of failures seems to be that iptables is
> > mapping to nftables on the executor. And either nftables doesn't
> > support the functionality the tests expect or we're missing configs :(
> > E.g. the TTL module.  
> 
> I don't know if it is the same issue, but for MPTCP, we use
> 'iptables-legacy' if available.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

Great! Thanks for the pointer. I installed the packages now,
so folks should be able to fix up their scripts.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 06:35:14PM +0000, Matthieu Baerts wrote:
> Hello,
> 
> 24 Jan 2024 17:01:24 Jakub Kicinski <kuba@kernel.org>:
> 
> > On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
> >>> Going through the failing ksft-net series on
> >>> https://netdev.bots.linux.dev/status.html, all the tests I'm
> >>> responsible seem to be passing. 
> >>
> >> Here's a more handy link filtered down to failures (clicking on
> >> the test counts links here):
> >>
> >> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> >>
> >> I have been attributing the udpg[rs]o and timestamp tests to you,
> >> but I haven't actually checked.. are they not yours? :)
> >
> > Ah, BTW, a major source of failures seems to be that iptables is
> > mapping to nftables on the executor. And either nftables doesn't
> > support the functionality the tests expect or we're missing configs :(
> > E.g. the TTL module.
> 
> I don't know if it is the same issue, but for MPTCP, we use
> 'iptables-legacy' if available.
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400

I'd suggest you do the other way around, first check if iptables-nft
is available, otherwise fall back to iptables-nft

commit refers to 5.15 already have iptables-nft support, it should
work out of the box.

Re: [netfilter-core] [ANN] net-next is OPEN

[Matthieu Baerts]

Hi Pablo,

Thank you for your reply!

On 24/01/2024 20:18, Pablo Neira Ayuso wrote:
> On Wed, Jan 24, 2024 at 06:35:14PM +0000, Matthieu Baerts wrote:
>> Hello,
>>
>> 24 Jan 2024 17:01:24 Jakub Kicinski <kuba@kernel.org>:
>>
>>> On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
>>>>> Going through the failing ksft-net series on
>>>>> https://netdev.bots.linux.dev/status.html, all the tests I'm
>>>>> responsible seem to be passing. 
>>>>
>>>> Here's a more handy link filtered down to failures (clicking on
>>>> the test counts links here):
>>>>
>>>> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
>>>>
>>>> I have been attributing the udpg[rs]o and timestamp tests to you,
>>>> but I haven't actually checked.. are they not yours? :)
>>>
>>> Ah, BTW, a major source of failures seems to be that iptables is
>>> mapping to nftables on the executor. And either nftables doesn't
>>> support the functionality the tests expect or we're missing configs :(
>>> E.g. the TTL module.
>>
>> I don't know if it is the same issue, but for MPTCP, we use
>> 'iptables-legacy' if available.
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=0c4cd3f86a400
> 
> I'd suggest you do the other way around, first check if iptables-nft
> is available, otherwise fall back to iptables-nft
> 
> commit refers to 5.15 already have iptables-nft support, it should
> work out of the box.

Good point, I understand it sounds better to use 'iptables-nft' in new
kselftests. I should have added a bit of background and not just a link
to this commit: at that time (around ~v6.4), we didn't need to force
using 'iptables-legacy' on -net or net-next tree. But we needed that
when testing kernels <= v5.15.

When validating (old) stable kernels, the recommended practice is
apparently [1] to use the kselftests from the last stable version, e.g.
using the kselftests from v6.7.4 when validating kernel v5.15.148. The
kselftests are then supposed to support older kernels, e.g. by skipping
some parts if a feature is not available. I didn't know about that
before, and I don't know if all kselftests devs know about that.

I don't think that's easy to support old kernels, especially in the
networking area, where some features/behaviours are not directly exposed
to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
or use other (ugly?) workarounds [2] to predict what we are supposed to
have, depending on the kernel that is being used. But something has to
be done, not to have big kselftests, with many different subtests,
always marked as "failed" when validating new stable releases.

Back to the modification to use 'iptables-legacy', maybe a kernel config
was missing, but the same kselftest, with the same list of kconfig to
add, was not working with the v5.15 kernel, while everything was OK with
a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
will check if maybe an old kconfig option was not missing.

[1] https://lore.kernel.org/stable/ZAG8dla274kYfxoK@kroah.com/
[2]
https://lore.kernel.org/netdev/20230609-upstream-net-20230610-mptcp-selftests-support-old-kernels-part-3-v1-0-2896fe2ee8a3@tessares.net

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

Hi Matthieu,

On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
[...]
> Good point, I understand it sounds better to use 'iptables-nft' in new
> kselftests. I should have added a bit of background and not just a link
> to this commit: at that time (around ~v6.4), we didn't need to force
> using 'iptables-legacy' on -net or net-next tree. But we needed that
> when testing kernels <= v5.15.
> 
> When validating (old) stable kernels, the recommended practice is
> apparently [1] to use the kselftests from the last stable version, e.g.
> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
> kselftests are then supposed to support older kernels, e.g. by skipping
> some parts if a feature is not available. I didn't know about that
> before, and I don't know if all kselftests devs know about that.

We are sending backports to stable kernels, if one stable kernel
fails, then we have to fix it.

> I don't think that's easy to support old kernels, especially in the
> networking area, where some features/behaviours are not directly exposed
> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
> or use other (ugly?) workarounds [2] to predict what we are supposed to
> have, depending on the kernel that is being used. But something has to
> be done, not to have big kselftests, with many different subtests,
> always marked as "failed" when validating new stable releases.

iptables-nft is supported in all of the existing stable kernels.

> Back to the modification to use 'iptables-legacy', maybe a kernel config
> was missing, but the same kselftest, with the same list of kconfig to
> add, was not working with the v5.15 kernel, while everything was OK with
> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
> will check if maybe an old kconfig option was not missing.

I suspect this is most likely kernel config missing, as it happened to Jakub.

Thanks.

Re: [netfilter-core] [ANN] net-next is OPEN

[Matthieu Baerts]

Hi Pablo,

Thank you for your reply!

On 07/02/2024 10:49, Pablo Neira Ayuso wrote:
> Hi Matthieu,
> 
> On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
> [...]
>> Good point, I understand it sounds better to use 'iptables-nft' in new
>> kselftests. I should have added a bit of background and not just a link
>> to this commit: at that time (around ~v6.4), we didn't need to force
>> using 'iptables-legacy' on -net or net-next tree. But we needed that
>> when testing kernels <= v5.15.
>>
>> When validating (old) stable kernels, the recommended practice is
>> apparently [1] to use the kselftests from the last stable version, e.g.
>> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
>> kselftests are then supposed to support older kernels, e.g. by skipping
>> some parts if a feature is not available. I didn't know about that
>> before, and I don't know if all kselftests devs know about that.
> 
> We are sending backports to stable kernels, if one stable kernel
> fails, then we have to fix it.

Do you validate stable kernels by running the kselftests from the same
version (e.g. both from v5.15.x) or by using the kselftests from the
last stable one (e.g. kernel v5.15.148 validated using the kselftests
from v6.7.4)?

>> I don't think that's easy to support old kernels, especially in the
>> networking area, where some features/behaviours are not directly exposed
>> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
>> or use other (ugly?) workarounds [2] to predict what we are supposed to
>> have, depending on the kernel that is being used. But something has to
>> be done, not to have big kselftests, with many different subtests,
>> always marked as "failed" when validating new stable releases.
> 
> iptables-nft is supported in all of the existing stable kernels.

OK, then we should not have had the bug we had. I thought we were using
features that were not supported in v5.15.

>> Back to the modification to use 'iptables-legacy', maybe a kernel config
>> was missing, but the same kselftest, with the same list of kconfig to
>> add, was not working with the v5.15 kernel, while everything was OK with
>> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
>> will check if maybe an old kconfig option was not missing.
> 
> I suspect this is most likely kernel config missing, as it happened to Jakub.

Probably, yes. I just retried by testing a v5.15.148 kernel using the
kselftests from the net-next tree and forcing iptables-nft: I no longer
have the issue I had one year ago. Not sure why, we already had
NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and
similar, because we noticed some CI didn't have them?
Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

Hi,

Sorry for taking a while.

On Wed, Feb 07, 2024 at 12:33:44PM +0100, Matthieu Baerts wrote:
> Hi Pablo,
> 
> Thank you for your reply!
> 
> On 07/02/2024 10:49, Pablo Neira Ayuso wrote:
> > Hi Matthieu,
> > 
> > On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
> > [...]
> >> Good point, I understand it sounds better to use 'iptables-nft' in new
> >> kselftests. I should have added a bit of background and not just a link
> >> to this commit: at that time (around ~v6.4), we didn't need to force
> >> using 'iptables-legacy' on -net or net-next tree. But we needed that
> >> when testing kernels <= v5.15.
> >>
> >> When validating (old) stable kernels, the recommended practice is
> >> apparently [1] to use the kselftests from the last stable version, e.g.
> >> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
> >> kselftests are then supposed to support older kernels, e.g. by skipping
> >> some parts if a feature is not available. I didn't know about that
> >> before, and I don't know if all kselftests devs know about that.
> > 
> > We are sending backports to stable kernels, if one stable kernel
> > fails, then we have to fix it.
> 
> Do you validate stable kernels by running the kselftests from the same
> version (e.g. both from v5.15.x) or by using the kselftests from the
> last stable one (e.g. kernel v5.15.148 validated using the kselftests
> from v6.7.4)?

We have kselftests, but nftables/tests/shell probe for kernel
capabilities then it runs tests according to what the kernel
supports, this includes packet and control plane path tests. For
iptables, there are iptables-tests.py for the control plane path.

> >> I don't think that's easy to support old kernels, especially in the
> >> networking area, where some features/behaviours are not directly exposed
> >> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
> >> or use other (ugly?) workarounds [2] to predict what we are supposed to
> >> have, depending on the kernel that is being used. But something has to
> >> be done, not to have big kselftests, with many different subtests,
> >> always marked as "failed" when validating new stable releases.
> > 
> > iptables-nft is supported in all of the existing stable kernels.
> 
> OK, then we should not have had the bug we had. I thought we were using
> features that were not supported in v5.15.

I don't think so, iptables-nft supports the same features as
iptables-legacy.

> >> Back to the modification to use 'iptables-legacy', maybe a kernel config
> >> was missing, but the same kselftest, with the same list of kconfig to
> >> add, was not working with the v5.15 kernel, while everything was OK with
> >> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
> >> will check if maybe an old kconfig option was not missing.
> > 
> > I suspect this is most likely kernel config missing, as it happened to Jakub.
> 
> Probably, yes. I just retried by testing a v5.15.148 kernel using the
> kselftests from the net-next tree and forcing iptables-nft: I no longer
> have the issue I had one year ago. Not sure why, we already had
> NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and
> similar, because we noticed some CI didn't have them?
> Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!

Thanks. If you experience any issue, report back to netfilter-devel@

Re: [netfilter-core] [ANN] net-next is OPEN

[Matthieu Baerts]

Hi Pablo,

Thank you for your reply!

On 16/02/2024 16:38, Pablo Neira Ayuso wrote:
> Hi,
> 
> Sorry for taking a while.
> 
> On Wed, Feb 07, 2024 at 12:33:44PM +0100, Matthieu Baerts wrote:
>> Hi Pablo,
>>
>> Thank you for your reply!
>>
>> On 07/02/2024 10:49, Pablo Neira Ayuso wrote:
>>> Hi Matthieu,
>>>
>>> On Tue, Feb 06, 2024 at 07:31:44PM +0100, Matthieu Baerts wrote:
>>> [...]
>>>> Good point, I understand it sounds better to use 'iptables-nft' in new
>>>> kselftests. I should have added a bit of background and not just a link
>>>> to this commit: at that time (around ~v6.4), we didn't need to force
>>>> using 'iptables-legacy' on -net or net-next tree. But we needed that
>>>> when testing kernels <= v5.15.
>>>>
>>>> When validating (old) stable kernels, the recommended practice is
>>>> apparently [1] to use the kselftests from the last stable version, e.g.
>>>> using the kselftests from v6.7.4 when validating kernel v5.15.148. The
>>>> kselftests are then supposed to support older kernels, e.g. by skipping
>>>> some parts if a feature is not available. I didn't know about that
>>>> before, and I don't know if all kselftests devs know about that.
>>>
>>> We are sending backports to stable kernels, if one stable kernel
>>> fails, then we have to fix it.
>>
>> Do you validate stable kernels by running the kselftests from the same
>> version (e.g. both from v5.15.x) or by using the kselftests from the
>> last stable one (e.g. kernel v5.15.148 validated using the kselftests
>> from v6.7.4)?
> 
> We have kselftests, but nftables/tests/shell probe for kernel
> capabilities then it runs tests according to what the kernel
> supports, this includes packet and control plane path tests. For
> iptables, there are iptables-tests.py for the control plane path.

That's great! It is good to support all the different kernels.

>>>> I don't think that's easy to support old kernels, especially in the
>>>> networking area, where some features/behaviours are not directly exposed
>>>> to the userspace. Some MPTCP kselftests have to look at /proc/kallsyms
>>>> or use other (ugly?) workarounds [2] to predict what we are supposed to
>>>> have, depending on the kernel that is being used. But something has to
>>>> be done, not to have big kselftests, with many different subtests,
>>>> always marked as "failed" when validating new stable releases.
>>>
>>> iptables-nft is supported in all of the existing stable kernels.
>>
>> OK, then we should not have had the bug we had. I thought we were using
>> features that were not supported in v5.15.
> 
> I don't think so, iptables-nft supports the same features as
> iptables-legacy.

We were probably unlucky and hit a kernel/userspace bug that has been
fixed in between, sorry for the noise!

>>>> Back to the modification to use 'iptables-legacy', maybe a kernel config
>>>> was missing, but the same kselftest, with the same list of kconfig to
>>>> add, was not working with the v5.15 kernel, while everything was OK with
>>>> a v6.4 one. With 'iptables-legacy', the test was running fine on both. I
>>>> will check if maybe an old kconfig option was not missing.
>>>
>>> I suspect this is most likely kernel config missing, as it happened to Jakub.
>>
>> Probably, yes. I just retried by testing a v5.15.148 kernel using the
>> kselftests from the net-next tree and forcing iptables-nft: I no longer
>> have the issue I had one year ago. Not sure why, we already had
>> NFT_COMPAT=m back then. Maybe because we recently added IP_NF_FILTER and
>> similar, because we noticed some CI didn't have them?
>> Anyway, I will then switch back to iptables-nft. Thanks for the suggestion!
> 
> Thanks. If you experience any issue, report back to netfilter-devel@

Will do, thank you!

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 09:01:23AM -0800, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 08:22:55 -0800 Jakub Kicinski wrote:
> > > Going through the failing ksft-net series on
> > > https://netdev.bots.linux.dev/status.html, all the tests I'm
> > > responsible seem to be passing.  
> > 
> > Here's a more handy link filtered down to failures (clicking on 
> > the test counts links here):
> > 
> > https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> > 
> > I have been attributing the udpg[rs]o and timestamp tests to you,
> > but I haven't actually checked.. are they not yours? :)
> 
> Ah, BTW, a major source of failures seems to be that iptables is
> mapping to nftables on the executor. And either nftables doesn't
> support the functionality the tests expect or we're missing configs :(
> E.g. the TTL module.

I could only find in the listing above this:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout

which shows:

 ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory

which seems like setup is broken, ie. it could not find libxt_conntrack.so

What is the issue?

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 20:16:39 +0100 Pablo Neira Ayuso wrote:
> > Ah, BTW, a major source of failures seems to be that iptables is
> > mapping to nftables on the executor. And either nftables doesn't
> > support the functionality the tests expect or we're missing configs :(
> > E.g. the TTL module.  
> 
> I could only find in the listing above this:

Thanks for taking a look!

> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout
> 
> which shows:
> 
>  ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory
> 
> which seems like setup is broken, ie. it could not find libxt_conntrack.so

Hm, odd, it's there:

$ ls /lib64/xtables/libxt_conntrack.so
/lib64/xtables/libxt_conntrack.so

but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 
is in it (normal loaded always scans system paths)!

> What is the issue?

A lot of the tests print warning messages like the ones below.
Some of them pass some of them fail. Tweaking the kernel config
to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_*
are included seem to have made no difference, which I concluded was
because iptables CLI uses nf_tables here by default..

[435321]$ grep -nrI "Warning: Extension" .
./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./6-fib-tests-sh/stdout:308:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./6-fib-tests-sh/stdout:316:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./6-fib-tests-sh/stdout:319:# Warning: Extension MARK revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:12:# No GRO                                  Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:13:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:14:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:16:# GRO frag list                           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:17:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:18:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:19:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:21:# Warning: Extension DNAT revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:23:# GRO fwd                                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:24:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:38:# GRO frag list over UDP tunnel           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:39:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:40:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:41:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:43:# Warning: Extension DNAT revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:45:# GRO fwd over UDP tunnel                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:46:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:61:# No GRO                                  Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:62:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:63:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:65:# GRO frag list                           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:66:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:67:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:68:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:72:# GRO fwd                                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:73:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:88:# GRO frag list over UDP tunnel           Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:89:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:90:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:91:# Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:95:# GRO fwd over UDP tunnel                 Warning: Extension udp revision 0 not supported, missing kernel module?
./18-udpgro-fwd-sh/stdout:96:# Warning: Extension udp revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:17:# Warning: Extension length revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:19:# Warning: Extension length revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:22:# Warning: Extension length revision 0 not supported, missing kernel module?
./37-big-tcp-sh/stdout:24:# Warning: Extension length revision 0 not supported, missing kernel module?
./56-xfrm-policy-sh/stdout:11:# Warning: Extension policy revision 0 not supported, missing kernel module?
./56-xfrm-policy-sh/stdout:13:# Warning: Extension policy revision 0 not supported, missing kernel module?
./54-amt-sh/stdout:94:# Warning: Extension TTL revision 0 not supported, missing kernel module?

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 11:40:57AM -0800, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 20:16:39 +0100 Pablo Neira Ayuso wrote:
> > > Ah, BTW, a major source of failures seems to be that iptables is
> > > mapping to nftables on the executor. And either nftables doesn't
> > > support the functionality the tests expect or we're missing configs :(
> > > E.g. the TTL module.  
> > 
> > I could only find in the listing above this:
> 
> Thanks for taking a look!
> 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435141/37-ip-defrag-sh/stdout
> > 
> > which shows:
> > 
> >  ip6tables v1.8.8 (nf_tables): Couldn't load match `conntrack':No such file or directory
> > 
> > which seems like setup is broken, ie. it could not find libxt_conntrack.so
> 
> Hm, odd, it's there:
> 
> $ ls /lib64/xtables/libxt_conntrack.so
> /lib64/xtables/libxt_conntrack.so
>
> but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 
> is in it (normal loaded always scans system paths)!

Could you also check your ./configure output for iptables? It shows
the directory where the .so file are search and found:

  ...
  Xtables extension directory:          /usr/lib/xtables

> > What is the issue?
> 
> A lot of the tests print warning messages like the ones below.
> Some of them pass some of them fail. Tweaking the kernel config
> to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_*
> are included seem to have made no difference, which I concluded was
> because iptables CLI uses nf_tables here by default..

Please, check if the symlink refers to -legacy or -nft via:

$ ls -la /usr/sbin/iptables

> [435321]$ grep -nrI "Warning: Extension" .
> ./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?

This could come from either legacy or nftables:

libxtables/xtables.c:                                   "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
iptables/nft.c:                         "Warning: Extension %s revision 0 not supported, missing kernel module?\n",

both have the same error.

if that is the nftables backend, it might be also that .config is
missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
options.

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 21:02:16 +0100 Pablo Neira Ayuso wrote:
> On Wed, Jan 24, 2024 at 11:40:57AM -0800, Jakub Kicinski wrote:
> > Hm, odd, it's there:
> > 
> > $ ls /lib64/xtables/libxt_conntrack.so
> > /lib64/xtables/libxt_conntrack.so
> >
> > but I set a custom LD_LIBRARY_PATH, let me make sure that /lib64 
> > is in it (normal loaded always scans system paths)!  
> 
> Could you also check your ./configure output for iptables? It shows
> the directory where the .so file are search and found:
> 
>   ...
>   Xtables extension directory:          /usr/lib/xtables

I'm using the OS iptables, I was hoping that for net/ tests
OS iptables should be good enough :S Is there a way to get
the info you're after?

$ iptables -V 
iptables v1.8.8 (nf_tables)

> > > What is the issue?  
> > 
> > A lot of the tests print warning messages like the ones below.
> > Some of them pass some of them fail. Tweaking the kernel config
> > to make sure the right CONFIG_IP_NF_TARGET_* and CONFIG_IP_NF_MATCH_*
> > are included seem to have made no difference, which I concluded was
> > because iptables CLI uses nf_tables here by default..  
> 
> Please, check if the symlink refers to -legacy or -nft via:
> 
> $ ls -la /usr/sbin/iptables

Ah, neat:

$ ls -la /etc/alternatives/iptables
lrwxrwxrwx. 1 root root 22 Jan  7 22:10 /etc/alternatives/iptables -> /usr/sbin/iptables-nft
$ ls -la /usr/sbin/iptables
lrwxrwxrwx. 1 root root 26 Jan  7 22:10 /usr/sbin/iptables -> /etc/alternatives/iptables

> > [435321]$ grep -nrI "Warning: Extension" .
> > ./6-fib-tests-sh/stdout:305:# Warning: Extension MARK revision 0 not supported, missing kernel module?  
> 
> This could come from either legacy or nftables:
> 
> libxtables/xtables.c:                                   "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
> iptables/nft.c:                         "Warning: Extension %s revision 0 not supported, missing kernel module?\n",
> 
> both have the same error.
> 
> if that is the nftables backend, it might be also that .config is
> missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> options.

FWIW full config:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config

CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
> > if that is the nftables backend, it might be also that .config is
> > missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> > options.  
> 
> FWIW full config:
> 
> https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config
> 
> CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.

NFT_COMPAT fixed a lot! One remaining warning comes from using 
-m length. Which NFT config do we need for that one?

Re: [netfilter-core] [ANN] net-next is OPEN

[Florian Westphal]

Jakub Kicinski <kuba@kernel.org> wrote:
> On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
> > > if that is the nftables backend, it might be also that .config is
> > > missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> > > options.  
> > 
> > FWIW full config:
> > 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config
> > 
> > CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.
> 
> NFT_COMPAT fixed a lot! One remaining warning comes from using 
> -m length. Which NFT config do we need for that one?

CONFIG_NETFILTER_XT_MATCH_LENGTH=m|y

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Thu, 25 Jan 2024 09:52:11 +0100 Florian Westphal wrote:
> > NFT_COMPAT fixed a lot! One remaining warning comes from using 
> > -m length. Which NFT config do we need for that one?  
> 
> CONFIG_NETFILTER_XT_MATCH_LENGTH=m|y

Thank you, that seems to solve the remaining problem!

Re: [netfilter-core] [ANN] net-next is OPEN

[Pablo Neira Ayuso]

On Wed, Jan 24, 2024 at 09:07:24PM -0800, Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 12:13:43 -0800 Jakub Kicinski wrote:
> > > if that is the nftables backend, it might be also that .config is
> > > missing CONFIG_NF_TABLES and CONFIG_NFT_COMPAT there, among other
> > > options.  
> > 
> > FWIW full config:
> > 
> > https://netdev-2.bots.linux.dev/vmksft-net-mp/results/435321/config
> > 
> > CONFIG_NFT_COMPAT was indeed missing! Let's see how it fares with it enabled.
> 
> NFT_COMPAT fixed a lot! One remaining warning comes from using 
> -m length. Which NFT config do we need for that one?

May I have a look at the logs? How does the error look like?

Re: [netfilter-core] [ANN] net-next is OPEN

[Jakub Kicinski]

On Thu, 25 Jan 2024 10:29:41 +0100 Pablo Neira Ayuso wrote:
> > NFT_COMPAT fixed a lot! One remaining warning comes from using 
> > -m length. Which NFT config do we need for that one?  
> 
> May I have a look at the logs? How does the error look like?

With the config pointed out by Florian in addition to NFT_COMPAT
all the iptables errors in the logs are gone, without switching
to legacy. Thank you for the help!

Also LMK if you guys want us to try running netfilter tests.
I'm guessing we're unlikely to regress anything in net-next,
and you have running netfilter tests covered - but if it's 
not too hard we can try to hook them up to net-next, too!
(Or you can run them on our branch and report back :))

Re: [ANN] net-next is OPEN

[Willem de Bruijn]

Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 10:59:36 -0500 Willem de Bruijn wrote:
> > David Ahern wrote:
> > > On 1/23/24 8:20 AM, Jakub Kicinski wrote:  
> > > > It groups all patches outstanding in patchwork (which build cleanly).
> > > > I'm hoping we could also do HW testing using this setup, so batching
> > > > is a must. Not 100% sure it's the right direction for SW testing but
> > > > there's one way to find out :)
> > > >   
> > > 
> > > Really cool. Thanks for spending time to make this happen.  
> > 
> > Just to add to the choir: this is fantastic, thanks!
> > 
> > Hopefully it will encourage people to add kselftests, kunit tests or
> > other kinds that now get continuous coverage.
> 
> Fingers crossed :)
> 
> > Going through the failing ksft-net series on
> > https://netdev.bots.linux.dev/status.html, all the tests I'm
> > responsible seem to be passing.
> 
> Here's a more handy link filtered down to failures (clicking on 
> the test counts links here):
> 
> https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> 
> I have been attributing the udpg[rs]o and timestamp tests to you,
> but I haven't actually checked.. are they not yours? :)

I just looked at the result file and assumed 0 meant fine. Oops.

Technically udpgro is Paolo. But it is derived from udpgso.

But I'm on the hook for a few more. I added fq-band-pktlimit, our team
added gso and txtimestamp I supposed is owned by the maintainers by now.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Wed, 24 Jan 2024 12:42:46 -0500 Willem de Bruijn wrote:
> > Here's a more handy link filtered down to failures (clicking on 
> > the test counts links here):
> > 
> > https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> > 
> > I have been attributing the udpg[rs]o and timestamp tests to you,
> > but I haven't actually checked.. are they not yours? :)  
> 
> I just looked at the result file and assumed 0 meant fine. Oops.

Sorry about the confusion there, make run_tests apparently always
returns 0 and the result file holds the exit code :( It could be
improved by I figured, as long as the JSON output is correct, investing
the time in the web UI is probably a better choice than massaging 
the output files.

Re: [ANN] net-next is OPEN

[Willem de Bruijn]

Jakub Kicinski wrote:
> On Wed, 24 Jan 2024 12:42:46 -0500 Willem de Bruijn wrote:
> > > Here's a more handy link filtered down to failures (clicking on 
> > > the test counts links here):
> > > 
> > > https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> > > 
> > > I have been attributing the udpg[rs]o and timestamp tests to you,
> > > but I haven't actually checked.. are they not yours? :)  
> > 
> > I just looked at the result file and assumed 0 meant fine. Oops.
> 
> Sorry about the confusion there, make run_tests apparently always
> returns 0 and the result file holds the exit code :( It could be
> improved by I figured, as long as the JSON output is correct, investing
> the time in the web UI is probably a better choice than massaging 
> the output files.

Absolutely. My bad for jumping to conclusions.

Re: [ANN] net-next is OPEN

[Paolo Abeni]

On Wed, 2024-01-24 at 13:23 -0500, Willem de Bruijn wrote:
> Jakub Kicinski wrote:
> > On Wed, 24 Jan 2024 12:42:46 -0500 Willem de Bruijn wrote:
> > > > Here's a more handy link filtered down to failures (clicking on 
> > > > the test counts links here):
> > > > 
> > > > https://netdev.bots.linux.dev/contest.html?branch=net-next-2024-01-24--15-00&executor=vmksft-net-mp&pass=0
> > > > 
> > > > I have been attributing the udpg[rs]o and timestamp tests to you,
> > > > but I haven't actually checked.. are they not yours? :)  
> > > 
> > > I just looked at the result file and assumed 0 meant fine. Oops.
> > 
> > Sorry about the confusion there, make run_tests apparently always
> > returns 0 and the result file holds the exit code :( It could be
> > improved by I figured, as long as the JSON output is correct, investing
> > the time in the web UI is probably a better choice than massaging 
> > the output files.
> 
> Absolutely. My bad for jumping to conclusions.

Some tests are failing because they need an xdp program build under the
ebpf directory.

Since such program is used even by ebpf tests, I'll send a patch
duplicating it under net, and using the existing rule for nat6to4 to
build it.

Cheers,

Paolo

Re: [ANN] net-next is OPEN

[Petr Machata]

Jakub Kicinski <kuba@kernel.org> writes:

> If you authored any net or drivers/net selftests, please look around
> and see if they are passing. If not - send patches or LMK what I need
> to do to make them pass on the runner.. Make sure to scroll down to 
> the "Not reporting to patchwork" section.

A whole bunch of them fail because of no IPv6 support in the runner
kernel. E.g. this from bridge-mdb.sh[0]:

    # Error: Rule family not supported.
    # Error: Rule family not supported.
    # sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory
    # sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory
    # RTNETLINK answers: Operation not supported

I'm surprised any passed at all, it's super common for tests to validate
their topology by pinging through, but I guess it's often just IPv4. I
think the fix is just this?

    $ scripts/config -k -m CONFIG_IPV6

There are also a bunch of missing qdiscs, e.g. in [1], [2]. To fix:

    $ scripts/config -k -m CONFIG_NET_SCH_TBF
    $ scripts/config -k -m CONFIG_NET_SCH_PRIO
    $ scripts/config -k -m CONFIG_NET_SCH_ETS

Regarding sch_red.sh[3], I worry the test will be noisy, and suspect it
does not make sense to run it in automated fashion. But if you think
it's worth a try:

    $ scripts/config -k -m CONFIG_NET_SCH_RED

Then there are a bunch of missing netdevices. VXLAN[4]:

    $ scripts/config -k -m CONFIG_VXLAN

and GRE [5], which I think needs all of these:

    $ scripts/config -k -m CONFIG_NET_IPIP
    $ scripts/config -k -m CONFIG_IPV6_GRE
    $ scripts/config -k -m CONFIG_NET_IPGRE_DEMUX
    $ scripts/config -k -m CONFIG_NET_IPGRE

And TC actions [6]. I think the following will be necessary for some of
the tests (we enable BPF as well internally).

    $ scripts/config -k -m CONFIG_NET_ACT_GACT
    $ scripts/config -k -m CONFIG_NET_ACT_MIRRED
    $ scripts/config -k -m CONFIG_NET_ACT_SAMPLE
    $ scripts/config -k -m CONFIG_NET_ACT_VLAN
    $ scripts/config -k -m CONFIG_NET_ACT_SKBEDIT
    $ scripts/config -k -m CONFIG_NET_ACT_PEDIT
    $ scripts/config -k -m CONFIG_NET_ACT_POLICE

Hopefully the above should clean up the results a bit, I can take
another sweep afterwards.

[0] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/14-bridge-mdb-sh/stdout
[1] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/17-sch-ets-sh/stdout
[2] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/33-sch-tbf-prio-sh/stdout
[3] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/21-sch-red-sh/stdout
[4] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/27-mirror-gre-changes-sh/stdout
[5] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/18-vxlan-bridge-1d-sh/stdout
[6] https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/42-pedit-l4port-sh/stdout

Re: [ANN] net-next is OPEN

[Matthias May]

On 23/01/2024 10:55, Petr Machata wrote:
> Jakub Kicinski <kuba@kernel.org> writes:
>
>> If you authored any net or drivers/net selftests, please look around
>> and see if they are passing. If not - send patches or LMK what I need
>> to do to make them pass on the runner.. Make sure to scroll down to
>> the "Not reporting to patchwork" section.
> A whole bunch of them fail because of no IPv6 support in the runner
> kernel. E.g. this from bridge-mdb.sh[0]:
>
>      # Error: Rule family not supported.
>      # Error: Rule family not supported.
>      # sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory
>      # sysctl: cannot stat /proc/sys/net/ipv6/conf/all/forwarding: No such file or directory
>      # RTNETLINK answers: Operation not supported
>
> I'm surprised any passed at all, it's super common for tests to validate
> their topology by pinging through, but I guess it's often just IPv4. I
> think the fix is just this?
>
>      $ scripts/config -k -m CONFIG_IPV6
>
> There are also a bunch of missing qdiscs, e.g. in [1], [2]. To fix:
>
>      $ scripts/config -k -m CONFIG_NET_SCH_TBF
>      $ scripts/config -k -m CONFIG_NET_SCH_PRIO
>      $ scripts/config -k -m CONFIG_NET_SCH_ETS
>
> Regarding sch_red.sh[3], I worry the test will be noisy, and suspect it
> does not make sense to run it in automated fashion. But if you think
> it's worth a try:
>
>      $ scripts/config -k -m CONFIG_NET_SCH_RED
>
> Then there are a bunch of missing netdevices. VXLAN[4]:
>
>      $ scripts/config -k -m CONFIG_VXLAN
>
> and GRE [5], which I think needs all of these:
>
>      $ scripts/config -k -m CONFIG_NET_IPIP
>      $ scripts/config -k -m CONFIG_IPV6_GRE
>      $ scripts/config -k -m CONFIG_NET_IPGRE_DEMUX
>      $ scripts/config -k -m CONFIG_NET_IPGRE
>
> And TC actions [6]. I think the following will be necessary for some of
> the tests (we enable BPF as well internally).
>
>      $ scripts/config -k -m CONFIG_NET_ACT_GACT
>      $ scripts/config -k -m CONFIG_NET_ACT_MIRRED
>      $ scripts/config -k -m CONFIG_NET_ACT_SAMPLE
>      $ scripts/config -k -m CONFIG_NET_ACT_VLAN
>      $ scripts/config -k -m CONFIG_NET_ACT_SKBEDIT
>      $ scripts/config -k -m CONFIG_NET_ACT_PEDIT
>      $ scripts/config -k -m CONFIG_NET_ACT_POLICE
>
> Hopefully the above should clean up the results a bit, I can take
> another sweep afterwards.
>
> [0] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/14-bridge-mdb-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalB7DvzQR$
> [1] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/17-sch-ets-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalDWGO6rP$
> [2] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/33-sch-tbf-prio-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalMVxxlTD$
> [3] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/21-sch-red-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalMfr0cOk$
> [4] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/27-mirror-gre-changes-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalP3lTojb$
> [5] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/18-vxlan-bridge-1d-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalBLEoTPs$
> [6] https://urldefense.com/v3/__https://netdev-2.bots.linux.dev/vmksft-forwarding/results/433341/42-pedit-l4port-sh/stdout__;!!I9LPvj3b!Eb-NZpjOY2wkTb2sApNj0Hx-II6xvFO688SZN7feUAjC_6RFAz3dmeR3LzssrLqm_Kr7rJEalOO5rTam$
>
Hi

Probably also missing is CONFIG_GENEVE
81-l2-tos-ttl-inherit-sh check operation with gre, vxlan and geneve, but 
modprobes first if the module is actually available. If it isn't, it 
just continues and doesn't fail.

Also there seems to be something wrong with ending, see 
https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
The test outputs the results in a table with box drawing characters 
(┌─┬┐├─┼┤└─┴┘)

BR
Matthias

Re: [ANN] net-next is OPEN

[Petr Machata]

Matthias May <matthias.may@westermo.com> writes:

> Also there seems to be something wrong with ending, see
> https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
> The test outputs the results in a table with box drawing characters (┌─┬┐├─┼┤└─┴┘)

It looks like whatever is serving the output should use MIME of
"text/plain;charset=UTF-8" instead of just "text/plain".

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Tue, 23 Jan 2024 14:38:24 +0100 Petr Machata wrote:
> Matthias May <matthias.may@westermo.com> writes:
> 
> > Also there seems to be something wrong with ending, see
> > https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
> > The test outputs the results in a table with box drawing characters (┌─┬┐├─┼┤└─┴┘)  
> 
> It looks like whatever is serving the output should use MIME of
> "text/plain;charset=UTF-8" instead of just "text/plain".

😮️ interesting. The table characters are not part of ASCII, right? 
So it must be using some extended charset. Firefox hid the option 
to tweak encoding tho so IDK what it actually uses :S

Re: [ANN] net-next is OPEN

[Petr Machata]

Jakub Kicinski <kuba@kernel.org> writes:

> On Tue, 23 Jan 2024 14:38:24 +0100 Petr Machata wrote:
>> Matthias May <matthias.may@westermo.com> writes:
>> 
>> > Also there seems to be something wrong with ending, see
>> > https://netdev-2.bots.linux.dev/vmksft-net/results/433200/81-l2-tos-ttl-inherit-sh
>> > The test outputs the results in a table with box drawing characters (┌─┬┐├─┼┤└─┴┘)  
>> 
>> It looks like whatever is serving the output should use MIME of
>> "text/plain;charset=UTF-8" instead of just "text/plain".
>
> 😮️ interesting. The table characters are not part of ASCII, right? 

Yeah, the table is UTF-8.

> So it must be using some extended charset. Firefox hid the option 
> to tweak encoding tho so IDK what it actually uses :S

My guess would be ISO-8859-1, but dunno. The developer console shows
Firefox has had to guess the encoding, but not what guess it made.

It seems weird to guess anything but UTF-8 in this day and age. Maybe
the logic is that modern web has correct content-type, because it needs
UTF-8, so if it doesn't declare encoding, it's something ancient.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Tue, 23 Jan 2024 17:05:12 +0100 Petr Machata wrote:
> >> It looks like whatever is serving the output should use MIME of
> >> "text/plain;charset=UTF-8" instead of just "text/plain".  
> >
> > 😮️ interesting. The table characters are not part of ASCII, right?   
> 
> Yeah, the table is UTF-8.
> 
> > So it must be using some extended charset. Firefox hid the option 
> > to tweak encoding tho so IDK what it actually uses :S  
> 
> My guess would be ISO-8859-1, but dunno. The developer console shows
> Firefox has had to guess the encoding, but not what guess it made.
> 
> It seems weird to guess anything but UTF-8 in this day and age. Maybe
> the logic is that modern web has correct content-type, because it needs
> UTF-8, so if it doesn't declare encoding, it's something ancient.

Ah, sorry, I read your message backwards. I remember fixing the encoding
on the NIPA nginx to make it utf-8 so I thought you said it's utf-8
and should be something else... But the runner has a separate
instance, and there the charset is indeed not set. Fixed now!

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Tue, 23 Jan 2024 10:55:09 +0100 Petr Machata wrote:
> > If you authored any net or drivers/net selftests, please look around
> > and see if they are passing. If not - send patches or LMK what I need
> > to do to make them pass on the runner.. Make sure to scroll down to 
> > the "Not reporting to patchwork" section.  
> 
> A whole bunch of them fail because of no IPv6 support in the runner
> kernel. E.g. this from bridge-mdb.sh[0]:

Thanks a lot for investigating! I take it that you're looking at
forwarding? Please send a patch to add the missing configs to

tools/testing/selftests/net/forwarding/config

The runner uses that to configure the kernel on top of defconfig.

Unless I'm doing it wrong and the sub-directories are supposed to
inherit the parent directory's config? So net/forwarding/ should
be built with net/'s config? I could not find the info in docs,
does anyone know?

Re: [ANN] net-next is OPEN

[Petr Machata]

Jakub Kicinski <kuba@kernel.org> writes:

> On Tue, 23 Jan 2024 10:55:09 +0100 Petr Machata wrote:
>> > If you authored any net or drivers/net selftests, please look around
>> > and see if they are passing. If not - send patches or LMK what I need
>> > to do to make them pass on the runner.. Make sure to scroll down to 
>> > the "Not reporting to patchwork" section.  
>> 
>> A whole bunch of them fail because of no IPv6 support in the runner
>> kernel. E.g. this from bridge-mdb.sh[0]:
>
> Thanks a lot for investigating! I take it that you're looking at
> forwarding? Please send a patch to add the missing configs to
>
> tools/testing/selftests/net/forwarding/config

OK.

> The runner uses that to configure the kernel on top of defconfig.
>
> Unless I'm doing it wrong and the sub-directories are supposed to
> inherit the parent directory's config? So net/forwarding/ should
> be built with net/'s config? I could not find the info in docs,
> does anyone know?

I don't think they are, net/config defines CONFIG_VXLAN, but then the
vxlan tests still complain about unknown device type. Though maybe
there's another device type that it's missing...

What do I do to feed the config file to some build script to get a
kernel image to test? I can of course just do something like
cat config | xargs -n1 scripts/config -m, but I expect there's some
automation for it and I just can't find it.

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Tue, 23 Jan 2024 18:04:19 +0100 Petr Machata wrote:
> > Unless I'm doing it wrong and the sub-directories are supposed to
> > inherit the parent directory's config? So net/forwarding/ should
> > be built with net/'s config? I could not find the info in docs,
> > does anyone know?  
> 
> I don't think they are, net/config defines CONFIG_VXLAN, but then the
> vxlan tests still complain about unknown device type. Though maybe
> there's another device type that it's missing...
> 
> What do I do to feed the config file to some build script to get a
> kernel image to test? I can of course just do something like
> cat config | xargs -n1 scripts/config -m, but I expect there's some
> automation for it and I just can't find it.

The CI script is based on virtme-ng. So it does this:

# $target is net or net/forwarding or drivers/net/bonding etc.
make mrproper
vng -v -b -f tools/testing/selftests/$target
# build the scripts
make headers
make -C tools/testing/selftests/$target

vng -v -r arch/x86/boot/bzImage --user root
# inside the VM
make -C tools/testing/selftests TARGETS=$target run_tests

https://github.com/kuba-moo/nipa/blob/master/contest/remote/vmksft.py#L138

You're right, it definitely does not "inherit" net's config when
running forwarding/net. I can easily make it do so, but I'm not clear
what the expectation from the kselftest subsystem is. Because if other
testers (people testing stable, KernelCI etc. et.c) don't "inherit" we
better fill in the config completely so that the tests pass for
everyone.

Re: [ANN] net-next is OPEN

[Petr Machata]

Jakub Kicinski <kuba@kernel.org> writes:

> On Tue, 23 Jan 2024 18:04:19 +0100 Petr Machata wrote:
>> > Unless I'm doing it wrong and the sub-directories are supposed to
>> > inherit the parent directory's config? So net/forwarding/ should
>> > be built with net/'s config? I could not find the info in docs,
>> > does anyone know?  
>> 
>> I don't think they are, net/config defines CONFIG_VXLAN, but then the
>> vxlan tests still complain about unknown device type. Though maybe
>> there's another device type that it's missing...
>> 
>> What do I do to feed the config file to some build script to get a
>> kernel image to test? I can of course just do something like
>> cat config | xargs -n1 scripts/config -m, but I expect there's some
>> automation for it and I just can't find it.
>
> The CI script is based on virtme-ng. So it does this:
>
> # $target is net or net/forwarding or drivers/net/bonding etc.
> make mrproper
> vng -v -b -f tools/testing/selftests/$target

Actually:

# vng -v -b -f tools/testing/selftests/${target}/config

Didn't know about vng, it's way cool!
(Despite having used virtme for a long time.)

> # build the scripts
> make headers
> make -C tools/testing/selftests/$target
>
> vng -v -r arch/x86/boot/bzImage --user root
> # inside the VM
> make -C tools/testing/selftests TARGETS=$target run_tests

I'm working my way through the selftests.

> https://github.com/kuba-moo/nipa/blob/master/contest/remote/vmksft.py#L138
>
> You're right, it definitely does not "inherit" net's config when
> running forwarding/net. I can easily make it do so, but I'm not clear
>
> what the expectation from the kselftest subsystem is. Because if other
> testers (people testing stable, KernelCI etc. et.c) don't "inherit" we
> better fill in the config completely so that the tests pass for
> everyone.

Oh, gotcha, the question was not whether it does, but whether it's
supposed to. OK.

IMHO not necessarily. net/config is for net/*.sh, net/forwarding/config
for net/forwarding/*.sh. It's not a given that whatever is needed for
net/ is needed for net/forwarding/ as well.

It will also lead to simpler patches, where enabling config options in
X/ doesn't imply checking for newly useless duplicities in X's child
directories.

Re: [ANN] net-next is OPEN

[Ido Schimmel]

On Mon, Jan 22, 2024 at 09:16:12AM -0800, Jakub Kicinski wrote:
> If you authored any net or drivers/net selftests, please look around
> and see if they are passing. If not - send patches or LMK what I need
> to do to make them pass on the runner.. Make sure to scroll down to 
> the "Not reporting to patchwork" section.

selftests-net/test-bridge-neigh-suppress-sh should be fixed by:

dnf install ndisc6

selftests-net/test-bridge-backup-port-sh should be fixed by:

https://lore.kernel.org/netdev/20240129123703.1857843-1-idosch@nvidia.com/

selftests-net/drop-monitor-tests-sh should be fixed by:

dnf install dropwatch

Re: [ANN] net-next is OPEN

[Ido Schimmel]

On Mon, Jan 29, 2024 at 02:43:49PM +0200, Ido Schimmel wrote:
> On Mon, Jan 22, 2024 at 09:16:12AM -0800, Jakub Kicinski wrote:
> > If you authored any net or drivers/net selftests, please look around
> > and see if they are passing. If not - send patches or LMK what I need
> > to do to make them pass on the runner.. Make sure to scroll down to 
> > the "Not reporting to patchwork" section.

Forgot to mention: Thanks a lot for setting this up!

> 
> selftests-net/test-bridge-neigh-suppress-sh should be fixed by:
> 
> dnf install ndisc6
> 
> selftests-net/test-bridge-backup-port-sh should be fixed by:
> 
> https://lore.kernel.org/netdev/20240129123703.1857843-1-idosch@nvidia.com/
> 
> selftests-net/drop-monitor-tests-sh should be fixed by:
> 
> dnf install dropwatch

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Mon, 29 Jan 2024 14:46:32 +0200 Ido Schimmel wrote:
> On Mon, Jan 29, 2024 at 02:43:49PM +0200, Ido Schimmel wrote:
> > On Mon, Jan 22, 2024 at 09:16:12AM -0800, Jakub Kicinski wrote:  
> > > If you authored any net or drivers/net selftests, please look around
> > > and see if they are passing. If not - send patches or LMK what I need
> > > to do to make them pass on the runner.. Make sure to scroll down to 
> > > the "Not reporting to patchwork" section.  
> 
> Forgot to mention: Thanks a lot for setting this up!

Finger crossed that it ends up being useful :)

> > selftests-net/test-bridge-neigh-suppress-sh should be fixed by:
> > 
> > dnf install ndisc6
> > 
> > selftests-net/test-bridge-backup-port-sh should be fixed by:
> > 
> > https://lore.kernel.org/netdev/20240129123703.1857843-1-idosch@nvidia.com/
> > 
> > selftests-net/drop-monitor-tests-sh should be fixed by:
> > 
> > dnf install dropwatch  

Installed both (from source) just in time for the
net-next-2024-01-29--15-00 run.. let's see.

Re: [ANN] net-next is OPEN

[Ido Schimmel]

On Mon, Jan 29, 2024 at 07:00:57AM -0800, Jakub Kicinski wrote:
> On Mon, 29 Jan 2024 14:46:32 +0200 Ido Schimmel wrote:
> > On Mon, Jan 29, 2024 at 02:43:49PM +0200, Ido Schimmel wrote:
> > > On Mon, Jan 22, 2024 at 09:16:12AM -0800, Jakub Kicinski wrote:  
> > > > If you authored any net or drivers/net selftests, please look around
> > > > and see if they are passing. If not - send patches or LMK what I need
> > > > to do to make them pass on the runner.. Make sure to scroll down to 
> > > > the "Not reporting to patchwork" section.  
> > 
> > Forgot to mention: Thanks a lot for setting this up!
> 
> Finger crossed that it ends up being useful :)
> 
> > > selftests-net/test-bridge-neigh-suppress-sh should be fixed by:
> > > 
> > > dnf install ndisc6
> > > 
> > > selftests-net/test-bridge-backup-port-sh should be fixed by:
> > > 
> > > https://lore.kernel.org/netdev/20240129123703.1857843-1-idosch@nvidia.com/
> > > 
> > > selftests-net/drop-monitor-tests-sh should be fixed by:
> > > 
> > > dnf install dropwatch  
> 
> Installed both (from source) just in time for the
> net-next-2024-01-29--15-00 run.. let's see.

Thanks!

The last two tests look good now, but the first still fails. Can you
share the ndisc6 version information? I tested with [1] from [2].

If your copy of ndisc6 indeed works, then I might be missing some
sysctl. I will be AFK tomorrow so I will look into it later this week.

[1]
$ ./ndisc6 --version
ndisc6: IPv6 Neighbor/Router Discovery userland tool 1.0.8 ($Rev$)
 built Jan 29 2024 on <redacted>
Configured with: ./configure 
Written by Remi Denis-Courmont

Copyright (C) 2004-2007 Remi Denis-Courmont
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE.

[2] https://git.remlab.net/gitweb/?p=ndisc6.git

Re: [ANN] net-next is OPEN

[Jakub Kicinski]

On Mon, 29 Jan 2024 19:00:49 +0200 Ido Schimmel wrote:
> > Installed both (from source) just in time for the
> > net-next-2024-01-29--15-00 run.. let's see.  
> 
> Thanks!
> 
> The last two tests look good now, but the first still fails. Can you
> share the ndisc6 version information? I tested with [1] from [2].
> 
> If your copy of ndisc6 indeed works, then I might be missing some
> sysctl. I will be AFK tomorrow so I will look into it later this week.

Hm. Looks like our versions match. I put the entire tools root dir up on
HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna
fetch the exact binary, it only links with libc, it seems.

Re: [ANN] net-next is OPEN

[Ido Schimmel]

On Mon, Jan 29, 2024 at 09:18:10AM -0800, Jakub Kicinski wrote:
> On Mon, 29 Jan 2024 19:00:49 +0200 Ido Schimmel wrote:
> > > Installed both (from source) just in time for the
> > > net-next-2024-01-29--15-00 run.. let's see.  
> > 
> > Thanks!
> > 
> > The last two tests look good now, but the first still fails. Can you
> > share the ndisc6 version information? I tested with [1] from [2].
> > 
> > If your copy of ndisc6 indeed works, then I might be missing some
> > sysctl. I will be AFK tomorrow so I will look into it later this week.
> 
> Hm. Looks like our versions match. I put the entire tools root dir up on
> HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna
> fetch the exact binary, it only links with libc, it seems.

I tried with your binary and on other setups and I'm unable to reproduce
the failure. From the test output it seems the NS is never sent. If you
can, attaching the verbose test output might help:

./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v

Thanks

Re: [ANN] net-next is OPEN

[Heiner Kallweit]

On 31.01.2024 14:23, Ido Schimmel wrote:
> On Mon, Jan 29, 2024 at 09:18:10AM -0800, Jakub Kicinski wrote:
>> On Mon, 29 Jan 2024 19:00:49 +0200 Ido Schimmel wrote:
>>>> Installed both (from source) just in time for the
>>>> net-next-2024-01-29--15-00 run.. let's see.  
>>>
>>> Thanks!
>>>
>>> The last two tests look good now, but the first still fails. Can you
>>> share the ndisc6 version information? I tested with [1] from [2].
>>>
>>> If your copy of ndisc6 indeed works, then I might be missing some
>>> sysctl. I will be AFK tomorrow so I will look into it later this week.
>>
>> Hm. Looks like our versions match. I put the entire tools root dir up on
>> HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna
>> fetch the exact binary, it only links with libc, it seems.
> 
> I tried with your binary and on other setups and I'm unable to reproduce
> the failure. From the test output it seems the NS is never sent. If you
> can, attaching the verbose test output might help:
> 
> ./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v
> 
> Thanks
> 
Maybe change the subject of the mail thread? At least I'm confused if I
read several times a day "net-next is OPEN".

Re: [TEST] bridge tests (was: net-next is OPEN)

[Jakub Kicinski]

On Wed, 31 Jan 2024 15:23:50 +0200 Ido Schimmel wrote:
> > Hm. Looks like our versions match. I put the entire tools root dir up on
> > HTTP now: https://netdev-2.bots.linux.dev/tools/fs/ in case you wanna
> > fetch the exact binary, it only links with libc, it seems.  
> 
> I tried with your binary and on other setups and I'm unable to reproduce
> the failure. From the test output it seems the NS is never sent. If you
> can, attaching the verbose test output might help:
> 
> ./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v

FWIW I started two new instances on bare metal one with the same kernel
as the nested VM and one with debug options enabled.

selftests-net/test-bridge-neigh-suppress-sh
 - fails across all, so must be the OS rather than the "speed"

selftests-net/test-bridge-backup-port-sh
  - passes on VM, metal-dbg
  - fails on metal :S very reliably / every time:
https://netdev.bots.linux.dev/contest.html?test=test-bridge-backup-port-sh

  # TEST: No forwarding out of swp1                    [FAIL]

selftests-net/drop-monitor-tests-sh 
 - passes everywhere now

The info you asked for:

# ./test_bridge_neigh_suppress.sh -t neigh_suppress_ns -v
[   49.621534] eth0: renamed from veth0
[   49.654673] swp1: renamed from veth1
[   49.676235] ip (235) used greatest stack depth: 12032 bytes left
[   49.721434] veth0: renamed from veth1
[   49.739521] ip (241) used greatest stack depth: 11880 bytes left
[   49.760463] eth0: renamed from veth0
[   49.787523] swp1: renamed from veth1
[   50.019197] br0: port 1(swp1) entered blocking state
[   50.019610] br0: port 1(swp1) entered disabled state
[   50.019949] swp1: entered allmulticast mode
[   50.020307] swp1: entered promiscuous mode
[   50.021035] br0: port 1(swp1) entered blocking state
[   50.021415] br0: port 1(swp1) entered forwarding state
[   50.076445] br0: port 2(vx0) entered blocking state
[   50.076805] br0: port 2(vx0) entered disabled state
[   50.077139] vx0: entered allmulticast mode
[   50.077492] vx0: entered promiscuous mode
[   50.077812] br0: port 2(vx0) entered blocking state
[   50.078172] br0: port 2(vx0) entered forwarding state
[   50.238364] br0: port 1(swp1) entered blocking state
[   50.238735] br0: port 1(swp1) entered disabled state
[   50.239077] swp1: entered allmulticast mode
[   50.239434] swp1: entered promiscuous mode
[   50.240046] br0: port 1(swp1) entered blocking state
[   50.240395] br0: port 1(swp1) entered forwarding state
[   50.269453] br0: port 2(vx0) entered blocking state
[   50.269812] br0: port 2(vx0) entered disabled state
[   50.270142] vx0: entered allmulticast mode
[   50.270597] vx0: entered promiscuous mode
[   50.270893] br0: port 2(vx0) entered blocking state
[   50.271222] br0: port 2(vx0) entered forwarding state

Per-port NS suppression - VLAN 10
---------------------------------
COMMAND: tc -n sw1-hF6GFk qdisc replace dev vx0 clsact
COMMAND: tc -n sw1-hF6GFk filter replace dev vx0 egress pref 1 handle 101 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip ff02::1:ff00:2 src_ip 2001:db8:1::1 type 135 code 0 action pass
[   55.411592] GACT probability NOT on
COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress on
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress on"
        hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress on neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0 
TEST: "neigh_suppress" is on                                        [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk fdb replace 5a:1d:b4:4b:25:16 dev vx0 master static vlan 10
TEST: FDB entry installation                                        [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n sw1-hF6GFk neigh replace 2001:db8:1::2 lladdr 5a:1d:b4:4b:25:16 nud permanent dev br0.10
TEST: Neighbor entry installation                                   [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.10 down
TEST: H2 down                                                       [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.10 up
TEST: H2 up                                                         [ OK ]

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress off
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress off"
        hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress off neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0 
TEST: "neigh_suppress" is off                                       [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.10 down
TEST: H2 down                                                       [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 2

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0


Per-port NS suppression - VLAN 20
---------------------------------
COMMAND: tc -n sw1-hF6GFk qdisc replace dev vx0 clsact
COMMAND: tc -n sw1-hF6GFk filter replace dev vx0 egress pref 1 handle 101 proto ipv6 flower indev swp1 ip_proto icmpv6 dst_ip ff02::1:ff00:2 src_ip 2001:db8:2::1 type 135 code 0 action pass
COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress on
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress on"
        hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress on neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0 
TEST: "neigh_suppress" is on                                        [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: bridge -n sw1-hF6GFk fdb replace 5a:1d:b4:4b:25:16 dev vx0 master static vlan 20
TEST: FDB entry installation                                        [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n sw1-hF6GFk neigh replace 2001:db8:2::2 lladdr 5a:1d:b4:4b:25:16 nud permanent dev br0.20
TEST: Neighbor entry installation                                   [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.20 down
TEST: H2 down                                                       [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.20 up
TEST: H2 up                                                         [ OK ]

COMMAND: bridge -n sw1-hF6GFk link set dev vx0 neigh_suppress off
COMMAND: bridge -n sw1-hF6GFk -d link show dev vx0 | grep "neigh_suppress off"
        hairpin off guard off root_block off fastleave off learning off flood on mcast_flood on bcast_flood on mcast_router 1 mcast_to_unicast off neigh_suppress off neigh_vlan_suppress off vlan_tunnel on isolated off locked off mab off mcast_n_groups 0 mcast_max_groups 0 
TEST: "neigh_suppress" is off                                       [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 0

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

COMMAND: ip -n h2-g0sh0Q link set dev eth0.20 down
TEST: H2 down                                                       [ OK ]

COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:2::1 -w 5000 2001:db8:2::2 eth0.20
Raw IPv6 socket: Operation not permitted
TEST: ndisc6                                                        [FAIL]
    rc=1, expected 2

TEST: NS suppression                                                [FAIL]
    rc=1, expected 0

[   57.883133] br0: port 1(swp1) entered disabled state
[   57.884754] swp1 (unregistering): left allmulticast mode
[   57.885191] swp1 (unregistering): left promiscuous mode
[   57.885580] br0: port 1(swp1) entered disabled state

Tests passed:  14
Tests failed:  28
[   57.966498] vx0: left allmulticast mode
[   57.966802] vx0: left promiscuous mode
[   57.967091] br0: port 2(vx0) entered disabled state
[   57.992675] br0: port 1(swp1) entered disabled state
[   58.017115] swp1 (unregistering): left allmulticast mode
[   58.017520] swp1 (unregistering): left promiscuous mode
[   58.017898] br0: port 1(swp1) entered disabled state
bash-5.2# [   58.092405] vx0: left allmulticast mode
[   58.092704] vx0: left promiscuous mode
[   58.092972] br0: port 2(vx0) entered disabled state

Re: [TEST] bridge tests (was: net-next is OPEN)

[Ido Schimmel]

On Wed, Jan 31, 2024 at 08:01:37AM -0800, Jakub Kicinski wrote:
> FWIW I started two new instances on bare metal one with the same kernel
> as the nested VM and one with debug options enabled.
> 
> selftests-net/test-bridge-neigh-suppress-sh
>  - fails across all, so must be the OS rather than the "speed"

Yes, it's something related to the OS. From the log below:

```
 COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
 Raw IPv6 socket: Operation not permitted
 TEST: ndisc6                                                        [FAIL]
     rc=1, expected 0
```

The test is supposed to be run as root so I'm not sure what this error
is about. Do you have something like AppArmor or SELinux running? The
program creates an IPv6 raw socket and requires CAP_NET_RAW.

> selftests-net/test-bridge-backup-port-sh
>   - passes on VM, metal-dbg
>   - fails on metal :S very reliably / every time:
> https://netdev.bots.linux.dev/contest.html?test=test-bridge-backup-port-sh
> 
>   # TEST: No forwarding out of swp1                    [FAIL]

Passes on all setups:

https://netdev-2.bots.linux.dev/vmksft-net-mp/results/446482/12-test-bridge-backup-port-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net/results/446481/7-test-bridge-backup-port-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/446481/12-test-bridge-backup-port-sh/stdout

With this patch:

https://lore.kernel.org/netdev/20240201080516.3585867-1-idosch@nvidia.com/

Will submit v2 next week.

> selftests-net/drop-monitor-tests-sh 
>  - passes everywhere now

Nice, thanks

Re: [TEST] bridge tests (was: net-next is OPEN)

[Jakub Kicinski]

On Thu, 1 Feb 2024 15:46:23 +0200 Ido Schimmel wrote:
> > selftests-net/test-bridge-neigh-suppress-sh
> >  - fails across all, so must be the OS rather than the "speed"  
> 
> Yes, it's something related to the OS. From the log below:
> 
> ```
>  COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
>  Raw IPv6 socket: Operation not permitted
>  TEST: ndisc6                                                        [FAIL]
>      rc=1, expected 0
> ```
> 
> The test is supposed to be run as root so I'm not sure what this error
> is about. Do you have something like AppArmor or SELinux running? The
> program creates an IPv6 raw socket and requires CAP_NET_RAW.

Ah, ugh, sorry for the misdirection, you're right.

Looks like the binaries have SUID set:

# find tools/fs/ -perm -4000
tools/fs/usr/bin/ndisc6
tools/fs/usr/bin/rdisc6
tools/fs/usr/bin/rltraceroute6

But I install them as a normal user:

# ll tools/fs/usr/bin/ndisc6
-rwsr-xr-x. 1 virtme virtme 53840 Jan 29 14:36 tools/fs/usr/bin/ndisc6

so I guess they intend to SUID themselves into privileges but end up
SUIDing out to a lowly user :(

I cleared the SUID bits out, let's see the next run.

Re: [TEST] bridge tests (was: net-next is OPEN)

[Jakub Kicinski]

On Thu, 1 Feb 2024 07:30:25 -0800 Jakub Kicinski wrote:
> On Thu, 1 Feb 2024 15:46:23 +0200 Ido Schimmel wrote:
> > > selftests-net/test-bridge-neigh-suppress-sh
> > >  - fails across all, so must be the OS rather than the "speed"    
> > 
> > Yes, it's something related to the OS. From the log below:
> > 
> > ```
> >  COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
> >  Raw IPv6 socket: Operation not permitted
> >  TEST: ndisc6                                                        [FAIL]
> >      rc=1, expected 0
> > ```
> > 
> > The test is supposed to be run as root so I'm not sure what this error
> > is about. Do you have something like AppArmor or SELinux running? The
> > program creates an IPv6 raw socket and requires CAP_NET_RAW.  
> 
> Ah, ugh, sorry for the misdirection, you're right.

Confirmed, with the SUID cleared test-bridge-neigh-suppress-sh now
passes on everything with the exception of metal+debug kernel.

Re: [TEST] bridge tests (was: net-next is OPEN)

[Ido Schimmel]

On Thu, Feb 01, 2024 at 04:16:19PM -0800, Jakub Kicinski wrote:
> On Thu, 1 Feb 2024 07:30:25 -0800 Jakub Kicinski wrote:
> > On Thu, 1 Feb 2024 15:46:23 +0200 Ido Schimmel wrote:
> > > > selftests-net/test-bridge-neigh-suppress-sh
> > > >  - fails across all, so must be the OS rather than the "speed"    
> > > 
> > > Yes, it's something related to the OS. From the log below:
> > > 
> > > ```
> > >  COMMAND: ip netns exec h1-n8Aaip ndisc6 -q -r 1 -s 2001:db8:1::1 -w 5000 2001:db8:1::2 eth0.10
> > >  Raw IPv6 socket: Operation not permitted
> > >  TEST: ndisc6                                                        [FAIL]
> > >      rc=1, expected 0
> > > ```
> > > 
> > > The test is supposed to be run as root so I'm not sure what this error
> > > is about. Do you have something like AppArmor or SELinux running? The
> > > program creates an IPv6 raw socket and requires CAP_NET_RAW.  
> > 
> > Ah, ugh, sorry for the misdirection, you're right.
> 
> Confirmed, with the SUID cleared test-bridge-neigh-suppress-sh now
> passes on everything with the exception of metal+debug kernel.

I'm sorry for bothering you with this, but I checked today's logs and it
seems there is a similar problem with arping:
https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/456561/8-test-bridge-neigh-suppress-sh/stdout
https://netdev-3.bots.linux.dev/vmksft-net/results/456562/6-test-bridge-neigh-suppress-sh/stdout

And according to this log the ndisc6 problem resurfaced:
https://netdev-3.bots.linux.dev/vmksft-net/results/456382/6-test-bridge-neigh-suppress-sh/stdout

Any chance that something in the OS changed since last week?

Re: [TEST] bridge tests (was: net-next is OPEN)

[Jakub Kicinski]

On Thu, 8 Feb 2024 18:21:29 +0200 Ido Schimmel wrote:
> On Thu, Feb 01, 2024 at 04:16:19PM -0800, Jakub Kicinski wrote:
> > On Thu, 1 Feb 2024 07:30:25 -0800 Jakub Kicinski wrote:  
> > > Ah, ugh, sorry for the misdirection, you're right.  
> > 
> > Confirmed, with the SUID cleared test-bridge-neigh-suppress-sh now
> > passes on everything with the exception of metal+debug kernel.  
> 
> I'm sorry for bothering you with this, 

No worries at all :)

> but I checked today's logs and it
> seems there is a similar problem with arping:
> https://netdev-3.bots.linux.dev/vmksft-net-dbg/results/456561/8-test-bridge-neigh-suppress-sh/stdout
> https://netdev-3.bots.linux.dev/vmksft-net/results/456562/6-test-bridge-neigh-suppress-sh/stdout
> 
> And according to this log the ndisc6 problem resurfaced:
> https://netdev-3.bots.linux.dev/vmksft-net/results/456382/6-test-bridge-neigh-suppress-sh/stdout

It seems pretty faily:

https://netdev.bots.linux.dev/flakes.html?br-cnt=80&tn-needle=bridge-neigh&min-flip=0

I guess the short green streak does coincide with the fix, tho. Hm.

> Any chance that something in the OS changed since last week?

My memory is bad so I started keeping a log:

https://docs.google.com/spreadsheets/d/1mFnt91SKtA9ENIUfY0v2UmZSXJzmvsa4oqha4t6JVBs/edit?pli=1#gid=338019672

But it's not the suid:

$ find tools/fs/ -perm -4000
$ ll $(find tools/fs/ -name arping)
-rwxr-xr-x. 1 virtme virtme 79360 Jan 29 14:38 tools/fs/usr/bin/arping
$ ll $(find tools/fs/ -name ndisc6)
-rwxr-xr-x. 1 virtme virtme 53840 Jan 29 14:36 tools/fs/usr/bin/ndisc6

And it occasionally blips to "pass" so can't be completely broken env.

I run the test now with VERBOSE=1 and -q removed from arping and ndisc6:
https://netdev-3.bots.linux.dev/vmksft-net/test_bridge_neigh_suppress/

It just says: Received 0 response(s) :(

Repro is 100%, LMK if you'd like me to try with some custom diff on
top..