[ANN] Containers to run Netdev CI tests locally

[ANN] Containers to run Netdev CI tests locally

[Matthieu Baerts]

Hello,

TL;DR: containers are now available with all dependencies to easily
execute all static, unit, and functional tests in an environment closed
to the Netdev CI one:

- https://github.com/orgs/linux-netdev/packages

- https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally


                             -- ** -- ** --

As already mentioned on this list, and repeated at the last Netdev Conf,
developers are expected to validate the same tests as the ones executed
by the Netdev CI before any submissions. This includes all static, unit
and functional tests linked to the net subsystem -- and impacted by the
modifications --, but also LLM reviews. The goal is to reduce the number
of revisions and general traffic on this mailing list, not to overload
maintainers and reviewers even more.

A wiki page has been added to cover this and explain how to validate the
different tests:

  https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally

To help developers, containers are now built, packing all the required
dependencies to run these tests in order to run the tests in an
environment that would be as close as possible to the CI one. The goal
is to have an environment that is easy to maintain -- no heavy
optimisations to reduce the images size -- but also easy to use. In
short, people (and non-humans) can continue to use the different tools
as before, but add a prefix to the different commands to execute them
from a container, e.g.

  alias run="${nipa}/docker/selftests/run.sh"
  run vng --build
  run make headers
  (...)


Feedback on this is welcome. Do not hesitate to report issues on the
GitHub project (or here) and share fixes!

Note: currently only 'docker' is supported, but 'podman' can be used,
and plain 'chroot' as well, see the wiki. Patches are welcome!

Also, the following page has been modified to include more tips, visual
improvements, and references to the new containers:


https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

Cheers,
Matt
--
For the Netdev Foundation

Re: [ANN] Containers to run Netdev CI tests locally

[Hangbin Liu]

Hi Matthieu,

On Thu, Sep 03, 2026 at 06:19:25PM +0200, Matthieu Baerts wrote:
> Hello,
> 
> TL;DR: containers are now available with all dependencies to easily
> execute all static, unit, and functional tests in an environment closed
> to the Netdev CI one:
> 
> - https://github.com/orgs/linux-netdev/packages
> 
> - https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally

Great! I've been struggling to set up the local NIPA docker. I'm following
the steps from your git commit description. Now this article saved me.

Would you please add the article link to https://github.com/linux-netdev/nipa/wiki?
I could not find it and assumed there was no docker setup documentation yet.

Thanks
Hangbin

Re: [ANN] Containers to run Netdev CI tests locally

[Matthieu Baerts]

Hi Hangbin,

On 04/09/2026 08:59, Hangbin Liu wrote:
> Hi Matthieu,
> 
> On Thu, Sep 03, 2026 at 06:19:25PM +0200, Matthieu Baerts wrote:
>> Hello,
>>
>> TL;DR: containers are now available with all dependencies to easily
>> execute all static, unit, and functional tests in an environment closed
>> to the Netdev CI one:
>>
>> - https://github.com/orgs/linux-netdev/packages
>>
>> - https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
> 
> Great! I've been struggling to set up the local NIPA docker. I'm following
> the steps from your git commit description. Now this article saved me.

I'm glad it helped!

> Would you please add the article link to https://github.com/linux-netdev/nipa/wiki?
> I could not find it and assumed there was no docker setup documentation yet.
Good point, thanks, I forgot to update the parent page. Done!
I also just added a README in the "docker" folder.

Note that I'm also going to add a AGENTS.md file for those who would
find this useful.

Cheers,
Matt

Re: [ANN] Containers to run Netdev CI tests locally

[Hangbin Liu]

On Fri, Sep 04, 2026 at 10:33:18AM +0200, Matthieu Baerts wrote:
> Hi Hangbin,
> 
> On 04/09/2026 08:59, Hangbin Liu wrote:
> > Hi Matthieu,
> > 
> > On Thu, Sep 03, 2026 at 06:19:25PM +0200, Matthieu Baerts wrote:
> >> Hello,
> >>
> >> TL;DR: containers are now available with all dependencies to easily
> >> execute all static, unit, and functional tests in an environment closed
> >> to the Netdev CI one:
> >>
> >> - https://github.com/orgs/linux-netdev/packages
> >>
> >> - https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
> > 
> > Great! I've been struggling to set up the local NIPA docker. I'm following
> > the steps from your git commit description. Now this article saved me.
> 
> I'm glad it helped!
> 

From a user‑experience perspective, I think the static tests work well.
They display the check results clearly and save outputs to the result directory.

However, for selftests, I do not see any difference compared to running vng
directly on my local machine, i.e.

  vng -v --run . --user root -a mitigations=off --cpus 4 -- \
  	make -C tools/testing/selftests TARGETS=net run_tests

Do you know how to make the result looks like the CI shows[1], that each test
has a separate log dir and shows stdout/stderr.

[1] https://netdev.bots.linux.dev/contest.html

Thanks
Hangbin

Re: [ANN] Containers to run Netdev CI tests locally

[Ilya Maximets]

On 9/3/26 6:19 PM, Matthieu Baerts wrote:
> 
> A wiki page has been added to cover this and explain how to validate the
> different tests:
> 
>   https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
> 
> To help developers, containers are now built, packing all the required
> dependencies to run these tests in order to run the tests in an
> environment that would be as close as possible to the CI one. The goal
> is to have an environment that is easy to maintain -- no heavy
> optimisations to reduce the images size -- but also easy to use. In
> short, people (and non-humans) can continue to use the different tools
> as before, but add a prefix to the different commands to execute them
> from a container, e.g.
> 
>   alias run="${nipa}/docker/selftests/run.sh"
>   run vng --build
>   run make headers
>   (...)
The headers are not used unless the USERCFLAGS="-I$(pwd)/usr/include" is
passed into the actual make command for the selftests.  So, uAPI changes
will not be picked up.  Would be good to fix that in the wiki.

The $(pwd) part is a little tricky since it will be resolved outside of
the container.  I didn't try if that works.

Best regards, Ilya Maximets.

Re: [ANN] Containers to run Netdev CI tests locally

[Matthieu Baerts]

Hi Ilya,

Thank you for your reply!

On 04/09/2026 16:01, Ilya Maximets wrote:
> On 9/3/26 6:19 PM, Matthieu Baerts wrote:
>>
>> A wiki page has been added to cover this and explain how to validate the
>> different tests:
>>
>>   https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
>>
>> To help developers, containers are now built, packing all the required
>> dependencies to run these tests in order to run the tests in an
>> environment that would be as close as possible to the CI one. The goal
>> is to have an environment that is easy to maintain -- no heavy
>> optimisations to reduce the images size -- but also easy to use. In
>> short, people (and non-humans) can continue to use the different tools
>> as before, but add a prefix to the different commands to execute them
>> from a container, e.g.
>>
>>   alias run="${nipa}/docker/selftests/run.sh"
>>   run vng --build
>>   run make headers
>>   (...)
> The headers are not used unless the USERCFLAGS="-I$(pwd)/usr/include" is
> passed into the actual make command for the selftests.  So, uAPI changes
> will not be picked up.  Would be good to fix that in the wiki.

I think setting USERCFLAGS is not supposed to be required for this
purpose: KHDR_INCLUDES is there for that. According to the doc [1], each
target building binaries should have "CFLAGS = $(KHDR_INCLUDES)" in
their Makefile. If KHDR_INCLUDES is not set by the user, then lib.mk
will set it to "-I$(top_srcdir)/usr/include", so the default directory
when "make headers" is used. It needs to be adapted with building in a
different directory.

At least, when validating all net and drv-net selftests on my side, I
didn't need to specify extra CFLAGS. (I think the netfilter target is
missing KHDR_INCLUDES, I will fix that.)

Or did you have to do that for a specific target?


Just in case, here are the recommended commands to execute to run the
selftests per target:

    alias run="${nipa}/docker/selftests/run.sh"

    rm -f .config
    config="tools/testing/selftests/${target}/config"
    run vng --build -v --force --config "${config}"  ## + debug

    run make headers
    run make -C tools/testing/selftests TARGETS="${target}"

    run sudo vng -v --run . --user root -a mitigations=off --cpus 4 \
        --rw  --network loop -- \
        make -C tools/testing/selftests TARGETS="${target}" run_tests


> The $(pwd) part is a little tricky since it will be resolved outside of
> the container.  I didn't try if that works.

Note that in the container, the current directory is mounted at the same
place. So $(pwd) can be used.

[1]
https://docs.kernel.org/dev-tools/kselftest.html#contributing-new-tests-details

Cheers,
Matt

Re: [ANN] Containers to run Netdev CI tests locally

[Ilya Maximets]

On 9/4/26 4:54 PM, Matthieu Baerts wrote:
> Hi Ilya,
> 
> Thank you for your reply!
> 
> On 04/09/2026 16:01, Ilya Maximets wrote:
>> On 9/3/26 6:19 PM, Matthieu Baerts wrote:
>>>
>>> A wiki page has been added to cover this and explain how to validate the
>>> different tests:
>>>
>>>   https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
>>>
>>> To help developers, containers are now built, packing all the required
>>> dependencies to run these tests in order to run the tests in an
>>> environment that would be as close as possible to the CI one. The goal
>>> is to have an environment that is easy to maintain -- no heavy
>>> optimisations to reduce the images size -- but also easy to use. In
>>> short, people (and non-humans) can continue to use the different tools
>>> as before, but add a prefix to the different commands to execute them
>>> from a container, e.g.
>>>
>>>   alias run="${nipa}/docker/selftests/run.sh"
>>>   run vng --build
>>>   run make headers
>>>   (...)
>> The headers are not used unless the USERCFLAGS="-I$(pwd)/usr/include" is
>> passed into the actual make command for the selftests.  So, uAPI changes
>> will not be picked up.  Would be good to fix that in the wiki.
> 
> I think setting USERCFLAGS is not supposed to be required for this
> purpose: KHDR_INCLUDES is there for that. According to the doc [1], each
> target building binaries should have "CFLAGS = $(KHDR_INCLUDES)" in
> their Makefile. If KHDR_INCLUDES is not set by the user, then lib.mk
> will set it to "-I$(top_srcdir)/usr/include", so the default directory
> when "make headers" is used. It needs to be adapted with building in a
> different directory.
> 
> At least, when validating all net and drv-net selftests on my side, I
> didn't need to specify extra CFLAGS. (I think the netfilter target is
> missing KHDR_INCLUDES, I will fix that.)
> 
> Or did you have to do that for a specific target?

I was actually having trouble with the netfilter target specifically
as I'm working on a patch that changes uAPI and the selftest build
can't pick it up without the USERCFLAGS.  But if it's actually a bug
in the makefile for that target, then it's better to fix this target,
of course, instead of changing the docs. :)

Thanks for taking a look!

> 
> 
> Just in case, here are the recommended commands to execute to run the
> selftests per target:
> 
>     alias run="${nipa}/docker/selftests/run.sh"
> 
>     rm -f .config
>     config="tools/testing/selftests/${target}/config"
>     run vng --build -v --force --config "${config}"  ## + debug
> 
>     run make headers
>     run make -C tools/testing/selftests TARGETS="${target}"
> 
>     run sudo vng -v --run . --user root -a mitigations=off --cpus 4 \
>         --rw  --network loop -- \
>         make -C tools/testing/selftests TARGETS="${target}" run_tests
> 
> 
>> The $(pwd) part is a little tricky since it will be resolved outside of
>> the container.  I didn't try if that works.
> 
> Note that in the container, the current directory is mounted at the same
> place. So $(pwd) can be used.
> 
> [1]
> https://docs.kernel.org/dev-tools/kselftest.html#contributing-new-tests-details
> 
> Cheers,
> Matt

Re: [ANN] Containers to run Netdev CI tests locally

[Matthieu Baerts]

On 04/09/2026 17:25, Ilya Maximets wrote:
> On 9/4/26 4:54 PM, Matthieu Baerts wrote:
>> Hi Ilya,
>>
>> Thank you for your reply!
>>
>> On 04/09/2026 16:01, Ilya Maximets wrote:
>>> On 9/3/26 6:19 PM, Matthieu Baerts wrote:
>>>>
>>>> A wiki page has been added to cover this and explain how to validate the
>>>> different tests:
>>>>
>>>>   https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
>>>>
>>>> To help developers, containers are now built, packing all the required
>>>> dependencies to run these tests in order to run the tests in an
>>>> environment that would be as close as possible to the CI one. The goal
>>>> is to have an environment that is easy to maintain -- no heavy
>>>> optimisations to reduce the images size -- but also easy to use. In
>>>> short, people (and non-humans) can continue to use the different tools
>>>> as before, but add a prefix to the different commands to execute them
>>>> from a container, e.g.
>>>>
>>>>   alias run="${nipa}/docker/selftests/run.sh"
>>>>   run vng --build
>>>>   run make headers
>>>>   (...)
>>> The headers are not used unless the USERCFLAGS="-I$(pwd)/usr/include" is
>>> passed into the actual make command for the selftests.  So, uAPI changes
>>> will not be picked up.  Would be good to fix that in the wiki.
>>
>> I think setting USERCFLAGS is not supposed to be required for this
>> purpose: KHDR_INCLUDES is there for that. According to the doc [1], each
>> target building binaries should have "CFLAGS = $(KHDR_INCLUDES)" in
>> their Makefile. If KHDR_INCLUDES is not set by the user, then lib.mk
>> will set it to "-I$(top_srcdir)/usr/include", so the default directory
>> when "make headers" is used. It needs to be adapted with building in a
>> different directory.
>>
>> At least, when validating all net and drv-net selftests on my side, I
>> didn't need to specify extra CFLAGS. (I think the netfilter target is
>> missing KHDR_INCLUDES, I will fix that.)
>>
>> Or did you have to do that for a specific target?
> 
> I was actually having trouble with the netfilter target specifically
> as I'm working on a patch that changes uAPI and the selftest build
> can't pick it up without the USERCFLAGS.  But if it's actually a bug
> in the makefile for that target, then it's better to fix this target,
> of course, instead of changing the docs. :)

Indeed, it looks like you picked a target not setting KHDR_INCLUDES :)
Apparently the net/forwarding one also had the same issue.

Fixes have been sent. Don't hesitate to validate them ;)

https://lore.kernel.org/20260904-net-sft-nf-khdr_includes-v1-0-92455af428e9@kernel.org/

Cheers,
Matt

Re: [ANN] Containers to run Netdev CI tests locally

[Ilya Maximets]

On 9/4/26 6:17 PM, Matthieu Baerts wrote:
> On 04/09/2026 17:25, Ilya Maximets wrote:
>> On 9/4/26 4:54 PM, Matthieu Baerts wrote:
>>> Hi Ilya,
>>>
>>> Thank you for your reply!
>>>
>>> On 04/09/2026 16:01, Ilya Maximets wrote:
>>>> On 9/3/26 6:19 PM, Matthieu Baerts wrote:
>>>>>
>>>>> A wiki page has been added to cover this and explain how to validate the
>>>>> different tests:
>>>>>
>>>>>   https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
>>>>>
>>>>> To help developers, containers are now built, packing all the required
>>>>> dependencies to run these tests in order to run the tests in an
>>>>> environment that would be as close as possible to the CI one. The goal
>>>>> is to have an environment that is easy to maintain -- no heavy
>>>>> optimisations to reduce the images size -- but also easy to use. In
>>>>> short, people (and non-humans) can continue to use the different tools
>>>>> as before, but add a prefix to the different commands to execute them
>>>>> from a container, e.g.
>>>>>
>>>>>   alias run="${nipa}/docker/selftests/run.sh"
>>>>>   run vng --build
>>>>>   run make headers
>>>>>   (...)
>>>> The headers are not used unless the USERCFLAGS="-I$(pwd)/usr/include" is
>>>> passed into the actual make command for the selftests.  So, uAPI changes
>>>> will not be picked up.  Would be good to fix that in the wiki.
>>>
>>> I think setting USERCFLAGS is not supposed to be required for this
>>> purpose: KHDR_INCLUDES is there for that. According to the doc [1], each
>>> target building binaries should have "CFLAGS = $(KHDR_INCLUDES)" in
>>> their Makefile. If KHDR_INCLUDES is not set by the user, then lib.mk
>>> will set it to "-I$(top_srcdir)/usr/include", so the default directory
>>> when "make headers" is used. It needs to be adapted with building in a
>>> different directory.
>>>
>>> At least, when validating all net and drv-net selftests on my side, I
>>> didn't need to specify extra CFLAGS. (I think the netfilter target is
>>> missing KHDR_INCLUDES, I will fix that.)
>>>
>>> Or did you have to do that for a specific target?
>>
>> I was actually having trouble with the netfilter target specifically
>> as I'm working on a patch that changes uAPI and the selftest build
>> can't pick it up without the USERCFLAGS.  But if it's actually a bug
>> in the makefile for that target, then it's better to fix this target,
>> of course, instead of changing the docs. :)
> 
> Indeed, it looks like you picked a target not setting KHDR_INCLUDES :)
> Apparently the net/forwarding one also had the same issue.
> 
> Fixes have been sent. Don't hesitate to validate them ;)
> 
> https://lore.kernel.org/20260904-net-sft-nf-khdr_includes-v1-0-92455af428e9@kernel.org/
Will do.  Thanks!

Best regards, Ilya Maximets.

Re: [ANN] Containers to run Netdev CI tests locally

[Matthieu Baerts]

Hi Hangbin,

On 04/09/2026 13:44, Hangbin Liu wrote:
> On Fri, Sep 04, 2026 at 10:33:18AM +0200, Matthieu Baerts wrote:
>> Hi Hangbin,
>>
>> On 04/09/2026 08:59, Hangbin Liu wrote:
>>> Hi Matthieu,
>>>
>>> On Thu, Sep 03, 2026 at 06:19:25PM +0200, Matthieu Baerts wrote:
>>>> Hello,
>>>>
>>>> TL;DR: containers are now available with all dependencies to easily
>>>> execute all static, unit, and functional tests in an environment closed
>>>> to the Netdev CI one:
>>>>
>>>> - https://github.com/orgs/linux-netdev/packages
>>>>
>>>> - https://github.com/linux-netdev/nipa/wiki/Running-Netdev-CI-tests-locally
>>>
>>> Great! I've been struggling to set up the local NIPA docker. I'm following
>>> the steps from your git commit description. Now this article saved me.
>>
>> I'm glad it helped!
>>
> 
> From a user‑experience perspective, I think the static tests work well.
> They display the check results clearly and save outputs to the result directory.
> 
> However, for selftests, I do not see any difference compared to running vng
> directly on my local machine, i.e.
> 
>   vng -v --run . --user root -a mitigations=off --cpus 4 -- \
>   	make -C tools/testing/selftests TARGETS=net run_tests
> 
> Do you know how to make the result looks like the CI shows[1], that each test
> has a separate log dir and shows stdout/stderr.

Mmh, I understand the issue, but that's mainly kselftests related I think.

We could reuse python scripts from the NIPA repo to generate .json files
that will need to be processed to be presented in a local HTML page. But
that seems overcomplicated to put in place. At the end, the 'run_tests'
command will output a TAP format, which can be parsed with different
consumers [1] (e.g. tappy) or a simple "grep" command, e.g.

  grep -e "^not ok" -e "ok .* # SKIP"

An alternative I suggested a few months ago [2], is to print a summary
at the end of the "run_tests" command, e.g.

  # Totals: pass:1 fail:1 xfail:0 xpass:0 skip:1 error:0

Can you try it on your side to see if it improves your user experience?

Maybe this simple patch should be taken in the net-next tree. Often,
patches modifying the kselftest framework have to be taken via a
different tree I think...

[1] https://testanything.org/consumers.html
[2]
https://lore.kernel.org/20260625-sft-run-tests-print-totals-v1-1-bfe588266249@kernel.org

Cheers,
Matt