[TEST] Wiki / instructions

[TEST] Wiki / instructions

[Jakub Kicinski]

Hi folks!

We added a wiki page to the nipa repo on how to run the tests locally:

https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests

Also feel free to request membership in the linux-netdev org to be able
to edit!

Re: [TEST] Wiki / instructions

[Matthieu Baerts]

Hi Jakub,

On 02/02/2024 18:31, Jakub Kicinski wrote:
> We added a wiki page to the nipa repo on how to run the tests locally:
> 
> https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests

Thank you for this wiki page, and all the work with the CI infrastructure!

For the debug options, I see that you are using:

  kernel/configs/x86_debug.config

It looks like this is specific for the 'tip' tree:

  Debugging options for tip tree testing

I don't know if it is still maintained, e.g. it includes DEBUG_SLAB
option. But also, it enables options that are maybe not needed: GCOV?
X86_DEBUG_FPU?
Maybe it is better not to use this .config file, no?

For our CI validating MPTCP tests in a "debug" mode, we use
"debug.config" without "x86_debug.config". On top of that, we also
disable "SLUB_DEBUG_ON", because the impact on the perf is too
important, especially with slow environments. We think it is not worth
it for our case. You don't have the same hardware, but if you have perf
issues, don't hesitate to do the same ;)

> Also feel free to request membership in the linux-netdev org to be able
> to edit!

Linked to the discussion we had last week, I can describe which patches
are being tested if you think it can help others :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.
(matttbe on GitHub)

Re: [TEST] Wiki / instructions

[Jakub Kicinski]

On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:
> Thank you for this wiki page, and all the work with the CI infrastructure!
> 
> For the debug options, I see that you are using:
> 
>   kernel/configs/x86_debug.config
> 
> It looks like this is specific for the 'tip' tree:
> 
>   Debugging options for tip tree testing
> 
> I don't know if it is still maintained, e.g. it includes DEBUG_SLAB
> option. But also, it enables options that are maybe not needed: GCOV?
> X86_DEBUG_FPU?
> Maybe it is better not to use this .config file, no?

I haven't looked to closely. I noticed that the basic debug config
doesn't enable LOCKDEP ?! so I put the x86 one on top.

I added a local patch to cut out all the obviously pointless stuff from
x86_debug.config, we should probably start our own config for networking
at some stage.

> For our CI validating MPTCP tests in a "debug" mode, we use
> "debug.config" without "x86_debug.config". On top of that, we also
> disable "SLUB_DEBUG_ON", because the impact on the perf is too
> important, especially with slow environments. We think it is not worth
> it for our case. You don't have the same hardware, but if you have perf
> issues, don't hesitate to do the same ;)

The mptcp tests take <60min to run with debug enabled, and just 
a single thread / VM. I think that's fine for now. But thanks for 
the heads up that SLUB_DEBUG_ON is problematic, for it may matter for
forwarding or net tests.

Re: [TEST] Wiki / instructions

[Matthieu Baerts]

Hi Jakub,

Thank you for your reply!

On 07/02/2024 02:37, Jakub Kicinski wrote:
> On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:
>> Thank you for this wiki page, and all the work with the CI infrastructure!
>>
>> For the debug options, I see that you are using:
>>
>>   kernel/configs/x86_debug.config
>>
>> It looks like this is specific for the 'tip' tree:
>>
>>   Debugging options for tip tree testing
>>
>> I don't know if it is still maintained, e.g. it includes DEBUG_SLAB
>> option. But also, it enables options that are maybe not needed: GCOV?
>> X86_DEBUG_FPU?
>> Maybe it is better not to use this .config file, no?
> 
> I haven't looked to closely. I noticed that the basic debug config
> doesn't enable LOCKDEP ?! so I put the x86 one on top.

I was surprised not to see LOCKDEP there, but in fact, it is: it enables
PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.

So maybe it is not needed to include the x86 one?

> I added a local patch to cut out all the obviously pointless stuff from
> x86_debug.config

Thank you!

> we should probably start our own config for networking
> at some stage.

Good idea!

On our side, we always enable DEBUG_NET, and the "debug" environment
also has NET_NS_REFCNT_TRACKER. We should probably enable
NET_DEV_REFCNT_TRACKER too.

Do you want me to add a new file in "kernel/configs" for net including
these 3 options?

Not directly related to "Net", we also enable DEBUG_INFO (+ compressed)
everywhere + KFENCE in the "non-debug" env only, disable RETPOLINE (+
mitigations=off) not to slow down the tests in already slow envs, and
disable a few components we don't need to accelerate the build and boot:
DRM, SOUND, etc.

https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284

It is also possible to add some kconfig in the selftests if preferred,
e.g. in

  ./tools/testing/selftests/net/debug.config

>> For our CI validating MPTCP tests in a "debug" mode, we use
>> "debug.config" without "x86_debug.config". On top of that, we also
>> disable "SLUB_DEBUG_ON", because the impact on the perf is too
>> important, especially with slow environments. We think it is not worth
>> it for our case. You don't have the same hardware, but if you have perf
>> issues, don't hesitate to do the same ;)
> 
> The mptcp tests take <60min to run with debug enabled, and just 
> a single thread / VM. I think that's fine for now. But thanks for 
> the heads up that SLUB_DEBUG_ON is problematic, for it may matter for
> forwarding or net tests.

The longest MPTCP selftest is currently stopped after 30 minutes due to
the selftest timeout. I will see what we can do. That's not just because
of SLUB_DEBUG_ON, that's normal to be very slow in such particular env.

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [TEST] Wiki / instructions

[Jakub Kicinski]

On Wed, 7 Feb 2024 09:50:48 +0100 Matthieu Baerts wrote:
> On 07/02/2024 02:37, Jakub Kicinski wrote:
> > On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:  
> >> Thank you for this wiki page, and all the work with the CI infrastructure!
> >>
> >> For the debug options, I see that you are using:
> >>
> >>   kernel/configs/x86_debug.config
> >>
> >> It looks like this is specific for the 'tip' tree:
> >>
> >>   Debugging options for tip tree testing
> >>
> >> I don't know if it is still maintained, e.g. it includes DEBUG_SLAB
> >> option. But also, it enables options that are maybe not needed: GCOV?
> >> X86_DEBUG_FPU?
> >> Maybe it is better not to use this .config file, no?  
> > 
> > I haven't looked to closely. I noticed that the basic debug config
> > doesn't enable LOCKDEP ?! so I put the x86 one on top.  
> 
> I was surprised not to see LOCKDEP there, but in fact, it is: it enables
> PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.
> 
> So maybe it is not needed to include the x86 one?

I'm confused. Now doing:

  make O=built_test defconfig
  make O=built_test debug.config

I don't get KASAN but I get LOCKDEP :S Bleh, maybe because of the
options vng throws in?

> > I added a local patch to cut out all the obviously pointless stuff from
> > x86_debug.config  
> 
> Thank you!
> 
> > we should probably start our own config for networking
> > at some stage.  
> 
> Good idea!
> 
> On our side, we always enable DEBUG_NET, and the "debug" environment
> also has NET_NS_REFCNT_TRACKER. We should probably enable
> NET_DEV_REFCNT_TRACKER too.
> 
> Do you want me to add a new file in "kernel/configs" for net including
> these 3 options?

Just the three? What about KASAN, OBJECT debug, DEBUG_VM etc?
As much as we can without going over the 3h limit in the tests :)

> Not directly related to "Net", we also enable DEBUG_INFO (+ compressed)
> everywhere

We have debug_info, maybe vng adds it..

> + KFENCE in the "non-debug" env only,

We could do KFENCE I guess. I'm a bit surprised it's not on by default
for x86. My first choice would be to try to change that..

> disable RETPOLINE (+
> mitigations=off) not to slow down the tests in already slow envs, and
> disable a few components we don't need to accelerate the build and boot:

RETPOLINE we could kill, agreed

> DRM, SOUND, etc.
> 
> https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284

Yes, vng also adds some stuff we don't need in this area :(

> It is also possible to add some kconfig in the selftests if preferred,
> e.g. in
> 
>   ./tools/testing/selftests/net/debug.config

Would be great if everyone didn't have to go thru this exercise.
How about we start sending patches to kernel/configs/debug.config
and see if anyone screams at us?

> >> For our CI validating MPTCP tests in a "debug" mode, we use
> >> "debug.config" without "x86_debug.config". On top of that, we also
> >> disable "SLUB_DEBUG_ON", because the impact on the perf is too
> >> important, especially with slow environments. We think it is not worth
> >> it for our case. You don't have the same hardware, but if you have perf
> >> issues, don't hesitate to do the same ;)  
> > 
> > The mptcp tests take <60min to run with debug enabled, and just 
> > a single thread / VM. I think that's fine for now. But thanks for 
> > the heads up that SLUB_DEBUG_ON is problematic, for it may matter for
> > forwarding or net tests.  
> 
> The longest MPTCP selftest is currently stopped after 30 minutes due to
> the selftest timeout. I will see what we can do. That's not just because
> of SLUB_DEBUG_ON, that's normal to be very slow in such particular env.

I'll 2x the timeouts before reporting debug to patchwork, so don't
worry about it, yet. 

Re: [TEST] Wiki / instructions

[Matthieu Baerts]

On 07/02/2024 16:21, Jakub Kicinski wrote:
> On Wed, 7 Feb 2024 09:50:48 +0100 Matthieu Baerts wrote:
>> On 07/02/2024 02:37, Jakub Kicinski wrote:
>>> On Mon, 5 Feb 2024 18:21:56 +0100 Matthieu Baerts wrote:  
>>>> Thank you for this wiki page, and all the work with the CI infrastructure!
>>>>
>>>> For the debug options, I see that you are using:
>>>>
>>>>   kernel/configs/x86_debug.config
>>>>
>>>> It looks like this is specific for the 'tip' tree:
>>>>
>>>>   Debugging options for tip tree testing
>>>>
>>>> I don't know if it is still maintained, e.g. it includes DEBUG_SLAB
>>>> option. But also, it enables options that are maybe not needed: GCOV?
>>>> X86_DEBUG_FPU?
>>>> Maybe it is better not to use this .config file, no?  
>>>
>>> I haven't looked to closely. I noticed that the basic debug config
>>> doesn't enable LOCKDEP ?! so I put the x86 one on top.  
>>
>> I was surprised not to see LOCKDEP there, but in fact, it is: it enables
>> PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.
>>
>> So maybe it is not needed to include the x86 one?
> 
> I'm confused. Now doing:
> 
>   make O=built_test defconfig
>   make O=built_test debug.config
> 
> I don't get KASAN but I get LOCKDEP :S Bleh, maybe because of the
> options vng throws in?

Strange, I get both of them on my side.

Same if I include the two targets:

  make O=built_test defconfig debug.config

>>> I added a local patch to cut out all the obviously pointless stuff from
>>> x86_debug.config  
>>
>> Thank you!
>>
>>> we should probably start our own config for networking
>>> at some stage.  
>>
>> Good idea!
>>
>> On our side, we always enable DEBUG_NET, and the "debug" environment
>> also has NET_NS_REFCNT_TRACKER. We should probably enable
>> NET_DEV_REFCNT_TRACKER too.
>>
>> Do you want me to add a new file in "kernel/configs" for net including
>> these 3 options?
> 
> Just the three? What about KASAN, OBJECT debug, DEBUG_VM etc?
> As much as we can without going over the 3h limit in the tests :)

Sorry, I meant to say: we would use the existing "debug.config" + a new
"net_debug.config" containing these 3 options (and maybe more that are
specific to the net tree).

>> Not directly related to "Net", we also enable DEBUG_INFO (+ compressed)
>> everywhere
> 
> We have debug_info, maybe vng adds it..

Great!

>> + KFENCE in the "non-debug" env only,
> 
> We could do KFENCE I guess. I'm a bit surprised it's not on by default
> for x86. My first choice would be to try to change that..

Good point. I can ask KFENCE maintainers what they think about that.

>> disable RETPOLINE (+
>> mitigations=off) not to slow down the tests in already slow envs, and
>> disable a few components we don't need to accelerate the build and boot:
> 
> RETPOLINE we could kill, agreed

I still need to check vrg: is it easy to add kconfig you want to
enable/disable? Do you need a new file? Should we maintain this file in
the tree?

>> DRM, SOUND, etc.
>>
>> https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284
> 
> Yes, vng also adds some stuff we don't need in this area :(

With the non-ng virtme, I disable those:

  -d PCCARD -d MACINTOSH_DRIVERS -d SOUND -d USB_SUPPORT -d NEW_LEDS
  -d SURFACE_PLATFORMS -d DRM -d FB

I will check when I will switch to vng to see what else we can disable.
But on your side, you probably have more CPU resources to compile the
kernel, and that's fine to keep them :)

>> It is also possible to add some kconfig in the selftests if preferred,
>> e.g. in
>>
>>   ./tools/testing/selftests/net/debug.config
> 
> Would be great if everyone didn't have to go thru this exercise.
> How about we start sending patches to kernel/configs/debug.config
> and see if anyone screams at us?

I guess the 3 net debug ones could go there indeed.

Do you want me to try?

>>>> For our CI validating MPTCP tests in a "debug" mode, we use
>>>> "debug.config" without "x86_debug.config". On top of that, we also
>>>> disable "SLUB_DEBUG_ON", because the impact on the perf is too
>>>> important, especially with slow environments. We think it is not worth
>>>> it for our case. You don't have the same hardware, but if you have perf
>>>> issues, don't hesitate to do the same ;)  
>>>
>>> The mptcp tests take <60min to run with debug enabled, and just 
>>> a single thread / VM. I think that's fine for now. But thanks for 
>>> the heads up that SLUB_DEBUG_ON is problematic, for it may matter for
>>> forwarding or net tests.  
>>
>> The longest MPTCP selftest is currently stopped after 30 minutes due to
>> the selftest timeout. I will see what we can do. That's not just because
>> of SLUB_DEBUG_ON, that's normal to be very slow in such particular env.
> 
> I'll 2x the timeouts before reporting debug to patchwork, so don't
> worry about it, yet. 

Thanks :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [TEST] Wiki / instructions

[Jakub Kicinski]

On Wed, 7 Feb 2024 16:59:31 +0100 Matthieu Baerts wrote:
> On 07/02/2024 16:21, Jakub Kicinski wrote:
> > On Wed, 7 Feb 2024 09:50:48 +0100 Matthieu Baerts wrote:  
> >> I was surprised not to see LOCKDEP there, but in fact, it is: it enables
> >> PROVE_LOCKING, which selects LOCKDEP and a few other DEBUG_xxx ones.
> >>
> >> So maybe it is not needed to include the x86 one?  
> > 
> > I'm confused. Now doing:
> > 
> >   make O=built_test defconfig
> >   make O=built_test debug.config
> > 
> > I don't get KASAN but I get LOCKDEP :S Bleh, maybe because of the
> > options vng throws in?  
> 
> Strange, I get both of them on my side.
> 
> Same if I include the two targets:
> 
>   make O=built_test defconfig debug.config

:o Maybe somehow doing it in two separate commands makes a difference?

I tried now and all that's changing without the x86 config is:

-CONFIG_DEBUG_PAGEALLOC=y
-# CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT is not set
+# CONFIG_DEBUG_PAGEALLOC is not set

-CONFIG_LOCK_STAT=y
+# CONFIG_LOCK_STAT is not set


I don't know if PAGEALLOC is useful but let's assume it's not important
enough to add the extra config.

> > Just the three? What about KASAN, OBJECT debug, DEBUG_VM etc?
> > As much as we can without going over the 3h limit in the tests :)  
> 
> Sorry, I meant to say: we would use the existing "debug.config" + a new
> "net_debug.config" containing these 3 options (and maybe more that are
> specific to the net tree).

Turns out that if I try with vng DEBUG_VM and KASAN _are_ enabled :(

> >> disable RETPOLINE (+
> >> mitigations=off) not to slow down the tests in already slow envs, and
> >> disable a few components we don't need to accelerate the build and boot:  
> > 
> > RETPOLINE we could kill, agreed  
> 
> I still need to check vrg: is it easy to add kconfig you want to
> enable/disable? Do you need a new file? Should we maintain this file in
> the tree?

Sorry for going in circles. Seems like you were right that we don't
need the x86 debug, so on the debug side all we need is the two
networking options? Can we try to add those to debug.config ?

> >> DRM, SOUND, etc.
> >>
> >> https://github.com/multipath-tcp/mptcp-upstream-virtme-docker/blob/latest/entrypoint.sh#L284  
> > 
> > Yes, vng also adds some stuff we don't need in this area :(  
> 
> With the non-ng virtme, I disable those:
> 
>   -d PCCARD -d MACINTOSH_DRIVERS -d SOUND -d USB_SUPPORT -d NEW_LEDS
>   -d SURFACE_PLATFORMS -d DRM -d FB
> 
> I will check when I will switch to vng to see what else we can disable.
> But on your side, you probably have more CPU resources to compile the
> kernel, and that's fine to keep them :)

Yup!

> >> It is also possible to add some kconfig in the selftests if preferred,
> >> e.g. in
> >>
> >>   ./tools/testing/selftests/net/debug.config  
> > 
> > Would be great if everyone didn't have to go thru this exercise.
> > How about we start sending patches to kernel/configs/debug.config
> > and see if anyone screams at us?  
> 
> I guess the 3 net debug ones could go there indeed.
> 
> Do you want me to try?

Yes, please! (Unless you don't like being screamed at, then I can :))

Re: [TEST] Wiki / instructions

[Simon Horman]

On Fri, Feb 02, 2024 at 09:31:48AM -0800, Jakub Kicinski wrote:
> Hi folks!
> 
> We added a wiki page to the nipa repo on how to run the tests locally:
> 
> https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests

Thanks for this.

I am wondering if the URL should be / is now:

https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

...

Re: [TEST] Wiki / instructions

[Jakub Kicinski]

On Fri, 9 Feb 2024 10:25:10 +0000 Simon Horman wrote:
> On Fri, Feb 02, 2024 at 09:31:48AM -0800, Jakub Kicinski wrote:
> > Hi folks!
> > 
> > We added a wiki page to the nipa repo on how to run the tests locally:
> > 
> > https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests  
> 
> Thanks for this.
> 
> I am wondering if the URL should be / is now:
> 
> https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style

Yes, I didn't realize editing the title would change the URL.
There's no permanent link either, it seems.
I added some links to the home page:
https://github.com/linux-netdev/nipa/wiki
And also the contest page now links directly to the repro instructions
(top right corner). Is there any other place we need to update?

Re: [TEST] Wiki / instructions

[Simon Horman]

On Fri, Feb 09, 2024 at 07:30:04AM -0800, Jakub Kicinski wrote:
> On Fri, 9 Feb 2024 10:25:10 +0000 Simon Horman wrote:
> > On Fri, Feb 02, 2024 at 09:31:48AM -0800, Jakub Kicinski wrote:
> > > Hi folks!
> > > 
> > > We added a wiki page to the nipa repo on how to run the tests locally:
> > > 
> > > https://github.com/linux-netdev/nipa/wiki/How-to-test-netdev-selftests  
> > 
> > Thanks for this.
> > 
> > I am wondering if the URL should be / is now:
> > 
> > https://github.com/linux-netdev/nipa/wiki/How-to-run-netdev-selftests-CI-style
> 
> Yes, I didn't realize editing the title would change the URL.
> There's no permanent link either, it seems.
> I added some links to the home page:
> https://github.com/linux-netdev/nipa/wiki
> And also the contest page now links directly to the repro instructions
> (top right corner). Is there any other place we need to update?

Thanks, I'm not aware of any other places that need updating.