Re: [TEST] The no-kvm CI instances going away

Re: [TEST] The no-kvm CI instances going away

[Matthieu Baerts]

Hi Jakub,

On 06/02/2024 02:41, Jakub Kicinski wrote:
> because cloud computing is expensive I'm shutting down the instances
> which were running without KVM support. We're left with the KVM-enabled
> instances only (metal) - one normal and one with debug configs enabled.

Thank you for the notification!

It sounds like good news if the non-support of KVM was causing issues :)

I think we can then no longer ignore the two MPTCP tests that were
unstable in the previous environment.

The results from the different tests running on the -dbg instances don't
look good. Maybe some debug kconfig have a too big impact? [1]

For MPTCP, one test always hits the selftest timeout [2] when using a
debug kconfig. I don't know what to do in this case: if we need to set a
timeout value that is supported by debug environments, the value will be
so high, it will no longer catch issues "early enough" in "normal"
environments.
Or could it be possible to ignore or double the timeout value in this
debug environment?

Also, what is the plan with this debug env? It looks like the results
are not reported to patchwork for the moment. Maybe only "important"
issues, like kernel warnings, could be reported? Failed tests could be
reported as "Warning" instead of "Fail"?

[1]
https://lore.kernel.org/netdev/90c6d9b6-0bc4-468a-95fe-ebc2a23fffc1@kernel.org/
[2]
https://netdev-3.bots.linux.dev/vmksft-mptcp-dbg/results/453502/1-mptcp-join-sh/stdout

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [TEST] The no-kvm CI instances going away

[Jakub Kicinski]

On Tue, 6 Feb 2024 12:16:43 +0100 Matthieu Baerts wrote:
> Hi Jakub,
> 
> On 06/02/2024 02:41, Jakub Kicinski wrote:
> > because cloud computing is expensive I'm shutting down the instances
> > which were running without KVM support. We're left with the KVM-enabled
> > instances only (metal) - one normal and one with debug configs enabled.  
> 
> Thank you for the notification!
> 
> It sounds like good news if the non-support of KVM was causing issues :)
> 
> I think we can then no longer ignore the two MPTCP tests that were
> unstable in the previous environment.
> 
> The results from the different tests running on the -dbg instances don't
> look good. Maybe some debug kconfig have a too big impact? [1]

Sorry, I'm behind on the reading the list. FWIW if you want to reach me
quickly make sure the To: doesn't include anyone else. That gets sorted
to a higher prio folder :S

> For MPTCP, one test always hits the selftest timeout [2] when using a
> debug kconfig. I don't know what to do in this case: if we need to set a
> timeout value that is supported by debug environments, the value will be
> so high, it will no longer catch issues "early enough" in "normal"
> environments.
> Or could it be possible to ignore or double the timeout value in this
> debug environment?
> 
> Also, what is the plan with this debug env? It looks like the results
> are not reported to patchwork for the moment. Maybe only "important"
> issues, like kernel warnings, could be reported? Failed tests could be
> reported as "Warning" instead of "Fail"?

Unfortunately I'm really behind on my "real job". I don't have a clear
plan. I think we should scale the timeout by 2x or so, but I haven't
looked how to do that.

I wish the selftest subsystem had some basic guidance.

Re: [TEST] The no-kvm CI instances going away

[Matthieu Baerts]

Hi Jakub,

On 07/02/2024 02:44, Jakub Kicinski wrote:
> On Tue, 6 Feb 2024 12:16:43 +0100 Matthieu Baerts wrote:
>> Hi Jakub,
>>
>> On 06/02/2024 02:41, Jakub Kicinski wrote:
>>> because cloud computing is expensive I'm shutting down the instances
>>> which were running without KVM support. We're left with the KVM-enabled
>>> instances only (metal) - one normal and one with debug configs enabled.  
>>
>> Thank you for the notification!
>>
>> It sounds like good news if the non-support of KVM was causing issues :)
>>
>> I think we can then no longer ignore the two MPTCP tests that were
>> unstable in the previous environment.
>>
>> The results from the different tests running on the -dbg instances don't
>> look good. Maybe some debug kconfig have a too big impact? [1]
> 
> Sorry, I'm behind on the reading the list. FWIW if you want to reach me
> quickly make sure the To: doesn't include anyone else. That gets sorted
> to a higher prio folder :S

Sorry, there was no urgency, I only wanted to add a link to the previous
discussion for those who wanted more details about that.

Thank you for the note!

>> For MPTCP, one test always hits the selftest timeout [2] when using a
>> debug kconfig. I don't know what to do in this case: if we need to set a
>> timeout value that is supported by debug environments, the value will be
>> so high, it will no longer catch issues "early enough" in "normal"
>> environments.
>> Or could it be possible to ignore or double the timeout value in this
>> debug environment?
>>
>> Also, what is the plan with this debug env? It looks like the results
>> are not reported to patchwork for the moment. Maybe only "important"
>> issues, like kernel warnings, could be reported? Failed tests could be
>> reported as "Warning" instead of "Fail"?
> 
> Unfortunately I'm really behind on my "real job". I don't have a clear
> plan. I think we should scale the timeout by 2x or so, but I haven't
> looked how to do that.

No hurry, I understand.

It is not clear to me how the patches you add on top of the ones from
patchwork are managed. Then, I don't know if it can help, but on the
debug instance, this command could be launched before starting the tests
to double the timeout values in all the "net" selftests:

  $ find tools/testing/selftests/net -name settings -print0 | xargs -0 \
       awk -i inplace -F '=' \
           '{if ($1 == "timeout") { print $1 "=" $2*2 } else { print }}'

> I wish the selftest subsystem had some basic guidance.

Me too :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [TEST] The no-kvm CI instances going away

[Jakub Kicinski]

On Wed, 7 Feb 2024 10:44:14 +0100 Matthieu Baerts wrote:
> > Unfortunately I'm really behind on my "real job". I don't have a clear
> > plan. I think we should scale the timeout by 2x or so, but I haven't
> > looked how to do that.  
> 
> No hurry, I understand.
> 
> It is not clear to me how the patches you add on top of the ones from
> patchwork are managed. Then, I don't know if it can help, but on the
> debug instance, this command could be launched before starting the tests
> to double the timeout values in all the "net" selftests:
> 
>   $ find tools/testing/selftests/net -name settings -print0 | xargs -0 \
>        awk -i inplace -F '=' \
>            '{if ($1 == "timeout") { print $1 "=" $2*2 } else { print }}'

I'd rather not modify the tree. Poking around - this seems to work:

  export kselftest_override_timeout=1

Now it's just a matter of finding 15min to code it up :)

Re: [TEST] The no-kvm CI instances going away

[Matthieu Baerts]

On 07/02/2024 15:25, Jakub Kicinski wrote:
> On Wed, 7 Feb 2024 10:44:14 +0100 Matthieu Baerts wrote:
>>> Unfortunately I'm really behind on my "real job". I don't have a clear
>>> plan. I think we should scale the timeout by 2x or so, but I haven't
>>> looked how to do that.  
>>
>> No hurry, I understand.
>>
>> It is not clear to me how the patches you add on top of the ones from
>> patchwork are managed. Then, I don't know if it can help, but on the
>> debug instance, this command could be launched before starting the tests
>> to double the timeout values in all the "net" selftests:
>>
>>   $ find tools/testing/selftests/net -name settings -print0 | xargs -0 \
>>        awk -i inplace -F '=' \
>>            '{if ($1 == "timeout") { print $1 "=" $2*2 } else { print }}'
> 
> I'd rather not modify the tree. Poking around - this seems to work:
> 
>   export kselftest_override_timeout=1

Even better :)

  f=tools/testing/selftests/net/settings
  kselftest_override_timeout=$(awk -F = '/^timeout=/ {print $2*2}' $f)

> Now it's just a matter of finding 15min to code it up :)
I'm not sure if I can help here :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [TEST] The no-kvm CI instances going away

[Jakub Kicinski]

On Wed, 7 Feb 2024 15:37:22 +0100 Matthieu Baerts wrote:
> > I'd rather not modify the tree. Poking around - this seems to work:
> > 
> >   export kselftest_override_timeout=1  
> 
> Even better :)
> 
>   f=tools/testing/selftests/net/settings
>   kselftest_override_timeout=$(awk -F = '/^timeout=/ {print $2*2}' $f)
> 
> > Now it's just a matter of finding 15min to code it up :)  
> I'm not sure if I can help here :)

If you're willing to touch my nasty Python - that'd be very welcome! :)

Right now I put this in the configs:

[vm]
exports=KSFT_MACHINE_SLOW=yes

We can leave the support for adding exports in place, but we should add
a new config option like

[vm]
slowdown=2

(exact name up for debate), if it's present generate the
KSFT_MACHINE_SLOW=yes export automatically, without the explicit entry
in the config. And add the export for the timeout matching the logic
you propose (but in Python).

FWIW here's the configs we currently use:

$ cat remote.config
[remote]
branches=https://netdev.bots.linux.dev/static/nipa/branches.json
filters=https://netdev.bots.linux.dev/contest/filters.json
[local]
results_path=results
json_path=jsons
[env]
paths=/home/virtme/virtme-ng/
[vm]
paths=/home/virtme/tools/fs/bin:/home/virtme/tools/fs/sbin:/home/virtme/tools/fs/usr/bin:/home/virtme/tools/fs/usr/sbin
ld_paths=/lib64/:/home/virtme/tools/fs/usr/lib/:/home/virtme/tools/fs/lib64/:/home/virtme/tools/fs/usr/lib64/
init_prompt=bash-5.2#
default_timeout=200
boot_timeout=45


$ cat ksft-mptcp-dbg.config
[executor]
name=vmksft-mptcp-dbg
[local]
tree_path=/home/virtme/testing-12/
base_path=/home/virtme/outputs-12/
[www]
url=https://netdev-3.bots.linux.dev/vmksft-mptcp-dbg
[vm]
exports=KSFT_MACHINE_SLOW=yes
configs=kernel/configs/debug.config,kernel/configs/x86_debug.config
default_timeout=300
cpus=4
[ksft]
target=net/mptcp
[cfg]
thread_cnt=1

Re: [TEST] The no-kvm CI instances going away

[Matthieu Baerts]

On 07/02/2024 16:29, Jakub Kicinski wrote:
> On Wed, 7 Feb 2024 15:37:22 +0100 Matthieu Baerts wrote:
>>> I'd rather not modify the tree. Poking around - this seems to work:
>>>
>>>   export kselftest_override_timeout=1  
>>
>> Even better :)
>>
>>   f=tools/testing/selftests/net/settings
>>   kselftest_override_timeout=$(awk -F = '/^timeout=/ {print $2*2}' $f)
>>
>>> Now it's just a matter of finding 15min to code it up :)  
>> I'm not sure if I can help here :)
> 
> If you're willing to touch my nasty Python - that'd be very welcome! :)

Sure, I can have a look (but not today) :)

Thank you for the guide! Like with my PR to support subtests, I might
only test the code I modify not to have to set up everything, I hope it
is fine :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.