[TEST] kunit/cfg80211-ie-generation flaking ?

[TEST] kunit/cfg80211-ie-generation flaking ?

[Jakub Kicinski]

Hi!

Do you see any problems with the cfg80211-ie-generation kUnit test?

We hit 3 failures in the last 3 days.

https://netdev.bots.linux.dev/flakes.html?min-flip=0&tn-needle=cfg80211-ie-generation

But the kunit stuff likes to break because of cross-tests corruptions :(

We run:

  ./tools/testing/kunit/kunit.py run --alltests --json --arch=x86_64

Re: [TEST] kunit/cfg80211-ie-generation flaking ?

[Johannes Berg]

On Thu, 2025-02-13 at 17:37 +0000, Jakub Kicinski wrote:
> Hi!
> 
> Do you see any problems with the cfg80211-ie-generation kUnit test?

Nope, never, we must run it dozens of times a day ...

> We hit 3 failures in the last 3 days.

Four, actually, it seems? ;-)

> https://netdev.bots.linux.dev/flakes.html?min-flip=0&tn-needle=cfg80211-ie-generation
> 
> But the kunit stuff likes to break because of cross-tests corruptions :(

Hmm. Let's say ...

https://netdev-3.bots.linux.dev/kunit/results/987921/kunit-test.log

is your serial console simply too slow?

ok 70 cfg80211-inform-bss
    KTAP version 1
    # Subtest: cfg80211-ie-generation
    # module: cfg80211_tests
    1..2
        KTAP version 1
        # Subtest: test_gen_new_ie
        oaction: accept multicast without MFP

should say

"ok 4 public action: accept ..."

instead, I think?

johannes

Re: [TEST] kunit/cfg80211-ie-generation flaking ?

[Jakub Kicinski]

On Thu, 13 Feb 2025 19:22:04 +0100 Johannes Berg wrote:
> > We hit 3 failures in the last 3 days.  
> 
> Four, actually, it seems? ;-)

Yup! I jinxed it, it failed again after I sent the report :)

> > https://netdev.bots.linux.dev/flakes.html?min-flip=0&tn-needle=cfg80211-ie-generation
> > 
> > But the kunit stuff likes to break because of cross-tests corruptions :(  
> 
> Hmm. Let's say ...
> 
> https://netdev-3.bots.linux.dev/kunit/results/987921/kunit-test.log
> 
> is your serial console simply too slow?
> 
> ok 70 cfg80211-inform-bss
>     KTAP version 1
>     # Subtest: cfg80211-ie-generation
>     # module: cfg80211_tests
>     1..2
>         KTAP version 1
>         # Subtest: test_gen_new_ie
>         oaction: accept multicast without MFP
> 
> should say
> 
> "ok 4 public action: accept ..."
> 
> instead, I think?

Oh, that's annoying :( Thanks for investigating.
I think the CI runs when the machine is overloaded by builds.
I'll add some safety for that.

Re: [TEST] kunit/cfg80211-ie-generation flaking ?

[Johannes Berg]

On Thu, 2025-02-13 at 10:46 -0800, Jakub Kicinski wrote:
> On Thu, 13 Feb 2025 19:22:04 +0100 Johannes Berg wrote:
> > > We hit 3 failures in the last 3 days.  
> > 
> > Four, actually, it seems? ;-)
> 
> Yup! I jinxed it, it failed again after I sent the report :)

:)

It's weird that it happens in this test, or are others similar?

> > https://netdev-3.bots.linux.dev/kunit/results/987921/kunit-test.log
> > 
> > is your serial console simply too slow?
> > 
> > ok 70 cfg80211-inform-bss
> >     KTAP version 1
> >     # Subtest: cfg80211-ie-generation
> >     # module: cfg80211_tests
> >     1..2
> >         KTAP version 1
> >         # Subtest: test_gen_new_ie
> >         oaction: accept multicast without MFP
> > 
> > should say
> > 
> > "ok 4 public action: accept ..."
> > 
> > instead, I think?
> 
> Oh, that's annoying :( Thanks for investigating.
> I think the CI runs when the machine is overloaded by builds.
> I'll add some safety for that.

It almost feels like it shouldn't matter - couldn't qemu just kind of
'pause' the VM when the serial port isn't keeping up? I think you're
using qemu? But I guess I could also see why that might not be something
you want in other use cases...

Not sure, but it really seems more related to the output (buffering)
than anything else.

Are you using the tools/testing/kunit/kunit.py script?

johannes

Re: [TEST] kunit/cfg80211-ie-generation flaking ?

[Jakub Kicinski]

On Thu, 13 Feb 2025 20:03:04 +0100 Johannes Berg wrote:
> > Oh, that's annoying :( Thanks for investigating.
> > I think the CI runs when the machine is overloaded by builds.
> > I'll add some safety for that.  
> 
> It almost feels like it shouldn't matter - couldn't qemu just kind of
> 'pause' the VM when the serial port isn't keeping up? I think you're
> using qemu? But I guess I could also see why that might not be something
> you want in other use cases...
> 
> Not sure, but it really seems more related to the output (buffering)
> than anything else.
> 
> Are you using the tools/testing/kunit/kunit.py script?

Yes. We do set:

Environment=PYTHONUNBUFFERED=true

So that the logs reach journald without a huge delay.
I guess it disturbs the kunit wrapper.