[TEST] google runner needs nested cases

[TEST] google runner needs nested cases

[Jakub Kicinski]

Hi!

Another ask, please parse the TAP output of the tests and provide 
a nested array of results per-subcase, see:

https://github.com/linux-netdev/nipa/wiki/Netdev-CI-system/#results

right now you only seem to report top level pass/fail but most tests
have at least one sub-case the driver doesn't support.

Re: [TEST] google runner needs nested cases

[Sheena Mohan]

Thanks, Jakub. We plan to merge the changes to support nested arrays
for sub-test results by next week.


On Tue, Jul 21, 2026 at 6:38 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> Hi!
>
> Another ask, please parse the TAP output of the tests and provide
> a nested array of results per-subcase, see:
>
> https://github.com/linux-netdev/nipa/wiki/Netdev-CI-system/#results
>
> right now you only seem to report top level pass/fail but most tests
> have at least one sub-case the driver doesn't support.

Re: [TEST] google runner needs nested cases

[Jakub Kicinski]

On Thu, 23 Jul 2026 10:14:20 -0700 Sheena Mohan wrote:
> Thanks, Jakub. We plan to merge the changes to support nested arrays
> for sub-test results by next week.

SG, please ping again when you do, so I can check and close the request
on my side.

Re: [TEST] google runner needs nested cases

[Sheena Mohan]

Hello Jakub,

The recent updates have been merged, and the tests should now report
sub-case results.
Please let us know if any further improvements are needed.

Thanks,
Sheena

On Thu, Jul 23, 2026 at 10:36 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Thu, 23 Jul 2026 10:14:20 -0700 Sheena Mohan wrote:
> > Thanks, Jakub. We plan to merge the changes to support nested arrays
> > for sub-test results by next week.
>
> SG, please ping again when you do, so I can check and close the request
> on my side.

Re: [TEST] google runner needs nested cases

[Jakub Kicinski]

On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
> The recent updates have been merged, and the tests should now report
> sub-case results.
> Please let us know if any further improvements are needed.

Looks good, thanks!

I'm still waiting on the device node, but I assume we're stacking that
separately.

Re: [TEST] google runner needs nested cases

[Sheena Mohan]

On Tue, Jul 28, 2026 at 3:13 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
> > The recent updates have been merged, and the tests should now report
> > sub-case results.
> > Please let us know if any further improvements are needed.
>
> Looks good, thanks!
>
> I'm still waiting on the device node, but I assume we're stacking that
> separately.
Yes, we are tracking that separately. I will share an update on
devlink support once the patches are submitted for upstream review.

Re: [TEST] google runner needs nested cases

[Jakub Kicinski]

On Tue, 28 Jul 2026 15:38:56 -0700 Sheena Mohan wrote:
> On Tue, Jul 28, 2026 at 3:13 PM Jakub Kicinski <kuba@kernel.org> wrote:
> >
> > On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:  
> > > The recent updates have been merged, and the tests should now report
> > > sub-case results.
> > > Please let us know if any further improvements are needed.  
> >
> > Looks good, thanks!
> >
> > I'm still waiting on the device node, but I assume we're stacking that
> > separately.  
> Yes, we are tracking that separately. I will share an update on
> devlink support once the patches are submitted for upstream review.

To be clear -- it's not required to be real devlink.
The format has to be like devlink (JSON keys).
For QEMU virtio for example we have a two liner
bash script in NIPA:

qver=$(qemu-system-x86_64 --version | head -1)

echo '{"driver":"virtio_net","versions":{"fixed":{},"stored":{},"running":{"fw":"'"$qver"'"}}}'

https://github.com/linux-netdev/nipa/blob/main/contest/scripts/vm-virtio-dev-info.sh

Of course real devlink reporting is better because
it may also help users correlate problems in production.

Re: [TEST] google runner needs nested cases

[Sheena Mohan]

On Tue, Jul 28, 2026 at 4:22 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 28 Jul 2026 15:38:56 -0700 Sheena Mohan wrote:
> > On Tue, Jul 28, 2026 at 3:13 PM Jakub Kicinski <kuba@kernel.org> wrote:
> > >
> > > On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
> > > > The recent updates have been merged, and the tests should now report
> > > > sub-case results.
> > > > Please let us know if any further improvements are needed.
> > >
> > > Looks good, thanks!
> > >
> > > I'm still waiting on the device node, but I assume we're stacking that
> > > separately.
> > Yes, we are tracking that separately. I will share an update on
> > devlink support once the patches are submitted for upstream review.
>
> To be clear -- it's not required to be real devlink.
> The format has to be like devlink (JSON keys).
> For QEMU virtio for example we have a two liner
> bash script in NIPA:
>
> qver=$(qemu-system-x86_64 --version | head -1)
>
> echo '{"driver":"virtio_net","versions":{"fixed":{},"stored":{},"running":{"fw":"'"$qver"'"}}}'
>
> https://github.com/linux-netdev/nipa/blob/main/contest/scripts/vm-virtio-dev-info.sh
>
> Of course real devlink reporting is better because
> it may also help users correlate problems in production.
Thanks for the clarification. Yes, we can unblock NIPA testing by
providing the information via the script until the real devlink
feature is supported in the drivers.

Re: [TEST] google runner needs nested cases

[Jakub Kicinski]

On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
> The recent updates have been merged, and the tests should now report
> sub-case results.
> Please let us know if any further improvements are needed.

It doesn't look right.

https://gve-netdev-nipa.static.usercontent.goog/json/results-202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a.json

test	"napi-threaded-py"
group	"selftests-drivers-net"
result	"pass"
^^^^^^^^^^^^^
link	"https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
time	20.788391

# selftests: drivers/net: napi_threaded.py
not ok 1 selftests: drivers/net: napi_threaded.py # exit=1
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
# not ok 1 napi_threaded.napi_init
# not ok 2 napi_threaded.change_num_queues
# not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threaded

Does not look like a pass at all. Subcases not in JSON.
Not sure why the subcases are listed under the main return 
in the output, I've never seen that.


test	"psp-py"
group	"selftests-drivers-net"
result	"pass"
link	"https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html"
time	341.833939

# selftests: drivers/net: psp.py
not ok 1 selftests: drivers/net: psp.py # exit=1
# Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0

Again, doesn't seem to be a pass??


test	"queues-py"
group	"selftests-drivers-net"
result	"pass"
link	"https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html"
time	686.483495
results	
	test	"queues-check-xsk"
	result	"pass"

Okay, here we have a subcase..

# selftests: drivers/net: queues.py
ok 1 selftests: drivers/net: queues.py
# Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
check_down
# ok 4 queues.check_xsk
# # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0

But why only one? What happened to the other 3 that ksft clearly saw
since it enumerated check_xsk as the fourth??


Something is mangling the output on your runner, it seems.
And the parsing seems off.

Re: [TEST] google runner needs nested cases

[Sheena Mohan]

On Tue, Aug 4, 2026 at 7:25 AM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 28 Jul 2026 10:07:15 -0700 Sheena Mohan wrote:
> > The recent updates have been merged, and the tests should now report
> > sub-case results.
> > Please let us know if any further improvements are needed.
>
> It doesn't look right.
>
> https://gve-netdev-nipa.static.usercontent.goog/json/results-202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a.json
>
> test    "napi-threaded-py"
> group   "selftests-drivers-net"
> result  "pass"
> ^^^^^^^^^^^^^
> link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/napi-threaded-py/logs.html"
> time    20.788391
>
> # selftests: drivers/net: napi_threaded.py
> not ok 1 selftests: drivers/net: napi_threaded.py # exit=1
> # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
> # not ok 1 napi_threaded.napi_init
> # not ok 2 napi_threaded.change_num_queues
> # not ok 3 napi_threaded.enable_dev_threaded_disable_napi_threaded
>
> Does not look like a pass at all. Subcases not in JSON.
> Not sure why the subcases are listed under the main return
> in the output, I've never seen that.
>
>
> test    "psp-py"
> group   "selftests-drivers-net"
> result  "pass"
> link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/psp-py/logs.html"
> time    341.833939
>
> # selftests: drivers/net: psp.py
> not ok 1 selftests: drivers/net: psp.py # exit=1
> # Totals: pass:0 fail:1 xfail:0 xpass:0 skip:0 error:0
>
> Again, doesn't seem to be a pass??
>
>
> test    "queues-py"
> group   "selftests-drivers-net"
> result  "pass"
> link    "https://gve-netdev-nipa.static.usercontent.goog/results/202608040108-88cb54ab-ca48-426b-9c9a-50bfbb0de32a/queues-py/logs.html"
> time    686.483495
> results
>         test    "queues-check-xsk"
>         result  "pass"
>
> Okay, here we have a subcase..
>
> # selftests: drivers/net: queues.py
> ok 1 selftests: drivers/net: queues.py
> # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0
> check_down
> # ok 4 queues.check_xsk
> # # Totals: pass:4 fail:0 xfail:0 xpass:0 skip:0 error:0
>
> But why only one? What happened to the other 3 that ksft clearly saw
> since it enumerated check_xsk as the fourth??
>
>
> Something is mangling the output on your runner, it seems.
> And the parsing seems off.
Thanks for bringing this to our attention. We will look into the
parsing issues and work on getting this fixed.