Re: [PATCH nft 2/5] tests: shell: let netdev_chain_0 test indicate SKIP if kernel requires netdev device

[Thomas Haller <thaller@redhat.com>]

On Mon, 2023-09-04 at 11:06 +0200, Florian Westphal wrote:
> This test case only works on kernel 6.4+.
> Add feature probe for this and then exit early.
> 
> We don't want to indicate a test failure, as this test doesn't apply
> on older kernels.
> 
> But we should not indicate sucess either, else we might be fooled
> in case something went wrong during feature probe.
> 
> Add a special return value, 123, and let run-tests.sh count this
> as 'SKIPPED'.
> 

[...]

>  failed=0
> +skipped=0
>  taint=0
>  
>  check_features()
> @@ -270,6 +271,9 @@ do
>                                 msg_warn "[DUMP FAIL]   $testfile"
>                         fi
>                 fi
> +       elif [ "$rc_got" -eq 123 ]; then
> +               ((skipped++))
> +               msg_info "[SKIPPED]     $testfile"

I agree with Phil, I think this should return 77.

Btw, I did a similar patch on 

  [PATCH nft v5 08/19] tests/shell: interpret an exit code of 77 from scripts as "skipped"

Granted, you send your first version with this patch/idea a few hours
before mine. I just point out the overlap...


>         else
>                 ((failed++))
>                 if [ "$VERBOSE" == "y" ] ; then
> @@ -294,7 +298,12 @@ echo ""
>  kmemleak_found=0
>  check_kmemleak_force
>  
> -msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
> +msg_info "results: [OK] $ok [FAILED] $failed [SKIPPED] $skipped
> [TOTAL] $((ok+failed+skipped))"
> +
> +if [ $ok -eq 0 -a  $failed -eq 0 ]; then
> +       # no test cases were run, indicate a failure
> +       failed=1
> +fi

I think this should be dropped.

When I run

  ./tests/shell/run-tests.sh tests/shell/testcases/maps/typeof_maps_add_delete

on my Fedora38, the test gets skipped. But I think the command should
just give me a success. What would I do about the "failure" anyway?


Thomas