Re: [PATCH] testing: kselftest_harness: add filtering and enumerating tests

[Shuah Khan <skhan@linuxfoundation.org>]

On 1/25/23 16:13, Jakub Kicinski wrote:
> As the number of test cases and length of execution grows it's
> useful to select only a subset of tests. In TLS for instance we
> have a matrix of variants for different crypto protocols and
> during development mostly care about testing a handful.
> This is quicker and makes reading output easier.
> 
> This patch adds argument parsing to kselftest_harness.
> 
> It supports a couple of ways to filter things, I could not come
> up with one way which will cover all cases.
> 
> The first and simplest switch is -r which takes the name of
> a test to run (can be specified multiple times). For example:
> 
>    $ ./my_test -r some.test.name -r some.other.name
> 
> will run tests some.test.name and some.other.name (where "some"
> is the fixture, "test" and "other" and "name is the test.)
> 
> Then there is a handful of group filtering options. f/v/t for
> filtering by fixture/variant/test. They have both positive
> (match -> run) and negative versions (match -> skip).
> If user specifies any positive option we assume the default
> is not to run the tests. If only negative options are set
> we assume the tests are supposed to be run by default.
> 
>    Usage: ./tools/testing/selftests/net/tls [-h|-l] [-t|-T|-v|-V|-f|-F|-r name]
> 	-h       print help
> 	-l       list all tests
> 
> 	-t name  include test
> 	-T name  exclude test
> 	-v name  include variant
> 	-V name  exclude variant
> 	-f name  include fixture
> 	-F name  exclude fixture
> 	-r name  run specified test
> 
>    Test filter options can be specified multiple times. The filtering stops
>    at the first match. For example to include all tests from variant 'bla'
>    but not test 'foo' specify '-T foo -v bla'.
> 
> Here we can request for example all tests from fixture "foo" to run:
> 
>   ./my_test -f foo
> 
> or to skip variants var1 and var2:
> 
>   ./my_test -V var1 -V var2
> 
> Signed-off-by: Jakub Kicinski <kuba@kernel.org>
> ---
> v2:
>   - use getopt()
> 

Thank you. I applied this to linux-kselftest next for Linux 6.3-rc1

thanks,
-- Shuah