[RFC v2 0/9] kunit: Add test attributes API

[RFC v2 0/9] kunit: Add test attributes API

[Rae Moar]

Hello everyone,

This is an RFC patch series to propose the addition of a test attributes
framework to KUnit.

There has been interest in filtering out "slow" KUnit tests. Most notably,
a new config, CONFIG_MEMCPY_SLOW_KUNIT_TEST, has been added to exclude a
particularly slow memcpy test
(https://lore.kernel.org/all/20230118200653.give.574-kees@kernel.org/).

This proposed attributes framework would be used to save and access test
associated data, including whether a test is slow. These attributes would
be reportable (via KTAP and command line output) and some will be
filterable.

This framework is designed to allow for the addition of other attributes in
the future. These attributes could include whether the test is flaky,
associated test files, etc.

This is the second version of the RFC I have added a few big changes:
- Change method for inputting filters to allow for spaces in filtering
  values
- Add option to skip filtered tests instead of not run or show them with
  the --filter_skip flag
- Separate the new feature to list tests and their attributes into both
  --list_tests (lists just tests) and --list_tests_attr (lists all)
- Add new attribute to store module name associated with test
- Add Tests to executor_test.c
- Add Documentation
- A few small changes to code commented on previously

I would love to hear about the new features. If the series seems overall
good I will send out the next version as an official patch series.

Thanks!
Rae

Rae Moar (9):
  kunit: Add test attributes API structure
  kunit: Add speed attribute
  kunit: Add module attribute
  kunit: Add ability to filter attributes
  kunit: tool: Add command line interface to filter and report
    attributes
  kunit: memcpy: Mark tests as slow using test attributes
  kunit: time: Mark test as slow using test attributes
  kunit: add tests for filtering attributes
  kunit: Add documentation of KUnit test attributes

 .../dev-tools/kunit/running_tips.rst          | 163 +++++++
 include/kunit/attributes.h                    |  50 +++
 include/kunit/test.h                          |  68 ++-
 kernel/time/time_test.c                       |   2 +-
 lib/Kconfig.debug                             |   3 +
 lib/kunit/Makefile                            |   3 +-
 lib/kunit/attributes.c                        | 406 ++++++++++++++++++
 lib/kunit/executor.c                          | 115 ++++-
 lib/kunit/executor_test.c                     | 119 ++++-
 lib/kunit/kunit-example-test.c                |   9 +
 lib/kunit/test.c                              |  27 +-
 lib/memcpy_kunit.c                            |   8 +-
 tools/testing/kunit/kunit.py                  |  80 +++-
 tools/testing/kunit/kunit_kernel.py           |   6 +-
 tools/testing/kunit/kunit_tool_test.py        |  39 +-
 15 files changed, 1022 insertions(+), 76 deletions(-)
 create mode 100644 include/kunit/attributes.h
 create mode 100644 lib/kunit/attributes.c


base-commit: 2e66833579ed759d7b7da1a8f07eb727ec6e80db
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 1/9] kunit: Add test attributes API structure

[Rae Moar]

Add the basic structure of the test attribute API to KUnit, which can be
used to save and access test associated data.

Add attributes.c and attributes.h to hold associated structs and functions
for the API.

Create a struct that holds a variety of associated helper functions for
each test attribute. These helper functions will be used to get the
attribute value, convert the value to a string, and filter based on the
value. This struct is flexible by design to allow for attributes of
numerous types and contexts.

Add a method to print test attributes in the format of "# [<test_name if
not suite>.]<attribute_name>: <attribute_value>".

Example for a suite: "# speed: slow"

Example for a test case: "# test_case.speed: very_slow"

Use this method to report attributes in the KTAP output (KTAP spec:
https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when
kernel's new kunit.action=list_attr option is used. Note this is derivative
of the kunit.action=list option.

In test.h, add fields and associated helper functions to test cases and
suites to hold user-inputted test attributes.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- Add list_attr option to only include attribute in the _list_tests output
  when this module param is set
- Add printing options for attributes to print always, print only for
  suites, or print never.

 include/kunit/attributes.h | 19 +++++++++
 include/kunit/test.h       | 33 ++++++++++++++++
 lib/kunit/Makefile         |  3 +-
 lib/kunit/attributes.c     | 80 ++++++++++++++++++++++++++++++++++++++
 lib/kunit/executor.c       | 21 ++++++++--
 lib/kunit/test.c           | 17 ++++----
 6 files changed, 161 insertions(+), 12 deletions(-)
 create mode 100644 include/kunit/attributes.h
 create mode 100644 lib/kunit/attributes.c

diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
new file mode 100644
index 000000000000..9fcd184cce36
--- /dev/null
+++ b/include/kunit/attributes.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * KUnit API to save and access test attributes
+ *
+ * Copyright (C) 2023, Google LLC.
+ * Author: Rae Moar <rmoar@google.com>
+ */
+
+#ifndef _KUNIT_ATTRIBUTES_H
+#define _KUNIT_ATTRIBUTES_H
+
+/*
+ * Print all test attributes for a test case or suite.
+ * Output format for test cases: "# <test_name>.<attribute>: <value>"
+ * Output format for test suites: "# <attribute>: <value>"
+ */
+void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
+
+#endif /* _KUNIT_ATTRIBUTES_H */
diff --git a/include/kunit/test.h b/include/kunit/test.h
index 23120d50499e..1fc9155988e9 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -63,12 +63,16 @@ enum kunit_status {
 	KUNIT_SKIPPED,
 };
 
+/* Holds attributes for each test case and suite */
+struct kunit_attributes {};
+
 /**
  * struct kunit_case - represents an individual test case.
  *
  * @run_case: the function representing the actual test case.
  * @name:     the name of the test case.
  * @generate_params: the generator function for parameterized tests.
+ * @attr:     the attributes associated with the test
  *
  * A test case is a function with the signature,
  * ``void (*)(struct kunit *)``
@@ -104,6 +108,7 @@ struct kunit_case {
 	void (*run_case)(struct kunit *test);
 	const char *name;
 	const void* (*generate_params)(const void *prev, char *desc);
+	struct kunit_attributes attr;
 
 	/* private: internal use only. */
 	enum kunit_status status;
@@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
  */
 #define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
 
+/**
+ * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
+ * with attributes
+ *
+ * @test_name: a reference to a test case function.
+ * @attributes: a reference to a struct kunit_attributes object containing
+ * test attributes
+ */
+#define KUNIT_CASE_ATTR(test_name, attributes)			\
+		{ .run_case = test_name, .name = #test_name,	\
+		  .attr = attributes }
+
 /**
  * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
  *
@@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
 		{ .run_case = test_name, .name = #test_name,	\
 		  .generate_params = gen_params }
 
+/**
+ * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
+ * kunit_case with attributes
+ *
+ * @test_name: a reference to a test case function.
+ * @gen_params: a reference to a parameter generator function.
+ * @attributes: a reference to a struct kunit_attributes object containing
+ * test attributes
+ */
+#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes)	\
+		{ .run_case = test_name, .name = #test_name,	\
+		  .generate_params = gen_params,				\
+		  .attr = attributes }
+
 /**
  * struct kunit_suite - describes a related collection of &struct kunit_case
  *
@@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
  * @init:	called before every test case.
  * @exit:	called after every test case.
  * @test_cases:	a null terminated array of test cases.
+ * @attr:	the attributes associated with the test suite
  *
  * A kunit_suite is a collection of related &struct kunit_case s, such that
  * @init is called before every test case and @exit is called after every
@@ -182,6 +214,7 @@ struct kunit_suite {
 	int (*init)(struct kunit *test);
 	void (*exit)(struct kunit *test);
 	struct kunit_case *test_cases;
+	struct kunit_attributes attr;
 
 	/* private: internal use only */
 	char status_comment[KUNIT_STATUS_COMMENT_SIZE];
diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
index cb417f504996..46f75f23dfe4 100644
--- a/lib/kunit/Makefile
+++ b/lib/kunit/Makefile
@@ -6,7 +6,8 @@ kunit-objs +=				test.o \
 					string-stream.o \
 					assert.o \
 					try-catch.o \
-					executor.o
+					executor.o \
+					attributes.o
 
 ifeq ($(CONFIG_KUNIT_DEBUGFS),y)
 kunit-objs +=				debugfs.o
diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
new file mode 100644
index 000000000000..9bda5a5f4030
--- /dev/null
+++ b/lib/kunit/attributes.c
@@ -0,0 +1,80 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * KUnit API to save and access test attributes
+ *
+ * Copyright (C) 2023, Google LLC.
+ * Author: Rae Moar <rmoar@google.com>
+ */
+
+#include <kunit/test.h>
+#include <kunit/attributes.h>
+
+/* Options for printing attributes:
+ * PRINT_ALWAYS - attribute is printed for every test case and suite if set
+ * PRINT_SUITE - attribute is printed for every suite if set but not for test cases
+ * PRINT_NEVER - attribute is never printed
+ */
+enum print_ops {
+	PRINT_ALWAYS,
+	PRINT_SUITE,
+	PRINT_NEVER,
+};
+
+/**
+ * struct kunit_attr - represents a test attribute and holds flexible
+ * helper functions to interact with attribute.
+ *
+ * @name: name of test attribute, eg. speed
+ * @get_attr: function to return attribute value given a test
+ * @to_string: function to return string representation of given
+ * attribute value
+ * @filter: function to indicate whether a given attribute value passes a
+ * filter
+ */
+struct kunit_attr {
+	const char *name;
+	void *(*get_attr)(void *test_or_suite, bool is_test);
+	const char *(*to_string)(void *attr, bool *to_free);
+	int (*filter)(void *attr, const char *input, int *err);
+	void *attr_default;
+	enum print_ops print;
+};
+
+/* List of all Test Attributes */
+
+static struct kunit_attr kunit_attr_list[] = {};
+
+/* Helper Functions to Access Attributes */
+
+void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
+{
+	int i;
+	bool to_free;
+	void *attr;
+	const char *attr_name, *attr_str;
+	struct kunit_suite *suite = is_test ? NULL : test_or_suite;
+	struct kunit_case *test = is_test ? test_or_suite : NULL;
+
+	for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) {
+		if (kunit_attr_list[i].print == PRINT_NEVER ||
+				(test && kunit_attr_list[i].print == PRINT_SUITE))
+			continue;
+		attr = kunit_attr_list[i].get_attr(test_or_suite, is_test);
+		if (attr) {
+			attr_name = kunit_attr_list[i].name;
+			attr_str = kunit_attr_list[i].to_string(attr, &to_free);
+			if (test) {
+				kunit_log(KERN_INFO, test, "%*s# %s.%s: %s",
+					KUNIT_INDENT_LEN * test_level, "", test->name,
+					attr_name, attr_str);
+			} else {
+				kunit_log(KERN_INFO, suite, "%*s# %s: %s",
+					KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str);
+			}
+
+			/* Free to_string of attribute if needed */
+			if (to_free)
+				kfree(attr_str);
+		}
+	}
+}
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 74982b83707c..12e38a48a5cc 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -2,6 +2,7 @@
 
 #include <linux/reboot.h>
 #include <kunit/test.h>
+#include <kunit/attributes.h>
 #include <linux/glob.h>
 #include <linux/moduleparam.h>
 
@@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0);
 MODULE_PARM_DESC(action,
 		 "Changes KUnit executor behavior, valid values are:\n"
 		 "<none>: run the tests like normal\n"
-		 "'list' to list test names instead of running them.\n");
+		 "'list' to list test names instead of running them.\n"
+		 "'list_attr' to list test names and attributes instead of running them.\n");
 
 /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
 struct kunit_test_filter {
@@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set)
 	__kunit_test_suites_init(suite_set->start, num_suites);
 }
 
-static void kunit_exec_list_tests(struct suite_set *suite_set)
+static void kunit_exec_list_tests(struct suite_set *suite_set, bool include_attr)
 {
 	struct kunit_suite * const *suites;
 	struct kunit_case *test_case;
@@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *suite_set)
 	/* Hack: print a ktap header so kunit.py can find the start of KUnit output. */
 	pr_info("KTAP version 1\n");
 
-	for (suites = suite_set->start; suites < suite_set->end; suites++)
+	for (suites = suite_set->start; suites < suite_set->end; suites++) {
+		/* Print suite name and suite attributes */
+		pr_info("%s\n", (*suites)->name);
+		if (include_attr)
+			kunit_print_attr((void *)(*suites), false, 0);
+
+		/* Print test case name and attributes in suite */
 		kunit_suite_for_each_test_case((*suites), test_case) {
 			pr_info("%s.%s\n", (*suites)->name, test_case->name);
+			if (include_attr)
+				kunit_print_attr((void *)test_case, true, 0);
 		}
+	}
 }
 
 int kunit_run_all_tests(void)
@@ -206,7 +217,9 @@ int kunit_run_all_tests(void)
 	if (!action_param)
 		kunit_exec_run_tests(&suite_set);
 	else if (strcmp(action_param, "list") == 0)
-		kunit_exec_list_tests(&suite_set);
+		kunit_exec_list_tests(&suite_set, false);
+	else if (strcmp(action_param, "list_attr") == 0)
+		kunit_exec_list_tests(&suite_set, true);
 	else
 		pr_err("kunit executor: unknown action '%s'\n", action_param);
 
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 84e4666555c9..9ee55139ecd1 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -9,6 +9,7 @@
 #include <kunit/resource.h>
 #include <kunit/test.h>
 #include <kunit/test-bug.h>
+#include <kunit/attributes.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/moduleparam.h>
@@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite)
 }
 EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases);
 
+/* Currently supported test levels */
+enum {
+	KUNIT_LEVEL_SUITE = 0,
+	KUNIT_LEVEL_CASE,
+	KUNIT_LEVEL_CASE_PARAM,
+};
+
 static void kunit_print_suite_start(struct kunit_suite *suite)
 {
 	/*
@@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite)
 	pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n");
 	pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
 		  suite->name);
+	kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE);
 	pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n",
 		  kunit_suite_num_test_cases(suite));
 }
 
-/* Currently supported test levels */
-enum {
-	KUNIT_LEVEL_SUITE = 0,
-	KUNIT_LEVEL_CASE,
-	KUNIT_LEVEL_CASE_PARAM,
-};
-
 static void kunit_print_ok_not_ok(struct kunit *test,
 				  unsigned int test_level,
 				  enum kunit_status status,
@@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite)
 			}
 		}
 
+		kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE);
 
 		kunit_print_test_stats(&test, param_stats);
 
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 2/9] kunit: Add speed attribute

[Rae Moar]

Add speed attribute to the test attribute API. This attribute will allow
users to mark tests with a category of speed.

Currently the categories of speed proposed are: normal, slow, and very_slow
(outlined in enum kunit_speed). These are outlined in the enum kunit_speed.

The assumed default speed for tests is "normal". This indicates that the
test takes a relatively trivial amount of time (less than 1 second),
regardless of the machine it is running on. Any test slower than this could
be marked as "slow" or "very_slow".

Add the macro KUNIT_CASE_SLOW to set a test as slow, as this is likely a
common use of the attributes API.

Add an example of marking a slow test to kunit-example-test.c.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- Remove the "fast" category of speed.

 include/kunit/test.h           | 30 +++++++++++++++++++++-
 lib/kunit/attributes.c         | 46 +++++++++++++++++++++++++++++++++-
 lib/kunit/kunit-example-test.c |  9 +++++++
 3 files changed, 83 insertions(+), 2 deletions(-)

diff --git a/include/kunit/test.h b/include/kunit/test.h
index 1fc9155988e9..c255c98a58f7 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -63,8 +63,25 @@ enum kunit_status {
 	KUNIT_SKIPPED,
 };
 
+/* Attribute struct/enum definitions */
+
+/*
+ * Speed Attribute is stored as an enum and separated into categories of
+ * speed: very_slowm, slow, normal, and fast. These speeds are relative
+ * to other KUnit tests.
+ */
+enum kunit_speed {
+	KUNIT_SPEED_UNSET,
+	KUNIT_SPEED_VERY_SLOW,
+	KUNIT_SPEED_SLOW,
+	KUNIT_SPEED_NORMAL,
+	KUNIT_SPEED_MAX = KUNIT_SPEED_NORMAL,
+};
+
 /* Holds attributes for each test case and suite */
-struct kunit_attributes {};
+struct kunit_attributes {
+	enum kunit_speed speed;
+};
 
 /**
  * struct kunit_case - represents an individual test case.
@@ -150,6 +167,17 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
 		{ .run_case = test_name, .name = #test_name,	\
 		  .attr = attributes }
 
+/**
+ * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case
+ * with the slow attribute
+ *
+ * @test_name: a reference to a test case function.
+ */
+
+#define KUNIT_CASE_SLOW(test_name)			\
+		{ .run_case = test_name, .name = #test_name,	\
+		  .attr.speed = KUNIT_SPEED_SLOW }
+
 /**
  * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
  *
diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
index 9bda5a5f4030..e97096dbb3b1 100644
--- a/lib/kunit/attributes.c
+++ b/lib/kunit/attributes.c
@@ -40,9 +40,53 @@ struct kunit_attr {
 	enum print_ops print;
 };
 
+/* String Lists for enum Attributes */
+
+static const char * const speed_str_list[] = {"unset", "very_slow", "slow", "normal"};
+
+/* To String Methods */
+
+static const char *attr_enum_to_string(void *attr, const char * const str_list[], bool *to_free)
+{
+	long val = (long)attr;
+
+	*to_free = false;
+	if (!val)
+		return NULL;
+	return str_list[val];
+}
+
+static const char *attr_speed_to_string(void *attr, bool *to_free)
+{
+	return attr_enum_to_string(attr, speed_str_list, to_free);
+}
+
+/* Get Attribute Methods */
+
+static void *attr_speed_get(void *test_or_suite, bool is_test)
+{
+	struct kunit_suite *suite = is_test ? NULL : test_or_suite;
+	struct kunit_case *test = is_test ? test_or_suite : NULL;
+
+	if (test)
+		return ((void *) test->attr.speed);
+	else
+		return ((void *) suite->attr.speed);
+}
+
+/* Attribute Struct Definitions */
+
+static const struct kunit_attr speed_attr = {
+	.name = "speed",
+	.get_attr = attr_speed_get,
+	.to_string = attr_speed_to_string,
+	.attr_default = (void *)KUNIT_SPEED_NORMAL,
+	.print = PRINT_ALWAYS,
+};
+
 /* List of all Test Attributes */
 
-static struct kunit_attr kunit_attr_list[] = {};
+static struct kunit_attr kunit_attr_list[] = {speed_attr};
 
 /* Helper Functions to Access Attributes */
 
diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c
index b69b689ea850..01a769f35e1d 100644
--- a/lib/kunit/kunit-example-test.c
+++ b/lib/kunit/kunit-example-test.c
@@ -220,6 +220,14 @@ static void example_params_test(struct kunit *test)
 	KUNIT_EXPECT_EQ(test, param->value % param->value, 0);
 }
 
+/*
+ * This test should always pass. Can be used to practice filtering attributes.
+ */
+static void example_slow_test(struct kunit *test)
+{
+	KUNIT_EXPECT_EQ(test, 1 + 1, 2);
+}
+
 /*
  * Here we make a list of all the test cases we want to add to the test suite
  * below.
@@ -237,6 +245,7 @@ static struct kunit_case example_test_cases[] = {
 	KUNIT_CASE(example_all_expect_macros_test),
 	KUNIT_CASE(example_static_stub_test),
 	KUNIT_CASE_PARAM(example_params_test, example_gen_params),
+	KUNIT_CASE_SLOW(example_slow_test),
 	{}
 };
 
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 3/9] kunit: Add module attribute

[Rae Moar]

Add module attribute to the test attribute API. This attribute stores the
module name associated with the test using KBUILD_MODNAME.

The name of a test suite and the module name often do not match. A
reference to the module name associated with the suite could be extremely
helpful in running tests as modules without needing to check the codebase.

This attribute will be printed for each suite.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes: since v1:
- This is a new patch.

 include/kunit/test.h   | 13 ++++++++-----
 lib/kunit/attributes.c | 28 +++++++++++++++++++++++++++-
 2 files changed, 35 insertions(+), 6 deletions(-)

diff --git a/include/kunit/test.h b/include/kunit/test.h
index c255c98a58f7..cdfc3f42e899 100644
--- a/include/kunit/test.h
+++ b/include/kunit/test.h
@@ -129,6 +129,7 @@ struct kunit_case {
 
 	/* private: internal use only. */
 	enum kunit_status status;
+	char *module_name;
 	char *log;
 };
 
@@ -153,7 +154,9 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
  * &struct kunit_case object from it. See the documentation for
  * &struct kunit_case for an example on how to use it.
  */
-#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
+#define KUNIT_CASE(test_name)			\
+		{ .run_case = test_name, .name = #test_name,	\
+		  .module_name = KBUILD_MODNAME}
 
 /**
  * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
@@ -165,7 +168,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
  */
 #define KUNIT_CASE_ATTR(test_name, attributes)			\
 		{ .run_case = test_name, .name = #test_name,	\
-		  .attr = attributes }
+		  .attr = attributes, .module_name = KBUILD_MODNAME}
 
 /**
  * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case
@@ -176,7 +179,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
 
 #define KUNIT_CASE_SLOW(test_name)			\
 		{ .run_case = test_name, .name = #test_name,	\
-		  .attr.speed = KUNIT_SPEED_SLOW }
+		  .attr.speed = KUNIT_SPEED_SLOW, .module_name = KBUILD_MODNAME}
 
 /**
  * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
@@ -197,7 +200,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
  */
 #define KUNIT_CASE_PARAM(test_name, gen_params)			\
 		{ .run_case = test_name, .name = #test_name,	\
-		  .generate_params = gen_params }
+		  .generate_params = gen_params, .module_name = KBUILD_MODNAME}
 
 /**
  * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
@@ -211,7 +214,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
 #define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes)	\
 		{ .run_case = test_name, .name = #test_name,	\
 		  .generate_params = gen_params,				\
-		  .attr = attributes }
+		  .attr = attributes, .module_name = KBUILD_MODNAME}
 
 /**
  * struct kunit_suite - describes a related collection of &struct kunit_case
diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
index e97096dbb3b1..43dcb5de8b8f 100644
--- a/lib/kunit/attributes.c
+++ b/lib/kunit/attributes.c
@@ -61,6 +61,12 @@ static const char *attr_speed_to_string(void *attr, bool *to_free)
 	return attr_enum_to_string(attr, speed_str_list, to_free);
 }
 
+static const char *attr_string_to_string(void *attr, bool *to_free)
+{
+	*to_free = false;
+	return (char *) attr;
+}
+
 /* Get Attribute Methods */
 
 static void *attr_speed_get(void *test_or_suite, bool is_test)
@@ -74,6 +80,18 @@ static void *attr_speed_get(void *test_or_suite, bool is_test)
 		return ((void *) suite->attr.speed);
 }
 
+static void *attr_module_get(void *test_or_suite, bool is_test)
+{
+	struct kunit_suite *suite = is_test ? NULL : test_or_suite;
+	struct kunit_case *test = is_test ? test_or_suite : NULL;
+
+	// Suites get their module attribute from their first test_case
+	if (test)
+		return ((void *) test->module_name);
+	else
+		return ((void *) suite->test_cases[0].module_name);
+}
+
 /* Attribute Struct Definitions */
 
 static const struct kunit_attr speed_attr = {
@@ -84,9 +102,17 @@ static const struct kunit_attr speed_attr = {
 	.print = PRINT_ALWAYS,
 };
 
+static const struct kunit_attr module_attr = {
+	.name = "module",
+	.get_attr = attr_module_get,
+	.to_string = attr_string_to_string,
+	.attr_default = (void *)"",
+	.print = PRINT_SUITE,
+};
+
 /* List of all Test Attributes */
 
-static struct kunit_attr kunit_attr_list[] = {speed_attr};
+static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};
 
 /* Helper Functions to Access Attributes */
 
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 4/9] kunit: Add ability to filter attributes

[Rae Moar]

Add filtering of test attributes. Users can filter tests using the
module_param called "filter".

Filters are imputed in the format: <attribute_name><operation><value>

Example: kunit.filter="speed>slow"

Operations include: >, <, >=, <=, !=, and =. These operations will act the
same for attributes of the same type but may not between types.

Note multiple filters can be inputted by separating them with a comma.
Example: kunit.filter="speed=slow, module!=example"

Since both suites and test cases can have attributes, there may be
conflicts. The process of filtering follows these rules:
- Filtering always operates at a per-test level.
- If a test has an attribute set, then the test's value is filtered on.
- Otherwise, the value falls back to the suite's value.
- If neither are set, the attribute has a global "default" value, which
  is used.

Filtered tests will not be run or show in output. The tests can instead be
skipped using the configurable option "kunit.filter_action=skip".

Note the default settings for running tests remains unfiltered.

Finally, add "filter" methods for the speed and module attributes to parse
and compare attribute values.

Note this filtering functionality will be added to kunit.py in the next
patch.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- Change method for inputting filters to allow for spaces in filtering
  values
- Add option to skip filtered tests instead of not run or show them with
  the --filter_skip flag

 include/kunit/attributes.h |  31 +++++
 lib/kunit/attributes.c     | 256 +++++++++++++++++++++++++++++++++++++
 lib/kunit/executor.c       |  94 +++++++++++---
 lib/kunit/executor_test.c  |  12 +-
 lib/kunit/test.c           |  10 +-
 5 files changed, 375 insertions(+), 28 deletions(-)

diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
index 9fcd184cce36..bc76a0b786d2 100644
--- a/include/kunit/attributes.h
+++ b/include/kunit/attributes.h
@@ -9,6 +9,20 @@
 #ifndef _KUNIT_ATTRIBUTES_H
 #define _KUNIT_ATTRIBUTES_H
 
+/*
+ * struct kunit_attr_filter - representation of attributes filter with the
+ * attribute object and string input
+ */
+struct kunit_attr_filter {
+	struct kunit_attr *attr;
+	char *input;
+};
+
+/*
+ * Returns the name of the filter's attribute.
+ */
+const char *kunit_attr_filter_name(struct kunit_attr_filter filter);
+
 /*
  * Print all test attributes for a test case or suite.
  * Output format for test cases: "# <test_name>.<attribute>: <value>"
@@ -16,4 +30,21 @@
  */
 void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
 
+/*
+ * Returns the number of fitlers in input.
+ */
+int kunit_get_filter_count(char *input);
+
+/*
+ * Parse attributes filter input and return an objects containing the
+ * attribute object and the string input of the next filter.
+ */
+struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err);
+
+/*
+ * Returns a copy of the suite containing only tests that pass the filter.
+ */
+struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
+		struct kunit_attr_filter filter, char *action, int *err);
+
 #endif /* _KUNIT_ATTRIBUTES_H */
diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
index 43dcb5de8b8f..91cbcacafba9 100644
--- a/lib/kunit/attributes.c
+++ b/lib/kunit/attributes.c
@@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bool *to_free)
 	return (char *) attr;
 }
 
+/* Filter Methods */
+
+static const char op_list[] = "<>!=";
+
+/*
+ * Returns whether the inputted integer value matches the filter given
+ * by the operation string and inputted integer.
+ */
+static int int_filter(long val, const char *op, int input, int *err)
+{
+	if (!strncmp(op, "<=", 2))
+		return (val <= input);
+	else if (!strncmp(op, ">=", 2))
+		return (val >= input);
+	else if (!strncmp(op, "!=", 2))
+		return (val != input);
+	else if (!strncmp(op, ">", 1))
+		return (val > input);
+	else if (!strncmp(op, "<", 1))
+		return (val < input);
+	else if (!strncmp(op, "=", 1))
+		return (val == input);
+	*err = -EINVAL;
+	pr_err("kunit executor: invalid filter operation: %s\n", op);
+	return false;
+}
+
+/*
+ * Returns whether the inputted enum value "attr" matches the filter given
+ * by the input string. Note: the str_list includes the corresponding string
+ * list to the enum values.
+ */
+static int attr_enum_filter(void *attr, const char *input, int *err,
+		const char * const str_list[], int max)
+{
+	int i, j, input_int;
+	long test_val = (long)attr;
+	const char *input_val;
+
+	for (i = 0; input[i]; i++) {
+		if (!strchr(op_list, input[i])) {
+			input_val = input + i;
+			break;
+		}
+	}
+
+	if (!input_val) {
+		*err = -EINVAL;
+		pr_err("kunit executor: filter value not found: %s\n", input);
+		return false;
+	}
+
+	for (j = 0; j <= max; j++) {
+		if (!strcmp(input_val, str_list[j]))
+			input_int = j;
+	}
+
+	if (!input_int) {
+		*err = -EINVAL;
+		pr_err("kunit executor: invalid filter input: %s\n", input);
+		return false;
+	}
+
+	return int_filter(test_val, input, input_int, err);
+}
+
+static int attr_speed_filter(void *attr, const char *input, int *err)
+{
+	return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX);
+}
+
+/*
+ * Returns whether the inputted string value (attr) matches the filter given
+ * by the input string.
+ */
+static int attr_string_filter(void *attr, const char *input, int *err)
+{
+	char *str = attr;
+
+	if (!strncmp(input, "<", 1)) {
+		*err = -EINVAL;
+		pr_err("kunit executor: invalid filter input: %s\n", input);
+		return false;
+	} else if (!strncmp(input, ">", 1)) {
+		*err = -EINVAL;
+		pr_err("kunit executor: invalid filter input: %s\n", input);
+		return false;
+	} else if (!strncmp(input, "!=", 2)) {
+		return (strcmp(input + 2, str) != 0);
+	} else if (!strncmp(input, "=", 1)) {
+		return (strcmp(input + 1, str) == 0);
+	}
+	*err = -EINVAL;
+	pr_err("kunit executor: invalid filter operation: %s\n", input);
+	return false;
+}
+
+
 /* Get Attribute Methods */
 
 static void *attr_speed_get(void *test_or_suite, bool is_test)
@@ -98,6 +196,7 @@ static const struct kunit_attr speed_attr = {
 	.name = "speed",
 	.get_attr = attr_speed_get,
 	.to_string = attr_speed_to_string,
+	.filter = attr_speed_filter,
 	.attr_default = (void *)KUNIT_SPEED_NORMAL,
 	.print = PRINT_ALWAYS,
 };
@@ -106,6 +205,7 @@ static const struct kunit_attr module_attr = {
 	.name = "module",
 	.get_attr = attr_module_get,
 	.to_string = attr_string_to_string,
+	.filter = attr_string_filter,
 	.attr_default = (void *)"",
 	.print = PRINT_SUITE,
 };
@@ -116,6 +216,11 @@ static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};
 
 /* Helper Functions to Access Attributes */
 
+const char *kunit_attr_filter_name(struct kunit_attr_filter filter)
+{
+	return filter.attr->name;
+}
+
 void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
 {
 	int i;
@@ -148,3 +253,154 @@ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level
 		}
 	}
 }
+
+/* Helper Functions to Filter Attributes */
+
+int kunit_get_filter_count(char *input)
+{
+	int i, comma_index, count = 0;
+
+	for (i = 0; input[i]; i++) {
+		if (input[i] == ',') {
+			if ((i - comma_index) > 1)
+				count++;
+			comma_index = i;
+		}
+	}
+	if ((i - comma_index) > 1)
+		count++;
+	return count;
+}
+
+struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err)
+{
+	struct kunit_attr_filter filter;
+	int i, j, comma_index, new_start_index;
+	int op_index = -1, attr_index = -1;
+	char op;
+	char *input = *filters;
+
+	/* Parse input until operation */
+	for (i = 0; input[i]; i++) {
+		if (op_index < 0 && strchr(op_list, input[i])) {
+			op_index = i;
+		} else if (!comma_index && input[i] == ',') {
+			comma_index = i;
+		} else if (comma_index && input[i] != ' ') {
+			new_start_index = i;
+			break;
+		}
+	}
+
+	if (op_index <= 0) {
+		*err = -EINVAL;
+		pr_err("kunit executor: filter operation not found: %s\n", input);
+		return filter;
+	}
+
+	/* Temporarily set operator to \0 character. */
+	op = input[op_index];
+	input[op_index] = '\0';
+
+	/* Find associated kunit_attr object */
+	for (j = 0; j < ARRAY_SIZE(kunit_attr_list); j++) {
+		if (!strcmp(input, kunit_attr_list[j].name)) {
+			attr_index = j;
+			break;
+		}
+	}
+
+	input[op_index] = op;
+
+	if (attr_index < 0) {
+		*err = -EINVAL;
+		pr_err("kunit executor: attribute not found: %s\n", input);
+	} else {
+		filter.attr = &kunit_attr_list[attr_index];
+	}
+
+	if (comma_index) {
+		input[comma_index] = '\0';
+		filter.input = input + op_index;
+		input = input + new_start_index;
+	} else {
+		filter.input = input + op_index;
+		input = NULL;
+	}
+
+	*filters = input;
+
+	return filter;
+}
+
+struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
+		struct kunit_attr_filter filter, char *action, int *err)
+{
+	int n = 0;
+	struct kunit_case *filtered, *test_case;
+	struct kunit_suite *copy;
+	void *suite_val, *test_val;
+	bool suite_result, test_result, default_result, result;
+
+	/* Allocate memory for new copy of suite and list of test cases */
+	copy = kmemdup(suite, sizeof(*copy), GFP_KERNEL);
+	if (!copy)
+		return ERR_PTR(-ENOMEM);
+
+	kunit_suite_for_each_test_case(suite, test_case) { n++; }
+
+	filtered = kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL);
+	if (!filtered) {
+		kfree(copy);
+		return ERR_PTR(-ENOMEM);
+	}
+
+	n = 0;
+
+	/* Save filtering result on default value */
+	default_result = filter.attr->filter(filter.attr->attr_default, filter.input, err);
+
+	/* Save suite attribute value and filtering result on that value */
+	suite_val = filter.attr->get_attr((void *)suite, false);
+	suite_result = filter.attr->filter(suite_val, filter.input, err);
+
+	/* For each test case, save test case if passes filtering. */
+	kunit_suite_for_each_test_case(suite, test_case) {
+		test_val = filter.attr->get_attr((void *) test_case, true);
+		test_result = filter.attr->filter(filter.attr->get_attr(test_case, true),
+				filter.input, err);
+
+		/*
+		 * If attribute value of test case is set, filter on that value.
+		 * If not, filter on suite value if set. If not, filter on
+		 * default value.
+		 */
+		result = false;
+		if (test_val) {
+			if (test_result)
+				result = true;
+		} else if (suite_val) {
+			if (suite_result)
+				result = true;
+		} else if (default_result) {
+			result = true;
+		}
+
+		if (result) {
+			filtered[n++] = *test_case;
+		} else if (action && strcmp(action, "skip") == 0) {
+			test_case->status = KUNIT_SKIPPED;
+			filtered[n++] = *test_case;
+		}
+	}
+
+	if (n == 0) {
+		kfree(copy);
+		kfree(filtered);
+		return NULL;
+	}
+
+	copy->test_cases = filtered;
+
+	return copy;
+}
diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
index 12e38a48a5cc..c286ae47435a 100644
--- a/lib/kunit/executor.c
+++ b/lib/kunit/executor.c
@@ -17,6 +17,9 @@ extern struct kunit_suite * const __kunit_suites_end[];
 
 static char *filter_glob_param;
 static char *action_param;
+static char *filter_param;
+static char *filter_action_param;
+
 
 module_param_named(filter_glob, filter_glob_param, charp, 0);
 MODULE_PARM_DESC(filter_glob,
@@ -27,15 +30,23 @@ MODULE_PARM_DESC(action,
 		 "<none>: run the tests like normal\n"
 		 "'list' to list test names instead of running them.\n"
 		 "'list_attr' to list test names and attributes instead of running them.\n");
+module_param_named(filter, filter_param, charp, 0);
+MODULE_PARM_DESC(filter,
+		"Filter which KUnit test suites/tests run at boot-time using attributes, e.g. speed>slow");
+module_param_named(filter_action, filter_action_param, charp, 0);
+MODULE_PARM_DESC(filter_action,
+		"Changes behavior of filtered tests using attributes, valid values are:\n"
+		"<none>: do not run filtered tests as normal\n"
+		"'skip': skip all filtered tests instead so tests will appear in output\n");
 
 /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
-struct kunit_test_filter {
+struct kunit_glob_filter {
 	char *suite_glob;
 	char *test_glob;
 };
 
 /* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty. */
-static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
+static void kunit_parse_glob_filter(struct kunit_glob_filter *parsed,
 				    const char *filter_glob)
 {
 	const int len = strlen(filter_glob);
@@ -57,7 +68,7 @@ static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
 
 /* Create a copy of suite with only tests that match test_glob. */
 static struct kunit_suite *
-kunit_filter_tests(const struct kunit_suite *const suite, const char *test_glob)
+kunit_filter_glob_tests(const struct kunit_suite *const suite, const char *test_glob)
 {
 	int n = 0;
 	struct kunit_case *filtered, *test_case;
@@ -111,12 +122,15 @@ static void kunit_free_suite_set(struct suite_set suite_set)
 
 static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
 					    const char *filter_glob,
+						char *filters,
+						char *filter_action,
 					    int *err)
 {
-	int i;
-	struct kunit_suite **copy, *filtered_suite;
+	int i, j, k, filter_count;
+	struct kunit_suite **copy, *filtered_suite, *new_filtered_suite;
 	struct suite_set filtered;
-	struct kunit_test_filter filter;
+	struct kunit_glob_filter parsed_glob;
+	struct kunit_attr_filter *parsed_filters;
 
 	const size_t max = suite_set->end - suite_set->start;
 
@@ -127,17 +141,52 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
 		return filtered;
 	}
 
-	kunit_parse_filter_glob(&filter, filter_glob);
-
-	for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
-		if (!glob_match(filter.suite_glob, suite_set->start[i]->name))
-			continue;
+	if (filter_glob)
+		kunit_parse_glob_filter(&parsed_glob, filter_glob);
 
-		filtered_suite = kunit_filter_tests(suite_set->start[i], filter.test_glob);
-		if (IS_ERR(filtered_suite)) {
-			*err = PTR_ERR(filtered_suite);
+	/* Parse attribute filters */
+	if (filters) {
+		filter_count = kunit_get_filter_count(filters);
+		parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);
+		for (j = 0; j < filter_count; j++)
+			parsed_filters[j] = kunit_next_attr_filter(&filters, err);
+		if (*err)
 			return filtered;
+	}
+
+	for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
+		filtered_suite = suite_set->start[i];
+		if (filter_glob) {
+			if (!glob_match(parsed_glob.suite_glob, filtered_suite->name))
+				continue;
+			filtered_suite = kunit_filter_glob_tests(filtered_suite,
+					parsed_glob.test_glob);
+			if (IS_ERR(filtered_suite)) {
+				*err = PTR_ERR(filtered_suite);
+				return filtered;
+			}
+		}
+		if (filter_count) {
+			for (k = 0; k < filter_count; k++) {
+				new_filtered_suite = kunit_filter_attr_tests(filtered_suite,
+						parsed_filters[k], filter_action, err);
+
+				/* Free previous copy of suite */
+				if (k > 0 || filter_glob)
+					kfree(filtered_suite);
+				filtered_suite = new_filtered_suite;
+
+				if (*err)
+					return filtered;
+				if (IS_ERR(filtered_suite)) {
+					*err = PTR_ERR(filtered_suite);
+					return filtered;
+				}
+				if (!filtered_suite)
+					break;
+			}
 		}
+
 		if (!filtered_suite)
 			continue;
 
@@ -145,8 +194,14 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
 	}
 	filtered.end = copy;
 
-	kfree(filter.suite_glob);
-	kfree(filter.test_glob);
+	if (filter_glob) {
+		kfree(parsed_glob.suite_glob);
+		kfree(parsed_glob.test_glob);
+	}
+
+	if (filter_count)
+		kfree(parsed_filters);
+
 	return filtered;
 }
 
@@ -206,8 +261,9 @@ int kunit_run_all_tests(void)
 		goto out;
 	}
 
-	if (filter_glob_param) {
-		suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err);
+	if (filter_glob_param || filter_param) {
+		suite_set = kunit_filter_suites(&suite_set, filter_glob_param,
+				filter_param, filter_action_param, &err);
 		if (err) {
 			pr_err("kunit executor: error filtering suites: %d\n", err);
 			goto out;
@@ -223,7 +279,7 @@ int kunit_run_all_tests(void)
 	else
 		pr_err("kunit executor: unknown action '%s'\n", action_param);
 
-	if (filter_glob_param) { /* a copy was made of each suite */
+	if (filter_glob_param || filter_param) { /* a copy was made of each suite */
 		kunit_free_suite_set(suite_set);
 	}
 
diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
index ce6749af374d..d7ab069324b5 100644
--- a/lib/kunit/executor_test.c
+++ b/lib/kunit/executor_test.c
@@ -24,15 +24,15 @@ static struct kunit_case dummy_test_cases[] = {
 
 static void parse_filter_test(struct kunit *test)
 {
-	struct kunit_test_filter filter = {NULL, NULL};
+	struct kunit_glob_filter filter = {NULL, NULL};
 
-	kunit_parse_filter_glob(&filter, "suite");
+	kunit_parse_glob_filter(&filter, "suite");
 	KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
 	KUNIT_EXPECT_FALSE(test, filter.test_glob);
 	kfree(filter.suite_glob);
 	kfree(filter.test_glob);
 
-	kunit_parse_filter_glob(&filter, "suite.test");
+	kunit_parse_glob_filter(&filter, "suite.test");
 	KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
 	KUNIT_EXPECT_STREQ(test, filter.test_glob, "test");
 	kfree(filter.suite_glob);
@@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test)
 	subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
 
 	/* Want: suite1, suite2, NULL -> suite2, NULL */
-	got = kunit_filter_suites(&suite_set, "suite2", &err);
+	got = kunit_filter_suites(&suite_set, "suite2", NULL, NULL, &err);
 	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
 	KUNIT_ASSERT_EQ(test, err, 0);
 	kfree_at_end(test, got.start);
@@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *test)
 	subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
 
 	/* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */
-	got = kunit_filter_suites(&suite_set, "suite2.test2", &err);
+	got = kunit_filter_suites(&suite_set, "suite2.test2", NULL, NULL, &err);
 	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
 	KUNIT_ASSERT_EQ(test, err, 0);
 	kfree_at_end(test, got.start);
@@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *test)
 	subsuite[0] = alloc_fake_suite(test, "suite1", dummy_test_cases);
 	subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
 
-	got = kunit_filter_suites(&suite_set, "not_found", &err);
+	got = kunit_filter_suites(&suite_set, "not_found", NULL, NULL, &err);
 	KUNIT_ASSERT_EQ(test, err, 0);
 	kfree_at_end(test, got.start); /* just in case */
 
diff --git a/lib/kunit/test.c b/lib/kunit/test.c
index 9ee55139ecd1..cb9797fa6303 100644
--- a/lib/kunit/test.c
+++ b/lib/kunit/test.c
@@ -613,18 +613,22 @@ int kunit_run_tests(struct kunit_suite *suite)
 	kunit_suite_for_each_test_case(suite, test_case) {
 		struct kunit test = { .param_value = NULL, .param_index = 0 };
 		struct kunit_result_stats param_stats = { 0 };
-		test_case->status = KUNIT_SKIPPED;
 
 		kunit_init_test(&test, test_case->name, test_case->log);
-
-		if (!test_case->generate_params) {
+		if (test_case->status == KUNIT_SKIPPED) {
+			/* Test marked as skip */
+			test.status = KUNIT_SKIPPED;
+			kunit_update_stats(&param_stats, test.status);
+		} else if (!test_case->generate_params) {
 			/* Non-parameterised test. */
+			test_case->status = KUNIT_SKIPPED;
 			kunit_run_case_catch_errors(suite, test_case, &test);
 			kunit_update_stats(&param_stats, test.status);
 		} else {
 			/* Get initial param. */
 			param_desc[0] = '\0';
 			test.param_value = test_case->generate_params(NULL, param_desc);
+			test_case->status = KUNIT_SKIPPED;
 			kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
 				  "KTAP version 1\n");
 			kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 5/9] kunit: tool: Add command line interface to filter and report attributes

[Rae Moar]

Add ability to kunit.py to filter attributes and report a list of tests
including attributes without running tests.

Add flag "--filter" to input filters on test attributes. Tests will be
filtered out if they do not match all inputted filters.

Example: --filter speed=slow (This filter would run only the tests that are
marked as slow)

Filters have operations: <, >, <=, >=, !=, and =. But note that the
characters < and > are often interpreted by the shell, so they may need to
be quoted or escaped.

Example: --filter "speed>slow" or --filter speed\>slow (This filter would
run only the tests that have the speed faster than slow.

Additionally, multiple filters can be used.

Example: --filter "speed=slow, module!=example" (This filter would run
only the tests that have the speed slow and are not in the "example"
module)

Note if the user wants to skip filtered tests instead of not
running/showing them use the "--filter_skip" flag instead.

Expose the output of kunit.action=list option with flag "--list_tests" to
output a list of tests. Additionally, add flag "--list_tests_attr" to
output a list of tests and their attributes. These flags are useful to see
tests and test attributes without needing to run tests.

Example of the output of "--list_tests_attr":
  example
  example.test_1
  example.test_2
  # example.test_2.speed: slow

This output includes a suite, example, with two test cases, test_1 and
test_2. And in this instance test_2 has been marked as slow.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- Change method for inputting filters to allow for spaces in filtering
  values
- Add option to skip filtered tests instead of not run or show them with
  the --filter_skip flag
- Separate the new feature to list tests and their attributes into both
  --list_tests (lists just tests) and --list_tests_attr (lists all)

 tools/testing/kunit/kunit.py           | 80 ++++++++++++++++++++++++--
 tools/testing/kunit/kunit_kernel.py    |  6 +-
 tools/testing/kunit/kunit_tool_test.py | 39 ++++++-------
 3 files changed, 96 insertions(+), 29 deletions(-)

diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
index 3905c43369c3..6104e622ce20 100755
--- a/tools/testing/kunit/kunit.py
+++ b/tools/testing/kunit/kunit.py
@@ -55,8 +55,12 @@ class KunitExecRequest(KunitParseRequest):
 	build_dir: str
 	timeout: int
 	filter_glob: str
+	filter: str
+	filter_skip: str
 	kernel_args: Optional[List[str]]
 	run_isolated: Optional[str]
+	list_tests: bool
+	list_tests_attr: bool
 
 @dataclass
 class KunitRequest(KunitExecRequest, KunitBuildRequest):
@@ -102,19 +106,39 @@ def config_and_build_tests(linux: kunit_kernel.LinuxSourceTree,
 
 def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> List[str]:
 	args = ['kunit.action=list']
+
+	if request.kernel_args:
+		args.extend(request.kernel_args)
+
+	output = linux.run_kernel(args=args,
+			   timeout=request.timeout,
+			   filter_glob=request.filter_glob,
+			   filter=request.filter,
+			   build_dir=request.build_dir)
+	lines = kunit_parser.extract_tap_lines(output)
+	# Hack! Drop the dummy TAP version header that the executor prints out.
+	lines.pop()
+
+	# Filter out any extraneous non-test output that might have gotten mixed in.
+	return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
+
+def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]:
+	args = ['kunit.action=list_attr']
+
 	if request.kernel_args:
 		args.extend(request.kernel_args)
 
 	output = linux.run_kernel(args=args,
 			   timeout=request.timeout,
 			   filter_glob=request.filter_glob,
+			   filter=request.filter,
 			   build_dir=request.build_dir)
 	lines = kunit_parser.extract_tap_lines(output)
 	# Hack! Drop the dummy TAP version header that the executor prints out.
 	lines.pop()
 
 	# Filter out any extraneous non-test output that might have gotten mixed in.
-	return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
+	return lines
 
 def _suites_from_test_list(tests: List[str]) -> List[str]:
 	"""Extracts all the suites from an ordered list of tests."""
@@ -128,10 +152,18 @@ def _suites_from_test_list(tests: List[str]) -> List[str]:
 			suites.append(suite)
 	return suites
 
-
-
 def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> KunitResult:
 	filter_globs = [request.filter_glob]
+	if request.list_tests:
+		output = _list_tests(linux, request)
+		for line in output:
+			print(line.rstrip())
+		return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0)
+	if request.list_tests_attr:
+		attr_output = _list_tests_attr(linux, request)
+		for line in attr_output:
+			print(line.rstrip())
+		return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0)
 	if request.run_isolated:
 		tests = _list_tests(linux, request)
 		if request.run_isolated == 'test':
@@ -145,6 +177,17 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
 
 	metadata = kunit_json.Metadata(arch=linux.arch(), build_dir=request.build_dir, def_config='kunit_defconfig')
 
+	filter = request.filter
+	if request.filter_skip:
+		args = ['kunit.filter_action=skip']
+		filter = request.filter_skip
+		if request.kernel_args:
+			args.extend(request.kernel_args)
+	elif request.kernel_args:
+		args = request.kernel_args
+	else:
+		args = None
+
 	test_counts = kunit_parser.TestCounts()
 	exec_time = 0.0
 	for i, filter_glob in enumerate(filter_globs):
@@ -152,9 +195,10 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
 
 		test_start = time.time()
 		run_result = linux.run_kernel(
-			args=request.kernel_args,
+			args=args,
 			timeout=request.timeout,
 			filter_glob=filter_glob,
+			filter=filter,
 			build_dir=request.build_dir)
 
 		_, test_result = parse_tests(request, metadata, run_result)
@@ -341,6 +385,18 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None:
 			    nargs='?',
 			    default='',
 			    metavar='filter_glob')
+	parser.add_argument('--filter',
+			    help='Filter KUnit tests with attributes, '
+				'filtered tests will not run, '
+			    'e.g. speed=fast or speed=>low',
+			    type=str,
+				default='')
+	parser.add_argument('--filter_skip',
+			    help='Filter KUnit tests run with attributes, '
+				'filtered tests will be skipped, '
+			    'e.g. speed=fast or speed=>low',
+			    type=str,
+				default='')
 	parser.add_argument('--kernel_args',
 			    help='Kernel command-line parameters. Maybe be repeated',
 			     action='append', metavar='')
@@ -350,6 +406,10 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None:
 			    'what ran before it.',
 			    type=str,
 			    choices=['suite', 'test'])
+	parser.add_argument('--list_tests', help='If set, list all tests',
+			    action='store_true')
+	parser.add_argument('--list_tests_attr', help='If set, list all tests and attributes.',
+			    action='store_true')
 
 def add_parse_opts(parser: argparse.ArgumentParser) -> None:
 	parser.add_argument('--raw_output', help='If set don\'t parse output from kernel. '
@@ -398,8 +458,12 @@ def run_handler(cli_args: argparse.Namespace) -> None:
 					json=cli_args.json,
 					timeout=cli_args.timeout,
 					filter_glob=cli_args.filter_glob,
+					filter=cli_args.filter,
+					filter_skip=cli_args.filter_skip,
 					kernel_args=cli_args.kernel_args,
-					run_isolated=cli_args.run_isolated)
+					run_isolated=cli_args.run_isolated,
+					list_tests=cli_args.list_tests,
+					list_tests_attr=cli_args.list_tests_attr)
 	result = run_tests(linux, request)
 	if result.status != KunitStatus.SUCCESS:
 		sys.exit(1)
@@ -441,8 +505,12 @@ def exec_handler(cli_args: argparse.Namespace) -> None:
 					json=cli_args.json,
 					timeout=cli_args.timeout,
 					filter_glob=cli_args.filter_glob,
+					filter=cli_args.filter,
+					filter_skip=cli_args.filter_skip,
 					kernel_args=cli_args.kernel_args,
-					run_isolated=cli_args.run_isolated)
+					run_isolated=cli_args.run_isolated,
+					list_tests=cli_args.list_tests,
+					list_tests_attr=cli_args.list_tests_attr)
 	result = exec_tests(linux, exec_request)
 	stdout.print_with_timestamp((
 		'Elapsed time: %.3fs\n') % (result.elapsed_time))
diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
index 7f648802caf6..281f062a4767 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -330,11 +330,13 @@ class LinuxSourceTree:
 			return False
 		return self.validate_config(build_dir)
 
-	def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', timeout: Optional[int]=None) -> Iterator[str]:
+	def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', filter: str='', timeout: Optional[int]=None) -> Iterator[str]:
 		if not args:
 			args = []
 		if filter_glob:
-			args.append('kunit.filter_glob='+filter_glob)
+			args.append('kunit.filter_glob=' + filter_glob)
+		if filter:
+			args.append('kunit.filter="' + filter + '"')
 		args.append('kunit.enable=1')
 
 		process = self._ops.start(args, build_dir)
diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
index be35999bb84f..85a1fb72735e 100755
--- a/tools/testing/kunit/kunit_tool_test.py
+++ b/tools/testing/kunit/kunit_tool_test.py
@@ -597,7 +597,7 @@ class KUnitMainTest(unittest.TestCase):
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0)
 		self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir='.kunit', filter_glob='', timeout=300)
+			args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_run_passes_args_pass(self):
@@ -605,7 +605,7 @@ class KUnitMainTest(unittest.TestCase):
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
 		self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir='.kunit', filter_glob='', timeout=300)
+			args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_exec_passes_args_fail(self):
@@ -629,7 +629,7 @@ class KUnitMainTest(unittest.TestCase):
 			kunit.main(['run'])
 		self.assertEqual(e.exception.code, 1)
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir='.kunit', filter_glob='', timeout=300)
+			args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
 		self.print_mock.assert_any_call(StrContains(' 0 tests run!'))
 
 	def test_exec_raw_output(self):
@@ -670,13 +670,13 @@ class KUnitMainTest(unittest.TestCase):
 		self.linux_source_mock.run_kernel = mock.Mock(return_value=[])
 		kunit.main(['run', '--raw_output', 'filter_glob'])
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir='.kunit', filter_glob='filter_glob', timeout=300)
+			args=None, build_dir='.kunit', filter_glob='filter_glob', filter='', timeout=300)
 
 	def test_exec_timeout(self):
 		timeout = 3453
 		kunit.main(['exec', '--timeout', str(timeout)])
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir='.kunit', filter_glob='', timeout=timeout)
+			args=None, build_dir='.kunit', filter_glob='', filter='', timeout=timeout)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_run_timeout(self):
@@ -684,7 +684,7 @@ class KUnitMainTest(unittest.TestCase):
 		kunit.main(['run', '--timeout', str(timeout)])
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir='.kunit', filter_glob='', timeout=timeout)
+			args=None, build_dir='.kunit', filter_glob='', filter='', timeout=timeout)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_run_builddir(self):
@@ -692,7 +692,7 @@ class KUnitMainTest(unittest.TestCase):
 		kunit.main(['run', '--build_dir=.kunit'])
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir=build_dir, filter_glob='', timeout=300)
+			args=None, build_dir=build_dir, filter_glob='', filter='', timeout=300)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_config_builddir(self):
@@ -710,7 +710,7 @@ class KUnitMainTest(unittest.TestCase):
 		build_dir = '.kunit'
 		kunit.main(['exec', '--build_dir', build_dir])
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=None, build_dir=build_dir, filter_glob='', timeout=300)
+			args=None, build_dir=build_dir, filter_glob='', filter='', timeout=300)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_run_kunitconfig(self):
@@ -786,7 +786,7 @@ class KUnitMainTest(unittest.TestCase):
 		kunit.main(['run', '--kernel_args=a=1', '--kernel_args=b=2'])
 		self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-		      args=['a=1','b=2'], build_dir='.kunit', filter_glob='', timeout=300)
+		      args=['a=1','b=2'], build_dir='.kunit', filter_glob='', filter='', timeout=300)
 		self.print_mock.assert_any_call(StrContains('Testing complete.'))
 
 	def test_list_tests(self):
@@ -794,13 +794,11 @@ class KUnitMainTest(unittest.TestCase):
 		self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want
 
 		got = kunit._list_tests(self.linux_source_mock,
-				     kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite'))
-
+				     kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', '', None, 'suite', False, False))
 		self.assertEqual(got, want)
 		# Should respect the user's filter glob when listing tests.
 		self.linux_source_mock.run_kernel.assert_called_once_with(
-			args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', timeout=300)
-
+			args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', timeout=300)
 
 	@mock.patch.object(kunit, '_list_tests')
 	def test_run_isolated_by_suite(self, mock_tests):
@@ -809,10 +807,10 @@ class KUnitMainTest(unittest.TestCase):
 
 		# Should respect the user's filter glob when listing tests.
 		mock_tests.assert_called_once_with(mock.ANY,
-				     kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite'))
+				     kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', '', '', None, 'suite', False, False))
 		self.linux_source_mock.run_kernel.assert_has_calls([
-			mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300),
-			mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300),
+			mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', filter='', timeout=300),
+			mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', filter='', timeout=300),
 		])
 
 	@mock.patch.object(kunit, '_list_tests')
@@ -822,13 +820,12 @@ class KUnitMainTest(unittest.TestCase):
 
 		# Should respect the user's filter glob when listing tests.
 		mock_tests.assert_called_once_with(mock.ANY,
-				     kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test'))
+				     kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', '', None, 'test', False, False))
 		self.linux_source_mock.run_kernel.assert_has_calls([
-			mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300),
-			mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300),
-			mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', timeout=300),
+			mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', filter='', timeout=300),
+			mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', filter='', timeout=300),
+			mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', filter='', timeout=300),
 		])
 
-
 if __name__ == '__main__':
 	unittest.main()
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 6/9] kunit: memcpy: Mark tests as slow using test attributes

[Rae Moar]

Mark slow memcpy KUnit tests using test attributes.

Tests marked as slow are as follows: memcpy_large_test, memmove_test,
memmove_large_test, and memmove_overlap_test. These tests were the slowest
of the memcpy tests and relatively slower to most other KUnit tests. Most
of these tests are already skipped when CONFIG_MEMCPY_SLOW_KUNIT_TEST is
not enabled.

These tests can now be filtered using the KUnit test attribute filtering
feature. Example: --filter "speed>slow". This will run only the tests that
have speeds faster than slow. The slow attribute will also be outputted in
KTAP.

Note: This patch is intended to replace the use of
CONFIG_MEMCPY_SLOW_KUNIT_TEST and to potentially deprecate this feature.
This patch does not remove the config option but does add a note to the
config definition commenting on this future shift.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- Added note under CONFIG_MEMCPY_SLOW_KUNIT_TEST.

 lib/Kconfig.debug  | 3 +++
 lib/memcpy_kunit.c | 8 ++++----
 2 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index ce51d4dc6803..f1e16bbbe491 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2607,6 +2607,9 @@ config MEMCPY_SLOW_KUNIT_TEST
 	  and bit ranges. These can be very slow, so they are split out
 	  as a separate config, in case they need to be disabled.
 
+	  Note this config option will be replaced by the use of KUnit test
+	  attributes.
+
 config IS_SIGNED_TYPE_KUNIT_TEST
 	tristate "Test is_signed_type() macro" if !KUNIT_ALL_TESTS
 	depends on KUNIT
diff --git a/lib/memcpy_kunit.c b/lib/memcpy_kunit.c
index 887926f04731..440aee705ccc 100644
--- a/lib/memcpy_kunit.c
+++ b/lib/memcpy_kunit.c
@@ -551,10 +551,10 @@ static void strtomem_test(struct kunit *test)
 static struct kunit_case memcpy_test_cases[] = {
 	KUNIT_CASE(memset_test),
 	KUNIT_CASE(memcpy_test),
-	KUNIT_CASE(memcpy_large_test),
-	KUNIT_CASE(memmove_test),
-	KUNIT_CASE(memmove_large_test),
-	KUNIT_CASE(memmove_overlap_test),
+	KUNIT_CASE_SLOW(memcpy_large_test),
+	KUNIT_CASE_SLOW(memmove_test),
+	KUNIT_CASE_SLOW(memmove_large_test),
+	KUNIT_CASE_SLOW(memmove_overlap_test),
 	KUNIT_CASE(strtomem_test),
 	{}
 };
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 7/9] kunit: time: Mark test as slow using test attributes

[Rae Moar]

Mark the time KUnit test, time64_to_tm_test_date_range, as slow using test
attributes.

This test ran relatively much slower than most other KUnit tests.

By marking this test as slow, the test can now be filtered using the KUnit
test attribute filtering feature. Example: --filter "speed>slow". This will
run only the tests that have speeds faster than slow. The slow attribute
will also be outputted in KTAP.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- No changes.

 kernel/time/time_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kernel/time/time_test.c b/kernel/time/time_test.c
index 831e8e779ace..ca058c8af6ba 100644
--- a/kernel/time/time_test.c
+++ b/kernel/time/time_test.c
@@ -86,7 +86,7 @@ static void time64_to_tm_test_date_range(struct kunit *test)
 }
 
 static struct kunit_case time_test_cases[] = {
-	KUNIT_CASE(time64_to_tm_test_date_range),
+	KUNIT_CASE_SLOW(time64_to_tm_test_date_range),
 	{}
 };
 
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 8/9] kunit: add tests for filtering attributes

[Rae Moar]

Add four tests to executor_test.c to test behavior of filtering attributes.

- parse_filter_attr_test - to test the parsing of inputted filters

- filter_attr_test - to test the filtering procedure on attributes

- filter_attr_empty_test - to test the behavior when all tests are filtered
  out

- filter_attr_skip_test - to test the configurable filter_skip option

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- This is a new patch

 lib/kunit/executor_test.c | 107 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 107 insertions(+)

diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
index d7ab069324b5..145a78ade33d 100644
--- a/lib/kunit/executor_test.c
+++ b/lib/kunit/executor_test.c
@@ -7,6 +7,7 @@
  */
 
 #include <kunit/test.h>
+#include <kunit/attributes.h>
 
 static void kfree_at_end(struct kunit *test, const void *to_free);
 static struct kunit_suite *alloc_fake_suite(struct kunit *test,
@@ -22,6 +23,14 @@ static struct kunit_case dummy_test_cases[] = {
 	{},
 };
 
+static struct kunit_case dummy_attr_test_cases[] = {
+	/* .run_case is not important, just needs to be non-NULL */
+	{ .name = "test1", .run_case = dummy_test, .module_name = "dummy",
+	  .attr.speed = KUNIT_SPEED_SLOW },
+	{ .name = "test2", .run_case = dummy_test, .module_name = "dummy" },
+	{},
+};
+
 static void parse_filter_test(struct kunit *test)
 {
 	struct kunit_glob_filter filter = {NULL, NULL};
@@ -108,11 +117,109 @@ static void filter_suites_to_empty_test(struct kunit *test)
 				"should be empty to indicate no match");
 }
 
+static void parse_filter_attr_test(struct kunit *test)
+{
+	int j, filter_count;
+	struct kunit_attr_filter *parsed_filters;
+	char *filters = "speed>slow, module!=example";
+	int err = 0;
+
+	filter_count = kunit_get_filter_count(filters);
+	KUNIT_EXPECT_EQ(test, filter_count, 2);
+
+	parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);
+	for (j = 0; j < filter_count; j++)
+		parsed_filters[j] = kunit_next_attr_filter(&filters, &err);
+
+	KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[0]), "speed");
+	KUNIT_EXPECT_STREQ(test, parsed_filters[0].input, ">slow");
+
+	KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[1]), "module");
+	KUNIT_EXPECT_STREQ(test, parsed_filters[1].input, "!=example");
+
+	kfree(parsed_filters);
+}
+
+static void filter_attr_test(struct kunit *test)
+{
+	struct kunit_suite *subsuite[3] = {NULL, NULL};
+	struct suite_set suite_set = {.start = subsuite, .end = &subsuite[2]};
+	struct suite_set got;
+	int err = 0;
+
+	subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
+	subsuite[1] = alloc_fake_suite(test, "suite2", dummy_attr_test_cases);
+	subsuite[1]->attr.speed = KUNIT_SPEED_SLOW; // Set suite attribute
+
+	/* Want: suite1(test1, test2), suite2(test1, test2), NULL -> suite1(test2), NULL */
+	got = kunit_filter_suites(&suite_set, NULL, "speed>slow", NULL, &err);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
+	KUNIT_ASSERT_EQ(test, err, 0);
+	kfree_at_end(test, got.start);
+
+	/* Validate we just have suite1 */
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]);
+	KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->name, "suite1");
+	KUNIT_ASSERT_EQ(test, got.end - got.start, 1);
+
+	/* Now validate we just have test2 */
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases);
+	KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[0].name, "test2");
+	KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].name);
+}
+
+static void filter_attr_empty_test(struct kunit *test)
+{
+	struct kunit_suite *subsuite[3] = {NULL, NULL};
+	struct suite_set suite_set = {.start = subsuite, .end = &subsuite[2]};
+	struct suite_set got;
+	int err = 0;
+
+	subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
+	subsuite[1] = alloc_fake_suite(test, "suite2", dummy_attr_test_cases);
+
+	got = kunit_filter_suites(&suite_set, NULL, "module!=dummy", NULL, &err);
+	KUNIT_ASSERT_EQ(test, err, 0);
+	kfree_at_end(test, got.start); /* just in case */
+
+	KUNIT_EXPECT_PTR_EQ_MSG(test, got.start, got.end,
+				"should be empty to indicate no match");
+}
+
+static void filter_attr_skip_test(struct kunit *test)
+{
+	struct kunit_suite *subsuite[2] = {NULL};
+	struct suite_set suite_set = {.start = subsuite, .end = &subsuite[1]};
+	struct suite_set got;
+	int err = 0;
+
+	subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
+
+	/* Want: suite1(test1, test2), NULL -> suite1(test1 with SKIP, test2), NULL */
+	got = kunit_filter_suites(&suite_set, NULL, "speed>slow", "skip", &err);
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
+	KUNIT_ASSERT_EQ(test, err, 0);
+	kfree_at_end(test, got.start);
+
+	/* Validate we have both test1 and test2 */
+	KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases);
+	KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[0].name, "test1");
+	KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[1].name, "test2");
+
+	/* Now ensure test1 is skipped and test2 is not */
+	KUNIT_EXPECT_EQ(test, got.start[0]->test_cases[0].status, KUNIT_SKIPPED);
+	KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].status);
+}
+
 static struct kunit_case executor_test_cases[] = {
 	KUNIT_CASE(parse_filter_test),
 	KUNIT_CASE(filter_suites_test),
 	KUNIT_CASE(filter_suites_test_glob_test),
 	KUNIT_CASE(filter_suites_to_empty_test),
+	KUNIT_CASE(parse_filter_attr_test),
+	KUNIT_CASE(filter_attr_test),
+	KUNIT_CASE(filter_attr_empty_test),
+	KUNIT_CASE(filter_attr_skip_test),
 	{}
 };
 
-- 
2.41.0.255.g8b1d071c50-goog

[RFC v2 9/9] kunit: Add documentation of KUnit test attributes

[Rae Moar]

Add documentation on the use of test attributes under the section "Tips for
Running KUnit Tests" in the KUnit docs.

Documentation includes three sections on how to mark tests with attributes,
how attributes are reported, and how the user can filter tests using test
attributes.

Signed-off-by: Rae Moar <rmoar@google.com>
---

Changes since v1:
- This is a new patch

 .../dev-tools/kunit/running_tips.rst          | 163 ++++++++++++++++++
 1 file changed, 163 insertions(+)

diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
index 8e8c493f17d1..c9bc5a6595d3 100644
--- a/Documentation/dev-tools/kunit/running_tips.rst
+++ b/Documentation/dev-tools/kunit/running_tips.rst
@@ -262,3 +262,166 @@ other code executed during boot, e.g.
 	# Reset coverage counters before running the test.
 	$ echo 0 > /sys/kernel/debug/gcov/reset
 	$ modprobe kunit-example-test
+
+
+Test Attributes and Filtering
+=============================
+
+Test suites and cases can be marked with test attributes, such as speed of
+test. These attributes will later be printed in test output and can be used to
+filter test execution.
+
+Marking Test Attributes
+-----------------------
+
+Tests are marked with an attribute by including a ``kunit_attributes`` object
+in the test definition.
+
+Test cases can be marked using the ``KUNIT_CASE_ATTR(test_name, attributes)``
+macro to define the test case instead of ``KUNIT_CASE(test_name)``.
+
+.. code-block:: c
+
+	static const struct kunit_attributes example_attr = {
+		.speed = KUNIT_VERY_SLOW,
+	};
+
+	static struct kunit_case example_test_cases[] = {
+		KUNIT_CASE_ATTR(example_test, example_attr),
+	};
+
+.. note::
+	To mark a test case as slow, you can also use ``KUNIT_CASE_SLOW(test_name)``.
+	This is a helpful macro as the slow attribute is the most commonly used.
+
+Test suites can be marked with an attribute by setting the "attr" field in the
+suite definition.
+
+.. code-block:: c
+
+	static const struct kunit_attributes example_attr = {
+		.speed = KUNIT_VERY_SLOW,
+	};
+
+	static struct kunit_suite example_test_suite = {
+		...,
+		.attr = example_attr,
+	};
+
+.. note::
+	Not all attributes need to be set in a ``kunit_attributes`` object. Unset
+	attributes will remain uninitialized and act as though the attribute is set
+	to 0 or NULL. Thus, if an attribute is set to 0, it is treated as unset.
+	These unset attributes will not be reported and may act as a default value
+	for filtering purposes.
+
+Reporting Attributes
+--------------------
+
+When a user runs tests, attributes will be present in kernel output (in KTAP
+format). This is an example of how test attributes for test cases will be formatted
+in Kernel output:
+
+.. code-block:: none
+
+	# example_test.speed: slow
+	ok 1 example_test
+
+This is an example of how test attributes for test suites will be formatted in
+Kernel output:
+
+.. code-block:: none
+
+	  KTAP version 2
+	  # Subtest: example_suite
+	  # module: kunit_example_test
+	  1..3
+	  ...
+	ok 1 example_suite
+
+Additionally, users can output a full attribute report of tests with their
+attributes, using the command line flag ``--list_tests_attr``:
+
+.. code-block:: bash
+
+	kunit.py run "example" --list_tests_attr
+
+.. note::
+	This report can be accessed when running KUnit manually by passing in the
+	module_param ``kunit.action=list_attr``.
+
+Filtering
+---------
+
+Users can filter tests using the ``--filter`` command line flag when running
+tests. As an example:
+
+.. code-block:: bash
+
+	kunit.py run --filter speed=slow
+
+
+You can also use the following operations on filters: "<", ">", "<=", ">=",
+"!=", and "=". Example:
+
+.. code-block:: bash
+
+	kunit.py run --filter "speed>slow"
+
+This example will run all tests with speeds faster than slow. Note that the
+characters < and > are often interpreted by the shell, so they may need to be
+quoted or escaped, as above.
+
+Additionally, you can use multiple filters at once. Simply separate filters
+using commas. Example:
+
+.. code-block:: bash
+
+	kunit.py run --filter "speed>slow, module=kunit_example_test"
+
+.. note::
+	You can use this filtering feature when running KUnit manually by passing
+	the filter as a module param: ``kunit.filter="speed>slow, speed<=normal"``.
+
+Filtered tests will not run or show up in the test output. You can use the
+``--filter_skip`` flag to skip filtered tests instead. These tests will be
+shown in the test output in the test but will not run. To use this feature when
+running KUnit manually, use the ``kunit.filter`` module param with
+``kunit.filter_action=skip``.
+
+Rules of Filtering Procedure
+----------------------------
+
+Since both suites and test cases can have attributes, there may be conflicts
+between attributes during filtering. The process of filtering follows these
+rules:
+
+- Filtering always operates at a per-test level.
+
+- If a test has an attribute set, then the test's value is filtered on.
+
+- Otherwise, the value falls back to the suite's value.
+
+- If neither are set, the attribute has a global "default" value, which is used.
+
+List of Current Attributes
+--------------------------
+
+``speed``
+
+This attribute indicates the speed of a test's execution (how slow or fast the
+test is).
+
+This attribute is saved as an enum with the following categories: "normal",
+"slow", or "very_slow". The assumed default speed for tests is "normal". This
+indicates that the test takes a relatively trivial amount of time (less than
+1 second), regardless of the machine it is running on. Any test slower than
+this could be marked as "slow" or "very_slow".
+
+``module``
+
+This attribute indicates the name of the module associated with the test.
+
+This attribute is automatically saved as a string and is printed for each suite.
+Tests can also be filtered using this attribute.
+
-- 
2.41.0.255.g8b1d071c50-goog

Re: [RFC v2 8/9] kunit: add tests for filtering attributes

[Daniel Latypov]

On Fri, Jul 7, 2023 at 2:10 PM Rae Moar <rmoar@google.com> wrote:
>
> Add four tests to executor_test.c to test behavior of filtering attributes.
>
> - parse_filter_attr_test - to test the parsing of inputted filters
>
> - filter_attr_test - to test the filtering procedure on attributes
>
> - filter_attr_empty_test - to test the behavior when all tests are filtered
>   out
>
> - filter_attr_skip_test - to test the configurable filter_skip option
>
> Signed-off-by: Rae Moar <rmoar@google.com>

I love that I'm able to read this patch first and get a feel for what
exactly the patch series is doing overall.

Some nits and suggestions below.

> ---
>
> Changes since v1:
> - This is a new patch
>
>  lib/kunit/executor_test.c | 107 ++++++++++++++++++++++++++++++++++++++
>  1 file changed, 107 insertions(+)
>
> diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
> index d7ab069324b5..145a78ade33d 100644
> --- a/lib/kunit/executor_test.c
> +++ b/lib/kunit/executor_test.c
> @@ -7,6 +7,7 @@
>   */
>
>  #include <kunit/test.h>
> +#include <kunit/attributes.h>
>
>  static void kfree_at_end(struct kunit *test, const void *to_free);
>  static struct kunit_suite *alloc_fake_suite(struct kunit *test,
> @@ -22,6 +23,14 @@ static struct kunit_case dummy_test_cases[] = {
>         {},
>  };
>
> +static struct kunit_case dummy_attr_test_cases[] = {
> +       /* .run_case is not important, just needs to be non-NULL */
> +       { .name = "test1", .run_case = dummy_test, .module_name = "dummy",
> +         .attr.speed = KUNIT_SPEED_SLOW },
> +       { .name = "test2", .run_case = dummy_test, .module_name = "dummy" },
> +       {},
> +};

1) can we move this array to be just above parse_filter_attr_test so
it's next to where it's used?

2) How about renaming "test1" to "slow" to make the assertions in the
test case a bit easier to follow?
Right now readers need to remember which test case was supposed to be
filtered out.


> +
>  static void parse_filter_test(struct kunit *test)
>  {
>         struct kunit_glob_filter filter = {NULL, NULL};
> @@ -108,11 +117,109 @@ static void filter_suites_to_empty_test(struct kunit *test)
>                                 "should be empty to indicate no match");
>  }
>
> +static void parse_filter_attr_test(struct kunit *test)
> +{
> +       int j, filter_count;
> +       struct kunit_attr_filter *parsed_filters;
> +       char *filters = "speed>slow, module!=example";
> +       int err = 0;
> +
> +       filter_count = kunit_get_filter_count(filters);
> +       KUNIT_EXPECT_EQ(test, filter_count, 2);
> +
> +       parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);

nit: kunit_kcalloc() instead?

> +       for (j = 0; j < filter_count; j++)
> +               parsed_filters[j] = kunit_next_attr_filter(&filters, &err);

then here we probably want to check err, i.e.
  KUNIT_ASSERT_EQ_MSG(test, err, 0, "failed to parse filter '%s'", filters[i]);


> +
> +       KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[0]), "speed");
> +       KUNIT_EXPECT_STREQ(test, parsed_filters[0].input, ">slow");
> +
> +       KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[1]), "module");
> +       KUNIT_EXPECT_STREQ(test, parsed_filters[1].input, "!=example");
> +
> +       kfree(parsed_filters);
> +}
> +
> +static void filter_attr_test(struct kunit *test)
> +{
> +       struct kunit_suite *subsuite[3] = {NULL, NULL};
> +       struct suite_set suite_set = {.start = subsuite, .end = &subsuite[2]};
> +       struct suite_set got;
> +       int err = 0;
> +
> +       subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
> +       subsuite[1] = alloc_fake_suite(test, "suite2", dummy_attr_test_cases);
> +       subsuite[1]->attr.speed = KUNIT_SPEED_SLOW; // Set suite attribute

Similarly, perhaps we can rename suite2 to "slow_suite"?
That would cause this line to go over 80 characters wide, but since
that's no longer a hard limit, I think this would be a decent place to
go past it.

> +
> +       /* Want: suite1(test1, test2), suite2(test1, test2), NULL -> suite1(test2), NULL */
> +       got = kunit_filter_suites(&suite_set, NULL, "speed>slow", NULL, &err);
> +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
> +       KUNIT_ASSERT_EQ(test, err, 0);
> +       kfree_at_end(test, got.start);
> +
> +       /* Validate we just have suite1 */
> +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]);
> +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->name, "suite1");
> +       KUNIT_ASSERT_EQ(test, got.end - got.start, 1);
> +
> +       /* Now validate we just have test2 */
> +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases);
> +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[0].name, "test2");
> +       KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].name);
> +}
> +
> +static void filter_attr_empty_test(struct kunit *test)
> +{
> +       struct kunit_suite *subsuite[3] = {NULL, NULL};
> +       struct suite_set suite_set = {.start = subsuite, .end = &subsuite[2]};
> +       struct suite_set got;
> +       int err = 0;
> +
> +       subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
> +       subsuite[1] = alloc_fake_suite(test, "suite2", dummy_attr_test_cases);
> +
> +       got = kunit_filter_suites(&suite_set, NULL, "module!=dummy", NULL, &err);
> +       KUNIT_ASSERT_EQ(test, err, 0);
> +       kfree_at_end(test, got.start); /* just in case */
> +
> +       KUNIT_EXPECT_PTR_EQ_MSG(test, got.start, got.end,
> +                               "should be empty to indicate no match");
> +}
> +
> +static void filter_attr_skip_test(struct kunit *test)
> +{
> +       struct kunit_suite *subsuite[2] = {NULL};
> +       struct suite_set suite_set = {.start = subsuite, .end = &subsuite[1]};
> +       struct suite_set got;
> +       int err = 0;
> +
> +       subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
> +
> +       /* Want: suite1(test1, test2), NULL -> suite1(test1 with SKIP, test2), NULL */
> +       got = kunit_filter_suites(&suite_set, NULL, "speed>slow", "skip", &err);
> +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
> +       KUNIT_ASSERT_EQ(test, err, 0);
> +       kfree_at_end(test, got.start);
> +
> +       /* Validate we have both test1 and test2 */
> +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases);

Should we assert that we have 2 test cases before we dereference the second one?
The other code in this file (that I wrote) is being a bit sloppy and
deref'ing test_cases[0] without checking. It's doing that since I was
relying on the fact that the filtering code drops suites with no test
cases, so we don't necessarily need to check len(test_cases) >= 1.
(In terms of best practices, we should be defensive and checking that, though).

But in this case, we have no such guarantee about the second element.

> +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[0].name, "test1");
> +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[1].name, "test2");

Trying to remember, I think the cast to `const char *` is no longer
necessary after one of David's changes...
I think we might just never have gotten around to cleaning that up due
to the ordering in which the patches went in...

> +
> +       /* Now ensure test1 is skipped and test2 is not */
> +       KUNIT_EXPECT_EQ(test, got.start[0]->test_cases[0].status, KUNIT_SKIPPED);
> +       KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].status);

Should we check that it's equal to KUNIT_SUCCESS instead?


> +}
> +
>  static struct kunit_case executor_test_cases[] = {
>         KUNIT_CASE(parse_filter_test),
>         KUNIT_CASE(filter_suites_test),
>         KUNIT_CASE(filter_suites_test_glob_test),
>         KUNIT_CASE(filter_suites_to_empty_test),
> +       KUNIT_CASE(parse_filter_attr_test),
> +       KUNIT_CASE(filter_attr_test),
> +       KUNIT_CASE(filter_attr_empty_test),
> +       KUNIT_CASE(filter_attr_skip_test),
>         {}
>  };
>
> --
> 2.41.0.255.g8b1d071c50-goog
>

Re: [RFC v2 8/9] kunit: add tests for filtering attributes

[Rae Moar]

On Mon, Jul 10, 2023 at 2:07 PM Daniel Latypov <dlatypov@google.com> wrote:
>
> On Fri, Jul 7, 2023 at 2:10 PM Rae Moar <rmoar@google.com> wrote:
> >
> > Add four tests to executor_test.c to test behavior of filtering attributes.
> >
> > - parse_filter_attr_test - to test the parsing of inputted filters
> >
> > - filter_attr_test - to test the filtering procedure on attributes
> >
> > - filter_attr_empty_test - to test the behavior when all tests are filtered
> >   out
> >
> > - filter_attr_skip_test - to test the configurable filter_skip option
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
>
> I love that I'm able to read this patch first and get a feel for what
> exactly the patch series is doing overall.


Thanks!

>
>
> Some nits and suggestions below.
>
> > ---
> >
> > Changes since v1:
> > - This is a new patch
> >
> >  lib/kunit/executor_test.c | 107 ++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 107 insertions(+)
> >
> > diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
> > index d7ab069324b5..145a78ade33d 100644
> > --- a/lib/kunit/executor_test.c
> > +++ b/lib/kunit/executor_test.c
> > @@ -7,6 +7,7 @@
> >   */
> >
> >  #include <kunit/test.h>
> > +#include <kunit/attributes.h>
> >
> >  static void kfree_at_end(struct kunit *test, const void *to_free);
> >  static struct kunit_suite *alloc_fake_suite(struct kunit *test,
> > @@ -22,6 +23,14 @@ static struct kunit_case dummy_test_cases[] = {
> >         {},
> >  };
> >
> > +static struct kunit_case dummy_attr_test_cases[] = {
> > +       /* .run_case is not important, just needs to be non-NULL */
> > +       { .name = "test1", .run_case = dummy_test, .module_name = "dummy",
> > +         .attr.speed = KUNIT_SPEED_SLOW },
> > +       { .name = "test2", .run_case = dummy_test, .module_name = "dummy" },
> > +       {},
> > +};
>
> 1) can we move this array to be just above parse_filter_attr_test so
> it's next to where it's used?

This seems like a great idea. I will move it down.

>
>
> 2) How about renaming "test1" to "slow" to make the assertions in the
> test case a bit easier to follow?
> Right now readers need to remember which test case was supposed to be
> filtered out.

Yes this sounds good. I think including "slow" would be helpful
although I might also consider the name "slow_test".

>
>
> > +
> >  static void parse_filter_test(struct kunit *test)
> >  {
> >         struct kunit_glob_filter filter = {NULL, NULL};
> > @@ -108,11 +117,109 @@ static void filter_suites_to_empty_test(struct kunit *test)
> >                                 "should be empty to indicate no match");
> >  }
> >
> > +static void parse_filter_attr_test(struct kunit *test)
> > +{
> > +       int j, filter_count;
> > +       struct kunit_attr_filter *parsed_filters;
> > +       char *filters = "speed>slow, module!=example";
> > +       int err = 0;
> > +
> > +       filter_count = kunit_get_filter_count(filters);
> > +       KUNIT_EXPECT_EQ(test, filter_count, 2);
> > +
> > +       parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);
>
> nit: kunit_kcalloc() instead?

Right, that makes sense. Thanks!

>
> > +       for (j = 0; j < filter_count; j++)
> > +               parsed_filters[j] = kunit_next_attr_filter(&filters, &err);
>
> then here we probably want to check err, i.e.
>   KUNIT_ASSERT_EQ_MSG(test, err, 0, "failed to parse filter '%s'", filters[i]);
>

Sounds good. I will add this.

>
> > +
> > +       KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[0]), "speed");
> > +       KUNIT_EXPECT_STREQ(test, parsed_filters[0].input, ">slow");
> > +
> > +       KUNIT_EXPECT_STREQ(test, kunit_attr_filter_name(parsed_filters[1]), "module");
> > +       KUNIT_EXPECT_STREQ(test, parsed_filters[1].input, "!=example");
> > +
> > +       kfree(parsed_filters);
> > +}
> > +
> > +static void filter_attr_test(struct kunit *test)
> > +{
> > +       struct kunit_suite *subsuite[3] = {NULL, NULL};
> > +       struct suite_set suite_set = {.start = subsuite, .end = &subsuite[2]};
> > +       struct suite_set got;
> > +       int err = 0;
> > +
> > +       subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
> > +       subsuite[1] = alloc_fake_suite(test, "suite2", dummy_attr_test_cases);
> > +       subsuite[1]->attr.speed = KUNIT_SPEED_SLOW; // Set suite attribute
>
> Similarly, perhaps we can rename suite2 to "slow_suite"?
> That would cause this line to go over 80 characters wide, but since
> that's no longer a hard limit, I think this would be a decent place to
> go past it.

Like above, I like this idea. I'll change the name. Interesting idea
about the 80 character limit also.

>
>
> > +
> > +       /* Want: suite1(test1, test2), suite2(test1, test2), NULL -> suite1(test2), NULL */
> > +       got = kunit_filter_suites(&suite_set, NULL, "speed>slow", NULL, &err);
> > +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
> > +       KUNIT_ASSERT_EQ(test, err, 0);
> > +       kfree_at_end(test, got.start);
> > +
> > +       /* Validate we just have suite1 */
> > +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]);
> > +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->name, "suite1");
> > +       KUNIT_ASSERT_EQ(test, got.end - got.start, 1);
> > +
> > +       /* Now validate we just have test2 */
> > +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases);
> > +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[0].name, "test2");
> > +       KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].name);
> > +}
> > +
> > +static void filter_attr_empty_test(struct kunit *test)
> > +{
> > +       struct kunit_suite *subsuite[3] = {NULL, NULL};
> > +       struct suite_set suite_set = {.start = subsuite, .end = &subsuite[2]};
> > +       struct suite_set got;
> > +       int err = 0;
> > +
> > +       subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
> > +       subsuite[1] = alloc_fake_suite(test, "suite2", dummy_attr_test_cases);
> > +
> > +       got = kunit_filter_suites(&suite_set, NULL, "module!=dummy", NULL, &err);
> > +       KUNIT_ASSERT_EQ(test, err, 0);
> > +       kfree_at_end(test, got.start); /* just in case */
> > +
> > +       KUNIT_EXPECT_PTR_EQ_MSG(test, got.start, got.end,
> > +                               "should be empty to indicate no match");
> > +}
> > +
> > +static void filter_attr_skip_test(struct kunit *test)
> > +{
> > +       struct kunit_suite *subsuite[2] = {NULL};
> > +       struct suite_set suite_set = {.start = subsuite, .end = &subsuite[1]};
> > +       struct suite_set got;
> > +       int err = 0;
> > +
> > +       subsuite[0] = alloc_fake_suite(test, "suite1", dummy_attr_test_cases);
> > +
> > +       /* Want: suite1(test1, test2), NULL -> suite1(test1 with SKIP, test2), NULL */
> > +       got = kunit_filter_suites(&suite_set, NULL, "speed>slow", "skip", &err);
> > +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
> > +       KUNIT_ASSERT_EQ(test, err, 0);
> > +       kfree_at_end(test, got.start);
> > +
> > +       /* Validate we have both test1 and test2 */
> > +       KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start[0]->test_cases);
>
> Should we assert that we have 2 test cases before we dereference the second one?
> The other code in this file (that I wrote) is being a bit sloppy and
> deref'ing test_cases[0] without checking. It's doing that since I was
> relying on the fact that the filtering code drops suites with no test
> cases, so we don't necessarily need to check len(test_cases) >= 1.
> (In terms of best practices, we should be defensive and checking that, though).
>
> But in this case, we have no such guarantee about the second element.

Good point. I'll add an assert statement here about the length of test_cases.

>
>
> > +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[0].name, "test1");
> > +       KUNIT_EXPECT_STREQ(test, (const char *)got.start[0]->test_cases[1].name, "test2");
>
> Trying to remember, I think the cast to `const char *` is no longer
> necessary after one of David's changes...
> I think we might just never have gotten around to cleaning that up due
> to the ordering in which the patches went in...

Ahh got it. That is my bad. I'll double check if these are necessary.

>
>
> > +
> > +       /* Now ensure test1 is skipped and test2 is not */
> > +       KUNIT_EXPECT_EQ(test, got.start[0]->test_cases[0].status, KUNIT_SKIPPED);
> > +       KUNIT_EXPECT_FALSE(test, got.start[0]->test_cases[1].status);
>
> Should we check that it's equal to KUNIT_SUCCESS instead?
>

I wouldn't expect the status to be set in this case. But the status is
returning as 0 so it would pass for both the assert statement above
and if it's equal to KUNIT_SUCCESS. But since it is not supposed to be
set to KUNIT_SUCCESS, I'm inclined to keep it this way.

Thanks for all the comments Daniel!
-Rae

>
>
> > +}
> > +
> >  static struct kunit_case executor_test_cases[] = {
> >         KUNIT_CASE(parse_filter_test),
> >         KUNIT_CASE(filter_suites_test),
> >         KUNIT_CASE(filter_suites_test_glob_test),
> >         KUNIT_CASE(filter_suites_to_empty_test),
> > +       KUNIT_CASE(parse_filter_attr_test),
> > +       KUNIT_CASE(filter_attr_test),
> > +       KUNIT_CASE(filter_attr_empty_test),
> > +       KUNIT_CASE(filter_attr_skip_test),
> >         {}
> >  };
> >
> > --
> > 2.41.0.255.g8b1d071c50-goog
> >

Re: [RFC v2 6/9] kunit: memcpy: Mark tests as slow using test attributes

[Kees Cook]

On July 7, 2023 2:09:44 PM PDT, Rae Moar <rmoar@google.com> wrote:
>Mark slow memcpy KUnit tests using test attributes.
>
>Tests marked as slow are as follows: memcpy_large_test, memmove_test,
>memmove_large_test, and memmove_overlap_test. These tests were the slowest
>of the memcpy tests and relatively slower to most other KUnit tests. Most
>of these tests are already skipped when CONFIG_MEMCPY_SLOW_KUNIT_TEST is
>not enabled.
>
>These tests can now be filtered using the KUnit test attribute filtering
>feature. Example: --filter "speed>slow". This will run only the tests that
>have speeds faster than slow. The slow attribute will also be outputted in
>KTAP.
>
>Note: This patch is intended to replace the use of
>CONFIG_MEMCPY_SLOW_KUNIT_TEST and to potentially deprecate this feature.
>This patch does not remove the config option but does add a note to the
>config definition commenting on this future shift.
>
>Signed-off-by: Rae Moar <rmoar@google.com>

I remain excited about this series. :)

Acked-by: Kees Cook <keescook@chromium.org>


-- 
Kees Cook

Re: [RFC v2 1/9] kunit: Add test attributes API structure

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 16415 bytes --]

On Sat, 8 Jul 2023 at 05:09, Rae Moar <rmoar@google.com> wrote:
>
> Add the basic structure of the test attribute API to KUnit, which can be
> used to save and access test associated data.
>
> Add attributes.c and attributes.h to hold associated structs and functions
> for the API.
>
> Create a struct that holds a variety of associated helper functions for
> each test attribute. These helper functions will be used to get the
> attribute value, convert the value to a string, and filter based on the
> value. This struct is flexible by design to allow for attributes of
> numerous types and contexts.
>
> Add a method to print test attributes in the format of "# [<test_name if
> not suite>.]<attribute_name>: <attribute_value>".
>
> Example for a suite: "# speed: slow"
>
> Example for a test case: "# test_case.speed: very_slow"

So, this is the one thing I'm a little unsure about here, and it's
really more of a problem with test names overall.

As noted in the KTAPv2 attributes and test name proposals, the names
and attributes are only really defined for "suites", hence the need to
have a different output format for test cases.

Personally, I'd prefer to keep the formats the same if we can (at
least for the actual KTAP output; I'm less concerned with the
list_attr option). That might make things a bit more difficult to
parse, though.

One possibility would be to combine the KTAP attributes and test name
specs and suggest that every test has a "test name" attribute, which
must be the first attribute output.

The output would then look something like:
KTAP version 2
# Name: my_suite
# Other-Attr: value
1..2
  KTAP version 2
  # Name: test_1
  # Other-Attr: value
  ok 1 test_1
  # Name: test_2
  # Other-Attr: value
  not ok 2 test_2
ok 1 my_suite

Would there be any problems with something like that?

I'm less concerned with the list_attr option, as that's not something
totally standardised in the way KTAP is.

>
> Use this method to report attributes in the KTAP output (KTAP spec:
> https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when
> kernel's new kunit.action=list_attr option is used. Note this is derivative
> of the kunit.action=list option.
>
> In test.h, add fields and associated helper functions to test cases and
> suites to hold user-inputted test attributes.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---

The only other thing I'd really like to support one day is having
attributes for individual parameters in parameterised tests. I think
it makes sense as a follow-up, though.

>
> Changes since v1:
> - Add list_attr option to only include attribute in the _list_tests output
>   when this module param is set
> - Add printing options for attributes to print always, print only for
>   suites, or print never.
>
>  include/kunit/attributes.h | 19 +++++++++
>  include/kunit/test.h       | 33 ++++++++++++++++
>  lib/kunit/Makefile         |  3 +-
>  lib/kunit/attributes.c     | 80 ++++++++++++++++++++++++++++++++++++++
>  lib/kunit/executor.c       | 21 ++++++++--
>  lib/kunit/test.c           | 17 ++++----
>  6 files changed, 161 insertions(+), 12 deletions(-)
>  create mode 100644 include/kunit/attributes.h
>  create mode 100644 lib/kunit/attributes.c
>
> diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
> new file mode 100644
> index 000000000000..9fcd184cce36
> --- /dev/null
> +++ b/include/kunit/attributes.h
> @@ -0,0 +1,19 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * KUnit API to save and access test attributes
> + *
> + * Copyright (C) 2023, Google LLC.
> + * Author: Rae Moar <rmoar@google.com>
> + */
> +
> +#ifndef _KUNIT_ATTRIBUTES_H
> +#define _KUNIT_ATTRIBUTES_H
> +
> +/*
> + * Print all test attributes for a test case or suite.
> + * Output format for test cases: "# <test_name>.<attribute>: <value>"
> + * Output format for test suites: "# <attribute>: <value>"
> + */
> +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
> +
> +#endif /* _KUNIT_ATTRIBUTES_H */
> diff --git a/include/kunit/test.h b/include/kunit/test.h
> index 23120d50499e..1fc9155988e9 100644
> --- a/include/kunit/test.h
> +++ b/include/kunit/test.h
> @@ -63,12 +63,16 @@ enum kunit_status {
>         KUNIT_SKIPPED,
>  };
>
> +/* Holds attributes for each test case and suite */
> +struct kunit_attributes {};

Do we want a separate set of attributes for test cases and suites?
(I think probably not, but it's worth making sure.)

> +
>  /**
>   * struct kunit_case - represents an individual test case.
>   *
>   * @run_case: the function representing the actual test case.
>   * @name:     the name of the test case.
>   * @generate_params: the generator function for parameterized tests.
> + * @attr:     the attributes associated with the test
>   *
>   * A test case is a function with the signature,
>   * ``void (*)(struct kunit *)``
> @@ -104,6 +108,7 @@ struct kunit_case {
>         void (*run_case)(struct kunit *test);
>         const char *name;
>         const void* (*generate_params)(const void *prev, char *desc);
> +       struct kunit_attributes attr;
>
>         /* private: internal use only. */
>         enum kunit_status status;
> @@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>   */
>  #define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
>
> +/**
> + * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
> + * with attributes
> + *
> + * @test_name: a reference to a test case function.
> + * @attributes: a reference to a struct kunit_attributes object containing
> + * test attributes
> + */
> +#define KUNIT_CASE_ATTR(test_name, attributes)                 \
> +               { .run_case = test_name, .name = #test_name,    \
> +                 .attr = attributes }
> +
>  /**
>   * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
>   *
> @@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>                 { .run_case = test_name, .name = #test_name,    \
>                   .generate_params = gen_params }
>
> +/**
> + * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
> + * kunit_case with attributes
> + *
> + * @test_name: a reference to a test case function.
> + * @gen_params: a reference to a parameter generator function.
> + * @attributes: a reference to a struct kunit_attributes object containing
> + * test attributes
> + */
> +#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes)       \
> +               { .run_case = test_name, .name = #test_name,    \
> +                 .generate_params = gen_params,                                \
> +                 .attr = attributes }
> +

I do worry a bit that we'll end up with a huge list of variants of the
KUNIT_CASE_* macros if we start adding more things here. I can't think
of a better way to handle it at the moment, though.



>  /**
>   * struct kunit_suite - describes a related collection of &struct kunit_case
>   *
> @@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>   * @init:      called before every test case.
>   * @exit:      called after every test case.
>   * @test_cases:        a null terminated array of test cases.
> + * @attr:      the attributes associated with the test suite
>   *
>   * A kunit_suite is a collection of related &struct kunit_case s, such that
>   * @init is called before every test case and @exit is called after every
> @@ -182,6 +214,7 @@ struct kunit_suite {
>         int (*init)(struct kunit *test);
>         void (*exit)(struct kunit *test);
>         struct kunit_case *test_cases;
> +       struct kunit_attributes attr;
>
>         /* private: internal use only */
>         char status_comment[KUNIT_STATUS_COMMENT_SIZE];
> diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
> index cb417f504996..46f75f23dfe4 100644
> --- a/lib/kunit/Makefile
> +++ b/lib/kunit/Makefile
> @@ -6,7 +6,8 @@ kunit-objs +=                           test.o \
>                                         string-stream.o \
>                                         assert.o \
>                                         try-catch.o \
> -                                       executor.o
> +                                       executor.o \
> +                                       attributes.o
>
>  ifeq ($(CONFIG_KUNIT_DEBUGFS),y)
>  kunit-objs +=                          debugfs.o
> diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> new file mode 100644
> index 000000000000..9bda5a5f4030
> --- /dev/null
> +++ b/lib/kunit/attributes.c
> @@ -0,0 +1,80 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * KUnit API to save and access test attributes
> + *
> + * Copyright (C) 2023, Google LLC.
> + * Author: Rae Moar <rmoar@google.com>
> + */
> +
> +#include <kunit/test.h>
> +#include <kunit/attributes.h>
> +
> +/* Options for printing attributes:
> + * PRINT_ALWAYS - attribute is printed for every test case and suite if set
> + * PRINT_SUITE - attribute is printed for every suite if set but not for test cases
> + * PRINT_NEVER - attribute is never printed
> + */
> +enum print_ops {
> +       PRINT_ALWAYS,
> +       PRINT_SUITE,
> +       PRINT_NEVER,
> +};
> +
> +/**
> + * struct kunit_attr - represents a test attribute and holds flexible
> + * helper functions to interact with attribute.
> + *
> + * @name: name of test attribute, eg. speed
> + * @get_attr: function to return attribute value given a test
> + * @to_string: function to return string representation of given
> + * attribute value
> + * @filter: function to indicate whether a given attribute value passes a
> + * filter
> + */
> +struct kunit_attr {
> +       const char *name;
> +       void *(*get_attr)(void *test_or_suite, bool is_test);
> +       const char *(*to_string)(void *attr, bool *to_free);
> +       int (*filter)(void *attr, const char *input, int *err);
> +       void *attr_default;
> +       enum print_ops print;
> +};
> +
> +/* List of all Test Attributes */
> +
> +static struct kunit_attr kunit_attr_list[] = {};
> +
> +/* Helper Functions to Access Attributes */
> +
> +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
> +{
> +       int i;
> +       bool to_free;
> +       void *attr;
> +       const char *attr_name, *attr_str;
> +       struct kunit_suite *suite = is_test ? NULL : test_or_suite;
> +       struct kunit_case *test = is_test ? test_or_suite : NULL;
> +
> +       for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) {
> +               if (kunit_attr_list[i].print == PRINT_NEVER ||
> +                               (test && kunit_attr_list[i].print == PRINT_SUITE))
> +                       continue;
> +               attr = kunit_attr_list[i].get_attr(test_or_suite, is_test);
> +               if (attr) {
> +                       attr_name = kunit_attr_list[i].name;
> +                       attr_str = kunit_attr_list[i].to_string(attr, &to_free);
> +                       if (test) {
> +                               kunit_log(KERN_INFO, test, "%*s# %s.%s: %s",
> +                                       KUNIT_INDENT_LEN * test_level, "", test->name,
> +                                       attr_name, attr_str);
> +                       } else {
> +                               kunit_log(KERN_INFO, suite, "%*s# %s: %s",
> +                                       KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str);
> +                       }
> +
> +                       /* Free to_string of attribute if needed */
> +                       if (to_free)
> +                               kfree(attr_str);
> +               }
> +       }
> +}
> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> index 74982b83707c..12e38a48a5cc 100644
> --- a/lib/kunit/executor.c
> +++ b/lib/kunit/executor.c
> @@ -2,6 +2,7 @@
>
>  #include <linux/reboot.h>
>  #include <kunit/test.h>
> +#include <kunit/attributes.h>
>  #include <linux/glob.h>
>  #include <linux/moduleparam.h>
>
> @@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0);
>  MODULE_PARM_DESC(action,
>                  "Changes KUnit executor behavior, valid values are:\n"
>                  "<none>: run the tests like normal\n"
> -                "'list' to list test names instead of running them.\n");
> +                "'list' to list test names instead of running them.\n"
> +                "'list_attr' to list test names and attributes instead of running them.\n");
>
>  /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
>  struct kunit_test_filter {
> @@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set)
>         __kunit_test_suites_init(suite_set->start, num_suites);
>  }
>
> -static void kunit_exec_list_tests(struct suite_set *suite_set)
> +static void kunit_exec_list_tests(struct suite_set *suite_set, bool include_attr)
>  {
>         struct kunit_suite * const *suites;
>         struct kunit_case *test_case;
> @@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *suite_set)
>         /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */
>         pr_info("KTAP version 1\n");
>
> -       for (suites = suite_set->start; suites < suite_set->end; suites++)
> +       for (suites = suite_set->start; suites < suite_set->end; suites++) {
> +               /* Print suite name and suite attributes */
> +               pr_info("%s\n", (*suites)->name);
> +               if (include_attr)
> +                       kunit_print_attr((void *)(*suites), false, 0);
> +
> +               /* Print test case name and attributes in suite */
>                 kunit_suite_for_each_test_case((*suites), test_case) {
>                         pr_info("%s.%s\n", (*suites)->name, test_case->name);
> +                       if (include_attr)
> +                               kunit_print_attr((void *)test_case, true, 0);
>                 }
> +       }
>  }
>
>  int kunit_run_all_tests(void)
> @@ -206,7 +217,9 @@ int kunit_run_all_tests(void)
>         if (!action_param)
>                 kunit_exec_run_tests(&suite_set);
>         else if (strcmp(action_param, "list") == 0)
> -               kunit_exec_list_tests(&suite_set);
> +               kunit_exec_list_tests(&suite_set, false);
> +       else if (strcmp(action_param, "list_attr") == 0)
> +               kunit_exec_list_tests(&suite_set, true);
>         else
>                 pr_err("kunit executor: unknown action '%s'\n", action_param);
>
> diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> index 84e4666555c9..9ee55139ecd1 100644
> --- a/lib/kunit/test.c
> +++ b/lib/kunit/test.c
> @@ -9,6 +9,7 @@
>  #include <kunit/resource.h>
>  #include <kunit/test.h>
>  #include <kunit/test-bug.h>
> +#include <kunit/attributes.h>
>  #include <linux/kernel.h>
>  #include <linux/module.h>
>  #include <linux/moduleparam.h>
> @@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite)
>  }
>  EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases);
>
> +/* Currently supported test levels */
> +enum {
> +       KUNIT_LEVEL_SUITE = 0,
> +       KUNIT_LEVEL_CASE,
> +       KUNIT_LEVEL_CASE_PARAM,
> +};
> +
>  static void kunit_print_suite_start(struct kunit_suite *suite)
>  {
>         /*
> @@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite)
>         pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n");
>         pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
>                   suite->name);
> +       kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE);
>         pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n",
>                   kunit_suite_num_test_cases(suite));
>  }
>
> -/* Currently supported test levels */
> -enum {
> -       KUNIT_LEVEL_SUITE = 0,
> -       KUNIT_LEVEL_CASE,
> -       KUNIT_LEVEL_CASE_PARAM,
> -};
> -
>  static void kunit_print_ok_not_ok(struct kunit *test,
>                                   unsigned int test_level,
>                                   enum kunit_status status,
> @@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite)
>                         }
>                 }
>
> +               kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE);
>
>                 kunit_print_test_stats(&test, param_stats);
>
> --
> 2.41.0.255.g8b1d071c50-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 2/9] kunit: Add speed attribute

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 5837 bytes --]

On Sat, 8 Jul 2023 at 05:09, Rae Moar <rmoar@google.com> wrote:
>
> Add speed attribute to the test attribute API. This attribute will allow
> users to mark tests with a category of speed.
>
> Currently the categories of speed proposed are: normal, slow, and very_slow
> (outlined in enum kunit_speed). These are outlined in the enum kunit_speed.
>
> The assumed default speed for tests is "normal". This indicates that the
> test takes a relatively trivial amount of time (less than 1 second),
> regardless of the machine it is running on. Any test slower than this could
> be marked as "slow" or "very_slow".
>
> Add the macro KUNIT_CASE_SLOW to set a test as slow, as this is likely a
> common use of the attributes API.
>
> Add an example of marking a slow test to kunit-example-test.c.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---

Looks good.

Reviewed-by: David Gow <davidgow@google.com>

>
> Changes since v1:
> - Remove the "fast" category of speed.
>
>  include/kunit/test.h           | 30 +++++++++++++++++++++-
>  lib/kunit/attributes.c         | 46 +++++++++++++++++++++++++++++++++-
>  lib/kunit/kunit-example-test.c |  9 +++++++
>  3 files changed, 83 insertions(+), 2 deletions(-)
>
> diff --git a/include/kunit/test.h b/include/kunit/test.h
> index 1fc9155988e9..c255c98a58f7 100644
> --- a/include/kunit/test.h
> +++ b/include/kunit/test.h
> @@ -63,8 +63,25 @@ enum kunit_status {
>         KUNIT_SKIPPED,
>  };
>
> +/* Attribute struct/enum definitions */
> +
> +/*
> + * Speed Attribute is stored as an enum and separated into categories of
> + * speed: very_slowm, slow, normal, and fast. These speeds are relative

Nit: we only have very_slow, slow & normal now.

> + * to other KUnit tests.
> + */
> +enum kunit_speed {
> +       KUNIT_SPEED_UNSET,
> +       KUNIT_SPEED_VERY_SLOW,
> +       KUNIT_SPEED_SLOW,
> +       KUNIT_SPEED_NORMAL,
> +       KUNIT_SPEED_MAX = KUNIT_SPEED_NORMAL,
> +};

A part of me feels that we could get away with reversing the order of
this and having KUNIT_SPEED_NORMAL == 0. (Possibly reversing the
comparisons in the filtering, too.)

That's probably not worth the added complexity though. Either way,
maybe add a note that "UNSET" == "NORMAL".


> +
>  /* Holds attributes for each test case and suite */
> -struct kunit_attributes {};
> +struct kunit_attributes {
> +       enum kunit_speed speed;
> +};
>
>  /**
>   * struct kunit_case - represents an individual test case.
> @@ -150,6 +167,17 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>                 { .run_case = test_name, .name = #test_name,    \
>                   .attr = attributes }
>
> +/**
> + * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case
> + * with the slow attribute
> + *
> + * @test_name: a reference to a test case function.
> + */
> +
> +#define KUNIT_CASE_SLOW(test_name)                     \
> +               { .run_case = test_name, .name = #test_name,    \
> +                 .attr.speed = KUNIT_SPEED_SLOW }
> +
>  /**
>   * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
>   *
> diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> index 9bda5a5f4030..e97096dbb3b1 100644
> --- a/lib/kunit/attributes.c
> +++ b/lib/kunit/attributes.c
> @@ -40,9 +40,53 @@ struct kunit_attr {
>         enum print_ops print;
>  };
>
> +/* String Lists for enum Attributes */
> +
> +static const char * const speed_str_list[] = {"unset", "very_slow", "slow", "normal"};
> +
> +/* To String Methods */
> +
> +static const char *attr_enum_to_string(void *attr, const char * const str_list[], bool *to_free)
> +{
> +       long val = (long)attr;
> +
> +       *to_free = false;
> +       if (!val)
> +               return NULL;
> +       return str_list[val];
> +}
> +
> +static const char *attr_speed_to_string(void *attr, bool *to_free)
> +{
> +       return attr_enum_to_string(attr, speed_str_list, to_free);
> +}
> +
> +/* Get Attribute Methods */
> +
> +static void *attr_speed_get(void *test_or_suite, bool is_test)
> +{
> +       struct kunit_suite *suite = is_test ? NULL : test_or_suite;
> +       struct kunit_case *test = is_test ? test_or_suite : NULL;
> +
> +       if (test)
> +               return ((void *) test->attr.speed);
> +       else
> +               return ((void *) suite->attr.speed);
> +}
> +
> +/* Attribute Struct Definitions */
> +
> +static const struct kunit_attr speed_attr = {
> +       .name = "speed",
> +       .get_attr = attr_speed_get,
> +       .to_string = attr_speed_to_string,
> +       .attr_default = (void *)KUNIT_SPEED_NORMAL,
> +       .print = PRINT_ALWAYS,
> +};
> +
>  /* List of all Test Attributes */
>
> -static struct kunit_attr kunit_attr_list[] = {};
> +static struct kunit_attr kunit_attr_list[] = {speed_attr};
>
>  /* Helper Functions to Access Attributes */
>
> diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c
> index b69b689ea850..01a769f35e1d 100644
> --- a/lib/kunit/kunit-example-test.c
> +++ b/lib/kunit/kunit-example-test.c
> @@ -220,6 +220,14 @@ static void example_params_test(struct kunit *test)
>         KUNIT_EXPECT_EQ(test, param->value % param->value, 0);
>  }
>
> +/*
> + * This test should always pass. Can be used to practice filtering attributes.
> + */
> +static void example_slow_test(struct kunit *test)
> +{
> +       KUNIT_EXPECT_EQ(test, 1 + 1, 2);
> +}
> +
>  /*
>   * Here we make a list of all the test cases we want to add to the test suite
>   * below.
> @@ -237,6 +245,7 @@ static struct kunit_case example_test_cases[] = {
>         KUNIT_CASE(example_all_expect_macros_test),
>         KUNIT_CASE(example_static_stub_test),
>         KUNIT_CASE_PARAM(example_params_test, example_gen_params),
> +       KUNIT_CASE_SLOW(example_slow_test),
>         {}
>  };
>
> --
> 2.41.0.255.g8b1d071c50-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 3/9] kunit: Add module attribute

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 5488 bytes --]

On Sat, 8 Jul 2023 at 05:09, Rae Moar <rmoar@google.com> wrote:
>
> Add module attribute to the test attribute API. This attribute stores the
> module name associated with the test using KBUILD_MODNAME.
>
> The name of a test suite and the module name often do not match. A
> reference to the module name associated with the suite could be extremely
> helpful in running tests as modules without needing to check the codebase.
>
> This attribute will be printed for each suite.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---
>

This has already been convenient for me, thanks.

Reviewed-by: David Gow <davidgow@google.com>


-- David

> Changes: since v1:
> - This is a new patch.
>
>  include/kunit/test.h   | 13 ++++++++-----
>  lib/kunit/attributes.c | 28 +++++++++++++++++++++++++++-
>  2 files changed, 35 insertions(+), 6 deletions(-)
>
> diff --git a/include/kunit/test.h b/include/kunit/test.h
> index c255c98a58f7..cdfc3f42e899 100644
> --- a/include/kunit/test.h
> +++ b/include/kunit/test.h
> @@ -129,6 +129,7 @@ struct kunit_case {
>
>         /* private: internal use only. */
>         enum kunit_status status;
> +       char *module_name;
>         char *log;
>  };
>
> @@ -153,7 +154,9 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>   * &struct kunit_case object from it. See the documentation for
>   * &struct kunit_case for an example on how to use it.
>   */
> -#define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
> +#define KUNIT_CASE(test_name)                  \
> +               { .run_case = test_name, .name = #test_name,    \
> +                 .module_name = KBUILD_MODNAME}
>
>  /**
>   * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
> @@ -165,7 +168,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>   */
>  #define KUNIT_CASE_ATTR(test_name, attributes)                 \
>                 { .run_case = test_name, .name = #test_name,    \
> -                 .attr = attributes }
> +                 .attr = attributes, .module_name = KBUILD_MODNAME}
>
>  /**
>   * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case
> @@ -176,7 +179,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>
>  #define KUNIT_CASE_SLOW(test_name)                     \
>                 { .run_case = test_name, .name = #test_name,    \
> -                 .attr.speed = KUNIT_SPEED_SLOW }
> +                 .attr.speed = KUNIT_SPEED_SLOW, .module_name = KBUILD_MODNAME}
>
>  /**
>   * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
> @@ -197,7 +200,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>   */
>  #define KUNIT_CASE_PARAM(test_name, gen_params)                        \
>                 { .run_case = test_name, .name = #test_name,    \
> -                 .generate_params = gen_params }
> +                 .generate_params = gen_params, .module_name = KBUILD_MODNAME}
>
>  /**
>   * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
> @@ -211,7 +214,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
>  #define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes)       \
>                 { .run_case = test_name, .name = #test_name,    \
>                   .generate_params = gen_params,                                \
> -                 .attr = attributes }
> +                 .attr = attributes, .module_name = KBUILD_MODNAME}
>
>  /**
>   * struct kunit_suite - describes a related collection of &struct kunit_case
> diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> index e97096dbb3b1..43dcb5de8b8f 100644
> --- a/lib/kunit/attributes.c
> +++ b/lib/kunit/attributes.c
> @@ -61,6 +61,12 @@ static const char *attr_speed_to_string(void *attr, bool *to_free)
>         return attr_enum_to_string(attr, speed_str_list, to_free);
>  }
>
> +static const char *attr_string_to_string(void *attr, bool *to_free)
> +{
> +       *to_free = false;
> +       return (char *) attr;
> +}
> +
>  /* Get Attribute Methods */
>
>  static void *attr_speed_get(void *test_or_suite, bool is_test)
> @@ -74,6 +80,18 @@ static void *attr_speed_get(void *test_or_suite, bool is_test)
>                 return ((void *) suite->attr.speed);
>  }
>
> +static void *attr_module_get(void *test_or_suite, bool is_test)
> +{
> +       struct kunit_suite *suite = is_test ? NULL : test_or_suite;
> +       struct kunit_case *test = is_test ? test_or_suite : NULL;
> +
> +       // Suites get their module attribute from their first test_case
> +       if (test)
> +               return ((void *) test->module_name);
> +       else
> +               return ((void *) suite->test_cases[0].module_name);
> +}
> +
>  /* Attribute Struct Definitions */
>
>  static const struct kunit_attr speed_attr = {
> @@ -84,9 +102,17 @@ static const struct kunit_attr speed_attr = {
>         .print = PRINT_ALWAYS,
>  };
>
> +static const struct kunit_attr module_attr = {
> +       .name = "module",
> +       .get_attr = attr_module_get,
> +       .to_string = attr_string_to_string,
> +       .attr_default = (void *)"",
> +       .print = PRINT_SUITE,
> +};
> +
>  /* List of all Test Attributes */
>
> -static struct kunit_attr kunit_attr_list[] = {speed_attr};
> +static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};
>
>  /* Helper Functions to Access Attributes */
>
> --
> 2.41.0.255.g8b1d071c50-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 4/9] kunit: Add ability to filter attributes

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 25542 bytes --]

On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
>
> Add filtering of test attributes. Users can filter tests using the
> module_param called "filter".
>
> Filters are imputed in the format: <attribute_name><operation><value>
>
> Example: kunit.filter="speed>slow"
>
> Operations include: >, <, >=, <=, !=, and =. These operations will act the
> same for attributes of the same type but may not between types.
>
> Note multiple filters can be inputted by separating them with a comma.
> Example: kunit.filter="speed=slow, module!=example"
>
> Since both suites and test cases can have attributes, there may be
> conflicts. The process of filtering follows these rules:
> - Filtering always operates at a per-test level.
> - If a test has an attribute set, then the test's value is filtered on.
> - Otherwise, the value falls back to the suite's value.
> - If neither are set, the attribute has a global "default" value, which
>   is used.
>
> Filtered tests will not be run or show in output. The tests can instead be
> skipped using the configurable option "kunit.filter_action=skip".
>
> Note the default settings for running tests remains unfiltered.
>
> Finally, add "filter" methods for the speed and module attributes to parse
> and compare attribute values.
>
> Note this filtering functionality will be added to kunit.py in the next
> patch.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---
>
> Changes since v1:
> - Change method for inputting filters to allow for spaces in filtering
>   values
> - Add option to skip filtered tests instead of not run or show them with
>   the --filter_skip flag
>
>  include/kunit/attributes.h |  31 +++++
>  lib/kunit/attributes.c     | 256 +++++++++++++++++++++++++++++++++++++
>  lib/kunit/executor.c       |  94 +++++++++++---
>  lib/kunit/executor_test.c  |  12 +-
>  lib/kunit/test.c           |  10 +-
>  5 files changed, 375 insertions(+), 28 deletions(-)
>
> diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
> index 9fcd184cce36..bc76a0b786d2 100644
> --- a/include/kunit/attributes.h
> +++ b/include/kunit/attributes.h
> @@ -9,6 +9,20 @@
>  #ifndef _KUNIT_ATTRIBUTES_H
>  #define _KUNIT_ATTRIBUTES_H
>
> +/*
> + * struct kunit_attr_filter - representation of attributes filter with the
> + * attribute object and string input
> + */
> +struct kunit_attr_filter {
> +       struct kunit_attr *attr;
> +       char *input;
> +};
> +
> +/*
> + * Returns the name of the filter's attribute.
> + */
> +const char *kunit_attr_filter_name(struct kunit_attr_filter filter);
> +
>  /*
>   * Print all test attributes for a test case or suite.
>   * Output format for test cases: "# <test_name>.<attribute>: <value>"
> @@ -16,4 +30,21 @@
>   */
>  void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
>
> +/*
> + * Returns the number of fitlers in input.
> + */
> +int kunit_get_filter_count(char *input);
> +
> +/*
> + * Parse attributes filter input and return an objects containing the
> + * attribute object and the string input of the next filter.
> + */
> +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err);
> +
> +/*
> + * Returns a copy of the suite containing only tests that pass the filter.
> + */
> +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
> +               struct kunit_attr_filter filter, char *action, int *err);
> +
>  #endif /* _KUNIT_ATTRIBUTES_H */
> diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> index 43dcb5de8b8f..91cbcacafba9 100644
> --- a/lib/kunit/attributes.c
> +++ b/lib/kunit/attributes.c
> @@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bool *to_free)
>         return (char *) attr;
>  }
>
> +/* Filter Methods */
> +
> +static const char op_list[] = "<>!=";
> +
> +/*
> + * Returns whether the inputted integer value matches the filter given
> + * by the operation string and inputted integer.
> + */
> +static int int_filter(long val, const char *op, int input, int *err)
> +{
> +       if (!strncmp(op, "<=", 2))
> +               return (val <= input);
> +       else if (!strncmp(op, ">=", 2))
> +               return (val >= input);
> +       else if (!strncmp(op, "!=", 2))
> +               return (val != input);
> +       else if (!strncmp(op, ">", 1))
> +               return (val > input);
> +       else if (!strncmp(op, "<", 1))
> +               return (val < input);
> +       else if (!strncmp(op, "=", 1))
> +               return (val == input);
> +       *err = -EINVAL;
> +       pr_err("kunit executor: invalid filter operation: %s\n", op);

More a nitpick for the kunit.py patch, but I'd love to have this shown
to the user as an error when run under kunit.py. It's very annoying to
miss this and only get a "no tests run" error (or worse, unfiltered
results) back.


> +       return false;
> +}
> +
> +/*
> + * Returns whether the inputted enum value "attr" matches the filter given
> + * by the input string. Note: the str_list includes the corresponding string
> + * list to the enum values.
> + */
> +static int attr_enum_filter(void *attr, const char *input, int *err,
> +               const char * const str_list[], int max)
> +{
> +       int i, j, input_int;
> +       long test_val = (long)attr;
> +       const char *input_val;
> +
> +       for (i = 0; input[i]; i++) {
> +               if (!strchr(op_list, input[i])) {
> +                       input_val = input + i;
> +                       break;
> +               }
> +       }
> +
> +       if (!input_val) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: filter value not found: %s\n", input);
> +               return false;
> +       }
> +
> +       for (j = 0; j <= max; j++) {
> +               if (!strcmp(input_val, str_list[j]))
> +                       input_int = j;
> +       }
> +
> +       if (!input_int) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: invalid filter input: %s\n", input);
> +               return false;
> +       }
> +
> +       return int_filter(test_val, input, input_int, err);
> +}
> +
> +static int attr_speed_filter(void *attr, const char *input, int *err)
> +{
> +       return attr_enum_filter(attr, input, err, speed_str_list, KUNIT_SPEED_MAX);
> +}
> +
> +/*
> + * Returns whether the inputted string value (attr) matches the filter given
> + * by the input string.
> + */
> +static int attr_string_filter(void *attr, const char *input, int *err)
> +{
> +       char *str = attr;
> +
> +       if (!strncmp(input, "<", 1)) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: invalid filter input: %s\n", input);
> +               return false;
> +       } else if (!strncmp(input, ">", 1)) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: invalid filter input: %s\n", input);
> +               return false;
> +       } else if (!strncmp(input, "!=", 2)) {
> +               return (strcmp(input + 2, str) != 0);
> +       } else if (!strncmp(input, "=", 1)) {
> +               return (strcmp(input + 1, str) == 0);
> +       }
> +       *err = -EINVAL;
> +       pr_err("kunit executor: invalid filter operation: %s\n", input);
> +       return false;
> +}
> +
> +
>  /* Get Attribute Methods */
>
>  static void *attr_speed_get(void *test_or_suite, bool is_test)
> @@ -98,6 +196,7 @@ static const struct kunit_attr speed_attr = {
>         .name = "speed",
>         .get_attr = attr_speed_get,
>         .to_string = attr_speed_to_string,
> +       .filter = attr_speed_filter,
>         .attr_default = (void *)KUNIT_SPEED_NORMAL,
>         .print = PRINT_ALWAYS,
>  };
> @@ -106,6 +205,7 @@ static const struct kunit_attr module_attr = {
>         .name = "module",
>         .get_attr = attr_module_get,
>         .to_string = attr_string_to_string,
> +       .filter = attr_string_filter,
>         .attr_default = (void *)"",
>         .print = PRINT_SUITE,
>  };
> @@ -116,6 +216,11 @@ static struct kunit_attr kunit_attr_list[] = {speed_attr, module_attr};
>
>  /* Helper Functions to Access Attributes */
>
> +const char *kunit_attr_filter_name(struct kunit_attr_filter filter)
> +{
> +       return filter.attr->name;
> +}
> +
>  void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
>  {
>         int i;
> @@ -148,3 +253,154 @@ void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level
>                 }
>         }
>  }
> +
> +/* Helper Functions to Filter Attributes */
> +
> +int kunit_get_filter_count(char *input)
> +{
> +       int i, comma_index, count = 0;
> +
> +       for (i = 0; input[i]; i++) {
> +               if (input[i] == ',') {
> +                       if ((i - comma_index) > 1)
> +                               count++;
> +                       comma_index = i;
> +               }
> +       }
> +       if ((i - comma_index) > 1)
> +               count++;
> +       return count;
> +}
> +
> +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err)
> +{
> +       struct kunit_attr_filter filter;
> +       int i, j, comma_index, new_start_index;
> +       int op_index = -1, attr_index = -1;
> +       char op;
> +       char *input = *filters;
> +
> +       /* Parse input until operation */
> +       for (i = 0; input[i]; i++) {
> +               if (op_index < 0 && strchr(op_list, input[i])) {
> +                       op_index = i;
> +               } else if (!comma_index && input[i] == ',') {
> +                       comma_index = i;
> +               } else if (comma_index && input[i] != ' ') {
> +                       new_start_index = i;
> +                       break;
> +               }
> +       }
> +
> +       if (op_index <= 0) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: filter operation not found: %s\n", input);
> +               return filter;
> +       }
> +
> +       /* Temporarily set operator to \0 character. */
> +       op = input[op_index];
> +       input[op_index] = '\0';
> +
> +       /* Find associated kunit_attr object */
> +       for (j = 0; j < ARRAY_SIZE(kunit_attr_list); j++) {
> +               if (!strcmp(input, kunit_attr_list[j].name)) {
> +                       attr_index = j;
> +                       break;
> +               }
> +       }
> +
> +       input[op_index] = op;
> +
> +       if (attr_index < 0) {
> +               *err = -EINVAL;
> +               pr_err("kunit executor: attribute not found: %s\n", input);
> +       } else {
> +               filter.attr = &kunit_attr_list[attr_index];
> +       }
> +
> +       if (comma_index) {
> +               input[comma_index] = '\0';
> +               filter.input = input + op_index;
> +               input = input + new_start_index;
> +       } else {
> +               filter.input = input + op_index;
> +               input = NULL;
> +       }
> +
> +       *filters = input;
> +
> +       return filter;
> +}
> +
> +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
> +               struct kunit_attr_filter filter, char *action, int *err)
> +{
> +       int n = 0;
> +       struct kunit_case *filtered, *test_case;
> +       struct kunit_suite *copy;
> +       void *suite_val, *test_val;
> +       bool suite_result, test_result, default_result, result;
> +
> +       /* Allocate memory for new copy of suite and list of test cases */
> +       copy = kmemdup(suite, sizeof(*copy), GFP_KERNEL);
> +       if (!copy)
> +               return ERR_PTR(-ENOMEM);
> +
> +       kunit_suite_for_each_test_case(suite, test_case) { n++; }
> +
> +       filtered = kcalloc(n + 1, sizeof(*filtered), GFP_KERNEL);
> +       if (!filtered) {
> +               kfree(copy);
> +               return ERR_PTR(-ENOMEM);
> +       }
> +
> +       n = 0;
> +
> +       /* Save filtering result on default value */
> +       default_result = filter.attr->filter(filter.attr->attr_default, filter.input, err);
> +
> +       /* Save suite attribute value and filtering result on that value */
> +       suite_val = filter.attr->get_attr((void *)suite, false);
> +       suite_result = filter.attr->filter(suite_val, filter.input, err);
> +
> +       /* For each test case, save test case if passes filtering. */
> +       kunit_suite_for_each_test_case(suite, test_case) {
> +               test_val = filter.attr->get_attr((void *) test_case, true);
> +               test_result = filter.attr->filter(filter.attr->get_attr(test_case, true),
> +                               filter.input, err);
> +
> +               /*
> +                * If attribute value of test case is set, filter on that value.
> +                * If not, filter on suite value if set. If not, filter on
> +                * default value.
> +                */
> +               result = false;
> +               if (test_val) {
> +                       if (test_result)
> +                               result = true;
> +               } else if (suite_val) {
> +                       if (suite_result)
> +                               result = true;
> +               } else if (default_result) {
> +                       result = true;
> +               }
> +
> +               if (result) {
> +                       filtered[n++] = *test_case;
> +               } else if (action && strcmp(action, "skip") == 0) {
> +                       test_case->status = KUNIT_SKIPPED;
> +                       filtered[n++] = *test_case;
> +               }
> +       }
> +
> +       if (n == 0) {
> +               kfree(copy);
> +               kfree(filtered);
> +               return NULL;
> +       }
> +
> +       copy->test_cases = filtered;
> +
> +       return copy;
> +}
> diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> index 12e38a48a5cc..c286ae47435a 100644
> --- a/lib/kunit/executor.c
> +++ b/lib/kunit/executor.c
> @@ -17,6 +17,9 @@ extern struct kunit_suite * const __kunit_suites_end[];
>
>  static char *filter_glob_param;
>  static char *action_param;
> +static char *filter_param;
> +static char *filter_action_param;
> +
>
>  module_param_named(filter_glob, filter_glob_param, charp, 0);
>  MODULE_PARM_DESC(filter_glob,
> @@ -27,15 +30,23 @@ MODULE_PARM_DESC(action,
>                  "<none>: run the tests like normal\n"
>                  "'list' to list test names instead of running them.\n"
>                  "'list_attr' to list test names and attributes instead of running them.\n");
> +module_param_named(filter, filter_param, charp, 0);
> +MODULE_PARM_DESC(filter,
> +               "Filter which KUnit test suites/tests run at boot-time using attributes, e.g. speed>slow");
> +module_param_named(filter_action, filter_action_param, charp, 0);
> +MODULE_PARM_DESC(filter_action,
> +               "Changes behavior of filtered tests using attributes, valid values are:\n"
> +               "<none>: do not run filtered tests as normal\n"
> +               "'skip': skip all filtered tests instead so tests will appear in output\n");
>
>  /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
> -struct kunit_test_filter {
> +struct kunit_glob_filter {
>         char *suite_glob;
>         char *test_glob;
>  };
>
>  /* Split "suite_glob.test_glob" into two. Assumes filter_glob is not empty. */
> -static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
> +static void kunit_parse_glob_filter(struct kunit_glob_filter *parsed,
>                                     const char *filter_glob)
>  {
>         const int len = strlen(filter_glob);
> @@ -57,7 +68,7 @@ static void kunit_parse_filter_glob(struct kunit_test_filter *parsed,
>
>  /* Create a copy of suite with only tests that match test_glob. */
>  static struct kunit_suite *
> -kunit_filter_tests(const struct kunit_suite *const suite, const char *test_glob)
> +kunit_filter_glob_tests(const struct kunit_suite *const suite, const char *test_glob)
>  {
>         int n = 0;
>         struct kunit_case *filtered, *test_case;
> @@ -111,12 +122,15 @@ static void kunit_free_suite_set(struct suite_set suite_set)
>
>  static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
>                                             const char *filter_glob,
> +                                               char *filters,
> +                                               char *filter_action,
>                                             int *err)
>  {
> -       int i;
> -       struct kunit_suite **copy, *filtered_suite;
> +       int i, j, k, filter_count;
> +       struct kunit_suite **copy, *filtered_suite, *new_filtered_suite;
>         struct suite_set filtered;
> -       struct kunit_test_filter filter;
> +       struct kunit_glob_filter parsed_glob;
> +       struct kunit_attr_filter *parsed_filters;
>
>         const size_t max = suite_set->end - suite_set->start;
>
> @@ -127,17 +141,52 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
>                 return filtered;
>         }
>
> -       kunit_parse_filter_glob(&filter, filter_glob);
> -
> -       for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
> -               if (!glob_match(filter.suite_glob, suite_set->start[i]->name))
> -                       continue;
> +       if (filter_glob)
> +               kunit_parse_glob_filter(&parsed_glob, filter_glob);
>
> -               filtered_suite = kunit_filter_tests(suite_set->start[i], filter.test_glob);
> -               if (IS_ERR(filtered_suite)) {
> -                       *err = PTR_ERR(filtered_suite);
> +       /* Parse attribute filters */
> +       if (filters) {
> +               filter_count = kunit_get_filter_count(filters);
> +               parsed_filters = kcalloc(filter_count + 1, sizeof(*parsed_filters), GFP_KERNEL);
> +               for (j = 0; j < filter_count; j++)
> +                       parsed_filters[j] = kunit_next_attr_filter(&filters, err);
> +               if (*err)
>                         return filtered;
> +       }
> +
> +       for (i = 0; &suite_set->start[i] != suite_set->end; i++) {
> +               filtered_suite = suite_set->start[i];
> +               if (filter_glob) {
> +                       if (!glob_match(parsed_glob.suite_glob, filtered_suite->name))
> +                               continue;
> +                       filtered_suite = kunit_filter_glob_tests(filtered_suite,
> +                                       parsed_glob.test_glob);
> +                       if (IS_ERR(filtered_suite)) {
> +                               *err = PTR_ERR(filtered_suite);
> +                               return filtered;
> +                       }
> +               }
> +               if (filter_count) {
> +                       for (k = 0; k < filter_count; k++) {
> +                               new_filtered_suite = kunit_filter_attr_tests(filtered_suite,
> +                                               parsed_filters[k], filter_action, err);
> +
> +                               /* Free previous copy of suite */
> +                               if (k > 0 || filter_glob)
> +                                       kfree(filtered_suite);
> +                               filtered_suite = new_filtered_suite;
> +
> +                               if (*err)
> +                                       return filtered;
> +                               if (IS_ERR(filtered_suite)) {
> +                                       *err = PTR_ERR(filtered_suite);
> +                                       return filtered;
> +                               }
> +                               if (!filtered_suite)
> +                                       break;
> +                       }
>                 }
> +
>                 if (!filtered_suite)
>                         continue;
>
> @@ -145,8 +194,14 @@ static struct suite_set kunit_filter_suites(const struct suite_set *suite_set,
>         }
>         filtered.end = copy;
>
> -       kfree(filter.suite_glob);
> -       kfree(filter.test_glob);
> +       if (filter_glob) {
> +               kfree(parsed_glob.suite_glob);
> +               kfree(parsed_glob.test_glob);
> +       }
> +
> +       if (filter_count)
> +               kfree(parsed_filters);
> +
>         return filtered;
>  }
>
> @@ -206,8 +261,9 @@ int kunit_run_all_tests(void)
>                 goto out;
>         }
>
> -       if (filter_glob_param) {
> -               suite_set = kunit_filter_suites(&suite_set, filter_glob_param, &err);
> +       if (filter_glob_param || filter_param) {
> +               suite_set = kunit_filter_suites(&suite_set, filter_glob_param,
> +                               filter_param, filter_action_param, &err);
>                 if (err) {
>                         pr_err("kunit executor: error filtering suites: %d\n", err);
>                         goto out;
> @@ -223,7 +279,7 @@ int kunit_run_all_tests(void)
>         else
>                 pr_err("kunit executor: unknown action '%s'\n", action_param);
>
> -       if (filter_glob_param) { /* a copy was made of each suite */
> +       if (filter_glob_param || filter_param) { /* a copy was made of each suite */
>                 kunit_free_suite_set(suite_set);
>         }
>
> diff --git a/lib/kunit/executor_test.c b/lib/kunit/executor_test.c
> index ce6749af374d..d7ab069324b5 100644
> --- a/lib/kunit/executor_test.c
> +++ b/lib/kunit/executor_test.c
> @@ -24,15 +24,15 @@ static struct kunit_case dummy_test_cases[] = {
>
>  static void parse_filter_test(struct kunit *test)
>  {
> -       struct kunit_test_filter filter = {NULL, NULL};
> +       struct kunit_glob_filter filter = {NULL, NULL};
>
> -       kunit_parse_filter_glob(&filter, "suite");
> +       kunit_parse_glob_filter(&filter, "suite");
>         KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
>         KUNIT_EXPECT_FALSE(test, filter.test_glob);
>         kfree(filter.suite_glob);
>         kfree(filter.test_glob);
>
> -       kunit_parse_filter_glob(&filter, "suite.test");
> +       kunit_parse_glob_filter(&filter, "suite.test");
>         KUNIT_EXPECT_STREQ(test, filter.suite_glob, "suite");
>         KUNIT_EXPECT_STREQ(test, filter.test_glob, "test");
>         kfree(filter.suite_glob);
> @@ -50,7 +50,7 @@ static void filter_suites_test(struct kunit *test)
>         subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
>
>         /* Want: suite1, suite2, NULL -> suite2, NULL */
> -       got = kunit_filter_suites(&suite_set, "suite2", &err);
> +       got = kunit_filter_suites(&suite_set, "suite2", NULL, NULL, &err);
>         KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
>         KUNIT_ASSERT_EQ(test, err, 0);
>         kfree_at_end(test, got.start);
> @@ -74,7 +74,7 @@ static void filter_suites_test_glob_test(struct kunit *test)
>         subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
>
>         /* Want: suite1, suite2, NULL -> suite2 (just test1), NULL */
> -       got = kunit_filter_suites(&suite_set, "suite2.test2", &err);
> +       got = kunit_filter_suites(&suite_set, "suite2.test2", NULL, NULL, &err);
>         KUNIT_ASSERT_NOT_ERR_OR_NULL(test, got.start);
>         KUNIT_ASSERT_EQ(test, err, 0);
>         kfree_at_end(test, got.start);
> @@ -100,7 +100,7 @@ static void filter_suites_to_empty_test(struct kunit *test)
>         subsuite[0] = alloc_fake_suite(test, "suite1", dummy_test_cases);
>         subsuite[1] = alloc_fake_suite(test, "suite2", dummy_test_cases);
>
> -       got = kunit_filter_suites(&suite_set, "not_found", &err);
> +       got = kunit_filter_suites(&suite_set, "not_found", NULL, NULL, &err);
>         KUNIT_ASSERT_EQ(test, err, 0);
>         kfree_at_end(test, got.start); /* just in case */
>
> diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> index 9ee55139ecd1..cb9797fa6303 100644
> --- a/lib/kunit/test.c
> +++ b/lib/kunit/test.c
> @@ -613,18 +613,22 @@ int kunit_run_tests(struct kunit_suite *suite)
>         kunit_suite_for_each_test_case(suite, test_case) {
>                 struct kunit test = { .param_value = NULL, .param_index = 0 };
>                 struct kunit_result_stats param_stats = { 0 };
> -               test_case->status = KUNIT_SKIPPED;
>
>                 kunit_init_test(&test, test_case->name, test_case->log);
> -
> -               if (!test_case->generate_params) {
> +               if (test_case->status == KUNIT_SKIPPED) {
> +                       /* Test marked as skip */
> +                       test.status = KUNIT_SKIPPED;
> +                       kunit_update_stats(&param_stats, test.status);
> +               } else if (!test_case->generate_params) {
>                         /* Non-parameterised test. */
> +                       test_case->status = KUNIT_SKIPPED;
>                         kunit_run_case_catch_errors(suite, test_case, &test);
>                         kunit_update_stats(&param_stats, test.status);
>                 } else {
>                         /* Get initial param. */
>                         param_desc[0] = '\0';
>                         test.param_value = test_case->generate_params(NULL, param_desc);
> +                       test_case->status = KUNIT_SKIPPED;
>                         kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
>                                   "KTAP version 1\n");
>                         kunit_log(KERN_INFO, &test, KUNIT_SUBTEST_INDENT KUNIT_SUBTEST_INDENT
> --
> 2.41.0.255.g8b1d071c50-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 5/9] kunit: tool: Add command line interface to filter and report attributes

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 21093 bytes --]

On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
>
> Add ability to kunit.py to filter attributes and report a list of tests
> including attributes without running tests.
>
> Add flag "--filter" to input filters on test attributes. Tests will be
> filtered out if they do not match all inputted filters.
>
> Example: --filter speed=slow (This filter would run only the tests that are
> marked as slow)
>
> Filters have operations: <, >, <=, >=, !=, and =. But note that the
> characters < and > are often interpreted by the shell, so they may need to
> be quoted or escaped.
>
> Example: --filter "speed>slow" or --filter speed\>slow (This filter would
> run only the tests that have the speed faster than slow.
>
> Additionally, multiple filters can be used.
>
> Example: --filter "speed=slow, module!=example" (This filter would run
> only the tests that have the speed slow and are not in the "example"
> module)
>
> Note if the user wants to skip filtered tests instead of not
> running/showing them use the "--filter_skip" flag instead.
>
> Expose the output of kunit.action=list option with flag "--list_tests" to
> output a list of tests. Additionally, add flag "--list_tests_attr" to
> output a list of tests and their attributes. These flags are useful to see
> tests and test attributes without needing to run tests.
>
> Example of the output of "--list_tests_attr":
>   example
>   example.test_1
>   example.test_2
>   # example.test_2.speed: slow
>
> This output includes a suite, example, with two test cases, test_1 and
> test_2. And in this instance test_2 has been marked as slow.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---
>
> Changes since v1:
> - Change method for inputting filters to allow for spaces in filtering
>   values
> - Add option to skip filtered tests instead of not run or show them with
>   the --filter_skip flag
> - Separate the new feature to list tests and their attributes into both
>   --list_tests (lists just tests) and --list_tests_attr (lists all)
>
>  tools/testing/kunit/kunit.py           | 80 ++++++++++++++++++++++++--
>  tools/testing/kunit/kunit_kernel.py    |  6 +-
>  tools/testing/kunit/kunit_tool_test.py | 39 ++++++-------
>  3 files changed, 96 insertions(+), 29 deletions(-)
>
> diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
> index 3905c43369c3..6104e622ce20 100755
> --- a/tools/testing/kunit/kunit.py
> +++ b/tools/testing/kunit/kunit.py
> @@ -55,8 +55,12 @@ class KunitExecRequest(KunitParseRequest):
>         build_dir: str
>         timeout: int
>         filter_glob: str
> +       filter: str
> +       filter_skip: str
>         kernel_args: Optional[List[str]]
>         run_isolated: Optional[str]
> +       list_tests: bool
> +       list_tests_attr: bool
>
>  @dataclass
>  class KunitRequest(KunitExecRequest, KunitBuildRequest):
> @@ -102,19 +106,39 @@ def config_and_build_tests(linux: kunit_kernel.LinuxSourceTree,
>
>  def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> List[str]:
>         args = ['kunit.action=list']
> +
> +       if request.kernel_args:
> +               args.extend(request.kernel_args)
> +
> +       output = linux.run_kernel(args=args,
> +                          timeout=request.timeout,
> +                          filter_glob=request.filter_glob,
> +                          filter=request.filter,
> +                          build_dir=request.build_dir)
> +       lines = kunit_parser.extract_tap_lines(output)
> +       # Hack! Drop the dummy TAP version header that the executor prints out.
> +       lines.pop()
> +
> +       # Filter out any extraneous non-test output that might have gotten mixed in.
> +       return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
> +
> +def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]:
> +       args = ['kunit.action=list_attr']
> +
>         if request.kernel_args:
>                 args.extend(request.kernel_args)
>
>         output = linux.run_kernel(args=args,
>                            timeout=request.timeout,
>                            filter_glob=request.filter_glob,
> +                          filter=request.filter,
>                            build_dir=request.build_dir)
>         lines = kunit_parser.extract_tap_lines(output)
>         # Hack! Drop the dummy TAP version header that the executor prints out.
>         lines.pop()
>
>         # Filter out any extraneous non-test output that might have gotten mixed in.
> -       return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
> +       return lines
>
>  def _suites_from_test_list(tests: List[str]) -> List[str]:
>         """Extracts all the suites from an ordered list of tests."""
> @@ -128,10 +152,18 @@ def _suites_from_test_list(tests: List[str]) -> List[str]:
>                         suites.append(suite)
>         return suites
>
> -
> -
>  def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> KunitResult:
>         filter_globs = [request.filter_glob]
> +       if request.list_tests:
> +               output = _list_tests(linux, request)
> +               for line in output:
> +                       print(line.rstrip())
> +               return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0)
> +       if request.list_tests_attr:
> +               attr_output = _list_tests_attr(linux, request)
> +               for line in attr_output:
> +                       print(line.rstrip())
> +               return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0)
>         if request.run_isolated:
>                 tests = _list_tests(linux, request)
>                 if request.run_isolated == 'test':
> @@ -145,6 +177,17 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
>
>         metadata = kunit_json.Metadata(arch=linux.arch(), build_dir=request.build_dir, def_config='kunit_defconfig')
>
> +       filter = request.filter
> +       if request.filter_skip:
> +               args = ['kunit.filter_action=skip']
> +               filter = request.filter_skip
> +               if request.kernel_args:
> +                       args.extend(request.kernel_args)

What happens if both filter and filter_skip are set? We should
probably either make those mutually exclusive (error if both are set),
or expose filter_action directly instead.

> +       elif request.kernel_args:
> +               args = request.kernel_args
> +       else:
> +               args = None
> +
>         test_counts = kunit_parser.TestCounts()
>         exec_time = 0.0
>         for i, filter_glob in enumerate(filter_globs):
> @@ -152,9 +195,10 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
>
>                 test_start = time.time()
>                 run_result = linux.run_kernel(
> -                       args=request.kernel_args,
> +                       args=args,
>                         timeout=request.timeout,
>                         filter_glob=filter_glob,
> +                       filter=filter,
>                         build_dir=request.build_dir)
>
>                 _, test_result = parse_tests(request, metadata, run_result)
> @@ -341,6 +385,18 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None:
>                             nargs='?',
>                             default='',
>                             metavar='filter_glob')
> +       parser.add_argument('--filter',
> +                           help='Filter KUnit tests with attributes, '
> +                               'filtered tests will not run, '
> +                           'e.g. speed=fast or speed=>low',

Neither fast or low are valid values for speed now.

> +                           type=str,
> +                               default='')
> +       parser.add_argument('--filter_skip',
> +                           help='Filter KUnit tests run with attributes, '
> +                               'filtered tests will be skipped, '
> +                           'e.g. speed=fast or speed=>low',

Neither fast or low are valid values for speed now.



> +                           type=str,
> +                               default='')
>         parser.add_argument('--kernel_args',
>                             help='Kernel command-line parameters. Maybe be repeated',
>                              action='append', metavar='')
> @@ -350,6 +406,10 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None:
>                             'what ran before it.',
>                             type=str,
>                             choices=['suite', 'test'])
> +       parser.add_argument('--list_tests', help='If set, list all tests',
> +                           action='store_true')
> +       parser.add_argument('--list_tests_attr', help='If set, list all tests and attributes.',
> +                           action='store_true')
>
>  def add_parse_opts(parser: argparse.ArgumentParser) -> None:
>         parser.add_argument('--raw_output', help='If set don\'t parse output from kernel. '
> @@ -398,8 +458,12 @@ def run_handler(cli_args: argparse.Namespace) -> None:
>                                         json=cli_args.json,
>                                         timeout=cli_args.timeout,
>                                         filter_glob=cli_args.filter_glob,
> +                                       filter=cli_args.filter,
> +                                       filter_skip=cli_args.filter_skip,
>                                         kernel_args=cli_args.kernel_args,
> -                                       run_isolated=cli_args.run_isolated)
> +                                       run_isolated=cli_args.run_isolated,
> +                                       list_tests=cli_args.list_tests,
> +                                       list_tests_attr=cli_args.list_tests_attr)
>         result = run_tests(linux, request)
>         if result.status != KunitStatus.SUCCESS:
>                 sys.exit(1)
> @@ -441,8 +505,12 @@ def exec_handler(cli_args: argparse.Namespace) -> None:
>                                         json=cli_args.json,
>                                         timeout=cli_args.timeout,
>                                         filter_glob=cli_args.filter_glob,
> +                                       filter=cli_args.filter,
> +                                       filter_skip=cli_args.filter_skip,
>                                         kernel_args=cli_args.kernel_args,
> -                                       run_isolated=cli_args.run_isolated)
> +                                       run_isolated=cli_args.run_isolated,
> +                                       list_tests=cli_args.list_tests,
> +                                       list_tests_attr=cli_args.list_tests_attr)
>         result = exec_tests(linux, exec_request)
>         stdout.print_with_timestamp((
>                 'Elapsed time: %.3fs\n') % (result.elapsed_time))
> diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> index 7f648802caf6..281f062a4767 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -330,11 +330,13 @@ class LinuxSourceTree:
>                         return False
>                 return self.validate_config(build_dir)
>
> -       def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', timeout: Optional[int]=None) -> Iterator[str]:
> +       def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', filter: str='', timeout: Optional[int]=None) -> Iterator[str]:
>                 if not args:
>                         args = []
>                 if filter_glob:
> -                       args.append('kunit.filter_glob='+filter_glob)
> +                       args.append('kunit.filter_glob=' + filter_glob)
> +               if filter:
> +                       args.append('kunit.filter="' + filter + '"')
>                 args.append('kunit.enable=1')
>
>                 process = self._ops.start(args, build_dir)
> diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> index be35999bb84f..85a1fb72735e 100755
> --- a/tools/testing/kunit/kunit_tool_test.py
> +++ b/tools/testing/kunit/kunit_tool_test.py
> @@ -597,7 +597,7 @@ class KUnitMainTest(unittest.TestCase):
>                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0)
>                 self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir='.kunit', filter_glob='', timeout=300)
> +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_run_passes_args_pass(self):
> @@ -605,7 +605,7 @@ class KUnitMainTest(unittest.TestCase):
>                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
>                 self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir='.kunit', filter_glob='', timeout=300)
> +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_exec_passes_args_fail(self):
> @@ -629,7 +629,7 @@ class KUnitMainTest(unittest.TestCase):
>                         kunit.main(['run'])
>                 self.assertEqual(e.exception.code, 1)
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir='.kunit', filter_glob='', timeout=300)
> +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
>                 self.print_mock.assert_any_call(StrContains(' 0 tests run!'))
>
>         def test_exec_raw_output(self):
> @@ -670,13 +670,13 @@ class KUnitMainTest(unittest.TestCase):
>                 self.linux_source_mock.run_kernel = mock.Mock(return_value=[])
>                 kunit.main(['run', '--raw_output', 'filter_glob'])
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir='.kunit', filter_glob='filter_glob', timeout=300)
> +                       args=None, build_dir='.kunit', filter_glob='filter_glob', filter='', timeout=300)
>
>         def test_exec_timeout(self):
>                 timeout = 3453
>                 kunit.main(['exec', '--timeout', str(timeout)])
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir='.kunit', filter_glob='', timeout=timeout)
> +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=timeout)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_run_timeout(self):
> @@ -684,7 +684,7 @@ class KUnitMainTest(unittest.TestCase):
>                 kunit.main(['run', '--timeout', str(timeout)])
>                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir='.kunit', filter_glob='', timeout=timeout)
> +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=timeout)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_run_builddir(self):
> @@ -692,7 +692,7 @@ class KUnitMainTest(unittest.TestCase):
>                 kunit.main(['run', '--build_dir=.kunit'])
>                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir=build_dir, filter_glob='', timeout=300)
> +                       args=None, build_dir=build_dir, filter_glob='', filter='', timeout=300)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_config_builddir(self):
> @@ -710,7 +710,7 @@ class KUnitMainTest(unittest.TestCase):
>                 build_dir = '.kunit'
>                 kunit.main(['exec', '--build_dir', build_dir])
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=None, build_dir=build_dir, filter_glob='', timeout=300)
> +                       args=None, build_dir=build_dir, filter_glob='', filter='', timeout=300)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_run_kunitconfig(self):
> @@ -786,7 +786,7 @@ class KUnitMainTest(unittest.TestCase):
>                 kunit.main(['run', '--kernel_args=a=1', '--kernel_args=b=2'])
>                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                     args=['a=1','b=2'], build_dir='.kunit', filter_glob='', timeout=300)
> +                     args=['a=1','b=2'], build_dir='.kunit', filter_glob='', filter='', timeout=300)
>                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
>
>         def test_list_tests(self):
> @@ -794,13 +794,11 @@ class KUnitMainTest(unittest.TestCase):
>                 self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want
>
>                 got = kunit._list_tests(self.linux_source_mock,
> -                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite'))
> -
> +                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', '', None, 'suite', False, False))
>                 self.assertEqual(got, want)
>                 # Should respect the user's filter glob when listing tests.
>                 self.linux_source_mock.run_kernel.assert_called_once_with(
> -                       args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', timeout=300)
> -
> +                       args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', timeout=300)
>
>         @mock.patch.object(kunit, '_list_tests')
>         def test_run_isolated_by_suite(self, mock_tests):
> @@ -809,10 +807,10 @@ class KUnitMainTest(unittest.TestCase):
>
>                 # Should respect the user's filter glob when listing tests.
>                 mock_tests.assert_called_once_with(mock.ANY,
> -                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite'))
> +                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', '', '', None, 'suite', False, False))
>                 self.linux_source_mock.run_kernel.assert_has_calls([
> -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300),
> -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300),
> +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', filter='', timeout=300),
> +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', filter='', timeout=300),
>                 ])
>
>         @mock.patch.object(kunit, '_list_tests')
> @@ -822,13 +820,12 @@ class KUnitMainTest(unittest.TestCase):
>
>                 # Should respect the user's filter glob when listing tests.
>                 mock_tests.assert_called_once_with(mock.ANY,
> -                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test'))
> +                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', '', None, 'test', False, False))
>                 self.linux_source_mock.run_kernel.assert_has_calls([
> -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300),
> -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300),
> -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', timeout=300),
> +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', filter='', timeout=300),
> +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', filter='', timeout=300),
> +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', filter='', timeout=300),
>                 ])
>
> -
>  if __name__ == '__main__':
>         unittest.main()
> --
> 2.41.0.255.g8b1d071c50-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 7/9] kunit: time: Mark test as slow using test attributes

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
>
> Mark the time KUnit test, time64_to_tm_test_date_range, as slow using test
> attributes.
>
> This test ran relatively much slower than most other KUnit tests.
>
> By marking this test as slow, the test can now be filtered using the KUnit
> test attribute filtering feature. Example: --filter "speed>slow". This will
> run only the tests that have speeds faster than slow. The slow attribute
> will also be outputted in KTAP.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---

Thanks: this test is slow enough to really annoy me on some machines.
(It's possibly even on the edge of "very_slow" territory, though I
suspect "slow" is better.)

Reviewed-by: David Gow <davidgow@google.com>

Cheers,
-- David


>
> Changes since v1:
> - No changes.
>
>  kernel/time/time_test.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/kernel/time/time_test.c b/kernel/time/time_test.c
> index 831e8e779ace..ca058c8af6ba 100644
> --- a/kernel/time/time_test.c
> +++ b/kernel/time/time_test.c
> @@ -86,7 +86,7 @@ static void time64_to_tm_test_date_range(struct kunit *test)
>  }
>
>  static struct kunit_case time_test_cases[] = {
> -       KUNIT_CASE(time64_to_tm_test_date_range),
> +       KUNIT_CASE_SLOW(time64_to_tm_test_date_range),
>         {}
>  };
>
> --
> 2.41.0.255.g8b1d071c50-goog
>

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 9/9] kunit: Add documentation of KUnit test attributes

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 7167 bytes --]

On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
>
> Add documentation on the use of test attributes under the section "Tips for
> Running KUnit Tests" in the KUnit docs.
>
> Documentation includes three sections on how to mark tests with attributes,
> how attributes are reported, and how the user can filter tests using test
> attributes.
>
> Signed-off-by: Rae Moar <rmoar@google.com>
> ---

Looks good overall. Some nitpicks below.

Reviewed-by: David Gow <davidgow@google.com>

>
> Changes since v1:
> - This is a new patch
>
>  .../dev-tools/kunit/running_tips.rst          | 163 ++++++++++++++++++
>  1 file changed, 163 insertions(+)
>
> diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
> index 8e8c493f17d1..c9bc5a6595d3 100644
> --- a/Documentation/dev-tools/kunit/running_tips.rst
> +++ b/Documentation/dev-tools/kunit/running_tips.rst
> @@ -262,3 +262,166 @@ other code executed during boot, e.g.
>         # Reset coverage counters before running the test.
>         $ echo 0 > /sys/kernel/debug/gcov/reset
>         $ modprobe kunit-example-test
> +
> +
> +Test Attributes and Filtering
> +=============================
> +
> +Test suites and cases can be marked with test attributes, such as speed of
> +test. These attributes will later be printed in test output and can be used to
> +filter test execution.
> +
> +Marking Test Attributes
> +-----------------------
> +
> +Tests are marked with an attribute by including a ``kunit_attributes`` object
> +in the test definition.
> +
> +Test cases can be marked using the ``KUNIT_CASE_ATTR(test_name, attributes)``
> +macro to define the test case instead of ``KUNIT_CASE(test_name)``.
> +
> +.. code-block:: c
> +
> +       static const struct kunit_attributes example_attr = {
> +               .speed = KUNIT_VERY_SLOW,
> +       };
> +
> +       static struct kunit_case example_test_cases[] = {
> +               KUNIT_CASE_ATTR(example_test, example_attr),
> +       };
> +
> +.. note::
> +       To mark a test case as slow, you can also use ``KUNIT_CASE_SLOW(test_name)``.
> +       This is a helpful macro as the slow attribute is the most commonly used.
> +
> +Test suites can be marked with an attribute by setting the "attr" field in the
> +suite definition.
> +
> +.. code-block:: c
> +
> +       static const struct kunit_attributes example_attr = {
> +               .speed = KUNIT_VERY_SLOW,
> +       };
> +
> +       static struct kunit_suite example_test_suite = {
> +               ...,
> +               .attr = example_attr,
> +       };
> +
> +.. note::
> +       Not all attributes need to be set in a ``kunit_attributes`` object. Unset
> +       attributes will remain uninitialized and act as though the attribute is set
> +       to 0 or NULL. Thus, if an attribute is set to 0, it is treated as unset.
> +       These unset attributes will not be reported and may act as a default value
> +       for filtering purposes.
> +
> +Reporting Attributes
> +--------------------
> +
> +When a user runs tests, attributes will be present in kernel output (in KTAP
> +format). This is an example of how test attributes for test cases will be formatted
> +in Kernel output:
> +
> +.. code-block:: none
> +
> +       # example_test.speed: slow
> +       ok 1 example_test
> +
> +This is an example of how test attributes for test suites will be formatted in
> +Kernel output:
> +
> +.. code-block:: none
> +
> +         KTAP version 2
> +         # Subtest: example_suite
> +         # module: kunit_example_test
> +         1..3
> +         ...
> +       ok 1 example_suite
> +

Maybe worth noting that kunit.py will hide these for passing tests by
default, and --raw_output is needed to see them?

> +Additionally, users can output a full attribute report of tests with their
> +attributes, using the command line flag ``--list_tests_attr``:
> +
> +.. code-block:: bash
> +
> +       kunit.py run "example" --list_tests_attr
> +
> +.. note::
> +       This report can be accessed when running KUnit manually by passing in the
> +       module_param ``kunit.action=list_attr``.
> +
> +Filtering
> +---------
> +
> +Users can filter tests using the ``--filter`` command line flag when running
> +tests. As an example:
> +
> +.. code-block:: bash
> +
> +       kunit.py run --filter speed=slow
> +
> +
> +You can also use the following operations on filters: "<", ">", "<=", ">=",
> +"!=", and "=". Example:
> +
> +.. code-block:: bash
> +
> +       kunit.py run --filter "speed>slow"
> +
> +This example will run all tests with speeds faster than slow. Note that the
> +characters < and > are often interpreted by the shell, so they may need to be
> +quoted or escaped, as above.
> +
> +Additionally, you can use multiple filters at once. Simply separate filters
> +using commas. Example:
> +
> +.. code-block:: bash
> +
> +       kunit.py run --filter "speed>slow, module=kunit_example_test"
> +
> +.. note::
> +       You can use this filtering feature when running KUnit manually by passing
> +       the filter as a module param: ``kunit.filter="speed>slow, speed<=normal"``.
> +
> +Filtered tests will not run or show up in the test output. You can use the
> +``--filter_skip`` flag to skip filtered tests instead. These tests will be
> +shown in the test output in the test but will not run. To use this feature when
> +running KUnit manually, use the ``kunit.filter`` module param with
> +``kunit.filter_action=skip``.
> +
> +Rules of Filtering Procedure
> +----------------------------
> +
> +Since both suites and test cases can have attributes, there may be conflicts
> +between attributes during filtering. The process of filtering follows these
> +rules:
> +
> +- Filtering always operates at a per-test level.
> +
> +- If a test has an attribute set, then the test's value is filtered on.
> +
> +- Otherwise, the value falls back to the suite's value.
> +
> +- If neither are set, the attribute has a global "default" value, which is used.
> +
> +List of Current Attributes
> +--------------------------

I wonder whether this should end up part of the KTAP spec (or as an
appendix/supplement to it). Or even as a separate page within the
KUnit documentation to avoid running_tips.rst from getting too huge.

> +
> +``speed``
> +
> +This attribute indicates the speed of a test's execution (how slow or fast the
> +test is).
> +
> +This attribute is saved as an enum with the following categories: "normal",
> +"slow", or "very_slow". The assumed default speed for tests is "normal". This
> +indicates that the test takes a relatively trivial amount of time (less than
> +1 second), regardless of the machine it is running on. Any test slower than
> +this could be marked as "slow" or "very_slow".

Is it worth noting that "KUNIT_CASE_SLOW()" can be used to easily set
this to slow?


> +
> +``module``
> +
> +This attribute indicates the name of the module associated with the test.
> +
> +This attribute is automatically saved as a string and is printed for each suite.
> +Tests can also be filtered using this attribute.
> +
> --
> 2.41.0.255.g8b1d071c50-goog

>

Error: new blank line at EOF.

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 2/9] kunit: Add speed attribute

[Rae Moar]

On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
>
> On Sat, 8 Jul 2023 at 05:09, Rae Moar <rmoar@google.com> wrote:
> >
> > Add speed attribute to the test attribute API. This attribute will allow
> > users to mark tests with a category of speed.
> >
> > Currently the categories of speed proposed are: normal, slow, and very_slow
> > (outlined in enum kunit_speed). These are outlined in the enum kunit_speed.
> >
> > The assumed default speed for tests is "normal". This indicates that the
> > test takes a relatively trivial amount of time (less than 1 second),
> > regardless of the machine it is running on. Any test slower than this could
> > be marked as "slow" or "very_slow".
> >
> > Add the macro KUNIT_CASE_SLOW to set a test as slow, as this is likely a
> > common use of the attributes API.
> >
> > Add an example of marking a slow test to kunit-example-test.c.
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
> > ---
>
> Looks good.
>
> Reviewed-by: David Gow <davidgow@google.com>
>

Thanks!

> >
> > Changes since v1:
> > - Remove the "fast" category of speed.
> >
> >  include/kunit/test.h           | 30 +++++++++++++++++++++-
> >  lib/kunit/attributes.c         | 46 +++++++++++++++++++++++++++++++++-
> >  lib/kunit/kunit-example-test.c |  9 +++++++
> >  3 files changed, 83 insertions(+), 2 deletions(-)
> >
> > diff --git a/include/kunit/test.h b/include/kunit/test.h
> > index 1fc9155988e9..c255c98a58f7 100644
> > --- a/include/kunit/test.h
> > +++ b/include/kunit/test.h
> > @@ -63,8 +63,25 @@ enum kunit_status {
> >         KUNIT_SKIPPED,
> >  };
> >
> > +/* Attribute struct/enum definitions */
> > +
> > +/*
> > + * Speed Attribute is stored as an enum and separated into categories of
> > + * speed: very_slowm, slow, normal, and fast. These speeds are relative
>
> Nit: we only have very_slow, slow & normal now.
>

Oops thanks for catching this. Will fix this before I send out the
official patch set.

> > + * to other KUnit tests.
> > + */
> > +enum kunit_speed {
> > +       KUNIT_SPEED_UNSET,
> > +       KUNIT_SPEED_VERY_SLOW,
> > +       KUNIT_SPEED_SLOW,
> > +       KUNIT_SPEED_NORMAL,
> > +       KUNIT_SPEED_MAX = KUNIT_SPEED_NORMAL,
> > +};
>
> A part of me feels that we could get away with reversing the order of
> this and having KUNIT_SPEED_NORMAL == 0. (Possibly reversing the
> comparisons in the filtering, too.)
>
> That's probably not worth the added complexity though. Either way,
> maybe add a note that "UNSET" == "NORMAL".
>
>

Yes, I definitely have been debating this order myself as well. This
order makes slightly more sense to me so I will likely keep this for
the next version but I will add that note. If there are more comments
on this, I am definitely open to reversing the order.

> > +
> >  /* Holds attributes for each test case and suite */
> > -struct kunit_attributes {};
> > +struct kunit_attributes {
> > +       enum kunit_speed speed;
> > +};
> >
> >  /**
> >   * struct kunit_case - represents an individual test case.
> > @@ -150,6 +167,17 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> >                 { .run_case = test_name, .name = #test_name,    \
> >                   .attr = attributes }
> >
> > +/**
> > + * KUNIT_CASE_SLOW - A helper for creating a &struct kunit_case
> > + * with the slow attribute
> > + *
> > + * @test_name: a reference to a test case function.
> > + */
> > +
> > +#define KUNIT_CASE_SLOW(test_name)                     \
> > +               { .run_case = test_name, .name = #test_name,    \
> > +                 .attr.speed = KUNIT_SPEED_SLOW }
> > +
> >  /**
> >   * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
> >   *
> > diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> > index 9bda5a5f4030..e97096dbb3b1 100644
> > --- a/lib/kunit/attributes.c
> > +++ b/lib/kunit/attributes.c
> > @@ -40,9 +40,53 @@ struct kunit_attr {
> >         enum print_ops print;
> >  };
> >
> > +/* String Lists for enum Attributes */
> > +
> > +static const char * const speed_str_list[] = {"unset", "very_slow", "slow", "normal"};
> > +
> > +/* To String Methods */
> > +
> > +static const char *attr_enum_to_string(void *attr, const char * const str_list[], bool *to_free)
> > +{
> > +       long val = (long)attr;
> > +
> > +       *to_free = false;
> > +       if (!val)
> > +               return NULL;
> > +       return str_list[val];
> > +}
> > +
> > +static const char *attr_speed_to_string(void *attr, bool *to_free)
> > +{
> > +       return attr_enum_to_string(attr, speed_str_list, to_free);
> > +}
> > +
> > +/* Get Attribute Methods */
> > +
> > +static void *attr_speed_get(void *test_or_suite, bool is_test)
> > +{
> > +       struct kunit_suite *suite = is_test ? NULL : test_or_suite;
> > +       struct kunit_case *test = is_test ? test_or_suite : NULL;
> > +
> > +       if (test)
> > +               return ((void *) test->attr.speed);
> > +       else
> > +               return ((void *) suite->attr.speed);
> > +}
> > +
> > +/* Attribute Struct Definitions */
> > +
> > +static const struct kunit_attr speed_attr = {
> > +       .name = "speed",
> > +       .get_attr = attr_speed_get,
> > +       .to_string = attr_speed_to_string,
> > +       .attr_default = (void *)KUNIT_SPEED_NORMAL,
> > +       .print = PRINT_ALWAYS,
> > +};
> > +
> >  /* List of all Test Attributes */
> >
> > -static struct kunit_attr kunit_attr_list[] = {};
> > +static struct kunit_attr kunit_attr_list[] = {speed_attr};
> >
> >  /* Helper Functions to Access Attributes */
> >
> > diff --git a/lib/kunit/kunit-example-test.c b/lib/kunit/kunit-example-test.c
> > index b69b689ea850..01a769f35e1d 100644
> > --- a/lib/kunit/kunit-example-test.c
> > +++ b/lib/kunit/kunit-example-test.c
> > @@ -220,6 +220,14 @@ static void example_params_test(struct kunit *test)
> >         KUNIT_EXPECT_EQ(test, param->value % param->value, 0);
> >  }
> >
> > +/*
> > + * This test should always pass. Can be used to practice filtering attributes.
> > + */
> > +static void example_slow_test(struct kunit *test)
> > +{
> > +       KUNIT_EXPECT_EQ(test, 1 + 1, 2);
> > +}
> > +
> >  /*
> >   * Here we make a list of all the test cases we want to add to the test suite
> >   * below.
> > @@ -237,6 +245,7 @@ static struct kunit_case example_test_cases[] = {
> >         KUNIT_CASE(example_all_expect_macros_test),
> >         KUNIT_CASE(example_static_stub_test),
> >         KUNIT_CASE_PARAM(example_params_test, example_gen_params),
> > +       KUNIT_CASE_SLOW(example_slow_test),
> >         {}
> >  };
> >
> > --
> > 2.41.0.255.g8b1d071c50-goog
> >

Re: [RFC v2 4/9] kunit: Add ability to filter attributes

[Rae Moar]

On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
>
> On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
> >
> > Add filtering of test attributes. Users can filter tests using the
> > module_param called "filter".
> >
> > Filters are imputed in the format: <attribute_name><operation><value>
> >
> > Example: kunit.filter="speed>slow"
> >
> > Operations include: >, <, >=, <=, !=, and =. These operations will act the
> > same for attributes of the same type but may not between types.
> >
> > Note multiple filters can be inputted by separating them with a comma.
> > Example: kunit.filter="speed=slow, module!=example"
> >
> > Since both suites and test cases can have attributes, there may be
> > conflicts. The process of filtering follows these rules:
> > - Filtering always operates at a per-test level.
> > - If a test has an attribute set, then the test's value is filtered on.
> > - Otherwise, the value falls back to the suite's value.
> > - If neither are set, the attribute has a global "default" value, which
> >   is used.
> >
> > Filtered tests will not be run or show in output. The tests can instead be
> > skipped using the configurable option "kunit.filter_action=skip".
> >
> > Note the default settings for running tests remains unfiltered.
> >
> > Finally, add "filter" methods for the speed and module attributes to parse
> > and compare attribute values.
> >
> > Note this filtering functionality will be added to kunit.py in the next
> > patch.
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
> > ---
> >
> > Changes since v1:
> > - Change method for inputting filters to allow for spaces in filtering
> >   values
> > - Add option to skip filtered tests instead of not run or show them with
> >   the --filter_skip flag
> >
> >  include/kunit/attributes.h |  31 +++++
> >  lib/kunit/attributes.c     | 256 +++++++++++++++++++++++++++++++++++++
> >  lib/kunit/executor.c       |  94 +++++++++++---
> >  lib/kunit/executor_test.c  |  12 +-
> >  lib/kunit/test.c           |  10 +-
> >  5 files changed, 375 insertions(+), 28 deletions(-)
> >
> > diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
> > index 9fcd184cce36..bc76a0b786d2 100644
> > --- a/include/kunit/attributes.h
> > +++ b/include/kunit/attributes.h
> > @@ -9,6 +9,20 @@
> >  #ifndef _KUNIT_ATTRIBUTES_H
> >  #define _KUNIT_ATTRIBUTES_H
> >
> > +/*
> > + * struct kunit_attr_filter - representation of attributes filter with the
> > + * attribute object and string input
> > + */
> > +struct kunit_attr_filter {
> > +       struct kunit_attr *attr;
> > +       char *input;
> > +};
> > +
> > +/*
> > + * Returns the name of the filter's attribute.
> > + */
> > +const char *kunit_attr_filter_name(struct kunit_attr_filter filter);
> > +
> >  /*
> >   * Print all test attributes for a test case or suite.
> >   * Output format for test cases: "# <test_name>.<attribute>: <value>"
> > @@ -16,4 +30,21 @@
> >   */
> >  void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
> >
> > +/*
> > + * Returns the number of fitlers in input.
> > + */
> > +int kunit_get_filter_count(char *input);
> > +
> > +/*
> > + * Parse attributes filter input and return an objects containing the
> > + * attribute object and the string input of the next filter.
> > + */
> > +struct kunit_attr_filter kunit_next_attr_filter(char **filters, int *err);
> > +
> > +/*
> > + * Returns a copy of the suite containing only tests that pass the filter.
> > + */
> > +struct kunit_suite *kunit_filter_attr_tests(const struct kunit_suite *const suite,
> > +               struct kunit_attr_filter filter, char *action, int *err);
> > +
> >  #endif /* _KUNIT_ATTRIBUTES_H */
> > diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> > index 43dcb5de8b8f..91cbcacafba9 100644
> > --- a/lib/kunit/attributes.c
> > +++ b/lib/kunit/attributes.c
> > @@ -67,6 +67,104 @@ static const char *attr_string_to_string(void *attr, bool *to_free)
> >         return (char *) attr;
> >  }
> >
> > +/* Filter Methods */
> > +
> > +static const char op_list[] = "<>!=";
> > +
> > +/*
> > + * Returns whether the inputted integer value matches the filter given
> > + * by the operation string and inputted integer.
> > + */
> > +static int int_filter(long val, const char *op, int input, int *err)
> > +{
> > +       if (!strncmp(op, "<=", 2))
> > +               return (val <= input);
> > +       else if (!strncmp(op, ">=", 2))
> > +               return (val >= input);
> > +       else if (!strncmp(op, "!=", 2))
> > +               return (val != input);
> > +       else if (!strncmp(op, ">", 1))
> > +               return (val > input);
> > +       else if (!strncmp(op, "<", 1))
> > +               return (val < input);
> > +       else if (!strncmp(op, "=", 1))
> > +               return (val == input);
> > +       *err = -EINVAL;
> > +       pr_err("kunit executor: invalid filter operation: %s\n", op);
>
> More a nitpick for the kunit.py patch, but I'd love to have this shown
> to the user as an error when run under kunit.py. It's very annoying to
> miss this and only get a "no tests run" error (or worse, unfiltered
> results) back.
>

I agree. I would love to change this. I am currently looking into how
easy this is to change.

It would be nice to at least see a general KUnit executor error in the
output instead of the "no tests run!" error.

>
> > +       return false;
> > +}

Re: [RFC v2 5/9] kunit: tool: Add command line interface to filter and report attributes

[Rae Moar]

On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
>
> On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
> >
> > Add ability to kunit.py to filter attributes and report a list of tests
> > including attributes without running tests.
> >
> > Add flag "--filter" to input filters on test attributes. Tests will be
> > filtered out if they do not match all inputted filters.
> >
> > Example: --filter speed=slow (This filter would run only the tests that are
> > marked as slow)
> >
> > Filters have operations: <, >, <=, >=, !=, and =. But note that the
> > characters < and > are often interpreted by the shell, so they may need to
> > be quoted or escaped.
> >
> > Example: --filter "speed>slow" or --filter speed\>slow (This filter would
> > run only the tests that have the speed faster than slow.
> >
> > Additionally, multiple filters can be used.
> >
> > Example: --filter "speed=slow, module!=example" (This filter would run
> > only the tests that have the speed slow and are not in the "example"
> > module)
> >
> > Note if the user wants to skip filtered tests instead of not
> > running/showing them use the "--filter_skip" flag instead.
> >
> > Expose the output of kunit.action=list option with flag "--list_tests" to
> > output a list of tests. Additionally, add flag "--list_tests_attr" to
> > output a list of tests and their attributes. These flags are useful to see
> > tests and test attributes without needing to run tests.
> >
> > Example of the output of "--list_tests_attr":
> >   example
> >   example.test_1
> >   example.test_2
> >   # example.test_2.speed: slow
> >
> > This output includes a suite, example, with two test cases, test_1 and
> > test_2. And in this instance test_2 has been marked as slow.
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
> > ---
> >
> > Changes since v1:
> > - Change method for inputting filters to allow for spaces in filtering
> >   values
> > - Add option to skip filtered tests instead of not run or show them with
> >   the --filter_skip flag
> > - Separate the new feature to list tests and their attributes into both
> >   --list_tests (lists just tests) and --list_tests_attr (lists all)
> >
> >  tools/testing/kunit/kunit.py           | 80 ++++++++++++++++++++++++--
> >  tools/testing/kunit/kunit_kernel.py    |  6 +-
> >  tools/testing/kunit/kunit_tool_test.py | 39 ++++++-------
> >  3 files changed, 96 insertions(+), 29 deletions(-)
> >
> > diff --git a/tools/testing/kunit/kunit.py b/tools/testing/kunit/kunit.py
> > index 3905c43369c3..6104e622ce20 100755
> > --- a/tools/testing/kunit/kunit.py
> > +++ b/tools/testing/kunit/kunit.py
> > @@ -55,8 +55,12 @@ class KunitExecRequest(KunitParseRequest):
> >         build_dir: str
> >         timeout: int
> >         filter_glob: str
> > +       filter: str
> > +       filter_skip: str
> >         kernel_args: Optional[List[str]]
> >         run_isolated: Optional[str]
> > +       list_tests: bool
> > +       list_tests_attr: bool
> >
> >  @dataclass
> >  class KunitRequest(KunitExecRequest, KunitBuildRequest):
> > @@ -102,19 +106,39 @@ def config_and_build_tests(linux: kunit_kernel.LinuxSourceTree,
> >
> >  def _list_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> List[str]:
> >         args = ['kunit.action=list']
> > +
> > +       if request.kernel_args:
> > +               args.extend(request.kernel_args)
> > +
> > +       output = linux.run_kernel(args=args,
> > +                          timeout=request.timeout,
> > +                          filter_glob=request.filter_glob,
> > +                          filter=request.filter,
> > +                          build_dir=request.build_dir)
> > +       lines = kunit_parser.extract_tap_lines(output)
> > +       # Hack! Drop the dummy TAP version header that the executor prints out.
> > +       lines.pop()
> > +
> > +       # Filter out any extraneous non-test output that might have gotten mixed in.
> > +       return [l for l in output if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
> > +
> > +def _list_tests_attr(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> Iterable[str]:
> > +       args = ['kunit.action=list_attr']
> > +
> >         if request.kernel_args:
> >                 args.extend(request.kernel_args)
> >
> >         output = linux.run_kernel(args=args,
> >                            timeout=request.timeout,
> >                            filter_glob=request.filter_glob,
> > +                          filter=request.filter,
> >                            build_dir=request.build_dir)
> >         lines = kunit_parser.extract_tap_lines(output)
> >         # Hack! Drop the dummy TAP version header that the executor prints out.
> >         lines.pop()
> >
> >         # Filter out any extraneous non-test output that might have gotten mixed in.
> > -       return [l for l in lines if re.match(r'^[^\s.]+\.[^\s.]+$', l)]
> > +       return lines
> >
> >  def _suites_from_test_list(tests: List[str]) -> List[str]:
> >         """Extracts all the suites from an ordered list of tests."""
> > @@ -128,10 +152,18 @@ def _suites_from_test_list(tests: List[str]) -> List[str]:
> >                         suites.append(suite)
> >         return suites
> >
> > -
> > -
> >  def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -> KunitResult:
> >         filter_globs = [request.filter_glob]
> > +       if request.list_tests:
> > +               output = _list_tests(linux, request)
> > +               for line in output:
> > +                       print(line.rstrip())
> > +               return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0)
> > +       if request.list_tests_attr:
> > +               attr_output = _list_tests_attr(linux, request)
> > +               for line in attr_output:
> > +                       print(line.rstrip())
> > +               return KunitResult(status=KunitStatus.SUCCESS, elapsed_time=0.0)
> >         if request.run_isolated:
> >                 tests = _list_tests(linux, request)
> >                 if request.run_isolated == 'test':
> > @@ -145,6 +177,17 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
> >
> >         metadata = kunit_json.Metadata(arch=linux.arch(), build_dir=request.build_dir, def_config='kunit_defconfig')
> >
> > +       filter = request.filter
> > +       if request.filter_skip:
> > +               args = ['kunit.filter_action=skip']
> > +               filter = request.filter_skip
> > +               if request.kernel_args:
> > +                       args.extend(request.kernel_args)
>
> What happens if both filter and filter_skip are set? We should
> probably either make those mutually exclusive (error if both are set),
> or expose filter_action directly instead.

That's a great idea. I wouldn't mind just using filter_action instead
of filter_skip. I will change this for the next version.

>
> > +       elif request.kernel_args:
> > +               args = request.kernel_args
> > +       else:
> > +               args = None
> > +
> >         test_counts = kunit_parser.TestCounts()
> >         exec_time = 0.0
> >         for i, filter_glob in enumerate(filter_globs):
> > @@ -152,9 +195,10 @@ def exec_tests(linux: kunit_kernel.LinuxSourceTree, request: KunitExecRequest) -
> >
> >                 test_start = time.time()
> >                 run_result = linux.run_kernel(
> > -                       args=request.kernel_args,
> > +                       args=args,
> >                         timeout=request.timeout,
> >                         filter_glob=filter_glob,
> > +                       filter=filter,
> >                         build_dir=request.build_dir)
> >
> >                 _, test_result = parse_tests(request, metadata, run_result)
> > @@ -341,6 +385,18 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None:
> >                             nargs='?',
> >                             default='',
> >                             metavar='filter_glob')
> > +       parser.add_argument('--filter',
> > +                           help='Filter KUnit tests with attributes, '
> > +                               'filtered tests will not run, '
> > +                           'e.g. speed=fast or speed=>low',
>
> Neither fast or low are valid values for speed now.

Sorry about this. I have changed this for the next version.

>
> > +                           type=str,
> > +                               default='')
> > +       parser.add_argument('--filter_skip',
> > +                           help='Filter KUnit tests run with attributes, '
> > +                               'filtered tests will be skipped, '
> > +                           'e.g. speed=fast or speed=>low',
>
> Neither fast or low are valid values for speed now.
>
>

Oops. I have changed this for the next version.

>
> > +                           type=str,
> > +                               default='')
> >         parser.add_argument('--kernel_args',
> >                             help='Kernel command-line parameters. Maybe be repeated',
> >                              action='append', metavar='')
> > @@ -350,6 +406,10 @@ def add_exec_opts(parser: argparse.ArgumentParser) -> None:
> >                             'what ran before it.',
> >                             type=str,
> >                             choices=['suite', 'test'])
> > +       parser.add_argument('--list_tests', help='If set, list all tests',
> > +                           action='store_true')
> > +       parser.add_argument('--list_tests_attr', help='If set, list all tests and attributes.',
> > +                           action='store_true')
> >
> >  def add_parse_opts(parser: argparse.ArgumentParser) -> None:
> >         parser.add_argument('--raw_output', help='If set don\'t parse output from kernel. '
> > @@ -398,8 +458,12 @@ def run_handler(cli_args: argparse.Namespace) -> None:
> >                                         json=cli_args.json,
> >                                         timeout=cli_args.timeout,
> >                                         filter_glob=cli_args.filter_glob,
> > +                                       filter=cli_args.filter,
> > +                                       filter_skip=cli_args.filter_skip,
> >                                         kernel_args=cli_args.kernel_args,
> > -                                       run_isolated=cli_args.run_isolated)
> > +                                       run_isolated=cli_args.run_isolated,
> > +                                       list_tests=cli_args.list_tests,
> > +                                       list_tests_attr=cli_args.list_tests_attr)
> >         result = run_tests(linux, request)
> >         if result.status != KunitStatus.SUCCESS:
> >                 sys.exit(1)
> > @@ -441,8 +505,12 @@ def exec_handler(cli_args: argparse.Namespace) -> None:
> >                                         json=cli_args.json,
> >                                         timeout=cli_args.timeout,
> >                                         filter_glob=cli_args.filter_glob,
> > +                                       filter=cli_args.filter,
> > +                                       filter_skip=cli_args.filter_skip,
> >                                         kernel_args=cli_args.kernel_args,
> > -                                       run_isolated=cli_args.run_isolated)
> > +                                       run_isolated=cli_args.run_isolated,
> > +                                       list_tests=cli_args.list_tests,
> > +                                       list_tests_attr=cli_args.list_tests_attr)
> >         result = exec_tests(linux, exec_request)
> >         stdout.print_with_timestamp((
> >                 'Elapsed time: %.3fs\n') % (result.elapsed_time))
> > diff --git a/tools/testing/kunit/kunit_kernel.py b/tools/testing/kunit/kunit_kernel.py
> > index 7f648802caf6..281f062a4767 100644
> > --- a/tools/testing/kunit/kunit_kernel.py
> > +++ b/tools/testing/kunit/kunit_kernel.py
> > @@ -330,11 +330,13 @@ class LinuxSourceTree:
> >                         return False
> >                 return self.validate_config(build_dir)
> >
> > -       def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', timeout: Optional[int]=None) -> Iterator[str]:
> > +       def run_kernel(self, args: Optional[List[str]]=None, build_dir: str='', filter_glob: str='', filter: str='', timeout: Optional[int]=None) -> Iterator[str]:
> >                 if not args:
> >                         args = []
> >                 if filter_glob:
> > -                       args.append('kunit.filter_glob='+filter_glob)
> > +                       args.append('kunit.filter_glob=' + filter_glob)
> > +               if filter:
> > +                       args.append('kunit.filter="' + filter + '"')
> >                 args.append('kunit.enable=1')
> >
> >                 process = self._ops.start(args, build_dir)
> > diff --git a/tools/testing/kunit/kunit_tool_test.py b/tools/testing/kunit/kunit_tool_test.py
> > index be35999bb84f..85a1fb72735e 100755
> > --- a/tools/testing/kunit/kunit_tool_test.py
> > +++ b/tools/testing/kunit/kunit_tool_test.py
> > @@ -597,7 +597,7 @@ class KUnitMainTest(unittest.TestCase):
> >                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 0)
> >                 self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir='.kunit', filter_glob='', timeout=300)
> > +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_run_passes_args_pass(self):
> > @@ -605,7 +605,7 @@ class KUnitMainTest(unittest.TestCase):
> >                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
> >                 self.assertEqual(self.linux_source_mock.run_kernel.call_count, 1)
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir='.kunit', filter_glob='', timeout=300)
> > +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_exec_passes_args_fail(self):
> > @@ -629,7 +629,7 @@ class KUnitMainTest(unittest.TestCase):
> >                         kunit.main(['run'])
> >                 self.assertEqual(e.exception.code, 1)
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir='.kunit', filter_glob='', timeout=300)
> > +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=300)
> >                 self.print_mock.assert_any_call(StrContains(' 0 tests run!'))
> >
> >         def test_exec_raw_output(self):
> > @@ -670,13 +670,13 @@ class KUnitMainTest(unittest.TestCase):
> >                 self.linux_source_mock.run_kernel = mock.Mock(return_value=[])
> >                 kunit.main(['run', '--raw_output', 'filter_glob'])
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir='.kunit', filter_glob='filter_glob', timeout=300)
> > +                       args=None, build_dir='.kunit', filter_glob='filter_glob', filter='', timeout=300)
> >
> >         def test_exec_timeout(self):
> >                 timeout = 3453
> >                 kunit.main(['exec', '--timeout', str(timeout)])
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir='.kunit', filter_glob='', timeout=timeout)
> > +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=timeout)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_run_timeout(self):
> > @@ -684,7 +684,7 @@ class KUnitMainTest(unittest.TestCase):
> >                 kunit.main(['run', '--timeout', str(timeout)])
> >                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir='.kunit', filter_glob='', timeout=timeout)
> > +                       args=None, build_dir='.kunit', filter_glob='', filter='', timeout=timeout)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_run_builddir(self):
> > @@ -692,7 +692,7 @@ class KUnitMainTest(unittest.TestCase):
> >                 kunit.main(['run', '--build_dir=.kunit'])
> >                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir=build_dir, filter_glob='', timeout=300)
> > +                       args=None, build_dir=build_dir, filter_glob='', filter='', timeout=300)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_config_builddir(self):
> > @@ -710,7 +710,7 @@ class KUnitMainTest(unittest.TestCase):
> >                 build_dir = '.kunit'
> >                 kunit.main(['exec', '--build_dir', build_dir])
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=None, build_dir=build_dir, filter_glob='', timeout=300)
> > +                       args=None, build_dir=build_dir, filter_glob='', filter='', timeout=300)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_run_kunitconfig(self):
> > @@ -786,7 +786,7 @@ class KUnitMainTest(unittest.TestCase):
> >                 kunit.main(['run', '--kernel_args=a=1', '--kernel_args=b=2'])
> >                 self.assertEqual(self.linux_source_mock.build_reconfig.call_count, 1)
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                     args=['a=1','b=2'], build_dir='.kunit', filter_glob='', timeout=300)
> > +                     args=['a=1','b=2'], build_dir='.kunit', filter_glob='', filter='', timeout=300)
> >                 self.print_mock.assert_any_call(StrContains('Testing complete.'))
> >
> >         def test_list_tests(self):
> > @@ -794,13 +794,11 @@ class KUnitMainTest(unittest.TestCase):
> >                 self.linux_source_mock.run_kernel.return_value = ['TAP version 14', 'init: random output'] + want
> >
> >                 got = kunit._list_tests(self.linux_source_mock,
> > -                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'suite'))
> > -
> > +                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', '', None, 'suite', False, False))
> >                 self.assertEqual(got, want)
> >                 # Should respect the user's filter glob when listing tests.
> >                 self.linux_source_mock.run_kernel.assert_called_once_with(
> > -                       args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', timeout=300)
> > -
> > +                       args=['kunit.action=list'], build_dir='.kunit', filter_glob='suite*', filter='', timeout=300)
> >
> >         @mock.patch.object(kunit, '_list_tests')
> >         def test_run_isolated_by_suite(self, mock_tests):
> > @@ -809,10 +807,10 @@ class KUnitMainTest(unittest.TestCase):
> >
> >                 # Should respect the user's filter glob when listing tests.
> >                 mock_tests.assert_called_once_with(mock.ANY,
> > -                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', None, 'suite'))
> > +                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*.test*', '', '', None, 'suite', False, False))
> >                 self.linux_source_mock.run_kernel.assert_has_calls([
> > -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', timeout=300),
> > -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', timeout=300),
> > +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test*', filter='', timeout=300),
> > +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test*', filter='', timeout=300),
> >                 ])
> >
> >         @mock.patch.object(kunit, '_list_tests')
> > @@ -822,13 +820,12 @@ class KUnitMainTest(unittest.TestCase):
> >
> >                 # Should respect the user's filter glob when listing tests.
> >                 mock_tests.assert_called_once_with(mock.ANY,
> > -                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', None, 'test'))
> > +                                    kunit.KunitExecRequest(None, None, '.kunit', 300, 'suite*', '', '', None, 'test', False, False))
> >                 self.linux_source_mock.run_kernel.assert_has_calls([
> > -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', timeout=300),
> > -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', timeout=300),
> > -                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', timeout=300),
> > +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test1', filter='', timeout=300),
> > +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite.test2', filter='', timeout=300),
> > +                       mock.call(args=None, build_dir='.kunit', filter_glob='suite2.test1', filter='', timeout=300),
> >                 ])
> >
> > -
> >  if __name__ == '__main__':
> >         unittest.main()
> > --
> > 2.41.0.255.g8b1d071c50-goog
> >

Re: [RFC v2 9/9] kunit: Add documentation of KUnit test attributes

[Rae Moar]

On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
>
> On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
> >
> > Add documentation on the use of test attributes under the section "Tips for
> > Running KUnit Tests" in the KUnit docs.
> >
> > Documentation includes three sections on how to mark tests with attributes,
> > how attributes are reported, and how the user can filter tests using test
> > attributes.
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
> > ---
>
> Looks good overall. Some nitpicks below.
>
> Reviewed-by: David Gow <davidgow@google.com>
>
> >
> > Changes since v1:
> > - This is a new patch
> >
> >  .../dev-tools/kunit/running_tips.rst          | 163 ++++++++++++++++++
> >  1 file changed, 163 insertions(+)
> >
> > diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
> > index 8e8c493f17d1..c9bc5a6595d3 100644
> > --- a/Documentation/dev-tools/kunit/running_tips.rst
> > +++ b/Documentation/dev-tools/kunit/running_tips.rst
> > @@ -262,3 +262,166 @@ other code executed during boot, e.g.
> >         # Reset coverage counters before running the test.
> >         $ echo 0 > /sys/kernel/debug/gcov/reset
> >         $ modprobe kunit-example-test
> > +
> > +
> > +Test Attributes and Filtering
> > +=============================
> > +
> > +Test suites and cases can be marked with test attributes, such as speed of
> > +test. These attributes will later be printed in test output and can be used to
> > +filter test execution.
> > +
> > +Marking Test Attributes
> > +-----------------------
> > +
> > +Tests are marked with an attribute by including a ``kunit_attributes`` object
> > +in the test definition.
> > +
> > +Test cases can be marked using the ``KUNIT_CASE_ATTR(test_name, attributes)``
> > +macro to define the test case instead of ``KUNIT_CASE(test_name)``.
> > +
> > +.. code-block:: c
> > +
> > +       static const struct kunit_attributes example_attr = {
> > +               .speed = KUNIT_VERY_SLOW,
> > +       };
> > +
> > +       static struct kunit_case example_test_cases[] = {
> > +               KUNIT_CASE_ATTR(example_test, example_attr),
> > +       };
> > +
> > +.. note::
> > +       To mark a test case as slow, you can also use ``KUNIT_CASE_SLOW(test_name)``.
> > +       This is a helpful macro as the slow attribute is the most commonly used.
> > +
> > +Test suites can be marked with an attribute by setting the "attr" field in the
> > +suite definition.
> > +
> > +.. code-block:: c
> > +
> > +       static const struct kunit_attributes example_attr = {
> > +               .speed = KUNIT_VERY_SLOW,
> > +       };
> > +
> > +       static struct kunit_suite example_test_suite = {
> > +               ...,
> > +               .attr = example_attr,
> > +       };
> > +
> > +.. note::
> > +       Not all attributes need to be set in a ``kunit_attributes`` object. Unset
> > +       attributes will remain uninitialized and act as though the attribute is set
> > +       to 0 or NULL. Thus, if an attribute is set to 0, it is treated as unset.
> > +       These unset attributes will not be reported and may act as a default value
> > +       for filtering purposes.
> > +
> > +Reporting Attributes
> > +--------------------
> > +
> > +When a user runs tests, attributes will be present in kernel output (in KTAP
> > +format). This is an example of how test attributes for test cases will be formatted
> > +in Kernel output:
> > +
> > +.. code-block:: none
> > +
> > +       # example_test.speed: slow
> > +       ok 1 example_test
> > +
> > +This is an example of how test attributes for test suites will be formatted in
> > +Kernel output:
> > +
> > +.. code-block:: none
> > +
> > +         KTAP version 2
> > +         # Subtest: example_suite
> > +         # module: kunit_example_test
> > +         1..3
> > +         ...
> > +       ok 1 example_suite
> > +
>
> Maybe worth noting that kunit.py will hide these for passing tests by
> default, and --raw_output is needed to see them?
>

I will definitely add this in. If attributes are popular in the
future, I could create a future patch to show attributes in the parser
output as well.

> > +Additionally, users can output a full attribute report of tests with their
> > +attributes, using the command line flag ``--list_tests_attr``:
> > +
> > +.. code-block:: bash
> > +
> > +       kunit.py run "example" --list_tests_attr
> > +
> > +.. note::
> > +       This report can be accessed when running KUnit manually by passing in the
> > +       module_param ``kunit.action=list_attr``.
> > +
> > +Filtering
> > +---------
> > +
> > +Users can filter tests using the ``--filter`` command line flag when running
> > +tests. As an example:
> > +
> > +.. code-block:: bash
> > +
> > +       kunit.py run --filter speed=slow
> > +
> > +
> > +You can also use the following operations on filters: "<", ">", "<=", ">=",
> > +"!=", and "=". Example:
> > +
> > +.. code-block:: bash
> > +
> > +       kunit.py run --filter "speed>slow"
> > +
> > +This example will run all tests with speeds faster than slow. Note that the
> > +characters < and > are often interpreted by the shell, so they may need to be
> > +quoted or escaped, as above.
> > +
> > +Additionally, you can use multiple filters at once. Simply separate filters
> > +using commas. Example:
> > +
> > +.. code-block:: bash
> > +
> > +       kunit.py run --filter "speed>slow, module=kunit_example_test"
> > +
> > +.. note::
> > +       You can use this filtering feature when running KUnit manually by passing
> > +       the filter as a module param: ``kunit.filter="speed>slow, speed<=normal"``.
> > +
> > +Filtered tests will not run or show up in the test output. You can use the
> > +``--filter_skip`` flag to skip filtered tests instead. These tests will be
> > +shown in the test output in the test but will not run. To use this feature when
> > +running KUnit manually, use the ``kunit.filter`` module param with
> > +``kunit.filter_action=skip``.
> > +
> > +Rules of Filtering Procedure
> > +----------------------------
> > +
> > +Since both suites and test cases can have attributes, there may be conflicts
> > +between attributes during filtering. The process of filtering follows these
> > +rules:
> > +
> > +- Filtering always operates at a per-test level.
> > +
> > +- If a test has an attribute set, then the test's value is filtered on.
> > +
> > +- Otherwise, the value falls back to the suite's value.
> > +
> > +- If neither are set, the attribute has a global "default" value, which is used.
> > +
> > +List of Current Attributes
> > +--------------------------
>
> I wonder whether this should end up part of the KTAP spec (or as an
> appendix/supplement to it). Or even as a separate page within the
> KUnit documentation to avoid running_tips.rst from getting too huge.

I am a bit hesitant to move this as part of the KTAP spec in case
there will exist KTAP attributes/data that are not supported by the
KUnit test attributes framework (could be runtime specific attributes
that use a different framework?).

However, I do worry about the size of this page. Do you think that I
should move all of the attributes to a new documentation page?

>
> > +
> > +``speed``
> > +
> > +This attribute indicates the speed of a test's execution (how slow or fast the
> > +test is).
> > +
> > +This attribute is saved as an enum with the following categories: "normal",
> > +"slow", or "very_slow". The assumed default speed for tests is "normal". This
> > +indicates that the test takes a relatively trivial amount of time (less than
> > +1 second), regardless of the machine it is running on. Any test slower than
> > +this could be marked as "slow" or "very_slow".
>
> Is it worth noting that "KUNIT_CASE_SLOW()" can be used to easily set
> this to slow?

This definitely seems important to add. I will add this to the documentation.

>
>
> > +
> > +``module``
> > +
> > +This attribute indicates the name of the module associated with the test.
> > +
> > +This attribute is automatically saved as a string and is printed for each suite.
> > +Tests can also be filtered using this attribute.
> > +
> > --
> > 2.41.0.255.g8b1d071c50-goog
>
> >
>
> Error: new blank line at EOF.

Oops. I will change this.

Re: [RFC v2 1/9] kunit: Add test attributes API structure

[Rae Moar]

On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
>
> On Sat, 8 Jul 2023 at 05:09, Rae Moar <rmoar@google.com> wrote:
> >
> > Add the basic structure of the test attribute API to KUnit, which can be
> > used to save and access test associated data.
> >
> > Add attributes.c and attributes.h to hold associated structs and functions
> > for the API.
> >
> > Create a struct that holds a variety of associated helper functions for
> > each test attribute. These helper functions will be used to get the
> > attribute value, convert the value to a string, and filter based on the
> > value. This struct is flexible by design to allow for attributes of
> > numerous types and contexts.
> >
> > Add a method to print test attributes in the format of "# [<test_name if
> > not suite>.]<attribute_name>: <attribute_value>".
> >
> > Example for a suite: "# speed: slow"
> >
> > Example for a test case: "# test_case.speed: very_slow"
>
> So, this is the one thing I'm a little unsure about here, and it's
> really more of a problem with test names overall.
>
> As noted in the KTAPv2 attributes and test name proposals, the names
> and attributes are only really defined for "suites", hence the need to
> have a different output format for test cases.
>
> Personally, I'd prefer to keep the formats the same if we can (at
> least for the actual KTAP output; I'm less concerned with the
> list_attr option). That might make things a bit more difficult to
> parse, though.
>
> One possibility would be to combine the KTAP attributes and test name
> specs and suggest that every test has a "test name" attribute, which
> must be the first attribute output.
>
> The output would then look something like:
> KTAP version 2
> # Name: my_suite
> # Other-Attr: value
> 1..2
>   KTAP version 2
>   # Name: test_1
>   # Other-Attr: value
>   ok 1 test_1
>   # Name: test_2
>   # Other-Attr: value
>   not ok 2 test_2
> ok 1 my_suite
>
> Would there be any problems with something like that?
>
> I'm less concerned with the list_attr option, as that's not something
> totally standardised in the way KTAP is.
>

This is a really interesting idea. I like that this standardizes the
concept of KTAP test metadata for both test suites and test cases. I
would love to discuss this concept further as KTAP v2 is developed.

My main concern would be that there is push back on stating the test
name when it is already present in the result line. This adds
potentially unnecessary lines to the output. However, one positive to
this is that diagnostic data could be printed under this header which
would reduce any confusion for which test the diagnostic data refers
to.

I would be interested if anyone else has any opinions on this.

> >
> > Use this method to report attributes in the KTAP output (KTAP spec:
> > https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when
> > kernel's new kunit.action=list_attr option is used. Note this is derivative
> > of the kunit.action=list option.
> >
> > In test.h, add fields and associated helper functions to test cases and
> > suites to hold user-inputted test attributes.
> >
> > Signed-off-by: Rae Moar <rmoar@google.com>
> > ---
>
> The only other thing I'd really like to support one day is having
> attributes for individual parameters in parameterised tests. I think
> it makes sense as a follow-up, though.
>

That is an exciting idea! I think that would be ideal as a follow-up.

> >
> > Changes since v1:
> > - Add list_attr option to only include attribute in the _list_tests output
> >   when this module param is set
> > - Add printing options for attributes to print always, print only for
> >   suites, or print never.
> >
> >  include/kunit/attributes.h | 19 +++++++++
> >  include/kunit/test.h       | 33 ++++++++++++++++
> >  lib/kunit/Makefile         |  3 +-
> >  lib/kunit/attributes.c     | 80 ++++++++++++++++++++++++++++++++++++++
> >  lib/kunit/executor.c       | 21 ++++++++--
> >  lib/kunit/test.c           | 17 ++++----
> >  6 files changed, 161 insertions(+), 12 deletions(-)
> >  create mode 100644 include/kunit/attributes.h
> >  create mode 100644 lib/kunit/attributes.c
> >
> > diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
> > new file mode 100644
> > index 000000000000..9fcd184cce36
> > --- /dev/null
> > +++ b/include/kunit/attributes.h
> > @@ -0,0 +1,19 @@
> > +/* SPDX-License-Identifier: GPL-2.0 */
> > +/*
> > + * KUnit API to save and access test attributes
> > + *
> > + * Copyright (C) 2023, Google LLC.
> > + * Author: Rae Moar <rmoar@google.com>
> > + */
> > +
> > +#ifndef _KUNIT_ATTRIBUTES_H
> > +#define _KUNIT_ATTRIBUTES_H
> > +
> > +/*
> > + * Print all test attributes for a test case or suite.
> > + * Output format for test cases: "# <test_name>.<attribute>: <value>"
> > + * Output format for test suites: "# <attribute>: <value>"
> > + */
> > +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
> > +
> > +#endif /* _KUNIT_ATTRIBUTES_H */
> > diff --git a/include/kunit/test.h b/include/kunit/test.h
> > index 23120d50499e..1fc9155988e9 100644
> > --- a/include/kunit/test.h
> > +++ b/include/kunit/test.h
> > @@ -63,12 +63,16 @@ enum kunit_status {
> >         KUNIT_SKIPPED,
> >  };
> >
> > +/* Holds attributes for each test case and suite */
> > +struct kunit_attributes {};
>
> Do we want a separate set of attributes for test cases and suites?
> (I think probably not, but it's worth making sure.)
>

I'm thinking if our goal is to eventually move to arbitrary nesting
for tests it would be easiest to try to keep this list the same. But I
agree. There may definitely be attributes that are more applicable for
test cases or suites. I'm inclined to keep it this way.

> > +
> >  /**
> >   * struct kunit_case - represents an individual test case.
> >   *
> >   * @run_case: the function representing the actual test case.
> >   * @name:     the name of the test case.
> >   * @generate_params: the generator function for parameterized tests.
> > + * @attr:     the attributes associated with the test
> >   *
> >   * A test case is a function with the signature,
> >   * ``void (*)(struct kunit *)``
> > @@ -104,6 +108,7 @@ struct kunit_case {
> >         void (*run_case)(struct kunit *test);
> >         const char *name;
> >         const void* (*generate_params)(const void *prev, char *desc);
> > +       struct kunit_attributes attr;
> >
> >         /* private: internal use only. */
> >         enum kunit_status status;
> > @@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> >   */
> >  #define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
> >
> > +/**
> > + * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
> > + * with attributes
> > + *
> > + * @test_name: a reference to a test case function.
> > + * @attributes: a reference to a struct kunit_attributes object containing
> > + * test attributes
> > + */
> > +#define KUNIT_CASE_ATTR(test_name, attributes)                 \
> > +               { .run_case = test_name, .name = #test_name,    \
> > +                 .attr = attributes }
> > +
> >  /**
> >   * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
> >   *
> > @@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> >                 { .run_case = test_name, .name = #test_name,    \
> >                   .generate_params = gen_params }
> >
> > +/**
> > + * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
> > + * kunit_case with attributes
> > + *
> > + * @test_name: a reference to a test case function.
> > + * @gen_params: a reference to a parameter generator function.
> > + * @attributes: a reference to a struct kunit_attributes object containing
> > + * test attributes
> > + */
> > +#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes)       \
> > +               { .run_case = test_name, .name = #test_name,    \
> > +                 .generate_params = gen_params,                                \
> > +                 .attr = attributes }
> > +
>
> I do worry a bit that we'll end up with a huge list of variants of the
> KUNIT_CASE_* macros if we start adding more things here. I can't think
> of a better way to handle it at the moment, though.
>
>

I agree. If this becomes an issue, this could be a follow up change?

>
> >  /**
> >   * struct kunit_suite - describes a related collection of &struct kunit_case
> >   *
> > @@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> >   * @init:      called before every test case.
> >   * @exit:      called after every test case.
> >   * @test_cases:        a null terminated array of test cases.
> > + * @attr:      the attributes associated with the test suite
> >   *
> >   * A kunit_suite is a collection of related &struct kunit_case s, such that
> >   * @init is called before every test case and @exit is called after every
> > @@ -182,6 +214,7 @@ struct kunit_suite {
> >         int (*init)(struct kunit *test);
> >         void (*exit)(struct kunit *test);
> >         struct kunit_case *test_cases;
> > +       struct kunit_attributes attr;
> >
> >         /* private: internal use only */
> >         char status_comment[KUNIT_STATUS_COMMENT_SIZE];
> > diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
> > index cb417f504996..46f75f23dfe4 100644
> > --- a/lib/kunit/Makefile
> > +++ b/lib/kunit/Makefile
> > @@ -6,7 +6,8 @@ kunit-objs +=                           test.o \
> >                                         string-stream.o \
> >                                         assert.o \
> >                                         try-catch.o \
> > -                                       executor.o
> > +                                       executor.o \
> > +                                       attributes.o
> >
> >  ifeq ($(CONFIG_KUNIT_DEBUGFS),y)
> >  kunit-objs +=                          debugfs.o
> > diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> > new file mode 100644
> > index 000000000000..9bda5a5f4030
> > --- /dev/null
> > +++ b/lib/kunit/attributes.c
> > @@ -0,0 +1,80 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * KUnit API to save and access test attributes
> > + *
> > + * Copyright (C) 2023, Google LLC.
> > + * Author: Rae Moar <rmoar@google.com>
> > + */
> > +
> > +#include <kunit/test.h>
> > +#include <kunit/attributes.h>
> > +
> > +/* Options for printing attributes:
> > + * PRINT_ALWAYS - attribute is printed for every test case and suite if set
> > + * PRINT_SUITE - attribute is printed for every suite if set but not for test cases
> > + * PRINT_NEVER - attribute is never printed
> > + */
> > +enum print_ops {
> > +       PRINT_ALWAYS,
> > +       PRINT_SUITE,
> > +       PRINT_NEVER,
> > +};
> > +
> > +/**
> > + * struct kunit_attr - represents a test attribute and holds flexible
> > + * helper functions to interact with attribute.
> > + *
> > + * @name: name of test attribute, eg. speed
> > + * @get_attr: function to return attribute value given a test
> > + * @to_string: function to return string representation of given
> > + * attribute value
> > + * @filter: function to indicate whether a given attribute value passes a
> > + * filter
> > + */
> > +struct kunit_attr {
> > +       const char *name;
> > +       void *(*get_attr)(void *test_or_suite, bool is_test);
> > +       const char *(*to_string)(void *attr, bool *to_free);
> > +       int (*filter)(void *attr, const char *input, int *err);
> > +       void *attr_default;
> > +       enum print_ops print;
> > +};
> > +
> > +/* List of all Test Attributes */
> > +
> > +static struct kunit_attr kunit_attr_list[] = {};
> > +
> > +/* Helper Functions to Access Attributes */
> > +
> > +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
> > +{
> > +       int i;
> > +       bool to_free;
> > +       void *attr;
> > +       const char *attr_name, *attr_str;
> > +       struct kunit_suite *suite = is_test ? NULL : test_or_suite;
> > +       struct kunit_case *test = is_test ? test_or_suite : NULL;
> > +
> > +       for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) {
> > +               if (kunit_attr_list[i].print == PRINT_NEVER ||
> > +                               (test && kunit_attr_list[i].print == PRINT_SUITE))
> > +                       continue;
> > +               attr = kunit_attr_list[i].get_attr(test_or_suite, is_test);
> > +               if (attr) {
> > +                       attr_name = kunit_attr_list[i].name;
> > +                       attr_str = kunit_attr_list[i].to_string(attr, &to_free);
> > +                       if (test) {
> > +                               kunit_log(KERN_INFO, test, "%*s# %s.%s: %s",
> > +                                       KUNIT_INDENT_LEN * test_level, "", test->name,
> > +                                       attr_name, attr_str);
> > +                       } else {
> > +                               kunit_log(KERN_INFO, suite, "%*s# %s: %s",
> > +                                       KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str);
> > +                       }
> > +
> > +                       /* Free to_string of attribute if needed */
> > +                       if (to_free)
> > +                               kfree(attr_str);
> > +               }
> > +       }
> > +}
> > diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> > index 74982b83707c..12e38a48a5cc 100644
> > --- a/lib/kunit/executor.c
> > +++ b/lib/kunit/executor.c
> > @@ -2,6 +2,7 @@
> >
> >  #include <linux/reboot.h>
> >  #include <kunit/test.h>
> > +#include <kunit/attributes.h>
> >  #include <linux/glob.h>
> >  #include <linux/moduleparam.h>
> >
> > @@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0);
> >  MODULE_PARM_DESC(action,
> >                  "Changes KUnit executor behavior, valid values are:\n"
> >                  "<none>: run the tests like normal\n"
> > -                "'list' to list test names instead of running them.\n");
> > +                "'list' to list test names instead of running them.\n"
> > +                "'list_attr' to list test names and attributes instead of running them.\n");
> >
> >  /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
> >  struct kunit_test_filter {
> > @@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set)
> >         __kunit_test_suites_init(suite_set->start, num_suites);
> >  }
> >
> > -static void kunit_exec_list_tests(struct suite_set *suite_set)
> > +static void kunit_exec_list_tests(struct suite_set *suite_set, bool include_attr)
> >  {
> >         struct kunit_suite * const *suites;
> >         struct kunit_case *test_case;
> > @@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *suite_set)
> >         /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */
> >         pr_info("KTAP version 1\n");
> >
> > -       for (suites = suite_set->start; suites < suite_set->end; suites++)
> > +       for (suites = suite_set->start; suites < suite_set->end; suites++) {
> > +               /* Print suite name and suite attributes */
> > +               pr_info("%s\n", (*suites)->name);
> > +               if (include_attr)
> > +                       kunit_print_attr((void *)(*suites), false, 0);
> > +
> > +               /* Print test case name and attributes in suite */
> >                 kunit_suite_for_each_test_case((*suites), test_case) {
> >                         pr_info("%s.%s\n", (*suites)->name, test_case->name);
> > +                       if (include_attr)
> > +                               kunit_print_attr((void *)test_case, true, 0);
> >                 }
> > +       }
> >  }
> >
> >  int kunit_run_all_tests(void)
> > @@ -206,7 +217,9 @@ int kunit_run_all_tests(void)
> >         if (!action_param)
> >                 kunit_exec_run_tests(&suite_set);
> >         else if (strcmp(action_param, "list") == 0)
> > -               kunit_exec_list_tests(&suite_set);
> > +               kunit_exec_list_tests(&suite_set, false);
> > +       else if (strcmp(action_param, "list_attr") == 0)
> > +               kunit_exec_list_tests(&suite_set, true);
> >         else
> >                 pr_err("kunit executor: unknown action '%s'\n", action_param);
> >
> > diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> > index 84e4666555c9..9ee55139ecd1 100644
> > --- a/lib/kunit/test.c
> > +++ b/lib/kunit/test.c
> > @@ -9,6 +9,7 @@
> >  #include <kunit/resource.h>
> >  #include <kunit/test.h>
> >  #include <kunit/test-bug.h>
> > +#include <kunit/attributes.h>
> >  #include <linux/kernel.h>
> >  #include <linux/module.h>
> >  #include <linux/moduleparam.h>
> > @@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite)
> >  }
> >  EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases);
> >
> > +/* Currently supported test levels */
> > +enum {
> > +       KUNIT_LEVEL_SUITE = 0,
> > +       KUNIT_LEVEL_CASE,
> > +       KUNIT_LEVEL_CASE_PARAM,
> > +};
> > +
> >  static void kunit_print_suite_start(struct kunit_suite *suite)
> >  {
> >         /*
> > @@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite)
> >         pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n");
> >         pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
> >                   suite->name);
> > +       kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE);
> >         pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n",
> >                   kunit_suite_num_test_cases(suite));
> >  }
> >
> > -/* Currently supported test levels */
> > -enum {
> > -       KUNIT_LEVEL_SUITE = 0,
> > -       KUNIT_LEVEL_CASE,
> > -       KUNIT_LEVEL_CASE_PARAM,
> > -};
> > -
> >  static void kunit_print_ok_not_ok(struct kunit *test,
> >                                   unsigned int test_level,
> >                                   enum kunit_status status,
> > @@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite)
> >                         }
> >                 }
> >
> > +               kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE);
> >
> >                 kunit_print_test_stats(&test, param_stats);
> >
> > --
> > 2.41.0.255.g8b1d071c50-goog
> >

Re: [RFC v2 9/9] kunit: Add documentation of KUnit test attributes

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 10026 bytes --]

On Wed, 19 Jul 2023 at 04:49, Rae Moar <rmoar@google.com> wrote:
>
> On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
> >
> > On Sat, 8 Jul 2023 at 05:10, Rae Moar <rmoar@google.com> wrote:
> > >
> > > Add documentation on the use of test attributes under the section "Tips for
> > > Running KUnit Tests" in the KUnit docs.
> > >
> > > Documentation includes three sections on how to mark tests with attributes,
> > > how attributes are reported, and how the user can filter tests using test
> > > attributes.
> > >
> > > Signed-off-by: Rae Moar <rmoar@google.com>
> > > ---
> >
> > Looks good overall. Some nitpicks below.
> >
> > Reviewed-by: David Gow <davidgow@google.com>
> >
> > >
> > > Changes since v1:
> > > - This is a new patch
> > >
> > >  .../dev-tools/kunit/running_tips.rst          | 163 ++++++++++++++++++
> > >  1 file changed, 163 insertions(+)
> > >
> > > diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst
> > > index 8e8c493f17d1..c9bc5a6595d3 100644
> > > --- a/Documentation/dev-tools/kunit/running_tips.rst
> > > +++ b/Documentation/dev-tools/kunit/running_tips.rst
> > > @@ -262,3 +262,166 @@ other code executed during boot, e.g.
> > >         # Reset coverage counters before running the test.
> > >         $ echo 0 > /sys/kernel/debug/gcov/reset
> > >         $ modprobe kunit-example-test
> > > +
> > > +
> > > +Test Attributes and Filtering
> > > +=============================
> > > +
> > > +Test suites and cases can be marked with test attributes, such as speed of
> > > +test. These attributes will later be printed in test output and can be used to
> > > +filter test execution.
> > > +
> > > +Marking Test Attributes
> > > +-----------------------
> > > +
> > > +Tests are marked with an attribute by including a ``kunit_attributes`` object
> > > +in the test definition.
> > > +
> > > +Test cases can be marked using the ``KUNIT_CASE_ATTR(test_name, attributes)``
> > > +macro to define the test case instead of ``KUNIT_CASE(test_name)``.
> > > +
> > > +.. code-block:: c
> > > +
> > > +       static const struct kunit_attributes example_attr = {
> > > +               .speed = KUNIT_VERY_SLOW,
> > > +       };
> > > +
> > > +       static struct kunit_case example_test_cases[] = {
> > > +               KUNIT_CASE_ATTR(example_test, example_attr),
> > > +       };
> > > +
> > > +.. note::
> > > +       To mark a test case as slow, you can also use ``KUNIT_CASE_SLOW(test_name)``.
> > > +       This is a helpful macro as the slow attribute is the most commonly used.
> > > +
> > > +Test suites can be marked with an attribute by setting the "attr" field in the
> > > +suite definition.
> > > +
> > > +.. code-block:: c
> > > +
> > > +       static const struct kunit_attributes example_attr = {
> > > +               .speed = KUNIT_VERY_SLOW,
> > > +       };
> > > +
> > > +       static struct kunit_suite example_test_suite = {
> > > +               ...,
> > > +               .attr = example_attr,
> > > +       };
> > > +
> > > +.. note::
> > > +       Not all attributes need to be set in a ``kunit_attributes`` object. Unset
> > > +       attributes will remain uninitialized and act as though the attribute is set
> > > +       to 0 or NULL. Thus, if an attribute is set to 0, it is treated as unset.
> > > +       These unset attributes will not be reported and may act as a default value
> > > +       for filtering purposes.
> > > +
> > > +Reporting Attributes
> > > +--------------------
> > > +
> > > +When a user runs tests, attributes will be present in kernel output (in KTAP
> > > +format). This is an example of how test attributes for test cases will be formatted
> > > +in Kernel output:
> > > +
> > > +.. code-block:: none
> > > +
> > > +       # example_test.speed: slow
> > > +       ok 1 example_test
> > > +
> > > +This is an example of how test attributes for test suites will be formatted in
> > > +Kernel output:
> > > +
> > > +.. code-block:: none
> > > +
> > > +         KTAP version 2
> > > +         # Subtest: example_suite
> > > +         # module: kunit_example_test
> > > +         1..3
> > > +         ...
> > > +       ok 1 example_suite
> > > +
> >
> > Maybe worth noting that kunit.py will hide these for passing tests by
> > default, and --raw_output is needed to see them?
> >
>
> I will definitely add this in. If attributes are popular in the
> future, I could create a future patch to show attributes in the parser
> output as well.

Yeah, that could definitely be useful as a follow-up patch.

> > > +Additionally, users can output a full attribute report of tests with their
> > > +attributes, using the command line flag ``--list_tests_attr``:
> > > +
> > > +.. code-block:: bash
> > > +
> > > +       kunit.py run "example" --list_tests_attr
> > > +
> > > +.. note::
> > > +       This report can be accessed when running KUnit manually by passing in the
> > > +       module_param ``kunit.action=list_attr``.
> > > +
> > > +Filtering
> > > +---------
> > > +
> > > +Users can filter tests using the ``--filter`` command line flag when running
> > > +tests. As an example:
> > > +
> > > +.. code-block:: bash
> > > +
> > > +       kunit.py run --filter speed=slow
> > > +
> > > +
> > > +You can also use the following operations on filters: "<", ">", "<=", ">=",
> > > +"!=", and "=". Example:
> > > +
> > > +.. code-block:: bash
> > > +
> > > +       kunit.py run --filter "speed>slow"
> > > +
> > > +This example will run all tests with speeds faster than slow. Note that the
> > > +characters < and > are often interpreted by the shell, so they may need to be
> > > +quoted or escaped, as above.
> > > +
> > > +Additionally, you can use multiple filters at once. Simply separate filters
> > > +using commas. Example:
> > > +
> > > +.. code-block:: bash
> > > +
> > > +       kunit.py run --filter "speed>slow, module=kunit_example_test"
> > > +
> > > +.. note::
> > > +       You can use this filtering feature when running KUnit manually by passing
> > > +       the filter as a module param: ``kunit.filter="speed>slow, speed<=normal"``.
> > > +
> > > +Filtered tests will not run or show up in the test output. You can use the
> > > +``--filter_skip`` flag to skip filtered tests instead. These tests will be
> > > +shown in the test output in the test but will not run. To use this feature when
> > > +running KUnit manually, use the ``kunit.filter`` module param with
> > > +``kunit.filter_action=skip``.
> > > +
> > > +Rules of Filtering Procedure
> > > +----------------------------
> > > +
> > > +Since both suites and test cases can have attributes, there may be conflicts
> > > +between attributes during filtering. The process of filtering follows these
> > > +rules:
> > > +
> > > +- Filtering always operates at a per-test level.
> > > +
> > > +- If a test has an attribute set, then the test's value is filtered on.
> > > +
> > > +- Otherwise, the value falls back to the suite's value.
> > > +
> > > +- If neither are set, the attribute has a global "default" value, which is used.
> > > +
> > > +List of Current Attributes
> > > +--------------------------
> >
> > I wonder whether this should end up part of the KTAP spec (or as an
> > appendix/supplement to it). Or even as a separate page within the
> > KUnit documentation to avoid running_tips.rst from getting too huge.
>
> I am a bit hesitant to move this as part of the KTAP spec in case
> there will exist KTAP attributes/data that are not supported by the
> KUnit test attributes framework (could be runtime specific attributes
> that use a different framework?).

This is probably something best worked out as part of the KTAP spec
process. Either attribute names are a free-for-all (albeit hopefully
one where there are some documented 'common' attributes), or we need
some sort of namespacing between "General KTAP attributes",
"KUnit-specific attributes", "subsystem-specific attributes",
"totally-made-up-on-the-spot attributes", etc.

e.g., email headers have a list of 'standard' ones, but anyone can add
their own as long as it starts with 'X-'. Or OpenGL extensions are
always of the form GL_blah_blah_blah_<vendor> (where vendor is the
code for the company that proposed it, or EXT or ARB for those which
have been agreed upon by everyone).

> However, I do worry about the size of this page. Do you think that I
> should move all of the attributes to a new documentation page?

While I don't think it's a problem with only two attributes, it'd
probably be the more futureproof thing to do.

That being said, maybe we wait until there's a decision on the KTAP
side? Up to you.



> >
> > > +
> > > +``speed``
> > > +
> > > +This attribute indicates the speed of a test's execution (how slow or fast the
> > > +test is).
> > > +
> > > +This attribute is saved as an enum with the following categories: "normal",
> > > +"slow", or "very_slow". The assumed default speed for tests is "normal". This
> > > +indicates that the test takes a relatively trivial amount of time (less than
> > > +1 second), regardless of the machine it is running on. Any test slower than
> > > +this could be marked as "slow" or "very_slow".
> >
> > Is it worth noting that "KUNIT_CASE_SLOW()" can be used to easily set
> > this to slow?
>
> This definitely seems important to add. I will add this to the documentation.
>
> >
> >
> > > +
> > > +``module``
> > > +
> > > +This attribute indicates the name of the module associated with the test.
> > > +
> > > +This attribute is automatically saved as a string and is printed for each suite.
> > > +Tests can also be filtered using this attribute.
> > > +
> > > --
> > > 2.41.0.255.g8b1d071c50-goog
> >
> > >
> >
> > Error: new blank line at EOF.
>
> Oops. I will change this.

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: [RFC v2 1/9] kunit: Add test attributes API structure

[David Gow]

[-- Attachment #1: Type: text/plain, Size: 20238 bytes --]

On Wed, 19 Jul 2023 at 05:01, Rae Moar <rmoar@google.com> wrote:
>
> On Tue, Jul 18, 2023 at 3:39 AM David Gow <davidgow@google.com> wrote:
> >
> > On Sat, 8 Jul 2023 at 05:09, Rae Moar <rmoar@google.com> wrote:
> > >
> > > Add the basic structure of the test attribute API to KUnit, which can be
> > > used to save and access test associated data.
> > >
> > > Add attributes.c and attributes.h to hold associated structs and functions
> > > for the API.
> > >
> > > Create a struct that holds a variety of associated helper functions for
> > > each test attribute. These helper functions will be used to get the
> > > attribute value, convert the value to a string, and filter based on the
> > > value. This struct is flexible by design to allow for attributes of
> > > numerous types and contexts.
> > >
> > > Add a method to print test attributes in the format of "# [<test_name if
> > > not suite>.]<attribute_name>: <attribute_value>".
> > >
> > > Example for a suite: "# speed: slow"
> > >
> > > Example for a test case: "# test_case.speed: very_slow"
> >
> > So, this is the one thing I'm a little unsure about here, and it's
> > really more of a problem with test names overall.
> >
> > As noted in the KTAPv2 attributes and test name proposals, the names
> > and attributes are only really defined for "suites", hence the need to
> > have a different output format for test cases.
> >
> > Personally, I'd prefer to keep the formats the same if we can (at
> > least for the actual KTAP output; I'm less concerned with the
> > list_attr option). That might make things a bit more difficult to
> > parse, though.
> >
> > One possibility would be to combine the KTAP attributes and test name
> > specs and suggest that every test has a "test name" attribute, which
> > must be the first attribute output.
> >
> > The output would then look something like:
> > KTAP version 2
> > # Name: my_suite
> > # Other-Attr: value
> > 1..2
> >   KTAP version 2
> >   # Name: test_1
> >   # Other-Attr: value
> >   ok 1 test_1
> >   # Name: test_2
> >   # Other-Attr: value
> >   not ok 2 test_2
> > ok 1 my_suite
> >
> > Would there be any problems with something like that?
> >
> > I'm less concerned with the list_attr option, as that's not something
> > totally standardised in the way KTAP is.
> >
>
> This is a really interesting idea. I like that this standardizes the
> concept of KTAP test metadata for both test suites and test cases. I
> would love to discuss this concept further as KTAP v2 is developed.
>
> My main concern would be that there is push back on stating the test
> name when it is already present in the result line. This adds
> potentially unnecessary lines to the output. However, one positive to
> this is that diagnostic data could be printed under this header which
> would reduce any confusion for which test the diagnostic data refers
> to.

Yeah, I think the main advantage is that the test name is known when
any diagnostic/other lines are read, and the main disadvantage is that
for trivial cases, it becomes pretty redundant.

>
> I would be interested if anyone else has any opinions on this.
>

Let's stick with what we're doing for now, and take this to the KTAP thread.

> > >
> > > Use this method to report attributes in the KTAP output (KTAP spec:
> > > https://docs.kernel.org/dev-tools/ktap.html) and _list_tests output when
> > > kernel's new kunit.action=list_attr option is used. Note this is derivative
> > > of the kunit.action=list option.
> > >
> > > In test.h, add fields and associated helper functions to test cases and
> > > suites to hold user-inputted test attributes.
> > >
> > > Signed-off-by: Rae Moar <rmoar@google.com>
> > > ---
> >
> > The only other thing I'd really like to support one day is having
> > attributes for individual parameters in parameterised tests. I think
> > it makes sense as a follow-up, though.
> >
>
> That is an exciting idea! I think that would be ideal as a follow-up.
>

Yeah, definitely no pressure to have that in the next version.

> > >
> > > Changes since v1:
> > > - Add list_attr option to only include attribute in the _list_tests output
> > >   when this module param is set
> > > - Add printing options for attributes to print always, print only for
> > >   suites, or print never.
> > >
> > >  include/kunit/attributes.h | 19 +++++++++
> > >  include/kunit/test.h       | 33 ++++++++++++++++
> > >  lib/kunit/Makefile         |  3 +-
> > >  lib/kunit/attributes.c     | 80 ++++++++++++++++++++++++++++++++++++++
> > >  lib/kunit/executor.c       | 21 ++++++++--
> > >  lib/kunit/test.c           | 17 ++++----
> > >  6 files changed, 161 insertions(+), 12 deletions(-)
> > >  create mode 100644 include/kunit/attributes.h
> > >  create mode 100644 lib/kunit/attributes.c
> > >
> > > diff --git a/include/kunit/attributes.h b/include/kunit/attributes.h
> > > new file mode 100644
> > > index 000000000000..9fcd184cce36
> > > --- /dev/null
> > > +++ b/include/kunit/attributes.h
> > > @@ -0,0 +1,19 @@
> > > +/* SPDX-License-Identifier: GPL-2.0 */
> > > +/*
> > > + * KUnit API to save and access test attributes
> > > + *
> > > + * Copyright (C) 2023, Google LLC.
> > > + * Author: Rae Moar <rmoar@google.com>
> > > + */
> > > +
> > > +#ifndef _KUNIT_ATTRIBUTES_H
> > > +#define _KUNIT_ATTRIBUTES_H
> > > +
> > > +/*
> > > + * Print all test attributes for a test case or suite.
> > > + * Output format for test cases: "# <test_name>.<attribute>: <value>"
> > > + * Output format for test suites: "# <attribute>: <value>"
> > > + */
> > > +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level);
> > > +
> > > +#endif /* _KUNIT_ATTRIBUTES_H */
> > > diff --git a/include/kunit/test.h b/include/kunit/test.h
> > > index 23120d50499e..1fc9155988e9 100644
> > > --- a/include/kunit/test.h
> > > +++ b/include/kunit/test.h
> > > @@ -63,12 +63,16 @@ enum kunit_status {
> > >         KUNIT_SKIPPED,
> > >  };
> > >
> > > +/* Holds attributes for each test case and suite */
> > > +struct kunit_attributes {};
> >
> > Do we want a separate set of attributes for test cases and suites?
> > (I think probably not, but it's worth making sure.)
> >
>
> I'm thinking if our goal is to eventually move to arbitrary nesting
> for tests it would be easiest to try to keep this list the same. But I
> agree. There may definitely be attributes that are more applicable for
> test cases or suites. I'm inclined to keep it this way.
>

Agreed: having them share the same thing is definitely more future proof.

> > > +
> > >  /**
> > >   * struct kunit_case - represents an individual test case.
> > >   *
> > >   * @run_case: the function representing the actual test case.
> > >   * @name:     the name of the test case.
> > >   * @generate_params: the generator function for parameterized tests.
> > > + * @attr:     the attributes associated with the test
> > >   *
> > >   * A test case is a function with the signature,
> > >   * ``void (*)(struct kunit *)``
> > > @@ -104,6 +108,7 @@ struct kunit_case {
> > >         void (*run_case)(struct kunit *test);
> > >         const char *name;
> > >         const void* (*generate_params)(const void *prev, char *desc);
> > > +       struct kunit_attributes attr;
> > >
> > >         /* private: internal use only. */
> > >         enum kunit_status status;
> > > @@ -133,6 +138,18 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> > >   */
> > >  #define KUNIT_CASE(test_name) { .run_case = test_name, .name = #test_name }
> > >
> > > +/**
> > > + * KUNIT_CASE_ATTR - A helper for creating a &struct kunit_case
> > > + * with attributes
> > > + *
> > > + * @test_name: a reference to a test case function.
> > > + * @attributes: a reference to a struct kunit_attributes object containing
> > > + * test attributes
> > > + */
> > > +#define KUNIT_CASE_ATTR(test_name, attributes)                 \
> > > +               { .run_case = test_name, .name = #test_name,    \
> > > +                 .attr = attributes }
> > > +
> > >  /**
> > >   * KUNIT_CASE_PARAM - A helper for creation a parameterized &struct kunit_case
> > >   *
> > > @@ -154,6 +171,20 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> > >                 { .run_case = test_name, .name = #test_name,    \
> > >                   .generate_params = gen_params }
> > >
> > > +/**
> > > + * KUNIT_CASE_PARAM_ATTR - A helper for creating a parameterized &struct
> > > + * kunit_case with attributes
> > > + *
> > > + * @test_name: a reference to a test case function.
> > > + * @gen_params: a reference to a parameter generator function.
> > > + * @attributes: a reference to a struct kunit_attributes object containing
> > > + * test attributes
> > > + */
> > > +#define KUNIT_CASE_PARAM_ATTR(test_name, gen_params, attributes)       \
> > > +               { .run_case = test_name, .name = #test_name,    \
> > > +                 .generate_params = gen_params,                                \
> > > +                 .attr = attributes }
> > > +
> >
> > I do worry a bit that we'll end up with a huge list of variants of the
> > KUNIT_CASE_* macros if we start adding more things here. I can't think
> > of a better way to handle it at the moment, though.
> >
> >
>
> I agree. If this becomes an issue, this could be a follow up change?
>
Yeah, sounds good.


> >
> > >  /**
> > >   * struct kunit_suite - describes a related collection of &struct kunit_case
> > >   *
> > > @@ -163,6 +194,7 @@ static inline char *kunit_status_to_ok_not_ok(enum kunit_status status)
> > >   * @init:      called before every test case.
> > >   * @exit:      called after every test case.
> > >   * @test_cases:        a null terminated array of test cases.
> > > + * @attr:      the attributes associated with the test suite
> > >   *
> > >   * A kunit_suite is a collection of related &struct kunit_case s, such that
> > >   * @init is called before every test case and @exit is called after every
> > > @@ -182,6 +214,7 @@ struct kunit_suite {
> > >         int (*init)(struct kunit *test);
> > >         void (*exit)(struct kunit *test);
> > >         struct kunit_case *test_cases;
> > > +       struct kunit_attributes attr;
> > >
> > >         /* private: internal use only */
> > >         char status_comment[KUNIT_STATUS_COMMENT_SIZE];
> > > diff --git a/lib/kunit/Makefile b/lib/kunit/Makefile
> > > index cb417f504996..46f75f23dfe4 100644
> > > --- a/lib/kunit/Makefile
> > > +++ b/lib/kunit/Makefile
> > > @@ -6,7 +6,8 @@ kunit-objs +=                           test.o \
> > >                                         string-stream.o \
> > >                                         assert.o \
> > >                                         try-catch.o \
> > > -                                       executor.o
> > > +                                       executor.o \
> > > +                                       attributes.o
> > >
> > >  ifeq ($(CONFIG_KUNIT_DEBUGFS),y)
> > >  kunit-objs +=                          debugfs.o
> > > diff --git a/lib/kunit/attributes.c b/lib/kunit/attributes.c
> > > new file mode 100644
> > > index 000000000000..9bda5a5f4030
> > > --- /dev/null
> > > +++ b/lib/kunit/attributes.c
> > > @@ -0,0 +1,80 @@
> > > +// SPDX-License-Identifier: GPL-2.0
> > > +/*
> > > + * KUnit API to save and access test attributes
> > > + *
> > > + * Copyright (C) 2023, Google LLC.
> > > + * Author: Rae Moar <rmoar@google.com>
> > > + */
> > > +
> > > +#include <kunit/test.h>
> > > +#include <kunit/attributes.h>
> > > +
> > > +/* Options for printing attributes:
> > > + * PRINT_ALWAYS - attribute is printed for every test case and suite if set
> > > + * PRINT_SUITE - attribute is printed for every suite if set but not for test cases
> > > + * PRINT_NEVER - attribute is never printed
> > > + */
> > > +enum print_ops {
> > > +       PRINT_ALWAYS,
> > > +       PRINT_SUITE,
> > > +       PRINT_NEVER,
> > > +};
> > > +
> > > +/**
> > > + * struct kunit_attr - represents a test attribute and holds flexible
> > > + * helper functions to interact with attribute.
> > > + *
> > > + * @name: name of test attribute, eg. speed
> > > + * @get_attr: function to return attribute value given a test
> > > + * @to_string: function to return string representation of given
> > > + * attribute value
> > > + * @filter: function to indicate whether a given attribute value passes a
> > > + * filter
> > > + */
> > > +struct kunit_attr {
> > > +       const char *name;
> > > +       void *(*get_attr)(void *test_or_suite, bool is_test);
> > > +       const char *(*to_string)(void *attr, bool *to_free);
> > > +       int (*filter)(void *attr, const char *input, int *err);
> > > +       void *attr_default;
> > > +       enum print_ops print;
> > > +};
> > > +
> > > +/* List of all Test Attributes */
> > > +
> > > +static struct kunit_attr kunit_attr_list[] = {};
> > > +
> > > +/* Helper Functions to Access Attributes */
> > > +
> > > +void kunit_print_attr(void *test_or_suite, bool is_test, unsigned int test_level)
> > > +{
> > > +       int i;
> > > +       bool to_free;
> > > +       void *attr;
> > > +       const char *attr_name, *attr_str;
> > > +       struct kunit_suite *suite = is_test ? NULL : test_or_suite;
> > > +       struct kunit_case *test = is_test ? test_or_suite : NULL;
> > > +
> > > +       for (i = 0; i < ARRAY_SIZE(kunit_attr_list); i++) {
> > > +               if (kunit_attr_list[i].print == PRINT_NEVER ||
> > > +                               (test && kunit_attr_list[i].print == PRINT_SUITE))
> > > +                       continue;
> > > +               attr = kunit_attr_list[i].get_attr(test_or_suite, is_test);
> > > +               if (attr) {
> > > +                       attr_name = kunit_attr_list[i].name;
> > > +                       attr_str = kunit_attr_list[i].to_string(attr, &to_free);
> > > +                       if (test) {
> > > +                               kunit_log(KERN_INFO, test, "%*s# %s.%s: %s",
> > > +                                       KUNIT_INDENT_LEN * test_level, "", test->name,
> > > +                                       attr_name, attr_str);
> > > +                       } else {
> > > +                               kunit_log(KERN_INFO, suite, "%*s# %s: %s",
> > > +                                       KUNIT_INDENT_LEN * test_level, "", attr_name, attr_str);
> > > +                       }
> > > +
> > > +                       /* Free to_string of attribute if needed */
> > > +                       if (to_free)
> > > +                               kfree(attr_str);
> > > +               }
> > > +       }
> > > +}
> > > diff --git a/lib/kunit/executor.c b/lib/kunit/executor.c
> > > index 74982b83707c..12e38a48a5cc 100644
> > > --- a/lib/kunit/executor.c
> > > +++ b/lib/kunit/executor.c
> > > @@ -2,6 +2,7 @@
> > >
> > >  #include <linux/reboot.h>
> > >  #include <kunit/test.h>
> > > +#include <kunit/attributes.h>
> > >  #include <linux/glob.h>
> > >  #include <linux/moduleparam.h>
> > >
> > > @@ -24,7 +25,8 @@ module_param_named(action, action_param, charp, 0);
> > >  MODULE_PARM_DESC(action,
> > >                  "Changes KUnit executor behavior, valid values are:\n"
> > >                  "<none>: run the tests like normal\n"
> > > -                "'list' to list test names instead of running them.\n");
> > > +                "'list' to list test names instead of running them.\n"
> > > +                "'list_attr' to list test names and attributes instead of running them.\n");
> > >
> > >  /* glob_match() needs NULL terminated strings, so we need a copy of filter_glob_param. */
> > >  struct kunit_test_filter {
> > > @@ -172,7 +174,7 @@ static void kunit_exec_run_tests(struct suite_set *suite_set)
> > >         __kunit_test_suites_init(suite_set->start, num_suites);
> > >  }
> > >
> > > -static void kunit_exec_list_tests(struct suite_set *suite_set)
> > > +static void kunit_exec_list_tests(struct suite_set *suite_set, bool include_attr)
> > >  {
> > >         struct kunit_suite * const *suites;
> > >         struct kunit_case *test_case;
> > > @@ -180,10 +182,19 @@ static void kunit_exec_list_tests(struct suite_set *suite_set)
> > >         /* Hack: print a ktap header so kunit.py can find the start of KUnit output. */
> > >         pr_info("KTAP version 1\n");
> > >
> > > -       for (suites = suite_set->start; suites < suite_set->end; suites++)
> > > +       for (suites = suite_set->start; suites < suite_set->end; suites++) {
> > > +               /* Print suite name and suite attributes */
> > > +               pr_info("%s\n", (*suites)->name);
> > > +               if (include_attr)
> > > +                       kunit_print_attr((void *)(*suites), false, 0);
> > > +
> > > +               /* Print test case name and attributes in suite */
> > >                 kunit_suite_for_each_test_case((*suites), test_case) {
> > >                         pr_info("%s.%s\n", (*suites)->name, test_case->name);
> > > +                       if (include_attr)
> > > +                               kunit_print_attr((void *)test_case, true, 0);
> > >                 }
> > > +       }
> > >  }
> > >
> > >  int kunit_run_all_tests(void)
> > > @@ -206,7 +217,9 @@ int kunit_run_all_tests(void)
> > >         if (!action_param)
> > >                 kunit_exec_run_tests(&suite_set);
> > >         else if (strcmp(action_param, "list") == 0)
> > > -               kunit_exec_list_tests(&suite_set);
> > > +               kunit_exec_list_tests(&suite_set, false);
> > > +       else if (strcmp(action_param, "list_attr") == 0)
> > > +               kunit_exec_list_tests(&suite_set, true);
> > >         else
> > >                 pr_err("kunit executor: unknown action '%s'\n", action_param);
> > >
> > > diff --git a/lib/kunit/test.c b/lib/kunit/test.c
> > > index 84e4666555c9..9ee55139ecd1 100644
> > > --- a/lib/kunit/test.c
> > > +++ b/lib/kunit/test.c
> > > @@ -9,6 +9,7 @@
> > >  #include <kunit/resource.h>
> > >  #include <kunit/test.h>
> > >  #include <kunit/test-bug.h>
> > > +#include <kunit/attributes.h>
> > >  #include <linux/kernel.h>
> > >  #include <linux/module.h>
> > >  #include <linux/moduleparam.h>
> > > @@ -168,6 +169,13 @@ size_t kunit_suite_num_test_cases(struct kunit_suite *suite)
> > >  }
> > >  EXPORT_SYMBOL_GPL(kunit_suite_num_test_cases);
> > >
> > > +/* Currently supported test levels */
> > > +enum {
> > > +       KUNIT_LEVEL_SUITE = 0,
> > > +       KUNIT_LEVEL_CASE,
> > > +       KUNIT_LEVEL_CASE_PARAM,
> > > +};
> > > +
> > >  static void kunit_print_suite_start(struct kunit_suite *suite)
> > >  {
> > >         /*
> > > @@ -181,17 +189,11 @@ static void kunit_print_suite_start(struct kunit_suite *suite)
> > >         pr_info(KUNIT_SUBTEST_INDENT "KTAP version 1\n");
> > >         pr_info(KUNIT_SUBTEST_INDENT "# Subtest: %s\n",
> > >                   suite->name);
> > > +       kunit_print_attr((void *)suite, false, KUNIT_LEVEL_CASE);
> > >         pr_info(KUNIT_SUBTEST_INDENT "1..%zd\n",
> > >                   kunit_suite_num_test_cases(suite));
> > >  }
> > >
> > > -/* Currently supported test levels */
> > > -enum {
> > > -       KUNIT_LEVEL_SUITE = 0,
> > > -       KUNIT_LEVEL_CASE,
> > > -       KUNIT_LEVEL_CASE_PARAM,
> > > -};
> > > -
> > >  static void kunit_print_ok_not_ok(struct kunit *test,
> > >                                   unsigned int test_level,
> > >                                   enum kunit_status status,
> > > @@ -651,6 +653,7 @@ int kunit_run_tests(struct kunit_suite *suite)
> > >                         }
> > >                 }
> > >
> > > +               kunit_print_attr((void *)test_case, true, KUNIT_LEVEL_CASE);
> > >
> > >                 kunit_print_test_stats(&test, param_stats);
> > >
> > > --
> > > 2.41.0.255.g8b1d071c50-goog
> > >

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]