[PATCH 0/3] kselftest framework to introduce TEST_CONFIG_DEPS

[PATCH 0/3] kselftest framework to introduce TEST_CONFIG_DEPS

[Siddharth Menon]

Currently, kselftests does not have a generalised mechanism to skip compilation
and run tests when required kernel configuration options are disabled.

This patch series addresses this limitation by introducing a new flag, 
'TEST_CONFIG_DEPS' in lib.mk, along with corresponding updates to the 
documentation. 
The selftests/livepatch/Makefile has been updated to utilize TEST_CONFIG_DEPS.

Siddharth Menon (3):
  docs/kselftests: Explain the usage of TEST_CONFIG_DEPS
  selftests/lib.mk: Introduce check to validate required configs
  selftests/livepatch: Check if required config options are enabled

 Documentation/dev-tools/kselftest.rst      |  3 +++
 tools/testing/selftests/lib.mk             | 18 ++++++++++++++++--
 tools/testing/selftests/livepatch/Makefile |  1 +
 3 files changed, 20 insertions(+), 2 deletions(-)

-- 
2.39.5

[PATCH 1/3] docs/kselftests: Explain the usage of TEST_CONFIG_DEPS

[Siddharth Menon]

Update documentation to explain the TEST_CONFIG_DEPS flag in lib.mk.
TEST_CONFIG_DEPS is used to validate the presence of required config flags
specified in the selftest makefile before compiling or running a test.

Signed-off-by: Siddharth Menon <simeddon@gmail.com>
---
 Documentation/dev-tools/kselftest.rst | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
index fdb1df86783a..e816b282363f 100644
--- a/Documentation/dev-tools/kselftest.rst
+++ b/Documentation/dev-tools/kselftest.rst
@@ -301,6 +301,9 @@ Contributing new tests (details)
 
    e.g: tools/testing/selftests/android/config
 
+ * Use TEST_CONFIG_DEPS to specify required config options to be enabled 
+   before a test is allowed to run or compile.
+
  * Create a .gitignore file inside test directory and add all generated objects
    in it.
 
-- 
2.39.5

Re: [PATCH 1/3] docs/kselftests: Explain the usage of TEST_CONFIG_DEPS

[Petr Mladek]

On Thu 2024-12-05 17:17:55, Siddharth Menon wrote:
> Update documentation to explain the TEST_CONFIG_DEPS flag in lib.mk.
> TEST_CONFIG_DEPS is used to validate the presence of required config flags
> specified in the selftest makefile before compiling or running a test.
> 
> Signed-off-by: Siddharth Menon <simeddon@gmail.com>
> ---
>  Documentation/dev-tools/kselftest.rst | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/Documentation/dev-tools/kselftest.rst b/Documentation/dev-tools/kselftest.rst
> index fdb1df86783a..e816b282363f 100644
> --- a/Documentation/dev-tools/kselftest.rst
> +++ b/Documentation/dev-tools/kselftest.rst
> @@ -301,6 +301,9 @@ Contributing new tests (details)
>  
>     e.g: tools/testing/selftests/android/config
>  
> + * Use TEST_CONFIG_DEPS to specify required config options to be enabled 
> +   before a test is allowed to run or compile.
> +
>   * Create a .gitignore file inside test directory and add all generated objects
>     in it.

It might be a matter of taste. It is a chicken & egg problem. I
personally find it weird to document something which does not exist yet.

Please, either update the documentation together with the code or
later :-)

Best Regards,
Petr

PS: I haven't got this mail. I have got only 2nd and 3rd patch.
    I prefer to see the entire patchset. I suggest to always
    send all patches to the same list of people and mailing lists.
    

[PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[Siddharth Menon]

Currently, kselftests does not have a generalised mechanism to skip compilation
and run tests when required kernel configuration flags are missing.

This patch introduces a check to validate the presence of required config flags
specified in the selftest makefile. In case scripts/config is not found,
this check is skipped.

Use TEST_CONFIG_DEPS to check for specific config options before compiling,
example usage:
```
TEST_CONFIG_DEPS := CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG
```
Here it checks whether CONFIG_LIVEPATCH and CONFIG_DYNAMIC_DEBUG are enabled.

Suggested-by: Petr Mladek <pmladek@suse.com>
Suggested-by: Miroslav Benes <mbenes@suse.cz>
Reviewed-by: Shuah Khan <skhan@linuxfoundation.org>
Signed-off-by: Siddharth Menon <simeddon@gmail.com>
---
 tools/testing/selftests/lib.mk | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/lib.mk b/tools/testing/selftests/lib.mk
index d6edcfcb5be8..7ca713237bf7 100644
--- a/tools/testing/selftests/lib.mk
+++ b/tools/testing/selftests/lib.mk
@@ -97,7 +97,21 @@ TEST_GEN_PROGS := $(patsubst %,$(OUTPUT)/%,$(TEST_GEN_PROGS))
 TEST_GEN_PROGS_EXTENDED := $(patsubst %,$(OUTPUT)/%,$(TEST_GEN_PROGS_EXTENDED))
 TEST_GEN_FILES := $(patsubst %,$(OUTPUT)/%,$(TEST_GEN_FILES))
 
-all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES) \
+KDIR ?= /lib/modules/$(shell uname -r)/build
+
+define CHECK_CONFIG_DEPS
+    $(if $(wildcard $(KDIR)/scripts/config),
+        $(eval MISSING_FLAGS := $(filter-out 1,$(foreach cfg,$(TEST_CONFIG_DEPS),\
+            $(shell cd $(KDIR) && scripts/config --state $(cfg) | grep -q '^\(y\|m\)$$' && echo 1 || echo $(cfg))))),
+        $(info Skipping CHECK_GEN_REQ: $(KDIR)/scripts/config not found)
+    )
+    $(if $(MISSING_FLAGS),$(error Missing required config flags: $(MISSING_FLAGS)))
+endef
+
+check_config_deps:
+	$(call CHECK_CONFIG_DEPS)
+
+all: check_config_deps $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES) \
 	$(if $(TEST_GEN_MODS_DIR),gen_mods_dir)
 
 define RUN_TESTS
@@ -228,4 +242,4 @@ $(OUTPUT)/%:%.S
 	$(LINK.S) $^ $(LDLIBS) -o $@
 endif
 
-.PHONY: run_tests all clean install emit_tests gen_mods_dir clean_mods_dir
+.PHONY: run_tests all clean install emit_tests gen_mods_dir clean_mods_dir check_config_deps
-- 
2.39.5

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

On Thu, Dec 05, 2024 at 05:17:56PM +0530, Siddharth Menon wrote:

> Currently, kselftests does not have a generalised mechanism to skip compilation
> and run tests when required kernel configuration flags are missing.

Should this be a build dependency or only a runtime dependency, or
should these be separate options for cases where the selftest builds a
module?  If people are building the selftests once and then using them
with a bunch of kernel builds it might be surprising if some of the
binaries vanish.

> -all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES) \
> +KDIR ?= /lib/modules/$(shell uname -r)/build
> +

Shouldn't we try the current kernel tree, and for runtime checks
/proc/config.gz would be good to check when it's enabled?

> +define CHECK_CONFIG_DEPS
> +    $(if $(wildcard $(KDIR)/scripts/config),
> +        $(eval MISSING_FLAGS := $(filter-out 1,$(foreach cfg,$(TEST_CONFIG_DEPS),\
> +            $(shell cd $(KDIR) && scripts/config --state $(cfg) | grep -q '^\(y\|m\)$$' && echo 1 || echo $(cfg))))),
> +        $(info Skipping CHECK_GEN_REQ: $(KDIR)/scripts/config not found)
> +    )
> +    $(if $(MISSING_FLAGS),$(error Missing required config flags: $(MISSING_FLAGS)))
> +endef

This is going to use a separate set of config options to those listed in
the config file in the selftest directory which is perhaps a bit
surprising.  OTOH we do have a lot of the selftests directories where
not every test needs all the options so that's probably a good choice
unless we make things finer grained which might be more trouble than
it's worth.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[BiscuitBobby]

On Thu, 5 Dec 2024 at 21:05, Mark Brown <broonie@kernel.org> wrote:
> Should this be a build dependency or only a runtime dependency, or
> should these be separate options for cases where the selftest builds a
> module?

Hello, thanks for looking through my patch. Some tests fail to compile and
throw errors in case their required config options are not enabled.
This optional check was created to prevent such issues.

> If people are building the selftests once and then using them
> with a bunch of kernel builds it might be surprising if some of the
> binaries vanish.

I'm not really familiar with packaging selftests, but this patch does not
remove existing binaries when running tests. If I misunderstood what
you are referring to, please let me know.

> Shouldn't we try the current kernel tree, and for runtime checks
> /proc/config.gz would be good to check when it's enabled?

When I looked into this, it seems (according to the config.gz man page)
that the contents of /proc/config.gz are the same as
/lib/modules/$(uname -r)/build/.config, but /proc/config.gz is only available if
the kernel was compiled with CONFIG_IKCONFIG_PROC enabled.

It does seem like /lib/modules/$(uname -r)/build/scripts is not always available
though, so will send in a new patch directly checking build/.config
after checking
it out on a few more systems.

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1908 bytes --]

On Sat, Dec 07, 2024 at 12:50:47AM +0530, BiscuitBobby wrote:
> On Thu, 5 Dec 2024 at 21:05, Mark Brown <broonie@kernel.org> wrote:

> > If people are building the selftests once and then using them
> > with a bunch of kernel builds it might be surprising if some of the
> > binaries vanish.

> I'm not really familiar with packaging selftests, but this patch does not
> remove existing binaries when running tests. If I misunderstood what
> you are referring to, please let me know.

It doesn't remove existing barriers but it's also not obvious that this
isn't just a generic dependency mechanism that should be used for any
old dependency rather than things that would actually break the build,
and it's one config list for both build and runtime checks.  If the
intention is that this should be infrequently used it probably needs to
be a bit clearer that that's the case.  As things stand I'd expect there
to be some confusion about the interaction between this and the existing
system with specifying config fragments.

> > Shouldn't we try the current kernel tree, and for runtime checks
> > /proc/config.gz would be good to check when it's enabled?

> When I looked into this, it seems (according to the config.gz man page)
> that the contents of /proc/config.gz are the same as
> /lib/modules/$(uname -r)/build/.config, but /proc/config.gz is only available if
> the kernel was compiled with CONFIG_IKCONFIG_PROC enabled.

> It does seem like /lib/modules/$(uname -r)/build/scripts is not always available
> though, so will send in a new patch directly checking build/.config
> after checking
> it out on a few more systems.

Indeed, when deploying a kernel for test people don't always deploy
modules onto the target filesystem, there may not be any modules at all
if CONFIG_MODULES is disabled, or people might just not install modules
that do get built if they're not needed on a given platform.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[BiscuitBobby]

On Sat, 7 Dec 2024 at 01:20, Mark Brown <broonie@kernel.org> wrote:
>
> It doesn't remove existing barriers but it's also not obvious that this
> isn't just a generic dependency mechanism that should be used for any
> old dependency rather than things that would actually break the build,
> and it's one config list for both build and runtime checks.  If the
> intention is that this should be infrequently used it probably needs to
> be a bit clearer that that's the case.  As things stand I'd expect there
> to be some confusion about the interaction between this and the existing
> system with specifying config fragments.

I shall update the commit and documentation to make this more clear.
Thank you for the feedback.

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[Petr Mladek]

On Thu 2024-12-05 17:17:56, Siddharth Menon wrote:
> Currently, kselftests does not have a generalised mechanism to skip compilation
> and run tests when required kernel configuration flags are missing.
> 
> This patch introduces a check to validate the presence of required config flags
> specified in the selftest makefile. In case scripts/config is not found,
> this check is skipped.
> 
> Use TEST_CONFIG_DEPS to check for specific config options before compiling,
> example usage:
> ```
> TEST_CONFIG_DEPS := CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG

What is the reason to add another set of dependencies, please?

Both CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG are already mentioned in
tools/testing/selftests/livepatch/config

IMHO, the new check should read the dependencies
from the existing tools/testing/selftests/<test>/config file.

> --- a/tools/testing/selftests/lib.mk
> +++ b/tools/testing/selftests/lib.mk
> @@ -97,7 +97,21 @@ TEST_GEN_PROGS := $(patsubst %,$(OUTPUT)/%,$(TEST_GEN_PROGS))
>  TEST_GEN_PROGS_EXTENDED := $(patsubst %,$(OUTPUT)/%,$(TEST_GEN_PROGS_EXTENDED))
>  TEST_GEN_FILES := $(patsubst %,$(OUTPUT)/%,$(TEST_GEN_FILES))
>  
> -all: $(TEST_GEN_PROGS) $(TEST_GEN_PROGS_EXTENDED) $(TEST_GEN_FILES) \
> +KDIR ?= /lib/modules/$(shell uname -r)/build
> +
> +define CHECK_CONFIG_DEPS
> +    $(if $(wildcard $(KDIR)/scripts/config),
> +        $(eval MISSING_FLAGS := $(filter-out 1,$(foreach cfg,$(TEST_CONFIG_DEPS),\
> +            $(shell cd $(KDIR) && scripts/config --state $(cfg) | grep -q '^\(y\|m\)$$' && echo 1 || echo $(cfg))))),
> +        $(info Skipping CHECK_GEN_REQ: $(KDIR)/scripts/config not found)
> +    )
> +    $(if $(MISSING_FLAGS),$(error Missing required config flags: $(MISSING_FLAGS)))
> +endef

It somehow does not work here. I get:

tools/testing/selftests/livepatch # make run_tests 
grep: .config: No such file or directory
grep: .config: No such file or directory
grep: .config: No such file or directory
grep: .config: No such file or directory
../lib.mk:112: *** Missing required config flags: CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG.  Stop.


I run the livepatch tests the following way.

1. On my workstation, I build the kernel RPMs using

     make rpm-pkg

2. In qemu test system, I mount the build directory from the
   workstation and install both kernel and kernel-devel packages:

    rpm -ivh rpmbuild/RPMS/x86_64/kernel-6.12.0_default+-35.x86_64.rpm
    rpm -ivh rpmbuild/RPMS/x86_64/kernel-devel-6.12.0_default+-35.x86_64.rpm

   and reboot

3. In rebooted qemu test system, I mount once again the build
   directory from the workstation and run the tests:

     cd tools/testing/selftests/livepatch
     make run_tests


The "grep" errors come from the "scripts/config" command. For example:

   tools/testing/selftests/livepatch # /lib/modules/6.12.0-default+/build/scripts/config -s CONFIG_LIVEPATCH
   grep: .config: No such file or directory
   grep: .config: No such file or directory
   undef

It helps to define patch to the config file installed by the devel
package:

   tools/testing/selftests/livepatch # /lib/modules/6.12.0-default+/build/scripts/config --file /lib/modules/6.12.0-default+/config -s CONFIG_LIVEPATCH
   y


But I am not sure if this works when people run the "make" in the
original build directory on the workstation.

Best Regards,
Petr

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[BiscuitBobby]

On Tue, 10 Dec 2024 at 20:26, Petr Mladek <pmladek@suse.com> wrote:
>
> What is the reason to add another set of dependencies, please?

I had done this because not every test required all the options specified in
tools/testing/selftests/<test>/config. I thought it would not be desirable to
prevent these tests from compiling/running.

> Both CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG are already mentioned in
> tools/testing/selftests/livepatch/config

This particular test only required CONFIG_LIVEPATCH to compile, but I
had included CONFIG_DYNAMIC_DEBUG, as Miroslav had expressed
wanting both of them checked.

> IMHO, the new check should read the dependencies
> from the existing tools/testing/selftests/<test>/config file.

I shall check tools/testing/selftests/<test>/config in my next patch as
suggested.

> I run the livepatch tests the following way.
>
> 1. On my workstation, I build the kernel RPMs using
>
>      make rpm-pkg
>
> 2. In qemu test system, I mount the build directory from the
>    workstation and install both kernel and kernel-devel packages:
>
>     rpm -ivh rpmbuild/RPMS/x86_64/kernel-6.12.0_default+-35.x86_64.rpm
>     rpm -ivh rpmbuild/RPMS/x86_64/kernel-devel-6.12.0_default+-35.x86_64.rpm
>
>    and reboot
>
> 3. In rebooted qemu test system, I mount once again the build
>    directory from the workstation and run the tests:
>
>      cd tools/testing/selftests/livepatch
>      make run_tests

Thanks, I will test building kernel RPMs when I update the check to be
more distro agnostic.

Sincerely,
Siddharth Menon

Re: [PATCH 2/3] selftests/lib.mk: Introduce check to validate required configs

[Petr Mladek]

On Tue 2024-12-10 22:40:51, BiscuitBobby wrote:
> On Tue, 10 Dec 2024 at 20:26, Petr Mladek <pmladek@suse.com> wrote:
> >
> > What is the reason to add another set of dependencies, please?
> 
> I had done this because not every test required all the options specified in
> tools/testing/selftests/<test>/config. I thought it would not be desirable to
> prevent these tests from compiling/running.

The biggest problem is that tools/testing/selftests/<test>/config are
not used during build or tests at the moment. It means that they
are not tested and might be outdated.

If we add the dependency then some <test>/config files might need
to get fixed.

I am not sure how many problems it might cause. But it might
be worth the effort.

> > Both CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG are already mentioned in
> > tools/testing/selftests/livepatch/config
>
> This particular test only required CONFIG_LIVEPATCH to compile, but I
> had included CONFIG_DYNAMIC_DEBUG, as Miroslav had expressed
> wanting both of them checked.

I see. The build succeeds even without CONFIG_DYNAMIC_DEBUG.
But it must be enabled on the kernel where the test modules
are loaded. Otherwise, the tests would fail.

Honestly, I think that this is rather an exception. It works
only because all the needed pr_debug() messages are in
the livepatch core code. The test modules do not use pr_debug()
on its own.

I believe that most options in tools/testing/selftests/<test>/config
have to be enabled on both compile and runtime. Otherwise, the test
binaries might not have access to the needed API.

I suggest to keep it simple and require all <test>/config options
at both compile and run time. IMHO, most people build and run
the tests on the same kernel anyway.

Best Regards,
Petr

[PATCH 3/3] selftests/livepatch: Check if required config options are enabled

[Siddharth Menon]

When CONFIG_LIVEPATCH is disabled, compilation fails due to the
required structs from the livepatch header file being undefined.
This checks for whether CONFIG_LIVEPATCH and CONFIG_DYNAMIC_DEBUG
are enabled before compiling livepatch self-tests.

Signed-off-by: Siddharth Menon <simeddon@gmail.com>
---
 tools/testing/selftests/livepatch/Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tools/testing/selftests/livepatch/Makefile b/tools/testing/selftests/livepatch/Makefile
index a080eb54a215..14b5c60663cd 100644
--- a/tools/testing/selftests/livepatch/Makefile
+++ b/tools/testing/selftests/livepatch/Makefile
@@ -14,5 +14,6 @@ TEST_PROGS := \
 	test-kprobe.sh
 
 TEST_FILES := settings
+TEST_CONFIG_DEPS := CONFIG_LIVEPATCH CONFIG_DYNAMIC_DEBUG
 
 include ../lib.mk
-- 
2.39.5