[PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Kees Cook]

 v2:
  - switch from -include to -I with a -D gated include compiler-version.h
 v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/

Hi,

This is my attempt to introduce dependencies that track the various
compiler behaviors that may globally change the build that aren't
represented by either compiler flags nor the compiler version
(CC_VERSION_TEXT). Namely, this is to detect when the contents of a
file the compiler uses changes. We have 3 such situations currently in
the tree:

- If any of the GCC plugins change, we need to rebuild everything that
  was built with them, as they may have changed their behavior and those
  behaviors may need to be synchronized across all translation units.
  (The most obvious of these is the randstruct GCC plugin, but is true
  for most of them.)

- If the randstruct seed itself changes (whether for GCC plugins or
  Clang), the entire tree needs to be rebuilt since the randomization of
  structures may change between compilation units if not.

- If the integer-wrap-ignore.scl file for Clang's integer wrapping
  sanitizer changes, a full rebuild is needed as the coverage for wrapping
  types may have changed, once again cause behavior differences between
  compilation units.

The best way I found to deal with this is to:
- Generate a .h file that is updated when the specific dependencies change.
  e.g.: randstruct_hash.h depends on randstruct.seed

- Add a -I argument globally to be able to locate the .h file.
  e.g.: -I$(objtree)/scripts/basic

- Add a conditional -D argument for each separate case
  e.g.: RANDSTRUCT_CFLAGS += -DRANDSTRUCT

- Include the .h file from compiler-version.h through an #ifdef for the define
  e.g.:
  #ifdef RANDSTUCT
  #include "randstruct_hash.h"
  #endif

This means that all targets gain the dependency (via fixdep), but only
when the defines are active, which means they are trivially controlled
by the existing CFLAGS removal mechanisms that are already being used
to turn off each of the above features.

-Kees

Kees Cook (3):
  gcc-plugins: Force full rebuild when plugins change
  randstruct: Force full rebuild when seed changes
  integer-wrap: Force full rebuild when .scl file changes

 Makefile                         |  1 +
 arch/um/Makefile                 |  2 ++
 include/linux/compiler-version.h | 10 ++++++++++
 include/linux/vermagic.h         |  1 -
 scripts/Makefile.gcc-plugins     |  2 +-
 scripts/Makefile.ubsan           |  1 +
 scripts/basic/Makefile           | 20 +++++++++++++++-----
 scripts/gcc-plugins/Makefile     |  8 ++++++++
 8 files changed, 38 insertions(+), 7 deletions(-)

-- 
2.34.1

[PATCH v2 1/3] gcc-plugins: Force full rebuild when plugins change

[Kees Cook]

There was no dependency between the plugins changing and the rest of the
kernel being built. Enforce this by including a synthetic header file
when using plugins, that is regenerated any time the plugins are built.

This cannot be included via '-include ...' because Makefiles use the
"filter-out" string function, which removes individual words. Removing
all instances of "-include" from the CFLAGS will cause a lot of
problems. :)

Instead, use -I to include the gcc-plugins directory, and depend on the
new -DGCC_PLUGINS_ENABLED flag to include the generated header file via
include/linux/compiler-version.h, which is already being used to control
full rebuilds. The UM build requires that the -I be explicitly added.

Signed-off-by: Kees Cook <kees@kernel.org>
---
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nicolas Schier <nicolas.schier@linux.dev>
Cc: <linux-hardening@vger.kernel.org>
Cc: <linux-kbuild@vger.kernel.org>
---
 arch/um/Makefile                 | 1 +
 include/linux/compiler-version.h | 4 ++++
 scripts/Makefile.gcc-plugins     | 2 +-
 scripts/gcc-plugins/Makefile     | 8 ++++++++
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/arch/um/Makefile b/arch/um/Makefile
index 1d36a613aad8..8cc0f22ebefa 100644
--- a/arch/um/Makefile
+++ b/arch/um/Makefile
@@ -72,6 +72,7 @@ USER_CFLAGS = $(patsubst $(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \
 		$(ARCH_INCLUDE) $(MODE_INCLUDE) $(filter -I%,$(CFLAGS)) \
 		-D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \
 		-idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \
+		-I$(objtree)/scripts/gcc-plugins \
 		-include $(srctree)/include/linux/compiler-version.h \
 		-include $(srctree)/include/linux/kconfig.h
 
diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
index 573fa85b6c0c..08943df04ebb 100644
--- a/include/linux/compiler-version.h
+++ b/include/linux/compiler-version.h
@@ -12,3 +12,7 @@
  * and add dependency on include/config/CC_VERSION_TEXT, which is touched
  * by Kconfig when the version string from the compiler changes.
  */
+
+#ifdef GCC_PLUGINS_ENABLED
+#include "gcc-plugins-deps.h"
+#endif
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 5b8a8378ca8a..468bb8faa9d1 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -38,7 +38,7 @@ export DISABLE_STACKLEAK_PLUGIN
 
 # All the plugin CFLAGS are collected here in case a build target needs to
 # filter them out of the KBUILD_CFLAGS.
-GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
+GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y)) -I$(objtree)/scripts/gcc-plugins -DGCC_PLUGINS_ENABLED
 export GCC_PLUGINS_CFLAGS
 
 # Add the flags to the build!
diff --git a/scripts/gcc-plugins/Makefile b/scripts/gcc-plugins/Makefile
index 320afd3cf8e8..24671d39ec90 100644
--- a/scripts/gcc-plugins/Makefile
+++ b/scripts/gcc-plugins/Makefile
@@ -66,3 +66,11 @@ quiet_cmd_plugin_cxx_o_c = HOSTCXX $@
 
 $(plugin-objs): $(obj)/%.o: $(src)/%.c FORCE
 	$(call if_changed_dep,plugin_cxx_o_c)
+
+quiet_cmd_gcc_plugins_updated = UPDATE  $@
+      cmd_gcc_plugins_updated = echo '/* $^ */' > $(obj)/gcc-plugins-deps.h
+
+$(obj)/gcc-plugins-deps.h: $(plugin-single) $(plugin-multi) FORCE
+	$(call if_changed,gcc_plugins_updated)
+
+always-y += gcc-plugins-deps.h
-- 
2.34.1

[PATCH v2 2/3] randstruct: Force full rebuild when seed changes

[Kees Cook]

While the randstruct GCC plugin was being rebuilt if the randstruct
seed changed, Clangs build did not notice the change. Include the hash
header directly so that it becomes a universal build dependency and full
rebuilds will happen if it changes.

Since we cannot use "-include ..." as the randstruct flags are removed
via "filter-out" (which would cause all instances of "-include" to be
removed), use the existing -DRANDSTRUCT to control the header inclusion
via include/linux/compiler-version.h. Universally add a -I for the
scripts/basic directory, where header exists. The UM build requires that
the -I be explicitly added.

Signed-off-by: Kees Cook <kees@kernel.org>
---
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nicolas Schier <nicolas.schier@linux.dev>
Cc: Petr Pavlu <petr.pavlu@suse.com>
Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
Cc: <linux-kbuild@vger.kernel.org>
---
 Makefile                         |  1 +
 arch/um/Makefile                 |  1 +
 include/linux/compiler-version.h |  3 +++
 include/linux/vermagic.h         |  1 -
 scripts/basic/Makefile           | 11 ++++++-----
 5 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/Makefile b/Makefile
index 5aa9ee52a765..cef652227843 100644
--- a/Makefile
+++ b/Makefile
@@ -567,6 +567,7 @@ LINUXINCLUDE    := \
 		-I$(objtree)/arch/$(SRCARCH)/include/generated \
 		-I$(srctree)/include \
 		-I$(objtree)/include \
+		-I$(objtree)/scripts/basic \
 		$(USERINCLUDE)
 
 KBUILD_AFLAGS   := -D__ASSEMBLY__ -fno-PIE
diff --git a/arch/um/Makefile b/arch/um/Makefile
index 8cc0f22ebefa..38f6024e75d7 100644
--- a/arch/um/Makefile
+++ b/arch/um/Makefile
@@ -73,6 +73,7 @@ USER_CFLAGS = $(patsubst $(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \
 		-D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \
 		-idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \
 		-I$(objtree)/scripts/gcc-plugins \
+		-I$(objtree)/scripts/basic \
 		-include $(srctree)/include/linux/compiler-version.h \
 		-include $(srctree)/include/linux/kconfig.h
 
diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
index 08943df04ebb..05d555320a0f 100644
--- a/include/linux/compiler-version.h
+++ b/include/linux/compiler-version.h
@@ -16,3 +16,6 @@
 #ifdef GCC_PLUGINS_ENABLED
 #include "gcc-plugins-deps.h"
 #endif
+#ifdef RANDSTRUCT
+#include "randstruct_hash.h"
+#endif
diff --git a/include/linux/vermagic.h b/include/linux/vermagic.h
index 939ceabcaf06..335c360d4f9b 100644
--- a/include/linux/vermagic.h
+++ b/include/linux/vermagic.h
@@ -33,7 +33,6 @@
 #define MODULE_VERMAGIC_MODVERSIONS ""
 #endif
 #ifdef RANDSTRUCT
-#include <generated/randstruct_hash.h>
 #define MODULE_RANDSTRUCT "RANDSTRUCT_" RANDSTRUCT_HASHED_SEED
 #else
 #define MODULE_RANDSTRUCT
diff --git a/scripts/basic/Makefile b/scripts/basic/Makefile
index dd289a6725ac..31637ce4dc5c 100644
--- a/scripts/basic/Makefile
+++ b/scripts/basic/Makefile
@@ -8,9 +8,10 @@ hostprogs-always-y	+= fixdep
 # before running a Clang kernel build.
 gen-randstruct-seed	:= $(srctree)/scripts/gen-randstruct-seed.sh
 quiet_cmd_create_randstruct_seed = GENSEED $@
-cmd_create_randstruct_seed = \
-	$(CONFIG_SHELL) $(gen-randstruct-seed) \
-		$@ $(objtree)/include/generated/randstruct_hash.h
-$(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
+      cmd_create_randstruct_seed = $(CONFIG_SHELL) $(gen-randstruct-seed) \
+		$(obj)/randstruct.seed $(obj)/randstruct_hash.h
+
+$(obj)/randstruct_hash.h $(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
 	$(call if_changed,create_randstruct_seed)
-always-$(CONFIG_RANDSTRUCT) += randstruct.seed
+
+always-$(CONFIG_RANDSTRUCT) += randstruct.seed randstruct_hash.h
-- 
2.34.1

[PATCH v2 3/3] integer-wrap: Force full rebuild when .scl file changes

[Kees Cook]

Since the integer wrapping sanitizer's behavior depends on its
associated .scl file, we must force a full rebuild if the file changes.
Universally include a synthetic header file that is rebuilt when the
.scl file changes, via compiler-version.h, since using "-include ..." is
not possible in the case of having compiler flags removed via
"filter-out" (which would remove all instances of "-include").

Signed-off-by: Kees Cook <kees@kernel.org>
---
Cc: Masahiro Yamada <masahiroy@kernel.org>
Cc: Justin Stitt <justinstitt@google.com>
Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Nicolas Schier <nicolas.schier@linux.dev>
Cc: Marco Elver <elver@google.com>
Cc: Andrey Konovalov <andreyknvl@gmail.com>
Cc: Andrey Ryabinin <ryabinin.a.a@gmail.com>
Cc: <linux-kbuild@vger.kernel.org>
Cc: <kasan-dev@googlegroups.com>
Cc: <linux-hardening@vger.kernel.org>
---
 include/linux/compiler-version.h | 3 +++
 scripts/Makefile.ubsan           | 1 +
 scripts/basic/Makefile           | 9 +++++++++
 3 files changed, 13 insertions(+)

diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
index 05d555320a0f..9d6b1890ffc7 100644
--- a/include/linux/compiler-version.h
+++ b/include/linux/compiler-version.h
@@ -19,3 +19,6 @@
 #ifdef RANDSTRUCT
 #include "randstruct_hash.h"
 #endif
+#ifdef INTEGER_WRAP
+#include "integer-wrap.h"
+#endif
diff --git a/scripts/Makefile.ubsan b/scripts/Makefile.ubsan
index 9e35198edbf0..653f7117819c 100644
--- a/scripts/Makefile.ubsan
+++ b/scripts/Makefile.ubsan
@@ -15,6 +15,7 @@ ubsan-cflags-$(CONFIG_UBSAN_TRAP)		+= $(call cc-option,-fsanitize-trap=undefined
 export CFLAGS_UBSAN := $(ubsan-cflags-y)
 
 ubsan-integer-wrap-cflags-$(CONFIG_UBSAN_INTEGER_WRAP)     +=	\
+	-DINTEGER_WRAP						\
 	-fsanitize-undefined-ignore-overflow-pattern=all	\
 	-fsanitize=signed-integer-overflow			\
 	-fsanitize=unsigned-integer-overflow			\
diff --git a/scripts/basic/Makefile b/scripts/basic/Makefile
index 31637ce4dc5c..04f5620a3f8b 100644
--- a/scripts/basic/Makefile
+++ b/scripts/basic/Makefile
@@ -15,3 +15,12 @@ $(obj)/randstruct_hash.h $(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
 	$(call if_changed,create_randstruct_seed)
 
 always-$(CONFIG_RANDSTRUCT) += randstruct.seed randstruct_hash.h
+
+# integer-wrap: if the .scl file changes, we need to do a full rebuild.
+quiet_cmd_integer_wrap_updated = UPDATE  $@
+      cmd_integer_wrap_updated = echo '/* $^ */' > $(obj)/integer-wrap.h
+
+$(obj)/integer-wrap.h: $(srctree)/scripts/integer-wrap-ignore.scl FORCE
+	$(call if_changed,integer_wrap_updated)
+
+always-$(CONFIG_UBSAN_INTEGER_WRAP) += integer-wrap.h
-- 
2.34.1

Re: [PATCH v2 1/3] gcc-plugins: Force full rebuild when plugins change

[Masahiro Yamada]

On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
>
> There was no dependency between the plugins changing and the rest of the
> kernel being built. Enforce this by including a synthetic header file
> when using plugins, that is regenerated any time the plugins are built.
>
> This cannot be included via '-include ...' because Makefiles use the
> "filter-out" string function, which removes individual words. Removing
> all instances of "-include" from the CFLAGS will cause a lot of
> problems. :)
>
> Instead, use -I to include the gcc-plugins directory, and depend on the
> new -DGCC_PLUGINS_ENABLED flag to include the generated header file via
> include/linux/compiler-version.h, which is already being used to control
> full rebuilds. The UM build requires that the -I be explicitly added.
>
> Signed-off-by: Kees Cook <kees@kernel.org>
> ---
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Nathan Chancellor <nathan@kernel.org>
> Cc: Nicolas Schier <nicolas.schier@linux.dev>
> Cc: <linux-hardening@vger.kernel.org>
> Cc: <linux-kbuild@vger.kernel.org>
> ---
>  arch/um/Makefile                 | 1 +
>  include/linux/compiler-version.h | 4 ++++
>  scripts/Makefile.gcc-plugins     | 2 +-
>  scripts/gcc-plugins/Makefile     | 8 ++++++++
>  4 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/arch/um/Makefile b/arch/um/Makefile
> index 1d36a613aad8..8cc0f22ebefa 100644
> --- a/arch/um/Makefile
> +++ b/arch/um/Makefile
> @@ -72,6 +72,7 @@ USER_CFLAGS = $(patsubst $(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \
>                 $(ARCH_INCLUDE) $(MODE_INCLUDE) $(filter -I%,$(CFLAGS)) \
>                 -D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \
>                 -idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \
> +               -I$(objtree)/scripts/gcc-plugins \
>                 -include $(srctree)/include/linux/compiler-version.h \
>                 -include $(srctree)/include/linux/kconfig.h
>
> diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
> index 573fa85b6c0c..08943df04ebb 100644
> --- a/include/linux/compiler-version.h
> +++ b/include/linux/compiler-version.h
> @@ -12,3 +12,7 @@
>   * and add dependency on include/config/CC_VERSION_TEXT, which is touched
>   * by Kconfig when the version string from the compiler changes.
>   */
> +
> +#ifdef GCC_PLUGINS_ENABLED
> +#include "gcc-plugins-deps.h"
> +#endif
> diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
> index 5b8a8378ca8a..468bb8faa9d1 100644
> --- a/scripts/Makefile.gcc-plugins
> +++ b/scripts/Makefile.gcc-plugins
> @@ -38,7 +38,7 @@ export DISABLE_STACKLEAK_PLUGIN
>
>  # All the plugin CFLAGS are collected here in case a build target needs to
>  # filter them out of the KBUILD_CFLAGS.
> -GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y))
> +GCC_PLUGINS_CFLAGS := $(strip $(addprefix -fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y)) $(gcc-plugin-cflags-y)) -I$(objtree)/scripts/gcc-plugins -DGCC_PLUGINS_ENABLED


This still relies on no-space after the -I option.



>  export GCC_PLUGINS_CFLAGS
>
>  # Add the flags to the build!
> diff --git a/scripts/gcc-plugins/Makefile b/scripts/gcc-plugins/Makefile
> index 320afd3cf8e8..24671d39ec90 100644
> --- a/scripts/gcc-plugins/Makefile
> +++ b/scripts/gcc-plugins/Makefile
> @@ -66,3 +66,11 @@ quiet_cmd_plugin_cxx_o_c = HOSTCXX $@
>
>  $(plugin-objs): $(obj)/%.o: $(src)/%.c FORCE
>         $(call if_changed_dep,plugin_cxx_o_c)
> +
> +quiet_cmd_gcc_plugins_updated = UPDATE  $@
> +      cmd_gcc_plugins_updated = echo '/* $^ */' > $(obj)/gcc-plugins-deps.h

I think 'touch' should be enough.

If some plugins are disabled, it is detected by the normal if_changed rule.


> +
> +$(obj)/gcc-plugins-deps.h: $(plugin-single) $(plugin-multi) FORCE
> +       $(call if_changed,gcc_plugins_updated)
> +
> +always-y += gcc-plugins-deps.h
> --
> 2.34.1
>


I think it is simpler to place the header
in include/generated/.

I attached my suggestion below:










diff --git a/arch/um/Makefile b/arch/um/Makefile
index 8cc0f22ebefa..1d36a613aad8 100644
--- a/arch/um/Makefile
+++ b/arch/um/Makefile
@@ -72,7 +72,6 @@ USER_CFLAGS = $(patsubst
$(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \
                $(ARCH_INCLUDE) $(MODE_INCLUDE) $(filter -I%,$(CFLAGS)) \
                -D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \
                -idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \
-               -I$(objtree)/scripts/gcc-plugins \
                -include $(srctree)/include/linux/compiler-version.h \
                -include $(srctree)/include/linux/kconfig.h

diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
index 08943df04ebb..ea3d533dc04a 100644
--- a/include/linux/compiler-version.h
+++ b/include/linux/compiler-version.h
@@ -14,5 +14,5 @@
  */

 #ifdef GCC_PLUGINS_ENABLED
-#include "gcc-plugins-deps.h"
+#include <generated/gcc-plugins-deps.h>
 #endif
diff --git a/scripts/Makefile.gcc-plugins b/scripts/Makefile.gcc-plugins
index 67b045a66157..f9b51c2c2158 100644
--- a/scripts/Makefile.gcc-plugins
+++ b/scripts/Makefile.gcc-plugins
@@ -44,7 +44,7 @@ export DISABLE_ARM_SSP_PER_TASK_PLUGIN

 # All the plugin CFLAGS are collected here in case a build target needs to
 # filter them out of the KBUILD_CFLAGS.
-GCC_PLUGINS_CFLAGS := $(strip $(addprefix
-fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y))
$(gcc-plugin-cflags-y)) -I$(objtree)/scripts/gcc-plugins
-DGCC_PLUGINS_ENABLED
+GCC_PLUGINS_CFLAGS := $(strip $(addprefix
-fplugin=$(objtree)/scripts/gcc-plugins/, $(gcc-plugin-y))
$(gcc-plugin-cflags-y)) -DGCC_PLUGINS_ENABLED
 export GCC_PLUGINS_CFLAGS

 # Add the flags to the build!
diff --git a/scripts/gcc-plugins/Makefile b/scripts/gcc-plugins/Makefile
index 24671d39ec90..b354c0f9f66d 100644
--- a/scripts/gcc-plugins/Makefile
+++ b/scripts/gcc-plugins/Makefile
@@ -67,10 +67,10 @@ quiet_cmd_plugin_cxx_o_c = HOSTCXX $@
 $(plugin-objs): $(obj)/%.o: $(src)/%.c FORCE
        $(call if_changed_dep,plugin_cxx_o_c)

-quiet_cmd_gcc_plugins_updated = UPDATE  $@
-      cmd_gcc_plugins_updated = echo '/* $^ */' > $(obj)/gcc-plugins-deps.h
+quiet_cmd_gcc_plugins_updated = TOUCH   $@
+      cmd_gcc_plugins_updated = touch $@

-$(obj)/gcc-plugins-deps.h: $(plugin-single) $(plugin-multi) FORCE
+$(obj)/../../include/generated/gcc-plugins-deps.h: $(plugin-single)
$(plugin-multi) FORCE
        $(call if_changed,gcc_plugins_updated)

-always-y += gcc-plugins-deps.h
+always-y += ../../include/generated/gcc-plugins-deps.h




--
Best Regards
Masahiro Yamada

Re: [PATCH v2 2/3] randstruct: Force full rebuild when seed changes

[Masahiro Yamada]

On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
>
> While the randstruct GCC plugin was being rebuilt if the randstruct
> seed changed, Clangs build did not notice the change. Include the hash
> header directly so that it becomes a universal build dependency and full
> rebuilds will happen if it changes.
>
> Since we cannot use "-include ..." as the randstruct flags are removed
> via "filter-out" (which would cause all instances of "-include" to be
> removed), use the existing -DRANDSTRUCT to control the header inclusion
> via include/linux/compiler-version.h. Universally add a -I for the
> scripts/basic directory, where header exists. The UM build requires that
> the -I be explicitly added.
>
> Signed-off-by: Kees Cook <kees@kernel.org>
> ---
> Cc: Masahiro Yamada <masahiroy@kernel.org>
> Cc: Nathan Chancellor <nathan@kernel.org>
> Cc: Nicolas Schier <nicolas.schier@linux.dev>
> Cc: Petr Pavlu <petr.pavlu@suse.com>
> Cc: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: <linux-kbuild@vger.kernel.org>
> ---
>  Makefile                         |  1 +
>  arch/um/Makefile                 |  1 +
>  include/linux/compiler-version.h |  3 +++
>  include/linux/vermagic.h         |  1 -
>  scripts/basic/Makefile           | 11 ++++++-----
>  5 files changed, 11 insertions(+), 6 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 5aa9ee52a765..cef652227843 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -567,6 +567,7 @@ LINUXINCLUDE    := \
>                 -I$(objtree)/arch/$(SRCARCH)/include/generated \
>                 -I$(srctree)/include \
>                 -I$(objtree)/include \
> +               -I$(objtree)/scripts/basic \


Now you are adding random header search paths everywhere.
This is very hacky.


I recommend keeping <generated/randstruct_hash.h>

Then,  -I$(objtree)/scripts/basic is unneeded.


>                 $(USERINCLUDE)
>
>  KBUILD_AFLAGS   := -D__ASSEMBLY__ -fno-PIE
> diff --git a/arch/um/Makefile b/arch/um/Makefile
> index 8cc0f22ebefa..38f6024e75d7 100644
> --- a/arch/um/Makefile
> +++ b/arch/um/Makefile
> @@ -73,6 +73,7 @@ USER_CFLAGS = $(patsubst $(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \
>                 -D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \
>                 -idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \
>                 -I$(objtree)/scripts/gcc-plugins \
> +               -I$(objtree)/scripts/basic \
>                 -include $(srctree)/include/linux/compiler-version.h \
>                 -include $(srctree)/include/linux/kconfig.h
>
> diff --git a/include/linux/compiler-version.h b/include/linux/compiler-version.h
> index 08943df04ebb..05d555320a0f 100644
> --- a/include/linux/compiler-version.h
> +++ b/include/linux/compiler-version.h
> @@ -16,3 +16,6 @@
>  #ifdef GCC_PLUGINS_ENABLED
>  #include "gcc-plugins-deps.h"
>  #endif
> +#ifdef RANDSTRUCT
> +#include "randstruct_hash.h"
> +#endif
> diff --git a/include/linux/vermagic.h b/include/linux/vermagic.h
> index 939ceabcaf06..335c360d4f9b 100644
> --- a/include/linux/vermagic.h
> +++ b/include/linux/vermagic.h
> @@ -33,7 +33,6 @@
>  #define MODULE_VERMAGIC_MODVERSIONS ""
>  #endif
>  #ifdef RANDSTRUCT
> -#include <generated/randstruct_hash.h>
>  #define MODULE_RANDSTRUCT "RANDSTRUCT_" RANDSTRUCT_HASHED_SEED
>  #else
>  #define MODULE_RANDSTRUCT
> diff --git a/scripts/basic/Makefile b/scripts/basic/Makefile
> index dd289a6725ac..31637ce4dc5c 100644
> --- a/scripts/basic/Makefile
> +++ b/scripts/basic/Makefile
> @@ -8,9 +8,10 @@ hostprogs-always-y     += fixdep
>  # before running a Clang kernel build.
>  gen-randstruct-seed    := $(srctree)/scripts/gen-randstruct-seed.sh
>  quiet_cmd_create_randstruct_seed = GENSEED $@
> -cmd_create_randstruct_seed = \
> -       $(CONFIG_SHELL) $(gen-randstruct-seed) \
> -               $@ $(objtree)/include/generated/randstruct_hash.h
> -$(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
> +      cmd_create_randstruct_seed = $(CONFIG_SHELL) $(gen-randstruct-seed) \
> +               $(obj)/randstruct.seed $(obj)/randstruct_hash.h
> +
> +$(obj)/randstruct_hash.h $(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
>         $(call if_changed,create_randstruct_seed)


This is wrong.


$(obj)/randstruct_hash.h $(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
         $(call if_changed,create_randstruct_seed)

is equivalent to:

$(obj)/randstruct_hash.h: $(gen-randstruct-seed) FORCE
         $(call if_changed,create_randstruct_seed)

$(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
         $(call if_changed,create_randstruct_seed)


So, this rule is executed twice; for randstruct_hash.h and for randstruct.seed

randstruct_hash.h and randstruct.seed will contain different hash values.

I recommend keeping the current code.













--
Best Regards
Masahiro Yamada

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Masahiro Yamada]

On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
>
>  v2:
>   - switch from -include to -I with a -D gated include compiler-version.h
>  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/


What do you think of my patch as a prerequisite?
https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
Perhaps, can you implement this series more simply?

My idea is to touch a single include/generated/global-rebuild.h
rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.

When the file is touched, the entire kernel source tree will be rebuilt.
This may rebuild more than needed (e.g. vdso) but I do not think
it is a big deal.




-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 1/3] gcc-plugins: Force full rebuild when plugins change

[Kees Cook]

On Sat, May 03, 2025 at 03:12:23PM +0900, Masahiro Yamada wrote:
> On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > +quiet_cmd_gcc_plugins_updated = UPDATE  $@
> > +      cmd_gcc_plugins_updated = echo '/* $^ */' > $(obj)/gcc-plugins-deps.h
> 
> I think 'touch' should be enough.
> 
> If some plugins are disabled, it is detected by the normal if_changed rule.

I kind of likely having the active plugins show up in there, but yes,
"touch" is enough (it's what I started with originally).

> > +$(obj)/gcc-plugins-deps.h: $(plugin-single) $(plugin-multi) FORCE
> > +       $(call if_changed,gcc_plugins_updated)
> > +
> > +always-y += gcc-plugins-deps.h
> > --
> > 2.34.1
> >
> 
> 
> I think it is simpler to place the header
> in include/generated/.

I couldn't figure out how to do this, but thankfully you did! :)

> I attached my suggestion below:
> [...]
> -quiet_cmd_gcc_plugins_updated = UPDATE  $@
> -      cmd_gcc_plugins_updated = echo '/* $^ */' > $(obj)/gcc-plugins-deps.h
> +quiet_cmd_gcc_plugins_updated = TOUCH   $@
> +      cmd_gcc_plugins_updated = touch $@
> 
> -$(obj)/gcc-plugins-deps.h: $(plugin-single) $(plugin-multi) FORCE
> +$(obj)/../../include/generated/gcc-plugins-deps.h: $(plugin-single)
> $(plugin-multi) FORCE
>         $(call if_changed,gcc_plugins_updated)
> 
> -always-y += gcc-plugins-deps.h
> +always-y += ../../include/generated/gcc-plugins-deps.h

Aaagh, thank you! I didn't even consider trying ".." for targets!

Yes, this is SO much better! I will read your other replies and work on
v3...

-- 
Kees Cook

Re: [PATCH v2 2/3] randstruct: Force full rebuild when seed changes

[Kees Cook]

On Sat, May 03, 2025 at 03:13:06PM +0900, Masahiro Yamada wrote:
> On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > +$(obj)/randstruct_hash.h $(obj)/randstruct.seed: $(gen-randstruct-seed) FORCE
> >         $(call if_changed,create_randstruct_seed)
> [...]
> So, this rule is executed twice; for randstruct_hash.h and for randstruct.seed
> 
> randstruct_hash.h and randstruct.seed will contain different hash values.
> 
> I recommend keeping the current code.

Agreed -- getting stuff into generated is much preferred. Thank you!

-- 
Kees Cook

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Kees Cook]

On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> >
> >  v2:
> >   - switch from -include to -I with a -D gated include compiler-version.h
> >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> 
> 
> What do you think of my patch as a prerequisite?
> https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> Perhaps, can you implement this series more simply?
> 
> My idea is to touch a single include/generated/global-rebuild.h
> rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> 
> When the file is touched, the entire kernel source tree will be rebuilt.
> This may rebuild more than needed (e.g. vdso) but I do not think
> it is a big deal.

This is roughly where I started when trying to implement this, but I
didn't like the ergonomics of needing to scatter "touch" calls all over,
which was especially difficult for targets that shared a build rule but
may not all need to trigger a global rebuild. But what ultimately pushed
me away from it was when I needed to notice if a non-built source file
changed (the Clang .scl file), and I saw that I need to be dependency
driven rather than target driven. (Though perhaps there is a way to
address this with your global-rebuild.h?)

As far as doing a full rebuild, if it had been available last week, I
probably would have used it, but now given the work that Nicolas, you,
and I have put into this, we have a viable way (I think) to make this
more specific. It does end up being a waste of time/resources to rebuild
stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
does add up when I'm iterating on something that keeps triggering a full
rebuild. We already have to do the argument filtering for targets that
don't want randstruct, etc, so why not capitalize on that and make the
rebuild avoid those files too?

So, I think the global-rebuild.h idea is a good one (though I think it
should maybe be included in compiler-version.h just to avoid yet more
compiler command line arguments), I'd really like to try to have the
specific dependency-based way to get it done.

I'll send a v3, and see what you think?

-Kees

-- 
Kees Cook

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Masahiro Yamada]

On Sun, May 4, 2025 at 2:37 AM Kees Cook <kees@kernel.org> wrote:
>
> On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> > On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > >
> > >  v2:
> > >   - switch from -include to -I with a -D gated include compiler-version.h
> > >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> >
> >
> > What do you think of my patch as a prerequisite?
> > https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> > Perhaps, can you implement this series more simply?
> >
> > My idea is to touch a single include/generated/global-rebuild.h
> > rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> >
> > When the file is touched, the entire kernel source tree will be rebuilt.
> > This may rebuild more than needed (e.g. vdso) but I do not think
> > it is a big deal.
>
> This is roughly where I started when trying to implement this, but I
> didn't like the ergonomics of needing to scatter "touch" calls all over,
> which was especially difficult for targets that shared a build rule but
> may not all need to trigger a global rebuild. But what ultimately pushed
> me away from it was when I needed to notice if a non-built source file
> changed (the Clang .scl file), and I saw that I need to be dependency
> driven rather than target driven. (Though perhaps there is a way to
> address this with your global-rebuild.h?)
>
> As far as doing a full rebuild, if it had been available last week, I
> probably would have used it, but now given the work that Nicolas, you,
> and I have put into this, we have a viable way (I think) to make this
> more specific. It does end up being a waste of time/resources to rebuild
> stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
> does add up when I'm iterating on something that keeps triggering a full
> rebuild. We already have to do the argument filtering for targets that
> don't want randstruct, etc, so why not capitalize on that and make the
> rebuild avoid those files too?


efi-stub, vdso are very small.

Unless this turns out to be painful, I prefer
a simpler implementation.

You will see how .scl file is handled.

See the below code:


diff --git a/Kbuild b/Kbuild
index f327ca86990c..85747239314c 100644
--- a/Kbuild
+++ b/Kbuild
@@ -67,10 +67,20 @@ targets += $(atomic-checks)
 $(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
        $(call if_changed,check_sha1)

+rebuild-$(CONFIG_GCC_PLUGINS)          += $(addprefix
scripts/gcc-plugins/, $(GCC_PLUGIN))
+rebuild-$(CONFIG_RANDSTRUCT)           += include/generated/randstruct_hash.h
+rebuild-$(CONFIG_UBSAN_INTEGER_WRAP)   += scripts/integer-wrap-ignore.scl
+
+quiet_cmd_touch = TOUCH   $@
+      cmd_touch = touch $@
+
+include/generated/global-rebuild.h: $(rebuild-y)
+       $(call cmd,touch)
+
 # A phony target that depends on all the preparation targets

 PHONY += prepare
-prepare: $(offsets-file) missing-syscalls $(atomic-checks)
+prepare: $(offsets-file) missing-syscalls $(atomic-checks)
include/generated/global-rebuild.h
        @:

 # Ordinary directory descending
diff --git a/Makefile b/Makefile
index b29cc321ffd9..f963a72b0761 100644
--- a/Makefile
+++ b/Makefile
@@ -558,7 +558,8 @@ USERINCLUDE    := \
                -I$(srctree)/include/uapi \
                -I$(objtree)/include/generated/uapi \
                 -include $(srctree)/include/linux/compiler-version.h \
-                -include $(srctree)/include/linux/kconfig.h
+                -include $(srctree)/include/linux/kconfig.h \
+                -include $(objtree)/include/generated/global-rebuild.h

 # Use LINUXINCLUDE when you must reference the include/ directory.
 # Needed to be compatible with the O= option
@@ -1250,6 +1251,12 @@ endif
 include/config/kernel.release: FORCE
        $(call filechk,kernel.release)

+quiet_cmd_touch = TOUCH   $@
+      cmd_touch = touch $@
+
+include/generated/global-rebuild.h:
+       $(call cmd,touch)
+
 # Additional helpers built in scripts/
 # Carefully list dependencies so we do not try to build scripts twice
 # in parallel
@@ -1266,6 +1273,7 @@ scripts: scripts_basic scripts_dtc
 PHONY += prepare archprepare

 archprepare: outputmakefile archheaders archscripts scripts
include/config/kernel.release \
+       include/generated/global-rebuild.h \
        asm-generic $(version_h) include/generated/utsrelease.h \
        include/generated/compile.h include/generated/autoconf.h \
        include/generated/rustc_cfg remove-stale-files
diff --git a/arch/um/Makefile b/arch/um/Makefile
index 1d36a613aad8..f564a26c1364 100644
--- a/arch/um/Makefile
+++ b/arch/um/Makefile
@@ -73,7 +73,8 @@ USER_CFLAGS = $(patsubst
$(KERNEL_DEFINES),,$(patsubst -I%,,$(KBUILD_CFLAGS))) \
                -D_FILE_OFFSET_BITS=64 -idirafter $(srctree)/include \
                -idirafter $(objtree)/include -D__KERNEL__ -D__UM_HOST__ \
                -include $(srctree)/include/linux/compiler-version.h \
-               -include $(srctree)/include/linux/kconfig.h
+               -include $(srctree)/include/linux/kconfig.h \
+               -include $(objtree)/include/generated/global-rebuild.h

 #This will adjust *FLAGS accordingly to the platform.
 include $(srctree)/$(ARCH_DIR)/Makefile-os-Linux









-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Kees Cook]

On Fri, May 09, 2025 at 01:44:09AM +0900, Masahiro Yamada wrote:
> On Sun, May 4, 2025 at 2:37 AM Kees Cook <kees@kernel.org> wrote:
> >
> > On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> > > On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > > >
> > > >  v2:
> > > >   - switch from -include to -I with a -D gated include compiler-version.h
> > > >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> > >
> > >
> > > What do you think of my patch as a prerequisite?
> > > https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> > > Perhaps, can you implement this series more simply?
> > >
> > > My idea is to touch a single include/generated/global-rebuild.h
> > > rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> > >
> > > When the file is touched, the entire kernel source tree will be rebuilt.
> > > This may rebuild more than needed (e.g. vdso) but I do not think
> > > it is a big deal.
> >
> > This is roughly where I started when trying to implement this, but I
> > didn't like the ergonomics of needing to scatter "touch" calls all over,
> > which was especially difficult for targets that shared a build rule but
> > may not all need to trigger a global rebuild. But what ultimately pushed
> > me away from it was when I needed to notice if a non-built source file
> > changed (the Clang .scl file), and I saw that I need to be dependency
> > driven rather than target driven. (Though perhaps there is a way to
> > address this with your global-rebuild.h?)
> >
> > As far as doing a full rebuild, if it had been available last week, I
> > probably would have used it, but now given the work that Nicolas, you,
> > and I have put into this, we have a viable way (I think) to make this
> > more specific. It does end up being a waste of time/resources to rebuild
> > stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
> > does add up when I'm iterating on something that keeps triggering a full
> > rebuild. We already have to do the argument filtering for targets that
> > don't want randstruct, etc, so why not capitalize on that and make the
> > rebuild avoid those files too?
> 
> 
> efi-stub, vdso are very small.
> 
> Unless this turns out to be painful, I prefer
> a simpler implementation.
> 
> You will see how .scl file is handled.
> 
> See the below code:
> 
> 
> diff --git a/Kbuild b/Kbuild
> index f327ca86990c..85747239314c 100644
> --- a/Kbuild
> +++ b/Kbuild
> @@ -67,10 +67,20 @@ targets += $(atomic-checks)
>  $(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
>         $(call if_changed,check_sha1)
> 
> +rebuild-$(CONFIG_GCC_PLUGINS)          += $(addprefix
> scripts/gcc-plugins/, $(GCC_PLUGIN))
> +rebuild-$(CONFIG_RANDSTRUCT)           += include/generated/randstruct_hash.h

These are in $(objtree)

> +rebuild-$(CONFIG_UBSAN_INTEGER_WRAP)   += scripts/integer-wrap-ignore.scl

This is in $(srctree)

> +
> +quiet_cmd_touch = TOUCH   $@
> +      cmd_touch = touch $@
> +
> +include/generated/global-rebuild.h: $(rebuild-y)
> +       $(call cmd,touch)

Is this rule going to find the right versions of the dependencies?

> --- a/Makefile
> +++ b/Makefile
> @@ -558,7 +558,8 @@ USERINCLUDE    := \
>                 -I$(srctree)/include/uapi \
>                 -I$(objtree)/include/generated/uapi \
>                  -include $(srctree)/include/linux/compiler-version.h \
> -                -include $(srctree)/include/linux/kconfig.h
> +                -include $(srctree)/include/linux/kconfig.h \
> +                -include $(objtree)/include/generated/global-rebuild.h

Instead of adding a new file, why not just touch compiler-version.h?

But whatever the case, sure, I can live with this. :)

-Kees

-- 
Kees Cook

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Masahiro Yamada]

On Fri, May 9, 2025 at 1:56 AM Kees Cook <kees@kernel.org> wrote:
>
> On Fri, May 09, 2025 at 01:44:09AM +0900, Masahiro Yamada wrote:
> > On Sun, May 4, 2025 at 2:37 AM Kees Cook <kees@kernel.org> wrote:
> > >
> > > On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> > > > On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > > > >
> > > > >  v2:
> > > > >   - switch from -include to -I with a -D gated include compiler-version.h
> > > > >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> > > >
> > > >
> > > > What do you think of my patch as a prerequisite?
> > > > https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> > > > Perhaps, can you implement this series more simply?
> > > >
> > > > My idea is to touch a single include/generated/global-rebuild.h
> > > > rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> > > >
> > > > When the file is touched, the entire kernel source tree will be rebuilt.
> > > > This may rebuild more than needed (e.g. vdso) but I do not think
> > > > it is a big deal.
> > >
> > > This is roughly where I started when trying to implement this, but I
> > > didn't like the ergonomics of needing to scatter "touch" calls all over,
> > > which was especially difficult for targets that shared a build rule but
> > > may not all need to trigger a global rebuild. But what ultimately pushed
> > > me away from it was when I needed to notice if a non-built source file
> > > changed (the Clang .scl file), and I saw that I need to be dependency
> > > driven rather than target driven. (Though perhaps there is a way to
> > > address this with your global-rebuild.h?)
> > >
> > > As far as doing a full rebuild, if it had been available last week, I
> > > probably would have used it, but now given the work that Nicolas, you,
> > > and I have put into this, we have a viable way (I think) to make this
> > > more specific. It does end up being a waste of time/resources to rebuild
> > > stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
> > > does add up when I'm iterating on something that keeps triggering a full
> > > rebuild. We already have to do the argument filtering for targets that
> > > don't want randstruct, etc, so why not capitalize on that and make the
> > > rebuild avoid those files too?
> >
> >
> > efi-stub, vdso are very small.
> >
> > Unless this turns out to be painful, I prefer
> > a simpler implementation.
> >
> > You will see how .scl file is handled.
> >
> > See the below code:
> >
> >
> > diff --git a/Kbuild b/Kbuild
> > index f327ca86990c..85747239314c 100644
> > --- a/Kbuild
> > +++ b/Kbuild
> > @@ -67,10 +67,20 @@ targets += $(atomic-checks)
> >  $(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
> >         $(call if_changed,check_sha1)
> >
> > +rebuild-$(CONFIG_GCC_PLUGINS)          += $(addprefix
> > scripts/gcc-plugins/, $(GCC_PLUGIN))
> > +rebuild-$(CONFIG_RANDSTRUCT)           += include/generated/randstruct_hash.h
>
> These are in $(objtree)

Yes.

> > +rebuild-$(CONFIG_UBSAN_INTEGER_WRAP)   += scripts/integer-wrap-ignore.scl
>
> This is in $(srctree)

Yes.

> > +
> > +quiet_cmd_touch = TOUCH   $@
> > +      cmd_touch = touch $@
> > +
> > +include/generated/global-rebuild.h: $(rebuild-y)
> > +       $(call cmd,touch)
>
> Is this rule going to find the right versions of the dependencies?

I think so, but please test it.


> > --- a/Makefile
> > +++ b/Makefile
> > @@ -558,7 +558,8 @@ USERINCLUDE    := \
> >                 -I$(srctree)/include/uapi \
> >                 -I$(objtree)/include/generated/uapi \
> >                  -include $(srctree)/include/linux/compiler-version.h \
> > -                -include $(srctree)/include/linux/kconfig.h
> > +                -include $(srctree)/include/linux/kconfig.h \
> > +                -include $(objtree)/include/generated/global-rebuild.h
>
> Instead of adding a new file, why not just touch compiler-version.h?

Because the compiler-version.h is in $(srctree), which might be
in the read-only file system.


> But whatever the case, sure, I can live with this. :)
>
> -Kees
>
> --
> Kees Cook



-- 
Best Regards
Masahiro Yamada

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Kees Cook]

On Fri, May 09, 2025 at 08:13:18AM +0900, Masahiro Yamada wrote:
> On Fri, May 9, 2025 at 1:56 AM Kees Cook <kees@kernel.org> wrote:
> >
> > On Fri, May 09, 2025 at 01:44:09AM +0900, Masahiro Yamada wrote:
> > > On Sun, May 4, 2025 at 2:37 AM Kees Cook <kees@kernel.org> wrote:
> > > >
> > > > On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> > > > > On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > > > > >
> > > > > >  v2:
> > > > > >   - switch from -include to -I with a -D gated include compiler-version.h
> > > > > >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> > > > >
> > > > >
> > > > > What do you think of my patch as a prerequisite?
> > > > > https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> > > > > Perhaps, can you implement this series more simply?
> > > > >
> > > > > My idea is to touch a single include/generated/global-rebuild.h
> > > > > rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> > > > >
> > > > > When the file is touched, the entire kernel source tree will be rebuilt.
> > > > > This may rebuild more than needed (e.g. vdso) but I do not think
> > > > > it is a big deal.
> > > >
> > > > This is roughly where I started when trying to implement this, but I
> > > > didn't like the ergonomics of needing to scatter "touch" calls all over,
> > > > which was especially difficult for targets that shared a build rule but
> > > > may not all need to trigger a global rebuild. But what ultimately pushed
> > > > me away from it was when I needed to notice if a non-built source file
> > > > changed (the Clang .scl file), and I saw that I need to be dependency
> > > > driven rather than target driven. (Though perhaps there is a way to
> > > > address this with your global-rebuild.h?)
> > > >
> > > > As far as doing a full rebuild, if it had been available last week, I
> > > > probably would have used it, but now given the work that Nicolas, you,
> > > > and I have put into this, we have a viable way (I think) to make this
> > > > more specific. It does end up being a waste of time/resources to rebuild
> > > > stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
> > > > does add up when I'm iterating on something that keeps triggering a full
> > > > rebuild. We already have to do the argument filtering for targets that
> > > > don't want randstruct, etc, so why not capitalize on that and make the
> > > > rebuild avoid those files too?
> > >
> > >
> > > efi-stub, vdso are very small.
> > >
> > > Unless this turns out to be painful, I prefer
> > > a simpler implementation.
> > >
> > > You will see how .scl file is handled.
> > >
> > > See the below code:
> > >
> > >
> > > diff --git a/Kbuild b/Kbuild
> > > index f327ca86990c..85747239314c 100644
> > > --- a/Kbuild
> > > +++ b/Kbuild
> > > @@ -67,10 +67,20 @@ targets += $(atomic-checks)
> > >  $(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
> > >         $(call if_changed,check_sha1)
> > >
> > > +rebuild-$(CONFIG_GCC_PLUGINS)          += $(addprefix
> > > scripts/gcc-plugins/, $(GCC_PLUGIN))
> > > +rebuild-$(CONFIG_RANDSTRUCT)           += include/generated/randstruct_hash.h
> >
> > These are in $(objtree)
> 
> Yes.
> 
> > > +rebuild-$(CONFIG_UBSAN_INTEGER_WRAP)   += scripts/integer-wrap-ignore.scl
> >
> > This is in $(srctree)
> 
> Yes.
> 
> > > +
> > > +quiet_cmd_touch = TOUCH   $@
> > > +      cmd_touch = touch $@
> > > +
> > > +include/generated/global-rebuild.h: $(rebuild-y)
> > > +       $(call cmd,touch)
> >
> > Is this rule going to find the right versions of the dependencies?
> 
> I think so, but please test it.

The patch was white-space damaged and wrapped, but I rebuilt it manually
and it mostly works. There still seems to be some ordering issues, as
some stuff gets rebuilt on a record build:

# Clean the tree and pick an "everything" build
$ make O=gcc-test clean allmodconfig -s

# Make a target normally
$ make O=gcc-test kernel/seccomp.o -s

# Touch a gcc plugin that was in .config
$ touch scripts/gcc-plugins/stackleak_plugin.c

# Build and a full rebuild is triggered (good)
$ make O=gcc-test kernel/seccomp.o
make[1]: Entering directory '/srv/code/gcc-test'
  GEN     Makefile
  DESCEND objtool
  HOSTCXX scripts/gcc-plugins/stackleak_plugin.so
  INSTALL libsubcmd_headers
  TOUCH   include/generated/global-rebuild.h
  CC      kernel/bounds.s
  CC      arch/x86/kernel/asm-offsets.s
  CALL    ../scripts/checksyscalls.sh
  CC      kernel/seccomp.o
make[1]: Leaving directory '/srv/code/gcc-test'

# Build again, but more stuff gets built
$ make O=gcc-test kernel/seccomp.o
make[1]: Entering directory '/srv/code/gcc-test'
  GEN     Makefile
  DESCEND objtool
  CC      scripts/mod/empty.o
  CC      scripts/mod/devicetable-offsets.s
  INSTALL libsubcmd_headers
  MKELF   scripts/mod/elfconfig.h
  HOSTCC  scripts/mod/modpost.o
  HOSTCC  scripts/mod/sumversion.o
  HOSTCC  scripts/mod/symsearch.o
  HOSTCC  scripts/mod/file2alias.o
  HOSTLD  scripts/mod/modpost
  CALL    ../scripts/checksyscalls.sh
make[1]: Leaving directory '/srv/code/gcc-test'

# Third time finally everything is stable
$ hmake O=gcc-test kernel/seccomp.o
make[1]: Entering directory '/srv/code/gcc-test'
  GEN     Makefile
  DESCEND objtool
  CALL    ../scripts/checksyscalls.sh
  INSTALL libsubcmd_headers
make[1]: Leaving directory '/srv/code/gcc-test'


Note that scripts/mod/* gets rebuilt on the second rebuild.


-- 
Kees Cook

Re: [PATCH v2 0/3] Detect changed compiler dependencies for full rebuild

[Masahiro Yamada]

On Fri, May 9, 2025 at 8:59 AM Kees Cook <kees@kernel.org> wrote:
>
> On Fri, May 09, 2025 at 08:13:18AM +0900, Masahiro Yamada wrote:
> > On Fri, May 9, 2025 at 1:56 AM Kees Cook <kees@kernel.org> wrote:
> > >
> > > On Fri, May 09, 2025 at 01:44:09AM +0900, Masahiro Yamada wrote:
> > > > On Sun, May 4, 2025 at 2:37 AM Kees Cook <kees@kernel.org> wrote:
> > > > >
> > > > > On Sat, May 03, 2025 at 06:39:28PM +0900, Masahiro Yamada wrote:
> > > > > > On Sat, May 3, 2025 at 7:54 AM Kees Cook <kees@kernel.org> wrote:
> > > > > > >
> > > > > > >  v2:
> > > > > > >   - switch from -include to -I with a -D gated include compiler-version.h
> > > > > > >  v1: https://lore.kernel.org/lkml/20250501193839.work.525-kees@kernel.org/
> > > > > >
> > > > > >
> > > > > > What do you think of my patch as a prerequisite?
> > > > > > https://lore.kernel.org/linux-kbuild/20250503084145.1994176-1-masahiroy@kernel.org/T/#u
> > > > > > Perhaps, can you implement this series more simply?
> > > > > >
> > > > > > My idea is to touch a single include/generated/global-rebuild.h
> > > > > > rather than multiple files such as gcc-plugins-deps.h, integer-wrap.h, etc.
> > > > > >
> > > > > > When the file is touched, the entire kernel source tree will be rebuilt.
> > > > > > This may rebuild more than needed (e.g. vdso) but I do not think
> > > > > > it is a big deal.
> > > > >
> > > > > This is roughly where I started when trying to implement this, but I
> > > > > didn't like the ergonomics of needing to scatter "touch" calls all over,
> > > > > which was especially difficult for targets that shared a build rule but
> > > > > may not all need to trigger a global rebuild. But what ultimately pushed
> > > > > me away from it was when I needed to notice if a non-built source file
> > > > > changed (the Clang .scl file), and I saw that I need to be dependency
> > > > > driven rather than target driven. (Though perhaps there is a way to
> > > > > address this with your global-rebuild.h?)
> > > > >
> > > > > As far as doing a full rebuild, if it had been available last week, I
> > > > > probably would have used it, but now given the work that Nicolas, you,
> > > > > and I have put into this, we have a viable way (I think) to make this
> > > > > more specific. It does end up being a waste of time/resources to rebuild
> > > > > stuff that doesn't need to be (efi-stub, vdso, boot code, etc), and that
> > > > > does add up when I'm iterating on something that keeps triggering a full
> > > > > rebuild. We already have to do the argument filtering for targets that
> > > > > don't want randstruct, etc, so why not capitalize on that and make the
> > > > > rebuild avoid those files too?
> > > >
> > > >
> > > > efi-stub, vdso are very small.
> > > >
> > > > Unless this turns out to be painful, I prefer
> > > > a simpler implementation.
> > > >
> > > > You will see how .scl file is handled.
> > > >
> > > > See the below code:
> > > >
> > > >
> > > > diff --git a/Kbuild b/Kbuild
> > > > index f327ca86990c..85747239314c 100644
> > > > --- a/Kbuild
> > > > +++ b/Kbuild
> > > > @@ -67,10 +67,20 @@ targets += $(atomic-checks)
> > > >  $(atomic-checks): $(obj)/.checked-%: include/linux/atomic/%  FORCE
> > > >         $(call if_changed,check_sha1)
> > > >
> > > > +rebuild-$(CONFIG_GCC_PLUGINS)          += $(addprefix
> > > > scripts/gcc-plugins/, $(GCC_PLUGIN))
> > > > +rebuild-$(CONFIG_RANDSTRUCT)           += include/generated/randstruct_hash.h
> > >
> > > These are in $(objtree)
> >
> > Yes.
> >
> > > > +rebuild-$(CONFIG_UBSAN_INTEGER_WRAP)   += scripts/integer-wrap-ignore.scl
> > >
> > > This is in $(srctree)
> >
> > Yes.
> >
> > > > +
> > > > +quiet_cmd_touch = TOUCH   $@
> > > > +      cmd_touch = touch $@
> > > > +
> > > > +include/generated/global-rebuild.h: $(rebuild-y)
> > > > +       $(call cmd,touch)
> > >
> > > Is this rule going to find the right versions of the dependencies?
> >
> > I think so, but please test it.
>
> The patch was white-space damaged and wrapped, but I rebuilt it manually
> and it mostly works. There still seems to be some ordering issues, as
> some stuff gets rebuilt on a record build:
>
> # Clean the tree and pick an "everything" build
> $ make O=gcc-test clean allmodconfig -s
>
> # Make a target normally
> $ make O=gcc-test kernel/seccomp.o -s
>
> # Touch a gcc plugin that was in .config
> $ touch scripts/gcc-plugins/stackleak_plugin.c
>
> # Build and a full rebuild is triggered (good)
> $ make O=gcc-test kernel/seccomp.o
> make[1]: Entering directory '/srv/code/gcc-test'
>   GEN     Makefile
>   DESCEND objtool
>   HOSTCXX scripts/gcc-plugins/stackleak_plugin.so
>   INSTALL libsubcmd_headers
>   TOUCH   include/generated/global-rebuild.h
>   CC      kernel/bounds.s
>   CC      arch/x86/kernel/asm-offsets.s
>   CALL    ../scripts/checksyscalls.sh
>   CC      kernel/seccomp.o
> make[1]: Leaving directory '/srv/code/gcc-test'
>
> # Build again, but more stuff gets built
> $ make O=gcc-test kernel/seccomp.o
> make[1]: Entering directory '/srv/code/gcc-test'
>   GEN     Makefile
>   DESCEND objtool
>   CC      scripts/mod/empty.o
>   CC      scripts/mod/devicetable-offsets.s
>   INSTALL libsubcmd_headers
>   MKELF   scripts/mod/elfconfig.h
>   HOSTCC  scripts/mod/modpost.o
>   HOSTCC  scripts/mod/sumversion.o
>   HOSTCC  scripts/mod/symsearch.o
>   HOSTCC  scripts/mod/file2alias.o
>   HOSTLD  scripts/mod/modpost
>   CALL    ../scripts/checksyscalls.sh
> make[1]: Leaving directory '/srv/code/gcc-test'
>
> # Third time finally everything is stable
> $ hmake O=gcc-test kernel/seccomp.o
> make[1]: Entering directory '/srv/code/gcc-test'
>   GEN     Makefile
>   DESCEND objtool
>   CALL    ../scripts/checksyscalls.sh
>   INSTALL libsubcmd_headers
> make[1]: Leaving directory '/srv/code/gcc-test'
>
>
> Note that scripts/mod/* gets rebuilt on the second rebuild.

Hmm.
OK, my code did not work.

I accept your patch set (although I am not a big fan
of the added complexity...)

Could you move the normalize_path macro
to scripts/Kbuild?


-- 
Best Regards
Masahiro Yamada