[PATCH 00/27] kbuild: yet another series of cleanups (modpost and LTO)

[PATCH 00/27] kbuild: yet another series of cleanups (modpost and LTO)

[Masahiro Yamada]

This is the third batch of cleanups in this development cycle.

This weekend, I wrote up the code I have been planning.

After a bunch of modpost refactoring, I got rid of the ugly code
in Makefiles.

With this, Kbuild will get back much simpler and cleaner.



Masahiro Yamada (27):
  modpost: use snprintf() instead of sprintf() for safety
  modpost: do not write out any file when error occurred
  modpost: remove stale comment about sym_add_exported()
  modpost: add a separate error for exported symbols without definition
  modpost: retrieve the module dependency and CRCs in check_exports()
  modpost: use bool type where appropriate
  modpost: import include/linux/list.h
  modpost: traverse modules in order
  modpost: add sym_add_unresolved() helper
  modpost: traverse unresolved symbols in order
  modpost: use doubly linked list for dump_lists
  modpost: move struct namespace_list to modpost.c
  modpost: traverse the namespace_list in order
  modpost: dump Module.symvers in the same order of modules.order
  modpost: move static EXPORT_SYMBOL check to check_exports()
  modpost: make multiple export error
  modpost: make sym_add_exported() always allocate a new symbol
  modpost: make sym_add_exported() a void function
  modpost: use hlist for hash table implementation
  modpost: mitigate false-negatives for static EXPORT_SYMBOL checks
  kbuild: record symbol versions in *.cmd files
  kbuild: generate a list of objects in vmlinux
  modpost: retrieve symbol versions by parsing *.cmd files
  modpost: generate linker script to collect symbol versions
  kbuild: embed symbol versions at final link of vmlinux or modules
  kbuild: stop generating *.symversions
  kbuild: do not create *.prelink.o for Clang LTO or IBT

 .gitignore                |   1 +
 Makefile                  |   1 +
 scripts/Kbuild.include    |   4 +
 scripts/Makefile.build    | 107 ++------
 scripts/Makefile.lib      |   7 -
 scripts/Makefile.modfinal |   6 +-
 scripts/Makefile.modpost  |   9 +-
 scripts/link-vmlinux.sh   |  38 ++-
 scripts/mod/file2alias.c  |   2 -
 scripts/mod/list.h        | 336 ++++++++++++++++++++++++
 scripts/mod/modpost.c     | 529 +++++++++++++++++++++++---------------
 scripts/mod/modpost.h     |  27 +-
 12 files changed, 731 insertions(+), 336 deletions(-)
 create mode 100644 scripts/mod/list.h

-- 
2.32.0

[PATCH 21/27] kbuild: record symbol versions in *.cmd files

[Masahiro Yamada]

When CONFIG_MODVERSIONS=y, the output from genksyms is saved in
separate *.symversions files, and will be used much later when
CONFIG_LTO_CLANG=y because it is impossible to update LLVM bit code
here.

This approach is not robust because:

 - *.symversions may or may not exist. If *.symversions does not
   exist, we never know if it is missing for legitimate reason
   (i.e. no EXPORT_SYMBOL) or something bad has happened (for
   example, the user accidentally deleted it). Once it occurs,
   it is not self-healing because *.symversions is generated
   as a side effect of the build rule of the object.

 - stale (i.e. invalid) *.symversions might be picked up if an
   object is generated in a non-ordinary way, and corresponding
   *.symversions (, which was generated by old builds) just happen
   to exist.

A more robust approach is to save symbol versions in *.cmd files
because:

 - *.cmd always exists (if the object is generated by if_changed
   rule or friends). Even if the user accidentally deletes it,
   it will be regenerated in the next build.

 - *.cmd is always re-generated when the object is updated. This
   avoid stale version information being picked up.

I will remove *.symversions later.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/Makefile.build | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index f6a506318795..e03e85c90b26 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -175,6 +175,8 @@ gen_symversions =								\
 	if $(NM) $@ 2>/dev/null | grep -q __ksymtab; then			\
 		$(call cmd_gensymtypes_$(1),$(KBUILD_SYMTYPES),$(@:.o=.symtypes)) \
 		    > $@.symversions;						\
+		echo >> $(dot-target).cmd ;					\
+		sed 's/\(.*\) = \(.*\);/$(pound)\1=\2/' $@.symversions >> $(dot-target).cmd ; \
 	else									\
 		rm -f $@.symversions;						\
 	fi
-- 
2.32.0

Re: [PATCH 21/27] kbuild: record symbol versions in *.cmd files

[Nicolas Schier]

On ma. 25. april 2022 kl. 04.08 +0000 Masahiro Yamada wrote:
> When CONFIG_MODVERSIONS=y, the output from genksyms is saved in
> separate *.symversions files, and will be used much later when
> CONFIG_LTO_CLANG=y because it is impossible to update LLVM bit code
> here.
> 
> This approach is not robust because:
> 
>  - *.symversions may or may not exist. If *.symversions does not
>    exist, we never know if it is missing for legitimate reason
>    (i.e. no EXPORT_SYMBOL) or something bad has happened (for
>    example, the user accidentally deleted it). Once it occurs,
>    it is not self-healing because *.symversions is generated
>    as a side effect of the build rule of the object.
> 
>  - stale (i.e. invalid) *.symversions might be picked up if an
>    object is generated in a non-ordinary way, and corresponding
>    *.symversions (, which was generated by old builds) just happen
>    to exist.
> 
> A more robust approach is to save symbol versions in *.cmd files
> because:
> 
>  - *.cmd always exists (if the object is generated by if_changed
>    rule or friends). Even if the user accidentally deletes it,
>    it will be regenerated in the next build.
> 
>  - *.cmd is always re-generated when the object is updated. This
>    avoid stale version information being picked up.
> 
> I will remove *.symversions later.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  scripts/Makefile.build | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index f6a506318795..e03e85c90b26 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -175,6 +175,8 @@ gen_symversions =								\
>  	if $(NM) $@ 2>/dev/null | grep -q __ksymtab; then			\
>  		$(call cmd_gensymtypes_$(1),$(KBUILD_SYMTYPES),$(@:.o=.symtypes)) \
>  		    > $@.symversions;						\
> +		echo >> $(dot-target).cmd ;					\
> +		sed 's/\(.*\) = \(.*\);/$(pound)\1=\2/' $@.symversions >> $(dot-target).cmd ; \
>  	else									\
>  		rm -f $@.symversions;						\
>  	fi

While reviewing this patch, I was missing the update of the comment a 
few lines above.  But I understand that it makes more sense to update 
it in the follow-up patch.

Tested-by: Nicolas Schier <nicolas@fjasle.eu>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

> -- 
> 2.32.0

-- 
epost|xmpp: nicolas@fjasle.eu          irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

[PATCH 23/27] modpost: retrieve symbol versions by parsing *.cmd files

[Masahiro Yamada]

Currently, CONFIG_MODVERSIONS needs extra link to embed the symbol
versions into objects. Then, modpost parses the ELF objects to retrieve
the version CRCs.

See the current build flow below.

Current implementation
======================

                  embed CRC        |---------|              |------------|
       $(CC)        $(LD)          |         |              | final link |
  *.c ------> *.o --------> *.o -->| modpost |-- *.o ------>| for        |
                      /            |         |              | vmlinux    |
     genksyms        /             |         |-- *.mod.c -->| or module  |
  *.c ------> *.symversions        |---------|              |------------|

Genksyms outputs the calculated CRCs in the form of linker script
(*.symversions), which is used by $(LD) to update the object.

If CONFIG_LTO_CLANG=y, the build process becomes much more complex.
Embedding the CRCs is postponed until the LLVM bitcode is converted
into ELF, creating another intermediate *.prelink.o.

However, this complexity is unneeded in the first place. There is no
reason why we must embed version CRCs in objects so early.

There is final link stage for vmlinux (scripts/link-vmlinux.sh) and
modules (scripts/Makefile.modfinal). We can embed CRCs at the very
last moment.

See the following figure, which explains what I want to do.

New implementation
==================

       $(CC)           |---------|                    |------------|
  *.c ------> *.o  --->|         |-- *.o ------------>| final link |
                       | modpost |-- *.mod.c -------->| for        |
      genksyms         |         |-- *.symver.lds --->| vmlinux    |
  *.c ------> *.cmd -->|         |                    | or module  |
                       |---------|                    |------------|

We can pass the symbol versions to modpost as separate text data.
(The previous commit output the versions in *.cmd files.)

This commit changes modpost to retrieve CRCs from *.cmd files instead
of from ELF objects.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/mod/modpost.c | 177 ++++++++++++++++++++++++++++++++----------
 1 file changed, 136 insertions(+), 41 deletions(-)

diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index fb50927cd241..43ab4f000ae3 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -246,6 +246,9 @@ static inline unsigned int tdb_hash(const char *name)
 /* hash table of all exported symbols */
 HASHTABLE_DECLARE(exported_symbols, 8192);
 
+/* hash table of CRCs */
+HASHTABLE_DECLARE(crc_symbols, 1024);
+
 /**
  * Allocate a new symbols for use in the hash of exported symbols or
  * the list of unresolved symbols per module
@@ -420,7 +423,27 @@ static void sym_add_exported(const char *name, struct module *mod,
 		      s->module->is_vmlinux ? "" : ".ko");
 	}
 
-	s = alloc_symbol(name);
+	s = NULL;
+
+	if (modversions) {
+		struct hlist_node *n;
+
+		hash_for_matched_symbol_safe(s, n, crc_symbols, name) {
+			if (s->module == mod)
+				hash_del_symbol(s);
+			break;
+		}
+
+		if (!s)
+			warn("EXPORT symbol \"%s\" [%s%s] version generation failed, symbol will not be versioned.\n"
+			     "Is \"%s\" prototyped in <asm/asm-prototypes.h>?\n",
+			     name, mod->name, mod->is_vmlinux ? "" : ".ko",
+			     name);
+	}
+
+	if (!s)
+		s = alloc_symbol(name);
+
 	s->module = mod;
 	s->is_static = !mod->from_dump;
 	s->export    = export;
@@ -428,19 +451,116 @@ static void sym_add_exported(const char *name, struct module *mod,
 	hash_add_symbol(s, exported_symbols);
 }
 
-static void sym_set_crc(const char *name, unsigned int crc)
+static void sym_add_crc(const char *name, unsigned int crc, struct module *mod)
 {
-	struct symbol *s = find_symbol(name);
+	struct symbol *sym;
 
-	/*
-	 * Ignore stand-alone __crc_*, which might be auto-generated symbols
-	 * such as __*_veneer in ARM ELF.
-	 */
-	if (!s)
+	sym = alloc_symbol(name);
+
+	sym->crc = crc;
+	sym->crc_valid = true;
+	sym->module = mod;
+
+	hash_add_symbol(sym, crc_symbols);
+}
+
+/*
+ * The CRCs are recorded in .*.cmd files in the form of:
+ * #__crc_<name>=<crc>
+ */
+static void retrieve_crcs_from_cmdfile(const char *object, struct module *mod)
+{
+	char cmd_file[PATH_MAX];
+	char *buf, *p;
+	const char *base;
+	int dirlen, ret;
+
+	base = strrchr(object, '/');
+	if (base) {
+		base++;
+		dirlen = base - object;
+	} else {
+		dirlen = 0;
+		base = object;
+	}
+
+	ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%s.cmd",
+		       dirlen, object, base);
+	if (ret >= sizeof(cmd_file)) {
+		error("%s: too long path was truncated\n", cmd_file);
 		return;
+	}
+
+	buf = read_text_file(cmd_file);
+	p = buf;
+
+	while ((p = strstr(p, "\n#__crc_"))) {
+		char *name;
+		size_t namelen;
+		unsigned int crc;
+		char *end;
+
+		name = p + strlen("\n#__crc_");
+
+		p = strchr(name, '=');
+		if (!p) {
+			error("invalid\n");
+			goto out;
+		}
+
+		namelen = p - name;
+
+		p++;
 
-	s->crc = crc;
-	s->crc_valid = true;
+		if (!isdigit(*p)) {
+			error("invalid\n");
+			goto out;
+		}
+
+		crc = strtol(p, &end, 0);
+		p = end;
+
+		if (*p != '\n') {
+			error("invalid\n");
+			goto out;
+		}
+
+		name[namelen] = '\0';
+		sym_add_crc(name, crc, mod);
+	}
+
+out:
+	free(buf);
+}
+
+/*
+ * The symbol versions (CRC) are recorded in the .*.cmd files.
+ * Parse them to retrieve CRCs for the current module.
+ */
+static void retrieve_crcs(struct module *mod)
+{
+	char objlist[PATH_MAX];
+	char *buf, *p, *obj;
+	int ret;
+
+	if (mod->is_vmlinux) {
+		strcpy(objlist, ".vmlinux.objs");
+	} else {
+		/* objects for a module are listed in the *.mod file. */
+		ret = snprintf(objlist, sizeof(objlist), "%s.mod", mod->name);
+		if (ret >= sizeof(objlist)) {
+			error("%s: too long path was truncated\n", objlist);
+			return;
+		}
+	}
+
+	buf = read_text_file(objlist);
+	p = buf;
+
+	while ((obj = strsep(&p, "\n")) && obj[0])
+		retrieve_crcs_from_cmdfile(obj, mod);
+
+	free(buf);
 }
 
 static void *grab_file(const char *filename, size_t *size)
@@ -663,33 +783,6 @@ static int ignore_undef_symbol(struct elf_info *info, const char *symname)
 	return 0;
 }
 
-static void handle_modversion(const struct module *mod,
-			      const struct elf_info *info,
-			      const Elf_Sym *sym, const char *symname)
-{
-	unsigned int crc;
-
-	if (sym->st_shndx == SHN_UNDEF) {
-		warn("EXPORT symbol \"%s\" [%s%s] version generation failed, symbol will not be versioned.\n"
-		     "Is \"%s\" prototyped in <asm/asm-prototypes.h>?\n",
-		     symname, mod->name, mod->is_vmlinux ? "" : ".ko",
-		     symname);
-
-		return;
-	}
-
-	if (sym->st_shndx == SHN_ABS) {
-		crc = sym->st_value;
-	} else {
-		unsigned int *crcp;
-
-		/* symbol points to the CRC in the ELF object */
-		crcp = sym_get_data(info, sym);
-		crc = TO_NATIVE(*crcp);
-	}
-	sym_set_crc(symname, crc);
-}
-
 static void handle_symbol(struct module *mod, struct elf_info *info,
 			  const Elf_Sym *sym, const char *symname)
 {
@@ -2019,6 +2112,10 @@ static void read_symbols(const char *modname)
 		if (strends(tmp, ".prelink"))
 			tmp[strlen(tmp) - 8] = '\0';
 		mod = new_module(tmp);
+
+		if (modversions)
+			retrieve_crcs(mod);
+
 		free(tmp);
 	}
 
@@ -2058,9 +2155,6 @@ static void read_symbols(const char *modname)
 		if (strstarts(symname, "__kstrtabns_"))
 			sym_update_namespace(symname + strlen("__kstrtabns_"),
 					     sym_get_data(&info, sym));
-		if (strstarts(symname, "__crc_"))
-			handle_modversion(mod, &info, sym,
-					  symname + strlen("__crc_"));
 	}
 
 	// check for static EXPORT_SYMBOL_* functions && global vars
@@ -2451,8 +2545,9 @@ static void read_dump(const char *fname)
 			mod = new_module(modname);
 			mod->from_dump = true;
 		}
+
+		sym_add_crc(symname, crc, mod);
 		sym_add_exported(symname, mod, export_no(export));
-		sym_set_crc(symname, crc);
 		sym_update_namespace(symname, namespace);
 	}
 	free(buf);
-- 
2.32.0

[PATCH 27/27] kbuild: do not create *.prelink.o for Clang LTO or IBT

[Masahiro Yamada]

When CONFIG_LTO_CLANG=y, additional intermediate *.prelink.o is created
for each module. Also, objtool is postponed until LLVM bitcode is
converted to ELF.

CONFIG_X86_KERNEL_IBT works in a similar way to postpone objtool until
objects are merged together.

This commit stops generating *.prelink.o, so the build flow will look
the same with/without LTO.

The following figures show how the LTO build currently works, and
how this commit is changing it.

Current build flow
==================

 [1] single-object module

                                    [+objtool]
           $(CC)                      $(LD)                $(LD)
    foo.c --------------------> foo.o -----> foo.prelink.o -----> foo.ko
                           (LLVM bitcode)        (ELF)       |
                                                             |
                                                 foo.mod.o --/

 [2] multi-object module
                                    [+objtool]
           $(CC)         $(AR)        $(LD)                $(LD)
    foo1.c -----> foo1.o -----> foo.o -----> foo.prelink.o -----> foo.ko
                           |  (archive)          (ELF)       |
    foo2.c -----> foo2.o --/                                 |
                (LLVM bitcode)                   foo.mod.o --/

  One confusion is foo.o in multi-object module is an archive despite of
  its suffix.

New build flow
==============

 [1] single-object module

  Since there is only one object, we do not need to have the LLVM
  bitcode stage. Use $(CC)+$(LD) to generate an ELF object in one
  build rule. Of course, only $(CC) is used when LTO is disabled.

           $(CC)+$(LD)[+objtool]           $(LD)
    foo.c ------------------------> foo.o -------> foo.ko
                                    (ELF)    |
                                             |
                                 foo.mod.o --/

 [2] multi-object module

  Previously, $(AR) was used to combine LLVM bitcode into an archive,
  but there was not a technical reason to do so.
  This commit just uses $(LD) to combine and convert them into a single
  ELF object.

                          [+objtool]
            $(CC)           $(LD)          $(LD)
    foo1.c -------> foo1.o -------> foo.o -------> foo.ko
                              |     (ELF)    |
    foo2.c -------> foo2.o ---/              |
                (LLVM bitcode)   foo.mod.o --/

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/Kbuild.include    |  4 +++
 scripts/Makefile.build    | 58 ++++++++++++---------------------------
 scripts/Makefile.lib      |  7 -----
 scripts/Makefile.modfinal |  5 ++--
 scripts/Makefile.modpost  |  9 ++----
 scripts/mod/modpost.c     |  7 -----
 6 files changed, 25 insertions(+), 65 deletions(-)

diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 3514c2149e9d..455a0a6ce12d 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -15,6 +15,10 @@ pound := \#
 # Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
 dot-target = $(dir $@).$(notdir $@)
 
+###
+# Name of target with a '.tmp_' as filename prefix. foo/bar.o => foo/.tmp_bar.o
+tmp-target = $(dir $@).tmp_$(notdir $@)
+
 ###
 # The temporary file to save gcc -MMD generated dependencies must not
 # contain a comma
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index 7f199b0a5170..fe4d3a908dd0 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -88,10 +88,6 @@ endif
 targets-for-modules := $(foreach x, o mod $(if $(CONFIG_TRIM_UNUSED_KSYMS), usyms), \
 				$(patsubst %.o, %.$x, $(filter %.o, $(obj-m))))
 
-ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
-targets-for-modules += $(patsubst %.o, %.prelink.o, $(filter %.o, $(obj-m)))
-endif
-
 ifdef need-modorder
 targets-for-modules += $(obj)/modules.order
 endif
@@ -153,8 +149,16 @@ $(obj)/%.ll: $(src)/%.c FORCE
 # The C file is compiled and updated dependency information is generated.
 # (See cmd_cc_o_c + relevant part of rule_cc_o_c)
 
+is-single-obj-m = $(if $(part-of-module),$(if $(filter $@, $(obj-m)),y))
+
+ifdef CONFIG_LTO_CLANG
+cmd_ld_single = $(if $(is-single-obj-m), ; $(LD) $(ld_flags) -r -o $(tmp-target) $@; mv $(tmp-target) $@)
+endif
+
 quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
-      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< $(cmd_objtool)
+      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \
+		$(cmd_ld_single) \
+		$(cmd_objtool)
 
 ifdef CONFIG_MODVERSIONS
 # When module versioning is enabled the following steps are executed:
@@ -228,21 +232,16 @@ cmd_gen_objtooldep = $(if $(objtool-enabled), { echo ; echo '$@: $$(wildcard $(o
 
 endif # CONFIG_STACK_VALIDATION
 
-ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
-
-# Skip objtool for LLVM bitcode
-$(obj)/%.o: objtool-enabled :=
-
-else
 
 # 'OBJECT_FILES_NON_STANDARD := y': skip objtool checking for a directory
 # 'OBJECT_FILES_NON_STANDARD_foo.o := 'y': skip objtool checking for a file
 # 'OBJECT_FILES_NON_STANDARD_foo.o := 'n': override directory skip for a file
 
-$(obj)/%.o: objtool-enabled = $(if $(filter-out y%, \
-	$(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n),y)
+is-standard-object = $(if $(filter-out y%, $(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n),y)
 
-endif
+delay-objtool := $(or $(CONFIG_LTO_CLANG),$(CONFIG_X86_KERNEL_IBT))
+
+$(obj)/%.o: objtool-enabled = $(if $(is-standard-object),$(if $(delay-objtool),$(is-single-obj-m),y))
 
 ifdef CONFIG_TRIM_UNUSED_KSYMS
 cmd_gen_ksymdeps = \
@@ -271,24 +270,6 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
 	$(call if_changed_rule,cc_o_c)
 	$(call cmd,force_checksrc)
 
-ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
-# Module .o files may contain LLVM bitcode, compile them into native code
-# before ELF processing
-quiet_cmd_cc_prelink_modules = LD [M]  $@
-      cmd_cc_prelink_modules =						\
-	$(LD) $(ld_flags) -r -o $@					\
-		--whole-archive $(filter-out FORCE,$^)			\
-		$(cmd_objtool)
-
-# objtool was skipped for LLVM bitcode, run it now that we have compiled
-# modules into native code
-$(obj)/%.prelink.o: objtool-enabled = y
-$(obj)/%.prelink.o: part-of-module := y
-
-$(obj)/%.prelink.o: $(obj)/%.o FORCE
-	$(call if_changed,cc_prelink_modules)
-endif
-
 cmd_mod = echo $(addprefix $(obj)/, $(call real-search, $*.o, .o, -objs -y -m)) | \
 	$(AWK) -v RS='( |\n)' '!x[$$0]++' > $@
 
@@ -298,7 +279,7 @@ $(obj)/%.mod: FORCE
 # List module undefined symbols
 cmd_undefined_syms = $(NM) $< | sed -n 's/^  *U //p' > $@
 
-$(obj)/%.usyms: $(obj)/%$(mod-prelink-ext).o FORCE
+$(obj)/%.usyms: $(obj)/%.o FORCE
 	$(call if_changed,undefined_syms)
 
 quiet_cmd_cc_lst_c = MKLST   $@
@@ -420,16 +401,11 @@ $(obj)/modules.order: $(obj-m) FORCE
 $(obj)/lib.a: $(lib-y) FORCE
 	$(call if_changed,ar)
 
-ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
-quiet_cmd_link_multi-m = AR [M]  $@
-cmd_link_multi-m =						\
-	rm -f $@; 						\
-	$(AR) cDPrsT $@ @$(patsubst %.o,%.mod,$@)
-else
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ @$(patsubst %.o,%.mod,$@)
-endif
+      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ @$(patsubst %.o,%.mod,$@) $(cmd_objtool)
 
+$(multi-obj-m): objtool-enabled := $(delay-objtool)
+$(multi-obj-m): part-of-module := y
 $(multi-obj-m): %.o: %.mod FORCE
 	$(call if_changed,link_multi-m)
 $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 0453a1904646..f75138385449 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -225,13 +225,6 @@ dtc_cpp_flags  = -Wp,-MMD,$(depfile).pre.tmp -nostdinc                    \
 		 $(addprefix -I,$(DTC_INCLUDE))                          \
 		 -undef -D__DTS__
 
-ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
-# With CONFIG_LTO_CLANG, .o files in modules might be LLVM bitcode, so we
-# need to run LTO to compile them into native code (.lto.o) before further
-# processing.
-mod-prelink-ext := .prelink
-endif
-
 # Useful for describing the dependency of composite objects
 # Usage:
 #   $(call multi_depend, multi_used_targets, suffix_to_remove, suffix_to_add)
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index d429e3f9ae1d..51d384a0e4f9 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -9,7 +9,7 @@ __modfinal:
 include include/config/auto.conf
 include $(srctree)/scripts/Kbuild.include
 
-# for c_flags and mod-prelink-ext
+# for c_flags
 include $(srctree)/scripts/Makefile.lib
 
 # find all modules listed in modules.order
@@ -55,9 +55,8 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
 	$(cmd);                                                              \
 	printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
 
-
 # Re-generate module BTFs if either module's .ko or vmlinux changed
-$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o $(if $(CONFIG_MODVERSIONS), %.symver.lds) scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
+$(modules): %.ko: %.o %.mod.o $(if $(CONFIG_MODVERSIONS), %.symver.lds) scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index 48585c4d04ad..f2ce411acd59 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -41,9 +41,6 @@ __modpost:
 include include/config/auto.conf
 include $(srctree)/scripts/Kbuild.include
 
-# for mod-prelink-ext
-include $(srctree)/scripts/Makefile.lib
-
 MODPOST = scripts/mod/modpost								\
 	$(if $(CONFIG_MODVERSIONS),-m)							\
 	$(if $(CONFIG_MODULE_SRCVERSION_ALL),-a)					\
@@ -118,8 +115,6 @@ $(input-symdump):
 	@echo >&2 '         Modules may not have dependencies or modversions.'
 	@echo >&2 '         You may get many unresolved symbol warnings.'
 
-modules := $(sort $(shell cat $(MODORDER)))
-
 # KBUILD_MODPOST_WARN can be set to avoid error out in case of undefined symbols
 ifneq ($(KBUILD_MODPOST_WARN)$(filter-out $(existing-input-symdump), $(input-symdump)),)
 MODPOST += -w
@@ -128,9 +123,9 @@ endif
 # Read out modules.order to pass in modpost.
 # Otherwise, allmodconfig would fail with "Argument list too long".
 quiet_cmd_modpost = MODPOST $@
-      cmd_modpost = sed 's/\.ko$$/$(mod-prelink-ext)\.o/' $< | $(MODPOST) -T -
+      cmd_modpost = sed 's/ko$$/o/' $< | $(MODPOST) -T -
 
-$(output-symdump): $(MODORDER) $(input-symdump) $(modules:.ko=$(mod-prelink-ext).o) FORCE
+$(output-symdump): $(MODORDER) $(input-symdump) FORCE
 	$(call if_changed,modpost)
 
 targets += $(output-symdump)
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index ef779ada04c6..d8de62506939 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -2081,10 +2081,6 @@ static char *remove_dot(char *s)
 		size_t m = strspn(s + n + 1, "0123456789");
 		if (m && (s[n + m] == '.' || s[n + m] == 0))
 			s[n] = 0;
-
-		/* strip trailing .prelink */
-		if (strends(s, ".prelink"))
-			s[strlen(s) - 8] = '\0';
 	}
 	return s;
 }
@@ -2108,9 +2104,6 @@ static void read_symbols(const char *modname)
 		/* strip trailing .o */
 		tmp = NOFAIL(strdup(modname));
 		tmp[strlen(tmp) - 2] = '\0';
-		/* strip trailing .prelink */
-		if (strends(tmp, ".prelink"))
-			tmp[strlen(tmp) - 8] = '\0';
 		mod = new_module(tmp);
 
 		if (modversions)
-- 
2.32.0

Re: [PATCH 27/27] kbuild: do not create *.prelink.o for Clang LTO or IBT

[Nicolas Schier]

On Mon 25 Apr 2022 04:08:11 GMT Masahiro Yamada wrote:
> When CONFIG_LTO_CLANG=y, additional intermediate *.prelink.o is 
> created
> for each module. Also, objtool is postponed until LLVM bitcode is
> converted to ELF.
> 
> CONFIG_X86_KERNEL_IBT works in a similar way to postpone objtool until
> objects are merged together.
> 
> This commit stops generating *.prelink.o, so the build flow will look
> the same with/without LTO.
> 
> The following figures show how the LTO build currently works, and
> how this commit is changing it.
> 
> Current build flow
> ==================
> 
>  [1] single-object module
> 
>                                     [+objtool]
>            $(CC)                      $(LD)                $(LD)
>     foo.c --------------------> foo.o -----> foo.prelink.o -----> foo.ko
>                            (LLVM bitcode)        (ELF)       |
>                                                              |
>                                                  foo.mod.o --/
> 
>  [2] multi-object module
>                                     [+objtool]
>            $(CC)         $(AR)        $(LD)                $(LD)
>     foo1.c -----> foo1.o -----> foo.o -----> foo.prelink.o -----> foo.ko
>                            |  (archive)          (ELF)       |
>     foo2.c -----> foo2.o --/                                 |
>                 (LLVM bitcode)                   foo.mod.o --/
> 
>   One confusion is foo.o in multi-object module is an archive despite of
>   its suffix.
> 
> New build flow
> ==============
> 
>  [1] single-object module
> 
>   Since there is only one object, we do not need to have the LLVM
>   bitcode stage. Use $(CC)+$(LD) to generate an ELF object in one
>   build rule. Of course, only $(CC) is used when LTO is disabled.
> 
>            $(CC)+$(LD)[+objtool]           $(LD)
>     foo.c ------------------------> foo.o -------> foo.ko
>                                     (ELF)    |
>                                              |
>                                  foo.mod.o --/
> 
>  [2] multi-object module
> 
>   Previously, $(AR) was used to combine LLVM bitcode into an archive,
>   but there was not a technical reason to do so.
>   This commit just uses $(LD) to combine and convert them into a single
>   ELF object.
> 
>                           [+objtool]
>             $(CC)           $(LD)          $(LD)
>     foo1.c -------> foo1.o -------> foo.o -------> foo.ko
>                               |     (ELF)    |
>     foo2.c -------> foo2.o ---/              |
>                 (LLVM bitcode)   foo.mod.o --/
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  scripts/Kbuild.include    |  4 +++
>  scripts/Makefile.build    | 58 ++++++++++++---------------------------
>  scripts/Makefile.lib      |  7 -----
>  scripts/Makefile.modfinal |  5 ++--
>  scripts/Makefile.modpost  |  9 ++----
>  scripts/mod/modpost.c     |  7 -----
>  6 files changed, 25 insertions(+), 65 deletions(-)
> 
> diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
> index 3514c2149e9d..455a0a6ce12d 100644
> --- a/scripts/Kbuild.include
> +++ b/scripts/Kbuild.include
> @@ -15,6 +15,10 @@ pound := \#
>  # Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
>  dot-target = $(dir $@).$(notdir $@)
>  
> +###
> +# Name of target with a '.tmp_' as filename prefix. foo/bar.o => foo/.tmp_bar.o
> +tmp-target = $(dir $@).tmp_$(notdir $@)

This matches the dot-target definition above, otherwise $(@D).tmp_$(@F) 
would be an alternative.

> +
>  ###
>  # The temporary file to save gcc -MMD generated dependencies must not
>  # contain a comma
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index 7f199b0a5170..fe4d3a908dd0 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -88,10 +88,6 @@ endif
>  targets-for-modules := $(foreach x, o mod $(if $(CONFIG_TRIM_UNUSED_KSYMS), usyms), \
>  				$(patsubst %.o, %.$x, $(filter %.o, $(obj-m))))
>  
> -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> -targets-for-modules += $(patsubst %.o, %.prelink.o, $(filter %.o, $(obj-m)))
> -endif
> -
>  ifdef need-modorder
>  targets-for-modules += $(obj)/modules.order
>  endif
> @@ -153,8 +149,16 @@ $(obj)/%.ll: $(src)/%.c FORCE
>  # The C file is compiled and updated dependency information is generated.
>  # (See cmd_cc_o_c + relevant part of rule_cc_o_c)
>  
> +is-single-obj-m = $(if $(part-of-module),$(if $(filter $@, $(obj-m)),y))

Perhaps using $(and ..,..,y) instead if $(if ..,$(if ..,y))?

is-single-obj-m = $(and $(part-of-module),$(filter $@, $(obj-m)),y)

> +
> +ifdef CONFIG_LTO_CLANG
> +cmd_ld_single = $(if $(is-single-obj-m), ; $(LD) $(ld_flags) -r -o $(tmp-target) $@; mv $(tmp-target) $@)
> +endif
> +
>  quiet_cmd_cc_o_c = CC $(quiet_modtag)  $@
> -      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< $(cmd_objtool)
> +      cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \
> +		$(cmd_ld_single) \
> +		$(cmd_objtool)
>  
>  ifdef CONFIG_MODVERSIONS
>  # When module versioning is enabled the following steps are executed:
> @@ -228,21 +232,16 @@ cmd_gen_objtooldep = $(if $(objtool-enabled), { echo ; echo '$@: $$(wildcard $(o
>  
>  endif # CONFIG_STACK_VALIDATION
>  
> -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> -
> -# Skip objtool for LLVM bitcode
> -$(obj)/%.o: objtool-enabled :=
> -
> -else
>  
>  # 'OBJECT_FILES_NON_STANDARD := y': skip objtool checking for a directory
>  # 'OBJECT_FILES_NON_STANDARD_foo.o := 'y': skip objtool checking for a file
>  # 'OBJECT_FILES_NON_STANDARD_foo.o := 'n': override directory skip for a file
>  
> -$(obj)/%.o: objtool-enabled = $(if $(filter-out y%, \
> -	$(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n),y)
> +is-standard-object = $(if $(filter-out y%, $(OBJECT_FILES_NON_STANDARD_$(basetarget).o)$(OBJECT_FILES_NON_STANDARD)n),y)
>  
> -endif
> +delay-objtool := $(or $(CONFIG_LTO_CLANG),$(CONFIG_X86_KERNEL_IBT))
> +
> +$(obj)/%.o: objtool-enabled = $(if $(is-standard-object),$(if $(delay-objtool),$(is-single-obj-m),y))

same here?  $(and) versus $(if ..,$(if ..,y))

Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

>  
>  ifdef CONFIG_TRIM_UNUSED_KSYMS
>  cmd_gen_ksymdeps = \
> @@ -271,24 +270,6 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
>  	$(call if_changed_rule,cc_o_c)
>  	$(call cmd,force_checksrc)
>  
> -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> -# Module .o files may contain LLVM bitcode, compile them into native code
> -# before ELF processing
> -quiet_cmd_cc_prelink_modules = LD [M]  $@
> -      cmd_cc_prelink_modules =						\
> -	$(LD) $(ld_flags) -r -o $@					\
> -		--whole-archive $(filter-out FORCE,$^)			\
> -		$(cmd_objtool)
> -
> -# objtool was skipped for LLVM bitcode, run it now that we have compiled
> -# modules into native code
> -$(obj)/%.prelink.o: objtool-enabled = y
> -$(obj)/%.prelink.o: part-of-module := y
> -
> -$(obj)/%.prelink.o: $(obj)/%.o FORCE
> -	$(call if_changed,cc_prelink_modules)
> -endif
> -
>  cmd_mod = echo $(addprefix $(obj)/, $(call real-search, $*.o, .o, -objs -y -m)) | \
>  	$(AWK) -v RS='( |\n)' '!x[$$0]++' > $@
>  
> @@ -298,7 +279,7 @@ $(obj)/%.mod: FORCE
>  # List module undefined symbols
>  cmd_undefined_syms = $(NM) $< | sed -n 's/^  *U //p' > $@
>  
> -$(obj)/%.usyms: $(obj)/%$(mod-prelink-ext).o FORCE
> +$(obj)/%.usyms: $(obj)/%.o FORCE
>  	$(call if_changed,undefined_syms)
>  
>  quiet_cmd_cc_lst_c = MKLST   $@
> @@ -420,16 +401,11 @@ $(obj)/modules.order: $(obj-m) FORCE
>  $(obj)/lib.a: $(lib-y) FORCE
>  	$(call if_changed,ar)
>  
> -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> -quiet_cmd_link_multi-m = AR [M]  $@
> -cmd_link_multi-m =						\
> -	rm -f $@; 						\
> -	$(AR) cDPrsT $@ @$(patsubst %.o,%.mod,$@)
> -else
>  quiet_cmd_link_multi-m = LD [M]  $@
> -      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ @$(patsubst %.o,%.mod,$@)
> -endif
> +      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ @$(patsubst %.o,%.mod,$@) $(cmd_objtool)
>  
> +$(multi-obj-m): objtool-enabled := $(delay-objtool)
> +$(multi-obj-m): part-of-module := y
>  $(multi-obj-m): %.o: %.mod FORCE
>  	$(call if_changed,link_multi-m)
>  $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 0453a1904646..f75138385449 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -225,13 +225,6 @@ dtc_cpp_flags  = -Wp,-MMD,$(depfile).pre.tmp -nostdinc                    \
>  		 $(addprefix -I,$(DTC_INCLUDE))                          \
>  		 -undef -D__DTS__
>  
> -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> -# With CONFIG_LTO_CLANG, .o files in modules might be LLVM bitcode, so we
> -# need to run LTO to compile them into native code (.lto.o) before further
> -# processing.
> -mod-prelink-ext := .prelink
> -endif
> -
>  # Useful for describing the dependency of composite objects
>  # Usage:
>  #   $(call multi_depend, multi_used_targets, suffix_to_remove, suffix_to_add)
> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> index d429e3f9ae1d..51d384a0e4f9 100644
> --- a/scripts/Makefile.modfinal
> +++ b/scripts/Makefile.modfinal
> @@ -9,7 +9,7 @@ __modfinal:
>  include include/config/auto.conf
>  include $(srctree)/scripts/Kbuild.include
>  
> -# for c_flags and mod-prelink-ext
> +# for c_flags
>  include $(srctree)/scripts/Makefile.lib
>  
>  # find all modules listed in modules.order
> @@ -55,9 +55,8 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
>  	$(cmd);                                                              \
>  	printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
>  
> -
>  # Re-generate module BTFs if either module's .ko or vmlinux changed
> -$(modules): %.ko: %$(mod-prelink-ext).o %.mod.o $(if $(CONFIG_MODVERSIONS), %.symver.lds) scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
> +$(modules): %.ko: %.o %.mod.o $(if $(CONFIG_MODVERSIONS), %.symver.lds) scripts/module.lds $(if $(KBUILD_BUILTIN),vmlinux) FORCE
>  	+$(call if_changed_except,ld_ko_o,vmlinux)
>  ifdef CONFIG_DEBUG_INFO_BTF_MODULES
>  	+$(if $(newer-prereqs),$(call cmd,btf_ko))
> diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
> index 48585c4d04ad..f2ce411acd59 100644
> --- a/scripts/Makefile.modpost
> +++ b/scripts/Makefile.modpost
> @@ -41,9 +41,6 @@ __modpost:
>  include include/config/auto.conf
>  include $(srctree)/scripts/Kbuild.include
>  
> -# for mod-prelink-ext
> -include $(srctree)/scripts/Makefile.lib
> -
>  MODPOST = scripts/mod/modpost								\
>  	$(if $(CONFIG_MODVERSIONS),-m)							\
>  	$(if $(CONFIG_MODULE_SRCVERSION_ALL),-a)					\
> @@ -118,8 +115,6 @@ $(input-symdump):
>  	@echo >&2 '         Modules may not have dependencies or modversions.'
>  	@echo >&2 '         You may get many unresolved symbol warnings.'
>  
> -modules := $(sort $(shell cat $(MODORDER)))
> -
>  # KBUILD_MODPOST_WARN can be set to avoid error out in case of undefined symbols
>  ifneq ($(KBUILD_MODPOST_WARN)$(filter-out $(existing-input-symdump), $(input-symdump)),)
>  MODPOST += -w
> @@ -128,9 +123,9 @@ endif
>  # Read out modules.order to pass in modpost.
>  # Otherwise, allmodconfig would fail with "Argument list too long".
>  quiet_cmd_modpost = MODPOST $@
> -      cmd_modpost = sed 's/\.ko$$/$(mod-prelink-ext)\.o/' $< | $(MODPOST) -T -
> +      cmd_modpost = sed 's/ko$$/o/' $< | $(MODPOST) -T -
>  
> -$(output-symdump): $(MODORDER) $(input-symdump) $(modules:.ko=$(mod-prelink-ext).o) FORCE
> +$(output-symdump): $(MODORDER) $(input-symdump) FORCE
>  	$(call if_changed,modpost)
>  
>  targets += $(output-symdump)
> diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
> index ef779ada04c6..d8de62506939 100644
> --- a/scripts/mod/modpost.c
> +++ b/scripts/mod/modpost.c
> @@ -2081,10 +2081,6 @@ static char *remove_dot(char *s)
>  		size_t m = strspn(s + n + 1, "0123456789");
>  		if (m && (s[n + m] == '.' || s[n + m] == 0))
>  			s[n] = 0;
> -
> -		/* strip trailing .prelink */
> -		if (strends(s, ".prelink"))
> -			s[strlen(s) - 8] = '\0';
>  	}
>  	return s;
>  }
> @@ -2108,9 +2104,6 @@ static void read_symbols(const char *modname)
>  		/* strip trailing .o */
>  		tmp = NOFAIL(strdup(modname));
>  		tmp[strlen(tmp) - 2] = '\0';
> -		/* strip trailing .prelink */
> -		if (strends(tmp, ".prelink"))
> -			tmp[strlen(tmp) - 8] = '\0';
>  		mod = new_module(tmp);
>  
>  		if (modversions)
> -- 
> 2.32.0

-- 
epost|xmpp: nicolas@fjasle.eu          irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

Re: [PATCH 27/27] kbuild: do not create *.prelink.o for Clang LTO or IBT

[Masahiro Yamada]

On Thu, Apr 28, 2022 at 12:31 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
>
> On Mon 25 Apr 2022 04:08:11 GMT Masahiro Yamada wrote:
> > When CONFIG_LTO_CLANG=y, additional intermediate *.prelink.o is
> > created
> > for each module. Also, objtool is postponed until LLVM bitcode is
> > converted to ELF.
> >
> > CONFIG_X86_KERNEL_IBT works in a similar way to postpone objtool until
> > objects are merged together.
> >
> > This commit stops generating *.prelink.o, so the build flow will look
> > the same with/without LTO.
> >
> > The following figures show how the LTO build currently works, and
> > how this commit is changing it.
> >
> > Current build flow
> > ==================
> >
> >  [1] single-object module
> >
> >                                     [+objtool]
> >            $(CC)                      $(LD)                $(LD)
> >     foo.c --------------------> foo.o -----> foo.prelink.o -----> foo.ko
> >                            (LLVM bitcode)        (ELF)       |
> >                                                              |
> >                                                  foo.mod.o --/
> >
> >  [2] multi-object module
> >                                     [+objtool]
> >            $(CC)         $(AR)        $(LD)                $(LD)
> >     foo1.c -----> foo1.o -----> foo.o -----> foo.prelink.o -----> foo.ko
> >                            |  (archive)          (ELF)       |
> >     foo2.c -----> foo2.o --/                                 |
> >                 (LLVM bitcode)                   foo.mod.o --/
> >
> >   One confusion is foo.o in multi-object module is an archive despite of
> >   its suffix.
> >
> > New build flow
> > ==============
> >
> >  [1] single-object module
> >
> >   Since there is only one object, we do not need to have the LLVM
> >   bitcode stage. Use $(CC)+$(LD) to generate an ELF object in one
> >   build rule. Of course, only $(CC) is used when LTO is disabled.
> >
> >            $(CC)+$(LD)[+objtool]           $(LD)
> >     foo.c ------------------------> foo.o -------> foo.ko
> >                                     (ELF)    |
> >                                              |
> >                                  foo.mod.o --/
> >
> >  [2] multi-object module
> >
> >   Previously, $(AR) was used to combine LLVM bitcode into an archive,
> >   but there was not a technical reason to do so.
> >   This commit just uses $(LD) to combine and convert them into a single
> >   ELF object.
> >
> >                           [+objtool]
> >             $(CC)           $(LD)          $(LD)
> >     foo1.c -------> foo1.o -------> foo.o -------> foo.ko
> >                               |     (ELF)    |
> >     foo2.c -------> foo2.o ---/              |
> >                 (LLVM bitcode)   foo.mod.o --/
> >
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > ---
> >
> >  scripts/Kbuild.include    |  4 +++
> >  scripts/Makefile.build    | 58 ++++++++++++---------------------------
> >  scripts/Makefile.lib      |  7 -----
> >  scripts/Makefile.modfinal |  5 ++--
> >  scripts/Makefile.modpost  |  9 ++----
> >  scripts/mod/modpost.c     |  7 -----
> >  6 files changed, 25 insertions(+), 65 deletions(-)
> >
> > diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
> > index 3514c2149e9d..455a0a6ce12d 100644
> > --- a/scripts/Kbuild.include
> > +++ b/scripts/Kbuild.include
> > @@ -15,6 +15,10 @@ pound := \#
> >  # Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
> >  dot-target = $(dir $@).$(notdir $@)
> >
> > +###
> > +# Name of target with a '.tmp_' as filename prefix. foo/bar.o => foo/.tmp_bar.o
> > +tmp-target = $(dir $@).tmp_$(notdir $@)
>
> This matches the dot-target definition above, otherwise $(@D).tmp_$(@F)
> would be an alternative.

Yes, I intentionally made dot-target and tmp-target look similar.

The difference is $(dir ...) leaves the trailing slash, but $(@D) does not.

The alternative would be
$(@D)/.tmp_$(@F)





>
> > +
> >  ###
> >  # The temporary file to save gcc -MMD generated dependencies must not
> >  # contain a comma
> > diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> > index 7f199b0a5170..fe4d3a908dd0 100644
> > --- a/scripts/Makefile.build
> > +++ b/scripts/Makefile.build
> > @@ -88,10 +88,6 @@ endif
> >  targets-for-modules := $(foreach x, o mod $(if $(CONFIG_TRIM_UNUSED_KSYMS), usyms), \
> >                               $(patsubst %.o, %.$x, $(filter %.o, $(obj-m))))
> >
> > -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> > -targets-for-modules += $(patsubst %.o, %.prelink.o, $(filter %.o, $(obj-m)))
> > -endif
> > -
> >  ifdef need-modorder
> >  targets-for-modules += $(obj)/modules.order
> >  endif
> > @@ -153,8 +149,16 @@ $(obj)/%.ll: $(src)/%.c FORCE
> >  # The C file is compiled and updated dependency information is generated.
> >  # (See cmd_cc_o_c + relevant part of rule_cc_o_c)
> >
> > +is-single-obj-m = $(if $(part-of-module),$(if $(filter $@, $(obj-m)),y))
>
> Perhaps using $(and ..,..,y) instead if $(if ..,$(if ..,y))?


Good idea!
I did not notice this.  I will do it in v2.



> >
> > -endif
> > +delay-objtool := $(or $(CONFIG_LTO_CLANG),$(CONFIG_X86_KERNEL_IBT))
> > +
> > +$(obj)/%.o: objtool-enabled = $(if $(is-standard-object),$(if $(delay-objtool),$(is-single-obj-m),y))
>
> same here?  $(and) versus $(if ..,$(if ..,y))

I am not sure about this case.
The second if-func is  $(if  cond, A, B) form.



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 27/27] kbuild: do not create *.prelink.o for Clang LTO or IBT

[Nicolas Schier]

On Thu 28 Apr 2022 13:38:39 GMT Masahiro Yamada wrote:
> 
> On Thu, Apr 28, 2022 at 12:31 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
> >
> > On Mon 25 Apr 2022 04:08:11 GMT Masahiro Yamada wrote:
> > > When CONFIG_LTO_CLANG=y, additional intermediate *.prelink.o is
> > > created
> > > for each module. Also, objtool is postponed until LLVM bitcode is
> > > converted to ELF.
> > >
> > > CONFIG_X86_KERNEL_IBT works in a similar way to postpone objtool until
> > > objects are merged together.
> > >
> > > This commit stops generating *.prelink.o, so the build flow will look
> > > the same with/without LTO.
> > >
> > > The following figures show how the LTO build currently works, and
> > > how this commit is changing it.
> > >
> > > Current build flow
> > > ==================
> > >
> > >  [1] single-object module
> > >
> > >                                     [+objtool]
> > >            $(CC)                      $(LD)                $(LD)
> > >     foo.c --------------------> foo.o -----> foo.prelink.o -----> foo.ko
> > >                            (LLVM bitcode)        (ELF)       |
> > >                                                              |
> > >                                                  foo.mod.o --/
> > >
> > >  [2] multi-object module
> > >                                     [+objtool]
> > >            $(CC)         $(AR)        $(LD)                $(LD)
> > >     foo1.c -----> foo1.o -----> foo.o -----> foo.prelink.o -----> foo.ko
> > >                            |  (archive)          (ELF)       |
> > >     foo2.c -----> foo2.o --/                                 |
> > >                 (LLVM bitcode)                   foo.mod.o --/
> > >
> > >   One confusion is foo.o in multi-object module is an archive despite of
> > >   its suffix.
> > >
> > > New build flow
> > > ==============
> > >
> > >  [1] single-object module
> > >
> > >   Since there is only one object, we do not need to have the LLVM
> > >   bitcode stage. Use $(CC)+$(LD) to generate an ELF object in one
> > >   build rule. Of course, only $(CC) is used when LTO is disabled.
> > >
> > >            $(CC)+$(LD)[+objtool]           $(LD)
> > >     foo.c ------------------------> foo.o -------> foo.ko
> > >                                     (ELF)    |
> > >                                              |
> > >                                  foo.mod.o --/
> > >
> > >  [2] multi-object module
> > >
> > >   Previously, $(AR) was used to combine LLVM bitcode into an archive,
> > >   but there was not a technical reason to do so.
> > >   This commit just uses $(LD) to combine and convert them into a single
> > >   ELF object.
> > >
> > >                           [+objtool]
> > >             $(CC)           $(LD)          $(LD)
> > >     foo1.c -------> foo1.o -------> foo.o -------> foo.ko
> > >                               |     (ELF)    |
> > >     foo2.c -------> foo2.o ---/              |
> > >                 (LLVM bitcode)   foo.mod.o --/
> > >
> > > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > > ---
> > >
> > >  scripts/Kbuild.include    |  4 +++
> > >  scripts/Makefile.build    | 58 ++++++++++++---------------------------
> > >  scripts/Makefile.lib      |  7 -----
> > >  scripts/Makefile.modfinal |  5 ++--
> > >  scripts/Makefile.modpost  |  9 ++----
> > >  scripts/mod/modpost.c     |  7 -----
> > >  6 files changed, 25 insertions(+), 65 deletions(-)
> > >
> > > diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
> > > index 3514c2149e9d..455a0a6ce12d 100644
> > > --- a/scripts/Kbuild.include
> > > +++ b/scripts/Kbuild.include
> > > @@ -15,6 +15,10 @@ pound := \#
> > >  # Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
> > >  dot-target = $(dir $@).$(notdir $@)
> > >
> > > +###
> > > +# Name of target with a '.tmp_' as filename prefix. foo/bar.o => foo/.tmp_bar.o
> > > +tmp-target = $(dir $@).tmp_$(notdir $@)
> >
> > This matches the dot-target definition above, otherwise $(@D).tmp_$(@F)
> > would be an alternative.
> 
> Yes, I intentionally made dot-target and tmp-target look similar.
> 
> The difference is $(dir ...) leaves the trailing slash, but $(@D) does not.
> 
> The alternative would be
> $(@D)/.tmp_$(@F)

ah, yes, thanks.

> >
> > > +
> > >  ###
> > >  # The temporary file to save gcc -MMD generated dependencies must not
> > >  # contain a comma
> > > diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> > > index 7f199b0a5170..fe4d3a908dd0 100644
> > > --- a/scripts/Makefile.build
> > > +++ b/scripts/Makefile.build
> > > @@ -88,10 +88,6 @@ endif
> > >  targets-for-modules := $(foreach x, o mod $(if $(CONFIG_TRIM_UNUSED_KSYMS), usyms), \
> > >                               $(patsubst %.o, %.$x, $(filter %.o, $(obj-m))))
> > >
> > > -ifneq ($(CONFIG_LTO_CLANG)$(CONFIG_X86_KERNEL_IBT),)
> > > -targets-for-modules += $(patsubst %.o, %.prelink.o, $(filter %.o, $(obj-m)))
> > > -endif
> > > -
> > >  ifdef need-modorder
> > >  targets-for-modules += $(obj)/modules.order
> > >  endif
> > > @@ -153,8 +149,16 @@ $(obj)/%.ll: $(src)/%.c FORCE
> > >  # The C file is compiled and updated dependency information is generated.
> > >  # (See cmd_cc_o_c + relevant part of rule_cc_o_c)
> > >
> > > +is-single-obj-m = $(if $(part-of-module),$(if $(filter $@, $(obj-m)),y))
> >
> > Perhaps using $(and ..,..,y) instead if $(if ..,$(if ..,y))?
> 
> 
> Good idea!
> I did not notice this.  I will do it in v2.
> 
> 
> 
> > >
> > > -endif
> > > +delay-objtool := $(or $(CONFIG_LTO_CLANG),$(CONFIG_X86_KERNEL_IBT))
> > > +
> > > +$(obj)/%.o: objtool-enabled = $(if $(is-standard-object),$(if $(delay-objtool),$(is-single-obj-m),y))
> >
> > same here?  $(and) versus $(if ..,$(if ..,y))
> 
> I am not sure about this case.
> The second if-func is  $(if  cond, A, B) form.

Oh, I didn't see it.  Then it might be the best to keep it the way it 
is.

Re: [PATCH 00/27] kbuild: yet another series of cleanups (modpost and LTO)

[Nicolas Schier]

On Mon, Apr 25, 2022 at 04:07:44AM +0900 Masahiro Yamada wrote:
> This is the third batch of cleanups in this development cycle.
> 
> This weekend, I wrote up the code I have been planning.
> 
> After a bunch of modpost refactoring, I got rid of the ugly code
> in Makefiles.
> 
> With this, Kbuild will get back much simpler and cleaner.
> 

Hi Masahiro,

I tried applying the patch set onto your kbuild and
kbuild-fixes-v5.18, but it didn't apply.  Can you give me
a hint on your commit base?

Kind regards,
Nicolas

> 
> Masahiro Yamada (27):
>   modpost: use snprintf() instead of sprintf() for safety
>   modpost: do not write out any file when error occurred
>   modpost: remove stale comment about sym_add_exported()
>   modpost: add a separate error for exported symbols without definition
>   modpost: retrieve the module dependency and CRCs in check_exports()
>   modpost: use bool type where appropriate
>   modpost: import include/linux/list.h
>   modpost: traverse modules in order
>   modpost: add sym_add_unresolved() helper
>   modpost: traverse unresolved symbols in order
>   modpost: use doubly linked list for dump_lists
>   modpost: move struct namespace_list to modpost.c
>   modpost: traverse the namespace_list in order
>   modpost: dump Module.symvers in the same order of modules.order
>   modpost: move static EXPORT_SYMBOL check to check_exports()
>   modpost: make multiple export error
>   modpost: make sym_add_exported() always allocate a new symbol
>   modpost: make sym_add_exported() a void function
>   modpost: use hlist for hash table implementation
>   modpost: mitigate false-negatives for static EXPORT_SYMBOL checks
>   kbuild: record symbol versions in *.cmd files
>   kbuild: generate a list of objects in vmlinux
>   modpost: retrieve symbol versions by parsing *.cmd files
>   modpost: generate linker script to collect symbol versions
>   kbuild: embed symbol versions at final link of vmlinux or modules
>   kbuild: stop generating *.symversions
>   kbuild: do not create *.prelink.o for Clang LTO or IBT
> 
>  .gitignore                |   1 +
>  Makefile                  |   1 +
>  scripts/Kbuild.include    |   4 +
>  scripts/Makefile.build    | 107 ++------
>  scripts/Makefile.lib      |   7 -
>  scripts/Makefile.modfinal |   6 +-
>  scripts/Makefile.modpost  |   9 +-
>  scripts/link-vmlinux.sh   |  38 ++-
>  scripts/mod/file2alias.c  |   2 -
>  scripts/mod/list.h        | 336 ++++++++++++++++++++++++
>  scripts/mod/modpost.c     | 529 +++++++++++++++++++++++---------------
>  scripts/mod/modpost.h     |  27 +-
>  12 files changed, 731 insertions(+), 336 deletions(-)
>  create mode 100644 scripts/mod/list.h
> 
> -- 
> 2.32.0

-- 
epost|xmpp: nicolas@fjasle.eu          irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

Re: [PATCH 00/27] kbuild: yet another series of cleanups (modpost and LTO)

[Masahiro Yamada]

On Wed, Apr 27, 2022 at 5:11 AM Nicolas Schier <nicolas@fjasle.eu> wrote:
>
> On Mon, Apr 25, 2022 at 04:07:44AM +0900 Masahiro Yamada wrote:
> > This is the third batch of cleanups in this development cycle.
> >
> > This weekend, I wrote up the code I have been planning.
> >
> > After a bunch of modpost refactoring, I got rid of the ugly code
> > in Makefiles.
> >
> > With this, Kbuild will get back much simpler and cleaner.
> >
>
> Hi Masahiro,
>
> I tried applying the patch set onto your kbuild and
> kbuild-fixes-v5.18, but it didn't apply.  Can you give me
> a hint on your commit base?


This series is based on  linux-kbuild/kbuild branch
7c39c50dcb74 ("scripts: dummy-tools, add pahole")


Anyway, this series is too big.
For convenience, I pushed this to a topic branch.

Please try:

git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
 lto-cleanup





> Kind regards,
> Nicolas
>
> >
> > Masahiro Yamada (27):
> >   modpost: use snprintf() instead of sprintf() for safety
> >   modpost: do not write out any file when error occurred
> >   modpost: remove stale comment about sym_add_exported()
> >   modpost: add a separate error for exported symbols without definition
> >   modpost: retrieve the module dependency and CRCs in check_exports()
> >   modpost: use bool type where appropriate
> >   modpost: import include/linux/list.h
> >   modpost: traverse modules in order
> >   modpost: add sym_add_unresolved() helper
> >   modpost: traverse unresolved symbols in order
> >   modpost: use doubly linked list for dump_lists
> >   modpost: move struct namespace_list to modpost.c
> >   modpost: traverse the namespace_list in order
> >   modpost: dump Module.symvers in the same order of modules.order
> >   modpost: move static EXPORT_SYMBOL check to check_exports()
> >   modpost: make multiple export error
> >   modpost: make sym_add_exported() always allocate a new symbol
> >   modpost: make sym_add_exported() a void function
> >   modpost: use hlist for hash table implementation
> >   modpost: mitigate false-negatives for static EXPORT_SYMBOL checks
> >   kbuild: record symbol versions in *.cmd files
> >   kbuild: generate a list of objects in vmlinux
> >   modpost: retrieve symbol versions by parsing *.cmd files
> >   modpost: generate linker script to collect symbol versions
> >   kbuild: embed symbol versions at final link of vmlinux or modules
> >   kbuild: stop generating *.symversions
> >   kbuild: do not create *.prelink.o for Clang LTO or IBT
> >
> >  .gitignore                |   1 +
> >  Makefile                  |   1 +
> >  scripts/Kbuild.include    |   4 +
> >  scripts/Makefile.build    | 107 ++------
> >  scripts/Makefile.lib      |   7 -
> >  scripts/Makefile.modfinal |   6 +-
> >  scripts/Makefile.modpost  |   9 +-
> >  scripts/link-vmlinux.sh   |  38 ++-
> >  scripts/mod/file2alias.c  |   2 -
> >  scripts/mod/list.h        | 336 ++++++++++++++++++++++++
> >  scripts/mod/modpost.c     | 529 +++++++++++++++++++++++---------------
> >  scripts/mod/modpost.h     |  27 +-
> >  12 files changed, 731 insertions(+), 336 deletions(-)
> >  create mode 100644 scripts/mod/list.h
> >
> > --
> > 2.32.0
>
> --
> epost|xmpp: nicolas@fjasle.eu          irc://oftc.net/nsc
> ↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
>      -- frykten for herren er opphav til kunnskap --



-- 
Best Regards
Masahiro Yamada

Re: [PATCH 00/27] kbuild: yet another series of cleanups (modpost and LTO)

[Masahiro Yamada]

On Wed, Apr 27, 2022 at 12:18 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> On Wed, Apr 27, 2022 at 5:11 AM Nicolas Schier <nicolas@fjasle.eu> wrote:
> >
> > On Mon, Apr 25, 2022 at 04:07:44AM +0900 Masahiro Yamada wrote:
> > > This is the third batch of cleanups in this development cycle.
> > >
> > > This weekend, I wrote up the code I have been planning.
> > >
> > > After a bunch of modpost refactoring, I got rid of the ugly code
> > > in Makefiles.
> > >
> > > With this, Kbuild will get back much simpler and cleaner.
> > >
> >
> > Hi Masahiro,
> >
> > I tried applying the patch set onto your kbuild and
> > kbuild-fixes-v5.18, but it didn't apply.  Can you give me
> > a hint on your commit base?
>
>
> This series is based on  linux-kbuild/kbuild branch
> 7c39c50dcb74 ("scripts: dummy-tools, add pahole")
>
>
> Anyway, this series is too big.
> For convenience, I pushed this to a topic branch.
>
> Please try:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild.git
>  lto-cleanup
>
>
>
>
>
> > Kind regards,
> > Nicolas
> >
> > >
> > > Masahiro Yamada (27):
> > >   modpost: use snprintf() instead of sprintf() for safety
> > >   modpost: do not write out any file when error occurred
> > >   modpost: remove stale comment about sym_add_exported()
> > >   modpost: add a separate error for exported symbols without definition
> > >   modpost: retrieve the module dependency and CRCs in check_exports()
> > >   modpost: use bool type where appropriate
> > >   modpost: import include/linux/list.h
> > >   modpost: traverse modules in order
> > >   modpost: add sym_add_unresolved() helper
> > >   modpost: traverse unresolved symbols in order
> > >   modpost: use doubly linked list for dump_lists
> > >   modpost: move struct namespace_list to modpost.c
> > >   modpost: traverse the namespace_list in order
> > >   modpost: dump Module.symvers in the same order of modules.order
> > >   modpost: move static EXPORT_SYMBOL check to check_exports()
> > >   modpost: make multiple export error
> > >   modpost: make sym_add_exported() always allocate a new symbol
> > >   modpost: make sym_add_exported() a void function
> > >   modpost: use hlist for hash table implementation
> > >   modpost: mitigate false-negatives for static EXPORT_SYMBOL checks
> > >   kbuild: record symbol versions in *.cmd files
> > >   kbuild: generate a list of objects in vmlinux
> > >   modpost: retrieve symbol versions by parsing *.cmd files
> > >   modpost: generate linker script to collect symbol versions
> > >   kbuild: embed symbol versions at final link of vmlinux or modules
> > >   kbuild: stop generating *.symversions
> > >   kbuild: do not create *.prelink.o for Clang LTO or IBT

01-05 and 12 applied.

I will send v2 for the rest.




-- 
Best Regards
Masahiro Yamada