From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-dl1-f74.google.com (mail-dl1-f74.google.com [74.125.82.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33DBF345CAF for ; Thu, 28 May 2026 20:39:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.82.74 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780000753; cv=none; b=I/UJMX2mzcCXW7CSxTPgTLqJjgGmgqrJI7ysGQC9pdcjpyAWUSI4qgpx+NSFtTeUeGsPAJPdrGtNlwguNUwEfy5bTqIz/YbMV+pwianNZza+Q9Ju1s4MUPGEXInZqOCrTCcTkfhsO8Ybmq9erEEcjU8LKpxEbTJGdWx9P9n0xaM= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780000753; c=relaxed/simple; bh=42WC/By/dX9EITF7nq1D6hTSVATuFilfGWFdJOkp8tc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NX91xKZMKS1FVRFQImEIKh2LYTAdquHfUnX64PdMoYVtDjHRviLTcRgDukCcDklZDMOvFiWfG+9aXLcVtyuAtQLmiyzWGB+7wrRVr0AtUWK6cUh7GC06Cb2NDDVkr85Nkc/d7r2hdsX/kiU2fTwf9+yU5RGbXmEkDezHMKn18X4= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--xur.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=KeLSszww; arc=none smtp.client-ip=74.125.82.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--xur.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KeLSszww" Received: by mail-dl1-f74.google.com with SMTP id a92af1059eb24-1353a6f29deso749586c88.1 for ; Thu, 28 May 2026 13:39:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780000750; x=1780605550; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AZCotzTpKJgc9BhnMM3hhg81ozDab1UAqZSdE2I8bQQ=; b=KeLSszwwuNvbvYxSo1QrbWrKa6u77V5ZDKh8sYFhncFFLedlqJKFIRFdDPzDqiNtTs LrcaGYR3xWuFiyimOkAQjkO3mBrxDnIYgPYy6u1uwvLzrsDgl+ihMg6jgPmapabsJi+D //aJP8b66Ms8dWgeG8pYup3ZOihu+TB5gN5PrdiJaC8dmSq0aLwLxDoaVTPUdH28F72z 5+Ol2UXqzyrjgqe5gRyDFl6otd9LLxCSjXecVdll/tpUijFOt+nybYXoxWHFHb++BheT uTzSB3G0SHjiZhGt90AVuUGSPPe/oCG3McyZzviQPu94DHiX+5oPSX9j0CSw2VOKXOmy KB3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780000750; x=1780605550; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AZCotzTpKJgc9BhnMM3hhg81ozDab1UAqZSdE2I8bQQ=; b=mB4FJjnl8ZcR03ourY0/BZvN9p9njUmGUAaHCGJm9L7fLDGpPr/htzcY/KdD25j6x+ glSHPAdqNV6awvI8j5v9QDIIT9umCiZw9M2KvxMGRxx/RZ2J8ym3E0esTQ6EUCXRbcX7 rDC8/QmdbalhqB7KQeZvrBtJiyLS5le3tDMHUQQzuiA7xWTfB/Rbs1dkr/HNHUyZ2WzD CDrC5XH58/Jhqoi5XWVYGDLRRP7xFjlWGw6W+DEpCshI4/OTRR0JKXO2kAPy3YlnLyHP BBYmucMLm4bs4E50fybAPD+VOvG7Q+olH4jjaagHejJN5Nq9nzRu5d/JE2xxQZqTDUwG r2yA== X-Forwarded-Encrypted: i=1; AFNElJ9lHac/y2IFNCeeOn8YrjyBKxoFuDNGxAPc1Kw++GkH59RApj1VkOliHt6O/dStNnAMyzJ7nQJnMtueG4M=@vger.kernel.org X-Gm-Message-State: AOJu0Yzzyw8FY9CEHv7cvtdnRQs4zvk0tlh3lIS7iU+RMMnflwS2sIwl Yd93KYXrjtsQDRVaygLEq5qNoml3fuywyntKJWJPRetjFvott/BMYj8F0x/E72Q7XWtBkw== X-Received: from dlak11.prod.google.com ([2002:a05:701b:290b:b0:136:ac69:b0eb]) (user=xur job=prod-delivery.src-stubby-dispatcher) by 2002:a05:7022:7e0e:b0:130:68a1:a221 with SMTP id a92af1059eb24-137a87cb8d7mr253333c88.38.1780000750190; Thu, 28 May 2026 13:39:10 -0700 (PDT) Date: Thu, 28 May 2026 13:39:05 -0700 In-Reply-To: <20260528203905.687687-1-xur@google.com> Precedence: bulk X-Mailing-List: linux-kbuild@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260528203905.687687-1-xur@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260528203905.687687-4-xur@google.com> Subject: [PATCH v11 3/3] kbuild: distributed build support for Clang ThinLTO From: xur@google.com To: Masahiro Yamada , Nathan Chancellor , Nicolas Schier , Nick Desaulniers , Yonghong Song , Bill Wendling , Justin Stitt , Miguel Ojeda , Thomas Gleixner , Alice Ryhl , Sami Tolvanen , "Mike Rapoport (Microsoft)" , Rafael Aquini , Michael Ellerman , Stafford Horne , Christophe Leroy , Piotr Gorski , Rong Xu , Venkat Rao Bagalkote , Miguel Ojeda , Teresa Johnson Cc: linux-kernel@vger.kernel.org, linux-kbuild@vger.kernel.org, llvm@lists.linux.dev Content-Type: text/plain; charset="UTF-8" From: Rong Xu Add distributed ThinLTO build support for the Linux kernel. This new mode offers several advantages: (1) Increased flexibility in handling user-specified build options. (2) Improved user-friendliness for developers. (3) Greater convenience for integrating with objtool and livepatch. Note that "distributed" in this context refers to a term that differentiates in-process ThinLTO builds by invoking backend compilation through the linker, not necessarily building in distributed environments. Distributed ThinLTO is enabled via the `CONFIG_LTO_CLANG_THIN_DIST` Kconfig option. For example: > make LLVM=1 defconfig > scripts/config -e LTO_CLANG_THIN_DIST > make LLVM=1 oldconfig > make LLVM=1 vmlinux -j <..> The build flow proceeds in four stages: 1. Perform FE compilation, mirroring the in-process ThinLTO mode. 2. Thin-link the generated IR files and object files. 3. Find all IR files and perform BE compilation, using the flags stored in the .*.o.cmd files. 4. Link the BE results to generate the final vmlinux.o. NOTE: This patch currently implements the build for the main kernel image (vmlinux) only. Kernel module support is planned for a subsequent patch. Tested on the following arch: x86, arm64, loongarch, and riscv. The earlier implementation details can be found here: https://discourse.llvm.org/t/rfc-distributed-thinlto-build-for-kernel/85934 Signed-off-by: Rong Xu Co-developed-by: Masahiro Yamada Signed-off-by: Masahiro Yamada Tested-by: Piotr Gorski Tested-by: Nathan Chancellor --- .gitignore | 2 ++ Makefile | 14 +++++++++---- arch/Kconfig | 19 ++++++++++++++++++ scripts/Makefile.lib | 8 ++++++++ scripts/Makefile.thinlto | 41 ++++++++++++++++++++++++++++++++++++++ scripts/Makefile.vmlinux_a | 37 ++++++++++++++++++++++++++++++++++ scripts/mod/modpost.c | 15 +++++++++++--- 7 files changed, 129 insertions(+), 7 deletions(-) create mode 100644 scripts/Makefile.thinlto diff --git a/.gitignore b/.gitignore index 3044b9590f05..1cc34c9a5523 100644 --- a/.gitignore +++ b/.gitignore @@ -57,6 +57,7 @@ *.zst Module.symvers dtbs-list +builtin.order modules.order # @@ -68,6 +69,7 @@ modules.order /vmlinux.32 /vmlinux.map /vmlinux.symvers +/vmlinux.thinlto-index /vmlinux.unstripped /vmlinux-gdb.py /vmlinuz diff --git a/Makefile b/Makefile index 0fce9557a115..857f08dcc952 100644 --- a/Makefile +++ b/Makefile @@ -1071,11 +1071,16 @@ export CC_FLAGS_SCS endif ifdef CONFIG_LTO_CLANG -ifdef CONFIG_LTO_CLANG_THIN +ifdef CONFIG_LTO_CLANG_FULL +CC_FLAGS_LTO := -flto +else CC_FLAGS_LTO := -flto=thin -fsplit-lto-unit + +# These LLVM options were initially added with only in-process ThinLTO +# support, so avoid distributed ThinLTO support for now. +ifdef CONFIG_LTO_CLANG_THIN KBUILD_LDFLAGS += $(call ld-option,--lto-whole-program-visibility -mllvm -always-rename-promoted-locals=false) -else -CC_FLAGS_LTO := -flto +endif endif CC_FLAGS_LTO += -fvisibility=hidden @@ -1684,6 +1689,7 @@ endif # CONFIG_MODULES CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin modules.builtin.modinfo modules.nsdeps \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ + vmlinux.thinlto-index builtin.order \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ .builtin-dtbs-list .builtin-dtbs.S @@ -2145,7 +2151,7 @@ clean: $(clean-dirs) $(call cmd,rmfiles) @find . $(RCS_FIND_IGNORE) \ \( -name '*.[aios]' -o -name '*.rsi' -o -name '*.ko' -o -name '.*.cmd' \ - -o -name '*.ko.*' \ + -o -name '*.ko.*' -o -name '*.o.thinlto.bc' \ -o -name '*.dtb' -o -name '*.dtbo' \ -o -name '*.dtb.S' -o -name '*.dtbo.S' \ -o -name '*.dt.yaml' -o -name 'dtbs-list' \ diff --git a/arch/Kconfig b/arch/Kconfig index e86880045158..082483a3bbf8 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -861,6 +861,25 @@ config LTO_CLANG_THIN https://clang.llvm.org/docs/ThinLTO.html If unsure, say Y. + +config LTO_CLANG_THIN_DIST + bool "Clang ThinLTO in distributed mode (EXPERIMENTAL)" + depends on HAS_LTO_CLANG && ARCH_SUPPORTS_LTO_CLANG_THIN + select LTO_CLANG + help + This option enables Clang's ThinLTO in distributed build mode. + In this mode, the linker performs the thin-link, generating + ThinLTO index files. Subsequently, the build system explicitly + invokes ThinLTO backend compilation using these index files + and pre-linked IR objects. The resulting native object files + are with the .thinlto-native.o suffix. + + This build mode offers improved visibility into the ThinLTO + process through explicit subcommand exposure. It also makes + final native object files directly available, benefiting + tools like objtool and kpatch. Additionally, it provides + crucial granular control over back-end options, enabling + module-specific compiler options, and simplifies debugging. endchoice config ARCH_SUPPORTS_AUTOFDO_CLANG diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib index 0718e39cedda..86e1428cc55d 100644 --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -249,6 +249,13 @@ ifdef CONFIG_LTO_CLANG cmd_ld_single = $(if $(objtool-enabled)$(is-single-obj-m), ; $(LD) $(ld_flags) -r -o $(tmp-target) $@; mv $(tmp-target) $@) endif +ifdef CONFIG_LTO_CLANG_THIN_DIST +# Save the _c_flags, sliently. +quiet_cmd_save_c_flags = + saved_c_flags = $(_c_flags) $(modkern_cflags) + cmd_save_c_flags = printf '\n%s\n' 'saved_c_flags_$@ := $(call escsq,$(saved_c_flags))' >> $(dot-target).cmd +endif + quiet_cmd_cc_o_c = CC $(quiet_modtag) $@ cmd_cc_o_c = $(CC) $(c_flags) -c -o $@ $< \ $(cmd_ld_single) \ @@ -256,6 +263,7 @@ quiet_cmd_cc_o_c = CC $(quiet_modtag) $@ define rule_cc_o_c $(call cmd_and_fixdep,cc_o_c) + $(call cmd,save_c_flags) $(call cmd,checksrc) $(call cmd,checkdoc) $(call cmd,gen_objtooldep) diff --git a/scripts/Makefile.thinlto b/scripts/Makefile.thinlto new file mode 100644 index 000000000000..408b63786a08 --- /dev/null +++ b/scripts/Makefile.thinlto @@ -0,0 +1,41 @@ +PHONY := __default +__default: + +include include/config/auto.conf +include $(srctree)/scripts/Kbuild.include +include $(srctree)/scripts/Makefile.lib + +native-objs := $(patsubst %.o,%.thinlto-native.o,$(call read-file, vmlinux.thinlto-index)) + +__default: $(native-objs) + +# Generate .thinlto-native.o (obj) from .o (bitcode) and .thinlto.bc (summary) files +# --------------------------------------------------------------------------- +quiet_cmd_cc_o_bc = CC $(quiet_modtag) $@ + be_flags = $(saved_c_flags_$(<)) + cmd_cc_o_bc = \ + $(CC) $(be_flags) -x ir -fno-lto -Wno-unused-command-line-argument \ + -fthinlto-index=$(word 2, $^) -c -o $@ $< + +targets += $(native-objs) +$(native-objs): %.thinlto-native.o: %.o %.o.thinlto.bc FORCE + $(call if_changed,cc_o_bc) + +# Add FORCE to the prerequisites of a target to force it to be always rebuilt. +# --------------------------------------------------------------------------- + +PHONY += FORCE +FORCE: + +# Read all saved command lines and dependencies for the $(targets) we +# may be building above, using $(if_changed{,_dep}). As an +# optimization, we don't need to read them if the target does not +# exist, we will rebuild anyway in that case. + +existing-targets := $(wildcard $(sort $(targets))) +prereq-ir-objs := $(targets:%.thinlto-native.o=%.o) +existing-ir-objs := $(wildcard $(sort $(prereq-ir-objs))) + +-include $(foreach f, $(existing-targets) $(existing-ir-objs), $(dir $(f)).$(notdir $(f)).cmd) + +.PHONY: $(PHONY) diff --git a/scripts/Makefile.vmlinux_a b/scripts/Makefile.vmlinux_a index 650d44330d1f..a46d5b8eccb6 100644 --- a/scripts/Makefile.vmlinux_a +++ b/scripts/Makefile.vmlinux_a @@ -21,6 +21,41 @@ targets += built-in-fixup.a built-in-fixup.a: $(KBUILD_VMLINUX_OBJS) scripts/head-object-list.txt FORCE $(call if_changed,ar_builtin_fixup) +ifdef CONFIG_LTO_CLANG_THIN_DIST + +quiet_cmd_builtin.order = GEN $@ + cmd_builtin.order = $(AR) t $< > $@ + +targets += builtin.order +builtin.order: built-in-fixup.a FORCE + $(call if_changed,builtin.order) + +quiet_cmd_ld_thinlto_index = LD $@ + cmd_ld_thinlto_index = \ + $(LD) $(KBUILD_LDFLAGS) -r --thinlto-index-only=$@ @$< + +targets += vmlinux.thinlto-index +vmlinux.thinlto-index: builtin.order FORCE + $(call if_changed,ld_thinlto_index) + +quiet_cmd_ar_vmlinux.a = GEN $@ + cmd_ar_vmlinux.a = \ + rm -f $@; \ + while read -r obj; do \ + if grep -qx $${obj} $(word 2, $^); then \ + echo $${obj%.o}.thinlto-native.o; \ + else \ + echo $${obj}; \ + fi; \ + done < $< | xargs $(AR) cDPrS --thin $@ + +targets += vmlinux.a +vmlinux.a: builtin.order vmlinux.thinlto-index FORCE + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.thinlto + $(call if_changed,ar_vmlinux.a) + +else + # vmlinux.a # --------------------------------------------------------------------------- @@ -28,6 +63,8 @@ targets += vmlinux.a vmlinux.a: built-in-fixup.a FORCE $(call if_changed,copy) +endif + # Add FORCE to the prerequisites of a target to force it to be always rebuilt. # --------------------------------------------------------------------------- diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c index abbcd3fc1394..171886ac1726 100644 --- a/scripts/mod/modpost.c +++ b/scripts/mod/modpost.c @@ -1487,13 +1487,22 @@ static void extract_crcs_for_object(const char *object, struct module *mod) char cmd_file[PATH_MAX]; char *buf, *p; const char *base; - int dirlen, ret; + int dirlen, baselen_without_suffix, ret; base = get_basename(object); dirlen = base - object; - ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%s.cmd", - dirlen, object, base); + baselen_without_suffix = strlen(object) - dirlen - strlen(".o"); + + /* + * When CONFIG_LTO_CLANG_THIN_DIST=y, the ELF is *.thinlto-native.o + * but the symbol CRCs are recorded in *.o.cmd file. + */ + if (strends(object, ".thinlto-native.o")) + baselen_without_suffix -= strlen(".thinlto-native"); + + ret = snprintf(cmd_file, sizeof(cmd_file), "%.*s.%.*s.o.cmd", + dirlen, object, baselen_without_suffix, base); if (ret >= sizeof(cmd_file)) { error("%s: too long path was truncated\n", cmd_file); return; -- 2.54.0.823.g6e5bcc1fc9-goog