Re: [PATCH v3 14/20] kbuild: batch module finalisation

["Lorenzo Stoakes (ARM)" <ljs@kernel.org>]

+cc Petr

FYI that were some discussion on this in the v2 thread at [0], which will result
in some changes for v4.

[0]:https://lore.kernel.org/all/arEZrFmXeCO4hZpP@gremlin

On Thu, Sep 17, 2026 at 05:06:24PM +0100, Lorenzo Stoakes (ARM) wrote:
> With the .mod.S change in place, module finalisation on allmodconfig builds
> consists of a large number of very short-lived jobs.
>
> For allmodconfig x86-64 this can be on the order of ~22,000 jobs of a few
> milliseconds in duration each.
>
> Each job entails processing ~22k .cmd files, so the combination of heavy
> overhead and small individual job results in a lot of unnecessary and
> repeated work even with all cores being utilised.
>
> The solution is to batch by a number of jobs. Determining which value makes
> sense was done empirically.
>
> On a 128-thread threadripper box doing an allmodconfig build, best of
> 2, *.ko, *.mod.o deleted each time:
>
>     modules per chunk    instances    wall
>     -----------------    ---------    ------
>                     1        11171    10.22s
>                     2         5586     6.72s
>                     4         2793     4.97s
>                     8         1397     4.26s
>                    16          699     4.05s
>                    32          350     4.05s
>                    64          175     4.07s
>                   128           88     4.05s
>                   256           44     4.03s
>                   512           22     4.18s
>
> Wall time flattens for 16-256 module batches.
>
> A slower/lower core machine will do better with fewer modules-per-batch, a
> faster/higher core machine will do better with more modules-per-batch.
>
> Therefore, take the midpoint which works in the most margin in either
> direction - 128 modules per batch.
>
> This naturally scales with module count too as the optimum gains are
> obtained with higher module count, so fewer batches in this case costs less
> overhead.
>
> Each instance holds only its own modules' variables and the top-level one
> reads no per-module .cmd files at all, the same rules serve both levels,
> and an instance is told its chunk with modfinal-first=<index>.
>
> Only the top-level instance builds .module-common.o, to the chunks it is a
> plain prerequisite, so no two instances ever write it.
>
> "make modules" with every *.mod.o and *.ko deleted goes from 28.9s to 15.9s
> with clang 22. No-op "make modules" goes from 5.6s to 4.8s, as checking the
> 22,000 targets is spread over the chunks too.
>
> Whole build, 128-thread Threadripper 9980X, best of N runs:
>
>                                          before   after     delta
>                                          -------------------------------
>   x86 allmodconfig, no-op make, gcc         2.3s     1.4s    -0.85s (-38%)
>   x86 allmodconfig, no-op make, clang       2.8s     1.8s    -0.94s (-34%)
>   x86 allmodconfig, clean, gcc            306.6s   294.1s     -12.6s (-4%)
>   x86 allmodconfig, clean, clang          290.2s   283.9s     -6.4s (-2%)
>
> Assisted-by: LLM
> Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
> ---
>  scripts/Makefile.modfinal | 30 +++++++++++++++++++++++++++++-
>  1 file changed, 29 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> index 75e9effdf02c..4d5e6e1a0ff9 100644
> --- a/scripts/Makefile.modfinal
> +++ b/scripts/Makefile.modfinal
> @@ -13,9 +13,30 @@ include $(srctree)/scripts/Makefile.lib
>  # find all modules listed in modules.order
>  modules := $(call read-file, modules.order)
>
> +modfinal-chunk-size := 128
> +
> +ifdef modfinal-first
> +
> +# this instance handles the chunk of modules.order starting at $(modfinal-first)
> +modules := $(wordlist $(modfinal-first), $(words $(modules)), $(modules))
> +modules := $(wordlist 1, $(modfinal-chunk-size), $(modules))
> +
>  __modfinal: $(modules:%.o=%.ko)
>  	@:
>
> +else
> +
> +modfinal-chunks := $(addprefix chunk-, $(shell seq 1 $(modfinal-chunk-size) $(words $(modules))))
> +
> +PHONY += $(modfinal-chunks)
> +$(modfinal-chunks): .module-common.o
> +	$(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal modfinal-first=$(@:chunk-%=%)
> +
> +__modfinal: $(modfinal-chunks)
> +	@:
> +
> +endif
> +
>  # modname and part-of-module are set to make c_flags define proper module flags
>  modname = $(notdir $(@:.mod.o=))
>  part-of-module = y
> @@ -29,8 +50,11 @@ quiet_cmd_as_mod_o = AS [M]  $@
>  %.mod.o: %.mod.S FORCE
>  	$(call if_changed,as_mod_o)
>
> +# Built by the top-level instance alone, the chunks take it as a plain file.
> +ifndef modfinal-first
>  .module-common.o: $(srctree)/scripts/module-common.c FORCE
>  	$(call if_changed_rule,cc_o_c)
> +endif
>
>  ifneq ($(WARN_ON_UNUSED_TRACEPOINTS),)
>  cmd_check_tracepoint = $(objtree)/scripts/tracepoint-update --module $<;
> @@ -58,7 +82,11 @@ ifdef CONFIG_DEBUG_INFO_BTF_MODULES
>  endif
>  	+$(call cmd,check_tracepoint)
>
> -targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o) .module-common.o
> +ifdef modfinal-first
> +targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o)
> +else
> +targets += .module-common.o
> +endif
>
>  # Add FORCE to the prerequisites of a target to force it to be always rebuilt.
>  # ---------------------------------------------------------------------------
>
> --
> 2.55.0
>

--
Cheers, Lorenzo