[PATCH v2 0/2] kbuild: Give jobserver-exec recipes the jobserver

[PATCH v2 0/2] kbuild: Give jobserver-exec recipes the jobserver

[Kees Cook]

Hi!

Paul reported a build failure on CentOS Stream 9 with linux-next[1]: the
kernel builds, but scripts/jobserver-exec prints

  WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')

and rcutorture's build check counts the "Error" in that line. CentOS's
make 4.3 carries a backport of the fix for GNU Make bug 58232, so
sub-makes, like the top-level make, give the jobserver pipe only to
recipes marked recursive, while MAKEFLAGS still names it. The new pigz
steps (1/2), for the kernel image, the built-in initramfs, and the other
gzip rules, and the older Clang LTO initcall ordering step (2/2) run
jobserver-exec from recipes without "+", so pigz gets one thread and the
initcall script caps its children at the CPU count instead of the free
job slots. Mark these recipes recursive, as was done for rustc.

Thanks!

-Kees

 v2:
 - 1/2: also mark the gzip rules that choose gzip through a variable,
   including the built-in initramfs in usr/Makefile, which rcutorture's
   kvm.sh builds and v1 missed (Paul)
 - 1/2: update the gzip entry in Documentation/kbuild/makefiles.rst
   (Nicolas)
 - 2/2: cite Masahiro's earlier version of this change and say when the
   script gets more than one archive (Nicolas)
 - v1..v2 diff: https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/diff/?id=dev/next-20261001/pigz-recursive/v2&id2=dev/next-20261001/pigz-recursive/v1
 v1: https://lore.kernel.org/all/20261009062722.i.253-kees@kernel.org/

[1] https://lore.kernel.org/all/0ddb5e5e-f801-444c-aa98-0cfe814a3026@paulmck-laptop/

Kees Cook (2):
  kbuild: Mark the gzip recipes as recursive
  kbuild: Mark the initcall ordering recipe as recursive

 arch/alpha/boot/Makefile                    | 4 ++--
 arch/arc/boot/Makefile                      | 2 +-
 arch/arm/boot/compressed/Makefile           | 2 +-
 arch/arm64/boot/Makefile                    | 2 +-
 arch/csky/boot/Makefile                     | 2 +-
 arch/microblaze/boot/Makefile               | 2 +-
 arch/mips/boot/Makefile                     | 2 +-
 arch/mips/boot/compressed/Makefile          | 2 +-
 arch/nios2/boot/Makefile                    | 2 +-
 arch/parisc/boot/compressed/Makefile        | 2 +-
 arch/riscv/boot/Makefile                    | 2 +-
 arch/s390/boot/Makefile                     | 2 +-
 arch/sh/boot/Makefile                       | 2 +-
 arch/sh/boot/compressed/Makefile            | 2 +-
 arch/sparc/boot/Makefile                    | 4 ++--
 arch/x86/boot/compressed/Makefile           | 2 +-
 arch/xtensa/boot/Makefile                   | 2 +-
 usr/Makefile                                | 2 +-
 drivers/firmware/efi/libstub/Makefile.zboot | 2 +-
 scripts/Makefile.vmlinux_o                  | 2 +-
 Documentation/kbuild/makefiles.rst          | 7 +++++--
 21 files changed, 27 insertions(+), 24 deletions(-)

-- 
2.55.0

[PATCH v2 2/2] kbuild: Mark the initcall ordering recipe as recursive

[Kees Cook]

generate_initcall_order.pl, which orders initcalls for Clang LTO, forks
a child per input file, up to the job count scripts/jobserver-exec
passes it in PARALLELISM. It used to run from link-vmlinux.sh, which
the top-level Makefile invokes as a recursive recipe, but
commit 5d45950dfbb15 ("kbuild: move vmlinux.o link to
scripts/Makefile.vmlinux_o") moved it into a recipe without "+".

GNU Make passes the jobserver pipe only to recipes it treats as
recursive, and the make 4.3 in CentOS Stream 9 also hides it in
sub-makes (a backport of the fix for GNU Make bug 58232), so
jobserver-exec cannot open it and prints:

  WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')

PARALLELISM is then left unset, and the script caps its children at the
number of online CPUs instead of the free job slots.

Mark the recipe as recursive again (this makes "make -n" runs the script
again too, as it did from link-vmlinux.sh). Masahiro posted the same
change and withdrew it because the script was given only vmlinux.a, and
so never ran more than one child [1]. But on arm64, LoongArch, and RISC-V
with CONFIG_EFI_STUB, KBUILD_VMLINUX_LIBS also gives it the EFI stub's
lib.a. Either way, the warning is silenced.

Tested ARCH=x86_64 defconfig with CONFIG_LTO_CLANG_THIN=y, LLVM=1
(Clang 24.0.0git), and GNU Make 4.3 plus CentOS Stream 9's
make-4.3-cloexec.patch.

Fixes: 5d45950dfbb15 ("kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o")
Link: https://lore.kernel.org/all/CAK7LNARCM=rUm8mA8GRQ7ufeyfneGf4OEvHmESKt=zuxs2KrHw@mail.gmail.com/ [1]
Assisted-by: LLM
Reviewed-by: Nicolas Schier <n.schier@fritz.com>
Signed-off-by: Kees Cook <kees@kernel.org>
---
 scripts/Makefile.vmlinux_o | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.vmlinux_o b/scripts/Makefile.vmlinux_o
index 24a3a4fd271c..0b37bd5dc3a4 100644
--- a/scripts/Makefile.vmlinux_o
+++ b/scripts/Makefile.vmlinux_o
@@ -19,7 +19,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
 
 .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
 		vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE
-	$(call if_changed,gen_initcalls_lds)
+	+$(call if_changed,gen_initcalls_lds)
 
 targets := .tmp_initcalls.lds
 
-- 
2.55.0

Re: [PATCH v2 0/2] kbuild: Give jobserver-exec recipes the jobserver

[Nathan Chancellor]

On Sat, Oct 10, 2026 at 01:19:09AM -0700, Kees Cook wrote:
> Hi!
> 
> Paul reported a build failure on CentOS Stream 9 with linux-next[1]: the
> kernel builds, but scripts/jobserver-exec prints
> 
>   WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')
> 
> and rcutorture's build check counts the "Error" in that line. CentOS's
> make 4.3 carries a backport of the fix for GNU Make bug 58232, so
> sub-makes, like the top-level make, give the jobserver pipe only to
> recipes marked recursive, while MAKEFLAGS still names it. The new pigz
> steps (1/2), for the kernel image, the built-in initramfs, and the other
> gzip rules, and the older Clang LTO initcall ordering step (2/2) run
> jobserver-exec from recipes without "+", so pigz gets one thread and the
> initcall script caps its children at the CPU count instead of the free
> job slots. Mark these recipes recursive, as was done for rustc.
> 
> Thanks!
> 
> -Kees
> 
>  v2:
>  - 1/2: also mark the gzip rules that choose gzip through a variable,
>    including the built-in initramfs in usr/Makefile, which rcutorture's
>    kvm.sh builds and v1 missed (Paul)
>  - 1/2: update the gzip entry in Documentation/kbuild/makefiles.rst
>    (Nicolas)
>  - 2/2: cite Masahiro's earlier version of this change and say when the
>    script gets more than one archive (Nicolas)
>  - v1..v2 diff: https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/diff/?id=dev/next-20261001/pigz-recursive/v2&id2=dev/next-20261001/pigz-recursive/v1
>  v1: https://lore.kernel.org/all/20261009062722.i.253-kees@kernel.org/
> 
> [1] https://lore.kernel.org/all/0ddb5e5e-f801-444c-aa98-0cfe814a3026@paulmck-laptop/
> 
> Kees Cook (2):
>   kbuild: Mark the gzip recipes as recursive
>   kbuild: Mark the initcall ordering recipe as recursive

I've kicked these into -next, I will formally apply them to
kbuild-next-speedups once Paul can confirm that it resolves his
regression. Thanks a lot for the assist here!

-- 
Cheers,
Nathan

Re: [PATCH v2 0/2] kbuild: Give jobserver-exec recipes the jobserver

[Paul E. McKenney]

On Sat, Oct 10, 2026 at 02:48:29PM +0100, Nathan Chancellor wrote:
> On Sat, Oct 10, 2026 at 01:19:09AM -0700, Kees Cook wrote:
> > Hi!
> > 
> > Paul reported a build failure on CentOS Stream 9 with linux-next[1]: the
> > kernel builds, but scripts/jobserver-exec prints
> > 
> >   WARNING: Unable to reopen jobserver read-side pipe: FileNotFoundError(2, 'No such file or directory')
> > 
> > and rcutorture's build check counts the "Error" in that line. CentOS's
> > make 4.3 carries a backport of the fix for GNU Make bug 58232, so
> > sub-makes, like the top-level make, give the jobserver pipe only to
> > recipes marked recursive, while MAKEFLAGS still names it. The new pigz
> > steps (1/2), for the kernel image, the built-in initramfs, and the other
> > gzip rules, and the older Clang LTO initcall ordering step (2/2) run
> > jobserver-exec from recipes without "+", so pigz gets one thread and the
> > initcall script caps its children at the CPU count instead of the free
> > job slots. Mark these recipes recursive, as was done for rustc.
> > 
> > Thanks!
> > 
> > -Kees
> > 
> >  v2:
> >  - 1/2: also mark the gzip rules that choose gzip through a variable,
> >    including the built-in initramfs in usr/Makefile, which rcutorture's
> >    kvm.sh builds and v1 missed (Paul)
> >  - 1/2: update the gzip entry in Documentation/kbuild/makefiles.rst
> >    (Nicolas)
> >  - 2/2: cite Masahiro's earlier version of this change and say when the
> >    script gets more than one archive (Nicolas)
> >  - v1..v2 diff: https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git/diff/?id=dev/next-20261001/pigz-recursive/v2&id2=dev/next-20261001/pigz-recursive/v1
> >  v1: https://lore.kernel.org/all/20261009062722.i.253-kees@kernel.org/
> > 
> > [1] https://lore.kernel.org/all/0ddb5e5e-f801-444c-aa98-0cfe814a3026@paulmck-laptop/
> > 
> > Kees Cook (2):
> >   kbuild: Mark the gzip recipes as recursive
> >   kbuild: Mark the initcall ordering recipe as recursive
> 
> I've kicked these into -next, I will formally apply them to
> kbuild-next-speedups once Paul can confirm that it resolves his
> regression. Thanks a lot for the assist here!

And it works!!!  ;-)

Tested-by: Paul E. McKenney <paulmck@kernel.org>

Re: [PATCH v2 0/2] kbuild: Give jobserver-exec recipes the jobserver

[Nathan Chancellor]

On Sat, 10 Oct 2026 01:19:09 -0700, Kees Cook wrote:
> kbuild: Give jobserver-exec recipes the jobserver
> 
> Hi!
> 
> Paul reported a build failure on CentOS Stream 9 with linux-next[1]: the
> kernel builds, but scripts/jobserver-exec prints
> 
> [...]

Applied to

  https://git.kernel.org/pub/scm/linux/kernel/git/kbuild/linux.git kbuild-next-speedups

Thanks!

[1/2] kbuild: Mark the gzip recipes as recursive
      https://git.kernel.org/kbuild/c/0d4d70da348b0
[2/2] kbuild: Mark the initcall ordering recipe as recursive
      https://git.kernel.org/kbuild/c/ae5f6044c4cdf

Please look out for regression or issue reports or other follow up
comments, as they may result in the patch/series getting dropped or
reverted. Patches applied to an "unstable" branch are accepted pending
wider testing in -next and any post-commit review; they will generally
be moved to the main branch in a week if no issues are found.

Best regards,
-- 
Cheers,
Nathan