[PATCH] kbuild: modinst: remove the multithread option from zstd compression

[PATCH] kbuild: modinst: remove the multithread option from zstd compression

[Masahiro Yamada]

Parallel execution is supported by GNU Make:

  $ make -j<N> modules_install

It is questionable to enable multithreading within each zstd process
by default.

If you still want to do it, you can use the environment variable:

  $ ZSTD_NBTHREADS=<N> make modules_install

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/Makefile.modinst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
index 0afd75472679..04f5229efa6b 100644
--- a/scripts/Makefile.modinst
+++ b/scripts/Makefile.modinst
@@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
 quiet_cmd_xz = XZ      $@
       cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
 quiet_cmd_zstd = ZSTD    $@
-      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
+      cmd_zstd = $(ZSTD) --rm -f -q $<
 
 $(dst)/%.ko.gz: $(dst)/%.ko FORCE
 	$(call cmd,gzip)
-- 
2.43.0

Re: [PATCH] kbuild: modinst: remove the multithread option from zstd compression

[Sedat Dilek]

On Sun, Aug 4, 2024 at 5:33 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Parallel execution is supported by GNU Make:
>
>   $ make -j<N> modules_install
>
> It is questionable to enable multithreading within each zstd process
> by default.
>
> If you still want to do it, you can use the environment variable:
>
>   $ ZSTD_NBTHREADS=<N> make modules_install
>

Hi Masahiro,

I have some understanding problems.

[ start-build.txt ]
dileks     24225   24217  0 17:55 tty2     00:00:00 /usr/bin/perf stat
make V=1 -k -j4 ARCH=x86_64 LLVM=1 LLVM=/opt/llvm/bin/
PAHOLE=/opt/pahole/bin/pahole KBUILD_BUILD_HOST=iniza
KBUILD_BUILD_USER=sedat.dilek@gmail.com
KBUILD_BUILD_TIMESTAMP=2024-08-03
KDEB_PKGVERSION=6.10.3-1~trixie+dileks1
LOCALVERSION=-1-amd64-clang18-kcfi olddefconfig bindeb-pkg

^^ How shall someone pass so that ... ZSTD_NBTHREADS=<N> make
modules_install ... is used?

As far as I understood, each kernel-module file is taken - in the
Debian build-process - sequentially file for file - ZSTD compressed
and afterwards deleted.
Is there a benefit when 'make -j<N>' is used?

[ EXAMPLE - with my patch from [1] ]
zstd -T0 -19 --rm -f -q
debian/linux-image-6.10.3-1-amd64-clang18-kcfi/lib/modules/6.10.3-1-amd64-clang18-kcfi/kernel/drivers/gpu/drm/amd/amdgpu/amdgpu.ko

^^ How should ZSTD_NBTHREADS=<N> replace 'zstd -T0'?

When I pass '-T0 -L19' to ZSTD two cores are used on my CPU and a
better compress-level produces smaller kernel-module files.
I would like to benefit from both.

Thanks.

Best regards,
-Sedat-

Link: https://lore.kernel.org/all/CA+icZUUQadYjAXiCNx7PmKDV20WctvnzkXC3R2F7FM4Gzcm39Q@mail.gmail.com/
[1]

> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
>  scripts/Makefile.modinst | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index 0afd75472679..04f5229efa6b 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
>  quiet_cmd_xz = XZ      $@
>        cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
>  quiet_cmd_zstd = ZSTD    $@
> -      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
> +      cmd_zstd = $(ZSTD) --rm -f -q $<
>
>  $(dst)/%.ko.gz: $(dst)/%.ko FORCE
>         $(call cmd,gzip)
> --
> 2.43.0
>
>

Re: [PATCH] kbuild: modinst: remove the multithread option from zstd compression

[Masahiro Yamada]

On Sun, Aug 4, 2024 at 3:36 PM Sedat Dilek <sedat.dilek@gmail.com> wrote:
>
> On Sun, Aug 4, 2024 at 5:33 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > Parallel execution is supported by GNU Make:
> >
> >   $ make -j<N> modules_install
> >
> > It is questionable to enable multithreading within each zstd process
> > by default.
> >
> > If you still want to do it, you can use the environment variable:
> >
> >   $ ZSTD_NBTHREADS=<N> make modules_install
> >
>
> Hi Masahiro,
>
> I have some understanding problems.
>
> [ start-build.txt ]
> dileks     24225   24217  0 17:55 tty2     00:00:00 /usr/bin/perf stat
> make V=1 -k -j4 ARCH=x86_64 LLVM=1 LLVM=/opt/llvm/bin/
> PAHOLE=/opt/pahole/bin/pahole KBUILD_BUILD_HOST=iniza
> KBUILD_BUILD_USER=sedat.dilek@gmail.com
> KBUILD_BUILD_TIMESTAMP=2024-08-03
> KDEB_PKGVERSION=6.10.3-1~trixie+dileks1
> LOCALVERSION=-1-amd64-clang18-kcfi olddefconfig bindeb-pkg
>
> ^^ How shall someone pass so that ... ZSTD_NBTHREADS=<N> make
> modules_install ... is used?


Option 1

$ export ZSTD_NBTHREADS=<N>
$ make V=1 -k  [snip] olddefconfig bindeb-pkg



Option 2

$ ZSTD_NBTHREADS=<N> make V=1 -k [snip] olddefconfig bindeb-pkg




>
> As far as I understood, each kernel-module file is taken - in the
> Debian build-process - sequentially file for file - ZSTD compressed
> and afterwards deleted.
> Is there a benefit when 'make -j<N>' is used?



zstd --rm -f -q foo.ko
zstd --rm -f -q bar.ko
zstd --rm -f -q baz.ko

will run in parallel with Make's -j<N> option.








-- 
Best Regards
Masahiro Yamada

Re: [PATCH] kbuild: modinst: remove the multithread option from zstd compression

[Nicolas Schier]

On Sun, Aug 04, 2024 at 12:33:07PM +0900, Masahiro Yamada wrote:
> Parallel execution is supported by GNU Make:
> 
>   $ make -j<N> modules_install
> 
> It is questionable to enable multithreading within each zstd process
> by default.
> 
> If you still want to do it, you can use the environment variable:
> 
>   $ ZSTD_NBTHREADS=<N> make modules_install
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  scripts/Makefile.modinst | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> index 0afd75472679..04f5229efa6b 100644
> --- a/scripts/Makefile.modinst
> +++ b/scripts/Makefile.modinst
> @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
>  quiet_cmd_xz = XZ      $@
>        cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
>  quiet_cmd_zstd = ZSTD    $@
> -      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
> +      cmd_zstd = $(ZSTD) --rm -f -q $<
>  
>  $(dst)/%.ko.gz: $(dst)/%.ko FORCE
>  	$(call cmd,gzip)
> -- 
> 2.43.0
> 

Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>

Re: [PATCH] kbuild: modinst: remove the multithread option from zstd compression

[Sedat Dilek]

On Tue, Aug 6, 2024 at 1:02 PM Nicolas Schier <nicolas@fjasle.eu> wrote:
>
> On Sun, Aug 04, 2024 at 12:33:07PM +0900, Masahiro Yamada wrote:
> > Parallel execution is supported by GNU Make:
> >
> >   $ make -j<N> modules_install
> >
> > It is questionable to enable multithreading within each zstd process
> > by default.
> >
> > If you still want to do it, you can use the environment variable:
> >
> >   $ ZSTD_NBTHREADS=<N> make modules_install
> >
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > ---
> >
> >  scripts/Makefile.modinst | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/scripts/Makefile.modinst b/scripts/Makefile.modinst
> > index 0afd75472679..04f5229efa6b 100644
> > --- a/scripts/Makefile.modinst
> > +++ b/scripts/Makefile.modinst
> > @@ -146,7 +146,7 @@ quiet_cmd_gzip = GZIP    $@
> >  quiet_cmd_xz = XZ      $@
> >        cmd_xz = $(XZ) --check=crc32 --lzma2=dict=1MiB -f $<
> >  quiet_cmd_zstd = ZSTD    $@
> > -      cmd_zstd = $(ZSTD) -T0 --rm -f -q $<
> > +      cmd_zstd = $(ZSTD) --rm -f -q $<
> >
> >  $(dst)/%.ko.gz: $(dst)/%.ko FORCE
> >       $(call cmd,gzip)
> > --
> > 2.43.0
> >
>
> Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
>

Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # Linux v6.11-rc4+ on x86-64

-Sedat-