public inbox for linux-kbuild@vger.kernel.org
 help / color / mirror / Atom feed
* Re: [PATCH] kbuild: fix broken builds because of GZIP,BZIP2,LZOP variables
@ 2020-06-08 18:03 Guenter Roeck
  0 siblings, 0 replies; 5+ messages in thread
From: Guenter Roeck @ 2020-06-08 18:03 UTC (permalink / raw)
  To: Denis Efremov
  Cc: Masahiro Yamada, Linux Kernel Mailing List,
	Linux Kbuild mailing list

On Mon, Jun 08, 2020 at 12:59:44PM +0300, Denis Efremov wrote:
> Redefine GZIP, BZIP2, LZOP variables as KGZIP, KBZIP2, KLZOP resp.
> GZIP, BZIP2, LZOP env variables are reserved by the tools. The original
> attempt to redefine them internally doesn't work in makefiles/scripts
> intercall scenarios, e.g., "make GZIP=gzip bindeb-pkg" and results in
> broken builds. There can be other broken build commands because of this,
> so the universal solution is to use non-reserved env variables for the
> compression tools.
> 
> Fixes: 8dfb61dcbace ("kbuild: add variables for compression tools")
> Signed-off-by: Denis Efremov <efremov@linux.com>

Tested-by: Guenter Roeck <linux@roeck-us.net>

> ---
>  Makefile                          | 24 +++++-------------------
>  arch/arm/boot/deflate_xip_data.sh |  2 +-
>  arch/ia64/Makefile                |  2 +-
>  arch/m68k/Makefile                |  8 ++++----
>  arch/parisc/Makefile              |  2 +-
>  scripts/Makefile.lib              |  6 +++---
>  scripts/Makefile.package          |  6 +++---
>  scripts/package/buildtar          |  4 ++--
>  8 files changed, 20 insertions(+), 34 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 839f9fee22cb..e43d193bb3b2 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -458,27 +458,13 @@ PYTHON		= python
>  PYTHON3		= python3
>  CHECK		= sparse
>  BASH		= bash
> -GZIP		= gzip
> -BZIP2		= bzip2
> -LZOP		= lzop
> +KGZIP		= gzip
> +KBZIP2		= bzip2
> +KLZOP		= lzop
>  LZMA		= lzma
>  LZ4		= lz4c
>  XZ		= xz
>  
> -# GZIP, BZIP2, LZOP env vars are used by the tools. Support them as the command
> -# line interface, but use _GZIP, _BZIP2, _LZOP internally.
> -_GZIP          := $(GZIP)
> -_BZIP2         := $(BZIP2)
> -_LZOP          := $(LZOP)
> -
> -# Reset GZIP, BZIP2, LZOP in this Makefile
> -override GZIP=
> -override BZIP2=
> -override LZOP=
> -
> -# Reset GZIP, BZIP2, LZOP in recursive invocations
> -MAKEOVERRIDES += GZIP= BZIP2= LZOP=
> -
>  CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
>  		  -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF)
>  NOSTDINC_FLAGS :=
> @@ -526,7 +512,7 @@ CLANG_FLAGS :=
>  export ARCH SRCARCH CONFIG_SHELL BASH HOSTCC KBUILD_HOSTCFLAGS CROSS_COMPILE LD CC
>  export CPP AR NM STRIP OBJCOPY OBJDUMP OBJSIZE READELF PAHOLE LEX YACC AWK INSTALLKERNEL
>  export PERL PYTHON PYTHON3 CHECK CHECKFLAGS MAKE UTS_MACHINE HOSTCXX
> -export _GZIP _BZIP2 _LZOP LZMA LZ4 XZ
> +export KGZIP KBZIP2 KLZOP LZMA LZ4 XZ
>  export KBUILD_HOSTCXXFLAGS KBUILD_HOSTLDFLAGS KBUILD_HOSTLDLIBS LDFLAGS_MODULE
>  
>  export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS KBUILD_LDFLAGS
> @@ -1047,7 +1033,7 @@ export mod_strip_cmd
>  mod_compress_cmd = true
>  ifdef CONFIG_MODULE_COMPRESS
>    ifdef CONFIG_MODULE_COMPRESS_GZIP
> -    mod_compress_cmd = $(_GZIP) -n -f
> +    mod_compress_cmd = $(KGZIP) -n -f
>    endif # CONFIG_MODULE_COMPRESS_GZIP
>    ifdef CONFIG_MODULE_COMPRESS_XZ
>      mod_compress_cmd = $(XZ) -f
> diff --git a/arch/arm/boot/deflate_xip_data.sh b/arch/arm/boot/deflate_xip_data.sh
> index 739f0464321e..304495c3c2c5 100755
> --- a/arch/arm/boot/deflate_xip_data.sh
> +++ b/arch/arm/boot/deflate_xip_data.sh
> @@ -56,7 +56,7 @@ trap 'rm -f "$XIPIMAGE.tmp"; exit 1' 1 2 3
>  # substitute the data section by a compressed version
>  $DD if="$XIPIMAGE" count=$data_start iflag=count_bytes of="$XIPIMAGE.tmp"
>  $DD if="$XIPIMAGE"  skip=$data_start iflag=skip_bytes |
> -$_GZIP -9 >> "$XIPIMAGE.tmp"
> +$KGZIP -9 >> "$XIPIMAGE.tmp"
>  
>  # replace kernel binary
>  mv -f "$XIPIMAGE.tmp" "$XIPIMAGE"
> diff --git a/arch/ia64/Makefile b/arch/ia64/Makefile
> index f817f3d5e758..2876a7df1b0a 100644
> --- a/arch/ia64/Makefile
> +++ b/arch/ia64/Makefile
> @@ -40,7 +40,7 @@ $(error Sorry, you need a newer version of the assember, one that is built from
>  endif
>  
>  quiet_cmd_gzip = GZIP    $@
> -cmd_gzip = cat $(real-prereqs) | $(_GZIP) -n -f -9 > $@
> +cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
>  
>  quiet_cmd_objcopy = OBJCOPY $@
>  cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
> diff --git a/arch/m68k/Makefile b/arch/m68k/Makefile
> index ce6db5e5a5a3..0415d28dbe4f 100644
> --- a/arch/m68k/Makefile
> +++ b/arch/m68k/Makefile
> @@ -135,10 +135,10 @@ vmlinux.gz: vmlinux
>  ifndef CONFIG_KGDB
>  	cp vmlinux vmlinux.tmp
>  	$(STRIP) vmlinux.tmp
> -	$(_GZIP) -9c vmlinux.tmp >vmlinux.gz
> +	$(KGZIP) -9c vmlinux.tmp >vmlinux.gz
>  	rm vmlinux.tmp
>  else
> -	$(_GZIP) -9c vmlinux >vmlinux.gz
> +	$(KGZIP) -9c vmlinux >vmlinux.gz
>  endif
>  
>  bzImage: vmlinux.bz2
> @@ -148,10 +148,10 @@ vmlinux.bz2: vmlinux
>  ifndef CONFIG_KGDB
>  	cp vmlinux vmlinux.tmp
>  	$(STRIP) vmlinux.tmp
> -	$(_BZIP2) -1c vmlinux.tmp >vmlinux.bz2
> +	$(KBZIP2) -1c vmlinux.tmp >vmlinux.bz2
>  	rm vmlinux.tmp
>  else
> -	$(_BZIP2) -1c vmlinux >vmlinux.bz2
> +	$(KBZIP2) -1c vmlinux >vmlinux.bz2
>  endif
>  
>  archclean:
> diff --git a/arch/parisc/Makefile b/arch/parisc/Makefile
> index 182a5bca3e2c..5140c602207f 100644
> --- a/arch/parisc/Makefile
> +++ b/arch/parisc/Makefile
> @@ -162,7 +162,7 @@ vmlinuz: bzImage
>  	$(OBJCOPY) $(boot)/bzImage $@
>  else
>  vmlinuz: vmlinux
> -	@$(_GZIP) -cf -9 $< > $@
> +	@$(KGZIP) -cf -9 $< > $@
>  endif
>  
>  install:
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 127f2a7e3ced..94eeddb2e599 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -244,7 +244,7 @@ cmd_objcopy = $(OBJCOPY) $(OBJCOPYFLAGS) $(OBJCOPYFLAGS_$(@F)) $< $@
>  # ---------------------------------------------------------------------------
>  
>  quiet_cmd_gzip = GZIP    $@
> -      cmd_gzip = cat $(real-prereqs) | $(_GZIP) -n -f -9 > $@
> +      cmd_gzip = cat $(real-prereqs) | $(KGZIP) -n -f -9 > $@
>  
>  # DTC
>  # ---------------------------------------------------------------------------
> @@ -337,7 +337,7 @@ printf "%08x\n" $$dec_size |						\
>  )
>  
>  quiet_cmd_bzip2 = BZIP2   $@
> -      cmd_bzip2 = { cat $(real-prereqs) | $(_BZIP2) -9; $(size_append); } > $@
> +      cmd_bzip2 = { cat $(real-prereqs) | $(KBZIP2) -9; $(size_append); } > $@
>  
>  # Lzma
>  # ---------------------------------------------------------------------------
> @@ -346,7 +346,7 @@ quiet_cmd_lzma = LZMA    $@
>        cmd_lzma = { cat $(real-prereqs) | $(LZMA) -9; $(size_append); } > $@
>  
>  quiet_cmd_lzo = LZO     $@
> -      cmd_lzo = { cat $(real-prereqs) | $(_LZOP) -9; $(size_append); } > $@
> +      cmd_lzo = { cat $(real-prereqs) | $(KLZOP) -9; $(size_append); } > $@
>  
>  quiet_cmd_lz4 = LZ4     $@
>        cmd_lz4 = { cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout; \
> diff --git a/scripts/Makefile.package b/scripts/Makefile.package
> index b2b6153af63a..f952fb64789d 100644
> --- a/scripts/Makefile.package
> +++ b/scripts/Makefile.package
> @@ -45,7 +45,7 @@ if test "$(objtree)" != "$(srctree)"; then \
>  	false; \
>  fi ; \
>  $(srctree)/scripts/setlocalversion --save-scmversion; \
> -tar -I $(_GZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
> +tar -I $(KGZIP) -c $(RCS_TAR_IGNORE) -f $(2).tar.gz \
>  	--transform 's:^:$(2)/:S' $(TAR_CONTENT) $(3); \
>  rm -f $(objtree)/.scmversion
>  
> @@ -127,8 +127,8 @@ util/PERF-VERSION-GEN $(CURDIR)/$(perf-tar)/);              \
>  tar rf $(perf-tar).tar $(perf-tar)/HEAD $(perf-tar)/PERF-VERSION-FILE; \
>  rm -r $(perf-tar);                                                  \
>  $(if $(findstring tar-src,$@),,                                     \
> -$(if $(findstring bz2,$@),$(_BZIP2),                                 \
> -$(if $(findstring gz,$@),$(_GZIP),                                  \
> +$(if $(findstring bz2,$@),$(KBZIP2),                                 \
> +$(if $(findstring gz,$@),$(KGZIP),                                  \
>  $(if $(findstring xz,$@),$(XZ),                                     \
>  $(error unknown target $@))))                                       \
>  	-f -9 $(perf-tar).tar)
> diff --git a/scripts/package/buildtar b/scripts/package/buildtar
> index ad62c6879622..fb1578e72ab9 100755
> --- a/scripts/package/buildtar
> +++ b/scripts/package/buildtar
> @@ -28,11 +28,11 @@ case "${1}" in
>  		opts=
>  		;;
>  	targz-pkg)
> -		opts="-I ${_GZIP}"
> +		opts="-I ${KGZIP}"
>  		tarball=${tarball}.gz
>  		;;
>  	tarbz2-pkg)
> -		opts="-I ${_BZIP2}"
> +		opts="-I ${KBZIP2}"
>  		tarball=${tarball}.bz2
>  		;;
>  	tarxz-pkg)
> -- 
> 2.26.2
> 

^ permalink raw reply	[flat|nested] 5+ messages in thread
* Re: [PATCH v5] kbuild: add variables for compression tools
@ 2020-06-08  4:59 Masahiro Yamada
  2020-06-08  9:59 ` [PATCH] kbuild: fix broken builds because of GZIP,BZIP2,LZOP variables Denis Efremov
  0 siblings, 1 reply; 5+ messages in thread
From: Masahiro Yamada @ 2020-06-08  4:59 UTC (permalink / raw)
  To: Guenter Roeck
  Cc: Denis Efremov, Stephen Rothwell, Linux Kbuild mailing list,
	Linux Kernel Mailing List

On Mon, Jun 8, 2020 at 10:30 AM Guenter Roeck <linux@roeck-us.net> wrote:
>
> Hi,
>
> On Fri, Jun 05, 2020 at 10:39:55AM +0300, Denis Efremov wrote:
> > Allow user to use alternative implementations of compression tools,
> > such as pigz, pbzip2, pxz. For example, multi-threaded tools to
> > speed up the build:
> > $ make GZIP=pigz BZIP2=pbzip2
> >
> > Variables _GZIP, _BZIP2, _LZOP are used internally because original env
> > vars are reserved by the tools. The use of GZIP in gzip tool is obsolete
> > since 2015. However, alternative implementations (e.g., pigz) still rely
> > on it. BZIP2, BZIP, LZOP vars are not obsolescent.
> >
>
> When building mips:defconfig, this patch results in:
>
> Building mips:defconfig ... failed
> --------------
> Error log:
> /bin/sh: -n: command not found
> make[3]: *** [kernel/config_data.gz] Error 127
> make[3]: *** Deleting file 'kernel/config_data.gz'
> make[3]: *** Waiting for unfinished jobs....
> make[2]: *** [kernel] Error 2
> make[2]: *** Waiting for unfinished jobs....
> make[1]: *** [autoksyms_recursive] Error 2
> make: *** [__sub-make] Error 2
>
> Reverting this patch fixes the problem. Bisect log is attached.
>
> Guenter


Agh, this is because of CONFIG_TRIM_UNUSED_KSYMS.

Also, the distro package builds are broken
e.g.  make GZIP=gzip bindeb-pkg


Denis,

I think we should go back to the original
KGZIP, KBZIP2, KLZOP.




>
> ---
> # bad: [cf0c97f148e9e50aa5a7ddd1984a604dd2bde4af] Merge tag 'pinctrl-v5.8-1' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-pinctrl
> # good: [aaa2faab4ed8e5fe0111e04d6e168c028fe2987f] Merge tag 'for-linus-5.8-ofs1' of git://git.kernel.org/pub/scm/linux/kernel/git/hubcap/linux
> git bisect start 'HEAD' 'aaa2faab4ed8'
> # good: [77f55d1305c11fb729b88f2c3f7881ba0831fa6f] staging: rtl8723bs: Use common packet header constants
> git bisect good 77f55d1305c11fb729b88f2c3f7881ba0831fa6f
> # bad: [e611c0fe318c6d6827ee2bba660fbc23cf73f7dc] Merge tag 'usb-5.8-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
> git bisect bad e611c0fe318c6d6827ee2bba660fbc23cf73f7dc
> # bad: [cff11abeca78aa782378401ca2800bd2194aa14e] Merge tag 'kbuild-v5.8' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
> git bisect bad cff11abeca78aa782378401ca2800bd2194aa14e
> # good: [2bd81cd04a3f5eb873cc81fa16c469377be3b092] Merge branch 'remotes/lorenzo/pci/vmd'
> git bisect good 2bd81cd04a3f5eb873cc81fa16c469377be3b092
> # good: [269a535ca931b754a40dda3ab60514e68773c759] modpost: generate vmlinux.symvers and reuse it for the second modpost
> git bisect good 269a535ca931b754a40dda3ab60514e68773c759
> # good: [e542e0dc3ee3eafc46dd8e3073388079d69cace0] Merge branch 'dmi-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/jdelvare/staging
> git bisect good e542e0dc3ee3eafc46dd8e3073388079d69cace0
> # good: [4de7b62936122570408357417f21072e78292926] modpost: remove is_vmlinux() helper
> git bisect good 4de7b62936122570408357417f21072e78292926
> # good: [1ee18de92927f37e6948d5a6fc73cbf89f806905] Merge tag 'dma-mapping-5.8' of git://git.infradead.org/users/hch/dma-mapping
> git bisect good 1ee18de92927f37e6948d5a6fc73cbf89f806905
> # bad: [8dfb61dcbaceb19a5ded5e9c9dcf8d05acc32294] kbuild: add variables for compression tools
> git bisect bad 8dfb61dcbaceb19a5ded5e9c9dcf8d05acc32294
> # good: [c0901577e1dcc8d1c0fd1a11c8d571f650df845f] kbuild: doc: rename LDFLAGS to KBUILD_LDFLAGS
> git bisect good c0901577e1dcc8d1c0fd1a11c8d571f650df845f
> # good: [e0b250b57dcf403529081e5898a9de717f96b76b] Makefile: install modules.builtin even if CONFIG_MODULES=n
> git bisect good e0b250b57dcf403529081e5898a9de717f96b76b
> # first bad commit: [8dfb61dcbaceb19a5ded5e9c9dcf8d05acc32294] kbuild: add variables for compression tools



-- 
Best Regards
Masahiro Yamada

^ permalink raw reply	[flat|nested] 5+ messages in thread
end of thread, other threads:[~2020-06-09  1:04 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-06-08 18:03 [PATCH] kbuild: fix broken builds because of GZIP,BZIP2,LZOP variables Guenter Roeck
  -- strict thread matches above, loose matches on Subject: below --
2020-06-08  4:59 [PATCH v5] kbuild: add variables for compression tools Masahiro Yamada
2020-06-08  9:59 ` [PATCH] kbuild: fix broken builds because of GZIP,BZIP2,LZOP variables Denis Efremov
2020-06-08 15:36   ` Adam Borowski
2020-06-08 16:23     ` Denis Efremov
2020-06-09  1:03   ` Masahiro Yamada

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox