From: Nicolas Schier <n.schier@avm.de>
To: Masahiro Yamada <masahiroy@kernel.org>
Cc: linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-arch@vger.kernel.org
Subject: Re: [PATCH 3/5] kbuild: rename cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}
Date: Mon, 10 Jan 2022 12:33:15 +0100 [thread overview]
Message-ID: <YdwZe9DHJZUaa6aO@buildd.core.avm.de> (raw)
In-Reply-To: <20220109181529.351420-3-masahiroy@kernel.org>
On Mon, Jan 10, 2022 at 03:15:27AM +0900, Masahiro Yamada wrote:
> GZIP-compressed files end with 4 byte data that represents the size
> of the original input. The decompressors (the self-extracting kernel)
> exploit it to know the vmlinux size beforehand. To mimic the GZIP's
> trailer, Kbuild provides cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22}.
> Unfortunately these macros are used everywhere despite the appended
> size data is only useful for the decompressors.
>
> There is no guarantee that such hand-crafted trailers are safely ignored.
> In fact, the kernel refuses compressed initramdisks with the garbage
> data. That is why usr/Makefile overrides size_append to make it no-op.
>
> To limit the use of such broken compressed files, this commit renames
> the existing macros as follows:
>
> cmd_bzip2 --> cmd_bzip2_with_size
> cmd_lzma --> cmd_lzma_with_size
> cmd_lzo --> cmd_lzo_with_size
> cmd_lz4 --> cmd_lz4_with_size
> cmd_xzkern --> cmd_xzkern_with_size
> cmd_zstd22 --> cmd_zstd22_with_size
>
> To keep the decompressors working, I updated the following Makefiles
> accordingly:
>
> arch/arm/boot/compressed/Makefile
> arch/h8300/boot/compressed/Makefile
> arch/mips/boot/compressed/Makefile
> arch/parisc/boot/compressed/Makefile
> arch/s390/boot/compressed/Makefile
> arch/sh/boot/compressed/Makefile
> arch/x86/boot/compressed/Makefile
>
> I reused the current macro names for the normal usecases; they produce
> the compressed data in the proper format.
>
> I did not touch the following:
>
> arch/arc/boot/Makefile
> arch/arm64/boot/Makefile
> arch/csky/boot/Makefile
> arch/mips/boot/Makefile
> arch/riscv/boot/Makefile
> arch/sh/boot/Makefile
> kernel/Makefile
>
> This means those Makefiles will stop appending the size data.
>
> I dropped the 'override size_append' hack from usr/Makefile.
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
Reviewed-by: Nicolas Schier <n.schier@avm.de>
>
> arch/arm/boot/compressed/Makefile | 8 ++++----
> arch/h8300/boot/compressed/Makefile | 4 +++-
> arch/mips/boot/compressed/Makefile | 12 +++++------
> arch/parisc/boot/compressed/Makefile | 10 +++++-----
> arch/s390/boot/compressed/Makefile | 12 +++++------
> arch/sh/boot/compressed/Makefile | 8 ++++----
> arch/x86/boot/compressed/Makefile | 12 +++++------
> scripts/Makefile.lib | 30 ++++++++++++++++++++++------
> usr/Makefile | 5 -----
> 9 files changed, 58 insertions(+), 43 deletions(-)
>
> diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
> index 91265e7ff672..adc0e318a1ea 100644
> --- a/arch/arm/boot/compressed/Makefile
> +++ b/arch/arm/boot/compressed/Makefile
> @@ -77,10 +77,10 @@ CPPFLAGS_vmlinux.lds += -DTEXT_OFFSET="$(TEXT_OFFSET)"
> CPPFLAGS_vmlinux.lds += -DMALLOC_SIZE="$(MALLOC_SIZE)"
>
> compress-$(CONFIG_KERNEL_GZIP) = gzip
> -compress-$(CONFIG_KERNEL_LZO) = lzo
> -compress-$(CONFIG_KERNEL_LZMA) = lzma
> -compress-$(CONFIG_KERNEL_XZ) = xzkern
> -compress-$(CONFIG_KERNEL_LZ4) = lz4
> +compress-$(CONFIG_KERNEL_LZO) = lzo_with_size
> +compress-$(CONFIG_KERNEL_LZMA) = lzma_with_size
> +compress-$(CONFIG_KERNEL_XZ) = xzkern_with_size
> +compress-$(CONFIG_KERNEL_LZ4) = lz4_with_size
>
> libfdt_objs := fdt_rw.o fdt_ro.o fdt_wip.o fdt.o
>
> diff --git a/arch/h8300/boot/compressed/Makefile b/arch/h8300/boot/compressed/Makefile
> index 5942793f77a0..6ab2fa5ba105 100644
> --- a/arch/h8300/boot/compressed/Makefile
> +++ b/arch/h8300/boot/compressed/Makefile
> @@ -30,9 +30,11 @@ $(obj)/vmlinux.bin: vmlinux FORCE
>
> suffix-$(CONFIG_KERNEL_GZIP) := gzip
> suffix-$(CONFIG_KERNEL_LZO) := lzo
> +compress-$(CONFIG_KERNEL_GZIP) := gzip
> +compress-$(CONFIG_KERNEL_LZO) := lzo_with_size
>
> $(obj)/vmlinux.bin.$(suffix-y): $(obj)/vmlinux.bin FORCE
> - $(call if_changed,$(suffix-y))
> + $(call if_changed,$(compress-y))
>
> LDFLAGS_piggy.o := -r --format binary --oformat elf32-h8300-linux -T
> OBJCOPYFLAGS := -O binary
> diff --git a/arch/mips/boot/compressed/Makefile b/arch/mips/boot/compressed/Makefile
> index f27cf31b4140..832f8001d7d9 100644
> --- a/arch/mips/boot/compressed/Makefile
> +++ b/arch/mips/boot/compressed/Makefile
> @@ -64,12 +64,12 @@ $(obj)/vmlinux.bin: $(KBUILD_IMAGE) FORCE
> $(call if_changed,objcopy)
>
> tool_$(CONFIG_KERNEL_GZIP) = gzip
> -tool_$(CONFIG_KERNEL_BZIP2) = bzip2
> -tool_$(CONFIG_KERNEL_LZ4) = lz4
> -tool_$(CONFIG_KERNEL_LZMA) = lzma
> -tool_$(CONFIG_KERNEL_LZO) = lzo
> -tool_$(CONFIG_KERNEL_XZ) = xzkern
> -tool_$(CONFIG_KERNEL_ZSTD) = zstd22
> +tool_$(CONFIG_KERNEL_BZIP2) = bzip2_with_size
> +tool_$(CONFIG_KERNEL_LZ4) = lz4_with_size
> +tool_$(CONFIG_KERNEL_LZMA) = lzma_with_size
> +tool_$(CONFIG_KERNEL_LZO) = lzo_with_size
> +tool_$(CONFIG_KERNEL_XZ) = xzkern_with_size
> +tool_$(CONFIG_KERNEL_ZSTD) = zstd22_with_size
>
> targets += vmlinux.bin.z
>
> diff --git a/arch/parisc/boot/compressed/Makefile b/arch/parisc/boot/compressed/Makefile
> index bf4f2891d0b7..2640f72d69ce 100644
> --- a/arch/parisc/boot/compressed/Makefile
> +++ b/arch/parisc/boot/compressed/Makefile
> @@ -70,15 +70,15 @@ suffix-$(CONFIG_KERNEL_XZ) := xz
> $(obj)/vmlinux.bin.gz: $(vmlinux.bin.all-y) FORCE
> $(call if_changed,gzip)
> $(obj)/vmlinux.bin.bz2: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,bzip2)
> + $(call if_changed,bzip2_with_size)
> $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lz4)
> + $(call if_changed,lz4_with_size)
> $(obj)/vmlinux.bin.lzma: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzma)
> + $(call if_changed,lzma_with_size)
> $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzo)
> + $(call if_changed,lzo_with_size)
> $(obj)/vmlinux.bin.xz: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,xzkern)
> + $(call if_changed,xzkern_with_size)
>
> LDFLAGS_piggy.o := -r --format binary --oformat $(LD_BFD) -T
> $(obj)/piggy.o: $(obj)/vmlinux.scr $(obj)/vmlinux.bin.$(suffix-y) FORCE
> diff --git a/arch/s390/boot/compressed/Makefile b/arch/s390/boot/compressed/Makefile
> index 3b860061e84d..8ea880b7c3ec 100644
> --- a/arch/s390/boot/compressed/Makefile
> +++ b/arch/s390/boot/compressed/Makefile
> @@ -71,17 +71,17 @@ suffix-$(CONFIG_KERNEL_ZSTD) := .zst
> $(obj)/vmlinux.bin.gz: $(vmlinux.bin.all-y) FORCE
> $(call if_changed,gzip)
> $(obj)/vmlinux.bin.bz2: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,bzip2)
> + $(call if_changed,bzip2_with_size)
> $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lz4)
> + $(call if_changed,lz4_with_size)
> $(obj)/vmlinux.bin.lzma: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzma)
> + $(call if_changed,lzma_with_size)
> $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzo)
> + $(call if_changed,lzo_with_size)
> $(obj)/vmlinux.bin.xz: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,xzkern)
> + $(call if_changed,xzkern_with_size)
> $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,zstd22)
> + $(call if_changed,zstd22_with_size)
>
> OBJCOPYFLAGS_piggy.o := -I binary -O elf64-s390 -B s390:64-bit --rename-section .data=.vmlinux.bin.compressed
> $(obj)/piggy.o: $(obj)/vmlinux.bin$(suffix-y) FORCE
> diff --git a/arch/sh/boot/compressed/Makefile b/arch/sh/boot/compressed/Makefile
> index c1eb9a62de55..a6808a403f4b 100644
> --- a/arch/sh/boot/compressed/Makefile
> +++ b/arch/sh/boot/compressed/Makefile
> @@ -52,13 +52,13 @@ vmlinux.bin.all-y := $(obj)/vmlinux.bin
> $(obj)/vmlinux.bin.gz: $(vmlinux.bin.all-y) FORCE
> $(call if_changed,gzip)
> $(obj)/vmlinux.bin.bz2: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,bzip2)
> + $(call if_changed,bzip2_with_size)
> $(obj)/vmlinux.bin.lzma: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzma)
> + $(call if_changed,lzma_with_size)
> $(obj)/vmlinux.bin.xz: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,xzkern)
> + $(call if_changed,xzkern_with_size)
> $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzo)
> + $(call if_changed,lzo_with_size)
>
> OBJCOPYFLAGS += -R .empty_zero_page
>
> diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
> index 431bf7f846c3..2825c74bcae3 100644
> --- a/arch/x86/boot/compressed/Makefile
> +++ b/arch/x86/boot/compressed/Makefile
> @@ -123,17 +123,17 @@ vmlinux.bin.all-$(CONFIG_X86_NEED_RELOCS) += $(obj)/vmlinux.relocs
> $(obj)/vmlinux.bin.gz: $(vmlinux.bin.all-y) FORCE
> $(call if_changed,gzip)
> $(obj)/vmlinux.bin.bz2: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,bzip2)
> + $(call if_changed,bzip2_with_size)
> $(obj)/vmlinux.bin.lzma: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzma)
> + $(call if_changed,lzma_with_size)
> $(obj)/vmlinux.bin.xz: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,xzkern)
> + $(call if_changed,xzkern_with_size)
> $(obj)/vmlinux.bin.lzo: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lzo)
> + $(call if_changed,lzo_with_size)
> $(obj)/vmlinux.bin.lz4: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,lz4)
> + $(call if_changed,lz4_with_size)
> $(obj)/vmlinux.bin.zst: $(vmlinux.bin.all-y) FORCE
> - $(call if_changed,zstd22)
> + $(call if_changed,zstd22_with_size)
>
> suffix-$(CONFIG_KERNEL_GZIP) := gz
> suffix-$(CONFIG_KERNEL_BZIP2) := bz2
> diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
> index 5366466ea0e4..4207a72d429f 100644
> --- a/scripts/Makefile.lib
> +++ b/scripts/Makefile.lib
> @@ -395,19 +395,31 @@ printf "%08x\n" $$dec_size | \
> )
>
> quiet_cmd_bzip2 = BZIP2 $@
> - cmd_bzip2 = { cat $(real-prereqs) | $(KBZIP2) -9; $(size_append); } > $@
> + cmd_bzip2 = cat $(real-prereqs) | $(KBZIP2) -9 > $@
> +
> +quiet_cmd_bzip2_with_size = BZIP2 $@
> + cmd_bzip2_with_size = { cat $(real-prereqs) | $(KBZIP2) -9; $(size_append); } > $@
>
> # Lzma
> # ---------------------------------------------------------------------------
>
> quiet_cmd_lzma = LZMA $@
> - cmd_lzma = { cat $(real-prereqs) | $(LZMA) -9; $(size_append); } > $@
> + cmd_lzma = cat $(real-prereqs) | $(LZMA) -9 > $@
> +
> +quiet_cmd_lzma_with_size = LZMA $@
> + cmd_lzma_with_size = { cat $(real-prereqs) | $(LZMA) -9; $(size_append); } > $@
>
> quiet_cmd_lzo = LZO $@
> - cmd_lzo = { cat $(real-prereqs) | $(KLZOP) -9; $(size_append); } > $@
> + cmd_lzo = cat $(real-prereqs) | $(KLZOP) -9 > $@
> +
> +quiet_cmd_lzo_with_size = LZO $@
> + cmd_lzo_with_size = { cat $(real-prereqs) | $(KLZOP) -9; $(size_append); } > $@
>
> quiet_cmd_lz4 = LZ4 $@
> - cmd_lz4 = { cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout; \
> + cmd_lz4 = cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout > $@
> +
> +quiet_cmd_lz4_with_size = LZ4 $@
> + cmd_lz4_with_size = { cat $(real-prereqs) | $(LZ4) -l -c1 stdin stdout; \
> $(size_append); } > $@
>
> # U-Boot mkimage
> @@ -450,7 +462,10 @@ quiet_cmd_uimage = UIMAGE $@
> # big dictionary would increase the memory usage too much in the multi-call
> # decompression mode. A BCJ filter isn't used either.
> quiet_cmd_xzkern = XZKERN $@
> - cmd_xzkern = { cat $(real-prereqs) | sh $(srctree)/scripts/xz_wrap.sh; \
> + cmd_xzkern = cat $(real-prereqs) | sh $(srctree)/scripts/xz_wrap.sh > $@
> +
> +quiet_cmd_xzkern_with_size = XZKERN $@
> + cmd_xzkern_with_size = { cat $(real-prereqs) | sh $(srctree)/scripts/xz_wrap.sh; \
> $(size_append); } > $@
>
> quiet_cmd_xzmisc = XZMISC $@
> @@ -476,7 +491,10 @@ quiet_cmd_zstd = ZSTD $@
> cmd_zstd = cat $(real-prereqs) | $(ZSTD) -19 > $@
>
> quiet_cmd_zstd22 = ZSTD22 $@
> - cmd_zstd22 = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
> + cmd_zstd22 = cat $(real-prereqs) | $(ZSTD) -22 --ultra > $@
> +
> +quiet_cmd_zstd22_with_size = ZSTD22 $@
> + cmd_zstd22_with_size = { cat $(real-prereqs) | $(ZSTD) -22 --ultra; $(size_append); } > $@
>
> # ASM offsets
> # ---------------------------------------------------------------------------
> diff --git a/usr/Makefile b/usr/Makefile
> index b1a81a40eab1..7b89c0175a3a 100644
> --- a/usr/Makefile
> +++ b/usr/Makefile
> @@ -3,11 +3,6 @@
> # kbuild file for usr/ - including initramfs image
> #
>
> -# cmd_bzip2, cmd_lzma, cmd_lzo, cmd_lz4 from scripts/Makefile.lib appends the
> -# size at the end of the compressed file, which unfortunately does not work
> -# with unpack_to_rootfs(). Make size_append no-op.
> -override size_append := :
> -
> compress-y := shipped
> compress-$(CONFIG_INITRAMFS_COMPRESSION_GZIP) := gzip
> compress-$(CONFIG_INITRAMFS_COMPRESSION_BZIP2) := bzip2
> --
> 2.32.0
>
next prev parent reply other threads:[~2022-01-10 11:34 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-01-09 18:15 [PATCH 1/5] sh: rename suffix-y to suffix_y Masahiro Yamada
2022-01-09 18:15 ` [PATCH 2/5] kbuild: drop $(size_append) from cmd_zstd Masahiro Yamada
2022-01-10 11:32 ` Nicolas Schier
2022-01-09 18:15 ` [PATCH 3/5] kbuild: rename cmd_{bzip2,lzma,lzo,lz4,xzkern,zstd22} Masahiro Yamada
2022-01-10 11:33 ` Nicolas Schier [this message]
2023-06-23 14:45 ` Eugeniu Rosca
2023-07-19 19:09 ` Eugeniu Rosca
2023-07-22 16:08 ` Masahiro Yamada
2023-07-25 9:24 ` Eugeniu Rosca
2023-08-02 9:21 ` Masahiro Yamada
2023-08-11 11:25 ` Eugeniu Rosca
2022-01-09 18:15 ` [PATCH 4/5] arch: decompressor: remove useless vmlinux.bin.all-y Masahiro Yamada
2022-01-10 11:33 ` Nicolas Schier
2022-01-09 18:15 ` [PATCH 5/5] kbuild: add cmd_file_size Masahiro Yamada
2022-01-10 11:33 ` Nicolas Schier
2022-01-10 11:31 ` [PATCH 1/5] sh: rename suffix-y to suffix_y Nicolas Schier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YdwZe9DHJZUaa6aO@buildd.core.avm.de \
--to=n.schier@avm.de \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kbuild@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=masahiroy@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.