Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Kees Cook]

On Thu, Mar 19, 2020 at 12:59:26PM -0700, Nick Terrell wrote:
> Hi all,
> 
> This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> are supported on all architectures. The ZSTD-compressed kernel is only
> hooked up to x86 in this patch set.
> 
> Zstandard requires slightly more memory during the kernel decompression
> on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> window size.
> 
> Zstandard requires memory proprortional to the window size used during
> compression for decompressing the ramdisk image, since streaming mode is
> used. Newer versions of zstd (1.3.2+) list the window size of a file
> with `zstd -lv <file>'. The absolute maximum amount of memory required
> is just over 8 MB, but it can be controlled at compression time.
> 
> This patch set has been boot tested with buildroot and QEMU based off
> of linux-5.6-rc6.
> 
> On i386 and x86_64 I have tested the following configurations:
> * zstd compressed kernel and a separate zstd compressed initramfs
> * zstd compressed kernel and a built-in zstd compressed initramfs
> * gzip compressed kernel and a separate gzip compressed initramfs
> * gzip compressed kernel and a built-in gzip compressed initramfs
> 
> On arm and aarch64 I tested the same configurations, except that the kernel is
> always gzip compressed.
> 
> Facebook has been using v1 of these patches on x86_64 devices for more than 6
> months. When we switched from a xz compressed initramfs to a zstd compressed
> initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
> switched from a xz compressed kernel to a zstd compressed kernel we saved 2
> seconds of boot time.
> 
> Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
> When we switched from an lzma compressed initramfs to a zstd compressed initramfs
> decompression time shrunk from 27 seconds to 8 seconds.
> 
> The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> than the xz or lzma compressed kernels, and it decompresses faster than
> everything except lz4. See the table below for the measurement of an x86_64
> kernel ordered by compressed size:
> 
> algo	size
> xz  	 6,509,792
> lzma	 6,856,576
> zstd	 7,399,157
> gzip	 8,522,527
> bzip	 8,629,603
> lzo 	 9,808,035
> lz4 	10,705,570
> none	32,565,672
> 
> v1 -> v2:
> - Rebase
>   - usr/Makefile and init/Kconfig were changed so the patches were updated
> - No functional changes except to rebase
> - Split the patches up into smaller chunks
> 
> v2 -> v3:
> - Add *.zst to the .gitignore in patch 8
> - Style nits in patch 3
> - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
>   1 through 3
> 
> Best,
> Nick Terrell
> 
> Adam Borowski (1):
>   .gitignore: add ZSTD-compressed files
> 
> Nick Terrell (7):
>   lib: prepare zstd for preboot environment
>   lib: prepare xxhash for preboot environment
>   lib: add zstd support to decompress
>   init: add support for zstd compressed kernel
>   usr: add support for zstd compressed initramfs
>   x86: bump ZO_z_extra_bytes margin for zstd
>   x86: Add support for ZSTD compressed kernel
> 
>  .gitignore                        |   1 +
>  Documentation/x86/boot.rst        |   6 +-
>  arch/x86/Kconfig                  |   1 +
>  arch/x86/boot/compressed/Makefile |   5 +-
>  arch/x86/boot/compressed/misc.c   |   4 +
>  arch/x86/boot/header.S            |   8 +-
>  arch/x86/include/asm/boot.h       |   6 +-
>  include/linux/decompress/unzstd.h |  11 +
>  init/Kconfig                      |  15 +-
>  lib/Kconfig                       |   4 +
>  lib/Makefile                      |   1 +
>  lib/decompress.c                  |   5 +
>  lib/decompress_unzstd.c           | 338 ++++++++++++++++++++++++++++++
>  lib/xxhash.c                      |  21 +-
>  lib/zstd/decompress.c             |   2 +
>  lib/zstd/fse_decompress.c         |   9 +-
>  lib/zstd/zstd_internal.h          |  14 +-
>  scripts/Makefile.lib              |  15 ++
>  usr/Kconfig                       |  20 ++
>  usr/Makefile                      |   1 +
>  20 files changed, 460 insertions(+), 27 deletions(-)
>  create mode 100644 include/linux/decompress/unzstd.h
>  create mode 100644 lib/decompress_unzstd.c

This looks good to me! Please consider the series:

Reviewed-by: Kees Cook <keescook@chromium.org>

Thanks!

-Kees

-- 
Kees Cook

[PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Nick Terrell]

From: Nick Terrell <terrelln@fb.com>

Hi all,

I attempted to send this patch on March 19, but it got rejected by the
mailing lists for some reason, so I'm resending it.

This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
are supported on all architectures. The ZSTD-compressed kernel is only
hooked up to x86 in this patch set.

Zstandard requires slightly more memory during the kernel decompression
on x86 (192 KB vs 64 KB), and the memory usage is independent of the
window size.

Zstandard requires memory proprortional to the window size used during
compression for decompressing the ramdisk image, since streaming mode is
used. Newer versions of zstd (1.3.2+) list the window size of a file
with `zstd -lv <file>'. The absolute maximum amount of memory required
is just over 8 MB, but it can be controlled at compression time.

This patch set has been boot tested with buildroot and QEMU based off
of linux-5.6-rc6.

On i386 and x86_64 I have tested the following configurations:
* zstd compressed kernel and a separate zstd compressed initramfs
* zstd compressed kernel and a built-in zstd compressed initramfs
* gzip compressed kernel and a separate gzip compressed initramfs
* gzip compressed kernel and a built-in gzip compressed initramfs

On arm and aarch64 I tested the same configurations, except that the kernel is
always gzip compressed.

Facebook has been using v1 of these patches on x86_64 devices for more than 6
months. When we switched from a xz compressed initramfs to a zstd compressed
initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
switched from a xz compressed kernel to a zstd compressed kernel we saved 2
seconds of boot time.

Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
When we switched from an lzma compressed initramfs to a zstd compressed initramfs
decompression time shrunk from 27 seconds to 8 seconds.

The zstd compressed kernel is smaller than the gzip compressed kernel but larger
than the xz or lzma compressed kernels, and it decompresses faster than
everything except lz4. See the table below for the measurement of an x86_64
kernel ordered by compressed size:

algo	size
xz  	 6,509,792
lzma	 6,856,576
zstd	 7,399,157
gzip	 8,522,527
bzip	 8,629,603
lzo 	 9,808,035
lz4 	10,705,570
none	32,565,672

v1 -> v2:
- Rebase
  - usr/Makefile and init/Kconfig were changed so the patches were updated
- No functional changes except to rebase
- Split the patches up into smaller chunks

v2 -> v3:
- Add *.zst to the .gitignore in patch 8
- Style nits in patch 3
- Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
  1 through 3

Best,
Nick Terrell

Adam Borowski (1):
  .gitignore: add ZSTD-compressed files

Nick Terrell (7):
  lib: prepare zstd for preboot environment
  lib: prepare xxhash for preboot environment
  lib: add zstd support to decompress
  init: add support for zstd compressed kernel
  usr: add support for zstd compressed initramfs
  x86: bump ZO_z_extra_bytes margin for zstd
  x86: Add support for ZSTD compressed kernel

 .gitignore                        |   1 +
 Documentation/x86/boot.rst        |   6 +-
 arch/x86/Kconfig                  |   1 +
 arch/x86/boot/compressed/Makefile |   5 +-
 arch/x86/boot/compressed/misc.c   |   4 +
 arch/x86/boot/header.S            |   8 +-
 arch/x86/include/asm/boot.h       |   6 +-
 include/linux/decompress/unzstd.h |  11 +
 init/Kconfig                      |  15 +-
 lib/Kconfig                       |   4 +
 lib/Makefile                      |   1 +
 lib/decompress.c                  |   5 +
 lib/decompress_unzstd.c           | 338 ++++++++++++++++++++++++++++++
 lib/xxhash.c                      |  21 +-
 lib/zstd/decompress.c             |   2 +
 lib/zstd/fse_decompress.c         |   9 +-
 lib/zstd/zstd_internal.h          |  14 +-
 scripts/Makefile.lib              |  15 ++
 usr/Kconfig                       |  20 ++
 usr/Makefile                      |   1 +
 20 files changed, 460 insertions(+), 27 deletions(-)
 create mode 100644 include/linux/decompress/unzstd.h
 create mode 100644 lib/decompress_unzstd.c

-- 
2.25.1

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Kees Cook]

On Wed, Mar 25, 2020 at 12:58:41PM -0700, Nick Terrell wrote:
> From: Nick Terrell <terrelln@fb.com>
> 
> Hi all,
> 
> I attempted to send this patch on March 19, but it got rejected by the
> mailing lists for some reason, so I'm resending it.
> 
> This patch set adds support for a ZSTD-compressed kernel, ramdisk, and
> initramfs in the kernel boot process. ZSTD-compressed ramdisk and initramfs
> are supported on all architectures. The ZSTD-compressed kernel is only
> hooked up to x86 in this patch set.
> 
> Zstandard requires slightly more memory during the kernel decompression
> on x86 (192 KB vs 64 KB), and the memory usage is independent of the
> window size.
> 
> Zstandard requires memory proprortional to the window size used during
> compression for decompressing the ramdisk image, since streaming mode is
> used. Newer versions of zstd (1.3.2+) list the window size of a file
> with `zstd -lv <file>'. The absolute maximum amount of memory required
> is just over 8 MB, but it can be controlled at compression time.
> 
> This patch set has been boot tested with buildroot and QEMU based off
> of linux-5.6-rc6.
> 
> On i386 and x86_64 I have tested the following configurations:
> * zstd compressed kernel and a separate zstd compressed initramfs
> * zstd compressed kernel and a built-in zstd compressed initramfs
> * gzip compressed kernel and a separate gzip compressed initramfs
> * gzip compressed kernel and a built-in gzip compressed initramfs
> 
> On arm and aarch64 I tested the same configurations, except that the kernel is
> always gzip compressed.
> 
> Facebook has been using v1 of these patches on x86_64 devices for more than 6
> months. When we switched from a xz compressed initramfs to a zstd compressed
> initramfs decompression time shrunk from 12 seconds to 3 seconds. When we
> switched from a xz compressed kernel to a zstd compressed kernel we saved 2
> seconds of boot time.
> 
> Facebook has been using v2 of these patches on aarch64 devices for a few weeks.
> When we switched from an lzma compressed initramfs to a zstd compressed initramfs
> decompression time shrunk from 27 seconds to 8 seconds.
> 
> The zstd compressed kernel is smaller than the gzip compressed kernel but larger
> than the xz or lzma compressed kernels, and it decompresses faster than
> everything except lz4. See the table below for the measurement of an x86_64
> kernel ordered by compressed size:
> 
> algo	size
> xz  	 6,509,792
> lzma	 6,856,576
> zstd	 7,399,157
> gzip	 8,522,527
> bzip	 8,629,603
> lzo 	 9,808,035
> lz4 	10,705,570
> none	32,565,672
> 
> v1 -> v2:
> - Rebase
>   - usr/Makefile and init/Kconfig were changed so the patches were updated
> - No functional changes except to rebase
> - Split the patches up into smaller chunks
> 
> v2 -> v3:
> - Add *.zst to the .gitignore in patch 8
> - Style nits in patch 3
> - Rename the PREBOOT macro to ZSTD_PREBOOT and XXH_PREBOOT in patches
>   1 through 3
> 
> Best,
> Nick Terrell
> 

Hi! Just resending Reviewed-by, since it looks like the earlier v3
didn't make it to the mailing list, as Nick just pointed out to me:

https://lore.kernel.org/lkml/202003201109.00FDE311A7@keescook/

Reviewed-by: Kees Cook <keescook@chromium.org>

-Kees

> Adam Borowski (1):
>   .gitignore: add ZSTD-compressed files
> 
> Nick Terrell (7):
>   lib: prepare zstd for preboot environment
>   lib: prepare xxhash for preboot environment
>   lib: add zstd support to decompress
>   init: add support for zstd compressed kernel
>   usr: add support for zstd compressed initramfs
>   x86: bump ZO_z_extra_bytes margin for zstd
>   x86: Add support for ZSTD compressed kernel
> 
>  .gitignore                        |   1 +
>  Documentation/x86/boot.rst        |   6 +-
>  arch/x86/Kconfig                  |   1 +
>  arch/x86/boot/compressed/Makefile |   5 +-
>  arch/x86/boot/compressed/misc.c   |   4 +
>  arch/x86/boot/header.S            |   8 +-
>  arch/x86/include/asm/boot.h       |   6 +-
>  include/linux/decompress/unzstd.h |  11 +
>  init/Kconfig                      |  15 +-
>  lib/Kconfig                       |   4 +
>  lib/Makefile                      |   1 +
>  lib/decompress.c                  |   5 +
>  lib/decompress_unzstd.c           | 338 ++++++++++++++++++++++++++++++
>  lib/xxhash.c                      |  21 +-
>  lib/zstd/decompress.c             |   2 +
>  lib/zstd/fse_decompress.c         |   9 +-
>  lib/zstd/zstd_internal.h          |  14 +-
>  scripts/Makefile.lib              |  15 ++
>  usr/Kconfig                       |  20 ++
>  usr/Makefile                      |   1 +
>  20 files changed, 460 insertions(+), 27 deletions(-)
>  create mode 100644 include/linux/decompress/unzstd.h
>  create mode 100644 lib/decompress_unzstd.c
> 
> -- 
> 2.25.1
> 

-- 
Kees Cook

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Nick Terrell]

> On Mar 29, 2020, at 8:45 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> 
> [ CC: Nick Desaulniers ]
> 
> Hi Nick,
> 
> I have tested your patchset v3 on top of Linux v5.6-rc7+
> compiled/linked with Clang/LLD v9.0.1.
> 
> My Linux system is Debian/testing (will be "bullseye" aka version 11) AMD64.
> I used zstd package version 1.4.4+dfsg-3.
> 
> I have no big numbers for you expect a comparison of initrd.img files
> compressed with GZIP (default by Debian) and ZSTD:
> 
> root# du -k /boot/initrd.img-$(uname -r).{gz,zst}
> 29272   /boot/initrd.img-5.6.0-rc7-1-amd64-clang.gz
> 23540   /boot/initrd.img-5.6.0-rc7-1-amd64-clang.zst
> 
> I needed to do some modifications to stuff from
> initramfs-tools/initramfs-tools-core (see end-titles).
> 
> Do you plan to push this for Linux v5.7?

I hope that it will be accepted. From my point of view this patch set is
ready for merge, except for the maximum window size increase
requested by Petr.

> Feel free to add credits for the whole series:
> 
>   Tested-by: Sedat Dilek <sedat.dilek@gmail.com>

Thanks for testing the patch set Sedat!

-Nick

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Sedat Dilek]

On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:

[...]

> > Do you plan to push this for Linux v5.7?
>
> I hope that it will be accepted. From my point of view this patch set is
> ready for merge, except for the maximum window size increase
> requested by Petr.
>

Hi Nick,

thanks for your answer.

Did you send out a pull-request already?

Can you point me to that request or patch of Petr?
Is it relevant or optional for the pull-request?

> > Feel free to add credits for the whole series:
> >
> >   Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
>
> Thanks for testing the patch set Sedat!
>

I have re-tested zstd-v3 patchset with Linux version 5.6 final and
Clang/LLD version 10.0.0 final (from Debian/unstable repository).

Is it possible to mention that there might distro-specific changes
needed to initramfs-handling?
For Debian you are welcome to include below Link [1].
Not sure I will send/ask to/on the debian-kernel mailing list in this topic.

Thanks and bonne chance.

Regards,
- Sedat -

[1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Nick Terrell]

> On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> 
> On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:
> 
> [...]
> 
>>> Do you plan to push this for Linux v5.7?
>> 
>> I hope that it will be accepted. From my point of view this patch set is
>> ready for merge, except for the maximum window size increase
>> requested by Petr.
>> 
> 
> Hi Nick,
> 
> thanks for your answer.
> 
> Did you send out a pull-request already?

I haven’t sent out a pull request. If that is something that I need to
do, or if it makes the process easier for a maintainer to merge then
I can do that.

> Can you point me to that request or patch of Petr?
> Is it relevant or optional for the pull-request?

The patches from Petr are a parallel implementation of initramfs
decompression and are available here [0]. They support initramfs
compression well, but are missing some pieces for correct and
performant zstd kernel decompression. Namely the equivalent of
patch 6 in this series to increase ZO_z_extra_bytes, the
performance fix in patch 1 that switches memcpy() to
__builtin_memcpy(), and the decompress_single() function in patch 3.

Without those changes a very compressible kernel may fail to
decompress successfully, and will be corrupted. And the performance
is about 4x worse on x86_64. My patches took 70 ms to decompress vs
318 ms with Petr’s. See the detailed comparison here [1].

>>> Feel free to add credits for the whole series:
>>> 
>>>  Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
>> 
>> Thanks for testing the patch set Sedat!
>> 
> 
> I have re-tested zstd-v3 patchset with Linux version 5.6 final and
> Clang/LLD version 10.0.0 final (from Debian/unstable repository).
> 
> Is it possible to mention that there might distro-specific changes
> needed to initramfs-handling?
> For Debian you are welcome to include below Link [1].
> Not sure I will send/ask to/on the debian-kernel mailing list in this topic.

I don’t expect any distro specific changes are required to continue operating
as-is. However, if a distro wanted to switch to a zstd compressed initramfs 
they would need to update their toolchain to compress with zstd.

> Thanks and bonne chance.
> 
> Regards,
> - Sedat -
> 
> [1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/

[0] https://lkml.org/lkml/2020/3/16/461
[1] https://lkml.org/lkml/2020/3/31/1573

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Adam Borowski]

On Tue, Mar 31, 2020 at 05:11:35PM +0200, Sedat Dilek wrote:
> I have re-tested zstd-v3 patchset with Linux version 5.6 final and
> Clang/LLD version 10.0.0 final (from Debian/unstable repository).

So did I and a bunch of other folks.  I for one run v1 since 2017 on
a bunch of boxes (amd64/BIOS, amd64/EFI, arm64, armhf), without a glitch
(not counting trying to boot zstd initrd _without_ the patchset :p).

I've tried v2 on all of the above configurations, v3 on all but armhf, v4 on
amd64/EFI -- all is fine.

Back in the days, folks reported success on IIRC ppc64, sparc64 and more.

(Obviously, initrd only on !x86.)

> Is it possible to mention that there might distro-specific changes
> needed to initramfs-handling?
> For Debian you are welcome to include below Link [1].
> [1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/

+zstd)  compress="zstd -19 -v" ;;

I'm not sure if hard-coding -19 is always right.  It's good for production
machines but slows down dev cycles.

But that's a matter for userland, not a problem with Nick's patchset.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
⠈⠳⣄⠀⠀⠀⠀

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Adam Borowski]

On Wed, Apr 01, 2020 at 02:52:06AM +0000, Nick Terrell wrote:
> > On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:
> >>> Do you plan to push this for Linux v5.7?
> >> 
> >> I hope that it will be accepted. From my point of view this patch set is
> >> ready for merge, except for the maximum window size increase
> >> requested by Petr.

If you target 5.7, remember that the merge window for 5.7 is already going
on, and Linus is extremely cranky about pulling anything that hasn't stewed
in next for a while.

Your patchset hasn't been in next yet, so the timing is very very late. 
You'd need to bother Stephen Rothwell or someone who has a tree in next
about right now to have any chances.

> > Is it possible to mention that there might distro-specific changes
> > needed to initramfs-handling?
> > For Debian you are welcome to include below Link [1].
> > Not sure I will send/ask to/on the debian-kernel mailing list in this topic.
> 
> I don’t expect any distro specific changes are required to continue operating
> as-is. However, if a distro wanted to switch to a zstd compressed initramfs 
> they would need to update their toolchain to compress with zstd.

Just setting COMPRESS=zstd is enough -- it'll say:
W: Unknown compression command zstd
but will do everything right.

Also, just minutes ago someone filed https://bugs.debian.org/955469


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
⠈⠳⣄⠀⠀⠀⠀

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Sedat Dilek]

On Wed, Apr 1, 2020 at 4:52 AM Nick Terrell <terrelln@fb.com> wrote:
>
>
>
> > On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:
> >
> > [...]
> >
> >>> Do you plan to push this for Linux v5.7?
> >>
> >> I hope that it will be accepted. From my point of view this patch set is
> >> ready for merge, except for the maximum window size increase
> >> requested by Petr.
> >>
> >
> > Hi Nick,
> >
> > thanks for your answer.
> >
> > Did you send out a pull-request already?
>
> I haven’t sent out a pull request. If that is something that I need to
> do, or if it makes the process easier for a maintainer to merge then
> I can do that.
>

[ CC  Stephen Rothwell (maintainer of Linux-next) ]

Hi Nick,

as Adam pointed out it might be good to ask "Stephen Rothwell"
<sfr@canb.auug.org.au>
to include your Git tree into linux-next. Stephen will give you some
informations about that.

> > Can you point me to that request or patch of Petr?
> > Is it relevant or optional for the pull-request?
>
> The patches from Petr are a parallel implementation of initramfs
> decompression and are available here [0]. They support initramfs
> compression well, but are missing some pieces for correct and
> performant zstd kernel decompression. Namely the equivalent of
> patch 6 in this series to increase ZO_z_extra_bytes, the
> performance fix in patch 1 that switches memcpy() to
> __builtin_memcpy(), and the decompress_single() function in patch 3.
>
> Without those changes a very compressible kernel may fail to
> decompress successfully, and will be corrupted. And the performance
> is about 4x worse on x86_64. My patches took 70 ms to decompress vs
> 318 ms with Petr’s. See the detailed comparison here [1].
>

Thanks for version 4 of your patchset and these clarifications.

I will try zstd-v4.

Regards,
- Sedat -



> >>> Feel free to add credits for the whole series:
> >>>
> >>>  Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
> >>
> >> Thanks for testing the patch set Sedat!
> >>
> >
> > I have re-tested zstd-v3 patchset with Linux version 5.6 final and
> > Clang/LLD version 10.0.0 final (from Debian/unstable repository).
> >
> > Is it possible to mention that there might distro-specific changes
> > needed to initramfs-handling?
> > For Debian you are welcome to include below Link [1].
> > Not sure I will send/ask to/on the debian-kernel mailing list in this topic.
>
> I don’t expect any distro specific changes are required to continue operating
> as-is. However, if a distro wanted to switch to a zstd compressed initramfs
> they would need to update their toolchain to compress with zstd.
>
> > Thanks and bonne chance.
> >
> > Regards,
> > - Sedat -
> >
> > [1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/
>
> [0] https://lkml.org/lkml/2020/3/16/461
> [1] https://lkml.org/lkml/2020/3/31/1573
>

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Sedat Dilek]

On Wed, Apr 1, 2020 at 9:27 AM Adam Borowski <kilobyte@angband.pl> wrote:
>
> On Wed, Apr 01, 2020 at 02:52:06AM +0000, Nick Terrell wrote:
> > > On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> > > On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:

[...]

> > I don’t expect any distro specific changes are required to continue operating
> > as-is. However, if a distro wanted to switch to a zstd compressed initramfs
> > they would need to update their toolchain to compress with zstd.
>
> Just setting COMPRESS=zstd is enough -- it'll say:
> W: Unknown compression command zstd
> but will do everything right.
>
> Also, just minutes ago someone filed https://bugs.debian.org/955469
>

Hi Adam,

thanks for your feedback and the pointer to the Debian bug #955469.

With "distro-specific changes" I meant the ones to userland - for
Debian - namely initramfs-tools{,-core} packages.
For generating, inspecting and unpacking an initramfs the Debian way
you need a bit more.

In the Bug-BR a patch called "enable_zstd.patch" is attached - seems
to have all what I mentioned in [1].

Regards,
- Sedat -

[1] LINK: https://lore.kernel.org/lkml/CA+icZUXCn2an9aNDrm+-eneSAOyGibz0W1xYhwkA5k3B3U-5vQ@mail.gmail.com/

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Nick Terrell]

> On Apr 1, 2020, at 2:50 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> 
> On Wed, Apr 1, 2020 at 4:52 AM Nick Terrell <terrelln@fb.com> wrote:
>> 
>> 
>> 
>>> On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
>>> 
>>> On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:
>>> 
>>> [...]
>>> 
>>>>> Do you plan to push this for Linux v5.7?
>>>> 
>>>> I hope that it will be accepted. From my point of view this patch set is
>>>> ready for merge, except for the maximum window size increase
>>>> requested by Petr.
>>>> 
>>> 
>>> Hi Nick,
>>> 
>>> thanks for your answer.
>>> 
>>> Did you send out a pull-request already?
>> 
>> I haven’t sent out a pull request. If that is something that I need to
>> do, or if it makes the process easier for a maintainer to merge then
>> I can do that.
>> 
> 
> [ CC  Stephen Rothwell (maintainer of Linux-next) ]
> 
> Hi Nick,
> 
> as Adam pointed out it might be good to ask "Stephen Rothwell"
> <sfr@canb.auug.org.au>
> to include your Git tree into linux-next. Stephen will give you some
> informations about that.

Thanks for the pointers Sedat and Adam! I will reach out and ask
about the best way forward. I don’t particularly care if it makes it in
5.7 or 5.8, I just want to ensure that it does get merged this time.

Best,
Nick

Re: [PATCH v3 0/8] Add support for ZSTD-compressed kernel and initramfs

[Sedat Dilek]

On Wed, Apr 1, 2020 at 9:00 PM Nick Terrell <terrelln@fb.com> wrote:
>
> > On Apr 1, 2020, at 2:50 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >
> > On Wed, Apr 1, 2020 at 4:52 AM Nick Terrell <terrelln@fb.com> wrote:
> >>
> >>
> >>
> >>> On Mar 31, 2020, at 8:11 AM, Sedat Dilek <sedat.dilek@gmail.com> wrote:
> >>>
> >>> On Tue, Mar 31, 2020 at 6:15 AM Nick Terrell <terrelln@fb.com> wrote:
> >>>
> >>> [...]
> >>>
> >>>>> Do you plan to push this for Linux v5.7?
> >>>>
> >>>> I hope that it will be accepted. From my point of view this patch set is
> >>>> ready for merge, except for the maximum window size increase
> >>>> requested by Petr.
> >>>>
> >>>
> >>> Hi Nick,
> >>>
> >>> thanks for your answer.
> >>>
> >>> Did you send out a pull-request already?
> >>
> >> I haven’t sent out a pull request. If that is something that I need to
> >> do, or if it makes the process easier for a maintainer to merge then
> >> I can do that.
> >>
> >
> > [ CC  Stephen Rothwell (maintainer of Linux-next) ]
> >
> > Hi Nick,
> >
> > as Adam pointed out it might be good to ask "Stephen Rothwell"
> > <sfr@canb.auug.org.au>
> > to include your Git tree into linux-next. Stephen will give you some
> > informations about that.
>
> Thanks for the pointers Sedat and Adam! I will reach out and ask
> about the best way forward. I don’t particularly care if it makes it in
> 5.7 or 5.8, I just want to ensure that it does get merged this time.
>

Hi Nick,

just in case you did not know:
The Documentation directory in linux-Git has a subfolder called
"process" (RST format) which gives a lot of hints.

All stuff is also available as HTML if you prefer this format.
See [0] "A guide to the Kernel Development Process".

Hope that helps you.

Regards,
- Sedat -

[0] https://www.kernel.org/doc/html/latest/process/index.html
[1] https://www.kernel.org/doc/html/latest/process/2.Process.html#next-trees
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/2.Process.rst
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/process/submitting-patches.rst