[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Peter Korsgaard]

Similar to the other compressors.  Notice that we use the -l (legacy format)
for Linux kernel initrd compatibility.

Lz4 decompression is supported by the Linux kernel since 3.11.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 fs/common.mk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 9a7758ff49..0fc151b294 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -59,6 +59,11 @@ ROOTFS_$(2)_DEPENDENCIES += host-lzma
 ROOTFS_$(2)_COMPRESS_EXT = .lzma
 ROOTFS_$(2)_COMPRESS_CMD = $$(LZMA) -9 -c
 endif
+ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZ4),y)
+ROOTFS_$(2)_DEPENDENCIES += host-lz4
+ROOTFS_$(2)_COMPRESS_EXT = .lz4
+ROOTFS_$(2)_COMPRESS_CMD = lz4 -l -9 -c
+endif
 ifeq ($$(BR2_TARGET_ROOTFS_$(2)_LZO),y)
 ROOTFS_$(2)_DEPENDENCIES += host-lzop
 ROOTFS_$(2)_COMPRESS_EXT = .lzo
-- 
2.11.0

[Buildroot] [PATCH 2/4] fs/cpio: add option for lz4 compression

[Peter Korsgaard]

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 fs/cpio/Config.in | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/cpio/Config.in b/fs/cpio/Config.in
index 206baca677..679959d222 100644
--- a/fs/cpio/Config.in
+++ b/fs/cpio/Config.in
@@ -31,6 +31,11 @@ config BR2_TARGET_ROOTFS_CPIO_BZIP2
 	help
 	  Do compress the cpio filesystem with bzip2.
 
+config BR2_TARGET_ROOTFS_CPIO_LZ4
+	bool "lz4"
+	help
+	  Do compress the cpio filesystem with lz4.
+
 config BR2_TARGET_ROOTFS_CPIO_LZMA
 	bool "lzma"
 	help
-- 
2.11.0

[Buildroot] [PATCH 3/4] fs/ext2: add option for lz4 compression

[Peter Korsgaard]

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 fs/ext2/Config.in | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/ext2/Config.in b/fs/ext2/Config.in
index 3884b08b76..27ad69fde9 100644
--- a/fs/ext2/Config.in
+++ b/fs/ext2/Config.in
@@ -108,6 +108,11 @@ config BR2_TARGET_ROOTFS_EXT2_BZIP2
 	help
 	  Do compress the ext2/3/4 filesystem with bzip2.
 
+config BR2_TARGET_ROOTFS_EXT2_LZ4
+	bool "lz4"
+	help
+	  Do compress the ext2 filesystem with lz4.
+
 config BR2_TARGET_ROOTFS_EXT2_LZMA
 	bool "lzma"
 	help
-- 
2.11.0

[Buildroot] [PATCH 4/4] fs/tar: add option for lz4 compression

[Peter Korsgaard]

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 fs/tar/Config.in | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/fs/tar/Config.in b/fs/tar/Config.in
index 63663ecd6f..2116edf143 100644
--- a/fs/tar/Config.in
+++ b/fs/tar/Config.in
@@ -26,6 +26,11 @@ config BR2_TARGET_ROOTFS_TAR_BZIP2
 	help
 	  Do compress the tarball with bzip2.
 
+config BR2_TARGET_ROOTFS_TAR_LZ4
+	bool "lz4"
+	help
+	  Do compress the tarball with lz4.
+
 config BR2_TARGET_ROOTFS_TAR_LZMA
 	bool "lzma"
 	help
-- 
2.11.0

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Thomas Petazzoni]

Hello,

On Fri, 13 Oct 2017 21:16:52 +0200, Peter Korsgaard wrote:
> Similar to the other compressors.  Notice that we use the -l (legacy format)
> for Linux kernel initrd compatibility.
> 
> Lz4 decompression is supported by the Linux kernel since 3.11.
> 
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

I don't know in which direction we want to go, but Yann has a series
doing cleanups/improvements in the filesystem stuff, and as part of
this, he is dropping the compression options:

  https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/fs-2

(This has not been submitted so far I believe.)

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Yann E. MORIN]

Peter, Thomas, All,

On 2017-10-14 00:05 +0200, Thomas Petazzoni spake thusly:
> On Fri, 13 Oct 2017 21:16:52 +0200, Peter Korsgaard wrote:
> > Similar to the other compressors.  Notice that we use the -l (legacy format)
> > for Linux kernel initrd compatibility.
> > 
> > Lz4 decompression is supported by the Linux kernel since 3.11.
> > 
> > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
> 
> I don't know in which direction we want to go, but Yann has a series
> doing cleanups/improvements in the filesystem stuff,

I already pointed Peter to this on-going work 10 days ago, while
discussing on IRC, and he said "ahh, nice!"  ;-)

> and as part of
> this, he is dropping the compression options:
> 
>   https://git.buildroot.org/~ymorin/git/buildroot/log/?h=yem/fs-2

Indeed. It does not make sense to get a compressedd ext2/3/4 (or
whatever other uncompressed filesystem). That is semantically useless.

One may argue that for very big images, there might not be enough place
to store it, but the image is first generated uncomoressed anyway, so we
need at least that much space to start with. Compressing the image will
not help for big images sanyway.

If one still needs to compress the image (e.g. to send to Q/A), then it
is still possible to do it in a post-iamge script, which is even more
versatile (it can sign and/or encrypt as well, for example).

The cpio case is a bit different, but nonetheless the outcome is the
same: we don't need to compress it.

When used as an initramfs, the kernel will compress it anyway, so that
would be a double compression; bad. When used as an initrd with the
iso9660 filesystem, it is it that should compress it.

The only questionable case that remains is the cpio as used as a pure
initrd. Does that still exist? And for that single use-case, I would
still suggest a post-image script, because I don't see the point of
having support in the infra for compression, when only a single image
uses it. OR at the very least, that compressing cpio is entirely moved
to the cpio fs.

> (This has not been submitted so far I believe.)

I still need to do a few tests of it, and I shall submit it here by the
end of the week (so we can discuss it during the dev-days).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Peter, Thomas, All,
 > On 2017-10-14 00:05 +0200, Thomas Petazzoni spake thusly:
 >> On Fri, 13 Oct 2017 21:16:52 +0200, Peter Korsgaard wrote:
 >> > Similar to the other compressors.  Notice that we use the -l (legacy format)
 >> > for Linux kernel initrd compatibility.
 >> > 
 >> > Lz4 decompression is supported by the Linux kernel since 3.11.
 >> > 
 >> > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
 >> 
 >> I don't know in which direction we want to go, but Yann has a series
 >> doing cleanups/improvements in the filesystem stuff,

 > I already pointed Peter to this on-going work 10 days ago, while
 > discussing on IRC, and he said "ahh, nice!"  ;-)

Ahh, the discussion about fixing the issues around toplevel parallel
build and filesystem creation? I haven't looked at your patches, I just
commented that it was nice that you were taking care of it ;)


 > Indeed. It does not make sense to get a compressedd ext2/3/4 (or
 > whatever other uncompressed filesystem). That is semantically useless.

I agree that we would most likely not introduce such options today, but
they have been supported for a very long time. I have personally used
the .gz option to save archival disk space for buildbot, so they are not
completely useless ;)

It is not completely semantically useless as it can be used for
old-style initrds (but yeah, using an initramfs makes more sense).


 > One may argue that for very big images, there might not be enough place
 > to store it, but the image is first generated uncomoressed anyway, so we
 > need at least that much space to start with. Compressing the image will
 > not help for big images sanyway.

Except if you are archiving the output/images directory and want to
limit long term disk space like explained above.


 > If one still needs to compress the image (e.g. to send to Q/A), then it
 > is still possible to do it in a post-iamge script, which is even more
 > versatile (it can sign and/or encrypt as well, for example).

Indeed.


 > The cpio case is a bit different, but nonetheless the outcome is the
 > same: we don't need to compress it.

 > When used as an initramfs, the kernel will compress it anyway, so that
 > would be a double compression; bad. When used as an initrd with the
 > iso9660 filesystem, it is it that should compress it.

You are missing the external initramfs usecase (E.G. not built into the
kernel) - How just about all PC distributions boot. This usecase
probably doesn't make much sense for Buildroot, but such a setup does
make sense when you use a dm-verity / dm-crypt rootfs and need to mount
it in the initramfs.

For thes use cases, the compression options IMHO makes sense.


 > The only questionable case that remains is the cpio as used as a pure
 > initrd. Does that still exist?

Yes, that is similar to the initramfs-inside-kernel usecase, but here
the compression options do make sense. There may be some usecases where
it is interesting to keep the rootfs and kernel separately from each
other.

 > And for that single use-case, I would still suggest a post-image
 > script, because I don't see the point of having support in the infra
 > for compression, when only a single image uses it. OR at the very
 > least, that compressing cpio is entirely moved to the cpio fs.

In concept I agree, but you could use that logic on a lot of the things
Buildroot does. We have supported these options for a very long time,
and they imho add very little complexity, so I suggest we keep them for
compatibility reasons.

Regarding the question of if we should add lz4 as well.  I'm fairly
pragmatic about it. Most likely other people may also want it, so either
we add it or create a visible (Config.in.host) host package so it can be
used from post-image scripts.

For consistency I would prefer to add it to the compression options.


 >> (This has not been submitted so far I believe.)

 > I still need to do a few tests of it, and I shall submit it here by the
 > end of the week (so we can discuss it during the dev-days).

Great, thanks!

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Yann E. MORIN]

Peter, All,

On 2017-10-15 11:03 +0200, Peter Korsgaard spake thusly:
> >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> 
>  > Peter, Thomas, All,
>  > On 2017-10-14 00:05 +0200, Thomas Petazzoni spake thusly:
>  >> On Fri, 13 Oct 2017 21:16:52 +0200, Peter Korsgaard wrote:
>  >> > Similar to the other compressors.  Notice that we use the -l (legacy format)
>  >> > for Linux kernel initrd compatibility.
>  >> > 
>  >> > Lz4 decompression is supported by the Linux kernel since 3.11.
>  >> > 
>  >> > Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[--SNIP--]
>  > Indeed. It does not make sense to get a compressedd ext2/3/4 (or
>  > whatever other uncompressed filesystem). That is semantically useless.
> 
> I agree that we would most likely not introduce such options today, but
> they have been supported for a very long time. I have personally used
> the .gz option to save archival disk space for buildbot, so they are not
> completely useless ;)

But since the filesystem infra does not get rid of the uncompresed
image, surely you have a post-image (or post-buildroot) script that gets
rid of the uncompressed stuff, don't you? In which case that archival
script could very well be the one doing the compresion...

And Buildroot should not be concerned about the archiving requirements
of the users: there will be almost as many such archiving requirements
as there are users.

> It is not completely semantically useless as it can be used for
> old-style initrds (but yeah, using an initramfs makes more sense).

The external initrd is what I was talking about below, yes.

>  > One may argue that for very big images, there might not be enough place
>  > to store it, but the image is first generated uncomoressed anyway, so we
>  > need at least that much space to start with. Compressing the image will
>  > not help for big images sanyway.
> 
> Except if you are archiving the output/images directory and want to
> limit long term disk space like explained above.

That should not be the responsibility of Buildroot. And if you want to
archive something, then you have much more to take care of than just the
images, so you'll anyway have to provide a post-image (or post-buildroot)
script anyway...

>  > If one still needs to compress the image (e.g. to send to Q/A), then it
>  > is still possible to do it in a post-iamge script, which is even more
>  > versatile (it can sign and/or encrypt as well, for example).
> Indeed.
> 
>  > The cpio case is a bit different, but nonetheless the outcome is the
>  > same: we don't need to compress it.
> 
>  > When used as an initramfs, the kernel will compress it anyway, so that
>  > would be a double compression; bad. When used as an initrd with the
>  > iso9660 filesystem, it is it that should compress it.
> 
> You are missing the external initramfs usecase (E.G. not built into the
> kernel) - How just about all PC distributions boot. This usecase
> probably doesn't make much sense for Buildroot, but such a setup does
> make sense when you use a dm-verity / dm-crypt rootfs and need to mount
> it in the initramfs.
> 
> For thes use cases, the compression options IMHO makes sense.

Well, the compression is indeed only valid when using an initrd (cpio,
ext, whatev') or an external initramfs.

But again, I still think that it's better to offload that to an external
script.

>  > The only questionable case that remains is the cpio as used as a pure
>  > initrd. Does that still exist?
> 
> Yes, that is similar to the initramfs-inside-kernel usecase, but here
> the compression options do make sense. There may be some usecases where
> it is interesting to keep the rootfs and kernel separately from each
> other.
> 
>  > And for that single use-case, I would still suggest a post-image
>  > script, because I don't see the point of having support in the infra
>  > for compression, when only a single image uses it. OR at the very
>  > least, that compressing cpio is entirely moved to the cpio fs.
> 
> In concept I agree, but you could use that logic on a lot of the things
> Buildroot does. We have supported these options for a very long time,
> and they imho add very little complexity, so I suggest we keep them for
> compatibility reasons.

Well, we historically had support for the toolchain on the target as
well, but we did eventually get rid of it! ;-)

> Regarding the question of if we should add lz4 as well.  I'm fairly
> pragmatic about it. Most likely other people may also want it, so either
> we add it or create a visible (Config.in.host) host package so it can be
> used from post-image scripts.
> 
> For consistency I would prefer to add it to the compression options.

Note: I am not arguing against adding lz4. I'm arguing against keeping
the compresion options altogether. Slight difference.

What I find cumbersome to maintain is the duplication of compression
lists in each filesystem types.

Maybe a middle ground would be to remove all the compression options,
and move them to the top-level filesystem menu, something like:

    Image compression:
      (X) none
      ( ) gzip
      ( ) bzip2
      ( ) xz
      ( ) lzo
      ( ) lz4
      ( ) lzp

With this help text:

    For filesystems that inherently do not support compression
    (e.g. ext2, cpio, tarball...), compress them with this
    compresion algorithm (using the default compression level)

    Filesystem that already support internal compression (e.g.
    squashfs, cramfs...) are not impacted by this option.

Note that we can *not* remove the uncompresed image, because it may
serve as input to another filesystem.

Regards,
Yann E. MORIN.

>  >> (This has not been submitted so far I believe.)
>  > I still need to do a few tests of it, and I shall submit it here by the
>  > end of the week (so we can discuss it during the dev-days).
> Great, thanks!

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

Hi,

 >> > Indeed. It does not make sense to get a compressedd ext2/3/4 (or
 >> > whatever other uncompressed filesystem). That is semantically useless.
 >> 
 >> I agree that we would most likely not introduce such options today, but
 >> they have been supported for a very long time. I have personally used
 >> the .gz option to save archival disk space for buildbot, so they are not
 >> completely useless ;)

 > But since the filesystem infra does not get rid of the uncompresed
 > image, surely you have a post-image (or post-buildroot) script that gets
 > rid of the uncompressed stuff, don't you? In which case that archival
 > script could very well be the one doing the compresion...

Correct. This was just a (poor) example where I personally have used the
option.


 > And Buildroot should not be concerned about the archiving requirements
 > of the users: there will be almost as many such archiving requirements
 > as there are users.

 >> It is not completely semantically useless as it can be used for
 >> old-style initrds (but yeah, using an initramfs makes more sense).

 > The external initrd is what I was talking about below, yes.

Ok. Keep in mind that there's 3 variants:

- Initramfs built into the kernel
- External initramfs (separate cpio archive)
- External initrd (filesystem image, E.G. extN)


 >> You are missing the external initramfs usecase (E.G. not built into the
 >> kernel) - How just about all PC distributions boot. This usecase
 >> probably doesn't make much sense for Buildroot, but such a setup does
 >> make sense when you use a dm-verity / dm-crypt rootfs and need to mount
 >> it in the initramfs.
 >> 
 >> For thes use cases, the compression options IMHO makes sense.

 > Well, the compression is indeed only valid when using an initrd (cpio,
 > ext, whatev') or an external initramfs.

 > But again, I still think that it's better to offload that to an external
 > script.

Tradeoffs, a script is more flexible, but an option is more user friendly.


 >> In concept I agree, but you could use that logic on a lot of the things
 >> Buildroot does. We have supported these options for a very long time,
 >> and they imho add very little complexity, so I suggest we keep them for
 >> compatibility reasons.

 > Well, we historically had support for the toolchain on the target as
 > well, but we did eventually get rid of it! ;-)

The reason why we got rid of it was that it was hard to test, nontrivial
to maintain and against the philosophy of Buildroot (cross compilation).

I don't think those things really applies here. Sure, it could be done
by a script, but so could E.G. the root password option, remount rw,
purge locales, ..


 >> Regarding the question of if we should add lz4 as well.  I'm fairly
 >> pragmatic about it. Most likely other people may also want it, so either
 >> we add it or create a visible (Config.in.host) host package so it can be
 >> used from post-image scripts.
 >> 
 >> For consistency I would prefer to add it to the compression options.

 > Note: I am not arguing against adding lz4. I'm arguing against keeping
 > the compresion options altogether. Slight difference.

I understand that.


 > What I find cumbersome to maintain is the duplication of compression
 > lists in each filesystem types.

I agree that it isn't very nice, but we only rarely add new filesystem
types or compression algorithms (and if we do, the patches are trivial
as this lz4 series).


 > Maybe a middle ground would be to remove all the compression options,
 > and move them to the top-level filesystem menu, something like:

 >     Image compression:
 >       (X) none
 >       ( ) gzip
 >       ( ) bzip2
 >       ( ) xz
 >       ( ) lzo
 >       ( ) lz4
 >       ( ) lzp

 > With this help text:

 >     For filesystems that inherently do not support compression
 >     (e.g. ext2, cpio, tarball...), compress them with this
 >     compresion algorithm (using the default compression level)

 >     Filesystem that already support internal compression (e.g.
 >     squashfs, cramfs...) are not impacted by this option.

Hmm, I don't really like this. It still breaks the build for existing
users, and it doesn't really simplify anything in our infrastructure
(except for a few Config.in entries).

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 1/4] fs/common.mk: support lz4 compression

[Thomas Petazzoni]

Hello,

On Fri, 13 Oct 2017 21:16:52 +0200, Peter Korsgaard wrote:
> Similar to the other compressors.  Notice that we use the -l (legacy format)
> for Linux kernel initrd compatibility.
> 
> Lz4 decompression is supported by the Linux kernel since 3.11.
> 
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
> ---
>  fs/common.mk | 5 +++++
>  1 file changed, 5 insertions(+)

I've applied the series. I'm also not super convinced we need to
support all those compression options, but we already have plenty of
them, and Peter is just adding one more, which doesn't make the whole
thing much more complicated.

We can continue the discussion on whether we want to keep this or not,
especially in light of http://patchwork.ozlabs.org/patch/850055/ that
proposes to allow compressing the sdcard.img generated by genimage.

If we allow compression rootfs.tar, why not sdcard.img ?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com