[Buildroot] Why can't I see any effects by different rootfs compression methods ??

[Buildroot] Why can't I see any effects by different rootfs compression methods ??

[Frank Ihle]

Hi,

I'm using Buildroot to generate a uImage with a kernel, which
 in the end starts busybox. I tried the different compression modes 
(kernel compression, built-in initramfs compression and the rootfs 
compression). The first 2 options worked quiet well but when it comes to
 apply different compression methodes (e.g. LZMA, bzip2) on  rootfs in make menuconfig nothing changes, neither size nor decompress time.

Why is it like that ? And how can I extract files of the generated uImage, so that i can see what really has been compressed ?

Thank you for your help.
Kind regards

Frank Ihle
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140206/05e72e76/attachment.html>

[Buildroot] Why can't I see any effects by different rootfs compression methods ??

[Arnout Vandecappelle]

On 06/02/14 10:19, Frank Ihle wrote:
> Hi,
> 
> I'm using Buildroot to generate a uImage with a kernel, which in the end
> starts busybox. I tried the different compression modes (kernel
> compression, built-in initramfs compression and the rootfs compression).
> The first 2 options worked quiet well but when it comes to apply
> different compression methodes (e.g. LZMA, bzip2) on  rootfs in make
> menuconfig nothing changes, neither size nor decompress time.
> 
> Why is it like that ? And how can I extract files of the generated
> uImage, so that i can see what really has been compressed ?

 The rootfs compression only affects the rootfs image that is generated
in output/images, not the initramfs that is linked into the kernel. The
latter is purely controlled by kernel options (which you did).

 Note that built-in initramfs compression is pointless, because it will
be compressed again with the kernel compression method - and that usually
increases the size rather than decreasing it.


 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Antw: Re: Why can't I see any effects by different rootfs compression methods ??

[Frank Ihle]

Thanks for the response, that gave me a better view. 

But now i got a question which may sound a bit confusing: what can I do with these compressed archives (like rootfs.cpio.lzo) ? I always thought I can only create bootable Images with an Initramfs/initrd ? Because when I put the (e.g) rootfs.cpio as Initramfs source file(s) in make linux-menuconfig then i got a working image in the end, but if i put rootfs.cpio.lzo then he skips the .lzo ending and uses the rootfs.cpio again (I guess that's nothing new to you).

so then I'm not sure why the rootfs can be compressed if it only can be used on host, or is it there so it can be mount e.g. in userspace ?

The other question is: about the "double" compression you mentioned. Someone told me these two compression modes are independent from each other, unfortunately i can't open those u/zImages to look what's right now. Anyway, if you want to have a fast booting linux, can't some combination of these 2 compression mode lead to a quicker startup?.

Thanks a lot for your help!

Kind Regards,

Frank

>>> Arnout Vandecappelle <arnout@mind.be> 06.02.14 20.38 Uhr >>>
On 06/02/14 10:19, Frank Ihle wrote:
> Hi,
> 
> I'm using Buildroot to generate a uImage with a kernel, which in the end
> starts busybox. I tried the different compression modes (kernel
> compression, built-in initramfs compression and the rootfs compression).
> The first 2 options worked quiet well but when it comes to apply
> different compression methodes (e.g. LZMA, bzip2) on  rootfs in make
> menuconfig nothing changes, neither size nor decompress time.
> 
> Why is it like that ? And how can I extract files of the generated
> uImage, so that i can see what really has been compressed ?

 The rootfs compression only affects the rootfs image that is generated
in output/images, not the initramfs that is linked into the kernel. The
latter is purely controlled by kernel options (which you did).

 Note that built-in initramfs compression is pointless, because it will
be compressed again with the kernel compression method - and that usually
increases the size rather than decreasing it.


 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20140207/f60fead9/attachment.html>

[Buildroot] Antw: Re: Why can't I see any effects by different rootfs compression methods ??

[Arnout Vandecappelle]

 Hi Frank,

 [Please don't top-post, but reply in-line and prune the message to the
relevant parts.]

On 07/02/14 08:14, Frank Ihle wrote:
> Thanks for the response, that gave me a better view.
> 
> But now i got a question which may sound a bit confusing: what can I do
> with these compressed archives (like rootfs.cpio.lzo) ? I always thought
> I can only create bootable Images with an Initramfs/initrd ? Because when
> I put the (e.g) rootfs.cpio as /Initramfs source file(s)/ in /make
> linux-menuconfig/ then i got a working image in the end, but if i put
> rootfs.cpio.lzo then he skips the .lzo ending and uses the rootfs.cpio
> again (I guess that's nothing new to you).

 An initrd/initramfs is normally passed by the bootloader to the kernel
as part of the boot arguments. On your PC, for instance, you'll see in /boot:

initrd.img-3.12-1-amd64
vmlinuz-3.12-1-amd64

 As an extra option, the kernel has the feature that you can link the
initramfs directly into the kernel - which has the advantage that you
only need to manage a single image instead of two. But the initramfs
linked into the kernel is not really the normal use case (although I'll
admit buildroot users use it quite often, because it's a convenient way
to make sure kernel and rootfs are in sync).

> 
> so then I'm not sure why the rootfs can be compressed if it only can be
> used on host, or is it there so it can be mount e.g. in userspace ?

 You can't mount it, but you can unpack it: it's a simple cpio image. In
fact, what the kernel does at boot time is exactly that: it creates a
tmpfs as root (the equivalent of "mount -t tmpfs root /"), then unpacks
the image into it ("lzop -d <img> | cpio -i").


> 
> The other question is: about the "double" compression you mentioned.
> Someone told me these two compression modes are independent from each
> other, 

 They are independent in the sense that you can configure them
independently. However, they are also cumulative: the initramfs is linked
into the vmlinux ELF binary, which is subsequently compressed into a
self-extractor. If you look carefully at the output of the kernel build,
you can see it from the order in which things are done:

GEN     usr/initramfs_data.cpio.gz
LINK    vmlinux
LD      arch/x86/boot/compressed/vmlinux


> unfortunately i can't open those u/zImages to look what's right
> now. Anyway, if you want to have a fast booting linux, can't some
> combination of these 2 compression mode lead to a quicker startup?.

 I expect an uncompressed initramfs inside a lzo or lz4 kernel will be
the fastest. The most important thing, however, is to keep things as
small as possible - 10% size reduction has about the same boot time
impact as a two times faster decompression algorithm.

 If your rootfs is a bit larger, it's probably best to move it to a
squashfs or something similar, because the kernel can often access
storage media much faster than the boot loader.

 Regards,
 Arnout

> 
> Thanks a lot for your help!
> 
> Kind Regards,
> 
> Frank
[snip]


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] Antw: Re: Why can't I see any effects by different rootfs compression methods ??

[Mike Zick]

On Fri, 07 Feb 2014 10:15:15 +0100
Arnout Vandecappelle <arnout@mind.be> wrote:

>  An initrd/initramfs is normally passed by the bootloader to the
> kernel as part of the boot arguments. On your PC, for instance,
> you'll see in /boot:
> 
> initrd.img-3.12-1-amd64
> vmlinuz-3.12-1-amd64
> 
>  As an extra option, the kernel has the feature that you can link the
> initramfs directly into the kernel - which has the advantage that you
> only need to manage a single image instead of two.
>

Leaving the choice to the final implementor of the system can be
important.

Linking the initramfs (or older initrd format) into the kernel binary
makes its contents subject to the kernel's (viral) GPLv2.

Keeping the kernel and the file system as separate files stops the
viral spread of the GPL license from the kernel to the file system.

When the initial file system contains proprietary code, an important
point to keep in mind.

Another point arises in the case of the GPLv3 -
If the initial file system is statically linked into the kernel, it
becomes much more difficult to replace a GPLv3 binary in the file
system for the end user (or even for the vendor) without re-building
the kernel.

- - - -

Two everyday common instances:

Google Android: Their "bootloader image" (their terms) carries the
kernel and the initial file system as two binary images.

Amazon Kindle (the e-ink models): Their u-boot loaded kernel image
has the initial file system statically linked with the kernel.
(and contains what Amazon considers proprietary code, which they
do not release, but that is off-topic here).

Mike