[Buildroot] [PATCH 0/2] fs/iso9660: fix transparent (de)compression

[Buildroot] [PATCH 0/2] fs/iso9660: fix transparent (de)compression

[Yann E. MORIN]

Hello All!

We need mkzftree, from zisofs_tools, bundled in cdrkit. This was missing
so far, so here we add it as a new package, and make iso9660 depend on
it when needed.

The kernel also needs support for compressed isofs, so we update the
test kernel config.

Finally, the kernel images must be stored uncompressed in the iso9660
filesystem, because grub does not decompresses the files it reads from
the filesystem.


Regards,
Yann E. MORIN.


The following changes since commit 55d79ed93e44e1876334f795c6313ea6f406d229

  eeprog: remove package (2018-01-02 10:10:25 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to e793abe185c20ebd853db927552156e203813613

  fs/iso9660: fix transparent (de)compression (2018-01-02 20:57:10 +0100)


----------------------------------------------------------------
Yann E. MORIN (2):
      package/zisofs_tools: new package
      fs/iso9660: fix transparent (de)compression

 fs/iso9660/iso9660.mk                               | 12 ++++++++----
 package/zisofs_tools/zisofs_tools.hash              |  5 +++++
 package/zisofs_tools/zisofs_tools.mk                | 15 +++++++++++++++
 support/testing/conf/minimal-x86-qemu-kernel.config |  1 +
 4 files changed, 29 insertions(+), 4 deletions(-)
 create mode 100644 package/zisofs_tools/zisofs_tools.hash
 create mode 100644 package/zisofs_tools/zisofs_tools.mk

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2] package/zisofs_tools: new package

[Yann E. MORIN]

To support transparent (de)compression in iso9660, we need mkzftree,
which comes from zisofs_tools, biundled with cdrkit.

However, cdrkit is a cmake package, but zisofs_tools is an autotools
package, so we need a separate package just to get mkzftree, but it is
pretty lightweight.

We just need the host variant for now,so we just add that.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---
 package/zisofs_tools/zisofs_tools.hash |  5 +++++
 package/zisofs_tools/zisofs_tools.mk   | 15 +++++++++++++++
 2 files changed, 20 insertions(+)
 create mode 100644 package/zisofs_tools/zisofs_tools.hash
 create mode 100644 package/zisofs_tools/zisofs_tools.mk

diff --git a/package/zisofs_tools/zisofs_tools.hash b/package/zisofs_tools/zisofs_tools.hash
new file mode 100644
index 0000000000..d2f594c757
--- /dev/null
+++ b/package/zisofs_tools/zisofs_tools.hash
@@ -0,0 +1,5 @@
+# From http://snapshot.debian.org/archive/debian/20141023T043132Z/pool/main/c/cdrkit/cdrkit_1.1.11-3.dsc
+sha256  d1c030756ecc182defee9fe885638c1785d35a2c2a297b4604c0e0dcc78e47da  cdrkit_1.1.11.orig.tar.gz
+
+# Locally computed
+sha256  32b1062f7da84967e7019d01ab805935caa7ab7321a7ced0e30ebe75e5df1670  3rd-party/zisofs_tools/COPYING
diff --git a/package/zisofs_tools/zisofs_tools.mk b/package/zisofs_tools/zisofs_tools.mk
new file mode 100644
index 0000000000..fd8c371fea
--- /dev/null
+++ b/package/zisofs_tools/zisofs_tools.mk
@@ -0,0 +1,15 @@
+################################################################################
+#
+# zisofs-tools
+#
+################################################################################
+
+ZISOFS_TOOLS_VERSION = 1.1.11
+ZISOFS_TOOLS_SOURCE = cdrkit_$(ZISOFS_TOOLS_VERSION).orig.tar.gz
+ZISOFS_TOOLS_SITE = http://snapshot.debian.org/archive/debian/20141023T043132Z/pool/main/c/cdrkit
+ZISOFS_TOOLS_LICENSE = GPL-2.0+
+ZISOFS_TOOLS_LICENSE_FILES = 3rd-party/zisofs_tools/COPYING
+ZISOFS_TOOLS_SUBDIR = 3rd-party/zisofs_tools
+HOST_ZISOFS_TOOLS_DEPENDENCIES = host-zlib
+
+$(eval $(host-autotools-package))
-- 
2.11.0

[Buildroot] [PATCH 2/2] fs/iso9660: fix transparent (de)compression

[Yann E. MORIN]

It needs mkzftree from zisofs_tools, so we add a dependency to it, and
we call that one explicitly (to avoid using the one from the host in
PATH.

It also needs the the uncompressed kernel image, but because it is
already in target/ so it gets compressed by mkzftree. We have two
options:
  - compress everything but the kernel image,
  - compress everything, kernel included, and recopy it later.

We choose the latter, because it is the simplest solution. So, we alwats
define the kernel-copy hook, but only register it when needed.

Finally, itneeds a kernel with support for transparent (de)compression,
so we update the existing test config.

Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 fs/iso9660/iso9660.mk                               | 12 ++++++++----
 support/testing/conf/minimal-x86-qemu-kernel.config |  1 +
 2 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index 06467262e0..66929a9ccd 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -42,11 +42,12 @@ define ROOTFS_ISO9660_CREATE_TEMPDIR
 endef
 ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_CREATE_TEMPDIR
 else ifeq ($(BR2_TARGET_ROOTFS_ISO9660_TRANSPARENT_COMPRESSION),y)
+ROOTFS_ISO9660_DEPENDENCIES += host-zisofs_tools
 ROOTFS_ISO9660_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp
-# This must be early, before we copy the bootloader files
+# This must be early, before we copy the bootloader files.
 define ROOTFS_ISO9660_MKZFTREE
 	$(RM) -rf $(ROOTFS_ISO9660_TARGET_DIR)
-	mkzftree -X -z 9 -p $(PARALLEL_JOBS) \
+	$(HOST_DIR)/bin/mkzftree -X -z 9 -p $(PARALLEL_JOBS) \
 		$(TARGET_DIR) \
 		$(ROOTFS_ISO9660_TARGET_DIR)
 endef
@@ -92,14 +93,13 @@ define ROOTFS_ISO9660_DISABLE_EXTERNAL_INITRD
 	$(SED) '/__INITRD_PATH__/d'  $(ROOTFS_ISO9660_BOOTLOADER_CONFIG_PATH)
 endef
 
-ifeq ($(ROOTFS_ISO9660_USE_INITRD),YES)
-
 # Copy the kernel to temporary filesystem
 define ROOTFS_ISO9660_COPY_KERNEL
 	$(INSTALL) -D -m 0644 $(LINUX_IMAGE_PATH) \
 		$(ROOTFS_ISO9660_TARGET_DIR)/boot/$(LINUX_IMAGE_NAME)
 endef
 
+ifeq ($(ROOTFS_ISO9660_USE_INITRD),YES)
 ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_COPY_KERNEL
 
 # If initramfs is used, disable loading the initrd as the rootfs is
@@ -119,6 +119,10 @@ ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_COPY_INITRD
 endif
 
 else # ROOTFS_ISO9660_USE_INITRD
+ifeq ($(BR2_TARGET_ROOTFS_ISO9660_TRANSPARENT_COMPRESSION),y)
+# We must use the uncompressed kernel image
+ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_COPY_KERNEL
+endif
 
 ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_DISABLE_EXTERNAL_INITRD
 
diff --git a/support/testing/conf/minimal-x86-qemu-kernel.config b/support/testing/conf/minimal-x86-qemu-kernel.config
index 4aaaab7218..cc8ee6a713 100644
--- a/support/testing/conf/minimal-x86-qemu-kernel.config
+++ b/support/testing/conf/minimal-x86-qemu-kernel.config
@@ -43,6 +43,7 @@ CONFIG_VIRTIO_MMIO=y
 CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES=y
 CONFIG_EXT4_FS=y
 CONFIG_ISO9660_FS=y
+CONFIG_ZISOFS=y
 CONFIG_JOLIET=y
 CONFIG_TMPFS=y
 CONFIG_TMPFS_POSIX_ACL=y
-- 
2.11.0

[Buildroot] [PATCH 1/2] package/zisofs_tools: new package

[Thomas Petazzoni]

Hello,

On Tue,  2 Jan 2018 21:04:11 +0100, Yann E. MORIN wrote:
> To support transparent (de)compression in iso9660, we need mkzftree,
> which comes from zisofs_tools, biundled with cdrkit.
> 
> However, cdrkit is a cmake package, but zisofs_tools is an autotools
> package, so we need a separate package just to get mkzftree, but it is
> pretty lightweight.
> 
> We just need the host variant for now,so we just add that.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

I've renamed to zisofs-tools (we more commonly use - as a separator,
and indeed your comment header in the .mk file was using this name!),
added an entry to the DEVELOPERS file, and applied. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 2/2] fs/iso9660: fix transparent (de)compression

[Thomas Petazzoni]

Hello,

On Tue,  2 Jan 2018 21:04:12 +0100, Yann E. MORIN wrote:
> It needs mkzftree from zisofs_tools, so we add a dependency to it, and
> we call that one explicitly (to avoid using the one from the host in
> PATH.
> 
> It also needs the the uncompressed kernel image, but because it is
> already in target/ so it gets compressed by mkzftree. We have two
> options:
>   - compress everything but the kernel image,
>   - compress everything, kernel included, and recopy it later.
> 
> We choose the latter, because it is the simplest solution. So, we alwats
> define the kernel-copy hook, but only register it when needed.
> 
> Finally, itneeds a kernel with support for transparent (de)compression,
> so we update the existing test config.
> 
> Reported-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> ---
>  fs/iso9660/iso9660.mk                               | 12 ++++++++----
>  support/testing/conf/minimal-x86-qemu-kernel.config |  1 +
>  2 files changed, 9 insertions(+), 4 deletions(-)

Applied to master after changing the dependency on host-zisofs-tools
following my change in PATCH 1/2, and fixing a few typos in the commit
log.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com