[Buildroot] [PATCH 0/2] fs/iso9660: add transparent compression

[Buildroot] [PATCH 0/2] fs/iso9660: add transparent compression

[Yann E. MORIN]

Hello All!

This two-patch series introduces support for generating iso9660 images
with compressed files in them, using transparent (de)compression.


Regards,
Yann E. MORIN.


The following changes since commit 5b85c6a038cc210355d8d5715cdf6fa73d18e8ac

  pulseaudio: fix libsamplerate dependency (2017-12-20 21:37:18 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to d9148095910e211f1b38a85f93a10ab88fa1af04

  core/tests: add test for compressed iso9660 (2017-12-20 22:12:23 +0100)


----------------------------------------------------------------
Yann E. MORIN (2):
      fs/iso9660: add option for transparent (de)compression
      core/tests: add test for compressed iso9660

 .gitlab-ci.yml                           |  2 ++
 fs/iso9660/Config.in                     |  9 +++++++
 fs/iso9660/iso9660.mk                    | 12 ++++++++++
 support/testing/tests/fs/test_iso9660.py | 41 ++++++++++++++++++++++++++++++++
 4 files changed, 64 insertions(+)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2] fs/iso9660: add option for transparent (de)compression

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 fs/iso9660/Config.in  |  9 +++++++++
 fs/iso9660/iso9660.mk | 12 ++++++++++++
 2 files changed, 21 insertions(+)

diff --git a/fs/iso9660/Config.in b/fs/iso9660/Config.in
index 842c987f21..1457c80569 100644
--- a/fs/iso9660/Config.in
+++ b/fs/iso9660/Config.in
@@ -63,6 +63,15 @@ config BR2_TARGET_ROOTFS_ISO9660_INITRD
 	  contain a kernel image, an initrd image (unless an initramfs
 	  linked into the kernel is used) and the bootloader.
 
+config BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS
+	bool "transparent compression"
+	depends on !BR2_TARGET_ROOTFS_ISO9660_INITRD
+	depends on !BR2_TARGET_ROOTFS_INITRAMFS
+	help
+	  Say 'y' to enable use of transparent (de)compression. Files
+	  are stored compressed and will be decompressed on-the-fly
+	  upon access at runtime.
+
 config BR2_TARGET_ROOTFS_ISO9660_HYBRID
 	bool "Build hybrid image"
 	depends on BR2_TARGET_ROOTFS_ISO9660_ISOLINUX
diff --git a/fs/iso9660/iso9660.mk b/fs/iso9660/iso9660.mk
index c2de27101a..771ad4eceb 100644
--- a/fs/iso9660/iso9660.mk
+++ b/fs/iso9660/iso9660.mk
@@ -41,6 +41,17 @@ define ROOTFS_ISO9660_CREATE_TEMPDIR
 	mkdir -p $(ROOTFS_ISO9660_TARGET_DIR)
 endef
 ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_CREATE_TEMPDIR
+else ifeq ($(BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS),y)
+ROOTFS_ISO9660_TARGET_DIR = $(FS_DIR)/rootfs.iso9660.tmp
+# This must be early, before we copy the bootloader files
+define ROOTFS_ISO9660_MKZFTREE
+	$(RM) -rf $(ROOTFS_ISO9660_TARGET_DIR)
+	mkzftree -X -z 9 -p $(PARALLEL_JOBS) \
+		$(TARGET_DIR) \
+		$(ROOTFS_ISO9660_TARGET_DIR)
+endef
+ROOTFS_ISO9660_PRE_GEN_HOOKS += ROOTFS_ISO9660_MKZFTREE
+ROOTFS_ISO9660_GENISOIMAGE_OPTS += -z
 else
 ROOTFS_ISO9660_TARGET_DIR = $(TARGET_DIR)
 endif
@@ -117,6 +128,7 @@ endif # ROOTFS_ISO9660_USE_INITRD
 define ROOTFS_ISO9660_CMD
 	$(HOST_DIR)/bin/genisoimage -J -R -b $(ROOTFS_ISO9660_BOOT_IMAGE) \
 		-no-emul-boot -boot-load-size 4 -boot-info-table \
+		$(ROOTFS_ISO9660_GENISOIMAGE_OPTS) \
 		-o $@ $(ROOTFS_ISO9660_TARGET_DIR)
 endef
 
-- 
2.11.0

[Buildroot] [PATCH 2/2] core/tests: add test for compressed iso9660

[Yann E. MORIN]

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
---
 .gitlab-ci.yml                           |  2 ++
 support/testing/tests/fs/test_iso9660.py | 41 ++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c81697920b..8b35e18003 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -256,8 +256,10 @@ tests.fs.test_ext.TestExt2r1: *runtime_test
 tests.fs.test_ext.TestExt3: *runtime_test
 tests.fs.test_ext.TestExt4: *runtime_test
 tests.fs.test_iso9660.TestIso9660Grub2External: *runtime_test
+tests.fs.test_iso9660.TestIso9660Grub2ExternalCompress: *runtime_test
 tests.fs.test_iso9660.TestIso9660Grub2Internal: *runtime_test
 tests.fs.test_iso9660.TestIso9660SyslinuxExternal: *runtime_test
+tests.fs.test_iso9660.TestIso9660SyslinuxExternalCompress: *runtime_test
 tests.fs.test_iso9660.TestIso9660SyslinuxInternal: *runtime_test
 tests.fs.test_jffs2.TestJffs2: *runtime_test
 tests.fs.test_squashfs.TestSquashfs: *runtime_test
diff --git a/support/testing/tests/fs/test_iso9660.py b/support/testing/tests/fs/test_iso9660.py
index 5d945a347a..157105d04b 100644
--- a/support/testing/tests/fs/test_iso9660.py
+++ b/support/testing/tests/fs/test_iso9660.py
@@ -67,6 +67,27 @@ class TestIso9660Grub2External(infra.basetest.BRTest):
         self.assertEqual(exit_code, 1)
 
 
+class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest):
+    config = BASIC_CONFIG + \
+        """
+        BR2_TARGET_ROOTFS_ISO9660=y
+        # BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+        BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS
+        BR2_TARGET_GRUB2=y
+        BR2_TARGET_GRUB2_BOOT_PARTITION="cd"
+        BR2_TARGET_GRUB2_BUILTIN_MODULES="boot linux ext2 fat part_msdos part_gpt normal biosdisk iso9660"
+        BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+        """.format(infra.filepath("conf/grub2.cfg"))
+
+    def test_run(self):
+        exit_code = test_mount_internal_external(self.emulator,
+                                                 self.builddir, internal=False)
+        self.assertEqual(exit_code, 0)
+
+        exit_code = test_touch_file(self.emulator)
+        self.assertEqual(exit_code, 1)
+
+
 class TestIso9660Grub2Internal(infra.basetest.BRTest):
     config = BASIC_CONFIG + \
         """
@@ -109,6 +130,26 @@ class TestIso9660SyslinuxExternal(infra.basetest.BRTest):
         self.assertEqual(exit_code, 1)
 
 
+class TestIso9660SyslinuxExternalCompress(infra.basetest.BRTest):
+    config = BASIC_CONFIG + \
+        """
+        BR2_TARGET_ROOTFS_ISO9660=y
+        # BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
+        BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS
+        BR2_TARGET_ROOTFS_ISO9660_HYBRID=y
+        BR2_TARGET_ROOTFS_ISO9660_BOOT_MENU="{}"
+        BR2_TARGET_SYSLINUX=y
+        """.format(infra.filepath("conf/isolinux.cfg"))
+
+    def test_run(self):
+        exit_code = test_mount_internal_external(self.emulator,
+                                                 self.builddir, internal=False)
+        self.assertEqual(exit_code, 0)
+
+        exit_code = test_touch_file(self.emulator)
+        self.assertEqual(exit_code, 1)
+
+
 class TestIso9660SyslinuxInternal(infra.basetest.BRTest):
     config = BASIC_CONFIG + \
         """
-- 
2.11.0

[Buildroot] [PATCH 1/2] fs/iso9660: add option for transparent (de)compression

[Thomas Petazzoni]

Hello,

On Wed, 20 Dec 2017 22:27:42 +0100, Yann E. MORIN wrote:

> +config BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS

I found the abbreviation to TRANSP_COMPRESS a little bit useless, so
I've changed the name of the option to _TRANSPARENT_COMPRESSION
instead. Applied with this change. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 2/2] core/tests: add test for compressed iso9660

[Thomas Petazzoni]

Hello,

On Wed, 20 Dec 2017 22:27:43 +0100, Yann E. MORIN wrote:

> +class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest):
> +    config = BASIC_CONFIG + \
> +        """
> +        BR2_TARGET_ROOTFS_ISO9660=y
> +        # BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
> +        BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS

Not sure you really tested with transparent compression enabled, since
=y was missing. So I've fixed that, taken into account the option
rename, and applied. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 2/2] core/tests: add test for compressed iso9660

[Yann E. MORIN]

Thomas, All,

On 2017-12-31 18:23 +0100, Thomas Petazzoni spake thusly:
> On Wed, 20 Dec 2017 22:27:43 +0100, Yann E. MORIN wrote:
> > +class TestIso9660Grub2ExternalCompress(infra.basetest.BRTest):
> > +    config = BASIC_CONFIG + \
> > +        """
> > +        BR2_TARGET_ROOTFS_ISO9660=y
> > +        # BR2_TARGET_ROOTFS_ISO9660_INITRD is not set
> > +        BR2_TARGET_ROOTFS_ISO9660_TRANSP_COMPRESS
> Not sure you really tested with transparent compression enabled, since
> =y was missing. So I've fixed that, taken into account the option
> rename, and applied. Thanks!

I'm pretty sure that got tested:
    https://gitlab.com/ymorin/buildroot-ci/pipelines/15356311

But it proabably got ignored, so the filesystem was not compression
enabled.

Thanks for the fix.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'