[master][PATCH] image_types: add 7-Zip support in conversion types and commands

[master][PATCH] image_types: add 7-Zip support in conversion types and commands

[Livius]

[-- Attachment #1: Type: text/plain, Size: 3131 bytes --]

I added to support 7-Zip in conversion types/commands. It is fully configurable in compression level, method and file extension.

From: "Benjamin Szőke" <egyszeregy@freemail.hu>
Date: Thu, 25 Aug 2022 21:45:55 +0200
Subject: [PATCH] image_types: add 7-Zip support in conversion types and commands

---
meta/classes-recipe/image_types.bbclass | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index a731e585b2..94aa1d9510 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -59,6 +59,10 @@ XZ_INTEGRITY_CHECK ?= "crc32"

ZIP_COMPRESSION_LEVEL ?= "-9"

+7ZIP_COMPRESSION_LEVEL ?= "9"
+7ZIP_COMPRESSION_METHOD ?= "BZip2"
+7ZIP_EXTENSION ?= "zip"
+
ZSTD_COMPRESSION_LEVEL ?= "-3"

JFFS2_SUM_EXTRA_ARGS ?= ""
@@ -296,7 +300,7 @@ IMAGE_TYPES:append:x86-64 = " hddimg iso"
# CONVERSION_CMD/DEPENDS.
COMPRESSIONTYPES ?= ""

-CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip zst sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vhd vhdx vdi qcow2 base64 gzsync zsync ${COMPRESSIONTYPES}"
+CONVERSIONTYPES = "gz bz2 lzma xz lz4 lzo zip 7zip zst sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap u-boot vmdk vhd vhdx vdi qcow2 base64 gzsync zsync ${COMPRESSIONTYPES}"
CONVERSION_CMD:lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD:gz = "gzip -f -9 -n -c --rsyncable ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
CONVERSION_CMD:bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -304,6 +308,7 @@ CONVERSION_CMD:xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_DEFAULTS} --check=
CONVERSION_CMD:lz4 = "lz4 -9 -z -l ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
CONVERSION_CMD:lzo = "lzop -9 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD:zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
+CONVERSION_CMD:7zip = "7za a -mx=${7ZIP_COMPRESSION_LEVEL} -mm=${7ZIP_COMPRESSION_METHOD} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.${7ZIP_EXTENSION} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
CONVERSION_CMD:zst = "zstd -f -k -T0 -c ${ZSTD_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zst"
CONVERSION_CMD:sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
CONVERSION_CMD:md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
@@ -329,6 +334,7 @@ CONVERSION_DEPENDS_xz = "xz-native"
CONVERSION_DEPENDS_lz4 = "lz4-native"
CONVERSION_DEPENDS_lzo = "lzop-native"
CONVERSION_DEPENDS_zip = "zip-native"
+CONVERSION_DEPENDS_7zip = "p7zip-native"
CONVERSION_DEPENDS_zst = "zstd-native"
CONVERSION_DEPENDS_sum = "mtd-utils-native"
CONVERSION_DEPENDS_bmap = "bmap-tools-native"
--
2.35.1.windows.2

[-- Attachment #2: Type: text/html, Size: 3948 bytes --]

Re: [OE-core] [master][PATCH] image_types: add 7-Zip support in conversion types and commands

[Luca Ceresoli]

Hello Livius,

thank you for your patch!

I have a few comments, see below.

On Thu, 25 Aug 2022 12:59:34 -0700
"Livius" <egyszeregy@freemail.hu> wrote:

> I added to support 7-Zip in conversion types/commands. It is fully configurable in compression level, method and file extension.
> 
> From: "Benjamin Szőke" <egyszeregy@freemail.hu>
> Date: Thu, 25 Aug 2022 21:45:55 +0200
> Subject: [PATCH] image_types: add 7-Zip support in conversion types and commands

These 3 lines should be in the mail header, not in the body. All the
lines that appear in the body before the '---' line will be part of the
commit message, and of course we don't want these 3 lines in the commit
message.

I recommend you to use 'git send-email' to send patches.

See this link  (from README.md in the meta-yocto sources) for more info:
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded

> ---
> meta/classes-recipe/image_types.bbclass | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index a731e585b2..94aa1d9510 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -59,6 +59,10 @@ XZ_INTEGRITY_CHECK ?= "crc32"
> 
> ZIP_COMPRESSION_LEVEL ?= "-9"
> 
> +7ZIP_COMPRESSION_LEVEL ?= "9"
> +7ZIP_COMPRESSION_METHOD ?= "BZip2"

Did you run some performance test to check that this is actually a good
balance between compression ratio and compression time?

> +7ZIP_EXTENSION ?= "zip"

I think the 7zip default extension is '7z', why defaulting to 'zip'?
Seems like generating confusion for no reason.

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [master][PATCH] image_types: add 7-Zip support in conversion types and commands

[Livius]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

I can not use 'git send-email' because my workstation is not configured to be able to send any mail. You can use my patch, the content is available under '--' what is need to for it, or I can copy and paste it and re-send to this thread. But in 2022, I really recommend to you to improve your contributing processes more. It could be much better in a modern pull-request ticket in GitHub, GitLab or Bitbucket portal then this historical mode. That kind of process is too legacy, and it can not help in developing speed and to get more-more users/contributors who want to help in daily developing.

My goal was replacing wic.zip conversion, because it is extremely slowly for an image which is several GB sizes. Problem is that normal zip can operate only in single-thread mode, but 7-zip can do it in multi-thread mode in much faster if I use BZip2 method for a zip archive. I liked to provide .zip extension because BZip2 support it, and it is fine for multiplatform usage, in 100% you can unzip it in Windows, Linux and OSX too. So, my new wic.7zip can operate in 100% CPU load in each 8 cores of my CPU, therefore this is the best for making a .zip file for wic image. But if someone wants to use any other mode, they can do with variables overriding.

[-- Attachment #2: Type: text/html, Size: 1297 bytes --]

Re: [master][PATCH] image_types: add 7-Zip support in conversion types and commands

[Livius]

[-- Attachment #1: Type: text/plain, Size: 111 bytes --]

I resend this patch via " git send-email".
https://lists.openembedded.org/g/openembedded-core/message/172765

[-- Attachment #2: Type: text/html, Size: 158 bytes --]