[Buildroot] [PATCH V2] package/lzma-alone: new package

[Buildroot] [PATCH V2] package/lzma-alone: new package

[Rafał Miłecki]

From: Rafał Miłecki <rafal@milecki.pl>

This new package provides "lzma_alone" host binary based on the original
LZMA SDK. It provides few extra options when compared to the LZMA Utils
/ XZ Utils project "lzma" binary (already packaged as the "lzma").

This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows
Debian's solution. Please note that Debian also uses LZMA SDK for the
base "lzma" tool which may be considered for Buildroot too - as an
independent change. Similar packaging is also used by Ubuntu & Arch.

lzma_alone is a requirement for preparing firmware images for some
Broadcom based home routers with a picky CFE bootloader. It has limited
LZMA support and building compatible images requires specifying
dictionary size and lc/lp/pb LZMA values manually.

Version 9.22 is used as it's the last release using .tar.bz2 format. The
same version is used by Debian.

Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
---
Hi, this is V2 of my following old attempt:

[PATCH] package/lzma-alone: new package
https://patchwork.ozlabs.org/project/buildroot/patch/20201026213433.4309-1-zajec5@gmail.com/

It doesn't seem like I'm going to get V1 properly reviewed / commented
so let me try with V2 with changes I think were expected.
---
 package/Config.in.host             |  1 +
 package/lzma-alone/Config.in.host  | 18 ++++++++++++++++++
 package/lzma-alone/lzma-alone.hash |  1 +
 package/lzma-alone/lzma-alone.mk   | 22 ++++++++++++++++++++++
 4 files changed, 42 insertions(+)
 create mode 100644 package/lzma-alone/Config.in.host
 create mode 100644 package/lzma-alone/lzma-alone.hash
 create mode 100644 package/lzma-alone/lzma-alone.mk

diff --git a/package/Config.in.host b/package/Config.in.host
index e5fbb07fc2..cd1c34b5a3 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -50,6 +50,7 @@ menu "Host utilities"
 	source "package/lld/Config.in.host"
 	source "package/lpc3250loader/Config.in.host"
 	source "package/lttng-babeltrace/Config.in.host"
+	source "package/lzma-alone/Config.in.host"
 	source "package/mender-artifact/Config.in.host"
 	source "package/meson-tools/Config.in.host"
 	source "package/mfgtools/Config.in.host"
diff --git a/package/lzma-alone/Config.in.host b/package/lzma-alone/Config.in.host
new file mode 100644
index 0000000000..053b869635
--- /dev/null
+++ b/package/lzma-alone/Config.in.host
@@ -0,0 +1,18 @@
+config BR2_PACKAGE_HOST_LZMA_ALONE
+	bool "host lzma-alone"
+	help
+	  LZMA has two well-known command-line tools:
+	  1. lzma from LZMA SDK
+	  2. lzma from XZ Utils (previously LZMA Utils)
+
+	  This package provides the former one which:
+	  1. Gives more control over compression parameters
+	  2. Gets named lzma_alone to allow coexistence
+
+	  This tool may be used to prepare firmwares for bootloaders
+	  with limited LZMA support.
+	  E.g. Broadcom's CFE bootloader for MIPS BCM47xx devices is
+	  known to fail to boot images using LZMA compression with
+	  dictionary.
+
+	  https://www.7-zip.org/sdk.html
diff --git a/package/lzma-alone/lzma-alone.hash b/package/lzma-alone/lzma-alone.hash
new file mode 100644
index 0000000000..39759c4e3b
--- /dev/null
+++ b/package/lzma-alone/lzma-alone.hash
@@ -0,0 +1 @@
+sha256	9aade84f229fb25f7aef39d8866b375fe6d35a9e18098d7cd86a99e294902944	lzma922.tar.bz2
diff --git a/package/lzma-alone/lzma-alone.mk b/package/lzma-alone/lzma-alone.mk
new file mode 100644
index 0000000000..6258bb4a95
--- /dev/null
+++ b/package/lzma-alone/lzma-alone.mk
@@ -0,0 +1,22 @@
+################################################################################
+#
+# lzma-alone
+#
+################################################################################
+
+LZMA_ALONE_VERSION = 9.22
+LZMA_ALONE_SITE = https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK
+LZMA_ALONE_SOURCE = lzma922.tar.bz2
+LZMA_ALONE_STRIP_COMPONENTS = 0
+LZMA_ALONE_LICENSE = LGPL-2.1-or-later, BSD-3-Clause
+
+define HOST_LZMA_ALONE_BUILD_CMDS
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/C/Util/Lzma -f makefile.gcc
+	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/CPP/7zip/Bundles/LzmaCon -f makefile.gcc
+endef
+
+define HOST_LZMA_ALONE_INSTALL_CMDS
+	$(INSTALL) -D -m 0755 $(@D)/CPP/7zip/Bundles/LzmaCon/lzma $(HOST_DIR)/bin/lzma_alone
+endef
+
+$(eval $(host-generic-package))
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2] package/lzma-alone: new package

[Yann E. MORIN]

Rafał, All,

On 2022-03-08 19:35 +0100, Rafał Miłecki spake thusly:
> From: Rafał Miłecki <rafal@milecki.pl>
> 
> This new package provides "lzma_alone" host binary based on the original
> LZMA SDK. It provides few extra options when compared to the LZMA Utils
> / XZ Utils project "lzma" binary (already packaged as the "lzma").
> 
> This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows
> Debian's solution. Please note that Debian also uses LZMA SDK for the
> base "lzma" tool which may be considered for Buildroot too - as an
> independent change. Similar packaging is also used by Ubuntu & Arch.
> 
> lzma_alone is a requirement for preparing firmware images for some
> Broadcom based home routers with a picky CFE bootloader. It has limited
> LZMA support and building compatible images requires specifying
> dictionary size and lc/lp/pb LZMA values manually.
> 
> Version 9.22 is used as it's the last release using .tar.bz2 format.

We could introduce a host variant of p7zip, so that we could
automatically add that to FOO_EXTRACT_DEPENDENCIES if the host does not
already have 7za (or 7zr), like we do for lzip archives for example.

> The
> same version is used by Debian.
> 
> Signed-off-by: Rafał Miłecki <rafal@milecki.pl>
> ---
[--SNIP--]
> diff --git a/package/lzma-alone/lzma-alone.hash b/package/lzma-alone/lzma-alone.hash
> new file mode 100644
> index 0000000000..39759c4e3b
> --- /dev/null
> +++ b/package/lzma-alone/lzma-alone.hash
> @@ -0,0 +1 @@
> +sha256	9aade84f229fb25f7aef39d8866b375fe6d35a9e18098d7cd86a99e294902944	lzma922.tar.bz2

Fields must be separated by two spaces; fixed.

I also added the sha1 that is provided int he SF page, and approipriate
comments.

> diff --git a/package/lzma-alone/lzma-alone.mk b/package/lzma-alone/lzma-alone.mk
> new file mode 100644
> index 0000000000..6258bb4a95
> --- /dev/null
> +++ b/package/lzma-alone/lzma-alone.mk
> @@ -0,0 +1,22 @@
> +################################################################################
> +#
> +# lzma-alone
> +#
> +################################################################################
> +
> +LZMA_ALONE_VERSION = 9.22
> +LZMA_ALONE_SITE = https://sourceforge.net/projects/sevenzip/files/LZMA%20SDK
> +LZMA_ALONE_SOURCE = lzma922.tar.bz2
> +LZMA_ALONE_STRIP_COMPONENTS = 0
> +LZMA_ALONE_LICENSE = LGPL-2.1-or-later, BSD-3-Clause

The license stated in lzma.txt (at the root of the source tree) states
the "LZMA SDK is written and placed in the public domain by Igor
Pavlov."

The 'GPL' term only appears in that file, and BSD appears nowhere. Also,
the usual words in the BSD or GPL licenses do not appear in the source
at all: 'redistribution', 'provided', 'disclaimer'.

WikiPedia seems to agree that it is Public Domain since 2008 and version
4.62:
    https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm#7-Zip_reference_implementation

Fixed.

> +define HOST_LZMA_ALONE_BUILD_CMDS
> +	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/C/Util/Lzma -f makefile.gcc
> +	$(HOST_MAKE_ENV) $(HOST_CONFIGURE_OPTS) $(MAKE) -C $(@D)/CPP/7zip/Bundles/LzmaCon -f makefile.gcc

Lines too long, I've split them.

> +endef
> +
> +define HOST_LZMA_ALONE_INSTALL_CMDS
> +	$(INSTALL) -D -m 0755 $(@D)/CPP/7zip/Bundles/LzmaCon/lzma $(HOST_DIR)/bin/lzma_alone

Ditto.

Applied to master, thanks.

Could you see at using a more recent version, with the p7zip host
variant I suggested above, please?

Regards,
Yann E. MORIN.

> +endef
> +
> +$(eval $(host-generic-package))
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2] package/lzma-alone: new package

[Rafał Miłecki]

On 9.03.2022 23:08, Yann E. MORIN wrote:
> On 2022-03-08 19:35 +0100, Rafał Miłecki spake thusly:
>> From: Rafał Miłecki <rafal@milecki.pl>
>>
>> This new package provides "lzma_alone" host binary based on the original
>> LZMA SDK. It provides few extra options when compared to the LZMA Utils
>> / XZ Utils project "lzma" binary (already packaged as the "lzma").
>>
>> This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows
>> Debian's solution. Please note that Debian also uses LZMA SDK for the
>> base "lzma" tool which may be considered for Buildroot too - as an
>> independent change. Similar packaging is also used by Ubuntu & Arch.
>>
>> lzma_alone is a requirement for preparing firmware images for some
>> Broadcom based home routers with a picky CFE bootloader. It has limited
>> LZMA support and building compatible images requires specifying
>> dictionary size and lc/lp/pb LZMA values manually.
>>
>> Version 9.22 is used as it's the last release using .tar.bz2 format.
> 
> We could introduce a host variant of p7zip, so that we could
> automatically add that to FOO_EXTRACT_DEPENDENCIES if the host does not
> already have 7za (or 7zr), like we do for lzip archives for example.

I can see 7z format support is no less messy than LZMA tools.

It seems that 7-Zip project didn't provide command-line tool initially
and that resulted in developing p7zip. It seems it has changed though.

So we have two projects:

1. p7zip
* Provides "7z" and "7za"
* Last updated in 2016
* Suffers from security issues
* Requires patching (11 Debian patches, 2 for CVEs)

2. 7-Zip
* Provides "7zr" (documented as "7za")
* Last updated in 2021
* Messy source
* Doesn't even compile on some systems:
../../../C/Util/7z/7zMain.c:363:5: error: initializer element is not constant
      (const UInt64)60 * 60 * 24 * (89 + 365 * (kUnixTimeStartYear - kFileTimeStartYear));
      ^

Which one of above we should try to package?


> Applied to master, thanks.

Huge thanks for your help.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH V2] package/lzma-alone: new package

[Arnout Vandecappelle]

On 10/03/2022 15:12, Rafał Miłecki wrote:
> On 9.03.2022 23:08, Yann E. MORIN wrote:
>> On 2022-03-08 19:35 +0100, Rafał Miłecki spake thusly:
>>> From: Rafał Miłecki <rafal@milecki.pl>
>>>
>>> This new package provides "lzma_alone" host binary based on the original
>>> LZMA SDK. It provides few extra options when compared to the LZMA Utils
>>> / XZ Utils project "lzma" binary (already packaged as the "lzma").
>>>
>>> This packaging schema (LZMA SDK with lzma -> lzma_alone rename) follows
>>> Debian's solution. Please note that Debian also uses LZMA SDK for the
>>> base "lzma" tool which may be considered for Buildroot too - as an
>>> independent change. Similar packaging is also used by Ubuntu & Arch.
>>>
>>> lzma_alone is a requirement for preparing firmware images for some
>>> Broadcom based home routers with a picky CFE bootloader. It has limited
>>> LZMA support and building compatible images requires specifying
>>> dictionary size and lc/lp/pb LZMA values manually.
>>>
>>> Version 9.22 is used as it's the last release using .tar.bz2 format.
>>
>> We could introduce a host variant of p7zip, so that we could
>> automatically add that to FOO_EXTRACT_DEPENDENCIES if the host does not
>> already have 7za (or 7zr), like we do for lzip archives for example.
> 
> I can see 7z format support is no less messy than LZMA tools.
> 
> It seems that 7-Zip project didn't provide command-line tool initially
> and that resulted in developing p7zip. It seems it has changed though.
> 
> So we have two projects:
> 
> 1. p7zip
> * Provides "7z" and "7za"
> * Last updated in 2016
> * Suffers from security issues
> * Requires patching (11 Debian patches, 2 for CVEs)

  Last commit in the fork that we use in package/p7zip [1] is from December '21, 
so not that bad I think.

  Regards,
  Arnout

[1] https://github.com/jinfeihan57/p7zip


> 
> 2. 7-Zip
> * Provides "7zr" (documented as "7za")
> * Last updated in 2021
> * Messy source
> * Doesn't even compile on some systems:
> ../../../C/Util/7z/7zMain.c:363:5: error: initializer element is not constant
>       (const UInt64)60 * 60 * 24 * (89 + 365 * (kUnixTimeStartYear - 
> kFileTimeStartYear));
>       ^
> 
> Which one of above we should try to package?
> 
> 
>> Applied to master, thanks.
> 
> Huge thanks for your help.
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot