deb: custom compression

deb: custom compression

[Tim Blechmann]

[-- Attachment #1: Type: text/plain, Size: 367 bytes --]

hi!

poky compresses deb packages with the default flags, which these days
boils down to using xz compression.
trying to cut down turnaround times for local builds, i tried to modify
the `package_deb` class to be able to force a low gzip compression via a
`DPKG_FAST` parameter (patch attached).

could this patch be accepted upstream?

thanks a lot,
tim

[-- Attachment #2: 0001-package_deb-add-DPKG_FAST-flag.patch --]
[-- Type: text/plain, Size: 1808 bytes --]

From c36ead7c4cc9b0cb81b274ddcd288f0d13c90662 Mon Sep 17 00:00:00 2001
From: Tim Blechmann <tim@klingt.org>
Date: Fri, 2 Aug 2019 06:05:47 +0200
Subject: [PATCH] package_deb: add DPKG_FAST flag

by default dpkg-deb compresses debian packages with xz. xz gives great
compression ratios, at the cost of compression speed.

we therefore introduce a DPKG_FAST flag, which compresses with gzip, which
should be an order of magnitude faster than xz at the cost of almost
doubling package sizes.

numbers:
https://www.rootusers.com/gzip-vs-bzip2-vs-xz-performance-comparison/

Signed-off-by: Tim Blechmann <tim@klingt.org>
---
 meta/classes/package_deb.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 6f81591653..28c6b9c19f 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -15,6 +15,10 @@ APTCONF_TARGET = "${WORKDIR}"
 
 APT_ARGS = "${@['', '--no-install-recommends'][d.getVar("NO_RECOMMENDATIONS") == "1"]}"
 
+DPKG_FAST ??= "0"
+
+BB_HASHCONFIG_WHITELIST += "DPKG_FAST"
+
 def debian_arch_map(arch, tune):
     tune_features = tune.split()
     if arch == "allarch":
@@ -269,7 +273,13 @@ def deb_write_pkg(pkg, d):
             conffiles.close()
 
         os.chdir(basedir)
-        subprocess.check_output("PATH=\"%s\" dpkg-deb -b %s %s" % (localdata.getVar("PATH"), root, pkgoutdir),
+
+        if d.getVar("DPKG_FAST") != "0":
+            dpkg_args = "-Zgzip -z1"
+        else:
+            dpkg_args = ""
+
+        subprocess.check_output("PATH=\"%s\" dpkg-deb %s -b %s %s" % (localdata.getVar("PATH"), dpkg_args, root, pkgoutdir),
                                 stderr=subprocess.STDOUT,
                                 shell=True)
 
-- 
2.18.0

Re: deb: custom compression

[Adrian Bunk]

On Wed, Aug 07, 2019 at 11:28:32AM +0800, Tim Blechmann wrote:
> hi!
> 
> poky compresses deb packages with the default flags, which these days
> boils down to using xz compression.
> trying to cut down turnaround times for local builds, i tried to modify
> the `package_deb` class to be able to force a low gzip compression via a
> `DPKG_FAST` parameter (patch attached).
> 
> could this patch be accepted upstream?

It is specific to your problem (and sent to the wrong mailing list).

A more generic solution would be similar to OPKGBUILDCMD,
allowing the user to pass any parameters (including any
compression settings) to dpkg-deb.

This would also support the opposite usecase of "xz -9",
which is both faster to download and faster to decompress
than the default "xz -6".

> thanks a lot,
> tim
>...

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: deb: custom compression

[Tim Blechmann]

>> poky compresses deb packages with the default flags, which these days
>> boils down to using xz compression.
>> trying to cut down turnaround times for local builds, i tried to modify
>> the `package_deb` class to be able to force a low gzip compression via a
>> `DPKG_FAST` parameter (patch attached).
>>
>> could this patch be accepted upstream?
> 
> It is specific to your problem (and sent to the wrong mailing list).

my bad (thought it's this one, as the file resides in the poky repo).
which is the right place for it?

> A more generic solution would be similar to OPKGBUILDCMD,
> allowing the user to pass any parameters (including any
> compression settings) to dpkg-deb.

something like `DPKG_OPTIONS` i'm happy to adapt the patch?

thanks,
tim

Re: deb: custom compression

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1096 bytes --]

openembedded-core list is the right one.

Alex

On Wed, 7 Aug 2019 at 08:36, Tim Blechmann <tim@klingt.org> wrote:

> >> poky compresses deb packages with the default flags, which these days
> >> boils down to using xz compression.
> >> trying to cut down turnaround times for local builds, i tried to modify
> >> the `package_deb` class to be able to force a low gzip compression via a
> >> `DPKG_FAST` parameter (patch attached).
> >>
> >> could this patch be accepted upstream?
> >
> > It is specific to your problem (and sent to the wrong mailing list).
>
> my bad (thought it's this one, as the file resides in the poky repo).
> which is the right place for it?
>
> > A more generic solution would be similar to OPKGBUILDCMD,
> > allowing the user to pass any parameters (including any
> > compression settings) to dpkg-deb.
>
> something like `DPKG_OPTIONS` i'm happy to adapt the patch?
>
> thanks,
> tim
>
> --
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>

[-- Attachment #2: Type: text/html, Size: 1677 bytes --]

Re: deb: custom compression

[Adrian Bunk]

On Wed, Aug 07, 2019 at 02:35:37PM +0800, Tim Blechmann wrote:
> >> poky compresses deb packages with the default flags, which these days
> >> boils down to using xz compression.
> >> trying to cut down turnaround times for local builds, i tried to modify
> >> the `package_deb` class to be able to force a low gzip compression via a
> >> `DPKG_FAST` parameter (patch attached).
> >>
> >> could this patch be accepted upstream?
> > 
> > It is specific to your problem (and sent to the wrong mailing list).
> 
> my bad (thought it's this one, as the file resides in the poky repo).
> which is the right place for it?

Not really your bad since it is complicated,
README.poky contains the details.

> > A more generic solution would be similar to OPKGBUILDCMD,
> > allowing the user to pass any parameters (including any
> > compression settings) to dpkg-deb.
> 
> something like `DPKG_OPTIONS` i'm happy to adapt the patch?

opkg (upstream defaults to gzip) has
  OPKGBUILDCMD ??= 'opkg-build -Z xz -a "${XZ_DEFAULTS}"'

For dpkg-deb I would therefore suggest
  DPKGDEBCMD ??= 'dpkg-deb'

> thanks,
> tim

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed