[PATCH 0/1] Add support for 7-Zip compressed files

[PATCH 0/1] Add support for 7-Zip compressed files

[Juro Bystricky]

7-Zip offers substantially better compression than other compression
formats. In fact, 7-Zip explicitly claims it provides the best compression ratio.
To quickly test the claim, I compressed a gcc 5.2 538.8 MB tree snapshot in various ways:

tar.bz2: 114.0 MB
tar.gz:  112.4 MB
tar.7z:   70.3 MB

At present, there is no support for 7-Zip compressed files in Bitbake.
Various public Git repositories only allow pushing files of certain max size,
AFAIK Github only allows 100MB, so there are cases where it is desirable
to use maximum compression possible.


Juro Bystricky (1):
  bitbake:fetch2/__init__.py: Add support for 7-Zip

 bitbake/lib/bb/fetch2/__init__.py | 4 ++++
 1 file changed, 4 insertions(+)

-- 
1.9.1

[PATCH 1/1] bitbake:fetch2/__init__.py: Add support for 7-Zip

[Juro Bystricky]

7-Zip is a file archiver claiming the highest compression ratio.
This patch allows using 7-Zip commpressed files in bitbake recipes.
Two common formats are supported:

SRC_URI = "file://abc.tar.7z"
SRC_URI = "file://abc.7z"

Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
 bitbake/lib/bb/fetch2/__init__.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/bitbake/lib/bb/fetch2/__init__.py b/bitbake/lib/bb/fetch2/__init__.py
index 68ebf8b..0f9c643 100644
--- a/bitbake/lib/bb/fetch2/__init__.py
+++ b/bitbake/lib/bb/fetch2/__init__.py
@@ -1407,6 +1407,10 @@ class FetchMethod(object):
                     cmd = 'rpm2cpio.sh %s | cpio -id' % (file)
             elif file.endswith('.deb') or file.endswith('.ipk'):
                 cmd = 'ar -p %s data.tar.gz | zcat | tar --no-same-owner -xpf -' % file
+            elif file.endswith('.tar.7z'):
+                cmd = '7z x -so %s | tar xf - ' % file
+            elif file.endswith('.7z'):
+                cmd = '7za x -y %s 1>/dev/null' % file
 
         if not unpack or not cmd:
             # If file == dest, then avoid any copies, as we already put the file into dest!
-- 
1.9.1

Re: [PATCH 0/1] Add support for 7-Zip compressed files

[Mark Hatle]

On 1/4/16 12:51 PM, Juro Bystricky wrote:
> 7-Zip offers substantially better compression than other compression
> formats. In fact, 7-Zip explicitly claims it provides the best compression ratio.
> To quickly test the claim, I compressed a gcc 5.2 538.8 MB tree snapshot in various ways:
> 
> tar.bz2: 114.0 MB
> tar.gz:  112.4 MB
> tar.7z:   70.3 MB

I thought 7z was an archive format, that included compression information.

Since the compression that makes 7z better then bz2 is lzma, and 'xz' uses lzma
as it's compression format.  (In my experience, .xz is really common on Linux --
7z is only common on Windows.)

I'm not saying don't support .7z, but adding support for .tar.xz and others
would make equal sense to me.  (There are a number of formats out there that
could and likely should be added.)

--Mark

> At present, there is no support for 7-Zip compressed files in Bitbake.
> Various public Git repositories only allow pushing files of certain max size,
> AFAIK Github only allows 100MB, so there are cases where it is desirable
> to use maximum compression possible.
> 
> 
> Juro Bystricky (1):
>   bitbake:fetch2/__init__.py: Add support for 7-Zip
> 
>  bitbake/lib/bb/fetch2/__init__.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 

Re: [PATCH 0/1] Add support for 7-Zip compressed files

[Bystricky, Juro]

I agree.
I compressed the same tree with tar.xz, the compressed size is:

tar.xz: 88.9 MB

which is quite respectable and markedly better than tar.bz2.
To add bitbake support for this compression format should be quite straightforward.

Juro


> -----Original Message-----
> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> Sent: Monday, January 4, 2016 12:36 PM
> To: Bystricky, Juro; bitbake-devel@lists.openembedded.org
> Cc: Purdie, Richard
> Subject: Re: [bitbake-devel] [PATCH 0/1] Add support for 7-Zip compressed
> files
> 
> On 1/4/16 12:51 PM, Juro Bystricky wrote:
> > 7-Zip offers substantially better compression than other compression
> > formats. In fact, 7-Zip explicitly claims it provides the best compression
> ratio.
> > To quickly test the claim, I compressed a gcc 5.2 538.8 MB tree snapshot in
> various ways:
> >
> > tar.bz2: 114.0 MB
> > tar.gz:  112.4 MB
> > tar.7z:   70.3 MB
> 
> I thought 7z was an archive format, that included compression information.
> 
> Since the compression that makes 7z better then bz2 is lzma, and 'xz' uses
> lzma as it's compression format.  (In my experience, .xz is really common on
> Linux -- 7z is only common on Windows.)
> 
> I'm not saying don't support .7z, but adding support for .tar.xz and others
> would make equal sense to me.  (There are a number of formats out there
> that could and likely should be added.)
> 
> --Mark
> 
> > At present, there is no support for 7-Zip compressed files in Bitbake.
> > Various public Git repositories only allow pushing files of certain
> > max size, AFAIK Github only allows 100MB, so there are cases where it
> > is desirable to use maximum compression possible.
> >
> >
> > Juro Bystricky (1):
> >   bitbake:fetch2/__init__.py: Add support for 7-Zip
> >
> >  bitbake/lib/bb/fetch2/__init__.py | 4 ++++
> >  1 file changed, 4 insertions(+)
> >

Re: [PATCH 0/1] Add support for 7-Zip compressed files

[Paul Eggleton]

On Mon, 04 Jan 2016 14:35:41 Mark Hatle wrote:
> On 1/4/16 12:51 PM, Juro Bystricky wrote:
> > 7-Zip offers substantially better compression than other compression
> > formats. In fact, 7-Zip explicitly claims it provides the best compression
> > ratio. To quickly test the claim, I compressed a gcc 5.2 538.8 MB tree
> > snapshot in various ways:
> > 
> > tar.bz2: 114.0 MB
> > tar.gz:  112.4 MB
> > tar.7z:   70.3 MB
> 
> I thought 7z was an archive format, that included compression information.
> 
> Since the compression that makes 7z better then bz2 is lzma, and 'xz' uses
> lzma as it's compression format.  (In my experience, .xz is really common
> on Linux -- 7z is only common on Windows.)
> 
> I'm not saying don't support .7z, but adding support for .tar.xz and others
> would make equal sense to me.  (There are a number of formats out there that
> could and likely should be added.)

We already support tar.xz FYI (and use it within the metadata quite a lot).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre