[PATCH] pixz: Add 1.0.6

[PATCH] pixz: Add 1.0.6

[Richard Purdie]

xz gives better compression results than bzip/gz but is often slower.
Using parallel compression mitigates this somewhat and is particularly
useful for the SDK.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb b/meta/recipes-support/pixz/pixz_1.0.6.bb
new file mode 100644
index 0000000..e6e4ac2
--- /dev/null
+++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
@@ -0,0 +1,14 @@
+SUMMARY = "Parallel, indexed xz compressor"
+
+DEPENDS = "xz libarchive"
+
+SRC_URI = "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.xz"
+SRC_URI[md5sum] = "f6dc5909c9a31b192f69aa397ae8df48"
+SRC_URI[sha256sum] = "02c50746b134fa1b1aae41fcc314d7c6f1919b3d48bcdea01bf11769f83f72e8"
+
+LICENSE = "BSD-2-Clause"
+LIC_FILES_CHKSUM = "file://LICENSE;md5=5cf6d164086105f1512ccb81bfff1926"
+
+inherit autotools
+
+BBCLASSEXTEND = "native"

Re: [PATCH] pixz: Add 1.0.6

[Khem Raj]

On Fri, Jan 8, 2016 at 10:22 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> xz gives better compression results than bzip/gz but is often slower.
> Using parallel compression mitigates this somewhat and is particularly
> useful for the SDK.

I have proposed another tool called pxz back in Sept last year
http://lists.openembedded.org/pipermail/openembedded-core/2015-September/110971.html

although pixz seems to be well maintained and active, would be nice to
know what do people mostly use.

>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb b/meta/recipes-support/pixz/pixz_1.0.6.bb
> new file mode 100644
> index 0000000..e6e4ac2
> --- /dev/null
> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
> @@ -0,0 +1,14 @@
> +SUMMARY = "Parallel, indexed xz compressor"
> +
> +DEPENDS = "xz libarchive"
> +
> +SRC_URI = "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.xz"
> +SRC_URI[md5sum] = "f6dc5909c9a31b192f69aa397ae8df48"
> +SRC_URI[sha256sum] = "02c50746b134fa1b1aae41fcc314d7c6f1919b3d48bcdea01bf11769f83f72e8"
> +
> +LICENSE = "BSD-2-Clause"
> +LIC_FILES_CHKSUM = "file://LICENSE;md5=5cf6d164086105f1512ccb81bfff1926"
> +
> +inherit autotools
> +
> +BBCLASSEXTEND = "native"
>
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] pixz: Add 1.0.6

[Paul Eggleton]

On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
> xz gives better compression results than bzip/gz but is often slower.
> Using parallel compression mitigates this somewhat and is particularly
> useful for the SDK.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
> b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
> index 0000000..e6e4ac2
> --- /dev/null
> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
> @@ -0,0 +1,14 @@
> +SUMMARY = "Parallel, indexed xz compressor"
> +
> +DEPENDS = "xz libarchive"
> +
> +SRC_URI =
> "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.xz"

Can we rely on this never changing? I thought we'd experienced problems with 
github's release tarballs being generated on the fly in the past...

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] pixz: Add 1.0.6

[Paul Eggleton]

On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
> On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
> > xz gives better compression results than bzip/gz but is often slower.
> > Using parallel compression mitigates this somewhat and is particularly
> > useful for the SDK.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > 
> > diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
> > b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
> > index 0000000..e6e4ac2
> > --- /dev/null
> > +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
> > @@ -0,0 +1,14 @@
> > +SUMMARY = "Parallel, indexed xz compressor"
> > +
> > +DEPENDS = "xz libarchive"
> > +
> > +SRC_URI =
> > "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.xz
> > "
> 
> Can we rely on this never changing? I thought we'd experienced problems with
> github's release tarballs being generated on the fly in the past...

Another thing, this seems to fail to build without asciidoc:

------------ snip ------------
checking for src/pixz.1... no
checking for a2x... no
configure: error: AsciiDoc not found, not able to generate the man page.
------------ snip ------------

This is also related to not supporting B != S, since src/pixz.1 does exist, 
just in S and not B. If you inherit autotools-brokensep instead of autotools 
it works.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] pixz: Add 1.0.6

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 731 bytes --]

On 10 January 2016 at 20:26, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> > "
> https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.xz"
>
> Can we rely on this never changing? I thought we'd experienced problems
> with
> github's release tarballs being generated on the fly in the past...


I've spoken to github engineers who swear that because github's archives
are generated using git archive they should be identical (it forces
timestamps to match the data in git).  Either way, on the releases page the
links to "Source code" are github's dynamically generated tarballs, the
other files (with sizes) were created by the maintainer and uploaded so
they won't be changing.

Ross

[-- Attachment #2: Type: text/html, Size: 1272 bytes --]

Re: [PATCH] pixz: Add 1.0.6

[Andre McCurdy]

On Sat, Jan 9, 2016 at 9:14 AM, Khem Raj <raj.khem@gmail.com> wrote:
> On Fri, Jan 8, 2016 at 10:22 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> xz gives better compression results than bzip/gz but is often slower.
>> Using parallel compression mitigates this somewhat and is particularly
>> useful for the SDK.
>
> I have proposed another tool called pxz back in Sept last year
> http://lists.openembedded.org/pipermail/openembedded-core/2015-September/110971.html
>
> although pixz seems to be well maintained and active, would be nice to
> know what do people mostly use.

The pixz README contains a summary of some of the alternatives - and
does mention high memory usage as a potential issue for pxz.

  https://github.com/vasi/pixz#comparison-to-other-tools

>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>
>> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb b/meta/recipes-support/pixz/pixz_1.0.6.bb
>> new file mode 100644
>> index 0000000..e6e4ac2
>> --- /dev/null
>> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
>> @@ -0,0 +1,14 @@
>> +SUMMARY = "Parallel, indexed xz compressor"
>> +
>> +DEPENDS = "xz libarchive"
>> +
>> +SRC_URI = "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.xz"
>> +SRC_URI[md5sum] = "f6dc5909c9a31b192f69aa397ae8df48"
>> +SRC_URI[sha256sum] = "02c50746b134fa1b1aae41fcc314d7c6f1919b3d48bcdea01bf11769f83f72e8"
>> +
>> +LICENSE = "BSD-2-Clause"
>> +LIC_FILES_CHKSUM = "file://LICENSE;md5=5cf6d164086105f1512ccb81bfff1926"
>> +
>> +inherit autotools
>> +
>> +BBCLASSEXTEND = "native"
>>
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] pixz: Add 1.0.6

[Randy Witt]

On 01/08/2016 10:22 AM, Richard Purdie wrote:
> xz gives better compression results than bzip/gz but is often slower.
> Using parallel compression mitigates this somewhat and is particularly
> useful for the SDK.

xz as of 5.2.0 allows for multi-threaded compression.

http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD

Re: [PATCH] pixz: Add 1.0.6

[Paul Eggleton]

On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
> On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
> > On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
> > > xz gives better compression results than bzip/gz but is often slower.
> > > Using parallel compression mitigates this somewhat and is particularly
> > > useful for the SDK.
> > > 
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > 
> > > diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
> > > b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
> > > index 0000000..e6e4ac2
> > > --- /dev/null
> > > +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
> > > @@ -0,0 +1,14 @@
> > > +SUMMARY = "Parallel, indexed xz compressor"
> > > +
> > > +DEPENDS = "xz libarchive"
> > > +
> > > +SRC_URI =
> > > "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.
> > > xz
> > > "
> > 
> > Can we rely on this never changing? I thought we'd experienced problems
> > with github's release tarballs being generated on the fly in the past...
> Another thing, this seems to fail to build without asciidoc:
> 
> ------------ snip ------------
> checking for src/pixz.1... no
> checking for a2x... no
> configure: error: AsciiDoc not found, not able to generate the man page.
> ------------ snip ------------
> 
> This is also related to not supporting B != S, since src/pixz.1 does exist,
> just in S and not B. If you inherit autotools-brokensep instead of autotools
> it works.

Possibly a bit obvious, but even inheriting autotools-brokensep isn't enough, 
because if it runs "make clean" on re-executing do_configure, src/pixz.1 gets 
deleted and you get the same issue.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] pixz: Add 1.0.6

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2168 bytes --]


> On Jan 11, 2016, at 1:07 PM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> 
> On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
>> On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
>>> On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
>>>> xz gives better compression results than bzip/gz but is often slower.
>>>> Using parallel compression mitigates this somewhat and is particularly
>>>> useful for the SDK.
>>>> 
>>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>>> 
>>>> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>> b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
>>>> index 0000000..e6e4ac2
>>>> --- /dev/null
>>>> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>> @@ -0,0 +1,14 @@
>>>> +SUMMARY = "Parallel, indexed xz compressor"
>>>> +
>>>> +DEPENDS = "xz libarchive"
>>>> +
>>>> +SRC_URI =
>>>> "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar.
>>>> xz
>>>> "
>>> 
>>> Can we rely on this never changing? I thought we'd experienced problems
>>> with github's release tarballs being generated on the fly in the past...
>> Another thing, this seems to fail to build without asciidoc:
>> 
>> ------------ snip ------------
>> checking for src/pixz.1... no
>> checking for a2x... no
>> configure: error: AsciiDoc not found, not able to generate the man page.
>> ------------ snip ------------
>> 
>> This is also related to not supporting B != S, since src/pixz.1 does exist,
>> just in S and not B. If you inherit autotools-brokensep instead of autotools
>> it works.
> 
> Possibly a bit obvious, but even inheriting autotools-brokensep isn't enough,
> because if it runs "make clean" on re-executing do_configure, src/pixz.1 gets
> deleted and you get the same issue.

Adding --without-manpage might get you past this issue.

> 
> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] pixz: Add 1.0.6

[Paul Eggleton]

On Mon, 11 Jan 2016 13:17:14 Khem Raj wrote:
> > On Jan 11, 2016, at 1:07 PM, Paul Eggleton <paul.eggleton@linux.intel.com>
> > wrote:> 
> > On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
> >> On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
> >>> On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
> >>>> xz gives better compression results than bzip/gz but is often slower.
> >>>> Using parallel compression mitigates this somewhat and is particularly
> >>>> useful for the SDK.
> >>>> 
> >>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >>>> 
> >>>> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
> >>>> b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
> >>>> index 0000000..e6e4ac2
> >>>> --- /dev/null
> >>>> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
> >>>> @@ -0,0 +1,14 @@
> >>>> +SUMMARY = "Parallel, indexed xz compressor"
> >>>> +
> >>>> +DEPENDS = "xz libarchive"
> >>>> +
> >>>> +SRC_URI =
> >>>> "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar
> >>>> .
> >>>> xz
> >>>> "
> >>> 
> >>> Can we rely on this never changing? I thought we'd experienced problems
> >>> with github's release tarballs being generated on the fly in the past...
> >> 
> >> Another thing, this seems to fail to build without asciidoc:
> >> 
> >> ------------ snip ------------
> >> checking for src/pixz.1... no
> >> checking for a2x... no
> >> configure: error: AsciiDoc not found, not able to generate the man page.
> >> ------------ snip ------------
> >> 
> >> This is also related to not supporting B != S, since src/pixz.1 does
> >> exist,
> >> just in S and not B. If you inherit autotools-brokensep instead of
> >> autotools it works.
> > 
> > Possibly a bit obvious, but even inheriting autotools-brokensep isn't
> > enough, because if it runs "make clean" on re-executing do_configure,
> > src/pixz.1 gets deleted and you get the same issue.
> 
> Adding --without-manpage might get you past this issue.

I'm afraid that's not a valid option for this configure script.

I had more shenanigans trying to build the target version. In the end I needed 
to add "ac_cv_file_src_pixz_1=yes" to EXTRA_OECONF and inherit pkgconfig.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH] pixz: Add 1.0.6

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2940 bytes --]


> On Jan 11, 2016, at 2:36 PM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> 
> On Mon, 11 Jan 2016 13:17:14 Khem Raj wrote:
>>> On Jan 11, 2016, at 1:07 PM, Paul Eggleton <paul.eggleton@linux.intel.com>
>>> wrote:>
>>> On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
>>>> On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
>>>>> On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
>>>>>> xz gives better compression results than bzip/gz but is often slower.
>>>>>> Using parallel compression mitigates this somewhat and is particularly
>>>>>> useful for the SDK.
>>>>>> 
>>>>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>>>>> 
>>>>>> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>>>> b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
>>>>>> index 0000000..e6e4ac2
>>>>>> --- /dev/null
>>>>>> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>>>> @@ -0,0 +1,14 @@
>>>>>> +SUMMARY = "Parallel, indexed xz compressor"
>>>>>> +
>>>>>> +DEPENDS = "xz libarchive"
>>>>>> +
>>>>>> +SRC_URI =
>>>>>> "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar
>>>>>> .
>>>>>> xz
>>>>>> "
>>>>> 
>>>>> Can we rely on this never changing? I thought we'd experienced problems
>>>>> with github's release tarballs being generated on the fly in the past...
>>>> 
>>>> Another thing, this seems to fail to build without asciidoc:
>>>> 
>>>> ------------ snip ------------
>>>> checking for src/pixz.1... no
>>>> checking for a2x... no
>>>> configure: error: AsciiDoc not found, not able to generate the man page.
>>>> ------------ snip ------------
>>>> 
>>>> This is also related to not supporting B != S, since src/pixz.1 does
>>>> exist,
>>>> just in S and not B. If you inherit autotools-brokensep instead of
>>>> autotools it works.
>>> 
>>> Possibly a bit obvious, but even inheriting autotools-brokensep isn't
>>> enough, because if it runs "make clean" on re-executing do_configure,
>>> src/pixz.1 gets deleted and you get the same issue.
>> 
>> Adding --without-manpage might get you past this issue.
> 
> I'm afraid that's not a valid option for this configure script.

I was checking the source code on github and now I tried it locally.
the option does exist

git clone https://github.com/vasi/pixz.git
cd pixz
/configure

and it complains

configure: error: AsciiDoc not found, not able to generate the man page.


./configure --without-manpage

succeeds.


> 
> I had more shenanigans trying to build the target version. In the end I needed
> to add "ac_cv_file_src_pixz_1=yes" to EXTRA_OECONF and inherit pkgconfig.
> 

it should be set to ‘no’ if manpages were to be disabled. so I wonder setting it to yes
is just bypassing the configure time check but should fail at compile/install time.

> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] pixz: Add 1.0.6

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2754 bytes --]


> On Jan 11, 2016, at 2:36 PM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:
> 
> On Mon, 11 Jan 2016 13:17:14 Khem Raj wrote:
>>> On Jan 11, 2016, at 1:07 PM, Paul Eggleton <paul.eggleton@linux.intel.com>
>>> wrote:>
>>> On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
>>>> On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
>>>>> On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
>>>>>> xz gives better compression results than bzip/gz but is often slower.
>>>>>> Using parallel compression mitigates this somewhat and is particularly
>>>>>> useful for the SDK.
>>>>>> 
>>>>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>>>>> 
>>>>>> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>>>> b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode 100644
>>>>>> index 0000000..e6e4ac2
>>>>>> --- /dev/null
>>>>>> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>>>> @@ -0,0 +1,14 @@
>>>>>> +SUMMARY = "Parallel, indexed xz compressor"
>>>>>> +
>>>>>> +DEPENDS = "xz libarchive"
>>>>>> +
>>>>>> +SRC_URI =
>>>>>> "https://github.com/vasi/pixz/releases/download/v${PV}/${BPN}-${PV}.tar
>>>>>> .
>>>>>> xz
>>>>>> "
>>>>> 
>>>>> Can we rely on this never changing? I thought we'd experienced problems
>>>>> with github's release tarballs being generated on the fly in the past...
>>>> 
>>>> Another thing, this seems to fail to build without asciidoc:
>>>> 
>>>> ------------ snip ------------
>>>> checking for src/pixz.1... no
>>>> checking for a2x... no
>>>> configure: error: AsciiDoc not found, not able to generate the man page.
>>>> ------------ snip ------------
>>>> 
>>>> This is also related to not supporting B != S, since src/pixz.1 does
>>>> exist,
>>>> just in S and not B. If you inherit autotools-brokensep instead of
>>>> autotools it works.
>>> 
>>> Possibly a bit obvious, but even inheriting autotools-brokensep isn't
>>> enough, because if it runs "make clean" on re-executing do_configure,
>>> src/pixz.1 gets deleted and you get the same issue.
>> 
>> Adding --without-manpage might get you past this issue.
> 
> I'm afraid that's not a valid option for this configure script.
> 
> I had more shenanigans trying to build the target version. In the end I needed
> to add "ac_cv_file_src_pixz_1=yes" to EXTRA_OECONF and inherit pkgconfig.
> 

I think you are missing
https://github.com/vasi/pixz/commit/936d8068ae19d95260d3058f41dd6cf718101cd6

which is committed after 1.0.6 release. It should be back ported.
may be not use tarball but straight use SRCREV=“936d8068ae19d95260d3058f41dd6cf718101cd6”
with git fetcher might be better.

> Cheers,
> Paul
> 
> --
> 
> Paul Eggleton
> Intel Open Source Technology Centre


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]

Re: [PATCH] pixz: Add 1.0.6

[Richard Purdie]

On Mon, 2016-01-11 at 15:21 -0800, Khem Raj wrote:
> > On Jan 11, 2016, at 2:36 PM, Paul Eggleton <
> > paul.eggleton@linux.intel.com> wrote:
> > 
> > On Mon, 11 Jan 2016 13:17:14 Khem Raj wrote:
> > > > On Jan 11, 2016, at 1:07 PM, Paul Eggleton <
> > > > paul.eggleton@linux.intel.com>
> > > > wrote:>
> > > > On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
> > > > > On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
> > > > > > On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
> > > > > > > xz gives better compression results than bzip/gz but is
> > > > > > > often slower.
> > > > > > > Using parallel compression mitigates this somewhat and is
> > > > > > > particularly
> > > > > > > useful for the SDK.
> > > > > > > 
> > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > 
> > > > > > > diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
> > > > > > > b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode
> > > > > > > 100644
> > > > > > > index 0000000..e6e4ac2
> > > > > > > --- /dev/null
> > > > > > > +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
> > > > > > > @@ -0,0 +1,14 @@
> > > > > > > +SUMMARY = "Parallel, indexed xz compressor"
> > > > > > > +
> > > > > > > +DEPENDS = "xz libarchive"
> > > > > > > +
> > > > > > > +SRC_URI =
> > > > > > > "https://github.com/vasi/pixz/releases/download/v${PV}/${
> > > > > > > BPN}-${PV}.tar
> > > > > > > .
> > > > > > > xz
> > > > > > > "
> > > > > > 
> > > > > > Can we rely on this never changing? I thought we'd
> > > > > > experienced problems
> > > > > > with github's release tarballs being generated on the fly
> > > > > > in the past...
> > > > > 
> > > > > Another thing, this seems to fail to build without asciidoc:
> > > > > 
> > > > > ------------ snip ------------
> > > > > checking for src/pixz.1... no
> > > > > checking for a2x... no
> > > > > configure: error: AsciiDoc not found, not able to generate
> > > > > the man page.
> > > > > ------------ snip ------------
> > > > > 
> > > > > This is also related to not supporting B != S, since
> > > > > src/pixz.1 does
> > > > > exist,
> > > > > just in S and not B. If you inherit autotools-brokensep
> > > > > instead of
> > > > > autotools it works.
> > > > 
> > > > Possibly a bit obvious, but even inheriting autotools-brokensep
> > > > isn't
> > > > enough, because if it runs "make clean" on re-executing
> > > > do_configure,
> > > > src/pixz.1 gets deleted and you get the same issue.
> > > 
> > > Adding --without-manpage might get you past this issue.
> > 
> > I'm afraid that's not a valid option for this configure script.
> > 
> > I had more shenanigans trying to build the target version. In the
> > end I needed
> > to add "ac_cv_file_src_pixz_1=yes" to EXTRA_OECONF and inherit
> > pkgconfig.
> > 
> 
> I think you are missing
> https://github.com/vasi/pixz/commit/936d8068ae19d95260d3058f41dd6cf71
> 8101cd6
> 
> which is committed after 1.0.6 release. It should be back ported.
> may be not use tarball but straight use
> SRCREV=“936d8068ae19d95260d3058f41dd6cf718101cd6”
> with git fetcher might be better.

As Randy mentions, should we use the -T option to xz instead though?

Cheers,

Richard

Re: [PATCH] pixz: Add 1.0.6

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 3714 bytes --]


> On Jan 11, 2016, at 3:35 PM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
> On Mon, 2016-01-11 at 15:21 -0800, Khem Raj wrote:
>>> On Jan 11, 2016, at 2:36 PM, Paul Eggleton <
>>> paul.eggleton@linux.intel.com> wrote:
>>> 
>>> On Mon, 11 Jan 2016 13:17:14 Khem Raj wrote:
>>>>> On Jan 11, 2016, at 1:07 PM, Paul Eggleton <
>>>>> paul.eggleton@linux.intel.com>
>>>>> wrote:>
>>>>> On Mon, 11 Jan 2016 09:36:36 Paul Eggleton wrote:
>>>>>> On Mon, 11 Jan 2016 09:26:39 Paul Eggleton wrote:
>>>>>>> On Fri, 08 Jan 2016 18:22:49 Richard Purdie wrote:
>>>>>>>> xz gives better compression results than bzip/gz but is
>>>>>>>> often slower.
>>>>>>>> Using parallel compression mitigates this somewhat and is
>>>>>>>> particularly
>>>>>>>> useful for the SDK.
>>>>>>>> 
>>>>>>>> Signed-off-by: Richard Purdie <
>>>>>>>> richard.purdie@linuxfoundation.org>
>>>>>>>> 
>>>>>>>> diff --git a/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>>>>>> b/meta/recipes-support/pixz/pixz_1.0.6.bb new file mode
>>>>>>>> 100644
>>>>>>>> index 0000000..e6e4ac2
>>>>>>>> --- /dev/null
>>>>>>>> +++ b/meta/recipes-support/pixz/pixz_1.0.6.bb
>>>>>>>> @@ -0,0 +1,14 @@
>>>>>>>> +SUMMARY = "Parallel, indexed xz compressor"
>>>>>>>> +
>>>>>>>> +DEPENDS = "xz libarchive"
>>>>>>>> +
>>>>>>>> +SRC_URI =
>>>>>>>> "https://github.com/vasi/pixz/releases/download/v${PV}/${
>>>>>>>> BPN}-${PV}.tar
>>>>>>>> .
>>>>>>>> xz
>>>>>>>> "
>>>>>>> 
>>>>>>> Can we rely on this never changing? I thought we'd
>>>>>>> experienced problems
>>>>>>> with github's release tarballs being generated on the fly
>>>>>>> in the past...
>>>>>> 
>>>>>> Another thing, this seems to fail to build without asciidoc:
>>>>>> 
>>>>>> ------------ snip ------------
>>>>>> checking for src/pixz.1... no
>>>>>> checking for a2x... no
>>>>>> configure: error: AsciiDoc not found, not able to generate
>>>>>> the man page.
>>>>>> ------------ snip ------------
>>>>>> 
>>>>>> This is also related to not supporting B != S, since
>>>>>> src/pixz.1 does
>>>>>> exist,
>>>>>> just in S and not B. If you inherit autotools-brokensep
>>>>>> instead of
>>>>>> autotools it works.
>>>>> 
>>>>> Possibly a bit obvious, but even inheriting autotools-brokensep
>>>>> isn't
>>>>> enough, because if it runs "make clean" on re-executing
>>>>> do_configure,
>>>>> src/pixz.1 gets deleted and you get the same issue.
>>>> 
>>>> Adding --without-manpage might get you past this issue.
>>> 
>>> I'm afraid that's not a valid option for this configure script.
>>> 
>>> I had more shenanigans trying to build the target version. In the
>>> end I needed
>>> to add "ac_cv_file_src_pixz_1=yes" to EXTRA_OECONF and inherit
>>> pkgconfig.
>>> 
>> 
>> I think you are missing
>> https://github.com/vasi/pixz/commit/936d8068ae19d95260d3058f41dd6cf71
>> 8101cd6
>> 
>> which is committed after 1.0.6 release. It should be back ported.
>> may be not use tarball but straight use
>> SRCREV=“936d8068ae19d95260d3058f41dd6cf718101cd6”
>> with git fetcher might be better.
> 
> As Randy mentions, should we use the -T option to xz instead though?

that might be ok too

xz /tmp/xx.tar  19.65s user 0.07s system 100% cpu 19.714 total
XZ_DEFAULTS="-T 0" xz /tmp/xx.tar  22.35s user 0.49s system 360% cpu 6.340 total
pixz /tmp/xx.tar /tmp/xx.tar.xz  26.56s user 0.45s system 456% cpu 5.917 total

a little slower than pixz and doesnt seem to use all cores with pixz I had 450% CPU on same load.

using pixz is simpler though. It can also be called integrated with tar. for XZ we need to set this env
variable to enable parallelism with pixz its not needed.

> 
> Cheers,
> 
> Richard


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 211 bytes --]