[PATCH 1/2] yocto-uninative: Update to 2.5 release

[PATCH 1/2] yocto-uninative: Update to 2.5 release

[Richard Purdie]

This includes libstdc++ changes from gcc 9.X.

It also switches uninative from bz2 to xz compression.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/distro/include/yocto-uninative.inc | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/conf/distro/include/yocto-uninative.inc b/meta/conf/distro/include/yocto-uninative.inc
index 59ccd6931ba..0bb8f7af176 100644
--- a/meta/conf/distro/include/yocto-uninative.inc
+++ b/meta/conf/distro/include/yocto-uninative.inc
@@ -8,7 +8,7 @@
 
 UNINATIVE_MAXGLIBCVERSION = "2.29"
 
-UNINATIVE_URL ?= "http://downloads.yoctoproject.org/releases/uninative/2.4/"
-UNINATIVE_CHECKSUM[aarch64] ?= "af2e2faf6cf00ff45cc1bcd5e3fb00cee7f79b3ec7c3be15917ad4ff8c154cfe"
-UNINATIVE_CHECKSUM[i686] ?= "fafacfc537a6ce2bd122bd16c146881ab5ac69bd575abf6cb68a0dd33fa70ea2"
-UNINATIVE_CHECKSUM[x86_64] ?= "06f91685b782f2ccfedf3070b3ba0fe4a5ba2f0766dad5c9d1642dccf95accd0"
+UNINATIVE_URL ?= "http://downloads.yoctoproject.org/releases/uninative/2.5/"
+UNINATIVE_CHECKSUM[aarch64] ?= "ca977ff95c77f983570141908d451ff7d78add2864471605af404302bb36a1fa"
+UNINATIVE_CHECKSUM[i686] ?= "7b5822891c293795faf8a4a80586b36f8cde405387524916a24f9055ea82f7ca"
+UNINATIVE_CHECKSUM[x86_64] ?= "ed0ac07c710b711925cb976685dd855fb1d442dd840d00194751c18bf480c4ed"
-- 
2.20.1

[PATCH 2/2] uninative: Switch from bz2 to xz

[Richard Purdie]

(From OE-Core rev: 29fc9210b973be68de474e75068e4c72371afe5a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/uninative.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index ba99fb6e8fd..3326c0db3da 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -2,7 +2,7 @@ UNINATIVE_LOADER ?= "${UNINATIVE_STAGING_DIR}-uninative/${BUILD_ARCH}-linux/lib/
 UNINATIVE_STAGING_DIR ?= "${STAGING_DIR}"
 
 UNINATIVE_URL ?= "unset"
-UNINATIVE_TARBALL ?= "${BUILD_ARCH}-nativesdk-libc.tar.bz2"
+UNINATIVE_TARBALL ?= "${BUILD_ARCH}-nativesdk-libc.tar.xz"
 # Example checksums
 #UNINATIVE_CHECKSUM[aarch64] = "dead"
 #UNINATIVE_CHECKSUM[i686] = "dead"
@@ -89,7 +89,7 @@ python uninative_event_fetchloader() {
         cmd = d.expand("\
 mkdir -p ${UNINATIVE_STAGING_DIR}-uninative; \
 cd ${UNINATIVE_STAGING_DIR}-uninative; \
-tar -xjf ${UNINATIVE_DLDIR}/%s/${UNINATIVE_TARBALL}; \
+tar -xJf ${UNINATIVE_DLDIR}/%s/${UNINATIVE_TARBALL}; \
 ${UNINATIVE_STAGING_DIR}-uninative/relocate_sdk.py \
   ${UNINATIVE_STAGING_DIR}-uninative/${BUILD_ARCH}-linux \
   ${UNINATIVE_LOADER} \
-- 
2.20.1

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[akuster808]

On 5/29/19 11:43 AM, Richard Purdie wrote:
> (From OE-Core rev: 29fc9210b973be68de474e75068e4c72371afe5a)
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

With back port to warrior and Thud too?

- armin
> ---
>  meta/classes/uninative.bbclass | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
> index ba99fb6e8fd..3326c0db3da 100644
> --- a/meta/classes/uninative.bbclass
> +++ b/meta/classes/uninative.bbclass
> @@ -2,7 +2,7 @@ UNINATIVE_LOADER ?= "${UNINATIVE_STAGING_DIR}-uninative/${BUILD_ARCH}-linux/lib/
>  UNINATIVE_STAGING_DIR ?= "${STAGING_DIR}"
>  
>  UNINATIVE_URL ?= "unset"
> -UNINATIVE_TARBALL ?= "${BUILD_ARCH}-nativesdk-libc.tar.bz2"
> +UNINATIVE_TARBALL ?= "${BUILD_ARCH}-nativesdk-libc.tar.xz"
>  # Example checksums
>  #UNINATIVE_CHECKSUM[aarch64] = "dead"
>  #UNINATIVE_CHECKSUM[i686] = "dead"
> @@ -89,7 +89,7 @@ python uninative_event_fetchloader() {
>          cmd = d.expand("\
>  mkdir -p ${UNINATIVE_STAGING_DIR}-uninative; \
>  cd ${UNINATIVE_STAGING_DIR}-uninative; \
> -tar -xjf ${UNINATIVE_DLDIR}/%s/${UNINATIVE_TARBALL}; \
> +tar -xJf ${UNINATIVE_DLDIR}/%s/${UNINATIVE_TARBALL}; \
>  ${UNINATIVE_STAGING_DIR}-uninative/relocate_sdk.py \
>    ${UNINATIVE_STAGING_DIR}-uninative/${BUILD_ARCH}-linux \
>    ${UNINATIVE_LOADER} \

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> 
> On 5/29/19 11:43 AM, Richard Purdie wrote:
> > (From OE-Core rev: 29fc9210b973be68de474e75068e4c72371afe5a)
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> With back port to warrior and Thud too?

Yes, we'll need to do that if we bring fedora 30 workers online on the
autobuilder.

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[akuster808]

On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
>> On 5/29/19 11:43 AM, Richard Purdie wrote:
>>> (From OE-Core rev: 29fc9210b973be68de474e75068e4c72371afe5a)
>>>
>>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>> With back port to warrior and Thud too?
> Yes, we'll need to do that if we bring fedora 30 workers online on the
> autobuilder.
I know the uninative update is required for FC30 but changing the
compression does not.

This question is about the compression extension question.

Does this mean YP will no longer be providing bz2 uninatives?


- armin
> Cheers,
>
> Richard
>

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> 
> On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > (From OE-Core rev: 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > 
> > > > Signed-off-by: Richard Purdie <
> > > > richard.purdie@linuxfoundation.org>
> > > With back port to warrior and Thud too?
> > Yes, we'll need to do that if we bring fedora 30 workers online on
> > the
> > autobuilder.
> I know the uninative update is required for FC30 but changing the
> compression does not.
> 
> This question is about the compression extension question.
> 
> Does this mean YP will no longer be providing bz2 uninatives?

Correct, the patch should be straight forward to include with the
upgrade.

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

On Wed, May 29, 2019 at 09:50:22PM +0100, richard.purdie@linuxfoundation.org wrote:
> On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > 
> > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > (From OE-Core rev: 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > 
> > > > > Signed-off-by: Richard Purdie <
> > > > > richard.purdie@linuxfoundation.org>
> > > > With back port to warrior and Thud too?
> > > Yes, we'll need to do that if we bring fedora 30 workers online on
> > > the
> > > autobuilder.
> > I know the uninative update is required for FC30 but changing the
> > compression does not.
> > 
> > This question is about the compression extension question.
> > 
> > Does this mean YP will no longer be providing bz2 uninatives?
> 
> Correct, the patch should be straight forward to include with the
> upgrade.

Does that in turn mean we must have host xz-utils?

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> On Wed, May 29, 2019 at 09:50:22PM +0100, 
> richard.purdie@linuxfoundation.org wrote:
> > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > (From OE-Core rev:
> > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > 
> > > > > > Signed-off-by: Richard Purdie <
> > > > > > richard.purdie@linuxfoundation.org>
> > > > > With back port to warrior and Thud too?
> > > > Yes, we'll need to do that if we bring fedora 30 workers online
> > > > on
> > > > the
> > > > autobuilder.
> > > I know the uninative update is required for FC30 but changing the
> > > compression does not.
> > > 
> > > This question is about the compression extension question.
> > > 
> > > Does this mean YP will no longer be providing bz2 uninatives?
> > 
> > Correct, the patch should be straight forward to include with the
> > upgrade.
> 
> Does that in turn mean we must have host xz-utils?

It will mean you need a tar which supports the -J option. The tests
I've see so far suggest all our oldest supported distros do...

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Adrian Bunk]

On Wed, May 29, 2019 at 10:29:31PM +0100, richard.purdie@linuxfoundation.org wrote:
> On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > On Wed, May 29, 2019 at 09:50:22PM +0100, 
> > richard.purdie@linuxfoundation.org wrote:
> > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > (From OE-Core rev:
> > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > 
> > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > With back port to warrior and Thud too?
> > > > > Yes, we'll need to do that if we bring fedora 30 workers online
> > > > > on
> > > > > the
> > > > > autobuilder.
> > > > I know the uninative update is required for FC30 but changing the
> > > > compression does not.
> > > > 
> > > > This question is about the compression extension question.
> > > > 
> > > > Does this mean YP will no longer be providing bz2 uninatives?
> > > 
> > > Correct, the patch should be straight forward to include with the
> > > upgrade.
> > 
> > Does that in turn mean we must have host xz-utils?
> 
> It will mean you need a tar which supports the -J option. The tests
> I've see so far suggest all our oldest supported distros do...

For thud tar >= 1.27 is required, -J was added in 1.22.

But Tom is asking about something different:
tar calls "xz -d" for decompression, which needs xz-utils installed.

https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
documents that xz has to be installed, but looking at 
meta/conf/bitbake.conf it might be missing in HOSTTOOLS.

> Cheers,
> 
> Richard

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 2171 bytes --]

On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
> On Wed, May 29, 2019 at 10:29:31PM +0100, richard.purdie@linuxfoundation.org wrote:
> > On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > > On Wed, May 29, 2019 at 09:50:22PM +0100, 
> > > richard.purdie@linuxfoundation.org wrote:
> > > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > > (From OE-Core rev:
> > > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > > 
> > > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > With back port to warrior and Thud too?
> > > > > > Yes, we'll need to do that if we bring fedora 30 workers online
> > > > > > on
> > > > > > the
> > > > > > autobuilder.
> > > > > I know the uninative update is required for FC30 but changing the
> > > > > compression does not.
> > > > > 
> > > > > This question is about the compression extension question.
> > > > > 
> > > > > Does this mean YP will no longer be providing bz2 uninatives?
> > > > 
> > > > Correct, the patch should be straight forward to include with the
> > > > upgrade.
> > > 
> > > Does that in turn mean we must have host xz-utils?
> > 
> > It will mean you need a tar which supports the -J option. The tests
> > I've see so far suggest all our oldest supported distros do...
> 
> For thud tar >= 1.27 is required, -J was added in 1.22.
> 
> But Tom is asking about something different:
> tar calls "xz -d" for decompression, which needs xz-utils installed.
> 
> https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
> documents that xz has to be installed, but looking at 
> meta/conf/bitbake.conf it might be missing in HOSTTOOLS.

I think it's also missing in for example the crops containers.  So if
we're going to backport it, I think maybe we should keep it bz2?  The
size difference can't be that much.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Adrian Bunk]

On Wed, May 29, 2019 at 06:29:20PM -0400, Tom Rini wrote:
> On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
> > On Wed, May 29, 2019 at 10:29:31PM +0100, richard.purdie@linuxfoundation.org wrote:
> > > On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > > > On Wed, May 29, 2019 at 09:50:22PM +0100, 
> > > > richard.purdie@linuxfoundation.org wrote:
> > > > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > > > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > > > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > > > (From OE-Core rev:
> > > > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > > > 
> > > > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > > With back port to warrior and Thud too?
> > > > > > > Yes, we'll need to do that if we bring fedora 30 workers online
> > > > > > > on
> > > > > > > the
> > > > > > > autobuilder.
> > > > > > I know the uninative update is required for FC30 but changing the
> > > > > > compression does not.
> > > > > > 
> > > > > > This question is about the compression extension question.
> > > > > > 
> > > > > > Does this mean YP will no longer be providing bz2 uninatives?
> > > > > 
> > > > > Correct, the patch should be straight forward to include with the
> > > > > upgrade.
> > > > 
> > > > Does that in turn mean we must have host xz-utils?
> > > 
> > > It will mean you need a tar which supports the -J option. The tests
> > > I've see so far suggest all our oldest supported distros do...
> > 
> > For thud tar >= 1.27 is required, -J was added in 1.22.
> > 
> > But Tom is asking about something different:
> > tar calls "xz -d" for decompression, which needs xz-utils installed.
> > 
> > https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
> > documents that xz has to be installed, but looking at 
> > meta/conf/bitbake.conf it might be missing in HOSTTOOLS.
> 
> I think it's also missing in for example the crops containers.  So if
> we're going to backport it, I think maybe we should keep it bz2?  The
> size difference can't be that much.

The size difference is surprisingly high - a factor of 5 (sic).
But in practice the difference between 5 MB and 26 MB
shouldn't really matter.

> Tom

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Thu, 2019-05-30 at 02:02 +0300, Adrian Bunk wrote:
> On Wed, May 29, 2019 at 06:29:20PM -0400, Tom Rini wrote:
> > On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
> > > On Wed, May 29, 2019 at 10:29:31PM +0100, 
> > > richard.purdie@linuxfoundation.org wrote:
> > > > On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > > > > On Wed, May 29, 2019 at 09:50:22PM +0100, 
> > > > > richard.purdie@linuxfoundation.org wrote:
> > > > > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > > > > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org
> > > > > > > wrote:
> > > > > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > > > > (From OE-Core rev:
> > > > > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > > > > 
> > > > > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > > > With back port to warrior and Thud too?
> > > > > > > > Yes, we'll need to do that if we bring fedora 30
> > > > > > > > workers online
> > > > > > > > on
> > > > > > > > the
> > > > > > > > autobuilder.
> > > > > > > I know the uninative update is required for FC30 but
> > > > > > > changing the
> > > > > > > compression does not.
> > > > > > > 
> > > > > > > This question is about the compression extension
> > > > > > > question.
> > > > > > > 
> > > > > > > Does this mean YP will no longer be providing bz2
> > > > > > > uninatives?
> > > > > > 
> > > > > > Correct, the patch should be straight forward to include
> > > > > > with the
> > > > > > upgrade.
> > > > > 
> > > > > Does that in turn mean we must have host xz-utils?
> > > > 
> > > > It will mean you need a tar which supports the -J option. The
> > > > tests
> > > > I've see so far suggest all our oldest supported distros do...
> > > 
> > > For thud tar >= 1.27 is required, -J was added in 1.22.
> > > 
> > > But Tom is asking about something different:
> > > tar calls "xz -d" for decompression, which needs xz-utils
> > > installed.
> > > 
> > > https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
> > > documents that xz has to be installed, but looking at 
> > > meta/conf/bitbake.conf it might be missing in HOSTTOOLS.

uninative runs in a weird early non-worker context and doesn't actually
use HOSTTOOLS so luckily we're covered from that angle.

> > I think it's also missing in for example the crops containers.  So
> > if
> > we're going to backport it, I think maybe we should keep it
> > bz2?  The
> > size difference can't be that much.
> 
> The size difference is surprisingly high - a factor of 5 (sic).
> But in practice the difference between 5 MB and 26 MB
> shouldn't really matter.

I'm torn, partly as if we stick with bz2, we effectively do that
perpetually and given the size difference, we should switch.

Tim mentions we could fix the crops container and I'm tempted to switch
given we're so close with the current patchset...

We can add xz to HOSTTOOLS in master and that makes sense for a number
of other reasons but gets tricky as we can't add it to ASSUME_PROVIDED
as easily due to the libs it provides. I think we only need to worry
about this on master though.

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[akuster808]

On 5/30/19 1:13 AM, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2019-05-30 at 02:02 +0300, Adrian Bunk wrote:
>> On Wed, May 29, 2019 at 06:29:20PM -0400, Tom Rini wrote:
>>> On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
>>>> On Wed, May 29, 2019 at 10:29:31PM +0100, 
>>>> richard.purdie@linuxfoundation.org wrote:
>>>>> On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
>>>>>> On Wed, May 29, 2019 at 09:50:22PM +0100, 
>>>>>> richard.purdie@linuxfoundation.org wrote:
>>>>>>> On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
>>>>>>>> On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org
>>>>>>>> wrote:
>>>>>>>>> On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
>>>>>>>>>> On 5/29/19 11:43 AM, Richard Purdie wrote:
>>>>>>>>>>> (From OE-Core rev:
>>>>>>>>>>> 29fc9210b973be68de474e75068e4c72371afe5a)
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Richard Purdie <
>>>>>>>>>>> richard.purdie@linuxfoundation.org>
>>>>>>>>>> With back port to warrior and Thud too?
>>>>>>>>> Yes, we'll need to do that if we bring fedora 30
>>>>>>>>> workers online
>>>>>>>>> on
>>>>>>>>> the
>>>>>>>>> autobuilder.
>>>>>>>> I know the uninative update is required for FC30 but
>>>>>>>> changing the
>>>>>>>> compression does not.
>>>>>>>>
>>>>>>>> This question is about the compression extension
>>>>>>>> question.
>>>>>>>>
>>>>>>>> Does this mean YP will no longer be providing bz2
>>>>>>>> uninatives?
>>>>>>> Correct, the patch should be straight forward to include
>>>>>>> with the
>>>>>>> upgrade.
>>>>>> Does that in turn mean we must have host xz-utils?
>>>>> It will mean you need a tar which supports the -J option. The
>>>>> tests
>>>>> I've see so far suggest all our oldest supported distros do...
>>>> For thud tar >= 1.27 is required, -J was added in 1.22.
>>>>
>>>> But Tom is asking about something different:
>>>> tar calls "xz -d" for decompression, which needs xz-utils
>>>> installed.
>>>>
>>>> https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
>>>> documents that xz has to be installed, but looking at 
>>>> meta/conf/bitbake.conf it might be missing in HOSTTOOLS.
> uninative runs in a weird early non-worker context and doesn't actually
> use HOSTTOOLS so luckily we're covered from that angle.
>
>>> I think it's also missing in for example the crops containers.  So
>>> if
>>> we're going to backport it, I think maybe we should keep it
>>> bz2?  The
>>> size difference can't be that much.
>> The size difference is surprisingly high - a factor of 5 (sic).
>> But in practice the difference between 5 MB and 26 MB
>> shouldn't really matter.
> I'm torn, partly as if we stick with bz2, we effectively do that
> perpetually and given the size difference, we should switch.
>
> Tim mentions we could fix the crops container and I'm tempted to switch
> given we're so close with the current patchset...
>
> We can add xz to HOSTTOOLS in master and that makes sense for a number
> of other reasons but gets tricky as we can't add it to ASSUME_PROVIDED
> as easily due to the libs it provides. I think we only need to worry
> about this on master though.
So.. Can we have a bz and xz version built until thud is in community
supported and update Warrior to use xz ?

- Armin
> Cheers,
>
> Richard
>
>
>

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Thu, 2019-05-30 at 06:32 -0700, akuster808 wrote:
> 
> On 5/30/19 1:13 AM, richard.purdie@linuxfoundation.org wrote:
> > On Thu, 2019-05-30 at 02:02 +0300, Adrian Bunk wrote:
> > > On Wed, May 29, 2019 at 06:29:20PM -0400, Tom Rini wrote:
> > > > On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
> > > > > On Wed, May 29, 2019 at 10:29:31PM +0100, 
> > > > > richard.purdie@linuxfoundation.org wrote:
> > > > > > On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > > > > > > On Wed, May 29, 2019 at 09:50:22PM +0100, 
> > > > > > > richard.purdie@linuxfoundation.org wrote:
> > > > > > > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > > > > > > On 5/29/19 12:26 PM, 
> > > > > > > > > richard.purdie@linuxfoundation.org
> > > > > > > > > wrote:
> > > > > > > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808
> > > > > > > > > > wrote:
> > > > > > > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > > > > > > (From OE-Core rev:
> > > > > > > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > > > > > > 
> > > > > > > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > > > > > With back port to warrior and Thud too?
> > > > > > > > > > Yes, we'll need to do that if we bring fedora 30
> > > > > > > > > > workers online
> > > > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > autobuilder.
> > > > > > > > > I know the uninative update is required for FC30 but
> > > > > > > > > changing the
> > > > > > > > > compression does not.
> > > > > > > > > 
> > > > > > > > > This question is about the compression extension
> > > > > > > > > question.
> > > > > > > > > 
> > > > > > > > > Does this mean YP will no longer be providing bz2
> > > > > > > > > uninatives?
> > > > > > > > Correct, the patch should be straight forward to
> > > > > > > > include
> > > > > > > > with the
> > > > > > > > upgrade.
> > > > > > > Does that in turn mean we must have host xz-utils?
> > > > > > It will mean you need a tar which supports the -J option.
> > > > > > The
> > > > > > tests
> > > > > > I've see so far suggest all our oldest supported distros
> > > > > > do...
> > > > > For thud tar >= 1.27 is required, -J was added in 1.22.
> > > > > 
> > > > > But Tom is asking about something different:
> > > > > tar calls "xz -d" for decompression, which needs xz-utils
> > > > > installed.
> > > > > 
> > > > > https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
> > > > > documents that xz has to be installed, but looking at 
> > > > > meta/conf/bitbake.conf it might be missing in HOSTTOOLS.
> > uninative runs in a weird early non-worker context and doesn't
> > actually
> > use HOSTTOOLS so luckily we're covered from that angle.
> > 
> > > > I think it's also missing in for example the crops
> > > > containers.  So
> > > > if
> > > > we're going to backport it, I think maybe we should keep it
> > > > bz2?  The
> > > > size difference can't be that much.
> > > The size difference is surprisingly high - a factor of 5 (sic).
> > > But in practice the difference between 5 MB and 26 MB
> > > shouldn't really matter.
> > I'm torn, partly as if we stick with bz2, we effectively do that
> > perpetually and given the size difference, we should switch.
> > 
> > Tim mentions we could fix the crops container and I'm tempted to
> > switch
> > given we're so close with the current patchset...
> > 
> > We can add xz to HOSTTOOLS in master and that makes sense for a
> > number
> > of other reasons but gets tricky as we can't add it to
> > ASSUME_PROVIDED
> > as easily due to the libs it provides. I think we only need to
> > worry
> > about this on master though.
> So.. Can we have a bz and xz version built until thud is in community
> supported and update Warrior to use xz ?

I think thud and warrior will work fine if we backport just the upgrade
and the change to uninative.bbclass?

I really don't want to try and do both.

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Tom Rini]

[-- Attachment #1: Type: text/plain, Size: 4842 bytes --]

On Thu, May 30, 2019 at 02:55:35PM +0100, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2019-05-30 at 06:32 -0700, akuster808 wrote:
> > 
> > On 5/30/19 1:13 AM, richard.purdie@linuxfoundation.org wrote:
> > > On Thu, 2019-05-30 at 02:02 +0300, Adrian Bunk wrote:
> > > > On Wed, May 29, 2019 at 06:29:20PM -0400, Tom Rini wrote:
> > > > > On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
> > > > > > On Wed, May 29, 2019 at 10:29:31PM +0100, 
> > > > > > richard.purdie@linuxfoundation.org wrote:
> > > > > > > On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > > > > > > > On Wed, May 29, 2019 at 09:50:22PM +0100, 
> > > > > > > > richard.purdie@linuxfoundation.org wrote:
> > > > > > > > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > > > > > > > On 5/29/19 12:26 PM, 
> > > > > > > > > > richard.purdie@linuxfoundation.org
> > > > > > > > > > wrote:
> > > > > > > > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808
> > > > > > > > > > > wrote:
> > > > > > > > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > > > > > > > (From OE-Core rev:
> > > > > > > > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > > > > > > With back port to warrior and Thud too?
> > > > > > > > > > > Yes, we'll need to do that if we bring fedora 30
> > > > > > > > > > > workers online
> > > > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > autobuilder.
> > > > > > > > > > I know the uninative update is required for FC30 but
> > > > > > > > > > changing the
> > > > > > > > > > compression does not.
> > > > > > > > > > 
> > > > > > > > > > This question is about the compression extension
> > > > > > > > > > question.
> > > > > > > > > > 
> > > > > > > > > > Does this mean YP will no longer be providing bz2
> > > > > > > > > > uninatives?
> > > > > > > > > Correct, the patch should be straight forward to
> > > > > > > > > include
> > > > > > > > > with the
> > > > > > > > > upgrade.
> > > > > > > > Does that in turn mean we must have host xz-utils?
> > > > > > > It will mean you need a tar which supports the -J option.
> > > > > > > The
> > > > > > > tests
> > > > > > > I've see so far suggest all our oldest supported distros
> > > > > > > do...
> > > > > > For thud tar >= 1.27 is required, -J was added in 1.22.
> > > > > > 
> > > > > > But Tom is asking about something different:
> > > > > > tar calls "xz -d" for decompression, which needs xz-utils
> > > > > > installed.
> > > > > > 
> > > > > > https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host 
> > > > > > documents that xz has to be installed, but looking at 
> > > > > > meta/conf/bitbake.conf it might be missing in HOSTTOOLS.
> > > uninative runs in a weird early non-worker context and doesn't
> > > actually
> > > use HOSTTOOLS so luckily we're covered from that angle.
> > > 
> > > > > I think it's also missing in for example the crops
> > > > > containers.  So
> > > > > if
> > > > > we're going to backport it, I think maybe we should keep it
> > > > > bz2?  The
> > > > > size difference can't be that much.
> > > > The size difference is surprisingly high - a factor of 5 (sic).
> > > > But in practice the difference between 5 MB and 26 MB
> > > > shouldn't really matter.
> > > I'm torn, partly as if we stick with bz2, we effectively do that
> > > perpetually and given the size difference, we should switch.
> > > 
> > > Tim mentions we could fix the crops container and I'm tempted to
> > > switch
> > > given we're so close with the current patchset...
> > > 
> > > We can add xz to HOSTTOOLS in master and that makes sense for a
> > > number
> > > of other reasons but gets tricky as we can't add it to
> > > ASSUME_PROVIDED
> > > as easily due to the libs it provides. I think we only need to
> > > worry
> > > about this on master though.
> > So.. Can we have a bz and xz version built until thud is in community
> > supported and update Warrior to use xz ?
> 
> I think thud and warrior will work fine if we backport just the upgrade
> and the change to uninative.bbclass?
> 
> I really don't want to try and do both.

To be honest, I'd really like to see thud/warrior keep the bz2 and
master switch.  Changing underlying host dependencies especially ones
that we hit on in special early threads seems like a bad idea.  I've
been doing too many "fun" things in containers of late, and CROPS might
be fine but bare Ubuntu containers aren't and this would be a very funky
error to diagnose.  Given we've released I think we need to err on the
side of consistency.

-- 
Tom

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Thu, 2019-05-30 at 11:57 -0400, Tom Rini wrote:
> On Thu, May 30, 2019 at 02:55:35PM +0100, 
> richard.purdie@linuxfoundation.org wrote:
> > On Thu, 2019-05-30 at 06:32 -0700, akuster808 wrote:
> > > On 5/30/19 1:13 AM, richard.purdie@linuxfoundation.org wrote:
> > > > On Thu, 2019-05-30 at 02:02 +0300, Adrian Bunk wrote:
> > > So.. Can we have a bz and xz version built until thud is in
> > > community
> > > supported and update Warrior to use xz ?
> > 
> > I think thud and warrior will work fine if we backport just the
> > upgrade
> > and the change to uninative.bbclass?
> > 
> > I really don't want to try and do both.
> 
> To be honest, I'd really like to see thud/warrior keep the bz2 and
> master switch.  Changing underlying host dependencies especially ones
> that we hit on in special early threads seems like a bad idea.  I've
> been doing too many "fun" things in containers of late, and CROPS
> might be fine but bare Ubuntu containers aren't and this would be a
> very funky error to diagnose.  Given we've released I think we need
> to err on the side of consistency.

Whilst we could in theory start hacking around the infrastructure to
put dual tarballs in place, I'd really prefer not to be doing that for
uninative releases for the next 1-2 years.

xz appears to be documented as a host dependency even if it isn't in
HOSTTOOLS and isn't very obscure.

I can see both sides of this :/.

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Adrian Bunk]

On Thu, May 30, 2019 at 09:13:02AM +0100, richard.purdie@linuxfoundation.org wrote:
>...
> I'm torn, partly as if we stick with bz2, we effectively do that
> perpetually and given the size difference, we should switch.
> 
> Tim mentions we could fix the crops container and I'm tempted to switch
> given we're so close with the current patchset...
> 
> We can add xz to HOSTTOOLS in master and that makes sense for a number
> of other reasons but gets tricky as we can't add it to ASSUME_PROVIDED
> as easily due to the libs it provides. I think we only need to worry
> about this on master though.

There are also some other related topics that might be considered
on how this should develop long-term:

1. How important is it to support host distributions that are not 
listed as supported? Not limited to uninative it can be painful
adding support for new distributions to stable branches.

2. Does using uninative by default bring enough benefits to outweight 
the problems with new host distributions, or should it become 
non-default in master?

The sum of the two points above is especially problematic:
"re-use of native shared state artifacts across different host 
distributions" is a more exotic feature, and effectively supporting
this for unsupported host distributions is the problem here.

3. uninative is so fragile because it tries to fix the C library
only. A lot of problems with host distributions (not limited to
uninative) would go away if one host distribution would be picked
and provided as container for people who are using other distributions.

> Cheers,
> 
> Richard

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[richard.purdie]

On Thu, 2019-05-30 at 16:59 +0300, Adrian Bunk wrote:
> There are also some other related topics that might be considered
> on how this should develop long-term:
> 
> 1. How important is it to support host distributions that are not 
> listed as supported? Not limited to uninative it can be painful
> adding support for new distributions to stable branches.

The distros we do support were chosen to give a decent cross section of
the Linux ecosystem. Whilst not full coverage, the delta between those
and other systems should be relatively minimal as we have both older
and modern in there.

> 2. Does using uninative by default bring enough benefits to
> outweight the problems with new host distributions, or should it
> become non-default in master?

We test it heavily on the autobuilder and also benefit from it
massively there so in my view its a significant win for us alone. We
chose to share it with the ecosystem. You're not forced to use it.

> The sum of the two points above is especially problematic:
> "re-use of native shared state artifacts across different host 
> distributions" is a more exotic feature, and effectively supporting
> this for unsupported host distributions is the problem here.

Which distros are you seeing a problem with and what kinds of problems?
I'd hoped/believed the problems should be relatively minor unless
they're really old.

You could also just disable uninative on specific distros if that was
needed.

> 3. uninative is so fragile because it tries to fix the C library
> only. A lot of problems with host distributions (not limited to
> uninative) would go away if one host distribution would be picked
> and provided as container for people who are using other
> distributions.

We could make lots of things easier if we mandated a specific
underlying configuration but that isn't how people use the project
(including commercially), or why people are attracted to it so whilst I
understand the attraction and it would make things simpler, I don't see
it as a viable option.

I am curious what issues you're running into specifically though.

Cheers,

Richard

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Adrian Bunk]

On Thu, May 30, 2019 at 03:06:56PM +0100, richard.purdie@linuxfoundation.org wrote:
> On Thu, 2019-05-30 at 16:59 +0300, Adrian Bunk wrote:
> > There are also some other related topics that might be considered
> > on how this should develop long-term:
> > 
> > 1. How important is it to support host distributions that are not 
> > listed as supported? Not limited to uninative it can be painful
> > adding support for new distributions to stable branches.
> 
> The distros we do support were chosen to give a decent cross section of
> the Linux ecosystem. Whilst not full coverage, the delta between those
> and other systems should be relatively minimal as we have both older
> and modern in there.

I agree that it usually works on the range between the oldest and the 
newest supported host distribution.

But adding support for more recent host distributions to stable 
branches could end up being any amount of troubles if you are
unlucky.

> > 2. Does using uninative by default bring enough benefits to
> > outweight the problems with new host distributions, or should it
> > become non-default in master?
> 
> We test it heavily on the autobuilder and also benefit from it
> massively there so in my view its a significant win for us alone. We
> chose to share it with the ecosystem. You're not forced to use it.
> 
> > The sum of the two points above is especially problematic:
> > "re-use of native shared state artifacts across different host 
> > distributions" is a more exotic feature, and effectively supporting
> > this for unsupported host distributions is the problem here.
> 
> Which distros are you seeing a problem with and what kinds of problems?
> I'd hoped/believed the problems should be relatively minor unless
> they're really old.
>...

The problem is that uninative is expected to break 3 times per year, 
with every new release of gcc or glibc.

Are the problems people regularly see on bleeding-edge distributions
really offset by the benefits of having it enabled by default?

> Cheers,
> 
> Richard

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[akuster808]

On 5/30/19 6:59 AM, Adrian Bunk wrote:
> On Thu, May 30, 2019 at 09:13:02AM +0100, richard.purdie@linuxfoundation.org wrote:
>> ...
>> I'm torn, partly as if we stick with bz2, we effectively do that
>> perpetually and given the size difference, we should switch.
>>
>> Tim mentions we could fix the crops container and I'm tempted to switch
>> given we're so close with the current patchset...
>>
>> We can add xz to HOSTTOOLS in master and that makes sense for a number
>> of other reasons but gets tricky as we can't add it to ASSUME_PROVIDED
>> as easily due to the libs it provides. I think we only need to worry
>> about this on master though.
> There are also some other related topics that might be considered
> on how this should develop long-term:
>
> 1. How important is it to support host distributions that are not 
> listed as supported? Not limited to uninative it can be painful
> adding support for new distributions to stable branches.

When new hosts are added to the AB and a stable branch builds and all
tests pass on that host, we have updated the docs to reflect such an
addition.
its not a fun exercise and gets harder the older the stable branch is.

It is also noted in the Stable branch maintenance story on wiki and has
been SOP for years.

- armin
>
> 2. Does using uninative by default bring enough benefits to outweight 
> the problems with new host distributions, or should it become 
> non-default in master?
>
> The sum of the two points above is especially problematic:
> "re-use of native shared state artifacts across different host 
> distributions" is a more exotic feature, and effectively supporting
> this for unsupported host distributions is the problem here.
>
> 3. uninative is so fragile because it tries to fix the C library
> only. A lot of problems with host distributions (not limited to
> uninative) would go away if one host distribution would be picked
> and provided as container for people who are using other distributions.
>
>> Cheers,
>>
>> Richard
> cu
> Adrian
>

Re: [PATCH 2/2] uninative: Switch from bz2 to xz

[Tim Orling]

[-- Attachment #1: Type: text/plain, Size: 2584 bytes --]

On Wed, May 29, 2019 at 3:29 PM Tom Rini <trini@konsulko.com> wrote:

> On Thu, May 30, 2019 at 01:17:26AM +0300, Adrian Bunk wrote:
> > On Wed, May 29, 2019 at 10:29:31PM +0100,
> richard.purdie@linuxfoundation.org wrote:
> > > On Wed, 2019-05-29 at 17:25 -0400, Tom Rini wrote:
> > > > On Wed, May 29, 2019 at 09:50:22PM +0100,
> > > > richard.purdie@linuxfoundation.org wrote:
> > > > > On Wed, 2019-05-29 at 13:39 -0700, akuster808 wrote:
> > > > > > On 5/29/19 12:26 PM, richard.purdie@linuxfoundation.org wrote:
> > > > > > > On Wed, 2019-05-29 at 12:21 -0700, akuster808 wrote:
> > > > > > > > On 5/29/19 11:43 AM, Richard Purdie wrote:
> > > > > > > > > (From OE-Core rev:
> > > > > > > > > 29fc9210b973be68de474e75068e4c72371afe5a)
> > > > > > > > >
> > > > > > > > > Signed-off-by: Richard Purdie <
> > > > > > > > > richard.purdie@linuxfoundation.org>
> > > > > > > > With back port to warrior and Thud too?
> > > > > > > Yes, we'll need to do that if we bring fedora 30 workers online
> > > > > > > on
> > > > > > > the
> > > > > > > autobuilder.
> > > > > > I know the uninative update is required for FC30 but changing the
> > > > > > compression does not.
> > > > > >
> > > > > > This question is about the compression extension question.
> > > > > >
> > > > > > Does this mean YP will no longer be providing bz2 uninatives?
> > > > >
> > > > > Correct, the patch should be straight forward to include with the
> > > > > upgrade.
> > > >
> > > > Does that in turn mean we must have host xz-utils?
> > >
> > > It will mean you need a tar which supports the -J option. The tests
> > > I've see so far suggest all our oldest supported distros do...
> >
> > For thud tar >= 1.27 is required, -J was added in 1.22.
> >
> > But Tom is asking about something different:
> > tar calls "xz -d" for decompression, which needs xz-utils installed.
> >
> >
> https://www.yoctoproject.org/docs/2.6/ref-manual/ref-manual.html#required-packages-for-the-build-host
> > documents that xz has to be installed, but looking at
> > meta/conf/bitbake.conf it might be missing in HOSTTOOLS.
>
> I think it's also missing in for example the crops containers.


We can add it to the crops containers.

So if
> we're going to backport it, I think maybe we should keep it bz2?  The
> size difference can't be that much.
>
> --
> Tom
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 4304 bytes --]