[PATCH] bitbake.conf: add bunzip2 in HOSTTOOLS

[PATCH] bitbake.conf: add bunzip2 in HOSTTOOLS

[aplanas]

From: Alberto Planas <aplanas@suse.com>

rpm2cpio.sh can make calls to bunzip2 to uncompress the RPM payload that
conform the cpio file.

bzip2 is already part of HOSTTOOLS, as a link to the system installed
bzip2.

This patch add bunzip2 in HOSTOOLS list as a non-optional binary, so is
available to rpm2cpio.sh when it is required.

The currect rpm2cpio.sh has other calls to gunzip (present in HOSTOOLS),
xzcat (missing), unlzma (missing) and unzstd (present in HOSTTOOLS since
bff58d337890e804d33d7decbaa46065a4d3bba4)

Adding the missing ones will not add any new host dependency as xz-utils
is already listed as a requirement.

Signed-off-by: Alberto Planas <aplanas@suse.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 599bbc4ba8..b19f265b3d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -516,7 +516,7 @@ HOSTTOOLS_DIR = "${TMPDIR}/hosttools"
 
 # Tools needed to run builds with OE-Core
 HOSTTOOLS += " \
-    [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
+    [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
     cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
     fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
     head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
-- 
2.41.0

Re: [OE-core] [PATCH] bitbake.conf: add bunzip2 in HOSTTOOLS

[Pavel Zhukov]

[-- Attachment #1: Type: text/plain, Size: 2545 bytes --]

Hi Alberto,
Looks like the regression has been  introduced in  previous update. This patch https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=eb3ec7469fff857c819332371ad1d586f43c79c3 (which was reverted by https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=3aba44a75dd565b192f7328f2a0150a313de3cc1 )  solved the issue without bringing bunzip to every system

-- 
Pavel

On Fri, Aug 18, 2023, at 11:30, Alberto Planas via lists.openembedded.org wrote:
> From: Alberto Planas <aplanas@suse.com>
> 
> rpm2cpio.sh can make calls to bunzip2 to uncompress the RPM payload that
> conform the cpio file.
> 
> bzip2 is already part of HOSTTOOLS, as a link to the system installed
> bzip2.
> 
> This patch add bunzip2 in HOSTOOLS list as a non-optional binary, so is
> available to rpm2cpio.sh when it is required.
> 
> The currect rpm2cpio.sh has other calls to gunzip (present in HOSTOOLS),
> xzcat (missing), unlzma (missing) and unzstd (present in HOSTTOOLS since
> bff58d337890e804d33d7decbaa46065a4d3bba4)
> 
> Adding the missing ones will not add any new host dependency as xz-utils
> is already listed as a requirement.
> 
> Signed-off-by: Alberto Planas <aplanas@suse.com>
> ---
> meta/conf/bitbake.conf | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 599bbc4ba8..b19f265b3d 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -516,7 +516,7 @@ HOSTTOOLS_DIR = "${TMPDIR}/hosttools"
>  
> # Tools needed to run builds with OE-Core
> HOSTTOOLS += " \
> -    [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
> +    [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
>      head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
> -- 
> 2.41.0
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186353): https://lists.openembedded.org/g/openembedded-core/message/186353
> Mute This Topic: https://lists.openembedded.org/mt/100817328/6390638
> Group Owner: openembedded-core+owner@lists.openembedded.org <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [pavel@zhukoff.net]
> -=-=-=-=-=-=-=-=-=-=-=-
> 
> 

[-- Attachment #2: Type: text/html, Size: 4226 bytes --]

Re: [OE-core] [PATCH] bitbake.conf: add bunzip2 in HOSTTOOLS

[Alexandre Belloni]

Hi Pavel,

On 18/08/2023 11:39:12+0200, Pavel Zhukov wrote:
> Hi Alberto,
> Looks like the regression has been  introduced in  previous update. This patch https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=eb3ec7469fff857c819332371ad1d586f43c79c3 (which was reverted by https://git.openembedded.org/openembedded-core/commit/?h=master-next&id=3aba44a75dd565b192f7328f2a0150a313de3cc1 )  solved the issue without bringing bunzip to every system
> 

I guess you should submit the oe-core patch again and maybe it would be
a good idea to get it upstream.

> -- 
> Pavel
> 
> On Fri, Aug 18, 2023, at 11:30, Alberto Planas via lists.openembedded.org wrote:
> > From: Alberto Planas <aplanas@suse.com>
> > 
> > rpm2cpio.sh can make calls to bunzip2 to uncompress the RPM payload that
> > conform the cpio file.
> > 
> > bzip2 is already part of HOSTTOOLS, as a link to the system installed
> > bzip2.
> > 
> > This patch add bunzip2 in HOSTOOLS list as a non-optional binary, so is
> > available to rpm2cpio.sh when it is required.
> > 
> > The currect rpm2cpio.sh has other calls to gunzip (present in HOSTOOLS),
> > xzcat (missing), unlzma (missing) and unzstd (present in HOSTTOOLS since
> > bff58d337890e804d33d7decbaa46065a4d3bba4)
> > 
> > Adding the missing ones will not add any new host dependency as xz-utils
> > is already listed as a requirement.
> > 
> > Signed-off-by: Alberto Planas <aplanas@suse.com>
> > ---
> > meta/conf/bitbake.conf | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index 599bbc4ba8..b19f265b3d 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -516,7 +516,7 @@ HOSTTOOLS_DIR = "${TMPDIR}/hosttools"
> >  
> > # Tools needed to run builds with OE-Core
> > HOSTTOOLS += " \
> > -    [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
> > +    [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
> >      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
> >      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
> >      head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \
> > -- 
> > 2.41.0
> > 
> > 
> > 
> > 
> > 
> > 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#186354): https://lists.openembedded.org/g/openembedded-core/message/186354
> Mute This Topic: https://lists.openembedded.org/mt/100817328/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] bitbake.conf: add bunzip2 in HOSTTOOLS

[Alberto Planas Dominguez]

From: Pavel Zhukov <pavel@zhukoff.net>

> Looks like the regression has been  introduced in  previous update. This 
> patch https://git.openembedded.org/openembedded-core/commit/?h=master-
> next&id=eb3ec7469fff857c819332371ad1d586f43c79c3 (which was reverted 
> by https://git.openembedded.org/openembedded-core/commit/?h=master-next&
> id=3aba44a75dd565b192f7328f2a0150a313de3cc1 )  solved the issue without 
> bringing bunzip to every system

I see! Good catch! Seem that I am indeed the culprit on this regression.

I have some questions to evaluate the alternatives:

1) Is it OK to change rpm2cpio.sh?

This is a verbatim copy of the upstream version from [1]. If we edit it we take the risk of reverting again the changes done in the oe-core copy, changes that are hard to track.

2) Is it OK to drag more stuff into HOSTTOOLS?

I do not know the cost, but IMHO it is pretty cheap to populate links in HOSTTOOLS *only* if the link does not introduce any new dependency on the host machine. In the case of bunzip, unzstd et al. those are already listed in the packages that should be installed on the host, so there is no new dependency or new package to install.

I think that the safe option is that, for already installed dependencies, it is OK to add a new link in "tmp/hosttools", because the risk of changing rpm2cpio.sh is that those changes can be discarded accidentally when this script gets updated (as I sadly did)

If we do not agree and the first solution is selected, then 9e14d2ab801e7f35af6d43dd5d071fe29821d27c should be reverted too and more changes in rpm2cpio needs to be done.

[1] https://github.com/rpm-software-management/rpm/blob/master/scripts/rpm2cpio.sh

Re: [OE-core] [PATCH] bitbake.conf: add bunzip2 in HOSTTOOLS

[Richard Purdie]

On Fri, 2023-08-18 at 11:30 +0200, Alberto Planas via
lists.openembedded.org wrote:
> From: Alberto Planas <aplanas@suse.com>
> 
> rpm2cpio.sh can make calls to bunzip2 to uncompress the RPM payload that
> conform the cpio file.
> 
> bzip2 is already part of HOSTTOOLS, as a link to the system installed
> bzip2.
> 
> This patch add bunzip2 in HOSTOOLS list as a non-optional binary, so is
> available to rpm2cpio.sh when it is required.
> 
> The currect rpm2cpio.sh has other calls to gunzip (present in HOSTOOLS),
> xzcat (missing), unlzma (missing) and unzstd (present in HOSTTOOLS since
> bff58d337890e804d33d7decbaa46065a4d3bba4)
> 
> Adding the missing ones will not add any new host dependency as xz-utils
> is already listed as a requirement.
> 
> Signed-off-by: Alberto Planas <aplanas@suse.com>
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index 599bbc4ba8..b19f265b3d 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -516,7 +516,7 @@ HOSTTOOLS_DIR = "${TMPDIR}/hosttools"
>  
>  # Tools needed to run builds with OE-Core
>  HOSTTOOLS += " \
> -    [ ar as awk basename bash bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
> +    [ ar as awk basename bash bunzip2 bzip2 cat chgrp chmod chown chrpath cmp comm cp cpio \
>      cpp cut date dd diff diffstat dirname du echo egrep env expand expr false \
>      fgrep file find flock g++ gawk gcc getconf getopt git grep gunzip gzip \
>      head hostname iconv id install ld ldd ln ls lz4c make md5sum mkdir mkfifo mknod \

I ended up thinking we should take this in favour of an easier life. It
resulted in this repoducibility issue:

http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230829-oldigjb2/packages/diff-html/

sadly I only discovered that after merging the patch :(

Cheers,

Richard