[PATCH] poky.yaml.in: simplify host package requirements

[PATCH] poky.yaml.in: simplify host package requirements

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

Some of the listed host packages were no longer necessary.

[YOCTO #14873]

Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
---
 documentation/poky.yaml.in | 28 ++++++++--------------------
 1 file changed, 8 insertions(+), 20 deletions(-)

diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
index 1c072c1d05..7f71412fee 100644
--- a/documentation/poky.yaml.in
+++ b/documentation/poky.yaml.in
@@ -10,31 +10,19 @@ BITBAKE_SERIES : ""
 YOCTO_DL_URL : "https://downloads.yoctoproject.org"
 YOCTO_AB_URL : "https://autobuilder.yoctoproject.org"
 YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
-UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip texinfo gcc \
-     build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
-     xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev \
-     python3-subunit mesa-common-dev zstd liblz4-tool file locales
+UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat build-essential chrpath cpio \
+     python3 zstd liblz4-tool file locales
      \n\   $ sudo locale-gen en_US.UTF-8"
-FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip perl patch \
-     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
-     ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
-     python3-pexpect findutils which file cpio python python3-pip xz python3-GitPython \
-     python3-jinja2 SDL-devel rpcgen mesa-libGL-devel perl-FindBin perl-File-Compare \
-     perl-File-Copy perl-locale zstd lz4 hostname glibc-langpack-en"
-OPENSUSE_HOST_PACKAGES_ESSENTIAL : "python gcc gcc-c++ git chrpath make wget python-xml \
-     diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip \
-     python3-pexpect xz which python3-Jinja2 Mesa-libEGL1 libSDL-devel rpcgen Mesa-dri-devel \
-     zstd lz4 bzip2 gzip hostname
-     \n\   $ sudo pip3 install GitPython"
+FEDORA_HOST_PACKAGES_ESSENTIAL : "bzip2 perl patch diffutils diffstat git gcc g++ chrpath \
+     which glibc-langpack-en which cpio file hostname patch rpcgen wget zstd lz4 xz"
+OPENSUSE_HOST_PACKAGES_ESSENTIAL : "git python3 gcc gcc-c++ zstd bzip2 chrpath diffstat \
+     gzip hostname lz4 make patch rpcgen tar wget python3-curses xz" 
 ALMALINUX_HOST_PACKAGES_ESSENTIAL : "-y epel-release
      \n\   $ sudo yum install dnf-plugins-core
      \n\   $ sudo dnf config-manager --set-enabled crb
      \n\   $ sudo dnf makecache
-     \n\   $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
-     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache \
-     socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip \
-     python3-GitPython python3-jinja2 python3-pexpect xz which SDL-devel \
-     rpcgen mesa-libGL-devel zstd lz4 cpio glibc-langpack-en"
+     \n\   $ sudo dnf install bzip2 perl patch diffutils diffstat git gcc g++ chrpath \
+     which glibc-langpack-en which cpio file hostname patch rpcgen wget zstd lz4 xz"
 PIP3_HOST_PACKAGES_DOC : "$ sudo pip3 install sphinx sphinx_rtd_theme pyyaml"
 MIN_PYTHON_VERSION : "3.8.0"
 MIN_TAR_VERSION : "1.28"
-- 
2.34.1

Re: [docs] [PATCH] poky.yaml.in: simplify host package requirements

[Richard Purdie]

On Wed, 2023-05-03 at 11:01 +0200, Michael Opdenacker via
lists.yoctoproject.org wrote:
> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
> 
> Some of the listed host packages were no longer necessary.
> 
> [YOCTO #14873]
> 
> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> ---
>  documentation/poky.yaml.in | 28 ++++++++--------------------
>  1 file changed, 8 insertions(+), 20 deletions(-)

I think this is another "it depends" situation.

The cut down lists you have below probably works fine if you just want
to build and image and hopefully let you run it under runqemu.

Where things will start to get problematic is some of the testing
workflows such as "oe-selftest -j X" which definitely needs python3-
subunit, or resulttool which does need python3-jinja2 for some options
iirc.

As such, which this change is right for some workflows, it isn't
correct for others.

Ideally there would be two levels of dependencies, those needed for the
basics and those needed to cover all the needs.

Looking at the original bug, I think we did remove some of the
SDL/xterm/vnc requirements. Even then some vnc pieces are still needed
for some autobuilder setups though.

We likely should define which workflows we expect to work with this
"essential" dependency list?

Cheers,

Richard


> diff --git a/documentation/poky.yaml.in b/documentation/poky.yaml.in
> index 1c072c1d05..7f71412fee 100644
> --- a/documentation/poky.yaml.in
> +++ b/documentation/poky.yaml.in
> @@ -10,31 +10,19 @@ BITBAKE_SERIES : ""
>  YOCTO_DL_URL : "https://downloads.yoctoproject.org"
>  YOCTO_AB_URL : "https://autobuilder.yoctoproject.org"
>  YOCTO_RELEASE_DL_URL : "&YOCTO_DL_URL;/releases/yocto/yocto-&DISTRO;"
> -UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat unzip texinfo gcc \
> -     build-essential chrpath socat cpio python3 python3-pip python3-pexpect \
> -     xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev \
> -     python3-subunit mesa-common-dev zstd liblz4-tool file locales
> +UBUNTU_HOST_PACKAGES_ESSENTIAL : "gawk wget git diffstat build-essential chrpath cpio \
> +     python3 zstd liblz4-tool file locales
>       \n\   $ sudo locale-gen en_US.UTF-8"
> -FEDORA_HOST_PACKAGES_ESSENTIAL : "gawk make wget tar bzip2 gzip python3 unzip perl patch \
> -     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath \
> -     ccache perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue perl-bignum socat \
> -     python3-pexpect findutils which file cpio python python3-pip xz python3-GitPython \
> -     python3-jinja2 SDL-devel rpcgen mesa-libGL-devel perl-FindBin perl-File-Compare \
> -     perl-File-Copy perl-locale zstd lz4 hostname glibc-langpack-en"
> -OPENSUSE_HOST_PACKAGES_ESSENTIAL : "python gcc gcc-c++ git chrpath make wget python-xml \
> -     diffstat makeinfo python-curses patch socat python3 python3-curses tar python3-pip \
> -     python3-pexpect xz which python3-Jinja2 Mesa-libEGL1 libSDL-devel rpcgen Mesa-dri-devel \
> -     zstd lz4 bzip2 gzip hostname
> -     \n\   $ sudo pip3 install GitPython"
> +FEDORA_HOST_PACKAGES_ESSENTIAL : "bzip2 perl patch diffutils diffstat git gcc g++ chrpath \
> +     which glibc-langpack-en which cpio file hostname patch rpcgen wget zstd lz4 xz"
> +OPENSUSE_HOST_PACKAGES_ESSENTIAL : "git python3 gcc gcc-c++ zstd bzip2 chrpath diffstat \
> +     gzip hostname lz4 make patch rpcgen tar wget python3-curses xz" 
>  ALMALINUX_HOST_PACKAGES_ESSENTIAL : "-y epel-release
>       \n\   $ sudo yum install dnf-plugins-core
>       \n\   $ sudo dnf config-manager --set-enabled crb
>       \n\   $ sudo dnf makecache
> -     \n\   $ sudo dnf install gawk make wget tar bzip2 gzip python3 unzip perl patch \
> -     diffutils diffstat git cpp gcc gcc-c++ glibc-devel texinfo chrpath ccache \
> -     socat perl-Data-Dumper perl-Text-ParseWords perl-Thread-Queue python3-pip \
> -     python3-GitPython python3-jinja2 python3-pexpect xz which SDL-devel \
> -     rpcgen mesa-libGL-devel zstd lz4 cpio glibc-langpack-en"
> +     \n\   $ sudo dnf install bzip2 perl patch diffutils diffstat git gcc g++ chrpath \
> +     which glibc-langpack-en which cpio file hostname patch rpcgen wget zstd lz4 xz"
>  PIP3_HOST_PACKAGES_DOC : "$ sudo pip3 install sphinx sphinx_rtd_theme pyyaml"
>  MIN_PYTHON_VERSION : "3.8.0"
>  MIN_TAR_VERSION : "1.28"
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#3895): https://lists.yoctoproject.org/g/docs/message/3895
> Mute This Topic: https://lists.yoctoproject.org/mt/98657701/1686473
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [richard.purdie@linuxfoundation.org]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [docs] [PATCH] poky.yaml.in: simplify host package requirements

[Michael Opdenacker]

Hi Richard,

Thank you for reviewing this patch!

On 03.05.23 at 11:13, Richard Purdie wrote:
> On Wed, 2023-05-03 at 11:01 +0200, Michael Opdenacker via
> lists.yoctoproject.org wrote:
>> From: Michael Opdenacker <michael.opdenacker@bootlin.com>
>>
>> Some of the listed host packages were no longer necessary.
>>
>> [YOCTO #14873]
>>
>> Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
>> ---
>>   documentation/poky.yaml.in | 28 ++++++++--------------------
>>   1 file changed, 8 insertions(+), 20 deletions(-)
> I think this is another "it depends" situation.
>
> The cut down lists you have below probably works fine if you just want
> to build and image and hopefully let you run it under runqemu.
>
> Where things will start to get problematic is some of the testing
> workflows such as "oe-selftest -j X" which definitely needs python3-
> subunit, or resulttool which does need python3-jinja2 for some options
> iirc.
>
> As such, which this change is right for some workflows, it isn't
> correct for others.


Right, I just tested for the use case advertised in the manual, that is 
running "bitbake core-image-minimal" and "runqemu qemux86-64". The 
latter command didn't need extra SDL host packages.

>
> Ideally there would be two levels of dependencies, those needed for the
> basics and those needed to cover all the needs.
>
> Looking at the original bug, I think we did remove some of the
> SDL/xterm/vnc requirements. Even then some vnc pieces are still needed
> for some autobuilder setups though.
>
> We likely should define which workflows we expect to work with this
> "essential" dependency list?


Maybe we could document this elsewhere (in a section for developers / 
testers ?), to keep the list short for regular users.

The thing is it's easier to add new requirements than to remove the ones 
which are no longer needed.

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] poky.yaml.in: simplify host package requirements

[Richard Purdie]

On Wed, 2023-05-03 at 11:37 +0200, Michael Opdenacker wrote:
> Hi Richard,
> 
> Thank you for reviewing this patch!
> 
> On 03.05.23 at 11:13, Richard Purdie wrote:
> > On Wed, 2023-05-03 at 11:01 +0200, Michael Opdenacker via
> > lists.yoctoproject.org wrote:
> > > From: Michael Opdenacker <michael.opdenacker@bootlin.com>
> > > 
> > > Some of the listed host packages were no longer necessary.
> > > 
> > > [YOCTO #14873]
> > > 
> > > Signed-off-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
> > > ---
> > >   documentation/poky.yaml.in | 28 ++++++++--------------------
> > >   1 file changed, 8 insertions(+), 20 deletions(-)
> > I think this is another "it depends" situation.
> > 
> > The cut down lists you have below probably works fine if you just want
> > to build and image and hopefully let you run it under runqemu.
> > 
> > Where things will start to get problematic is some of the testing
> > workflows such as "oe-selftest -j X" which definitely needs python3-
> > subunit, or resulttool which does need python3-jinja2 for some options
> > iirc.
> > 
> > As such, which this change is right for some workflows, it isn't
> > correct for others.
> 
> 
> Right, I just tested for the use case advertised in the manual, that is 
> running "bitbake core-image-minimal" and "runqemu qemux86-64". The 
> latter command didn't need extra SDL host packages.
> 
> > 
> > Ideally there would be two levels of dependencies, those needed for the
> > basics and those needed to cover all the needs.
> > 
> > Looking at the original bug, I think we did remove some of the
> > SDL/xterm/vnc requirements. Even then some vnc pieces are still needed
> > for some autobuilder setups though.
> > 
> > We likely should define which workflows we expect to work with this
> > "essential" dependency list?
> 
> 
> Maybe we could document this elsewhere (in a section for developers / 
> testers ?), to keep the list short for regular users.

I agree, I think we do likely need to have two different lists with
pointers to move the user to the appropriate one. Would you be able to
propose a change?

> The thing is it's easier to add new requirements than to remove the ones 
> which are no longer needed.

Definitely, I understand the challenges here. I just don't want to lose
the completeness of the information either.

Cheers,

Richard

Re: [docs] [PATCH] poky.yaml.in: simplify host package requirements

[Michael Opdenacker]

On 03.05.23 at 12:44, Richard Purdie wrote:
>
>>> Ideally there would be two levels of dependencies, those needed for the
>>> basics and those needed to cover all the needs.
>>>
>>> Looking at the original bug, I think we did remove some of the
>>> SDL/xterm/vnc requirements. Even then some vnc pieces are still needed
>>> for some autobuilder setups though.
>>>
>>> We likely should define which workflows we expect to work with this
>>> "essential" dependency list?
>>
>> Maybe we could document this elsewhere (in a section for developers /
>> testers ?), to keep the list short for regular users.
> I agree, I think we do likely need to have two different lists with
> pointers to move the user to the appropriate one. Would you be able to
> propose a change?


Sure, but I'd need to know what commands I should test to figure out 
which extra packages are needed.

I'm asking because I suspect that some of the host packages we're 
listing are no longer necessary anyway, either because of distro 
changes, or because of our own changes. It's also good to document what 
the host packages are necessary for.

So, would you (or others) have such commands to run? I'd install the 
"minimum" requirements first for each distro, and then run these 
commands until I've found all the extra host packages they actually need.

Thanks in advance
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH] poky.yaml.in: simplify host package requirements

[Michael Opdenacker]

Hello,

On 03.05.23 at 16:46, Michael Opdenacker via lists.yoctoproject.org wrote:
>
> On 03.05.23 at 12:44, Richard Purdie wrote:
>>
>>>> Ideally there would be two levels of dependencies, those needed for 
>>>> the
>>>> basics and those needed to cover all the needs.
>>>>
>>>> Looking at the original bug, I think we did remove some of the
>>>> SDL/xterm/vnc requirements. Even then some vnc pieces are still needed
>>>> for some autobuilder setups though.
>>>>
>>>> We likely should define which workflows we expect to work with this
>>>> "essential" dependency list?
>>>
>>> Maybe we could document this elsewhere (in a section for developers /
>>> testers ?), to keep the list short for regular users.
>> I agree, I think we do likely need to have two different lists with
>> pointers to move the user to the appropriate one. Would you be able to
>> propose a change?
>
>
> Sure, but I'd need to know what commands I should test to figure out 
> which extra packages are needed.
>
> I'm asking because I suspect that some of the host packages we're 
> listing are no longer necessary anyway, either because of distro 
> changes, or because of our own changes. It's also good to document 
> what the host packages are necessary for.
>
> So, would you (or others) have such commands to run? I'd install the 
> "minimum" requirements first for each distro, and then run these 
> commands until I've found all the extra host packages they actually need.


So, any clue about the commands I should run to find the extra host 
packages that are needed to generate more than images, such as running 
Autobuilder and some testing jobs?

Anyway, if we document such extra host packages, we'll need to explain 
what they are needed for.
Thanks in advance
Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com