[PATCH] package.py: Add Requires.private field in process_pkgconfig

[PATCH] package.py: Add Requires.private field in process_pkgconfig

[Sreejith Ravi]

Currently, the process_pkgconfig() only handles the "Requires" key
field from the .pc file and ignores the "Requires.private" field while
generating the dev dependency chain. This results in a broken dependency
list and requires the installation of recommended packages to resolve
the build dependencies when using the dev IPKs. This increases the
hard disk space usage and download time and installs many unnecessary
packages as part of the recommendations.

This patch ensures that the "Requires.private" field is also considered
when creating the dependency list for the dev IPKs. With this, the dev
IPK will have the proper dependencies listed, eliminating the need to
install recommended packages to resolve the build time dependencies.

Example: usr/lib/pkgconfig/libical.pc
----------
Libs: -L${libdir} -lical -licalss -licalvcal
Libs.private: -lpthread
Requires.private: icu-i18n
----------
Depends field generated for libical-dev
Depends: glib-2.0-dev, libical (= 3.0.7-r0)
------------

When trying to resolve the build time dependency with libical
package using “-dev” ipk generated, it will throw the below error.
-----------
Package icu-i18n was not found in the pkg-config search path.
Perhaps you should add the directory containing `icu-i18n.pc'
to the PKG_CONFIG_PATH environment variable
Package 'icu-i18n', required by 'libical', not found
-----------

This patch will fix the broken dependency list.

-------
libical-dev depends field generated with this patch
Depends: glib-2.0-dev, icu-dev, libical (= 3.0.7-r0)
-------

Other examples of packages generated with broken dev dependency.

libflac-dev : https://packages.debian.org/sid/libflac-dev
Without patch:
Depends: flac (= 1.3.3-r0), libflac, libflac++
with patch:
Depends: flac (= 1.3.3-r0), libflac, libflac++, libogg-dev

libglib2.0-dev : https://packages.debian.org/buster/libglib2.0-dev
without patch:
Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev
with patch:
Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev,
         util-linux-dev, zlib-dev

Signed-off-by: Sreejith Ravi <sreejith.ravi087@gmail.com>
---
 meta/lib/oe/package.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index d1303f32bf..e6b46a0dc5 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1853,7 +1853,7 @@ def process_pkgconfig(pkgfiles, d):
                         if m:
                             hdr = m.group(1)
                             exp = pd.expand(m.group(2))
-                            if hdr == 'Requires':
+                            if hdr == 'Requires' or hdr == 'Requires.private':
                                 pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
                                 continue
                         m = var_re.match(l)
-- 
2.34.1

Re: [OE-core] [PATCH] package.py: Add Requires.private field in process_pkgconfig

[Alexandre Belloni]

I believe this causes the following reproducibility failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4998/steps/13/logs/stdio

https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240627-3tyeozgn/packages/diff-html/



On 26/06/2024 14:05:00+0000, Sreejith Ravi wrote:
> Currently, the process_pkgconfig() only handles the "Requires" key
> field from the .pc file and ignores the "Requires.private" field while
> generating the dev dependency chain. This results in a broken dependency
> list and requires the installation of recommended packages to resolve
> the build dependencies when using the dev IPKs. This increases the
> hard disk space usage and download time and installs many unnecessary
> packages as part of the recommendations.
> 
> This patch ensures that the "Requires.private" field is also considered
> when creating the dependency list for the dev IPKs. With this, the dev
> IPK will have the proper dependencies listed, eliminating the need to
> install recommended packages to resolve the build time dependencies.
> 
> Example: usr/lib/pkgconfig/libical.pc
> ----------
> Libs: -L${libdir} -lical -licalss -licalvcal
> Libs.private: -lpthread
> Requires.private: icu-i18n
> ----------
> Depends field generated for libical-dev
> Depends: glib-2.0-dev, libical (= 3.0.7-r0)
> ------------
> 
> When trying to resolve the build time dependency with libical
> package using “-dev” ipk generated, it will throw the below error.
> -----------
> Package icu-i18n was not found in the pkg-config search path.
> Perhaps you should add the directory containing `icu-i18n.pc'
> to the PKG_CONFIG_PATH environment variable
> Package 'icu-i18n', required by 'libical', not found
> -----------
> 
> This patch will fix the broken dependency list.
> 
> -------
> libical-dev depends field generated with this patch
> Depends: glib-2.0-dev, icu-dev, libical (= 3.0.7-r0)
> -------
> 
> Other examples of packages generated with broken dev dependency.
> 
> libflac-dev : https://packages.debian.org/sid/libflac-dev
> Without patch:
> Depends: flac (= 1.3.3-r0), libflac, libflac++
> with patch:
> Depends: flac (= 1.3.3-r0), libflac, libflac++, libogg-dev
> 
> libglib2.0-dev : https://packages.debian.org/buster/libglib2.0-dev
> without patch:
> Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev
> with patch:
> Depends: libffi-dev, libglib-2.0-0 (= 1:2.62.6-r0), libpcre-dev,
>          util-linux-dev, zlib-dev
> 
> Signed-off-by: Sreejith Ravi <sreejith.ravi087@gmail.com>
> ---
>  meta/lib/oe/package.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index d1303f32bf..e6b46a0dc5 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -1853,7 +1853,7 @@ def process_pkgconfig(pkgfiles, d):
>                          if m:
>                              hdr = m.group(1)
>                              exp = pd.expand(m.group(2))
> -                            if hdr == 'Requires':
> +                            if hdr == 'Requires' or hdr == 'Requires.private':
>                                  pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
>                                  continue
>                          m = var_re.match(l)
> -- 
> 2.34.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201166): https://lists.openembedded.org/g/openembedded-core/message/201166
> Mute This Topic: https://lists.openembedded.org/mt/106889869/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH] package.py: Add Requires.private field in process_pkgconfig

[Richard Purdie]

On Thu, 2024-06-27 at 11:29 +0200, Alexandre Belloni via
lists.openembedded.org wrote:
> I believe this causes the following reproducibility failures:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4998/steps/13/logs/stdio
> 
> https://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20240627-3tyeozgn/packages/diff-html/
> 

This is exposing a problem with the way we're handling python library
functions and their effect or lack of effect on the task hashes.

If you do something like:

bitbake libdrm -S none

then look at 

bitbake-dumpsig tmp/stamps/core2-64-poky-linux/libdrm/2.4.120.do_package.sigdata.*

you'll see:

List of dependencies for variable oe.package.process_pkgconfig is ['PACKAGES', 'PKGDEST', 'SHLIBSDIRS', 'SHLIBSWORKDIR']
List of dependencies for variable package_do_pkgconfig is ['oe.package.process_pkgconfig']
Variable package_do_pkgconfig value is     oe.package.process_pkgconfig(pkgfiles, d)
Variable oe.package.process_pkgconfig value is 

i.e. it knows the function is being used but isn't caching its value.
That was by design but it does mean when you change funcitonality, you
have to manually tweak the tashhash which I'm not sure is a great idea
when put like this.

I'm pondering the best way to improve/fix this.

Cheers,

Richard