About multilib packages rpm pkg name

About multilib packages rpm pkg name

[Robert Yang]

Hello,

For the multilib package like lib32-bash, its rpm package name
is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is
lib32-bash.ipk and lib32-bash.deb, there is a side effect for
the naming of rpm, for example, if packagegroup-xx rdepends on bash,
the lib32-packagegroup-xx should rdepend on lib32-bash, but
the its rpm pkg knows nothing about lib32-bash, so it still rdepends
on bash, this causes 32bit toolchain libs not installed when export
multilib sdk, for example:

Configured multilib:
MACHINE ?= "qemux86-64"
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"

$ bitbake core-image-minimal -cpopulate_sdk

When PACKAGE_CLASSES = "package_rpm",  the lib32-libgcc,
lib32-libgcc-dev or other lib32-xx packages are not installed,
so that the 32bit toolchain doesn't work in SDK, this is because
lib32-packagegroup-core-standalone-sdk-target can't pull in the
lib32-xx correctly.

When PACKAGE_CLASSES = "package_ipk", it works well.

Does anyone why we don't use the name like lib32-bash.rpm, please ?
Can we use lib32-bash, please ?

-- 
Thanks

Robert

Re: About multilib packages rpm pkg name

[Mark Hatle]

On 8/5/15 1:45 AM, Robert Yang wrote:
> 
> Hello,
> 
> For the multilib package like lib32-bash, its rpm package name
> is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is
> lib32-bash.ipk and lib32-bash.deb, there is a side effect for
> the naming of rpm, for example, if packagegroup-xx rdepends on bash,
> the lib32-packagegroup-xx should rdepend on lib32-bash, but
> the its rpm pkg knows nothing about lib32-bash, so it still rdepends
> on bash, this causes 32bit toolchain libs not installed when export
> multilib sdk, for example:
> 
> Configured multilib:
> MACHINE ?= "qemux86-64"
> require conf/multilib.conf
> MULTILIBS = "multilib:lib32"
> DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
> 
> $ bitbake core-image-minimal -cpopulate_sdk
> 
> When PACKAGE_CLASSES = "package_rpm",  the lib32-libgcc,
> lib32-libgcc-dev or other lib32-xx packages are not installed,
> so that the 32bit toolchain doesn't work in SDK, this is because
> lib32-packagegroup-core-standalone-sdk-target can't pull in the
> lib32-xx correctly.
> 
> When PACKAGE_CLASSES = "package_ipk", it works well.
> 
> Does anyone why we don't use the name like lib32-bash.rpm, please ?
> Can we use lib32-bash, please ?

Because for something like bash, there is absolutely no reason to choose a
64-bit or 32-bit version over the other.  They have equivalent functionality for
the end user, and only one version can be installed at a time.

This is the RPM behavior that is being copied from Red Hat and other places, and
what our customers are expecting.

RPM supports multiple packages, with different arch fields, being installed at
the same time.  ipkg and deb have a limitation that the package name MUST be
unique.  So this is the first reason why we have different style names.

The idea here being that you can build the .i586 packages once, and re-use them
on an i586 system and an x86-64 system.  (Note, this doesn't work in practice in
the Yocto Project due to sstate issues... but that is the underlying design from
a package management perspective.)

Dependency processing is the second reason, see below for an explanation of the
difference and how it affects us.

There are different types of dependencies within the system.

* Package level dependency
This is of the format 'bash' or 'lib32-bash'.  The system requires a specific
package.  All three package formats support this.

* Virtual dependency
This is similar, in that the format can look like a package, or be something
like 'virtual/foo'.  Again all three package formats support this.

* Library SONAME
This is a correlation of the library SONAME into the dependency set, such as
'libz.so(64)'.  This is pulled directly from any ELF binaries in the system
during processing and added as a direct dependency to other shared libraries.
It is significantly more exact then the package or virtual dependencies above.
HOWEVER, it only works in the RPM case.

This is the second reason why deb and ipkg need a unique 'lib32' style name for
things.  They don't have a mechanism for a package to say "I need the 64-bit
version of libz.so" to run properly, while that is built into RPM itself.

* Filename dependency
These are dependencies that start with '/'.  This is unique to RPM as well, and
allows RPM to express dependencies on '/bin/bash', vs deb/ipkg having
dependencies on 'bash'.  The difference is that it puts a hard marker in place
that if 'bash' doesn't provide '/bin/bash', it's not enough.

This doesn't directly affect the naming, but occasionally has some indirect
adjustments on things.


So when building a system using RPM as the package type, the rootfs construction
may be slightly different because the package manager rules are different.

RPM (smart) will look at the overall dependency tree and attempt to pick the
shortest path so that all of the dependencies are correct.  You can 'weight' the
dependency resolver to favor one style of package over the other.  This is a
place in smart where we likely need to change the behavior to make it a bit more
friendly for the Yocto Project.

The dependency generator is already configured to weigh a package of the same
type (arch) as a bit higher then others.  The problem is that a 'noarch' (all)
package is of a different type then say i586.  So when an all package says "give
me bash".  There is no 'bash.all', so it looks at the list and finds 'bash' of
the highest priority, IF it had not previously been selected by something else.
 (How would it have previously been selected?  If the image had selected
'lib32-bash' explicitly, or the more likely case, a lib32-* package that
requires bash had been selected by the image, and there were not explicit
requirements for 'bash' [main lib version]).


RPM itself has a setting that defines which ABI type becomes the default if you
attempt to choose multiple packages that have the same executables listed.  The
setting here is:

# Set RPM_PREFER_ELF_ARCH to configure preferred ABI when using rpm packaging
# backend to generate a rootfs, choices are:
# 1: ELF32 wins
# 2: ELF64 wins
# 4: ELF64 N32 wins (for mips64 or mips64el only)

Note, this doesn't prevent multiple packages of the same name, different arch
from being installed -- what this does is selects which one wins.  When not set,
I believe it's last-in wins.


This setting should also help weigh the smart dep generator, but I'm not sure it
does.  (Pretty easy to test though.. set the value, build a multilib rootfs, and
select something (all package) that requires 'bash' and see which version is
selected.)

But for most things in the system this really doesn't matter at all.  For things
where a 32-bit or 64-bit does matter (executable and/or libraries that are not
implicitly linked elsewhere), we really do need to add dependencies into the
system so that the correct set of items are selected.

PAM, Perl, Python, and the on-target toolchains are things that adding a few
runtime dependencies are needed.  Basically anything in the system that may have
runtime loadable libraries of a specific ABI, using something like dlopen.
(Remember SONAME dependencies are automatic as mentioned above.)

These new runtime dependencies will not harm deb/ipkg resolution (and may help
it), but they give RPM the additional information it needs to select the
explicit set that is required.  For example PAM:

RPROVIDES_${PN} += "${PN}-${libpam_suffix}"
RPROVIDES_${PN}-runtime += "${PN}-runtime-${libpam_suffix}"

RDEPENDS_${PN}-runtime = "${PN}-${libpam_suffix} \
    ${MLPREFIX}pam-plugin-deny-${libpam_suffix} \
    ${MLPREFIX}pam-plugin-permit-${libpam_suffix} \
    ${MLPREFIX}pam-plugin-warn-${libpam_suffix} \
    ${MLPREFIX}pam-plugin-unix-${libpam_suffix} \
    "

(other code later constructs the RPROVIDES for the plugins as well)

The MLPREFIX will be filtered off when RPM constructs it's packages, as it
doesn't want/need them.  However the suffix won't be filtered.

libpam_suffix = "suffix${@get_multilib_bit(d)}"

This way the -runtime package ensures the right other packages (with their
specific PROVIDES sufffixes) are included in all cases.

This is a unique case, but should be pretty easy to help fix.

For the case where you want to specify 'lib32-packagegroup' and expect to ONLY
get 32-bit packages... While we can't ensure it with the RPM mechanism, we can
weigh things to more likely happens -if- the lib32-packagegroup is of the same
.arch as the packages it's installing, instead of being 'all' packages.

Hopefully this explains a bit more how all of these items fit together, and the
choice made when RPM was implemented to follow more of an expected multilib
workflow.  (This won't affect you if you don't use multilibs in your design!)

--Mark

Re: About multilib packages rpm pkg name

[Robert Yang]

Hi Mark,

Thank you very much, this makes things more clear now, I've tried
to change packagegroup-core-standalone-sdk-target's PACKAGE_ARCH
to MACHINE_ARCH or TUNE_PKGARCH, it didn't work (the rdepends lib32
are still not installed), I will try to use read_subpkgdata() to get
the required pkgs and fix the problem.

// Robert

On 08/05/2015 09:10 PM, Mark Hatle wrote:
> On 8/5/15 1:45 AM, Robert Yang wrote:
>>
>> Hello,
>>
>> For the multilib package like lib32-bash, its rpm package name
>> is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is
>> lib32-bash.ipk and lib32-bash.deb, there is a side effect for
>> the naming of rpm, for example, if packagegroup-xx rdepends on bash,
>> the lib32-packagegroup-xx should rdepend on lib32-bash, but
>> the its rpm pkg knows nothing about lib32-bash, so it still rdepends
>> on bash, this causes 32bit toolchain libs not installed when export
>> multilib sdk, for example:
>>
>> Configured multilib:
>> MACHINE ?= "qemux86-64"
>> require conf/multilib.conf
>> MULTILIBS = "multilib:lib32"
>> DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
>>
>> $ bitbake core-image-minimal -cpopulate_sdk
>>
>> When PACKAGE_CLASSES = "package_rpm",  the lib32-libgcc,
>> lib32-libgcc-dev or other lib32-xx packages are not installed,
>> so that the 32bit toolchain doesn't work in SDK, this is because
>> lib32-packagegroup-core-standalone-sdk-target can't pull in the
>> lib32-xx correctly.
>>
>> When PACKAGE_CLASSES = "package_ipk", it works well.
>>
>> Does anyone why we don't use the name like lib32-bash.rpm, please ?
>> Can we use lib32-bash, please ?
>
> Because for something like bash, there is absolutely no reason to choose a
> 64-bit or 32-bit version over the other.  They have equivalent functionality for
> the end user, and only one version can be installed at a time.
>
> This is the RPM behavior that is being copied from Red Hat and other places, and
> what our customers are expecting.
>
> RPM supports multiple packages, with different arch fields, being installed at
> the same time.  ipkg and deb have a limitation that the package name MUST be
> unique.  So this is the first reason why we have different style names.
>
> The idea here being that you can build the .i586 packages once, and re-use them
> on an i586 system and an x86-64 system.  (Note, this doesn't work in practice in
> the Yocto Project due to sstate issues... but that is the underlying design from
> a package management perspective.)
>
> Dependency processing is the second reason, see below for an explanation of the
> difference and how it affects us.
>
> There are different types of dependencies within the system.
>
> * Package level dependency
> This is of the format 'bash' or 'lib32-bash'.  The system requires a specific
> package.  All three package formats support this.
>
> * Virtual dependency
> This is similar, in that the format can look like a package, or be something
> like 'virtual/foo'.  Again all three package formats support this.
>
> * Library SONAME
> This is a correlation of the library SONAME into the dependency set, such as
> 'libz.so(64)'.  This is pulled directly from any ELF binaries in the system
> during processing and added as a direct dependency to other shared libraries.
> It is significantly more exact then the package or virtual dependencies above.
> HOWEVER, it only works in the RPM case.
>
> This is the second reason why deb and ipkg need a unique 'lib32' style name for
> things.  They don't have a mechanism for a package to say "I need the 64-bit
> version of libz.so" to run properly, while that is built into RPM itself.
>
> * Filename dependency
> These are dependencies that start with '/'.  This is unique to RPM as well, and
> allows RPM to express dependencies on '/bin/bash', vs deb/ipkg having
> dependencies on 'bash'.  The difference is that it puts a hard marker in place
> that if 'bash' doesn't provide '/bin/bash', it's not enough.
>
> This doesn't directly affect the naming, but occasionally has some indirect
> adjustments on things.
>
>
> So when building a system using RPM as the package type, the rootfs construction
> may be slightly different because the package manager rules are different.
>
> RPM (smart) will look at the overall dependency tree and attempt to pick the
> shortest path so that all of the dependencies are correct.  You can 'weight' the
> dependency resolver to favor one style of package over the other.  This is a
> place in smart where we likely need to change the behavior to make it a bit more
> friendly for the Yocto Project.
>
> The dependency generator is already configured to weigh a package of the same
> type (arch) as a bit higher then others.  The problem is that a 'noarch' (all)
> package is of a different type then say i586.  So when an all package says "give
> me bash".  There is no 'bash.all', so it looks at the list and finds 'bash' of
> the highest priority, IF it had not previously been selected by something else.
>   (How would it have previously been selected?  If the image had selected
> 'lib32-bash' explicitly, or the more likely case, a lib32-* package that
> requires bash had been selected by the image, and there were not explicit
> requirements for 'bash' [main lib version]).
>
>
> RPM itself has a setting that defines which ABI type becomes the default if you
> attempt to choose multiple packages that have the same executables listed.  The
> setting here is:
>
> # Set RPM_PREFER_ELF_ARCH to configure preferred ABI when using rpm packaging
> # backend to generate a rootfs, choices are:
> # 1: ELF32 wins
> # 2: ELF64 wins
> # 4: ELF64 N32 wins (for mips64 or mips64el only)
>
> Note, this doesn't prevent multiple packages of the same name, different arch
> from being installed -- what this does is selects which one wins.  When not set,
> I believe it's last-in wins.
>
>
> This setting should also help weigh the smart dep generator, but I'm not sure it
> does.  (Pretty easy to test though.. set the value, build a multilib rootfs, and
> select something (all package) that requires 'bash' and see which version is
> selected.)
>
> But for most things in the system this really doesn't matter at all.  For things
> where a 32-bit or 64-bit does matter (executable and/or libraries that are not
> implicitly linked elsewhere), we really do need to add dependencies into the
> system so that the correct set of items are selected.
>
> PAM, Perl, Python, and the on-target toolchains are things that adding a few
> runtime dependencies are needed.  Basically anything in the system that may have
> runtime loadable libraries of a specific ABI, using something like dlopen.
> (Remember SONAME dependencies are automatic as mentioned above.)
>
> These new runtime dependencies will not harm deb/ipkg resolution (and may help
> it), but they give RPM the additional information it needs to select the
> explicit set that is required.  For example PAM:
>
> RPROVIDES_${PN} += "${PN}-${libpam_suffix}"
> RPROVIDES_${PN}-runtime += "${PN}-runtime-${libpam_suffix}"
>
> RDEPENDS_${PN}-runtime = "${PN}-${libpam_suffix} \
>      ${MLPREFIX}pam-plugin-deny-${libpam_suffix} \
>      ${MLPREFIX}pam-plugin-permit-${libpam_suffix} \
>      ${MLPREFIX}pam-plugin-warn-${libpam_suffix} \
>      ${MLPREFIX}pam-plugin-unix-${libpam_suffix} \
>      "
>
> (other code later constructs the RPROVIDES for the plugins as well)
>
> The MLPREFIX will be filtered off when RPM constructs it's packages, as it
> doesn't want/need them.  However the suffix won't be filtered.
>
> libpam_suffix = "suffix${@get_multilib_bit(d)}"
>
> This way the -runtime package ensures the right other packages (with their
> specific PROVIDES sufffixes) are included in all cases.
>
> This is a unique case, but should be pretty easy to help fix.
>
> For the case where you want to specify 'lib32-packagegroup' and expect to ONLY
> get 32-bit packages... While we can't ensure it with the RPM mechanism, we can
> weigh things to more likely happens -if- the lib32-packagegroup is of the same
> .arch as the packages it's installing, instead of being 'all' packages.
>
> Hopefully this explains a bit more how all of these items fit together, and the
> choice made when RPM was implemented to follow more of an expected multilib
> workflow.  (This won't affect you if you don't use multilibs in your design!)
>
> --Mark
>
>

Re: About multilib packages rpm pkg name

[Mark Hatle]

On 8/5/15 9:33 PM, Robert Yang wrote:
> 
> Hi Mark,
> 
> Thank you very much, this makes things more clear now, I've tried
> to change packagegroup-core-standalone-sdk-target's PACKAGE_ARCH
> to MACHINE_ARCH or TUNE_PKGARCH, it didn't work (the rdepends lib32
> are still not installed), I will try to use read_subpkgdata() to get
> the required pkgs and fix the problem.

During package creation we strip off the multilib prefix (since it's not
needed).  Then during dependency calculations things often favor a 'shortest
path' to the dependency which may not give you the 32-bit version you are expecting.

What I'd like to see is a patch to smart where you can say "Use this ARCH or
word size, unless explicitly selected".  That would help eliminate -some- of
these issues.

The other side of the issue then is for things that do have explicit word size
requirements (like gcc and some of the build tools) we need to fix the packages
to have additions RDEPENDS and RPROVIDES that dictate the word size they need
and the word size they provide.  libpam already does this, and it resolved the
issues there.

--Mark

> // Robert
> 
> On 08/05/2015 09:10 PM, Mark Hatle wrote:
>> On 8/5/15 1:45 AM, Robert Yang wrote:
>>>
>>> Hello,
>>>
>>> For the multilib package like lib32-bash, its rpm package name
>>> is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is
>>> lib32-bash.ipk and lib32-bash.deb, there is a side effect for
>>> the naming of rpm, for example, if packagegroup-xx rdepends on bash,
>>> the lib32-packagegroup-xx should rdepend on lib32-bash, but
>>> the its rpm pkg knows nothing about lib32-bash, so it still rdepends
>>> on bash, this causes 32bit toolchain libs not installed when export
>>> multilib sdk, for example:
>>>
>>> Configured multilib:
>>> MACHINE ?= "qemux86-64"
>>> require conf/multilib.conf
>>> MULTILIBS = "multilib:lib32"
>>> DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
>>>
>>> $ bitbake core-image-minimal -cpopulate_sdk
>>>
>>> When PACKAGE_CLASSES = "package_rpm",  the lib32-libgcc,
>>> lib32-libgcc-dev or other lib32-xx packages are not installed,
>>> so that the 32bit toolchain doesn't work in SDK, this is because
>>> lib32-packagegroup-core-standalone-sdk-target can't pull in the
>>> lib32-xx correctly.
>>>
>>> When PACKAGE_CLASSES = "package_ipk", it works well.
>>>
>>> Does anyone why we don't use the name like lib32-bash.rpm, please ?
>>> Can we use lib32-bash, please ?
>>
>> Because for something like bash, there is absolutely no reason to choose a
>> 64-bit or 32-bit version over the other.  They have equivalent functionality for
>> the end user, and only one version can be installed at a time.
>>
>> This is the RPM behavior that is being copied from Red Hat and other places, and
>> what our customers are expecting.
>>
>> RPM supports multiple packages, with different arch fields, being installed at
>> the same time.  ipkg and deb have a limitation that the package name MUST be
>> unique.  So this is the first reason why we have different style names.
>>
>> The idea here being that you can build the .i586 packages once, and re-use them
>> on an i586 system and an x86-64 system.  (Note, this doesn't work in practice in
>> the Yocto Project due to sstate issues... but that is the underlying design from
>> a package management perspective.)
>>
>> Dependency processing is the second reason, see below for an explanation of the
>> difference and how it affects us.
>>
>> There are different types of dependencies within the system.
>>
>> * Package level dependency
>> This is of the format 'bash' or 'lib32-bash'.  The system requires a specific
>> package.  All three package formats support this.
>>
>> * Virtual dependency
>> This is similar, in that the format can look like a package, or be something
>> like 'virtual/foo'.  Again all three package formats support this.
>>
>> * Library SONAME
>> This is a correlation of the library SONAME into the dependency set, such as
>> 'libz.so(64)'.  This is pulled directly from any ELF binaries in the system
>> during processing and added as a direct dependency to other shared libraries.
>> It is significantly more exact then the package or virtual dependencies above.
>> HOWEVER, it only works in the RPM case.
>>
>> This is the second reason why deb and ipkg need a unique 'lib32' style name for
>> things.  They don't have a mechanism for a package to say "I need the 64-bit
>> version of libz.so" to run properly, while that is built into RPM itself.
>>
>> * Filename dependency
>> These are dependencies that start with '/'.  This is unique to RPM as well, and
>> allows RPM to express dependencies on '/bin/bash', vs deb/ipkg having
>> dependencies on 'bash'.  The difference is that it puts a hard marker in place
>> that if 'bash' doesn't provide '/bin/bash', it's not enough.
>>
>> This doesn't directly affect the naming, but occasionally has some indirect
>> adjustments on things.
>>
>>
>> So when building a system using RPM as the package type, the rootfs construction
>> may be slightly different because the package manager rules are different.
>>
>> RPM (smart) will look at the overall dependency tree and attempt to pick the
>> shortest path so that all of the dependencies are correct.  You can 'weight' the
>> dependency resolver to favor one style of package over the other.  This is a
>> place in smart where we likely need to change the behavior to make it a bit more
>> friendly for the Yocto Project.
>>
>> The dependency generator is already configured to weigh a package of the same
>> type (arch) as a bit higher then others.  The problem is that a 'noarch' (all)
>> package is of a different type then say i586.  So when an all package says "give
>> me bash".  There is no 'bash.all', so it looks at the list and finds 'bash' of
>> the highest priority, IF it had not previously been selected by something else.
>>   (How would it have previously been selected?  If the image had selected
>> 'lib32-bash' explicitly, or the more likely case, a lib32-* package that
>> requires bash had been selected by the image, and there were not explicit
>> requirements for 'bash' [main lib version]).
>>
>>
>> RPM itself has a setting that defines which ABI type becomes the default if you
>> attempt to choose multiple packages that have the same executables listed.  The
>> setting here is:
>>
>> # Set RPM_PREFER_ELF_ARCH to configure preferred ABI when using rpm packaging
>> # backend to generate a rootfs, choices are:
>> # 1: ELF32 wins
>> # 2: ELF64 wins
>> # 4: ELF64 N32 wins (for mips64 or mips64el only)
>>
>> Note, this doesn't prevent multiple packages of the same name, different arch
>> from being installed -- what this does is selects which one wins.  When not set,
>> I believe it's last-in wins.
>>
>>
>> This setting should also help weigh the smart dep generator, but I'm not sure it
>> does.  (Pretty easy to test though.. set the value, build a multilib rootfs, and
>> select something (all package) that requires 'bash' and see which version is
>> selected.)
>>
>> But for most things in the system this really doesn't matter at all.  For things
>> where a 32-bit or 64-bit does matter (executable and/or libraries that are not
>> implicitly linked elsewhere), we really do need to add dependencies into the
>> system so that the correct set of items are selected.
>>
>> PAM, Perl, Python, and the on-target toolchains are things that adding a few
>> runtime dependencies are needed.  Basically anything in the system that may have
>> runtime loadable libraries of a specific ABI, using something like dlopen.
>> (Remember SONAME dependencies are automatic as mentioned above.)
>>
>> These new runtime dependencies will not harm deb/ipkg resolution (and may help
>> it), but they give RPM the additional information it needs to select the
>> explicit set that is required.  For example PAM:
>>
>> RPROVIDES_${PN} += "${PN}-${libpam_suffix}"
>> RPROVIDES_${PN}-runtime += "${PN}-runtime-${libpam_suffix}"
>>
>> RDEPENDS_${PN}-runtime = "${PN}-${libpam_suffix} \
>>      ${MLPREFIX}pam-plugin-deny-${libpam_suffix} \
>>      ${MLPREFIX}pam-plugin-permit-${libpam_suffix} \
>>      ${MLPREFIX}pam-plugin-warn-${libpam_suffix} \
>>      ${MLPREFIX}pam-plugin-unix-${libpam_suffix} \
>>      "
>>
>> (other code later constructs the RPROVIDES for the plugins as well)
>>
>> The MLPREFIX will be filtered off when RPM constructs it's packages, as it
>> doesn't want/need them.  However the suffix won't be filtered.
>>
>> libpam_suffix = "suffix${@get_multilib_bit(d)}"
>>
>> This way the -runtime package ensures the right other packages (with their
>> specific PROVIDES sufffixes) are included in all cases.
>>
>> This is a unique case, but should be pretty easy to help fix.
>>
>> For the case where you want to specify 'lib32-packagegroup' and expect to ONLY
>> get 32-bit packages... While we can't ensure it with the RPM mechanism, we can
>> weigh things to more likely happens -if- the lib32-packagegroup is of the same
>> .arch as the packages it's installing, instead of being 'all' packages.
>>
>> Hopefully this explains a bit more how all of these items fit together, and the
>> choice made when RPM was implemented to follow more of an expected multilib
>> workflow.  (This won't affect you if you don't use multilibs in your design!)
>>
>> --Mark
>>
>>

Re: About multilib packages rpm pkg name

[Robert Yang]

On 08/07/2015 04:57 AM, Mark Hatle wrote:
> On 8/5/15 9:33 PM, Robert Yang wrote:
>>
>> Hi Mark,
>>
>> Thank you very much, this makes things more clear now, I've tried
>> to change packagegroup-core-standalone-sdk-target's PACKAGE_ARCH
>> to MACHINE_ARCH or TUNE_PKGARCH, it didn't work (the rdepends lib32
>> are still not installed), I will try to use read_subpkgdata() to get
>> the required pkgs and fix the problem.
>
> During package creation we strip off the multilib prefix (since it's not
> needed).  Then during dependency calculations things often favor a 'shortest
> path' to the dependency which may not give you the 32-bit version you are expecting.
>
> What I'd like to see is a patch to smart where you can say "Use this ARCH or
> word size, unless explicitly selected".  That would help eliminate -some- of
> these issues.

Thanks, sounds good, I will try smart later, at the moment, I made a
simple patch to make the populate_sdk pull in multilib toolchains works,
there are still other issues, I will fix them and send patches.

diff --git a/meta/classes/populate_sdk_base.bbclass 
b/meta/classes/populate_sdk_base.bbclass
index a9e9bd7..c2491ea 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -83,8 +83,40 @@ POPULATE_SDK_POST_HOST_COMMAND_append = " 
write_host_sdk_manifest; "
  fakeroot python do_populate_sdk() {
      from oe.sdk import populate_sdk
      from oe.manifest import create_manifest, Manifest
+    import oe.packagedata

      pn = d.getVar('PN', True)
+    # Help rpm to pull in multilib requires since it may not pull in
+    # them correctly, for example,
+    # lib32-packagegroup-core-standalone-sdk-target requires
+    # lib32-libc6, rpm may pull in libc6 rather than lib32-libc6, but
+    # directly use
+    # TOOLCHAIN_TARGET_TASK_append = " lib32-libc6" works well.
+    for pkg in (d.getVar('TOOLCHAIN_TARGET_TASK', True) or "").split():
+        sub_data = oe.packagedata.read_subpkgdata(pkg, d)
+        sub_rdep = sub_data.get("RDEPENDS_" + pkg)
+        if not sub_rdep:
+            continue
+        done = sub_rdep.split()
+        next = done
+        # Find all the rdepends on the dependency chain
+        while next:
+            new = []
+            for sub_pkg in next:
+                sub_data = oe.packagedata.read_subpkgdata(sub_pkg, d)
+                sub_pkg_rdep = sub_data.get("RDEPENDS_" + sub_pkg)
+                if not sub_pkg_rdep:
+                    continue
+                for p in sub_pkg_rdep.split():
+                    if p in done:
+                        continue
+                    if not p.startswith('(') and \
+                            oe.packagedata.has_subpkgdata(p, d):
+                        # It's a new rdep
+                        done.append(p)
+                        new.append(p)
+            next = new
+        d.appendVar('TOOLCHAIN_TARGET_TASK', ' ' + " ".join(done))

// Robert

>
> The other side of the issue then is for things that do have explicit word size
> requirements (like gcc and some of the build tools) we need to fix the packages
> to have additions RDEPENDS and RPROVIDES that dictate the word size they need
> and the word size they provide.  libpam already does this, and it resolved the
> issues there.


>
> --Mark
>
>> // Robert
>>
>> On 08/05/2015 09:10 PM, Mark Hatle wrote:
>>> On 8/5/15 1:45 AM, Robert Yang wrote:
>>>>
>>>> Hello,
>>>>
>>>> For the multilib package like lib32-bash, its rpm package name
>>>> is bash-4.3.30-r0.lib32_x86.rpm, but for ipk and deb, its name is
>>>> lib32-bash.ipk and lib32-bash.deb, there is a side effect for
>>>> the naming of rpm, for example, if packagegroup-xx rdepends on bash,
>>>> the lib32-packagegroup-xx should rdepend on lib32-bash, but
>>>> the its rpm pkg knows nothing about lib32-bash, so it still rdepends
>>>> on bash, this causes 32bit toolchain libs not installed when export
>>>> multilib sdk, for example:
>>>>
>>>> Configured multilib:
>>>> MACHINE ?= "qemux86-64"
>>>> require conf/multilib.conf
>>>> MULTILIBS = "multilib:lib32"
>>>> DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
>>>>
>>>> $ bitbake core-image-minimal -cpopulate_sdk
>>>>
>>>> When PACKAGE_CLASSES = "package_rpm",  the lib32-libgcc,
>>>> lib32-libgcc-dev or other lib32-xx packages are not installed,
>>>> so that the 32bit toolchain doesn't work in SDK, this is because
>>>> lib32-packagegroup-core-standalone-sdk-target can't pull in the
>>>> lib32-xx correctly.
>>>>
>>>> When PACKAGE_CLASSES = "package_ipk", it works well.
>>>>
>>>> Does anyone why we don't use the name like lib32-bash.rpm, please ?
>>>> Can we use lib32-bash, please ?
>>>
>>> Because for something like bash, there is absolutely no reason to choose a
>>> 64-bit or 32-bit version over the other.  They have equivalent functionality for
>>> the end user, and only one version can be installed at a time.
>>>
>>> This is the RPM behavior that is being copied from Red Hat and other places, and
>>> what our customers are expecting.
>>>
>>> RPM supports multiple packages, with different arch fields, being installed at
>>> the same time.  ipkg and deb have a limitation that the package name MUST be
>>> unique.  So this is the first reason why we have different style names.
>>>
>>> The idea here being that you can build the .i586 packages once, and re-use them
>>> on an i586 system and an x86-64 system.  (Note, this doesn't work in practice in
>>> the Yocto Project due to sstate issues... but that is the underlying design from
>>> a package management perspective.)
>>>
>>> Dependency processing is the second reason, see below for an explanation of the
>>> difference and how it affects us.
>>>
>>> There are different types of dependencies within the system.
>>>
>>> * Package level dependency
>>> This is of the format 'bash' or 'lib32-bash'.  The system requires a specific
>>> package.  All three package formats support this.
>>>
>>> * Virtual dependency
>>> This is similar, in that the format can look like a package, or be something
>>> like 'virtual/foo'.  Again all three package formats support this.
>>>
>>> * Library SONAME
>>> This is a correlation of the library SONAME into the dependency set, such as
>>> 'libz.so(64)'.  This is pulled directly from any ELF binaries in the system
>>> during processing and added as a direct dependency to other shared libraries.
>>> It is significantly more exact then the package or virtual dependencies above.
>>> HOWEVER, it only works in the RPM case.
>>>
>>> This is the second reason why deb and ipkg need a unique 'lib32' style name for
>>> things.  They don't have a mechanism for a package to say "I need the 64-bit
>>> version of libz.so" to run properly, while that is built into RPM itself.
>>>
>>> * Filename dependency
>>> These are dependencies that start with '/'.  This is unique to RPM as well, and
>>> allows RPM to express dependencies on '/bin/bash', vs deb/ipkg having
>>> dependencies on 'bash'.  The difference is that it puts a hard marker in place
>>> that if 'bash' doesn't provide '/bin/bash', it's not enough.
>>>
>>> This doesn't directly affect the naming, but occasionally has some indirect
>>> adjustments on things.
>>>
>>>
>>> So when building a system using RPM as the package type, the rootfs construction
>>> may be slightly different because the package manager rules are different.
>>>
>>> RPM (smart) will look at the overall dependency tree and attempt to pick the
>>> shortest path so that all of the dependencies are correct.  You can 'weight' the
>>> dependency resolver to favor one style of package over the other.  This is a
>>> place in smart where we likely need to change the behavior to make it a bit more
>>> friendly for the Yocto Project.
>>>
>>> The dependency generator is already configured to weigh a package of the same
>>> type (arch) as a bit higher then others.  The problem is that a 'noarch' (all)
>>> package is of a different type then say i586.  So when an all package says "give
>>> me bash".  There is no 'bash.all', so it looks at the list and finds 'bash' of
>>> the highest priority, IF it had not previously been selected by something else.
>>>    (How would it have previously been selected?  If the image had selected
>>> 'lib32-bash' explicitly, or the more likely case, a lib32-* package that
>>> requires bash had been selected by the image, and there were not explicit
>>> requirements for 'bash' [main lib version]).
>>>
>>>
>>> RPM itself has a setting that defines which ABI type becomes the default if you
>>> attempt to choose multiple packages that have the same executables listed.  The
>>> setting here is:
>>>
>>> # Set RPM_PREFER_ELF_ARCH to configure preferred ABI when using rpm packaging
>>> # backend to generate a rootfs, choices are:
>>> # 1: ELF32 wins
>>> # 2: ELF64 wins
>>> # 4: ELF64 N32 wins (for mips64 or mips64el only)
>>>
>>> Note, this doesn't prevent multiple packages of the same name, different arch
>>> from being installed -- what this does is selects which one wins.  When not set,
>>> I believe it's last-in wins.
>>>
>>>
>>> This setting should also help weigh the smart dep generator, but I'm not sure it
>>> does.  (Pretty easy to test though.. set the value, build a multilib rootfs, and
>>> select something (all package) that requires 'bash' and see which version is
>>> selected.)
>>>
>>> But for most things in the system this really doesn't matter at all.  For things
>>> where a 32-bit or 64-bit does matter (executable and/or libraries that are not
>>> implicitly linked elsewhere), we really do need to add dependencies into the
>>> system so that the correct set of items are selected.
>>>
>>> PAM, Perl, Python, and the on-target toolchains are things that adding a few
>>> runtime dependencies are needed.  Basically anything in the system that may have
>>> runtime loadable libraries of a specific ABI, using something like dlopen.
>>> (Remember SONAME dependencies are automatic as mentioned above.)
>>>
>>> These new runtime dependencies will not harm deb/ipkg resolution (and may help
>>> it), but they give RPM the additional information it needs to select the
>>> explicit set that is required.  For example PAM:
>>>
>>> RPROVIDES_${PN} += "${PN}-${libpam_suffix}"
>>> RPROVIDES_${PN}-runtime += "${PN}-runtime-${libpam_suffix}"
>>>
>>> RDEPENDS_${PN}-runtime = "${PN}-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-deny-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-permit-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-warn-${libpam_suffix} \
>>>       ${MLPREFIX}pam-plugin-unix-${libpam_suffix} \
>>>       "
>>>
>>> (other code later constructs the RPROVIDES for the plugins as well)
>>>
>>> The MLPREFIX will be filtered off when RPM constructs it's packages, as it
>>> doesn't want/need them.  However the suffix won't be filtered.
>>>
>>> libpam_suffix = "suffix${@get_multilib_bit(d)}"
>>>
>>> This way the -runtime package ensures the right other packages (with their
>>> specific PROVIDES sufffixes) are included in all cases.
>>>
>>> This is a unique case, but should be pretty easy to help fix.
>>>
>>> For the case where you want to specify 'lib32-packagegroup' and expect to ONLY
>>> get 32-bit packages... While we can't ensure it with the RPM mechanism, we can
>>> weigh things to more likely happens -if- the lib32-packagegroup is of the same
>>> .arch as the packages it's installing, instead of being 'all' packages.
>>>
>>> Hopefully this explains a bit more how all of these items fit together, and the
>>> choice made when RPM was implemented to follow more of an expected multilib
>>> workflow.  (This won't affect you if you don't use multilibs in your design!)
>>>
>>> --Mark
>>>
>>>
>
>
>