Multilib status

Multilib status

[Richard Purdie]

Hi Mark/Dongxaio,

We really need to get the remainder of the multilib bugs wrapped up. I
think there is some confusion about the exact approach to take to fix
some of them so I'm hoping to try and summarise the problems here so we
can work out some resolutions.

Problem A
=========

We can have multiple forms of "machine specific" packages now, one for
each multilib e.g.:

task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")

(lets assume the first uses /lib/, the second /lib64/ and the
thrid /lib32/, an artificial example I realise)

I know one proposal was to map:

task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
since the library directories are different. In this specific case it
might not matter but in general it would. What is also important is that
the different versions imply different dependencies. The lib64 bit
version would pull in and select lib64 bit libs. Its these dependencies
which are a key reason these are not "all" arch packages.

The end result is therefore we likely want:

task-core-x11-base-1.0-r34.qemux86_64.rpm
task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm

and I believe there are some patches Dongxiao has created which do this.

Problem B
=========

If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends on
"connman", how does the system figure out to associate this to mean we
want the "x86_64" architecture packages installed?

As far as I can tell there are two proposals around:

1) Set the architecture in the package provides and depends (a bit ugly)

2) Configure libzypp to understand that qemux86_64 does really map to
x86_64 architecture and imply x86_64 dependencies.

Solving problem A above is the first step towards resolving this either
way but we don't seem to have a resolution on this second part.

For reference, we really do want these task packages to have an
associated architecture so the user can easily build up blocks of the
system using a given ABI.



What I really need now is an idea of which patches you both agree on
that we need to merge (I think we have some for problem A) and a
resolution on problem B along with some patches so we can close this set
of issues out.

If there are further issues can you reply to this email either with the
details or a pointer to the specific additional problems.

Cheers,

Richard

Re: Multilib status

[Xu, Dongxiao]

Hi Richard and Mark,

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Friday, September 16, 2011 5:32 PM
> To: Mark Hatle
> Cc: Xu, Dongxiao; openembedded-core
> Subject: Multilib status
> 
> Hi Mark/Dongxaio,
> 
> We really need to get the remainder of the multilib bugs wrapped up. I think
> there is some confusion about the exact approach to take to fix some of them so
> I'm hoping to try and summarise the problems here so we can work out some
> resolutions.
> 
> Problem A
> =========
> 
> We can have multiple forms of "machine specific" packages now, one for each
> multilib e.g.:
> 
> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> 
> (lets assume the first uses /lib/, the second /lib64/ and the thrid /lib32/, an
> artificial example I realise)
> 
> I know one proposal was to map:
> 
> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct since the
> library directories are different. In this specific case it might not matter but in
> general it would. What is also important is that the different versions imply
> different dependencies. The lib64 bit version would pull in and select lib64 bit
> libs. Its these dependencies which are a key reason these are not "all" arch
> packages.
> 
> The end result is therefore we likely want:
> 
> task-core-x11-base-1.0-r34.qemux86_64.rpm
> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> 
> and I believe there are some patches Dongxiao has created which do this.
> 
> Problem B
> =========
> 
> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends on
> "connman", how does the system figure out to associate this to mean we want
> the "x86_64" architecture packages installed?
> 
> As far as I can tell there are two proposals around:
> 
> 1) Set the architecture in the package provides and depends (a bit ugly)
> 
> 2) Configure libzypp to understand that qemux86_64 does really map to
> x86_64 architecture and imply x86_64 dependencies.

In our current rootfs_rpm logic, we don't use zypper tool to make the file system, what we use is pure rpm plus db.
I am not sure whether the rpm database or sat-solver has such kind of bindings between different repo folders? (like qemux86_64 and x86_64)

Thanks,
Dongxiao

> 
> Solving problem A above is the first step towards resolving this either way but
> we don't seem to have a resolution on this second part.
> 
> For reference, we really do want these task packages to have an associated
> architecture so the user can easily build up blocks of the system using a given
> ABI.
> 
> 
> 
> What I really need now is an idea of which patches you both agree on that we
> need to merge (I think we have some for problem A) and a resolution on
> problem B along with some patches so we can close this set of issues out.
> 
> If there are further issues can you reply to this email either with the details or a
> pointer to the specific additional problems.
> 
> Cheers,
> 
> Richard
> 

Re: Multilib status

[Xu, Dongxiao]

Hi Richard and Mark,

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf Of
> Xu, Dongxiao
> Sent: Friday, September 16, 2011 8:01 PM
> To: Richard Purdie; Mark Hatle
> Cc: openembedded-core
> Subject: Re: [OE-core] Multilib status
> 
> Hi Richard and Mark,
> 
> > -----Original Message-----
> > From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> > Sent: Friday, September 16, 2011 5:32 PM
> > To: Mark Hatle
> > Cc: Xu, Dongxiao; openembedded-core
> > Subject: Multilib status
> >
> > Hi Mark/Dongxaio,
> >
> > We really need to get the remainder of the multilib bugs wrapped up. I
> > think there is some confusion about the exact approach to take to fix
> > some of them so I'm hoping to try and summarise the problems here so
> > we can work out some resolutions.
> >
> > Problem A
> > =========
> >
> > We can have multiple forms of "machine specific" packages now, one for
> > each multilib e.g.:
> >
> > task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> > task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> > task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >
> > (lets assume the first uses /lib/, the second /lib64/ and the thrid
> > /lib32/, an artificial example I realise)
> >
> > I know one proposal was to map:
> >
> > task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> > task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
> > since the library directories are different. In this specific case it
> > might not matter but in general it would. What is also important is
> > that the different versions imply different dependencies. The lib64
> > bit version would pull in and select lib64 bit libs. Its these
> > dependencies which are a key reason these are not "all" arch packages.
> >
> > The end result is therefore we likely want:
> >
> > task-core-x11-base-1.0-r34.qemux86_64.rpm
> > task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> > task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >
> > and I believe there are some patches Dongxiao has created which do this.
> >
> > Problem B
> > =========
> >
> > If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends
> > on "connman", how does the system figure out to associate this to mean
> > we want the "x86_64" architecture packages installed?
> >
> > As far as I can tell there are two proposals around:
> >
> > 1) Set the architecture in the package provides and depends (a bit
> > ugly)
> >
> > 2) Configure libzypp to understand that qemux86_64 does really map to
> > x86_64 architecture and imply x86_64 dependencies.
> 
> In our current rootfs_rpm logic, we don't use zypper tool to make the file
> system, what we use is pure rpm plus db.
> I am not sure whether the rpm database or sat-solver has such kind of bindings
> between different repo folders? (like qemux86_64 and x86_64)

I updated my branch at:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml4

Patch 1/3 is to solve the Problem B by approach 1) that Richard mentioned.
Patch 2/3 is to solve the Problem A which defines a new architecture (lib32_qemux86_64) when PACKAGE_ARCH == MACHINE_ARCH.
Patch 3/3 is to install MULTILIB_IMAGE_INSTALL packages into final image.

Thanks,
Dongxiao

> 
> Thanks,
> Dongxiao
> 
> >
> > Solving problem A above is the first step towards resolving this
> > either way but we don't seem to have a resolution on this second part.
> >
> > For reference, we really do want these task packages to have an
> > associated architecture so the user can easily build up blocks of the
> > system using a given ABI.
> >
> >
> >
> > What I really need now is an idea of which patches you both agree on
> > that we need to merge (I think we have some for problem A) and a
> > resolution on problem B along with some patches so we can close this set of
> issues out.
> >
> > If there are further issues can you reply to this email either with
> > the details or a pointer to the specific additional problems.
> >
> > Cheers,
> >
> > Richard
> >
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

Re: Multilib status

[Mark Hatle]

On 9/16/11 4:32 AM, Richard Purdie wrote:
> Hi Mark/Dongxaio,
> 
> We really need to get the remainder of the multilib bugs wrapped up. I
> think there is some confusion about the exact approach to take to fix
> some of them so I'm hoping to try and summarise the problems here so we
> can work out some resolutions.

Let me explain where I am quickly.  I just spent two days working on reproducing
the issue(s).  So far I've not reproduced the direct failures but a series of
others.  I finally figured out part of the reason I wasn't seeing this.  I had a
typo in my configuration:

DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"

This resulted in two sets of binaries normal and lib32 that were more or less
identical in the RPM case.  We -really- need a sanity check that stupid typos
like that don't cause problems.

---

So now that I finally have a built with that done... see below.

> Problem A
> =========
> 
> We can have multiple forms of "machine specific" packages now, one for
> each multilib e.g.:
> 
> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> 
> (lets assume the first uses /lib/, the second /lib64/ and the
> thrid /lib32/, an artificial example I realise)
> 
> I know one proposal was to map:
> 
> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
> since the library directories are different. In this specific case it
> might not matter but in general it would. What is also important is that
> the different versions imply different dependencies. The lib64 bit
> version would pull in and select lib64 bit libs. Its these dependencies
> which are a key reason these are not "all" arch packages.
> 
> The end result is therefore we likely want:
> 
> task-core-x11-base-1.0-r34.qemux86_64.rpm
> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> 
> and I believe there are some patches Dongxiao has created which do this.

Yes, I'm currently testing his new MACHINE_virtclass-multilib-... patch.  So far
it seems to be working.. assuming this is the approach we go with, we'll want to
add some type of sanity check so that it's not possible to collide between the
different multilib configurations.  (Alternatively we could, for each multilib,
specify a default machine virtclass in the format you list above.

An alternative I was thinking of over night would be to avoid the RPM remapping
on the "machine" packages.. but I'm not sure if that is really a good idea.
However, it would could simplify the configuration aspects.

> Problem B
> =========
> 
> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends on
> "connman", how does the system figure out to associate this to mean we
> want the "x86_64" architecture packages installed?
> 
> As far as I can tell there are two proposals around:
> 
> 1) Set the architecture in the package provides and depends (a bit ugly)
> 
> 2) Configure libzypp to understand that qemux86_64 does really map to
> x86_64 architecture and imply x86_64 dependencies.
> 
> Solving problem A above is the first step towards resolving this either
> way but we don't seem to have a resolution on this second part.

Agreed.

> For reference, we really do want these task packages to have an
> associated architecture so the user can easily build up blocks of the
> system using a given ABI.

This is going to be somewhat of a problem I suspect, simply because that is not
the way RPM is/was designed.  We can adjust the rootfs install time, but this
won't address field upgrade/installs.

RPM (and Red Hat Linux/Fedora systems) appear to be designed that any package
that reasonably meets the run-time dependencies can be used.  There is a concept
of a "best" machine based on the resolver hierarchy, but ELF size may or may not
be a factor in that decision.

Doing this is quite powerful, but it also has a conscious decision set behind
it.  If "bash" is required by a script, it really doesn't matter if it's ELF32,
ELF64 or something else, as long as something provides bash.

> What I really need now is an idea of which patches you both agree on
> that we need to merge (I think we have some for problem A) and a
> resolution on problem B along with some patches so we can close this set
> of issues out.
> 
> If there are further issues can you reply to this email either with the
> details or a pointer to the specific additional problems.

Now that I have my stupid typo out of the way, I expect to have further
information in a few hours as to the regularly dependency resolution failure
Dongxiao was reporting.  (Library dependencies -are- ELF specific, so those have
to work properly!)

--Mark

> Cheers,
> 
> Richard
> 
> 

Re: Multilib status

[Xu, Dongxiao]

> -----Original Message-----
> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> Sent: Friday, September 16, 2011 10:51 PM
> To: Richard Purdie
> Cc: Xu, Dongxiao; openembedded-core
> Subject: Re: Multilib status
> 
> On 9/16/11 4:32 AM, Richard Purdie wrote:
> > Hi Mark/Dongxaio,
> >
> > We really need to get the remainder of the multilib bugs wrapped up. I
> > think there is some confusion about the exact approach to take to fix
> > some of them so I'm hoping to try and summarise the problems here so
> > we can work out some resolutions.
> 
> Let me explain where I am quickly.  I just spent two days working on
> reproducing the issue(s).  So far I've not reproduced the direct failures but a
> series of others.  I finally figured out part of the reason I wasn't seeing this.  I
> had a typo in my configuration:
> 
> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
> 
> This resulted in two sets of binaries normal and lib32 that were more or less
> identical in the RPM case.  We -really- need a sanity check that stupid typos
> like that don't cause problems.
> 
> ---
> 
> So now that I finally have a built with that done... see below.
> 
> > Problem A
> > =========
> >
> > We can have multiple forms of "machine specific" packages now, one for
> > each multilib e.g.:
> >
> > task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> > task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> > task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >
> > (lets assume the first uses /lib/, the second /lib64/ and the thrid
> > /lib32/, an artificial example I realise)
> >
> > I know one proposal was to map:
> >
> > task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> > task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
> > since the library directories are different. In this specific case it
> > might not matter but in general it would. What is also important is
> > that the different versions imply different dependencies. The lib64
> > bit version would pull in and select lib64 bit libs. Its these
> > dependencies which are a key reason these are not "all" arch packages.
> >
> > The end result is therefore we likely want:
> >
> > task-core-x11-base-1.0-r34.qemux86_64.rpm
> > task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> > task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >
> > and I believe there are some patches Dongxiao has created which do this.
> 
> Yes, I'm currently testing his new MACHINE_virtclass-multilib-... patch.  So far
> it seems to be working.. assuming this is the approach we go with, we'll want to
> add some type of sanity check so that it's not possible to collide between the
> different multilib configurations.  (Alternatively we could, for each multilib,
> specify a default machine virtclass in the format you list above.
> 
> An alternative I was thinking of over night would be to avoid the RPM
> remapping on the "machine" packages.. but I'm not sure if that is really a good
> idea.
> However, it would could simplify the configuration aspects.

There are two implementations towards the above result:
1) automatically setting MLPREFIX before MACHINE_ARCH.
2) user manually sets MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.

Which one do you prefer?

> 
> > Problem B
> > =========
> >
> > If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends
> > on "connman", how does the system figure out to associate this to mean
> > we want the "x86_64" architecture packages installed?
> >
> > As far as I can tell there are two proposals around:
> >
> > 1) Set the architecture in the package provides and depends (a bit
> > ugly)
> >
> > 2) Configure libzypp to understand that qemux86_64 does really map to
> > x86_64 architecture and imply x86_64 dependencies.
> >
> > Solving problem A above is the first step towards resolving this
> > either way but we don't seem to have a resolution on this second part.
> 
> Agreed.
> 
> > For reference, we really do want these task packages to have an
> > associated architecture so the user can easily build up blocks of the
> > system using a given ABI.
> 
> This is going to be somewhat of a problem I suspect, simply because that is not
> the way RPM is/was designed.  We can adjust the rootfs install time, but this
> won't address field upgrade/installs.
> 
> RPM (and Red Hat Linux/Fedora systems) appear to be designed that any
> package that reasonably meets the run-time dependencies can be used.
> There is a concept of a "best" machine based on the resolver hierarchy, but ELF
> size may or may not be a factor in that decision.
> 
> Doing this is quite powerful, but it also has a conscious decision set behind it.  If
> "bash" is required by a script, it really doesn't matter if it's ELF32,
> ELF64 or something else, as long as something provides bash.

If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a qemux86-64 image, it means that he wants to install all elements required by task-core-boot by lib32 version. If the dependency is selected *randomly*, users would not have multiple libraries in his system.

Thanks,
Dongxiao
> 
> > What I really need now is an idea of which patches you both agree on
> > that we need to merge (I think we have some for problem A) and a
> > resolution on problem B along with some patches so we can close this
> > set of issues out.
> >
> > If there are further issues can you reply to this email either with
> > the details or a pointer to the specific additional problems.
> 
> Now that I have my stupid typo out of the way, I expect to have further
> information in a few hours as to the regularly dependency resolution failure
> Dongxiao was reporting.  (Library dependencies -are- ELF specific, so those
> have to work properly!)
> 
> --Mark
> 
> > Cheers,
> >
> > Richard
> >
> >

Re: Multilib status

[Mark Hatle]

On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>> -----Original Message-----
>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>> Sent: Friday, September 16, 2011 10:51 PM
>> To: Richard Purdie
>> Cc: Xu, Dongxiao; openembedded-core
>> Subject: Re: Multilib status
>>
>> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>> Hi Mark/Dongxaio,
>>>
>>> We really need to get the remainder of the multilib bugs wrapped up. I
>>> think there is some confusion about the exact approach to take to fix
>>> some of them so I'm hoping to try and summarise the problems here so
>>> we can work out some resolutions.
>>
>> Let me explain where I am quickly.  I just spent two days working on
>> reproducing the issue(s).  So far I've not reproduced the direct failures but a
>> series of others.  I finally figured out part of the reason I wasn't seeing this.  I
>> had a typo in my configuration:
>>
>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>
>> This resulted in two sets of binaries normal and lib32 that were more or less
>> identical in the RPM case.  We -really- need a sanity check that stupid typos
>> like that don't cause problems.
>>
>> ---
>>
>> So now that I finally have a built with that done... see below.
>>
>>> Problem A
>>> =========
>>>
>>> We can have multiple forms of "machine specific" packages now, one for
>>> each multilib e.g.:
>>>
>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>
>>> (lets assume the first uses /lib/, the second /lib64/ and the thrid
>>> /lib32/, an artificial example I realise)
>>>
>>> I know one proposal was to map:
>>>
>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
>>> since the library directories are different. In this specific case it
>>> might not matter but in general it would. What is also important is
>>> that the different versions imply different dependencies. The lib64
>>> bit version would pull in and select lib64 bit libs. Its these
>>> dependencies which are a key reason these are not "all" arch packages.
>>>
>>> The end result is therefore we likely want:
>>>
>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>
>>> and I believe there are some patches Dongxiao has created which do this.
>>
>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-... patch.  So far
>> it seems to be working.. assuming this is the approach we go with, we'll want to
>> add some type of sanity check so that it's not possible to collide between the
>> different multilib configurations.  (Alternatively we could, for each multilib,
>> specify a default machine virtclass in the format you list above.
>>
>> An alternative I was thinking of over night would be to avoid the RPM
>> remapping on the "machine" packages.. but I'm not sure if that is really a good
>> idea.
>> However, it would could simplify the configuration aspects.
> 
> There are two implementations towards the above result:
> 1) automatically setting MLPREFIX before MACHINE_ARCH.
> 2) user manually sets MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
> 
> Which one do you prefer?

Right now I'm thinking the best approach is:

*) Allow the user to set MACHINE_virtclass-multilib-...="...."

*) If the user has NOT set it, fall back and use the setting of lib..-machine as
in 2

This way for people who have specific requirements for external compatibility or
just desire more control they can do it.  But for everyone else it will still
work properly.

FYI, I'm currently working in the rootfs_rpm stuff, the system is currently
ignoring the alternative (multilib) machine packages.  I'm hoping this will be
fairly simple to resolve.

>>
>>> Problem B
>>> =========
>>>
>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends
>>> on "connman", how does the system figure out to associate this to mean
>>> we want the "x86_64" architecture packages installed?
>>>
>>> As far as I can tell there are two proposals around:
>>>
>>> 1) Set the architecture in the package provides and depends (a bit
>>> ugly)
>>>
>>> 2) Configure libzypp to understand that qemux86_64 does really map to
>>> x86_64 architecture and imply x86_64 dependencies.
>>>
>>> Solving problem A above is the first step towards resolving this
>>> either way but we don't seem to have a resolution on this second part.
>>
>> Agreed.
>>
>>> For reference, we really do want these task packages to have an
>>> associated architecture so the user can easily build up blocks of the
>>> system using a given ABI.
>>
>> This is going to be somewhat of a problem I suspect, simply because that is not
>> the way RPM is/was designed.  We can adjust the rootfs install time, but this
>> won't address field upgrade/installs.
>>
>> RPM (and Red Hat Linux/Fedora systems) appear to be designed that any
>> package that reasonably meets the run-time dependencies can be used.
>> There is a concept of a "best" machine based on the resolver hierarchy, but ELF
>> size may or may not be a factor in that decision.
>>
>> Doing this is quite powerful, but it also has a conscious decision set behind it.  If
>> "bash" is required by a script, it really doesn't matter if it's ELF32,
>> ELF64 or something else, as long as something provides bash.
> 
> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a qemux86-64 image, it means that he wants to install all elements required by task-core-boot by lib32 version. If the dependency is selected *randomly*, users would not have multiple libraries in his system.

This is a special use case, yes one that I think we need to resolve, but it's
not the typical case.

The typical case will be someone wants the 64-bit (or 32-bit) version of a
select package for compatibility.  And they only want the dependencies for that
package to be loaded.  Anything satisfying the dependency will do.

My recommendation right now is, lets get that working ASAP.  Once that does,
refocus the efforts on getting the multilib "tasks" to do something reasonable.
(i.e. select a group of packages)

If we try to solve both at once, I don't think we're going to come up with a
reasonable solution at this time.

--Mark

> Thanks,
> Dongxiao
>>
>>> What I really need now is an idea of which patches you both agree on
>>> that we need to merge (I think we have some for problem A) and a
>>> resolution on problem B along with some patches so we can close this
>>> set of issues out.
>>>
>>> If there are further issues can you reply to this email either with
>>> the details or a pointer to the specific additional problems.
>>
>> Now that I have my stupid typo out of the way, I expect to have further
>> information in a few hours as to the regularly dependency resolution failure
>> Dongxiao was reporting.  (Library dependencies -are- ELF specific, so those
>> have to work properly!)
>>
>> --Mark
>>
>>> Cheers,
>>>
>>> Richard
>>>
>>>
> 

Re: Multilib status

[Mark Hatle]

Just an FYI.  I'm working through a bunch of issues.  I found out that the RPM
dependency resolution was completely broken, due to a small bug in the
package_rpm.bbclass..  Packages were not being given the proper provide
information, so RPM was attempting best match -- and failing as noted in the
previous discussion..

Unfortunately this has opened up a small can of worms which I'm trying to deal
with.  The simplistic fix for the bug is:

--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -840,7 +840,7 @@ python do_package_rpm () {
        os.chmod(outdepends, 0755)

        # Poky / RPM Provides
-       outprovides = workdir + "/" + srcname + ".requires"
+       outprovides = workdir + "/" + srcname + ".provides"

        try:
                from __builtin__ import file

Unfortunately this has exposed a large set of problems that I didn't realize we
had....

Will do a more formal pull request as soon as I get things working again.

--Mark

On 9/16/11 10:26 AM, Mark Hatle wrote:
> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>>> >> -----Original Message-----
>>> >> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>>> >> Sent: Friday, September 16, 2011 10:51 PM
>>> >> To: Richard Purdie
>>> >> Cc: Xu, Dongxiao; openembedded-core
>>> >> Subject: Re: Multilib status
>>> >>
>>> >> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>>> >>> Hi Mark/Dongxaio,
>>>> >>>
>>>> >>> We really need to get the remainder of the multilib bugs wrapped up. I
>>>> >>> think there is some confusion about the exact approach to take to fix
>>>> >>> some of them so I'm hoping to try and summarise the problems here so
>>>> >>> we can work out some resolutions.
>>> >>
>>> >> Let me explain where I am quickly.  I just spent two days working on
>>> >> reproducing the issue(s).  So far I've not reproduced the direct failures but a
>>> >> series of others.  I finally figured out part of the reason I wasn't seeing this.  I
>>> >> had a typo in my configuration:
>>> >>
>>> >> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>> >>
>>> >> This resulted in two sets of binaries normal and lib32 that were more or less
>>> >> identical in the RPM case.  We -really- need a sanity check that stupid typos
>>> >> like that don't cause problems.
>>> >>
>>> >> ---
>>> >>
>>> >> So now that I finally have a built with that done... see below.
>>> >>
>>>> >>> Problem A
>>>> >>> =========
>>>> >>>
>>>> >>> We can have multiple forms of "machine specific" packages now, one for
>>>> >>> each multilib e.g.:
>>>> >>>
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>> >>>
>>>> >>> (lets assume the first uses /lib/, the second /lib64/ and the thrid
>>>> >>> /lib32/, an artificial example I realise)
>>>> >>>
>>>> >>> I know one proposal was to map:
>>>> >>>
>>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't correct
>>>> >>> since the library directories are different. In this specific case it
>>>> >>> might not matter but in general it would. What is also important is
>>>> >>> that the different versions imply different dependencies. The lib64
>>>> >>> bit version would pull in and select lib64 bit libs. Its these
>>>> >>> dependencies which are a key reason these are not "all" arch packages.
>>>> >>>
>>>> >>> The end result is therefore we likely want:
>>>> >>>
>>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>> >>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>> >>>
>>>> >>> and I believe there are some patches Dongxiao has created which do this.
>>> >>
>>> >> Yes, I'm currently testing his new MACHINE_virtclass-multilib-... patch.  So far
>>> >> it seems to be working.. assuming this is the approach we go with, we'll want to
>>> >> add some type of sanity check so that it's not possible to collide between the
>>> >> different multilib configurations.  (Alternatively we could, for each multilib,
>>> >> specify a default machine virtclass in the format you list above.
>>> >>
>>> >> An alternative I was thinking of over night would be to avoid the RPM
>>> >> remapping on the "machine" packages.. but I'm not sure if that is really a good
>>> >> idea.
>>> >> However, it would could simplify the configuration aspects.
>> > 
>> > There are two implementations towards the above result:
>> > 1) automatically setting MLPREFIX before MACHINE_ARCH.
>> > 2) user manually sets MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
>> > 
>> > Which one do you prefer?
> Right now I'm thinking the best approach is:
> 
> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> 
> *) If the user has NOT set it, fall back and use the setting of lib..-machine as
> in 2
> 
> This way for people who have specific requirements for external compatibility or
> just desire more control they can do it.  But for everyone else it will still
> work properly.
> 
> FYI, I'm currently working in the rootfs_rpm stuff, the system is currently
> ignoring the alternative (multilib) machine packages.  I'm hoping this will be
> fairly simple to resolve.
> 
>>> >>
>>>> >>> Problem B
>>>> >>> =========
>>>> >>>
>>>> >>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which depends
>>>> >>> on "connman", how does the system figure out to associate this to mean
>>>> >>> we want the "x86_64" architecture packages installed?
>>>> >>>
>>>> >>> As far as I can tell there are two proposals around:
>>>> >>>
>>>> >>> 1) Set the architecture in the package provides and depends (a bit
>>>> >>> ugly)
>>>> >>>
>>>> >>> 2) Configure libzypp to understand that qemux86_64 does really map to
>>>> >>> x86_64 architecture and imply x86_64 dependencies.
>>>> >>>
>>>> >>> Solving problem A above is the first step towards resolving this
>>>> >>> either way but we don't seem to have a resolution on this second part.
>>> >>
>>> >> Agreed.
>>> >>
>>>> >>> For reference, we really do want these task packages to have an
>>>> >>> associated architecture so the user can easily build up blocks of the
>>>> >>> system using a given ABI.
>>> >>
>>> >> This is going to be somewhat of a problem I suspect, simply because that is not
>>> >> the way RPM is/was designed.  We can adjust the rootfs install time, but this
>>> >> won't address field upgrade/installs.
>>> >>
>>> >> RPM (and Red Hat Linux/Fedora systems) appear to be designed that any
>>> >> package that reasonably meets the run-time dependencies can be used.
>>> >> There is a concept of a "best" machine based on the resolver hierarchy, but ELF
>>> >> size may or may not be a factor in that decision.
>>> >>
>>> >> Doing this is quite powerful, but it also has a conscious decision set behind it.  If
>>> >> "bash" is required by a script, it really doesn't matter if it's ELF32,
>>> >> ELF64 or something else, as long as something provides bash.
>> > 
>> > If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a qemux86-64 image, it means that he wants to install all elements required by task-core-boot by lib32 version. If the dependency is selected *randomly*, users would not have multiple libraries in his system.
> This is a special use case, yes one that I think we need to resolve, but it's
> not the typical case.
> 
> The typical case will be someone wants the 64-bit (or 32-bit) version of a
> select package for compatibility.  And they only want the dependencies for that
> package to be loaded.  Anything satisfying the dependency will do.
> 
> My recommendation right now is, lets get that working ASAP.  Once that does,
> refocus the efforts on getting the multilib "tasks" to do something reasonable.
> (i.e. select a group of packages)
> 
> If we try to solve both at once, I don't think we're going to come up with a
> reasonable solution at this time.
> 
> --Mark
> 
>> > Thanks,
>> > Dongxiao
>>> >>
>>>> >>> What I really need now is an idea of which patches you both agree on
>>>> >>> that we need to merge (I think we have some for problem A) and a
>>>> >>> resolution on problem B along with some patches so we can close this
>>>> >>> set of issues out.
>>>> >>>
>>>> >>> If there are further issues can you reply to this email either with
>>>> >>> the details or a pointer to the specific additional problems.
>>> >>
>>> >> Now that I have my stupid typo out of the way, I expect to have further
>>> >> information in a few hours as to the regularly dependency resolution failure
>>> >> Dongxiao was reporting.  (Library dependencies -are- ELF specific, so those
>>> >> have to work properly!)
>>> >>
>>> >> --Mark
>>> >>
>>>> >>> Cheers,
>>>> >>>
>>>> >>> Richard
>>>> >>>
>>>> >>>
>> > 
> 

Re: Multilib status

[Xu, Dongxiao]

Hi Mark,

> -----Original Message-----
> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> Sent: Saturday, September 17, 2011 2:19 AM
> To: Mark Hatle
> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
> Subject: Re: Multilib status
> 
> Just an FYI.  I'm working through a bunch of issues.  I found out that the RPM
> dependency resolution was completely broken, due to a small bug in the
> package_rpm.bbclass..  Packages were not being given the proper provide
> information, so RPM was attempting best match -- and failing as noted in the
> previous discussion..
> 
> Unfortunately this has opened up a small can of worms which I'm trying to deal
> with.  The simplistic fix for the bug is:
> 
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -840,7 +840,7 @@ python do_package_rpm () {
>         os.chmod(outdepends, 0755)
> 
>         # Poky / RPM Provides
> -       outprovides = workdir + "/" + srcname + ".requires"
> +       outprovides = workdir + "/" + srcname + ".provides"
> 
>         try:
>                 from __builtin__ import file
> 
> Unfortunately this has exposed a large set of problems that I didn't realize we
> had....
> 
> Will do a more formal pull request as soon as I get things working again.
> 
> --Mark

I just had a test after cherry-pick some of your changes in mhatle/rpm, here are some problems I found and some investigations:

1. do_rootfs failed.
The error log reports unmet dependency of pkgconfig(libpng).
This is due to the dangling link in libpng package. libpng.pc links to libpng12.pc which has been packaged into libpng12.

2. There is a wrong commit I suppose it should be reverted. When you debugging your code, please revert it.
I will discuss it with Richard tomorrow.
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mhatle/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f

3. After the above issue got fixed, rootfs succeed but we saw the final image is still in a mess. We need some x86_64 rpm to be installed, however what we got is x86 rpm package. Then I tried another approach we ever talked about that, firstly install base image, then install the multilib packages. I added a parameter "--replacepkgs" to rpm command to avoid errors like "package xxx has already installed". With this approach, the final image could boot up with multilib library installed.

Some of my testing code is located under http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml-testing

Thanks,
Dongxiao

	
> 
> On 9/16/11 10:26 AM, Mark Hatle wrote:
> > On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
> >>> >> -----Original Message-----
> >>> >> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> >>> >> Sent: Friday, September 16, 2011 10:51 PM
> >>> >> To: Richard Purdie
> >>> >> Cc: Xu, Dongxiao; openembedded-core
> >>> >> Subject: Re: Multilib status
> >>> >>
> >>> >> On 9/16/11 4:32 AM, Richard Purdie wrote:
> >>>> >>> Hi Mark/Dongxaio,
> >>>> >>>
> >>>> >>> We really need to get the remainder of the multilib bugs
> >>>> >>> wrapped up. I think there is some confusion about the exact
> >>>> >>> approach to take to fix some of them so I'm hoping to try and
> >>>> >>> summarise the problems here so we can work out some resolutions.
> >>> >>
> >>> >> Let me explain where I am quickly.  I just spent two days working
> >>> >> on reproducing the issue(s).  So far I've not reproduced the
> >>> >> direct failures but a series of others.  I finally figured out
> >>> >> part of the reason I wasn't seeing this.  I had a typo in my
> configuration:
> >>> >>
> >>> >> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
> >>> >>
> >>> >> This resulted in two sets of binaries normal and lib32 that were
> >>> >> more or less identical in the RPM case.  We -really- need a
> >>> >> sanity check that stupid typos like that don't cause problems.
> >>> >>
> >>> >> ---
> >>> >>
> >>> >> So now that I finally have a built with that done... see below.
> >>> >>
> >>>> >>> Problem A
> >>>> >>> =========
> >>>> >>>
> >>>> >>> We can have multiple forms of "machine specific" packages now,
> >>>> >>> one for each multilib e.g.:
> >>>> >>>
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >>>> >>>
> >>>> >>> (lets assume the first uses /lib/, the second /lib64/ and the
> >>>> >>> thrid /lib32/, an artificial example I realise)
> >>>> >>>
> >>>> >>> I know one proposal was to map:
> >>>> >>>
> >>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> >>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
> >>>> >>> correct since the library directories are different. In this
> >>>> >>> specific case it might not matter but in general it would. What
> >>>> >>> is also important is that the different versions imply
> >>>> >>> different dependencies. The lib64 bit version would pull in and
> >>>> >>> select lib64 bit libs. Its these dependencies which are a key reason
> these are not "all" arch packages.
> >>>> >>>
> >>>> >>> The end result is therefore we likely want:
> >>>> >>>
> >>>> >>> task-core-x11-base-1.0-r34.qemux86_64.rpm
> >>>> >>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> >>>> >>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >>>> >>>
> >>>> >>> and I believe there are some patches Dongxiao has created which do
> this.
> >>> >>
> >>> >> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
> >>> >> patch.  So far it seems to be working.. assuming this is the
> >>> >> approach we go with, we'll want to add some type of sanity check
> >>> >> so that it's not possible to collide between the different
> >>> >> multilib configurations.  (Alternatively we could, for each multilib,
> specify a default machine virtclass in the format you list above.
> >>> >>
> >>> >> An alternative I was thinking of over night would be to avoid the
> >>> >> RPM remapping on the "machine" packages.. but I'm not sure if
> >>> >> that is really a good idea.
> >>> >> However, it would could simplify the configuration aspects.
> >> >
> >> > There are two implementations towards the above result:
> >> > 1) automatically setting MLPREFIX before MACHINE_ARCH.
> >> > 2) user manually sets
> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
> >> >
> >> > Which one do you prefer?
> > Right now I'm thinking the best approach is:
> >
> > *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> >
> > *) If the user has NOT set it, fall back and use the setting of
> > lib..-machine as in 2
> >
> > This way for people who have specific requirements for external
> > compatibility or just desire more control they can do it.  But for
> > everyone else it will still work properly.
> >
> > FYI, I'm currently working in the rootfs_rpm stuff, the system is
> > currently ignoring the alternative (multilib) machine packages.  I'm
> > hoping this will be fairly simple to resolve.
> >
> >>> >>
> >>>> >>> Problem B
> >>>> >>> =========
> >>>> >>>
> >>>> >>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which
> >>>> >>> depends on "connman", how does the system figure out to
> >>>> >>> associate this to mean we want the "x86_64" architecture packages
> installed?
> >>>> >>>
> >>>> >>> As far as I can tell there are two proposals around:
> >>>> >>>
> >>>> >>> 1) Set the architecture in the package provides and depends (a
> >>>> >>> bit
> >>>> >>> ugly)
> >>>> >>>
> >>>> >>> 2) Configure libzypp to understand that qemux86_64 does really
> >>>> >>> map to
> >>>> >>> x86_64 architecture and imply x86_64 dependencies.
> >>>> >>>
> >>>> >>> Solving problem A above is the first step towards resolving
> >>>> >>> this either way but we don't seem to have a resolution on this second
> part.
> >>> >>
> >>> >> Agreed.
> >>> >>
> >>>> >>> For reference, we really do want these task packages to have an
> >>>> >>> associated architecture so the user can easily build up blocks
> >>>> >>> of the system using a given ABI.
> >>> >>
> >>> >> This is going to be somewhat of a problem I suspect, simply
> >>> >> because that is not the way RPM is/was designed.  We can adjust
> >>> >> the rootfs install time, but this won't address field upgrade/installs.
> >>> >>
> >>> >> RPM (and Red Hat Linux/Fedora systems) appear to be designed that
> >>> >> any package that reasonably meets the run-time dependencies can be
> used.
> >>> >> There is a concept of a "best" machine based on the resolver
> >>> >> hierarchy, but ELF size may or may not be a factor in that decision.
> >>> >>
> >>> >> Doing this is quite powerful, but it also has a conscious
> >>> >> decision set behind it.  If "bash" is required by a script, it
> >>> >> really doesn't matter if it's ELF32,
> >>> >> ELF64 or something else, as long as something provides bash.
> >> >
> >> > If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a
> qemux86-64 image, it means that he wants to install all elements required by
> task-core-boot by lib32 version. If the dependency is selected *randomly*, users
> would not have multiple libraries in his system.
> > This is a special use case, yes one that I think we need to resolve,
> > but it's not the typical case.
> >
> > The typical case will be someone wants the 64-bit (or 32-bit) version
> > of a select package for compatibility.  And they only want the
> > dependencies for that package to be loaded.  Anything satisfying the
> dependency will do.
> >
> > My recommendation right now is, lets get that working ASAP.  Once that
> > does, refocus the efforts on getting the multilib "tasks" to do something
> reasonable.
> > (i.e. select a group of packages)
> >
> > If we try to solve both at once, I don't think we're going to come up
> > with a reasonable solution at this time.
> >
> > --Mark
> >
> >> > Thanks,
> >> > Dongxiao
> >>> >>
> >>>> >>> What I really need now is an idea of which patches you both
> >>>> >>> agree on that we need to merge (I think we have some for
> >>>> >>> problem A) and a resolution on problem B along with some
> >>>> >>> patches so we can close this set of issues out.
> >>>> >>>
> >>>> >>> If there are further issues can you reply to this email either
> >>>> >>> with the details or a pointer to the specific additional problems.
> >>> >>
> >>> >> Now that I have my stupid typo out of the way, I expect to have
> >>> >> further information in a few hours as to the regularly dependency
> >>> >> resolution failure Dongxiao was reporting.  (Library dependencies
> >>> >> -are- ELF specific, so those have to work properly!)
> >>> >>
> >>> >> --Mark
> >>> >>
> >>>> >>> Cheers,
> >>>> >>>
> >>>> >>> Richard
> >>>> >>>
> >>>> >>>
> >> >
> >

Re: Multilib status

[Mark Hatle]

On 9/20/11 10:01 AM, Xu, Dongxiao wrote:
> Hi Mark,
> 
>> -----Original Message-----
>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>> Sent: Saturday, September 17, 2011 2:19 AM
>> To: Mark Hatle
>> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
>> Subject: Re: Multilib status
>>
>> Just an FYI.  I'm working through a bunch of issues.  I found out that the RPM
>> dependency resolution was completely broken, due to a small bug in the
>> package_rpm.bbclass..  Packages were not being given the proper provide
>> information, so RPM was attempting best match -- and failing as noted in the
>> previous discussion..
>>
>> Unfortunately this has opened up a small can of worms which I'm trying to deal
>> with.  The simplistic fix for the bug is:
>>
>> --- a/meta/classes/package_rpm.bbclass
>> +++ b/meta/classes/package_rpm.bbclass
>> @@ -840,7 +840,7 @@ python do_package_rpm () {
>>         os.chmod(outdepends, 0755)
>>
>>         # Poky / RPM Provides
>> -       outprovides = workdir + "/" + srcname + ".requires"
>> +       outprovides = workdir + "/" + srcname + ".provides"
>>
>>         try:
>>                 from __builtin__ import file
>>
>> Unfortunately this has exposed a large set of problems that I didn't realize we
>> had....
>>
>> Will do a more formal pull request as soon as I get things working again.
>>
>> --Mark
> 
> I just had a test after cherry-pick some of your changes in mhatle/rpm, here are some problems I found and some investigations:
> 
> 1. do_rootfs failed.
> The error log reports unmet dependency of pkgconfig(libpng).
> This is due to the dangling link in libpng package. libpng.pc links to libpng12.pc which has been packaged into libpng12.

Ahh this is a bit different then the failures I've seen.  We'll have to fix that
one.

> 2. There is a wrong commit I suppose it should be reverted. When you debugging your code, please revert it.
> I will discuss it with Richard tomorrow.
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mhatle/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f

Ahh, ya I just ran into that problem... I have a fix for that commit I just
finished.  (Do you have a newer version, if so I'll update to that and see if my
change is still needed.)

> 3. After the above issue got fixed, rootfs succeed but we saw the final image is still in a mess. We need some x86_64 rpm to be installed, however what we got is x86 rpm package. Then I tried another approach we ever talked about that, firstly install base image, then install the multilib packages. I added a parameter "--replacepkgs" to rpm command to avoid errors like "package xxx has already installed". With this approach, the final image could boot up with multilib library installed.
> 
> Some of my testing code is located under http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml-testing

I'll run through the testing code and see if I can figure out what is different.

--Mark

> Thanks,
> Dongxiao
> 
> 	
>>
>> On 9/16/11 10:26 AM, Mark Hatle wrote:
>>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>>>>>>> Sent: Friday, September 16, 2011 10:51 PM
>>>>>>> To: Richard Purdie
>>>>>>> Cc: Xu, Dongxiao; openembedded-core
>>>>>>> Subject: Re: Multilib status
>>>>>>>
>>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>>>>>>>> Hi Mark/Dongxaio,
>>>>>>>>>
>>>>>>>>> We really need to get the remainder of the multilib bugs
>>>>>>>>> wrapped up. I think there is some confusion about the exact
>>>>>>>>> approach to take to fix some of them so I'm hoping to try and
>>>>>>>>> summarise the problems here so we can work out some resolutions.
>>>>>>>
>>>>>>> Let me explain where I am quickly.  I just spent two days working
>>>>>>> on reproducing the issue(s).  So far I've not reproduced the
>>>>>>> direct failures but a series of others.  I finally figured out
>>>>>>> part of the reason I wasn't seeing this.  I had a typo in my
>> configuration:
>>>>>>>
>>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>>>>>>
>>>>>>> This resulted in two sets of binaries normal and lib32 that were
>>>>>>> more or less identical in the RPM case.  We -really- need a
>>>>>>> sanity check that stupid typos like that don't cause problems.
>>>>>>>
>>>>>>> ---
>>>>>>>
>>>>>>> So now that I finally have a built with that done... see below.
>>>>>>>
>>>>>>>>> Problem A
>>>>>>>>> =========
>>>>>>>>>
>>>>>>>>> We can have multiple forms of "machine specific" packages now,
>>>>>>>>> one for each multilib e.g.:
>>>>>>>>>
>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>>>>>>>
>>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and the
>>>>>>>>> thrid /lib32/, an artificial example I realise)
>>>>>>>>>
>>>>>>>>> I know one proposal was to map:
>>>>>>>>>
>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
>>>>>>>>> correct since the library directories are different. In this
>>>>>>>>> specific case it might not matter but in general it would. What
>>>>>>>>> is also important is that the different versions imply
>>>>>>>>> different dependencies. The lib64 bit version would pull in and
>>>>>>>>> select lib64 bit libs. Its these dependencies which are a key reason
>> these are not "all" arch packages.
>>>>>>>>>
>>>>>>>>> The end result is therefore we likely want:
>>>>>>>>>
>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>>>>>>>
>>>>>>>>> and I believe there are some patches Dongxiao has created which do
>> this.
>>>>>>>
>>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
>>>>>>> patch.  So far it seems to be working.. assuming this is the
>>>>>>> approach we go with, we'll want to add some type of sanity check
>>>>>>> so that it's not possible to collide between the different
>>>>>>> multilib configurations.  (Alternatively we could, for each multilib,
>> specify a default machine virtclass in the format you list above.
>>>>>>>
>>>>>>> An alternative I was thinking of over night would be to avoid the
>>>>>>> RPM remapping on the "machine" packages.. but I'm not sure if
>>>>>>> that is really a good idea.
>>>>>>> However, it would could simplify the configuration aspects.
>>>>>
>>>>> There are two implementations towards the above result:
>>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
>>>>> 2) user manually sets
>> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
>>>>>
>>>>> Which one do you prefer?
>>> Right now I'm thinking the best approach is:
>>>
>>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
>>>
>>> *) If the user has NOT set it, fall back and use the setting of
>>> lib..-machine as in 2
>>>
>>> This way for people who have specific requirements for external
>>> compatibility or just desire more control they can do it.  But for
>>> everyone else it will still work properly.
>>>
>>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
>>> currently ignoring the alternative (multilib) machine packages.  I'm
>>> hoping this will be fairly simple to resolve.
>>>
>>>>>>>
>>>>>>>>> Problem B
>>>>>>>>> =========
>>>>>>>>>
>>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which
>>>>>>>>> depends on "connman", how does the system figure out to
>>>>>>>>> associate this to mean we want the "x86_64" architecture packages
>> installed?
>>>>>>>>>
>>>>>>>>> As far as I can tell there are two proposals around:
>>>>>>>>>
>>>>>>>>> 1) Set the architecture in the package provides and depends (a
>>>>>>>>> bit
>>>>>>>>> ugly)
>>>>>>>>>
>>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does really
>>>>>>>>> map to
>>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
>>>>>>>>>
>>>>>>>>> Solving problem A above is the first step towards resolving
>>>>>>>>> this either way but we don't seem to have a resolution on this second
>> part.
>>>>>>>
>>>>>>> Agreed.
>>>>>>>
>>>>>>>>> For reference, we really do want these task packages to have an
>>>>>>>>> associated architecture so the user can easily build up blocks
>>>>>>>>> of the system using a given ABI.
>>>>>>>
>>>>>>> This is going to be somewhat of a problem I suspect, simply
>>>>>>> because that is not the way RPM is/was designed.  We can adjust
>>>>>>> the rootfs install time, but this won't address field upgrade/installs.
>>>>>>>
>>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed that
>>>>>>> any package that reasonably meets the run-time dependencies can be
>> used.
>>>>>>> There is a concept of a "best" machine based on the resolver
>>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
>>>>>>>
>>>>>>> Doing this is quite powerful, but it also has a conscious
>>>>>>> decision set behind it.  If "bash" is required by a script, it
>>>>>>> really doesn't matter if it's ELF32,
>>>>>>> ELF64 or something else, as long as something provides bash.
>>>>>
>>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a
>> qemux86-64 image, it means that he wants to install all elements required by
>> task-core-boot by lib32 version. If the dependency is selected *randomly*, users
>> would not have multiple libraries in his system.
>>> This is a special use case, yes one that I think we need to resolve,
>>> but it's not the typical case.
>>>
>>> The typical case will be someone wants the 64-bit (or 32-bit) version
>>> of a select package for compatibility.  And they only want the
>>> dependencies for that package to be loaded.  Anything satisfying the
>> dependency will do.
>>>
>>> My recommendation right now is, lets get that working ASAP.  Once that
>>> does, refocus the efforts on getting the multilib "tasks" to do something
>> reasonable.
>>> (i.e. select a group of packages)
>>>
>>> If we try to solve both at once, I don't think we're going to come up
>>> with a reasonable solution at this time.
>>>
>>> --Mark
>>>
>>>>> Thanks,
>>>>> Dongxiao
>>>>>>>
>>>>>>>>> What I really need now is an idea of which patches you both
>>>>>>>>> agree on that we need to merge (I think we have some for
>>>>>>>>> problem A) and a resolution on problem B along with some
>>>>>>>>> patches so we can close this set of issues out.
>>>>>>>>>
>>>>>>>>> If there are further issues can you reply to this email either
>>>>>>>>> with the details or a pointer to the specific additional problems.
>>>>>>>
>>>>>>> Now that I have my stupid typo out of the way, I expect to have
>>>>>>> further information in a few hours as to the regularly dependency
>>>>>>> resolution failure Dongxiao was reporting.  (Library dependencies
>>>>>>> -are- ELF specific, so those have to work properly!)
>>>>>>>
>>>>>>> --Mark
>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Richard
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>

Re: Multilib status

[Xu, Dongxiao]

> >> -----Original Message-----
> >> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> >> Sent: Saturday, September 17, 2011 2:19 AM
> >> To: Mark Hatle
> >> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
> >> Subject: Re: Multilib status
> >>
> >> Just an FYI.  I'm working through a bunch of issues.  I found out
> >> that the RPM dependency resolution was completely broken, due to a
> >> small bug in the package_rpm.bbclass..  Packages were not being given
> >> the proper provide information, so RPM was attempting best match --
> >> and failing as noted in the previous discussion..
> >>
> >> Unfortunately this has opened up a small can of worms which I'm
> >> trying to deal with.  The simplistic fix for the bug is:
> >>
> >> --- a/meta/classes/package_rpm.bbclass
> >> +++ b/meta/classes/package_rpm.bbclass
> >> @@ -840,7 +840,7 @@ python do_package_rpm () {
> >>         os.chmod(outdepends, 0755)
> >>
> >>         # Poky / RPM Provides
> >> -       outprovides = workdir + "/" + srcname + ".requires"
> >> +       outprovides = workdir + "/" + srcname + ".provides"
> >>
> >>         try:
> >>                 from __builtin__ import file
> >>
> >> Unfortunately this has exposed a large set of problems that I didn't
> >> realize we had....
> >>
> >> Will do a more formal pull request as soon as I get things working again.
> >>
> >> --Mark
> >
> > I just had a test after cherry-pick some of your changes in mhatle/rpm, here
> are some problems I found and some investigations:
> >
> > 1. do_rootfs failed.
> > The error log reports unmet dependency of pkgconfig(libpng).
> > This is due to the dangling link in libpng package. libpng.pc links to libpng12.pc
> which has been packaged into libpng12.
> 
> Ahh this is a bit different then the failures I've seen.  We'll have to fix that one.
> 
> > 2. There is a wrong commit I suppose it should be reverted. When you
> debugging your code, please revert it.
> > I will discuss it with Richard tomorrow.
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mhatl
> > e/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f
> 
> Ahh, ya I just ran into that problem... I have a fix for that commit I just finished.
> (Do you have a newer version, if so I'll update to that and see if my change is still
> needed.)

No I haven't worked on a new version of that patch. Could you share yours?

Thanks,
Dongxiao

> 
> > 3. After the above issue got fixed, rootfs succeed but we saw the final image is
> still in a mess. We need some x86_64 rpm to be installed, however what we got
> is x86 rpm package. Then I tried another approach we ever talked about that,
> firstly install base image, then install the multilib packages. I added a parameter
> "--replacepkgs" to rpm command to avoid errors like "package xxx has already
> installed". With this approach, the final image could boot up with multilib library
> installed.
> >
> > Some of my testing code is located under
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml-
> > testing
> 
> I'll run through the testing code and see if I can figure out what is different.
> 
> --Mark
> 
> > Thanks,
> > Dongxiao
> >
> >
> >>
> >> On 9/16/11 10:26 AM, Mark Hatle wrote:
> >>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
> >>>>>>> -----Original Message-----
> >>>>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> >>>>>>> Sent: Friday, September 16, 2011 10:51 PM
> >>>>>>> To: Richard Purdie
> >>>>>>> Cc: Xu, Dongxiao; openembedded-core
> >>>>>>> Subject: Re: Multilib status
> >>>>>>>
> >>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
> >>>>>>>>> Hi Mark/Dongxaio,
> >>>>>>>>>
> >>>>>>>>> We really need to get the remainder of the multilib bugs
> >>>>>>>>> wrapped up. I think there is some confusion about the exact
> >>>>>>>>> approach to take to fix some of them so I'm hoping to try and
> >>>>>>>>> summarise the problems here so we can work out some resolutions.
> >>>>>>>
> >>>>>>> Let me explain where I am quickly.  I just spent two days
> >>>>>>> working on reproducing the issue(s).  So far I've not reproduced
> >>>>>>> the direct failures but a series of others.  I finally figured
> >>>>>>> out part of the reason I wasn't seeing this.  I had a typo in my
> >> configuration:
> >>>>>>>
> >>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
> >>>>>>>
> >>>>>>> This resulted in two sets of binaries normal and lib32 that were
> >>>>>>> more or less identical in the RPM case.  We -really- need a
> >>>>>>> sanity check that stupid typos like that don't cause problems.
> >>>>>>>
> >>>>>>> ---
> >>>>>>>
> >>>>>>> So now that I finally have a built with that done... see below.
> >>>>>>>
> >>>>>>>>> Problem A
> >>>>>>>>> =========
> >>>>>>>>>
> >>>>>>>>> We can have multiple forms of "machine specific" packages now,
> >>>>>>>>> one for each multilib e.g.:
> >>>>>>>>>
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >>>>>>>>>
> >>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and the
> >>>>>>>>> thrid /lib32/, an artificial example I realise)
> >>>>>>>>>
> >>>>>>>>> I know one proposal was to map:
> >>>>>>>>>
> >>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> >>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
> >>>>>>>>> correct since the library directories are different. In this
> >>>>>>>>> specific case it might not matter but in general it would.
> >>>>>>>>> What is also important is that the different versions imply
> >>>>>>>>> different dependencies. The lib64 bit version would pull in
> >>>>>>>>> and select lib64 bit libs. Its these dependencies which are a
> >>>>>>>>> key reason
> >> these are not "all" arch packages.
> >>>>>>>>>
> >>>>>>>>> The end result is therefore we likely want:
> >>>>>>>>>
> >>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
> >>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> >>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >>>>>>>>>
> >>>>>>>>> and I believe there are some patches Dongxiao has created
> >>>>>>>>> which do
> >> this.
> >>>>>>>
> >>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
> >>>>>>> patch.  So far it seems to be working.. assuming this is the
> >>>>>>> approach we go with, we'll want to add some type of sanity check
> >>>>>>> so that it's not possible to collide between the different
> >>>>>>> multilib configurations.  (Alternatively we could, for each
> >>>>>>> multilib,
> >> specify a default machine virtclass in the format you list above.
> >>>>>>>
> >>>>>>> An alternative I was thinking of over night would be to avoid
> >>>>>>> the RPM remapping on the "machine" packages.. but I'm not sure
> >>>>>>> if that is really a good idea.
> >>>>>>> However, it would could simplify the configuration aspects.
> >>>>>
> >>>>> There are two implementations towards the above result:
> >>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
> >>>>> 2) user manually sets
> >> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
> >>>>>
> >>>>> Which one do you prefer?
> >>> Right now I'm thinking the best approach is:
> >>>
> >>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> >>>
> >>> *) If the user has NOT set it, fall back and use the setting of
> >>> lib..-machine as in 2
> >>>
> >>> This way for people who have specific requirements for external
> >>> compatibility or just desire more control they can do it.  But for
> >>> everyone else it will still work properly.
> >>>
> >>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
> >>> currently ignoring the alternative (multilib) machine packages.  I'm
> >>> hoping this will be fairly simple to resolve.
> >>>
> >>>>>>>
> >>>>>>>>> Problem B
> >>>>>>>>> =========
> >>>>>>>>>
> >>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which
> >>>>>>>>> depends on "connman", how does the system figure out to
> >>>>>>>>> associate this to mean we want the "x86_64" architecture
> >>>>>>>>> packages
> >> installed?
> >>>>>>>>>
> >>>>>>>>> As far as I can tell there are two proposals around:
> >>>>>>>>>
> >>>>>>>>> 1) Set the architecture in the package provides and depends (a
> >>>>>>>>> bit
> >>>>>>>>> ugly)
> >>>>>>>>>
> >>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does really
> >>>>>>>>> map to
> >>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
> >>>>>>>>>
> >>>>>>>>> Solving problem A above is the first step towards resolving
> >>>>>>>>> this either way but we don't seem to have a resolution on this
> >>>>>>>>> second
> >> part.
> >>>>>>>
> >>>>>>> Agreed.
> >>>>>>>
> >>>>>>>>> For reference, we really do want these task packages to have
> >>>>>>>>> an associated architecture so the user can easily build up
> >>>>>>>>> blocks of the system using a given ABI.
> >>>>>>>
> >>>>>>> This is going to be somewhat of a problem I suspect, simply
> >>>>>>> because that is not the way RPM is/was designed.  We can adjust
> >>>>>>> the rootfs install time, but this won't address field upgrade/installs.
> >>>>>>>
> >>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed
> >>>>>>> that any package that reasonably meets the run-time dependencies
> >>>>>>> can be
> >> used.
> >>>>>>> There is a concept of a "best" machine based on the resolver
> >>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
> >>>>>>>
> >>>>>>> Doing this is quite powerful, but it also has a conscious
> >>>>>>> decision set behind it.  If "bash" is required by a script, it
> >>>>>>> really doesn't matter if it's ELF32,
> >>>>>>> ELF64 or something else, as long as something provides bash.
> >>>>>
> >>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a
> >> qemux86-64 image, it means that he wants to install all elements
> >> required by task-core-boot by lib32 version. If the dependency is
> >> selected *randomly*, users would not have multiple libraries in his system.
> >>> This is a special use case, yes one that I think we need to resolve,
> >>> but it's not the typical case.
> >>>
> >>> The typical case will be someone wants the 64-bit (or 32-bit)
> >>> version of a select package for compatibility.  And they only want
> >>> the dependencies for that package to be loaded.  Anything satisfying
> >>> the
> >> dependency will do.
> >>>
> >>> My recommendation right now is, lets get that working ASAP.  Once
> >>> that does, refocus the efforts on getting the multilib "tasks" to do
> >>> something
> >> reasonable.
> >>> (i.e. select a group of packages)
> >>>
> >>> If we try to solve both at once, I don't think we're going to come
> >>> up with a reasonable solution at this time.
> >>>
> >>> --Mark
> >>>
> >>>>> Thanks,
> >>>>> Dongxiao
> >>>>>>>
> >>>>>>>>> What I really need now is an idea of which patches you both
> >>>>>>>>> agree on that we need to merge (I think we have some for
> >>>>>>>>> problem A) and a resolution on problem B along with some
> >>>>>>>>> patches so we can close this set of issues out.
> >>>>>>>>>
> >>>>>>>>> If there are further issues can you reply to this email either
> >>>>>>>>> with the details or a pointer to the specific additional problems.
> >>>>>>>
> >>>>>>> Now that I have my stupid typo out of the way, I expect to have
> >>>>>>> further information in a few hours as to the regularly
> >>>>>>> dependency resolution failure Dongxiao was reporting.  (Library
> >>>>>>> dependencies
> >>>>>>> -are- ELF specific, so those have to work properly!)
> >>>>>>>
> >>>>>>> --Mark
> >>>>>>>
> >>>>>>>>> Cheers,
> >>>>>>>>>
> >>>>>>>>> Richard
> >>>>>>>>>
> >>>>>>>>>
> >>>>>
> >>>

Re: Multilib status

[Mark Hatle]

commit id aea5b9ef458099efd8f9a83428af84bbbe69eed2

I'm really not sure what I have is right though.. it seems to be not failing..
;)  But you likely know what should trigger these changes and if what I did is
correct or not.

--Mark

On 9/20/11 11:57 AM, Xu, Dongxiao wrote:
>>>> -----Original Message-----
>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>>>> Sent: Saturday, September 17, 2011 2:19 AM
>>>> To: Mark Hatle
>>>> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
>>>> Subject: Re: Multilib status
>>>>
>>>> Just an FYI.  I'm working through a bunch of issues.  I found out
>>>> that the RPM dependency resolution was completely broken, due to a
>>>> small bug in the package_rpm.bbclass..  Packages were not being given
>>>> the proper provide information, so RPM was attempting best match --
>>>> and failing as noted in the previous discussion..
>>>>
>>>> Unfortunately this has opened up a small can of worms which I'm
>>>> trying to deal with.  The simplistic fix for the bug is:
>>>>
>>>> --- a/meta/classes/package_rpm.bbclass
>>>> +++ b/meta/classes/package_rpm.bbclass
>>>> @@ -840,7 +840,7 @@ python do_package_rpm () {
>>>>         os.chmod(outdepends, 0755)
>>>>
>>>>         # Poky / RPM Provides
>>>> -       outprovides = workdir + "/" + srcname + ".requires"
>>>> +       outprovides = workdir + "/" + srcname + ".provides"
>>>>
>>>>         try:
>>>>                 from __builtin__ import file
>>>>
>>>> Unfortunately this has exposed a large set of problems that I didn't
>>>> realize we had....
>>>>
>>>> Will do a more formal pull request as soon as I get things working again.
>>>>
>>>> --Mark
>>>
>>> I just had a test after cherry-pick some of your changes in mhatle/rpm, here
>> are some problems I found and some investigations:
>>>
>>> 1. do_rootfs failed.
>>> The error log reports unmet dependency of pkgconfig(libpng).
>>> This is due to the dangling link in libpng package. libpng.pc links to libpng12.pc
>> which has been packaged into libpng12.
>>
>> Ahh this is a bit different then the failures I've seen.  We'll have to fix that one.
>>
>>> 2. There is a wrong commit I suppose it should be reverted. When you
>> debugging your code, please revert it.
>>> I will discuss it with Richard tomorrow.
>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mhatl
>>> e/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f
>>
>> Ahh, ya I just ran into that problem... I have a fix for that commit I just finished.
>> (Do you have a newer version, if so I'll update to that and see if my change is still
>> needed.)
> 
> No I haven't worked on a new version of that patch. Could you share yours?
> 
> Thanks,
> Dongxiao
> 
>>
>>> 3. After the above issue got fixed, rootfs succeed but we saw the final image is
>> still in a mess. We need some x86_64 rpm to be installed, however what we got
>> is x86 rpm package. Then I tried another approach we ever talked about that,
>> firstly install base image, then install the multilib packages. I added a parameter
>> "--replacepkgs" to rpm command to avoid errors like "package xxx has already
>> installed". With this approach, the final image could boot up with multilib library
>> installed.
>>>
>>> Some of my testing code is located under
>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/ml-
>>> testing
>>
>> I'll run through the testing code and see if I can figure out what is different.
>>
>> --Mark
>>
>>> Thanks,
>>> Dongxiao
>>>
>>>
>>>>
>>>> On 9/16/11 10:26 AM, Mark Hatle wrote:
>>>>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>>>>>>>>> Sent: Friday, September 16, 2011 10:51 PM
>>>>>>>>> To: Richard Purdie
>>>>>>>>> Cc: Xu, Dongxiao; openembedded-core
>>>>>>>>> Subject: Re: Multilib status
>>>>>>>>>
>>>>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>>>>>>>>>> Hi Mark/Dongxaio,
>>>>>>>>>>>
>>>>>>>>>>> We really need to get the remainder of the multilib bugs
>>>>>>>>>>> wrapped up. I think there is some confusion about the exact
>>>>>>>>>>> approach to take to fix some of them so I'm hoping to try and
>>>>>>>>>>> summarise the problems here so we can work out some resolutions.
>>>>>>>>>
>>>>>>>>> Let me explain where I am quickly.  I just spent two days
>>>>>>>>> working on reproducing the issue(s).  So far I've not reproduced
>>>>>>>>> the direct failures but a series of others.  I finally figured
>>>>>>>>> out part of the reason I wasn't seeing this.  I had a typo in my
>>>> configuration:
>>>>>>>>>
>>>>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>>>>>>>>
>>>>>>>>> This resulted in two sets of binaries normal and lib32 that were
>>>>>>>>> more or less identical in the RPM case.  We -really- need a
>>>>>>>>> sanity check that stupid typos like that don't cause problems.
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>>
>>>>>>>>> So now that I finally have a built with that done... see below.
>>>>>>>>>
>>>>>>>>>>> Problem A
>>>>>>>>>>> =========
>>>>>>>>>>>
>>>>>>>>>>> We can have multiple forms of "machine specific" packages now,
>>>>>>>>>>> one for each multilib e.g.:
>>>>>>>>>>>
>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>>>>>>>>>
>>>>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and the
>>>>>>>>>>> thrid /lib32/, an artificial example I realise)
>>>>>>>>>>>
>>>>>>>>>>> I know one proposal was to map:
>>>>>>>>>>>
>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
>>>>>>>>>>> correct since the library directories are different. In this
>>>>>>>>>>> specific case it might not matter but in general it would.
>>>>>>>>>>> What is also important is that the different versions imply
>>>>>>>>>>> different dependencies. The lib64 bit version would pull in
>>>>>>>>>>> and select lib64 bit libs. Its these dependencies which are a
>>>>>>>>>>> key reason
>>>> these are not "all" arch packages.
>>>>>>>>>>>
>>>>>>>>>>> The end result is therefore we likely want:
>>>>>>>>>>>
>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>>>>>>>>>
>>>>>>>>>>> and I believe there are some patches Dongxiao has created
>>>>>>>>>>> which do
>>>> this.
>>>>>>>>>
>>>>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
>>>>>>>>> patch.  So far it seems to be working.. assuming this is the
>>>>>>>>> approach we go with, we'll want to add some type of sanity check
>>>>>>>>> so that it's not possible to collide between the different
>>>>>>>>> multilib configurations.  (Alternatively we could, for each
>>>>>>>>> multilib,
>>>> specify a default machine virtclass in the format you list above.
>>>>>>>>>
>>>>>>>>> An alternative I was thinking of over night would be to avoid
>>>>>>>>> the RPM remapping on the "machine" packages.. but I'm not sure
>>>>>>>>> if that is really a good idea.
>>>>>>>>> However, it would could simplify the configuration aspects.
>>>>>>>
>>>>>>> There are two implementations towards the above result:
>>>>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
>>>>>>> 2) user manually sets
>>>> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
>>>>>>>
>>>>>>> Which one do you prefer?
>>>>> Right now I'm thinking the best approach is:
>>>>>
>>>>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
>>>>>
>>>>> *) If the user has NOT set it, fall back and use the setting of
>>>>> lib..-machine as in 2
>>>>>
>>>>> This way for people who have specific requirements for external
>>>>> compatibility or just desire more control they can do it.  But for
>>>>> everyone else it will still work properly.
>>>>>
>>>>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
>>>>> currently ignoring the alternative (multilib) machine packages.  I'm
>>>>> hoping this will be fairly simple to resolve.
>>>>>
>>>>>>>>>
>>>>>>>>>>> Problem B
>>>>>>>>>>> =========
>>>>>>>>>>>
>>>>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm which
>>>>>>>>>>> depends on "connman", how does the system figure out to
>>>>>>>>>>> associate this to mean we want the "x86_64" architecture
>>>>>>>>>>> packages
>>>> installed?
>>>>>>>>>>>
>>>>>>>>>>> As far as I can tell there are two proposals around:
>>>>>>>>>>>
>>>>>>>>>>> 1) Set the architecture in the package provides and depends (a
>>>>>>>>>>> bit
>>>>>>>>>>> ugly)
>>>>>>>>>>>
>>>>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does really
>>>>>>>>>>> map to
>>>>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
>>>>>>>>>>>
>>>>>>>>>>> Solving problem A above is the first step towards resolving
>>>>>>>>>>> this either way but we don't seem to have a resolution on this
>>>>>>>>>>> second
>>>> part.
>>>>>>>>>
>>>>>>>>> Agreed.
>>>>>>>>>
>>>>>>>>>>> For reference, we really do want these task packages to have
>>>>>>>>>>> an associated architecture so the user can easily build up
>>>>>>>>>>> blocks of the system using a given ABI.
>>>>>>>>>
>>>>>>>>> This is going to be somewhat of a problem I suspect, simply
>>>>>>>>> because that is not the way RPM is/was designed.  We can adjust
>>>>>>>>> the rootfs install time, but this won't address field upgrade/installs.
>>>>>>>>>
>>>>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed
>>>>>>>>> that any package that reasonably meets the run-time dependencies
>>>>>>>>> can be
>>>> used.
>>>>>>>>> There is a concept of a "best" machine based on the resolver
>>>>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
>>>>>>>>>
>>>>>>>>> Doing this is quite powerful, but it also has a conscious
>>>>>>>>> decision set behind it.  If "bash" is required by a script, it
>>>>>>>>> really doesn't matter if it's ELF32,
>>>>>>>>> ELF64 or something else, as long as something provides bash.
>>>>>>>
>>>>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with a
>>>> qemux86-64 image, it means that he wants to install all elements
>>>> required by task-core-boot by lib32 version. If the dependency is
>>>> selected *randomly*, users would not have multiple libraries in his system.
>>>>> This is a special use case, yes one that I think we need to resolve,
>>>>> but it's not the typical case.
>>>>>
>>>>> The typical case will be someone wants the 64-bit (or 32-bit)
>>>>> version of a select package for compatibility.  And they only want
>>>>> the dependencies for that package to be loaded.  Anything satisfying
>>>>> the
>>>> dependency will do.
>>>>>
>>>>> My recommendation right now is, lets get that working ASAP.  Once
>>>>> that does, refocus the efforts on getting the multilib "tasks" to do
>>>>> something
>>>> reasonable.
>>>>> (i.e. select a group of packages)
>>>>>
>>>>> If we try to solve both at once, I don't think we're going to come
>>>>> up with a reasonable solution at this time.
>>>>>
>>>>> --Mark
>>>>>
>>>>>>> Thanks,
>>>>>>> Dongxiao
>>>>>>>>>
>>>>>>>>>>> What I really need now is an idea of which patches you both
>>>>>>>>>>> agree on that we need to merge (I think we have some for
>>>>>>>>>>> problem A) and a resolution on problem B along with some
>>>>>>>>>>> patches so we can close this set of issues out.
>>>>>>>>>>>
>>>>>>>>>>> If there are further issues can you reply to this email either
>>>>>>>>>>> with the details or a pointer to the specific additional problems.
>>>>>>>>>
>>>>>>>>> Now that I have my stupid typo out of the way, I expect to have
>>>>>>>>> further information in a few hours as to the regularly
>>>>>>>>> dependency resolution failure Dongxiao was reporting.  (Library
>>>>>>>>> dependencies
>>>>>>>>> -are- ELF specific, so those have to work properly!)
>>>>>>>>>
>>>>>>>>> --Mark
>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Richard
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>
> 

Re: Multilib status

[Xu, Dongxiao]

> -----Original Message-----
> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> Sent: Wednesday, September 21, 2011 1:06 AM
> To: Xu, Dongxiao
> Cc: Richard Purdie; openembedded-core
> Subject: Re: Multilib status
> 
> commit id aea5b9ef458099efd8f9a83428af84bbbe69eed2
> 
> I'm really not sure what I have is right though.. it seems to be not failing..
> ;)  But you likely know what should trigger these changes and if what I did is
> correct or not.

What I was trying to solve with the previous commit is to handle the multiple (r)provider issue of those allarch recipes.

It seems that although your changes don't solve the above problem, they are needed to handle RPROVIDES mappings.

For the multiple provider issue, I will discuss it with Richard tomorrow.

Thanks,
Dongxiao

> 
> --Mark
> 
> On 9/20/11 11:57 AM, Xu, Dongxiao wrote:
> >>>> -----Original Message-----
> >>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> >>>> Sent: Saturday, September 17, 2011 2:19 AM
> >>>> To: Mark Hatle
> >>>> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
> >>>> Subject: Re: Multilib status
> >>>>
> >>>> Just an FYI.  I'm working through a bunch of issues.  I found out
> >>>> that the RPM dependency resolution was completely broken, due to a
> >>>> small bug in the package_rpm.bbclass..  Packages were not being
> >>>> given the proper provide information, so RPM was attempting best
> >>>> match -- and failing as noted in the previous discussion..
> >>>>
> >>>> Unfortunately this has opened up a small can of worms which I'm
> >>>> trying to deal with.  The simplistic fix for the bug is:
> >>>>
> >>>> --- a/meta/classes/package_rpm.bbclass
> >>>> +++ b/meta/classes/package_rpm.bbclass
> >>>> @@ -840,7 +840,7 @@ python do_package_rpm () {
> >>>>         os.chmod(outdepends, 0755)
> >>>>
> >>>>         # Poky / RPM Provides
> >>>> -       outprovides = workdir + "/" + srcname + ".requires"
> >>>> +       outprovides = workdir + "/" + srcname + ".provides"
> >>>>
> >>>>         try:
> >>>>                 from __builtin__ import file
> >>>>
> >>>> Unfortunately this has exposed a large set of problems that I
> >>>> didn't realize we had....
> >>>>
> >>>> Will do a more formal pull request as soon as I get things working again.
> >>>>
> >>>> --Mark
> >>>
> >>> I just had a test after cherry-pick some of your changes in
> >>> mhatle/rpm, here
> >> are some problems I found and some investigations:
> >>>
> >>> 1. do_rootfs failed.
> >>> The error log reports unmet dependency of pkgconfig(libpng).
> >>> This is due to the dangling link in libpng package. libpng.pc links
> >>> to libpng12.pc
> >> which has been packaged into libpng12.
> >>
> >> Ahh this is a bit different then the failures I've seen.  We'll have to fix that
> one.
> >>
> >>> 2. There is a wrong commit I suppose it should be reverted. When you
> >> debugging your code, please revert it.
> >>> I will discuss it with Richard tomorrow.
> >>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mha
> >>> tl e/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f
> >>
> >> Ahh, ya I just ran into that problem... I have a fix for that commit I just
> finished.
> >> (Do you have a newer version, if so I'll update to that and see if my
> >> change is still
> >> needed.)
> >
> > No I haven't worked on a new version of that patch. Could you share yours?
> >
> > Thanks,
> > Dongxiao
> >
> >>
> >>> 3. After the above issue got fixed, rootfs succeed but we saw the
> >>> final image is
> >> still in a mess. We need some x86_64 rpm to be installed, however
> >> what we got is x86 rpm package. Then I tried another approach we ever
> >> talked about that, firstly install base image, then install the
> >> multilib packages. I added a parameter "--replacepkgs" to rpm command
> >> to avoid errors like "package xxx has already installed". With this
> >> approach, the final image could boot up with multilib library installed.
> >>>
> >>> Some of my testing code is located under
> >>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/m
> >>> l-
> >>> testing
> >>
> >> I'll run through the testing code and see if I can figure out what is different.
> >>
> >> --Mark
> >>
> >>> Thanks,
> >>> Dongxiao
> >>>
> >>>
> >>>>
> >>>> On 9/16/11 10:26 AM, Mark Hatle wrote:
> >>>>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
> >>>>>>>>> Sent: Friday, September 16, 2011 10:51 PM
> >>>>>>>>> To: Richard Purdie
> >>>>>>>>> Cc: Xu, Dongxiao; openembedded-core
> >>>>>>>>> Subject: Re: Multilib status
> >>>>>>>>>
> >>>>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
> >>>>>>>>>>> Hi Mark/Dongxaio,
> >>>>>>>>>>>
> >>>>>>>>>>> We really need to get the remainder of the multilib bugs
> >>>>>>>>>>> wrapped up. I think there is some confusion about the exact
> >>>>>>>>>>> approach to take to fix some of them so I'm hoping to try
> >>>>>>>>>>> and summarise the problems here so we can work out some
> resolutions.
> >>>>>>>>>
> >>>>>>>>> Let me explain where I am quickly.  I just spent two days
> >>>>>>>>> working on reproducing the issue(s).  So far I've not
> >>>>>>>>> reproduced the direct failures but a series of others.  I
> >>>>>>>>> finally figured out part of the reason I wasn't seeing this.
> >>>>>>>>> I had a typo in my
> >>>> configuration:
> >>>>>>>>>
> >>>>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
> >>>>>>>>>
> >>>>>>>>> This resulted in two sets of binaries normal and lib32 that
> >>>>>>>>> were more or less identical in the RPM case.  We -really- need
> >>>>>>>>> a sanity check that stupid typos like that don't cause problems.
> >>>>>>>>>
> >>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>>> So now that I finally have a built with that done... see below.
> >>>>>>>>>
> >>>>>>>>>>> Problem A
> >>>>>>>>>>> =========
> >>>>>>>>>>>
> >>>>>>>>>>> We can have multiple forms of "machine specific" packages
> >>>>>>>>>>> now, one for each multilib e.g.:
> >>>>>>>>>>>
> >>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
> >>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
> >>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
> >>>>>>>>>>>
> >>>>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and
> >>>>>>>>>>> the thrid /lib32/, an artificial example I realise)
> >>>>>>>>>>>
> >>>>>>>>>>> I know one proposal was to map:
> >>>>>>>>>>>
> >>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
> >>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
> >>>>>>>>>>> correct since the library directories are different. In this
> >>>>>>>>>>> specific case it might not matter but in general it would.
> >>>>>>>>>>> What is also important is that the different versions imply
> >>>>>>>>>>> different dependencies. The lib64 bit version would pull in
> >>>>>>>>>>> and select lib64 bit libs. Its these dependencies which are
> >>>>>>>>>>> a key reason
> >>>> these are not "all" arch packages.
> >>>>>>>>>>>
> >>>>>>>>>>> The end result is therefore we likely want:
> >>>>>>>>>>>
> >>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
> >>>>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
> >>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
> >>>>>>>>>>>
> >>>>>>>>>>> and I believe there are some patches Dongxiao has created
> >>>>>>>>>>> which do
> >>>> this.
> >>>>>>>>>
> >>>>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
> >>>>>>>>> patch.  So far it seems to be working.. assuming this is the
> >>>>>>>>> approach we go with, we'll want to add some type of sanity
> >>>>>>>>> check so that it's not possible to collide between the
> >>>>>>>>> different multilib configurations.  (Alternatively we could,
> >>>>>>>>> for each multilib,
> >>>> specify a default machine virtclass in the format you list above.
> >>>>>>>>>
> >>>>>>>>> An alternative I was thinking of over night would be to avoid
> >>>>>>>>> the RPM remapping on the "machine" packages.. but I'm not sure
> >>>>>>>>> if that is really a good idea.
> >>>>>>>>> However, it would could simplify the configuration aspects.
> >>>>>>>
> >>>>>>> There are two implementations towards the above result:
> >>>>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
> >>>>>>> 2) user manually sets
> >>>> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
> >>>>>>>
> >>>>>>> Which one do you prefer?
> >>>>> Right now I'm thinking the best approach is:
> >>>>>
> >>>>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
> >>>>>
> >>>>> *) If the user has NOT set it, fall back and use the setting of
> >>>>> lib..-machine as in 2
> >>>>>
> >>>>> This way for people who have specific requirements for external
> >>>>> compatibility or just desire more control they can do it.  But for
> >>>>> everyone else it will still work properly.
> >>>>>
> >>>>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
> >>>>> currently ignoring the alternative (multilib) machine packages.
> >>>>> I'm hoping this will be fairly simple to resolve.
> >>>>>
> >>>>>>>>>
> >>>>>>>>>>> Problem B
> >>>>>>>>>>> =========
> >>>>>>>>>>>
> >>>>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm
> >>>>>>>>>>> which depends on "connman", how does the system figure out
> >>>>>>>>>>> to associate this to mean we want the "x86_64" architecture
> >>>>>>>>>>> packages
> >>>> installed?
> >>>>>>>>>>>
> >>>>>>>>>>> As far as I can tell there are two proposals around:
> >>>>>>>>>>>
> >>>>>>>>>>> 1) Set the architecture in the package provides and depends
> >>>>>>>>>>> (a bit
> >>>>>>>>>>> ugly)
> >>>>>>>>>>>
> >>>>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does
> >>>>>>>>>>> really map to
> >>>>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
> >>>>>>>>>>>
> >>>>>>>>>>> Solving problem A above is the first step towards resolving
> >>>>>>>>>>> this either way but we don't seem to have a resolution on
> >>>>>>>>>>> this second
> >>>> part.
> >>>>>>>>>
> >>>>>>>>> Agreed.
> >>>>>>>>>
> >>>>>>>>>>> For reference, we really do want these task packages to have
> >>>>>>>>>>> an associated architecture so the user can easily build up
> >>>>>>>>>>> blocks of the system using a given ABI.
> >>>>>>>>>
> >>>>>>>>> This is going to be somewhat of a problem I suspect, simply
> >>>>>>>>> because that is not the way RPM is/was designed.  We can
> >>>>>>>>> adjust the rootfs install time, but this won't address field
> upgrade/installs.
> >>>>>>>>>
> >>>>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed
> >>>>>>>>> that any package that reasonably meets the run-time
> >>>>>>>>> dependencies can be
> >>>> used.
> >>>>>>>>> There is a concept of a "best" machine based on the resolver
> >>>>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
> >>>>>>>>>
> >>>>>>>>> Doing this is quite powerful, but it also has a conscious
> >>>>>>>>> decision set behind it.  If "bash" is required by a script, it
> >>>>>>>>> really doesn't matter if it's ELF32,
> >>>>>>>>> ELF64 or something else, as long as something provides bash.
> >>>>>>>
> >>>>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with
> >>>>>>> a
> >>>> qemux86-64 image, it means that he wants to install all elements
> >>>> required by task-core-boot by lib32 version. If the dependency is
> >>>> selected *randomly*, users would not have multiple libraries in his system.
> >>>>> This is a special use case, yes one that I think we need to
> >>>>> resolve, but it's not the typical case.
> >>>>>
> >>>>> The typical case will be someone wants the 64-bit (or 32-bit)
> >>>>> version of a select package for compatibility.  And they only want
> >>>>> the dependencies for that package to be loaded.  Anything
> >>>>> satisfying the
> >>>> dependency will do.
> >>>>>
> >>>>> My recommendation right now is, lets get that working ASAP.  Once
> >>>>> that does, refocus the efforts on getting the multilib "tasks" to
> >>>>> do something
> >>>> reasonable.
> >>>>> (i.e. select a group of packages)
> >>>>>
> >>>>> If we try to solve both at once, I don't think we're going to come
> >>>>> up with a reasonable solution at this time.
> >>>>>
> >>>>> --Mark
> >>>>>
> >>>>>>> Thanks,
> >>>>>>> Dongxiao
> >>>>>>>>>
> >>>>>>>>>>> What I really need now is an idea of which patches you both
> >>>>>>>>>>> agree on that we need to merge (I think we have some for
> >>>>>>>>>>> problem A) and a resolution on problem B along with some
> >>>>>>>>>>> patches so we can close this set of issues out.
> >>>>>>>>>>>
> >>>>>>>>>>> If there are further issues can you reply to this email
> >>>>>>>>>>> either with the details or a pointer to the specific additional
> problems.
> >>>>>>>>>
> >>>>>>>>> Now that I have my stupid typo out of the way, I expect to
> >>>>>>>>> have further information in a few hours as to the regularly
> >>>>>>>>> dependency resolution failure Dongxiao was reporting.
> >>>>>>>>> (Library dependencies
> >>>>>>>>> -are- ELF specific, so those have to work properly!)
> >>>>>>>>>
> >>>>>>>>> --Mark
> >>>>>>>>>
> >>>>>>>>>>> Cheers,
> >>>>>>>>>>>
> >>>>>>>>>>> Richard
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
> >>>>>
> >

Re: Multilib status

[Mark Hatle]

On 9/20/11 12:34 PM, Xu, Dongxiao wrote:
>> -----Original Message-----
>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>> Sent: Wednesday, September 21, 2011 1:06 AM
>> To: Xu, Dongxiao
>> Cc: Richard Purdie; openembedded-core
>> Subject: Re: Multilib status
>>
>> commit id aea5b9ef458099efd8f9a83428af84bbbe69eed2
>>
>> I'm really not sure what I have is right though.. it seems to be not failing..
>> ;)  But you likely know what should trigger these changes and if what I did is
>> correct or not.
> 
> What I was trying to solve with the previous commit is to handle the multiple (r)provider issue of those allarch recipes.
> 
> It seems that although your changes don't solve the above problem, they are needed to handle RPROVIDES mappings.
> 
> For the multiple provider issue, I will discuss it with Richard tomorrow.

Ya, what I sent out early was broken.. I think I have it working now:

poky-contrib.git -- commit id: d971a73d970b8f0beeded599f95a6d8ce5c32a4c

This seems to correctly specify the PROVIDES, RPROVIDES and RPROVIDES_<PN> for
the multilibs now.. I'm still not sure that this is right though, because it may
change the PROVIDES/RPROVIDES for packages that were previously cached from non
multilib builds..  but it's progress.

--Mark

> Thanks,
> Dongxiao
> 
>>
>> --Mark
>>
>> On 9/20/11 11:57 AM, Xu, Dongxiao wrote:
>>>>>> -----Original Message-----
>>>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>>>>>> Sent: Saturday, September 17, 2011 2:19 AM
>>>>>> To: Mark Hatle
>>>>>> Cc: Xu, Dongxiao; Richard Purdie; openembedded-core
>>>>>> Subject: Re: Multilib status
>>>>>>
>>>>>> Just an FYI.  I'm working through a bunch of issues.  I found out
>>>>>> that the RPM dependency resolution was completely broken, due to a
>>>>>> small bug in the package_rpm.bbclass..  Packages were not being
>>>>>> given the proper provide information, so RPM was attempting best
>>>>>> match -- and failing as noted in the previous discussion..
>>>>>>
>>>>>> Unfortunately this has opened up a small can of worms which I'm
>>>>>> trying to deal with.  The simplistic fix for the bug is:
>>>>>>
>>>>>> --- a/meta/classes/package_rpm.bbclass
>>>>>> +++ b/meta/classes/package_rpm.bbclass
>>>>>> @@ -840,7 +840,7 @@ python do_package_rpm () {
>>>>>>         os.chmod(outdepends, 0755)
>>>>>>
>>>>>>         # Poky / RPM Provides
>>>>>> -       outprovides = workdir + "/" + srcname + ".requires"
>>>>>> +       outprovides = workdir + "/" + srcname + ".provides"
>>>>>>
>>>>>>         try:
>>>>>>                 from __builtin__ import file
>>>>>>
>>>>>> Unfortunately this has exposed a large set of problems that I
>>>>>> didn't realize we had....
>>>>>>
>>>>>> Will do a more formal pull request as soon as I get things working again.
>>>>>>
>>>>>> --Mark
>>>>>
>>>>> I just had a test after cherry-pick some of your changes in
>>>>> mhatle/rpm, here
>>>> are some problems I found and some investigations:
>>>>>
>>>>> 1. do_rootfs failed.
>>>>> The error log reports unmet dependency of pkgconfig(libpng).
>>>>> This is due to the dangling link in libpng package. libpng.pc links
>>>>> to libpng12.pc
>>>> which has been packaged into libpng12.
>>>>
>>>> Ahh this is a bit different then the failures I've seen.  We'll have to fix that
>> one.
>>>>
>>>>> 2. There is a wrong commit I suppose it should be reverted. When you
>>>> debugging your code, please revert it.
>>>>> I will discuss it with Richard tomorrow.
>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=mha
>>>>> tl e/rpm&id=329d864f9bbf94ad3aae8df43d63fe10e4237e4f
>>>>
>>>> Ahh, ya I just ran into that problem... I have a fix for that commit I just
>> finished.
>>>> (Do you have a newer version, if so I'll update to that and see if my
>>>> change is still
>>>> needed.)
>>>
>>> No I haven't worked on a new version of that patch. Could you share yours?
>>>
>>> Thanks,
>>> Dongxiao
>>>
>>>>
>>>>> 3. After the above issue got fixed, rootfs succeed but we saw the
>>>>> final image is
>>>> still in a mess. We need some x86_64 rpm to be installed, however
>>>> what we got is x86 rpm package. Then I tried another approach we ever
>>>> talked about that, firstly install base image, then install the
>>>> multilib packages. I added a parameter "--replacepkgs" to rpm command
>>>> to avoid errors like "package xxx has already installed". With this
>>>> approach, the final image could boot up with multilib library installed.
>>>>>
>>>>> Some of my testing code is located under
>>>>> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=dxu4/m
>>>>> l-
>>>>> testing
>>>>
>>>> I'll run through the testing code and see if I can figure out what is different.
>>>>
>>>> --Mark
>>>>
>>>>> Thanks,
>>>>> Dongxiao
>>>>>
>>>>>
>>>>>>
>>>>>> On 9/16/11 10:26 AM, Mark Hatle wrote:
>>>>>>> On 9/16/11 10:21 AM, Xu, Dongxiao wrote:
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Mark Hatle [mailto:mark.hatle@windriver.com]
>>>>>>>>>>> Sent: Friday, September 16, 2011 10:51 PM
>>>>>>>>>>> To: Richard Purdie
>>>>>>>>>>> Cc: Xu, Dongxiao; openembedded-core
>>>>>>>>>>> Subject: Re: Multilib status
>>>>>>>>>>>
>>>>>>>>>>> On 9/16/11 4:32 AM, Richard Purdie wrote:
>>>>>>>>>>>>> Hi Mark/Dongxaio,
>>>>>>>>>>>>>
>>>>>>>>>>>>> We really need to get the remainder of the multilib bugs
>>>>>>>>>>>>> wrapped up. I think there is some confusion about the exact
>>>>>>>>>>>>> approach to take to fix some of them so I'm hoping to try
>>>>>>>>>>>>> and summarise the problems here so we can work out some
>> resolutions.
>>>>>>>>>>>
>>>>>>>>>>> Let me explain where I am quickly.  I just spent two days
>>>>>>>>>>> working on reproducing the issue(s).  So far I've not
>>>>>>>>>>> reproduced the direct failures but a series of others.  I
>>>>>>>>>>> finally figured out part of the reason I wasn't seeing this.
>>>>>>>>>>> I had a typo in my
>>>>>> configuration:
>>>>>>>>>>>
>>>>>>>>>>> DEFAULTTUNE-virtclass-mulitlib-lib32 = "x86"
>>>>>>>>>>>
>>>>>>>>>>> This resulted in two sets of binaries normal and lib32 that
>>>>>>>>>>> were more or less identical in the RPM case.  We -really- need
>>>>>>>>>>> a sanity check that stupid typos like that don't cause problems.
>>>>>>>>>>>
>>>>>>>>>>> ---
>>>>>>>>>>>
>>>>>>>>>>> So now that I finally have a built with that done... see below.
>>>>>>>>>>>
>>>>>>>>>>>>> Problem A
>>>>>>>>>>>>> =========
>>>>>>>>>>>>>
>>>>>>>>>>>>> We can have multiple forms of "machine specific" packages
>>>>>>>>>>>>> now, one for each multilib e.g.:
>>>>>>>>>>>>>
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("normal")
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib64")
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm ("lib32")
>>>>>>>>>>>>>
>>>>>>>>>>>>> (lets assume the first uses /lib/, the second /lib64/ and
>>>>>>>>>>>>> the thrid /lib32/, an artificial example I realise)
>>>>>>>>>>>>>
>>>>>>>>>>>>> I know one proposal was to map:
>>>>>>>>>>>>>
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm to
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86.rpm but this isn't
>>>>>>>>>>>>> correct since the library directories are different. In this
>>>>>>>>>>>>> specific case it might not matter but in general it would.
>>>>>>>>>>>>> What is also important is that the different versions imply
>>>>>>>>>>>>> different dependencies. The lib64 bit version would pull in
>>>>>>>>>>>>> and select lib64 bit libs. Its these dependencies which are
>>>>>>>>>>>>> a key reason
>>>>>> these are not "all" arch packages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The end result is therefore we likely want:
>>>>>>>>>>>>>
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib64_qemux86_64.rpm
>>>>>>>>>>>>> task-core-x11-base-1.0-r34.lib32_qemux86_64.rpm
>>>>>>>>>>>>>
>>>>>>>>>>>>> and I believe there are some patches Dongxiao has created
>>>>>>>>>>>>> which do
>>>>>> this.
>>>>>>>>>>>
>>>>>>>>>>> Yes, I'm currently testing his new MACHINE_virtclass-multilib-...
>>>>>>>>>>> patch.  So far it seems to be working.. assuming this is the
>>>>>>>>>>> approach we go with, we'll want to add some type of sanity
>>>>>>>>>>> check so that it's not possible to collide between the
>>>>>>>>>>> different multilib configurations.  (Alternatively we could,
>>>>>>>>>>> for each multilib,
>>>>>> specify a default machine virtclass in the format you list above.
>>>>>>>>>>>
>>>>>>>>>>> An alternative I was thinking of over night would be to avoid
>>>>>>>>>>> the RPM remapping on the "machine" packages.. but I'm not sure
>>>>>>>>>>> if that is really a good idea.
>>>>>>>>>>> However, it would could simplify the configuration aspects.
>>>>>>>>>
>>>>>>>>> There are two implementations towards the above result:
>>>>>>>>> 1) automatically setting MLPREFIX before MACHINE_ARCH.
>>>>>>>>> 2) user manually sets
>>>>>> MACHINE_virtclass-multilib-lib32="lib32-qemux86-64" in local.conf.
>>>>>>>>>
>>>>>>>>> Which one do you prefer?
>>>>>>> Right now I'm thinking the best approach is:
>>>>>>>
>>>>>>> *) Allow the user to set MACHINE_virtclass-multilib-...="...."
>>>>>>>
>>>>>>> *) If the user has NOT set it, fall back and use the setting of
>>>>>>> lib..-machine as in 2
>>>>>>>
>>>>>>> This way for people who have specific requirements for external
>>>>>>> compatibility or just desire more control they can do it.  But for
>>>>>>> everyone else it will still work properly.
>>>>>>>
>>>>>>> FYI, I'm currently working in the rootfs_rpm stuff, the system is
>>>>>>> currently ignoring the alternative (multilib) machine packages.
>>>>>>> I'm hoping this will be fairly simple to resolve.
>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> Problem B
>>>>>>>>>>>>> =========
>>>>>>>>>>>>>
>>>>>>>>>>>>> If we install task-core-x11-base-1.0-r34.qemux86_64.rpm
>>>>>>>>>>>>> which depends on "connman", how does the system figure out
>>>>>>>>>>>>> to associate this to mean we want the "x86_64" architecture
>>>>>>>>>>>>> packages
>>>>>> installed?
>>>>>>>>>>>>>
>>>>>>>>>>>>> As far as I can tell there are two proposals around:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1) Set the architecture in the package provides and depends
>>>>>>>>>>>>> (a bit
>>>>>>>>>>>>> ugly)
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2) Configure libzypp to understand that qemux86_64 does
>>>>>>>>>>>>> really map to
>>>>>>>>>>>>> x86_64 architecture and imply x86_64 dependencies.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Solving problem A above is the first step towards resolving
>>>>>>>>>>>>> this either way but we don't seem to have a resolution on
>>>>>>>>>>>>> this second
>>>>>> part.
>>>>>>>>>>>
>>>>>>>>>>> Agreed.
>>>>>>>>>>>
>>>>>>>>>>>>> For reference, we really do want these task packages to have
>>>>>>>>>>>>> an associated architecture so the user can easily build up
>>>>>>>>>>>>> blocks of the system using a given ABI.
>>>>>>>>>>>
>>>>>>>>>>> This is going to be somewhat of a problem I suspect, simply
>>>>>>>>>>> because that is not the way RPM is/was designed.  We can
>>>>>>>>>>> adjust the rootfs install time, but this won't address field
>> upgrade/installs.
>>>>>>>>>>>
>>>>>>>>>>> RPM (and Red Hat Linux/Fedora systems) appear to be designed
>>>>>>>>>>> that any package that reasonably meets the run-time
>>>>>>>>>>> dependencies can be
>>>>>> used.
>>>>>>>>>>> There is a concept of a "best" machine based on the resolver
>>>>>>>>>>> hierarchy, but ELF size may or may not be a factor in that decision.
>>>>>>>>>>>
>>>>>>>>>>> Doing this is quite powerful, but it also has a conscious
>>>>>>>>>>> decision set behind it.  If "bash" is required by a script, it
>>>>>>>>>>> really doesn't matter if it's ELF32,
>>>>>>>>>>> ELF64 or something else, as long as something provides bash.
>>>>>>>>>
>>>>>>>>> If user sets MULTILIB_IMAGE_INSTALL="lib32-task-core-boot" with
>>>>>>>>> a
>>>>>> qemux86-64 image, it means that he wants to install all elements
>>>>>> required by task-core-boot by lib32 version. If the dependency is
>>>>>> selected *randomly*, users would not have multiple libraries in his system.
>>>>>>> This is a special use case, yes one that I think we need to
>>>>>>> resolve, but it's not the typical case.
>>>>>>>
>>>>>>> The typical case will be someone wants the 64-bit (or 32-bit)
>>>>>>> version of a select package for compatibility.  And they only want
>>>>>>> the dependencies for that package to be loaded.  Anything
>>>>>>> satisfying the
>>>>>> dependency will do.
>>>>>>>
>>>>>>> My recommendation right now is, lets get that working ASAP.  Once
>>>>>>> that does, refocus the efforts on getting the multilib "tasks" to
>>>>>>> do something
>>>>>> reasonable.
>>>>>>> (i.e. select a group of packages)
>>>>>>>
>>>>>>> If we try to solve both at once, I don't think we're going to come
>>>>>>> up with a reasonable solution at this time.
>>>>>>>
>>>>>>> --Mark
>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Dongxiao
>>>>>>>>>>>
>>>>>>>>>>>>> What I really need now is an idea of which patches you both
>>>>>>>>>>>>> agree on that we need to merge (I think we have some for
>>>>>>>>>>>>> problem A) and a resolution on problem B along with some
>>>>>>>>>>>>> patches so we can close this set of issues out.
>>>>>>>>>>>>>
>>>>>>>>>>>>> If there are further issues can you reply to this email
>>>>>>>>>>>>> either with the details or a pointer to the specific additional
>> problems.
>>>>>>>>>>>
>>>>>>>>>>> Now that I have my stupid typo out of the way, I expect to
>>>>>>>>>>> have further information in a few hours as to the regularly
>>>>>>>>>>> dependency resolution failure Dongxiao was reporting.
>>>>>>>>>>> (Library dependencies
>>>>>>>>>>> -are- ELF specific, so those have to work properly!)
>>>>>>>>>>>
>>>>>>>>>>> --Mark
>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Richard
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>
>