zypper and poky architectures

zypper and poky architectures

[Qing He]

I recently reported several zypper bugs specifically for arm, after
some deeper investigation, the problem seems to be of higher level than
I originally thought.

The root cause is that zypper and poky use different way to represent
architectures, as we are putting them together, these two ways are
not compatible, causing many minor glitches that need to modify at
least one of them.

Poky has three kinds of representations in a single target image, which
are independent, cpu-dependent and machine-dependent (all, armv5te,
qemuarm, respectively), e.g.

	update-rc.d-0.7-r3.all.rpm
	curl-7.21.0-r0.armv5te.rpm
	task-base-1.0-r69.qemuarm.rpm

(note that armv5te is the same with gcc's -march option, meaning little
endian)

This is natural until zypper is involved. Zypper supports only one arch
at one time (and this arch should not be changed in fly), and use the
idea of arch compatibility (e.g. _noarch is compatible with _i586), it
then hardcodes the available archs in a different way than poky does,
thus creating several problems:
  1. what uses for independent packages is called "noarch", "all" is not
     recognized, something depends on update-rc.d won't be installed
     because of missing dependency
  2. the arch automatic detection system uses "uname -m", thus producing
     armv5tejl, which can only be resolved as armv5tel, conflicting with
     "armv5te" in rpm
  3. many archs are missing in zypper, like mips, armeb, etc.
  4. there is no concept of machine-dependent packages (task-base) in
     zypper, although we can work around.

Currently, at least zypper is broken on all of mips, arm, ppc, with
slightly different problems.

The ideal situation is to use consistent arch specification, the
following can be a solution:
  1. rename *.all.rpm to *.noarch.rpm
  2. removing the concept of machine-dependent packages, change all
     *.qemuarm.rpm to *.armv5te.rpm
  3. enhance zypper arch module, make the addition more flexible,
     allowing arch alias (e.g. armv5te = armv5tel = armel = arm)

That would be some work to do, maybe 1.0 is a good time to get zypper
and package upgrade truely working.

Any ideas and comments?

Thanks,
Qing

Re: zypper and poky architectures

[Richard Purdie]

On Thu, 2010-10-21 at 16:33 +0800, Qing He wrote:
> I recently reported several zypper bugs specifically for arm, after
> some deeper investigation, the problem seems to be of higher level than
> I originally thought.
> 
> The root cause is that zypper and poky use different way to represent
> architectures, as we are putting them together, these two ways are
> not compatible, causing many minor glitches that need to modify at
> least one of them.
> 
> Poky has three kinds of representations in a single target image, which
> are independent, cpu-dependent and machine-dependent (all, armv5te,
> qemuarm, respectively), e.g.
> 
> 	update-rc.d-0.7-r3.all.rpm
> 	curl-7.21.0-r0.armv5te.rpm
> 	task-base-1.0-r69.qemuarm.rpm
> 
> (note that armv5te is the same with gcc's -march option, meaning little
> endian)

This is a good analysis and summary. It actually gets more complicated
as for some machines we have a long list of compatible package types
that can be installed, e.g. for qemuarm, if you list PACKAGE_ARCHES
you'll see armv4t armv5 armv5t armv5te which are all accepted by the
opkg backend.

> This is natural until zypper is involved. Zypper supports only one arch
> at one time (and this arch should not be changed in fly), and use the
> idea of arch compatibility (e.g. _noarch is compatible with _i586), it
> then hardcodes the available archs in a different way than poky does,
> thus creating several problems:
>   1. what uses for independent packages is called "noarch", "all" is not
>      recognized, something depends on update-rc.d won't be installed
>      because of missing dependency
>   2. the arch automatic detection system uses "uname -m", thus producing
>      armv5tejl, which can only be resolved as armv5tel, conflicting with
>      "armv5te" in rpm
>   3. many archs are missing in zypper, like mips, armeb, etc.
>   4. there is no concept of machine-dependent packages (task-base) in
>      zypper, although we can work around.
> 
> Currently, at least zypper is broken on all of mips, arm, ppc, with
> slightly different problems.
> 
> The ideal situation is to use consistent arch specification, the
> following can be a solution:
>   1. rename *.all.rpm to *.noarch.rpm

This would only solve part of the problem though?

>   2. removing the concept of machine-dependent packages, change all
>      *.qemuarm.rpm to *.armv5te.rpm

This could mean making a copy of each rpm per machine so I'm not keen on
this.

>   3. enhance zypper arch module, make the addition more flexible,
>      allowing arch alias (e.g. armv5te = armv5tel = armel = arm)
> 
> That would be some work to do, maybe 1.0 is a good time to get zypper
> and package upgrade truely working.
> 
> Any ideas and comments?

I think we're going to have to teach zypper to read a list of compatible
"architectures" from a configuration file. There is a config file opkg
writes to the filesystem continaing this list and we'll have to do
similar for RPM.

It does raise the question of how given two possible rpm's it would
chose between the two (for opkg, the list is in order).

Is the problem just in zypper and is rpm free from any issues in this
area?

Cheers,

Richard

Re: zypper and poky architectures

[Mark Hatle]

On 10/21/10 6:21 AM, Richard Purdie wrote:
> On Thu, 2010-10-21 at 16:33 +0800, Qing He wrote:
>> I recently reported several zypper bugs specifically for arm, after
>> some deeper investigation, the problem seems to be of higher level than
>> I originally thought.
>>
>> The root cause is that zypper and poky use different way to represent
>> architectures, as we are putting them together, these two ways are
>> not compatible, causing many minor glitches that need to modify at
>> least one of them.
>>
>> Poky has three kinds of representations in a single target image, which
>> are independent, cpu-dependent and machine-dependent (all, armv5te,
>> qemuarm, respectively), e.g.
>>
>> 	update-rc.d-0.7-r3.all.rpm
>> 	curl-7.21.0-r0.armv5te.rpm
>> 	task-base-1.0-r69.qemuarm.rpm
>>
>> (note that armv5te is the same with gcc's -march option, meaning little
>> endian)
>
> This is a good analysis and summary. It actually gets more complicated
> as for some machines we have a long list of compatible package types
> that can be installed, e.g. for qemuarm, if you list PACKAGE_ARCHES
> you'll see armv4t armv5 armv5t armv5te which are all accepted by the
> opkg backend.
>
>> This is natural until zypper is involved. Zypper supports only one arch
>> at one time (and this arch should not be changed in fly), and use the
>> idea of arch compatibility (e.g. _noarch is compatible with _i586), it
>> then hardcodes the available archs in a different way than poky does,
>> thus creating several problems:
>>    1. what uses for independent packages is called "noarch", "all" is not
>>       recognized, something depends on update-rc.d won't be installed
>>       because of missing dependency
>>    2. the arch automatic detection system uses "uname -m", thus producing
>>       armv5tejl, which can only be resolved as armv5tel, conflicting with
>>       "armv5te" in rpm
>>    3. many archs are missing in zypper, like mips, armeb, etc.
>>    4. there is no concept of machine-dependent packages (task-base) in
>>       zypper, although we can work around.
>>
>> Currently, at least zypper is broken on all of mips, arm, ppc, with
>> slightly different problems.
>>
>> The ideal situation is to use consistent arch specification, the
>> following can be a solution:
>>    1. rename *.all.rpm to *.noarch.rpm
>
> This would only solve part of the problem though?
>
>>    2. removing the concept of machine-dependent packages, change all
>>       *.qemuarm.rpm to *.armv5te.rpm
>
> This could mean making a copy of each rpm per machine so I'm not keen on
> this.
>
>>    3. enhance zypper arch module, make the addition more flexible,
>>       allowing arch alias (e.g. armv5te = armv5tel = armel = arm)
>>
>> That would be some work to do, maybe 1.0 is a good time to get zypper
>> and package upgrade truely working.
>>
>> Any ideas and comments?
>
> I think we're going to have to teach zypper to read a list of compatible
> "architectures" from a configuration file. There is a config file opkg
> writes to the filesystem continaing this list and we'll have to do
> similar for RPM.
>
> It does raise the question of how given two possible rpm's it would
> chose between the two (for opkg, the list is in order).
>
> Is the problem just in zypper and is rpm free from any issues in this
> area?

We will need to do some additional verification on RPM -- but it appears to me 
that RPM should not have a problem with this.  As far as I'm aware, RPM5 simply 
ignores the ARCH field for the most part.  (My concern as we move in this 
direction will be ordering of priority between architecture types..  perhaps 
thats not even in the scope of RPM to worry about... but replacing say an armv4 
w/ an armv5 is...)

--Mark

> Cheers,
>
> Richard
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.pokylinux.org/listinfo/yocto

Re: zypper and poky architectures

[Qing He]

On Thu, 2010-10-21 at 19:21 +0800, Richard Purdie wrote:
> On Thu, 2010-10-21 at 16:33 +0800, Qing He wrote:
> > I recently reported several zypper bugs specifically for arm, after
> > some deeper investigation, the problem seems to be of higher level than
> > I originally thought.
> > 
> > The root cause is that zypper and poky use different way to represent
> > architectures, as we are putting them together, these two ways are
> > not compatible, causing many minor glitches that need to modify at
> > least one of them.
> > 
> > Poky has three kinds of representations in a single target image, which
> > are independent, cpu-dependent and machine-dependent (all, armv5te,
> > qemuarm, respectively), e.g.
> > 
> > 	update-rc.d-0.7-r3.all.rpm
> > 	curl-7.21.0-r0.armv5te.rpm
> > 	task-base-1.0-r69.qemuarm.rpm
> > 
> > (note that armv5te is the same with gcc's -march option, meaning little
> > endian)
> 
> This is a good analysis and summary. It actually gets more complicated
> as for some machines we have a long list of compatible package types
> that can be installed, e.g. for qemuarm, if you list PACKAGE_ARCHES
> you'll see armv4t armv5 armv5t armv5te which are all accepted by the
> opkg backend.

Zypper also does something similar:

    defCompatibleWith(_armv5tejl,   _noarch,_armv3l,_armv4l,_armv4tl,_armv5l,_armv5tel);

> 
> > Currently, at least zypper is broken on all of mips, arm, ppc, with
> > slightly different problems.
> > 
> > The ideal situation is to use consistent arch specification, the
> > following can be a solution:
> >   1. rename *.all.rpm to *.noarch.rpm
> 
> This would only solve part of the problem though?

Yes, only a small part. And the way I prefer is to add an alias in zypper
instead of changing poky.

> 
> >   2. removing the concept of machine-dependent packages, change all
> >      *.qemuarm.rpm to *.armv5te.rpm
> 
> This could mean making a copy of each rpm per machine so I'm not keen on
> this.

Agreed.

> 
> >   3. enhance zypper arch module, make the addition more flexible,
> >      allowing arch alias (e.g. armv5te = armv5tel = armel = arm)
> > 
> > That would be some work to do, maybe 1.0 is a good time to get zypper
> > and package upgrade truely working.
> > 
> > Any ideas and comments?
> 
> I think we're going to have to teach zypper to read a list of compatible
> "architectures" from a configuration file. There is a config file opkg
> writes to the filesystem continaing this list and we'll have to do
> similar for RPM.

The problem is that current zypper uses hardcoded "archs" and
compatibility list, i.e. at build time with C++ code. Changing it to
read the list at run time is probably substantial.

Another possibility is to generate the list at build time and do it by
scripts and conditional compiling, this does have the problem of
creating logically different versions for different archs.

> 
> It does raise the question of how given two possible rpm's it would
> chose between the two (for opkg, the list is in order).
> 
> Is the problem just in zypper and is rpm free from any issues in this
> area?

I don't know much about rpm internals, there is a bug regarding the rpm,
http://bugzilla.pokylinux.org/show_bug.cgi?id=498

But it looks fair limited in impact.

Thanks,
Qing

Re: zypper and poky architectures

[Mark Hatle]

On 10/21/10 3:33 AM, Qing He wrote:
> I recently reported several zypper bugs specifically for arm, after
> some deeper investigation, the problem seems to be of higher level than
> I originally thought.
>
> The root cause is that zypper and poky use different way to represent
> architectures, as we are putting them together, these two ways are
> not compatible, causing many minor glitches that need to modify at
> least one of them.
>
> Poky has three kinds of representations in a single target image, which
> are independent, cpu-dependent and machine-dependent (all, armv5te,
> qemuarm, respectively), e.g.
>
> 	update-rc.d-0.7-r3.all.rpm
> 	curl-7.21.0-r0.armv5te.rpm
> 	task-base-1.0-r69.qemuarm.rpm
>
> (note that armv5te is the same with gcc's -march option, meaning little
> endian)
>
> This is natural until zypper is involved. Zypper supports only one arch
> at one time (and this arch should not be changed in fly), and use the
> idea of arch compatibility (e.g. _noarch is compatible with _i586), it
> then hardcodes the available archs in a different way than poky does,
> thus creating several problems:
>    1. what uses for independent packages is called "noarch", "all" is not
>       recognized, something depends on update-rc.d won't be installed
>       because of missing dependency

We can certainly look into translating "all" to "noarch" post 0.9.  That might 
make it easier for people coming from the RPM world, to understand what is in 
the package.

>    2. the arch automatic detection system uses "uname -m", thus producing
>       armv5tejl, which can only be resolved as armv5tel, conflicting with
>       "armv5te" in rpm

This is a bug in Zypper.  The machine names should come from somewhere other 
then uname -m.  (The value of uname -m is very much ia32 specific for the most 
part.. other architectures have way too many possible namings for it to be 
useful.)   There is a line in "/etc/rpm/platform" that contains the name of Poky 
architecture.  This file should be referenced (instead of -m) for all cases.

>    3. many archs are missing in zypper, like mips, armeb, etc.
>    4. there is no concept of machine-dependent packages (task-base) in
>       zypper, although we can work around.

Generally speaking, this is true of most RPM installations.  However, within RPM 
itself.. there really isn't any concept of "arch" anymore.. They're really only 
used for grouping and ordering.  So Zypper may need to be updated to query the 
arch of a package and use it for it's various operations.

> Currently, at least zypper is broken on all of mips, arm, ppc, with
> slightly different problems.
>
> The ideal situation is to use consistent arch specification, the
> following can be a solution:
>    1. rename *.all.rpm to *.noarch.rpm

We can certainly do this easily.

>    2. removing the concept of machine-dependent packages, change all
>       *.qemuarm.rpm to *.armv5te.rpm

I'm a bit worried about doing this, as we'll end up with (potentially) 
incompatible packages with exactly the same name and versions...  Perhaps we 
need to think about embedding the machine type into the name of the packages 
instead?

>    3. enhance zypper arch module, make the addition more flexible,
>       allowing arch alias (e.g. armv5te = armv5tel = armel = arm)

Zypper should read the rpm platform file.

> That would be some work to do, maybe 1.0 is a good time to get zypper
> and package upgrade truely working.

Yes, we also need to get multi-arch as well.. (i.e. 32-bit and 64-bit at the 
same time) working.  I'm guessing there will be some Zypper interactions there 
as well.

--Mark

> Any ideas and comments?
>
> Thanks,
> Qing
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.pokylinux.org/listinfo/yocto

Re: zypper and poky architectures

[Qing He]

On Thu, 2010-10-21 at 23:18 +0800, Mark Hatle wrote:
> On 10/21/10 3:33 AM, Qing He wrote:
> >    1. what uses for independent packages is called "noarch", "all" is not
> >       recognized, something depends on update-rc.d won't be installed
> >       because of missing dependency
> 
> We can certainly look into translating "all" to "noarch" post 0.9.  That might 
> make it easier for people coming from the RPM world, to understand what is in 
> the package.
>
> >    1. rename *.all.rpm to *.noarch.rpm
> 
> We can certainly do this easily.

If noarch is universally used in RPM word, I think we should use it.

> 
> >    2. the arch automatic detection system uses "uname -m", thus producing
> >       armv5tejl, which can only be resolved as armv5tel, conflicting with
> >       "armv5te" in rpm
> 
> This is a bug in Zypper.  The machine names should come from somewhere other 
> then uname -m.  (The value of uname -m is very much ia32 specific for the most 
> part.. other architectures have way too many possible namings for it to be 
> useful.)   There is a line in "/etc/rpm/platform" that contains the name of Poky 
> architecture.  This file should be referenced (instead of -m) for all cases.
>
> >    3. enhance zypper arch module, make the addition more flexible,
> >       allowing arch alias (e.g. armv5te = armv5tel = armel = arm)
> 
> Zypper should read the rpm platform file.

Sounds reasonable. After all, zypper is only intended to be a frontend
utility to the lower end package tool. Then we won't need to worry about
alias and different naming, and this detaches zypper from hardware.

> 
> >    3. many archs are missing in zypper, like mips, armeb, etc.
> >    4. there is no concept of machine-dependent packages (task-base) in
> >       zypper, although we can work around.
> 
> Generally speaking, this is true of most RPM installations.  However, within RPM 
> itself.. there really isn't any concept of "arch" anymore.. They're really only 
> used for grouping and ordering.  So Zypper may need to be updated to query the 
> arch of a package and use it for it's various operations.
> 
> >    2. removing the concept of machine-dependent packages, change all
> >       *.qemuarm.rpm to *.armv5te.rpm
> 
> I'm a bit worried about doing this, as we'll end up with (potentially) 
> incompatible packages with exactly the same name and versions...  Perhaps we 
> need to think about embedding the machine type into the name of the packages 
> instead?

Thanks for the info. If we are going for dynamic platform specs, it
doesn't really matter whether we have things like qemuarm or not, does it?

> 
> 
> > That would be some work to do, maybe 1.0 is a good time to get zypper
> > and package upgrade truely working.
> 
> Yes, we also need to get multi-arch as well.. (i.e. 32-bit and 64-bit at the 
> same time) working.  I'm guessing there will be some Zypper interactions there 
> as well.
> 

I don't really have ideas how this is done. I think on debian this is
actually avoided and i386 packages are repackaged as lib32xxx for x86_64
platform.

Thanks,
Qing

Re: zypper and poky architectures

[Mark Hatle]

(Sorry for the late response, today's my first day back from CELF)

On 10/21/10 8:47 PM, Qing He wrote:
> On Thu, 2010-10-21 at 23:18 +0800, Mark Hatle wrote:
>> On 10/21/10 3:33 AM, Qing He wrote:
>>>     1. what uses for independent packages is called "noarch", "all" is not
>>>        recognized, something depends on update-rc.d won't be installed
>>>        because of missing dependency
>>
>> We can certainly look into translating "all" to "noarch" post 0.9.  That might
>> make it easier for people coming from the RPM world, to understand what is in
>> the package.
>>
>>>     1. rename *.all.rpm to *.noarch.rpm
>>
>> We can certainly do this easily.
>
> If noarch is universally used in RPM word, I think we should use it.

My preference is staying with the Poky 'arch' naming...  but renaming to noarch 
is fine, and unless Richard or someone else sees an issue it could be used as a 
temporary workaround.  (There are a few places like rootfs generation that we'll 
have to translate "all" to "noarch".. if we decide to do this.)

>>
>>>     2. the arch automatic detection system uses "uname -m", thus producing
>>>        armv5tejl, which can only be resolved as armv5tel, conflicting with
>>>        "armv5te" in rpm
>>
>> This is a bug in Zypper.  The machine names should come from somewhere other
>> then uname -m.  (The value of uname -m is very much ia32 specific for the most
>> part.. other architectures have way too many possible namings for it to be
>> useful.)   There is a line in "/etc/rpm/platform" that contains the name of Poky
>> architecture.  This file should be referenced (instead of -m) for all cases.
>>
>>>     3. enhance zypper arch module, make the addition more flexible,
>>>        allowing arch alias (e.g. armv5te = armv5tel = armel = arm)
>>
>> Zypper should read the rpm platform file.
>
> Sounds reasonable. After all, zypper is only intended to be a frontend
> utility to the lower end package tool. Then we won't need to worry about
> alias and different naming, and this detaches zypper from hardware.
>
>>
>>>     3. many archs are missing in zypper, like mips, armeb, etc.
>>>     4. there is no concept of machine-dependent packages (task-base) in
>>>        zypper, although we can work around.
>>
>> Generally speaking, this is true of most RPM installations.  However, within RPM
>> itself.. there really isn't any concept of "arch" anymore.. They're really only
>> used for grouping and ordering.  So Zypper may need to be updated to query the
>> arch of a package and use it for it's various operations.
>>
>>>     2. removing the concept of machine-dependent packages, change all
>>>        *.qemuarm.rpm to *.armv5te.rpm
>>
>> I'm a bit worried about doing this, as we'll end up with (potentially)
>> incompatible packages with exactly the same name and versions...  Perhaps we
>> need to think about embedding the machine type into the name of the packages
>> instead?
>
> Thanks for the info. If we are going for dynamic platform specs, it
> doesn't really matter whether we have things like qemuarm or not, does it?

Ya, if we are able to do things dynamically, then the naming is no longer 
important.  That's really my hope as to how we implement the RPM components.

>>
>>
>>> That would be some work to do, maybe 1.0 is a good time to get zypper
>>> and package upgrade truely working.
>>
>> Yes, we also need to get multi-arch as well.. (i.e. 32-bit and 64-bit at the
>> same time) working.  I'm guessing there will be some Zypper interactions there
>> as well.
>>
>
> I don't really have ideas how this is done. I think on debian this is
> actually avoided and i386 packages are repackaged as lib32xxx for x86_64
> platform.

Since Poky does not yet have the ability to deal with Multiarch builds, this is 
something we will have to work on designing as we get closer to Yocto 1.0.

Within RPM, the rpm package manager understands all of the "types" of each file 
in the system.  When you ask to install (note, not upgrade) two packages of the 
same name the system checks the files.

When a conflict is identified, if the contents of the files are the same, 
nothing is done -- no error is generated.

If the contents of the file are different, and the file is tagged as a 
configuration file, then either first or last in wins (I don't remember which) 
-- no error is generated.

If the contents of the file are different, and the file type is NOT ELF (and the 
above has no already detected), then an error is generated and installation stops.

if the contents of the file are different, and the file type is ELF... then 
there is a weighting algorithm that is used.  Depending on the configuration the 
following could happen:

multiarch is not allowed -- an error is generated

multiarch is allowed -- one of the components though is not an allowed multiarch 
-- an error is generated (this could be the mips case of o32, n32 and n64 on the 
same system.  You could prevent someone from installing say o32 binaries.)

multiarch is allowed -- a 'winner' is chosen based on the system configuration. 
  The winner is installed, and the loser is not installed -- no error is generated.

---

If DEB and IPK don't support this (which I wouldn't be surprised if they don't), 
then we'll need to: extend them, modify the package naming to include some type 
of architecture keying to avoid conflicts, or simply state multiarch support 
isn't available if the rootfs type is DEB or IPK.  (I suspect the middle case is 
what we'll end up with.)

--Mark

> Thanks,
> Qing

Re: zypper and poky architectures

[Qing He]

On Mon, 2010-11-01 at 23:37 +0800, Mark Hatle wrote:
> (Sorry for the late response, today's my first day back from CELF)
> 
> On 10/21/10 8:47 PM, Qing He wrote:
> > On Thu, 2010-10-21 at 23:18 +0800, Mark Hatle wrote:
> >> On 10/21/10 3:33 AM, Qing He wrote:
> >>>     1. what uses for independent packages is called "noarch", "all" is not
> >>>        recognized, something depends on update-rc.d won't be installed
> >>>        because of missing dependency
> >>
> >> We can certainly look into translating "all" to "noarch" post 0.9.  That might
> >> make it easier for people coming from the RPM world, to understand what is in
> >> the package.
> >>
> >>>     1. rename *.all.rpm to *.noarch.rpm
> >>
> >> We can certainly do this easily.
> >
> > If noarch is universally used in RPM word, I think we should use it.
> 
> My preference is staying with the Poky 'arch' naming...  but renaming to noarch 
> is fine, and unless Richard or someone else sees an issue it could be used as a 
> temporary workaround.  (There are a few places like rootfs generation that we'll 
> have to translate "all" to "noarch".. if we decide to do this.)

This is good. I may test if this works and let's see if Richard has any
comments on it.

> > Thanks for the info. If we are going for dynamic platform specs, it
> > doesn't really matter whether we have things like qemuarm or not, does it?
> 
> Ya, if we are able to do things dynamically, then the naming is no longer 
> important.  That's really my hope as to how we implement the RPM components.

It seems to be an elegant solution, but need some efforts to find out
how this can fit in current zypper.

> 
> >>
> >>
> >>> That would be some work to do, maybe 1.0 is a good time to get zypper
> >>> and package upgrade truely working.
> >>
> >> Yes, we also need to get multi-arch as well.. (i.e. 32-bit and 64-bit at the
> >> same time) working.  I'm guessing there will be some Zypper interactions there
> >> as well.
> >>
> >
> > I don't really have ideas how this is done. I think on debian this is
> > actually avoided and i386 packages are repackaged as lib32xxx for x86_64
> > platform.
> 
> Since Poky does not yet have the ability to deal with Multiarch builds, this is 
> something we will have to work on designing as we get closer to Yocto 1.0.
> 
> Within RPM, the rpm package manager understands all of the "types" of each file 
> in the system.  When you ask to install (note, not upgrade) two packages of the 
> same name the system checks the files.
> 
> When a conflict is identified, if the contents of the files are the same, 
> nothing is done -- no error is generated.
> 
> If the contents of the file are different, and the file is tagged as a 
> configuration file, then either first or last in wins (I don't remember which) 
> -- no error is generated.
> 
> If the contents of the file are different, and the file type is NOT ELF (and the 
> above has no already detected), then an error is generated and installation stops.
> 
> if the contents of the file are different, and the file type is ELF... then 
> there is a weighting algorithm that is used.  Depending on the configuration the 
> following could happen:
> 
> multiarch is not allowed -- an error is generated
> 
> multiarch is allowed -- one of the components though is not an allowed multiarch 
> -- an error is generated (this could be the mips case of o32, n32 and n64 on the 
> same system.  You could prevent someone from installing say o32 binaries.)
> 
> multiarch is allowed -- a 'winner' is chosen based on the system configuration. 
>   The winner is installed, and the loser is not installed -- no error is generated.
> 

Hmm, this is not quite what I've been thinking about. The problem is
the shared library, and the library path renaming.

The above winner works fine for executables, nobody needs different arch
versions of a same binary, but it's possible that several different
archs are used for different binaries, that's where the library problem
comes in.

Let's say we have an i586 `ls', and an x86_64 `cp' that coexist in the same
box, they would possibly link to two different ld-linux.so and libc.so (this
scenario is common requirement on x86, esp. x86_64). If 32bit rpms can
be installed to 64bit platforms directly without any modification, that
would be great.

I guess the deb way to solve this is to create a special kind of
package, namely `lib32c_2.10.1-r1.x86_64.deb', which installs something
like /lib32/ld-linux-i586.so.2. If the executable can links to it, the
dedicated ld.so cache can get most of its library path right. However,
sadly, this special lib32 and maybe the 32bit executable package, may
not be installable on pure i586 archs.


So should this kind of multiarch be concerned, where multiarch packages
coexist instead of being exlusive?

Thanks,
Qing

Re: zypper and poky architectures

[Mark Hatle]

On 11/3/10 2:58 AM, Qing He wrote:
> On Mon, 2010-11-01 at 23:37 +0800, Mark Hatle wrote:
>> (Sorry for the late response, today's my first day back from CELF)
>>
>> On 10/21/10 8:47 PM, Qing He wrote:
>>> On Thu, 2010-10-21 at 23:18 +0800, Mark Hatle wrote:
>>>> On 10/21/10 3:33 AM, Qing He wrote:
>>>>>      1. what uses for independent packages is called "noarch", "all" is not
>>>>>         recognized, something depends on update-rc.d won't be installed
>>>>>         because of missing dependency
>>>>
>>>> We can certainly look into translating "all" to "noarch" post 0.9.  That might
>>>> make it easier for people coming from the RPM world, to understand what is in
>>>> the package.
>>>>
>>>>>      1. rename *.all.rpm to *.noarch.rpm
>>>>
>>>> We can certainly do this easily.
>>>
>>> If noarch is universally used in RPM word, I think we should use it.
>>
>> My preference is staying with the Poky 'arch' naming...  but renaming to noarch
>> is fine, and unless Richard or someone else sees an issue it could be used as a
>> temporary workaround.  (There are a few places like rootfs generation that we'll
>> have to translate "all" to "noarch".. if we decide to do this.)
>
> This is good. I may test if this works and let's see if Richard has any
> comments on it.
>
>>> Thanks for the info. If we are going for dynamic platform specs, it
>>> doesn't really matter whether we have things like qemuarm or not, does it?
>>
>> Ya, if we are able to do things dynamically, then the naming is no longer
>> important.  That's really my hope as to how we implement the RPM components.
>
> It seems to be an elegant solution, but need some efforts to find out
> how this can fit in current zypper.
>
>>
>>>>
>>>>
>>>>> That would be some work to do, maybe 1.0 is a good time to get zypper
>>>>> and package upgrade truely working.
>>>>
>>>> Yes, we also need to get multi-arch as well.. (i.e. 32-bit and 64-bit at the
>>>> same time) working.  I'm guessing there will be some Zypper interactions there
>>>> as well.
>>>>
>>>
>>> I don't really have ideas how this is done. I think on debian this is
>>> actually avoided and i386 packages are repackaged as lib32xxx for x86_64
>>> platform.
>>
>> Since Poky does not yet have the ability to deal with Multiarch builds, this is
>> something we will have to work on designing as we get closer to Yocto 1.0.
>>
>> Within RPM, the rpm package manager understands all of the "types" of each file
>> in the system.  When you ask to install (note, not upgrade) two packages of the
>> same name the system checks the files.
>>
>> When a conflict is identified, if the contents of the files are the same,
>> nothing is done -- no error is generated.
>>
>> If the contents of the file are different, and the file is tagged as a
>> configuration file, then either first or last in wins (I don't remember which)
>> -- no error is generated.
>>
>> If the contents of the file are different, and the file type is NOT ELF (and the
>> above has no already detected), then an error is generated and installation stops.
>>
>> if the contents of the file are different, and the file type is ELF... then
>> there is a weighting algorithm that is used.  Depending on the configuration the
>> following could happen:
>>
>> multiarch is not allowed -- an error is generated
>>
>> multiarch is allowed -- one of the components though is not an allowed multiarch
>> -- an error is generated (this could be the mips case of o32, n32 and n64 on the
>> same system.  You could prevent someone from installing say o32 binaries.)
>>
>> multiarch is allowed -- a 'winner' is chosen based on the system configuration.
>>    The winner is installed, and the loser is not installed -- no error is generated.
>>
>
> Hmm, this is not quite what I've been thinking about. The problem is
> the shared library, and the library path renaming.
>
> The above winner works fine for executables, nobody needs different arch
> versions of a same binary, but it's possible that several different
> archs are used for different binaries, that's where the library problem
> comes in.

Thats the full purpose of the winner is to deal with conflicts.  Libraries 
should never conflict, if they do it's a bug in the library.

> Let's say we have an i586 `ls', and an x86_64 `cp' that coexist in the same
> box, they would possibly link to two different ld-linux.so and libc.so (this
> scenario is common requirement on x86, esp. x86_64). If 32bit rpms can
> be installed to 64bit platforms directly without any modification, that
> would be great.

The file dependencies that I added in the 0.9 work will resolve this.  Each 
library (file) is tagged with various dependency information.  The information 
includes ELF types and will automatically choose the correct version (or 
versions) of a given library in order to resolve dependencies.   (We'll have to 
double check it's working correctly of course, but the work is already there, 
just not verified.  This is true for all packaging mechanisms...)

> I guess the deb way to solve this is to create a special kind of
> package, namely `lib32c_2.10.1-r1.x86_64.deb', which installs something
> like /lib32/ld-linux-i586.so.2. If the executable can links to it, the
> dedicated ld.so cache can get most of its library path right. However,
> sadly, this special lib32 and maybe the 32bit executable package, may
> not be installable on pure i586 archs.

That is something I would like to avoid as we move toward multiarch support in 
Poky.  I want to be able to re-use packages from any architecture and not 
generate "special" 32-bit versions.  There really should be no reason to do so 
in any packaging method.  (The 32-bit/64-bit executable collision workaround 
really shouldn't be necessary if the rest of the system is done right..)

> So should this kind of multiarch be concerned, where multiarch packages
> coexist instead of being exlusive?

Multiarch need to co-exist.. The three primary cases I'm worried about are:

ia32:  x86_32 & x86_64
ppc:  ppc32 & ppc64
mips: mips_o32 & mips_n32 & mips_n64

I've seen situations in each where the "default" ABI type is different depending 
on customer needs, but generally speaking the defaults become:

ia32: x86_64
ppc: ppc32
mips: mips_o32 or mips_n32

(note the mips case is only for a mips64 compatible processor)

> Thanks,
> Qing
>
>