trying to reconcile OE builds with rpm4-format rpm files built on centos 6

trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Robert P. J. Day]

  looking for advice on how to most cleanly deal with the following.
currently working on BSP layer to which some folks want to add RPMs
that are currently (and for quite some time have been) built on a
centos 6 system, which means those RPM files are in rpm4 format; ergo,
they obviously don't play well with a system built with current OE/YP.
so ... what to do?

  first, current muttering is, "grrrrrr ... why did OE migrate to
rpm5?" i actually didn't really know the answer to that, so i poked
around and found this:

  http://rpm5.org/community/rpm-users/0998.html

the mention of "more flexibility in cross compilation and control"
makes perfect sense to me, but is there a more comprehensive writeup
somewhere that lays out the rationale for the move to rpm5 that can be
used in its defense?

  next bit of muttering is, "can we downgrade the OE build to use
rpm4-format packages?", which is not a path down which i want to walk.

  another option is to simply install a totally independent rpm5 on
the centos 6 box, and use that exclusively for building packages to be
installed on an OE system. has anyone done this? does it represent a
sane/reasonable approach?

  finally, my reaction to all of this is, "why not just write recipes
for all that software so it can be built by OE?" but, as explained to
me, the OE package build system is *heavily* tied to a much larger
internal build process that resides on the centos 6 box, and there is
a real reluctance to try to extract the OE component from the larger
build process. the phrase that seems to pop up is, "not a chance in
hell."

  so ... thoughts? short of ripping out that part of the build process
and properly adding it as additional recipes to the OE build, is it
possible to build and install rpm5 on a centos 6 box? as a test, i can
try it for my fedora 23 system and, if that works, at least i can
demonstrate proof-of-concept.

  anyone done this? if it's possible, it would seem to be the simplest
solution under the circumstances.

rday

p.s. does adding the smart package manager into all of this make any
difference? i'm pretty familiar with RPM packaging, but have barely
looked at smart, and i don't know whether it would have any relevance
to this issue.

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca> wrote:

>   next bit of muttering is, "can we downgrade the OE build to use
> rpm4-format packages?", which is not a path down which i want to walk.
>

Assuming that the obviously correct option of "build the packages inside
OE" really is being written off for mysterious reasons, rpm4 was only just
removed from oe-core (though depending on what releases you're using you
may have never noticed it be added and removed again).  So you could just
recover that from history (oe-core a6e7a86f1635be9a688c56c25e9d215ea4d2cc84
removed it) and fix it up.

Ross

[-- Attachment #2: Type: text/html, Size: 1136 bytes --]

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

On Thu, 21 Apr 2016, Burton, Ross wrote:

>
> On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>         next bit of muttering is, "can we downgrade the OE build to use
>       rpm4-format packages?", which is not a path down which i want to walk.
>
>
> Assuming that the obviously correct option of "build the packages
> inside OE" really is being written off for mysterious reasons, rpm4
> was only just removed from oe-core (though depending on what
> releases you're using you may have never noticed it be added and
> removed again).  So you could just recover that from history
> (oe-core a6e7a86f1635be9a688c56c25e9d215ea4d2cc84 removed it) and
> fix it up.

  under the circumstances, that sounds like the simplest approach.
would we be losing any significant functionality downgrading to rpm4?
it's not my first choice, but if all that's required is to do basic
installs and upgrades, i suspect it will work just fine.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Joshua G Lock]

On Thu, 2016-04-21 at 08:24 -0400, Robert P. J. Day wrote:
> On Thu, 21 Apr 2016, Burton, Ross wrote:
> > 
> > On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca>
> > wrote:
> >         next bit of muttering is, "can we downgrade the OE build to
> > use
> >       rpm4-format packages?", which is not a path down which i want
> > to walk.
> > 
> > 
> > Assuming that the obviously correct option of "build the packages
> > inside OE" really is being written off for mysterious reasons, rpm4
> > was only just removed from oe-core (though depending on what
> > releases you're using you may have never noticed it be added and
> > removed again).  So you could just recover that from history
> > (oe-core a6e7a86f1635be9a688c56c25e9d215ea4d2cc84 removed it) and
> > fix it up.
>   under the circumstances, that sounds like the simplest approach.
> would we be losing any significant functionality downgrading to rpm4?
> it's not my first choice, but if all that's required is to do basic
> installs and upgrades, i suspect it will work just fine.

On the topic of RPM4 vs. RPM5 you should be able to find more details
in the list archives, I easily found a brief summary by Mark Hatle in
the Yocto Project mailing list archive:

"There are some specific uses of RPM 4 in the YP, but I do caution
against people just using it "because".  The RPM 5 version is generally
better suited for the embedded world.  (There are been posts on more
reasons on the oe-core lists in the past.  But as quick summary --
dynamic architecture support, better cross compilation support, cross-
endian support, more configurable for custom distributions, etc.)"
http://article.gmane.org/gmane.linux.embedded.yocto.general/28654/

We ended up removing RPM4 support because it was clear that it was
broken when using SMART as a package manager and we didn't have the
resources to fix it, nor any objections to its removal.

Some recent examples of the kinds of breakage:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8968
https://bugzilla.yoctoproject.org/show_bug.cgi?id=8969

Regards,

Joshua

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Robert P. J. Day]

On Thu, 21 Apr 2016, Joshua G Lock wrote:

... snip ...

> On the topic of RPM4 vs. RPM5 you should be able to find more
> details in the list archives, I easily found a brief summary by Mark
> Hatle in the Yocto Project mailing list archive:
>
> "There are some specific uses of RPM 4 in the YP, but I do caution
> against people just using it "because".  The RPM 5 version is
> generally better suited for the embedded world.  (There are been
> posts on more reasons on the oe-core lists in the past.  But as
> quick summary -- dynamic architecture support, better cross
> compilation support, cross- endian support, more configurable for
> custom distributions, etc.)"
> http://article.gmane.org/gmane.linux.embedded.yocto.general/28654/
>
> We ended up removing RPM4 support because it was clear that it was
> broken when using SMART as a package manager and we didn't have the
> resources to fix it, nor any objections to its removal.

  ah, i was unaware that smart does not play well with RPM4 ... i
think i'd better collect all this info so i can present it in one
shot. i think i'm going to look into the feasibility of installing
RPM5 on that centos build server, unless someone cautions me against
it.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Robert P. J. Day]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

On Thu, 21 Apr 2016, Burton, Ross wrote:

>
> On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca> wrote:
>         next bit of muttering is, "can we downgrade the OE build to use
>       rpm4-format packages?", which is not a path down which i want to walk.
>
>
> Assuming that the obviously correct option of "build the packages
> inside OE" really is being written off for mysterious reasons, rpm4
> was only just removed from oe-core (though depending on what
> releases you're using you may have never noticed it be added and
> removed again).  So you could just recover that from history
> (oe-core a6e7a86f1635be9a688c56c25e9d215ea4d2cc84 removed it) and
> fix it up.

  just to be clear, if i can dredge up the recipe for rpm_4, i'm
assuming i'd want to specify that i want the "package-management"
image feature, as well as stating:

PREFERRED_VERSION_rpm = "4.%"
PREFERRED_VERSION_rpm-native = "4.%"

correct?

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Richard Purdie]

On Thu, 2016-04-21 at 08:50 -0400, Robert P. J. Day wrote:
> On Thu, 21 Apr 2016, Burton, Ross wrote:
> 
> > 
> > On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca>
> > wrote:
> >         next bit of muttering is, "can we downgrade the OE build to
> > use
> >       rpm4-format packages?", which is not a path down which i want
> > to walk.
> > 
> > 
> > Assuming that the obviously correct option of "build the packages
> > inside OE" really is being written off for mysterious reasons, rpm4
> > was only just removed from oe-core (though depending on what
> > releases you're using you may have never noticed it be added and
> > removed again).  So you could just recover that from history
> > (oe-core a6e7a86f1635be9a688c56c25e9d215ea4d2cc84 removed it) and
> > fix it up.
> 
>   just to be clear, if i can dredge up the recipe for rpm_4, i'm
> assuming i'd want to specify that i want the "package-management"
> image feature, as well as stating:
> 
> PREFERRED_VERSION_rpm = "4.%"
> PREFERRED_VERSION_rpm-native = "4.%"
> 
> correct?

Just to confuse things further, you could write an OE recipe which took
the v4 rpm files from the other system and then simply repackaged them
into v5 rpms files. Nothing says you *must* compile from source, the
input could be the v4 rpms.

Cheers,

Richard

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Robert P. J. Day]

On Thu, 21 Apr 2016, Richard Purdie wrote:

> On Thu, 2016-04-21 at 08:50 -0400, Robert P. J. Day wrote:
> > On Thu, 21 Apr 2016, Burton, Ross wrote:
> >
> > >
> > > On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca>
> > > wrote:
> > >         next bit of muttering is, "can we downgrade the OE build to
> > > use
> > >       rpm4-format packages?", which is not a path down which i want
> > > to walk.
> > >
> > >
> > > Assuming that the obviously correct option of "build the
> > > packages inside OE" really is being written off for mysterious
> > > reasons, rpm4 was only just removed from oe-core (though
> > > depending on what releases you're using you may have never
> > > noticed it be added and removed again).  So you could just
> > > recover that from history (oe-core
> > > a6e7a86f1635be9a688c56c25e9d215ea4d2cc84 removed it) and fix it
> > > up.
> >
> >   just to be clear, if i can dredge up the recipe for rpm_4, i'm
> > assuming i'd want to specify that i want the "package-management"
> > image feature, as well as stating:
> >
> > PREFERRED_VERSION_rpm = "4.%"
> > PREFERRED_VERSION_rpm-native = "4.%"
> >
> > correct?
>
> Just to confuse things further, you could write an OE recipe which
> took the v4 rpm files from the other system and then simply
> repackaged them into v5 rpms files. Nothing says you *must* compile
> from source, the input could be the v4 rpms.

  great, just what i needed ... yet *another* strategy. in any event,
can i confirm that if i have the recipe for RPM4, i can use
PREFERRED_VERSION to use it for the OE build and, later, to install
RPM4-format rpms built elsewhere?

  also, *if* i build an image based on RPM4, is it feasible (or even
possible) to upgrade the whole thing to RPM5 later? i'm not sure i
even want to think about the grief possibly involved in that.

  thanks muchly.

rday

-- 

========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                        http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Mark Hatle]

On 4/21/16 9:25 AM, Robert P. J. Day wrote:
> On Thu, 21 Apr 2016, Richard Purdie wrote:
> 
>> On Thu, 2016-04-21 at 08:50 -0400, Robert P. J. Day wrote:
>>> On Thu, 21 Apr 2016, Burton, Ross wrote:
>>>
>>>>
>>>> On 21 April 2016 at 13:06, Robert P. J. Day <rpjday@crashcourse.ca>
>>>> wrote:
>>>>         next bit of muttering is, "can we downgrade the OE build to
>>>> use
>>>>       rpm4-format packages?", which is not a path down which i want
>>>> to walk.
>>>>
>>>>
>>>> Assuming that the obviously correct option of "build the
>>>> packages inside OE" really is being written off for mysterious
>>>> reasons, rpm4 was only just removed from oe-core (though
>>>> depending on what releases you're using you may have never
>>>> noticed it be added and removed again).  So you could just
>>>> recover that from history (oe-core
>>>> a6e7a86f1635be9a688c56c25e9d215ea4d2cc84 removed it) and fix it
>>>> up.
>>>
>>>   just to be clear, if i can dredge up the recipe for rpm_4, i'm
>>> assuming i'd want to specify that i want the "package-management"
>>> image feature, as well as stating:
>>>
>>> PREFERRED_VERSION_rpm = "4.%"
>>> PREFERRED_VERSION_rpm-native = "4.%"
>>>
>>> correct?
>>
>> Just to confuse things further, you could write an OE recipe which
>> took the v4 rpm files from the other system and then simply
>> repackaged them into v5 rpms files. Nothing says you *must* compile
>> from source, the input could be the v4 rpms.
> 
>   great, just what i needed ... yet *another* strategy. in any event,
> can i confirm that if i have the recipe for RPM4, i can use
> PREFERRED_VERSION to use it for the OE build and, later, to install
> RPM4-format rpms built elsewhere?
> 
>   also, *if* i build an image based on RPM4, is it feasible (or even
> possible) to upgrade the whole thing to RPM5 later? i'm not sure i
> even want to think about the grief possibly involved in that.

Due to potential endian and header differences, I don't know.

It was certainly possible in the past to do this (and easy).  But at this point
it may no longer work.

I think this is a question that probably should go to the rpm5-users mailing
list, as they may have more information.

--Mark

>   thanks muchly.
> 
> rday
> 

Re: trying to reconcile OE builds with rpm4-format rpm files built on centos 6

[Mark Hatle]

On 4/21/16 7:06 AM, Robert P. J. Day wrote:
> 
>   looking for advice on how to most cleanly deal with the following.
> currently working on BSP layer to which some folks want to add RPMs
> that are currently (and for quite some time have been) built on a
> centos 6 system, which means those RPM files are in rpm4 format; ergo,
> they obviously don't play well with a system built with current OE/YP.
> so ... what to do?

What is not working about the packages build with RPM 4?  The RPM format is
compatible between RPM 4 and RPM 5.

There are specific configurations that may or may not be used in the RPM 4
compilations, that may need to be enabled in RPM 5.  Things like compression
format for instance.

I need to see the errors that are encountered to give a clue as to what may be
wrong.

>   first, current muttering is, "grrrrrr ... why did OE migrate to
> rpm5?" i actually didn't really know the answer to that, so i poked
> around and found this:
> 
>   http://rpm5.org/community/rpm-users/0998.html

It's been a while, but there was a thread a while back on 4 vs 5 on the OE list,
and there was a mini thread on the YP list a few days ago.

Unfortunately I am not finding the email.

I'll quote from another email I've sent (this is NOT the one I'm referring to BTW).

Comparing RPM 4 to RPM 5 (circa May 2014):

HHGG:
* Don't panic!  RPM Packages are compatible between RPM 4 and RPM 5.

Background:

* RPM 4 and RPM 5 split a number of years ago.
   Both still have their purposes.

- RPM 4:
   * License: GPLv2+
   * Uses beecrypt and nss for encryption
   * Maintained and supported by Fedora(Red Hat)

- RPM 5:
   * License: LGPLv2.1
   * Uses beecrypt, and (nss or openssl or others) for encryption
     - Can't mix OpenSSL w/ GPLv2 due to license issue
   * Maintained and supported by Jeff Johnson

Differences:

* RPM 4 vs RPM 5
RPM 4 is missing the following features:

- Cross compile support / Cross endian support
  ** rpmbuild v4 is intended for native builds only

The RPM data structures should be little endian for the most part, but
historically RPM 4 did not transform everything to little endian.  This caused
problems when a package was built on x86 and installed on Power.  The berkleydb
itself was susceptible to this -- RPM5 does proper endian translations.

- "no arch"  / "not arch" support

In RPM4, there is a set list of 'compatible' and 'known' architectures.  If your
package 'arch' is not in this list, everything fails.

In RPM5, there is no set list.  Instead, there is a configuration of compatible
platforms (thats the /etc/rpm/platforms file).  The package arch is not used for
compatibility, instead there is a package platform that is inspected and
compared.  (Note, RPM4 also sets the platform, it just doesn't use it in this way.)

- Support for arbitrary tags

Packages can specify arbitrary information to include in the header and be
retrieved during install and inspection.  There tags are used by OpenEmbedded
for the 'Suggests' and 'Enhances' fields.  (While rarely used, we do support the
values.  In addition, I know of a few people who use the arbitrary tags to add
additional support information into the RPM packages themselves.  This is
outside the normal OE usage though.)

- RPM 5 has supported the 'Recommends' field for a long time, only recently has
RPM 4 added it -- and it was added in an incompatible way.

RPM 5 puts recommended items into the 'requires' field, and adds a 'MISSINGOK' flag.

RPM 4 defined a new tag for Recommended items.

(Both RPM 5 and 4 support each others implementations)

- RPM 5 maintainer actively cares about OE and YP usages.

You may not see Jeff on the mailing list, but I talk with him regularly and
discuss the problems and enhancements we have.  He will work through the issues
with me (and other YP folks) to try to come up with a solution that fits into
the RPM framework without creating new incompatibilities.  This is a huge help
for us.

- RPM 5 does not have a plug-in interface like RPM 4

This is the one place where I've seen discussions for people demanding RPM 4.
They need specific IMA, Smack, Selinux, etc plugins.  Frankly I think trusting
any system the size and complexity of RPM with that responsibility is
incorrect... but from a compatibility standpoint that may be one of the few
reasons to justify RPM 4 at present.

> the mention of "more flexibility in cross compilation and control"
> makes perfect sense to me, but is there a more comprehensive writeup
> somewhere that lays out the rationale for the move to rpm5 that can be
> used in its defense?
> 
>   next bit of muttering is, "can we downgrade the OE build to use
> rpm4-format packages?", which is not a path down which i want to walk.

If you want package compatibility, this simply should not be necessary.

>   another option is to simply install a totally independent rpm5 on
> the centos 6 box, and use that exclusively for building packages to be
> installed on an OE system. has anyone done this? does it represent a
> sane/reasonable approach?

Yes this works, I've done it as have others.

>   finally, my reaction to all of this is, "why not just write recipes
> for all that software so it can be built by OE?" but, as explained to
> me, the OE package build system is *heavily* tied to a much larger
> internal build process that resides on the centos 6 box, and there is
> a real reluctance to try to extract the OE component from the larger
> build process. the phrase that seems to pop up is, "not a chance in
> hell."
> 
>   so ... thoughts? short of ripping out that part of the build process
> and properly adding it as additional recipes to the OE build, is it
> possible to build and install rpm5 on a centos 6 box? as a test, i can
> try it for my fedora 23 system and, if that works, at least i can
> demonstrate proof-of-concept.
> 
>   anyone done this? if it's possible, it would seem to be the simplest
> solution under the circumstances.

At ELC a person from Fujitsu demonstrated using Smart from inside of the SDK.
The reason I bring this up, if smart works -- frankly rpmbuild should work as
well (properly configured/setup).

It should be possible, but I doubt it's been well tested, to produce an SDK that
includes rpm/rpmbuild in it and macro files that match the behaviors we expect.

On the target itself, you can run rpmbuild and build many packages.  (This falls
down a bit when there is a distribution specific macro file required -- however,
often you can just make the macro available and it works.)

> rday
> 
> p.s. does adding the smart package manager into all of this make any
> difference? i'm pretty familiar with RPM packaging, but have barely
> looked at smart, and i don't know whether it would have any relevance
> to this issue.
> 

It should not have any effect.

--Mark