RPM vs IPK

RPM vs IPK

[Gary Thomas]

Building Poky for various targets, I see some striking differences
based on the packaging.  I'm building for the beagleboard (RPM)
and my own OMAP/3530 (IPK), so everything is the same for these
packages (same compiler, architecture, etc), only the package
method differs.  This was built on an otherwise idle box
4-way (Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz), with
   BB_NUMBER_THREADS ?= "4"
   PARALLEL_MAKE ?= "-j 4"

Each of these tests are a complete build of the package, with
all dependencies already built.  For example, I use this sequence:
   % bitbake perl
   % bitbake perl -c clean
   % rm sstate-cache/sstate-perl-arm*
   % time bitbake perl

perl -      RPM                         IPK
        real    12m15.520s          real    9m43.228s
        user    5m42.988s           user    4m40.692s
        sys     3m56.636s           sys     2m19.860s

eglibc     RPM                          IPK
        real    32m19.984s          real    23m52.124s
        user    15m32.732s          user    20m48.214s
        sys     17m28.087s          sys     9m3.936s

Bottom line - it seems to take 20-30% longer to package via RPM.

I know there are reasons and tradeoffs for different packaging
methods, but 30% extra?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RPM vs IPK

[Mark Hatle]

On 5/19/11 9:05 AM, Gary Thomas wrote:
> Building Poky for various targets, I see some striking differences
> based on the packaging.  I'm building for the beagleboard (RPM)
> and my own OMAP/3530 (IPK), so everything is the same for these
> packages (same compiler, architecture, etc), only the package
> method differs.  This was built on an otherwise idle box
> 4-way (Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz), with
>    BB_NUMBER_THREADS ?= "4"
>    PARALLEL_MAKE ?= "-j 4"
> 
> Each of these tests are a complete build of the package, with
> all dependencies already built.  For example, I use this sequence:
>    % bitbake perl
>    % bitbake perl -c clean
>    % rm sstate-cache/sstate-perl-arm*
>    % time bitbake perl
> 
> perl -      RPM                         IPK
>         real    12m15.520s          real    9m43.228s
>         user    5m42.988s           user    4m40.692s
>         sys     3m56.636s           sys     2m19.860s
> 
> eglibc     RPM                          IPK
>         real    32m19.984s          real    23m52.124s
>         user    15m32.732s          user    20m48.214s
>         sys     17m28.087s          sys     9m3.936s
> 
> Bottom line - it seems to take 20-30% longer to package via RPM.
> 
> I know there are reasons and tradeoffs for different packaging
> methods, but 30% extra?
> 

This matches my expectations for the most part.  RPM creates and processes a lot
more metadata then IPK.  IPK is well optimized for small systems, and certainly
I would advise someone generating a small system to use IPK.

For medium and (especially) large systems, RPM starts to give more abilities
that IPK due to the additional meta data.  This includes individual file type
information, file checksum generation and evaluation on install, sparse file
support, conflict detection and resolution [for multilib systems], ACID style
upgrade, repackaging abilities for rollback, etc..  (Some of the items for RPM
are not yet enabled in oe-core, but could be by individual developers.)

(I also wouldn't be surprised if we've got integration optimizations in the RPM
backend in oe-core that could help with those numbers.  The numbers seem to be
much worse on packages with large numbers of files, vs the typical package with
only a few to a hundred or so files.)

Downside of RPM for small systems is the Berkley DB and the amount of meta data.
 If you will be doing on-device upgrades the space required to handle the
berkley DB and related items is quite large..  Also many of the advanced
features available in RPM simple are not needed in small systems.

--Mark

Re: RPM vs IPK

[Gary Thomas]

On 05/19/2011 08:17 AM, Mark Hatle wrote:
> On 5/19/11 9:05 AM, Gary Thomas wrote:
>> Building Poky for various targets, I see some striking differences
>> based on the packaging.  I'm building for the beagleboard (RPM)
>> and my own OMAP/3530 (IPK), so everything is the same for these
>> packages (same compiler, architecture, etc), only the package
>> method differs.  This was built on an otherwise idle box
>> 4-way (Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz), with
>>     BB_NUMBER_THREADS ?= "4"
>>     PARALLEL_MAKE ?= "-j 4"
>>
>> Each of these tests are a complete build of the package, with
>> all dependencies already built.  For example, I use this sequence:
>>     % bitbake perl
>>     % bitbake perl -c clean
>>     % rm sstate-cache/sstate-perl-arm*
>>     % time bitbake perl
>>
>> perl -      RPM                         IPK
>>          real    12m15.520s          real    9m43.228s
>>          user    5m42.988s           user    4m40.692s
>>          sys     3m56.636s           sys     2m19.860s
>>
>> eglibc     RPM                          IPK
>>          real    32m19.984s          real    23m52.124s
>>          user    15m32.732s          user    20m48.214s
>>          sys     17m28.087s          sys     9m3.936s
>>
>> Bottom line - it seems to take 20-30% longer to package via RPM.
>>
>> I know there are reasons and tradeoffs for different packaging
>> methods, but 30% extra?
>>
>
> This matches my expectations for the most part.  RPM creates and processes a lot
> more metadata then IPK.  IPK is well optimized for small systems, and certainly
> I would advise someone generating a small system to use IPK.
>
> For medium and (especially) large systems, RPM starts to give more abilities
> that IPK due to the additional meta data.  This includes individual file type
> information, file checksum generation and evaluation on install, sparse file
> support, conflict detection and resolution [for multilib systems], ACID style
> upgrade, repackaging abilities for rollback, etc..  (Some of the items for RPM
> are not yet enabled in oe-core, but could be by individual developers.)
>
> (I also wouldn't be surprised if we've got integration optimizations in the RPM
> backend in oe-core that could help with those numbers.  The numbers seem to be
> much worse on packages with large numbers of files, vs the typical package with
> only a few to a hundred or so files.)
>
> Downside of RPM for small systems is the Berkley DB and the amount of meta data.
>   If you will be doing on-device upgrades the space required to handle the
> berkley DB and related items is quite large..  Also many of the advanced
> features available in RPM simple are not needed in small systems.

Fair enough.  Perhaps the packaging type should be chosen based on
the target, e.g. the BeagleBoard probably would be best served by IPK.

Certainly, these data/tradeoffs should be made clear somewhere in the documentation.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RPM vs IPK

[Stewart, David C]

> From: poky-bounces@yoctoproject.org [mailto:poky-
> bounces@yoctoproject.org] On Behalf Of Gary Thomas
> Sent: Thursday, May 19, 2011 7:26 AM
> 
> On 05/19/2011 08:17 AM, Mark Hatle wrote:
> > On 5/19/11 9:05 AM, Gary Thomas wrote:
> >> Building Poky for various targets, I see some striking differences
> >> based on the packaging.  I'm building for the beagleboard (RPM) and
> >> my own OMAP/3530 (IPK), so everything is the same for these packages
> >> (same compiler, architecture, etc), only the package method differs.
> >> This was built on an otherwise idle box
> >> 4-way (Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz), with
> >>     BB_NUMBER_THREADS ?= "4"
> >>     PARALLEL_MAKE ?= "-j 4"
> >>
> >> Each of these tests are a complete build of the package, with all
> >> dependencies already built.  For example, I use this sequence:
> >>     % bitbake perl
> >>     % bitbake perl -c clean
> >>     % rm sstate-cache/sstate-perl-arm*
> >>     % time bitbake perl
> >>
> >> perl -      RPM                         IPK
> >>          real    12m15.520s          real    9m43.228s
> >>          user    5m42.988s           user    4m40.692s
> >>          sys     3m56.636s           sys     2m19.860s
> >>
> >> eglibc     RPM                          IPK
> >>          real    32m19.984s          real    23m52.124s
> >>          user    15m32.732s          user    20m48.214s
> >>          sys     17m28.087s          sys     9m3.936s
> >>
> >> Bottom line - it seems to take 20-30% longer to package via RPM.
> >>
> >> I know there are reasons and tradeoffs for different packaging
> >> methods, but 30% extra?
> >>
> >
> > This matches my expectations for the most part.  RPM creates and
> > processes a lot more metadata then IPK.  IPK is well optimized for
> > small systems, and certainly I would advise someone generating a small
> system to use IPK.
> >
> > For medium and (especially) large systems, RPM starts to give more
> > abilities that IPK due to the additional meta data.  This includes
> > individual file type information, file checksum generation and
> > evaluation on install, sparse file support, conflict detection and
> > resolution [for multilib systems], ACID style upgrade, repackaging
> > abilities for rollback, etc..  (Some of the items for RPM are not yet
> > enabled in oe-core, but could be by individual developers.)
> >
> > (I also wouldn't be surprised if we've got integration optimizations
> > in the RPM backend in oe-core that could help with those numbers.  The
> > numbers seem to be much worse on packages with large numbers of files,
> > vs the typical package with only a few to a hundred or so files.)
> >
> > Downside of RPM for small systems is the Berkley DB and the amount of
> meta data.
> >   If you will be doing on-device upgrades the space required to handle
> > the berkley DB and related items is quite large..  Also many of the
> > advanced features available in RPM simple are not needed in small
> systems.
> 
> Fair enough.  Perhaps the packaging type should be chosen based on the
> target, e.g. the BeagleBoard probably would be best served by IPK.
> 
> Certainly, these data/tradeoffs should be made clear somewhere in the
> documentation.

I agree - hey Gary, can you please file a Bugzilla to add this to the Reference Manual?  Thanks...

Re: RPM vs IPK

[Gary Thomas]

On 05/19/2011 09:28 AM, Stewart, David C wrote:
>
>> From: poky-bounces@yoctoproject.org [mailto:poky-
>> bounces@yoctoproject.org] On Behalf Of Gary Thomas
>> Sent: Thursday, May 19, 2011 7:26 AM
>>
>> On 05/19/2011 08:17 AM, Mark Hatle wrote:
>>> On 5/19/11 9:05 AM, Gary Thomas wrote:
>>>> Building Poky for various targets, I see some striking differences
>>>> based on the packaging.  I'm building for the beagleboard (RPM) and
>>>> my own OMAP/3530 (IPK), so everything is the same for these packages
>>>> (same compiler, architecture, etc), only the package method differs.
>>>> This was built on an otherwise idle box
>>>> 4-way (Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz), with
>>>>      BB_NUMBER_THREADS ?= "4"
>>>>      PARALLEL_MAKE ?= "-j 4"
>>>>
>>>> Each of these tests are a complete build of the package, with all
>>>> dependencies already built.  For example, I use this sequence:
>>>>      % bitbake perl
>>>>      % bitbake perl -c clean
>>>>      % rm sstate-cache/sstate-perl-arm*
>>>>      % time bitbake perl
>>>>
>>>> perl -      RPM                         IPK
>>>>           real    12m15.520s          real    9m43.228s
>>>>           user    5m42.988s           user    4m40.692s
>>>>           sys     3m56.636s           sys     2m19.860s
>>>>
>>>> eglibc     RPM                          IPK
>>>>           real    32m19.984s          real    23m52.124s
>>>>           user    15m32.732s          user    20m48.214s
>>>>           sys     17m28.087s          sys     9m3.936s
>>>>
>>>> Bottom line - it seems to take 20-30% longer to package via RPM.
>>>>
>>>> I know there are reasons and tradeoffs for different packaging
>>>> methods, but 30% extra?
>>>>
>>>
>>> This matches my expectations for the most part.  RPM creates and
>>> processes a lot more metadata then IPK.  IPK is well optimized for
>>> small systems, and certainly I would advise someone generating a small
>> system to use IPK.
>>>
>>> For medium and (especially) large systems, RPM starts to give more
>>> abilities that IPK due to the additional meta data.  This includes
>>> individual file type information, file checksum generation and
>>> evaluation on install, sparse file support, conflict detection and
>>> resolution [for multilib systems], ACID style upgrade, repackaging
>>> abilities for rollback, etc..  (Some of the items for RPM are not yet
>>> enabled in oe-core, but could be by individual developers.)
>>>
>>> (I also wouldn't be surprised if we've got integration optimizations
>>> in the RPM backend in oe-core that could help with those numbers.  The
>>> numbers seem to be much worse on packages with large numbers of files,
>>> vs the typical package with only a few to a hundred or so files.)
>>>
>>> Downside of RPM for small systems is the Berkley DB and the amount of
>> meta data.
>>>    If you will be doing on-device upgrades the space required to handle
>>> the berkley DB and related items is quite large..  Also many of the
>>> advanced features available in RPM simple are not needed in small
>> systems.
>>
>> Fair enough.  Perhaps the packaging type should be chosen based on the
>> target, e.g. the BeagleBoard probably would be best served by IPK.
>>
>> Certainly, these data/tradeoffs should be made clear somewhere in the
>> documentation.
>
> I agree - hey Gary, can you please file a Bugzilla to add this to the Reference Manual?  Thanks...

Done - bug #1088

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

RPM vs IPK

[Gary Thomas]

I know that historically Poky has used 'ipk' as the primary packaging
mechanism.  It seems that now Poky/Yocto has move to 'rpm'.  My distribution
is still using ipk, but I'm happy to change, given a good argument.

* Is there such [a good reason] to use rpm over ipk?
* What are the pros and cons?  I'm mostly interested in very resource limited
   deeply embedded systems which often only run from FLASH.

Thanks for any comments

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RPM vs IPK

[Richard Purdie]

On Sun, 2011-03-20 at 19:58 -0600, Gary Thomas wrote:
> I know that historically Poky has used 'ipk' as the primary packaging
> mechanism.  It seems that now Poky/Yocto has move to 'rpm'.  My distribution
> is still using ipk, but I'm happy to change, given a good argument.
> 
> * Is there such [a good reason] to use rpm over ipk?
> * What are the pros and cons?  I'm mostly interested in very resource limited
>    deeply embedded systems which often only run from FLASH.
> 
> Thanks for any comments

My advice is that for such a resource limited system, you're probably
best of sticking to ipk, particularly if you have it working already.

opkg:

* Has a smaller disk footprint
* Is generally faster than rpm

rpm+zypper:

* More of an industry standard
* Emphasises correctness and robustness over speed (e.g. number of 
  fsync calls)
* Has desktop/enterprise features
* Not optimised for size (e.g. uses c++)

I'd not say one was better than the other, they're just different and
suit different use cases.

Cheers,

Richard

Re: RPM vs IPK

[Gary Thomas]

On 03/21/2011 05:57 AM, Richard Purdie wrote:
> On Sun, 2011-03-20 at 19:58 -0600, Gary Thomas wrote:
>> I know that historically Poky has used 'ipk' as the primary packaging
>> mechanism.  It seems that now Poky/Yocto has move to 'rpm'.  My distribution
>> is still using ipk, but I'm happy to change, given a good argument.
>>
>> * Is there such [a good reason] to use rpm over ipk?
>> * What are the pros and cons?  I'm mostly interested in very resource limited
>>     deeply embedded systems which often only run from FLASH.
>>
>> Thanks for any comments
>
> My advice is that for such a resource limited system, you're probably
> best of sticking to ipk, particularly if you have it working already.
>
> opkg:
>
> * Has a smaller disk footprint
> * Is generally faster than rpm
>
> rpm+zypper:
>
> * More of an industry standard
> * Emphasises correctness and robustness over speed (e.g. number of
>    fsync calls)

Does this mean ipk/opkg fails along these lines in any way?

> * Has desktop/enterprise features

Such as?

> * Not optimised for size (e.g. uses c++)
>
> I'd not say one was better than the other, they're just different and
> suit different use cases.

Pretty much what I thought, thanks.

My only concern is that if rpm is the primary emphasis, ipk/opkg might
suffer from rot.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RPM vs IPK

[Richard Purdie]

On Mon, 2011-03-21 at 08:02 -0600, Gary Thomas wrote:
> On 03/21/2011 05:57 AM, Richard Purdie wrote:
> > rpm+zypper:
> >
> > * More of an industry standard
> > * Emphasises correctness and robustness over speed (e.g. number of
> >    fsync calls)
> 
> Does this mean ipk/opkg fails along these lines in any way?

The opkg source code has certainly not been audited as thoroughly as the
rpm codebase for those kind of issues. I'd say the two codebases each
have their own set of problems though :).

> > * Has desktop/enterprise features

Off the top of my head, the ones I'm aware of are:

Possibility of DeltaRPM updates
Various more advanced dependency calculations (which we're adding for
the benefit of all packaging backends in Yocto) 
   * per-file dependenies
   * perl/python module dependencies
   * directory ownership
Multilib support
Tighter integration with features like prelink.

I'm sure some of the more rpm literate people on the list can detail
others.

> > * Not optimised for size (e.g. uses c++)
> >
> > I'd not say one was better than the other, they're just different and
> > suit different use cases.
> 
> Pretty much what I thought, thanks.
> 
> My only concern is that if rpm is the primary emphasis, ipk/opkg might
> suffer from rot.

I want to see opkg continue to work well, if you do see any problems
with it, let us know...

Cheers,

Richard

Re: RPM vs IPK

[Mark Hatle]

On 3/21/11 9:42 AM, Richard Purdie wrote:
> On Mon, 2011-03-21 at 08:02 -0600, Gary Thomas wrote:
>> On 03/21/2011 05:57 AM, Richard Purdie wrote:
>>> rpm+zypper:
>>>
>>> * More of an industry standard
>>> * Emphasises correctness and robustness over speed (e.g. number of
>>>    fsync calls)
>>
>> Does this mean ipk/opkg fails along these lines in any way?
> 
> The opkg source code has certainly not been audited as thoroughly as the
> rpm codebase for those kind of issues. I'd say the two codebases each
> have their own set of problems though :).

From my recent experience with opkg, it appears that it's a very good packaging
system for small to medium sized devices.  It's small, efficient, but may not
have all of the features needed in desktop/enterprise type designs.

I see RPM has very useful in medium to very large installations...

These two are very complementary within Yocto, and I expect it to stay that way.

>>> * Has desktop/enterprise features
> 
> Off the top of my head, the ones I'm aware of are:
> 
> Possibility of DeltaRPM updates
> Various more advanced dependency calculations (which we're adding for
> the benefit of all packaging backends in Yocto) 
>    * per-file dependenies

This is something we enabled internally for all of the system to help develop
tools overtime to discover and display dependency maps of not only recipes, or
(run-time) packages, but of the files themselves.

>    * perl/python module dependencies
>    * directory ownership

RPM currently has directory ownership, but we do not use this in Yocto properly.
 This will be something we'll hopefully be added in the near future.  (Directory
ownership means specific package(s) and users/group/permissions are defined for
packages.  By default in most OE based systems it appears that the directories
just end up being a by-product of the installation of files.. and the default
0755 root:root is set.)

> Multilib support

Multilib support is one of the key reasons we needed RPM support, and not just
"rpm", but fully integrated rpm support.

The trick with multilib support is it's easy to have to packages, one that
problems /lib/libfoo.so, and one that provides /lib64/libfoo.so.  But what
happens when there is a corresponding /bin/foo?  Do you use alternatives, do use
cause a fatal conflict that must be resolved by the packaging of the system?  or
in the case of RPM, do you use preprogrammed policy to either conflict -- or
decide which of the two (or three) versions to use.

For larger systems, carrier grade specificall, the preprogrammed policy is the
mechanism of choice for most installations.

> Tighter integration with features like prelink.
> 
> I'm sure some of the more rpm literate people on the list can detail
> others.
> 
>>> * Not optimised for size (e.g. uses c++)

(Note, this is Zypper, not RPM.. RPM itself is written in C...)

>>> I'd not say one was better than the other, they're just different and
>>> suit different use cases.
>>
>> Pretty much what I thought, thanks.
>>
>> My only concern is that if rpm is the primary emphasis, ipk/opkg might
>> suffer from rot.
> 
> I want to see opkg continue to work well, if you do see any problems
> with it, let us know...

I have a vested interest in RPM, but I acknowledge that we need more then one
packaging mechanism.  RPM isn't the right answer for a lot of the devices people
are producing.. neither is deb or ipk/opkg.  (Deb support should work, but I'm
pretty sure that is the least tested of what is supported in Yocto.  It would be
nice for someone familiar with the Debian packaging mechanisms to step up and
help verify and improve the current implementation.)

--Mark

> Cheers,
> 
> Richard
> 
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: RPM vs IPK

[Khem Raj]

> My only concern is that if rpm is the primary emphasis, ipk/opkg might
> suffer from rot.
>
I think that wont be the case

RPM vs IPK

[Gary Thomas]

I notice that in the latest master, rpm-native is always built,
even if I'm using ipk style packaging.

* Why?
* Why should I choose one over the other?  I'm happy using rpm
   (been using it for 15 years!) but I wonder why the switch?
   Is rpm to become the de-facto packaging for Poky builds?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RPM vs IPK

[Gary Thomas]

On 10/29/2010 03:33 PM, Gary Thomas wrote:
> I notice that in the latest master, rpm-native is always built,
> even if I'm using ipk style packaging.
>
> * Why?
> * Why should I choose one over the other? I'm happy using rpm
> (been using it for 15 years!) but I wonder why the switch?
> Is rpm to become the de-facto packaging for Poky builds?

Also, if it's to be rpm, will there be a yum (or equivalent)?
Otherwise, what tool will be used to manage dependency tracking, etc?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: RPM vs IPK

[Richard Purdie]

On Fri, 2010-10-29 at 15:39 -0600, Gary Thomas wrote:
> On 10/29/2010 03:33 PM, Gary Thomas wrote:
> > I notice that in the latest master, rpm-native is always built,
> > even if I'm using ipk style packaging.
> >
> > * Why?
> > * Why should I choose one over the other? I'm happy using rpm
> > (been using it for 15 years!) but I wonder why the switch?
> > Is rpm to become the de-facto packaging for Poky builds?
> 
> Also, if it's to be rpm, will there be a yum (or equivalent)?
> Otherwise, what tool will be used to manage dependency tracking, etc?

This was covered in some emails that were sent a couple of months ago. 

Poky continues to support multiple packaging backends including ipk. We
do however plan to use the best tools for the job and there is a tool
called something like rpmdeps which does some dependency calculations
really well. This tool is therefore used during do_package and the
results are shared between all the packaging backends. As an example of
the kind of dependency it can provide, it will look for scripts using
python/perl/bash and then add the appropriate dependencies but there are
other things as well.

So the dependency on rpm-native is expected, it doesn't mean we're going
rpm only, just that we use the best tools for the job.

Also, since we do support packaging management with rpm, we did add
zypper as our yum equivalent for the rpm use case.

Cheers,

Richard