[RFC] let PACKAGES_DYNAMIC be optional ?

[RFC] let PACKAGES_DYNAMIC be optional ?

[Robert Yang]

Currently, the recipe which uses PACKAGES_DYNAMIC usually generates
a lot of packages which costs a lot of time on building the recipe
and do_rootfs, for example, the perl and kernel:

$ ls tmp/deploy/rpm/armv5te/perl-module-* | wc -l
621

$ ls tmp/deploy/rpm/qemux86/kernel-module-* | wc -l
268

Also, the eglibc-locale generates more than 300 packages.

Take perl as an example:

1) We generate 621 perl-module-* packages, but the package *perl-modules*
    requires all of them, so once *perl-modules* is installed, all the other
    perl-module-* will be installed and we can't remove any of them since
    perl-modules rdepends on it, if there is a way to package all of these
    perl-module-* into one package (they are about 10MB), it would save a lot
    of time on do_package* and do_rootfs.

2) The nativesdk.bbclass can't support PACKAGES_DYNAMIC, for example, it can't
    change the perl-module-app-cpan to nativesdk-perl-module-app-cpan since
    there is no perl-module-app-cpan in PACKAGES when nativesdk.bbclass
    changes the variable's name.

Can we add a way to let the PACKAGES_DYNAMIC be optional ? for example,

PACKAGES_DYNAMIC[perl] = "0"

will disable the perl's PACKAGES_DYNAMIC, and will pack the files as other
recipes do, and of course we need to do some work on the recipe.

-- 
Thanks

Robert

Re: [RFC] let PACKAGES_DYNAMIC be optional ?

[Richard Purdie]

On Fri, 2013-11-15 at 11:31 +0800, Robert Yang wrote:
> Currently, the recipe which uses PACKAGES_DYNAMIC usually generates
> a lot of packages which costs a lot of time on building the recipe
> and do_rootfs, for example, the perl and kernel:
> 
> $ ls tmp/deploy/rpm/armv5te/perl-module-* | wc -l
> 621
> 
> $ ls tmp/deploy/rpm/qemux86/kernel-module-* | wc -l
> 268
> 
> Also, the eglibc-locale generates more than 300 packages.
> 
> Take perl as an example:
> 
> 1) We generate 621 perl-module-* packages, but the package *perl-modules*
>     requires all of them, so once *perl-modules* is installed, all the other
>     perl-module-* will be installed and we can't remove any of them since
>     perl-modules rdepends on it, if there is a way to package all of these
>     perl-module-* into one package (they are about 10MB), it would save a lot
>     of time on do_package* and do_rootfs.
> 
> 2) The nativesdk.bbclass can't support PACKAGES_DYNAMIC, for example, it can't
>     change the perl-module-app-cpan to nativesdk-perl-module-app-cpan since
>     there is no perl-module-app-cpan in PACKAGES when nativesdk.bbclass
>     changes the variable's name.
> 
> Can we add a way to let the PACKAGES_DYNAMIC be optional ? for example,
> 
> PACKAGES_DYNAMIC[perl] = "0"
> 
> will disable the perl's PACKAGES_DYNAMIC, and will pack the files as other
> recipes do, and of course we need to do some work on the recipe.

Before we consider doing this, I'd actually like to see real numbers
about how big this problem is.

Why? Speaking as someone who has looked specifically at perl and the
kernel, I don't believe there is a huge amount of time spent dealing
with the individual packages and that maintaining two build paths is
actually worse than they minimal performance impact this has.

In particular, I'd note that the locale generation happens in parallel
with other parts of the build and is not a significant factor in overall
build times.

The time would be better spent reducing the size of the kernel source
installed into the sysroot for example (Bruce is planning action on
this).

Cheers,

Richard

Re: [RFC] let PACKAGES_DYNAMIC be optional ?

[Otavio Salvador]

On Fri, Nov 15, 2013 at 10:18 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2013-11-15 at 11:31 +0800, Robert Yang wrote:
>> Currently, the recipe which uses PACKAGES_DYNAMIC usually generates
>> a lot of packages which costs a lot of time on building the recipe
>> and do_rootfs, for example, the perl and kernel:
>>
>> $ ls tmp/deploy/rpm/armv5te/perl-module-* | wc -l
>> 621
>>
>> $ ls tmp/deploy/rpm/qemux86/kernel-module-* | wc -l
>> 268
>>
>> Also, the eglibc-locale generates more than 300 packages.
>>
>> Take perl as an example:
>>
>> 1) We generate 621 perl-module-* packages, but the package *perl-modules*
>>     requires all of them, so once *perl-modules* is installed, all the other
>>     perl-module-* will be installed and we can't remove any of them since
>>     perl-modules rdepends on it, if there is a way to package all of these
>>     perl-module-* into one package (they are about 10MB), it would save a lot
>>     of time on do_package* and do_rootfs.
>>
>> 2) The nativesdk.bbclass can't support PACKAGES_DYNAMIC, for example, it can't
>>     change the perl-module-app-cpan to nativesdk-perl-module-app-cpan since
>>     there is no perl-module-app-cpan in PACKAGES when nativesdk.bbclass
>>     changes the variable's name.
>>
>> Can we add a way to let the PACKAGES_DYNAMIC be optional ? for example,
>>
>> PACKAGES_DYNAMIC[perl] = "0"
>>
>> will disable the perl's PACKAGES_DYNAMIC, and will pack the files as other
>> recipes do, and of course we need to do some work on the recipe.
>
> Before we consider doing this, I'd actually like to see real numbers
> about how big this problem is.
>
> Why? Speaking as someone who has looked specifically at perl and the
> kernel, I don't believe there is a huge amount of time spent dealing
> with the individual packages and that maintaining two build paths is
> actually worse than they minimal performance impact this has.
>
> In particular, I'd note that the locale generation happens in parallel
> with other parts of the build and is not a significant factor in overall
> build times.
>
> The time would be better spent reducing the size of the kernel source
> installed into the sysroot for example (Bruce is planning action on
> this).

I agree on Richard on this; keeping too many different build routes
introduces more test burden and it's only worth if the gain is huge.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [RFC] let PACKAGES_DYNAMIC be optional ?

[Robert Yang]

On 11/15/2013 09:25 PM, Otavio Salvador wrote:
> On Fri, Nov 15, 2013 at 10:18 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> On Fri, 2013-11-15 at 11:31 +0800, Robert Yang wrote:
>>> Currently, the recipe which uses PACKAGES_DYNAMIC usually generates
>>> a lot of packages which costs a lot of time on building the recipe
>>> and do_rootfs, for example, the perl and kernel:
>>>
>>> $ ls tmp/deploy/rpm/armv5te/perl-module-* | wc -l
>>> 621
>>>
>>> $ ls tmp/deploy/rpm/qemux86/kernel-module-* | wc -l
>>> 268
>>>
>>> Also, the eglibc-locale generates more than 300 packages.
>>>
>>> Take perl as an example:
>>>
>>> 1) We generate 621 perl-module-* packages, but the package *perl-modules*
>>>      requires all of them, so once *perl-modules* is installed, all the other
>>>      perl-module-* will be installed and we can't remove any of them since
>>>      perl-modules rdepends on it, if there is a way to package all of these
>>>      perl-module-* into one package (they are about 10MB), it would save a lot
>>>      of time on do_package* and do_rootfs.
>>>
>>> 2) The nativesdk.bbclass can't support PACKAGES_DYNAMIC, for example, it can't
>>>      change the perl-module-app-cpan to nativesdk-perl-module-app-cpan since
>>>      there is no perl-module-app-cpan in PACKAGES when nativesdk.bbclass
>>>      changes the variable's name.
>>>
>>> Can we add a way to let the PACKAGES_DYNAMIC be optional ? for example,
>>>
>>> PACKAGES_DYNAMIC[perl] = "0"
>>>
>>> will disable the perl's PACKAGES_DYNAMIC, and will pack the files as other
>>> recipes do, and of course we need to do some work on the recipe.
>>
>> Before we consider doing this, I'd actually like to see real numbers
>> about how big this problem is.
>>
>> Why? Speaking as someone who has looked specifically at perl and the
>> kernel, I don't believe there is a huge amount of time spent dealing
>> with the individual packages and that maintaining two build paths is
>> actually worse than they minimal performance impact this has.
>>
>> In particular, I'd note that the locale generation happens in parallel
>> with other parts of the build and is not a significant factor in overall
>> build times.
>>
>> The time would be better spent reducing the size of the kernel source
>> installed into the sysroot for example (Bruce is planning action on
>> this).
>
> I agree on Richard on this; keeping too many different build routes
> introduces more test burden and it's only worth if the gain is huge.
>

Sounds reasonable, I will try sooner to see the gain.

// Robert