[PATCH V2 0/1] Mark some tasks in extensible SDK as 'noexec'

[PATCH V2 0/1] Mark some tasks in extensible SDK as 'noexec'

[Chen Qi]

Changes since V1:
Use the 'for' loop to simplify codes.

The following changes since commit 2d09cb28f25080ed337d37d2d4381b846bbe6247:

  gcc5: Add PR65779 patch to fix powerpc compile issues (2015-05-12 13:10:17 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib ChenQi/SDK-noexec
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/SDK-noexec

Chen Qi (1):
  populate_sdk_ext: set noexec flag for some tasks

 meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

-- 
1.9.1

[PATCH V2 1/1] populate_sdk_ext: set noexec flag for some tasks

[Chen Qi]

For the SDK, what we are interested in is the sysroot part. So the
only useful task is do_populate_sysroot. Other tasks like do_package
don't need to run. So mark them as 'noexec' for SDK.

This would result in two benefits:
1) The size of extensible SDK is larged reduced. For example, extensible
   SDK for core-image-minimal is reduced from 1.8G to 793M.
2) The installation time is reduced as a lot of tasks are not run.

[YOCTO #7590]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index dc2c58e..6c265b8 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -141,8 +141,18 @@ python copy_buildsystem () {
     sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
     oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
 
+    # Mark tasks to be 'noexec' for the SDK as we only need the sysroot part
+    base_class_path = os.path.join(baseoutpath, core_meta_subdir, 'classes', 'base.bbclass')
+    if os.path.exists(base_class_path):
+        with open(base_class_path, 'a') as f:
+            noexec_tasks = ['do_package', 'do_packagedata', 'do_package_qa', 'do_package_write_rpm',
+                            'do_package_write_ipk', 'do_package_write_deb', 'do_deploy', 'do_populate_lic',
+                            'do_rootfs']
+            for t in noexec_tasks:
+                f.write('%s[noexec] = "1"\n' % t)
+
     # Filter the locked signatures file to just the sstate tasks we are interested in
-    allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
+    allowed_tasks = ['do_populate_sysroot']
     excluded_targets = d.getVar('SDK_TARGETS', True)
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
     oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
-- 
1.9.1

Re: [PATCH V2 1/1] populate_sdk_ext: set noexec flag for some tasks

[Randy Witt]

On 05/13/2015 07:19 PM, Chen Qi wrote:
> For the SDK, what we are interested in is the sysroot part. So the
> only useful task is do_populate_sysroot. Other tasks like do_package
> don't need to run. So mark them as 'noexec' for SDK.
>
> This would result in two benefits:
> 1) The size of extensible SDK is larged reduced. For example, extensible
>     SDK for core-image-minimal is reduced from 1.8G to 793M.
> 2) The installation time is reduced as a lot of tasks are not run.
>
> [YOCTO #7590]
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>   meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
>   1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
> index dc2c58e..6c265b8 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -141,8 +141,18 @@ python copy_buildsystem () {
>       sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
>       oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
>
> +    # Mark tasks to be 'noexec' for the SDK as we only need the sysroot part
> +    base_class_path = os.path.join(baseoutpath, core_meta_subdir, 'classes', 'base.bbclass')

Qi, I don't like the idea of modifying the source that is copied into the sdk. 
Also, this will continue to persist if the user ever wants to start generating 
images.

Ideally, we would only modify the behavior while setting up the sdk. And then 
the behavior afterward would remain as when using a regular bitbake workspace.

> +    if os.path.exists(base_class_path):
> +        with open(base_class_path, 'a') as f:
> +            noexec_tasks = ['do_package', 'do_packagedata', 'do_package_qa', 'do_package_write_rpm',
> +                            'do_package_write_ipk', 'do_package_write_deb', 'do_deploy', 'do_populate_lic',
> +                            'do_rootfs']
> +            for t in noexec_tasks:
> +                f.write('%s[noexec] = "1"\n' % t)
> +
>       # Filter the locked signatures file to just the sstate tasks we are interested in
> -    allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
> +    allowed_tasks = ['do_populate_sysroot']
>       excluded_targets = d.getVar('SDK_TARGETS', True)
>       lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
>       oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
>

Re: [PATCH V2 1/1] populate_sdk_ext: set noexec flag for some tasks

[Paul Eggleton]

On Thursday 14 May 2015 09:49:50 Randy Witt wrote:
> On 05/13/2015 07:19 PM, Chen Qi wrote:
> > For the SDK, what we are interested in is the sysroot part. So the
> > only useful task is do_populate_sysroot. Other tasks like do_package
> > don't need to run. So mark them as 'noexec' for SDK.
> > 
> > This would result in two benefits:
> > 1) The size of extensible SDK is larged reduced. For example, extensible
> > 
> >     SDK for core-image-minimal is reduced from 1.8G to 793M.
> > 
> > 2) The installation time is reduced as a lot of tasks are not run.
> > 
> > [YOCTO #7590]
> > 
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > ---
> > 
> >   meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
> >   1 file changed, 11 insertions(+), 1 deletion(-)
> > 
> > diff --git a/meta/classes/populate_sdk_ext.bbclass
> > b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..6c265b8 100644
> > --- a/meta/classes/populate_sdk_ext.bbclass
> > +++ b/meta/classes/populate_sdk_ext.bbclass
> > @@ -141,8 +141,18 @@ python copy_buildsystem () {
> > 
> >       sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
> >       oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
> > 
> > +    # Mark tasks to be 'noexec' for the SDK as we only need the sysroot
> > part +    base_class_path = os.path.join(baseoutpath, core_meta_subdir,
> > 'classes', 'base.bbclass')
>
> Qi, I don't like the idea of modifying the source that is copied into the
> sdk. Also, this will continue to persist if the user ever wants to start
> generating images.
> 
> Ideally, we would only modify the behavior while setting up the sdk. And
> then the behavior afterward would remain as when using a regular bitbake
> workspace.

Right. In order to fix this bug we really ought to only affect what happens at 
SDK install time. If we need to inject things into the configuration just for 
that execution, one way to do that would be to create a pre/post configuration 
file and pass that into the command that gets run with -r/-R.

Re allowed_tasks, don't you also need to change the recrdeptask value?

I'd also like to see this configurable at SDK construction time, so you can set 
a variable to provide and install the artifacts for the rest of the tasks if 
you want to.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH V2 1/1] populate_sdk_ext: set noexec flag for some tasks

[ChenQi]

On 05/15/2015 01:10 AM, Paul Eggleton wrote:
> On Thursday 14 May 2015 09:49:50 Randy Witt wrote:
>> On 05/13/2015 07:19 PM, Chen Qi wrote:
>>> For the SDK, what we are interested in is the sysroot part. So the
>>> only useful task is do_populate_sysroot. Other tasks like do_package
>>> don't need to run. So mark them as 'noexec' for SDK.
>>>
>>> This would result in two benefits:
>>> 1) The size of extensible SDK is larged reduced. For example, extensible
>>>
>>>      SDK for core-image-minimal is reduced from 1.8G to 793M.
>>>
>>> 2) The installation time is reduced as a lot of tasks are not run.
>>>
>>> [YOCTO #7590]
>>>
>>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>>> ---
>>>
>>>    meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
>>>    1 file changed, 11 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/meta/classes/populate_sdk_ext.bbclass
>>> b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..6c265b8 100644
>>> --- a/meta/classes/populate_sdk_ext.bbclass
>>> +++ b/meta/classes/populate_sdk_ext.bbclass
>>> @@ -141,8 +141,18 @@ python copy_buildsystem () {
>>>
>>>        sigfile = d.getVar('WORKDIR', True) + '/locked-sigs.inc'
>>>        oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
>>>
>>> +    # Mark tasks to be 'noexec' for the SDK as we only need the sysroot
>>> part +    base_class_path = os.path.join(baseoutpath, core_meta_subdir,
>>> 'classes', 'base.bbclass')
>> Qi, I don't like the idea of modifying the source that is copied into the
>> sdk. Also, this will continue to persist if the user ever wants to start
>> generating images.
>>
>> Ideally, we would only modify the behavior while setting up the sdk. And
>> then the behavior afterward would remain as when using a regular bitbake
>> workspace.
> Right. In order to fix this bug we really ought to only affect what happens at
> SDK install time. If we need to inject things into the configuration just for
> that execution, one way to do that would be to create a pre/post configuration
> file and pass that into the command that gets run with -r/-R.
>
> Re allowed_tasks, don't you also need to change the recrdeptask value?
>
> I'd also like to see this configurable at SDK construction time, so you can set
> a variable to provide and install the artifacts for the rest of the tasks if
> you want to.
>
> Cheers,
> Paul
>

Hi Paul and Randy,

Thanks for your feedback.
I totally agree with you.

And I'm going to follow Paul's suggestions and make a new patch.

Best Regards,
Chen Qi