[RFC PATCH] populate_sdk_ext: Change to include siginfo and non sstate task sigs

[RFC PATCH] populate_sdk_ext: Change to include siginfo and non sstate task sigs

[Richard Purdie]

Right now, the locked task hashes list for the extensible SDK locks
down only the sstate tasks.

Whilst aesthetically pleasing, this gives two problems:

* Half the task are left floating meaning checksum mismatches
  are a pain to debug
* The later code which copies relavent data files out the sstate
  cache can't use any of this data.

This patch modifies things so all the checksums are listed in the locked
file. An exclusion of tasks probably makes more sense for the library
function rather than an allowed list.

The only sstate task being deliberately excluded here was do_package
so add in a function to explictly exclude those sstate object files.

The net result of this that siginfo files for all tasks are included in
the SDK, which means commands like "bitbake -S printdiff" now function.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 317043d..3a65c07 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -171,10 +171,9 @@ python copy_buildsystem () {
     oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
 
     # Filter the locked signatures file to just the sstate tasks we are interested in
-    allowed_tasks = ['do_populate_lic', 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk', 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa', 'do_deploy']
     excluded_targets = d.getVar('SDK_TARGETS', True)
     lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
-    oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
+    oe.copy_buildsystem.prune_lockedsigs([],
                                          excluded_targets,
                                          sigfile,
                                          lockedsigs_pruned)
@@ -187,6 +186,12 @@ python copy_buildsystem () {
                                                    d.getVar('SSTATE_DIR', True),
                                                    sstate_out, d,
                                                    fixedlsbstring)
+    # We don't need sstate do_package files
+    for root, dirs, files in os.walk(sstate_out):
+        for name in files:
+            if name.endswith("_package.tgz"):
+                f = os.path.join(root, name)
+                os.remove(f)
 }
 
 def extsdk_get_buildtools_filename(d):
diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index 15af4eb..a5ca3df 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -75,7 +75,7 @@ def generate_locked_sigs(sigfile, d):
     tasks = ['%s.%s' % (v[2], v[1]) for v in depd.itervalues()]
     bb.parse.siggen.dump_lockedsigs(sigfile, tasks)
 
-def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output):
+def prune_lockedsigs(excluded_tasks, excluded_targets, lockedsigs, pruned_output):
     with open(lockedsigs, 'r') as infile:
         bb.utils.mkdirhier(os.path.dirname(pruned_output))
         with open(pruned_output, 'w') as f:
@@ -84,7 +84,7 @@ def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs, pruned_output)
                 if invalue:
                     if line.endswith('\\\n'):
                         splitval = line.strip().split(':')
-                        if splitval[1] in allowed_tasks and not splitval[0] in excluded_targets:
+                        if not splitval[1] in excluded_tasks and not splitval[0] in excluded_targets:
                             f.write(line)
                     else:
                         f.write(line)

Re: [RFC PATCH] populate_sdk_ext: Change to include siginfo and non sstate task sigs

[Paul Eggleton]

On Fri, 08 Jan 2016 18:25:22 Richard Purdie wrote:
> Right now, the locked task hashes list for the extensible SDK locks
> down only the sstate tasks.
> 
> Whilst aesthetically pleasing, this gives two problems:
> 
> * Half the task are left floating meaning checksum mismatches
>   are a pain to debug
> * The later code which copies relavent data files out the sstate
>   cache can't use any of this data.
> 
> This patch modifies things so all the checksums are listed in the locked
> file. An exclusion of tasks probably makes more sense for the library
> function rather than an allowed list.
> 
> The only sstate task being deliberately excluded here was do_package
> so add in a function to explictly exclude those sstate object files.
> 
> The net result of this that siginfo files for all tasks are included in
> the SDK, which means commands like "bitbake -S printdiff" now function.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index 317043d..3a65c07 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -171,10 +171,9 @@ python copy_buildsystem () {
>      oe.copy_buildsystem.generate_locked_sigs(sigfile, d)
> 
>      # Filter the locked signatures file to just the sstate tasks we are
> interested in -    allowed_tasks = ['do_populate_lic',
> 'do_populate_sysroot', 'do_packagedata', 'do_package_write_ipk',
> 'do_package_write_rpm', 'do_package_write_deb', 'do_package_qa',
> 'do_deploy'] excluded_targets = d.getVar('SDK_TARGETS', True)
>      lockedsigs_pruned = baseoutpath + '/conf/locked-sigs.inc'
> -    oe.copy_buildsystem.prune_lockedsigs(allowed_tasks,
> +    oe.copy_buildsystem.prune_lockedsigs([],
>                                           excluded_targets,
>                                           sigfile,
>                                           lockedsigs_pruned)
> @@ -187,6 +186,12 @@ python copy_buildsystem () {
>                                                     d.getVar('SSTATE_DIR',
> True), sstate_out, d, fixedlsbstring) +    # We don't need sstate
> do_package files
> +    for root, dirs, files in os.walk(sstate_out):
> +        for name in files:
> +            if name.endswith("_package.tgz"):
> +                f = os.path.join(root, name)
> +                os.remove(f)
>  }
> 
>  def extsdk_get_buildtools_filename(d):
> diff --git a/meta/lib/oe/copy_buildsystem.py
> b/meta/lib/oe/copy_buildsystem.py index 15af4eb..a5ca3df 100644
> --- a/meta/lib/oe/copy_buildsystem.py
> +++ b/meta/lib/oe/copy_buildsystem.py
> @@ -75,7 +75,7 @@ def generate_locked_sigs(sigfile, d):
>      tasks = ['%s.%s' % (v[2], v[1]) for v in depd.itervalues()]
>      bb.parse.siggen.dump_lockedsigs(sigfile, tasks)
> 
> -def prune_lockedsigs(allowed_tasks, excluded_targets, lockedsigs,
> pruned_output): +def prune_lockedsigs(excluded_tasks, excluded_targets,
> lockedsigs, pruned_output): with open(lockedsigs, 'r') as infile:
>          bb.utils.mkdirhier(os.path.dirname(pruned_output))
>          with open(pruned_output, 'w') as f:
> @@ -84,7 +84,7 @@ def prune_lockedsigs(allowed_tasks, excluded_targets,
> lockedsigs, pruned_output) if invalue:
>                      if line.endswith('\\\n'):
>                          splitval = line.strip().split(':')
> -                        if splitval[1] in allowed_tasks and not splitval[0]
> in excluded_targets: +                        if not splitval[1] in
> excluded_tasks and not splitval[0] in excluded_targets: f.write(line)
>                      else:
>                          f.write(line)

This looks good to me.

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Cheers,
Paul

-- 
Paul Eggleton
Intel Open Source Technology Centre