[PATCHv2] bitbake-layers: check layer dependencies before adding

[PATCHv2] bitbake-layers: check layer dependencies before adding

[Chang Rebecca Swee Fun]

Change log on v2 patch:
- Remove line "return 1", because bb.fatal raises an exception, hence
  return 1 is not needed.

Regards,
Rebecca

Chang Rebecca Swee Fun (1):
  bitbake-layers: check layer dependencies before adding

 bin/bitbake-layers     |  1 +
 lib/bblayers/action.py | 26 ++++++++++++++++++++++----
 2 files changed, 23 insertions(+), 4 deletions(-)

-- 
2.7.4

[PATCHv2] bitbake-layers: check layer dependencies before adding

[Chang Rebecca Swee Fun]

In the original implementation, "bitbake-layers add-layers <layer>"
succeeded without error checking. This will further introduce
failures in recipe parsing only when "bitbake" command is executed.
Adding a meta layer without its dependency layer(s) should failed
and exit the process gracefully.

Added extra argument "-f" to force add a layer without checking
layer dependency.

[YOCTO #10913]

Signed-off-by: Phoong Stanley Cheong Kwan <stanley.cheong.kwan.phoong@intel.com>
Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>
---
 bin/bitbake-layers     |  1 +
 lib/bblayers/action.py | 26 ++++++++++++++++++++++----
 2 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/bin/bitbake-layers b/bin/bitbake-layers
index 2b05d28..ced38a6 100755
--- a/bin/bitbake-layers
+++ b/bin/bitbake-layers
@@ -43,6 +43,7 @@ def main():
         add_help=False)
     parser.add_argument('-d', '--debug', help='Enable debug output', action='store_true')
     parser.add_argument('-q', '--quiet', help='Print only errors', action='store_true')
+    parser.add_argument('-f', '--force', help='Force add without recipe parse verification', action='store_true')
     parser.add_argument('--color', choices=['auto', 'always', 'never'], default='auto', help='Colorize output (where %(metavar)s is %(choices)s)', metavar='COLOR')
 
     global_args, unparsed_args = parser.parse_known_args()
diff --git a/lib/bblayers/action.py b/lib/bblayers/action.py
index cf94704..cc61555 100644
--- a/lib/bblayers/action.py
+++ b/lib/bblayers/action.py
@@ -1,7 +1,9 @@
 import fnmatch
 import logging
 import os
+import shutil
 import sys
+import tempfile
 
 import bb.utils
 
@@ -32,10 +34,26 @@ class ActionPlugin(LayerPlugin):
             sys.stderr.write("Unable to find bblayers.conf\n")
             return 1
 
-        notadded, _ = bb.utils.edit_bblayers_conf(bblayers_conf, layerdir, None)
-        if notadded:
-            for item in notadded:
-                sys.stderr.write("Specified layer %s is already in BBLAYERS\n" % item)
+        # Back up bblayers.conf to tempdir before we add layers
+        tempdir = tempfile.mkdtemp()
+        backup = tempdir + "/bblayers.conf.bak"
+        shutil.copy2(bblayers_conf, backup)
+
+        try:
+            notadded, _ = bb.utils.edit_bblayers_conf(bblayers_conf, layerdir, None)
+            if not (args.force or notadded):
+                try:
+                    self.tinfoil.parseRecipes()
+                except bb.tinfoil.TinfoilUIException:
+                    # Restore the back up copy of bblayers.conf
+                    shutil.copy2(backup, bblayers_conf)
+                    bb.fatal("Parse failure with the specified layer added")
+            else:
+                for item in notadded:
+                    sys.stderr.write("Specified layer %s is already in BBLAYERS\n" % item)
+        finally:
+            # Remove the back up copy of bblayers.conf
+            shutil.rmtree(tempdir)
 
     def do_remove_layer(self, args):
         """Remove a layer from bblayers.conf."""
-- 
2.7.4

Re: [PATCHv2] bitbake-layers: check layer dependencies before adding

[Richard Purdie]

On Mon, 2017-05-29 at 08:44 +0800, Chang Rebecca Swee Fun wrote:
> In the original implementation, "bitbake-layers add-layers <layer>"
> succeeded without error checking. This will further introduce
> failures in recipe parsing only when "bitbake" command is executed.
> Adding a meta layer without its dependency layer(s) should failed
> and exit the process gracefully.
> 
> Added extra argument "-f" to force add a layer without checking
> layer dependency.
> 
> [YOCTO #10913]
> 
> Signed-off-by: Phoong Stanley Cheong Kwan <stanley.cheong.kwan.phoong
> @intel.com>
> Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.c
> om>
> ---
>  bin/bitbake-layers     |  1 +
>  lib/bblayers/action.py | 26 ++++++++++++++++++++++----
>  2 files changed, 23 insertions(+), 4 deletions(-)

This patch seems to result in this:

https://autobuilder.yocto.io/builders/nightly-oe-selftest/builds/320/steps/Running%20oe-selftest/logs/stdio

Cheers,

Richard

Re: [PATCHv2] bitbake-layers: check layer dependencies before adding

[Chang, Rebecca Swee Fun]

Hi Richard,

Thanks! We have found out why it failed.
We were using "-f" to force add a layer without going through recipe parse.
I just realized the bitbake selftests were using "-f" in bitbake-layers show-recipes.
The argument flags were overriding each other.
I will fix that in v3 patch.

Regards,
Rebecca

> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: Sunday, June 4, 2017 10:13 PM
> To: Chang, Rebecca Swee Fun <rebecca.swee.fun.chang@intel.com>; Bitbake
> dev Mailing List <bitbake-devel@lists.openembedded.org>
> Cc: Phoong, Stanley Cheong Kwan <stanley.cheong.kwan.phoong@intel.com>
> Subject: Re: [bitbake-devel] [PATCHv2] bitbake-layers: check layer
> dependencies before adding
> 
> On Mon, 2017-05-29 at 08:44 +0800, Chang Rebecca Swee Fun wrote:
> > In the original implementation, "bitbake-layers add-layers <layer>"
> > succeeded without error checking. This will further introduce failures
> > in recipe parsing only when "bitbake" command is executed.
> > Adding a meta layer without its dependency layer(s) should failed and
> > exit the process gracefully.
> >
> > Added extra argument "-f" to force add a layer without checking layer
> > dependency.
> >
> > [YOCTO #10913]
> >
> > Signed-off-by: Phoong Stanley Cheong Kwan <stanley.cheong.kwan.phoong
> > @intel.com>
> > Signed-off-by: Chang Rebecca Swee Fun <rebecca.swee.fun.chang@intel.c
> > om>
> > ---
> >  bin/bitbake-layers     |  1 +
> >  lib/bblayers/action.py | 26 ++++++++++++++++++++++----
> >  2 files changed, 23 insertions(+), 4 deletions(-)
> 
> This patch seems to result in this:
> 
> https://autobuilder.yocto.io/builders/nightly-oe-
> selftest/builds/320/steps/Running%20oe-selftest/logs/stdio
> 
> Cheers,
> 
> Richard