* [PATCH 1/4] yocto-compat-layer.py: Add status for skipped items
2017-03-31 2:30 [PATCH 0/4] yocto-compat-layer.py updates Mark Hatle
@ 2017-03-31 2:30 ` Mark Hatle
2017-03-31 2:30 ` [PATCH 2/4] yocto-compat-layer.py: Add --dependency argument Mark Hatle
` (3 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2017-03-31 2:30 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
If items were skipped because the dependencies could not be found, we
want to record this was skipped so we can display it later.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
scripts/yocto-compat-layer.py | 10 +++++++---
1 file changed, 7 insertions(+), 3 deletions(-)
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index 9e74033..d13d8c0 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -107,6 +107,7 @@ def main():
td = {}
results = collections.OrderedDict()
+ results_status = collections.OrderedDict()
logger.info('')
logger.info('Getting initial bitbake variables ...')
@@ -125,19 +126,22 @@ def main():
shutil.copyfile(bblayersconf + '.backup', bblayersconf)
if not add_layer(bblayersconf, layer, layers, logger):
+ results[layer['name']] = None
+ results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
+ layers_tested = layers_tested + 1
continue
result = test_layer_compatibility(td, layer)
results[layer['name']] = result
+ results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
layers_tested = layers_tested + 1
if layers_tested:
logger.info('')
logger.info('Summary of results:')
logger.info('')
- for layer_name in results:
- logger.info('%s ... %s' % (layer_name, 'PASS' if \
- results[layer_name].wasSuccessful() else 'FAIL'))
+ for layer_name in results_status:
+ logger.info('%s ... %s' % (layer_name, results_status[layer_name]))
cleanup_bblayers(None, None)
--
1.8.3.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 2/4] yocto-compat-layer.py: Add --dependency argument
2017-03-31 2:30 [PATCH 0/4] yocto-compat-layer.py updates Mark Hatle
2017-03-31 2:30 ` [PATCH 1/4] yocto-compat-layer.py: Add status for skipped items Mark Hatle
@ 2017-03-31 2:30 ` Mark Hatle
2017-03-31 2:30 ` [PATCH 3/4] compatlayer/__init__.py: Allow add_layer to process recursive deps Mark Hatle
` (2 subsequent siblings)
4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2017-03-31 2:30 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
When processing a large number of items, there are times that it would be
nice to be able to pass in a series of layers that can be used as dependencies
for the layer that is being scanned. This avoids the significant overhead
of processing all of the layers to compatibility.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
scripts/yocto-compat-layer.py | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index d13d8c0..f8a1ac7 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -47,6 +47,8 @@ def main():
help='Layer to test compatibility with Yocto Project')
parser.add_argument('-o', '--output-log',
help='File to output log (optional)', action='store')
+ parser.add_argument('--dependency', nargs="+",
+ help='Layers to process for dependencies', action='store')
parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
action='store_true')
parser.add_argument('-d', '--debug', help='Enable debug output',
@@ -80,6 +82,11 @@ def main():
if not layers:
logger.error("Fail to detect layers")
return 1
+ if args.dependency:
+ dep_layers = detect_layers(args.dependency, args.no_auto)
+ dep_layers = dep_layers + layers
+ else:
+ dep_layers = layers
logger.info("Detected layers:")
for layer in layers:
@@ -125,7 +132,8 @@ def main():
shutil.copyfile(bblayersconf + '.backup', bblayersconf)
- if not add_layer(bblayersconf, layer, layers, logger):
+ if not add_layer(bblayersconf, layer, dep_layers, logger):
+ logger.info('Skipping %s due to missing dependencies.' % layer['name'])
results[layer['name']] = None
results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
layers_tested = layers_tested + 1
--
1.8.3.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 3/4] compatlayer/__init__.py: Allow add_layer to process recursive deps
2017-03-31 2:30 [PATCH 0/4] yocto-compat-layer.py updates Mark Hatle
2017-03-31 2:30 ` [PATCH 1/4] yocto-compat-layer.py: Add status for skipped items Mark Hatle
2017-03-31 2:30 ` [PATCH 2/4] yocto-compat-layer.py: Add --dependency argument Mark Hatle
@ 2017-03-31 2:30 ` Mark Hatle
2017-03-31 2:30 ` [PATCH 4/4] yocto-compat-layer.py: Fix the signature validation Mark Hatle
2017-04-03 19:03 ` [PATCH 0/4] yocto-compat-layer.py updates Aníbal Limón
4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2017-03-31 2:30 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
When processing a layer for dependencies, you have to process the layer
itself, it's dependencies, the dependencies dependencies and so forth until
all items have been processed.
i.e.: LayerA requires LayerB requires LayerC requires layerD
The end result should be LayerB, LayerC and LayerD are all dependencies of
LayerA.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
scripts/lib/compatlayer/__init__.py | 40 ++++++++++++++++++++++++++++++++-----
1 file changed, 35 insertions(+), 5 deletions(-)
diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 58f94b7..24edc88 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -142,10 +142,9 @@ def _find_layer_depends(depend, layers):
def add_layer(bblayersconf, layer, layers, logger):
logger.info('Adding layer %s' % layer['name'])
- for collection in layer['collections']:
- depends = layer['collections'][collection]['depends']
- if not depends:
- continue
+ def recurse_dependencies(depends, layer, layers, logger, ret = []):
+ logger.debug('Processing dependencies %s for layer %s.' % \
+ (depends, layer['name']))
for depend in depends.split():
# core (oe-core) is suppose to be provided
@@ -156,8 +155,39 @@ def add_layer(bblayersconf, layer, layers, logger):
if not layer_depend:
logger.error('Layer %s depends on %s and isn\'t found.' % \
(layer['name'], depend))
- return False
+ ret = None
+ continue
+
+ # We keep processing, even if ret is None, this allows us to report
+ # multiple errors at once
+ if ret is not None and layer_depend not in ret:
+ ret.append(layer_depend)
+
+ # Recursively process...
+ if 'collections' not in layer_depend:
+ continue
+
+ for collection in layer_depend['collections']:
+ collect_deps = layer_depend['collections'][collection]['depends']
+ if not collect_deps:
+ continue
+ ret = recurse_dependencies(collect_deps, layer_depend, layers, logger, ret)
+
+ return ret
+ layer_depends = []
+ for collection in layer['collections']:
+ depends = layer['collections'][collection]['depends']
+ if not depends:
+ continue
+
+ layer_depends = recurse_dependencies(depends, layer, layers, logger, layer_depends)
+
+ # Note: [] (empty) is allowed, None is not!
+ if layer_depends is None:
+ return False
+ else:
+ for layer_depend in layer_depends:
logger.info('Adding layer dependency %s' % layer_depend['name'])
with open(bblayersconf, 'a+') as f:
f.write("\nBBLAYERS += \"%s\"\n" % layer_depend['path'])
--
1.8.3.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH 4/4] yocto-compat-layer.py: Fix the signature validation
2017-03-31 2:30 [PATCH 0/4] yocto-compat-layer.py updates Mark Hatle
` (2 preceding siblings ...)
2017-03-31 2:30 ` [PATCH 3/4] compatlayer/__init__.py: Allow add_layer to process recursive deps Mark Hatle
@ 2017-03-31 2:30 ` Mark Hatle
2017-04-03 19:03 ` [PATCH 0/4] yocto-compat-layer.py updates Aníbal Limón
4 siblings, 0 replies; 6+ messages in thread
From: Mark Hatle @ 2017-03-31 2:30 UTC (permalink / raw)
To: openembedded-core; +Cc: saul.wold
The initial signatures need to be collected -after- the dependency layers have
been added to the system. Otherwise changes that happen due to dependencies,
outside of the layer being scanned, will show up as signature problems.
The add_layer function was split into two pieces so that we can process
the dependencies first, and then add the layer itself for the comparison.
Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
---
scripts/lib/compatlayer/__init__.py | 7 ++++---
scripts/yocto-compat-layer.py | 29 +++++++++++++++++++----------
2 files changed, 23 insertions(+), 13 deletions(-)
diff --git a/scripts/lib/compatlayer/__init__.py b/scripts/lib/compatlayer/__init__.py
index 24edc88..06c84f4 100644
--- a/scripts/lib/compatlayer/__init__.py
+++ b/scripts/lib/compatlayer/__init__.py
@@ -139,9 +139,7 @@ def _find_layer_depends(depend, layers):
return layer
return None
-def add_layer(bblayersconf, layer, layers, logger):
- logger.info('Adding layer %s' % layer['name'])
-
+def add_layer_dependencies(bblayersconf, layer, layers, logger):
def recurse_dependencies(depends, layer, layers, logger, ret = []):
logger.debug('Processing dependencies %s for layer %s.' % \
(depends, layer['name']))
@@ -191,7 +189,10 @@ def add_layer(bblayersconf, layer, layers, logger):
logger.info('Adding layer dependency %s' % layer_depend['name'])
with open(bblayersconf, 'a+') as f:
f.write("\nBBLAYERS += \"%s\"\n" % layer_depend['path'])
+ return True
+def add_layer(bblayersconf, layer, layers, logger):
+ logger.info('Adding layer %s' % layer['name'])
with open(bblayersconf, 'a+') as f:
f.write("\nBBLAYERS += \"%s\"\n" % layer['path'])
diff --git a/scripts/yocto-compat-layer.py b/scripts/yocto-compat-layer.py
index f8a1ac7..22c0c2d 100755
--- a/scripts/yocto-compat-layer.py
+++ b/scripts/yocto-compat-layer.py
@@ -22,7 +22,7 @@ import scriptpath
scriptpath.add_oe_lib_path()
scriptpath.add_bitbake_lib_path()
-from compatlayer import LayerType, detect_layers, add_layer, get_signatures
+from compatlayer import LayerType, detect_layers, add_layer, add_layer_dependencies, get_signatures
from oeqa.utils.commands import get_bb_vars
PROGNAME = 'yocto-compat-layer'
@@ -116,29 +116,38 @@ def main():
results = collections.OrderedDict()
results_status = collections.OrderedDict()
- logger.info('')
- logger.info('Getting initial bitbake variables ...')
- td['bbvars'] = get_bb_vars()
- logger.info('Getting initial signatures ...')
- td['builddir'] = builddir
- td['sigs'] = get_signatures(td['builddir'])
- logger.info('')
-
layers_tested = 0
for layer in layers:
if layer['type'] == LayerType.ERROR_NO_LAYER_CONF or \
layer['type'] == LayerType.ERROR_BSP_DISTRO:
continue
+ logger.info('')
+ logger.info("Setting up for %s(%s), %s" % (layer['name'], layer['type'],
+ layer['path']))
+
shutil.copyfile(bblayersconf + '.backup', bblayersconf)
- if not add_layer(bblayersconf, layer, dep_layers, logger):
+ if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger):
logger.info('Skipping %s due to missing dependencies.' % layer['name'])
results[layer['name']] = None
results_status[layer['name']] = 'SKIPPED (Missing dependencies)'
layers_tested = layers_tested + 1
continue
+ logger.info('Getting initial bitbake variables ...')
+ td['bbvars'] = get_bb_vars()
+ logger.info('Getting initial signatures ...')
+ td['builddir'] = builddir
+ td['sigs'] = get_signatures(td['builddir'])
+
+ if not add_layer(bblayersconf, layer, dep_layers, logger):
+ logger.info('Skipping %s ???.' % layer['name'])
+ results[layer['name']] = None
+ results_status[layer['name']] = 'SKIPPED (Unknown)'
+ layers_tested = layers_tested + 1
+ continue
+
result = test_layer_compatibility(td, layer)
results[layer['name']] = result
results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
--
1.8.3.1
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH 0/4] yocto-compat-layer.py updates
2017-03-31 2:30 [PATCH 0/4] yocto-compat-layer.py updates Mark Hatle
` (3 preceding siblings ...)
2017-03-31 2:30 ` [PATCH 4/4] yocto-compat-layer.py: Fix the signature validation Mark Hatle
@ 2017-04-03 19:03 ` Aníbal Limón
4 siblings, 0 replies; 6+ messages in thread
From: Aníbal Limón @ 2017-04-03 19:03 UTC (permalink / raw)
To: Mark Hatle, openembedded-core; +Cc: saul.wold
[-- Attachment #1: Type: text/plain, Size: 1651 bytes --]
Acked-by: Aníbal Limón <anibal.limon@linux.intel.com>
On 03/30/2017 08:30 PM, Mark Hatle wrote:
> The changes in this series fix a number of issues with the yocto-compat-layer
> script.
>
> - When a layer is skipped due to missing dependencies, it was difficult to
> see this without inspecting the whole log.
>
> - There needs to be a way to pass in a list of layers that can be used
> for dependencies, but should not be evaluated in this pass. The overall
> evaluation is very time consuming and having to parse/validate 15
> dependent layers to see if the layer you are working on passes is not
> useful!
>
> - The dependency processing was not working properly. It was only processing
> dependencies at one level. Move this to a recursive process so we're
> sure that all needed dependencies are present.
>
> - Change how the initial signatures are stored to be compared to the layers
> signatures for distro/BSP checks. Before the signatures were stored very
> early (before any dependencies were calculated), causing the dependencies
> for a BSP layer to trigger a failure in signature validation.
>
> Mark Hatle (4):
> yocto-compat-layer.py: Add status for skipped items
> yocto-compat-layer.py: Add --dependency argument
> compatlayer/__init__.py: Allow add_layer to process recursive deps
> yocto-compat-layer.py: Fix the signature validation
>
> scripts/lib/compatlayer/__init__.py | 47 ++++++++++++++++++++++++++++++-------
> scripts/yocto-compat-layer.py | 47 +++++++++++++++++++++++++++----------
> 2 files changed, 73 insertions(+), 21 deletions(-)
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread