[layerindex-web][PATCH 0/4] update.py: several fixes

[layerindex-web][PATCH 0/4] update.py: several fixes

[Robert Yang]

The following changes since commit 5cfdfdca8b9200b5a6e4aa14661b14059a799310:

  rrs_upstream_history.py: fix set_regexes function (2018-06-06 11:25:46 +1200)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/li_recs
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/li_recs

Robert Yang (4):
  update_layer.py: avoid calling setup_core_layer_sys_path() when
    --initial
  utils.py: fix checkout_repo when no HEAD
  update.py: add layers when RECOMMENDS isn't satisfied
  update.py: check whether branch existed when nocheckout

 layerindex/update.py       | 71 +++++++++++++++++++++++++++++++---------------
 layerindex/update_layer.py | 22 +++++++-------
 layerindex/utils.py        | 25 ++++++++++------
 3 files changed, 76 insertions(+), 42 deletions(-)

-- 
2.7.4

[layerindex-web][PATCH 1/4] update_layer.py: avoid calling setup_core_layer_sys_path() when --initial

[Robert Yang]

Fixed:
$ update.py -b <new_branch>
[snip]
NOTE: Starting bitbake server...
Traceback (most recent call last):
  File "update_layer.py", line 471, in main
    utils.setup_core_layer_sys_path(settings, branch.name)
  File "/buildarea1/lyang1/layerindex-web/layerindex/utils.py", line 376, in setup_core_layer_sys_path
    core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
AttributeError: 'NoneType' object has no attribute 'vcs_subdir'
[snip]

This is because core_layerbranch is not in database yet for completely new
branch, so it is None and we will get the error. Avoid calling
setup_core_layer_sys_path() when "update_layer.py --initial" will fix the
problem.

And also only add core layer's sys path when it is present, since core layer
may not be added yet for completely new branch.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update_layer.py | 19 ++++++++++---------
 layerindex/utils.py        |  9 +++++----
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index bbfaba9..81b730a 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -426,15 +426,6 @@ def main():
                 # why won't they just fix that?!)
                 tinfoil.config_data.setVar('LICENSE', '')
 
-                # Set up for recording patch info
-                utils.setup_core_layer_sys_path(settings, branch.name)
-                skip_patches = False
-                try:
-                    import oe.recipeutils
-                except ImportError:
-                    logger.warn('Failed to find lib/oe/recipeutils.py in layers - patch information will not be collected')
-                    skip_patches = True
-
                 layerconfparser = layerconfparse.LayerConfParse(logger=logger, tinfoil=tinfoil)
                 layer_config_data = layerconfparser.parse_layer(layerdir)
                 if not layer_config_data:
@@ -447,6 +438,16 @@ def main():
                     for i in ["BBFILE_COLLECTIONS", "LAYERVERSION", "LAYERDEPENDS", "LAYERRECOMMENDS"]:
                         print('%s = "%s"' % (i, utils.get_layer_var(layer_config_data, i, logger)))
                     sys.exit(0)
+
+                # Set up for recording patch info
+                utils.setup_core_layer_sys_path(settings, branch.name)
+                skip_patches = False
+                try:
+                    import oe.recipeutils
+                except ImportError:
+                    logger.warn('Failed to find lib/oe/recipeutils.py in layers - patch information will not be collected')
+                    skip_patches = True
+
                 utils.add_dependencies(layerbranch, layer_config_data, logger=logger)
                 utils.add_recommends(layerbranch, layer_config_data, logger=logger)
                 layerbranch.save()
diff --git a/layerindex/utils.py b/layerindex/utils.py
index f8c5fd4..c30038d 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -371,10 +371,11 @@ def setup_core_layer_sys_path(settings, branchname):
     """
     core_layer = get_layer(settings.CORE_LAYER_NAME)
     core_layerbranch = core_layer.get_layerbranch(branchname)
-    core_urldir = core_layer.get_fetch_dir()
-    core_repodir = os.path.join(settings.LAYER_FETCH_DIR, core_urldir)
-    core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
-    sys.path.insert(0, os.path.join(core_layerdir, 'lib'))
+    if core_layerbranch:
+        core_urldir = core_layer.get_fetch_dir()
+        core_repodir = os.path.join(settings.LAYER_FETCH_DIR, core_urldir)
+        core_layerdir = os.path.join(core_repodir, core_layerbranch.vcs_subdir)
+        sys.path.insert(0, os.path.join(core_layerdir, 'lib'))
 
 def run_command_interruptible(cmd):
     """
-- 
2.7.4

[layerindex-web][PATCH 2/4] utils.py: fix checkout_repo when no HEAD

[Robert Yang]

Fixed:
$ git clone <url>
warning: remote HEAD refers to nonexistent ref, unable to checkout.
$ git rev-parse HEAD
HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

Catch the error and avoid that.

And use "git reset --hard" to replace of "git reset --hard HEAD", HEAD is
default for git reset, so they are the same, but the later one reports error
when remote HEAD doesn't exist:
$ git reset --hard HEAD
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
[snip]

$ git reset --hard
No errors.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/utils.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/layerindex/utils.py b/layerindex/utils.py
index c30038d..3dc54a1 100644
--- a/layerindex/utils.py
+++ b/layerindex/utils.py
@@ -217,17 +217,25 @@ def checkout_repo(repodir, commit, logger, force=False):
     if force:
         currentref = ''
     else:
-        currentref = runcmd("git rev-parse HEAD", repodir, logger=logger).strip()
+        try:
+            # The "git rev-parse HEAD" returns "fatal: ambiguous argument 'HEAD'"
+            # when a repo is unable to check out after git clone:
+            # git clone <url>
+            # warning: remote HEAD refers to nonexistent ref, unable to checkout.
+            # So check and avoid that
+            currentref = runcmd("git rev-parse HEAD", repodir, logger=logger).strip()
+        except Exception as esc:
+            logger.warn(esc)
+            currentref = ''
     if currentref != commit:
         # Reset in case there are added but uncommitted changes
-        runcmd("git reset --hard HEAD", repodir, logger=logger)
+        runcmd("git reset --hard", repodir, logger=logger)
         # Drop any untracked files in case these cause problems (either because
         # they will exist in the revision we're checking out, or will otherwise
         # interfere with operation, e.g. stale pyc files)
         runcmd("git clean -qdfx", repodir, logger=logger)
         # Now check out the revision
-        runcmd("git checkout %s" % commit,
-                        repodir, logger=logger)
+        runcmd("git checkout %s" % commit, repodir, logger=logger)
 
 def checkout_layer_branch(layerbranch, repodir, logger=None):
     branchname = layerbranch.branch.name
-- 
2.7.4

[layerindex-web][PATCH 3/4] update.py: add layers when RECOMMENDS isn't satisfied

[Robert Yang]

When layer_a RECOMMENDS layer_b, try to add layer_b before layer_a, but if
layer_b is not found, still add layer_a.

And print summary error mesage:

$ update.py -b master

ERROR: Issues found on branch master:
    openembedded-core: Added without LAYERRECOMMENDS
    meta-secure-env: Failed to add since LAYERDEPENDS is not satisfied

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py | 67 ++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 46 insertions(+), 21 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index 06c61a7..5b5fc43 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -299,6 +299,7 @@ def main():
             last_rev = {}
             failed_layers = {}
             for branch in branches:
+                failed_layers[branch] = []
                 # If layer_A depends(or recommends) on layer_B, add layer_B before layer_A
                 deps_dict_all = {}
                 layerquery_sorted = []
@@ -399,28 +400,34 @@ def main():
                     deps = re.search("^LAYERDEPENDS = \"(.*)\"", output, re.M).group(1) or ''
                     recs = re.search("^LAYERRECOMMENDS = \"(.*)\"", output, re.M).group(1) or ''
 
-                    deps_dict = utils.explode_dep_versions2(bitbakepath, deps + ' ' + recs)
-                    if len(deps_dict) == 0:
+                    deps_dict = utils.explode_dep_versions2(bitbakepath, deps)
+                    recs_dict = utils.explode_dep_versions2(bitbakepath, recs)
+                    if not (deps_dict or recs_dict):
                         # No depends, add it firstly
                         layerquery_sorted.append(layer)
                         collections.add((col, ver))
                         continue
-                    deps_dict_all[layer] = {'requires': deps_dict, 'collection': col, 'version': ver}
+                    deps_dict_all[layer] = {'deps': deps_dict, \
+                                            'recs': recs_dict, \
+                                            'collection': col, \
+                                            'version': ver}
 
                 # Move deps_dict_all to layerquery_sorted orderly
-                logger.info("Sorting layers for branch %s" % branch)
+                if deps_dict_all:
+                    logger.info("Sorting layers for branch %s" % branch)
                 while True:
                     deps_dict_all_copy = deps_dict_all.copy()
                     for layer, value in deps_dict_all_copy.items():
-                        for req_col, req_ver_list in value['requires'].copy().items():
-                            matched = False
-                            if req_ver_list:
-                                req_ver = req_ver_list[0]
-                            else:
-                                req_ver = None
-                            if utils.is_deps_satisfied(req_col, req_ver, collections):
-                                del(value['requires'][req_col])
-                        if not value['requires']:
+                        for deps_recs in ('deps', 'recs'):
+                            for req_col, req_ver_list in value[deps_recs].copy().items():
+                                matched = False
+                                if req_ver_list:
+                                    req_ver = req_ver_list[0]
+                                else:
+                                    req_ver = None
+                                if utils.is_deps_satisfied(req_col, req_ver, collections):
+                                    del(value[deps_recs][req_col])
+                        if not (value['deps'] or value['recs']):
                             # All the depends are in collections:
                             del(deps_dict_all[layer])
                             layerquery_sorted.append(layer)
@@ -429,15 +436,32 @@ def main():
                     if not len(deps_dict_all):
                         break
 
-                    # If nothing changed after a run then some dependencies couldn't be resolved
+                    finished = True
+                    # If nothing changed after a run, drop recs and try again
                     if operator.eq(deps_dict_all_copy, deps_dict_all):
+                        for layer, value in deps_dict_all.items():
+                            if value['recs'] and not value['deps']:
+                                # Add it if recs isn't satisfied only.
+                                logger.warn('Adding %s without LAYERRECOMMENDS...' % layer.name)
+                                del(deps_dict_all[layer])
+                                layerquery_sorted.append(layer)
+                                collections.add((value['collection'], value['version']))
+                                failed_msg = '%s: Added without LAYERRECOMMENDS' % layer.name
+                                failed_layers[branch].append(failed_msg)
+                                finished = False
+                                break
+                        if not finished:
+                            continue
                         logger.warning("Cannot find required collections on branch %s:" % branch)
-                        layer_names = []
                         for layer, value in deps_dict_all.items():
-                            logger.error('%s: %s' % (layer.name, value['requires']))
-                            layer_names.append(layer.name)
+                            logger.warn('%s: LAYERDEPENDS: %s LAYERRECOMMENDS: %s' % (layer.name, value['deps'], value['recs']))
+                            if value['deps']:
+                                failed_layers[branch].append('%s: Failed to add since LAYERDEPENDS is not satisfied' % layer.name)
+                            else:
+                                # Should never come here
+                                logger.error("Unexpected errors when sorting layers")
+                                sys.exit(1)
                         logger.warning("Known collections on branch %s: %s" % (branch, collections))
-                        failed_layers[branch] = layer_names
                         break
 
                 for layer in layerquery_sorted:
@@ -479,10 +503,11 @@ def main():
                         logger.info('Update interrupted, exiting')
                         sys.exit(254)
             if failed_layers:
-                print()
                 for branch, err_msg_list in failed_layers.items():
-                    logger.error("Failed layers on branch %s: %s" % (branch, " ".join(err_msg_list)))
-                print()
+                    if err_msg_list:
+                        print()
+                        logger.error("Issues found on branch %s:\n    %s" % (branch, "\n    ".join(err_msg_list)))
+                        print()
         finally:
             utils.unlock_file(lockfile)
 
-- 
2.7.4

[layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

[Robert Yang]

Fixed:
Assume there is no master branch in hello layer:
$ update.py -l hello -b master
INFO: Skipping update of layer hello - branch master doesn't exist

This is correct since hello layer doesn't have master branch, but when --nocheckout:
$ update.py -l hello -b master  --nocheckout
[snip]
INFO: Sorting layers for branch mater:
WARNING: Cannot find required collections on branch master:
WARNING: hello: LAYERDEPENDS: <snip>

This is incorrect, this patch fixed the problem, now it skips it since the
branch doesn't exists when --nocheckout.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 layerindex/update.py       | 4 ++--
 layerindex/update_layer.py | 3 +--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/layerindex/update.py b/layerindex/update.py
index 5b5fc43..c0a9f88 100755
--- a/layerindex/update.py
+++ b/layerindex/update.py
@@ -345,10 +345,10 @@ def main():
                     repo = git.Repo(repodir)
                     assert repo.bare == False
                     try:
+                        # Always get origin/branchname, so it raises error when branch doesn't exist when nocheckout
+                        topcommit = repo.commit('origin/%s' % branchname)
                         if options.nocheckout:
                             topcommit = repo.commit('HEAD')
-                        else:
-                            topcommit = repo.commit('origin/%s' % branchname)
                     except:
                         if newbranch:
                             logger.info("Skipping update of layer %s - branch %s doesn't exist" % (layer.name, branchdesc))
diff --git a/layerindex/update_layer.py b/layerindex/update_layer.py
index 81b730a..d941ae6 100644
--- a/layerindex/update_layer.py
+++ b/layerindex/update_layer.py
@@ -362,10 +362,9 @@ def main():
     # Collect repo info
     repo = git.Repo(repodir)
     assert repo.bare == False
+    topcommit = repo.commit('origin/%s' % branchname)
     if options.nocheckout:
         topcommit = repo.commit('HEAD')
-    else:
-        topcommit = repo.commit('origin/%s' % branchname)
 
     tinfoil = None
     tempdir = None
-- 
2.7.4

Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

[Paul Eggleton]

Hi Robert,

On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote:
> Fixed:
> Assume there is no master branch in hello layer:
> $ update.py -l hello -b master
> INFO: Skipping update of layer hello - branch master doesn't exist
> 
> This is correct since hello layer doesn't have master branch, but when --nocheckout:
> $ update.py -l hello -b master  --nocheckout
> [snip]
> INFO: Sorting layers for branch mater:
> WARNING: Cannot find required collections on branch master:
> WARNING: hello: LAYERDEPENDS: <snip>
> 
> This is incorrect, this patch fixed the problem, now it skips it since the
> branch doesn't exists when --nocheckout.

This fix doesn't seem right. The intention of --nocheckout is to not check anything
out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct
to get the commit from the origin in this case.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

[Paul Eggleton]

On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote:
> Hi Robert,
> 
> On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote:
> > Fixed:
> > Assume there is no master branch in hello layer:
> > $ update.py -l hello -b master
> > INFO: Skipping update of layer hello - branch master doesn't exist
> > 
> > This is correct since hello layer doesn't have master branch, but when --nocheckout:
> > $ update.py -l hello -b master  --nocheckout
> > [snip]
> > INFO: Sorting layers for branch mater:
> > WARNING: Cannot find required collections on branch master:
> > WARNING: hello: LAYERDEPENDS: <snip>
> > 
> > This is incorrect, this patch fixed the problem, now it skips it since the
> > branch doesn't exists when --nocheckout.
> 
> This fix doesn't seem right. The intention of --nocheckout is to not check anything
> out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct
> to get the commit from the origin in this case.

Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

[Robert Yang]

Hi Paul,

On 07/09/2018 04:34 PM, Paul Eggleton wrote:
> Hi Robert,
> 
> On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote:
>> Fixed:
>> Assume there is no master branch in hello layer:
>> $ update.py -l hello -b master
>> INFO: Skipping update of layer hello - branch master doesn't exist
>>
>> This is correct since hello layer doesn't have master branch, but when --nocheckout:
>> $ update.py -l hello -b master  --nocheckout
>> [snip]
>> INFO: Sorting layers for branch mater:
>> WARNING: Cannot find required collections on branch master:
>> WARNING: hello: LAYERDEPENDS: <snip>
>>
>> This is incorrect, this patch fixed the problem, now it skips it since the
>> branch doesn't exists when --nocheckout.
> 
> This fix doesn't seem right. The intention of --nocheckout is to not check anything
> out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct
> to get the commit from the origin in this case.

It still uses HEAD when nocheckout, please see the code:

# Always get origin/branchname, so it raises error when branch doesn't exist 
when nocheckout
     topcommit = repo.commit('origin/%s' % branchname)
     if options.nocheckout:
         topcommit = repo.commit('HEAD')

I just want to make sure origin/branchname is existed, and raise error when it
doesn't exist (update.py catches the error), it still uses HEAD when --nocheckout.

// Robert
> 
> Cheers,
> Paul
> 

Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

[Robert Yang]

On 07/09/2018 04:36 PM, Paul Eggleton wrote:
> On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote:
>> Hi Robert,
>>
>> On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote:
>>> Fixed:
>>> Assume there is no master branch in hello layer:
>>> $ update.py -l hello -b master
>>> INFO: Skipping update of layer hello - branch master doesn't exist
>>>
>>> This is correct since hello layer doesn't have master branch, but when --nocheckout:
>>> $ update.py -l hello -b master  --nocheckout
>>> [snip]
>>> INFO: Sorting layers for branch mater:
>>> WARNING: Cannot find required collections on branch master:
>>> WARNING: hello: LAYERDEPENDS: <snip>
>>>
>>> This is incorrect, this patch fixed the problem, now it skips it since the
>>> branch doesn't exists when --nocheckout.
>>
>> This fix doesn't seem right. The intention of --nocheckout is to not check anything
>> out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct
>> to get the commit from the origin in this case.
> 
> Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it.

Do I need update commit message, please ?

// Robert

> 
> Cheers,
> Paul
> 

Re: [layerindex-web][PATCH 4/4] update.py: check whether branch existed when nocheckout

[Paul Eggleton]

On Monday, 9 July 2018 10:45:58 AM CEST Robert Yang wrote:
> On 07/09/2018 04:36 PM, Paul Eggleton wrote:
> > On Monday, 9 July 2018 10:34:07 AM CEST Paul Eggleton wrote:
> >> On Monday, 9 July 2018 6:11:30 AM CEST Robert Yang wrote:
> >>> Fixed:
> >>> Assume there is no master branch in hello layer:
> >>> $ update.py -l hello -b master
> >>> INFO: Skipping update of layer hello - branch master doesn't exist
> >>>
> >>> This is correct since hello layer doesn't have master branch, but when --nocheckout:
> >>> $ update.py -l hello -b master  --nocheckout
> >>> [snip]
> >>> INFO: Sorting layers for branch mater:
> >>> WARNING: Cannot find required collections on branch master:
> >>> WARNING: hello: LAYERDEPENDS: <snip>
> >>>
> >>> This is incorrect, this patch fixed the problem, now it skips it since the
> >>> branch doesn't exists when --nocheckout.
> >>
> >> This fix doesn't seem right. The intention of --nocheckout is to not check anything
> >> out, i.e. use whatever HEAD is in the current repository. It wouldn't be correct
> >> to get the commit from the origin in this case.
> > 
> > Actually, on a second reading of the change. It's not obvious why this is needed but I suppose the comment clarifies it.
> 
> Do I need update commit message, please ?

No, it's OK, I'll merge as-is.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre