[1.52][PATCH 0/5] Patch review

[1.52][PATCH 0/5] Patch review

[Anuj Mittal]

Please review and merge these changes for 1.52/honister. No issues seen
while testing on autobuilder.

Thanks,

Anuj

The following changes since commit 1428128c787642dca690a7d51f2be793e001ae37:

  tests/fetch: Handle upstream master -> main branch change (2022-02-23 18:09:52 +0000)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.52-next

Daniel Gomez (1):
  contrib: Fix hash server Dockerfile dependencies

Daniel Wagenknecht (1):
  fetch2: ssh: username and password are optional

Richard Purdie (3):
  data_smart: Fix overrides file/line message additions
  cooker: Improve parsing failure from handled exception usability
  utils: Ensure shell function failure in python logging is correct

 contrib/hashserv/Dockerfile | 6 +++++-
 lib/bb/cooker.py            | 4 +++-
 lib/bb/data_smart.py        | 8 ++++----
 lib/bb/fetch2/ssh.py        | 2 +-
 lib/bb/utils.py             | 2 +-
 5 files changed, 14 insertions(+), 8 deletions(-)

-- 
2.35.1

[1.52][PATCH 1/5] contrib: Fix hash server Dockerfile dependencies

[Anuj Mittal]

From: Daniel Gomez <daniel@qtec.com>

When building the Hash Equivalence server Dockerfile, some
dependencies are missing in order to run the hash server properly:

Traceback errors:

Traceback (most recent call last):
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
    ret = main()
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
    server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
  File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
    from . import server
  File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
    import bb.asyncrpc
ModuleNotFoundError: No module named 'bb'

Traceback (most recent call last):
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
    ret = main()
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
    server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
  File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
    from . import server
  File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
    import bb.asyncrpc
  File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
    from bb import fetch2 as fetch
  File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
    from . import git
  File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
    import bb.progress
  File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
    import bb.build
  File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
    from bb import data, event, utils
  File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
    from bb import data_smart
  File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
    import bb, bb.codeparser
  File "/opt/bbhashserv/lib/bb/codeparser.py", line 26, in <module>
    import codegen
ModuleNotFoundError: No module named 'codegen'

Traceback (most recent call last):
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
    ret = main()
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
    server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
  File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
    from . import server
  File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
    import bb.asyncrpc
  File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
    from bb import fetch2 as fetch
  File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1930, in <module>
    from . import git
  File "/opt/bbhashserv/lib/bb/fetch2/git.py", line 70, in <module>
    import bb.progress
  File "/opt/bbhashserv/lib/bb/progress.py", line 14, in <module>
    import bb.build
  File "/opt/bbhashserv/lib/bb/build.py", line 27, in <module>
    from bb import data, event, utils
  File "/opt/bbhashserv/lib/bb/data.py", line 36, in <module>
    from bb import data_smart
  File "/opt/bbhashserv/lib/bb/data_smart.py", line 23, in <module>
    import bb, bb.codeparser
  File "/opt/bbhashserv/lib/bb/codeparser.py", line 32, in <module>
    from bb.pysh import pyshyacc, pyshlex
  File "/opt/bbhashserv/lib/bb/pysh/pyshyacc.py", line 13, in <module>
    import bb.pysh.pyshlex as pyshlex
  File "/opt/bbhashserv/lib/bb/pysh/pyshlex.py", line 17, in <module>
    from ply import lex
ModuleNotFoundError: No module named 'ply'

Traceback (most recent call last):
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 61, in <module>
    ret = main()
  File "/opt/bbhashserv/bin/bitbake-hashserv", line 54, in main
    server = hashserv.create_server(args.bind, args.database, upstream=args.upstream, read_only=args.read_only)
  File "/opt/bbhashserv/lib/hashserv/__init__.py", line 120, in create_server
    from . import server
  File "/opt/bbhashserv/lib/hashserv/server.py", line 14, in <module>
    import bb.asyncrpc
  File "/opt/bbhashserv/lib/bb/__init__.py", line 128, in <module>
    from bb import fetch2 as fetch
  File "/opt/bbhashserv/lib/bb/fetch2/__init__.py", line 1935, in <module>
    from . import wget
  File "/opt/bbhashserv/lib/bb/fetch2/wget.py", line 30, in <module>
    from   bs4 import BeautifulSoup
ModuleNotFoundError: No module named 'bs4'

Signed-off-by: Daniel Gomez <daniel@qtec.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0bd637acfcba5a44230c291889d2a5ff571cb8c6)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 contrib/hashserv/Dockerfile | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/contrib/hashserv/Dockerfile b/contrib/hashserv/Dockerfile
index d6fc728f..74b4a3be 100644
--- a/contrib/hashserv/Dockerfile
+++ b/contrib/hashserv/Dockerfile
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: MIT
 #
 # Copyright (c) 2021 Joshua Watt <JPEWhacker@gmail.com>
-# 
+#
 # Dockerfile to build a bitbake hash equivalence server container
 #
 # From the root of the bitbake repository, run:
@@ -15,5 +15,9 @@ RUN apk add --no-cache python3
 
 COPY bin/bitbake-hashserv /opt/bbhashserv/bin/
 COPY lib/hashserv /opt/bbhashserv/lib/hashserv/
+COPY lib/bb /opt/bbhashserv/lib/bb/
+COPY lib/codegen.py /opt/bbhashserv/lib/codegen.py
+COPY lib/ply /opt/bbhashserv/lib/ply/
+COPY lib/bs4 /opt/bbhashserv/lib/bs4/
 
 ENTRYPOINT ["/opt/bbhashserv/bin/bitbake-hashserv"]
-- 
2.35.1

[1.52][PATCH 2/5] data_smart: Fix overrides file/line message additions

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The overrides warning message is meant to show filename and line
numbers but the variable names are incorrect and this wasn't working.
Fix it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 551c1cb20fc9b9d0dab5d830182c2bf626e72845)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/data_smart.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 5ffedebc..e9603278 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -494,10 +494,10 @@ class DataSmart(MutableMapping):
 
         if not var.startswith("__anon_") and ("_append" in var or "_prepend" in var or "_remove" in var):
             info = "%s" % var
-            if "filename" in loginfo:
-                info += " file: %s" % loginfo[filename]
-            if "lineno" in loginfo:
-                info += " line: %s" % loginfo[lineno]
+            if "file" in loginfo:
+                info += " file: %s" % loginfo["file"]
+            if "line" in loginfo:
+                info += " line: %s" % loginfo["line"]
             bb.fatal("Variable %s contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake." % info)
 
         self.expand_cache = {}
-- 
2.35.1

[1.52][PATCH 3/5] cooker: Improve parsing failure from handled exception usability

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

When a recipe raises a BBHandledException, it means the error was already
shown to the user. Adding an additional one here isn't helpful. What is
helpful is to mention that parsing was halted.

Tweak the code to do this with improves the messages the user sees
and helps understand what happened.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit fdf6ebc8b603fcfd3ed7c64baf486a4adabd25be)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 3688aa08..288d73fd 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2167,6 +2167,8 @@ class CookerParser(object):
                                             self.total)
 
             bb.event.fire(event, self.cfgdata)
+        else:
+            bb.error("Parsing halted due to errors")
 
         for process in self.processes:
             self.parser_quit.put(None)
@@ -2257,7 +2259,7 @@ class CookerParser(object):
             return False
         except bb.BBHandledException as exc:
             self.error += 1
-            logger.error('Failed to parse recipe: %s' % exc.recipe)
+            logger.debug('Failed to parse recipe: %s' % exc.recipe)
             self.shutdown(clean=False, force=True)
             return False
         except ParsingFailure as exc:
-- 
2.35.1

[1.52][PATCH 4/5] utils: Ensure shell function failure in python logging is correct

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If a python function exec_func() calls a shell task, the logging wasn't working
correctly in all cases since the exception was turned into a BBHandledException()
and the logfile piece was lost which is handled at the top task level.

The easiest way to avoid this is to allow the ExecutionError exceptions to
be raised to a higher level, we don't need the traceback for them.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 7cae11f558f9ff5fd05ef23b789aaef92fb5a327)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/utils.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 1a515897..cd442dcd 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -401,7 +401,7 @@ def better_exec(code, context, text = None, realfile = "<code>", pythonexception
         code = better_compile(code, realfile, realfile)
     try:
         exec(code, get_context(), context)
-    except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError):
+    except (bb.BBHandledException, bb.parse.SkipRecipe, bb.data_smart.ExpansionError, bb.process.ExecutionError):
         # Error already shown so passthrough, no need for traceback
         raise
     except Exception as e:
-- 
2.35.1

[1.52][PATCH 5/5] fetch2: ssh: username and password are optional

[Anuj Mittal]

From: Daniel Wagenknecht <dwagenknecht@emlix.com>

Support URLs like ssh://HOST/PATH. They were previously not recognized
due to a missing @ sign.

Signed-off-by: Daniel Wagenknecht <dwagenknecht@emlix.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit a2aa18bd27dac8902e52b466cb7118f71367d3dc)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/fetch2/ssh.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/ssh.py b/lib/bb/fetch2/ssh.py
index 2c8557e1..a104c9eb 100644
--- a/lib/bb/fetch2/ssh.py
+++ b/lib/bb/fetch2/ssh.py
@@ -40,9 +40,9 @@ __pattern__ = re.compile(r'''
  (                   # Optional username/password block
   (?P<user>\S+)      # username
   (:(?P<pass>\S+))?  # colon followed by the password (optional)
- )?
  (?P<cparam>(;[^;]+)*)?  # connection parameters block (optional)
  @
+ )?
  (?P<host>\S+?)          # non-greedy match of the host
  (:(?P<port>[0-9]+))?    # colon followed by the port (optional)
  /
-- 
2.35.1