[1.50][PATCH 0/8] Review request

[1.50][PATCH 0/8] Review request

[Anuj Mittal]

Please review and merge these changes for 1.50. No issues seen while
testing on autobuilder.

Thanks,

Anuj

The following changes since commit 15dadb69b4c5d29b80770e55d1e9afbe47084aa4:

  runqueue: Fix runall option handling (2021-11-16 11:28:01 +0800)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.50-next

Alexander Kanavin (3):
  bitbake: correct the collections vs collections.abc deprecation
  bitbake: correct deprecation warning in process.py
  bitbake: adjust parser error check for python 3.10 compatibility

Justin Bronder (1):
  hashserv: let asyncio discover the running loop

Richard Purdie (4):
  cooker: Handle parse threads disappearing to avoid hangs
  cooker: Remove debug code, oops :(
  cooker: Handle parsing results queue race
  cooker: Fix task-depends.dot for multiconfig targets

 lib/bb/cache.py          |  3 ++-
 lib/bb/cooker.py         | 18 ++++++++++++++++--
 lib/bb/data_smart.py     |  4 ++--
 lib/bb/persist_data.py   |  5 +++--
 lib/bb/server/process.py |  2 +-
 lib/hashserv/server.py   |  4 ++--
 6 files changed, 26 insertions(+), 10 deletions(-)

-- 
2.33.1

[1.50][PATCH 1/8] bitbake: correct the collections vs collections.abc deprecation

[Anuj Mittal]

From: Alexander Kanavin <alex.kanavin@gmail.com>

This becomes a hard error in python 3.10.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit ae219e1f7460077f4492b31ac91cef4cf9b17277)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cache.py        | 3 ++-
 lib/bb/data_smart.py   | 2 +-
 lib/bb/persist_data.py | 5 +++--
 3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/lib/bb/cache.py b/lib/bb/cache.py
index 27eb2717..5f9c0a77 100644
--- a/lib/bb/cache.py
+++ b/lib/bb/cache.py
@@ -19,7 +19,8 @@
 import os
 import logging
 import pickle
-from collections import defaultdict, Mapping
+from collections import defaultdict
+from collections.abc import Mapping
 import bb.utils
 from bb import PrefixLoggerAdapter
 import re
diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 8291ca65..aa9ac2c8 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -17,7 +17,7 @@ BitBake build tools.
 # Based on functions from the base bb module, Copyright 2003 Holger Schurig
 
 import copy, re, sys, traceback
-from collections import MutableMapping
+from collections.abc import MutableMapping
 import logging
 import hashlib
 import bb, bb.codeparser
diff --git a/lib/bb/persist_data.py b/lib/bb/persist_data.py
index c6a209fb..6f32d81a 100644
--- a/lib/bb/persist_data.py
+++ b/lib/bb/persist_data.py
@@ -12,6 +12,7 @@ currently, providing a key/value store accessed by 'domain'.
 #
 
 import collections
+import collections.abc
 import contextlib
 import functools
 import logging
@@ -19,7 +20,7 @@ import os.path
 import sqlite3
 import sys
 import warnings
-from collections import Mapping
+from collections.abc import Mapping
 
 sqlversion = sqlite3.sqlite_version_info
 if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
@@ -29,7 +30,7 @@ if sqlversion[0] < 3 or (sqlversion[0] == 3 and sqlversion[1] < 3):
 logger = logging.getLogger("BitBake.PersistData")
 
 @functools.total_ordering
-class SQLTable(collections.MutableMapping):
+class SQLTable(collections.abc.MutableMapping):
     class _Decorators(object):
         @staticmethod
         def retry(*, reconnect=True):
-- 
2.33.1

[1.50][PATCH 2/8] bitbake: correct deprecation warning in process.py

[Anuj Mittal]

From: Alexander Kanavin <alex.kanavin@gmail.com>

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit aff52fe21a0b27f6302555c1e52a864550eb46ce)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/server/process.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/server/process.py b/lib/bb/server/process.py
index 07bb785a..fcdce197 100644
--- a/lib/bb/server/process.py
+++ b/lib/bb/server/process.py
@@ -659,7 +659,7 @@ class BBUIEventQueue:
         self.reader = ConnectionReader(readfd)
 
         self.t = threading.Thread()
-        self.t.setDaemon(True)
+        self.t.daemon = True
         self.t.run = self.startCallbackHandler
         self.t.start()
 
-- 
2.33.1

[1.50][PATCH 3/8] bitbake: adjust parser error check for python 3.10 compatibility

[Anuj Mittal]

From: Alexander Kanavin <alex.kanavin@gmail.com>

The change was introduced in
https://github.com/python/cpython/commit/a698d52c3975c80b45b139b2f08402ec514dce75

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8d3c6cbbe6ee734495713ae3b99c609527842506)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index aa9ac2c8..65857a9c 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -403,7 +403,7 @@ class DataSmart(MutableMapping):
                     s = __expand_python_regexp__.sub(varparse.python_sub, s)
                 except SyntaxError as e:
                     # Likely unmatched brackets, just don't expand the expression
-                    if e.msg != "EOL while scanning string literal":
+                    if e.msg != "EOL while scanning string literal" and not e.msg.startswith("unterminated string literal"):
                         raise
                 if s == olds:
                     break
-- 
2.33.1

[1.50][PATCH 4/8] hashserv: let asyncio discover the running loop

[Anuj Mittal]

From: Justin Bronder <jsbronder@cold-front.org>

>From 3.10 documentation [1]:
    Deprecated since version 3.8, removed in version 3.10: The loop
    parameter. This function has been implicitly getting the current
    running loop since 3.7

This is fixed in master as a side-effect of
cf9bc0310b0092bf52b61057405aeb51c86ba137 which is more intrusive but
likewise drops the loop parameter.

1. https://docs.python.org/3/library/asyncio-stream.html#asyncio.open_connection

Signed-off-by: Justin Bronder <jsbronder@cold-front.org>
Signed-off-by: Steve Sakoman <steve@sakoman.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 74a1e71b1e677a482fdedc685a71a1798ad63920)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/hashserv/server.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/hashserv/server.py b/lib/hashserv/server.py
index a0dc0c17..df0fa0a0 100644
--- a/lib/hashserv/server.py
+++ b/lib/hashserv/server.py
@@ -521,7 +521,7 @@ class Server(object):
 
     def start_tcp_server(self, host, port):
         self.server = self.loop.run_until_complete(
-            asyncio.start_server(self.handle_client, host, port, loop=self.loop)
+            asyncio.start_server(self.handle_client, host, port)
         )
 
         for s in self.server.sockets:
@@ -546,7 +546,7 @@ class Server(object):
             # Work around path length limits in AF_UNIX
             os.chdir(os.path.dirname(path))
             self.server = self.loop.run_until_complete(
-                asyncio.start_unix_server(self.handle_client, os.path.basename(path), loop=self.loop)
+                asyncio.start_unix_server(self.handle_client, os.path.basename(path))
             )
         finally:
             os.chdir(cwd)
-- 
2.33.1

[1.50][PATCH 5/8] cooker: Handle parse threads disappearing to avoid hangs

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

If one of the parse threads disappears during parsing for some reason, bitbake
currently hangs. Avoid this (and zombie threads hanging around) by joining()
threads which have exited.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit dc86a533d951d13643ce446533370da804782afc)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index b041d2a0..f12f4caa 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2036,6 +2036,7 @@ class Parser(multiprocessing.Process):
                 result = pending.pop()
             else:
                 try:
+                    time.sleep(0.25)
                     job = self.jobs.pop()
                 except IndexError:
                     self.results.close()
@@ -2214,7 +2215,7 @@ class CookerParser(object):
             yield not cached, mc, infos
 
     def parse_generator(self):
-        while True:
+        while self.processes:
             if self.parsed >= self.toparse:
                 break
 
@@ -2228,6 +2229,14 @@ class CookerParser(object):
                     raise value
                 else:
                     yield result
+            for process in self.processes.copy():
+                if not process.is_alive():
+                    process.join()
+                    self.processes.remove(process)
+
+        if not (self.parsed >= self.toparse):
+            raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None)
+
 
     def parse_next(self):
         result = []
-- 
2.33.1

[1.50][PATCH 6/8] cooker: Remove debug code, oops :(

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 19291665fa8b6cc331290f2542af3e8e653203f1)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 1 -
 1 file changed, 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index f12f4caa..c952c574 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2036,7 +2036,6 @@ class Parser(multiprocessing.Process):
                 result = pending.pop()
             else:
                 try:
-                    time.sleep(0.25)
                     job = self.jobs.pop()
                 except IndexError:
                     self.results.close()
-- 
2.33.1

[1.50][PATCH 7/8] cooker: Handle parsing results queue race

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The previous fix introduced a race where the queue might not be empty
but all the parser processes have exited. Handle this correctly to avoid
occasional errors.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8e7f2b6500e26610f52d128b48ca0a09bf6fb2cb)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index c952c574..e2a5dc43 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2214,24 +2214,28 @@ class CookerParser(object):
             yield not cached, mc, infos
 
     def parse_generator(self):
-        while self.processes:
+        empty = False
+        while self.processes or not empty:
+            for process in self.processes.copy():
+                if not process.is_alive():
+                    process.join()
+                    self.processes.remove(process)
+
             if self.parsed >= self.toparse:
                 break
 
             try:
                 result = self.result_queue.get(timeout=0.25)
             except queue.Empty:
+                empty = True
                 pass
             else:
+                empty = False
                 value = result[1]
                 if isinstance(value, BaseException):
                     raise value
                 else:
                     yield result
-            for process in self.processes.copy():
-                if not process.is_alive():
-                    process.join()
-                    self.processes.remove(process)
 
         if not (self.parsed >= self.toparse):
             raise bb.parse.ParseError("Not all recipes parsed, parser thread killed/died? Exiting.", None)
-- 
2.33.1

[1.50][PATCH 8/8] cooker: Fix task-depends.dot for multiconfig targets

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

The right hand side of dependencies in the task dependency file generated
by bitbake -g was missing multiconfig prefixes, corrupting the data. Fix
this.

[YOCTO #14621]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1d5ca721040c5e39aefa11219f62710de6587701)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index e2a5dc43..c946800a 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -815,7 +815,9 @@ class BBCooker:
             for dep in rq.rqdata.runtaskentries[tid].depends:
                 (depmc, depfn, _, deptaskfn) = bb.runqueue.split_tid_mcfn(dep)
                 deppn = self.recipecaches[depmc].pkg_fn[deptaskfn]
-                depend_tree["tdepends"][dotname].append("%s.%s" % (deppn, bb.runqueue.taskname_from_tid(dep)))
+                if depmc:
+                    depmc = "mc:" + depmc + ":"
+                depend_tree["tdepends"][dotname].append("%s%s.%s" % (depmc, deppn, bb.runqueue.taskname_from_tid(dep)))
             if taskfn not in seen_fns:
                 seen_fns.append(taskfn)
                 packages = []
-- 
2.33.1

[1.50][PATCH 0/8] Review request

[Anuj Mittal]

Please review these changes for 1.50. Tested with oe-core and no issues
seen on autobuilder.

Thanks,

Anuj

The following changes since commit bf59c0080763e8d03f75fe130ae9708e572391f3:

  fetch/wget: Add timeout for checkstatus calls (30s) (2021-11-11 10:59:06 +0000)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.50-next

Jon Mason (1):
  bitbake:toaster:test: Update SSTATE URL

Jose Quaresma (1):
  cooker: check if upstream hash equivalence server is available

Richard Purdie (6):
  fetch2/perforce: Fix typo
  tests/fetch: Update github urls
  fetch: Handle mirror user/password replacements correctly
  tests/fetch: Update pcre.org address after github changes
  runqueue: Fix runall option task deletion ordering issue
  runqueue: Fix runall option handling

 lib/bb/cooker.py                      | 12 +++++++++-
 lib/bb/fetch2/__init__.py             |  4 ++++
 lib/bb/fetch2/perforce.py             |  2 +-
 lib/bb/runqueue.py                    | 34 +++++++++++++--------------
 lib/bb/tests/fetch.py                 | 32 ++++++++++++++-----------
 lib/toaster/tests/builds/buildtest.py |  2 +-
 6 files changed, 51 insertions(+), 35 deletions(-)

-- 
2.33.1