[1.48][PATCH 0/3] Patch review request

[1.48][PATCH 0/3] Patch review request

[Anuj Mittal]

Please review these next set of changes for 1.48. Clean a-full on
autobuilder.

Thanks,

Anuj

The following changes since commit 7a6517243dbfecfd78f82f6709fbeaab2fb03e2c:

  __init__.py: Fix bitbake debug log handling (2021-03-09 00:03:19 +0000)

are available in the Git repository at:

  git://push.openembedded.org/bitbake-contrib stable/1.48-next

Jan Brzezanski (1):
  Force parser shutdown after catching an exception

Richard Purdie (2):
  runqueue: Fix task execution corruption issue
  runqueue: Add setscene task overlap sanity check

 lib/bb/cooker.py   | 10 +++++-----
 lib/bb/runqueue.py |  7 +++++++
 2 files changed, 12 insertions(+), 5 deletions(-)

-- 
2.30.2

[1.48][PATCH 1/3] runqueue: Fix task execution corruption issue

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We've seen occasional issues where linux-yocto:do_compile_kernelmodules would
run without do_shared_workdir running before it. do_shared_workdir is an
setscene task but never has an sstate object generated so it will always
rerun. This should not happen since compile_kernemodules should only
execute if a setscene that depends on it didn't run and that should trigger
do_shared_workdir not to be marked as covered.

The issue is that build-appliance-image:do_package is one of the tasks which
covers linux-yocto:do_compile_kernelmodules but it is also a noexec task
and has a dependecy on pseudo-native:do_populate_sysroot.

In the problem case, pseudo-native:do_populate_sysroot is unavailable but
marked as covered since it is noexec. The "harddeps" code then also marks it
as notcovered. No task should ever be both covered and notcovered and this
is where the problems come from.

The solution is for the harddeps code only to to fail tasks if they've not
already been handled in some way. The code is assuming code couldn't have
handled revdeps at this point but we now have clear evidence they can.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f66556bbb38449789ceea2fd105e9f68df7fb660)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 28bdadb45..e22ca72ce 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2430,6 +2430,9 @@ class RunQueueExecute:
 
         for dep in sorted(self.sqdata.sq_deps[task]):
             if fail and task in self.sqdata.sq_harddeps and dep in self.sqdata.sq_harddeps[task]:
+                if dep in self.scenequeue_covered or dep in self.scenequeue_notcovered:
+                    # dependency could be already processed, e.g. noexec setscene task
+                    continue
                 logger.debug(2, "%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
                 self.sq_task_failoutright(dep)
                 continue
-- 
2.30.2

[1.48][PATCH 2/3] runqueue: Add setscene task overlap sanity check

[Anuj Mittal]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

We've seen hard to debug issues where a task ends up in both the
covered and notcovered list. Add a sanity check to ensure if this
happens in future, we see it in the logs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 6e001410854792f9bb66a0409a2ac176171b0507)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/runqueue.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index e22ca72ce..acc128ec4 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1942,6 +1942,10 @@ class RunQueueExecute:
             logger.error("Scenequeue had holdoff tasks: %s" % pprint.pformat(self.holdoff_tasks))
             err = True
 
+        for tid in self.scenequeue_covered.intersection(self.scenequeue_notcovered):
+            # No task should end up in both covered and uncovered, that is a bug.
+            logger.error("Setscene task %s in both covered and notcovered." % tid)
+
         for tid in self.rqdata.runq_setscene_tids:
             if tid not in self.scenequeue_covered and tid not in self.scenequeue_notcovered:
                 err = True
-- 
2.30.2

[1.48][PATCH 3/3] Force parser shutdown after catching an exception

[Anuj Mittal]

From: Jan Brzezanski <jan.brzezanski@gmail.com>

Commit bebef58b21bdff7a3ee1fa2449b7df19144f26fd introduced forcing
parser shutdown as default in case of build abort.

In this case bitbake sometimes hangs after facing error during parsing,
waiting for child processes to finish. Killing it then will spawn zombie
processes.

Thus we force the shutdown after catching an exception.

Signed-off-by: Jan Brzezanski <jan.brzezanski@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 915330e1dbae1ee8fd9a0358decf2c294f771961)
Signed-off-by: Anuj Mittal <anuj.mittal@intel.com>
---
 lib/bb/cooker.py | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 1f4cc1e96..7ed0b802c 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -2207,18 +2207,18 @@ class CookerParser(object):
         except bb.BBHandledException as exc:
             self.error += 1
             logger.error('Failed to parse recipe: %s' % exc.recipe)
-            self.shutdown(clean=False)
+            self.shutdown(clean=False, force=True)
             return False
         except ParsingFailure as exc:
             self.error += 1
             logger.error('Unable to parse %s: %s' %
                      (exc.recipe, bb.exceptions.to_string(exc.realexception)))
-            self.shutdown(clean=False)
+            self.shutdown(clean=False, force=True)
             return False
         except bb.parse.ParseError as exc:
             self.error += 1
             logger.error(str(exc))
-            self.shutdown(clean=False)
+            self.shutdown(clean=False, force=True)
             return False
         except bb.data_smart.ExpansionError as exc:
             self.error += 1
@@ -2227,7 +2227,7 @@ class CookerParser(object):
             tb = list(itertools.dropwhile(lambda e: e.filename.startswith(bbdir), exc.traceback))
             logger.error('ExpansionError during parsing %s', value.recipe,
                          exc_info=(etype, value, tb))
-            self.shutdown(clean=False)
+            self.shutdown(clean=False, force=True)
             return False
         except Exception as exc:
             self.error += 1
@@ -2239,7 +2239,7 @@ class CookerParser(object):
                 # Most likely, an exception occurred during raising an exception
                 import traceback
                 logger.error('Exception during parse: %s' % traceback.format_exc())
-            self.shutdown(clean=False)
+            self.shutdown(clean=False, force=True)
             return False
 
         self.current += 1
-- 
2.30.2