All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/7] bitbake/runqueue: rework 'bitbake -S printdiff' logic
@ 2023-12-18  8:43 Alexander Kanavin
  2023-12-18  8:43 ` [PATCH 2/7] selftest/sstatetests: fix up printdiff test to match rework of printdiff logic Alexander Kanavin
                   ` (5 more replies)
  0 siblings, 6 replies; 13+ messages in thread
From: Alexander Kanavin @ 2023-12-18  8:43 UTC (permalink / raw)
  To: openembedded-core; +Cc: Alexander Kanavin

Previously printdiff code would iterate over tasks that were reported as invalid or absent,
trying to follow dependency chains that would reach the most basic invalid items in the tree.

While this works in tightly controlled local builds, it can lead to bizarre reports
against industrial-sized sstate caches, as the code would not consider whether the
overall target can be fulfilled from valid sstate objects, and instead report
missing sstate signature files that perhaps were never even created due to hash
equivalency providing shortcuts in builds.

This commit reworks the logic in two ways:

- start the iteration over final targets rather than missing objects
and try to recursively arrive at the root of the invalid object dependency.

A previous version of this patch relied relies on finding the most 'recent'
signature in stamps or sstate in a different function later, and recursively
comparing that to the current signature, which is unreliable on real world caches.

- if a given object can be fulfilled from sstate, recurse only into
its setscene dependencies; bitbake wouldn't care if dependencies
for the actual task are absent, and neither should printdiff

I wrote a recursive function for following dependencies, as
doing recursive algorithms non-recursively can result in write-only
code, as was the case here.

[YOCTO #15289]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 bitbake/lib/bb/runqueue.py | 41 ++++++++++++++++++++++++--------------
 1 file changed, 26 insertions(+), 15 deletions(-)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 24497c5c173..f54d9b85541 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1685,6 +1685,17 @@ class RunQueue:
         return
 
     def print_diffscenetasks(self):
+        def get_root_invalid_tasks(task, taskdepends, valid, noexec, visited_invalid):
+            invalidtasks = []
+            for t in taskdepends[task].depends:
+                if t not in valid and t not in visited_invalid:
+                    invalidtasks.extend(get_root_invalid_tasks(t, taskdepends, valid, noexec, visited_invalid))
+                    visited_invalid.add(t)
+
+            direct_invalid = [t for t in taskdepends[task].depends if t not in valid]
+            if not direct_invalid and task not in noexec:
+                invalidtasks = [task]
+            return invalidtasks
 
         noexec = []
         tocheck = set()
@@ -1718,35 +1729,35 @@ class RunQueue:
                     valid_new.add(dep)
 
         invalidtasks = set()
-        for tid in self.rqdata.runtaskentries:
-            if tid not in valid_new and tid not in noexec:
-                invalidtasks.add(tid)
 
-        found = set()
-        processed = set()
-        for tid in invalidtasks:
+        toptasks = set(["{}:{}".format(t[3], t[2]) for t in self.rqdata.targets])
+        for tid in toptasks:
             toprocess = set([tid])
             while toprocess:
                 next = set()
+                visited_invalid = set()
                 for t in toprocess:
-                    for dep in self.rqdata.runtaskentries[t].depends:
-                        if dep in invalidtasks:
-                            found.add(tid)
-                        if dep not in processed:
-                            processed.add(dep)
+                    if t not in valid_new and t not in noexec:
+                        invalidtasks.update(get_root_invalid_tasks(t, self.rqdata.runtaskentries, valid_new, noexec, visited_invalid))
+                        continue
+                    if t in self.rqdata.runq_setscene_tids:
+                        for dep in self.rqexe.sqdata.sq_deps[t]:
                             next.add(dep)
+                        continue
+
+                    for dep in self.rqdata.runtaskentries[t].depends:
+                        next.add(dep)
+
                 toprocess = next
-                if tid in found:
-                    toprocess = set()
 
         tasklist = []
-        for tid in invalidtasks.difference(found):
+        for tid in invalidtasks:
             tasklist.append(tid)
 
         if tasklist:
             bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist))
 
-        return invalidtasks.difference(found)
+        return invalidtasks
 
     def write_diffscenetasks(self, invalidtasks):
 
-- 
2.39.2



^ permalink raw reply related	[flat|nested] 13+ messages in thread
end of thread, other threads:[~2024-01-05 12:02 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-12-18  8:43 [PATCH 1/7] bitbake/runqueue: rework 'bitbake -S printdiff' logic Alexander Kanavin
2023-12-18  8:43 ` [PATCH 2/7] selftest/sstatetests: fix up printdiff test to match rework of printdiff logic Alexander Kanavin
2024-01-01 13:00   ` [OE-core] " Alexandre Belloni
2024-01-01 18:27     ` Alexander Kanavin
     [not found]     ` <17A64C894AA0CBF1.19767@lists.openembedded.org>
2024-01-02 14:51       ` Alexander Kanavin
2023-12-18  8:43 ` [PATCH 3/7] sstatesig/find_siginfo: unify a disjointed API Alexander Kanavin
2024-01-05 11:22   ` [OE-core] " Richard Purdie
2024-01-05 11:42     ` Alexander Kanavin
2024-01-05 12:02       ` Richard Purdie
2023-12-18  8:44 ` [PATCH 4/7] bitbake-diffsigs/runqueue: adapt to reworked find_siginfo() Alexander Kanavin
2023-12-18  8:44 ` [PATCH 5/7] bitbake/runqueue: prioritize local stamps over sstate signatures in printdiff Alexander Kanavin
2023-12-18  8:44 ` [PATCH 6/7] bitbake/runqueue: add debugging for find_siginfo() calls Alexander Kanavin
2023-12-18  8:44 ` [PATCH 7/7] selftest/sstatetest: re-enable gcc printdiff test Alexander Kanavin

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.