[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[Steve Sakoman]

Please review this set of changes for scarthgap/2.8 and have comments back by
end of day Wednesday, June 19

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/7047

The following changes since commit 0791e66a47185d6f202af6be0d39e36a8e41850a:

  tests/fetch: Tweak test to match upstream repo url change (2024-06-10 05:57:30 -0700)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (2):
  tests/fetch: Tweak to work on Fedora40
  fetch2/wget: Fix failure path for files that are empty or don't exist

Rudolf J Streif (1):
  fetch2/wget: Canonicalize DL_DIR paths for wget2 compatibility

 lib/bb/fetch2/wget.py | 25 +++++++++++++------------
 lib/bb/tests/fetch.py |  6 +++---
 2 files changed, 16 insertions(+), 15 deletions(-)

-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[Steve Sakoman]

Please review this set of changes for scarthgap/2.8 and have comments back
by end of day Friday, December 6

Passed a-full on autobuilder:

https://valkyrie.yoctoproject.org/#/builders/29/builds/582

The following changes since commit 6c2641f7a9e92c1b82e306f59ddd3c1249c52cbf:

  fetch2: use persist_data context managers (2024-11-28 13:59:58 +0000)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Richard Purdie (3):
  runqueue: Fix performance of multiconfigs with large overlap
  runqueue: Optimise setscene loop processing
  runqueue: Fix scenetask processing performance issue

 lib/bb/runqueue.py | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 1/3] runqueue: Fix performance of multiconfigs with large overlap

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

There have been complaints about the performance of large multiconfig builds
for a while. The key missing data point was that the builds needed to have large
overlaps in sstate objects. This can be simulated by building the same things with
just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
deferred tasks.

The issue is that the expensive checks in the setscene loop were hit every time
through runqueue's execute function before the check on deferred tasks. This leads
to task execution starvation as that only happens once per iteration.

Move the skip check earlier in the function which speeds things up enormously
and should improve performance of such builds for users.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/runqueue.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 93079a977..744542b08 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2195,6 +2195,9 @@ class RunQueueExecute:
             # Find the next setscene to run
             for nexttask in self.sorted_setscene_tids:
                 if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values() and nexttask not in self.sq_harddep_deferred:
+                    if nexttask in self.sq_deferred and self.sq_deferred[nexttask] not in self.runq_complete:
+                        # Skip deferred tasks quickly before the 'expensive' tests below - this is key to performant multiconfig builds
+                        continue
                     if nexttask not in self.sqdata.unskippable and self.sqdata.sq_revdeps[nexttask] and \
                             nexttask not in self.sq_needed_harddeps and \
                             self.sqdata.sq_revdeps[nexttask].issubset(self.scenequeue_covered) and \
@@ -2224,8 +2227,7 @@ class RunQueueExecute:
                         if t in self.runq_running and t not in self.runq_complete:
                             continue
                     if nexttask in self.sq_deferred:
-                        if self.sq_deferred[nexttask] not in self.runq_complete:
-                            continue
+                        # Deferred tasks that were still deferred were skipped above so we now need to process
                         logger.debug("Task %s no longer deferred" % nexttask)
                         del self.sq_deferred[nexttask]
                         valid = self.rq.validate_hashes(set([nexttask]), self.cooker.data, 0, False, summary=False)
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 2/3] runqueue: Optimise setscene loop processing

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Rather than looping through things we looped through on the previous execution,
start looping where we left off for setscene processing. This gives speed
improvements depending on the kind of build being executed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 00f4d932e3af0eeb333339cbe942010fc76dee0f)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/runqueue.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 744542b08..75aef96a0 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -14,6 +14,7 @@ import os
 import sys
 import stat
 import errno
+import itertools
 import logging
 import re
 import bb
@@ -2189,11 +2190,16 @@ class RunQueueExecute:
         if not hasattr(self, "sorted_setscene_tids"):
             # Don't want to sort this set every execution
             self.sorted_setscene_tids = sorted(self.rqdata.runq_setscene_tids)
+            # Resume looping where we left off when we returned to feed the mainloop
+            self.setscene_tids_generator = itertools.cycle(self.rqdata.runq_setscene_tids)
 
         task = None
         if not self.sqdone and self.can_start_task():
-            # Find the next setscene to run
-            for nexttask in self.sorted_setscene_tids:
+            loopcount = 0
+            # Find the next setscene to run, exit the loop when we've processed all tids or found something to execute
+            while loopcount < len(self.rqdata.runq_setscene_tids):
+                loopcount += 1
+                nexttask = next(self.setscene_tids_generator)
                 if nexttask in self.sq_buildable and nexttask not in self.sq_running and self.sqdata.stamps[nexttask] not in self.build_stamps.values() and nexttask not in self.sq_harddep_deferred:
                     if nexttask in self.sq_deferred and self.sq_deferred[nexttask] not in self.runq_complete:
                         # Skip deferred tasks quickly before the 'expensive' tests below - this is key to performant multiconfig builds
-- 
2.34.1

[bitbake][scarthgap][2.8][PATCH 3/3] runqueue: Fix scenetask processing performance issue

[Steve Sakoman]

From: Richard Purdie <richard.purdie@linuxfoundation.org>

Analysis shows that "bitbake core-image-ptest-all" spends a lot of
time in scenequeue_updatecounters and much of it is rebuilding a set
which doens't change. Reorder the code to avoid that performance
glitch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 923c19b9713e398d8e66e6d4422dfd4c18a03486)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
 lib/bb/runqueue.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 75aef96a0..439da2bb4 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2759,8 +2759,12 @@ class RunQueueExecute:
                 logger.debug2("%s was unavailable and is a hard dependency of %s so skipping" % (task, dep))
                 self.sq_task_failoutright(dep)
                 continue
+
+        # For performance, only compute allcovered once if needed
+        if self.sqdata.sq_deps[task]:
+            allcovered = self.scenequeue_covered | self.scenequeue_notcovered
         for dep in sorted(self.sqdata.sq_deps[task]):
-            if self.sqdata.sq_revdeps[dep].issubset(self.scenequeue_covered | self.scenequeue_notcovered):
+            if self.sqdata.sq_revdeps[dep].issubset(allcovered):
                 if dep not in self.sq_buildable:
                     self.sq_buildable.add(dep)
 
-- 
2.34.1

Re: [bitbake][scarthgap][2.8][PATCH 1/3] runqueue: Fix performance of multiconfigs with large overlap

[Jeroen Hofstee]

Hi,

Op 04-12-2024 om 21:33 schreef Steve Sakoman:
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
>
> There have been complaints about the performance of large multiconfig builds
> for a while. The key missing data point was that the builds needed to have large
> overlaps in sstate objects. This can be simulated by building the same things with
> just different TMPDIRs. In runqueue/bitbake terms this equates to large numbers of
> deferred tasks.
>
[..]
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> (cherry picked from commit 9c6c506757f2b3e28c8b20513b45da6b4659c95f)
> Signed-off-by: Steve Sakoman <steve@sakoman.com>

Our CI build uses a multiconfig setup to build for 10 machines with 3
different archs. On scarthgap with bitbake 2.8, that resulted in a
really slow build:

NOTE: Executing Tasks
Bitbake still alive (no events for 600s). Active tasks:
Bitbake still alive (no events for 1200s). Active tasks:
Bitbake still alive (no events for 1800s). Active tasks:
Bitbake still alive (no events for 2400s). Active tasks:
Bitbake still alive (no events for 3000s). Active tasks:
Bitbake still alive (no events for 3600s). Active tasks:
Bitbake still alive (no events for 4200s). Active tasks:
NOTE: Running task 1 of 79788 ...
[...]

After almost 14 hours it was at 'Running task 8881 of 79788', and I
canceled the job. The cooker was running at 100% cpu, but there were
hardly any bakers running.

With this patch cherry-picked it builds successfully in 3h and 20min.
(I did add INHERIT:remove = "create-spdx", since it triggers an sstate
error causing the build to fail)

So this makes a huge difference, at least for a setup like ours.

Regards,
Jeroen

[bitbake][scarthgap][2.8][PATCH 0/3] Patch review

[Steve Sakoman]

Please reviwe this set of changes for 2.8/scarthgap and have comments back
by end of day Thursday, January 23

Passed a-full on autobuilder:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/856

The following changes since commit f40a3a477d5241b697bf2fb030dd804c1ff5839f:

  runqueue: Fix scenetask processing performance issue (2024-12-03 06:34:38 -0800)

are available in the Git repository at:

  https://git.openembedded.org/bitbake-contrib stable/2.8-nut
  https://git.openembedded.org/bitbake-contrib/log/?h=stable/2.8-nut

Chris Laplante (3):
  ui/knotty: print log paths for failed tasks in summary
  ui/knotty: respect NO_COLOR & check for tty; rename print_hyperlink =>
    format_hyperlink
  cooker: Make cooker 'skiplist' per-multiconfig/mc

 lib/bb/command.py     | 21 ++++++++++++++++++---
 lib/bb/cooker.py      | 11 ++++++-----
 lib/bb/tinfoil.py     | 16 ++++++++++++----
 lib/bb/ui/knotty.py   | 20 +++++++++++++++++---
 lib/bblayers/query.py | 14 +++++++-------
 5 files changed, 60 insertions(+), 22 deletions(-)

-- 
2.43.0