All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks
@ 2026-11-15 22:09 Mark Hatle
  2026-11-16 13:32 ` [bitbake-devel] " Richard Purdie
  2026-11-17 11:50 ` Richard Purdie
  0 siblings, 2 replies; 4+ messages in thread
From: Mark Hatle @ 2026-11-15 22:09 UTC (permalink / raw)
  To: bitbake-devel

When a task completes, the first setscene task deferred against it is
released and any others are re-deferred behind that task. If the released
task had already completed (e.g. its setscene was skipped due to a valid
stamp), it never completes again, so the re-deferred tasks are never
released. Setscene then can't finish until the deadlock fallback forces
each one, printing "Runqueue deadlocked on deferred tasks".

This shows up with three or more multiconfigs sharing a task hash. Skip
already completed tasks when picking the new blocker.

AI-Generated: Uses GitHub Copilot
Signed-off-by: Mark Hatle <mark.hatle@amd.com>
---
 lib/bb/runqueue.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index ef95eb46b..2a6b099b8 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2112,6 +2112,10 @@ class RunQueueExecute:
                     bb.debug(1, "Deferred task %s now buildable" % t)
                     del self.sq_deferred[t]
                     update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
+                    # A task which already completed will never complete again so must
+                    # not become the new blocker, else the remaining tasks never get released
+                    if t in self.runq_complete:
+                        continue
                     found = t
                 else:
                     bb.debug(1, "Deferring %s after %s" % (t, found))
-- 
2.34.1


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [bitbake-devel] [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks
  2026-11-15 22:09 [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks Mark Hatle
@ 2026-11-16 13:32 ` Richard Purdie
  2026-11-16 14:16   ` Mark Hatle
  2026-11-17 11:50 ` Richard Purdie
  1 sibling, 1 reply; 4+ messages in thread
From: Richard Purdie @ 2026-11-16 13:32 UTC (permalink / raw)
  To: mark.hatle, bitbake-devel

On Sun, 2026-11-15 at 15:09 -0700, Mark Hatle via bitbake-devel wrote:
> When a task completes, the first setscene task deferred against it is
> released and any others are re-deferred behind that task. If the released
> task had already completed (e.g. its setscene was skipped due to a valid
> stamp), it never completes again, so the re-deferred tasks are never
> released. Setscene then can't finish until the deadlock fallback forces
> each one, printing "Runqueue deadlocked on deferred tasks".
> 
> This shows up with three or more multiconfigs sharing a task hash. Skip
> already completed tasks when picking the new blocker.
> 
> AI-Generated: Uses GitHub Copilot
> Signed-off-by: Mark Hatle <mark.hatle@amd.com>
> ---
>  lib/bb/runqueue.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index ef95eb46b..2a6b099b8 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -2112,6 +2112,10 @@ class RunQueueExecute:
>                      bb.debug(1, "Deferred task %s now buildable" % t)
>                      del self.sq_deferred[t]
>                      update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
> +                    # A task which already completed will never complete again so must
> +                    # not become the new blocker, else the remaining tasks never get released
> +                    if t in self.runq_complete:
> +                        continue
>                      found = t
>                  else:
>                      bb.debug(1, "Deferring %s after %s" % (t, found))

This doesn't look quite right to me.

If you do that, you need to also then follow through and handle
anything that has a dependency on t.

Also, the code above this is the only codepath where
self.runq_complete.add(task) is called. For something to be in
runq_complete, this function has to have been called for t, so that
deferral should have already been handled too.

I have a different suspicion. I think the loop ordering is subtly
wrong, which is leading to the deadlocking.

We need to process deferred tasks in multilib order. The main code
already does this but this section just has a sorted() call. I think it
may need to be:

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 63615ac6c..b098ac170 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2109,7 +2109,10 @@ class RunQueueExecute:
                 logger.debug("Marking task %s as buildable", revdep)
 
         found = None
-        for t in sorted(self.sq_deferred.copy()):
+        for mc in sorted(self.sqdata.multiconfigs):
+          for t in sorted(self.sq_deferred.copy()):
+            if mc_from_tid(t) != mc:
+                continue
             if self.sq_deferred[t] == task:
                 # Allow the next deferred task to run. Any other deferred tasks should be deferred after that task.
                 # We shouldn't allow all to run at once as it is prone to races.

which then matches the other code. We probably need a small helper
function to give that sorting. (yes, I know the indentation needs
fixing, I left it like this for clarity on what the change is).

Can you see if that helps please?

Thanks,

Richard



^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [bitbake-devel] [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks
  2026-11-16 13:32 ` [bitbake-devel] " Richard Purdie
@ 2026-11-16 14:16   ` Mark Hatle
  0 siblings, 0 replies; 4+ messages in thread
From: Mark Hatle @ 2026-11-16 14:16 UTC (permalink / raw)
  To: Richard Purdie, bitbake-devel



On 11/16/26 7:32 AM, Richard Purdie wrote:
...

> 
> This doesn't look quite right to me.
> 
> If you do that, you need to also then follow through and handle
> anything that has a dependency on t.
> 
> Also, the code above this is the only codepath where
> self.runq_complete.add(task) is called. For something to be in
> runq_complete, this function has to have been called for t, so that
> deferral should have already been handled too.
> 
> I have a different suspicion. I think the loop ordering is subtly
> wrong, which is leading to the deadlocking.
> 
> We need to process deferred tasks in multilib order. The main code
> already does this but this section just has a sorted() call. I think it
> may need to be:
> 
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index 63615ac6c..b098ac170 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -2109,7 +2109,10 @@ class RunQueueExecute:
>                   logger.debug("Marking task %s as buildable", revdep)
>   
>           found = None
> -        for t in sorted(self.sq_deferred.copy()):
> +        for mc in sorted(self.sqdata.multiconfigs):
> +          for t in sorted(self.sq_deferred.copy()):
> +            if mc_from_tid(t) != mc:
> +                continue
>               if self.sq_deferred[t] == task:
>                   # Allow the next deferred task to run. Any other deferred tasks should be deferred after that task.
>                   # We shouldn't allow all to run at once as it is prone to races.
> 
> which then matches the other code. We probably need a small helper
> function to give that sorting. (yes, I know the indentation needs
> fixing, I left it like this for clarity on what the change is).
> 
> Can you see if that helps please?

Assuming I applied this properly, I'm still getting the same results:

WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:zynqmp-zcu102-sdt-full-microblaze-pmu:virtual:native:/scratch2/mhatle/git/external/experimental/release/meta/recipes-support/sqlite/sqlite3_3.54.1.bb:do_populate_lic blocked by mc:zynqmp-zcu102-sdt-full-cortexa53-fsbl:virtual:native:/scratch2/mhatle/git/external/experimental/release/meta/recipes-support/sqlite/sqlite3_3.54.1.bb:do_populate_lic
WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:zynqmp-zcu102-sdt-full-microblaze-pmu:virtual:native:/scratch2/mhatle/git/external/experimental/release/meta/recipes-support/sqlite/sqlite3_3.54.1.bb:do_populate_sysroot blocked by mc:zynqmp-zcu102-sdt-full-cortexa53-fsbl:virtual:native:/scratch2/mhatle/git/external/experimental/release/meta/recipes-support/sqlite/sqlite3_3.54.1.bb:do_populate_sysroot
WARNING: Runqueue deadlocked on deferred tasks, forcing task mc:zynqmp-zcu102-sdt-full-microblaze-pmu:virtual:native:/scratch2/mhatle/git/external/experimental/release/meta/recipes-support/sqlite/sqlite3_3.54.1.bb:do_recipe_qa blocked by mc:zynqmp-zcu102-sdt-full-cortexa53-fsbl:virtual:native:/scratch2/mhatle/git/external/experimental/release/meta/recipes-support/sqlite/sqlite3_3.54.1.bb:do_recipe_qa

(and many more)

--Mark

  
> Thanks,
> 
> Richard
> 
> 


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: [bitbake-devel] [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks
  2026-11-15 22:09 [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks Mark Hatle
  2026-11-16 13:32 ` [bitbake-devel] " Richard Purdie
@ 2026-11-17 11:50 ` Richard Purdie
  1 sibling, 0 replies; 4+ messages in thread
From: Richard Purdie @ 2026-11-17 11:50 UTC (permalink / raw)
  To: mark.hatle, bitbake-devel

On Sun, 2026-11-15 at 15:09 -0700, Mark Hatle via bitbake-devel wrote:
> When a task completes, the first setscene task deferred against it is
> released and any others are re-deferred behind that task. If the released
> task had already completed (e.g. its setscene was skipped due to a valid
> stamp), it never completes again, so the re-deferred tasks are never
> released. Setscene then can't finish until the deadlock fallback forces
> each one, printing "Runqueue deadlocked on deferred tasks".
> 
> This shows up with three or more multiconfigs sharing a task hash. Skip
> already completed tasks when picking the new blocker.
> 
> AI-Generated: Uses GitHub Copilot
> Signed-off-by: Mark Hatle <mark.hatle@amd.com>
> ---
>  lib/bb/runqueue.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
> index ef95eb46b..2a6b099b8 100644
> --- a/lib/bb/runqueue.py
> +++ b/lib/bb/runqueue.py
> @@ -2112,6 +2112,10 @@ class RunQueueExecute:
>                      bb.debug(1, "Deferred task %s now buildable" % t)
>                      del self.sq_deferred[t]
>                      update_scenequeue_data([t], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
> +                    # A task which already completed will never complete again so must
> +                    # not become the new blocker, else the remaining tasks never get released
> +                    if t in self.runq_complete:
> +                        continue
>                      found = t
>                  else:
>                      bb.debug(1, "Deferring %s after %s" % (t, found))

Just to confirm, this is not quite the right fix. I did work out
exactly why it was breaking and whilst this works around the problem,
this breaks things. There is a better fix and I've sent patches.

Cheers,

Richard

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-17 11:50 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-15 22:09 [PATCH] runqueue: Don't defer setscene tasks behind already completed tasks Mark Hatle
2026-11-16 13:32 ` [bitbake-devel] " Richard Purdie
2026-11-16 14:16   ` Mark Hatle
2026-11-17 11:50 ` Richard Purdie

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.