[PATCH 1/2] runqueue: add 'default' mc translation to mcdepends

[PATCH 1/2] runqueue: add 'default' mc translation to mcdepends

[Mark Hatle]

From: Mark Hatle <mark.hatle@amd.com>

The default mc is '' (blank), however BB_CURRENT_MC returns 'default'.
Allow a user to specify a mcdepend such as:

  mc:${BB_CURRENT_MC}:m2:recipe:task

Both '' (blank) and 'default' are now supported and have equivalent
meaning.

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 bitbake/lib/bb/runqueue.py                        | 4 ++++
 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb | 2 ++
 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb | 0
 bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb | 0
 bitbake/lib/bb/tests/runqueue.py                  | 7 +++++++
 5 files changed, 13 insertions(+)
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index ffb2d28494..f941749b0d 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -723,7 +723,11 @@ class RunQueueData:
                 mcdependency = dep.split(':')
                 pn = mcdependency[3]
                 frommc = mcdependency[1]
+                if frommc == 'default':
+                    frommc = ''
                 mcdep = mcdependency[2]
+                if mcdep == 'default':
+                    mcdep = ''
                 deptask = mcdependency[4]
                 if mcdep not in taskData:
                     bb.fatal("Multiconfig '%s' is referenced in multiconfig dependency '%s' but not enabled in BBMULTICONFIG?" % (mcdep, dep))
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
new file mode 100644
index 0000000000..20ce8dd3ad
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/g1.bb
@@ -0,0 +1,2 @@
+do_build[mcdepends] = "mc::mc-1:h1:do_build mc:default:mc_2:i1:do_build"
+
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/h1.bb
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/i1.bb
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runqueue.py
index cc87e8d6a8..cea2ca13fe 100644
--- a/bitbake/lib/bb/tests/runqueue.py
+++ b/bitbake/lib/bb/tests/runqueue.py
@@ -314,6 +314,13 @@ class RunQueueTests(unittest.TestCase):
                        ["mc_2:a1:%s" % t for t in rerun_tasks]
             self.assertEqual(set(tasks), set(expected))
 
+            # Test that mc::... and mc:default:... both work
+            tasks = self.run_bitbakecmd(["bitbake", "g1"], tempdir, "", extraenv=extraenv, cleanup=True)
+            expected = ["g1:%s" % t for t in self.alltasks] + \
+                       ["mc-1:h1:%s" % t for t in self.alltasks] + \
+                       ["mc_2:i1:%s" % t for t in self.alltasks]
+            self.assertEqual(set(tasks), set(expected))
+
             self.shutdown(tempdir)
 
     def test_hashserv_single(self):
-- 
2.34.1

[PATCH 2/2] runqueue: Verify mcdepends are valid

[Mark Hatle]

From: Mark Hatle <mark.hatle@amd.com>

In order to avoid a potentially confusing backtrace, check that the mcdepend
is valid when we add it.

Add a test case to ensure invalid configurations are caught and trigger an
error.

Signed-off-by: Mark Hatle <mark.hatle@amd.com>
Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
---
 bitbake/lib/bb/runqueue.py                        | 2 ++
 bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb | 2 ++
 bitbake/lib/bb/tests/runqueue.py                  | 9 +++++++++
 3 files changed, 13 insertions(+)
 create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index f941749b0d..90f606469b 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -734,6 +734,8 @@ class RunQueueData:
                 if mc == frommc:
                     fn = taskData[mcdep].build_targets[pn][0]
                     newdep = '%s:%s' % (fn,deptask)
+                    if newdep not in taskData[mcdep].taskentries:
+                        bb.fatal("Task mcdepends on non-existent task %s" % (newdep))
                     taskData[mc].taskentries[tid].tdepends.append(newdep)
 
         for mc in taskData:
diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb
new file mode 100644
index 0000000000..3c7dca0257
--- /dev/null
+++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb
@@ -0,0 +1,2 @@
+do_build[mcdepends] = "mc::mc-1:h1:do_invalid"
+
diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runqueue.py
index cea2ca13fe..4662efbf55 100644
--- a/bitbake/lib/bb/tests/runqueue.py
+++ b/bitbake/lib/bb/tests/runqueue.py
@@ -321,6 +321,15 @@ class RunQueueTests(unittest.TestCase):
                        ["mc_2:i1:%s" % t for t in self.alltasks]
             self.assertEqual(set(tasks), set(expected))
 
+            # Check what happens with an invalid task depednency
+            passed = False
+            try:
+                self.run_bitbakecmd(["bitbake", "j1"], tempdir, "", extraenv=extraenv, cleanup=True)
+            except AssertionError as e:
+                print("Expected failure: %s" % e)
+                passed = True
+            self.assertEqual(True, passed)
+
             self.shutdown(tempdir)
 
     def test_hashserv_single(self):
-- 
2.34.1

Re: [bitbake-devel] [PATCH 2/2] runqueue: Verify mcdepends are valid

[Richard Purdie]

On Thu, 2025-01-23 at 20:50 -0600, Mark Hatle via lists.openembedded.org wrote:
> From: Mark Hatle <mark.hatle@amd.com>
> 
> In order to avoid a potentially confusing backtrace, check that the mcdepend
> is valid when we add it.
> 
> Add a test case to ensure invalid configurations are caught and trigger an
> error.
> 
> Signed-off-by: Mark Hatle <mark.hatle@amd.com>
> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
> ---
>  bitbake/lib/bb/runqueue.py                        | 2 ++
>  bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb | 2 ++
>  bitbake/lib/bb/tests/runqueue.py                  | 9 +++++++++
>  3 files changed, 13 insertions(+)
>  create mode 100644 bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb
> 
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index f941749b0d..90f606469b 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -734,6 +734,8 @@ class RunQueueData:
>                  if mc == frommc:
>                      fn = taskData[mcdep].build_targets[pn][0]
>                      newdep = '%s:%s' % (fn,deptask)
> +                    if newdep not in taskData[mcdep].taskentries:
> +                        bb.fatal("Task mcdepends on non-existent task %s" % (newdep))
>                      taskData[mc].taskentries[tid].tdepends.append(newdep)
>  
>          for mc in taskData:
> diff --git a/bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb b/bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb
> new file mode 100644
> index 0000000000..3c7dca0257
> --- /dev/null
> +++ b/bitbake/lib/bb/tests/runqueue-tests/recipes/j1.bb
> @@ -0,0 +1,2 @@
> +do_build[mcdepends] = "mc::mc-1:h1:do_invalid"
> +
> diff --git a/bitbake/lib/bb/tests/runqueue.py b/bitbake/lib/bb/tests/runqueue.py
> index cea2ca13fe..4662efbf55 100644
> --- a/bitbake/lib/bb/tests/runqueue.py
> +++ b/bitbake/lib/bb/tests/runqueue.py
> @@ -321,6 +321,15 @@ class RunQueueTests(unittest.TestCase):
>                         ["mc_2:i1:%s" % t for t in self.alltasks]
>              self.assertEqual(set(tasks), set(expected))
>  
> +            # Check what happens with an invalid task depednency
> +            passed = False
> +            try:
> +                self.run_bitbakecmd(["bitbake", "j1"], tempdir, "", extraenv=extraenv, cleanup=True)
> +            except AssertionError as e:
> +                print("Expected failure: %s" % e)
> +                passed = True
> +            self.assertEqual(True, passed)
> +
>              self.shutdown(tempdir)
>  
>      def test_hashserv_single(self):


Firstly, I *really* appreciate people adding tests, its great to see so
thanks for that.

The test needs a few tweaks, just to follow best practice.

Basically you can use assertRaises() to test it raises an exception and
we should really check that it says "Command.*failed" and "mcdepends on
non-existent task.*do_invalid".

https://www.geeksforgeeks.org/test-if-a-function-throws-an-exception-in-python/

Another option would be to have a new parameter to run_bitbakecmd which
just makes it return the failure output.

Cheers,

Richard

Re: [bitbake-devel] [PATCH 1/2] runqueue: add 'default' mc translation to mcdepends

[Richard Purdie]

On Thu, 2025-01-23 at 20:50 -0600, Mark Hatle via lists.openembedded.org wrote:
> From: Mark Hatle <mark.hatle@amd.com>
> 
> The default mc is '' (blank), however BB_CURRENT_MC returns 'default'.
> Allow a user to specify a mcdepend such as:
> 
>   mc:${BB_CURRENT_MC}:m2:recipe:task
> 
> Both '' (blank) and 'default' are now supported and have equivalent
> meaning.
> 
> Signed-off-by: Mark Hatle <mark.hatle@amd.com>
> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>

I'm wondering if we should remove "default" entirely, it is confusing
and I'm not sure we want to complicate the code with this. I'm going to
test a patch doing that.

Cheers,

Richard

Re: [bitbake-devel] [PATCH 1/2] runqueue: add 'default' mc translation to mcdepends

[Mathieu Dubois-Briand]

On Fri Jan 24, 2025 at 3:50 AM CET, Mark Hatle wrote:
> From: Mark Hatle <mark.hatle@amd.com>
>
> The default mc is '' (blank), however BB_CURRENT_MC returns 'default'.
> Allow a user to specify a mcdepend such as:
>
>   mc:${BB_CURRENT_MC}:m2:recipe:task
>
> Both '' (blank) and 'default' are now supported and have equivalent
> meaning.
>
> Signed-off-by: Mark Hatle <mark.hatle@amd.com>
> Signed-off-by: Mark Hatle <mark.hatle@kernel.crashing.org>
> ---

Hi Mark,

Thanks for your patch.

I believe one of the patch from this series is responsible for some
failures we can see on the autobuilder:

FAIL: test_multiconfig_mcdepends (bb.tests.runqueue.RunQueueTests.test_multiconfig_mcdepends)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/bitbake/lib/bb/tests/runqueue.py", line 41, in run_bitbakecmd
    output = subprocess.check_output(cmd, env=env, stderr=subprocess.STDOUT,universal_newlines=True, cwd=builddir)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['bitbake', 'g1']' returned non-zero exit status 1.


https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/846/steps/11/logs/stdio

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com