[PATCH v2 0/5] Implement RECIPE_UPDATE_EXTRA_TASKS

[PATCH v2 0/5] Implement RECIPE_UPDATE_EXTRA_TASKS

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

Whenever a recipe which inherits cargo-update-recipe-crates is upgraded with
devtool (such as with the Auto Upgrade Helper) we frequently see failures.
If the upstream Cargo.toml or Cargo.lock has been changed, the corresponding
<recipe_name>-crates.inc needs to be updated. This is done by running the
"bitbake -c update_crates <recipe_name>" task, but this has until now been
a manual operation.

Add a new variable RECIPE_UPDATE_EXTRA_TASKS which is a string delimited
list of tasks to be run during the upgrade() method of
meta/lib/devtool/upgrade.py.

To solve the problem described above, the solution is for all recipes which
inherit cargo-update-recipe-crates to add the "do_update_crates" task to
the RECIPE_UPDATE_EXTRA_TASKS variable. Add this to the bbclass.

Add test case based on the guessing-game source we currently use for the
maturin runtime test cases, based on https://www.maturin.rs/tutorial.
This oe-selftest case checks for both the change in the recipe file and
the recipe -crates.inc file.

This relies on a new "guessing-game" git repository which has been added to
git.yoctoproject.org/guessing-game .

Currently, update_crates is the only task tested in this
RECIPE_UPDATE_EXTRA_TASKS use case. Other use cases that have been discussed
are the python3-manifest.json and perl-rdepends.txt files which are generated
by manual tasks after an upgrade of python3 and perl respectively.

Two obvious new use cases that would be nice to add with changes in oe-core are:
(1) some kind of "go-mod://" update class to use a similar <recipe>-go-mods.inc
(2) some kind of rework of the "npm://" fetcher to use <recipe>-npm-mods.inc
But both of those are out of scope for this proposal.

The _run_recipe_update_extra_tasks() method added to lib/devtool/upgrade.py
checks the result of each of the upgrade tasks and raises a DevtoolError if it
fails.

An attempt at documenting the new variable has been added to ref-manual,
but if anyone has some ideas for better clarification, input is welcome.

The following changes since commit 46b5c2ea6cb97bb6dd0a85bef3be9eaa9a19b4b5:

  linuxloader: add -armhf on arm only for TARGET_FPU 'hard' (2024-06-11 11:41:33 +0100)

are available in the Git repository at:

  https://git.yoctoproject.org/poky-contrib timo/RECIPE_UPDATE_EXTRA_TASKS_v2
  https://git.yoctoproject.org/poky-contrib/log/?h=timo/RECIPE_UPDATE_EXTRA_TASKS_v2

Tim Orling (5):
  devtool upgrade: enable RECIPE_UPDATE_EXTRA_TASKS
  cargo-update-recipe-crates: add RECIPE_UPDATE_EXTRA_TASKS
  ref-manual: add RECIPE_UPDATE_EXTRA_TASKS variable
  meta-selftest: add python3-guessing-game
  oe-selftest: add RECIPE_UPDATE_EXTRA_TASKS test

 documentation/ref-manual/variables.rst        |  7 ++
 .../python/python3-guessing-game-crates.inc   | 82 +++++++++++++++++
 .../python3-guessing-game-crates.inc.upgraded | 88 +++++++++++++++++++
 .../python/python3-guessing-game_git.bb       | 19 ++++
 .../python3-guessing-game_git.bb.upgraded     | 19 ++++
 .../cargo-update-recipe-crates.bbclass        |  2 +
 meta/lib/oeqa/selftest/cases/devtool.py       | 46 ++++++++++
 scripts/lib/devtool/upgrade.py                | 11 +++
 8 files changed, 274 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc.upgraded
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.upgraded

-- 
2.45.2

[PATCH v2 1/5] devtool upgrade: enable RECIPE_UPDATE_EXTRA_TASKS

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

For some recipes, such as those that inherit cargo-update-recipe-crates,
we need to run additional tasks once the new sources have been unpacked.

Introduce a new variable RECIPE_UPDATE_EXTRA_TASKS which is a space-
delimited list of tasks to run after the new sources have been
unpacked in scripts/lib/devtool/upgrade.py ugrade() method.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v2:
  * Drop the try/except in the _run_recipe_update_extra_tasks method
  * Check the result of each task (tinfoil.build_targets) and raise
    a DevtoolError on failure

 scripts/lib/devtool/upgrade.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index a8130ed23f5..87227df1a8f 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -534,6 +534,15 @@ def _generate_license_diff(old_licenses, new_licenses):
             diff = diff + line
     return diff
 
+def _run_recipe_update_extra_tasks(pn, rd, tinfoil):
+    tasks = []
+    for task in (rd.getVar('RECIPE_UPDATE_EXTRA_TASKS:%s' % pn) or '').split():
+        logger.info('Running extra recipe update task: %s' % task)
+        res = tinfoil.build_targets(pn, task, handle_events=True)
+
+        if not res:
+            raise DevtoolError('Running extra recipe update task %s for %s failed' % (task, pn))
+
 def upgrade(args, config, basepath, workspace):
     """Entry point for the devtool 'upgrade' subcommand"""
 
@@ -609,6 +618,8 @@ def upgrade(args, config, basepath, workspace):
                         copied, config.workspace_path, rd)
         standard._add_md5(config, pn, af)
 
+        _run_recipe_update_extra_tasks(pn, rd, tinfoil)
+
         update_unlockedsigs(basepath, workspace, args.fixed_setup, [pn])
 
         logger.info('Upgraded source extracted to %s' % srctree)
-- 
2.45.2

[PATCH v2 2/5] cargo-update-recipe-crates: add RECIPE_UPDATE_EXTRA_TASKS

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

When we upgrade a recipe that inherits cargo-update-recipe-crates and
the upstream Cargo.toml/Cargo.lock have been changed, we need to run
the update_crates task or else the devtool upgrade (and therefore
AUH upgrade) will fail.

Add "do_update_crates" task to RECIPE_UPDATE_EXTRA_TASKS for all
recipes that inherit this class.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
No changes in v2

 meta/classes-recipe/cargo-update-recipe-crates.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
index 8980137d02c..a405951f4b5 100644
--- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -18,6 +18,8 @@ do_update_crates[depends] = "python3-native:do_populate_sysroot"
 do_update_crates[nostamp] = "1"
 do_update_crates[doc] = "Update the recipe by reading Cargo.lock and write in ${THISDIR}/${BPN}-crates.inc"
 
+RECIPE_UPDATE_EXTRA_TASKS:${PN} += "do_update_crates"
+
 # The directory where to search for Cargo.lock files
 CARGO_LOCK_SRC_DIR ??= "${S}"
 
-- 
2.45.2

[PATCH v2 3/5] ref-manual: add RECIPE_UPDATE_EXTRA_TASKS variable

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

Document the new RECIPE_UPDATE_EXTRA_TASKS variable
and give cargo-update-recipe-crates as a concrete
example.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
No changes in v2

 documentation/ref-manual/variables.rst | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 07b5b6f95cf..255de3e396a 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -7080,6 +7080,13 @@ system and gives an overview of their function and contents.
       The default value is ``"${WORKDIR}/recipe-sysroot-native"``.
       Do not modify it.
 
+   :term:`RECIPE_UPDATE_EXTRA_TASKS`
+      For some recipes, after the new source has been unpacked, additional tasks
+      may need to be run during an upgrade. A good example of this is recipes
+      which inherit :ref:`ref-classes-cargo-update-recipe-crates`, where the
+      `do_update_crates` task needs to be run whenever Cargo.toml/Cargo.lock have
+      changed in the source.
+
    :term:`REPODIR`
       See :term:`bitbake:REPODIR` in the BitBake manual.
 
-- 
2.45.2

[PATCH v2 4/5] meta-selftest: add python3-guessing-game

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

Add v0.1.0 of python3-guessing-game which is used as the baseline
for an upgrade to v0.2.0 in test_devtool_upgrade_recipe_update_extra_tasks
test case.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v2:
  * Change to the git.yoctoproject.org url

 .../python/python3-guessing-game-crates.inc   | 82 +++++++++++++++++++
 .../python/python3-guessing-game_git.bb       | 19 +++++
 2 files changed, 101 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb

diff --git a/meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc b/meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc
new file mode 100644
index 00000000000..7a1bfe101f0
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc
@@ -0,0 +1,82 @@
+# Autogenerated with 'bitbake -c update_crates python3-guessing-game'
+
+# from Cargo.lock
+SRC_URI += " \
+    crate://crates.io/autocfg/1.2.0 \
+    crate://crates.io/bitflags/1.3.2 \
+    crate://crates.io/cfg-if/1.0.0 \
+    crate://crates.io/getrandom/0.2.14 \
+    crate://crates.io/indoc/1.0.9 \
+    crate://crates.io/libc/0.2.153 \
+    crate://crates.io/lock_api/0.4.11 \
+    crate://crates.io/memoffset/0.9.0 \
+    crate://crates.io/once_cell/1.19.0 \
+    crate://crates.io/parking_lot/0.12.1 \
+    crate://crates.io/parking_lot_core/0.9.9 \
+    crate://crates.io/ppv-lite86/0.2.17 \
+    crate://crates.io/proc-macro2/1.0.79 \
+    crate://crates.io/pyo3/0.19.2 \
+    crate://crates.io/pyo3-build-config/0.19.2 \
+    crate://crates.io/pyo3-ffi/0.19.2 \
+    crate://crates.io/pyo3-macros/0.19.2 \
+    crate://crates.io/pyo3-macros-backend/0.19.2 \
+    crate://crates.io/quote/1.0.35 \
+    crate://crates.io/rand/0.8.5 \
+    crate://crates.io/rand_chacha/0.3.1 \
+    crate://crates.io/rand_core/0.6.4 \
+    crate://crates.io/redox_syscall/0.4.1 \
+    crate://crates.io/scopeguard/1.2.0 \
+    crate://crates.io/smallvec/1.13.2 \
+    crate://crates.io/syn/1.0.109 \
+    crate://crates.io/target-lexicon/0.12.14 \
+    crate://crates.io/unicode-ident/1.0.12 \
+    crate://crates.io/unindent/0.1.11 \
+    crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \
+    crate://crates.io/windows-targets/0.48.5 \
+    crate://crates.io/windows_aarch64_gnullvm/0.48.5 \
+    crate://crates.io/windows_aarch64_msvc/0.48.5 \
+    crate://crates.io/windows_i686_gnu/0.48.5 \
+    crate://crates.io/windows_i686_msvc/0.48.5 \
+    crate://crates.io/windows_x86_64_gnu/0.48.5 \
+    crate://crates.io/windows_x86_64_gnullvm/0.48.5 \
+    crate://crates.io/windows_x86_64_msvc/0.48.5 \
+"
+
+SRC_URI[autocfg-1.2.0.sha256sum] = "f1fdabc7756949593fe60f30ec81974b613357de856987752631dea1e3394c80"
+SRC_URI[bitflags-1.3.2.sha256sum] = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
+SRC_URI[cfg-if-1.0.0.sha256sum] = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+SRC_URI[getrandom-0.2.14.sha256sum] = "94b22e06ecb0110981051723910cbf0b5f5e09a2062dd7663334ee79a9d1286c"
+SRC_URI[indoc-1.0.9.sha256sum] = "bfa799dd5ed20a7e349f3b4639aa80d74549c81716d9ec4f994c9b5815598306"
+SRC_URI[libc-0.2.153.sha256sum] = "9c198f91728a82281a64e1f4f9eeb25d82cb32a5de251c6bd1b5154d63a8e7bd"
+SRC_URI[lock_api-0.4.11.sha256sum] = "3c168f8615b12bc01f9c17e2eb0cc07dcae1940121185446edc3744920e8ef45"
+SRC_URI[memoffset-0.9.0.sha256sum] = "5a634b1c61a95585bd15607c6ab0c4e5b226e695ff2800ba0cdccddf208c406c"
+SRC_URI[once_cell-1.19.0.sha256sum] = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
+SRC_URI[parking_lot-0.12.1.sha256sum] = "3742b2c103b9f06bc9fff0a37ff4912935851bee6d36f3c02bcc755bcfec228f"
+SRC_URI[parking_lot_core-0.9.9.sha256sum] = "4c42a9226546d68acdd9c0a280d17ce19bfe27a46bf68784e4066115788d008e"
+SRC_URI[ppv-lite86-0.2.17.sha256sum] = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de"
+SRC_URI[proc-macro2-1.0.79.sha256sum] = "e835ff2298f5721608eb1a980ecaee1aef2c132bf95ecc026a11b7bf3c01c02e"
+SRC_URI[pyo3-0.19.2.sha256sum] = "e681a6cfdc4adcc93b4d3cf993749a4552018ee0a9b65fc0ccfad74352c72a38"
+SRC_URI[pyo3-build-config-0.19.2.sha256sum] = "076c73d0bc438f7a4ef6fdd0c3bb4732149136abd952b110ac93e4edb13a6ba5"
+SRC_URI[pyo3-ffi-0.19.2.sha256sum] = "e53cee42e77ebe256066ba8aa77eff722b3bb91f3419177cf4cd0f304d3284d9"
+SRC_URI[pyo3-macros-0.19.2.sha256sum] = "dfeb4c99597e136528c6dd7d5e3de5434d1ceaf487436a3f03b2d56b6fc9efd1"
+SRC_URI[pyo3-macros-backend-0.19.2.sha256sum] = "947dc12175c254889edc0c02e399476c2f652b4b9ebd123aa655c224de259536"
+SRC_URI[quote-1.0.35.sha256sum] = "291ec9ab5efd934aaf503a6466c5d5251535d108ee747472c3977cc5acc868ef"
+SRC_URI[rand-0.8.5.sha256sum] = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
+SRC_URI[rand_chacha-0.3.1.sha256sum] = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
+SRC_URI[rand_core-0.6.4.sha256sum] = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
+SRC_URI[redox_syscall-0.4.1.sha256sum] = "4722d768eff46b75989dd134e5c353f0d6296e5aaa3132e776cbdb56be7731aa"
+SRC_URI[scopeguard-1.2.0.sha256sum] = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
+SRC_URI[smallvec-1.13.2.sha256sum] = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67"
+SRC_URI[syn-1.0.109.sha256sum] = "72b64191b275b66ffe2469e8af2c1cfe3bafa67b529ead792a6d0160888b4237"
+SRC_URI[target-lexicon-0.12.14.sha256sum] = "e1fc403891a21bcfb7c37834ba66a547a8f402146eba7265b5a6d88059c9ff2f"
+SRC_URI[unicode-ident-1.0.12.sha256sum] = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
+SRC_URI[unindent-0.1.11.sha256sum] = "e1766d682d402817b5ac4490b3c3002d91dfa0d22812f341609f97b08757359c"
+SRC_URI[wasi-0.11.0+wasi-snapshot-preview1.sha256sum] = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
+SRC_URI[windows-targets-0.48.5.sha256sum] = "9a2fa6e2155d7247be68c096456083145c183cbbbc2764150dda45a87197940c"
+SRC_URI[windows_aarch64_gnullvm-0.48.5.sha256sum] = "2b38e32f0abccf9987a4e3079dfb67dcd799fb61361e53e2882c3cbaf0d905d8"
+SRC_URI[windows_aarch64_msvc-0.48.5.sha256sum] = "dc35310971f3b2dbbf3f0690a219f40e2d9afcf64f9ab7cc1be722937c26b4bc"
+SRC_URI[windows_i686_gnu-0.48.5.sha256sum] = "a75915e7def60c94dcef72200b9a8e58e5091744960da64ec734a6c6e9b3743e"
+SRC_URI[windows_i686_msvc-0.48.5.sha256sum] = "8f55c233f70c4b27f66c523580f78f1004e8b5a8b659e05a4eb49d4166cca406"
+SRC_URI[windows_x86_64_gnu-0.48.5.sha256sum] = "53d40abd2583d23e4718fddf1ebec84dbff8381c07cae67ff7768bbf19c6718e"
+SRC_URI[windows_x86_64_gnullvm-0.48.5.sha256sum] = "0b7b52767868a23d5bab768e390dc5f5c55825b6d30b86c844ff2dc7414044cc"
+SRC_URI[windows_x86_64_msvc-0.48.5.sha256sum] = "ed94fce61571a4006852b7389a063ab983c02eb1bb37b47f8272ce92d06d9538"
diff --git a/meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb b/meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb
new file mode 100644
index 00000000000..50246a8a118
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb
@@ -0,0 +1,19 @@
+SUMMARY = "The guessing game from The Rust Book using pyo3."
+DESCRIPTION = "Wrap a version of the guessing game from The Rust Book \
+to run in Python using pyo3."
+HOMEPAGE = "https://www.maturin.rs/tutorial"
+SECTION = "devel/python"
+LICENSE = "MIT & Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE-APACHE;md5=1836efb2eb779966696f473ee8540542 \
+                    file://LICENSE-MIT;md5=85fd3b67069cff784d98ebfc7d5c0797"
+
+SRC_URI = "git://git.yoctoproject.org/guessing-game.git;protocol=https;branch=main"
+
+PV = "0.1.0"
+SRCREV = "469c9e2230ca4fa9e391c94be6e697733e769500"
+
+S = "${WORKDIR}/git"
+
+inherit python_maturin cargo-update-recipe-crates
+
+require ${BPN}-crates.inc
-- 
2.45.2

[PATCH v2 5/5] oe-selftest: add RECIPE_UPDATE_EXTRA_TASKS test

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

Add test_devtool_upgrade_recipe_update_extra_tasks test case
to test upgrade of python3-guessing-game from v0.1.0 to v0.2.0
which will exercise the update_crates task during the upgrade.

Add python3-guessing-game_git.bb.upgraded and
python3-guessing-game-crates.inc.upgraded  which are the 0.2.0
variants.

Check that the new recipe file has the expected differences.

Check that the new -crates.inc file has the expected differences,
which should be reproducible because of Cargo.lock.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v2:
  * Switch to git.yoctoproject.org url

 .../python3-guessing-game-crates.inc.upgraded | 88 +++++++++++++++++++
 .../python3-guessing-game_git.bb.upgraded     | 19 ++++
 meta/lib/oeqa/selftest/cases/devtool.py       | 46 ++++++++++
 3 files changed, 153 insertions(+)
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc.upgraded
 create mode 100644 meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.upgraded

diff --git a/meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc.upgraded b/meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc.upgraded
new file mode 100644
index 00000000000..b19b42832d9
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-guessing-game-crates.inc.upgraded
@@ -0,0 +1,88 @@
+# Autogenerated with 'bitbake -c update_crates python3-guessing-game'
+
+# from Cargo.lock
+SRC_URI += " \
+    crate://crates.io/autocfg/1.3.0 \
+    crate://crates.io/bitflags/2.5.0 \
+    crate://crates.io/cfg-if/1.0.0 \
+    crate://crates.io/getrandom/0.2.15 \
+    crate://crates.io/heck/0.4.1 \
+    crate://crates.io/indoc/2.0.5 \
+    crate://crates.io/libc/0.2.155 \
+    crate://crates.io/lock_api/0.4.12 \
+    crate://crates.io/memoffset/0.9.1 \
+    crate://crates.io/once_cell/1.19.0 \
+    crate://crates.io/parking_lot/0.12.3 \
+    crate://crates.io/parking_lot_core/0.9.10 \
+    crate://crates.io/portable-atomic/1.6.0 \
+    crate://crates.io/ppv-lite86/0.2.17 \
+    crate://crates.io/proc-macro2/1.0.84 \
+    crate://crates.io/pyo3/0.21.2 \
+    crate://crates.io/pyo3-build-config/0.21.2 \
+    crate://crates.io/pyo3-ffi/0.21.2 \
+    crate://crates.io/pyo3-macros/0.21.2 \
+    crate://crates.io/pyo3-macros-backend/0.21.2 \
+    crate://crates.io/quote/1.0.36 \
+    crate://crates.io/rand/0.8.5 \
+    crate://crates.io/rand_chacha/0.3.1 \
+    crate://crates.io/rand_core/0.6.4 \
+    crate://crates.io/redox_syscall/0.5.1 \
+    crate://crates.io/scopeguard/1.2.0 \
+    crate://crates.io/smallvec/1.13.2 \
+    crate://crates.io/syn/2.0.66 \
+    crate://crates.io/target-lexicon/0.12.14 \
+    crate://crates.io/unicode-ident/1.0.12 \
+    crate://crates.io/unindent/0.2.3 \
+    crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \
+    crate://crates.io/windows-targets/0.52.5 \
+    crate://crates.io/windows_aarch64_gnullvm/0.52.5 \
+    crate://crates.io/windows_aarch64_msvc/0.52.5 \
+    crate://crates.io/windows_i686_gnu/0.52.5 \
+    crate://crates.io/windows_i686_gnullvm/0.52.5 \
+    crate://crates.io/windows_i686_msvc/0.52.5 \
+    crate://crates.io/windows_x86_64_gnu/0.52.5 \
+    crate://crates.io/windows_x86_64_gnullvm/0.52.5 \
+    crate://crates.io/windows_x86_64_msvc/0.52.5 \
+"
+
+SRC_URI[autocfg-1.3.0.sha256sum] = "0c4b4d0bd25bd0b74681c0ad21497610ce1b7c91b1022cd21c80c6fbdd9476b0"
+SRC_URI[bitflags-2.5.0.sha256sum] = "cf4b9d6a944f767f8e5e0db018570623c85f3d925ac718db4e06d0187adb21c1"
+SRC_URI[cfg-if-1.0.0.sha256sum] = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
+SRC_URI[getrandom-0.2.15.sha256sum] = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7"
+SRC_URI[heck-0.4.1.sha256sum] = "95505c38b4572b2d910cecb0281560f54b440a19336cbbcb27bf6ce6adc6f5a8"
+SRC_URI[indoc-2.0.5.sha256sum] = "b248f5224d1d606005e02c97f5aa4e88eeb230488bcc03bc9ca4d7991399f2b5"
+SRC_URI[libc-0.2.155.sha256sum] = "97b3888a4aecf77e811145cadf6eef5901f4782c53886191b2f693f24761847c"
+SRC_URI[lock_api-0.4.12.sha256sum] = "07af8b9cdd281b7915f413fa73f29ebd5d55d0d3f0155584dade1ff18cea1b17"
+SRC_URI[memoffset-0.9.1.sha256sum] = "488016bfae457b036d996092f6cb448677611ce4449e970ceaf42695203f218a"
+SRC_URI[once_cell-1.19.0.sha256sum] = "3fdb12b2476b595f9358c5161aa467c2438859caa136dec86c26fdd2efe17b92"
+SRC_URI[parking_lot-0.12.3.sha256sum] = "f1bf18183cf54e8d6059647fc3063646a1801cf30896933ec2311622cc4b9a27"
+SRC_URI[parking_lot_core-0.9.10.sha256sum] = "1e401f977ab385c9e4e3ab30627d6f26d00e2c73eef317493c4ec6d468726cf8"
+SRC_URI[portable-atomic-1.6.0.sha256sum] = "7170ef9988bc169ba16dd36a7fa041e5c4cbeb6a35b76d4c03daded371eae7c0"
+SRC_URI[ppv-lite86-0.2.17.sha256sum] = "5b40af805b3121feab8a3c29f04d8ad262fa8e0561883e7653e024ae4479e6de"
+SRC_URI[proc-macro2-1.0.84.sha256sum] = "ec96c6a92621310b51366f1e28d05ef11489516e93be030060e5fc12024a49d6"
+SRC_URI[pyo3-0.21.2.sha256sum] = "a5e00b96a521718e08e03b1a622f01c8a8deb50719335de3f60b3b3950f069d8"
+SRC_URI[pyo3-build-config-0.21.2.sha256sum] = "7883df5835fafdad87c0d888b266c8ec0f4c9ca48a5bed6bbb592e8dedee1b50"
+SRC_URI[pyo3-ffi-0.21.2.sha256sum] = "01be5843dc60b916ab4dad1dca6d20b9b4e6ddc8e15f50c47fe6d85f1fb97403"
+SRC_URI[pyo3-macros-0.21.2.sha256sum] = "77b34069fc0682e11b31dbd10321cbf94808394c56fd996796ce45217dfac53c"
+SRC_URI[pyo3-macros-backend-0.21.2.sha256sum] = "08260721f32db5e1a5beae69a55553f56b99bd0e1c3e6e0a5e8851a9d0f5a85c"
+SRC_URI[quote-1.0.36.sha256sum] = "0fa76aaf39101c457836aec0ce2316dbdc3ab723cdda1c6bd4e6ad4208acaca7"
+SRC_URI[rand-0.8.5.sha256sum] = "34af8d1a0e25924bc5b7c43c079c942339d8f0a8b57c39049bef581b46327404"
+SRC_URI[rand_chacha-0.3.1.sha256sum] = "e6c10a63a0fa32252be49d21e7709d4d4baf8d231c2dbce1eaa8141b9b127d88"
+SRC_URI[rand_core-0.6.4.sha256sum] = "ec0be4795e2f6a28069bec0b5ff3e2ac9bafc99e6a9a7dc3547996c5c816922c"
+SRC_URI[redox_syscall-0.5.1.sha256sum] = "469052894dcb553421e483e4209ee581a45100d31b4018de03e5a7ad86374a7e"
+SRC_URI[scopeguard-1.2.0.sha256sum] = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
+SRC_URI[smallvec-1.13.2.sha256sum] = "3c5e1a9a646d36c3599cd173a41282daf47c44583ad367b8e6837255952e5c67"
+SRC_URI[syn-2.0.66.sha256sum] = "c42f3f41a2de00b01c0aaad383c5a45241efc8b2d1eda5661812fda5f3cdcff5"
+SRC_URI[target-lexicon-0.12.14.sha256sum] = "e1fc403891a21bcfb7c37834ba66a547a8f402146eba7265b5a6d88059c9ff2f"
+SRC_URI[unicode-ident-1.0.12.sha256sum] = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
+SRC_URI[unindent-0.2.3.sha256sum] = "c7de7d73e1754487cb58364ee906a499937a0dfabd86bcb980fa99ec8c8fa2ce"
+SRC_URI[wasi-0.11.0+wasi-snapshot-preview1.sha256sum] = "9c8d87e72b64a3b4db28d11ce29237c246188f4f51057d65a7eab63b7987e423"
+SRC_URI[windows-targets-0.52.5.sha256sum] = "6f0713a46559409d202e70e28227288446bf7841d3211583a4b53e3f6d96e7eb"
+SRC_URI[windows_aarch64_gnullvm-0.52.5.sha256sum] = "7088eed71e8b8dda258ecc8bac5fb1153c5cffaf2578fc8ff5d61e23578d3263"
+SRC_URI[windows_aarch64_msvc-0.52.5.sha256sum] = "9985fd1504e250c615ca5f281c3f7a6da76213ebd5ccc9561496568a2752afb6"
+SRC_URI[windows_i686_gnu-0.52.5.sha256sum] = "88ba073cf16d5372720ec942a8ccbf61626074c6d4dd2e745299726ce8b89670"
+SRC_URI[windows_i686_gnullvm-0.52.5.sha256sum] = "87f4261229030a858f36b459e748ae97545d6f1ec60e5e0d6a3d32e0dc232ee9"
+SRC_URI[windows_i686_msvc-0.52.5.sha256sum] = "db3c2bf3d13d5b658be73463284eaf12830ac9a26a90c717b7f771dfe97487bf"
+SRC_URI[windows_x86_64_gnu-0.52.5.sha256sum] = "4e4246f76bdeff09eb48875a0fd3e2af6aada79d409d33011886d3e1581517d9"
+SRC_URI[windows_x86_64_gnullvm-0.52.5.sha256sum] = "852298e482cd67c356ddd9570386e2862b5673c85bd5f88df9ab6802b334c596"
+SRC_URI[windows_x86_64_msvc-0.52.5.sha256sum] = "bec47e5bfd1bff0eeaf6d8b485cc1074891a197ab4225d504cb7a1ab88b02bf0"
diff --git a/meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.upgraded b/meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.upgraded
new file mode 100644
index 00000000000..f60a62718db
--- /dev/null
+++ b/meta-selftest/recipes-devtools/python/python3-guessing-game_git.bb.upgraded
@@ -0,0 +1,19 @@
+SUMMARY = "The guessing game from The Rust Book using pyo3."
+DESCRIPTION = "Wrap a version of the guessing game from The Rust Book \
+to run in Python using pyo3."
+HOMEPAGE = "https://www.maturin.rs/tutorial"
+SECTION = "devel/python"
+LICENSE = "MIT & Apache-2.0"
+LIC_FILES_CHKSUM = "file://LICENSE-APACHE;md5=1836efb2eb779966696f473ee8540542 \
+                    file://LICENSE-MIT;md5=85fd3b67069cff784d98ebfc7d5c0797"
+
+SRC_URI = "git://git.yoctoproject.org/guessing-game.git;protocol=https;branch=main"
+
+PV = "0.2.0"
+SRCREV = "40cf004c2772ffa20ea803fa3be1528a75be3e98"
+
+S = "${WORKDIR}/git"
+
+inherit python_maturin cargo-update-recipe-crates
+
+require ${BPN}-crates.inc
diff --git a/meta/lib/oeqa/selftest/cases/devtool.py b/meta/lib/oeqa/selftest/cases/devtool.py
index 1cafb922ea4..432d9c9a673 100644
--- a/meta/lib/oeqa/selftest/cases/devtool.py
+++ b/meta/lib/oeqa/selftest/cases/devtool.py
@@ -2017,6 +2017,52 @@ class DevtoolUpgradeTests(DevtoolBase):
             newlines = f.readlines()
         self.assertEqual(desiredlines, newlines)
 
+    def test_devtool_upgrade_recipe_update_extra_tasks(self):
+        # Check preconditions
+        self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+        recipe = 'python3-guessing-game'
+        version = '0.2.0'
+        commit = '40cf004c2772ffa20ea803fa3be1528a75be3e98'
+        oldrecipefile = get_bb_var('FILE', recipe)
+        oldcratesincfile = os.path.join(os.path.dirname(oldrecipefile), os.path.basename(oldrecipefile).strip('_git.bb') + '-crates.inc')
+        tempdir = tempfile.mkdtemp(prefix='devtoolqa')
+        self.track_for_cleanup(tempdir)
+        # Check that recipe is not already under devtool control
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        # Check upgrade
+        result = runCmd('devtool upgrade %s %s --version %s --srcrev %s' % (recipe, tempdir, version, commit))
+        # Check if srctree at least is populated
+        self.assertTrue(len(os.listdir(tempdir)) > 0, 'srctree (%s) should be populated with new (%s) source code' % (tempdir, commit))
+        # Check new recipe file and new -crates.inc files are present
+        newrecipefile = os.path.join(self.workspacedir, 'recipes', recipe, os.path.basename(oldrecipefile))
+        newcratesincfile = os.path.join(self.workspacedir, 'recipes', recipe, os.path.basename(oldcratesincfile))
+        self.assertExists(newrecipefile, 'Recipe file should exist after upgrade')
+        self.assertExists(newcratesincfile, 'Recipe crates.inc file should exist after upgrade')
+        # Check devtool status and make sure recipe is present
+        result = runCmd('devtool status')
+        self.assertIn(recipe, result.output)
+        self.assertIn(tempdir, result.output)
+        # Check recipe got changed as expected
+        with open(oldrecipefile + '.upgraded', 'r') as f:
+            desiredlines = f.readlines()
+        with open(newrecipefile, 'r') as f:
+            newlines = f.readlines()
+        self.assertEqual(desiredlines, newlines)
+        # Check crates.inc got changed as expected
+        with open(oldcratesincfile + '.upgraded', 'r') as f:
+            desiredlines = f.readlines()
+        with open(newcratesincfile, 'r') as f:
+            newlines = f.readlines()
+        self.assertEqual(desiredlines, newlines)
+        # Check devtool reset recipe
+        result = runCmd('devtool reset %s -n' % recipe)
+        result = runCmd('devtool status')
+        self.assertNotIn(recipe, result.output)
+        self.assertNotExists(os.path.join(self.workspacedir, 'recipes', recipe), 'Recipe directory should not exist after resetting')
+
     def test_devtool_layer_plugins(self):
         """Test that devtool can use plugins from other layers.
 
-- 
2.45.2

Re: [OE-core] [PATCH v2 2/5] cargo-update-recipe-crates: add RECIPE_UPDATE_EXTRA_TASKS

[Ross Burton]

> On 11 Jun 2024, at 16:55, Tim Orling via lists.openembedded.org <tim.orling=konsulko.com@lists.openembedded.org> wrote:
> +RECIPE_UPDATE_EXTRA_TASKS:${PN} += "do_update_crates"

Why the ${PN}?  It’s not like you’d have RECIPE_UPDATE_EXTRA_TASKS:${PN}-doc.

Ross

[PATCH v3 1/5] devtool upgrade: enable RECIPE_UPDATE_EXTRA_TASKS

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

For some recipes, such as those that inherit cargo-update-recipe-crates,
we need to run additional tasks once the new sources have been unpacked.

Introduce a new variable RECIPE_UPDATE_EXTRA_TASKS which is a space-
delimited list of tasks to run after the new sources have been
unpacked in scripts/lib/devtool/upgrade.py ugrade() method.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v3:
  * Address rburton review comment
  * Since the variable is recipe scoped (as is the class being tested),
    we do not need the :${PN} appended to RECIPE_UPDATE_EXTRA_TASKS

Changes in v2:
  * Drop the try/except in the _run_recipe_update_extra_tasks method
  * Check the result of each task (tinfoil.build_targets) and raise
    a DevtoolError on failure

 scripts/lib/devtool/upgrade.py | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/scripts/lib/devtool/upgrade.py b/scripts/lib/devtool/upgrade.py
index a8130ed23f5..8e13833b51c 100644
--- a/scripts/lib/devtool/upgrade.py
+++ b/scripts/lib/devtool/upgrade.py
@@ -534,6 +534,15 @@ def _generate_license_diff(old_licenses, new_licenses):
             diff = diff + line
     return diff
 
+def _run_recipe_update_extra_tasks(pn, rd, tinfoil):
+    tasks = []
+    for task in (rd.getVar('RECIPE_UPDATE_EXTRA_TASKS') or '').split():
+        logger.info('Running extra recipe update task: %s' % task)
+        res = tinfoil.build_targets(pn, task, handle_events=True)
+
+        if not res:
+            raise DevtoolError('Running extra recipe update task %s for %s failed' % (task, pn))
+
 def upgrade(args, config, basepath, workspace):
     """Entry point for the devtool 'upgrade' subcommand"""
 
@@ -609,6 +618,8 @@ def upgrade(args, config, basepath, workspace):
                         copied, config.workspace_path, rd)
         standard._add_md5(config, pn, af)
 
+        _run_recipe_update_extra_tasks(pn, rd, tinfoil)
+
         update_unlockedsigs(basepath, workspace, args.fixed_setup, [pn])
 
         logger.info('Upgraded source extracted to %s' % srctree)
-- 
2.45.2

[PATCH v3 2/5] cargo-update-recipe-crates: add RECIPE_UPDATE_EXTRA_TASKS

[tim.orling]

From: Tim Orling <tim.orling@konsulko.com>

When we upgrade a recipe that inherits cargo-update-recipe-crates and
the upstream Cargo.toml/Cargo.lock have been changed, we need to run
the update_crates task or else the devtool upgrade (and therefore
AUH upgrade) will fail.

Add "do_update_crates" task to RECIPE_UPDATE_EXTRA_TASKS for all
recipes that inherit this class.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
Changes in v3:
  * Address rburton review comment
  * Since the variable is recipe scoped (as is the class being tested),
    we do not need the :${PN} appended to RECIPE_UPDATE_EXTRA_TASKS

No Changes in v2

 meta/classes-recipe/cargo-update-recipe-crates.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
index 8980137d02c..a19ce16b467 100644
--- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -18,6 +18,8 @@ do_update_crates[depends] = "python3-native:do_populate_sysroot"
 do_update_crates[nostamp] = "1"
 do_update_crates[doc] = "Update the recipe by reading Cargo.lock and write in ${THISDIR}/${BPN}-crates.inc"
 
+RECIPE_UPDATE_EXTRA_TASKS += "do_update_crates"
+
 # The directory where to search for Cargo.lock files
 CARGO_LOCK_SRC_DIR ??= "${S}"
 
-- 
2.45.2