[PATCH] cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock

[PATCH] cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock

[Martin Schwan]

Do not fail if only some Cargo.lock files are empty. Only fail, if there
are absolutely no dependencies found in any Cargo.lock.

This fixes the following error message, which would occur with "bitbake
-c update_crates python3-orjson":

    ERROR: python3-orjson-3.10.17-r0 do_update_crates: Execution of '.../python3-orjson/3.10.17/temp/run.do_update_crates.70693' failed with exit code 1
    ERROR: Logfile of failure stored in: .../python3-orjson/3.10.17/temp/log.do_update_crates.70693
    Log data follows:
    | DEBUG: Executing python function extend_recipe_sysroot
    | NOTE: Direct dependencies are ['.../sources/oe-core/../oe-core/meta/recipes-devtools/quilt/quilt-native_0.69.bb:do_populate_sysroot', 'virtual:native:.../sources/oe-core/../oe-core/meta/recipes-devtools/patch/patch_2.8.bb:do_populate_sysroot', 'virtual:native:.../sources/oe-core/../oe-core/meta/recipes-devtools/python/python3_3.13.9.bb:do_populate_sysroot']
    | NOTE: Installed into sysroot: []
    | NOTE: Skipping as already exists in sysroot: ['gettext-minimal-native', 'cmake-native', 'libtool-native', 'quilt-native', 'texinfo-dummy-native', 'openssl-native', 'expat-native', 'ncurses-native', 'util-linux-libuuid-native', 'zlib-native', 'libedit-native', 'make-native', 'patch-native', 'perl-native', 'python3-native', 'bzip2-native', 'xz-native', 'zstd-native', 'attr-native', 'gdbm-native', 'libffi-native', 'sqlite3-native']
    | DEBUG: Python function extend_recipe_sysroot finished
    | DEBUG: Executing shell function do_update_crates
    | Traceback (most recent call last):
    |   File "<stdin>", line 41, in <module>
    |   File "<stdin>", line 12, in get_crates
    | ValueError: Unable to find any candidate crates that use crates.io
    |
    | The above exception was the direct cause of the following exception:
    |
    | Traceback (most recent call last):
    |   File "<stdin>", line 43, in <module>
    | ValueError: Cannot parse '.../python3-orjson/3.10.17/sources/orjson-3.10.17/include/cargo/simdutf8-0.1.5/Cargo.lock'
    | WARNING: exit code 1 from a shell command.
    ERROR: Task (.../sources/oe-core/../meta-openembedded/meta-python/recipes-devtools/python/python3-orjson_3.10.17.bb:do_update_crates) failed with exit code '1'

After this change, a working crates include file is produced and warning
messages are output instead to inform about potential issues in the
corresponding project:

    WARNING: python3-orjson-3.10.17-r0 do_update_crates: Unable to find any candidate crates that use crates.io
    WARNING: python3-orjson-3.10.17-r0 do_update_crates: Cannot parse '.../python3-orjson/3.10.17/sources/orjson-3.10.17/include/cargo/simdutf8-0.1.5/Cargo.lock'

While at it, restructure do_update_crates() to be a real Python function
and use the corresponding code directly inline.

Signed-off-by: Martin Schwan <m.schwan@phytec.de>
---
 .../cargo-update-recipe-crates.bbclass        | 91 +++++++++----------
 1 file changed, 45 insertions(+), 46 deletions(-)

diff --git a/meta/classes-recipe/cargo-update-recipe-crates.bbclass b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
index 3251d5ef2e..589f497680 100644
--- a/meta/classes-recipe/cargo-update-recipe-crates.bbclass
+++ b/meta/classes-recipe/cargo-update-recipe-crates.bbclass
@@ -23,59 +23,58 @@ RECIPE_UPGRADE_EXTRA_TASKS += "do_update_crates"
 # The directory where to search for Cargo.lock files
 CARGO_LOCK_SRC_DIR ??= "${S}"
 
-do_update_crates() {
-    TARGET_FILE="${THISDIR}/${BPN}-crates.inc"
+python do_update_crates() {
+    import tomllib
 
-    nativepython3 - <<EOF
+    target_file = d.expand('${THISDIR}/${BPN}-crates.inc')
+    cargo_lock_src_dir = d.getVar('CARGO_LOCK_SRC_DIR')
 
-def get_crates(f):
-    import tomllib
-    c_list = '# from %s' % os.path.relpath(f, '${CARGO_LOCK_SRC_DIR}')
-    c_list += '\nSRC_URI += " \\\'
-    crates = tomllib.load(open(f, 'rb'))
+    def get_crates(f):
+        c_list = '# from %s' % os.path.relpath(f, cargo_lock_src_dir)
+        c_list += '\nSRC_URI += " \\'
+        crates = tomllib.load(open(f, 'rb'))
 
-    # Build a list with crates info that have crates.io in the source
-    crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package']))
+        # Build a list with crates info that have crates.io in the source
+        crates_candidates = list(filter(lambda c: 'crates.io' in c.get('source', ''), crates['package']))
 
-    if not crates_candidates:
-        raise ValueError("Unable to find any candidate crates that use crates.io")
+        if not crates_candidates:
+            bb.warn('Unable to find any candidate crates that use crates.io')
+            return None
 
-    # Update crates uri and their checksum, to avoid name clashing on the checksum
-    # we need to rename crates with name and version to have a unique key
-    cksum_list = ''
-    for c in crates_candidates:
-        rename = "%s-%s" % (c['name'], c['version'])
-        c_list += '\n    crate://crates.io/%s/%s \\\' % (c['name'], c['version'])
-        if 'checksum' in c:
-            cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum'])
+        # Update crates uri and their checksum, to avoid name clashing on the checksum
+        # we need to rename crates with name and version to have a unique key
+        cksum_list = ''
+        for c in crates_candidates:
+            rename = "%s-%s" % (c['name'], c['version'])
+            c_list += '\n    crate://crates.io/%s/%s \\' % (c['name'], c['version'])
+            if 'checksum' in c:
+                cksum_list += '\nSRC_URI[%s.sha256sum] = "%s"' % (rename, c['checksum'])
 
-    c_list += '\n"\n'
-    c_list += cksum_list
-    c_list += '\n'
-    return c_list
+        c_list += '\n"\n'
+        c_list += cksum_list
+        c_list += '\n'
+        return c_list
 
-import os
-crates = "# Autogenerated with 'bitbake -c update_crates ${PN}'\n\n"
-found = False
-for root, dirs, files in os.walk('${CARGO_LOCK_SRC_DIR}'):
-    # ignore git and patches directories
-    if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.pc')):
-        continue
-    if root.startswith(os.path.join('${CARGO_LOCK_SRC_DIR}', '.git')):
-        continue
-    for file in files:
-        if file == 'Cargo.lock':
-            try:
+    crates = d.expand('# Autogenerated with \'bitbake -c update_crates ${PN}\'\n\n')
+    for root, dirs, files in os.walk(cargo_lock_src_dir):
+        # ignore git and patches directories
+        if root.startswith(os.path.join(cargo_lock_src_dir, '.pc')):
+            continue
+        if root.startswith(os.path.join(cargo_lock_src_dir, '.git')):
+            continue
+        for file in files:
+            if file == 'Cargo.lock':
                 cargo_lock_path = os.path.join(root, file)
-                crates += get_crates(os.path.join(root, file))
-            except Exception as e:
-                raise ValueError("Cannot parse '%s'" % cargo_lock_path) from e
-            else:
-                found = True
-if not found:
-    raise ValueError("Unable to find any Cargo.lock in ${CARGO_LOCK_SRC_DIR}")
-open("${TARGET_FILE}", 'w').write(crates)
-EOF
+                c = get_crates(cargo_lock_path)
+                if c is None:
+                    bb.warn('Cannot parse \'%s\'' % cargo_lock_path)
+                else:
+                    crates += c
+    if crates is None:
+        bb.error(f'Unable to find any Cargo.lock in {cargo_lock_src_dir}')
+
+    with open(target_file, 'w') as f:
+        f.write(crates)
 
-    bbnote "Successfully update crates inside '${TARGET_FILE}'"
+    bb.note(f'Successfully updated crates inside \'{target_file}\'')
 }
-- 
2.43.0

Re: [OE-core] [PATCH] cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock

[Alexander Kanavin]

On Tue, 25 Nov 2025 at 10:01, Martin Schwan via lists.openembedded.org
<m.schwan=phytec.de@lists.openembedded.org> wrote:
>
> Do not fail if only some Cargo.lock files are empty. Only fail, if there
> are absolutely no dependencies found in any Cargo.lock.

...

> While at it, restructure do_update_crates() to be a real Python function
> and use the corresponding code directly inline.

This, as a diff on the mailing list, is incredibly hard to review, and
needs to be split up. Refactoring in one commit (explaining the
benefits), functional changes in another please. Even better if you
can split things further, and explain what the particular
modifications are (rather than use terms like 'restructure').

Aelx

Re: [OE-core] [PATCH] cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock

[Martin Schwan]

On Tue, 2025-11-25 at 11:23 +0100, Alexander Kanavin wrote:
> On Tue, 25 Nov 2025 at 10:01, Martin Schwan via
> lists.openembedded.org
> <m.schwan=phytec.de@lists.openembedded.org> wrote:
> > 
> > Do not fail if only some Cargo.lock files are empty. Only fail, if
> > there
> > are absolutely no dependencies found in any Cargo.lock.
> 
> ...
> 
> > While at it, restructure do_update_crates() to be a real Python
> > function
> > and use the corresponding code directly inline.
> 
> This, as a diff on the mailing list, is incredibly hard to review,
> and
> needs to be split up. Refactoring in one commit (explaining the
> benefits), functional changes in another please. Even better if you
> can split things further, and explain what the particular
> modifications are (rather than use terms like 'restructure').

Thanks for the feedback. Admittedly, I was a bit lazy with this one, as
the changes were quite interwoven. I'll send a v2 and split things up,
as best as I can.

Regards,
Martin

> 
> Aelx

Re: [OE-core] [PATCH] cargo-update-recipe-crates: Don't fail for partially empty Cargo.lock

[Clayton Casciato]

> Do not fail if only some Cargo.lock files are empty. Only fail, if there
> are absolutely no dependencies found in any Cargo.lock.
meta-security's suricata also has this problem.

I currently use a local patch to change the exception to a warning.

See https://lists.yoctoproject.org/g/yocto-patches/message/2577

--
Clayton Casciato