Re: [OE-core] [RFC PATCH] cargo_common.bbclass: use source replacement instead of dependency patching

["Mathieu Dubois-Briand" <mathieu.dubois-briand@bootlin.com>]

On Fri Oct 3, 2025 at 11:30 PM CEST, Gyorgy Sarvari via lists.openembedded.org wrote:
> Cargo.toml files usually contain a list of dependencies in one of two forms:
> either a crate name that can be fetched from some registry (like crates.io), or
> as a source crate, which is most often fetched from a git repository.
>
> Normally cargo handles fetching the crates from both the registry and from git,
> however with Yocto this task is taken over by Bitbake.
>
> After fetching these crates, they are made available to cargo by adding the location
> to $CARGO_HOME/config.toml. The source crates are of interest here: each git repository
> that can be found in the SRC_URI is added as one source crate.
>
> This works most of the time, as long as the repository really contains one crate only.
>
> However in case the repository is a cargo workspace, it contains multiple crates in
> different subfolders, and in order to allow cargo to process them, they need to be
> listed separately. This is not happening with the current implementation of cargo_common.
>
> This change introduces the following:
> - instead of patching the dependencies, use source replacement (the primary motivation for
>   this was that maturin seems to ignore source crate patches from config.toml)
> - the above also allows to keep the original Cargo.lock untouched (the original implementation
>   deleted git repository lines from it)
> - it adds a new folder, currently ${UNPACKDIR}/yocto-vendored-source-crates. During processing
>   the separate crate folders are copied into this folder, and it is used as the central
>   vendoring folder. This is needed for source replacements: the folder that is used for
>   vendoring needs to contain the crates separately, one crate in one folder. Each folder
>   has the name of the crate that it contains. Workspaces are not included here (unless the
>   given manifest is a workspace AND a package at once)
> - previuosly the SRC_URI had to contain a "name" and a "destsuffix" parameter to be considered
>   to be a rust crate. The name is not derived from the Cargo.toml file, not from the SRC_URI.
>   Having destsuffix is still mandatory though.
>
> The change does not handle nested workspaces, only the top level Cargo.toml is processed.
>
> Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
> Cc: Tom Geelen <t.f.g.geelen@gmail.com>
>
> ---

Hi Gyorgy,

I know this is just an RFC, but I still took it for a run on the
autobuilder. It looks like id does break rust build:

ERROR: rust-native-1.90.0-r0 do_configure: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:cargo_common_do_patch_paths(d)
     0003:
File: '/srv/pokybuild/yocto-worker/genericx86-64/build/meta/classes-recipe/cargo_common.bbclass', lineno: 182, function: cargo_common_do_patch_paths
     0178:    lockfile = d.getVar("CARGO_LOCK_PATH")
     0179:    if not os.path.exists(lockfile):
     0180:        bb.fatal(f"{lockfile} file doesn't exist")
     0181:
 *** 0182:    lockfile = load_toml_file(lockfile)
     0183:
     0184:    # key is the repo url, value is a boolean, which is used later
     0185:    # to indicate if there is a matching repository in SRC_URI also
     0186:    lockfile_git_repos = {}
File: '/srv/pokybuild/yocto-worker/genericx86-64/build/meta/classes-recipe/cargo_common.bbclass', lineno: 137, function: load_toml_file
     0133:        cargo_toml_path = os.path.join(path, 'Cargo.toml')
     0134:        return os.path.exists(cargo_toml_path)
     0135:
     0136:    def load_toml_file(toml_path):
 *** 0137:        import tomllib
     0138:        with open(toml_path, 'rb') as f:
     0139:            toml = tomllib.load(f)
     0140:        return toml
     0141:
Exception: ModuleNotFoundError: No module named 'tomllib'

https://autobuilder.yoctoproject.org/valkyrie/#/builders/4/builds/2528
https://autobuilder.yoctoproject.org/valkyrie/#/builders/9/builds/2505
https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/2543
https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/2497

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com