Re: [OE-core] [RFC PATCH] cargo_common.bbclass: use source replacement instead of dependency patching

[Gyorgy Sarvari <skandigraun@gmail.com>]

On 10/5/25 15:23, Mathieu Dubois-Briand wrote:
> On Fri Oct 3, 2025 at 11:30 PM CEST, Gyorgy Sarvari via lists.openembedded.org wrote:
>> Cargo.toml files usually contain a list of dependencies in one of two forms:
>> either a crate name that can be fetched from some registry (like crates.io), or
>> as a source crate, which is most often fetched from a git repository.
>>
>> Normally cargo handles fetching the crates from both the registry and from git,
>> however with Yocto this task is taken over by Bitbake.
>>
>> After fetching these crates, they are made available to cargo by adding the location
>> to $CARGO_HOME/config.toml. The source crates are of interest here: each git repository
>> that can be found in the SRC_URI is added as one source crate.
>>
>> This works most of the time, as long as the repository really contains one crate only.
>>
>> However in case the repository is a cargo workspace, it contains multiple crates in
>> different subfolders, and in order to allow cargo to process them, they need to be
>> listed separately. This is not happening with the current implementation of cargo_common.
>>
>> This change introduces the following:
>> - instead of patching the dependencies, use source replacement (the primary motivation for
>>   this was that maturin seems to ignore source crate patches from config.toml)
>> - the above also allows to keep the original Cargo.lock untouched (the original implementation
>>   deleted git repository lines from it)
>> - it adds a new folder, currently ${UNPACKDIR}/yocto-vendored-source-crates. During processing
>>   the separate crate folders are copied into this folder, and it is used as the central
>>   vendoring folder. This is needed for source replacements: the folder that is used for
>>   vendoring needs to contain the crates separately, one crate in one folder. Each folder
>>   has the name of the crate that it contains. Workspaces are not included here (unless the
>>   given manifest is a workspace AND a package at once)
>> - previuosly the SRC_URI had to contain a "name" and a "destsuffix" parameter to be considered
>>   to be a rust crate. The name is not derived from the Cargo.toml file, not from the SRC_URI.
>>   Having destsuffix is still mandatory though.
>>
>> The change does not handle nested workspaces, only the top level Cargo.toml is processed.
>>
>> Signed-off-by: Gyorgy Sarvari <skandigraun@gmail.com>
>> Cc: Tom Geelen <t.f.g.geelen@gmail.com>
>>
>> ---
> Hi Gyorgy,
>
> I know this is just an RFC, but I still took it for a run on the
> autobuilder. It looks like id does break rust build:
>
> ERROR: rust-native-1.90.0-r0 do_configure: Error executing a python function in exec_func_python() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>      0001:
>  *** 0002:cargo_common_do_patch_paths(d)
>      0003:
> File: '/srv/pokybuild/yocto-worker/genericx86-64/build/meta/classes-recipe/cargo_common.bbclass', lineno: 182, function: cargo_common_do_patch_paths
>      0178:    lockfile = d.getVar("CARGO_LOCK_PATH")
>      0179:    if not os.path.exists(lockfile):
>      0180:        bb.fatal(f"{lockfile} file doesn't exist")
>      0181:
>  *** 0182:    lockfile = load_toml_file(lockfile)
>      0183:
>      0184:    # key is the repo url, value is a boolean, which is used later
>      0185:    # to indicate if there is a matching repository in SRC_URI also
>      0186:    lockfile_git_repos = {}
> File: '/srv/pokybuild/yocto-worker/genericx86-64/build/meta/classes-recipe/cargo_common.bbclass', lineno: 137, function: load_toml_file
>      0133:        cargo_toml_path = os.path.join(path, 'Cargo.toml')
>      0134:        return os.path.exists(cargo_toml_path)
>      0135:
>      0136:    def load_toml_file(toml_path):
>  *** 0137:        import tomllib
>      0138:        with open(toml_path, 'rb') as f:
>      0139:            toml = tomllib.load(f)
>      0140:        return toml
>      0141:
> Exception: ModuleNotFoundError: No module named 'tomllib'

Thanks a lot for testing.
Looks like I got to target older Python versions than 3.11... is 3.9 the
oldest that is still supported?

> https://autobuilder.yoctoproject.org/valkyrie/#/builders/4/builds/2528
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/9/builds/2505
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/6/builds/2543
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/20/builds/2497
>
> Thanks,
> Mathieu
>