Re: [Buildroot] [PATCH 1/1] support/download/cargo-post-process: use cargo output for vendor config

["Yann E. MORIN" <yann.morin.1998@free.fr>]

Simon, All,

On 2022-10-20 18:16 +0000, Richter Simon spake thusly:
> Am 19.10.22 um 22:52 schrieb Yann E. MORIN:
> > Could you provide a bit more details on how this works? 
[--SNIP--]
> But if a project needs dependencies from other sources, say github,
> there needs to be an extra entry for each non-crates.io dependency.
> As an example, here is the needed vendor config for conduit - a 
> matrix homeserver.
> (https://gitlab.com/famedly/conduit
> rev cb2b5beea8a3fb27ed514ee9e46b5729aa272179)
> 
>     [source.crates-io]
>     replace-with = "vendored-sources"
>     
>     [source."https://github.com/ruma/ruma"]
>     git = "https://github.com/ruma/ruma"
>     rev = "fba6f70c2df8294f96567f56464a46e3d237a8e9"
>     replace-with = "vendored-sources"
>     
>     [source."https://github.com/timokoesters/heed.git"]
>     git = "https://github.com/timokoesters/heed.git"
>     rev = "f6f825da7fb2c758867e05ad973ef800a6fe1d5d"
>     replace-with = "vendored-sources"
>     
>     [source."https://github.com/timokoesters/reqwest"]
>     git = "https://github.com/timokoesters/reqwest"
>     rev = "57b7cf4feb921573dfafad7d34b9ac6e44ead0bd"
>     replace-with = "vendored-sources"
>     
>     [source.vendored-sources]
>     directory = "vendor"

Ok, thanks, that helps see the overal concept.

I see all the above use git repositories directly, so I suspect this is
going to cause explicit git-clone + build (that's OK). However, I was
also wondering if one could have a "registry" like crates.io, either
internal to a company, or another public resource elsewhere, and how
that would work...

> > Also, can we have a package that uses non-crates.io dependencies, so
> > that we have a test-case to ensure we have no regression?
> I'm not sure if i understood that correctly. Do you mean bringing a
> package into buildroot

Yes, I was wondering if there was a publicly visible package that we
could cary in Buildroot, so that we know the feature would be regularly
exercised. You pointed to conduit: it wouldbe good to have it packaged
in Buildrooot.

[--SNIP--]
> > Can we keep the vendoring to stdout as it currently is, and re-run it
> > right after, just to generate the config.toml file? I.e. something like:
[--SNIP--]
> While digging around i found the following solution:
>     cargo vendor \
>         --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} \
>         --locked VENDOR \
>     | tee .cargo/config
> 
> Cargo seems to print all progress output to stderr but not the needed
> output for the config. This goes to stdout. So we only have to tee stdout 
> to the .cargo/config file. I will update the patch accordingly.

Great! :-)

Please, also note that this has the potential to generate different
hashes, so please check existing cargo packages to see what breaks.

If we can't keep generating the same archives, then we must find a way
to dfferentiate the newer archives so they do not clash with the older
ones. Something like:

    diff --git a/package/pkg-download.mk b/package/pkg-download.mk
    index 0718f21aad..e247fd38c5 100644
    --- a/package/pkg-download.mk
    +++ b/package/pkg-download.mk
    @@ -22,6 +22,7 @@ export LOCALFILES := $(call qstrip,$(BR2_LOCALFILES))
     # download backend:
     BR_FMT_VERSION_git = -br1
     BR_FMT_VERSION_svn = -br2
    +BR_FMT_VERSION_cargp = -cargo1

     DL_WRAPPER = support/download/dl-wrapper

    diff --git a/package/pkg-utils.mk b/package/pkg-utils.mk
    index 0945e6ed31..7486d12499 100644
    --- a/package/pkg-utils.mk
    +++ b/package/pkg-utils.mk
    @@ -48,7 +48,7 @@ pkgname = $(lastword $(subst /, ,$(pkgdir)))
     # Helper to build the extension for a package archive, based on various
     # conditions.
     # $(1): upper-case package name
    -pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD)).tar.gz
    +pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD))$(BR_FMT_VERSION_$($(1)_DOWNLOAD_POST_PROCESS)).tar.gz

     # Define extractors for different archive suffixes
     INFLATE.bz2  = $(BZCAT)

Note: this is probably not sufficient, but that gives you a lead if
needed. Also, we'd need a comment at the top of the cargo-post-process
helper, like we have at the top of the git helper for example.

> > Finally, unrelated to your change: if the package already has a
> > .cargo/config.toml, then we irremediably override it. Is there something
> > in that file that would otherwise be required to do the build?
> 
> Yes, i noticed that too. Overwriting a possibly existing project specific
> configuration is not good.
> I think this also should be changed (in a different patch, as mentioned).

Or we can wait for until something actually needs it, so that we do have
an actual case to look at.

> > Can we simply append to the file rather than overwrite it, i.e.:
> > .... >> .cargo/config.toml
> 
> Unfortunately it is not that simple because of two reasons:
> 
> 1. cargo looks for a .cargo/config or (up from cargo 1.39) a 
> .cargo/config.toml file for configuration. If both files exist, cargo uses 
> .cargo/config and ignores the .cargo/config.toml file.
> 
> 2. if configuration entries for vendoring already exist in a configuration, 
> it is not allowed to add them again. Cargo will not build the project if a 
> corresponding entry exists more than once.
> 
> The first case would be quiet easy to address but the second one would
> need some sort of merging.

Yes, so that all calls for waiting until we do have an actual issue.

> So, speaking for this patch, i would keep the current behavior "overwrite
> every possibly existing config with the vendoring information".

Yes, because that has so far not caused an issue (at least that we
noticed), so this means it is not an actual problem.

Thanks!

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot