[PATCH] fetch:crate: create versioned 'name' entries

[PATCH] fetch:crate: create versioned 'name' entries

[Enrico Scholz]

From: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>

It is common for rust packages to depend on different versions of the
same crate.  E.g.

|    crate://crates.io/windows_x86_64_msvc/0.42.2 \
|    crate://crates.io/windows_x86_64_msvc/0.48.0 \

Identification only by the plain crate name makes the sha256sum
ambiguous

| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"

and requires lot of manual work to fix the SRC_URI list.

Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 lib/bb/fetch2/crate.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/crate.py b/lib/bb/fetch2/crate.py
index 590dc9c12681..a7021e5b367c 100644
--- a/lib/bb/fetch2/crate.py
+++ b/lib/bb/fetch2/crate.py
@@ -72,7 +72,7 @@ class Crate(Wget):
         ud.url = "https://%s/%s/%s/download" % (host, name, version)
         ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
         if 'name' not in ud.parm:
-            ud.parm['name'] = name
+            ud.parm['name'] = '%s-%s' % (name, version)
 
         logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))
 
-- 
2.39.2

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 3082 bytes --]

Hello,

There are multiple issues with this patch.

First, this is not compatible with recipes which will override the name
parameter in their crate:// entry like zvariant in meta-selftest layer.

Second, the cargo-update-recipe-crates class already did that modification
for you (it set name parameter to be pkg-vers and it expect bitbake fetcher
to respect that),
by the way, the zavariant I talk about was built with usage of this class

Last, this modification makes the bitbake unit test fail.

To bootstrap a cargo base recipe which have not this checksum list, below
are
the steps to make:
  - make the recipe inherit cargo-update-recipe-crates
  - remove all crate:// from your recipe
  - create an empty ${BPN}-crates.inc file and make your recipe requires it
  - execute "bitbake -c update_crates your_recipe"
  - copy paste the output of bitbake about the missing checksums in your
${BPN}-crates.inc

And you should have a buildable cargo based recipe with these.



On Wed, 5 Apr 2023 at 12:16, Enrico Scholz via lists.openembedded.org
<enrico.scholz=sigma-chemnitz.de@lists.openembedded.org> wrote:

> From: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
>
> It is common for rust packages to depend on different versions of the
> same crate.  E.g.
>
> |    crate://crates.io/windows_x86_64_msvc/0.42.2 \
> |    crate://crates.io/windows_x86_64_msvc/0.48.0 \
>
> Identification only by the plain crate name makes the sha256sum
> ambiguous
>
> | SRC_URI[windows_x86_64_msvc.sha256sum] =
> "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
> | SRC_URI[windows_x86_64_msvc.sha256sum] =
> "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"
>
> and requires lot of manual work to fix the SRC_URI list.
>
> Making the 'name' property unique by appending the version allows
> direct copy & paste of reported sha256sum errors to complete the
> crates list.
>
> Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> ---
>  lib/bb/fetch2/crate.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/lib/bb/fetch2/crate.py b/lib/bb/fetch2/crate.py
> index 590dc9c12681..a7021e5b367c 100644
> --- a/lib/bb/fetch2/crate.py
> +++ b/lib/bb/fetch2/crate.py
> @@ -72,7 +72,7 @@ class Crate(Wget):
>          ud.url = "https://%s/%s/%s/download" % (host, name, version)
>          ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
>          if 'name' not in ud.parm:
> -            ud.parm['name'] = name
> +            ud.parm['name'] = '%s-%s' % (name, version)
>
>          logger.debug2("Fetching %s to %s" % (ud.url,
> ud.parm['downloadfilename']))
>
> --
> 2.39.2
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14645):
> https://lists.openembedded.org/g/bitbake-devel/message/14645
> Mute This Topic: https://lists.openembedded.org/mt/98079257/6213388
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4795 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Enrico Scholz]

Frédéric Martinsons <frederic.martinsons@gmail.com> writes:

> First, this is not compatible with recipes which will override the name
> parameter in their crate:// entry like zvariant in meta-selftest
> layer.

When this is really a problem (I do not see why), it existed already
before.


> Second, the cargo-update-recipe-crates class already did that
> modification for you (it set name parameter to be pkg-vers and it
> expect bitbake fetcher to respect that),

cargo-update-recipe-crates is not very useful and has huge bootstrapping
problems (see below).


> Last, this modification makes the bitbake unit test fail.

ok; my fault.  I will send a v2


> To bootstrap a cargo base recipe which have not this checksum list, below
> are
> the steps to make:
>   - make the recipe inherit cargo-update-recipe-crates
>   - remove all crate:// from your recipe
>   - create an empty ${BPN}-crates.inc file and make your recipe requires it
>   - execute "bitbake -c update_crates your_recipe"

This does not work.  This class requires Cargo.lock, but to have
Cargo.lock, do_compile() must be executed.  This fails when required
crates are not available yet.


Enrico

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Wed, 5 Apr 2023 at 14:09, Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
wrote:

> > To bootstrap a cargo base recipe which have not this checksum list, below
> > are
> > the steps to make:
> >   - make the recipe inherit cargo-update-recipe-crates
> >   - remove all crate:// from your recipe
> >   - create an empty ${BPN}-crates.inc file and make your recipe requires
> it
> >   - execute "bitbake -c update_crates your_recipe"
>
> This does not work.  This class requires Cargo.lock, but to have
> Cargo.lock, do_compile() must be executed.  This fails when required
> crates are not available yet.
>
>
> Enrico
>

A rust project must have a Cargo.lock in its source (
https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
)
unless it is a pure library but I don't see the use case of building a rust
library that wouldn't be a simple
crate on crates.io.

[-- Attachment #2: Type: text/html, Size: 1527 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Enrico Scholz]

Frédéric Martinsons <frederic.martinsons@gmail.com> writes:

> A rust project must have a Cargo.lock in its source (
> https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries

It is recommended (due to documentation issues), but not required.


Enrico

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

It remains the problem with zvariant recipe in meta-selftest layer but I'll
let others answer since they may have a bigger picture on that

Le mer. 5 avr. 2023, 14:24, Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
a écrit :

> Frédéric Martinsons <frederic.martinsons@gmail.com> writes:
>
> > A rust project must have a Cargo.lock in its source (
> >
> https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
>
> It is recommended (due to documentation issues), but not required.
>
>
> Enrico
>

[-- Attachment #2: Type: text/html, Size: 1123 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 1696 bytes --]

No, I take that back, it should not be an issue since you set the name only
if it is not already there. But, to be sure, did you try to build some
existing cargo based recipe (zvariant but also python3-bcrypt for example).

Anyway, I tend to think we should keep using
cargo-update-update-recipe-crates as the only entry point to update those
recipe so maybe we want to made modifications there to handle the most
possible use case (missing Cargo.lock, bad Cargo.lock, some other corner
cases... Etc)

Le mer. 5 avr. 2023, 14:50, Frederic Martinsons via lists.openembedded.org
<frederic.martinsons=gmail.com@lists.openembedded.org> a écrit :

> It remains the problem with zvariant recipe in meta-selftest layer but
> I'll let others answer since they may have a bigger picture on that
>
> Le mer. 5 avr. 2023, 14:24, Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
> a écrit :
>
>> Frédéric Martinsons <frederic.martinsons@gmail.com> writes:
>>
>> > A rust project must have a Cargo.lock in its source (
>> >
>> https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries
>>
>> It is recommended (due to documentation issues), but not required.
>>
>>
>> Enrico
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#14656):
> https://lists.openembedded.org/g/bitbake-devel/message/14656
> Mute This Topic: https://lists.openembedded.org/mt/98079257/6213388
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> frederic.martinsons@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3351 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Enrico Scholz]

Frédéric Martinsons <frederic.martinsons@gmail.com> writes:

> No, I take that back, it should not be an issue since you set the name
> only if it is not already there. But, to be sure, did you try to build
> some existing cargo based recipe (zvariant but also python3-bcrypt for
> example).

It will break cargo related recipes which were created between 2023-03-16
and 2023-03-31.  Packages before this date do not contain sha256sum and
after this date, they have an explicit 'name' (which matches the one
generated by my patch).


Enrico

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 846 bytes --]

Le mer. 5 avr. 2023, 18:45, Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
a écrit :

> Frédéric Martinsons <frederic.martinsons@gmail.com> writes:
>
> > No, I take that back, it should not be an issue since you set the name
> > only if it is not already there. But, to be sure, did you try to build
> > some existing cargo based recipe (zvariant but also python3-bcrypt for
> > example).
>
> It will break cargo related recipes which were created between 2023-03-16
> and 2023-03-31.  Packages before this date do not contain sha256sum and
> after this date, they have an explicit 'name' (which matches the one
> generated by my patch).
>
>
> Enrico
>

Ok, I guess these dates match my patches series, if I do things rights, I
handle all the recipes I know of (in meta-openembedded and
openembedded-core at least)

>

[-- Attachment #2: Type: text/html, Size: 1481 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Richard Purdie]

On Wed, 2023-04-05 at 19:05 +0200, Frederic Martinsons wrote:
> 
> 
> Le mer. 5 avr. 2023, 18:45, Enrico Scholz
> <enrico.scholz@sigma-chemnitz.de> a écrit :
> > Frédéric Martinsons <frederic.martinsons@gmail.com> writes:
> > 
> > > No, I take that back, it should not be an issue since you set the
> > > name
> > > only if it is not already there. But, to be sure, did you try to
> > > build
> > > some existing cargo based recipe (zvariant but also python3-
> > > bcrypt for
> > > example).
> > 
> > It will break cargo related recipes which were created between
> > 2023-03-16
> > and 2023-03-31.  Packages before this date do not contain sha256sum
> > and
> > after this date, they have an explicit 'name' (which matches the
> > one
> > generated by my patch).
> > 
> > 
> > Enrico
> 
> Ok, I guess these dates match my patches series, if I do things
> rights, I handle all the recipes I know of (in meta-openembedded and
> openembedded-core at least) 

This now gives me a problem as rc1 for 4.2 built yesterday. I'm
guessing we don't want a stable release series with one form for urls
for six months and a different one in master?

Does this mean I need to scrap rc1, merge this, wait for the recipe
updates and then rebuild the release?

Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 1779 bytes --]

Le jeu. 6 avr. 2023, 10:45, Richard Purdie <
richard.purdie@linuxfoundation.org> a écrit :

> On Wed, 2023-04-05 at 19:05 +0200, Frederic Martinsons wrote:
> >
> >
> > Le mer. 5 avr. 2023, 18:45, Enrico Scholz
> > <enrico.scholz@sigma-chemnitz.de> a écrit :
> > > Frédéric Martinsons <frederic.martinsons@gmail.com> writes:
> > >
> > > > No, I take that back, it should not be an issue since you set the
> > > > name
> > > > only if it is not already there. But, to be sure, did you try to
> > > > build
> > > > some existing cargo based recipe (zvariant but also python3-
> > > > bcrypt for
> > > > example).
> > >
> > > It will break cargo related recipes which were created between
> > > 2023-03-16
> > > and 2023-03-31.  Packages before this date do not contain sha256sum
> > > and
> > > after this date, they have an explicit 'name' (which matches the
> > > one
> > > generated by my patch).
> > >
> > >
> > > Enrico
> >
> > Ok, I guess these dates match my patches series, if I do things
> > rights, I handle all the recipes I know of (in meta-openembedded and
> > openembedded-core at least)
>
> This now gives me a problem as rc1 for 4.2 built yesterday. I'm
> guessing we don't want a stable release series with one form for urls
> for six months and a different one in master?
>
> Does this mean I need to scrap rc1, merge this, wait for the recipe
> updates and then rebuild the release?
>
> Cheers,
>
> Richard
>

I'm not sure I understand, is the patch we talk already merged? In the 4.2?

This patch seems to me an improvement and without it, all the recipes I
know of was correctly built (even those in meta-security from what I read
from Armin in https://lists.openembedded.org/g/bitbake-devel/message/14658)

[-- Attachment #2: Type: text/html, Size: 2942 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Richard Purdie]

On Thu, 2023-04-06 at 12:05 +0200, Frédéric Martinsons wrote:
> 
> 
> Le jeu. 6 avr. 2023, 10:45, Richard Purdie
> <richard.purdie@linuxfoundation.org> a écrit :
> > On Wed, 2023-04-05 at 19:05 +0200, Frederic Martinsons wrote:
> > > 
> > > 
> > > Le mer. 5 avr. 2023, 18:45, Enrico Scholz
> > > <enrico.scholz@sigma-chemnitz.de> a écrit :
> > > > Frédéric Martinsons <frederic.martinsons@gmail.com> writes:
> > > > 
> > > > > No, I take that back, it should not be an issue since you set
> > > > > the
> > > > > name
> > > > > only if it is not already there. But, to be sure, did you try
> > > > > to
> > > > > build
> > > > > some existing cargo based recipe (zvariant but also python3-
> > > > > bcrypt for
> > > > > example).
> > > > 
> > > > It will break cargo related recipes which were created between
> > > > 2023-03-16
> > > > and 2023-03-31.  Packages before this date do not contain
> > > > sha256sum
> > > > and
> > > > after this date, they have an explicit 'name' (which matches
> > > > the
> > > > one
> > > > generated by my patch).
> > > > 
> > > > 
> > > > Enrico
> > > 
> > > Ok, I guess these dates match my patches series, if I do things
> > > rights, I handle all the recipes I know of (in meta-openembedded
> > > and
> > > openembedded-core at least) 
> > 
> > This now gives me a problem as rc1 for 4.2 built yesterday. I'm
> > guessing we don't want a stable release series with one form for
> > urls
> > for six months and a different one in master?
> > 
> > Does this mean I need to scrap rc1, merge this, wait for the recipe
> > updates and then rebuild the release?
> > 
> > Cheers,
> > 
> > Richard
> 
> I'm not sure I understand, is the patch we talk already merged? In
> the 4.2? 

4.2 is what is in the mickledore and bitbake 2.4 branches, i.e. the
recent changes but without the patch in this thread.

> This patch seems to me an improvement and without it, all the recipes
> I know of was correctly built (even those in meta-security from what
> I read from Armin in
> https://lists.openembedded.org/g/bitbake-devel/message/14658) 

I'm worried that people seem to be saying we need the patch in this
thread but it creates an incompatibility with what was already updated
previously.

Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

>> > It will break cargo related recipes which were created between
>> > 2023-03-16 and 2023-03-31.  Packages before this date do not
>> > contain sha256sum and after this date, they have an explicit
>> > 'name' (which matches the one generated by my patch).
>> 
>> Ok, I guess these dates match my patches series, if I do things
>> rights, I handle all the recipes I know of (in meta-openembedded and
>> openembedded-core at least) 
>
> This now gives me a problem as rc1 for 4.2 built yesterday. I'm
> guessing we don't want a stable release series with one form for urls
> for six months and a different one in master?
>
> Does this mean I need to scrap rc1, merge this, wait for the recipe
> updates and then rebuild the release?

To avoid breaking builds:

1. OE recipes without explicit ';name=' need to be updated; this can be
   done by 'bitbake -c update_crates'.  Afais, this affects only
   
   | meta/recipes-devtools/python/python3-bcrypt_4.0.1.bb
   | meta/recipes-devtools/python/python3-cryptography_39.0.2.bb

   and in oe-meta

   | meta-oe/recipes-core/uutils-coreutils/uutils-coreutils_0.0.17.bb
   | meta-python/recipes-devtools/python/python3-pyruvate_1.1.2.bb

2. the patch can be applied to bitbake

3. [optionally] change cargo-update-recipe-crates.bbclass to remove the
   explicit (and now redundant) ';name=' from generated SRC_URI list.
   This can be done post-mickledore



Enrico

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Richard Purdie]

On Thu, 2023-04-06 at 12:20 +0200, Enrico Scholz wrote:
> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> 
> > > > It will break cargo related recipes which were created between
> > > > 2023-03-16 and 2023-03-31.  Packages before this date do not
> > > > contain sha256sum and after this date, they have an explicit
> > > > 'name' (which matches the one generated by my patch).
> > > 
> > > Ok, I guess these dates match my patches series, if I do things
> > > rights, I handle all the recipes I know of (in meta-openembedded and
> > > openembedded-core at least) 
> > 
> > This now gives me a problem as rc1 for 4.2 built yesterday. I'm
> > guessing we don't want a stable release series with one form for urls
> > for six months and a different one in master?
> > 
> > Does this mean I need to scrap rc1, merge this, wait for the recipe
> > updates and then rebuild the release?
> 
> To avoid breaking builds:
> 
> 1. OE recipes without explicit ';name=' need to be updated; this can be
>    done by 'bitbake -c update_crates'.  Afais, this affects only
>    
>    | meta/recipes-devtools/python/python3-bcrypt_4.0.1.bb
>    | meta/recipes-devtools/python/python3-cryptography_39.0.2.bb
> 
>    and in oe-meta
> 
>    | meta-oe/recipes-core/uutils-coreutils/uutils-coreutils_0.0.17.bb
>    | meta-python/recipes-devtools/python/python3-pyruvate_1.1.2.bb
> 
> 2. the patch can be applied to bitbake
> 
> 3. [optionally] change cargo-update-recipe-crates.bbclass to remove the
>    explicit (and now redundant) ';name=' from generated SRC_URI list.
>    This can be done post-mickledore

I've gone ahead and stopped rc1 since I know in the long run this issue
is going to cause problems.

Short term I'll merge this patch and the fixes needed to the oe-core
recipes. 

Now I've been looking more closely at this, I did start to wonder if we
shouldn't just put ;sha256sum=XXX parameters onto the crate urls we
generate though? If we do that, we don't need to name them at all.

That is something we could move to at a later date if it make sense so
it is less important from a release standpoint.

Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 2517 bytes --]

Oh my, I see a lot of patches going around at the moment, sorry for the
trouble, I didn't expect the changes I made would be so soon in a release

Le jeu. 6 avr. 2023, 14:14, Richard Purdie <
richard.purdie@linuxfoundation.org> a écrit :

> On Thu, 2023-04-06 at 12:20 +0200, Enrico Scholz wrote:
> > Richard Purdie <richard.purdie@linuxfoundation.org> writes:
> >
> > > > > It will break cargo related recipes which were created between
> > > > > 2023-03-16 and 2023-03-31.  Packages before this date do not
> > > > > contain sha256sum and after this date, they have an explicit
> > > > > 'name' (which matches the one generated by my patch).
> > > >
> > > > Ok, I guess these dates match my patches series, if I do things
> > > > rights, I handle all the recipes I know of (in meta-openembedded and
> > > > openembedded-core at least)
> > >
> > > This now gives me a problem as rc1 for 4.2 built yesterday. I'm
> > > guessing we don't want a stable release series with one form for urls
> > > for six months and a different one in master?
> > >
> > > Does this mean I need to scrap rc1, merge this, wait for the recipe
> > > updates and then rebuild the release?
> >
> > To avoid breaking builds:
> >
> > 1. OE recipes without explicit ';name=' need to be updated; this can be
> >    done by 'bitbake -c update_crates'.  Afais, this affects only
> >
> >    | meta/recipes-devtools/python/python3-bcrypt_4.0.1.bb
> >    | meta/recipes-devtools/python/python3-cryptography_39.0.2.bb
> >
> >    and in oe-meta
> >
> >    | meta-oe/recipes-core/uutils-coreutils/uutils-coreutils_0.0.17.bb
> >    | meta-python/recipes-devtools/python/python3-pyruvate_1.1.2.bb
> >
> > 2. the patch can be applied to bitbake
> >
> > 3. [optionally] change cargo-update-recipe-crates.bbclass to remove the
> >    explicit (and now redundant) ';name=' from generated SRC_URI list.
> >    This can be done post-mickledore
>
> I've gone ahead and stopped rc1 since I know in the long run this issue
> is going to cause problems.
>
> Short term I'll merge this patch and the fixes needed to the oe-core
> recipes.
>
> Now I've been looking more closely at this, I did start to wonder if we
> shouldn't just put ;sha256sum=XXX parameters onto the crate urls we
> generate though? If we do that, we don't need to name them at all.
>
> That is something we could move to at a later date if it make sense so
> it is less important from a release standpoint.
>
> Cheers,
>
> Richard
>

[-- Attachment #2: Type: text/html, Size: 3653 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Richard Purdie]

On Thu, 2023-04-06 at 15:00 +0200, Frédéric Martinsons wrote:
> Oh my, I see a lot of patches going around at the moment, sorry for
> the trouble, I didn't expect the changes I made would be so soon in a
> release 

In hindsight I probably shouldn't have taken things so close to the
release but there are pros and cons to taking or not taking. I thought
things were sorted out and it turned out they weren't quite done.

I suspect we'd not have had some of the feedback had I not merged them.

I've merged a few changes so hopefully this resolves things for bitbake
and OE-Core. The other layers should be able to work things out from
there.

Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 831 bytes --]

Le jeu. 6 avr. 2023, 15:44, Richard Purdie <
richard.purdie@linuxfoundation.org> a écrit :

> On Thu, 2023-04-06 at 15:00 +0200, Frédéric Martinsons wrote:
> > Oh my, I see a lot of patches going around at the moment, sorry for
> > the trouble, I didn't expect the changes I made would be so soon in a
> > release
>
> In hindsight I probably shouldn't have taken things so close to the
> release but there are pros and cons to taking or not taking. I thought
> things were sorted out and it turned out they weren't quite done.
>
> I suspect we'd not have had some of the feedback had I not merged them.
>
> I've merged a few changes so hopefully this resolves things for bitbake
> and OE-Core. The other layers should be able to work things out from
> there.
>
> Cheers,
>
> Richard
>

Hope so too.

>

[-- Attachment #2: Type: text/html, Size: 1399 bytes --]

Re: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> Now I've been looking more closely at this, I did start to wonder if we
> shouldn't just put ;sha256sum=XXX parameters onto the crate urls we
> generate though? If we do that, we don't need to name them at all.

yes; I think, it is a good idea for autogenerated code.

But for manually written one, I somehow still prefer the dedicated
SRC_URI[<name>.sha256sum] syntax because it easier to read and fix.


Enrico

';sha256sum=' attribute in SRC_URI (was: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries)

[Enrico Scholz]

Richard Purdie <richard.purdie@linuxfoundation.org> writes:

> Now I've been looking more closely at this, I did start to wonder if we
> shouldn't just put ;sha256sum=XXX parameters onto the crate urls we
> generate though? If we do that, we don't need to name them at all.

I just tested this with

SRC_URI += " \
    crate://crates.io/addr2line/0.19.0;sha256sum=a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97 \
    crate://crates.io/adler/1.0.2;adler-1.0.2.sha256sum=f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe \
"

and got

| do_fetch: Missing SRC_URI checksum, please add those to the recipe: 
| SRC_URI[addr2line-0.19.0.sha256sum] = "a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97"


It seems that ';sha256sum' does not suffice but that you have to dup the
'name'.



Enrico

Re: ';sha256sum=' attribute in SRC_URI (was: [bitbake-devel] [PATCH] fetch:crate: create versioned 'name' entries)

[Frédéric Martinsons]

[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]

Maybe it missed something in crate fetcher but, providing this fetcher
inherits wget one, and if wget supports this attribute, so should crate
fetcher do.

I quickly look into differents recipe of oe-core, meta-oe, meta-security
etc and I don't
see a SRC_URI which uses sha256sum inline attribute, so it may not be
correctly parsed.

On Thu, 6 Apr 2023 at 16:18, Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
wrote:

> Richard Purdie <richard.purdie@linuxfoundation.org> writes:
>
> > Now I've been looking more closely at this, I did start to wonder if we
> > shouldn't just put ;sha256sum=XXX parameters onto the crate urls we
> > generate though? If we do that, we don't need to name them at all.
>
> I just tested this with
>
> SRC_URI += " \
>     crate://
> crates.io/addr2line/0.19.0;sha256sum=a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97
> \
>     crate://
> crates.io/adler/1.0.2;adler-1.0.2.sha256sum=f26201604c87b1e01bd3d98f8d5d9a8fcbb815e8cedb41ffccbeb4bf593a35fe
> \
> "
>
> and got
>
> | do_fetch: Missing SRC_URI checksum, please add those to the recipe:
> | SRC_URI[addr2line-0.19.0.sha256sum] =
> "a76fd60b23679b7d19bd066031410fb7e458ccc5e958eb5c325888ce4baedc97"
>
>
> It seems that ';sha256sum' does not suffice but that you have to dup the
> 'name'.
>
>
>
> Enrico
>

[-- Attachment #2: Type: text/html, Size: 2180 bytes --]

[PATCH, v2] fetch2/crate: create versioned 'name' entries

[Enrico Scholz]

From: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>

It is common for rust packages to depend on different versions of the
same crate.  E.g.

|    crate://crates.io/windows_x86_64_msvc/0.42.2 \
|    crate://crates.io/windows_x86_64_msvc/0.48.0 \

Identification only by the plain crate name makes the sha256sum
ambiguous

| SRC_URI[windows_x86_64_msvc.sha256sum] = "9aec5da331524158c6d1a4ac0ab1541149c0b9505fde06423b02f5ef0106b9f0"
| SRC_URI[windows_x86_64_msvc.sha256sum] = "1a515f5799fe4961cb532f983ce2b23082366b898e52ffbce459c86f67c8378a"

and requires lot of manual work to fix the SRC_URI list.

Making the 'name' property unique by appending the version allows
direct copy & paste of reported sha256sum errors to complete the
crates list.

Signed-off-by: Enrico Scholz <enrico.scholz@sigma-chemnitz.de>
---
 lib/bb/fetch2/crate.py | 2 +-
 lib/bb/tests/fetch.py  | 8 ++++----
 2 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/lib/bb/fetch2/crate.py b/lib/bb/fetch2/crate.py
index 590dc9c12681..a7021e5b367c 100644
--- a/lib/bb/fetch2/crate.py
+++ b/lib/bb/fetch2/crate.py
@@ -72,7 +72,7 @@ class Crate(Wget):
         ud.url = "https://%s/%s/%s/download" % (host, name, version)
         ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
         if 'name' not in ud.parm:
-            ud.parm['name'] = name
+            ud.parm['name'] = '%s-%s' % (name, version)
 
         logger.debug2("Fetching %s to %s" % (ud.url, ud.parm['downloadfilename']))
 
diff --git a/lib/bb/tests/fetch.py b/lib/bb/tests/fetch.py
index da6716890070..6ef0836f2be9 100644
--- a/lib/bb/tests/fetch.py
+++ b/lib/bb/tests/fetch.py
@@ -2380,7 +2380,7 @@ class CrateTest(FetcherTest):
         ud = fetcher.ud[fetcher.urls[0]]
 
         self.assertIn("name", ud.parm)
-        self.assertEqual(ud.parm["name"], "glob")
+        self.assertEqual(ud.parm["name"], "glob-0.2.11")
         self.assertIn("downloadfilename", ud.parm)
         self.assertEqual(ud.parm["downloadfilename"], "glob-0.2.11.crate")
 
@@ -2428,13 +2428,13 @@ class CrateTest(FetcherTest):
         ud = fetcher.ud[fetcher.urls[0]]
 
         self.assertIn("name", ud.parm)
-        self.assertEqual(ud.parm["name"], "glob")
+        self.assertEqual(ud.parm["name"], "glob-0.2.11")
         self.assertIn("downloadfilename", ud.parm)
         self.assertEqual(ud.parm["downloadfilename"], "glob-0.2.11.crate")
 
         ud = fetcher.ud[fetcher.urls[1]]
         self.assertIn("name", ud.parm)
-        self.assertEqual(ud.parm["name"], "time")
+        self.assertEqual(ud.parm["name"], "time-0.1.35")
         self.assertIn("downloadfilename", ud.parm)
         self.assertEqual(ud.parm["downloadfilename"], "time-0.1.35.crate")
 
@@ -2451,7 +2451,7 @@ class CrateTest(FetcherTest):
     def test_crate_incorrect_cksum(self):
         uri = "crate://crates.io/aho-corasick/0.7.20"
         self.d.setVar('SRC_URI', uri)
-        self.d.setVarFlag("SRC_URI", "aho-corasick.sha256sum", hashlib.sha256("Invalid".encode("utf-8")).hexdigest())
+        self.d.setVarFlag("SRC_URI", "aho-corasick-0.7.20.sha256sum", hashlib.sha256("Invalid".encode("utf-8")).hexdigest())
 
         uris = self.d.getVar('SRC_URI').split()
 
-- 
2.39.2