[PATCH 0/1] crates.io data access policy

[PATCH 0/1] crates.io data access policy

[Johan Anderholm]

It appears that the crate fetcher does not adhere to the data access policy
of crates.io as is described here:

https://crates.io/data-access#api

Namely the part about adding a user-agent identifying the application and
information on how to reach the project in case of issues.

Also users are requested to use the CDN instead of the download API, as that
does not have a 1 request per second limit:

https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751

This patch addresses the request to use a CDN and seems to fix a recent
breakage of the crate fetcher. It does not address the part about a user-agent
since I do not know how to do that, and I do not know what to put in it anyway.

Johan Anderholm (1):
  fetch2/crate: use CDN for fetching crates

 lib/bb/fetch2/crate.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

-- 
2.34.1

[PATCH 1/1] fetch2/crate: use CDN for fetching crates

[Johan Anderholm]

This avoids the 1 req/sec that the API has.

Reference: https://github.com/rust-lang/crates.io/issues/13482
Reference: https://blog.rust-lang.org/2024/03/11/crates-io-download-changes/
Signed-off-by: Johan Anderholm <johanam@axis.com>
---
 lib/bb/fetch2/crate.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/crate.py b/lib/bb/fetch2/crate.py
index e611736f0..2d3078899 100644
--- a/lib/bb/fetch2/crate.py
+++ b/lib/bb/fetch2/crate.py
@@ -68,8 +68,11 @@ class Crate(Wget):
         # if using upstream just fix it up nicely
         if host == 'crates.io':
             host = 'crates.io/api/v1/crates'
+            cdn_host = 'static.crates.io/crates'
+        else:
+            cdn_host = host
 
-        ud.url = "https://%s/%s/%s/download" % (host, name, version)
+        ud.url = "https://%s/%s/%s/download" % (cdn_host, name, version)
         ud.versionsurl = "https://%s/%s/versions" % (host, name)
         ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
         if 'name' not in ud.parm:
-- 
2.34.1

Re: [bitbake-devel] [PATCH 0/1] crates.io data access policy

[Quentin Schulz]

Hi Johan,

On 4/27/26 3:51 PM, Johan Anderholm via lists.openembedded.org wrote:
> [You don't often get email from johan.anderholm=axis.com@lists.openembedded.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> It appears that the crate fetcher does not adhere to the data access policy
> of crates.io as is described here:
> 
> https://crates.io/data-access#api
> 
> Namely the part about adding a user-agent identifying the application and
> information on how to reach the project in case of issues.
> 
> Also users are requested to use the CDN instead of the download API, as that
> does not have a 1 request per second limit:
> 
> https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751
> 
> This patch addresses the request to use a CDN and seems to fix a recent
> breakage of the crate fetcher. It does not address the part about a user-agent
> since I do not know how to do that, and I do not know what to put in it anyway.
> 

The crate fetcher is derived from the wget fetch method which does seem 
to implement a UA reporting bitbake and its version, see 234f9e810494 
("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). 
The best thing would be to try yourself (I'm assuming having a webserver 
you control and change the fetcher to use your local webserver (or 
possibly even setting up a PREMIRRORS for crates)).

Cheers,
Quentin

Re: [bitbake-devel] [PATCH 0/1] crates.io data access policy

[Ross Burton]

On 27 Apr 2026, at 18:35, Quentin Schulz via lists.openembedded.org <quentin.schulz=cherry.de@lists.openembedded.org> wrote:
> 
> The crate fetcher is derived from the wget fetch method which does seem to implement a UA reporting bitbake and its version, see 234f9e810494 ("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). The best thing would be to try yourself (I'm assuming having a webserver you control and change the fetcher to use your local webserver (or possibly even setting up a PREMIRRORS for crates)).

That’s checkstatus, not the actual fetch.

Put this on the list of reasons why I dislike the wget fetcher: two entirely distinct codepaths.

Ross