Re: [bitbake-devel] [PATCH v2 0/4] fetch2: add alternative fetch method based on curl

[Richard Purdie <richard.purdie@linuxfoundation.org>]

Hi Pascal,

On Tue, 2026-03-10 at 00:29 +0100, Pascal Eberhard via B4 Relay via lists.openembedded.org wrote:
> wget fetch method is used for web downloads with http, https, ftp, ftps
> protocols. wget cmdline tool is missing some features such as hostname
> resolution by the proxy when using SOCKS5 proxy. SSH tunnel provides
> this feature for example.
> 
> This curl fetch method is based on curl cmdline tool and provides the
> same protocols as wget and supports SOCKS5 hostname resolution.
> This class inherits wget method in order to avoid code duplication as
> much as possible.
> 
> wget remains the default download method. curl fetch method can be
> enabled by setting a new bitbake variable:
>   BB_FETCH_METHOD_HTTP = "curl"
> 
> The hostname resolution by SOCKS5 proxy is activated by setting
> environment variable:
>   all_proxy="socks5h://...""
> 
> a patch on ce-core will be sent if this proposal is accepted to
> conditionnaly add curl to HOSTTOOLS variable in meta/conf/bitbake.conf:
> HOSTTOOLS += "${@bb.utils.contains('BB_FETCH_METHOD_HTTP', 'curl', 'curl','', d)}"
> 
> Signed-off-by: Pascal Eberhard <pascal.eberhard@se.com>
> ---
> Changes in v2:
> - removed type hint unions and tested with Python 3.9, thanks Mathieu
>   Dubois-Briand for reporting this.
> - curl tests are now skipped when curl cmdline tool is not found.
> - Link to v1: https://lore.kernel.org/r/20260305-add_alt_fetch_method_curl-v1-0-0d0220e5fa59@se.com

We discussed this today a bit on the review call and we're all very
nervous about this change.

One of our guiding principles is not to have multiple ways of doing the
same thing. The reason is simple, so that we don't end up with two semi
functional things instead of one good one. Adding this "curl" mode to
the fetcher creates this scenario where one thing works for some and
the other works for everyone else. We'd then have to document when to
use one, when to use the other and potentially which functionalities
work in each different case.

The fetcher changes don't appear to change everything, they look like
you changed just enough to get your use cases to work.

The "wget" fetcher is already a hybrid where it uses wget for some
things and urllib for others. It does do that consistently though,
always using wget for certain operations and urllib for the status
tests.

We're not wedded to using wget, what we do want to do is provide a
strong stable and single API to enable fetching of http/https urls.

Given this, adding in the switch isn't an approach the project wants to
take. Equally, we've used wget reliably for a long time and it is the
thing we know. Swapping it for something else isn't something we'd
undertake lightly either. There are likely things that wget does better
than curl too for example. There are also factors like documentation to
consider if we were to switch to curl, our prerequisites would need
updating for all the distros, our autobuilder workers would need to be
reconfigured and so on.

I think we therefore need to have a wider discussion about how to move
forward here and work out what the best solution would be to aim for
long term, then come up with a plan to get there.

I'm conscious the LTS/feature freeze is looming but this isn't the kind
of change we can rush in before then unfortunately, we are simply too
late for that.

I'm also really aware of the limited bandwidth many of us have to
contribute to discussions like this. I personally find that really
frustrating as I want to try and help but there simply aren't enough
hours in the day :/.

So whilst I appreciate this isn't the answer you wanted to hear, I
don't think we can take these changes right now and we need to find a
different way to help you solve your issue.

Cheers,

Richard