Re: [bitbake-devel] [RFC PATCH 0/6] fetch2: add support for implicit urls

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2025-09-08 at 11:20 +0200, Stefan Herbrechtsmeier wrote:
>  Am 07.09.2025 um 17:52 schrieb Richard Purdie via lists.openembedded.org:
>  On Tue, 2025-09-02 at 08:55 +0200, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> > > The patch series add support for implicit URLs inside the fetcher. The
> > > implicit URLs could be defined inside a source like a version control
> > > system (git submodule) or a lock file (package-lock.json, cargo.lock or
> > > go.sum). The integration of implicit URLs beside explicit URLs
> > > simplifies the fetcher classes and avoid bugs because of iterations
> > > between the Fetch and FetchMethod classes.
> > > 
> > > The series remove most methods inside the gitsm fetcher and only leaves
> > > the parsing of the git submodules and the unpack functionality. It
> > > allows the gitsm fetcher to use the premirror only feature. The current
> > > implementation leads to problems because the download of the git
> > > submodules is triggered via the download method which is called deeply
> > > inside the fetcher code.

> >  We had the discussion a while back and the conclusion seemed to be that
> > implict urls were disliked by a significant number of people as the
> > were too unclear about what was going on behind the scenes and also
> > made things like software manifests harder.
> >   
> > 
> 
> It looks like I miss some discussion and especially the conclusion.
>  
> 
>  I assume you mean recipes by software manifests.

No, I did mean software manifests. If the urls are explict, it makes
generating manifests of the sources being used more obvious for people
to understand. Yes, there are programmatic ways of doing it with
implicit urls but people don't like them.

By conclusions, I was taking that as the outcome of the last set of
discussions but there was a lot of different emails and it was hard to
follow. Perhaps i got the conclusion wrong, I don't know.

> >  There was a strong
> > preference for metadata helpers and explicit lists of components
> > which
> > we have for crates/rust and now for go too.
> >  
>  
> Does this mean the npmsw and gitsm fetchers are obsolete and should
> be replace by a metadata helpers to fix open issues?

I really don't know about npmsw. I don't use it and I don't really
follow development there. I'm don't know much about the current set of
issues it may have.

With gitsm, I think that is generally accepted by people and I don't
see a strong reason to change it at present. I would be interested in a
clear summary of what the known issues are (e.g. the premirror issue
you mentioned).

> > It feels like this series is moving us back to the other direction.
> > Is
> > that correct and if so, what has changed in the approach since the
> > last
> > discussion?
> >  
>  
> Do you mean the response to my last RFC? In this case it wasn't clear
> to me that the project is against implicit URLs and that the npmsw
> and gitsm fetcher are the wrong direction.

I'm trying to read the "mood" of our developer community and right now,
it feels like putting a lot of complexity hidden in the fetcher isn't
what people want to see as they don't understand it and can't "see"
what is going on. During the discussions, I think we identified some
key fundamental issues with implict urls for some fetch types too.

There is some hard work needs to be done in trying to summarise those
discussions and writing down the "results" so that we don't have to
redo this every time a new patch series comes along. By that, I mean a
non-emotive list of the current advantages, disdtantages and known bugs
of the current approach and any proposed alternatives we might choose.
 It perhaps falls to me as the developer lead for bitbake to try and do
it but I'd very much welcome help from anyone else in trying to do it
as I simply don't have the mental bandwidth to try and do that for this
topic right now (due to e.g. bitbake-setup). I certainly don't have to
be the one who does it. I appreciate it isn't a fun task though.


> This series is only a cleanup of the existing functionally. The gitsm
> fetcher uses implicit URLs but doesn't work correct because of the
> misuses of the download code.

I'm worried about where the series is trying to take the project and
codebase though, hence the questions about intent.


> It is useless to start the discussion again. The project doesn't like
> implicit URLs. It prefers a special task as a replacement for the
> separate recipetool. It decides against the on-the-fly parse inside
> the fetcher.

The developers using the project feel much happier with that approach,
yes.

> How should I proceed? I have working code which parse the cargo.lock,
> go.sum and package-lock.json files and only use Git and Wget
> fetchers. The code uses the vendor feature of the package managers to
> create a patchable folder of the sources. It simplifies the npm class
> and add additional classes to build packages. The code integrates
> valid URLs inside the SBOM and creates components with name and
> version per dependency. Furthermore, I have rework the gitsm fetcher
> to hopefully fix some open issues. I can convert my cargo, go, npm
> and gitsm parser into metadata helpers but this is useless if the
> project dislike python functions to generate URLs like the pypi class
> or prefer package manager specific code inside the fetcher.

So you are proposing we drop the crate and gomod fetchers in favour of
implict urls?

I'd suggest sharing a branch of your changes so that others can see and
understand the implications and hopefully experiment a bit, see if it
can convince some people that implict urls are the way forward.

Cheers,

Richard