Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier <stefan.herbrechtsmeier-oss@weidmueller.com>]

[-- Attachment #1: Type: text/plain, Size: 5132 bytes --]

Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
> I've pulled this to a separate email/thread since I'd like to take a
> slight step back and put some different ideas into the mix as well as
> explain where my own thoughts are right now. I'm also doing this so
> that we can focus the discussion and give others a place to catch up
> from. For that reason I may restate some information below.
Thanks for the restate.

> We have two fundamentally different approaches:
>
> a) a single entry in SRC_URI with magic behind the scenes expanding
> this into a list of dependencies
>
> b) multiple entries in SRC_URI generated with tooling
>
> gitsm is similar to a), partly because it can contain recursive
> references to other submodules so the second approach wouldn't work.
The second approach works. The only difference is that a lock file of a 
package manager already resolves the recursive references / inter 
dependencies. In case of gitsm you have to handle the dependencies 
inside the tool and fetch a dependency to resolve it dependencies. In 
both cases a change of a single dependency could influence the other 
dependencies.

> crates are handled with the b) approach today.
>
> Developers in general are more comfortable with b) since they can more
> easily see what is going on but it is also the harder one since it
> deviates most from the underlying tools and has two sets of data that
> require to be in sync.
>
> Some feel the .inc files enabling b) are effectively machine generated
> and could be removed with the work happening transparently instead,
> simplifying the recipes and commits. This allows easier use of the
> underlying tooling too.
>
> I think firstly, we need to document some key principles. Behind the
> scenes, any given url should expand to a defined list of components and
> that list has to be deterministic and not "floating", i.e. always the
> same regardless of changes in any registry or other upstream. If there
> are changes, they need to be detected and there needs to be a hard
> error. Also, if the code sees things declared in a way they could vary,
> that also needs to be a hard error.
The purpose of the lock file is to lock the source of a package to be 
reproducible.

>
> Most of the concerns I've seen are about how easy it is to understand
> what is going on behind the scenes. The move of code to OE and
> splitting everything into multiple tasks/stages does do that to some
> extent but it does it in a way which I think is going to create a new
> and different set of problems.
Okay, but please keep in mind that some of my oe patches are reasonable 
independent of the native bitbake fetcher and it is possible to 
integrate the steps from the early class into the fetch task.

> I'm therefore wondering if there is a different way. The changes I'm
> wondering about would be to:
>
> a) embrace the single SRC_URI entry
>
> b) require a checksum of the internal "URL list" that is included
> in SRC_URI, much in the same way that we have checksums of tarballs.
The list isn't fix because it depends on the configured package manager 
proxy or registry. We have to remove this feature. But the user could 
use a PREMIRROR to redirect the upstream proxy to its private proxy.

> For better or worse, we have low trust in the underlying tools to get
> this right (they are getting better).

We don't need to trust the tools. We parse the lock file and enrich it 
with fix values. The resolve is deterministic. The output only depends 
on the resolve function, variable values and lock file content.

> c) if the checksum doesn't match, we know something went wrong and
> error

Can you please elaborate this point. We already check the integrity of 
the lock file and we have deterministically resolve the SRC_URIs.

> d) require the new modules to write the URL list into a known location
> as part of unpack
Why is this needed? The generated SRC_URIs could be resolved via 
fetcher.expanded_urldata().

> e) add the ability to add custom hooks in the fetch process to handle
> the cases of needing to alter the flow for patching the components list
I'm afraid this will be complicated since PATCH is applied in S and not 
in UNPACKDIR.

> f) create new tools that allow the fetcher to be stepped through and
> for example partially run, or run with clear debug output showing what
> was happening at each stage (show the list of components?). This may be
> standalone tools, maybe a devtool module, I don't know. We may want to
> make the fetch/unpack logs more useful in general as right now you
> don't get much useful data about what it is doing.
>
>
> If we do those things, where does that get us? How much buy in do our
> different stakeholders have?
>
> FWIW I am leaning towards having this code in the bitbake fetcher as a
> first class citizen as to do otherwise is going to create layers of
> abstraction and we probably have enough of those already.

The advantage is that the SRC_URI still contains the dependencies if you 
expand the urldata. On the other side the integration of the patches in 
the fetcher sounds complicated.

[-- Attachment #2: Type: text/html, Size: 7403 bytes --]