Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2025-02-17 at 12:00 +0100, Stefan Herbrechtsmeier wrote:
>  Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
>  
> > e) add the ability to add custom hooks in the fetch process to
> > handle
> > the cases of needing to alter the flow for patching the components
> > list
> >  
> I fear this isn't easy possible and we have a design problem. Bitbake
> and oe-core assume that a patch doesn't influence the fetch and that
> the SRC_URI contains all fetched sources. Many code use the Fetch
> class or the SRC_URI direct and doesn't expect recursive implicit
> URIs of gitsm. In reality the SRC_URI describe the given URIs only
> and doesn't contain the additional implicit URIs. The task flow
> applies patches after the unpack and prevent patches to the gitsm. In
> reality a patch could influence the implicit URIs and isn't
> independent.
> 
> I would recommend to keep the patching outside of bitbake and instead
> handle the implicit URIs inside oe-core. We could write the implicit
> URIs into files in the work directory and migrate the scattered
> direct users of the SRC_URI / Fetch class to a common oe-core
> function. This functions returns the given URIs and optional any
> additional implicit URIs. This allows the usage of the task
> dependencies to ensure that the implicit URIs are resolved before
> use.
> 
> The package manager lock file needs a single unpack and patch step
> whereas the gitsm needs a recursive unpack and patch of every
> successive gitsm. This makes it impossible to use additional tasks.
> We could add the resolve of the implicit URIs to the fetch task to
> support recursive resolve (gitsm).
>  
> 1. download SRC_URIs
> 2. exit if no URI is marked as unresolved
> 3. unpack unresolved URIs
> 4. apply associated patches
> 5. resolve URIs
> 6. goto 2 
>  
> This will eliminate the need for the partial implicit URI and gitsm
> support in bitbake and makes the fetcher code simpler. The patching
> could remain in oe-core and patches could be applied to gitsm and
> package manager lock files. Additionally we remove code duplication
> and simplify future changes to the SRC_URI determination.
> 
> It will add an indirection for the SRC_URI but the current code
> already shows that this makes problems. Only the archiver class uses
> the expanded_urldata function and all other classes including the
> spdx class ignore the implicit SRC_URIs.

I'm a bit worried we're getting caught up by the terminology. We don't
necessarily have to "patch" the component list so much as allow
something like a function to hook and adjust things. That then removes
the need for it to be a specific task.

I'm also not sure that drawing gitsm into this is a great idea when it
is effectively already working  quite well. I appreciate the desire to
have neat abstractions applying accross everything equally but I'm not
sure we can achieve that with the level of usability we need and I'd
rather improve usability at the cost of the inclusion of gitsm, which
is the most functional fetcher we currently have in this group.

I will repeat again that I do strongly feel that a strong API in
bitbake is going to lead to an overall better design that something
bolted onto the fetcher in OE-Core.

Cheers,

Richard