'vendor' fetching discussion cont.

'vendor' fetching discussion cont.

[Richard Purdie]

I've pulled this to a separate email/thread since I'd like to take a
slight step back and put some different ideas into the mix as well as
explain where my own thoughts are right now. I'm also doing this so
that we can focus the discussion and give others a place to catch up
from. For that reason I may restate some information below.

We have two fundamentally different approaches:

a) a single entry in SRC_URI with magic behind the scenes expanding
this into a list of dependencies

b) multiple entries in SRC_URI generated with tooling

gitsm is similar to a), partly because it can contain recursive
references to other submodules so the second approach wouldn't work.
crates are handled with the b) approach today.

Developers in general are more comfortable with b) since they can more
easily see what is going on but it is also the harder one since it
deviates most from the underlying tools and has two sets of data that
require to be in sync.

Some feel the .inc files enabling b) are effectively machine generated
and could be removed with the work happening transparently instead,
simplifying the recipes and commits. This allows easier use of the
underlying tooling too.

I think firstly, we need to document some key principles. Behind the
scenes, any given url should expand to a defined list of components and
that list has to be deterministic and not "floating", i.e. always the
same regardless of changes in any registry or other upstream. If there
are changes, they need to be detected and there needs to be a hard
error. Also, if the code sees things declared in a way they could vary,
that also needs to be a hard error.

Most of the concerns I've seen are about how easy it is to understand
what is going on behind the scenes. The move of code to OE and
splitting everything into multiple tasks/stages does do that to some
extent but it does it in a way which I think is going to create a new
and different set of problems.

I'm therefore wondering if there is a different way. The changes I'm
wondering about would be to:

a) embrace the single SRC_URI entry

b) require a checksum of the internal "URL list" that is included
in SRC_URI, much in the same way that we have checksums of tarballs.
For better or worse, we have low trust in the underlying tools to get
this right (they are getting better).

c) if the checksum doesn't match, we know something went wrong and
error

d) require the new modules to write the URL list into a known location
as part of unpack

e) add the ability to add custom hooks in the fetch process to handle
the cases of needing to alter the flow for patching the components list

f) create new tools that allow the fetcher to be stepped through and
for example partially run, or run with clear debug output showing what
was happening at each stage (show the list of components?). This may be
standalone tools, maybe a devtool module, I don't know. We may want to
make the fetch/unpack logs more useful in general as right now you
don't get much useful data about what it is doing.


If we do those things, where does that get us? How much buy in do our
different stakeholders have?

FWIW I am leaning towards having this code in the bitbake fetcher as a
first class citizen as to do otherwise is going to create layers of
abstraction and we probably have enough of those already.

Cheers,

Richard

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 5132 bytes --]

Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
> I've pulled this to a separate email/thread since I'd like to take a
> slight step back and put some different ideas into the mix as well as
> explain where my own thoughts are right now. I'm also doing this so
> that we can focus the discussion and give others a place to catch up
> from. For that reason I may restate some information below.
Thanks for the restate.

> We have two fundamentally different approaches:
>
> a) a single entry in SRC_URI with magic behind the scenes expanding
> this into a list of dependencies
>
> b) multiple entries in SRC_URI generated with tooling
>
> gitsm is similar to a), partly because it can contain recursive
> references to other submodules so the second approach wouldn't work.
The second approach works. The only difference is that a lock file of a 
package manager already resolves the recursive references / inter 
dependencies. In case of gitsm you have to handle the dependencies 
inside the tool and fetch a dependency to resolve it dependencies. In 
both cases a change of a single dependency could influence the other 
dependencies.

> crates are handled with the b) approach today.
>
> Developers in general are more comfortable with b) since they can more
> easily see what is going on but it is also the harder one since it
> deviates most from the underlying tools and has two sets of data that
> require to be in sync.
>
> Some feel the .inc files enabling b) are effectively machine generated
> and could be removed with the work happening transparently instead,
> simplifying the recipes and commits. This allows easier use of the
> underlying tooling too.
>
> I think firstly, we need to document some key principles. Behind the
> scenes, any given url should expand to a defined list of components and
> that list has to be deterministic and not "floating", i.e. always the
> same regardless of changes in any registry or other upstream. If there
> are changes, they need to be detected and there needs to be a hard
> error. Also, if the code sees things declared in a way they could vary,
> that also needs to be a hard error.
The purpose of the lock file is to lock the source of a package to be 
reproducible.

>
> Most of the concerns I've seen are about how easy it is to understand
> what is going on behind the scenes. The move of code to OE and
> splitting everything into multiple tasks/stages does do that to some
> extent but it does it in a way which I think is going to create a new
> and different set of problems.
Okay, but please keep in mind that some of my oe patches are reasonable 
independent of the native bitbake fetcher and it is possible to 
integrate the steps from the early class into the fetch task.

> I'm therefore wondering if there is a different way. The changes I'm
> wondering about would be to:
>
> a) embrace the single SRC_URI entry
>
> b) require a checksum of the internal "URL list" that is included
> in SRC_URI, much in the same way that we have checksums of tarballs.
The list isn't fix because it depends on the configured package manager 
proxy or registry. We have to remove this feature. But the user could 
use a PREMIRROR to redirect the upstream proxy to its private proxy.

> For better or worse, we have low trust in the underlying tools to get
> this right (they are getting better).

We don't need to trust the tools. We parse the lock file and enrich it 
with fix values. The resolve is deterministic. The output only depends 
on the resolve function, variable values and lock file content.

> c) if the checksum doesn't match, we know something went wrong and
> error

Can you please elaborate this point. We already check the integrity of 
the lock file and we have deterministically resolve the SRC_URIs.

> d) require the new modules to write the URL list into a known location
> as part of unpack
Why is this needed? The generated SRC_URIs could be resolved via 
fetcher.expanded_urldata().

> e) add the ability to add custom hooks in the fetch process to handle
> the cases of needing to alter the flow for patching the components list
I'm afraid this will be complicated since PATCH is applied in S and not 
in UNPACKDIR.

> f) create new tools that allow the fetcher to be stepped through and
> for example partially run, or run with clear debug output showing what
> was happening at each stage (show the list of components?). This may be
> standalone tools, maybe a devtool module, I don't know. We may want to
> make the fetch/unpack logs more useful in general as right now you
> don't get much useful data about what it is doing.
>
>
> If we do those things, where does that get us? How much buy in do our
> different stakeholders have?
>
> FWIW I am leaning towards having this code in the bitbake fetcher as a
> first class citizen as to do otherwise is going to create layers of
> abstraction and we probably have enough of those already.

The advantage is that the SRC_URI still contains the dependencies if you 
expand the urldata. On the other side the integration of the patches in 
the fetcher sounds complicated.

[-- Attachment #2: Type: text/html, Size: 7403 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Richard Purdie]

On Thu, 2025-02-13 at 17:33 +0100, Stefan Herbrechtsmeier wrote:
>  Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
> 
> > Most of the concerns I've seen are about how easy it is to understand
> > what is going on behind the scenes. The move of code to OE and
> > splitting everything into multiple tasks/stages does do that to some
> > extent but it does it in a way which I think is going to create a new
> > and different set of problems.
> >  
> > 
> > 
> > 
>  Okay, but please keep in mind that some of my oe patches are
> reasonable independent of the native bitbake fetcher and it is
> possible to integrate the steps from the early class into the fetch
> task.

I appreciate that and I appreciate the desire to push things into OE as
it appears easier. It can lead to much looser APIs and less structured
code and I'm wary of it here as we create a two layered system which I
think will be harder to understand (and hence harder to debug and use).

  
> > I'm therefore wondering if there is a different way. The changes
> > I'm
> > wondering about would be to:
> > 
> > a) embrace the single SRC_URI entry
> > 
> > b) require a checksum of the internal "URL list" that is included
> > in SRC_URI, much in the same way that we have checksums of
> > tarballs.
> >  
>  The list isn't fix because it depends on the configured package
> manager proxy or registry. We have to remove this feature. But the
> user could use a PREMIRROR to redirect the upstream proxy to its
> private proxy.

If that is true we have a huge problem.

By list I mean a list of something like (component, version) pairs
where component uniquely identifies the component and version is a
specific verifiable version of that component. If we can create that
list, we can checksum it and use it as above. If we can't create that
list, we have no idea what is in our builds and we may as well give up
as it isn't reproducible.

> > For better or worse, we have low trust in the underlying tools to
> > get this right (they are getting better).
> >  
>  
> We don't need to trust the tools. We parse the lock file and enrich
> it with fix values. The resolve is deterministic. The output only
> depends on the resolve function, variable values and lock file
> content.

This assumes the "resolve" always does the same thing. I'm afraid
experience shows these can have issues. I'd much rather we have some
kind of backup in the system which tells whether we did get the same
resolution which is what this checksum represents.

> > c) if the checksum doesn't match, we know something went wrong and
> > error
> >  
> Can you please elaborate this point. We already check the integrity
> of the lock file and we have deterministically resolve the SRC_URIs.

See above. I'd like to know that the list of components and versions we
resolve everything to matches what we expect it to look like.

> > d) require the new modules to write the URL list into a known
> > location as part of unpack
> > 
>  Why is this needed? The generated SRC_URIs could be resolved via
> fetcher.expanded_urldata().

If someone is trying to debug what the code did or resolved things too,
suggesting they run python functions to work it out will be a poor user
experience. If on the other hand they know the result is always stored
in WORKDIR/xyz/ABC, the know where and what to look at.

The user experience of using this code will make or break it's
adoption.

> > e) add the ability to add custom hooks in the fetch process to
> > handle
> > the cases of needing to alter the flow for patching the components
> > list
> >  
>  I'm afraid this will be complicated since PATCH is applied in S and
> not in UNPACKDIR.

Then we should work out how to handle that. We could allow the recipes
to specify the top level dir to apply patches from for example?


> > f) create new tools that allow the fetcher to be stepped through
> > and
> > for example partially run, or run with clear debug output showing
> > what
> > was happening at each stage (show the list of components?). This
> > may be
> > standalone tools, maybe a devtool module, I don't know. We may want
> > to
> > make the fetch/unpack logs more useful in general as right now you
> > don't get much useful data about what it is doing.
> > 
> > 
> > If we do those things, where does that get us? How much buy in do
> > our
> > different stakeholders have?
> > 
> > FWIW I am leaning towards having this code in the bitbake fetcher
> > as a
> > first class citizen as to do otherwise is going to create layers of
> > abstraction and we probably have enough of those already.
> >  
>  
> The advantage is that the SRC_URI still contains the dependencies if
> you expand the urldata. On the other side the integration of the
> patches in the fetcher sounds complicated.

Patches would stay where they are in the system in do_patch and use the
code in OE-Core. I'm just thinking we could add some hooks in the fetch
process to allow adjustment of things like the resolved component list.
It doesn't have to be a patch, it could be a function passed data.

Cheers,

Richard

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 9409 bytes --]

Am 13.02.2025 um 21:32 schrieb Richard Purdie:
> On Thu, 2025-02-13 at 17:33 +0100, Stefan Herbrechtsmeier wrote:
>>   Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
>>
>>> Most of the concerns I've seen are about how easy it is to understand
>>> what is going on behind the scenes. The move of code to OE and
>>> splitting everything into multiple tasks/stages does do that to some
>>> extent but it does it in a way which I think is going to create a new
>>> and different set of problems.
>>>   
>>>
>>>
>>>
>>   Okay, but please keep in mind that some of my oe patches are
>> reasonable independent of the native bitbake fetcher and it is
>> possible to integrate the steps from the early class into the fetch
>> task.
> I appreciate that and I appreciate the desire to push things into OE as
> it appears easier. It can lead to much looser APIs and less structured
> code and I'm wary of it here as we create a two layered system which I
> think will be harder to understand (and hence harder to debug and use).
>
>    
>>> I'm therefore wondering if there is a different way. The changes
>>> I'm
>>> wondering about would be to:
>>>
>>> a) embrace the single SRC_URI entry
>>>
>>> b) require a checksum of the internal "URL list" that is included
>>> in SRC_URI, much in the same way that we have checksums of
>>> tarballs.
>>>   
>>   The list isn't fix because it depends on the configured package
>> manager proxy or registry. We have to remove this feature. But the
>> user could use a PREMIRROR to redirect the upstream proxy to its
>> private proxy.
> If that is true we have a huge problem.
>
> By list I mean a list of something like (component, version) pairs
> where component uniquely identifies the component and version is a
> specific verifiable version of that component. If we can create that
> list, we can checksum it and use it as above. If we can't create that
> list, we have no idea what is in our builds and we may as well give up
> as it isn't reproducible.

What does reproducible mean? The lock file ensure that you always use 
the same dependencies.

go.sum:
cloud.google.com/go v0.110.0 h1:Zc8gqp3+a9/Eyph2KDmcGaPtbKRIoqq4YTlL4NMD0Ys=

Cargo.lock:
[[package]]
name = "addr2line"
version = "0.24.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = 
"dfbe277e56a376000877090da837660b4427aad530e3028d44e0bffe4f89a1c1"
dependencies = [
  "gimli",
]

package-lock.json:
     "node_modules/@adobe/css-tools": {
       "version": "4.3.3",
       "resolved": 
"https://registry.npmjs.org/@adobe/css-tools/-/css-tools-4.3.3.tgz",
       "integrity": 
"sha512-rE0Pygv0sEZ4vBWHlAgJLGDU7Pm8xoO6p3wsEceb7GYAjScrOHpEo8KK/eVkAcnSM+slAEtXjA2JpdjLp4fJQQ==",
       "dev": true
     },

The URL could be extracted from the lock file or could be generated 
based on the name and version or git revision. The integrity of the 
dependencies is ensured via a checksum or git revision. This means the 
same lock file always generate the same downloads and is fully 
reproducible. We only need to ensure the integrity of the lock file. 
This is already ensure by the existing fetchers.

At the moment the download URL could be manipulated by an variable 
because it is common to change the registry / proxy / server. The 
"https://registry.npmjs.org" or 
"registry+https://github.com/rust-lang/crates.io-index" are placeholders 
and could be replaced with a local registry / proxy / server. Additional 
the downloads are unpacked into a subfolder of the project. Therefore 
the SRC_URI parameters depends on S.

I already expand the SRC_URIs with a name and version (pn and pv) to 
enrich the SBOM with the name and version of the dependencies. Because 
the name parameter is already in use and the recipe could use a 
dependency in different versions we need to append the version to the 
name. Either we remove the version from the name to receive the real 
name, use an other parameter for the name or use the name and version as 
variable flag instead of the name alone.

It is already possible to create a list of the dependencies.

>>> For better or worse, we have low trust in the underlying tools to
>>> get this right (they are getting better).
>>>   
>>   
>> We don't need to trust the tools. We parse the lock file and enrich
>> it with fix values. The resolve is deterministic. The output only
>> depends on the resolve function, variable values and lock file
>> content.
> This assumes the "resolve" always does the same thing. I'm afraid
> experience shows these can have issues. I'd much rather we have some
> kind of backup in the system which tells whether we did get the same
> resolution which is what this checksum represents.

This sounds like a problem with the test coverage. But this should be 
solved by a high test coverage for the resolve function.

>>> c) if the checksum doesn't match, we know something went wrong and
>>> error
>>>   
>> Can you please elaborate this point. We already check the integrity
>> of the lock file and we have deterministically resolve the SRC_URIs.
> See above. I'd like to know that the list of components and versions we
> resolve everything to matches what we expect it to look like.

We can extract the dependency name and version from the SRC_URIs to hash 
it but in case of npm the destination folder could also influence the build.

>>> d) require the new modules to write the URL list into a known
>>> location as part of unpack
>>>
>>   Why is this needed? The generated SRC_URIs could be resolved via
>> fetcher.expanded_urldata().
> If someone is trying to debug what the code did or resolved things too,
> suggesting they run python functions to work it out will be a poor user
> experience. If on the other hand they know the result is always stored
> in WORKDIR/xyz/ABC, the know where and what to look at.
>
> The user experience of using this code will make or break it's
> adoption.
Can we instead extend the lock file in the download folder which the 
SRC_URIs? This allows use to use the file as cache and bypass the resolve.

>>> e) add the ability to add custom hooks in the fetch process to
>>> handle
>>> the cases of needing to alter the flow for patching the components
>>> list
>>>   
>>   I'm afraid this will be complicated since PATCH is applied in S and
>> not in UNPACKDIR.
> Then we should work out how to handle that. We could allow the recipes
> to specify the top level dir to apply patches from for example?
Is it okay to use variables like S inside the fetcher or should we pass 
everything via SRC_URI parameter to the fetcher?

The fetcher only knows its SRC_URI. This means we have to add the 
patches to the SRC_URI parameters or pass it via additional function 
parameter to the download function.

>>> f) create new tools that allow the fetcher to be stepped through
>>> and
>>> for example partially run, or run with clear debug output showing
>>> what
>>> was happening at each stage (show the list of components?). This
>>> may be
>>> standalone tools, maybe a devtool module, I don't know. We may want
>>> to
>>> make the fetch/unpack logs more useful in general as right now you
>>> don't get much useful data about what it is doing.
>>>
>>>
>>> If we do those things, where does that get us? How much buy in do
>>> our
>>> different stakeholders have?
>>>
>>> FWIW I am leaning towards having this code in the bitbake fetcher
>>> as a
>>> first class citizen as to do otherwise is going to create layers of
>>> abstraction and we probably have enough of those already.
>>>   
>>   
>> The advantage is that the SRC_URI still contains the dependencies if
>> you expand the urldata. On the other side the integration of the
>> patches in the fetcher sounds complicated.
> Patches would stay where they are in the system in do_patch and use the
> code in OE-Core. I'm just thinking we could add some hooks in the fetch
> process to allow adjustment of things like the resolved component list.
> It doesn't have to be a patch, it could be a function passed data.
Please take a look at the following patch to fix a security issue in 
librsvg:
https://gitlab.gnome.org/GNOME/librsvg/-/commit/aaaa6b68b024b2adbfdf5f8493dfce1f60e5e331

How should the integration into the recipe should look like and how long 
does it take to integrate the changes into the recipe? In case of my OE 
series you could simply apply the patch and mark it as early:

SRC_URI += 
"file://0001-update-url-crate-to-get-an-updated-idna-rustsec-2024.patch;early=1"

We need support for plain patches. Otherwise it is impossible to back 
port patches to fix a security issue or to reuse common tools.

The advantage of the OE based implementation is the possibility to patch 
the lock file. The bitbake based implementation only support a complete 
local lock file to manipulate the dependencies.

Any OE specific solution (.inc or hook) is useless because of the 
complexity of the dependency update. The lock file is a temporary flat 
view onto a dependency tree with inter dependencies, replacements, 
constraints, compatibilities and other meta data. A change without this 
information could lead to anything. It's like the list of git 
repositories of recursive git submodules. It isn't useful to change the 
revision of an arbitrary git repository in the list because the change 
could recursively influence other git submodules.



[-- Attachment #2: Type: text/html, Size: 13470 bytes --]

Re: 'vendor' fetching discussion cont.

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 5604 bytes --]

I did some replies to the other threads before seeing this, we can feel
free to let those other threads go unanswered, to unify things here.

On Thu, Feb 13, 2025 at 5:43 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> I've pulled this to a separate email/thread since I'd like to take a
> slight step back and put some different ideas into the mix as well as
> explain where my own thoughts are right now. I'm also doing this so
> that we can focus the discussion and give others a place to catch up
> from. For that reason I may restate some information below.
>
> We have two fundamentally different approaches:
>
> a) a single entry in SRC_URI with magic behind the scenes expanding
> this into a list of dependencies
>
> b) multiple entries in SRC_URI generated with tooling
>
> gitsm is similar to a), partly because it can contain recursive
> references to other submodules so the second approach wouldn't work.
> crates are handled with the b) approach today.
>
> Developers in general are more comfortable with b) since they can more
> easily see what is going on but it is also the harder one since it
> deviates most from the underlying tools and has two sets of data that
> require to be in sync.
>
> Some feel the .inc files enabling b) are effectively machine generated
> and could be removed with the work happening transparently instead,
> simplifying the recipes and commits. This allows easier use of the
> underlying tooling too.
>
> I think firstly, we need to document some key principles. Behind the
> scenes, any given url should expand to a defined list of components and
> that list has to be deterministic and not "floating", i.e. always the
> same regardless of changes in any registry or other upstream. If there
> are changes, they need to be detected and there needs to be a hard
> error. Also, if the code sees things declared in a way they could vary,
> that also needs to be a hard error.
>

And of course be expanded into something that is compatible
with our mirroring, but that was implied, I just wanted to say it :)


> Most of the concerns I've seen are about how easy it is to understand
> what is going on behind the scenes. The move of code to OE and
> splitting everything into multiple tasks/stages does do that to some
> extent but it does it in a way which I think is going to create a new
> and different set of problems.
>
> I'm therefore wondering if there is a different way. The changes I'm
> wondering about would be to:
>
> a) embrace the single SRC_URI entry
>

I still wonder how we'd be able to debug and/or override parts of
the single SRC_URI entry.  Do you consider a lock file or a language
dependency file that could be overwritten from recipe space as
a single SRC_URI entry ? If so, I can get on board with that.

I still prefer the expanded dependencies into some sort of base /
simple fetch format, but a single file that describes all the dependencies
is close enough. As long as there's a way to inspect what the file
was processed into for fetching, then there is some visibility in times
of need.

The remaining question for me is .. how recursive are the dependencies
in the file described on the SRC_URI ? If each line in the single file
is being expanded into multiple different dependencies, then the
visibility into the final list is low, the reproducibility and mirroring of
what
eventually gets fetched need to be guaranteed as well. That of course
isn't different from the issues which could arise with gitsm.


> b) require a checksum of the internal "URL list" that is included
> in SRC_URI, much in the same way that we have checksums of tarballs.
> For better or worse, we have low trust in the underlying tools to get
> this right (they are getting better).
>

This would be a checksum of the fully expanded dependencies of
the SRC_URI entry ?


>
> c) if the checksum doesn't match, we know something went wrong and
> error
>
> d) require the new modules to write the URL list into a known location
> as part of unpack
>

Aha. That answers the question that I had above.


> e) add the ability to add custom hooks in the fetch process to handle
> the cases of needing to alter the flow for patching the components list
>

Or potentially detect the output of d) being somehow supplied and not
do the dependency resolution ?


> f) create new tools that allow the fetcher to be stepped through and
> for example partially run, or run with clear debug output showing what
> was happening at each stage (show the list of components?). This may be
> standalone tools, maybe a devtool module, I don't know. We may want to
> make the fetch/unpack logs more useful in general as right now you
> don't get much useful data about what it is doing.
>
>
> If we do those things, where does that get us? How much buy in do our
> different stakeholders have?
>

I think it is getting closer!

If there's a way to see all of the individual fetches, and change those
fetches,
then it solves most of the issues that I've been using git:// fetches for
in my
go recipes.

Bruce



>
> FWIW I am leaning towards having this code in the bitbake fetcher as a
> first class citizen as to do otherwise is going to create layers of
> abstraction and we probably have enough of those already.
>
> Cheers,
>
> Richard
>
>
>
>
>
>
>
>
>

-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 9198 bytes --]

Re: 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 7104 bytes --]

Am 13.02.2025 um 18:34 schrieb Bruce Ashfield:
> I did some replies to the other threads before seeing this, we can feel
> free to let those other threads go unanswered, to unify things here.
Okay

> On Thu, Feb 13, 2025 at 5:43 AM Richard Purdie 
> <richard.purdie@linuxfoundation.org> wrote:
>
>     I've pulled this to a separate email/thread since I'd like to take a
>     slight step back and put some different ideas into the mix as well as
>     explain where my own thoughts are right now. I'm also doing this so
>     that we can focus the discussion and give others a place to catch up
>     from. For that reason I may restate some information below.
>
>     We have two fundamentally different approaches:
>
>     a) a single entry in SRC_URI with magic behind the scenes expanding
>     this into a list of dependencies
>
>     b) multiple entries in SRC_URI generated with tooling
>
>     gitsm is similar to a), partly because it can contain recursive
>     references to other submodules so the second approach wouldn't work.
>     crates are handled with the b) approach today.
>
>     Developers in general are more comfortable with b) since they can more
>     easily see what is going on but it is also the harder one since it
>     deviates most from the underlying tools and has two sets of data that
>     require to be in sync.
>
>     Some feel the .inc files enabling b) are effectively machine generated
>     and could be removed with the work happening transparently instead,
>     simplifying the recipes and commits. This allows easier use of the
>     underlying tooling too.
>
>     I think firstly, we need to document some key principles. Behind the
>     scenes, any given url should expand to a defined list of
>     components and
>     that list has to be deterministic and not "floating", i.e. always the
>     same regardless of changes in any registry or other upstream. If there
>     are changes, they need to be detected and there needs to be a hard
>     error. Also, if the code sees things declared in a way they could
>     vary,
>     that also needs to be a hard error.
>
>
> And of course be expanded into something that is compatible
> with our mirroring, but that was implied, I just wanted to say it :)
Therefore we need to fix the mirror regex. Otherwise we have to mimic 
the upstream layout in the download directory.

>
>     Most of the concerns I've seen are about how easy it is to understand
>     what is going on behind the scenes. The move of code to OE and
>     splitting everything into multiple tasks/stages does do that to some
>     extent but it does it in a way which I think is going to create a new
>     and different set of problems.
>
>     I'm therefore wondering if there is a different way. The changes I'm
>     wondering about would be to:
>
>     a) embrace the single SRC_URI entry
>
>
> I still wonder how we'd be able to debug and/or override parts of
> the single SRC_URI entry.  Do you consider a lock file or a language
> dependency file that could be overwritten from recipe space as
> a single SRC_URI entry ? If so, I can get on board with that.
>
> I still prefer the expanded dependencies into some sort of base /
> simple fetch format, but a single file that describes all the dependencies
> is close enough. As long as there's a way to inspect what the file
> was processed into for fetching, then there is some visibility in times
> of need.
Do you mean a Cargo.lock, go.sum and package-lock.json file or a 
proprietary file?

We could save the generated SRC_URIs and a hash of the lock file in a 
file in the download directory. The file could be used for inspection 
and as cache to avoid a regeneration.

> The remaining question for me is .. how recursive are the dependencies
> in the file described on the SRC_URI ?
Go, Rust and NPM already resolve the recursive dependencies in its lock 
file. Only gitsm need to resolve the recursive dependencies.

> If each line in the single file
> is being expanded into multiple different dependencies, then the
> visibility into the final list is low,

We have the fetcher.expanded_urldata function to receive the expanded 
SRC_URI list. Maybe we could make it easy accessible.

> the reproducibility
The generated SRC_URIs only depends on the lock file and some variables. 
We could reduce the variables if we depend on the PREMIRROR to configure 
a local proxy or registry.

> and mirroring of what
> eventually gets fetched need to be guaranteed as well. That of course
> isn't different from the issues which could arise with gitsm.
My last implementation are based on the wget and git fetcher. The 
mirroring will work as soon as we fix it.

>     b) require a checksum of the internal "URL list" that is included
>     in SRC_URI, much in the same way that we have checksums of tarballs.
>     For better or worse, we have low trust in the underlying tools to get
>     this right (they are getting better).
>
>
> This would be a checksum of the fully expanded dependencies of
> the SRC_URI entry ?
>
>
>     c) if the checksum doesn't match, we know something went wrong and
>     error
>
>     d) require the new modules to write the URL list into a known location
>     as part of unpack
>
>
> Aha. That answers the question that I had above.
>
>
>     e) add the ability to add custom hooks in the fetch process to handle
>     the cases of needing to alter the flow for patching the components
>     list
>
>
> Or potentially detect the output of d) being somehow supplied and not
> do the dependency resolution ?
I would focus on patch files for the lock file. It is always possible to 
remove the SRC_URI line and replace it with the desired dependency list. 
The list could be extracted from the cache file in the download 
directory or could be received via the expanded_urldata function. The 
function already returns the git

>     f) create new tools that allow the fetcher to be stepped through and
>     for example partially run, or run with clear debug output showing what
>     was happening at each stage (show the list of components?). This
>     may be
>     standalone tools, maybe a devtool module, I don't know. We may want to
>     make the fetch/unpack logs more useful in general as right now you
>     don't get much useful data about what it is doing.
>
>
>     If we do those things, where does that get us? How much buy in do our
>     different stakeholders have?
>
>
> I think it is getting closer!
>
> If there's a way to see all of the individual fetches, and change 
> those fetches,
> then it solves most of the issues that I've been using git:// fetches 
> for in my
> go recipes.
>
> Bruce
>
>
>     FWIW I am leaning towards having this code in the bitbake fetcher as a
>     first class citizen as to do otherwise is going to create layers of
>     abstraction and we probably have enough of those already.
>
>     Cheers,
>
>     Richard
>
>
>
>
>
>
>
>
>
>
> -- 
> - Thou shalt not follow the NULL pointer, for chaos and madness await 
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>

[-- Attachment #2: Type: text/html, Size: 16200 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 2266 bytes --]

Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
> e) add the ability to add custom hooks in the fetch process to handle
> the cases of needing to alter the flow for patching the components list

I fear this isn't easy possible and we have a design problem. Bitbake 
and oe-core assume that a patch doesn't influence the fetch and that the 
SRC_URI contains all fetched sources. Many code use the Fetch class or 
the SRC_URI direct and doesn't expect recursive implicit URIs of gitsm. 
In reality the SRC_URI describe the given URIs only and doesn't contain 
the additional implicit URIs. The task flow applies patches after the 
unpack and prevent patches to the gitsm. In reality a patch could 
influence the implicit URIs and isn't independent. I would recommend to 
keep the patching outside of bitbake and instead handle the implicit 
URIs inside oe-core. We could write the implicit URIs into files in the 
work directory and migrate the scattered direct users of the SRC_URI / 
Fetch class to a common oe-core function. This functions returns the 
given URIs and optional any additional implicit URIs. This allows the 
usage of the task dependencies to ensure that the implicit URIs are 
resolved before use. The package manager lock file needs a single unpack 
and patch step whereas the gitsm needs a recursive unpack and patch of 
every successive gitsm. This makes it impossible to use additional 
tasks. We could add the resolve of the implicit URIs to the fetch task 
to support recursive resolve (gitsm).

1. download SRC_URIs 2. exit if no URI is marked as unresolved 3. unpack 
unresolved URIs 4. apply associated patches 5. resolve URIs 6. goto 2

This will eliminate the need for the partial implicit URI and gitsm 
support in bitbake and makes the fetcher code simpler. The patching 
could remain in oe-core and patches could be applied to gitsm and 
package manager lock files. Additionally we remove code duplication and 
simplify future changes to the SRC_URI determination. It will add an 
indirection for the SRC_URI but the current code already shows that this 
makes problems. Only the archiver class uses the expanded_urldata 
function and all other classes including the spdx class ignore the 
implicit SRC_URIs.

[-- Attachment #2: Type: text/html, Size: 2970 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Richard Purdie]

On Mon, 2025-02-17 at 12:00 +0100, Stefan Herbrechtsmeier wrote:
>  Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
>  
> > e) add the ability to add custom hooks in the fetch process to
> > handle
> > the cases of needing to alter the flow for patching the components
> > list
> >  
> I fear this isn't easy possible and we have a design problem. Bitbake
> and oe-core assume that a patch doesn't influence the fetch and that
> the SRC_URI contains all fetched sources. Many code use the Fetch
> class or the SRC_URI direct and doesn't expect recursive implicit
> URIs of gitsm. In reality the SRC_URI describe the given URIs only
> and doesn't contain the additional implicit URIs. The task flow
> applies patches after the unpack and prevent patches to the gitsm. In
> reality a patch could influence the implicit URIs and isn't
> independent.
> 
> I would recommend to keep the patching outside of bitbake and instead
> handle the implicit URIs inside oe-core. We could write the implicit
> URIs into files in the work directory and migrate the scattered
> direct users of the SRC_URI / Fetch class to a common oe-core
> function. This functions returns the given URIs and optional any
> additional implicit URIs. This allows the usage of the task
> dependencies to ensure that the implicit URIs are resolved before
> use.
> 
> The package manager lock file needs a single unpack and patch step
> whereas the gitsm needs a recursive unpack and patch of every
> successive gitsm. This makes it impossible to use additional tasks.
> We could add the resolve of the implicit URIs to the fetch task to
> support recursive resolve (gitsm).
>  
> 1. download SRC_URIs
> 2. exit if no URI is marked as unresolved
> 3. unpack unresolved URIs
> 4. apply associated patches
> 5. resolve URIs
> 6. goto 2 
>  
> This will eliminate the need for the partial implicit URI and gitsm
> support in bitbake and makes the fetcher code simpler. The patching
> could remain in oe-core and patches could be applied to gitsm and
> package manager lock files. Additionally we remove code duplication
> and simplify future changes to the SRC_URI determination.
> 
> It will add an indirection for the SRC_URI but the current code
> already shows that this makes problems. Only the archiver class uses
> the expanded_urldata function and all other classes including the
> spdx class ignore the implicit SRC_URIs.

I'm a bit worried we're getting caught up by the terminology. We don't
necessarily have to "patch" the component list so much as allow
something like a function to hook and adjust things. That then removes
the need for it to be a specific task.

I'm also not sure that drawing gitsm into this is a great idea when it
is effectively already working  quite well. I appreciate the desire to
have neat abstractions applying accross everything equally but I'm not
sure we can achieve that with the level of usability we need and I'd
rather improve usability at the cost of the inclusion of gitsm, which
is the most functional fetcher we currently have in this group.

I will repeat again that I do strongly feel that a strong API in
bitbake is going to lead to an overall better design that something
bolted onto the fetcher in OE-Core.

Cheers,

Richard

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 6585 bytes --]

Am 17.02.2025 um 13:43 schrieb Richard Purdie:
> On Mon, 2025-02-17 at 12:00 +0100, Stefan Herbrechtsmeier wrote:
>>   Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
>>   
>>> e) add the ability to add custom hooks in the fetch process to
>>> handle
>>> the cases of needing to alter the flow for patching the components
>>> list
>>>   
>> I fear this isn't easy possible and we have a design problem. Bitbake
>> and oe-core assume that a patch doesn't influence the fetch and that
>> the SRC_URI contains all fetched sources. Many code use the Fetch
>> class or the SRC_URI direct and doesn't expect recursive implicit
>> URIs of gitsm. In reality the SRC_URI describe the given URIs only
>> and doesn't contain the additional implicit URIs. The task flow
>> applies patches after the unpack and prevent patches to the gitsm. In
>> reality a patch could influence the implicit URIs and isn't
>> independent.
>>
>> I would recommend to keep the patching outside of bitbake and instead
>> handle the implicit URIs inside oe-core. We could write the implicit
>> URIs into files in the work directory and migrate the scattered
>> direct users of the SRC_URI / Fetch class to a common oe-core
>> function. This functions returns the given URIs and optional any
>> additional implicit URIs. This allows the usage of the task
>> dependencies to ensure that the implicit URIs are resolved before
>> use.
>>
>> The package manager lock file needs a single unpack and patch step
>> whereas the gitsm needs a recursive unpack and patch of every
>> successive gitsm. This makes it impossible to use additional tasks.
>> We could add the resolve of the implicit URIs to the fetch task to
>> support recursive resolve (gitsm).
>>   
>> 1. download SRC_URIs
>> 2. exit if no URI is marked as unresolved
>> 3. unpack unresolved URIs
>> 4. apply associated patches
>> 5. resolve URIs
>> 6. goto 2
>>   
>> This will eliminate the need for the partial implicit URI and gitsm
>> support in bitbake and makes the fetcher code simpler. The patching
>> could remain in oe-core and patches could be applied to gitsm and
>> package manager lock files. Additionally we remove code duplication
>> and simplify future changes to the SRC_URI determination.
>>
>> It will add an indirection for the SRC_URI but the current code
>> already shows that this makes problems. Only the archiver class uses
>> the expanded_urldata function and all other classes including the
>> spdx class ignore the implicit SRC_URIs.
> I'm a bit worried we're getting caught up by the terminology. We don't
> necessarily have to "patch" the component list so much as allow
> something like a function to hook and adjust things. That then removes
> the need for it to be a specific task.
We can use a common patch without a do_patch task. We could even call 
the patch code from the do_patch without an extra task. I propose to 
call the code from the do_patch task in the do_fetch task. It doesn't 
matter if we use a hook or task. We should minimize the work for the 
user. The common solution to manipulate a package manager lock file is a 
patch file and we have the code to handle patch files.

> I'm also not sure that drawing gitsm into this is a great idea when it
> is effectively already working  quite well.

To my knowledge it isn't possible to patch a sub module revision. Is it 
intended that the SBOM doesn't contain the recursive implicit git 
repositories?

>   I appreciate the desire to
> have neat abstractions applying accross everything equally but I'm not
> sure we can achieve that with the level of usability we need and I'd
> rather improve usability at the cost of the inclusion of gitsm, which
> is the most functional fetcher we currently have in this group.

I include gitsm because it is similar to the package manager and have 
some drawbacks which I solve with my approach. It is the only fetcher 
that parse a file to determine additional URIs. Its integration into 
oe-core is sub-optimal.

> I will repeat again that I do strongly feel that a strong API in
> bitbake is going to lead to an overall better design that something
> bolted onto the fetcher in OE-Core.

What is a fetcher? We have fetchers for protocols (wget, git), wrapper 
around other fetchers (crate, gomod and gomodgit) and wrapper around 
other fetchers which embedded file parser (gitsm, npmsw). Thereby the 
gitsm fetcher fetch a source and parse the embedded file. The npmsw 
fetcher parse a file from the meta layer. The gitsm fetcher recursive 
combines the download and unpack function call in mostly every class 
function.

The wrappers has the disadvantage that they use uncommon URIs which are 
useless for the SBOM. We support PyPI without the need for an extra fetcher.

Is PyPI or crate the desired way to go?

Instead of expand the fetcher I would focus the fetcher on the download 
of provides URIs and keep the file parsing and post unpack processing 
outside of it.

The package manager lock file is a simple file which references to 
package sources. Thereby a main source could contain multiple package 
manager lock files. At the moment we have no fetcher which reference an 
other source as base or combines multiple fetchers.

Based on my two proposed solutions and the work the last month, I have 
the following feasible requirements:
* Support patches for the package manager lock file to simply updates, 
back ports and upsteaming
* Support multiple package manager lock files per main source
* Include the real https or git URI in the SBOM
* Include the dependency name and version in the SBOM
* No manual bitbake command call after a SRC_URI, PV or SRCREV change

The following code is mostly used in oe-core and ignore the implicit URIs:

     src_uri = (d.getVar('SRC_URI') or "").split()
     fetcher = bb.fetch2.Fetch(src_uri, d)
     for url in fetcher.urls:

The following code is used in the archiver class only and includes the 
implicit URIs:

     src_uri = get_src_uris(d)
     fetcher = bb.fetch2.Fetch(src_uri, d)
     for ud in fetcher.expanded_urldata():

My last series enable oe-core to generate implicit URIs and therefore 
replace the usage of the plain SRC_URI with a function:

     fetcher = bb.fetch.Fetch(get_src_uris(d), d)
     for url in fetcher.urls:

The SRC_URI contains the given URIs and the get_src_uri function 
contains the given and implicit URIs. The concept is taken from other 
classes like the spdx classes which use files to exchange data between 
tasks.

I have no idea how this "strong API" should look like.

[-- Attachment #2: Type: text/html, Size: 8414 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Richard Purdie]

FWIW I did try and discuss this in the OE TSC meeting today but it is a
US holiday. There were only two people who showed up.

On Mon, 2025-02-17 at 17:38 +0100, Stefan Herbrechtsmeier wrote:
>  Am 17.02.2025 um 13:43 schrieb Richard Purdie:
>  On Mon, 2025-02-17 at 12:00 +0100, Stefan Herbrechtsmeier wrote:
> > >  Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
> > > > 
> > > > e) add the ability to add custom hooks in the fetch process to
> > > > handle
> > > > the cases of needing to alter the flow for patching the components
> > > > list
> > > >  
> > > I fear this isn't easy possible and we have a design problem. Bitbake
> > > and oe-core assume that a patch doesn't influence the fetch and that
> > > the SRC_URI contains all fetched sources. Many code use the Fetch
> > > class or the SRC_URI direct and doesn't expect recursive implicit
> > > URIs of gitsm. In reality the SRC_URI describe the given URIs only
> > > and doesn't contain the additional implicit URIs. The task flow
> > > applies patches after the unpack and prevent patches to the gitsm. In
> > > reality a patch could influence the implicit URIs and isn't
> > > independent.
> > > 
> > > I would recommend to keep the patching outside of bitbake and instead
> > > handle the implicit URIs inside oe-core. We could write the implicit
> > > URIs into files in the work directory and migrate the scattered
> > > direct users of the SRC_URI / Fetch class to a common oe-core
> > > function. This functions returns the given URIs and optional any
> > > additional implicit URIs. This allows the usage of the task
> > > dependencies to ensure that the implicit URIs are resolved before
> > > use.
> > > 
> > > The package manager lock file needs a single unpack and patch step
> > > whereas the gitsm needs a recursive unpack and patch of every
> > > successive gitsm. This makes it impossible to use additional tasks.
> > > We could add the resolve of the implicit URIs to the fetch task to
> > > support recursive resolve (gitsm).
> > >  
> > > 1. download SRC_URIs
> > > 2. exit if no URI is marked as unresolved
> > > 3. unpack unresolved URIs
> > > 4. apply associated patches
> > > 5. resolve URIs
> > > 6. goto 2 
> > >  
> > > This will eliminate the need for the partial implicit URI and gitsm
> > > support in bitbake and makes the fetcher code simpler. The patching
> > > could remain in oe-core and patches could be applied to gitsm and
> > > package manager lock files. Additionally we remove code duplication
> > > and simplify future changes to the SRC_URI determination.
> > > 
> > > It will add an indirection for the SRC_URI but the current code
> > > already shows that this makes problems. Only the archiver class uses
> > > the expanded_urldata function and all other classes including the
> > > spdx class ignore the implicit SRC_URIs.
> > 
> > I'm a bit worried we're getting caught up by the terminology. We don't
> > necessarily have to "patch" the component list so much as allow
> > something like a function to hook and adjust things. That then removes
> > the need for it to be a specific task.
> > 
>
>  We can use a common patch without a do_patch task. We could even
> call the patch code from the do_patch without an extra task. I
> propose to call the code from the do_patch task in the do_fetch task.
> It doesn't matter if we use a hook or task. We should minimize the
> work for the user. The common solution to manipulate a package
> manager lock file is a patch file and we have the code to handle
> patch files.

One of the challenges is the "patch" code/module lives in OE-Core. That
means the fetcher can't call into it.

I appreciate this means you'll argue the fetcher code should move to
OE, or the vendor code should be in OE, however I am pretty decided
that is a really bad idea. Whether I can convey why, I don't know but I
have tried.

I also very strongly want to maintain our separation of do_fetch,
do_unpack and do_patch.

> > I'm also not sure that drawing gitsm into this is a great idea when it
> > is effectively already working  quite well.
> 
> To my knowledge it isn't possible to patch a sub module revision. Is
> it intended that the SBOM doesn't contain the recursive implicit git
> repositories?

A patch was merged to allow the data about implicit git repos (amongst
other things) to be available:

https://git.yoctoproject.org/poky/commit/bitbake/?id=ef3e46afd910d4b7727d42c4c18b501525c65695

That tracer code was added after a lot of discussion that took up a ton
of my time, then people moved on and it hasn't seem much
development/use :(.

So there were mechanisms added to make the data available.

> >  I appreciate the desire to
> > have neat abstractions applying accross everything equally but I'm not
> > sure we can achieve that with the level of usability we need and I'd
> > rather improve usability at the cost of the inclusion of gitsm, which
> > is the most functional fetcher we currently have in this group.
> 
> I include gitsm because it is similar to the package manager and have
> some drawbacks which I solve with my approach. It is the only fetcher
> that parse a file to determine additional URIs. Its integration into
> oe-core is sub-optimal.

I believe there was a way added to solve the drawback you mention. The
same mechanism could be used for other fetchers too yet we're talking
about much more radical re-designs.

> > I will repeat again that I do strongly feel that a strong API in
> > bitbake is going to lead to an overall better design that something
> > bolted onto the fetcher in OE-Core.

The "fetcher" is an API exposed by Bitbake with the intent of
standardising the fetching of source code. It's aim is to support
objectives such as allowing resilience through source mirrors and
support build reproducibility. There is a document in bitbake which
tries to at least summarise some of that after it was requested it be
written down.

> What is a fetcher? We have fetchers for protocols (wget, git),

These were the original fetchers using the original APIs.

> wrapper around other fetchers (crate, gomod and gomodgit)

These are newer and some work better than others are people have
attempted to git the newer languages into the older framework.

>  and wrapper around other fetchers which embedded file parser (gitsm,
> npmsw).

Of these, gitsm is now relatively successfully integrated and
functional.

>  Thereby the gitsm fetcher fetch a source and parse the embedded
> file. The npmsw fetcher parse a file from the meta layer. The gitsm
> fetcher recursive combines the download and unpack function call in
> mostly every class function.

Sadly there did not seem to be any other way to handle that. I'd
imagine that some of the other fetchers are going to have to work
similarly, unfortunately.


> The wrappers has the disadvantage that they use uncommon URIs which
> are useless for the SBOM. We support PyPI without the need for an
> extra fetcher.

We support pypi in so much as we have a recipe per dependency. That was
not deemed possible for npm or crates. I would disagree that uncommon
URIS are useless for SBOMs. They can be problematic but having the data
there means it is at least listed. There are also ways they can be
extended (e.g. the intercept code I link to above).

> Is PyPI or crate the desired way to go?

Compared to the complexity in some of the patches proposed, I'd
actually say yes.

> Instead of expand the fetcher I would focus the fetcher on the
> download of provides URIs and keep the file parsing and post unpack
> processing outside of it.

The challenge is that once you process it, you need to fetch more
things. Our ecosystem is built on the idea that fetching happens in
do_fetch. If we change that most mirroring scripts break for example.
It is like compiling happening during do_configure or do_install - it
is not what the user expects and that is problematic. I am against
changing behaviour in ways users do not expect.

> The package manager lock file is a simple file which references to
> package sources. Thereby a main source could contain multiple package
> manager lock files. At the moment we have no fetcher which reference
> an other source as base or combines multiple fetchers.

This is the case for good reason. If the lock file references all the
sources, perhaps we should just include it alongside the recipe and be
done with it?

> Based on my two proposed solutions and the work the last month, I
> have the following feasible requirements:
>  * Support patches for the package manager lock file to simply
     updates, back ports and upsteaming
>  * Support multiple package manager lock files per main source
>  * Include the real https or git URI in the SBOM
>  * Include the dependency name and version in the SBOM
>  * No manual bitbake command call after a SRC_URI, PV or SRCREV
> change

I think we may have to relax some of these requirements as we can't do
everything.
 
> The following code is mostly used in oe-core and ignore the implicit
> URIs:
> 
>     src_uri = (d.getVar('SRC_URI') or "").split()
>     fetcher = bb.fetch2.Fetch(src_uri, d)
>     for url in fetcher.urls:
> 
> The following code is used in the archiver class only and includes
> the implicit URIs:
> 
>     src_uri = get_src_uris(d)
>     fetcher = bb.fetch2.Fetch(src_uri, d)
>     for ud in fetcher.expanded_urldata():
 
Ironically, I think this an example of the kind of "strong API" I'm
referring to, where a standard API is exposed by bitbake. In some ways
this was already attempted with the tracer API above where the
expand_urldata call would actually be a call into the tracer API to
obtain the actual urls used.
 
 
> My last series enable oe-core to generate implicit URIs and therefore
> replace the usage of the plain SRC_URI with a function:
> 
>      fetcher = bb.fetch.Fetch(get_src_uris(d), d)
>     for url in fetcher.urls:
>
>  The SRC_URI contains the given URIs and the get_src_uri function
> contains the given and implicit URIs. The concept is taken from other
> classes like the spdx classes which use files to exchange data
> between tasks.

I believe the get_src_uris() functionality should happen within the
fetcher module and not outside of it as a bolt on interface.

I do not know where we go from here. You disagree with me on several
key things such as whether the code should be in bitbake or OE-Core,
whether gitsm should be changed, whether we need to have some kind of
checksum to verify the "lockfile" is correct as originally intended in
the recipe and probably more.

I'm in the position of trying to mediate things (i.e. see the different
viewpoints and bring people together, trying to find common ground) and
yet at the same time, express views as someone who as spent a lot of
years trying to maintain and improve this code.

I have tried to seek help from the people who effectively oversee me as
a maintainer (i.e. the TSC) with limited success. I'm not sure what to
do from here.
 
Regards,

Richard 

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 14626 bytes --]

On Mon, Feb 17, 2025 at 12:29 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> FWIW I did try and discuss this in the OE TSC meeting today but it is a
> US holiday. There were only two people who showed up.
>

Also a Canadian Holiday! I was one of the missing attendees, so I'll follow
up here.


>
> On Mon, 2025-02-17 at 17:38 +0100, Stefan Herbrechtsmeier wrote:
> >  Am 17.02.2025 um 13:43 schrieb Richard Purdie:
> >  On Mon, 2025-02-17 at 12:00 +0100, Stefan Herbrechtsmeier wrote:
> > > >  Am 13.02.2025 um 11:43 schrieb Richard Purdie via
> lists.openembedded.org:
> > > > >
> > > > > e) add the ability to add custom hooks in the fetch process to
> > > > > handle
> > > > > the cases of needing to alter the flow for patching the components
> > > > > list
> > > > >
> > > > I fear this isn't easy possible and we have a design problem. Bitbake
> > > > and oe-core assume that a patch doesn't influence the fetch and that
> > > > the SRC_URI contains all fetched sources. Many code use the Fetch
> > > > class or the SRC_URI direct and doesn't expect recursive implicit
> > > > URIs of gitsm. In reality the SRC_URI describe the given URIs only
> > > > and doesn't contain the additional implicit URIs. The task flow
> > > > applies patches after the unpack and prevent patches to the gitsm. In
> > > > reality a patch could influence the implicit URIs and isn't
> > > > independent.
> > > >
> > > > I would recommend to keep the patching outside of bitbake and instead
> > > > handle the implicit URIs inside oe-core. We could write the implicit
> > > > URIs into files in the work directory and migrate the scattered
> > > > direct users of the SRC_URI / Fetch class to a common oe-core
> > > > function. This functions returns the given URIs and optional any
> > > > additional implicit URIs. This allows the usage of the task
> > > > dependencies to ensure that the implicit URIs are resolved before
> > > > use.
> > > >
> > > > The package manager lock file needs a single unpack and patch step
> > > > whereas the gitsm needs a recursive unpack and patch of every
> > > > successive gitsm. This makes it impossible to use additional tasks.
> > > > We could add the resolve of the implicit URIs to the fetch task to
> > > > support recursive resolve (gitsm).
> > > >
> > > > 1. download SRC_URIs
> > > > 2. exit if no URI is marked as unresolved
> > > > 3. unpack unresolved URIs
> > > > 4. apply associated patches
> > > > 5. resolve URIs
> > > > 6. goto 2
> > > >
> > > > This will eliminate the need for the partial implicit URI and gitsm
> > > > support in bitbake and makes the fetcher code simpler. The patching
> > > > could remain in oe-core and patches could be applied to gitsm and
> > > > package manager lock files. Additionally we remove code duplication
> > > > and simplify future changes to the SRC_URI determination.
> > > >
> > > > It will add an indirection for the SRC_URI but the current code
> > > > already shows that this makes problems. Only the archiver class uses
> > > > the expanded_urldata function and all other classes including the
> > > > spdx class ignore the implicit SRC_URIs.
> > >
> > > I'm a bit worried we're getting caught up by the terminology. We don't
> > > necessarily have to "patch" the component list so much as allow
> > > something like a function to hook and adjust things. That then removes
> > > the need for it to be a specific task.
> > >
> >
> >  We can use a common patch without a do_patch task. We could even
> > call the patch code from the do_patch without an extra task. I
> > propose to call the code from the do_patch task in the do_fetch task.
> > It doesn't matter if we use a hook or task. We should minimize the
> > work for the user. The common solution to manipulate a package
> > manager lock file is a patch file and we have the code to handle
> > patch files.
>
> One of the challenges is the "patch" code/module lives in OE-Core. That
> means the fetcher can't call into it.
>
> I appreciate this means you'll argue the fetcher code should move to
> OE, or the vendor code should be in OE, however I am pretty decided
> that is a really bad idea. Whether I can convey why, I don't know but I
> have tried.
>
> I also very strongly want to maintain our separation of do_fetch,
> do_unpack and do_patch.
>

I also agree with this, which is why I've been suggesting the "drop-in"
lock files. That's a fetch and copy and not a patching.  I ended up
doing a custom patch step for some of the kernel meta data and to
this day it is still on my list to change (and I'll be doing that shortly).


> > > I'm also not sure that drawing gitsm into this is a great idea when it
> > > is effectively already working  quite well.
> >
> > To my knowledge it isn't possible to patch a sub module revision. Is
> > it intended that the SBOM doesn't contain the recursive implicit git
> > repositories?
>
> A patch was merged to allow the data about implicit git repos (amongst
> other things) to be available:
>
>
> https://git.yoctoproject.org/poky/commit/bitbake/?id=ef3e46afd910d4b7727d42c4c18b501525c65695
>
> That tracer code was added after a lot of discussion that took up a ton
> of my time, then people moved on and it hasn't seem much
> development/use :(.
>
> So there were mechanisms added to make the data available.
>
> > >  I appreciate the desire to
> > > have neat abstractions applying accross everything equally but I'm not
> > > sure we can achieve that with the level of usability we need and I'd
> > > rather improve usability at the cost of the inclusion of gitsm, which
> > > is the most functional fetcher we currently have in this group.
> >
> > I include gitsm because it is similar to the package manager and have
> > some drawbacks which I solve with my approach. It is the only fetcher
> > that parse a file to determine additional URIs. Its integration into
> > oe-core is sub-optimal.
>
> I believe there was a way added to solve the drawback you mention. The
> same mechanism could be used for other fetchers too yet we're talking
> about much more radical re-designs.
>
> > > I will repeat again that I do strongly feel that a strong API in
> > > bitbake is going to lead to an overall better design that something
> > > bolted onto the fetcher in OE-Core.
>
> The "fetcher" is an API exposed by Bitbake with the intent of
> standardising the fetching of source code. It's aim is to support
> objectives such as allowing resilience through source mirrors and
> support build reproducibility. There is a document in bitbake which
> tries to at least summarise some of that after it was requested it be
> written down.
>

FWIW. This is the mental model that I've always worked with
during my reviews of the series.


> > What is a fetcher? We have fetchers for protocols (wget, git),
>
> These were the original fetchers using the original APIs.
>
> > wrapper around other fetchers (crate, gomod and gomodgit)
>
> These are newer and some work better than others are people have
> attempted to git the newer languages into the older framework.
>
> >  and wrapper around other fetchers which embedded file parser (gitsm,
> > npmsw).
>
> Of these, gitsm is now relatively successfully integrated and
> functional.
>
> >  Thereby the gitsm fetcher fetch a source and parse the embedded
> > file. The npmsw fetcher parse a file from the meta layer. The gitsm
> > fetcher recursive combines the download and unpack function call in
> > mostly every class function.
>
> Sadly there did not seem to be any other way to handle that. I'd
> imagine that some of the other fetchers are going to have to work
> similarly, unfortunately.
>
>
> > The wrappers has the disadvantage that they use uncommon URIs which
> > are useless for the SBOM. We support PyPI without the need for an
> > extra fetcher.
>
> We support pypi in so much as we have a recipe per dependency. That was
> not deemed possible for npm or crates. I would disagree that uncommon
> URIS are useless for SBOMs. They can be problematic but having the data
> there means it is at least listed. There are also ways they can be
> extended (e.g. the intercept code I link to above).
>
> > Is PyPI or crate the desired way to go?
>
> Compared to the complexity in some of the patches proposed, I'd
> actually say yes.
>

I think I'm following this part correctly. When we say "crate the desired
way to go", I read that as "one dependency fetch by the language
specific 'protocol' (crate:// in this example)".  If so, then I also agree.

It became clear very quickly that packaging go dependencies as
separate recipes wasn't going to work, which I assume is the pypi
reference (that works for languages with "stricter" package release
processes).


>
> > Instead of expand the fetcher I would focus the fetcher on the
> > download of provides URIs and keep the file parsing and post unpack
> > processing outside of it.
>
> The challenge is that once you process it, you need to fetch more
> things. Our ecosystem is built on the idea that fetching happens in
> do_fetch. If we change that most mirroring scripts break for example.
> It is like compiling happening during do_configure or do_install - it
> is not what the user expects and that is problematic. I am against
> changing behaviour in ways users do not expect.
>

Which is how we ended up with the create:// lines in the .inc files
and also how I ended up with all my git:// fetches in my bigger
golang recipes. That we could loop, recurse, do whatever we needed
to fully resolve the dependencies, but when the actual fetch and
build of the recipe happened, that has all been detangled and is
a simple iterative processing of the lines.

For me, that separates my debugging into distinct phases. At one
point I'm scratching my head and looking at the language dependency
files, and fixing any issues there. Later on, if there are any issues my
problems are with mirrors, downloading, etc.

I'd also add a last stage of "getting the files in the position that
the language needs for vendoring". I then debug that as a separate
thing.


> > The package manager lock file is a simple file which references to
> > package sources. Thereby a main source could contain multiple package
> > manager lock files. At the moment we have no fetcher which reference
> > an other source as base or combines multiple fetchers.
>
> This is the case for good reason. If the lock file references all the
> sources, perhaps we should just include it alongside the recipe and be
> done with it?
>

I'm definitely ok with this as well, as long as it is complete and is
fully expanded, we don't need the SRC_URIs explicitly expanded in
the recipes. I may still prefer the fully expanded SRC_URIs, but I
also see a route to add that later in my own recipes as needed.


> > Based on my two proposed solutions and the work the last month, I
> > have the following feasible requirements:
> >  * Support patches for the package manager lock file to simply
>      updates, back ports and upsteaming
> >  * Support multiple package manager lock files per main source
> >  * Include the real https or git URI in the SBOM
> >  * Include the dependency name and version in the SBOM
> >  * No manual bitbake command call after a SRC_URI, PV or SRCREV
> > change
>
> I think we may have to relax some of these requirements as we can't do
> everything.
>

Avoiding both some bikeshedding and feature creep! :)

Richard: Which ones would you say don't need to be in something initial ?

To me, I'd say the "multiple package manager files" or the "no manual
bitbake call after"  and "support for patching the lockfile".


> > The following code is mostly used in oe-core and ignore the implicit
> > URIs:
> >
> >     src_uri = (d.getVar('SRC_URI') or "").split()
> >     fetcher = bb.fetch2.Fetch(src_uri, d)
> >     for url in fetcher.urls:
> >
> > The following code is used in the archiver class only and includes
> > the implicit URIs:
> >
> >     src_uri = get_src_uris(d)
> >     fetcher = bb.fetch2.Fetch(src_uri, d)
> >     for ud in fetcher.expanded_urldata():
>
> Ironically, I think this an example of the kind of "strong API" I'm
> referring to, where a standard API is exposed by bitbake. In some ways
> this was already attempted with the tracer API above where the
> expand_urldata call would actually be a call into the tracer API to
> obtain the actual urls used.
>
>
> > My last series enable oe-core to generate implicit URIs and therefore
> > replace the usage of the plain SRC_URI with a function:
> >
> >      fetcher = bb.fetch.Fetch(get_src_uris(d), d)
> >     for url in fetcher.urls:
> >
> >  The SRC_URI contains the given URIs and the get_src_uri function
> > contains the given and implicit URIs. The concept is taken from other
> > classes like the spdx classes which use files to exchange data
> > between tasks.
>
> I believe the get_src_uris() functionality should happen within the
> fetcher module and not outside of it as a bolt on interface.
>
> I do not know where we go from here. You disagree with me on several
> key things such as whether the code should be in bitbake or OE-Core,
> whether gitsm should be changed, whether we need to have some kind of
> checksum to verify the "lockfile" is correct as originally intended in
> the recipe and probably more.
>

As for where something belongs vs another, let's just say I have
no experience to distrust Richard's experience in these areas. I've
only rarely had to delve deep enough to even know some of the
pitfalls. But I do strongly agree that we need to keep the phases
separate with some sort of clear handoff (via files, but not necessarily
files in recipes) between them.

Cheers,

Bruce


> I'm in the position of trying to mediate things (i.e. see the different
> viewpoints and bring people together, trying to find common ground) and
> yet at the same time, express views as someone who as spent a lot of
> years trying to maintain and improve this code.
>
> I have tried to seek help from the people who effectively oversee me as
> a maintainer (i.e. the TSC) with limited success. I'm not sure what to
> do from here.
>
> Regards,
>
> Richard
>
>

-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 21101 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 22636 bytes --]

Am 17.02.2025 um 18:29 schrieb Richard Purdie:
> FWIW I did try and discuss this in the OE TSC meeting today but it is a
> US holiday. There were only two people who showed up.

I answer your points below but does it make sense to create a RFC to 
show how an alternative source URI management in oe-core could look 
like. I could demonstrate my approach and you could perhaps illustrate 
your concerns in the code.

> On Mon, 2025-02-17 at 17:38 +0100, Stefan Herbrechtsmeier wrote:
>>   Am 17.02.2025 um 13:43 schrieb Richard Purdie:
>>   On Mon, 2025-02-17 at 12:00 +0100, Stefan Herbrechtsmeier wrote:
>>>>   Am 13.02.2025 um 11:43 schrieb Richard Purdie via lists.openembedded.org:
>>>>> e) add the ability to add custom hooks in the fetch process to
>>>>> handle
>>>>> the cases of needing to alter the flow for patching the components
>>>>> list
>>>>>   
>>>> I fear this isn't easy possible and we have a design problem. Bitbake
>>>> and oe-core assume that a patch doesn't influence the fetch and that
>>>> the SRC_URI contains all fetched sources. Many code use the Fetch
>>>> class or the SRC_URI direct and doesn't expect recursive implicit
>>>> URIs of gitsm. In reality the SRC_URI describe the given URIs only
>>>> and doesn't contain the additional implicit URIs. The task flow
>>>> applies patches after the unpack and prevent patches to the gitsm. In
>>>> reality a patch could influence the implicit URIs and isn't
>>>> independent.
>>>>
>>>> I would recommend to keep the patching outside of bitbake and instead
>>>> handle the implicit URIs inside oe-core. We could write the implicit
>>>> URIs into files in the work directory and migrate the scattered
>>>> direct users of the SRC_URI / Fetch class to a common oe-core
>>>> function. This functions returns the given URIs and optional any
>>>> additional implicit URIs. This allows the usage of the task
>>>> dependencies to ensure that the implicit URIs are resolved before
>>>> use.
>>>>
>>>> The package manager lock file needs a single unpack and patch step
>>>> whereas the gitsm needs a recursive unpack and patch of every
>>>> successive gitsm. This makes it impossible to use additional tasks.
>>>> We could add the resolve of the implicit URIs to the fetch task to
>>>> support recursive resolve (gitsm).
>>>>   
>>>> 1. download SRC_URIs
>>>> 2. exit if no URI is marked as unresolved
>>>> 3. unpack unresolved URIs
>>>> 4. apply associated patches
>>>> 5. resolve URIs
>>>> 6. goto 2
>>>>   
>>>> This will eliminate the need for the partial implicit URI and gitsm
>>>> support in bitbake and makes the fetcher code simpler. The patching
>>>> could remain in oe-core and patches could be applied to gitsm and
>>>> package manager lock files. Additionally we remove code duplication
>>>> and simplify future changes to the SRC_URI determination.
>>>>
>>>> It will add an indirection for the SRC_URI but the current code
>>>> already shows that this makes problems. Only the archiver class uses
>>>> the expanded_urldata function and all other classes including the
>>>> spdx class ignore the implicit SRC_URIs.
>>> I'm a bit worried we're getting caught up by the terminology. We don't
>>> necessarily have to "patch" the component list so much as allow
>>> something like a function to hook and adjust things. That then removes
>>> the need for it to be a specific task.
>>>
>>   We can use a common patch without a do_patch task. We could even
>> call the patch code from the do_patch without an extra task. I
>> propose to call the code from the do_patch task in the do_fetch task.
>> It doesn't matter if we use a hook or task. We should minimize the
>> work for the user. The common solution to manipulate a package
>> manager lock file is a patch file and we have the code to handle
>> patch files.
> One of the challenges is the "patch" code/module lives in OE-Core. That
> means the fetcher can't call into it.

That's the reason I would restrict the fetcher to the plain fetch.

> I appreciate this means you'll argue the fetcher code should move to
> OE, or the vendor code should be in OE, however I am pretty decided
> that is a really bad idea. Whether I can convey why, I don't know but I
> have tried.

I argue that the fetcher should focus on the download and unpack of the 
provides URIs. I don't understand why the parsing of a source file and 
the preparation of a cache or vendor folder belongs to the fetcher. The 
fetcher downloads and unpack the provide URIs. The fetcher doesn't care 
if the provided URIs are static defined inside the recipe or implicit 
generated from a source file.

At the moment a lot of code inside oe-core assume that the SRC_URI 
contains all source URIs. This is already wrong because gitsm expand the 
SRC_URI. Only the archiver bitbake class use the SRC_URI in the expanded 
way. This means we have to fix a lot of users anyway. The gitsm approach 
embed the unpack step into the fetcher and avoid a patch step between 
the unpack and parsing of the source. This works for git because the 
destsuffix is known. In case of an archive we have to provide the 
relative directory inside the archive. This means we have to use 
variables inside the fetcher commands, pass more function arguments or 
extent the SRC_URI parameters. The alternative is to introduce implicit 
URIs. The SRC_URI only contains the static defined URIs from the recipe 
and we introduce a new API to collect the dynamic implicit URIs or all 
URIs together. The URIs are saved inside a fixed folder inside the 
WORKDOR. Its like an .inc file but the file is generated on-the-fly. The 
URIs still use the fetcher for download an unpack but we have a separate 
step for the SRC_URI and the implicit URIs. We have different 
possibilities to combine and distribute the steps onto tasks or 
postfuncs. We could add tasks to resolve, fetch, unpack and patch the 
implicit sources after the do_unpack. We could add tasks to fetch, 
unpack and patch the sources from the SRC_URI or only parts of it before 
the do_fetch or we could integrate the steps into the do_fetch. We could 
reuse the output from an previews task or use a fresh start like a plain 
unpack and patch of all sources. We could reuse the code to provide a 
manual update step to create a inc file. We need to improve the usage of 
the SRC_URI in any case because it is used like any other variable and 
it is unclear that an entry inside the SRC_URI could be expanded to 
multiple URIs.

> I also very strongly want to maintain our separation of do_fetch,
> do_unpack and do_patch.

What does this mean? The do_fetch of embedded dependencies requires an 
unpack. Therefore we have to split the tasks or disallow any dynamic. 
The oe-core solution allows a full separation whereas the bitbake 
solution require duplication inside bitbake.

>>> I'm also not sure that drawing gitsm into this is a great idea when it
>>> is effectively already working  quite well.
>> To my knowledge it isn't possible to patch a sub module revision. Is
>> it intended that the SBOM doesn't contain the recursive implicit git
>> repositories?
> A patch was merged to allow the data about implicit git repos (amongst
> other things) to be available:
>
> https://git.yoctoproject.org/poky/commit/bitbake/?id=ef3e46afd910d4b7727d42c4c18b501525c65695
>
> That tracer code was added after a lot of discussion that took up a ton
> of my time, then people moved on and it hasn't seem much
> development/use :(.
>
> So there were mechanisms added to make the data available.

Why was this needed? We already have a function for the fetchers to 
provide the expand URLs.

>>>   I appreciate the desire to
>>> have neat abstractions applying accross everything equally but I'm not
>>> sure we can achieve that with the level of usability we need and I'd
>>> rather improve usability at the cost of the inclusion of gitsm, which
>>> is the most functional fetcher we currently have in this group.
>> I include gitsm because it is similar to the package manager and have
>> some drawbacks which I solve with my approach. It is the only fetcher
>> that parse a file to determine additional URIs. Its integration into
>> oe-core is sub-optimal.
> I believe there was a way added to solve the drawback you mention. The
> same mechanism could be used for other fetchers too yet we're talking
> about much more radical re-designs.

Do you have a hint for that way so that I can look into it.

>>> I will repeat again that I do strongly feel that a strong API in
>>> bitbake is going to lead to an overall better design that something
>>> bolted onto the fetcher in OE-Core.
> The "fetcher" is an API exposed by Bitbake with the intent of
> standardising the fetching of source code. It's aim is to support
> objectives such as allowing resilience through source mirrors and
> support build reproducibility. There is a document in bitbake which
> tries to at least summarise some of that after it was requested it be
> written down.

What is the fetching of the source code and why doesn't the oe-core 
solution doesn't fit? It uses the fetcher to fetch the sources. The only 
different is that it pass the URIs explicit and doesn't fetch the 
SRC_URI only.

>> What is a fetcher? We have fetchers for protocols (wget, git),
> These were the original fetchers using the original APIs.
>
>> wrapper around other fetchers (crate, gomod and gomodgit)
> These are newer and some work better than others are people have
> attempted to git the newer languages into the older framework.

But this is possible without a fetcher. The crate, gomod and my npm 
fetcher mainly override the urldata_init and unpack function. The same 
functionality can be provided by a python function to define the SRC_URI 
(replace urldata_init function) and a task or postfunc for unpack 
(replace unpack function). The advantage is a plain URL in the SRC_URI 
and the cache or vendor folder handling isn't scattered over a fetcher 
and bbclass. If we use a oe library for the important code we could 
simplify the tests because we could test individual steps. Even the 
debugging could be simpler if we use a separate task to populate the 
cache or vendor folder from the downloaded artifacts. The appropriate 
code is already in my oe-core series.

>>   and wrapper around other fetchers which embedded file parser (gitsm,
>> npmsw).
> Of these, gitsm is now relatively successfully integrated and
> functional.

Doesn't Bruce have problems with it and replace it by plain git URIs?

I think the main part of the successful integration is that you doesn't 
need any manual step to fetch the embedded dependencies. The main 
difference to the package managers is that it is uncommon to use patches 
for gitsm. Every sub module defines the version of its sub module 
whereas a package only defines it compatibility. The package versions 
are defined inside the main source.

>>   Thereby the gitsm fetcher fetch a source and parse the embedded
>> file. The npmsw fetcher parse a file from the meta layer. The gitsm
>> fetcher recursive combines the download and unpack function call in
>> mostly every class function.
> Sadly there did not seem to be any other way to handle that. I'd
> imagine that some of the other fetchers are going to have to work
> similarly, unfortunately.
The problem of the bitbake fetcher is the unclear caching and the 
missing dependency between the steps. This is the advantage of oe-core. 
We can simple arrange everything which need this information after the 
resolve task.

>> The wrappers has the disadvantage that they use uncommon URIs which
>> are useless for the SBOM. We support PyPI without the need for an
>> extra fetcher.
> We support pypi in so much as we have a recipe per dependency. That was
> not deemed possible for npm or crates.
That's no reason for or against a pypi fetcher or a crate_src_uri function.

> I would disagree that uncommon
> URIS are useless for SBOMs. They can be problematic but having the data
> there means it is at least listed.
But it reduce the usability of the SBOM.

>   There are also ways they can be
> extended (e.g. the intercept code I link to above).
>
>> Is PyPI or crate the desired way to go?
> Compared to the complexity in some of the patches proposed, I'd
> actually say yes.
What complexity do you mean? Some of the complexity of my patches comes 
from the limitation and fixed use cases of the existing code.

The gomod fetcher contain a feature to extract the .mod from a .zip 
file. This isn't needed because the go.sum file contains entries for the 
.mod and .zip file. The compatible solution would be do download the 
.mod and .zip file.

The crate fetcher implements its own unpack code. It behavior depends on 
the BP variable. The unpack function writes a meta file which is need to 
use the source inside the vendor folder. This makes it impossible to 
reuse the code for plain URIs. Alternative the URI could be marked as 
crate vendor source and an extra step could create the meta file for all 
marked URIs.

>> nstead of expand the fetcher I would focus the fetcher on the
>> download of provides URIs and keep the file parsing and post unpack
>> processing outside of it.
> The challenge is that once you process it, you need to fetch more
> things. Our ecosystem is built on the idea that fetching happens in
> do_fetch. If we change that most mirroring scripts break for example.
> It is like compiling happening during do_configure or do_install - it
> is not what the user expects and that is problematic. I am against
> changing behaviour in ways users do not expect.

I don't change the exception. The exceptions are the reason for the 
state of the series.

The missing point is that the expectation is related to the end of a 
dependency chain and not to a task itself. In any case the real 
expectation isn't clear. What happens if the configuration need a 
compiled source and it isn't feasible to create a native package. My 
series satisfy the following expectations:

* All sources are downloaded after the do_fetch
* All sources are unpacked after the do_unpack
* All sources are patched after the do_patch

The series contains one possible solutions. There are multiple other 
solutions to reach the same inside oe-core by distributing the following 
steps over task:

run_early_fetch - Download the source with the lock files from SRC_URI
run_early_unpack - Unpack the source with the lock files from SRC_URI 
(after run_early_fetch)
run_early_patch - Apply patch which manipulate the lock files from 
SRC_URI (after run_early_patch)
run_resolve - Parse the lock files to resolve the dependency sources 
(after run_early_patch)
run_vendor_fetch - Download the dependency sources (after run_resolve)
run_vendor_unpack - Unpack the dependency sources (after run_vendor_fetch)
run_vendor_patch - Patch the dependency sources (after run_vendor_unpack)
run_fetch - Download the additional source from SRC_URI
run_unpack - Unpack the additional source from SRC_URI (after run_fetch)
run_patch - Apply the additional source from SRC_URI (after run_unpack)

Some of the steps could be combined, repeat or integrated into other 
task. For example the run_early_fetch and run_fetch are a simple fetch 
of the SRC_URI list whereas the run_early_fetch alone is a fetch of a 
filtered SRC_URI list.

Series:

do_early_fetch : run_early_fetch
do_early_unpack : run_early_unpack
do_early_patch : run_early_patch
do_resolve : run_resolve
do_fetch : run_fetch, run_early_fetch, run_vendor_fetch
do_unpack: run_unpack, run_early_unpack, run_vendor_unpack
do_patch: run_patch, run_early_patch, run_vendor_patch

WIP:

do_fetch : run_fetch, 
run_early_fetch, run_early_unpack, run_early_patch, run_resolve, 
run_vendor_fetch
do_unpack: run_unpack, run_early_unpack, run_vendor_fetch
do_patch: run_patch, run_early_patch, run_vendor_patch

The bitbake approach looks like the WIP but makes the run_early_patch 
complicated or impossible.

I prefer something like the first approach because it is similar to the 
.inc file approach and the separate tasks are easier to debug. The 
do_fetch, do_unpack and do_patch task works like before. The only 
difference is that the implicit URIs are managed outside of the fetcher. 
At the moment this only influence a single user outside of the fetch, 
unpack and patch tasks but we could use the opportunity to fix other 
users as well (ex. SBOM).

>> The package manager lock file is a simple file which references to
>> package sources. Thereby a main source could contain multiple package
>> manager lock files. At the moment we have no fetcher which reference
>> an other source as base or combines multiple fetchers.
> This is the case for good reason. If the lock file references all the
> sources, perhaps we should just include it alongside the recipe and be
> done with it?

This is possible but the user have manual copy the lock file from the 
source to the meta layer, manual apply patches to it and ensure that the 
source and lock file are in sync.

Why is the bitbake fetcher the correct place to parse a lock file?

Why is it a bad idea to generate and manage the URIs outside of bitbake?

>> Based on my two proposed solutions and the work the last month, I
>> have the following feasible requirements:
>>   * Support patches for the package manager lock file to simply
>       updates, back ports and upsteaming
>>   * Support multiple package manager lock files per main source
>>   * Include the real https or git URI in the SBOM
>>   * Include the dependency name and version in the SBOM
>>   * No manual bitbake command call after a SRC_URI, PV or SRCREV
>> change
> I think we may have to relax some of these requirements as we can't do
> everything.
>> The following code is mostly used in oe-core and ignore the implicit
>> URIs:
>>
>>      src_uri = (d.getVar('SRC_URI') or "").split()
>>      fetcher = bb.fetch2.Fetch(src_uri, d)
>>      for url in fetcher.urls:
>>
>> The following code is used in the archiver class only and includes
>> the implicit URIs:
>>
>>      src_uri = get_src_uris(d)
>>      fetcher = bb.fetch2.Fetch(src_uri, d)
>>      for ud in fetcher.expanded_urldata():
>   
> Ironically, I think this an example of the kind of "strong API" I'm
> referring to, where a standard API is exposed by bitbake. In some ways
> this was already attempted with the tracer API above where the
> expand_urldata call would actually be a call into the tracer API to
> obtain the actual urls used.

Sorry but I think the tracer API isn't needed because we have already 
all information and possible functions inside the fetcher.

This "strong API" make it complicated to patch the source. Either the 
API depends on bitbake variables or it depends on the provides hooks. 
This means if somebody miss a hook or a bitbake variable he will get an 
other output.

The primary function of the fetcher is to download and unpack of the 
sources. Why do we use it for management of the source URIs? Why don't 
we manage the source URIs outside of bitbake? We can save the source 
URIs inside the WORKDIR and provide an additional common api to it. At 
the moment every user create a new instance of the Fetch object and in 
many cases manual pass the SRC_URI to it:

     for url in oe.fetcher.get_source_uris(d):

We could add a new task to create the files and allows the user to 
inspect it. I could provide a RFC if it helps the discussion.

>> My last series enable oe-core to generate implicit URIs and therefore
>> replace the usage of the plain SRC_URI with a function:
>>
>>       fetcher = bb.fetch.Fetch(get_src_uris(d), d)
>>      for url in fetcher.urls:
>>
>>   The SRC_URI contains the given URIs and the get_src_uri function
>> contains the given and implicit URIs. The concept is taken from other
>> classes like the spdx classes which use files to exchange data
>> between tasks.
> I believe the get_src_uris() functionality should happen within the
> fetcher module and not outside of it as a bolt on interface.

It isn't bolt on the interface. It provides an interface on top of the 
fetcher. The fetcher download and unpack the source but the sources are 
managed inside oe-core. At the moment the management doesn't really 
exist because the sources are only consists of the SRC_URI variable.

> I do not know where we go from here. You disagree with me on several
> key things

It's not about consent, it's about trying to understand the reasons. 
I've been working on this topic for years and have tried out different 
options. I think I have a solution that is very easy to use. In order to 
improve the solution, I try to understand the reasons for the other 
decisions. I don't understand why it is desired to add code to the 
fetcher which works independent of network access and why it is desired 
to create a fetcher object to receive the list of source URIs.

>   such as whether the code should be in bitbake or OE-Core,
I have the fear that the bitbake solution limit the possibilities 
without an advantage. The complexity depends on the feature not on the 
point of implementation.

> whether gitsm should be changed,

I use gitsm as a benchmark to evaluate concepts and approaches. It is 
better known as a lock file and a complex use case because of the 
recursive sources. It doesn't matter if we change it but the selected 
approach should support a replacement without limitations or clearly 
state why it isn't supported.

>   whether we need to have some kind of
> checksum to verify the "lockfile" is correct as originally intended in
> the recipe and probably more.
I want to prevent us from having the wrong motivation due to a lack of 
understanding of the lock file. Why don't we add a look file across all 
expanded URIs, regardless of the source of the URI, because we could 
have the same problems in the other fetchers.

> I'm in the position of trying to mediate things (i.e. see the different
> viewpoints and bring people together, trying to find common ground) and
> yet at the same time, express views as someone who as spent a lot of
> years trying to maintain and improve this code.
I appreciate your work.

But I sometimes have the feeling that the code is a patchwork quilt and 
every attempt to replace several patches with something common is met 
with rejection.

> I have tried to seek help from the people who effectively oversee me as
> a maintainer (i.e. the TSC) with limited success. I'm not sure what to
> do from here.

I really like to upstream my work because I think it would simplify the 
package manager support, but there doesn't seem to be much interest in 
the community for it.

[-- Attachment #2: Type: text/html, Size: 31262 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 3861 bytes --]

On Wed, 2025-02-19 at 17:48 +0100, Stefan Herbrechtsmeier wrote:
>  
> I don't change the exception. The exceptions are the reason for the
> state of the series.
> 
> The missing point is that the expectation is related to the end of a
> dependency chain and not to a task itself. In any case the real
> expectation isn't clear. What happens if the configuration need a
> compiled source and it isn't feasible to create a native package. My
> series satisfy the following expectations:
> 
> * All sources are downloaded after the do_fetch
> * All sources are unpacked after the do_unpack
> * All sources are patched after the do_patch
I could try and reply to all your points but this one jumps out as a
really fundamental thing we disagree on. All our existing code assumes
that:

"All sources are downloaded during do_fetch"

which means once do_fetch runs, we have all the sources. That is quite
different to what you say above. I fully appreciate it imposes quite
some constraints but it doesn't change the fact that this is the
situation.

> WIP:
> 
> do_fetch : run_fetch,
> run_early_fetch, run_early_unpack, run_early_patch, run_resolve,
> run_vendor_fetch
> do_unpack: run_unpack, run_early_unpack, run_vendor_fetch
> do_patch: run_patch, run_early_patch, run_vendor_patch 
> 
> The bitbake approach looks like the WIP but makes the run_early_patch
> complicated or impossible.

Which is why I'm saying that I'd prefer to do this in bitbake and if we
have to drop that patching as a result, so be it.


> >  whether we need to have some kind of
> > checksum to verify the "lockfile" is correct as originally intended in
> > the recipe and probably more.
> >  
> I want to prevent us from having the wrong motivation due to a lack
> of understanding of the lock file. Why don't we add a look file
> across all expanded URIs, regardless of the source of the URI,
> because we could have the same problems in the other fetchers.
> 
> 
> >  
> > I'm in the position of trying to mediate things (i.e. see the different
> > viewpoints and bring people together, trying to find common ground) and
> > yet at the same time, express views as someone who as spent a lot of
> > years trying to maintain and improve this code.
> >  
> I appreciate your work.
> 
> But I sometimes have the feeling that the code is a patchwork quilt
> and every attempt to replace several patches with something common is
> met with rejection.


We have a patchwork where people trust the current boundaries between
the different pieces. You want to change the boundaries, and the
assumptions made about those boundaries and replace it with something
that isn't going to be easily understood by most of the userbase as the
concepts are complex and break the existing models people work with.
I'm doing my best to explain the problem but either I'm failing at that
or we simply disagree on them.

> > I have tried to seek help from the people who effectively oversee me as
> > a maintainer (i.e. the TSC) with limited success. I'm not sure what to
> > do from here.
> >  
>  
> I really like to upstream my work because I think it would simplify
> the package manager support, but there doesn't seem to be much
> interest in the community for it.
>  
People are too focused in their own problems to be able to spend enough
time on these architectural type pieces. We do badly need this kind of
work which is why I'm trying to help this move forward but equally we
have to do it in a way which doesn't destablise our existing userbase,
in other words we may have to make some compromises. I'd like to hope
we pick the right ones. I'm worried as we have tried solutions for this
space before and the fact we're discussing them again shows we haven't
got it right yet.

Cheers,

Richard



[-- Attachment #2: Type: text/html, Size: 5541 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 6531 bytes --]

Am 19.02.2025 um 18:33 schrieb Richard Purdie:
> On Wed, 2025-02-19 at 17:48 +0100, Stefan Herbrechtsmeier wrote:
>> I don't change the exception. The exceptions are the reason for the 
>> state of the series.
>>
>>
>> The missing point is that the expectation is related to the end of a 
>> dependency chain and not to a task itself. In any case the real 
>> expectation isn't clear. What happens if the configuration need a 
>> compiled source and it isn't feasible to create a native package. My 
>> series satisfy the following expectations:
>>
>> * All sources are downloaded after the do_fetch
>> * All sources are unpacked after the do_unpack
>> * All sources are patched after the do_patch
>>
> I could try and reply to all your points but this one jumps out as a 
> really fundamental thing we disagree on. All our existing code assumes 
> that:
>
> "All sources are downloaded during do_fetch"
>
>
> which means once do_fetch runs, we have all the sources. That is quite 
> different to what you say above. I fully appreciate it imposes quite 
> some constraints but it doesn't change the fact that this is the 
> situation.

I don't get the point. Do you say that only the do_fetch is allow to 
download anything? So my assumption that I can download something 
outside the do_fetch task as long as the download happens if something 
depend on the fetch task (like bitbake -c fetch ...) is wrong?

I don't understand that requirement but I can fulfill it. The only 
consequence is a complex do_fetch task.

>
>
>> WIP:
>>
>> do_fetch : run_fetch, 
>> run_early_fetch, run_early_unpack, run_early_patch, run_resolve, 
>> run_vendor_fetch
>> do_unpack: run_unpack, run_early_unpack, run_vendor_fetch
>> do_patch: run_patch, run_early_patch, run_vendor_patch
>>
>> The bitbake approach looks like the WIP but makes the run_early_patch 
>> complicated or impossible.
>>
>
> Which is why I'm saying that I'd prefer to do this in bitbake and if 
> we have to drop that patching as a result, so be it.

But this is an essential feature because it simplify the usability. It 
allows inexperienced user to fix a security issue in some minutes by 
simple back port a patch from the upstream project or fix the problem 
with common tools outside of oe.

>
>>>   whether we need to have some kind of
>>> checksum to verify the "lockfile" is correct as originally intended in
>>> the recipe and probably more.
>> I want to prevent us from having the wrong motivation due to a lack 
>> of understanding of the lock file. Why don't we add a look file 
>> across all expanded URIs, regardless of the source of the URI, 
>> because we could have the same problems in the other fetchers.
>>
>>
>>> I'm in the position of trying to mediate things (i.e. see the different
>>> viewpoints and bring people together, trying to find common ground) and
>>> yet at the same time, express views as someone who as spent a lot of
>>> years trying to maintain and improve this code.
>> I appreciate your work.
>>
>> But I sometimes have the feeling that the code is a patchwork quilt 
>> and every attempt to replace several patches with something common is 
>> met with rejection.
>
>
> We have a patchwork where people trust the current boundaries between 
> the different pieces. You want to change the boundaries, and the 
> assumptions made about those boundaries and replace it with something 
> that isn't going to be easily understood by most of the userbase as 
> the concepts are complex and break the existing models people work 
> with. I'm doing my best to explain the problem but either I'm failing 
> at that or we simply disagree on them.

I don't change the boundaries. The boundaries are not clean at the 
moment. Look at the code. Most users of the URIs assume that only the 
SRC_URI exists. Because of gitsm this isn't true. But only the archiver 
bbclass use it correct. The spdx classes notice it but fix it in a wrong 
way. It simple replace the gitsm:// with a git:// and lose all the 
dependent sources. I want to sharpen the concept by introduce a fetcher 
/ source library in oe. The library abstract the bitbake fetcher details 
and provide a simple api on top of it. This allows us to remove all the 
duplicated code. The bitbake fetcher focus on the download and the oe 
fetcher class on the management of the URIs. This allows use to 
introduce new concept for the URI management and allows use to remove 
the repeat instantiation and parsing of the SRC_URI.

Based on the code I would argue that the user base don't know that the 
SRC_URI could be expand to additional URIs nor that a SRC_URI is 
replaced by an other URI underneath. We have to fix this in any way and 
I would take the opportunity to introduce a simpler and feature rich 
solution.

We have a go-vendor bbclass and gomod fetcher. The first use the wget 
and git fetcher and the second add a specific fetcher. We have two 
solutions for the same problem with different concepts.

>>> I have tried to seek help from the people who effectively oversee me as
>>> a maintainer (i.e. the TSC) with limited success. I'm not sure what to
>>> do from here.
>>
>> I really like to upstream my work because I think it would simplify 
>> the package manager support, but there doesn't seem to be much 
>> interest in the community for it.
>>
> People are too focused in their own problems to be able to spend 
> enough time on these architectural type pieces. We do badly need this 
> kind of work which is why I'm trying to help this move forward but 
> equally we have to do it in a way which doesn't destablise our 
> existing userbase, in other words we may have to make some 
> compromises. I'd like to hope we pick the right ones. I'm worried as 
> we have tried solutions for this space before and the fact we're 
> discussing them again shows we haven't got it right yet.

I always try my best to not break existing users. Therefore I keep the 
old behavior of the existing tasks or function calls. But we need to 
sharpen the concepts and unify the code base. We have different 
solutions (like go-vendor.bbclass and gomod.py) for the same problem and 
partial integration of features (like expanded_urldata).

I think we have to much theoretical discussion and to less code. I would 
like to integrate a Fetcher OE library to remove the redundant code, 
unify the codebase and use the extended URIs where appropriate. This 
would fix some existing problems and could help in the next step 
independence of the solution. Would that be okay with you?

[-- Attachment #2: Type: text/html, Size: 10208 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Richard Purdie]

[-- Attachment #1: Type: text/plain, Size: 4993 bytes --]

On Thu, 2025-02-20 at 10:48 +0100, Stefan Herbrechtsmeier wrote:
>  
> Am 19.02.2025 um 18:33 schrieb Richard Purdie:
> > 
> > On Wed, 2025-02-19 at 17:48 +0100, Stefan Herbrechtsmeier wrote:
> > >  
> > >  
> > >  
> > > I don't change the exception. The exceptions are the reason for
> > > the state of the series.
> > >  
> > > 
> > > The missing point is that the expectation is related to the end
> > > of a dependency chain and not to a task itself. In any case the
> > > real expectation isn't clear. What happens if the configuration
> > > need a compiled source and it isn't feasible to create a native
> > > package. My series satisfy the following expectations:
> > > 
> > > * All sources are downloaded after the do_fetch
> > > * All sources are unpacked after the do_unpack
> > > * All sources are patched after the do_patch
> > >  
> >  
> > I could try and reply to all your points but this one jumps out as
> > a really fundamental thing we disagree on. All our existing code
> > assumes that:
> >  
> > 
> >  
> >  
> > "All sources are downloaded during do_fetch"
> > 
> >  
> > 
> >  
> >  
> > which means once do_fetch runs, we have all the sources. That is
> > quite different to what you say above. I fully appreciate it
> > imposes quite some constraints but it doesn't change the fact that
> > this is the situation.
> >  
>  
> I don't get the point. Do you say that only the do_fetch is allow to
> download anything? So my assumption that I can download something
> outside the do_fetch task as long as the download happens if
> something depend on the fetch task (like bitbake -c fetch ...) is
> wrong?
> 
> I don't understand that requirement but I can fulfill it. The only
> consequence is a complex do_fetch task.

The fetch task is the only task allowed to access the network and is
the only place where the sources should be being downloaded. Once the
do_fetch task completes, we have everything we need from the network
and no further network access is allowed.

So yes, your assumption is therefore wrong. I'm focusing on this as if
we don't have that common understanding, nothing built upon those
different understandings is going to work.

> > > >  
> > > > I have tried to seek help from the people who effectively oversee me as
> > > >  
> > > > a maintainer (i.e. the TSC) with limited success. I'm not sure what to
> > > >  
> > > > do from here.
> > > >  
> > > >  
> > > >  
> > >  
> > >  
> > >  
> > > I really like to upstream my work because I think it would
> > > simplify the package manager support, but there doesn't seem to
> > > be much interest in the community for it.
> > >  
> > >  
> > >  
> >  
> > People are too focused in their own problems to be able to spend
> > enough time on these architectural type pieces. We do badly need
> > this kind of work which is why I'm trying to help this move forward
> > but equally we have to do it in a way which doesn't destablise our
> > existing userbase, in other words we may have to make some
> > compromises. I'd like to hope we pick the right ones. I'm worried
> > as we have tried solutions for this space before and the fact we're
> > discussing them again shows we haven't got it right yet.
> >  
>  
> I always try my best to not break existing users. Therefore I keep
> the old behavior of the existing tasks or function calls. But we need
> to sharpen the concepts and unify the code base. We have different
> solutions (like go-vendor.bbclass and gomod.py) for the same problem
> and partial integration of features (like expanded_urldata).
> 
> I think we have to much theoretical discussion and to less code. I
> would like to integrate a Fetcher OE library to remove the redundant
> code, unify the codebase and use the extended URIs where appropriate.
> This would fix some existing problems and could help in the next step
> independence of the solution. Would that be okay with you?

It would not be okay at all, no. 

I appreciate theoretical discussion is hard work but we've tried taking
patches in the past and we've ended up making things worse. I can tell
the direction you've been going isn't correct and is going to lead to
patches which are unable to be merged.

I've made it very very clear that the fetch module stays in bitbake and
that we need to find a way to support this in the fetch module, not OE.
Yes, that complicates things unfortunately but I have good reasons for
doing it, whether you understand or agree with them or not. I'm not
changing my mind on this.

I will say your original patch series against bitbake was probably a
lot closer to what I think we probably need. I didn't like the "magic"
going on to extend the data so perhaps if that piece were reworked and
made a more default/simpler/clearer part of the fetcher rather than
hidden, we'd probably reach some code we could agree on.

Cheers,

Richard


>  


[-- Attachment #2: Type: text/html, Size: 6933 bytes --]

Re: [bitbake-devel] 'vendor' fetching discussion cont.

[Stefan Herbrechtsmeier]

[-- Attachment #1: Type: text/plain, Size: 5775 bytes --]

Am 20.02.2025 um 11:00 schrieb Richard Purdie via lists.openembedded.org:
> On Thu, 2025-02-20 at 10:48 +0100, Stefan Herbrechtsmeier wrote:
>> Am 19.02.2025 um 18:33 schrieb Richard Purdie:
>>> On Wed, 2025-02-19 at 17:48 +0100, Stefan Herbrechtsmeier wrote:
>>>> I don't change the exception. The exceptions are the reason for the 
>>>> state of the series.
>>>>
>>>>
>>>> The missing point is that the expectation is related to the end of 
>>>> a dependency chain and not to a task itself. In any case the real 
>>>> expectation isn't clear. What happens if the configuration need a 
>>>> compiled source and it isn't feasible to create a native package. 
>>>> My series satisfy the following expectations:
>>>>
>>>> * All sources are downloaded after the do_fetch
>>>> * All sources are unpacked after the do_unpack
>>>> * All sources are patched after the do_patch
>>>>
>>> I could try and reply to all your points but this one jumps out as a 
>>> really fundamental thing we disagree on. All our existing code 
>>> assumes that:
>>>
>>> "All sources are downloaded during do_fetch"
>>>
>>>
>>> which means once do_fetch runs, we have all the sources. That is 
>>> quite different to what you say above. I fully appreciate it imposes 
>>> quite some constraints but it doesn't change the fact that this is 
>>> the situation.
>>
>> I don't get the point. Do you say that only the do_fetch is allow to 
>> download anything? So my assumption that I can download something 
>> outside the do_fetch task as long as the download happens if 
>> something depend on the fetch task (like bitbake -c fetch ...) is wrong?
>>
>> I don't understand that requirement but I can fulfill it. The only 
>> consequence is a complex do_fetch task.
>>
>
> The fetch task is the only task allowed to access the network and is 
> the only place where the sources should be being downloaded. Once the 
> do_fetch task completes, we have everything we need from the network 
> and no further network access is allowed.
I don't understand why the network need to be restricted to the do_fetch 
task only but okay.


> So yes, your assumption is therefore wrong. I'm focusing on this as if 
> we don't have that common understanding, nothing built upon those 
> different understandings is going to work.
>
>>>>> I have tried to seek help from the people who effectively oversee me as
>>>>> a maintainer (i.e. the TSC) with limited success. I'm not sure what to
>>>>> do from here.
>>>>
>>>> I really like to upstream my work because I think it would simplify 
>>>> the package manager support, but there doesn't seem to be much 
>>>> interest in the community for it.
>>>>
>>> People are too focused in their own problems to be able to spend 
>>> enough time on these architectural type pieces. We do badly need 
>>> this kind of work which is why I'm trying to help this move forward 
>>> but equally we have to do it in a way which doesn't destablise our 
>>> existing userbase, in other words we may have to make some 
>>> compromises. I'd like to hope we pick the right ones. I'm worried as 
>>> we have tried solutions for this space before and the fact we're 
>>> discussing them again shows we haven't got it right yet.
>>
>> I always try my best to not break existing users. Therefore I keep 
>> the old behavior of the existing tasks or function calls. But we need 
>> to sharpen the concepts and unify the code base. We have different 
>> solutions (like go-vendor.bbclass and gomod.py) for the same problem 
>> and partial integration of features (like expanded_urldata).
>>
>> I think we have to much theoretical discussion and to less code. I 
>> would like to integrate a Fetcher OE library to remove the redundant 
>> code, unify the codebase and use the extended URIs where appropriate. 
>> This would fix some existing problems and could help in the next step 
>> independence of the solution. Would that be okay with you?
>>
>
> It would not be okay at all, no.

I want to fix existing gitsm problems which are independent of the 
solution. We have duplicated code inside oe-core and need to fix them 
for both solutions. Otherwise the sources from the lock file aren't 
handle by the classes.

We have the archiver.bbclass, cargo_common.bbclass, 
create-spdx-2.2.bbclass and spdx30_tasks.py files with use slightly 
different code to receive the URIs and only one use the expanded list of 
gitsm. How should I fix them?

> I appreciate theoretical discussion is hard work but we've tried 
> taking patches in the past and we've ended up making things worse. I 
> can tell the direction you've been going isn't correct and is going to 
> lead to patches which are unable to be merged.
>
> I've made it very very clear that the fetch module stays in bitbake 
> and that we need to find a way to support this in the fetch module, 
> not OE. Yes, that complicates things unfortunately but I have good 
> reasons for doing it, whether you understand or agree with them or 
> not. I'm not changing my mind on this.
Sorry, but I don't know how do go on. I don't understand the reasons nor 
goals. I propose to change the code to remove code duplication and you 
refuse it. I ask for the desired concept (pypi / go-vendor vs gomod) and 
don't get a clear answer. I have a lot of improvements, harmonization 
and fixes in the last series and doesn't know if they are desired.


> I will say your original patch series against bitbake was probably a 
> lot closer to what I think we probably need. I didn't like the "magic" 
> going on to extend the data so perhaps if that piece were reworked and 
> made a more default/simpler/clearer part of the fetcher rather than 
> hidden, we'd probably reach some code we could agree on.

Which "magic" do you mean and what isn't clean enough?

[-- Attachment #2: Type: text/html, Size: 10150 bytes --]