Re: [bitbake-devel] [RFC PATCH 00/15] Make mirror replacement syntax explicit

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2025-02-05 at 13:12 +0100, Stefan Herbrechtsmeier wrote:
>  Am 05.02.2025 um 11:34 schrieb Richard Purdie:
>  On Wed, 2025-02-05 at 08:15 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> > > The mirror replacement syntax contains many implicit transformations.
> > > The path of the URI always contains the base name of the downloaded
> > > filename. This makes it impossible to rename or remove the base name of
> > > the original path. It prevents upstream mirror for SRC_URIS with a
> > > downloadfilename parameter. The base name of the downloaded filename
> > > makes it impossible to use the download mirror for SRC_URIs with
> > > subfolders in the downloadfilename parameter. Altogether the implicit
> > > transformation complicates the understanding of the replacements.
> > > 
> > > This series adds an additional replacement named DOWNLOADFILENAME. This
> > > replacement contains the relative filename of the downloaded file or
> > > mirror archive for git and hg. This allows the user to explicitly define
> > > the behavior. The usage is equivalent to the PATH replacement for the
> > > sstate mirror from file to https scheme.
> > > 
> > > git://.*/.*  http://downloads.yoctoproject.org/mirror/sources/DOWNLOADFILENAME
> > > https?://.*/.*  http://downloads.yoctoproject.org/mirror/sources/DOWNLOADFILENAME
> > > file://.*  https://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH
> > > 
> > > Without a replacement variable the mirror will use the same base name as
> > > the origin SRC_URI. This allows the usage of private package manager
> > > registry together with a downloadfilename parameter or the rename of the
> > > base name.
> > > 
> > > https://registry.npmjs.org/  https://example.com/npm/registry/
> > > https://example.com/example/1.0.0.tgz  https://example.com/example/example-1.0.0.tgz
> > > 
> > > The series adds heuristics to keep a backward compatibility to common
> > > styles. Because of the ambiguity of the old style, it is advisable to
> > > remove this compatibility sooner or later to avoid unexpected behavior.
> > >  
> > Thanks for the patches, these look interesting with some good
> > improvements in there. A lot of the series looks like cleanups and
> > those look like good fixes to have. It may make sense to split this
> > series into two, the cleanups/fixes and the behaviour changes.
> > 
> > I'm not entirely "sold" on the naming of DOWNLOADFILENAME. You have to
> > think about this from the perspective of someone writing a MIRROR or
> > PREMIRROR entry - would they understand what that means vs some of the
> > other names?
> >  
> I’m open for suggestions. Even ARCHIVE or TARBALL are hard to
> understand because it is only a relative path on the download mirror.
> Alternative we can mark the lines as upstream or download mirror and
> give the replacement different meanings. The path could be the
> original PATH for an upstream mirror or the relative path of the
> downloaded file for the download mirror.

I've been giving this topic some thought. One idea I wondered about was
to instead markup the mirror urls with how they're expected to work
with a new parameter. For example:

git://.*/.*  http://downloads.yoctoproject.org/mirror/sources/?mirrorformat=mirrortarball

The possible options would be something like:

mirrortarball - mirror tarballs taken from DL_DIR
flattened - copy of DL_DIR so DL_DIR layout (maybe call it dldir?)
upstream - layout is the same as the upstream directory structure so a direct url replacement

If using a mirrortarball mirror url, we'd know to use the values from
urldata.mirrortarballs. We could add parameters to the fetcher to have
two parameters, one will be the DL_DIR path and the other would be the
upstream url path. One key question I have is how we might need to
shorted the url path for some mirror urls to add/remove a path prefix
in the mirroring.

I think that would cover most of our scenarios and make the mirror urls
much more useful/explicit.

It also would give us a migration path as the code can simply error if
it sees a mirror url without a mirrorformat parameter. This may mean we
could have a clean slate for the mirror urls and switch to one clear
format. I think we can make a case for such a breaking architecture
change.

To move it forward we'd need to work out which pieces of mirror syntax
we should drop, which mirror replacement strings we need (I'm not
convinced the current ones are right/useful). We'd probably need a some
proof of concept patches showing it in action and a proposal to the oe-
arch list explaining why the change was needed and what the change
would do/look like.

I think this can be done independently of the other fetcher/vendor
changing but should be able to work for the vendoring issues too.

Thoughts? I'm sure I'm missing things here :/.

Cheers,

Richard