Re: [bitbake-devel] [RFC PATCH 00/15] Make mirror replacement syntax explicit

[Richard Purdie <richard.purdie@linuxfoundation.org>]

[-- Attachment #1: Type: text/plain, Size: 5595 bytes --]

On Thu, 2025-02-20 at 12:45 +0100, Stefan Herbrechtsmeier via
lists.openembedded.org wrote:
>  
> Am 20.02.2025 um 11:22 schrieb Richard Purdie via
> lists.openembedded.org:
> > On Wed, 2025-02-05 at 13:12 +0100, Stefan Herbrechtsmeier wrote:
> > >  Am 05.02.2025 um 11:34 schrieb Richard Purdie:
> > >  On Wed, 2025-02-05 at 08:15 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
> > > I’m open for suggestions. Even ARCHIVE or TARBALL are hard to
> > > understand because it is only a relative path on the download mirror.
> > > Alternative we can mark the lines as upstream or download mirror and
> > > give the replacement different meanings. The path could be the
> > > original PATH for an upstream mirror or the relative path of the
> > > downloaded file for the download mirror.
> > >  
> >  
> > I've been giving this topic some thought. One idea I wondered about was
> > to instead markup the mirror urls with how they're expected to work
> > with a new parameter. For example:
> > 
> > git://.*/.*  http://downloads.yoctoproject.org/mirror/sources/?mirrorformat=mirrortarball
> >  
> The ? could be problematic because it is the separator for the query.
> It is unlikely that the user really use this query parameter but it
> could complicate the code because we have to handle additional query
> parameters.
> What does the "?mirrorformat=mirrortarball" mean? Will it work like a
> MIRRORTARBALL replacement?

The mirrorformat parameter would be used by the mirroring code itself
to understand how to handle the url. It would be dropped from the
modified url so is only therefore our code's use. If there are
additional parameters they would be passed through as they are now.

> How does a simple replacement should look like?
> 
> http://  https://
> 
> Because of the backward compatible this will replace the basename of
> the path.

It would depend how the mirror is laid out. Some mirrors flatten the
urls like DL_DIR is laid out, some potentially don't. The standard
usage would likely have a mirrorformat=dldir parameter added.


> > The possible options would be something like:
> > 
> > mirrortarball - mirror tarballs taken from DL_DIR
> > flattened - copy of DL_DIR so DL_DIR layout (maybe call it dldir?)
> > upstream - layout is the same as the upstream directory structure so a direct url replacement
> >  
> Do you think we have to handle the mirror tarball explicit? The
> mirror tarball is required for a scheme change.

If we do that, we can avoid having to guess at too many urls to test to
figure out a mirror format so I think it would be an improvement on
where we are today.

> > If using a mirrortarball mirror url, we'd know to use the values from
> > urldata.mirrortarballs. We could add parameters to the fetcher to have
> > two parameters, one will be the DL_DIR path and the other would be the
> > upstream url path.
> >  
> I don't understand where this is needed, because the mirror tarball
> and downloadfilename are used by different fetchers.

Please keep in mind that downloadfilename is pretty much a misfeature.
It was added as we couldn't control collisions inside dl_dir but it
creates all kind of other problems. I think we do need to handle that
problem case but it does then mean we have to indicate whether any
given mirror uses "dldir" or "upstream" names and paths.

> > One key question I have is how we might need to
> > shorted the url path for some mirror urls to add/remove a path prefix
> > in the mirroring.
> >  
> What do you mean by this? The downloadfilename could contain a path
> without any problem after my change.

See above, downloadfilename is not something I'm keen to promote and is
creating several of the problems we have by badly trying to hack extra
functionality onto the fetcher without thinking through all the issues
like mirroring.

This is about the fact that you could have:

htttps://some.server/some/deep/multi/level/path/

and a mirror of things there at:

htttps://some.server/shortpath/path/

so the mirror urls need to be able to map

htttps://some.server/some/deep/multi/level/path/a/b/c.tgz

to

htttps://some.server/shortpath/path/a/b/c.tgz

but also:

htttps://some.server/shortpath/path/c.tgz

or possibly

htttps://some.server/shortpath/path.d.tgz

if downloadfilename is in action.

> >  
> > I think that would cover most of our scenarios and make the mirror urls
> > much more useful/explicit.
> > 
> > It also would give us a migration path as the code can simply error if
> > it sees a mirror url without a mirrorformat parameter. This may mean we
> > could have a clean slate for the mirror urls and switch to one clear
> > format. I think we can make a case for such a breaking architecture
> > change.
> >  
> Does this means we doesn't need a backward compatibility?

If we can come up with sufficient justification to break things and can
detect and error correctly for old usage and get this signed off by the
TSC/community, potentially, yes.

> Alternative we could make some replacement mandatory if a wildcard is
> used to detect obsolete entries.

I don't understand that. 

I do think we have too many problems in the existing mirroring url
mapping and we probably need to rework this rather than try and pile
more patches into it and complicate it further.

The question is whether the proposal fixes the issues it needs to and
has enough simplification and benefit to justify making the change.

Cheers,

Richard

[-- Attachment #2: Type: text/html, Size: 8819 bytes --]