On Thu, 2025-02-20 at 12:45 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote:

Am 20.02.2025 um 11:22 schrieb Richard Purdie via lists.openembedded.org:

On Wed, 2025-02-05 at 13:12 +0100, Stefan Herbrechtsmeier wrote:

 Am 05.02.2025 um 11:34 schrieb Richard Purdie:

 On Wed, 2025-02-05 at 08:15 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote:

I’m open for suggestions. Even ARCHIVE or TARBALL are hard to

understand because it is only a relative path on the download mirror.

Alternative we can mark the lines as upstream or download mirror and

give the replacement different meanings. The path could be the

original PATH for an upstream mirror or the relative path of the

downloaded file for the download mirror.

I've been giving this topic some thought. One idea I wondered about was

to instead markup the mirror urls with how they're expected to work

with a new parameter. For example:

git://.*/.*  http://downloads.yoctoproject.org/mirror/sources/?mirrorformat=mirrortarball

The ? could be problematic because it is the separator for the query. It is unlikely that the user really use this query parameter but it could complicate the code because we have to handle additional query parameters.
What does the "?mirrorformat=mirrortarball" mean? Will it work like a MIRRORTARBALL replacement?

The mirrorformat parameter would be used by the mirroring code itself to understand how to handle the url.

What is the different to a MIRRORTARBALL replacement? The code will replace the word with the content.

It would be dropped from the modified url so is only therefore our code's use. If there are additional parameters they would be passed through as they are now.

How does a simple replacement should look like?

http://  https://

Because of the backward compatible this will replace the basename of the path.

It would depend how the mirror is laid out. Some mirrors flatten the urls like DL_DIR is laid out, some potentially don't. The standard usage would likely have a mirrorformat=dldir parameter added.

How does the user specify an entry that replace the http scheme with https and keeps everything else like it is (upstream mirror)?

The possible options would be something like:

mirrortarball - mirror tarballs taken from DL_DIR

flattened - copy of DL_DIR so DL_DIR layout (maybe call it dldir?)

upstream - layout is the same as the upstream directory structure so a direct url replacement

Do you think we have to handle the mirror tarball explicit? The mirror tarball is required for a scheme change.

If we do that, we can avoid having to guess at too many urls to test to figure out a mirror format so I think it would be an improvement on where we are today.

Do you mean we will test if the URL have a mirrortarball and if not skip the entry?

If using a mirrortarball mirror url, we'd know to use the values from

urldata.mirrortarballs. We could add parameters to the fetcher to have

two parameters, one will be the DL_DIR path and the other would be the

upstream url path.

I don't understand where this is needed, because the mirror tarball and downloadfilename are used by different fetchers.

Please keep in mind that downloadfilename is pretty much a misfeature. It was added as we couldn't control collisions inside dl_dir but it creates all kind of other problems. I think we do need to handle that problem case but it does then mean we have to indicate whether any given mirror uses "dldir" or "upstream" names and paths.

I don't understand the problem. The download mirror will use the downloadfilename or its default the basename of the localpath. The upstream mirror will use the path. The mirrortarball will use the mirrortarball. Why the fetcher need two parameters?

One key question I have is how we might need to

shorted the url path for some mirror urls to add/remove a path prefix

in the mirroring.

What do you mean by this? The downloadfilename could contain a path without any problem after my change.

See above, downloadfilename is not something I'm keen to promote and is creating several of the problems we have by badly trying to hack extra functionality onto the fetcher without thinking through all the issues like mirroring.

What is the desired way to avoid name clashes? The package manager fetcher need an generic way to override the basename.

This is about the fact that you could have:

htttps://some.server/some/deep/multi/level/path/

and a mirror of things there at:

htttps://some.server/shortpath/path/

so the mirror urls need to be able to map

htttps://some.server/some/deep/multi/level/path/a/b/c.tgz

to

htttps://some.server/shortpath/path/a/b/c.tgz

htttps://some.server/some/deep/multi/level/path/  htttps://some.server/shortpath/path/

but also:

htttps://some.server/shortpath/path/c.tgz

htttps://some.server/some/deep/multi/level/path/.*  htttps://some.server/shortpath/path/DOWNLOADFILENAME

or possibly

htttps://some.server/shortpath/path.d.tgz

if downloadfilename is in action.

Because the fallback for the downloadfilename is the basename the same entry works.

htttps://some.server/some/deep/multi/level/path/.*  htttps://some.server/shortpath/DOWNLOADFILENAME

I think that would cover most of our scenarios and make the mirror urls

much more useful/explicit.

It also would give us a migration path as the code can simply error if

it sees a mirror url without a mirrorformat parameter. This may mean we

could have a clean slate for the mirror urls and switch to one clear

format. I think we can make a case for such a breaking architecture

change.

Does this means we doesn't need a backward compatibility?

If we can come up with sufficient justification to break things and can detect and error correctly for old usage and get this signed off by the TSC/community, potentially, yes.

Okay

Alternative we could make some replacement mandatory if a wildcard is used to detect obsolete entries.

I don't understand that.

If we have a wildcard .* in the path we need to know how to replace it. This could be the PATH, BASENAME, DOWNLOADFILENAME, MIRRORARCHIVE or re group. But in case of the re group this could still be an old entry.

Do support all cases we need a fix prefix or delimiter:

r:http https

http#https
http|https

http?://.*/.*|http://downloads.yoctoproject.org/mirror/sources/DOWNLOADARCHIVE
git://.*/.*|http://downloads.yoctoproject.org/mirror/sources/MIRRORARCHIVE

I do think we have too many problems in the existing mirroring url mapping and we probably need to rework this rather than try and pile more patches into it and complicate it further.

The question is whether the proposal fixes the issues it needs to and has enough simplification and benefit to justify making the change.