On Thu, 2025-02-20 at 12:45 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
Am 20.02.2025 um 11:22 schrieb Richard Purdie via lists.openembedded.org:
On Wed, 2025-02-05 at 13:12 +0100, Stefan Herbrechtsmeier wrote:
Am 05.02.2025 um 11:34 schrieb Richard Purdie:
On Wed, 2025-02-05 at 08:15 +0100, Stefan Herbrechtsmeier via lists.openembedded.org wrote:
I’m open for suggestions. Even ARCHIVE or TARBALL are hard to
understand because it is only a relative path on the download mirror.
Alternative we can mark the lines as upstream or download mirror and
give the replacement different meanings. The path could be the
original PATH for an upstream mirror or the relative path of the
downloaded file for the download mirror.
I've been giving this topic some thought. One idea I wondered about was
to instead markup the mirror urls with how they're expected to work
with a new parameter. For example:
git://.*/.* http://downloads.yoctoproject.org/mirror/sources/?mirrorformat=mirrortarball
The ? could be problematic because it is the separator for the query. It is unlikely that the user really use this query parameter but it could complicate the code because we have to handle additional query parameters.
What does the "?mirrorformat=mirrortarball" mean? Will it work like a MIRRORTARBALL replacement?
The mirrorformat parameter would be used by the mirroring code itself to understand how to handle the url. It would be dropped from the modified url so is only therefore our code's use. If there are additional parameters they would be passed through as they are now.
How does a simple replacement should look like?
http:// https://
Because of the backward compatible this will replace the basename of the path.
It would depend how the mirror is laid out. Some mirrors flatten the urls like DL_DIR is laid out, some potentially don't. The standard usage would likely have a mirrorformat=dldir parameter added.
The possible options would be something like:
mirrortarball - mirror tarballs taken from DL_DIR
flattened - copy of DL_DIR so DL_DIR layout (maybe call it dldir?)
upstream - layout is the same as the upstream directory structure so a direct url replacement
Do you think we have to handle the mirror tarball explicit? The mirror tarball is required for a scheme change.
If we do that, we can avoid having to guess at too many urls to test to figure out a mirror format so I think it would be an improvement on where we are today.
If using a mirrortarball mirror url, we'd know to use the values from
urldata.mirrortarballs. We could add parameters to the fetcher to have
two parameters, one will be the DL_DIR path and the other would be the
upstream url path.
I don't understand where this is needed, because the mirror tarball and downloadfilename are used by different fetchers.
Please keep in mind that downloadfilename is pretty much a misfeature. It was added as we couldn't control collisions inside dl_dir but it creates all kind of other problems. I think we do need to handle that problem case but it does then mean we have to indicate whether any given mirror uses "dldir" or "upstream" names and paths.
One key question I have is how we might need to
shorted the url path for some mirror urls to add/remove a path prefix
in the mirroring.
What do you mean by this? The downloadfilename could contain a path without any problem after my change.
See above, downloadfilename is not something I'm keen to promote and is creating several of the problems we have by badly trying to hack extra functionality onto the fetcher without thinking through all the issues like mirroring.
This is about the fact that you could have:
htttps://some.server/some/deep/multi/level/path/
and a mirror of things there at:
htttps://some.server/shortpath/path/
so the mirror urls need to be able to map
htttps://some.server/some/deep/multi/level/path/a/b/c.tgz
to
htttps://some.server/shortpath/path/a/b/c.tgz
but also:
htttps://some.server/shortpath/path/c.tgz
or possibly
htttps://some.server/shortpath/path.d.tgz
if downloadfilename is in action.
I think that would cover most of our scenarios and make the mirror urls
much more useful/explicit.
It also would give us a migration path as the code can simply error if
it sees a mirror url without a mirrorformat parameter. This may mean we
could have a clean slate for the mirror urls and switch to one clear
format. I think we can make a case for such a breaking architecture
change.
Does this means we doesn't need a backward compatibility?
If we can come up with sufficient justification to break things and can detect and error correctly for old usage and get this signed off by the TSC/community, potentially, yes.
Alternative we could make some replacement mandatory if a wildcard is used to detect obsolete entries.
I don't understand that.
I do think we have too many problems in the existing mirroring url mapping and we probably need to rework this rather than try and pile more patches into it and complicate it further.
The question is whether the proposal fixes the issues it needs to and has enough simplification and benefit to justify making the change.
Cheers,
Richard