Plumbing for mapping from a remote tracking ref to the remote ref?

Plumbing for mapping from a remote tracking ref to the remote ref?

[Tao Klerks]

Hi folks,

Given the following configured fetch refspec for a remote:

[remote "origin"]
        url = git@someserver:somerepo.git
        fetch = +refs/heads/*:refs/remotes/somepath/*

And given a ref of the form "refs/remotes/somepath/branch_A",

I'm wondering whether there is any plumbing that would be able to tell
me what to put in a "fetch" command, to get
"refs/remotes/somepath/branch_A" fetched - in other words, is there
any plumbing that can use the configured fetch refspecs to map
"refs/remotes/somepath/branch_A" to "refs/heads/branch_A" for me, so
that I can then do "git fetch origin refs/heads/branch_A".

I understand I can parse the fetch refspecs myself, assuming any
asterisk is only ever on the tail end of the ref pattern... but that
seems very complicated, given that this is *probably* something others
have needed to do in the past?

Fwiw I've noticed that "git rev-parse --symbolic-full-name" knows how
to do almost exactly the *opposite* of that, when presented with the
"@{u}" pattern - it looks up the "branch.XXX.merge" value, which is
written in a remote-relative form ("refs/heads/branch_A" in this
example), and converts that to the "local" fetch destination (eg
"refs/remotes/somepath/branch_A"). But I don't know how to go the
opposite way, given only a local fetch destination and wanting to tell
fetch what to get - it expects a remote-relative ref.

Any help appreciated!

Thanks,
Tao

Re: Plumbing for mapping from a remote tracking ref to the remote ref?

[Junio C Hamano]

Tao Klerks <tao@klerks.biz> writes:

> Given the following configured fetch refspec for a remote:
>
> [remote "origin"]
>         url = git@someserver:somerepo.git
>         fetch = +refs/heads/*:refs/remotes/somepath/*
>
> And given a ref of the form "refs/remotes/somepath/branch_A",
>
> I'm wondering whether there is any plumbing that would be able to tell
> me what to put in a "fetch" command, to get
> "refs/remotes/somepath/branch_A" fetched - in other words, is there
> any plumbing that can use the configured fetch refspecs to map
> "refs/remotes/somepath/branch_A" to "refs/heads/branch_A" for me, so
> that I can then do "git fetch origin refs/heads/branch_A".

I am fairly certain that I never have written one myself ;-)

I wonder how the end-user experience should look like.

	$ git refmap refs/remotes/somepath/branch-A
	origin refs/heads/branch-A

	$ git refmap refs/remotes/somepath/{branch-A,branch-B}
	origin refs/heads/branch-A
	origin refs/heads/branch-B

IOW, you give name(s) of remote-tracking branches and then you get
the remote and their ref for these?

I do not oppose to such a command existing, but I do not know what
the right answer should be for a case like this:

	[remote "origin"]
		url = ... the official project repository ...
		fetch = +refs/heads/*:refs/remotes/upstream/*

	[remote "mirror"]
		url = ... a local mirror you'd use regularly ...
		fetch = +refs/heads/*:refs/remotes/upstream/*

In order to support such a "more than one can update the same" case
sensibly, the output may have to repeat the input, e.g.

	$ git refmap refs/remotes/upstream/main
	refs/remotes/upstream/main	origin refs/heads/main
	refs/remotes/upstream/main	mirror refs/heads/main

perhaps?

Re: Plumbing for mapping from a remote tracking ref to the remote ref?

[Johannes Schindelin]

Hi Junio,

On Wed, 15 Jun 2022, Junio C Hamano wrote:

> Tao Klerks <tao@klerks.biz> writes:
>
> > Given the following configured fetch refspec for a remote:
> >
> > [remote "origin"]
> >         url = git@someserver:somerepo.git
> >         fetch = +refs/heads/*:refs/remotes/somepath/*
> >
> > And given a ref of the form "refs/remotes/somepath/branch_A",
> >
> > I'm wondering whether there is any plumbing that would be able to tell
> > me what to put in a "fetch" command, to get
> > "refs/remotes/somepath/branch_A" fetched - in other words, is there
> > any plumbing that can use the configured fetch refspecs to map
> > "refs/remotes/somepath/branch_A" to "refs/heads/branch_A" for me, so
> > that I can then do "git fetch origin refs/heads/branch_A".
>
> I am fairly certain that I never have written one myself ;-)

I looked for something like that, but did not find it. We seem to have the
functions `apply_refspecs()`, `query_refspecs()` and
`remote_find_tracking()` that could be used to that end, but I do not see
any of them being used in plumbing that would expose the ref mapping in
the desired way.

> I wonder how the end-user experience should look like.
>
> 	$ git refmap refs/remotes/somepath/branch-A
> 	origin refs/heads/branch-A
>
> 	$ git refmap refs/remotes/somepath/{branch-A,branch-B}
> 	origin refs/heads/branch-A
> 	origin refs/heads/branch-B
>
> IOW, you give name(s) of remote-tracking branches and then you get
> the remote and their ref for these?

Modulo introducing a new top-level command (a subcommand of `git remote`
would make much more sense and make the feature eminently more
discoverable), and modulo allowing patterns in the ref to match, I agree.

> I do not oppose to such a command existing, but I do not know what
> the right answer should be for a case like this:
>
> 	[remote "origin"]
> 		url = ... the official project repository ...
> 		fetch = +refs/heads/*:refs/remotes/upstream/*
>
> 	[remote "mirror"]
> 		url = ... a local mirror you'd use regularly ...
> 		fetch = +refs/heads/*:refs/remotes/upstream/*
>
> In order to support such a "more than one can update the same" case
> sensibly, the output may have to repeat the input, e.g.
>
> 	$ git refmap refs/remotes/upstream/main
> 	refs/remotes/upstream/main	origin refs/heads/main
> 	refs/remotes/upstream/main	mirror refs/heads/main
>
> perhaps?

Sounds like a good plan to me.

Ciao,
Dscho

Re: Plumbing for mapping from a remote tracking ref to the remote ref?

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> 	$ git refmap refs/remotes/somepath/{branch-A,branch-B}
>> 	origin refs/heads/branch-A
>> 	origin refs/heads/branch-B
>>
>> IOW, you give name(s) of remote-tracking branches and then you get
>> the remote and their ref for these?
>
> Modulo introducing a new top-level command (a subcommand of `git remote`
> would make much more sense and make the feature eminently more
> discoverable), and modulo allowing patterns in the ref to match, I agree.

"git remote" is primarily about "I have this remote---tell me more
about it", but this query goes in the other direction, and that is
why I threw a non-existing command to solicit alternatives that are
potentially better than "git remote".

FWIW, I did not have any opinion on where the feature should appear
or what the syntax to query the remote.<nick>.fetch refmap should
be, when I wrote the above.  I still do not (yet) have a strong
opinion.

I do not oppose to the "find remotes I can fetch from to update
these remote-tracking branches" feature existing.  I just wanted to
make sure whoever will work on it is aware of the fact that they
must consider the case with overlapping destinations.

Thanks.

Re: Plumbing for mapping from a remote tracking ref to the remote ref?

[Tao Klerks]

On Sun, Jun 19, 2022 at 1:04 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> >>      $ git refmap refs/remotes/somepath/{branch-A,branch-B}
> >>      origin refs/heads/branch-A
> >>      origin refs/heads/branch-B
> >>
> >> IOW, you give name(s) of remote-tracking branches and then you get
> >> the remote and their ref for these?
> >
> > Modulo introducing a new top-level command (a subcommand of `git remote`
> > would make much more sense and make the feature eminently more
> > discoverable), and modulo allowing patterns in the ref to match, I agree.
>
> "git remote" is primarily about "I have this remote---tell me more
> about it", but this query goes in the other direction, and that is
> why I threw a non-existing command to solicit alternatives that are
> potentially better than "git remote".

Thank you for the responses here, and my apologies for not following
up (much) earlier.

Given that "git remote" already deals with different types of args
(remotes, URLs, remote branches), could it make sense to introduce a
dedicated new subcommand, not directly related to "set-branches", eg
"map-refs"? I agree with Dscho that keeping it under "git remote"
would help with discoverability and avoid clutter in the global
namespace: Git already has many top-level commands, the "theme" under
which this one fits is definitely "remote stuff", and "git remote"
already does a number of substantially-different things all related
with *remote configuration*.

In fact that's another way of seeing things: most of "git remote"'s
current subcommands are just syntactic sugar over "git config" (the
two that operate outside of the config are "set-head" and "update"),
and this new one would be config-focused in exactly the same way.

To Junio's question along the lines of "what if someone mapped
multiple remote namespaces to a single 'tracking namespace' location
in the local repo?" (which I hope is rare - I seem to recall there are
at least some operations that warn when this is detected), this
ambiguity would be absent in a "git remote" subcommand, as it would
take a remote name.

I had also considered some new weird "@{remotemapping}"-style syntax
to rev-parse, but here precisely there *would* be no implicit remote
context, and so getting more than one answer would be an option, and
that doesn't make sense for rev-parse.

Regarding patterns and wildcards, for *my* purpose at least, they
don't make much sense: The whole purpose of the exercise is to say "I
know the ref I want updated in my repo, I know what remote that it is
mapped to or that I want to update it from, I want to know exactly
what to put in a "git fetch <remote_name> <remote_ref>..." call, to
get that ref updated correctly/consistently for the current repo,
without affecting any other refs that this repo has mapped for that
remote.

Would something like the following be mutually agreeable?

       $ git remote origin map-ref
refs/remotes/my-favorite-remotes/origin/someref
      refs/heads/someref

       $ git remote origin map-ref
refs/remotes/my-favorite-remotes/origin/someref
refs/remotes/my-favorite-remotes/origin/someotherref
      refs/heads/someref
      refs/heads/someotherref

     $ git remote origin map-ref refs/remotes/someotherpath/someref
      error: the ref "refs/remotes/someotherpath/someref" is not
mapped in any configured refspec of remote "origin".


Thanks,
Tao

Re: Plumbing for mapping from a remote tracking ref to the remote ref?

[Junio C Hamano]

Tao Klerks <tao@klerks.biz> writes:

> On Sun, Jun 19, 2022 at 1:04 AM Junio C Hamano <gitster@pobox.com> wrote:
>>
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>>
>> >>      $ git refmap refs/remotes/somepath/{branch-A,branch-B}
>> >>      origin refs/heads/branch-A
>> >>      origin refs/heads/branch-B
>> >>
>> >> IOW, you give name(s) of remote-tracking branches and then you get
>> >> the remote and their ref for these?
>> >
>> > Modulo introducing a new top-level command (a subcommand of `git remote`
>> > would make much more sense and make the feature eminently more
>> > discoverable), and modulo allowing patterns in the ref to match, I agree.
>>
>> "git remote" is primarily about "I have this remote---tell me more
>> about it", but this query goes in the other direction, and that is
>> why I threw a non-existing command to solicit alternatives that are
>> potentially better than "git remote".
>
> Thank you for the responses here, and my apologies for not following
> up (much) earlier.
> ...
> Would something like the following be mutually agreeable?
>
>        $ git remote origin map-ref
> refs/remotes/my-favorite-remotes/origin/someref
>       refs/heads/someref

With strainge line wrapping, I cannot quite tell what is the input
and what is the output, but if you meant that the part up to the
long-ish refname in the refs/remotes is the command line, and map-ref
is the new subcommand name in the "git remote" command, i.e.

    $ git remote map-ref origin refs/remotes/my-favorite-remotes/origin/someref

is the input, to which the output 

    refs/heads/someref

is given, I am not sure what its value is.  First of all, the user
is giving a ref in a hierarchy that is usually used for the remote
whose name is "my-favorite-remotes".  What made this user _know_
that the remote reference belongs to "origin"?  Isn't that part of
what the user may want to _find_ _out_, instead of required to give
as input?

So, no, I do not think it is agreeable at least not from this end,
but I may be misunderstanding what you meant to present as your
design.

I would understand if it were more like

    $ git remote refmap refs/remotes/somepath/{branch-A,branch-B}
    origin:refs/heads/branch-A refs/remotes/somepath/branch-A
    origin:refs/heads/branch-B refs/remotes/somepath/branch-B

that is,

 (1) the new subcommand (refmap) takes one or more refs from the
     command line; they typically are in the refs/remotes hiearchy
     and each asks which remote's which ref needs to be fetched to
     update the ref.  Note that the user does *not* need to know
     which remote the refs will be updated from.

 (2) the subcommand goes through the "remote.*.fetch" configuration
     items (and its older equivalents in .git/remotes, whose support
     should come free if you used remote.c API properly) to find
     what ref from what remote is fetched to update the refs given
     from the command line.

 (3) the output is "<remote>:<ref-at-remote> <our-remote-tracking-branch>"
     one line at a time.

Note that this format allows the "two remotes can both update the
same remote tracking branches we have" arrangement.

Re: Plumbing for mapping from a remote tracking ref to the remote ref?

[Tao Klerks]

On Wed, Sep 6, 2023 at 12:18 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Tao Klerks <tao@klerks.biz> writes:
>
> > ...
> > Would something like the following be mutually agreeable?
> >
> >        $ git remote origin map-ref
> > refs/remotes/my-favorite-remotes/origin/someref
> >       refs/heads/someref
>
> With strainge line wrapping, I cannot quite tell what is the input
> and what is the output, but if you meant that the part up to the
> long-ish refname in the refs/remotes is the command line, and map-ref
> is the new subcommand name in the "git remote" command, i.e.
>
>     $ git remote map-ref origin refs/remotes/my-favorite-remotes/origin/someref
>
> is the input, to which the output
>
>     refs/heads/someref
>
> is given,

My apologies: in addition to automatic line wrapping, I also got the
arg order wrong.

Yes, this is what I meant.

> I am not sure what its value is.  First of all, the user
> is giving a ref in a hierarchy that is usually used for the remote
> whose name is "my-favorite-remotes".  What made this user _know_
> that the remote reference belongs to "origin"?

Understanding that it's dangerous to make assumptions about what's
typical, I am positing that the user typically knows what remote
they're working with / looking for stuff in. I would guess that the
set of repos, in the world, that have multiple remotes with different
ref path structures, mapped onto the same remote tracking ref
namespace, is much smaller than the set of repos that have some set of
refs mapped differently to the standard
"refs/heads/*:refs/remotes/originname/*" mapping. My "selfish" intent
was to address the latter, without worrying much about the former.

> Isn't that part of
> what the user may want to _find_ _out_, instead of required to give
> as input?

There's certainly value in enabling them to do so!

>
> So, no, I do not think it is agreeable at least not from this end,
> but I may be misunderstanding what you meant to present as your
> design.

No misunderstanding. I was unfortunately more concerned with "fitting
in" with other "git remote" subcommands (which take a remote name)
than making the most useful functionality.

>
> I would understand if it were more like
>
>     $ git remote refmap refs/remotes/somepath/{branch-A,branch-B}
>     origin:refs/heads/branch-A refs/remotes/somepath/branch-A
>     origin:refs/heads/branch-B refs/remotes/somepath/branch-B
>

This is a much better proposal, from my perspective!

> that is,
>
>  (1) the new subcommand (refmap) takes one or more refs from the
>      command line; they typically are in the refs/remotes hiearchy
>      and each asks which remote's which ref needs to be fetched to
>      update the ref.  Note that the user does *not* need to know
>      which remote the refs will be updated from.
>
>  (2) the subcommand goes through the "remote.*.fetch" configuration
>      items (and its older equivalents in .git/remotes, whose support
>      should come free if you used remote.c API properly) to find
>      what ref from what remote is fetched to update the refs given
>      from the command line.
>
>  (3) the output is "<remote>:<ref-at-remote> <our-remote-tracking-branch>"
>      one line at a time.
>
> Note that this format allows the "two remotes can both update the
> same remote tracking branches we have" arrangement.
>