`git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

`git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Han Jiang]

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

cd '/'; cd '/'; rm --force --recursive -- './test_git'; mkdir "$_"; cd "$_";
mkdir --parents -- './server' './client';
git -C './server' init --bare './repo.git'
git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
git -C './server/repo.git' branch branch1; git -C './server/repo.git'
branch branch2

git -C './client' init './repo'
git -C './client/repo' remote add server 'file://'"$(realpath
'./server/repo.git')"
git -C './client/repo' remote --verbose
git -C './client/repo' config list --local --show-scope --show-origin
git -C './client/repo' fetch --dry-run
git -C './client/repo' fetch --dry-run
--refmap='+refs/heads/*:refs/remotes/server/*' server branch1 branch2
git -C './client/repo' fetch --dry-run
--refmap='+refs/heads/branch*:refs/remotes/server/branch*' server
branch1 branch2
git -C './client/repo' fetch --dry-run
--refmap='+refs/heads/branch1:refs/remotes/server/branch1' server
branch1 branch2
git -C './client/repo' fetch
--refmap='+refs/heads/branch1:refs/remotes/server/branch1' server
branch1 branch2
git -C './client/repo' branch --list --all --verbose --verbose

What did you expect to happen? (Expected behavior)

In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
server branch1 branch2`,
`remote.server.fetch=+refs/heads/*:refs/remotes/server/*` merges with
`--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
so these should be what would be done:
branch1 -> server/branch1
branch2 -> server/branch2

What happened instead? (Actual behavior)

In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
server branch1 branch2`,
`remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
so these are what would be done:
branch1 -> server/branch1

What's different between what you expected and what actually happened?

Anything else you want to add:

Please review the rest of the bug report below.
You can delete any lines you don't wish to share.


[System Info]
git version:
git version 2.46.0.windows.1
cpu: x86_64
built from commit: 2e6a859ffc0471f60f79c1256f766042b0d5d17d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.9.0
OpenSSL: OpenSSL 3.2.2 4 Jun 2024
zlib: 1.3.1
uname: Windows 10.0 22631
compiler info: gnuc: 14.1
libc info: no libc information available
$SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe


[Enabled Hooks]
not run from a git repository - no hooks to show

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Johannes Sixt]

Am 04.09.24 um 06:09 schrieb Han Jiang:
> What did you expect to happen? (Expected behavior)
> 
> In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> server branch1 branch2`,
> `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` merges with
> `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,

The way I read the documentation of --refmap, I cannot infer any sort of
merging of refspec, only that a replacement occurs.

> so these should be what would be done:
> branch1 -> server/branch1
> branch2 -> server/branch2
> 
> What happened instead? (Actual behavior)
> 
> In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> server branch1 branch2`,
> `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> so these are what would be done:
> branch1 -> server/branch1

So, this seems the correct output, because it is no longer specified how
branch2 should be stored locally.

-- Hannes

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Han Jiang]

1. If the doc says "providing an empty <refspec> to the --refmap
option causes Git to ignore the configured refspecs and rely entirely
on the refspecs supplied as command-line arguments" then it's
reasonable to guess that "providing a non-empty <refspec> will not do
those". (Otherwise the doc could just say "providing a <refspec> will
do those" without the word "empty".) This is both logical fallacy and
common sense.
2. Searching "empty string" "empty value" in
https://git-scm.com/docs/git-config shows that using empty value to
clear lower priority configurations (and non-empty values just stack
up) is common behavior of multi-valued configuration variables.

On Wed, Sep 4, 2024 at 6:15 PM Johannes Sixt <j6t@kdbg.org> wrote:
>
> Am 04.09.24 um 06:09 schrieb Han Jiang:
> > What did you expect to happen? (Expected behavior)
> >
> > In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> > server branch1 branch2`,
> > `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` merges with
> > `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
>
> The way I read the documentation of --refmap, I cannot infer any sort of
> merging of refspec, only that a replacement occurs.
>
> > so these should be what would be done:
> > branch1 -> server/branch1
> > branch2 -> server/branch2
> >
> > What happened instead? (Actual behavior)
> >
> > In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> > server branch1 branch2`,
> > `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> > by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> > so these are what would be done:
> > branch1 -> server/branch1
>
> So, this seems the correct output, because it is no longer specified how
> branch2 should be stored locally.
>
> -- Hannes
>

Re: `git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Junio C Hamano]

Han Jiang <jhcarl0814@gmail.com> writes:

> What happened instead? (Actual behavior)
>
> In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> server branch1 branch2`,
> `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> so these are what would be done:
> branch1 -> server/branch1

This is totally expected.

A refspec specifies "what to fetch" and "where to store locally what
we fetched" at the same time.  A refmap only specifies the "where
the stuff is stored locally" part, and what to fetch is specified
separately.  You asked branch1 and branch2 to be fetched (peek into
the .git/FETCH_HEAD file to see both objects are fetched) from the
command line arguments, and your refmap specified where to store
what you got from their refs/heads/branch1.  As the refmap does not
say anything about any other branches, they are not stored anywhere
in your local refs/ namespace.

Typically --refmap is given with a glob, but the story is the same
there.  If you fetched refs/notes/commits from that same server with
a refmap to map "refs/heads/*:refs/remotes/server/*", the notes commit
is not mapped to anywhere with the refmap, so nothing will happen to
it.

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Johannes Sixt]

Am 04.09.24 um 13:12 schrieb Han Jiang:
> 1. If the doc says "providing an empty <refspec> to the --refmap
> option causes Git to ignore the configured refspecs and rely entirely
> on the refspecs supplied as command-line arguments" then it's
> reasonable to guess that "providing a non-empty <refspec> will not do
> those".

Fair enough. But then in section "Configured Remote-tracking Branches"
it is said that "The latter use of the remote.<repository>.fetch values
can be overridden by giving the --refmap=<refspec> parameter(s) on the
command line." So... I dunno. Just my 2c.

-- Hannes

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Junio C Hamano]

Johannes Sixt <j6t@kdbg.org> writes:

> Am 04.09.24 um 13:12 schrieb Han Jiang:
>> 1. If the doc says "providing an empty <refspec> to the --refmap
>> option causes Git to ignore the configured refspecs and rely entirely
>> on the refspecs supplied as command-line arguments" then it's
>> reasonable to guess that "providing a non-empty <refspec> will not do
>> those".
>
> Fair enough. But then in section "Configured Remote-tracking Branches"
> it is said that "The latter use of the remote.<repository>.fetch values
> can be overridden by giving the --refmap=<refspec> parameter(s) on the
> command line." So... I dunno. Just my 2c.

This cuts both ways, I suspect.  It is so unusual to say

    git fetch --refmap="" foo bar baz

to use an empty string as <refspec> in "--refmap=<refspec>", simply
because you would not say

    git fetch origin ""

and expect the empty string is a <refspec> in "git fetch origin <refspec>".

Any string other than an empty string, e.g. 

    git fetch --refmap="refs/heads/*:refs/prefetch/x/*" foo bar baz

would have an extra side effect of storing what gets fetched, when
your primary focus is "I want the command to ignore the configured
refspec".

For these reasons, I would actually say it is quite fair for the
documentation to single out the empty string case as an idiom that
may be hard for readers to figure out for themselves without being
told.

I would not be opposed to add a sentence or two to clarify what
happens when a non-empty <refspec> is used, though.

Thanks.

Re: `git fetch --refmap=<refspec>… <repository> <refspec>…​` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Han Jiang]

@Junio C Hamano Thank you for the explanation!
As you said, "refmap only specifies the where the stuff is stored
locally" (what to fetch is specified by last command line arguments),
but the issue with this report is a bit complicated because it's about
whether "only refmap specifies the where the stuff is stored locally".
After I search "empty value" or "empty string" in
https://git-scm.com/docs/git-config , the most intuitive behavior of
`git fetch --refmap` (for me) is:
`git fetch <remote> <branch>...` uses `remote.<remote>.fetch`;
`git fetch --refmap=<refspec> --refmap=<refspec> <remote> <branch>...`
uses `remote.<remote>.fetch` and all the `--refmap=<refspec>`s;
`git fetch --refmap='' --refmap=<refspec> --refmap=<refspec> <remote>
<branch>...` uses all the `--refmap=<refspec>`s;
I'll post an example of all of these being used at the same time to both of you.

On Thu, Sep 5, 2024 at 2:34 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Han Jiang <jhcarl0814@gmail.com> writes:
>
> > What happened instead? (Actual behavior)
> >
> > In `git fetch --refmap='+refs/heads/branch1:refs/remotes/server/branch1'
> > server branch1 branch2`,
> > `remote.server.fetch=+refs/heads/*:refs/remotes/server/*` is replaced
> > by `--refmap='+refs/heads/branch1:refs/remotes/server/branch1'`,
> > so these are what would be done:
> > branch1 -> server/branch1
>
> This is totally expected.
>
> A refspec specifies "what to fetch" and "where to store locally what
> we fetched" at the same time.  A refmap only specifies the "where
> the stuff is stored locally" part, and what to fetch is specified
> separately.  You asked branch1 and branch2 to be fetched (peek into
> the .git/FETCH_HEAD file to see both objects are fetched) from the
> command line arguments, and your refmap specified where to store
> what you got from their refs/heads/branch1.  As the refmap does not
> say anything about any other branches, they are not stored anywhere
> in your local refs/ namespace.
>
> Typically --refmap is given with a glob, but the story is the same
> there.  If you fetched refs/notes/commits from that same server with
> a refmap to map "refs/heads/*:refs/remotes/server/*", the notes commit
> is not mapped to anywhere with the refmap, so nothing will happen to
> it.
>
>

Re: `git fetch --refmap=<refspec>… <repository> <refspec>… ` providing NON-empty <refspec> to the --refmap ALSO causes Git to ignore the configured refspecs

[Han Jiang]

> But then in section "Configured Remote-tracking Branches" it is said that
The problem with the doc is "how to override". From
https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---refmapltrefspecgt
it seems we need to `git fetch --refmap='' --refmap=<refspec>` to
really override configuration variables.
I guess the inner working is like this: first for-loop traverses
configuration variable `remote.<remote>.fetch` collecting <refspec>s
into a list, then second for-loop traverses `--refmap=<refspec>`s,
clears the list when an empty string is encountered, appends to the
list when a non-empty string is encountered. Maybe I'm just
daydreaming...

> git fetch --refmap="" foo bar baz
I think the correct use case of empty string, is to specify more
--refmap=<refspec>s after it (like in
https://git-scm.com/docs/git-config#Documentation/git-config.txt-pushpushOption
), otherwise the mapping (to operate upon foo,bar and baz) would be
empty.

Sorry that I didn't use examples to express my intended use scenarios clearly!

After I searched "empty value" or "empty string" in
https://git-scm.com/docs/git-config yesterday, the most intuitive
behavior of `git fetch --refmap` (for me) was:
`git fetch <remote> <branch>...` uses `remote.<remote>.fetch`;
`git fetch --refmap=<refspec> --refmap=<refspec> <remote> <branch>...`
uses `remote.<remote>.fetch` and all the `--refmap=<refspec>`s;
`git fetch --refmap='' --refmap=<refspec> --refmap=<refspec> <remote>
<branch>...` uses all the `--refmap=<refspec>`s;

Here is an example of all of these being used at the same time.
1. Normally, user uses `fetch-safe` which uses `git fetch
--refmap=<refspec>` to use whatever mappings specified by
`remote.<remote>.fetch` and also let our backup logic sneak in.
2. Occasionally, upstream is rebased and our backup logic (refspec
without `+`) blocks the process because of `--atomic`.
2.1. When that happens, user uses `fetch-bak-current-time` which uses
`git fetch --refmap='' --refmap=<refspec>` to clear whatever mappings
specified by `remote.<remote>.fetch` and use our another backup logic
(refspec with `+`).
2.2. After old tips are saved, user uses `git fetch --force` to use
whatever mappings specified by `remote.<remote>.fetch`;

Currently [1.] is not working because --refmap=<not empty string!>
clears `remote.<remote>.fetch` instead of merging with it. Hence this
bug report.

cd '/'; cd '/'; rm --force --recursive -- './test_git2'; mkdir "$_"; cd "$_";
mkdir --parents -- './server' './client';

git -C './server' init --bare './repo.git'
git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
git -C './server/repo.git' branch branch1; git -C './server/repo.git'
branch branch2

git -C './client' init './repo'
git -C './client/repo' config set --local 'alias.fetch-safe'
'!git_fetch_safe(){ server="$1"; shift; git fetch --atomic
--refmap='\''refs/*:refs/namespaces/'\''"$server"'\''_backup/refs/*'\''
"$server" "$@"; };git_fetch_safe'
git -C './client/repo' config set --local
'alias.fetch-bak-current-time' '!git_fetch_bak_current_time(){
server="$1"; shift; git -c
'\''remote.'\''"$server"'\''_backup.url=.'\''
--namespace="$server"'\''_backup'\'' fetch
--refmap='\''refs/*:refs/namespaces/'\''"$server"'\''_backup_'\''"$(date
+'%Y%m%d%H%M%S')"'\''/refs/*'\'' "$server"'_backup' "$@"; git fetch
--atomic --refmap='\'\''
--refmap='\''+refs/*:refs/namespaces/'\''"$server"'\''_backup/refs/*'\''
"$server" "$@"; };git_fetch_bak_current_time'
git -C './client/repo' remote add server 'file://'"$(realpath
'./server/repo.git')"
git -C './client/repo' remote --verbose
git -C './client/repo' config list --local --show-scope --show-origin
git -C './client/repo' fetch-safe server branch1 branch2
git -C './client/repo' show-ref
git -C './client/repo' lg --all --decorate-refs='*'

git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))"
git -C './server/repo.git' branch --force branch1; git -C
'./server/repo.git' branch --force branch2
git -C './client/repo' fetch-safe server branch1 branch2
git -C './client/repo' show-ref
git -C './client/repo' lg --all --decorate-refs='*'

git --git-dir='./server/repo.git' --work-tree='.' commit --allow-empty
-m "$((++number))" --amend
git -C './server/repo.git' branch --force branch1; git -C
'./server/repo.git' branch --force branch2
git -C './client/repo' fetch-safe server branch1 branch2
git -C './client/repo' show-ref
git -C './client/repo' lg --all --decorate-refs='*'
git -C './client/repo' fetch-bak-current-time server branch1 branch2
git -C './client/repo' show-ref
git -C './client/repo' lg --all --decorate-refs='*'
git -C './client/repo' fetch --force server branch1 branch2
git -C './client/repo' show-ref
git -C './client/repo' lg --all --decorate-refs='*'

On Thu, Sep 5, 2024 at 4:47 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Johannes Sixt <j6t@kdbg.org> writes:
>
> > Am 04.09.24 um 13:12 schrieb Han Jiang:
> >> 1. If the doc says "providing an empty <refspec> to the --refmap
> >> option causes Git to ignore the configured refspecs and rely entirely
> >> on the refspecs supplied as command-line arguments" then it's
> >> reasonable to guess that "providing a non-empty <refspec> will not do
> >> those".
> >
> > Fair enough. But then in section "Configured Remote-tracking Branches"
> > it is said that "The latter use of the remote.<repository>.fetch values
> > can be overridden by giving the --refmap=<refspec> parameter(s) on the
> > command line." So... I dunno. Just my 2c.
>
> This cuts both ways, I suspect.  It is so unusual to say
>
>     git fetch --refmap="" foo bar baz
>
> to use an empty string as <refspec> in "--refmap=<refspec>", simply
> because you would not say
>
>     git fetch origin ""
>
> and expect the empty string is a <refspec> in "git fetch origin <refspec>".
>
> Any string other than an empty string, e.g.
>
>     git fetch --refmap="refs/heads/*:refs/prefetch/x/*" foo bar baz
>
> would have an extra side effect of storing what gets fetched, when
> your primary focus is "I want the command to ignore the configured
> refspec".
>
> For these reasons, I would actually say it is quite fair for the
> documentation to single out the empty string case as an idiom that
> may be hard for readers to figure out for themselves without being
> told.
>
> I would not be opposed to add a sentence or two to clarify what
> happens when a non-empty <refspec> is used, though.
>
> Thanks.
>