git init BUG when gitconfig has includeIf

git init BUG when gitconfig has includeIf

[Heghedus Razvan]

Hello everybody

Yesterday I stumble upon a bug when doing git init. I didn't
find any references to it, so I don't know if is a known problem
or not.

Steps to reproduce:
# git init .
BUG: refs.c:2123: reference backend is unknown

I'm able to reproduce this on multiple systems where the git
version is newer than v2.44.0. This happens only when I have
an "includeIf" in the ~/.gitconfig . Bellow [1] I have a
minimal configuration for reproducing the bug. With same
gitconfig but versions lower than v2.43.4 the error is
no longer happening.

So I went and did a git bisection and the error appeared with
173761e21b setup: start tracking ref storage format
Previous commit ("0fcc285c5e") is working fine.

I cc'ed Patrick Steinhardt since the error appeared from his
commit.

Best regards,
Razvan

--

1:
content of ~/.gitconfig
```
[user]
        email = some_email@test.net
        name = test
[includeIf "onbranch:upstream*"]
        path=~/.gitconfig_upstream
```

content of ~/.gitconfig_upstream
```
[user]
        name = test test
```

Re: git init BUG when gitconfig has includeIf

[Junio C Hamano]

Heghedus Razvan <heghedus.razvan@protonmail.com> writes:

> Yesterday I stumble upon a bug when doing git init. I didn't
> find any references to it, so I don't know if is a known problem
> or not.
>
> Steps to reproduce:
> # git init .
> BUG: refs.c:2123: reference backend is unknown

Patrick, this looks similar to an earlier one during "git clone"
that was discussed at

https://lore.kernel.org/git/72771da0-a0ef-4fd9-8071-6467cd7b6a6b@kernel-space.org/

that was fixed with 199f44cb (builtin/clone: allow remote helpers to
detect repo, 2024-02-27)?  The fix was about "git clone", but the
crux of the fix went to setup.c:initialize_repository_version()
which is also called by setup.c:init_db() that is the workhorse of
"git init", so it may already have been fixed (I didn't try).

Even if it is already fixed in the current version by the same
199f44cb, we may want to follow up 0eab85b9 (t5601: exercise clones
with "includeIf.*.onbranch", 2024-03-12) with an additional test to
cover "git init".

Thanks.

Re: git init BUG when gitconfig has includeIf

[Heghedus Razvan]

On Tue May 21, 2024 at 7:36 PM EEST, Junio C Hamano wrote:
> Heghedus Razvan <heghedus.razvan@protonmail.com> writes:
>
> > Yesterday I stumble upon a bug when doing git init. I didn't
> > find any references to it, so I don't know if is a known problem
> > or not.
> >
> > Steps to reproduce:
> > # git init .
> > BUG: refs.c:2123: reference backend is unknown
>
> Patrick, this looks similar to an earlier one during "git clone"
> that was discussed at
>
> https://lore.kernel.org/git/72771da0-a0ef-4fd9-8071-6467cd7b6a6b@kernel-space.org/
>
> that was fixed with 199f44cb (builtin/clone: allow remote helpers to
> detect repo, 2024-02-27)?  The fix was about "git clone", but the
> crux of the fix went to setup.c:initialize_repository_version()
> which is also called by setup.c:init_db() that is the workhorse of
> "git init", so it may already have been fixed (I didn't try).

I guess I forgot to mention, but I tested the current master 4365c6fcf9
and the issue is still present.

>
> Even if it is already fixed in the current version by the same
> 199f44cb, we may want to follow up 0eab85b9 (t5601: exercise clones
> with "includeIf.*.onbranch", 2024-03-12) with an additional test to
> cover "git init".
>
> Thanks.

Re: git init BUG when gitconfig has includeIf

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2653 bytes --]

On Tue, May 21, 2024 at 04:46:23PM +0000, Heghedus Razvan wrote:
> On Tue May 21, 2024 at 7:36 PM EEST, Junio C Hamano wrote:
> > Heghedus Razvan <heghedus.razvan@protonmail.com> writes:
> >
> > > Yesterday I stumble upon a bug when doing git init. I didn't
> > > find any references to it, so I don't know if is a known problem
> > > or not.
> > >
> > > Steps to reproduce:
> > > # git init .
> > > BUG: refs.c:2123: reference backend is unknown
> >
> > Patrick, this looks similar to an earlier one during "git clone"
> > that was discussed at
> >
> > https://lore.kernel.org/git/72771da0-a0ef-4fd9-8071-6467cd7b6a6b@kernel-space.org/
> >
> > that was fixed with 199f44cb (builtin/clone: allow remote helpers to
> > detect repo, 2024-02-27)?  The fix was about "git clone", but the
> > crux of the fix went to setup.c:initialize_repository_version()
> > which is also called by setup.c:init_db() that is the workhorse of
> > "git init", so it may already have been fixed (I didn't try).
> 
> I guess I forgot to mention, but I tested the current master 4365c6fcf9
> and the issue is still present.

I cannot reproduce the issue as-is, neither on Git v2.44 nor on the
current master branch. So clearly, there must be something special to
your setup. The following testcase and variants of it do not reproduce:

    test_expect_success 'init with includeIf.onbranch condition' '
        git config -f ./config foo.bar baz &&
        include=$(test-tool path-utils absolute_path config) &&
        test_when_finished "rm -rf repo" &&
        git -c includeIf.onbranch:main.path="$(<include)" init repo
    '

Now digging into the code, the condition gets evaluated in
`include_by_branch()`. The call to `refs_resolve_ref_unsafe()` is
guarded by `the_repository->gitdir`, which is `NULL` the first time it
is called by git-init(1). It does get called a second time, but at that
point we already initialized the refdb and configured the ref storage
format as expected.

Aha! Seems like this only happens when re-initializing an already
existent repository, that's what's missing. In that case we do already
have `the_repository->gitdir` set even though we did not yet set up the
ref storage format. I'll investigate and send a patch.

Can you confirm that this is what you see, or do you also see this when
creating an entirely new repository?

> > Even if it is already fixed in the current version by the same
> > 199f44cb, we may want to follow up 0eab85b9 (t5601: exercise clones
> > with "includeIf.*.onbranch", 2024-03-12) with an additional test to
> > cover "git init".

Agreed.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: git init BUG when gitconfig has includeIf

[Heghedus Razvan]

On Wed May 22, 2024 at 11:06 AM EEST, Patrick Steinhardt wrote:
> On Tue, May 21, 2024 at 04:46:23PM +0000, Heghedus Razvan wrote:
> > On Tue May 21, 2024 at 7:36 PM EEST, Junio C Hamano wrote:
> > > Heghedus Razvan <heghedus.razvan@protonmail.com> writes:
> > >
> > > > Yesterday I stumble upon a bug when doing git init. I didn't
> > > > find any references to it, so I don't know if is a known problem
> > > > or not.
> > > >
> > > > Steps to reproduce:
> > > > # git init .
> > > > BUG: refs.c:2123: reference backend is unknown
> > >
> > > Patrick, this looks similar to an earlier one during "git clone"
> > > that was discussed at
> > >
> > > https://lore.kernel.org/git/72771da0-a0ef-4fd9-8071-6467cd7b6a6b@kernel-space.org/
> > >
> > > that was fixed with 199f44cb (builtin/clone: allow remote helpers to
> > > detect repo, 2024-02-27)?  The fix was about "git clone", but the
> > > crux of the fix went to setup.c:initialize_repository_version()
> > > which is also called by setup.c:init_db() that is the workhorse of
> > > "git init", so it may already have been fixed (I didn't try).
> > 
> > I guess I forgot to mention, but I tested the current master 4365c6fcf9
> > and the issue is still present.
>
> I cannot reproduce the issue as-is, neither on Git v2.44 nor on the
> current master branch. So clearly, there must be something special to
> your setup. The following testcase and variants of it do not reproduce:
>
>     test_expect_success 'init with includeIf.onbranch condition' '
>         git config -f ./config foo.bar baz &&
>         include=$(test-tool path-utils absolute_path config) &&
>         test_when_finished "rm -rf repo" &&
>         git -c includeIf.onbranch:main.path="$(<include)" init repo
>     '
>
> Now digging into the code, the condition gets evaluated in
> `include_by_branch()`. The call to `refs_resolve_ref_unsafe()` is
> guarded by `the_repository->gitdir`, which is `NULL` the first time it
> is called by git-init(1). It does get called a second time, but at that
> point we already initialized the refdb and configured the ref storage
> format as expected.
>
> Aha! Seems like this only happens when re-initializing an already
> existent repository, that's what's missing. In that case we do already
> have `the_repository->gitdir` set even though we did not yet set up the
> ref storage format. I'll investigate and send a patch.
>
> Can you confirm that this is what you see, or do you also see this when
> creating an entirely new repository?

Hi Patrick,

Thanks for looking into this. It seems that the issue happens only when
the folder for the repo exists.

Eg:
$ mkdir new_folder
$ cd new_folder
$ git init . 

or
$ mkdir new_folder
$ git init new_folder

But directly running `git init new_folder` when there is no `new_folder`
works fine.

FYI, I just did these tests on master (4365c6fcf9).

Regards,
Razvan

>
> > > Even if it is already fixed in the current version by the same
> > > 199f44cb, we may want to follow up 0eab85b9 (t5601: exercise clones
> > > with "includeIf.*.onbranch", 2024-03-12) with an additional test to
> > > cover "git init".
>
> Agreed.
>
> Patrick

Re: git init BUG when gitconfig has includeIf

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2176 bytes --]

On Wed, May 22, 2024 at 08:21:50AM +0000, Heghedus Razvan wrote:
> On Wed May 22, 2024 at 11:06 AM EEST, Patrick Steinhardt wrote:
> > On Tue, May 21, 2024 at 04:46:23PM +0000, Heghedus Razvan wrote:
> > > On Tue May 21, 2024 at 7:36 PM EEST, Junio C Hamano wrote:
> > > > Heghedus Razvan <heghedus.razvan@protonmail.com> writes:
> > Aha! Seems like this only happens when re-initializing an already
> > existent repository, that's what's missing. In that case we do already
> > have `the_repository->gitdir` set even though we did not yet set up the
> > ref storage format. I'll investigate and send a patch.
> >
> > Can you confirm that this is what you see, or do you also see this when
> > creating an entirely new repository?
> 
> Hi Patrick,
> 
> Thanks for looking into this. It seems that the issue happens only when
> the folder for the repo exists.
> 
> Eg:
> $ mkdir new_folder
> $ cd new_folder
> $ git init . 
> 
> or
> $ mkdir new_folder
> $ git init new_folder
> 
> But directly running `git init new_folder` when there is no `new_folder`
> works fine.
> 
> FYI, I just did these tests on master (4365c6fcf9).
> 
> Regards,
> Razvan

Indeed, thanks for clarifying. From the following three tests, the
latter two fail:

diff --git a/t/t0001-init.sh b/t/t0001-init.sh
index b131d665db..c1c7c307d3 100755
--- a/t/t0001-init.sh
+++ b/t/t0001-init.sh
@@ -678,4 +678,21 @@ test_expect_success 'branch -m with the initial branch' '
    test_cmp expect actual
 '
 
+test_expect_success 'init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git -c includeIf.onbranch:main.path=something init repo
+'
+
+test_expect_success 'init with includeIf.onbranch condition with existing directory' '
+	test_when_finished "rm -rf repo" &&
+	mkdir repo &&
+	git -c includeIf.onbranch:main.path=something init repo
+'
+
+test_expect_success 're-init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	git -c includeIf.onbranch:main.path=something init repo
+'
+
 test_done

I've got a fix ready that I'll send upstream later today.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH] setup: fix bug with "includeIf.onbranch" when initializing dir

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 8056 bytes --]

It was reported that git-init(1) can fail when initializing an existing
directory in case the config contains an "includeIf.onbranch:"
condition:

```shell
$ mkdir repo
$ git -c includeIf.onbranch:main.path=nonexistent init repo
BUG: refs.c:2056: reference backend is unknown
```

The same error can also be triggered when re-initializing an already
existing repository.

The bug has been introduced in 173761e21b (setup: start tracking ref
storage format, 2023-12-29), which wired up the ref storage format. The
root cause is in `init_db()`, which tries to read the config before we
have initialized `the_repository` and most importantly its ref storage
format. We eventually end up calling `include_by_branch()` and execute
`refs_resolve_ref_unsafe()`, but because we have not initialized the ref
storage format yet this will trigger the above bug.

Interestingly, `include_by_branch()` has a mechanism that will only
cause us to resolve the ref when `the_repository->gitdir` is set. This
is also the reason why this only happens when we initialize an already
existing directory or repository: `gitdir` is set in those cases, but
not when creating a new directory.

Now there are two ways to address the issue:

  - We can adapt `include_by_branch()` to also make the code conditional
    on whether `the_repository->ref_storage_format` is set.

  - We can shift around code such that we initialize the repository
    format before we read the config.

While the first approach would be safe, it may also cause us to paper
over issues where a ref store should have been set up. In our case for
example, it may be reasonable to expect that re-initializing the repo
will cause the "onbranch:" condition to trigger, but we would not do
that if the ref storage format was not set up yet. This also used to
work before the above commit that introduced this bug.

Rearrange the code such that we set up the repository format before
reading the config. This fixes the bug and ensures that "onbranch:"
conditions can trigger.

Reported-by: Heghedus Razvan <heghedus.razvan@protonmail.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 setup.c         |  21 +++++-----
 t/t0001-init.sh | 100 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 104 insertions(+), 17 deletions(-)

diff --git a/setup.c b/setup.c
index 9247cded6a..8c7bc7bdb1 100644
--- a/setup.c
+++ b/setup.c
@@ -2286,12 +2286,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	}
 	startup_info->have_repository = 1;
 
-	/* Ensure `core.hidedotfiles` is processed */
-	git_config(platform_core_config, NULL);
-
-	safe_create_dir(git_dir, 0);
-
-
 	/* Check to see if the repository version is right.
 	 * Note that a newly created repository does not have
 	 * config file, so this will not fail.  What we are catching
@@ -2302,9 +2296,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	validate_hash_algorithm(&repo_fmt, hash);
 	validate_ref_storage_format(&repo_fmt, ref_storage_format);
 
-	reinit = create_default_files(template_dir, original_git_dir,
-				      &repo_fmt, init_shared_repository);
-
 	/*
 	 * Now that we have set up both the hash algorithm and the ref storage
 	 * format we can update the repository's settings accordingly.
@@ -2312,6 +2303,18 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
 	repo_set_ref_storage_format(the_repository, repo_fmt.ref_storage_format);
 
+	/*
+	 * Ensure `core.hidedotfiles` is processed. This must happen after we
+	 * have set up the repository format such that we can evaluate
+	 * includeIf conditions correctly in the case of re-initialization.
+	 */
+	git_config(platform_core_config, NULL);
+
+	safe_create_dir(git_dir, 0);
+
+	reinit = create_default_files(template_dir, original_git_dir,
+				      &repo_fmt, init_shared_repository);
+
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo_fmt.ref_storage_format,
 					  initial_branch, flags & INIT_DB_QUIET);
diff --git a/t/t0001-init.sh b/t/t0001-init.sh
index b131d665db..2239bed198 100755
--- a/t/t0001-init.sh
+++ b/t/t0001-init.sh
@@ -584,14 +584,38 @@ test_expect_success 'init with --ref-format=files' '
 	test_cmp expect actual
 '
 
-test_expect_success 're-init with same format' '
-	test_when_finished "rm -rf refformat" &&
-	git init --ref-format=files refformat &&
-	git init --ref-format=files refformat &&
-	echo files >expect &&
-	git -C refformat rev-parse --show-ref-format >actual &&
-	test_cmp expect actual
-'
+for from_format in files reftable
+do
+	for to_format in files reftable
+	do
+		if test "$from_format" = "$to_format"
+		then
+			continue
+		fi
+
+		test_expect_success "re-init with same format ($from_format)" '
+			test_when_finished "rm -rf refformat" &&
+			git init --ref-format=$from_format refformat &&
+			git init --ref-format=$from_format refformat &&
+			echo $from_format >expect &&
+			git -C refformat rev-parse --show-ref-format >actual &&
+			test_cmp expect actual
+		'
+
+		test_expect_success "re-init with different format fails ($from_format -> $to_format)" '
+			test_when_finished "rm -rf refformat" &&
+			git init --ref-format=$from_format refformat &&
+			cat >expect <<-EOF &&
+			fatal: attempt to reinitialize repository with different reference storage format
+			EOF
+			test_must_fail git init --ref-format=$to_format refformat 2>err &&
+			test_cmp expect err &&
+			echo $from_format >expect &&
+			git -C refformat rev-parse --show-ref-format >actual &&
+			test_cmp expect actual
+		'
+	done
+done
 
 test_expect_success 'init with --ref-format=garbage' '
 	test_when_finished "rm -rf refformat" &&
@@ -678,4 +702,64 @@ test_expect_success 'branch -m with the initial branch' '
 	test_cmp expect actual
 '
 
+test_expect_success 'init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git -c includeIf.onbranch:main.path=nonexistent init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'init with includeIf.onbranch condition with existing directory' '
+	test_when_finished "rm -rf repo" &&
+	mkdir repo &&
+	git -c includeIf.onbranch:nonexistent.path=/does/not/exist init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 're-init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	git -c includeIf.onbranch:nonexistent.path=/does/not/exist init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 're-init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	git -c includeIf.onbranch:nonexistent.path=/does/not/exist init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 're-init skips non-matching includeIf.onbranch' '
+	test_when_finished "rm -rf repo config" &&
+	cat >config <<-EOF &&
+	[
+	garbage
+	EOF
+	git init repo &&
+	git -c includeIf.onbranch:nonexistent.path="$(test-tool path-utils absolute_path config)" init repo
+'
+
+test_expect_success 're-init reads matching includeIf.onbranch' '
+	test_when_finished "rm -rf repo config" &&
+	cat >config <<-EOF &&
+	[
+	garbage
+	EOF
+	path="$(test-tool path-utils absolute_path config)" &&
+	git init --initial-branch=branch repo &&
+	cat >expect <<-EOF &&
+	fatal: bad config line 1 in file $path
+	EOF
+	test_must_fail git -c includeIf.onbranch:branch.path="$path" init repo 2>err &&
+	test_cmp expect err
+'
+
 test_done
-- 
2.45.1.216.g4365c6fcf9.dirty


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH] setup: fix bug with "includeIf.onbranch" when initializing dir

[Heghedus Razvan]

On Wed May 22, 2024 at 1:38 PM EEST, Patrick Steinhardt wrote:
> It was reported that git-init(1) can fail when initializing an existing
> directory in case the config contains an "includeIf.onbranch:"
> condition:
>
> ```shell
> $ mkdir repo
> $ git -c includeIf.onbranch:main.path=nonexistent init repo
> BUG: refs.c:2056: reference backend is unknown
> ```
>
> The same error can also be triggered when re-initializing an already
> existing repository.
>
> The bug has been introduced in 173761e21b (setup: start tracking ref
> storage format, 2023-12-29), which wired up the ref storage format. The
> root cause is in `init_db()`, which tries to read the config before we
> have initialized `the_repository` and most importantly its ref storage
> format. We eventually end up calling `include_by_branch()` and execute
> `refs_resolve_ref_unsafe()`, but because we have not initialized the ref
> storage format yet this will trigger the above bug.
>
> Interestingly, `include_by_branch()` has a mechanism that will only
> cause us to resolve the ref when `the_repository->gitdir` is set. This
> is also the reason why this only happens when we initialize an already
> existing directory or repository: `gitdir` is set in those cases, but
> not when creating a new directory.
>
> Now there are two ways to address the issue:
>
>   - We can adapt `include_by_branch()` to also make the code conditional
>     on whether `the_repository->ref_storage_format` is set.
>
>   - We can shift around code such that we initialize the repository
>     format before we read the config.
>
> While the first approach would be safe, it may also cause us to paper
> over issues where a ref store should have been set up. In our case for
> example, it may be reasonable to expect that re-initializing the repo
> will cause the "onbranch:" condition to trigger, but we would not do
> that if the ref storage format was not set up yet. This also used to
> work before the above commit that introduced this bug.
>
> Rearrange the code such that we set up the repository format before
> reading the config. This fixes the bug and ensures that "onbranch:"
> conditions can trigger.
>
> Reported-by: Heghedus Razvan <heghedus.razvan@protonmail.com>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>

I can confirm it's fixing the issue. Feel free to add:

Tested-by: Heghedus Razvan <heghedus.razvan@protonmail.com>

Re: [PATCH] setup: fix bug with "includeIf.onbranch" when initializing dir

[Junio C Hamano]

Heghedus Razvan <heghedus.razvan@protonmail.com> writes:

>> Reported-by: Heghedus Razvan <heghedus.razvan@protonmail.com>
>> Signed-off-by: Patrick Steinhardt <ps@pks.im>
>
> I can confirm it's fixing the issue. Feel free to add:
>
> Tested-by: Heghedus Razvan <heghedus.razvan@protonmail.com>

Thanks, both.

Re: [PATCH] setup: fix bug with "includeIf.onbranch" when initializing dir

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> The bug has been introduced in 173761e21b (setup: start tracking ref
> storage format, 2023-12-29), which wired up the ref storage format. The
> root cause is in `init_db()`, which tries to read the config before we
> have initialized `the_repository` and most importantly its ref storage
> format. We eventually end up calling `include_by_branch()` and execute
> `refs_resolve_ref_unsafe()`, but because we have not initialized the ref
> storage format yet this will trigger the above bug.

So I was trying to see how feasible it would be to backport this fix
on top of v2.44.0 which was the first version that was shipped with
173761e2 (setup: start tracking ref storage format, 2023-12-29),
which is v2.44.0-rc0~78^2~8 and noticed something curious.

As your addition to t0001-init.sh wants to try converting the ref
backends to and from files and reftable, it actually will barf until
b4385bf0 (Merge branch 'ps/reftable-backend', 2024-02-26), which is
v2.45.0-rc0~176, but even with a reinit with the same backend, the
problem should manifest itself, right?

> diff --git a/t/t0001-init.sh b/t/t0001-init.sh
> index b131d665db..2239bed198 100755
> --- a/t/t0001-init.sh
> +++ b/t/t0001-init.sh
> @@ -584,14 +584,38 @@ test_expect_success 'init with --ref-format=files' '
>  	test_cmp expect actual
>  '
>  
> -test_expect_success 're-init with same format' '
> -	test_when_finished "rm -rf refformat" &&
> -	git init --ref-format=files refformat &&
> -	git init --ref-format=files refformat &&
> -	echo files >expect &&
> -	git -C refformat rev-parse --show-ref-format >actual &&
> -	test_cmp expect actual
> -'

So we used to have "files to files" and nothing else.  But

> +for from_format in files reftable
> +do
> +	for to_format in files reftable
> +	do
> +		if test "$from_format" = "$to_format"
> +		then
> +			continue
> +		fi
> +
> +		test_expect_success "re-init with same format ($from_format)" '
> +			test_when_finished "rm -rf refformat" &&
> +			git init --ref-format=$from_format refformat &&
> +			git init --ref-format=$from_format refformat &&
> +			echo $from_format >expect &&
> +			git -C refformat rev-parse --show-ref-format >actual &&
> +			test_cmp expect actual
> +		'

This is very iffy, isn't it?  This one wants to do the same format
reinit, but it is behind "if from and to are the same, skip the rest"
in the loop.

I think the "same format" loop should be lifted outside and
immediately before the inner loop and we should be OK.

> +		test_expect_success "re-init with different format fails ($from_format -> $to_format)" '
> +			test_when_finished "rm -rf refformat" &&
> +			git init --ref-format=$from_format refformat &&
> +			cat >expect <<-EOF &&
> +			fatal: attempt to reinitialize repository with different reference storage format
> +			EOF
> +			test_must_fail git init --ref-format=$to_format refformat 2>err &&
> +			test_cmp expect err &&
> +			echo $from_format >expect &&
> +			git -C refformat rev-parse --show-ref-format >actual &&
> +			test_cmp expect actual
> +		'
> +	done
> +done

In any case, this "reinitialize to a different format is too late"
test has nothing to do with the problem we are fixing with this
patch, no?  It is a valuable set of tests in the post b4385bf0 world
where we can choose between the two, but it is somewhat out of scope
of the "we need to initialize the ref backend before we can do onbranch
config".

I'll send your patch backported to v2.44.0, plus the change needed
to review the merge of it into v2.45.0 codebase later.

Re: [PATCH] setup: fix bug with "includeIf.onbranch" when initializing dir

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2347 bytes --]

On Wed, May 22, 2024 at 05:41:28PM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
[snip]
> > +for from_format in files reftable
> > +do
> > +	for to_format in files reftable
> > +	do
> > +		if test "$from_format" = "$to_format"
> > +		then
> > +			continue
> > +		fi
> > +
> > +		test_expect_success "re-init with same format ($from_format)" '
> > +			test_when_finished "rm -rf refformat" &&
> > +			git init --ref-format=$from_format refformat &&
> > +			git init --ref-format=$from_format refformat &&
> > +			echo $from_format >expect &&
> > +			git -C refformat rev-parse --show-ref-format >actual &&
> > +			test_cmp expect actual
> > +		'
> 
> This is very iffy, isn't it?  This one wants to do the same format
> reinit, but it is behind "if from and to are the same, skip the rest"
> in the loop.
> 
> I think the "same format" loop should be lifted outside and
> immediately before the inner loop and we should be OK.

Yup, we can do that.

> > +		test_expect_success "re-init with different format fails ($from_format -> $to_format)" '
> > +			test_when_finished "rm -rf refformat" &&
> > +			git init --ref-format=$from_format refformat &&
> > +			cat >expect <<-EOF &&
> > +			fatal: attempt to reinitialize repository with different reference storage format
> > +			EOF
> > +			test_must_fail git init --ref-format=$to_format refformat 2>err &&
> > +			test_cmp expect err &&
> > +			echo $from_format >expect &&
> > +			git -C refformat rev-parse --show-ref-format >actual &&
> > +			test_cmp expect actual
> > +		'
> > +	done
> > +done
> 
> In any case, this "reinitialize to a different format is too late"
> test has nothing to do with the problem we are fixing with this
> patch, no?  It is a valuable set of tests in the post b4385bf0 world
> where we can choose between the two, but it is somewhat out of scope
> of the "we need to initialize the ref backend before we can do onbranch
> config".
> 
> I'll send your patch backported to v2.44.0, plus the change needed
> to review the merge of it into v2.45.0 codebase later.

Yeah, I wanted to play it safe and add some cases I felt were missing in
the existing test coverage so that there ideally aren't any other issues
and so that the proposed change doesn't regress functionality.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[PATCH v2] setup: fix bug with "includeIf.onbranch" when initializing dir

[Junio C Hamano]

It was reported that git-init(1) can fail when initializing an existing
directory in case the config contains an "includeIf.onbranch:"
condition:

    $ mkdir repo
    $ git -c includeIf.onbranch:main.path=nonexistent init repo
    BUG: refs.c:2056: reference backend is unknown

The same error can also be triggered when re-initializing an already
existing repository.

The bug has been introduced in 173761e21b (setup: start tracking ref
storage format, 2023-12-29), which wired up the ref storage format. The
root cause is in `init_db()`, which tries to read the config before we
have initialized `the_repository` and most importantly its ref storage
format. We eventually end up calling `include_by_branch()` and execute
`refs_resolve_ref_unsafe()`, but because we have not initialized the ref
storage format yet this will trigger the above bug.

Interestingly, `include_by_branch()` has a mechanism that will only
cause us to resolve the ref when `the_repository->gitdir` is set. This
is also the reason why this only happens when we initialize an already
existing directory or repository: `gitdir` is set in those cases, but
not when creating a new directory.

Now there are two ways to address the issue:

  - We can adapt `include_by_branch()` to also make the code conditional
    on whether `the_repository->ref_storage_format` is set.

  - We can shift around code such that we initialize the repository
    format before we read the config.

While the first approach would be safe, it may also cause us to paper
over issues where a ref store should have been set up. In our case for
example, it may be reasonable to expect that re-initializing the repo
will cause the "onbranch:" condition to trigger, but we would not do
that if the ref storage format was not set up yet. This also used to
work before the above commit that introduced this bug.

Rearrange the code such that we set up the repository format before
reading the config. This fixes the bug and ensures that "onbranch:"
conditions can trigger.

Reported-by: Heghedus Razvan <heghedus.razvan@protonmail.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
Tested-by: Heghedus Razvan <heghedus.razvan@protonmail.com>
[jc: backported to v2.44.0 codebase]
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 * So this is how it should look like if we later wanted to merge
   down the fix to the 2.44 maintenance track.

 setup.c         |  26 +++++++------
 t/t0001-init.sh | 101 ++++++++++++++++++++++++++++++++++++++++++++----
 2 files changed, 108 insertions(+), 19 deletions(-)

diff --git a/setup.c b/setup.c
index b69b1cbc2a..c6fffa0164 100644
--- a/setup.c
+++ b/setup.c
@@ -2196,13 +2196,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	}
 	startup_info->have_repository = 1;
 
-	/* Ensure `core.hidedotfiles` is processed */
-	git_config(platform_core_config, NULL);
-
-	safe_create_dir(git_dir, 0);
-
-	prev_bare_repository = is_bare_repository();
-
 	/* Check to see if the repository version is right.
 	 * Note that a newly created repository does not have
 	 * config file, so this will not fail.  What we are catching
@@ -2213,10 +2206,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	validate_hash_algorithm(&repo_fmt, hash);
 	validate_ref_storage_format(&repo_fmt, ref_storage_format);
 
-	reinit = create_default_files(template_dir, original_git_dir,
-				      &repo_fmt, prev_bare_repository,
-				      init_shared_repository);
-
 	/*
 	 * Now that we have set up both the hash algorithm and the ref storage
 	 * format we can update the repository's settings accordingly.
@@ -2224,6 +2213,21 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
 	repo_set_ref_storage_format(the_repository, repo_fmt.ref_storage_format);
 
+	/*
+	 * Ensure `core.hidedotfiles` is processed. This must happen after we
+	 * have set up the repository format such that we can evaluate
+	 * includeIf conditions correctly in the case of re-initialization.
+	 */
+	git_config(platform_core_config, NULL);
+
+	safe_create_dir(git_dir, 0);
+
+	prev_bare_repository = is_bare_repository();
+
+	reinit = create_default_files(template_dir, original_git_dir,
+				      &repo_fmt, prev_bare_repository,
+				      init_shared_repository);
+
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo_fmt.ref_storage_format,
 					  initial_branch, flags & INIT_DB_QUIET);
diff --git a/t/t0001-init.sh b/t/t0001-init.sh
index b131d665db..319ed81631 100755
--- a/t/t0001-init.sh
+++ b/t/t0001-init.sh
@@ -584,14 +584,39 @@ test_expect_success 'init with --ref-format=files' '
 	test_cmp expect actual
 '
 
-test_expect_success 're-init with same format' '
-	test_when_finished "rm -rf refformat" &&
-	git init --ref-format=files refformat &&
-	git init --ref-format=files refformat &&
-	echo files >expect &&
-	git -C refformat rev-parse --show-ref-format >actual &&
-	test_cmp expect actual
-'
+backends="files"
+for from_format in $backends
+do
+	test_expect_success "re-init with same format ($from_format)" '
+		test_when_finished "rm -rf refformat" &&
+		git init --ref-format=$from_format refformat &&
+		git init --ref-format=$from_format refformat &&
+		echo $from_format >expect &&
+		git -C refformat rev-parse --show-ref-format >actual &&
+		test_cmp expect actual
+	'
+
+	for to_format in $backends
+	do
+		if test "$from_format" = "$to_format"
+		then
+			continue
+		fi
+
+		test_expect_success "re-init with different format fails ($from_format -> $to_format)" '
+			test_when_finished "rm -rf refformat" &&
+			git init --ref-format=$from_format refformat &&
+			cat >expect <<-EOF &&
+			fatal: attempt to reinitialize repository with different reference storage format
+			EOF
+			test_must_fail git init --ref-format=$to_format refformat 2>err &&
+			test_cmp expect err &&
+			echo $from_format >expect &&
+			git -C refformat rev-parse --show-ref-format >actual &&
+			test_cmp expect actual
+		'
+	done
+done
 
 test_expect_success 'init with --ref-format=garbage' '
 	test_when_finished "rm -rf refformat" &&
@@ -678,4 +703,64 @@ test_expect_success 'branch -m with the initial branch' '
 	test_cmp expect actual
 '
 
+test_expect_success 'init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git -c includeIf.onbranch:main.path=nonexistent init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'init with includeIf.onbranch condition with existing directory' '
+	test_when_finished "rm -rf repo" &&
+	mkdir repo &&
+	git -c includeIf.onbranch:nonexistent.path=/does/not/exist init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 're-init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	git -c includeIf.onbranch:nonexistent.path=/does/not/exist init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 're-init with includeIf.onbranch condition' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	git -c includeIf.onbranch:nonexistent.path=/does/not/exist init repo &&
+	echo $GIT_DEFAULT_REF_FORMAT >expect &&
+	git -C repo rev-parse --show-ref-format >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 're-init skips non-matching includeIf.onbranch' '
+	test_when_finished "rm -rf repo config" &&
+	cat >config <<-EOF &&
+	[
+	garbage
+	EOF
+	git init repo &&
+	git -c includeIf.onbranch:nonexistent.path="$(test-tool path-utils absolute_path config)" init repo
+'
+
+test_expect_success 're-init reads matching includeIf.onbranch' '
+	test_when_finished "rm -rf repo config" &&
+	cat >config <<-EOF &&
+	[
+	garbage
+	EOF
+	path="$(test-tool path-utils absolute_path config)" &&
+	git init --initial-branch=branch repo &&
+	cat >expect <<-EOF &&
+	fatal: bad config line 1 in file $path
+	EOF
+	test_must_fail git -c includeIf.onbranch:branch.path="$path" init repo 2>err &&
+	test_cmp expect err
+'
+
 test_done
-- 
2.45.1-216-g4365c6fcf9

[rPATCH v2] setup: fix bug with "includeIf.onbranch" when initializing dir

[Junio C Hamano]

And this is "git show --remerge-diff" output for the merge that
merges the result of applying the previous patch to v2.44.0 into
v2.45.0-rc0~112 (which removed the 'prev_bare_repository' thing).

The resolution of conflicts in setup.c is trivial.  The change to
t/t0001-init.sh is an evil one.

diff --git a/t/t0001-init.sh b/t/t0001-init.sh
index 319ed81631..49e9bf77c6 100755
--- a/t/t0001-init.sh
+++ b/t/t0001-init.sh
@@ -584,7 +584,7 @@ test_expect_success 'init with --ref-format=files' '
 	test_cmp expect actual
 '
 
-backends="files"
+backends="files reftable"
 for from_format in $backends
 do
 	test_expect_success "re-init with same format ($from_format)" '
diff --git a/setup.c b/setup.c
remerge CONFLICT (content): Merge conflict in setup.c
index 2f23ab6efa..e01462863b 100644
--- a/setup.c
+++ b/setup.c
@@ -2175,23 +2175,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	}
 	startup_info->have_repository = 1;
 
-<<<<<<< 272fd9125a (Merge branch 'gt/core-bare-in-templates')
-	/* Ensure `core.hidedotfiles` is processed */
-	git_config(platform_core_config, NULL);
-
-	safe_create_dir(git_dir, 0);
-
-
-||||||| 3c2a3fdc38
-	/* Ensure `core.hidedotfiles` is processed */
-	git_config(platform_core_config, NULL);
-
-	safe_create_dir(git_dir, 0);
-
-	prev_bare_repository = is_bare_repository();
-
-=======
->>>>>>> 7a998665dd (setup: fix bug with "includeIf.onbranch" when initializing dir)
 	/* Check to see if the repository version is right.
 	 * Note that a newly created repository does not have
 	 * config file, so this will not fail.  What we are catching
@@ -2202,17 +2185,6 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	validate_hash_algorithm(&repo_fmt, hash);
 	validate_ref_storage_format(&repo_fmt, ref_storage_format);
 
-<<<<<<< 272fd9125a (Merge branch 'gt/core-bare-in-templates')
-	reinit = create_default_files(template_dir, original_git_dir,
-				      &repo_fmt, init_shared_repository);
-
-||||||| 3c2a3fdc38
-	reinit = create_default_files(template_dir, original_git_dir,
-				      &repo_fmt, prev_bare_repository,
-				      init_shared_repository);
-
-=======
->>>>>>> 7a998665dd (setup: fix bug with "includeIf.onbranch" when initializing dir)
 	/*
 	 * Now that we have set up both the hash algorithm and the ref storage
 	 * format we can update the repository's settings accordingly.
@@ -2229,11 +2201,8 @@ int init_db(const char *git_dir, const char *real_git_dir,
 
 	safe_create_dir(git_dir, 0);
 
-	prev_bare_repository = is_bare_repository();
-
 	reinit = create_default_files(template_dir, original_git_dir,
-				      &repo_fmt, prev_bare_repository,
-				      init_shared_repository);
+				      &repo_fmt, init_shared_repository);
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo_fmt.ref_storage_format,

Re: [PATCH v2] setup: fix bug with "includeIf.onbranch" when initializing dir

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 2269 bytes --]

On Wed, May 22, 2024 at 05:43:32PM -0700, Junio C Hamano wrote:
[snip]
> diff --git a/setup.c b/setup.c
> index b69b1cbc2a..c6fffa0164 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -2224,6 +2213,21 @@ int init_db(const char *git_dir, const char *real_git_dir,
>  	repo_set_hash_algo(the_repository, repo_fmt.hash_algo);
>  	repo_set_ref_storage_format(the_repository, repo_fmt.ref_storage_format);
>  
> +	/*
> +	 * Ensure `core.hidedotfiles` is processed. This must happen after we
> +	 * have set up the repository format such that we can evaluate
> +	 * includeIf conditions correctly in the case of re-initialization.
> +	 */
> +	git_config(platform_core_config, NULL);
> +
> +	safe_create_dir(git_dir, 0);
> +
> +	prev_bare_repository = is_bare_repository();

Okay, `prev_bare_repository` is another difference between v2.44 and
master.

> +	reinit = create_default_files(template_dir, original_git_dir,
> +				      &repo_fmt, prev_bare_repository,
> +				      init_shared_repository);
> +
>  	if (!(flags & INIT_DB_SKIP_REFDB))
>  		create_reference_database(repo_fmt.ref_storage_format,
>  					  initial_branch, flags & INIT_DB_QUIET);
> diff --git a/t/t0001-init.sh b/t/t0001-init.sh
> index b131d665db..319ed81631 100755
> --- a/t/t0001-init.sh
> +++ b/t/t0001-init.sh
> @@ -584,14 +584,39 @@ test_expect_success 'init with --ref-format=files' '
>  	test_cmp expect actual
>  '
>  
> -test_expect_success 're-init with same format' '
> -	test_when_finished "rm -rf refformat" &&
> -	git init --ref-format=files refformat &&
> -	git init --ref-format=files refformat &&
> -	echo files >expect &&
> -	git -C refformat rev-parse --show-ref-format >actual &&
> -	test_cmp expect actual
> -'
> +backends="files"
> +for from_format in $backends
> +do
> +	test_expect_success "re-init with same format ($from_format)" '
> +		test_when_finished "rm -rf refformat" &&
> +		git init --ref-format=$from_format refformat &&
> +		git init --ref-format=$from_format refformat &&
> +		echo $from_format >expect &&
> +		git -C refformat rev-parse --show-ref-format >actual &&
> +		test_cmp expect actual
> +	'

And this here is the change to the test setup, which makes sense.

Looks sensible, thanks!

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]