[PATCH 0/2] refs: allow setting the reference directory

[PATCH 0/2] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
reference backend and path in a URI form:

    <reference_backend>://<path>

For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

We could make the migration non-blocking by running the migration in the
background and capturing and replaying updates to both backends. This
would require Git to support writing references to different reference
backends and paths.

The first commit adds the required changes to create a 'ref_store' for a
given path. The second commit parses the URI if available when creating
the main ref store.

This is based on top of 9a2fb147f2 (Git 2.52, 2025-11-17).

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   8 ++++
 environment.h          |   1 +
 refs.c                 |  64 +++++++++++++++++++++++++++--
 t/meson.build          |   1 +
 t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 180 insertions(+), 3 deletions(-)

Karthik Nayak (2):
      refs: support obtaining ref_store for given dir
      refs: add GIT_REF_URI to specify reference backend and directory



base-commit: 9a2fb147f2c61d0cab52c883e7e26f5b7948e3ed
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH 1/2] refs: support obtaining ref_store for given dir

[Karthik Nayak]

The refs subsystem uses the `get_main_ref_store()` to obtain the main
ref_store for a given repository. In the upcoming patches we also want
to create a ref_store for any given reference directory, which may exist
in arbitrary paths. To support such behavior, extract out the core logic
for creating out the ref_store from `get_main_ref_store()` into a new
function `get_ref_store_for_dir()` which can provide the ref_store for a
given (repository, directory, reference format) combination.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/refs.c b/refs.c
index 965381367e..23f46867f2 100644
--- a/refs.c
+++ b/refs.c
@@ -2177,6 +2177,15 @@ void ref_store_release(struct ref_store *ref_store)
 	free(ref_store->gitdir);
 }
 
+static struct ref_store *get_ref_store_for_dir(struct repository *r,
+					       char *dir,
+					       enum ref_storage_format format)
+{
+	struct ref_store *ref_store = ref_store_init(r, format, dir,
+						     REF_STORE_ALL_CAPS);
+	return maybe_debug_wrap_ref_store(dir, ref_store);
+}
+
 struct ref_store *get_main_ref_store(struct repository *r)
 {
 	if (r->refs_private)
@@ -2185,9 +2194,7 @@ struct ref_store *get_main_ref_store(struct repository *r)
 	if (!r->gitdir)
 		BUG("attempting to get main_ref_store outside of repository");
 
-	r->refs_private = ref_store_init(r, r->ref_storage_format,
-					 r->gitdir, REF_STORE_ALL_CAPS);
-	r->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);
+	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
 	return r->refs_private;
 }
 

-- 
2.51.2

Re: [PATCH 1/2] refs: support obtaining ref_store for given dir

[Justin Tobler]

On 25/11/19 10:48PM, Karthik Nayak wrote:
> The refs subsystem uses the `get_main_ref_store()` to obtain the main
> ref_store for a given repository. In the upcoming patches we also want
> to create a ref_store for any given reference directory, which may exist
> in arbitrary paths. To support such behavior, extract out the core logic
> for creating out the ref_store from `get_main_ref_store()` into a new
> function `get_ref_store_for_dir()` which can provide the ref_store for a
> given (repository, directory, reference format) combination.

So when we refer to the "reference directory" in this case, we are not
refering to the "refs/" or "reftable/" directories directly, but one
level above that which is typically just the gitdir itself. This seems a
bit awkward at first, but makes sense since, for the files backend,
there may be symbolic references such as HEAD that exist outside of
"refs/" which must be considered. It might be helpful to clarify this in
the commit message.

Otherwise this patch looks good.

-Justin

Re: [PATCH 1/2] refs: support obtaining ref_store for given dir

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1384 bytes --]

Justin Tobler <jltobler@gmail.com> writes:

> On 25/11/19 10:48PM, Karthik Nayak wrote:
>> The refs subsystem uses the `get_main_ref_store()` to obtain the main
>> ref_store for a given repository. In the upcoming patches we also want
>> to create a ref_store for any given reference directory, which may exist
>> in arbitrary paths. To support such behavior, extract out the core logic
>> for creating out the ref_store from `get_main_ref_store()` into a new
>> function `get_ref_store_for_dir()` which can provide the ref_store for a
>> given (repository, directory, reference format) combination.
>
> So when we refer to the "reference directory" in this case, we are not
> refering to the "refs/" or "reftable/" directories directly, but one
> level above that which is typically just the gitdir itself. This seems a
> bit awkward at first, but makes sense since, for the files backend,
> there may be symbolic references such as HEAD that exist outside of
> "refs/" which must be considered. It might be helpful to clarify this in
> the commit message.
>

You're right, for the files and the reftable backend, this happens to be
the $GIT_DIR itself, due to how closely they are integrated with Git.
But if you build an external reference backend, this doesn't have to be
the $GIT_DIR.

I've modified the commit message accordingly

> Otherwise this patch looks good.
>
> -Justin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
This asymmetry makes it difficult to test different reference backends
or use alternative reference storage locations without modifying the
repository structure.

Add a new environment variable 'GIT_REF_URI' that specifies both the
reference backend and directory path using a URI format:

    <ref_backend>://<path>

When set, this variable is used to obtain the main reference store for
all Git commands. The variable is checked in `get_main_ref_store()`
when lazily assigning `repo->refs_private`. We cannot initialize this
earlier in `repo_set_gitdir()` because the repository's hash algorithm
isn't known at that point, and the reftable backend requires this
information during initialization.

When used with worktrees, the specified directory is treated as the
reference directory for all worktree operations.

Add a new test file 't1423-ref-backend.sh' to test this environment
variable.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   8 ++++
 environment.h          |   1 +
 refs.c                 |  53 +++++++++++++++++++++++-
 t/meson.build          |   1 +
 t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 171 insertions(+), 1 deletion(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..a1d1078f42 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REF_URI`::
+    Specify which reference backend and path to be used, if not specified the
+    backend is inferred from the configuration and $GIT_DIR is used as the
+    path.
++
+Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
+reference backend and the 'path' specifies the directory used by the backend.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 51898c99cd..9bc380bba4 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index 23f46867f2..0922f08c9f 100644
--- a/refs.c
+++ b/refs.c
@@ -2186,15 +2186,66 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
 	return maybe_debug_wrap_ref_store(dir, ref_store);
 }
 
+static struct ref_store *get_ref_store_from_uri(struct repository *repo,
+						const char *uri)
+{
+	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
+	enum ref_storage_format format;
+	struct ref_store *store = NULL;
+	char *format_string;
+	char *dir;
+
+	if (!uri || !uri[0]) {
+		error("reference backend uri is empty");
+		goto cleanup;
+	}
+
+	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
+		error("invalid reference backend uri format '%s'", uri);
+		goto cleanup;
+	}
+
+	format_string = ref_backend_info.items[0].string;
+	dir = ref_backend_info.items[1].string + 2;
+
+	if (!dir || !dir[0]) {
+		error("invalid path in uri '%s'", uri);
+		goto cleanup;
+	}
+
+	format = ref_storage_format_by_name(format_string);
+	if (format == REF_STORAGE_FORMAT_UNKNOWN) {
+		error("unknown reference backend '%s'", format_string);
+		goto cleanup;
+	}
+
+	store = get_ref_store_for_dir(repo, dir, format);
+
+cleanup:
+	string_list_clear(&ref_backend_info, 0);
+	return store;
+}
+
 struct ref_store *get_main_ref_store(struct repository *r)
 {
+	char *ref_uri;
+
 	if (r->refs_private)
 		return r->refs_private;
 
 	if (!r->gitdir)
 		BUG("attempting to get main_ref_store outside of repository");
 
-	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
+	ref_uri = getenv(GIT_REF_URI_ENVIRONMENT);
+	if (ref_uri) {
+		r->refs_private = get_ref_store_from_uri(r, ref_uri);
+		if (!r->refs_private)
+			die("failed to initialize ref store from URI: %s", ref_uri);
+
+	} else {
+		r->refs_private = get_ref_store_for_dir(r, r->gitdir,
+							r->ref_storage_format);
+	}
 	return r->refs_private;
 }
 
diff --git a/t/meson.build b/t/meson.build
index a5531df415..a66f8fafff 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -208,6 +208,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..e271708e02
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,109 @@
+#!/bin/sh
+
+test_description='Test different reference backend URIs'
+
+. ./test-lib.sh
+
+test_expect_success 'empty uri provided' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="" &&
+		export GIT_REF_URI &&
+		! git refs list 2>err &&
+		test_grep "reference backend uri is empty" err
+	)
+'
+
+test_expect_success 'invalid uri provided' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable@/home/reftable" &&
+		export GIT_REF_URI &&
+		! git refs list 2>err &&
+		test_grep "invalid reference backend uri format" err
+	)
+'
+
+test_expect_success 'empty path in uri' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable://" &&
+		export GIT_REF_URI &&
+		! git refs list 2>err &&
+		test_grep "invalid path in uri" err
+	)
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="db://.git" &&
+		export GIT_REF_URI &&
+		! git refs list 2>err &&
+		test_grep "unknown reference backend" err
+	)
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+	for to_format in $ref_formats
+	do
+		if test "$from_format" = "$to_format"
+		then
+			continue
+		fi
+
+		test_expect_success 'read from other reference backend' '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=files repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=reftable >out &&
+				REFTABLE_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
+				git refs list >expect &&
+				GIT_REF_URI="reftable://$REFTABLE_PATH" git refs list >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success 'write to other reference backend' '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=files repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=reftable >out &&
+				git refs list >expect &&
+
+				REFTABLE_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
+				GIT_REF_URI="reftable://$REFTABLE_PATH" git tag -d 1 &&
+
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				GIT_REF_URI="reftable://$REFTABLE_PATH" git refs list >expect &&
+				git refs list >out &&
+				cat out | grep -v "refs/tags/1" >actual &&
+				test_cmp expect actual
+			)
+		'
+	done
+done
+
+test_done

-- 
2.51.2

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Eric Sunshine]

On Wed, Nov 19, 2025 at 4:49 PM Karthik Nayak <karthik.188@gmail.com> wrote:
> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
> This asymmetry makes it difficult to test different reference backends
> or use alternative reference storage locations without modifying the
> repository structure.
>
> Add a new environment variable 'GIT_REF_URI' that specifies both the
> reference backend and directory path using a URI format:
> [...]
> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
> @@ -0,0 +1,109 @@
> +test_expect_success 'empty uri provided' '
> +       test_when_finished "rm -rf repo" &&
> +       git init --ref-format=files repo &&
> +       (
> +               cd repo &&
> +               GIT_REF_URI="" &&
> +               export GIT_REF_URI &&
> +               ! git refs list 2>err &&

Should this (and all other tests) be using `test_must_fail` rather than `!`?

> +               test_grep "reference backend uri is empty" err
> +       )
> +'
> +ref_formats="files reftable"
> +for from_format in $ref_formats
> +do
> +       for to_format in $ref_formats
> +       do
> +               if test "$from_format" = "$to_format"
> +               then
> +                       continue
> +               fi
> +
> +               test_expect_success 'read from other reference backend' '
> +                       test_when_finished "rm -rf repo" &&
> +                       git init --ref-format=files repo &&
> +                       (
> +                               cd repo &&
> +                               test_commit 1 &&
> +                               test_commit 2 &&
> +                               test_commit 3 &&
> +
> +                               git refs migrate --dry-run --ref-format=reftable >out &&
> +                               REFTABLE_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
> +                               git refs list >expect &&
> +                               GIT_REF_URI="reftable://$REFTABLE_PATH" git refs list >actual &&
> +                               test_cmp expect actual
> +                       )
> +               '
> +
> +               test_expect_success 'write to other reference backend' '
> +                       [...]
> +               '
> +       done
> +done

Something seems amiss here. Presumably, this nested loop wants to test
various combinations but the `from_format` and `to_format` variables
are never consulted in the tests; instead the tests just hardcode
specific ref-format values.

Also, if this is indeed meant to be loop-driven, then it would be
helpful for the test titles to include the values of `$from_format`
and `$to_format`.

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3203 bytes --]

Eric Sunshine <sunshine@sunshineco.com> writes:

> On Wed, Nov 19, 2025 at 4:49 PM Karthik Nayak <karthik.188@gmail.com> wrote:
>> Git allows setting a different object directory via
>> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
>> This asymmetry makes it difficult to test different reference backends
>> or use alternative reference storage locations without modifying the
>> repository structure.
>>
>> Add a new environment variable 'GIT_REF_URI' that specifies both the
>> reference backend and directory path using a URI format:
>> [...]
>> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
>> ---
>> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
>> @@ -0,0 +1,109 @@
>> +test_expect_success 'empty uri provided' '
>> +       test_when_finished "rm -rf repo" &&
>> +       git init --ref-format=files repo &&
>> +       (
>> +               cd repo &&
>> +               GIT_REF_URI="" &&
>> +               export GIT_REF_URI &&
>> +               ! git refs list 2>err &&
>
> Should this (and all other tests) be using `test_must_fail` rather than `!`?
>

Initially I used 'BUG()' instead of 'error()', which was wrong, but
meant that I couldn't use `test_must_fail`. I've fixed that now, but
this was missed. Thanks.

>> +               test_grep "reference backend uri is empty" err
>> +       )
>> +'
>> +ref_formats="files reftable"
>> +for from_format in $ref_formats
>> +do
>> +       for to_format in $ref_formats
>> +       do
>> +               if test "$from_format" = "$to_format"
>> +               then
>> +                       continue
>> +               fi
>> +
>> +               test_expect_success 'read from other reference backend' '
>> +                       test_when_finished "rm -rf repo" &&
>> +                       git init --ref-format=files repo &&
>> +                       (
>> +                               cd repo &&
>> +                               test_commit 1 &&
>> +                               test_commit 2 &&
>> +                               test_commit 3 &&
>> +
>> +                               git refs migrate --dry-run --ref-format=reftable >out &&
>> +                               REFTABLE_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
>> +                               git refs list >expect &&
>> +                               GIT_REF_URI="reftable://$REFTABLE_PATH" git refs list >actual &&
>> +                               test_cmp expect actual
>> +                       )
>> +               '
>> +
>> +               test_expect_success 'write to other reference backend' '
>> +                       [...]
>> +               '
>> +       done
>> +done
>
> Something seems amiss here. Presumably, this nested loop wants to test
> various combinations but the `from_format` and `to_format` variables
> are never consulted in the tests; instead the tests just hardcode
> specific ref-format values.
>
> Also, if this is indeed meant to be loop-driven, then it would be
> helpful for the test titles to include the values of `$from_format`
> and `$to_format`.

Indeed. I was hasty, will fix :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Jean-Noël Avila]

On 19/11/2025 at 22:48, Karthik Nayak wrote:
> ---
>  Documentation/git.adoc |   8 ++++
>  environment.h          |   1 +
>  refs.c                 |  53 +++++++++++++++++++++++-
>  t/meson.build          |   1 +
>  t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 171 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
> index ce099e78b8..a1d1078f42 100644
> --- a/Documentation/git.adoc
> +++ b/Documentation/git.adoc
> @@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
>  	repositories will be set to this value. The default is "files".
>  	See `--ref-format` in linkgit:git-init[1].
>  
> +`GIT_REF_URI`::
> +    Specify which reference backend and path to be used, if not specified the
> +    backend is inferred from the configuration and $GIT_DIR is used as the
> +    path.

Please use backquotes for environment variables: `$GIT_DIR`

> ++
> +Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
> +reference backend and the 'path' specifies the directory used by the backend.

Constant strings and keywords are back-quoted too but placeholders are
underscored:

Expects the format `<ref_backend>://<path>`, where the _<ref_backend>_
specifies the reference backend and the _<path>_ specifies the directory
used by the backend.

I'm only focusing on documentation.

Thanks

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1632 bytes --]

Jean-Noël Avila <jn.avila@free.fr> writes:

> On 19/11/2025 at 22:48, Karthik Nayak wrote:
>> ---
>>  Documentation/git.adoc |   8 ++++
>>  environment.h          |   1 +
>>  refs.c                 |  53 +++++++++++++++++++++++-
>>  t/meson.build          |   1 +
>>  t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
>>  5 files changed, 171 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
>> index ce099e78b8..a1d1078f42 100644
>> --- a/Documentation/git.adoc
>> +++ b/Documentation/git.adoc
>> @@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
>>  	repositories will be set to this value. The default is "files".
>>  	See `--ref-format` in linkgit:git-init[1].
>>
>> +`GIT_REF_URI`::
>> +    Specify which reference backend and path to be used, if not specified the
>> +    backend is inferred from the configuration and $GIT_DIR is used as the
>> +    path.
>
> Please use backquotes for environment variables: `$GIT_DIR`
>

Will do.

>> ++
>> +Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
>> +reference backend and the 'path' specifies the directory used by the backend.
>
> Constant strings and keywords are back-quoted too but placeholders are
> underscored:
>
> Expects the format `<ref_backend>://<path>`, where the _<ref_backend>_
> specifies the reference backend and the _<path>_ specifies the directory
> used by the backend.
>
> I'm only focusing on documentation.
>
> Thanks

Thanks for your review, will modify accordingly.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Justin Tobler]

On 25/11/19 10:48PM, Karthik Nayak wrote:
> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
> This asymmetry makes it difficult to test different reference backends
> or use alternative reference storage locations without modifying the
> repository structure.
> 
> Add a new environment variable 'GIT_REF_URI' that specifies both the
> reference backend and directory path using a URI format:
> 
>     <ref_backend>://<path>

Ok, we include the reference format as part of the URI here since it is
possible that the alternative reference store could be using a different
backend that what the repository is currently configured to use. Makes
sense.

> When set, this variable is used to obtain the main reference store for
> all Git commands. The variable is checked in `get_main_ref_store()`
> when lazily assigning `repo->refs_private`. We cannot initialize this
> earlier in `repo_set_gitdir()` because the repository's hash algorithm
> isn't known at that point, and the reftable backend requires this
> information during initialization.
>
> When used with worktrees, the specified directory is treated as the
> reference directory for all worktree operations.
> 
> Add a new test file 't1423-ref-backend.sh' to test this environment
> variable.
> 
> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
>  Documentation/git.adoc |   8 ++++
>  environment.h          |   1 +
>  refs.c                 |  53 +++++++++++++++++++++++-
>  t/meson.build          |   1 +
>  t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 171 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
> index ce099e78b8..a1d1078f42 100644
> --- a/Documentation/git.adoc
> +++ b/Documentation/git.adoc
> @@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
>  	repositories will be set to this value. The default is "files".
>  	See `--ref-format` in linkgit:git-init[1].
>  
> +`GIT_REF_URI`::
> +    Specify which reference backend and path to be used, if not specified the
> +    backend is inferred from the configuration and $GIT_DIR is used as the
> +    path.
> ++
> +Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
> +reference backend and the 'path' specifies the directory used by the backend.

I think some users may assume that the path to the reference backend
would be something like ".git/refs" similar to how
`GIT_OBJECT_DIRECTORY` is usually ".git/objects". It might be worth
clarifying this in the docs here.

> +
>  Git Commits
>  ~~~~~~~~~~~
>  `GIT_AUTHOR_NAME`::
> diff --git a/environment.h b/environment.h
> index 51898c99cd..9bc380bba4 100644
> --- a/environment.h
> +++ b/environment.h
> @@ -42,6 +42,7 @@
>  #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
>  #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
>  #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
> +#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
>  
>  /*
>   * Environment variable used to propagate the --no-advice global option to the
> diff --git a/refs.c b/refs.c
> index 23f46867f2..0922f08c9f 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2186,15 +2186,66 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
>  	return maybe_debug_wrap_ref_store(dir, ref_store);
>  }
>  
> +static struct ref_store *get_ref_store_from_uri(struct repository *repo,
> +						const char *uri)
> +{
> +	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
> +	enum ref_storage_format format;
> +	struct ref_store *store = NULL;
> +	char *format_string;
> +	char *dir;
> +
> +	if (!uri || !uri[0]) {
> +		error("reference backend uri is empty");
> +		goto cleanup;
> +	}
> +
> +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
> +		error("invalid reference backend uri format '%s'", uri);
> +		goto cleanup;
> +	}
> +
> +	format_string = ref_backend_info.items[0].string;
> +	dir = ref_backend_info.items[1].string + 2;
> +
> +	if (!dir || !dir[0]) {
> +		error("invalid path in uri '%s'", uri);
> +		goto cleanup;
> +	}
> +
> +	format = ref_storage_format_by_name(format_string);
> +	if (format == REF_STORAGE_FORMAT_UNKNOWN) {
> +		error("unknown reference backend '%s'", format_string);
> +		goto cleanup;
> +	}
> +
> +	store = get_ref_store_for_dir(repo, dir, format);

Since we don't update the reference format stored in repo, if we were to
run:

  $ GIT_REF_URI="reftable://<path> git repo info references.format

it would still report what ever the repository was originally configured
with. Since only a single reference backend can be used at time, I
wonder if we should go a bit further and update `r->ref_storage_format`
to be inline with how the repository reference backend is configured via
`GIT_REF_URI`.

-Justin

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 5620 bytes --]

Justin Tobler <jltobler@gmail.com> writes:

> On 25/11/19 10:48PM, Karthik Nayak wrote:
>> Git allows setting a different object directory via
>> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
>> This asymmetry makes it difficult to test different reference backends
>> or use alternative reference storage locations without modifying the
>> repository structure.
>>
>> Add a new environment variable 'GIT_REF_URI' that specifies both the
>> reference backend and directory path using a URI format:
>>
>>     <ref_backend>://<path>
>
> Ok, we include the reference format as part of the URI here since it is
> possible that the alternative reference store could be using a different
> backend that what the repository is currently configured to use. Makes
> sense.
>
>> When set, this variable is used to obtain the main reference store for
>> all Git commands. The variable is checked in `get_main_ref_store()`
>> when lazily assigning `repo->refs_private`. We cannot initialize this
>> earlier in `repo_set_gitdir()` because the repository's hash algorithm
>> isn't known at that point, and the reftable backend requires this
>> information during initialization.
>>
>> When used with worktrees, the specified directory is treated as the
>> reference directory for all worktree operations.
>>
>> Add a new test file 't1423-ref-backend.sh' to test this environment
>> variable.
>>
>> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
>> ---
>>  Documentation/git.adoc |   8 ++++
>>  environment.h          |   1 +
>>  refs.c                 |  53 +++++++++++++++++++++++-
>>  t/meson.build          |   1 +
>>  t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
>>  5 files changed, 171 insertions(+), 1 deletion(-)
>>
>> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
>> index ce099e78b8..a1d1078f42 100644
>> --- a/Documentation/git.adoc
>> +++ b/Documentation/git.adoc
>> @@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
>>  	repositories will be set to this value. The default is "files".
>>  	See `--ref-format` in linkgit:git-init[1].
>>
>> +`GIT_REF_URI`::
>> +    Specify which reference backend and path to be used, if not specified the
>> +    backend is inferred from the configuration and $GIT_DIR is used as the
>> +    path.
>> ++
>> +Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
>> +reference backend and the 'path' specifies the directory used by the backend.
>
> I think some users may assume that the path to the reference backend
> would be something like ".git/refs" similar to how
> `GIT_OBJECT_DIRECTORY` is usually ".git/objects". It might be worth
> clarifying this in the docs here.
>

Fair enough. I'll amend the commit.

>> +
>>  Git Commits
>>  ~~~~~~~~~~~
>>  `GIT_AUTHOR_NAME`::
>> diff --git a/environment.h b/environment.h
>> index 51898c99cd..9bc380bba4 100644
>> --- a/environment.h
>> +++ b/environment.h
>> @@ -42,6 +42,7 @@
>>  #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
>>  #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
>>  #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
>> +#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
>>
>>  /*
>>   * Environment variable used to propagate the --no-advice global option to the
>> diff --git a/refs.c b/refs.c
>> index 23f46867f2..0922f08c9f 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2186,15 +2186,66 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
>>  	return maybe_debug_wrap_ref_store(dir, ref_store);
>>  }
>>
>> +static struct ref_store *get_ref_store_from_uri(struct repository *repo,
>> +						const char *uri)
>> +{
>> +	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
>> +	enum ref_storage_format format;
>> +	struct ref_store *store = NULL;
>> +	char *format_string;
>> +	char *dir;
>> +
>> +	if (!uri || !uri[0]) {
>> +		error("reference backend uri is empty");
>> +		goto cleanup;
>> +	}
>> +
>> +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
>> +		error("invalid reference backend uri format '%s'", uri);
>> +		goto cleanup;
>> +	}
>> +
>> +	format_string = ref_backend_info.items[0].string;
>> +	dir = ref_backend_info.items[1].string + 2;
>> +
>> +	if (!dir || !dir[0]) {
>> +		error("invalid path in uri '%s'", uri);
>> +		goto cleanup;
>> +	}
>> +
>> +	format = ref_storage_format_by_name(format_string);
>> +	if (format == REF_STORAGE_FORMAT_UNKNOWN) {
>> +		error("unknown reference backend '%s'", format_string);
>> +		goto cleanup;
>> +	}
>> +
>> +	store = get_ref_store_for_dir(repo, dir, format);
>
> Since we don't update the reference format stored in repo, if we were to
> run:
>
>   $ GIT_REF_URI="reftable://<path> git repo info references.format
>
> it would still report what ever the repository was originally configured
> with. Since only a single reference backend can be used at time, I
> wonder if we should go a bit further and update `r->ref_storage_format`
> to be inline with how the repository reference backend is configured via
> `GIT_REF_URI`.
>

Updating it here won't here won't work, this flow is lazy and only
evaluated when you actually want to deal with references.

Commands like 'git repo info reference.format' will not trigger this
flow and will only read the config. I'm also not sure we should be
modifying it. Because the output of such a command is to note how the
repository is configured. We are not changing that configuration, but
instead we're simply asking the git to use a different backend for when
the env is provided. What do you think?

> -Justin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
> This asymmetry makes it difficult to test different reference backends
> or use alternative reference storage locations without modifying the
> repository structure.
>
> Add a new environment variable 'GIT_REF_URI' that specifies both the
> reference backend and directory path using a URI format:
>
>     <ref_backend>://<path>

I like this idea. This would allow us in the future to also do something
like:

    reftable+nfs://10.11.12.13/ref-dir

> When set, this variable is used to obtain the main reference store for
> all Git commands. The variable is checked in `get_main_ref_store()`
> when lazily assigning `repo->refs_private`. We cannot initialize this
> earlier in `repo_set_gitdir()` because the repository's hash algorithm
> isn't known at that point, and the reftable backend requires this
> information during initialization.
>
> When used with worktrees, the specified directory is treated as the
> reference directory for all worktree operations.
>
> Add a new test file 't1423-ref-backend.sh' to test this environment
> variable.
>
> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
>  Documentation/git.adoc |   8 ++++
>  environment.h          |   1 +
>  refs.c                 |  53 +++++++++++++++++++++++-
>  t/meson.build          |   1 +
>  t/t1423-ref-backend.sh | 109 +++++++++++++++++++++++++++++++++++++++++++++++++
>  5 files changed, 171 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
> index ce099e78b8..a1d1078f42 100644
> --- a/Documentation/git.adoc
> +++ b/Documentation/git.adoc
> @@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
>  	repositories will be set to this value. The default is "files".
>  	See `--ref-format` in linkgit:git-init[1].
>  
> +`GIT_REF_URI`::
> +    Specify which reference backend and path to be used, if not specified the
> +    backend is inferred from the configuration and $GIT_DIR is used as the
> +    path.
> ++
> +Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
> +reference backend and the 'path' specifies the directory used by the backend.
> +
>  Git Commits
>  ~~~~~~~~~~~
>  `GIT_AUTHOR_NAME`::
> diff --git a/environment.h b/environment.h
> index 51898c99cd..9bc380bba4 100644
> --- a/environment.h
> +++ b/environment.h
> @@ -42,6 +42,7 @@
>  #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
>  #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
>  #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
> +#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
>  
>  /*
>   * Environment variable used to propagate the --no-advice global option to the
> diff --git a/refs.c b/refs.c
> index 23f46867f2..0922f08c9f 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2186,15 +2186,66 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
>  	return maybe_debug_wrap_ref_store(dir, ref_store);
>  }
>  
> +static struct ref_store *get_ref_store_from_uri(struct repository *repo,
> +						const char *uri)
> +{
> +	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
> +	enum ref_storage_format format;
> +	struct ref_store *store = NULL;
> +	char *format_string;
> +	char *dir;
> +
> +	if (!uri || !uri[0]) {
> +		error("reference backend uri is empty");

I see no localization on any of the error() or die() messages. I think
it's worth to make them translatable.

> +		goto cleanup;
> +	}
> +
> +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
> +		error("invalid reference backend uri format '%s'", uri);
> +		goto cleanup;
> +	}
> +
> +	format_string = ref_backend_info.items[0].string;
> +	dir = ref_backend_info.items[1].string + 2;

Length check before jumping to the third char would be adviced. Also I
think it's worth to check if the first two chars are "//".

-- 
Cheers,
Toon

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Junio C Hamano]

Toon Claes <toon@iotcl.com> writes:

>>     <ref_backend>://<path>
>
> I like this idea. This would allow us in the future to also do something
> like:
>
>     reftable+nfs://10.11.12.13/ref-dir

I actually thought from Karthik's definition that what you are
trying to say is spelled more like this:

    reftable://nfs://10.11.12.13/ref-dir

IOW, the underlying URI to "reach the resource" is in the <path>
part (i.e., "nfs://<addr>/<directory>").  And I found it somewhat a
strange syntax, because the "to reach the resource, visit this" URI
may not necessarily look like <path>, and I also wondered if
spelling it like <ref_backend>:<URI-for-resource> is more
appropriate.

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1105 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Toon Claes <toon@iotcl.com> writes:
>
>>>     <ref_backend>://<path>
>>
>> I like this idea. This would allow us in the future to also do something
>> like:
>>
>>     reftable+nfs://10.11.12.13/ref-dir
>
> I actually thought from Karthik's definition that what you are
> trying to say is spelled more like this:
>
>     reftable://nfs://10.11.12.13/ref-dir
>

You're indeed correct.

> IOW, the underlying URI to "reach the resource" is in the <path>
> part (i.e., "nfs://<addr>/<directory>").  And I found it somewhat a
> strange syntax, because the "to reach the resource, visit this" URI
> may not necessarily look like <path>, and I also wondered if
> spelling it like <ref_backend>:<URI-for-resource> is more
> appropriate.

This is a much better way to state what I was going for, I was
considering the entire <ref_backend>:<path> as the URI since currently
we only deal with FS paths (for files and reftable). But that could
potentially change. As such, it makes sense to state it as
<ref_backend>:<URI-for-resource>. Will modify accordingly.

Thanks.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> Junio C Hamano <gitster@pobox.com> writes:
>
>> Toon Claes <toon@iotcl.com> writes:
>>
>>>>     <ref_backend>://<path>
>>>
>>> I like this idea. This would allow us in the future to also do something
>>> like:
>>>
>>>     reftable+nfs://10.11.12.13/ref-dir
>>
>> I actually thought from Karthik's definition that what you are
>> trying to say is spelled more like this:
>>
>>     reftable://nfs://10.11.12.13/ref-dir

LOL, I didn't realize since 07c7782cc8 (Disown ssh+git and git+ssh,
2016-02-15) 'ssh+git' isn't actually mentioned no more.

>> IOW, the underlying URI to "reach the resource" is in the <path>
>> part (i.e., "nfs://<addr>/<directory>").  And I found it somewhat a
>> strange syntax, because the "to reach the resource, visit this" URI
>> may not necessarily look like <path>, and I also wondered if
>> spelling it like <ref_backend>:<URI-for-resource> is more
>> appropriate.
>
> This is a much better way to state what I was going for, I was
> considering the entire <ref_backend>:<path> as the URI since currently
> we only deal with FS paths (for files and reftable). But that could
> potentially change. As such, it makes sense to state it as
> <ref_backend>:<URI-for-resource>. Will modify accordingly.

I don't really care on the format, as long as we're consistent. I know
Patrick is working on pluggable object databases and also there he's
suggesting to do `<backend_type>://<URI>` which might also lead to
double `://` use. But I guess that's still subjected to change.

That said. Should we change the current proposal of
`<ref_backend>://<path>` to `<ref_backend>:file://<path>`? Or can we
simply assume the latter implied from the former?


-- 
Cheers,
Toon

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

Toon Claes <toon@iotcl.com> writes:

[snip]

>> diff --git a/refs.c b/refs.c
>> index 23f46867f2..0922f08c9f 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2186,15 +2186,66 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
>>  	return maybe_debug_wrap_ref_store(dir, ref_store);
>>  }
>>
>> +static struct ref_store *get_ref_store_from_uri(struct repository *repo,
>> +						const char *uri)
>> +{
>> +	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
>> +	enum ref_storage_format format;
>> +	struct ref_store *store = NULL;
>> +	char *format_string;
>> +	char *dir;
>> +
>> +	if (!uri || !uri[0]) {
>> +		error("reference backend uri is empty");
>
> I see no localization on any of the error() or die() messages. I think
> it's worth to make them translatable.
>

Yeah, that makes sense.

>> +		goto cleanup;
>> +	}
>> +
>> +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
>> +		error("invalid reference backend uri format '%s'", uri);
>> +		goto cleanup;
>> +	}
>> +
>> +	format_string = ref_backend_info.items[0].string;
>> +	dir = ref_backend_info.items[1].string + 2;
>
> Length check before jumping to the third char would be adviced. Also I
> think it's worth to check if the first two chars are "//".
>

This is a good point, will add a test and fix this up.

> --
> Cheers,
> Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Patrick Steinhardt]

On Wed, Nov 19, 2025 at 10:48:53PM +0100, Karthik Nayak wrote:
> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
> This asymmetry makes it difficult to test different reference backends
> or use alternative reference storage locations without modifying the
> repository structure.
> 
> Add a new environment variable 'GIT_REF_URI' that specifies both the
> reference backend and directory path using a URI format:
> 
>     <ref_backend>://<path>
> 
> When set, this variable is used to obtain the main reference store for
> all Git commands. The variable is checked in `get_main_ref_store()`
> when lazily assigning `repo->refs_private`. We cannot initialize this
> earlier in `repo_set_gitdir()` because the repository's hash algorithm
> isn't known at that point, and the reftable backend requires this
> information during initialization.
> 
> When used with worktrees, the specified directory is treated as the
> reference directory for all worktree operations.
> 
> Add a new test file 't1423-ref-backend.sh' to test this environment
> variable.

Based on my reply in <aS2V4TKeS4V_oxAb@pks.im> I wonder whether we want
to take a bit of a different approach:

  - We extend the format understood by "extensions.refStorage" to
    understand "schema://data"-style strings and adapt the "data" part
    to be passed through to the reference backend.

  - We then use the same mechanism to parse both "extensions.refStorage"
    and the environment variable.

This would have a couple advantages:

  - We make the ref storage extension more flexible so that you can move
    your reference backends somewhere else entirely.

  - We prepare for a potential future ref format that _needs_ to receive
    data as input.

  - We have consistent behaviour between the environment variable and
    the extension. So basically, the environment variable starts to
    behave as an override to the extension.

One issue that we'd then have to solve is how to derive the worktree
references from the backend. Arguably though, I think that the extension
that was specified should also be sufficient to identify the location of
the worktree references.

We'd have to refactor the code base a bit though to properly reflect
that in our tree. One way to do this is to extend `ref_store_init()` so
that it receives the worktree (or NULL) as input. In that case, we would
continue to pass the combination of format and "data" to the init
function, and it would then know to locate the worktree references
itself.

What do you think?

Thanks!

Patrick

Re: [PATCH 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3421 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Wed, Nov 19, 2025 at 10:48:53PM +0100, Karthik Nayak wrote:
>> Git allows setting a different object directory via
>> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
>> This asymmetry makes it difficult to test different reference backends
>> or use alternative reference storage locations without modifying the
>> repository structure.
>>
>> Add a new environment variable 'GIT_REF_URI' that specifies both the
>> reference backend and directory path using a URI format:
>>
>>     <ref_backend>://<path>
>>
>> When set, this variable is used to obtain the main reference store for
>> all Git commands. The variable is checked in `get_main_ref_store()`
>> when lazily assigning `repo->refs_private`. We cannot initialize this
>> earlier in `repo_set_gitdir()` because the repository's hash algorithm
>> isn't known at that point, and the reftable backend requires this
>> information during initialization.
>>
>> When used with worktrees, the specified directory is treated as the
>> reference directory for all worktree operations.
>>
>> Add a new test file 't1423-ref-backend.sh' to test this environment
>> variable.
>
> Based on my reply in <aS2V4TKeS4V_oxAb@pks.im> I wonder whether we want
> to take a bit of a different approach:
>
>   - We extend the format understood by "extensions.refStorage" to
>     understand "schema://data"-style strings and adapt the "data" part
>     to be passed through to the reference backend.
>
>   - We then use the same mechanism to parse both "extensions.refStorage"
>     and the environment variable.
>
> This would have a couple advantages:
>
>   - We make the ref storage extension more flexible so that you can move
>     your reference backends somewhere else entirely.
>
>   - We prepare for a potential future ref format that _needs_ to receive
>     data as input.
>
>   - We have consistent behaviour between the environment variable and
>     the extension. So basically, the environment variable starts to
>     behave as an override to the extension.
>

I did read/respond to your reply there and I agree with your suggested
approach. An additional advantage would be that this would also mean the
ENV variable is more deeply integrated. So the backend override added by
the ENV variable would also show up when running `git repo info`.

> One issue that we'd then have to solve is how to derive the worktree
> references from the backend. Arguably though, I think that the extension
> that was specified should also be sufficient to identify the location of
> the worktree references.
>
> We'd have to refactor the code base a bit though to properly reflect
> that in our tree. One way to do this is to extend `ref_store_init()` so
> that it receives the worktree (or NULL) as input. In that case, we would
> continue to pass the combination of format and "data" to the init
> function, and it would then know to locate the worktree references
> itself.
>

Yeah, I'm considering adding this information to the `repository`
structure, so along with `ref_storage_format`, it would also contain a
`ref_storage_data` which would be passed down to `get_main_ref_store()`
which would in-turn call `ref_store_init()`.

In that sense, when in a worktree the $GIT_DIR is set appropriately and
this should all work accordingly.

> What do you think?
>
> Thanks!
>
> Patrick

Sounds great. Thanks for the input

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 0/2] refs: allow setting the reference directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> While Git allows users to select different reference backends, unlike
> with objects, there is no flexibility in selecting the reference
> directory. Currently, the reference format is obtained from the config
> of the repository and the reference directory is set to the $GIT_DIR.

I actually am not sure if I like the proposed environment variable.

The proposal is based on an assumption that any reference backend
should be able to move their backing store anywhere, and they should
be able to express the location of their backing store as a single
string <path>.  For a new backend, "where is your backing store" may
not even be a question that does not make much sense (as "somewhere
in the cloud that you do not even have to know" is certainly
possible), and even for a new backend design that does allow such a
question to have a meaningful answer, this "you have to be able to
use a random place specified by this environment variable as your
backing storage" is an additional requirement that its implementors
may not need to satisfy in order to please their user base.

For reftable and files backends, these assumptions may be true, but
then it is not too cumbersome if these stay to be backend specific,
as there are only two backends.

So I dunno.  In addition, if this is designed to help migration
(which is the impression I am getting from the cover letter
description), don't you need a way to specify more than one (i.e.,
source to migrate from and destination to migrate to)?  With a
single GIT_REF_URI, it would not be obvious what it refers to,
whether it is an additional place to write to, to read from, or
something completely unrelated.  For example ...

> This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
> reference backend and path in a URI form:
>
>     <reference_backend>://<path>
>
> For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.
>
> One use case for this is migration between different backends. On the
> server side, migrating from the files backend to the newly introduced
> reftable backend can be achieved by running 'git refs migrate'. However,
> for large repositories with millions of references, this migration can
> take from seconds to minutes.
>
> We could make the migration non-blocking by running the migration in the
> background and capturing and replaying updates to both backends. This
> would require Git to support writing references to different reference
> backends and paths.

... I am reading that the above is saying that the system will write
to whatever reference backend specified in the extension.refStorage,
plus also where GIT_REF_URI points at, but if that is the way how
the mechanism works, the variable should be named more specific to
what it does, no?  It is not just a random "REF URI"; it is an
additional ref backend that the updates are dumped to.  Maybe there
would be a different use case where you may want to read from two
reference backends, and you'd need to specify the secondary one with
an environment variable, but if the system behaves one specific way
for GIT_REF_URI (say, all updates are also copied to this additional
ref backend at the specified ref backing store), a different
environment variable name needs to be chosen to serve such a
different use case, no?

Re: [PATCH 0/2] refs: allow setting the reference directory

[Patrick Steinhardt]

On Sat, Nov 22, 2025 at 08:29:22PM -0800, Junio C Hamano wrote:
> Karthik Nayak <karthik.188@gmail.com> writes:
> 
> > While Git allows users to select different reference backends, unlike
> > with objects, there is no flexibility in selecting the reference
> > directory. Currently, the reference format is obtained from the config
> > of the repository and the reference directory is set to the $GIT_DIR.
> 
> I actually am not sure if I like the proposed environment variable.
> 
> The proposal is based on an assumption that any reference backend
> should be able to move their backing store anywhere, and they should
> be able to express the location of their backing store as a single
> string <path>.  For a new backend, "where is your backing store" may
> not even be a question that does not make much sense (as "somewhere
> in the cloud that you do not even have to know" is certainly
> possible), and even for a new backend design that does allow such a
> question to have a meaningful answer, this "you have to be able to
> use a random place specified by this environment variable as your
> backing storage" is an additional requirement that its implementors
> may not need to satisfy in order to please their user base.
> 
> For reftable and files backends, these assumptions may be true, but
> then it is not too cumbersome if these stay to be backend specific,
> as there are only two backends.

I think it's a reasonable assumption to make that the path _can_ be
represented as a single string. For now, we don't really require any
configuration for the backend in the first place. So all you need to do
is to say:

    [extension]
    refStorage = reftable

This implicitly identifies the location of the backend, too, as we
derive it from the commondir/gitdir. As you say that's sufficient for
the "files" and "reftable" backends, but it may be insufficient for
other backends.

Suppose that we for example have a Postgres database to store data. It's
clearly not sufficient to specify "extension.refStorage=postgres", as
that wouldn't give you enough information to also know how to connect to
the database.

It's a problem I have been thinking about quite a lot in the context of
pluggable object databases, as well. Ultimately, the solution I arrived
at is to extend the extension format itself. For pluggable ODBs this
would look like this:

    [extension]
    objectStorage = postgres://127.0.0.1:5432?database=myrepo

This is similar to a normal URI with a schema: everything before the
"://" identifies the format that is to be used, and everything after is
then passed as-is to the backend itself. I think this should give us
enough flexibility for any future formats and it is easy enough to
configure. The added benefit is that this can also work in contexts like
the GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES
environment variables, even though their naming is off now.

For the reference storage I think we should be moving into a similar
direction. Sure, for the current formats that we know its sufficient to
only specify their directory. But I think we should treat the directory
as an opaque string and then let the reference backend handle it, same
as with the proposed format for object databases:

    # A schema-only variable will be treated as if we specified the
    # common directory.
    [extension]
    refStorage = reftable

    # It's also possible to explicitly specify a different location for
    # the backend.
    [extension]
    refStorage = reftable:///foo/bar

    # And same as above, we can also specify non-locations.
    [extension]
    refStorage = postgres://127.0.0.1:5432?database=myrepo

As said, the important thing here is that the reference backends get the
string after the schema as opaque blobs that they can self-interpret.

> So I dunno.  In addition, if this is designed to help migration
> (which is the impression I am getting from the cover letter
> description), don't you need a way to specify more than one (i.e.,
> source to migrate from and destination to migrate to)?  With a
> single GIT_REF_URI, it would not be obvious what it refers to,
> whether it is an additional place to write to, to read from, or
> something completely unrelated.  For example ...

I think we cannot easily retrofit handling of multiple refdbs into Git
at this point in time anymore. The way to drive this would be that we
have two processes:

  - One `git refs list` process in the repository that uses the old
    format.

  - One `git update-ref --stdin` process in the repository that uses the
    new format specified via GIT_REF_URI.

This allows us to do an online migration of data into a separate ref
store.

> > This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
> > reference backend and path in a URI form:
> >
> >     <reference_backend>://<path>
> >
> > For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.
> >
> > One use case for this is migration between different backends. On the
> > server side, migrating from the files backend to the newly introduced
> > reftable backend can be achieved by running 'git refs migrate'. However,
> > for large repositories with millions of references, this migration can
> > take from seconds to minutes.
> >
> > We could make the migration non-blocking by running the migration in the
> > background and capturing and replaying updates to both backends. This
> > would require Git to support writing references to different reference
> > backends and paths.
> 
> ... I am reading that the above is saying that the system will write
> to whatever reference backend specified in the extension.refStorage,
> plus also where GIT_REF_URI points at, but if that is the way how
> the mechanism works, the variable should be named more specific to
> what it does, no?  It is not just a random "REF URI"; it is an
> additional ref backend that the updates are dumped to.  Maybe there
> would be a different use case where you may want to read from two
> reference backends, and you'd need to specify the secondary one with
> an environment variable, but if the system behaves one specific way
> for GIT_REF_URI (say, all updates are also copied to this additional
> ref backend at the specified ref backing store), a different
> environment variable name needs to be chosen to serve such a
> different use case, no?

Truth be told, I'm not realy a huge fan of the name, either. But as
said, I don't think we can easily "overlay" multiple refdbs, as it would
lead to various different questions due to our hierarchical layout of
references.

That being said, I personally would prefer `GIT_REFERENCE_BACKEND` as
variable name that accepts exactly the same kind of strings as the
`extension.refStorage` values I have proposed above.

Thanks!

Patrick

Re: [PATCH 0/2] refs: allow setting the reference directory

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> For the reference storage I think we should be moving into a similar
> direction. Sure, for the current formats that we know its sufficient to
> only specify their directory. But I think we should treat the directory
> as an opaque string and then let the reference backend handle it, same
> as with the proposed format for object databases:
>
>     # A schema-only variable will be treated as if we specified the
>     # common directory.
>     [extension]
>     refStorage = reftable
>
>     # It's also possible to explicitly specify a different location for
>     # the backend.
>     [extension]
>     refStorage = reftable:///foo/bar
>
>     # And same as above, we can also specify non-locations.
>     [extension]
>     refStorage = postgres://127.0.0.1:5432?database=myrepo

Cute.  I kinda like it ;-)

Re: [PATCH 0/2] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 7538 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Sat, Nov 22, 2025 at 08:29:22PM -0800, Junio C Hamano wrote:
>> Karthik Nayak <karthik.188@gmail.com> writes:
>>
>> > While Git allows users to select different reference backends, unlike
>> > with objects, there is no flexibility in selecting the reference
>> > directory. Currently, the reference format is obtained from the config
>> > of the repository and the reference directory is set to the $GIT_DIR.
>>
>> I actually am not sure if I like the proposed environment variable.
>>
>> The proposal is based on an assumption that any reference backend
>> should be able to move their backing store anywhere, and they should
>> be able to express the location of their backing store as a single
>> string <path>.  For a new backend, "where is your backing store" may
>> not even be a question that does not make much sense (as "somewhere
>> in the cloud that you do not even have to know" is certainly
>> possible), and even for a new backend design that does allow such a
>> question to have a meaningful answer, this "you have to be able to
>> use a random place specified by this environment variable as your
>> backing storage" is an additional requirement that its implementors
>> may not need to satisfy in order to please their user base.
>>
>> For reftable and files backends, these assumptions may be true, but
>> then it is not too cumbersome if these stay to be backend specific,
>> as there are only two backends.
>
> I think it's a reasonable assumption to make that the path _can_ be
> represented as a single string. For now, we don't really require any
> configuration for the backend in the first place. So all you need to do
> is to say:
>
>     [extension]
>     refStorage = reftable
>
> This implicitly identifies the location of the backend, too, as we
> derive it from the commondir/gitdir. As you say that's sufficient for
> the "files" and "reftable" backends, but it may be insufficient for
> other backends.
>
> Suppose that we for example have a Postgres database to store data. It's
> clearly not sufficient to specify "extension.refStorage=postgres", as
> that wouldn't give you enough information to also know how to connect to
> the database.
>
> It's a problem I have been thinking about quite a lot in the context of
> pluggable object databases, as well. Ultimately, the solution I arrived
> at is to extend the extension format itself. For pluggable ODBs this
> would look like this:
>
>     [extension]
>     objectStorage = postgres://127.0.0.1:5432?database=myrepo
>
> This is similar to a normal URI with a schema: everything before the
> "://" identifies the format that is to be used, and everything after is
> then passed as-is to the backend itself. I think this should give us
> enough flexibility for any future formats and it is easy enough to
> configure. The added benefit is that this can also work in contexts like
> the GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES
> environment variables, even though their naming is off now.
>
> For the reference storage I think we should be moving into a similar
> direction. Sure, for the current formats that we know its sufficient to
> only specify their directory. But I think we should treat the directory
> as an opaque string and then let the reference backend handle it, same
> as with the proposed format for object databases:
>
>     # A schema-only variable will be treated as if we specified the
>     # common directory.
>     [extension]
>     refStorage = reftable
>
>     # It's also possible to explicitly specify a different location for
>     # the backend.
>     [extension]
>     refStorage = reftable:///foo/bar
>
>     # And same as above, we can also specify non-locations.
>     [extension]
>     refStorage = postgres://127.0.0.1:5432?database=myrepo
>
> As said, the important thing here is that the reference backends get the
> string after the schema as opaque blobs that they can self-interpret.
>

I think you bring in some good points here, I didn't think of
`extension.refStorage` and I think we can extend that like you
mentioned, while staying backwards compatible.

>> So I dunno.  In addition, if this is designed to help migration
>> (which is the impression I am getting from the cover letter
>> description), don't you need a way to specify more than one (i.e.,
>> source to migrate from and destination to migrate to)?  With a
>> single GIT_REF_URI, it would not be obvious what it refers to,
>> whether it is an additional place to write to, to read from, or
>> something completely unrelated.  For example ...
>
> I think we cannot easily retrofit handling of multiple refdbs into Git
> at this point in time anymore. The way to drive this would be that we
> have two processes:
>
>   - One `git refs list` process in the repository that uses the old
>     format.
>
>   - One `git update-ref --stdin` process in the repository that uses the
>     new format specified via GIT_REF_URI.
>
> This allows us to do an online migration of data into a separate ref
> store.
>

That's exactly the mechanism I was talking about, thanks for explaining.

>> > This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
>> > reference backend and path in a URI form:
>> >
>> >     <reference_backend>://<path>
>> >
>> > For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.
>> >
>> > One use case for this is migration between different backends. On the
>> > server side, migrating from the files backend to the newly introduced
>> > reftable backend can be achieved by running 'git refs migrate'. However,
>> > for large repositories with millions of references, this migration can
>> > take from seconds to minutes.
>> >
>> > We could make the migration non-blocking by running the migration in the
>> > background and capturing and replaying updates to both backends. This
>> > would require Git to support writing references to different reference
>> > backends and paths.
>>
>> ... I am reading that the above is saying that the system will write
>> to whatever reference backend specified in the extension.refStorage,
>> plus also where GIT_REF_URI points at, but if that is the way how
>> the mechanism works, the variable should be named more specific to
>> what it does, no?  It is not just a random "REF URI"; it is an
>> additional ref backend that the updates are dumped to.  Maybe there
>> would be a different use case where you may want to read from two
>> reference backends, and you'd need to specify the secondary one with
>> an environment variable, but if the system behaves one specific way
>> for GIT_REF_URI (say, all updates are also copied to this additional
>> ref backend at the specified ref backing store), a different
>> environment variable name needs to be chosen to serve such a
>> different use case, no?
>
> Truth be told, I'm not realy a huge fan of the name, either. But as
> said, I don't think we can easily "overlay" multiple refdbs, as it would
> lead to various different questions due to our hierarchical layout of
> references.
>
> That being said, I personally would prefer `GIT_REFERENCE_BACKEND` as
> variable name that accepts exactly the same kind of strings as the
> `extension.refStorage` values I have proposed above.
>

Fair enough. Once both the env variable and `extension.refStorage` take
in the same input, it does make sense to rename the env variable to
`GIT_REFERENCE_BACKEND`.

> Thanks!
>
> Patrick

Thanks for your input. I'll make the necessary changes for v4 :)

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v2 0/2] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
reference backend and path in a URI form:

    <reference_backend>://<URI-for-resource>

For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

The first commit adds the required changes to create a 'ref_store' for a
given path. The second commit parses the URI if available when creating
the main ref store.

This is based on top of 9a2fb147f2 (Git 2.52, 2025-11-17).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/git.adoc |   8 ++++
 environment.h          |   1 +
 refs.c                 |  71 +++++++++++++++++++++++++++--
 t/meson.build          |   1 +
 t/t1423-ref-backend.sh | 121 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 199 insertions(+), 3 deletions(-)

Karthik Nayak (2):
      refs: support obtaining ref_store for given dir
      refs: add GIT_REF_URI to specify reference backend and directory

Range-diff versus v1:

1:  f6e8aa37fe ! 1:  c925726efd refs: support obtaining ref_store for given dir
    @@ Commit message
         The refs subsystem uses the `get_main_ref_store()` to obtain the main
         ref_store for a given repository. In the upcoming patches we also want
         to create a ref_store for any given reference directory, which may exist
    -    in arbitrary paths. To support such behavior, extract out the core logic
    -    for creating out the ref_store from `get_main_ref_store()` into a new
    -    function `get_ref_store_for_dir()` which can provide the ref_store for a
    +    in arbitrary paths. For the files backend and the reftable backend, the
    +    reference directory is generally the $GIT_DIR.
    +
    +    To support such behavior, extract out the core logic for creating out
    +    the ref_store from `get_main_ref_store()` into a new function
    +    `get_ref_store_for_dir()` which can provide the ref_store for a
         given (repository, directory, reference format) combination.
     
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
2:  5e30fa334e ! 2:  b859ebad64 refs: add GIT_REF_URI to specify reference backend and directory
    @@ Commit message
         Add a new environment variable 'GIT_REF_URI' that specifies both the
         reference backend and directory path using a URI format:
     
    -        <ref_backend>://<path>
    +        <ref_backend>://<URI-for-resource>
     
         When set, this variable is used to obtain the main reference store for
         all Git commands. The variable is checked in `get_main_ref_store()`
    @@ Commit message
         Add a new test file 't1423-ref-backend.sh' to test this environment
         variable.
     
    +    Helped-by: Jean-Noël Avila <jn.avila@free.fr>
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
     
      ## Documentation/git.adoc ##
    @@ Documentation/git.adoc: double-quotes and respecting backslash escapes. E.g., th
      	See `--ref-format` in linkgit:git-init[1].
      
     +`GIT_REF_URI`::
    -+    Specify which reference backend and path to be used, if not specified the
    -+    backend is inferred from the configuration and $GIT_DIR is used as the
    -+    path.
    ++    Specify which reference backend to be used along with its URI. Reference
    ++    backends like the files, reftable backend use the $GIT_DIR as their URI.
     ++
    -+Expects the format '<ref_backend>://<path>', where the 'backend' specifies the
    -+reference backend and the 'path' specifies the directory used by the backend.
    ++Expects the format `<ref_backend>://<URI-for-resource>`, where the
    ++_<ref_backend>_ specifies the reference backend and the _<URI-for-resource>_
    ++specifies the URI used by the backend.
     +
      Git Commits
      ~~~~~~~~~~~
    @@ refs.c: static struct ref_store *get_ref_store_for_dir(struct repository *r,
     +	}
     +
     +	format_string = ref_backend_info.items[0].string;
    ++	if (!starts_with(ref_backend_info.items[1].string, "//")) {
    ++		error("invalid reference backend uri format '%s'", uri);
    ++		goto cleanup;
    ++	}
    ++	dir = ref_backend_info.items[1].string + 2;
    ++
    ++	format_string = ref_backend_info.items[0].string;
     +	dir = ref_backend_info.items[1].string + 2;
     +
     +	if (!dir || !dir[0]) {
    @@ t/t1423-ref-backend.sh (new)
     +		cd repo &&
     +		GIT_REF_URI="" &&
     +		export GIT_REF_URI &&
    -+		! git refs list 2>err &&
    ++		test_must_fail git refs list 2>err &&
     +		test_grep "reference backend uri is empty" err
     +	)
     +'
    @@ t/t1423-ref-backend.sh (new)
     +		cd repo &&
     +		GIT_REF_URI="reftable@/home/reftable" &&
     +		export GIT_REF_URI &&
    -+		! git refs list 2>err &&
    ++		test_must_fail git refs list 2>err &&
     +		test_grep "invalid reference backend uri format" err
     +	)
     +'
    @@ t/t1423-ref-backend.sh (new)
     +		cd repo &&
     +		GIT_REF_URI="reftable://" &&
     +		export GIT_REF_URI &&
    -+		! git refs list 2>err &&
    ++		test_must_fail git refs list 2>err &&
     +		test_grep "invalid path in uri" err
     +	)
     +'
     +
    ++test_expect_success 'uri ends at colon' '
    ++	test_when_finished "rm -rf repo" &&
    ++	git init --ref-format=files repo &&
    ++	(
    ++		cd repo &&
    ++		GIT_REF_URI="reftable:" &&
    ++		export GIT_REF_URI &&
    ++		test_must_fail git refs list 2>err &&
    ++		test_grep "invalid reference backend uri format" err
    ++	)
    ++'
    ++
     +test_expect_success 'unknown reference backend' '
     +	test_when_finished "rm -rf repo" &&
     +	git init --ref-format=files repo &&
    @@ t/t1423-ref-backend.sh (new)
     +		cd repo &&
     +		GIT_REF_URI="db://.git" &&
     +		export GIT_REF_URI &&
    -+		! git refs list 2>err &&
    ++		test_must_fail git refs list 2>err &&
     +		test_grep "unknown reference backend" err
     +	)
     +'
    @@ t/t1423-ref-backend.sh (new)
     +			continue
     +		fi
     +
    -+		test_expect_success 'read from other reference backend' '
    ++		test_expect_success "read from $to_format backend" '
     +			test_when_finished "rm -rf repo" &&
    -+			git init --ref-format=files repo &&
    ++			git init --ref-format=$from_format repo &&
     +			(
     +				cd repo &&
     +				test_commit 1 &&
     +				test_commit 2 &&
     +				test_commit 3 &&
     +
    -+				git refs migrate --dry-run --ref-format=reftable >out &&
    -+				REFTABLE_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
    ++				git refs migrate --dry-run --ref-format=$to_format >out &&
    ++				BACKEND_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
     +				git refs list >expect &&
    -+				GIT_REF_URI="reftable://$REFTABLE_PATH" git refs list >actual &&
    ++				GIT_REF_URI="$to_format://$BACKEND_PATH" git refs list >actual &&
     +				test_cmp expect actual
     +			)
     +		'
     +
    -+		test_expect_success 'write to other reference backend' '
    ++		test_expect_success "write to $to_format backend" '
     +			test_when_finished "rm -rf repo" &&
    -+			git init --ref-format=files repo &&
    ++			git init --ref-format=$from_format repo &&
     +			(
     +				cd repo &&
     +				test_commit 1 &&
     +				test_commit 2 &&
     +				test_commit 3 &&
     +
    -+				git refs migrate --dry-run --ref-format=reftable >out &&
    ++				git refs migrate --dry-run --ref-format=$to_format >out &&
     +				git refs list >expect &&
     +
    -+				REFTABLE_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
    -+				GIT_REF_URI="reftable://$REFTABLE_PATH" git tag -d 1 &&
    ++				BACKEND_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
    ++				GIT_REF_URI="$to_format://$BACKEND_PATH" git tag -d 1 &&
     +
     +				git refs list >actual &&
     +				test_cmp expect actual &&
     +
    -+				GIT_REF_URI="reftable://$REFTABLE_PATH" git refs list >expect &&
    ++				GIT_REF_URI="$to_format://$BACKEND_PATH" git refs list >expect &&
     +				git refs list >out &&
     +				cat out | grep -v "refs/tags/1" >actual &&
     +				test_cmp expect actual


base-commit: 9a2fb147f2c61d0cab52c883e7e26f5b7948e3ed
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v2 1/2] refs: support obtaining ref_store for given dir

[Karthik Nayak]

The refs subsystem uses the `get_main_ref_store()` to obtain the main
ref_store for a given repository. In the upcoming patches we also want
to create a ref_store for any given reference directory, which may exist
in arbitrary paths. For the files backend and the reftable backend, the
reference directory is generally the $GIT_DIR.

To support such behavior, extract out the core logic for creating out
the ref_store from `get_main_ref_store()` into a new function
`get_ref_store_for_dir()` which can provide the ref_store for a
given (repository, directory, reference format) combination.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/refs.c b/refs.c
index 965381367e..23f46867f2 100644
--- a/refs.c
+++ b/refs.c
@@ -2177,6 +2177,15 @@ void ref_store_release(struct ref_store *ref_store)
 	free(ref_store->gitdir);
 }
 
+static struct ref_store *get_ref_store_for_dir(struct repository *r,
+					       char *dir,
+					       enum ref_storage_format format)
+{
+	struct ref_store *ref_store = ref_store_init(r, format, dir,
+						     REF_STORE_ALL_CAPS);
+	return maybe_debug_wrap_ref_store(dir, ref_store);
+}
+
 struct ref_store *get_main_ref_store(struct repository *r)
 {
 	if (r->refs_private)
@@ -2185,9 +2194,7 @@ struct ref_store *get_main_ref_store(struct repository *r)
 	if (!r->gitdir)
 		BUG("attempting to get main_ref_store outside of repository");
 
-	r->refs_private = ref_store_init(r, r->ref_storage_format,
-					 r->gitdir, REF_STORE_ALL_CAPS);
-	r->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);
+	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
 	return r->refs_private;
 }
 

-- 
2.51.2

Re: [PATCH v2 1/2] refs: support obtaining ref_store for given dir

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> The refs subsystem uses the `get_main_ref_store()` to obtain the main
> ref_store for a given repository. In the upcoming patches we also want
> to create a ref_store for any given reference directory, which may exist
> in arbitrary paths. For the files backend and the reftable backend, the
> reference directory is generally the $GIT_DIR.
>
> To support such behavior, extract out the core logic for creating out
> the ref_store from `get_main_ref_store()` into a new function
> `get_ref_store_for_dir()` which can provide the ref_store for a
> given (repository, directory, reference format) combination.

I am guessing that this is meant to work with the REF_URI thing, and
the <path> part in REF_URI=<backend>:<path> corresponds to the "dir"
parameter here.

Looks like a good no-op split.

> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
>  refs.c | 13 ++++++++++---
>  1 file changed, 10 insertions(+), 3 deletions(-)
>
> diff --git a/refs.c b/refs.c
> index 965381367e..23f46867f2 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2177,6 +2177,15 @@ void ref_store_release(struct ref_store *ref_store)
>  	free(ref_store->gitdir);
>  }
>  
> +static struct ref_store *get_ref_store_for_dir(struct repository *r,
> +					       char *dir,
> +					       enum ref_storage_format format)
> +{
> +	struct ref_store *ref_store = ref_store_init(r, format, dir,
> +						     REF_STORE_ALL_CAPS);
> +	return maybe_debug_wrap_ref_store(dir, ref_store);
> +}
> +
>  struct ref_store *get_main_ref_store(struct repository *r)
>  {
>  	if (r->refs_private)
> @@ -2185,9 +2194,7 @@ struct ref_store *get_main_ref_store(struct repository *r)
>  	if (!r->gitdir)
>  		BUG("attempting to get main_ref_store outside of repository");
>  
> -	r->refs_private = ref_store_init(r, r->ref_storage_format,
> -					 r->gitdir, REF_STORE_ALL_CAPS);
> -	r->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);
> +	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
>  	return r->refs_private;
>  }

[PATCH v2 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
This asymmetry makes it difficult to test different reference backends
or use alternative reference storage locations without modifying the
repository structure.

Add a new environment variable 'GIT_REF_URI' that specifies both the
reference backend and directory path using a URI format:

    <ref_backend>://<URI-for-resource>

When set, this variable is used to obtain the main reference store for
all Git commands. The variable is checked in `get_main_ref_store()`
when lazily assigning `repo->refs_private`. We cannot initialize this
earlier in `repo_set_gitdir()` because the repository's hash algorithm
isn't known at that point, and the reftable backend requires this
information during initialization.

When used with worktrees, the specified directory is treated as the
reference directory for all worktree operations.

Add a new test file 't1423-ref-backend.sh' to test this environment
variable.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   8 ++++
 environment.h          |   1 +
 refs.c                 |  60 +++++++++++++++++++++++-
 t/meson.build          |   1 +
 t/t1423-ref-backend.sh | 121 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 190 insertions(+), 1 deletion(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..8c6a3f6042 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REF_URI`::
+    Specify which reference backend to be used along with its URI. Reference
+    backends like the files, reftable backend use the $GIT_DIR as their URI.
++
+Expects the format `<ref_backend>://<URI-for-resource>`, where the
+_<ref_backend>_ specifies the reference backend and the _<URI-for-resource>_
+specifies the URI used by the backend.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 51898c99cd..9bc380bba4 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index 23f46867f2..a7af228799 100644
--- a/refs.c
+++ b/refs.c
@@ -2186,15 +2186,73 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
 	return maybe_debug_wrap_ref_store(dir, ref_store);
 }
 
+static struct ref_store *get_ref_store_from_uri(struct repository *repo,
+						const char *uri)
+{
+	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
+	enum ref_storage_format format;
+	struct ref_store *store = NULL;
+	char *format_string;
+	char *dir;
+
+	if (!uri || !uri[0]) {
+		error("reference backend uri is empty");
+		goto cleanup;
+	}
+
+	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
+		error("invalid reference backend uri format '%s'", uri);
+		goto cleanup;
+	}
+
+	format_string = ref_backend_info.items[0].string;
+	if (!starts_with(ref_backend_info.items[1].string, "//")) {
+		error("invalid reference backend uri format '%s'", uri);
+		goto cleanup;
+	}
+	dir = ref_backend_info.items[1].string + 2;
+
+	format_string = ref_backend_info.items[0].string;
+	dir = ref_backend_info.items[1].string + 2;
+
+	if (!dir || !dir[0]) {
+		error("invalid path in uri '%s'", uri);
+		goto cleanup;
+	}
+
+	format = ref_storage_format_by_name(format_string);
+	if (format == REF_STORAGE_FORMAT_UNKNOWN) {
+		error("unknown reference backend '%s'", format_string);
+		goto cleanup;
+	}
+
+	store = get_ref_store_for_dir(repo, dir, format);
+
+cleanup:
+	string_list_clear(&ref_backend_info, 0);
+	return store;
+}
+
 struct ref_store *get_main_ref_store(struct repository *r)
 {
+	char *ref_uri;
+
 	if (r->refs_private)
 		return r->refs_private;
 
 	if (!r->gitdir)
 		BUG("attempting to get main_ref_store outside of repository");
 
-	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
+	ref_uri = getenv(GIT_REF_URI_ENVIRONMENT);
+	if (ref_uri) {
+		r->refs_private = get_ref_store_from_uri(r, ref_uri);
+		if (!r->refs_private)
+			die("failed to initialize ref store from URI: %s", ref_uri);
+
+	} else {
+		r->refs_private = get_ref_store_for_dir(r, r->gitdir,
+							r->ref_storage_format);
+	}
 	return r->refs_private;
 }
 
diff --git a/t/meson.build b/t/meson.build
index a5531df415..a66f8fafff 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -208,6 +208,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..f6756bdd2b
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,121 @@
+#!/bin/sh
+
+test_description='Test different reference backend URIs'
+
+. ./test-lib.sh
+
+test_expect_success 'empty uri provided' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "reference backend uri is empty" err
+	)
+'
+
+test_expect_success 'invalid uri provided' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable@/home/reftable" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid reference backend uri format" err
+	)
+'
+
+test_expect_success 'empty path in uri' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable://" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid path in uri" err
+	)
+'
+
+test_expect_success 'uri ends at colon' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable:" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid reference backend uri format" err
+	)
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="db://.git" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "unknown reference backend" err
+	)
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+	for to_format in $ref_formats
+	do
+		if test "$from_format" = "$to_format"
+		then
+			continue
+		fi
+
+		test_expect_success "read from $to_format backend" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
+				git refs list >expect &&
+				GIT_REF_URI="$to_format://$BACKEND_PATH" git refs list >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "write to $to_format backend" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				git refs list >expect &&
+
+				BACKEND_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
+				GIT_REF_URI="$to_format://$BACKEND_PATH" git tag -d 1 &&
+
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				GIT_REF_URI="$to_format://$BACKEND_PATH" git refs list >expect &&
+				git refs list >out &&
+				cat out | grep -v "refs/tags/1" >actual &&
+				test_cmp expect actual
+			)
+		'
+	done
+done
+
+test_done

-- 
2.51.2

Re: [PATCH v2 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> +`GIT_REF_URI`::
> +    Specify which reference backend to be used along with its URI. Reference
> +    backends like the files, reftable backend use the $GIT_DIR as their URI.
> ++
> +Expects the format `<ref_backend>://<URI-for-resource>`, where the
> +_<ref_backend>_ specifies the reference backend and the _<URI-for-resource>_
> +specifies the URI used by the backend.

It is more like "<directory>" that specifies the local directory the
backend is told to use to store its data.  It feels way too broad
for what the initial implementation achieves and what the design can
potentially include, to say "URI-for-resource", I would think.

> diff --git a/environment.h b/environment.h
> index 51898c99cd..9bc380bba4 100644
> --- a/environment.h
> +++ b/environment.h
> @@ -42,6 +42,7 @@
>  #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
>  #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
>  #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
> +#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
>  
>  /*
>   * Environment variable used to propagate the --no-advice global option to the
> diff --git a/refs.c b/refs.c
> index 23f46867f2..a7af228799 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2186,15 +2186,73 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
>  	return maybe_debug_wrap_ref_store(dir, ref_store);
>  }
>  
> +static struct ref_store *get_ref_store_from_uri(struct repository *repo,
> +						const char *uri)
> +{
> +	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
> +	enum ref_storage_format format;
> +	struct ref_store *store = NULL;
> +	char *format_string;
> +	char *dir;
> +
> +	if (!uri || !uri[0]) {
> +		error("reference backend uri is empty");
> +		goto cleanup;
> +	}

Equating !uri and !uri[0] and giving the same message would not help
diagnosing an error, and not _("localizing") the message is of dubious
value (after all, the message is not being given to somebody coming
over the network, but meant to be given to the local user, right?).

If we remove the !uri[0] from the check, shouldn't the later check
catch it as "invalid format" anyway, and print '%s' it to show that
what was given was empty clearly enough?

> +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
> +		error("invalid reference backend uri format '%s'", uri);
> +		goto cleanup;
> +	}
> +
> +	format_string = ref_backend_info.items[0].string;
> +	if (!starts_with(ref_backend_info.items[1].string, "//")) {
> +		error("invalid reference backend uri format '%s'", uri);
> +		goto cleanup;
> +	}
> +	dir = ref_backend_info.items[1].string + 2;

Two questions.  (1) do we still want the double-slash after the
colon?  (2) if so, would it make it simpler to string-list-split
using "://" as the separator?

> +	format_string = ref_backend_info.items[0].string;
> +	dir = ref_backend_info.items[1].string + 2;

These two lines are fishy.  Perhaps leftover from an earlier draft
that did not have an error checking before the previous 5 lines were
added?

> +	if (!dir || !dir[0]) {
> +		error("invalid path in uri '%s'", uri);
> +		goto cleanup;
> +	}

At this point it is very unlikely for "dir" to be NULL, no?  Even if
the .string member after splitting were NULL, adding 2 to it would
not leave it NULL.

Being defensive and checking for NULL is good, but then exactly the
same question on "NULL vs an empty string" applies here.

>  struct ref_store *get_main_ref_store(struct repository *r)
>  {
> +	char *ref_uri;
> +
>  	if (r->refs_private)
>  		return r->refs_private;
>  
>  	if (!r->gitdir)
>  		BUG("attempting to get main_ref_store outside of repository");
>  
> -	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
> +	ref_uri = getenv(GIT_REF_URI_ENVIRONMENT);
> +	if (ref_uri) {
> +		r->refs_private = get_ref_store_from_uri(r, ref_uri);
> +		if (!r->refs_private)
> +			die("failed to initialize ref store from URI: %s", ref_uri);
> +
> +	} else {
> +		r->refs_private = get_ref_store_for_dir(r, r->gitdir,
> +							r->ref_storage_format);
> +	}
>  	return r->refs_private;
>  }

If this mechanism is for consumption by "git refs migrate", is it
possible to reduce the blast radius by giving the command a command
line option to do an equivalent of this?  I really am not happy with
this environment variable that can change the behaviour of such a
low level layer from unsuspecting programs that are not ready.

Instead of tweaking the behaviour of this function via environment
that can affect any programs, can't we give these callers like "git
refs migrate" with specific needs set_main_ref_store() function that
takes a ref_store and a repository.  Then they can use to call into
get_ref_store_for_dir() to obtain a ref they need.  "git refs migrate"
already takes "--ref-format" variable, so all it needs is another
"--ref-directory" command line option, right?

If the ability to set the ref backend location for arbitrary program
proves to be useful, we _could_ give the same --ref-format and
--ref-direcctory command line options to "git" itself (like "git -C
there" runs any subcommand in the named directory), which does the
the get_ref_store_for_dir() plus set_main_ref_store() dance,
modelled after how "git refs migrate" does them.

Hmm?

Re: [PATCH v2 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 7511 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> +`GIT_REF_URI`::
>> +    Specify which reference backend to be used along with its URI. Reference
>> +    backends like the files, reftable backend use the $GIT_DIR as their URI.
>> ++
>> +Expects the format `<ref_backend>://<URI-for-resource>`, where the
>> +_<ref_backend>_ specifies the reference backend and the _<URI-for-resource>_
>> +specifies the URI used by the backend.
>
> It is more like "<directory>" that specifies the local directory the
> backend is told to use to store its data.  It feels way too broad
> for what the initial implementation achieves and what the design can
> potentially include, to say "URI-for-resource", I would think.
>

Well I'm okay either ways, my first version was very specific as it
mention '<path>'. I changed it based on the discussion with you and Toon
about how the '<path>' is the URI for the reference backend.

>> diff --git a/environment.h b/environment.h
>> index 51898c99cd..9bc380bba4 100644
>> --- a/environment.h
>> +++ b/environment.h
>> @@ -42,6 +42,7 @@
>>  #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
>>  #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
>>  #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
>> +#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
>>
>>  /*
>>   * Environment variable used to propagate the --no-advice global option to the
>> diff --git a/refs.c b/refs.c
>> index 23f46867f2..a7af228799 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2186,15 +2186,73 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
>>  	return maybe_debug_wrap_ref_store(dir, ref_store);
>>  }
>>
>> +static struct ref_store *get_ref_store_from_uri(struct repository *repo,
>> +						const char *uri)
>> +{
>> +	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
>> +	enum ref_storage_format format;
>> +	struct ref_store *store = NULL;
>> +	char *format_string;
>> +	char *dir;
>> +
>> +	if (!uri || !uri[0]) {
>> +		error("reference backend uri is empty");
>> +		goto cleanup;
>> +	}
>
> Equating !uri and !uri[0] and giving the same message would not help
> diagnosing an error, and not _("localizing") the message is of dubious
> value (after all, the message is not being given to somebody coming
> over the network, but meant to be given to the local user, right?).
>

I think that's fair. I also missed localizing all the errors, I think
someone did point that out too.

> If we remove the !uri[0] from the check, shouldn't the later check
> catch it as "invalid format" anyway, and print '%s' it to show that
> what was given was empty clearly enough?
>

Yeah, it should I'll remove the latter and modify the test.

>> +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
>> +		error("invalid reference backend uri format '%s'", uri);
>> +		goto cleanup;
>> +	}
>> +
>> +	format_string = ref_backend_info.items[0].string;
>> +	if (!starts_with(ref_backend_info.items[1].string, "//")) {
>> +		error("invalid reference backend uri format '%s'", uri);
>> +		goto cleanup;
>> +	}
>> +	dir = ref_backend_info.items[1].string + 2;
>
> Two questions.  (1) do we still want the double-slash after the
> colon?  (2) if so, would it make it simpler to string-list-split
> using "://" as the separator?
>

(1) Yes.
(2) My understanding of `string_list_split()` was that the `delim`
argument are a set of characters to split the string on.

So:
    string_list_split(l, "abc:def/ghi/jkl", "://", -1) -> ["abc",
"def", "ghi", "jkl"]
    string_list_split(l, "reftable://foo", "://", -1) -> ["reftable",
"", "", "foo", "bar"]

But this isn't what we want.

>> +	format_string = ref_backend_info.items[0].string;
>> +	dir = ref_backend_info.items[1].string + 2;
>
> These two lines are fishy.  Perhaps leftover from an earlier draft
> that did not have an error checking before the previous 5 lines were
> added?
>

Yes, will cleanup.

>> +	if (!dir || !dir[0]) {
>> +		error("invalid path in uri '%s'", uri);
>> +		goto cleanup;
>> +	}
>
> At this point it is very unlikely for "dir" to be NULL, no?  Even if
> the .string member after splitting were NULL, adding 2 to it would
> not leave it NULL.
>
> Being defensive and checking for NULL is good, but then exactly the
> same question on "NULL vs an empty string" applies here.
>

Yea, the '!dir[0]' should definitely be enough here.

>>  struct ref_store *get_main_ref_store(struct repository *r)
>>  {
>> +	char *ref_uri;
>> +
>>  	if (r->refs_private)
>>  		return r->refs_private;
>>
>>  	if (!r->gitdir)
>>  		BUG("attempting to get main_ref_store outside of repository");
>>
>> -	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
>> +	ref_uri = getenv(GIT_REF_URI_ENVIRONMENT);
>> +	if (ref_uri) {
>> +		r->refs_private = get_ref_store_from_uri(r, ref_uri);
>> +		if (!r->refs_private)
>> +			die("failed to initialize ref store from URI: %s", ref_uri);
>> +
>> +	} else {
>> +		r->refs_private = get_ref_store_for_dir(r, r->gitdir,
>> +							r->ref_storage_format);
>> +	}
>>  	return r->refs_private;
>>  }
>
> If this mechanism is for consumption by "git refs migrate", is it
> possible to reduce the blast radius by giving the command a command
> line option to do an equivalent of this?  I really am not happy with
> this environment variable that can change the behaviour of such a
> low level layer from unsuspecting programs that are not ready.
>

But the mechanism isn't for 'git refs migrate', but rather we want to
add/update references via 'git update-ref' into the dry-run folder
created by the 'git refs migrate'. In the broader sense, we want to
manipulate references within this dry-run folder as if it is the
reference folder for the underlying repository.

I get the comprehension behind the environment variable and am happy to
work on something alternative if we can achieve something similar. The
reason to pick the ENV variable was mostly because this isn't a regular
user flag which we expect users to use. Also, this is very similar to
the already existing GIT_OBJECT_DIRECTORY.

> Instead of tweaking the behaviour of this function via environment
> that can affect any programs, can't we give these callers like "git
> refs migrate" with specific needs set_main_ref_store() function that
> takes a ref_store and a repository.  Then they can use to call into
> get_ref_store_for_dir() to obtain a ref they need.  "git refs migrate"
> already takes "--ref-format" variable, so all it needs is another
> "--ref-directory" command line option, right?
>

Something like this would require us to add these flags to all commands,
currently I can think of 'git update-ref' and 'git refs' but it could
spread to all reference oriented commands.

> If the ability to set the ref backend location for arbitrary program
> proves to be useful, we _could_ give the same --ref-format and
> --ref-direcctory command line options to "git" itself (like "git -C
> there" runs any subcommand in the named directory), which does the
> the get_ref_store_for_dir() plus set_main_ref_store() dance,
> modelled after how "git refs migrate" does them.
>
> Hmm?

This could work indeed, I would instead swap it out for a single
"--ref-uri=<backend>://<uri>" which would make it much simpler for users
and future implementations which might not have a 'directory' like the
current backends do.

Overall the ENV variable seemed the best based on the constraints and
the existing similar variables. Wdyt?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v2 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> (2) My understanding of `string_list_split()` was that the `delim`
> argument are a set of characters to split the string on.

Ah, silly me.

> But the mechanism isn't for 'git refs migrate', but rather we want to
> add/update references via 'git update-ref' into the dry-run folder
> created by the 'git refs migrate'. In the broader sense, we want to
> manipulate references within this dry-run folder as if it is the
> reference folder for the underlying repository.

OK, I took the cover letter description too literally, it seems.

If we want everybody in a single session to have a temporarily
distorted view of the world, it has been a tried and proven way to
use environment variables that override the default repository
layout, e.g., GIT_DIR, GIT_WORK_TREE, and this "no reference
interactions go there, not the usual place the repository
configuration says" environment variable fits very well in the
context.

Thanks.

Re: [PATCH v2 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1260 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> (2) My understanding of `string_list_split()` was that the `delim`
>> argument are a set of characters to split the string on.
>
> Ah, silly me.
>
>> But the mechanism isn't for 'git refs migrate', but rather we want to
>> add/update references via 'git update-ref' into the dry-run folder
>> created by the 'git refs migrate'. In the broader sense, we want to
>> manipulate references within this dry-run folder as if it is the
>> reference folder for the underlying repository.
>
> OK, I took the cover letter description too literally, it seems.
>

I did change the cover letter for this version with the plan of how this
would be used. Let me know if you think I could clarify further.

> If we want everybody in a single session to have a temporarily
> distorted view of the world, it has been a tried and proven way to
> use environment variables that override the default repository
> layout, e.g., GIT_DIR, GIT_WORK_TREE, and this "no reference
> interactions go there, not the usual place the repository
> configuration says" environment variable fits very well in the
> context.
>
> Thanks.

Yes! Exactly. Good to see we're on the same page :)

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v3 0/2] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch series adds a new ENV variable 'GIT_REF_URI' which takes the
reference backend and path in a URI form:

    <reference_backend>://<URI-for-resource>

For e.g. 'reftable:///foo' or 'files://$GIT_DIR/ref_migration.0xBsa0'.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

The first commit adds the required changes to create a 'ref_store' for a
given path. The second commit parses the URI if available when creating
the main ref store.

This is based on top of 9a2fb147f2 (Git 2.52, 2025-11-17).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/git.adoc |   8 ++++
 environment.h          |   1 +
 refs.c                 |  68 +++++++++++++++++++++++++--
 t/meson.build          |   1 +
 t/t1423-ref-backend.sh | 121 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 196 insertions(+), 3 deletions(-)

Karthik Nayak (2):
      refs: support obtaining ref_store for given dir
      refs: add GIT_REF_URI to specify reference backend and directory

Range-diff versus v2:

1:  5d37b2f0be = 1:  2b65f93e56 refs: support obtaining ref_store for given dir
2:  493c7ca098 ! 2:  d5dbb2f112 refs: add GIT_REF_URI to specify reference backend and directory
    @@ refs.c: static struct ref_store *get_ref_store_for_dir(struct repository *r,
     +	char *format_string;
     +	char *dir;
     +
    -+	if (!uri || !uri[0]) {
    -+		error("reference backend uri is empty");
    ++	if (!uri) {
    ++		error(_("reference backend uri is not provided"));
     +		goto cleanup;
     +	}
     +
     +	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
    -+		error("invalid reference backend uri format '%s'", uri);
    ++		error(_("invalid reference backend uri format '%s'"), uri);
     +		goto cleanup;
     +	}
     +
     +	format_string = ref_backend_info.items[0].string;
     +	if (!starts_with(ref_backend_info.items[1].string, "//")) {
    -+		error("invalid reference backend uri format '%s'", uri);
    ++		error(_("invalid reference backend uri format '%s'"), uri);
     +		goto cleanup;
     +	}
     +	dir = ref_backend_info.items[1].string + 2;
     +
    -+	format_string = ref_backend_info.items[0].string;
    -+	dir = ref_backend_info.items[1].string + 2;
    -+
    -+	if (!dir || !dir[0]) {
    -+		error("invalid path in uri '%s'", uri);
    ++	if (!dir[0]) {
    ++		error(_("invalid path in uri '%s'"), uri);
     +		goto cleanup;
     +	}
     +
     +	format = ref_storage_format_by_name(format_string);
     +	if (format == REF_STORAGE_FORMAT_UNKNOWN) {
    -+		error("unknown reference backend '%s'", format_string);
    ++		error(_("unknown reference backend '%s'"), format_string);
     +		goto cleanup;
     +	}
     +
    @@ t/t1423-ref-backend.sh (new)
     +		GIT_REF_URI="" &&
     +		export GIT_REF_URI &&
     +		test_must_fail git refs list 2>err &&
    -+		test_grep "reference backend uri is empty" err
    ++		test_grep "invalid reference backend uri format" err
     +	)
     +'
     +


base-commit: 9a2fb147f2c61d0cab52c883e7e26f5b7948e3ed
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v3 1/2] refs: support obtaining ref_store for given dir

[Karthik Nayak]

The refs subsystem uses the `get_main_ref_store()` to obtain the main
ref_store for a given repository. In the upcoming patches we also want
to create a ref_store for any given reference directory, which may exist
in arbitrary paths. For the files backend and the reftable backend, the
reference directory is generally the $GIT_DIR.

To support such behavior, extract out the core logic for creating out
the ref_store from `get_main_ref_store()` into a new function
`get_ref_store_for_dir()` which can provide the ref_store for a
given (repository, directory, reference format) combination.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/refs.c b/refs.c
index 965381367e..23f46867f2 100644
--- a/refs.c
+++ b/refs.c
@@ -2177,6 +2177,15 @@ void ref_store_release(struct ref_store *ref_store)
 	free(ref_store->gitdir);
 }
 
+static struct ref_store *get_ref_store_for_dir(struct repository *r,
+					       char *dir,
+					       enum ref_storage_format format)
+{
+	struct ref_store *ref_store = ref_store_init(r, format, dir,
+						     REF_STORE_ALL_CAPS);
+	return maybe_debug_wrap_ref_store(dir, ref_store);
+}
+
 struct ref_store *get_main_ref_store(struct repository *r)
 {
 	if (r->refs_private)
@@ -2185,9 +2194,7 @@ struct ref_store *get_main_ref_store(struct repository *r)
 	if (!r->gitdir)
 		BUG("attempting to get main_ref_store outside of repository");
 
-	r->refs_private = ref_store_init(r, r->ref_storage_format,
-					 r->gitdir, REF_STORE_ALL_CAPS);
-	r->refs_private = maybe_debug_wrap_ref_store(r->gitdir, r->refs_private);
+	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
 	return r->refs_private;
 }
 

-- 
2.51.2

[PATCH v3 2/2] refs: add GIT_REF_URI to specify reference backend and directory

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references.
This asymmetry makes it difficult to test different reference backends
or use alternative reference storage locations without modifying the
repository structure.

Add a new environment variable 'GIT_REF_URI' that specifies both the
reference backend and directory path using a URI format:

    <ref_backend>://<URI-for-resource>

When set, this variable is used to obtain the main reference store for
all Git commands. The variable is checked in `get_main_ref_store()`
when lazily assigning `repo->refs_private`. We cannot initialize this
earlier in `repo_set_gitdir()` because the repository's hash algorithm
isn't known at that point, and the reftable backend requires this
information during initialization.

When used with worktrees, the specified directory is treated as the
reference directory for all worktree operations.

Add a new test file 't1423-ref-backend.sh' to test this environment
variable.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   8 ++++
 environment.h          |   1 +
 refs.c                 |  57 ++++++++++++++++++++++-
 t/meson.build          |   1 +
 t/t1423-ref-backend.sh | 121 +++++++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 187 insertions(+), 1 deletion(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..8c6a3f6042 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,14 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REF_URI`::
+    Specify which reference backend to be used along with its URI. Reference
+    backends like the files, reftable backend use the $GIT_DIR as their URI.
++
+Expects the format `<ref_backend>://<URI-for-resource>`, where the
+_<ref_backend>_ specifies the reference backend and the _<URI-for-resource>_
+specifies the URI used by the backend.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 51898c99cd..9bc380bba4 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REF_URI_ENVIRONMENT "GIT_REF_URI"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index 23f46867f2..da76e0c54a 100644
--- a/refs.c
+++ b/refs.c
@@ -2186,15 +2186,70 @@ static struct ref_store *get_ref_store_for_dir(struct repository *r,
 	return maybe_debug_wrap_ref_store(dir, ref_store);
 }
 
+static struct ref_store *get_ref_store_from_uri(struct repository *repo,
+						const char *uri)
+{
+	struct string_list ref_backend_info = STRING_LIST_INIT_DUP;
+	enum ref_storage_format format;
+	struct ref_store *store = NULL;
+	char *format_string;
+	char *dir;
+
+	if (!uri) {
+		error(_("reference backend uri is not provided"));
+		goto cleanup;
+	}
+
+	if (string_list_split(&ref_backend_info, uri, ":", 2) != 2) {
+		error(_("invalid reference backend uri format '%s'"), uri);
+		goto cleanup;
+	}
+
+	format_string = ref_backend_info.items[0].string;
+	if (!starts_with(ref_backend_info.items[1].string, "//")) {
+		error(_("invalid reference backend uri format '%s'"), uri);
+		goto cleanup;
+	}
+	dir = ref_backend_info.items[1].string + 2;
+
+	if (!dir[0]) {
+		error(_("invalid path in uri '%s'"), uri);
+		goto cleanup;
+	}
+
+	format = ref_storage_format_by_name(format_string);
+	if (format == REF_STORAGE_FORMAT_UNKNOWN) {
+		error(_("unknown reference backend '%s'"), format_string);
+		goto cleanup;
+	}
+
+	store = get_ref_store_for_dir(repo, dir, format);
+
+cleanup:
+	string_list_clear(&ref_backend_info, 0);
+	return store;
+}
+
 struct ref_store *get_main_ref_store(struct repository *r)
 {
+	char *ref_uri;
+
 	if (r->refs_private)
 		return r->refs_private;
 
 	if (!r->gitdir)
 		BUG("attempting to get main_ref_store outside of repository");
 
-	r->refs_private = get_ref_store_for_dir(r, r->gitdir, r->ref_storage_format);
+	ref_uri = getenv(GIT_REF_URI_ENVIRONMENT);
+	if (ref_uri) {
+		r->refs_private = get_ref_store_from_uri(r, ref_uri);
+		if (!r->refs_private)
+			die("failed to initialize ref store from URI: %s", ref_uri);
+
+	} else {
+		r->refs_private = get_ref_store_for_dir(r, r->gitdir,
+							r->ref_storage_format);
+	}
 	return r->refs_private;
 }
 
diff --git a/t/meson.build b/t/meson.build
index a5531df415..a66f8fafff 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -208,6 +208,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..f36125bf64
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,121 @@
+#!/bin/sh
+
+test_description='Test different reference backend URIs'
+
+. ./test-lib.sh
+
+test_expect_success 'empty uri provided' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid reference backend uri format" err
+	)
+'
+
+test_expect_success 'invalid uri provided' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable@/home/reftable" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid reference backend uri format" err
+	)
+'
+
+test_expect_success 'empty path in uri' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable://" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid path in uri" err
+	)
+'
+
+test_expect_success 'uri ends at colon' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="reftable:" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "invalid reference backend uri format" err
+	)
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init --ref-format=files repo &&
+	(
+		cd repo &&
+		GIT_REF_URI="db://.git" &&
+		export GIT_REF_URI &&
+		test_must_fail git refs list 2>err &&
+		test_grep "unknown reference backend" err
+	)
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+	for to_format in $ref_formats
+	do
+		if test "$from_format" = "$to_format"
+		then
+			continue
+		fi
+
+		test_expect_success "read from $to_format backend" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
+				git refs list >expect &&
+				GIT_REF_URI="$to_format://$BACKEND_PATH" git refs list >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "write to $to_format backend" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				git refs list >expect &&
+
+				BACKEND_PATH=$(cat out | sed "s/.* ${SQ}\(.*\)${SQ}/\1/") &&
+				GIT_REF_URI="$to_format://$BACKEND_PATH" git tag -d 1 &&
+
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				GIT_REF_URI="$to_format://$BACKEND_PATH" git refs list >expect &&
+				git refs list >out &&
+				cat out | grep -v "refs/tags/1" >actual &&
+				test_cmp expect actual
+			)
+		'
+	done
+done
+
+test_done

-- 
2.51.2

Re: [PATCH v3 0/2] refs: allow setting the reference directory

[Patrick Steinhardt]

On Mon, Dec 01, 2025 at 12:24:57PM +0100, Karthik Nayak wrote:
> Changes in v3:
> - Cleanup some stale code which wasn't removed.
> - Localize strings which will be output to the user.
> - Remove additional defensive checks which are not needed.
> - Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

FYI: I did an internal review of this at [1] and mentioned a couple of
things that I think still need to be iterated on. Most importantly, I
think that we also need to handle worktrees:

    I think we need to do something about worktree ref stores in the
    context of this series, as we're now thinking a bit broader than
    before. It would be somewhat weird if you can change the main
    reference store, but it doesn't have any influence on the worktree
    ref stores. If I say e.g. that the ref store is located in
    `/tmp/dir`, then I'd expect that the main ref store would be in
    `/tmp/dir/refs` and that worktree refs are in
    `/tmp/dir/worktrees/$name/refs`.

    The reason I think this needs to be handled as part of this MR is
    that we are extending semantics of the extension now. This is only
    possible because the extensions are quite strict now, and thus we
    can strictly extend them. But once we have extended them we cannot
    just alter their semantics to also cover worktree refs.

So you can expect another iteration of this patch series once folks
settle into their usual post-holiday rhythms again :)

Patrick

[1]: https://gitlab.com/gitlab-org/git/-/merge_requests/451

Re: [PATCH v3 0/2] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 2945 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Dec 01, 2025 at 12:24:57PM +0100, Karthik Nayak wrote:
>> Changes in v3:
>> - Cleanup some stale code which wasn't removed.
>> - Localize strings which will be output to the user.
>> - Remove additional defensive checks which are not needed.
>> - Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com
>
> FYI: I did an internal review of this at [1] and mentioned a couple of
> things that I think still need to be iterated on. Most importantly, I
> think that we also need to handle worktrees:
>
>     I think we need to do something about worktree ref stores in the
>     context of this series, as we're now thinking a bit broader than
>     before. It would be somewhat weird if you can change the main
>     reference store, but it doesn't have any influence on the worktree
>     ref stores. If I say e.g. that the ref store is located in
>     `/tmp/dir`, then I'd expect that the main ref store would be in
>     `/tmp/dir/refs` and that worktree refs are in
>     `/tmp/dir/worktrees/$name/refs`.
>
>     The reason I think this needs to be handled as part of this MR is
>     that we are extending semantics of the extension now. This is only
>     possible because the extensions are quite strict now, and thus we
>     can strictly extend them. But once we have extended them we cannot
>     just alter their semantics to also cover worktree refs.
>
> So you can expect another iteration of this patch series once folks
> settle into their usual post-holiday rhythms again :)
>
> Patrick
>
> [1]: https://gitlab.com/gitlab-org/git/-/merge_requests/451

Thanks for putting that here. I did reply to one of Junio's what's
cooking emails [1] stating the same (mostly that I'm on vacation) and that
the topic isn't abandoned but would be delayed.

I've been looking at this with a fresh set of eyes post vacation. One of
the things I still need to solve is around how this works with Git
directory detection.

The current implementation works because it hijacks the refs
initialization to set the new reference directory. But this assumes that
the repository has a 'refs/' folder and 'HEAD' file within the $GITDIR.

So if we want this to work with worktrees the way that Patrick
mentioned, we'd have to create the worktree reference path within the
provided reference storage path. But this would mean that the $GITDIR
for the worktree wouldn't be initialized with the required files
currently required for being a valid Git directory.

The issue is that config parsing is only done post Git directory
detection. One hacky fix is to create a dummy 'refs/' folder and 'HEAD'
file within the $GITDIR, similar to how the 'reftable does it. I'm not a
big fan of adding something like that. Would be curious if anyone has
further thoughts on this.

[1]: https://lore.kernel.org/git/CAOLa=ZTeEEntiQdCnDd6B8a_ppJrrZqdURhQJV=bNktnp0JqMw@mail.gmail.com/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v3 0/2] refs: allow setting the reference directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

>> FYI: I did an internal review of this at [1] and mentioned a couple of
>> things that I think still need to be iterated on. Most importantly, I
>> think that we also need to handle worktrees:
>> ...
>> So you can expect another iteration of this patch series once folks
>> settle into their usual post-holiday rhythms again :)
>>
>> Patrick
>>
>> [1]: https://gitlab.com/gitlab-org/git/-/merge_requests/451
>
> Thanks for putting that here. I did reply to one of Junio's what's
> cooking emails [1] stating the same (mostly that I'm on vacation) and that
> the topic isn't abandoned but would be delayed.

OK.  The topic has been marked as [Stalled] and I'll probably eject
it from 'seen' soonish to declutter my tree, but it does not mean a
renewed effort on the same topic is unwelcome.  "Dismissed without
prejudice" so to speak.

Thanks.

Re: [PATCH v3 0/2] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>>> FYI: I did an internal review of this at [1] and mentioned a couple of
>>> things that I think still need to be iterated on. Most importantly, I
>>> think that we also need to handle worktrees:
>>> ...
>>> So you can expect another iteration of this patch series once folks
>>> settle into their usual post-holiday rhythms again :)
>>>
>>> Patrick
>>>
>>> [1]: https://gitlab.com/gitlab-org/git/-/merge_requests/451
>>
>> Thanks for putting that here. I did reply to one of Junio's what's
>> cooking emails [1] stating the same (mostly that I'm on vacation) and that
>> the topic isn't abandoned but would be delayed.
>
> OK.  The topic has been marked as [Stalled] and I'll probably eject
> it from 'seen' soonish to declutter my tree, but it does not mean a
> renewed effort on the same topic is unwelcome.  "Dismissed without
> prejudice" so to speak.
>
> Thanks.

Yeah I understand, I'm still working on it, but it has been a while.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v4 0/4] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:

   - A format name alone (e.g., `reftable` or `files`) which uses the
     default location (the repository's common directory).

   - A URI format `<format>://<location>` which explicitly specifies
     both the format and location (e.g., `reftable:///foo/bar`).

We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
  backends will store worktree references in 'ref_dir/worktrees/wt_id'
  and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
  relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/config/extensions.adoc |  16 ++-
 Documentation/git.adoc               |   5 +
 builtin/clone.c                      |  10 +-
 builtin/worktree.c                   |  35 +++++++
 environment.h                        |   1 +
 refs.c                               |  61 +++++++++++-
 refs.h                               |  13 +++
 refs/files-backend.c                 |  18 +++-
 refs/packed-backend.c                |   1 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  15 +++
 refs/reftable-backend.c              |  38 +++----
 repository.c                         |   9 +-
 repository.h                         |   5 +-
 setup.c                              |  59 ++++++++++-
 setup.h                              |   2 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 187 +++++++++++++++++++++++++++++++++++
 18 files changed, 433 insertions(+), 44 deletions(-)

Karthik Nayak (4):
      refs: allow reference location in refstorage config
      refs: extract out `refs_create_refdir_stubs()`
      refs: parse and use the reference storage payload
      refs: add GIT_REFERENCE_BACKEND to specify reference backend

Range-diff versus v3:

1:  a91a4d5ba3 < -:  ---------- refs: support obtaining ref_store for given dir
2:  a9f126d724 < -:  ---------- refs: add GIT_REF_URI to specify reference backend and directory
-:  ---------- > 1:  c8ddb8ec8d refs: allow reference location in refstorage config
-:  ---------- > 2:  02acdc16b4 refs: extract out `refs_create_refdir_stubs()`
-:  ---------- > 3:  1b0e17e705 refs: parse and use the reference storage payload
-:  ---------- > 4:  8ca379edd6 refs: add GIT_REFERENCE_BACKEND to specify reference backend


base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v4 1/4] refs: allow reference location in refstorage config

[Karthik Nayak]

The 'extensions.refStorage' config is used to specify the reference
backend for a given repository. Both the 'files' and 'reftable' backends
utilize the $GIT_DIR as the reference folder by default in
`get_main_ref_store()`.

Since the reference backends are pluggable, this means that they should
work with out-of-tree reference directories too. Extend the 'refStorage'
config to also support taking an URI input, where users can specify the
reference backend and the location.

Add the required changes to obtain and propagate this value to the
individual backends. A follow up commit will add the required changes on
the backends to parse this value.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c |  3 ++-
 repository.c    |  9 +++++++--
 repository.h    |  5 ++++-
 setup.c         | 39 ++++++++++++++++++++++++++++++++++-----
 setup.h         |  2 ++
 5 files changed, 49 insertions(+), 9 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index b40cee5968..28412576b3 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1442,7 +1442,8 @@ int cmd_clone(int argc,
 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
 	repo_set_hash_algo(the_repository, hash_algo);
-	create_reference_database(the_repository->ref_storage_format, NULL, 1);
+	create_reference_database(the_repository->ref_storage_format,
+				  the_repository->ref_storage_payload, NULL, 1);
 
 	/*
 	 * Before fetching from the remote, download and install bundle
diff --git a/repository.c b/repository.c
index c7e75215ac..9815f081ef 100644
--- a/repository.c
+++ b/repository.c
@@ -193,9 +193,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
 }
 
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format)
+				 enum ref_storage_format format,
+				 const char *payload)
 {
 	repo->ref_storage_format = format;
+	free(repo->ref_storage_payload);
+	repo->ref_storage_payload = xstrdup_or_null(payload);
 }
 
 /*
@@ -277,7 +280,8 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 	repo_set_compat_hash_algo(repo, format.compat_hash_algo);
-	repo_set_ref_storage_format(repo, format.ref_storage_format);
+	repo_set_ref_storage_format(repo, format.ref_storage_format,
+				    format.ref_storage_payload);
 	repo->repository_format_worktree_config = format.worktree_config;
 	repo->repository_format_relative_worktrees = format.relative_worktrees;
 	repo->repository_format_precious_objects = format.precious_objects;
@@ -369,6 +373,7 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
+	FREE_AND_NULL(repo->ref_storage_payload);
 
 	odb_free(repo->objects);
 	repo->objects = NULL;
diff --git a/repository.h b/repository.h
index 6063c4b846..c648dab196 100644
--- a/repository.h
+++ b/repository.h
@@ -150,6 +150,8 @@ struct repository {
 
 	/* Repository's reference storage format, as serialized on disk. */
 	enum ref_storage_format ref_storage_format;
+	/* Reference storage information as needed for the backend. */
+	char *ref_storage_payload;
 
 	/* A unique-id for tracing purposes. */
 	int trace2_repo_id;
@@ -204,7 +206,8 @@ void repo_set_worktree(struct repository *repo, const char *path);
 void repo_set_hash_algo(struct repository *repo, int algo);
 void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format);
+				 enum ref_storage_format format,
+				 const char *payload);
 void initialize_repository(struct repository *repo);
 RESULT_MUST_BE_USED
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index b723f8b339..44e393c251 100644
--- a/setup.c
+++ b/setup.c
@@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
 		return EXTENSION_UNKNOWN;
 }
 
+static void parse_reference_uri(const char *value, char **format,
+				char **payload)
+{
+	char *schema_end;
+
+	schema_end = strstr(value, "://");
+	if (!schema_end) {
+		*format = xstrdup(value);
+		*payload = NULL;
+	} else {
+		*format = xstrndup(value, schema_end - value);
+		*payload = xstrdup_or_null(schema_end + 3);
+	}
+}
+
 /*
  * Record any new extensions in this function.
  */
@@ -674,10 +689,17 @@ static enum extension_result handle_extension(const char *var,
 		return EXTENSION_OK;
 	} else if (!strcmp(ext, "refstorage")) {
 		unsigned int format;
+		char *format_str;
 
 		if (!value)
 			return config_error_nonbool(var);
-		format = ref_storage_format_by_name(value);
+
+		parse_reference_uri(value, &format_str,
+				    &data->ref_storage_payload);
+
+		format = ref_storage_format_by_name(format_str);
+		free(format_str);
+
 		if (format == REF_STORAGE_FORMAT_UNKNOWN)
 			return error(_("invalid value for '%s': '%s'"),
 				     "extensions.refstorage", value);
@@ -850,6 +872,7 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
 	free(format->partial_clone);
+	free(format->ref_storage_payload);
 	init_repository_format(format);
 }
 
@@ -1942,7 +1965,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			repo_set_compat_hash_algo(the_repository,
 						  repo_fmt.compat_hash_algo);
 			repo_set_ref_storage_format(the_repository,
-						    repo_fmt.ref_storage_format);
+						    repo_fmt.ref_storage_format,
+						    repo_fmt.ref_storage_payload);
 			the_repository->repository_format_worktree_config =
 				repo_fmt.worktree_config;
 			the_repository->repository_format_relative_worktrees =
@@ -2042,7 +2066,8 @@ void check_repository_format(struct repository_format *fmt)
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
 	repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo);
 	repo_set_ref_storage_format(the_repository,
-				    fmt->ref_storage_format);
+				    fmt->ref_storage_format,
+				    fmt->ref_storage_payload);
 	the_repository->repository_format_worktree_config =
 		fmt->worktree_config;
 	the_repository->repository_format_relative_worktrees =
@@ -2360,13 +2385,15 @@ static int is_reinit(void)
 }
 
 void create_reference_database(enum ref_storage_format ref_storage_format,
+			       const char *ref_storage_payload,
 			       const char *initial_branch, int quiet)
 {
 	struct strbuf err = STRBUF_INIT;
 	char *to_free = NULL;
 	int reinit = is_reinit();
 
-	repo_set_ref_storage_format(the_repository, ref_storage_format);
+	repo_set_ref_storage_format(the_repository, ref_storage_format,
+				    ref_storage_payload);
 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -2645,7 +2672,8 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
-	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format);
+	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
+				    repo_fmt->ref_storage_payload);
 }
 
 int init_db(const char *git_dir, const char *real_git_dir,
@@ -2702,6 +2730,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo_fmt.ref_storage_format,
+					  repo_fmt.ref_storage_payload,
 					  initial_branch, flags & INIT_DB_QUIET);
 	create_object_directory();
 
diff --git a/setup.h b/setup.h
index d55dcc6608..4d25a353ac 100644
--- a/setup.h
+++ b/setup.h
@@ -171,6 +171,7 @@ struct repository_format {
 	int hash_algo;
 	int compat_hash_algo;
 	enum ref_storage_format ref_storage_format;
+	char *ref_storage_payload;
 	int sparse_index;
 	char *work_tree;
 	struct string_list unknown_extensions;
@@ -241,6 +242,7 @@ void initialize_repository_version(int hash_algo,
 				   enum ref_storage_format ref_storage_format,
 				   int reinit);
 void create_reference_database(enum ref_storage_format ref_storage_format,
+			       const char *ref_storage_payload,
 			       const char *initial_branch, int quiet);
 
 /*

-- 
2.52.0

Re: [PATCH v4 1/4] refs: allow reference location in refstorage config

[Patrick Steinhardt]

On Mon, Feb 02, 2026 at 01:26:30PM +0100, Karthik Nayak wrote:
> The 'extensions.refStorage' config is used to specify the reference
> backend for a given repository. Both the 'files' and 'reftable' backends
> utilize the $GIT_DIR as the reference folder by default in
> `get_main_ref_store()`.
> 
> Since the reference backends are pluggable, this means that they should

s/should/could/ I guess. It's not like they are broken currently, we
rather want more.

> diff --git a/repository.h b/repository.h
> index 6063c4b846..c648dab196 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -150,6 +150,8 @@ struct repository {
>  
>  	/* Repository's reference storage format, as serialized on disk. */
>  	enum ref_storage_format ref_storage_format;
> +	/* Reference storage information as needed for the backend. */

Nit: it would be nice to document whether this only contains the
additional data after the schema, or whether it's the whole
"files:///dir" string.

Another nit: ideally, the step to pass through the payload to the
backends came before this patch so that we could already make them fail
in case they are passed a value they don't understand. Otherwise, a user
could now pass "reftable://foobar", and the "foobar" part would be
silently ignored without any kind of warning or error.

Other than that this patch looks good to me.

Patrick

Re: [PATCH v4 1/4] refs: allow reference location in refstorage config

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 02, 2026 at 01:26:30PM +0100, Karthik Nayak wrote:
>> The 'extensions.refStorage' config is used to specify the reference
>> backend for a given repository. Both the 'files' and 'reftable' backends
>> utilize the $GIT_DIR as the reference folder by default in
>> `get_main_ref_store()`.
>>
>> Since the reference backends are pluggable, this means that they should
>
> s/should/could/ I guess. It's not like they are broken currently, we
> rather want more.
>

Yeah, that's fair.

>> diff --git a/repository.h b/repository.h
>> index 6063c4b846..c648dab196 100644
>> --- a/repository.h
>> +++ b/repository.h
>> @@ -150,6 +150,8 @@ struct repository {
>>
>>  	/* Repository's reference storage format, as serialized on disk. */
>>  	enum ref_storage_format ref_storage_format;
>> +	/* Reference storage information as needed for the backend. */
>
> Nit: it would be nice to document whether this only contains the
> additional data after the schema, or whether it's the whole
> "files:///dir" string.
>

Will add some more information here.

> Another nit: ideally, the step to pass through the payload to the
> backends came before this patch so that we could already make them fail
> in case they are passed a value they don't understand. Otherwise, a user
> could now pass "reftable://foobar", and the "foobar" part would be
> silently ignored without any kind of warning or error.
>
> Other than that this patch looks good to me.
>
> Patrick

That's a good point, I hadn't considered, this should be simple enough
to do, so let me put that together.

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v4 2/4] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

For Git to recognize a directory as a Git directory, it requires the
directory to contain:

  1. 'HEAD' file
  2. object/ directory
  3. 'refs/' directory

Here, #1 and #3 are part of the reference storage mechanism,
specifically the files backend. Since then, newer backends such as the
reftable backend have moved to using their own path ('reftable/') for
storing references. But to ensure git still recognizes the directory as
a Git directory, we create stubs.

There are two locations we create stubs:

- In 'refs/reftable-backend.c' when creating the reftable backend.
- In 'clone.c' before spawning transport helpers.

In a following commit, we'll add another instance. So instead of
repeating the code, let's extract out this code to
`refs_create_refdir_stubs()` and use it.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c         |  7 +------
 refs.c                  | 24 ++++++++++++++++++++++++
 refs.h                  | 13 +++++++++++++
 refs/reftable-backend.c | 14 ++------------
 4 files changed, 40 insertions(+), 18 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 28412576b3..11854894d3 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1225,12 +1225,7 @@ int cmd_clone(int argc,
 	initialize_repository_version(GIT_HASH_UNKNOWN,
 				      the_repository->ref_storage_format, 1);
 
-	strbuf_addf(&buf, "%s/HEAD", git_dir);
-	write_file(buf.buf, "ref: refs/heads/.invalid");
-
-	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/refs", git_dir);
-	safe_create_dir(the_repository, buf.buf, 1);
+	refs_create_refdir_stubs(the_repository, git_dir, NULL);
 
 	/*
 	 * additional config can be injected with -c, make sure it's included
diff --git a/refs.c b/refs.c
index 627b7f8698..32b4edaf2d 100644
--- a/refs.c
+++ b/refs.c
@@ -3402,3 +3402,27 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
 		return "unknown failure";
 	}
 }
+
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_msg)
+{
+	struct strbuf path = STRBUF_INIT;
+
+
+	strbuf_addf(&path, "%s/HEAD", refdir);
+	write_file(path.buf, "ref: refs/heads/.invalid");
+	adjust_shared_perm(repo, path.buf);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/refs", refdir);
+	safe_create_dir(repo, path.buf, 1);
+
+	if (refs_heads_msg) {
+		strbuf_reset(&path);
+		strbuf_addf(&path, "%s/refs/heads", refdir);
+		write_file(path.buf, "%s", refs_heads_msg);
+		adjust_shared_perm(repo, path.buf);
+	}
+
+	strbuf_release(&path);
+}
diff --git a/refs.h b/refs.h
index f0abfa1d93..9d8890fdff 100644
--- a/refs.h
+++ b/refs.h
@@ -1427,4 +1427,17 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
+/*
+ * Git only recognizes a directory as a repository if it contains:
+ * - HEAD file
+ * - refs/ folder
+ * While it is necessary within the files backend, newer backends may not
+ * follow the same structure. To go around this, we create stubs as necessary.
+ *
+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file
+ * with the provided message.
+ */
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_msg);
+
 #endif /* REFS_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index fe74af73af..d8651fe779 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,18 +491,8 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	write_file(sb.buf, "ref: refs/heads/.invalid");
-	adjust_shared_perm(the_repository, sb.buf);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	safe_create_dir(the_repository, sb.buf, 1);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	write_file(sb.buf, "this repository uses the reftable format");
-	adjust_shared_perm(the_repository, sb.buf);
+	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
+				 "this repository uses the reftable format");
 
 	strbuf_release(&sb);
 	return 0;

-- 
2.52.0

Re: [PATCH v4 2/4] refs: extract out `refs_create_refdir_stubs()`

[Patrick Steinhardt]

On Mon, Feb 02, 2026 at 01:26:31PM +0100, Karthik Nayak wrote:
> For Git to recognize a directory as a Git directory, it requires the
> directory to contain:
> 
>   1. 'HEAD' file
>   2. object/ directory

s|object/|objects/|

>   3. 'refs/' directory
> 
> Here, #1 and #3 are part of the reference storage mechanism,
> specifically the files backend. Since then, newer backends such as the
> reftable backend have moved to using their own path ('reftable/') for
> storing references. But to ensure git still recognizes the directory as

s/git/Git/

> a Git directory, we create stubs.
> 
> There are two locations we create stubs:

s/we/where &/

> diff --git a/refs.c b/refs.c
> index 627b7f8698..32b4edaf2d 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -3402,3 +3402,27 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
>  		return "unknown failure";
>  	}
>  }
> +
> +void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
> +			      const char *refs_heads_msg)

It's not really a message, but much rather its content, right?

Patrick

Re: [PATCH v4 2/4] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 02, 2026 at 01:26:31PM +0100, Karthik Nayak wrote:
>> For Git to recognize a directory as a Git directory, it requires the
>> directory to contain:
>>
>>   1. 'HEAD' file
>>   2. object/ directory
>
> s|object/|objects/|
>
>>   3. 'refs/' directory
>>
>> Here, #1 and #3 are part of the reference storage mechanism,
>> specifically the files backend. Since then, newer backends such as the
>> reftable backend have moved to using their own path ('reftable/') for
>> storing references. But to ensure git still recognizes the directory as
>
> s/git/Git/
>
>> a Git directory, we create stubs.
>>
>> There are two locations we create stubs:
>
> s/we/where &/
>
>> diff --git a/refs.c b/refs.c
>> index 627b7f8698..32b4edaf2d 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -3402,3 +3402,27 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
>>  		return "unknown failure";
>>  	}
>>  }
>> +
>> +void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>> +			      const char *refs_heads_msg)
>
> It's not really a message, but much rather its content, right?
>
> Patrick

Indeed, will also patch up the typos. Thanks,

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v4 3/4] refs: parse and use the reference storage payload

[Karthik Nayak]

The previous commit extended the 'extensions.refStorage' config to add
support for a reference storage payload. The payload provides backend
specific information on where to store references for a given directory.

Propagate this information to individual backends when initializing them
via the 'init()' function. Both the files and reftable backends will
parse the information to be filesystem paths to store references.

To enable this, provide a 'refs_compute_filesystem_location()' function
which will parse the current 'gitdir' and the 'payload' to provide the
final reference directory and common reference directory (if working in
a linked worktree).

Finally, for linked worktrees, traditionally references were stored in
the '$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate
reference storage path, it doesn't make sense to store main worktree
references in the new path, and linked worktree references in the
$GIT_DIR path. So, let's store linked worktree references in
'$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id'. To do this, create the
necessary files and folders and also add stubs in the $GIT_DIR path to
ensure that it is still considered a Git directory.

Since this commit adds the required linking, also add the necessary
documentation and tests.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/config/extensions.adoc |  16 +++-
 builtin/worktree.c                   |  35 ++++++++
 refs.c                               |  37 +++++++-
 refs/files-backend.c                 |  18 ++--
 refs/packed-backend.c                |   1 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  15 ++++
 refs/reftable-backend.c              |  24 +++---
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
 10 files changed, 290 insertions(+), 17 deletions(-)

diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
index 532456644b..df86da6aa7 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
 `core.repositoryFormatVersion` setting.
 
 refStorage:::
-	Specify the ref storage format to use. The acceptable values are:
+	Specify the ref storage format and location to use. The value can be
+	either a format name or a URI:
 +
 --
+* A format name alone (e.g., `reftable` or `files`) uses the default
+  location (the repository's common directory).
+
+* A URI format `<format>://<location>` explicitly specifies both the
+  format and payload (e.g., `reftable:///foo/bar`).
+
+Supported format names are:
++
 include::../ref-storage-format.adoc[]
++
+The payload is passed directly to the reference backend. For the files and
+reftable backends, this must be a filesystem path. Relative paths are resolved
+relative to the $GIT_DIR. Future backends may support other payload schemes,
+e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +
 Note that this setting should only be set by linkgit:git-init[1] or
diff --git a/builtin/worktree.c b/builtin/worktree.c
index fbdaf2eb2e..800a376ac5 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -425,6 +425,40 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
 	return run_command(&cp);
 }
 
+/*
+ * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
+ * But when using alternate reference directories, we want to store the worktree
+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
+ *
+ * Create the necessary folder structure to facilitate the same. But to ensure
+ * that the former path is still considered a Git directory, add stubs (similar
+ *  to how we do in the reftable backend).
+ */
+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	char *path;
+
+	path = wt->repo->ref_storage_payload;
+	if (!path)
+		return;
+
+	if (!is_absolute_path(path))
+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
+
+	strbuf_addf(&sb, "%s/worktrees", path);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_addf(&sb, "/%s", wt->id);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_reset(&sb);
+
+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
+		   path, wt->id);
+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
+
+	strbuf_release(&sb);
+}
+
 static int add_worktree(const char *path, const char *refname,
 			const struct add_opts *opts)
 {
@@ -518,6 +552,7 @@ static int add_worktree(const char *path, const char *refname,
 		ret = error(_("could not find created worktree '%s'"), name);
 		goto done;
 	}
+	setup_alternate_ref_dir(wt, sb_repo.buf);
 	wt_refs = get_worktree_ref_store(wt);
 
 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
diff --git a/refs.c b/refs.c
index 32b4edaf2d..c1d69082a9 100644
--- a/refs.c
+++ b/refs.c
@@ -5,6 +5,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "abspath.h"
 #include "advice.h"
 #include "config.h"
 #include "environment.h"
@@ -2224,7 +2225,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, gitdir, flags);
+	/*
+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
+	 * allow the backend to handle how it wants to deal with worktrees.
+	 */
+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
 	return refs;
 }
 
@@ -3426,3 +3431,33 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
 
 	strbuf_release(&path);
 }
+
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	strbuf_addstr(refdir, gitdir);
+	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
+
+	if (!payload)
+		return;
+
+	if (!is_absolute_path(payload)) {
+		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
+		strbuf_realpath(ref_common_dir, sb.buf, 1);
+	} else {
+		strbuf_realpath(ref_common_dir, payload, 1);
+	}
+
+	strbuf_reset(refdir);
+	strbuf_addbuf(refdir, ref_common_dir);
+
+	if (*is_worktree) {
+		char *wt_id = strrchr(gitdir, '/') + 1;
+		strbuf_addf(refdir, "/worktrees/%s", wt_id);
+	}
+
+	strbuf_release(&sb);
+}
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 240d3c3b26..160ecb53b7 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
  * set of caches.
  */
 static struct ref_store *files_ref_store_init(struct repository *repo,
+					      const char *payload,
 					      const char *gitdir,
 					      unsigned int flags)
 {
 	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
 	struct ref_store *ref_store = (struct ref_store *)refs;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
+	bool is_worktree;
+
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
 
-	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
+	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
 	refs->store_flags = flags;
-	get_common_dir_noenv(&sb, gitdir);
-	refs->gitcommondir = strbuf_detach(&sb, NULL);
+	refs->gitcommondir = xstrdup(ref_common_dir.buf);
 	refs->packed_ref_store =
-		packed_ref_store_init(repo, refs->gitcommondir, flags);
+		packed_ref_store_init(repo, payload, ref_common_dir.buf, flags);
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
 	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
 
@@ -126,6 +131,9 @@ static struct ref_store *files_ref_store_init(struct repository *repo,
 	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
 			      &refs->gitcommondir);
 
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
+
 	return ref_store;
 }
 
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 4ea0c12299..028fbc0585 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -212,6 +212,7 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
 }
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload UNUSED,
 					const char *gitdir,
 					unsigned int store_flags)
 {
diff --git a/refs/packed-backend.h b/refs/packed-backend.h
index 9481d5e7c2..2c2377a356 100644
--- a/refs/packed-backend.h
+++ b/refs/packed-backend.h
@@ -14,6 +14,7 @@ struct ref_transaction;
  */
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload,
 					const char *gitdir,
 					unsigned int store_flags);
 
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index c7d2a6e50b..bd09b1280c 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -389,6 +389,7 @@ struct ref_store;
  * the ref_store and to record the ref_store for later lookup.
  */
 typedef struct ref_store *ref_store_init_fn(struct repository *repo,
+					    const char *payload,
 					    const char *gitdir,
 					    unsigned int flags);
 /*
@@ -666,4 +667,18 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 					  unsigned int initial_transaction,
 					  struct strbuf *err);
 
+/*
+ * Given a gitdir and the reference storage payload provided, retrieve the
+ * 'refdir' and 'ref_common_dir'. The former is where references should be
+ * stored for the current worktree, the latter is the common reference
+ * directory if working with a linked worktree. If working with the main
+ * worktree, both values will be the same.
+ *
+ * This is used by backends such as {files, reftable} which store references in
+ * dedicated filesystem paths.
+ */
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir);
+
 #endif /* REFS_REFS_INTERNAL_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d8651fe779..964b0b50fc 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -372,18 +372,24 @@ static int reftable_be_fsync(int fd)
 }
 
 static struct ref_store *reftable_be_init(struct repository *repo,
+					  const char *payload,
 					  const char *gitdir,
 					  unsigned int store_flags)
 {
 	struct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
 	struct strbuf path = STRBUF_INIT;
-	int is_worktree;
+	bool is_worktree;
 	mode_t mask;
 
 	mask = umask(0);
 	umask(mask);
 
-	base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
+
+	base_ref_store_init(&refs->base, repo, refdir.buf, &refs_be_reftable);
 	strmap_init(&refs->worktree_backends);
 	refs->store_flags = store_flags;
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
@@ -419,14 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	/*
 	 * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
 	 * This stack contains both the shared and the main worktree refs.
-	 *
-	 * Note that we don't try to resolve the path in case we have a
-	 * worktree because `get_common_dir_noenv()` already does it for us.
 	 */
-	is_worktree = get_common_dir_noenv(&path, gitdir);
+	strbuf_addbuf(&path, &ref_common_dir);
 	if (!is_worktree) {
 		strbuf_reset(&path);
-		strbuf_realpath(&path, gitdir, 0);
+		strbuf_realpath(&path, ref_common_dir.buf, 0);
 	}
 	strbuf_addstr(&path, "/reftable");
 	refs->err = reftable_backend_init(&refs->main_backend, path.buf,
@@ -443,10 +446,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	 * do it efficiently.
 	 */
 	if (is_worktree) {
-		strbuf_reset(&path);
-		strbuf_addf(&path, "%s/reftable", gitdir);
+		strbuf_addstr(&refdir, "/reftable");
 
-		refs->err = reftable_backend_init(&refs->worktree_backend, path.buf,
+		refs->err = reftable_backend_init(&refs->worktree_backend, refdir.buf,
 						  &refs->write_options);
 		if (refs->err)
 			goto done;
@@ -456,6 +458,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 
 done:
 	assert(refs->err != REFTABLE_API_ERROR);
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
 	strbuf_release(&path);
 	return &refs->base;
 }
diff --git a/t/meson.build b/t/meson.build
index 459c52a489..11fc5a49ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -210,6 +210,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..9c777b79f3
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='Test reference backend URIs'
+
+. ./test-lib.sh
+
+# Run a git command with the provided reference storage. Reset the backend
+# post running the command.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <cmd> is the git subcommand to be run in the repository.
+run_with_uri() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	cmd=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1
+	git -C "$repo" config set extensions.refStorage "$uri" &&
+	git -C "$repo" $cmd &&
+	git -C "$repo" config set extensions.refStorage "$backend"
+}
+
+# Test a repository with a given reference storage by running and comparing
+# 'git refs list' before and after setting the new reference backend. If
+# err_msg is set, expect the command to fail and grep for the provided err_msg.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
+test_refs_backend() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	err_msg=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test -n "$err_msg";
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		test_must_fail git -C "$repo" refs list 2>err &&
+		test_grep "$err_msg" err
+	else
+		git -C "$repo" refs list >expect &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		test_cmp expect actual
+	fi
+}
+
+test_expect_success 'URI is invalid' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable@/home/reftable" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'URI ends with colon' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable:" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "db://.git" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+
+for to_format in $ref_formats
+do
+	if test "$from_format" = "$to_format"
+	then
+		continue
+	fi
+
+
+	for dir in "$(pwd)/repo/.git" "./"
+	do
+
+		test_expect_success "$read from $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
+			)
+		'
+
+		test_expect_success "$write to $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+
+				git refs list >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				git refs list | grep -v "refs/tags/1" >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "with worktree and $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo wt" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				git config set core.repositoryformatversion 1 &&
+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+
+				git worktree add ../wt 2
+			) &&
+
+			git -C repo for-each-ref --include-root-refs >expect &&
+			git -C wt for-each-ref --include-root-refs >expect &&
+			! test_cmp expect actual &&
+
+			git -C wt rev-parse 2 >expect &&
+			git -C wt rev-parse HEAD >actual &&
+			test_cmp expect actual
+		'
+	done # closes dir
+done # closes to_format
+done # closes from_format
+
+test_done

-- 
2.52.0

Re: [PATCH v4 3/4] refs: parse and use the reference storage payload

[Patrick Steinhardt]

On Mon, Feb 02, 2026 at 01:26:32PM +0100, Karthik Nayak wrote:
> The previous commit extended the 'extensions.refStorage' config to add
> support for a reference storage payload. The payload provides backend
> specific information on where to store references for a given directory.
> 
> Propagate this information to individual backends when initializing them
> via the 'init()' function. Both the files and reftable backends will
> parse the information to be filesystem paths to store references.
> 
> To enable this, provide a 'refs_compute_filesystem_location()' function
> which will parse the current 'gitdir' and the 'payload' to provide the
> final reference directory and common reference directory (if working in
> a linked worktree).
> 
> Finally, for linked worktrees, traditionally references were stored in
> the '$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate
> reference storage path, it doesn't make sense to store main worktree
> references in the new path, and linked worktree references in the
> $GIT_DIR path. So, let's store linked worktree references in
> '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id'. To do this, create the

s/wt_id/&>/

> diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
> index 532456644b..df86da6aa7 100644
> --- a/Documentation/config/extensions.adoc
> +++ b/Documentation/config/extensions.adoc
> @@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
>  `core.repositoryFormatVersion` setting.
>  
>  refStorage:::
> -	Specify the ref storage format to use. The acceptable values are:
> +	Specify the ref storage format and location to use. The value can be
> +	either a format name or a URI:
>  +
>  --
> +* A format name alone (e.g., `reftable` or `files`) uses the default
> +  location (the repository's common directory).
> +
> +* A URI format `<format>://<location>` explicitly specifies both the

I think I mentioned this before, but shouldn't we say payload instead of
location here? It may not be a location for all backends.

> diff --git a/builtin/worktree.c b/builtin/worktree.c
> index fbdaf2eb2e..800a376ac5 100644
> --- a/builtin/worktree.c
> +++ b/builtin/worktree.c
> @@ -425,6 +425,40 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
>  	return run_command(&cp);
>  }
>  
> +/*
> + * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
> + * But when using alternate reference directories, we want to store the worktree
> + * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
> + *
> + * Create the necessary folder structure to facilitate the same. But to ensure
> + * that the former path is still considered a Git directory, add stubs (similar
> + *  to how we do in the reftable backend).
> + */

I think the last part in the braces isn't necessary to say anymore given
that we simply shell out to a function now.

> diff --git a/refs.c b/refs.c
> index 32b4edaf2d..c1d69082a9 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2224,7 +2225,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
>  	if (!be)
>  		BUG("reference backend is unknown");
>  
> -	refs = be->init(repo, gitdir, flags);
> +	/*
> +	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
> +	 * allow the backend to handle how it wants to deal with worktrees.
> +	 */

It would be nice if the commit message mentioned why this isn't done now
already.

> diff --git a/refs/files-backend.c b/refs/files-backend.c
> index 240d3c3b26..160ecb53b7 100644
> --- a/refs/files-backend.c
> +++ b/refs/files-backend.c
> @@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
>   * set of caches.
>   */
>  static struct ref_store *files_ref_store_init(struct repository *repo,
> +					      const char *payload,
>  					      const char *gitdir,
>  					      unsigned int flags)
>  {
>  	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
>  	struct ref_store *ref_store = (struct ref_store *)refs;
> -	struct strbuf sb = STRBUF_INIT;
> +	struct strbuf ref_common_dir = STRBUF_INIT;
> +	struct strbuf refdir = STRBUF_INIT;
> +	bool is_worktree;
> +
> +	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
> +					 &ref_common_dir);
>  
> -	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
> +	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
>  	refs->store_flags = flags;
> -	get_common_dir_noenv(&sb, gitdir);
> -	refs->gitcommondir = strbuf_detach(&sb, NULL);
> +	refs->gitcommondir = xstrdup(ref_common_dir.buf);

We can use `strbuf_detach()` instead to save an allocation and then use
`refs->gitcommondir` below.

>  	refs->packed_ref_store =
> -		packed_ref_store_init(repo, refs->gitcommondir, flags);
> +		packed_ref_store_init(repo, payload, ref_common_dir.buf, flags);
>  	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
>  	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);

Patrick

Re: [PATCH v4 3/4] refs: parse and use the reference storage payload

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 5354 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 02, 2026 at 01:26:32PM +0100, Karthik Nayak wrote:
>> The previous commit extended the 'extensions.refStorage' config to add
>> support for a reference storage payload. The payload provides backend
>> specific information on where to store references for a given directory.
>>
>> Propagate this information to individual backends when initializing them
>> via the 'init()' function. Both the files and reftable backends will
>> parse the information to be filesystem paths to store references.
>>
>> To enable this, provide a 'refs_compute_filesystem_location()' function
>> which will parse the current 'gitdir' and the 'payload' to provide the
>> final reference directory and common reference directory (if working in
>> a linked worktree).
>>
>> Finally, for linked worktrees, traditionally references were stored in
>> the '$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate
>> reference storage path, it doesn't make sense to store main worktree
>> references in the new path, and linked worktree references in the
>> $GIT_DIR path. So, let's store linked worktree references in
>> '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id'. To do this, create the
>
> s/wt_id/&>/
>

Thanks!

>> diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
>> index 532456644b..df86da6aa7 100644
>> --- a/Documentation/config/extensions.adoc
>> +++ b/Documentation/config/extensions.adoc
>> @@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
>>  `core.repositoryFormatVersion` setting.
>>
>>  refStorage:::
>> -	Specify the ref storage format to use. The acceptable values are:
>> +	Specify the ref storage format and location to use. The value can be
>> +	either a format name or a URI:
>>  +
>>  --
>> +* A format name alone (e.g., `reftable` or `files`) uses the default
>> +  location (the repository's common directory).
>> +
>> +* A URI format `<format>://<location>` explicitly specifies both the
>
> I think I mentioned this before, but shouldn't we say payload instead of
> location here? It may not be a location for all backends.
>

Yeah, you did. I missed this entirely somehow. Will re-do this.

>> diff --git a/builtin/worktree.c b/builtin/worktree.c
>> index fbdaf2eb2e..800a376ac5 100644
>> --- a/builtin/worktree.c
>> +++ b/builtin/worktree.c
>> @@ -425,6 +425,40 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
>>  	return run_command(&cp);
>>  }
>>
>> +/*
>> + * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
>> + * But when using alternate reference directories, we want to store the worktree
>> + * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
>> + *
>> + * Create the necessary folder structure to facilitate the same. But to ensure
>> + * that the former path is still considered a Git directory, add stubs (similar
>> + *  to how we do in the reftable backend).
>> + */
>
> I think the last part in the braces isn't necessary to say anymore given
> that we simply shell out to a function now.
>

Yeah, will remove.

>> diff --git a/refs.c b/refs.c
>> index 32b4edaf2d..c1d69082a9 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2224,7 +2225,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
>>  	if (!be)
>>  		BUG("reference backend is unknown");
>>
>> -	refs = be->init(repo, gitdir, flags);
>> +	/*
>> +	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
>> +	 * allow the backend to handle how it wants to deal with worktrees.
>> +	 */
>
> It would be nice if the commit message mentioned why this isn't done now
> already.

Will do!

>> diff --git a/refs/files-backend.c b/refs/files-backend.c
>> index 240d3c3b26..160ecb53b7 100644
>> --- a/refs/files-backend.c
>> +++ b/refs/files-backend.c
>> @@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
>>   * set of caches.
>>   */
>>  static struct ref_store *files_ref_store_init(struct repository *repo,
>> +					      const char *payload,
>>  					      const char *gitdir,
>>  					      unsigned int flags)
>>  {
>>  	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
>>  	struct ref_store *ref_store = (struct ref_store *)refs;
>> -	struct strbuf sb = STRBUF_INIT;
>> +	struct strbuf ref_common_dir = STRBUF_INIT;
>> +	struct strbuf refdir = STRBUF_INIT;
>> +	bool is_worktree;
>> +
>> +	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
>> +					 &ref_common_dir);
>>
>> -	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
>> +	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
>>  	refs->store_flags = flags;
>> -	get_common_dir_noenv(&sb, gitdir);
>> -	refs->gitcommondir = strbuf_detach(&sb, NULL);
>> +	refs->gitcommondir = xstrdup(ref_common_dir.buf);
>
> We can use `strbuf_detach()` instead to save an allocation and then use
> `refs->gitcommondir` below.
>

That's a good suggestion.

>>  	refs->packed_ref_store =
>> -		packed_ref_store_init(repo, refs->gitcommondir, flags);
>> +		packed_ref_store_init(repo, payload, ref_common_dir.buf, flags);
>>  	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
>>  	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
>
> Patrick

Thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v4 4/4] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
the previous commit we extended the 'extensions.refStorage' config to
also support an URI input for reference backend with location.

Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
takes in the same input as the config variable. Having an environment
variable allows us to modify the reference backend and location on the
fly for individual git commands.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |  5 +++
 environment.h          |  1 +
 setup.c                | 20 +++++++++++
 t/t1423-ref-backend.sh | 96 ++++++++++++++++++++++++++++++++------------------
 4 files changed, 88 insertions(+), 34 deletions(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..ed3191e8f6 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REFERENCE_BACKEND`::
+    Specify which reference backend to be used along with its URI.
+    See `extensions.refStorage` option in linkgit:git-config[1] for more
+    description. Overrides the config variable when used.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 27f657af04..540e0a7f6d 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/setup.c b/setup.c
index 44e393c251..b4a7b82cca 100644
--- a/setup.c
+++ b/setup.c
@@ -1838,6 +1838,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 	static struct strbuf cwd = STRBUF_INIT;
 	struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT;
 	const char *prefix = NULL;
+	const char *ref_backend_uri;
 	struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
 
 	/*
@@ -1995,6 +1996,25 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
+	/*
+	 * The env variable should override the repository config
+	 * for 'extensions.refStorage'.
+	 */
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *location;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &location);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+		repo_set_ref_storage_format(the_repository, format, location);
+
+		free(backend);
+		free(location);
+	}
+
 	setup_original_cwd();
 
 	strbuf_release(&dir);
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
index 9c777b79f3..10a9bb1a9b 100755
--- a/t/t1423-ref-backend.sh
+++ b/t/t1423-ref-backend.sh
@@ -11,16 +11,25 @@ test_description='Test reference backend URIs'
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
 #   <cmd> is the git subcommand to be run in the repository.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 run_with_uri() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
 	cmd=$4 &&
+	via=$5 &&
 
-	git -C "$repo" config set core.repositoryformatversion 1
-	git -C "$repo" config set extensions.refStorage "$uri" &&
-	git -C "$repo" $cmd &&
-	git -C "$repo" config set extensions.refStorage "$backend"
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test "$via" = "env"
+	then
+		test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd
+	elif test "$via" = "config"
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		git -C "$repo" $cmd &&
+		git -C "$repo" config set extensions.refStorage "$backend"
+	fi
 }
 
 # Test a repository with a given reference storage by running and comparing
@@ -30,44 +39,57 @@ run_with_uri() {
 #   <repo> is the relative path to the repo to run the command in.
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
 test_refs_backend() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
-	err_msg=$4 &&
+	via=$4 &&
+	err_msg=$5 &&
+
 
-	git -C "$repo" config set core.repositoryformatversion 1 &&
 	if test -n "$err_msg";
 	then
-		git -C "$repo" config set extensions.refStorage "$uri" &&
-		test_must_fail git -C "$repo" refs list 2>err &&
-		test_grep "$err_msg" err
+		if test "$via" = "env"
+		then
+			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
+		elif test "$via" = "config"
+		then
+			git -C "$repo" config set extensions.refStorage "$uri" &&
+			test_must_fail git -C "$repo" refs list 2>err &&
+			test_grep "$err_msg" err
+		fi
 	else
 		git -C "$repo" refs list >expect &&
-		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
 		test_cmp expect actual
 	fi
 }
 
-test_expect_success 'URI is invalid' '
+methods="config"
+for method in $methods
+do
+
+test_expect_success "$method: URI is invalid" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable@/home/reftable" \
+	test_refs_backend repo files "reftable@/home/reftable" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'URI ends with colon' '
+test_expect_success "$method: URI ends with colon" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable:" \
+	test_refs_backend repo files "reftable:" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'unknown reference backend' '
+test_expect_success "$method: unknown reference backend" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "db://.git" \
+	test_refs_backend repo files "db://.git" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
@@ -86,7 +108,7 @@ do
 	for dir in "$(pwd)/repo/.git" "./"
 	do
 
-		test_expect_success "$read from $to_format backend, $dir dir" '
+		test_expect_success "$method: $read from $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -101,7 +123,7 @@ do
 			)
 		'
 
-		test_expect_success "$write to $to_format backend, $dir dir" '
+		test_expect_success "$method: $write to $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -113,20 +135,22 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" &&
 
 				git refs list >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"tag -d 1" "$method" &&
 				git refs list >actual &&
 				test_cmp expect actual &&
 
 				git refs list | grep -v "refs/tags/1" >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"refs list" "$method" >actual &&
 				test_cmp expect actual
 			)
 		'
 
-		test_expect_success "with worktree and $to_format backend, $dir dir" '
+		test_expect_success "$method: with worktree and $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo wt" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -138,22 +162,26 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				git config set core.repositoryformatversion 1 &&
-				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"worktree add ../wt 2" "$method" &&
 
-				git worktree add ../wt 2
-			) &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >expect &&
+				! test_cmp expect actual &&
 
-			git -C repo for-each-ref --include-root-refs >expect &&
-			git -C wt for-each-ref --include-root-refs >expect &&
-			! test_cmp expect actual &&
-
-			git -C wt rev-parse 2 >expect &&
-			git -C wt rev-parse HEAD >actual &&
-			test_cmp expect actual
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse 2" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse HEAD" "$method" >expect &&
+				test_cmp expect actual
+			)
 		'
 	done # closes dir
 done # closes to_format
-done # closes from_format
+done # closes to_format
+
+done # closes method
 
 test_done

-- 
2.52.0

Re: [PATCH v4 4/4] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Patrick Steinhardt]

On Mon, Feb 02, 2026 at 01:26:33PM +0100, Karthik Nayak wrote:
> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
> the previous commit we extended the 'extensions.refStorage' config to
> also support an URI input for reference backend with location.
> 
> Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
> takes in the same input as the config variable. Having an environment
> variable allows us to modify the reference backend and location on the
> fly for individual git commands.

s/git/Git/

> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
> index ce099e78b8..ed3191e8f6 100644
> --- a/Documentation/git.adoc
> +++ b/Documentation/git.adoc
> @@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
>  	repositories will be set to this value. The default is "files".
>  	See `--ref-format` in linkgit:git-init[1].
>  
> +`GIT_REFERENCE_BACKEND`::
> +    Specify which reference backend to be used along with its URI.
> +    See `extensions.refStorage` option in linkgit:git-config[1] for more
> +    description. Overrides the config variable when used.

"for more description" reads a bit weird. How about "for more details"
instead?

Patrick

Re: [PATCH v4 4/4] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1392 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 02, 2026 at 01:26:33PM +0100, Karthik Nayak wrote:
>> Git allows setting a different object directory via
>> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
>> the previous commit we extended the 'extensions.refStorage' config to
>> also support an URI input for reference backend with location.
>>
>> Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
>> takes in the same input as the config variable. Having an environment
>> variable allows us to modify the reference backend and location on the
>> fly for individual git commands.
>
> s/git/Git/
>
>> diff --git a/Documentation/git.adoc b/Documentation/git.adoc
>> index ce099e78b8..ed3191e8f6 100644
>> --- a/Documentation/git.adoc
>> +++ b/Documentation/git.adoc
>> @@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
>>  	repositories will be set to this value. The default is "files".
>>  	See `--ref-format` in linkgit:git-init[1].
>>
>> +`GIT_REFERENCE_BACKEND`::
>> +    Specify which reference backend to be used along with its URI.
>> +    See `extensions.refStorage` option in linkgit:git-config[1] for more
>> +    description. Overrides the config variable when used.
>
> "for more description" reads a bit weird. How about "for more details"
> instead?
>
> Patrick

Will make both changes here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v4 0/4] refs: allow setting the reference directory

[Patrick Steinhardt]

On Mon, Feb 02, 2026 at 01:26:29PM +0100, Karthik Nayak wrote:
> Changes in v4:
> - Mostly re-wrote the code to also support worktree. Now, the existing
>   backends will store worktree references in 'ref_dir/worktrees/wt_id'
>   and add corresponding stubs in 'git_dir/worktrees/wt_id'.
> - We also support relative paths in the reference directories. These
>   relative paths are resolved relative to the GIT_DIR.
> - Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

I've got some of comments, but almost all of them are just nits. I quite
like the way this is shaping up.

Thanks!

Patrick

Re: [PATCH v4 0/4] refs: allow setting the reference directory

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 02, 2026 at 01:26:29PM +0100, Karthik Nayak wrote:
>> Changes in v4:
>> - Mostly re-wrote the code to also support worktree. Now, the existing
>>   backends will store worktree references in 'ref_dir/worktrees/wt_id'
>>   and add corresponding stubs in 'git_dir/worktrees/wt_id'.
>> - We also support relative paths in the reference directories. These
>>   relative paths are resolved relative to the GIT_DIR.
>> - Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com
>
> I've got some of comments, but almost all of them are just nits. I quite
> like the way this is shaping up.
>
> Thanks!
>
> Patrick

Looking good.  Thanks, both.

Re: [PATCH v4 0/4] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 739 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 02, 2026 at 01:26:29PM +0100, Karthik Nayak wrote:
>> Changes in v4:
>> - Mostly re-wrote the code to also support worktree. Now, the existing
>>   backends will store worktree references in 'ref_dir/worktrees/wt_id'
>>   and add corresponding stubs in 'git_dir/worktrees/wt_id'.
>> - We also support relative paths in the reference directories. These
>>   relative paths are resolved relative to the GIT_DIR.
>> - Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com
>
> I've got some of comments, but almost all of them are just nits. I quite
> like the way this is shaping up.
>
> Thanks!
>
> Patrick

Thanks for the review. Appreciate it.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v5 0/4] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:

  - A format name alone (e.g., `reftable` or `files`).

  - A URI format `<format>://<payload>` explicitly specifies both the
    format and payload (e.g., `reftable:///foo/bar`).

We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v5:
- Moved around the commits, to ensure that the code to handle the config
  in the backend is first. Previously, we added the config first, which
  meant the commit allowed users to provide a URI but it was simply
  ignore.
- Fix typos and grammar and rename variables.
- Clean up the description and documentation to actually specify
  protocol over location.
- Avoid an extra memory allocation by detaching the strbuf value.
- Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
  backends will store worktree references in 'ref_dir/worktrees/wt_id'
  and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
  relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/config/extensions.adoc |  16 ++-
 Documentation/git.adoc               |   5 +
 builtin/clone.c                      |  10 +-
 builtin/worktree.c                   |  34 +++++++
 environment.h                        |   1 +
 refs.c                               |  60 ++++++++++-
 refs.h                               |  13 +++
 refs/files-backend.c                 |  17 +++-
 refs/packed-backend.c                |   1 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  15 +++
 refs/reftable-backend.c              |  38 +++----
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  59 ++++++++++-
 setup.h                              |   2 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 187 +++++++++++++++++++++++++++++++++++
 18 files changed, 433 insertions(+), 44 deletions(-)

Karthik Nayak (4):
      refs: extract out `refs_create_refdir_stubs()`
      refs: forward and use the reference storage payload
      refs: allow reference location in refstorage config
      refs: add GIT_REFERENCE_BACKEND to specify reference backend

Range-diff versus v4:

2:  57943ad6f9 ! 1:  6ed61ba98b refs: extract out `refs_create_refdir_stubs()`
    @@ Commit message
         directory to contain:
     
           1. 'HEAD' file
    -      2. object/ directory
    +      2. 'objects/' directory
           3. 'refs/' directory
     
         Here, #1 and #3 are part of the reference storage mechanism,
         specifically the files backend. Since then, newer backends such as the
         reftable backend have moved to using their own path ('reftable/') for
    -    storing references. But to ensure git still recognizes the directory as
    +    storing references. But to ensure Git still recognizes the directory as
         a Git directory, we create stubs.
     
    -    There are two locations we create stubs:
    +    There are two locations where we create stubs:
     
         - In 'refs/reftable-backend.c' when creating the reftable backend.
         - In 'clone.c' before spawning transport helpers.
    @@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)
      }
     +
     +void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
    -+			      const char *refs_heads_msg)
    ++			      const char *refs_heads_content)
     +{
     +	struct strbuf path = STRBUF_INIT;
     +
    -+
     +	strbuf_addf(&path, "%s/HEAD", refdir);
     +	write_file(path.buf, "ref: refs/heads/.invalid");
     +	adjust_shared_perm(repo, path.buf);
    @@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)
     +	strbuf_addf(&path, "%s/refs", refdir);
     +	safe_create_dir(repo, path.buf, 1);
     +
    -+	if (refs_heads_msg) {
    ++	if (refs_heads_content) {
     +		strbuf_reset(&path);
     +		strbuf_addf(&path, "%s/refs/heads", refdir);
    -+		write_file(path.buf, "%s", refs_heads_msg);
    ++		write_file(path.buf, "%s", refs_heads_content);
     +		adjust_shared_perm(repo, path.buf);
     +	}
     +
3:  330a9bd253 ! 2:  a907b190d1 refs: parse and use the reference storage payload
    @@ Metadata
     Author: Karthik Nayak <karthik.188@gmail.com>
     
      ## Commit message ##
    -    refs: parse and use the reference storage payload
    +    refs: forward and use the reference storage payload
     
    -    The previous commit extended the 'extensions.refStorage' config to add
    -    support for a reference storage payload. The payload provides backend
    -    specific information on where to store references for a given directory.
    +    An upcoming commit will add support for providing an URI via the
    +    'extensions.refStorage' config. The URI will contain the reference
    +    backend and a corresponding payload. The payload can be then used for
    +    providing an alternate locations for the reference backend.
     
    -    Propagate this information to individual backends when initializing them
    -    via the 'init()' function. Both the files and reftable backends will
    -    parse the information to be filesystem paths to store references.
    +    To prepare for this, modify the existing backends to accept such an
    +    argument when initializing via the 'init()' function. Both the files
    +    and reftable backends will parse the information to be filesystem paths
    +    to store references.
     
         To enable this, provide a 'refs_compute_filesystem_location()' function
         which will parse the current 'gitdir' and the 'payload' to provide the
         final reference directory and common reference directory (if working in
         a linked worktree).
     
    -    Finally, for linked worktrees, traditionally references were stored in
    -    the '$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate
    -    reference storage path, it doesn't make sense to store main worktree
    -    references in the new path, and linked worktree references in the
    -    $GIT_DIR path. So, let's store linked worktree references in
    -    '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id'. To do this, create the
    -    necessary files and folders and also add stubs in the $GIT_DIR path to
    -    ensure that it is still considered a Git directory.
    -
    -    Since this commit adds the required linking, also add the necessary
    -    documentation and tests.
    +    The documentation and tests will be added alongside the extension of the
    +    config variable.
     
         Helped-by: Patrick Steinhardt <ps@pks.im>
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
     
    - ## Documentation/config/extensions.adoc ##
    -@@ Documentation/config/extensions.adoc: For historical reasons, this extension is respected regardless of the
    - `core.repositoryFormatVersion` setting.
    - 
    - refStorage:::
    --	Specify the ref storage format to use. The acceptable values are:
    -+	Specify the ref storage format and location to use. The value can be
    -+	either a format name or a URI:
    - +
    - --
    -+* A format name alone (e.g., `reftable` or `files`) uses the default
    -+  location (the repository's common directory).
    -+
    -+* A URI format `<format>://<location>` explicitly specifies both the
    -+  format and payload (e.g., `reftable:///foo/bar`).
    -+
    -+Supported format names are:
    -++
    - include::../ref-storage-format.adoc[]
    -++
    -+The payload is passed directly to the reference backend. For the files and
    -+reftable backends, this must be a filesystem path. Relative paths are resolved
    -+relative to the $GIT_DIR. Future backends may support other payload schemes,
    -+e.g., postgres://127.0.0.1:5432?database=myrepo.
    - --
    - +
    - Note that this setting should only be set by linkgit:git-init[1] or
    -
    - ## builtin/worktree.c ##
    -@@ builtin/worktree.c: static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
    - 	return run_command(&cp);
    - }
    - 
    -+/*
    -+ * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
    -+ * But when using alternate reference directories, we want to store the worktree
    -+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
    -+ *
    -+ * Create the necessary folder structure to facilitate the same. But to ensure
    -+ * that the former path is still considered a Git directory, add stubs (similar
    -+ *  to how we do in the reftable backend).
    -+ */
    -+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
    -+{
    -+	struct strbuf sb = STRBUF_INIT;
    -+	char *path;
    -+
    -+	path = wt->repo->ref_storage_payload;
    -+	if (!path)
    -+		return;
    -+
    -+	if (!is_absolute_path(path))
    -+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
    -+
    -+	strbuf_addf(&sb, "%s/worktrees", path);
    -+	safe_create_dir(wt->repo, sb.buf, 1);
    -+	strbuf_addf(&sb, "/%s", wt->id);
    -+	safe_create_dir(wt->repo, sb.buf, 1);
    -+	strbuf_reset(&sb);
    -+
    -+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
    -+		   path, wt->id);
    -+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
    -+
    -+	strbuf_release(&sb);
    -+}
    -+
    - static int add_worktree(const char *path, const char *refname,
    - 			const struct add_opts *opts)
    - {
    -@@ builtin/worktree.c: static int add_worktree(const char *path, const char *refname,
    - 		ret = error(_("could not find created worktree '%s'"), name);
    - 		goto done;
    - 	}
    -+	setup_alternate_ref_dir(wt, sb_repo.buf);
    - 	wt_refs = get_worktree_ref_store(wt);
    - 
    - 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
    -
      ## refs.c ##
     @@
      #define USE_THE_REPOSITORY_VARIABLE
    @@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
      		BUG("reference backend is unknown");
      
     -	refs = be->init(repo, gitdir, flags);
    -+	/*
    -+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
    -+	 * allow the backend to handle how it wants to deal with worktrees.
    -+	 */
    -+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
    ++	refs = be->init(repo, NULL, gitdir, flags);
      	return refs;
      }
      
    @@ refs/files-backend.c: static void clear_loose_ref_cache(struct files_ref_store *
      	refs->store_flags = flags;
     -	get_common_dir_noenv(&sb, gitdir);
     -	refs->gitcommondir = strbuf_detach(&sb, NULL);
    -+	refs->gitcommondir = xstrdup(ref_common_dir.buf);
    ++	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
      	refs->packed_ref_store =
     -		packed_ref_store_init(repo, refs->gitcommondir, flags);
    -+		packed_ref_store_init(repo, payload, ref_common_dir.buf, flags);
    ++		packed_ref_store_init(repo, payload, refs->gitcommondir, flags);
      	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
      	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
      
    @@ refs/files-backend.c: static struct ref_store *files_ref_store_init(struct repos
      	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
      			      &refs->gitcommondir);
      
    -+	strbuf_release(&ref_common_dir);
     +	strbuf_release(&refdir);
     +
      	return ref_store;
    @@ refs/reftable-backend.c: static struct ref_store *reftable_be_init(struct reposi
      	strbuf_release(&path);
      	return &refs->base;
      }
    -
    - ## t/meson.build ##
    -@@ t/meson.build: integration_tests = [
    -   't1420-lost-found.sh',
    -   't1421-reflog-write.sh',
    -   't1422-show-ref-exists.sh',
    -+  't1423-ref-backend.sh',
    -   't1430-bad-ref-name.sh',
    -   't1450-fsck.sh',
    -   't1451-fsck-buffer.sh',
    -
    - ## t/t1423-ref-backend.sh (new) ##
    -@@
    -+#!/bin/sh
    -+
    -+test_description='Test reference backend URIs'
    -+
    -+. ./test-lib.sh
    -+
    -+# Run a git command with the provided reference storage. Reset the backend
    -+# post running the command.
    -+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
    -+#   <repo> is the relative path to the repo to run the command in.
    -+#   <backend> is the original ref storage of the repo.
    -+#   <uri> is the new URI to be set for the ref storage.
    -+#   <cmd> is the git subcommand to be run in the repository.
    -+run_with_uri() {
    -+	repo=$1 &&
    -+	backend=$2 &&
    -+	uri=$3 &&
    -+	cmd=$4 &&
    -+
    -+	git -C "$repo" config set core.repositoryformatversion 1
    -+	git -C "$repo" config set extensions.refStorage "$uri" &&
    -+	git -C "$repo" $cmd &&
    -+	git -C "$repo" config set extensions.refStorage "$backend"
    -+}
    -+
    -+# Test a repository with a given reference storage by running and comparing
    -+# 'git refs list' before and after setting the new reference backend. If
    -+# err_msg is set, expect the command to fail and grep for the provided err_msg.
    -+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
    -+#   <repo> is the relative path to the repo to run the command in.
    -+#   <backend> is the original ref storage of the repo.
    -+#   <uri> is the new URI to be set for the ref storage.
    -+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
    -+test_refs_backend() {
    -+	repo=$1 &&
    -+	backend=$2 &&
    -+	uri=$3 &&
    -+	err_msg=$4 &&
    -+
    -+	git -C "$repo" config set core.repositoryformatversion 1 &&
    -+	if test -n "$err_msg";
    -+	then
    -+		git -C "$repo" config set extensions.refStorage "$uri" &&
    -+		test_must_fail git -C "$repo" refs list 2>err &&
    -+		test_grep "$err_msg" err
    -+	else
    -+		git -C "$repo" refs list >expect &&
    -+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
    -+		test_cmp expect actual
    -+	fi
    -+}
    -+
    -+test_expect_success 'URI is invalid' '
    -+	test_when_finished "rm -rf repo" &&
    -+	git init repo &&
    -+	test_refs_backend repo files "reftable@/home/reftable" \
    -+		"invalid value for ${SQ}extensions.refstorage${SQ}"
    -+'
    -+
    -+test_expect_success 'URI ends with colon' '
    -+	test_when_finished "rm -rf repo" &&
    -+	git init repo &&
    -+	test_refs_backend repo files "reftable:" \
    -+		"invalid value for ${SQ}extensions.refstorage${SQ}"
    -+'
    -+
    -+test_expect_success 'unknown reference backend' '
    -+	test_when_finished "rm -rf repo" &&
    -+	git init repo &&
    -+	test_refs_backend repo files "db://.git" \
    -+		"invalid value for ${SQ}extensions.refstorage${SQ}"
    -+'
    -+
    -+ref_formats="files reftable"
    -+for from_format in $ref_formats
    -+do
    -+
    -+for to_format in $ref_formats
    -+do
    -+	if test "$from_format" = "$to_format"
    -+	then
    -+		continue
    -+	fi
    -+
    -+
    -+	for dir in "$(pwd)/repo/.git" "./"
    -+	do
    -+
    -+		test_expect_success "$read from $to_format backend, $dir dir" '
    -+			test_when_finished "rm -rf repo" &&
    -+			git init --ref-format=$from_format repo &&
    -+			(
    -+				cd repo &&
    -+				test_commit 1 &&
    -+				test_commit 2 &&
    -+				test_commit 3 &&
    -+
    -+				git refs migrate --dry-run --ref-format=$to_format >out &&
    -+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
    -+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
    -+			)
    -+		'
    -+
    -+		test_expect_success "$write to $to_format backend, $dir dir" '
    -+			test_when_finished "rm -rf repo" &&
    -+			git init --ref-format=$from_format repo &&
    -+			(
    -+				cd repo &&
    -+				test_commit 1 &&
    -+				test_commit 2 &&
    -+				test_commit 3 &&
    -+
    -+				git refs migrate --dry-run --ref-format=$to_format >out &&
    -+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
    -+
    -+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
    -+
    -+				git refs list >expect &&
    -+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
    -+				git refs list >actual &&
    -+				test_cmp expect actual &&
    -+
    -+				git refs list | grep -v "refs/tags/1" >expect &&
    -+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
    -+				test_cmp expect actual
    -+			)
    -+		'
    -+
    -+		test_expect_success "with worktree and $to_format backend, $dir dir" '
    -+			test_when_finished "rm -rf repo wt" &&
    -+			git init --ref-format=$from_format repo &&
    -+			(
    -+				cd repo &&
    -+				test_commit 1 &&
    -+				test_commit 2 &&
    -+				test_commit 3 &&
    -+
    -+				git refs migrate --dry-run --ref-format=$to_format >out &&
    -+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
    -+
    -+				git config set core.repositoryformatversion 1 &&
    -+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
    -+
    -+				git worktree add ../wt 2
    -+			) &&
    -+
    -+			git -C repo for-each-ref --include-root-refs >expect &&
    -+			git -C wt for-each-ref --include-root-refs >expect &&
    -+			! test_cmp expect actual &&
    -+
    -+			git -C wt rev-parse 2 >expect &&
    -+			git -C wt rev-parse HEAD >actual &&
    -+			test_cmp expect actual
    -+		'
    -+	done # closes dir
    -+done # closes to_format
    -+done # closes from_format
    -+
    -+test_done
1:  77ec79dfc7 ! 3:  630aef7910 refs: allow reference location in refstorage config
    @@ Commit message
         utilize the $GIT_DIR as the reference folder by default in
         `get_main_ref_store()`.
     
    -    Since the reference backends are pluggable, this means that they should
    +    Since the reference backends are pluggable, this means that they could
         work with out-of-tree reference directories too. Extend the 'refStorage'
         config to also support taking an URI input, where users can specify the
         reference backend and the location.
     
         Add the required changes to obtain and propagate this value to the
    -    individual backends. A follow up commit will add the required changes on
    -    the backends to parse this value.
    +    individual backends also add the necessary documentation and tests.
    +
    +    Traditionally, for linked worktrees, references were stored in the
    +    '$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
    +    storage path, it doesn't make sense to store the main worktree
    +    references in the new path, and the linked worktree references in the
    +    $GIT_DIR. So, let's store linked worktree references in
    +    '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'. To do this, create the
    +    necessary files and folders while also adding stubs in the $GIT_DIR path
    +    to ensure that it is still considered a Git directory.
    +
    +    Ideally, we would want to pass in a `struct worktree *` to individual
    +    backends, instead of passing the `gitdir`. This allows them to handle
    +    worktree specific logic. Currently, that is not possible since the
    +    worktree code is:
    +
    +      - Tied to using the global `the_repository` variable.
    +
    +      - Is not setup before the reference database during initialization of
    +        the repository.
    +
    +    Add a TODO in 'refs.c' to ensure we can eventually make that change.
     
         Helped-by: Patrick Steinhardt <ps@pks.im>
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
     
    + ## Documentation/config/extensions.adoc ##
    +@@ Documentation/config/extensions.adoc: For historical reasons, this extension is respected regardless of the
    + `core.repositoryFormatVersion` setting.
    + 
    + refStorage:::
    +-	Specify the ref storage format to use. The acceptable values are:
    ++	Specify the ref storage format and a corresponding payload. The value
    ++	can be either a format name or a URI:
    + +
    + --
    ++* A format name alone (e.g., `reftable` or `files`).
    ++
    ++* A URI format `<format>://<payload>` explicitly specifies both the
    ++  format and payload (e.g., `reftable:///foo/bar`).
    ++
    ++Supported format names are:
    +++
    + include::../ref-storage-format.adoc[]
    +++
    ++The payload is passed directly to the reference backend. For the files and
    ++reftable backends, this must be a filesystem path where the references will
    ++be stored. Defaulting to the commondir when no payload is provided. Relative
    ++paths are resolved relative to the $GIT_DIR. Future backends may support
    ++other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
    + --
    + +
    + Note that this setting should only be set by linkgit:git-init[1] or
    +
      ## builtin/clone.c ##
     @@ builtin/clone.c: int cmd_clone(int argc,
      	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
    @@ builtin/clone.c: int cmd_clone(int argc,
      	/*
      	 * Before fetching from the remote, download and install bundle
     
    + ## builtin/worktree.c ##
    +@@ builtin/worktree.c: static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
    + 	return run_command(&cp);
    + }
    + 
    ++/*
    ++ * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
    ++ * But when using alternate reference directories, we want to store the worktree
    ++ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
    ++ *
    ++ * Create the necessary folder structure to facilitate the same. But to ensure
    ++ * that the former path is still considered a Git directory, add stubs.
    ++ */
    ++static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
    ++{
    ++	struct strbuf sb = STRBUF_INIT;
    ++	char *path;
    ++
    ++	path = wt->repo->ref_storage_payload;
    ++	if (!path)
    ++		return;
    ++
    ++	if (!is_absolute_path(path))
    ++		strbuf_addf(&sb, "%s/", wt->repo->commondir);
    ++
    ++	strbuf_addf(&sb, "%s/worktrees", path);
    ++	safe_create_dir(wt->repo, sb.buf, 1);
    ++	strbuf_addf(&sb, "/%s", wt->id);
    ++	safe_create_dir(wt->repo, sb.buf, 1);
    ++	strbuf_reset(&sb);
    ++
    ++	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
    ++		   path, wt->id);
    ++	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
    ++
    ++	strbuf_release(&sb);
    ++}
    ++
    + static int add_worktree(const char *path, const char *refname,
    + 			const struct add_opts *opts)
    + {
    +@@ builtin/worktree.c: static int add_worktree(const char *path, const char *refname,
    + 		ret = error(_("could not find created worktree '%s'"), name);
    + 		goto done;
    + 	}
    ++	setup_alternate_ref_dir(wt, sb_repo.buf);
    + 	wt_refs = get_worktree_ref_store(wt);
    + 
    + 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
    +
    + ## refs.c ##
    +@@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
    + 	if (!be)
    + 		BUG("reference backend is unknown");
    + 
    +-	refs = be->init(repo, NULL, gitdir, flags);
    ++	/*
    ++	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
    ++	 * allow the backend to handle how it wants to deal with worktrees.
    ++	 */
    ++	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
    + 	return refs;
    + }
    + 
    +
      ## repository.c ##
     @@ repository.c: void repo_set_compat_hash_algo(struct repository *repo, int algo)
      }
    @@ repository.h: struct repository {
      
      	/* Repository's reference storage format, as serialized on disk. */
      	enum ref_storage_format ref_storage_format;
    -+	/* Reference storage information as needed for the backend. */
    ++	/*
    ++	 * Reference storage information as needed for the backend. This contains
    ++	 * only the payload from the reference URI without the schema.
    ++	 */
     +	char *ref_storage_payload;
      
      	/* A unique-id for tracing purposes. */
    @@ setup.h: void initialize_repository_version(int hash_algo,
      			       const char *initial_branch, int quiet);
      
      /*
    +
    + ## t/meson.build ##
    +@@ t/meson.build: integration_tests = [
    +   't1420-lost-found.sh',
    +   't1421-reflog-write.sh',
    +   't1422-show-ref-exists.sh',
    ++  't1423-ref-backend.sh',
    +   't1430-bad-ref-name.sh',
    +   't1450-fsck.sh',
    +   't1451-fsck-buffer.sh',
    +
    + ## t/t1423-ref-backend.sh (new) ##
    +@@
    ++#!/bin/sh
    ++
    ++test_description='Test reference backend URIs'
    ++
    ++. ./test-lib.sh
    ++
    ++# Run a git command with the provided reference storage. Reset the backend
    ++# post running the command.
    ++# Usage: run_with_uri <repo> <backend> <uri> <cmd>
    ++#   <repo> is the relative path to the repo to run the command in.
    ++#   <backend> is the original ref storage of the repo.
    ++#   <uri> is the new URI to be set for the ref storage.
    ++#   <cmd> is the git subcommand to be run in the repository.
    ++run_with_uri() {
    ++	repo=$1 &&
    ++	backend=$2 &&
    ++	uri=$3 &&
    ++	cmd=$4 &&
    ++
    ++	git -C "$repo" config set core.repositoryformatversion 1
    ++	git -C "$repo" config set extensions.refStorage "$uri" &&
    ++	git -C "$repo" $cmd &&
    ++	git -C "$repo" config set extensions.refStorage "$backend"
    ++}
    ++
    ++# Test a repository with a given reference storage by running and comparing
    ++# 'git refs list' before and after setting the new reference backend. If
    ++# err_msg is set, expect the command to fail and grep for the provided err_msg.
    ++# Usage: run_with_uri <repo> <backend> <uri> <cmd>
    ++#   <repo> is the relative path to the repo to run the command in.
    ++#   <backend> is the original ref storage of the repo.
    ++#   <uri> is the new URI to be set for the ref storage.
    ++#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
    ++test_refs_backend() {
    ++	repo=$1 &&
    ++	backend=$2 &&
    ++	uri=$3 &&
    ++	err_msg=$4 &&
    ++
    ++	git -C "$repo" config set core.repositoryformatversion 1 &&
    ++	if test -n "$err_msg";
    ++	then
    ++		git -C "$repo" config set extensions.refStorage "$uri" &&
    ++		test_must_fail git -C "$repo" refs list 2>err &&
    ++		test_grep "$err_msg" err
    ++	else
    ++		git -C "$repo" refs list >expect &&
    ++		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
    ++		test_cmp expect actual
    ++	fi
    ++}
    ++
    ++test_expect_success 'URI is invalid' '
    ++	test_when_finished "rm -rf repo" &&
    ++	git init repo &&
    ++	test_refs_backend repo files "reftable@/home/reftable" \
    ++		"invalid value for ${SQ}extensions.refstorage${SQ}"
    ++'
    ++
    ++test_expect_success 'URI ends with colon' '
    ++	test_when_finished "rm -rf repo" &&
    ++	git init repo &&
    ++	test_refs_backend repo files "reftable:" \
    ++		"invalid value for ${SQ}extensions.refstorage${SQ}"
    ++'
    ++
    ++test_expect_success 'unknown reference backend' '
    ++	test_when_finished "rm -rf repo" &&
    ++	git init repo &&
    ++	test_refs_backend repo files "db://.git" \
    ++		"invalid value for ${SQ}extensions.refstorage${SQ}"
    ++'
    ++
    ++ref_formats="files reftable"
    ++for from_format in $ref_formats
    ++do
    ++
    ++for to_format in $ref_formats
    ++do
    ++	if test "$from_format" = "$to_format"
    ++	then
    ++		continue
    ++	fi
    ++
    ++
    ++	for dir in "$(pwd)/repo/.git" "./"
    ++	do
    ++
    ++		test_expect_success "$read from $to_format backend, $dir dir" '
    ++			test_when_finished "rm -rf repo" &&
    ++			git init --ref-format=$from_format repo &&
    ++			(
    ++				cd repo &&
    ++				test_commit 1 &&
    ++				test_commit 2 &&
    ++				test_commit 3 &&
    ++
    ++				git refs migrate --dry-run --ref-format=$to_format >out &&
    ++				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
    ++				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
    ++			)
    ++		'
    ++
    ++		test_expect_success "$write to $to_format backend, $dir dir" '
    ++			test_when_finished "rm -rf repo" &&
    ++			git init --ref-format=$from_format repo &&
    ++			(
    ++				cd repo &&
    ++				test_commit 1 &&
    ++				test_commit 2 &&
    ++				test_commit 3 &&
    ++
    ++				git refs migrate --dry-run --ref-format=$to_format >out &&
    ++				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
    ++
    ++				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
    ++
    ++				git refs list >expect &&
    ++				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
    ++				git refs list >actual &&
    ++				test_cmp expect actual &&
    ++
    ++				git refs list | grep -v "refs/tags/1" >expect &&
    ++				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
    ++				test_cmp expect actual
    ++			)
    ++		'
    ++
    ++		test_expect_success "with worktree and $to_format backend, $dir dir" '
    ++			test_when_finished "rm -rf repo wt" &&
    ++			git init --ref-format=$from_format repo &&
    ++			(
    ++				cd repo &&
    ++				test_commit 1 &&
    ++				test_commit 2 &&
    ++				test_commit 3 &&
    ++
    ++				git refs migrate --dry-run --ref-format=$to_format >out &&
    ++				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
    ++
    ++				git config set core.repositoryformatversion 1 &&
    ++				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
    ++
    ++				git worktree add ../wt 2
    ++			) &&
    ++
    ++			git -C repo for-each-ref --include-root-refs >expect &&
    ++			git -C wt for-each-ref --include-root-refs >expect &&
    ++			! test_cmp expect actual &&
    ++
    ++			git -C wt rev-parse 2 >expect &&
    ++			git -C wt rev-parse HEAD >actual &&
    ++			test_cmp expect actual
    ++		'
    ++	done # closes dir
    ++done # closes to_format
    ++done # closes from_format
    ++
    ++test_done
4:  dbb7d9c632 ! 4:  590bb706dd refs: add GIT_REFERENCE_BACKEND to specify reference backend
    @@ Commit message
         Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
         takes in the same input as the config variable. Having an environment
         variable allows us to modify the reference backend and location on the
    -    fly for individual git commands.
    +    fly for individual Git commands.
     
         Helped-by: Jean-Noël Avila <jn.avila@free.fr>
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
    @@ Documentation/git.adoc: double-quotes and respecting backslash escapes. E.g., th
     +`GIT_REFERENCE_BACKEND`::
     +    Specify which reference backend to be used along with its URI.
     +    See `extensions.refStorage` option in linkgit:git-config[1] for more
    -+    description. Overrides the config variable when used.
    ++    details. Overrides the config variable when used.
     +
      Git Commits
      ~~~~~~~~~~~


base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v5 1/4] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

For Git to recognize a directory as a Git directory, it requires the
directory to contain:

  1. 'HEAD' file
  2. 'objects/' directory
  3. 'refs/' directory

Here, #1 and #3 are part of the reference storage mechanism,
specifically the files backend. Since then, newer backends such as the
reftable backend have moved to using their own path ('reftable/') for
storing references. But to ensure Git still recognizes the directory as
a Git directory, we create stubs.

There are two locations where we create stubs:

- In 'refs/reftable-backend.c' when creating the reftable backend.
- In 'clone.c' before spawning transport helpers.

In a following commit, we'll add another instance. So instead of
repeating the code, let's extract out this code to
`refs_create_refdir_stubs()` and use it.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c         |  7 +------
 refs.c                  | 23 +++++++++++++++++++++++
 refs.h                  | 13 +++++++++++++
 refs/reftable-backend.c | 14 ++------------
 4 files changed, 39 insertions(+), 18 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index b40cee5968..535a257b10 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1225,12 +1225,7 @@ int cmd_clone(int argc,
 	initialize_repository_version(GIT_HASH_UNKNOWN,
 				      the_repository->ref_storage_format, 1);
 
-	strbuf_addf(&buf, "%s/HEAD", git_dir);
-	write_file(buf.buf, "ref: refs/heads/.invalid");
-
-	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/refs", git_dir);
-	safe_create_dir(the_repository, buf.buf, 1);
+	refs_create_refdir_stubs(the_repository, git_dir, NULL);
 
 	/*
 	 * additional config can be injected with -c, make sure it's included
diff --git a/refs.c b/refs.c
index 627b7f8698..36f3441632 100644
--- a/refs.c
+++ b/refs.c
@@ -3402,3 +3402,26 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
 		return "unknown failure";
 	}
 }
+
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_content)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	strbuf_addf(&path, "%s/HEAD", refdir);
+	write_file(path.buf, "ref: refs/heads/.invalid");
+	adjust_shared_perm(repo, path.buf);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/refs", refdir);
+	safe_create_dir(repo, path.buf, 1);
+
+	if (refs_heads_content) {
+		strbuf_reset(&path);
+		strbuf_addf(&path, "%s/refs/heads", refdir);
+		write_file(path.buf, "%s", refs_heads_content);
+		adjust_shared_perm(repo, path.buf);
+	}
+
+	strbuf_release(&path);
+}
diff --git a/refs.h b/refs.h
index f0abfa1d93..9d8890fdff 100644
--- a/refs.h
+++ b/refs.h
@@ -1427,4 +1427,17 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
+/*
+ * Git only recognizes a directory as a repository if it contains:
+ * - HEAD file
+ * - refs/ folder
+ * While it is necessary within the files backend, newer backends may not
+ * follow the same structure. To go around this, we create stubs as necessary.
+ *
+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file
+ * with the provided message.
+ */
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_msg);
+
 #endif /* REFS_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index fe74af73af..d8651fe779 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,18 +491,8 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	write_file(sb.buf, "ref: refs/heads/.invalid");
-	adjust_shared_perm(the_repository, sb.buf);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	safe_create_dir(the_repository, sb.buf, 1);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	write_file(sb.buf, "this repository uses the reftable format");
-	adjust_shared_perm(the_repository, sb.buf);
+	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
+				 "this repository uses the reftable format");
 
 	strbuf_release(&sb);
 	return 0;

-- 
2.52.0

[PATCH v5 2/4] refs: forward and use the reference storage payload

[Karthik Nayak]

An upcoming commit will add support for providing an URI via the
'extensions.refStorage' config. The URI will contain the reference
backend and a corresponding payload. The payload can be then used for
providing an alternate locations for the reference backend.

To prepare for this, modify the existing backends to accept such an
argument when initializing via the 'init()' function. Both the files
and reftable backends will parse the information to be filesystem paths
to store references.

To enable this, provide a 'refs_compute_filesystem_location()' function
which will parse the current 'gitdir' and the 'payload' to provide the
final reference directory and common reference directory (if working in
a linked worktree).

The documentation and tests will be added alongside the extension of the
config variable.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 33 ++++++++++++++++++++++++++++++++-
 refs/files-backend.c    | 17 ++++++++++++-----
 refs/packed-backend.c   |  1 +
 refs/packed-backend.h   |  1 +
 refs/refs-internal.h    | 15 +++++++++++++++
 refs/reftable-backend.c | 24 ++++++++++++++----------
 6 files changed, 75 insertions(+), 16 deletions(-)

diff --git a/refs.c b/refs.c
index 36f3441632..d9df25d7c0 100644
--- a/refs.c
+++ b/refs.c
@@ -5,6 +5,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "abspath.h"
 #include "advice.h"
 #include "config.h"
 #include "environment.h"
@@ -2224,7 +2225,7 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, gitdir, flags);
+	refs = be->init(repo, NULL, gitdir, flags);
 	return refs;
 }
 
@@ -3425,3 +3426,33 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
 
 	strbuf_release(&path);
 }
+
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	strbuf_addstr(refdir, gitdir);
+	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
+
+	if (!payload)
+		return;
+
+	if (!is_absolute_path(payload)) {
+		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
+		strbuf_realpath(ref_common_dir, sb.buf, 1);
+	} else {
+		strbuf_realpath(ref_common_dir, payload, 1);
+	}
+
+	strbuf_reset(refdir);
+	strbuf_addbuf(refdir, ref_common_dir);
+
+	if (*is_worktree) {
+		char *wt_id = strrchr(gitdir, '/') + 1;
+		strbuf_addf(refdir, "/worktrees/%s", wt_id);
+	}
+
+	strbuf_release(&sb);
+}
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 240d3c3b26..b192ce606d 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
  * set of caches.
  */
 static struct ref_store *files_ref_store_init(struct repository *repo,
+					      const char *payload,
 					      const char *gitdir,
 					      unsigned int flags)
 {
 	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
 	struct ref_store *ref_store = (struct ref_store *)refs;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
+	bool is_worktree;
+
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
 
-	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
+	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
 	refs->store_flags = flags;
-	get_common_dir_noenv(&sb, gitdir);
-	refs->gitcommondir = strbuf_detach(&sb, NULL);
+	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
 	refs->packed_ref_store =
-		packed_ref_store_init(repo, refs->gitcommondir, flags);
+		packed_ref_store_init(repo, payload, refs->gitcommondir, flags);
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
 	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
 
@@ -126,6 +131,8 @@ static struct ref_store *files_ref_store_init(struct repository *repo,
 	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
 			      &refs->gitcommondir);
 
+	strbuf_release(&refdir);
+
 	return ref_store;
 }
 
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 4ea0c12299..028fbc0585 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -212,6 +212,7 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
 }
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload UNUSED,
 					const char *gitdir,
 					unsigned int store_flags)
 {
diff --git a/refs/packed-backend.h b/refs/packed-backend.h
index 9481d5e7c2..2c2377a356 100644
--- a/refs/packed-backend.h
+++ b/refs/packed-backend.h
@@ -14,6 +14,7 @@ struct ref_transaction;
  */
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload,
 					const char *gitdir,
 					unsigned int store_flags);
 
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index c7d2a6e50b..bd09b1280c 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -389,6 +389,7 @@ struct ref_store;
  * the ref_store and to record the ref_store for later lookup.
  */
 typedef struct ref_store *ref_store_init_fn(struct repository *repo,
+					    const char *payload,
 					    const char *gitdir,
 					    unsigned int flags);
 /*
@@ -666,4 +667,18 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 					  unsigned int initial_transaction,
 					  struct strbuf *err);
 
+/*
+ * Given a gitdir and the reference storage payload provided, retrieve the
+ * 'refdir' and 'ref_common_dir'. The former is where references should be
+ * stored for the current worktree, the latter is the common reference
+ * directory if working with a linked worktree. If working with the main
+ * worktree, both values will be the same.
+ *
+ * This is used by backends such as {files, reftable} which store references in
+ * dedicated filesystem paths.
+ */
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir);
+
 #endif /* REFS_REFS_INTERNAL_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d8651fe779..964b0b50fc 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -372,18 +372,24 @@ static int reftable_be_fsync(int fd)
 }
 
 static struct ref_store *reftable_be_init(struct repository *repo,
+					  const char *payload,
 					  const char *gitdir,
 					  unsigned int store_flags)
 {
 	struct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
 	struct strbuf path = STRBUF_INIT;
-	int is_worktree;
+	bool is_worktree;
 	mode_t mask;
 
 	mask = umask(0);
 	umask(mask);
 
-	base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
+
+	base_ref_store_init(&refs->base, repo, refdir.buf, &refs_be_reftable);
 	strmap_init(&refs->worktree_backends);
 	refs->store_flags = store_flags;
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
@@ -419,14 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	/*
 	 * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
 	 * This stack contains both the shared and the main worktree refs.
-	 *
-	 * Note that we don't try to resolve the path in case we have a
-	 * worktree because `get_common_dir_noenv()` already does it for us.
 	 */
-	is_worktree = get_common_dir_noenv(&path, gitdir);
+	strbuf_addbuf(&path, &ref_common_dir);
 	if (!is_worktree) {
 		strbuf_reset(&path);
-		strbuf_realpath(&path, gitdir, 0);
+		strbuf_realpath(&path, ref_common_dir.buf, 0);
 	}
 	strbuf_addstr(&path, "/reftable");
 	refs->err = reftable_backend_init(&refs->main_backend, path.buf,
@@ -443,10 +446,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	 * do it efficiently.
 	 */
 	if (is_worktree) {
-		strbuf_reset(&path);
-		strbuf_addf(&path, "%s/reftable", gitdir);
+		strbuf_addstr(&refdir, "/reftable");
 
-		refs->err = reftable_backend_init(&refs->worktree_backend, path.buf,
+		refs->err = reftable_backend_init(&refs->worktree_backend, refdir.buf,
 						  &refs->write_options);
 		if (refs->err)
 			goto done;
@@ -456,6 +458,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 
 done:
 	assert(refs->err != REFTABLE_API_ERROR);
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
 	strbuf_release(&path);
 	return &refs->base;
 }

-- 
2.52.0

Re: [PATCH v5 2/4] refs: forward and use the reference storage payload

[Patrick Steinhardt]

On Mon, Feb 09, 2026 at 04:58:19PM +0100, Karthik Nayak wrote:
> An upcoming commit will add support for providing an URI via the
> 'extensions.refStorage' config. The URI will contain the reference
> backend and a corresponding payload. The payload can be then used for
> providing an alternate locations for the reference backend.
> 
> To prepare for this, modify the existing backends to accept such an
> argument when initializing via the 'init()' function. Both the files
> and reftable backends will parse the information to be filesystem paths
> to store references.

Maybe add: "to store references. Given that no callers pass any payload
yet this is essentially a no-op change for now."

> diff --git a/refs.c b/refs.c
> index 36f3441632..d9df25d7c0 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -3425,3 +3426,33 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>  
>  	strbuf_release(&path);
>  }
> +
> +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
> +				      bool *is_worktree, struct strbuf *refdir,
> +				      struct strbuf *ref_common_dir)
> +{
> +	struct strbuf sb = STRBUF_INIT;
> +
> +	strbuf_addstr(refdir, gitdir);
> +	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
> +
> +	if (!payload)
> +		return;

I think you should add a comment here that explains why it's not
necessary to modify the `refdir` in case `*is_worktree`. I'd arguably
even move that code into `if (!payload)`, as we otherwise only set it to
reset it later. So:

        if (!payload) {
                /*
                 * We can use `gitdir` as `refdir` without appending the
                 * worktree path because...
                 /
                 strbuf_addstr(refdir, gitdir);
        }

> +	if (!is_absolute_path(payload)) {
> +		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
> +		strbuf_realpath(ref_common_dir, sb.buf, 1);
> +	} else {
> +		strbuf_realpath(ref_common_dir, payload, 1);
> +	}
> +
> +	strbuf_reset(refdir);

And then you can drop this call to `strbuf_reset()`.

> diff --git a/refs/files-backend.c b/refs/files-backend.c
> index 240d3c3b26..b192ce606d 100644
> --- a/refs/files-backend.c
> +++ b/refs/files-backend.c
> @@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
>   * set of caches.
>   */
>  static struct ref_store *files_ref_store_init(struct repository *repo,
> +					      const char *payload,
>  					      const char *gitdir,
>  					      unsigned int flags)
>  {
>  	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
>  	struct ref_store *ref_store = (struct ref_store *)refs;
> -	struct strbuf sb = STRBUF_INIT;
> +	struct strbuf ref_common_dir = STRBUF_INIT;
> +	struct strbuf refdir = STRBUF_INIT;
> +	bool is_worktree;
> +
> +	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
> +					 &ref_common_dir);
>  
> -	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
> +	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
>  	refs->store_flags = flags;
> -	get_common_dir_noenv(&sb, gitdir);
> -	refs->gitcommondir = strbuf_detach(&sb, NULL);
> +	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
>  	refs->packed_ref_store =
> -		packed_ref_store_init(repo, refs->gitcommondir, flags);
> +		packed_ref_store_init(repo, payload, refs->gitcommondir, flags);

It's a bit weird that we end up passing the payload even though we
unconditionally ignore it in `packed_ref_store_init()`. I'd argue that
we should either pass a `NULL` pointer as payload, or let the packed
backend call `refs_compute_filesystem_location()` itsefl.

> diff --git a/refs/packed-backend.c b/refs/packed-backend.c
> index 4ea0c12299..028fbc0585 100644
> --- a/refs/packed-backend.c
> +++ b/refs/packed-backend.c
> @@ -212,6 +212,7 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
>  }
>  
>  struct ref_store *packed_ref_store_init(struct repository *repo,
> +					const char *payload UNUSED,
>  					const char *gitdir,
>  					unsigned int store_flags)
>  {

And here we should probably explain why we don't have to respect the
payload.

> diff --git a/refs/refs-internal.h b/refs/refs-internal.h
> index c7d2a6e50b..bd09b1280c 100644
> --- a/refs/refs-internal.h
> +++ b/refs/refs-internal.h
> @@ -666,4 +667,18 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
>  					  unsigned int initial_transaction,
>  					  struct strbuf *err);
>  
> +/*
> + * Given a gitdir and the reference storage payload provided, retrieve the
> + * 'refdir' and 'ref_common_dir'. The former is where references should be
> + * stored for the current worktree, the latter is the common reference
> + * directory if working with a linked worktree. If working with the main
> + * worktree, both values will be the same.
> + *
> + * This is used by backends such as {files, reftable} which store references in
> + * dedicated filesystem paths.
> + */

I guess we can say "This is used by backends that store store files in
the repository directly."

Patrick

Re: [PATCH v5 2/4] refs: forward and use the reference storage payload

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 5629 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 09, 2026 at 04:58:19PM +0100, Karthik Nayak wrote:
>> An upcoming commit will add support for providing an URI via the
>> 'extensions.refStorage' config. The URI will contain the reference
>> backend and a corresponding payload. The payload can be then used for
>> providing an alternate locations for the reference backend.
>>
>> To prepare for this, modify the existing backends to accept such an
>> argument when initializing via the 'init()' function. Both the files
>> and reftable backends will parse the information to be filesystem paths
>> to store references.
>
> Maybe add: "to store references. Given that no callers pass any payload
> yet this is essentially a no-op change for now."
>

Will add.

>> diff --git a/refs.c b/refs.c
>> index 36f3441632..d9df25d7c0 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -3425,3 +3426,33 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>>
>>  	strbuf_release(&path);
>>  }
>> +
>> +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
>> +				      bool *is_worktree, struct strbuf *refdir,
>> +				      struct strbuf *ref_common_dir)
>> +{
>> +	struct strbuf sb = STRBUF_INIT;
>> +
>> +	strbuf_addstr(refdir, gitdir);
>> +	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
>> +
>> +	if (!payload)
>> +		return;
>
> I think you should add a comment here that explains why it's not
> necessary to modify the `refdir` in case `*is_worktree`. I'd arguably
> even move that code into `if (!payload)`, as we otherwise only set it to
> reset it later. So:
>
>         if (!payload) {
>                 /*
>                  * We can use `gitdir` as `refdir` without appending the
>                  * worktree path because...
>                  /
>                  strbuf_addstr(refdir, gitdir);
>         }
>
>> +	if (!is_absolute_path(payload)) {
>> +		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
>> +		strbuf_realpath(ref_common_dir, sb.buf, 1);
>> +	} else {
>> +		strbuf_realpath(ref_common_dir, payload, 1);
>> +	}
>> +
>> +	strbuf_reset(refdir);
>
> And then you can drop this call to `strbuf_reset()`.
>

Fair enough, will do this.

>> diff --git a/refs/files-backend.c b/refs/files-backend.c
>> index 240d3c3b26..b192ce606d 100644
>> --- a/refs/files-backend.c
>> +++ b/refs/files-backend.c
>> @@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
>>   * set of caches.
>>   */
>>  static struct ref_store *files_ref_store_init(struct repository *repo,
>> +					      const char *payload,
>>  					      const char *gitdir,
>>  					      unsigned int flags)
>>  {
>>  	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
>>  	struct ref_store *ref_store = (struct ref_store *)refs;
>> -	struct strbuf sb = STRBUF_INIT;
>> +	struct strbuf ref_common_dir = STRBUF_INIT;
>> +	struct strbuf refdir = STRBUF_INIT;
>> +	bool is_worktree;
>> +
>> +	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
>> +					 &ref_common_dir);
>>
>> -	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
>> +	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
>>  	refs->store_flags = flags;
>> -	get_common_dir_noenv(&sb, gitdir);
>> -	refs->gitcommondir = strbuf_detach(&sb, NULL);
>> +	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
>>  	refs->packed_ref_store =
>> -		packed_ref_store_init(repo, refs->gitcommondir, flags);
>> +		packed_ref_store_init(repo, payload, refs->gitcommondir, flags);
>
> It's a bit weird that we end up passing the payload even though we
> unconditionally ignore it in `packed_ref_store_init()`. I'd argue that
> we should either pass a `NULL` pointer as payload, or let the packed
> backend call `refs_compute_filesystem_location()` itsefl.
>

I'm considering passing in a NULL and relying on the 'gitdir'. Mostly
because the packed-refs backend always comes linked to the files-backend
and as such should simply rely on the 'gitdir' provided by it. Let me
know if you think we should go the other way.

>> diff --git a/refs/packed-backend.c b/refs/packed-backend.c
>> index 4ea0c12299..028fbc0585 100644
>> --- a/refs/packed-backend.c
>> +++ b/refs/packed-backend.c
>> @@ -212,6 +212,7 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
>>  }
>>
>>  struct ref_store *packed_ref_store_init(struct repository *repo,
>> +					const char *payload UNUSED,
>>  					const char *gitdir,
>>  					unsigned int store_flags)
>>  {
>
> And here we should probably explain why we don't have to respect the
> payload.
>

Yeah, will add in a comment.

>> diff --git a/refs/refs-internal.h b/refs/refs-internal.h
>> index c7d2a6e50b..bd09b1280c 100644
>> --- a/refs/refs-internal.h
>> +++ b/refs/refs-internal.h
>> @@ -666,4 +667,18 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
>>  					  unsigned int initial_transaction,
>>  					  struct strbuf *err);
>>
>> +/*
>> + * Given a gitdir and the reference storage payload provided, retrieve the
>> + * 'refdir' and 'ref_common_dir'. The former is where references should be
>> + * stored for the current worktree, the latter is the common reference
>> + * directory if working with a linked worktree. If working with the main
>> + * worktree, both values will be the same.
>> + *
>> + * This is used by backends such as {files, reftable} which store references in
>> + * dedicated filesystem paths.
>> + */
>
> I guess we can say "This is used by backends that store store files in
> the repository directly."
>
> Patrick

Makes sense. Thanks.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v5 2/4] refs: forward and use the reference storage payload

[Jeff King]

On Mon, Feb 09, 2026 at 04:58:19PM +0100, Karthik Nayak wrote:

> +	if (*is_worktree) {
> +		char *wt_id = strrchr(gitdir, '/') + 1;
> +		strbuf_addf(refdir, "/worktrees/%s", wt_id);
> +	}

This is another instance where the new version of glibc will complain
about const-ness. The wt_id variable can just be marked as const.

(This and the other one I pointed out are the only two cases in your
series).

BTW, is it possible to have a gitdir string without a slash in it? I
guess not if it is absolute, but I didn't look at the caller to see if
that is always true. Anyway, I wonder if we should be checking the
result of strrchr() here.

-Peff

Re: [PATCH v5 2/4] refs: forward and use the reference storage payload

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

Jeff King <peff@peff.net> writes:

> On Mon, Feb 09, 2026 at 04:58:19PM +0100, Karthik Nayak wrote:
>
>> +	if (*is_worktree) {
>> +		char *wt_id = strrchr(gitdir, '/') + 1;
>> +		strbuf_addf(refdir, "/worktrees/%s", wt_id);
>> +	}
>
> This is another instance where the new version of glibc will complain
> about const-ness. The wt_id variable can just be marked as const.
>

Thanks, will amend.

> (This and the other one I pointed out are the only two cases in your
> series).
>
> BTW, is it possible to have a gitdir string without a slash in it? I
> guess not if it is absolute, but I didn't look at the caller to see if
> that is always true. Anyway, I wonder if we should be checking the
> result of strrchr() here.
>
> -Peff

Since this is called during the ref store initialization and that value
is generally set via:

1. `repo->gitdir`: Which is resolved to an absolute path
2. `repo->commondir`: Which is user input, can be without slash
3. `worktree dir`: Should contain a slash
4. `submodule dir`: Should contain a slash
5. `refs migration dir`: Should contain a slash

But even with this, Gits understanding of linked worktrees are that they
are in a `$gitdir/worktrees/$wt_id`. So it should be safe to do this.
Nevertheless, I'll add in a check.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v5 2/4] refs: forward and use the reference storage payload

[Jeff King]

On Fri, Feb 13, 2026 at 06:45:20AM -0800, Karthik Nayak wrote:

> > BTW, is it possible to have a gitdir string without a slash in it? I
> > guess not if it is absolute, but I didn't look at the caller to see if
> > that is always true. Anyway, I wonder if we should be checking the
> > result of strrchr() here.
> 
> Since this is called during the ref store initialization and that value
> is generally set via:
> 
> 1. `repo->gitdir`: Which is resolved to an absolute path
> 2. `repo->commondir`: Which is user input, can be without slash
> 3. `worktree dir`: Should contain a slash
> 4. `submodule dir`: Should contain a slash
> 5. `refs migration dir`: Should contain a slash
> 
> But even with this, Gits understanding of linked worktrees are that they
> are in a `$gitdir/worktrees/$wt_id`. So it should be safe to do this.
> Nevertheless, I'll add in a check.

Thanks. I figured it was something like that. Even a comment or a BUG()
would be more than enough, I'd think.

-Peff

[PATCH v5 3/4] refs: allow reference location in refstorage config

[Karthik Nayak]

The 'extensions.refStorage' config is used to specify the reference
backend for a given repository. Both the 'files' and 'reftable' backends
utilize the $GIT_DIR as the reference folder by default in
`get_main_ref_store()`.

Since the reference backends are pluggable, this means that they could
work with out-of-tree reference directories too. Extend the 'refStorage'
config to also support taking an URI input, where users can specify the
reference backend and the location.

Add the required changes to obtain and propagate this value to the
individual backends also add the necessary documentation and tests.

Traditionally, for linked worktrees, references were stored in the
'$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
storage path, it doesn't make sense to store the main worktree
references in the new path, and the linked worktree references in the
$GIT_DIR. So, let's store linked worktree references in
'$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'. To do this, create the
necessary files and folders while also adding stubs in the $GIT_DIR path
to ensure that it is still considered a Git directory.

Ideally, we would want to pass in a `struct worktree *` to individual
backends, instead of passing the `gitdir`. This allows them to handle
worktree specific logic. Currently, that is not possible since the
worktree code is:

  - Tied to using the global `the_repository` variable.

  - Is not setup before the reference database during initialization of
    the repository.

Add a TODO in 'refs.c' to ensure we can eventually make that change.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/config/extensions.adoc |  16 +++-
 builtin/clone.c                      |   3 +-
 builtin/worktree.c                   |  34 ++++++++
 refs.c                               |   6 +-
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  39 +++++++--
 setup.h                              |   2 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
 10 files changed, 266 insertions(+), 11 deletions(-)

diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
index 532456644b..3e51da36d3 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
 `core.repositoryFormatVersion` setting.
 
 refStorage:::
-	Specify the ref storage format to use. The acceptable values are:
+	Specify the ref storage format and a corresponding payload. The value
+	can be either a format name or a URI:
 +
 --
+* A format name alone (e.g., `reftable` or `files`).
+
+* A URI format `<format>://<payload>` explicitly specifies both the
+  format and payload (e.g., `reftable:///foo/bar`).
+
+Supported format names are:
++
 include::../ref-storage-format.adoc[]
++
+The payload is passed directly to the reference backend. For the files and
+reftable backends, this must be a filesystem path where the references will
+be stored. Defaulting to the commondir when no payload is provided. Relative
+paths are resolved relative to the $GIT_DIR. Future backends may support
+other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +
 Note that this setting should only be set by linkgit:git-init[1] or
diff --git a/builtin/clone.c b/builtin/clone.c
index 535a257b10..11854894d3 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1437,7 +1437,8 @@ int cmd_clone(int argc,
 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
 	repo_set_hash_algo(the_repository, hash_algo);
-	create_reference_database(the_repository->ref_storage_format, NULL, 1);
+	create_reference_database(the_repository->ref_storage_format,
+				  the_repository->ref_storage_payload, NULL, 1);
 
 	/*
 	 * Before fetching from the remote, download and install bundle
diff --git a/builtin/worktree.c b/builtin/worktree.c
index fbdaf2eb2e..94480be5c4 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
 	return run_command(&cp);
 }
 
+/*
+ * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
+ * But when using alternate reference directories, we want to store the worktree
+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
+ *
+ * Create the necessary folder structure to facilitate the same. But to ensure
+ * that the former path is still considered a Git directory, add stubs.
+ */
+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	char *path;
+
+	path = wt->repo->ref_storage_payload;
+	if (!path)
+		return;
+
+	if (!is_absolute_path(path))
+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
+
+	strbuf_addf(&sb, "%s/worktrees", path);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_addf(&sb, "/%s", wt->id);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_reset(&sb);
+
+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
+		   path, wt->id);
+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
+
+	strbuf_release(&sb);
+}
+
 static int add_worktree(const char *path, const char *refname,
 			const struct add_opts *opts)
 {
@@ -518,6 +551,7 @@ static int add_worktree(const char *path, const char *refname,
 		ret = error(_("could not find created worktree '%s'"), name);
 		goto done;
 	}
+	setup_alternate_ref_dir(wt, sb_repo.buf);
 	wt_refs = get_worktree_ref_store(wt);
 
 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
diff --git a/refs.c b/refs.c
index d9df25d7c0..f520d64a9d 100644
--- a/refs.c
+++ b/refs.c
@@ -2225,7 +2225,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, NULL, gitdir, flags);
+	/*
+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
+	 * allow the backend to handle how it wants to deal with worktrees.
+	 */
+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
 	return refs;
 }
 
diff --git a/repository.c b/repository.c
index c7e75215ac..9815f081ef 100644
--- a/repository.c
+++ b/repository.c
@@ -193,9 +193,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
 }
 
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format)
+				 enum ref_storage_format format,
+				 const char *payload)
 {
 	repo->ref_storage_format = format;
+	free(repo->ref_storage_payload);
+	repo->ref_storage_payload = xstrdup_or_null(payload);
 }
 
 /*
@@ -277,7 +280,8 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 	repo_set_compat_hash_algo(repo, format.compat_hash_algo);
-	repo_set_ref_storage_format(repo, format.ref_storage_format);
+	repo_set_ref_storage_format(repo, format.ref_storage_format,
+				    format.ref_storage_payload);
 	repo->repository_format_worktree_config = format.worktree_config;
 	repo->repository_format_relative_worktrees = format.relative_worktrees;
 	repo->repository_format_precious_objects = format.precious_objects;
@@ -369,6 +373,7 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
+	FREE_AND_NULL(repo->ref_storage_payload);
 
 	odb_free(repo->objects);
 	repo->objects = NULL;
diff --git a/repository.h b/repository.h
index 6063c4b846..95e2333bad 100644
--- a/repository.h
+++ b/repository.h
@@ -150,6 +150,11 @@ struct repository {
 
 	/* Repository's reference storage format, as serialized on disk. */
 	enum ref_storage_format ref_storage_format;
+	/*
+	 * Reference storage information as needed for the backend. This contains
+	 * only the payload from the reference URI without the schema.
+	 */
+	char *ref_storage_payload;
 
 	/* A unique-id for tracing purposes. */
 	int trace2_repo_id;
@@ -204,7 +209,8 @@ void repo_set_worktree(struct repository *repo, const char *path);
 void repo_set_hash_algo(struct repository *repo, int algo);
 void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format);
+				 enum ref_storage_format format,
+				 const char *payload);
 void initialize_repository(struct repository *repo);
 RESULT_MUST_BE_USED
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index b723f8b339..44e393c251 100644
--- a/setup.c
+++ b/setup.c
@@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
 		return EXTENSION_UNKNOWN;
 }
 
+static void parse_reference_uri(const char *value, char **format,
+				char **payload)
+{
+	char *schema_end;
+
+	schema_end = strstr(value, "://");
+	if (!schema_end) {
+		*format = xstrdup(value);
+		*payload = NULL;
+	} else {
+		*format = xstrndup(value, schema_end - value);
+		*payload = xstrdup_or_null(schema_end + 3);
+	}
+}
+
 /*
  * Record any new extensions in this function.
  */
@@ -674,10 +689,17 @@ static enum extension_result handle_extension(const char *var,
 		return EXTENSION_OK;
 	} else if (!strcmp(ext, "refstorage")) {
 		unsigned int format;
+		char *format_str;
 
 		if (!value)
 			return config_error_nonbool(var);
-		format = ref_storage_format_by_name(value);
+
+		parse_reference_uri(value, &format_str,
+				    &data->ref_storage_payload);
+
+		format = ref_storage_format_by_name(format_str);
+		free(format_str);
+
 		if (format == REF_STORAGE_FORMAT_UNKNOWN)
 			return error(_("invalid value for '%s': '%s'"),
 				     "extensions.refstorage", value);
@@ -850,6 +872,7 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
 	free(format->partial_clone);
+	free(format->ref_storage_payload);
 	init_repository_format(format);
 }
 
@@ -1942,7 +1965,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			repo_set_compat_hash_algo(the_repository,
 						  repo_fmt.compat_hash_algo);
 			repo_set_ref_storage_format(the_repository,
-						    repo_fmt.ref_storage_format);
+						    repo_fmt.ref_storage_format,
+						    repo_fmt.ref_storage_payload);
 			the_repository->repository_format_worktree_config =
 				repo_fmt.worktree_config;
 			the_repository->repository_format_relative_worktrees =
@@ -2042,7 +2066,8 @@ void check_repository_format(struct repository_format *fmt)
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
 	repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo);
 	repo_set_ref_storage_format(the_repository,
-				    fmt->ref_storage_format);
+				    fmt->ref_storage_format,
+				    fmt->ref_storage_payload);
 	the_repository->repository_format_worktree_config =
 		fmt->worktree_config;
 	the_repository->repository_format_relative_worktrees =
@@ -2360,13 +2385,15 @@ static int is_reinit(void)
 }
 
 void create_reference_database(enum ref_storage_format ref_storage_format,
+			       const char *ref_storage_payload,
 			       const char *initial_branch, int quiet)
 {
 	struct strbuf err = STRBUF_INIT;
 	char *to_free = NULL;
 	int reinit = is_reinit();
 
-	repo_set_ref_storage_format(the_repository, ref_storage_format);
+	repo_set_ref_storage_format(the_repository, ref_storage_format,
+				    ref_storage_payload);
 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -2645,7 +2672,8 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
-	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format);
+	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
+				    repo_fmt->ref_storage_payload);
 }
 
 int init_db(const char *git_dir, const char *real_git_dir,
@@ -2702,6 +2730,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo_fmt.ref_storage_format,
+					  repo_fmt.ref_storage_payload,
 					  initial_branch, flags & INIT_DB_QUIET);
 	create_object_directory();
 
diff --git a/setup.h b/setup.h
index d55dcc6608..4d25a353ac 100644
--- a/setup.h
+++ b/setup.h
@@ -171,6 +171,7 @@ struct repository_format {
 	int hash_algo;
 	int compat_hash_algo;
 	enum ref_storage_format ref_storage_format;
+	char *ref_storage_payload;
 	int sparse_index;
 	char *work_tree;
 	struct string_list unknown_extensions;
@@ -241,6 +242,7 @@ void initialize_repository_version(int hash_algo,
 				   enum ref_storage_format ref_storage_format,
 				   int reinit);
 void create_reference_database(enum ref_storage_format ref_storage_format,
+			       const char *ref_storage_payload,
 			       const char *initial_branch, int quiet);
 
 /*
diff --git a/t/meson.build b/t/meson.build
index 459c52a489..11fc5a49ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -210,6 +210,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..9c777b79f3
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='Test reference backend URIs'
+
+. ./test-lib.sh
+
+# Run a git command with the provided reference storage. Reset the backend
+# post running the command.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <cmd> is the git subcommand to be run in the repository.
+run_with_uri() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	cmd=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1
+	git -C "$repo" config set extensions.refStorage "$uri" &&
+	git -C "$repo" $cmd &&
+	git -C "$repo" config set extensions.refStorage "$backend"
+}
+
+# Test a repository with a given reference storage by running and comparing
+# 'git refs list' before and after setting the new reference backend. If
+# err_msg is set, expect the command to fail and grep for the provided err_msg.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
+test_refs_backend() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	err_msg=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test -n "$err_msg";
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		test_must_fail git -C "$repo" refs list 2>err &&
+		test_grep "$err_msg" err
+	else
+		git -C "$repo" refs list >expect &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		test_cmp expect actual
+	fi
+}
+
+test_expect_success 'URI is invalid' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable@/home/reftable" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'URI ends with colon' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable:" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "db://.git" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+
+for to_format in $ref_formats
+do
+	if test "$from_format" = "$to_format"
+	then
+		continue
+	fi
+
+
+	for dir in "$(pwd)/repo/.git" "./"
+	do
+
+		test_expect_success "$read from $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
+			)
+		'
+
+		test_expect_success "$write to $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+
+				git refs list >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				git refs list | grep -v "refs/tags/1" >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "with worktree and $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo wt" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				git config set core.repositoryformatversion 1 &&
+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+
+				git worktree add ../wt 2
+			) &&
+
+			git -C repo for-each-ref --include-root-refs >expect &&
+			git -C wt for-each-ref --include-root-refs >expect &&
+			! test_cmp expect actual &&
+
+			git -C wt rev-parse 2 >expect &&
+			git -C wt rev-parse HEAD >actual &&
+			test_cmp expect actual
+		'
+	done # closes dir
+done # closes to_format
+done # closes from_format
+
+test_done

-- 
2.52.0

Re: [PATCH v5 3/4] refs: allow reference location in refstorage config

[Patrick Steinhardt]

On Mon, Feb 09, 2026 at 04:58:20PM +0100, Karthik Nayak wrote:
> The 'extensions.refStorage' config is used to specify the reference
> backend for a given repository. Both the 'files' and 'reftable' backends
> utilize the $GIT_DIR as the reference folder by default in
> `get_main_ref_store()`.
> 
> Since the reference backends are pluggable, this means that they could
> work with out-of-tree reference directories too. Extend the 'refStorage'
> config to also support taking an URI input, where users can specify the
> reference backend and the location.
> 
> Add the required changes to obtain and propagate this value to the
> individual backends also add the necessary documentation and tests.

This reads as if this should have been two sentences.

> diff --git a/builtin/worktree.c b/builtin/worktree.c
> index fbdaf2eb2e..94480be5c4 100644
> --- a/builtin/worktree.c
> +++ b/builtin/worktree.c
> @@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
>  	return run_command(&cp);
>  }
>  
> +/*
> + * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.

s/worktress/worktrees/

> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
> new file mode 100755
> index 0000000000..9c777b79f3
> --- /dev/null
> +++ b/t/t1423-ref-backend.sh
> @@ -0,0 +1,159 @@
> +#!/bin/sh
> +
> +test_description='Test reference backend URIs'
> +
> +. ./test-lib.sh
> +
> +# Run a git command with the provided reference storage. Reset the backend
> +# post running the command.
> +# Usage: run_with_uri <repo> <backend> <uri> <cmd>
> +#   <repo> is the relative path to the repo to run the command in.
> +#   <backend> is the original ref storage of the repo.
> +#   <uri> is the new URI to be set for the ref storage.
> +#   <cmd> is the git subcommand to be run in the repository.
> +run_with_uri() {
> +	repo=$1 &&
> +	backend=$2 &&
> +	uri=$3 &&
> +	cmd=$4 &&
> +
> +	git -C "$repo" config set core.repositoryformatversion 1
> +	git -C "$repo" config set extensions.refStorage "$uri" &&
> +	git -C "$repo" $cmd &&
> +	git -C "$repo" config set extensions.refStorage "$backend"
> +}
> +
> +# Test a repository with a given reference storage by running and comparing
> +# 'git refs list' before and after setting the new reference backend. If
> +# err_msg is set, expect the command to fail and grep for the provided err_msg.
> +# Usage: run_with_uri <repo> <backend> <uri> <cmd>
> +#   <repo> is the relative path to the repo to run the command in.
> +#   <backend> is the original ref storage of the repo.
> +#   <uri> is the new URI to be set for the ref storage.
> +#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
> +test_refs_backend() {
> +	repo=$1 &&
> +	backend=$2 &&
> +	uri=$3 &&
> +	err_msg=$4 &&
> +
> +	git -C "$repo" config set core.repositoryformatversion 1 &&
> +	if test -n "$err_msg";
> +	then
> +		git -C "$repo" config set extensions.refStorage "$uri" &&
> +		test_must_fail git -C "$repo" refs list 2>err &&
> +		test_grep "$err_msg" err
> +	else
> +		git -C "$repo" refs list >expect &&
> +		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
> +		test_cmp expect actual
> +	fi
> +}
> +
> +test_expect_success 'URI is invalid' '
> +	test_when_finished "rm -rf repo" &&
> +	git init repo &&
> +	test_refs_backend repo files "reftable@/home/reftable" \
> +		"invalid value for ${SQ}extensions.refstorage${SQ}"
> +'
> +
> +test_expect_success 'URI ends with colon' '
> +	test_when_finished "rm -rf repo" &&
> +	git init repo &&
> +	test_refs_backend repo files "reftable:" \
> +		"invalid value for ${SQ}extensions.refstorage${SQ}"
> +'
> +
> +test_expect_success 'unknown reference backend' '
> +	test_when_finished "rm -rf repo" &&
> +	git init repo &&
> +	test_refs_backend repo files "db://.git" \
> +		"invalid value for ${SQ}extensions.refstorage${SQ}"
> +'
> +
> +ref_formats="files reftable"
> +for from_format in $ref_formats
> +do
> +
> +for to_format in $ref_formats
> +do
> +	if test "$from_format" = "$to_format"
> +	then
> +		continue
> +	fi
> +
> +
> +	for dir in "$(pwd)/repo/.git" "./"

As "./" is a relative directory I expect it to be resolved relative to
"$GIT_DIR", right? Also, I don't see any tests that create the ref
directory outside of the repository. Should we maybe add one?

> +	do
> +
> +		test_expect_success "$read from $to_format backend, $dir dir" '
> +			test_when_finished "rm -rf repo" &&
> +			git init --ref-format=$from_format repo &&
> +			(
> +				cd repo &&
> +				test_commit 1 &&
> +				test_commit 2 &&
> +				test_commit 3 &&
> +
> +				git refs migrate --dry-run --ref-format=$to_format >out &&

Okay, we do the migration, but with "--dry-run". This should result in a
temporary staging directory, which is in fact somewhat interesting given
that "to_format" can now contain a payload. I assume it wouldn't have an
impact if such a payload was set here?

> +				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&

Hm. I have no idea what this is doing :)

> +				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
> +			)
> +		'

In general I think it would be sensible to also have a couple tests here
that exercise specific formats directly. Like:

  - Are the files created in the right spots for the files and reftable
    backend with a specific backend?

  - Does `git refs migrate` know to write the files into the correct
    location in case "--dry-run" wasn't passed?

  - Does git-init(1) and git-clone(1) initialize the refstore in a
    different location as expected?

  - Does creating a worktree work?

Patrick

Re: [PATCH v5 3/4] refs: allow reference location in refstorage config

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3927 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 09, 2026 at 04:58:20PM +0100, Karthik Nayak wrote:
>> The 'extensions.refStorage' config is used to specify the reference
>> backend for a given repository. Both the 'files' and 'reftable' backends
>> utilize the $GIT_DIR as the reference folder by default in
>> `get_main_ref_store()`.
>>
>> Since the reference backends are pluggable, this means that they could
>> work with out-of-tree reference directories too. Extend the 'refStorage'
>> config to also support taking an URI input, where users can specify the
>> reference backend and the location.
>>
>> Add the required changes to obtain and propagate this value to the
>> individual backends also add the necessary documentation and tests.
>
> This reads as if this should have been two sentences.

Yeah, let me split it up.

>
>> diff --git a/builtin/worktree.c b/builtin/worktree.c
>> index fbdaf2eb2e..94480be5c4 100644
>> --- a/builtin/worktree.c
>> +++ b/builtin/worktree.c
>> @@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
>>  	return run_command(&cp);
>>  }
>>
>> +/*
>> + * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
>
> s/worktress/worktrees/
>

Oops.


>> +for to_format in $ref_formats
>> +do
>> +	if test "$from_format" = "$to_format"
>> +	then
>> +		continue
>> +	fi
>> +
>> +
>> +	for dir in "$(pwd)/repo/.git" "./"
>
> As "./" is a relative directory I expect it to be resolved relative to
> "$GIT_DIR", right?

Yup the './' is relative to the '$GIT_DIR'. It can be changed to '.', so
I'll do that and add a comment. So we do both

  BACKEND_PATH='/home/karthik/code/git/build/test-output/trash
directory.t1423-ref-backend/repo/.git/ref_migration.ZnKNpg'

and

  BACKEND_PATH=./ref_migration.BFyK5k

> Also, I don't see any tests that create the ref
> directory outside of the repository. Should we maybe add one?

Yeah, let me add that test.

>> +	do
>> +
>> +		test_expect_success "$read from $to_format backend, $dir dir" '
>> +			test_when_finished "rm -rf repo" &&
>> +			git init --ref-format=$from_format repo &&
>> +			(
>> +				cd repo &&
>> +				test_commit 1 &&
>> +				test_commit 2 &&
>> +				test_commit 3 &&
>> +
>> +				git refs migrate --dry-run --ref-format=$to_format >out &&
>
> Okay, we do the migration, but with "--dry-run". This should result in a
> temporary staging directory, which is in fact somewhat interesting given
> that "to_format" can now contain a payload. I assume it wouldn't have an
> impact if such a payload was set here?
>

It cannot, the 'git refs migrate' command parses the argument provided
to '--ref-format' as a reference-backend.

>> +				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
>
> Hm. I have no idea what this is doing :)
>

It extracts out the name of the dry-run folder created by 'git refs
migrate'. We append the $dir to it, which could be the absolute path or
the relative path.

>> +				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
>> +			)
>> +		'
>
> In general I think it would be sensible to also have a couple tests here
> that exercise specific formats directly. Like:
>
>   - Are the files created in the right spots for the files and reftable
>     backend with a specific backend?
>
>   - Does `git refs migrate` know to write the files into the correct
>     location in case "--dry-run" wasn't passed?
>

Will add tests for the following too.

>   - Does git-init(1) and git-clone(1) initialize the refstore in a
>     different location as expected?
>

Well so this only works when there is a reference store already
existing. I thought I had explicitly called this out, but doesn't seem
to be the case. I think it would actually make sense that it also works
with creation. So let me look into that.

>   - Does creating a worktree work?
>

This is currently already tested.

> Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v5 3/4] refs: allow reference location in refstorage config

[Jeff King]

On Mon, Feb 09, 2026 at 04:58:20PM +0100, Karthik Nayak wrote:

> +static void parse_reference_uri(const char *value, char **format,
> +				char **payload)
> +{
> +	char *schema_end;
> +
> +	schema_end = strstr(value, "://");
> +	if (!schema_end) {
> +		*format = xstrdup(value);
> +		*payload = NULL;
> +	} else {
> +		*format = xstrndup(value, schema_end - value);
> +		*payload = xstrdup_or_null(schema_end + 3);
> +	}
> +}

The schema_end variable should be "const" here. Otherwise new versions
of gcc/glibc will complain that the strstr() implicitly removes the
const from value (and compilation with DEVELOPER=1 fails). More details
in this thread:

  https://lore.kernel.org/git/e6f7e2eddbc9aef1c21f661420a4b8cb9cd8e2c1.1770095829.git.collin.funk1@gmail.com/

-Peff

Re: [PATCH v5 3/4] refs: allow reference location in refstorage config

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 965 bytes --]

Jeff King <peff@peff.net> writes:

> On Mon, Feb 09, 2026 at 04:58:20PM +0100, Karthik Nayak wrote:
>
>> +static void parse_reference_uri(const char *value, char **format,
>> +				char **payload)
>> +{
>> +	char *schema_end;
>> +
>> +	schema_end = strstr(value, "://");
>> +	if (!schema_end) {
>> +		*format = xstrdup(value);
>> +		*payload = NULL;
>> +	} else {
>> +		*format = xstrndup(value, schema_end - value);
>> +		*payload = xstrdup_or_null(schema_end + 3);
>> +	}
>> +}
>
> The schema_end variable should be "const" here. Otherwise new versions
> of gcc/glibc will complain that the strstr() implicitly removes the
> const from value (and compilation with DEVELOPER=1 fails). More details
> in this thread:
>
>   https://lore.kernel.org/git/e6f7e2eddbc9aef1c21f661420a4b8cb9cd8e2c1.1770095829.git.collin.funk1@gmail.com/
>
> -Peff

Thanks for the hint, my Linux distribution is still on glibc 2.42 and
didn't see the error. Will fix it in my next version.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v5 4/4] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
the previous commit we extended the 'extensions.refStorage' config to
also support an URI input for reference backend with location.

Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
takes in the same input as the config variable. Having an environment
variable allows us to modify the reference backend and location on the
fly for individual Git commands.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |  5 +++
 environment.h          |  1 +
 setup.c                | 20 +++++++++++
 t/t1423-ref-backend.sh | 96 ++++++++++++++++++++++++++++++++------------------
 4 files changed, 88 insertions(+), 34 deletions(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..66442735ea 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REFERENCE_BACKEND`::
+    Specify which reference backend to be used along with its URI.
+    See `extensions.refStorage` option in linkgit:git-config[1] for more
+    details. Overrides the config variable when used.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 27f657af04..540e0a7f6d 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/setup.c b/setup.c
index 44e393c251..b4a7b82cca 100644
--- a/setup.c
+++ b/setup.c
@@ -1838,6 +1838,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 	static struct strbuf cwd = STRBUF_INIT;
 	struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT;
 	const char *prefix = NULL;
+	const char *ref_backend_uri;
 	struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
 
 	/*
@@ -1995,6 +1996,25 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
+	/*
+	 * The env variable should override the repository config
+	 * for 'extensions.refStorage'.
+	 */
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *location;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &location);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+		repo_set_ref_storage_format(the_repository, format, location);
+
+		free(backend);
+		free(location);
+	}
+
 	setup_original_cwd();
 
 	strbuf_release(&dir);
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
index 9c777b79f3..10a9bb1a9b 100755
--- a/t/t1423-ref-backend.sh
+++ b/t/t1423-ref-backend.sh
@@ -11,16 +11,25 @@ test_description='Test reference backend URIs'
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
 #   <cmd> is the git subcommand to be run in the repository.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 run_with_uri() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
 	cmd=$4 &&
+	via=$5 &&
 
-	git -C "$repo" config set core.repositoryformatversion 1
-	git -C "$repo" config set extensions.refStorage "$uri" &&
-	git -C "$repo" $cmd &&
-	git -C "$repo" config set extensions.refStorage "$backend"
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test "$via" = "env"
+	then
+		test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd
+	elif test "$via" = "config"
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		git -C "$repo" $cmd &&
+		git -C "$repo" config set extensions.refStorage "$backend"
+	fi
 }
 
 # Test a repository with a given reference storage by running and comparing
@@ -30,44 +39,57 @@ run_with_uri() {
 #   <repo> is the relative path to the repo to run the command in.
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
 test_refs_backend() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
-	err_msg=$4 &&
+	via=$4 &&
+	err_msg=$5 &&
+
 
-	git -C "$repo" config set core.repositoryformatversion 1 &&
 	if test -n "$err_msg";
 	then
-		git -C "$repo" config set extensions.refStorage "$uri" &&
-		test_must_fail git -C "$repo" refs list 2>err &&
-		test_grep "$err_msg" err
+		if test "$via" = "env"
+		then
+			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
+		elif test "$via" = "config"
+		then
+			git -C "$repo" config set extensions.refStorage "$uri" &&
+			test_must_fail git -C "$repo" refs list 2>err &&
+			test_grep "$err_msg" err
+		fi
 	else
 		git -C "$repo" refs list >expect &&
-		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
 		test_cmp expect actual
 	fi
 }
 
-test_expect_success 'URI is invalid' '
+methods="config"
+for method in $methods
+do
+
+test_expect_success "$method: URI is invalid" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable@/home/reftable" \
+	test_refs_backend repo files "reftable@/home/reftable" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'URI ends with colon' '
+test_expect_success "$method: URI ends with colon" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable:" \
+	test_refs_backend repo files "reftable:" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'unknown reference backend' '
+test_expect_success "$method: unknown reference backend" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "db://.git" \
+	test_refs_backend repo files "db://.git" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
@@ -86,7 +108,7 @@ do
 	for dir in "$(pwd)/repo/.git" "./"
 	do
 
-		test_expect_success "$read from $to_format backend, $dir dir" '
+		test_expect_success "$method: $read from $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -101,7 +123,7 @@ do
 			)
 		'
 
-		test_expect_success "$write to $to_format backend, $dir dir" '
+		test_expect_success "$method: $write to $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -113,20 +135,22 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" &&
 
 				git refs list >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"tag -d 1" "$method" &&
 				git refs list >actual &&
 				test_cmp expect actual &&
 
 				git refs list | grep -v "refs/tags/1" >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"refs list" "$method" >actual &&
 				test_cmp expect actual
 			)
 		'
 
-		test_expect_success "with worktree and $to_format backend, $dir dir" '
+		test_expect_success "$method: with worktree and $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo wt" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -138,22 +162,26 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				git config set core.repositoryformatversion 1 &&
-				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"worktree add ../wt 2" "$method" &&
 
-				git worktree add ../wt 2
-			) &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >expect &&
+				! test_cmp expect actual &&
 
-			git -C repo for-each-ref --include-root-refs >expect &&
-			git -C wt for-each-ref --include-root-refs >expect &&
-			! test_cmp expect actual &&
-
-			git -C wt rev-parse 2 >expect &&
-			git -C wt rev-parse HEAD >actual &&
-			test_cmp expect actual
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse 2" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse HEAD" "$method" >expect &&
+				test_cmp expect actual
+			)
 		'
 	done # closes dir
 done # closes to_format
-done # closes from_format
+done # closes to_format
+
+done # closes method
 
 test_done

-- 
2.52.0

Re: [PATCH v5 0/4] refs: allow setting the reference directory

[Patrick Steinhardt]

On Mon, Feb 09, 2026 at 04:58:17PM +0100, Karthik Nayak wrote:
> Changes in v5:
> - Moved around the commits, to ensure that the code to handle the config
>   in the backend is first. Previously, we added the config first, which
>   meant the commit allowed users to provide a URI but it was simply
>   ignore.
> - Fix typos and grammar and rename variables.
> - Clean up the description and documentation to actually specify
>   protocol over location.
> - Avoid an extra memory allocation by detaching the strbuf value.
> - Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

I've got some more smallish nits, but this version looks mostly good to
me. The one thing that I think still needs to be extended quite a bit is
the tests we have. The new feature has quite a broad impact, and we
should at least verify that classes of commands work as expected:

  - Commands that create a repository should create the ref backend as
    expected.

  - Commands that access refs (read/write) should know to use the
    correct location.

  - Commands that create worktrees should know where to initialize the
    refs.

  - Reference migration should correctly migrate to/from an out-of-tree
    directory.

Thanks!

Patrick

Re: [PATCH v5 0/4] refs: allow setting the reference directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> Changes in v5:
> - Moved around the commits, to ensure that the code to handle the config
>   in the backend is first. Previously, we added the config first, which
>   meant the commit allowed users to provide a URI but it was simply
>   ignore.
> - Fix typos and grammar and rename variables.
> - Clean up the description and documentation to actually specify
>   protocol over location.

This one looked good.

> - Avoid an extra memory allocation by detaching the strbuf value.

So did this (thanks Stolee for spotting the opportunity).

> - Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

Replaced.  Hopefully this is now ready for 'next'?

Thanks.

Re: [PATCH v5 0/4] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 957 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> Changes in v5:
>> - Moved around the commits, to ensure that the code to handle the config
>>   in the backend is first. Previously, we added the config first, which
>>   meant the commit allowed users to provide a URI but it was simply
>>   ignore.
>> - Fix typos and grammar and rename variables.
>> - Clean up the description and documentation to actually specify
>>   protocol over location.
>
> This one looked good.
>
>> - Avoid an extra memory allocation by detaching the strbuf value.
>
> So did this (thanks Stolee for spotting the opportunity).
>
>> - Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com
>
> Replaced.  Hopefully this is now ready for 'next'?
>
> Thanks.

Hello Junio, I don't think so, I think we still need to address few
things as per the latest review. So let's hold off on it.

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v5 0/4] refs: allow setting the reference directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> Junio C Hamano <gitster@pobox.com> writes:
>
...
>> Replaced.  Hopefully this is now ready for 'next'?
>>
>> Thanks.
>
> Hello Junio, I don't think so, I think we still need to address few
> things as per the latest review. So let's hold off on it.

Thnaks, will do.

[PATCH v6 0/6] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:

  - A format name alone (e.g., `reftable` or `files`).

  - A URI format `<format>://<payload>` explicitly specifies both the
    format and payload (e.g., `reftable:///foo/bar`).

We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v6:
- The biggest change in this version is that we now support using the
  environment variable with 'git-clone(1)' and 'git-init(1)'. In such
  situations, the alternate reference directory is created and the
  config is added to the repository.
- Add a new commit which moves stub creation/removal to the generic
  layer.
- Cleanup logic flow in `refs_compute_filesystem_location()`.
- Add more tests for usage with 'git-clone(1)', 'git-init(1)' and
  migration of repositories using alternate refs backend.
- Fixup documentation, commit messages and typos.
- Link to v5: https://patch.msgid.link/20260209-kn-alternate-ref-dir-v5-0-740899834ceb@gmail.com

Changes in v5:
- Moved around the commits, to ensure that the code to handle the config
  in the backend is first. Previously, we added the config first, which
  meant the commit allowed users to provide a URI but it was simply
  ignore.
- Fix typos and grammar and rename variables.
- Clean up the description and documentation to actually specify
  protocol over location.
- Avoid an extra memory allocation by detaching the strbuf value.
- Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
  backends will store worktree references in 'ref_dir/worktrees/wt_id'
  and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
  relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/config/extensions.adoc |  16 ++-
 Documentation/git.adoc               |   5 +
 builtin/clone.c                      |   9 +-
 builtin/worktree.c                   |  34 +++++
 environment.h                        |   1 +
 refs.c                               | 127 +++++++++++++++++-
 refs.h                               |  13 ++
 refs/files-backend.c                 |  23 +++-
 refs/packed-backend.c                |   5 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  14 ++
 refs/reftable-backend.c              |  61 ++-------
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  96 ++++++++++++--
 setup.h                              |   4 +-
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 250 +++++++++++++++++++++++++++++++++++
 18 files changed, 596 insertions(+), 81 deletions(-)

Karthik Nayak (6):
      setup: don't modify repo in `create_reference_database()`
      refs: extract out `refs_create_refdir_stubs()`
      refs: receive and use the reference storage payload
      refs: move out stub modification to generic layer
      refs: allow reference location in refstorage config
      refs: add GIT_REFERENCE_BACKEND to specify reference backend

Range-diff versus v5:

-:  ---------- > 1:  f8dd9a2fd6 setup: don't modify repo in `create_reference_database()`
1:  d1f777120e ! 2:  07abde22c1 refs: extract out `refs_create_refdir_stubs()`
    @@ builtin/clone.c: int cmd_clone(int argc,
      	 * additional config can be injected with -c, make sure it's included
     
      ## refs.c ##
    -@@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)
    - 		return "unknown failure";
    - 	}
    +@@ refs.c: const char *refs_resolve_ref_unsafe(struct ref_store *refs,
    + 	return NULL;
      }
    -+
    + 
     +void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
     +			      const char *refs_heads_content)
     +{
    @@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)
     +
     +	strbuf_release(&path);
     +}
    ++
    + /* backend functions */
    + int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
    + {
     
      ## refs.h ##
     @@ refs.h: void ref_iterator_free(struct ref_iterator *ref_iterator);
2:  804720d11a ! 3:  aab87c8e39 refs: forward and use the reference storage payload
    @@ Metadata
     Author: Karthik Nayak <karthik.188@gmail.com>
     
      ## Commit message ##
    -    refs: forward and use the reference storage payload
    +    refs: receive and use the reference storage payload
     
         An upcoming commit will add support for providing an URI via the
         'extensions.refStorage' config. The URI will contain the reference
    @@ Commit message
         To prepare for this, modify the existing backends to accept such an
         argument when initializing via the 'init()' function. Both the files
         and reftable backends will parse the information to be filesystem paths
    -    to store references.
    +    to store references. Given that no callers pass any payload yet this is
    +    essentially a no-op change for now.
     
         To enable this, provide a 'refs_compute_filesystem_location()' function
         which will parse the current 'gitdir' and the 'payload' to provide the
    @@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
      	return refs;
      }
      
    -@@ refs.c: void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
    - 
    - 	strbuf_release(&path);
    +@@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)
    + 		return "unknown failure";
    + 	}
      }
     +
     +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
    @@ refs.c: void refs_create_refdir_stubs(struct repository *repo, const char *refdi
     +{
     +	struct strbuf sb = STRBUF_INIT;
     +
    -+	strbuf_addstr(refdir, gitdir);
     +	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
     +
    -+	if (!payload)
    ++	if (!payload) {
    ++		/*
    ++		 * We can use the 'gitdir' as the 'refdir' without appending the
    ++		 * worktree path, as the 'gitdir' here is already the worktree
    ++		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
    ++		 */
    ++		strbuf_addstr(refdir, gitdir);
     +		return;
    ++	}
     +
     +	if (!is_absolute_path(payload)) {
     +		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
    @@ refs.c: void refs_create_refdir_stubs(struct repository *repo, const char *refdi
     +		strbuf_realpath(ref_common_dir, payload, 1);
     +	}
     +
    -+	strbuf_reset(refdir);
     +	strbuf_addbuf(refdir, ref_common_dir);
     +
     +	if (*is_worktree) {
    -+		char *wt_id = strrchr(gitdir, '/') + 1;
    -+		strbuf_addf(refdir, "/worktrees/%s", wt_id);
    ++		const char *wt_id = strrchr(gitdir, '/');
    ++		if (!wt_id)
    ++			BUG("worktree path does not contain slash ");
    ++		strbuf_addf(refdir, "/worktrees/%s", wt_id + 1);
     +	}
     +
     +	strbuf_release(&sb);
    @@ refs/files-backend.c: static void clear_loose_ref_cache(struct files_ref_store *
     +	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
      	refs->packed_ref_store =
     -		packed_ref_store_init(repo, refs->gitcommondir, flags);
    -+		packed_ref_store_init(repo, payload, refs->gitcommondir, flags);
    ++		packed_ref_store_init(repo, NULL, refs->gitcommondir, flags);
      	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
      	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
      
    @@ refs/files-backend.c: static struct ref_store *files_ref_store_init(struct repos
     
      ## refs/packed-backend.c ##
     @@ refs/packed-backend.c: static size_t snapshot_hexsz(const struct snapshot *snapshot)
    + 	return snapshot->refs->base.repo->hash_algo->hexsz;
      }
      
    ++/*
    ++ * Since packed-refs is only stored in the common dir, don't parse the
    ++ * payload and rely on the files-backend to set 'gitdir' correctly.
    ++ */
      struct ref_store *packed_ref_store_init(struct repository *repo,
     +					const char *payload UNUSED,
      					const char *gitdir,
    @@ refs/refs-internal.h: enum ref_transaction_error refs_verify_refnames_available(
     + * directory if working with a linked worktree. If working with the main
     + * worktree, both values will be the same.
     + *
    -+ * This is used by backends such as {files, reftable} which store references in
    -+ * dedicated filesystem paths.
    ++ * This is used by backends that store store files in the repository directly.
     + */
     +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
     +				      bool *is_worktree, struct strbuf *refdir,
-:  ---------- > 4:  9ed74524d3 refs: move out stub modification to generic layer
3:  3a5058f39c ! 5:  331ac15000 refs: allow reference location in refstorage config
    @@ Commit message
         reference backend and the location.
     
         Add the required changes to obtain and propagate this value to the
    -    individual backends also add the necessary documentation and tests.
    +    individual backends. Add the necessary documentation and tests.
     
         Traditionally, for linked worktrees, references were stored in the
         '$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
    @@ Documentation/config/extensions.adoc: For historical reasons, this extension is
      +
      Note that this setting should only be set by linkgit:git-init[1] or
     
    - ## builtin/clone.c ##
    -@@ builtin/clone.c: int cmd_clone(int argc,
    - 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
    - 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
    - 	repo_set_hash_algo(the_repository, hash_algo);
    --	create_reference_database(the_repository->ref_storage_format, NULL, 1);
    -+	create_reference_database(the_repository->ref_storage_format,
    -+				  the_repository->ref_storage_payload, NULL, 1);
    - 
    - 	/*
    - 	 * Before fetching from the remote, download and install bundle
    -
      ## builtin/worktree.c ##
     @@ builtin/worktree.c: static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
      	return run_command(&cp);
      }
      
     +/*
    -+ * References for worktress are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
    ++ * References for worktrees are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
     + * But when using alternate reference directories, we want to store the worktree
     + * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
     + *
    @@ builtin/worktree.c: static int make_worktree_orphan(const char * ref, const stru
     +	strbuf_reset(&sb);
     +
     +	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
    -+		   path, wt->id);
    ++		    path, wt->id);
     +	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
     +
     +	strbuf_release(&sb);
    @@ setup.c: static enum extension_result handle_extension_v0(const char *var,
     +static void parse_reference_uri(const char *value, char **format,
     +				char **payload)
     +{
    -+	char *schema_end;
    ++	const char *schema_end;
     +
     +	schema_end = strstr(value, "://");
     +	if (!schema_end) {
    @@ setup.c: void check_repository_format(struct repository_format *fmt)
      	the_repository->repository_format_worktree_config =
      		fmt->worktree_config;
      	the_repository->repository_format_relative_worktrees =
    -@@ setup.c: static int is_reinit(void)
    - }
    - 
    - void create_reference_database(enum ref_storage_format ref_storage_format,
    -+			       const char *ref_storage_payload,
    - 			       const char *initial_branch, int quiet)
    - {
    - 	struct strbuf err = STRBUF_INIT;
    - 	char *to_free = NULL;
    - 	int reinit = is_reinit();
    - 
    --	repo_set_ref_storage_format(the_repository, ref_storage_format);
    -+	repo_set_ref_storage_format(the_repository, ref_storage_format,
    -+				    ref_storage_payload);
    - 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
    - 		die("failed to set up refs db: %s", err.buf);
    - 
     @@ setup.c: static void repository_format_configure(struct repository_format *repo_fmt,
      	} else {
      		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
    @@ setup.c: static void repository_format_configure(struct repository_format *repo_
      }
      
      int init_db(const char *git_dir, const char *real_git_dir,
    -@@ setup.c: int init_db(const char *git_dir, const char *real_git_dir,
    - 
    - 	if (!(flags & INIT_DB_SKIP_REFDB))
    - 		create_reference_database(repo_fmt.ref_storage_format,
    -+					  repo_fmt.ref_storage_payload,
    - 					  initial_branch, flags & INIT_DB_QUIET);
    - 	create_object_directory();
    - 
     
      ## setup.h ##
     @@ setup.h: struct repository_format {
    @@ setup.h: struct repository_format {
      	int sparse_index;
      	char *work_tree;
      	struct string_list unknown_extensions;
    -@@ setup.h: void initialize_repository_version(int hash_algo,
    - 				   enum ref_storage_format ref_storage_format,
    - 				   int reinit);
    - void create_reference_database(enum ref_storage_format ref_storage_format,
    -+			       const char *ref_storage_payload,
    - 			       const char *initial_branch, int quiet);
    - 
    - /*
     
      ## t/meson.build ##
     @@ t/meson.build: integration_tests = [
    @@ t/t1423-ref-backend.sh (new)
     +	fi
     +
     +
    -+	for dir in "$(pwd)/repo/.git" "./"
    ++	for dir in "$(pwd)/repo/.git" "."
     +	do
     +
    -+		test_expect_success "$read from $to_format backend, $dir dir" '
    ++		test_expect_success "read from $to_format backend, $dir dir" '
     +			test_when_finished "rm -rf repo" &&
     +			git init --ref-format=$from_format repo &&
     +			(
    @@ t/t1423-ref-backend.sh (new)
     +			)
     +		'
     +
    -+		test_expect_success "$write to $to_format backend, $dir dir" '
    ++		test_expect_success "write to $to_format backend, $dir dir" '
     +			test_when_finished "rm -rf repo" &&
     +			git init --ref-format=$from_format repo &&
     +			(
4:  12168d2e35 ! 6:  288bc77a98 refs: add GIT_REFERENCE_BACKEND to specify reference backend
    @@ Commit message
         variable allows us to modify the reference backend and location on the
         fly for individual Git commands.
     
    +    The environment variable also allows usage of alternate reference
    +    directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
    +    the repository when created with the environment variable set.
    +
    +    When initializing the repository with an alternate reference folder,
    +    create the required stubs in the repositories $GIT_DIR. The inverse,
    +    i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
    +    since that would render it unusable. Removal of ref store is only used
    +    when migrating between ref formats and cleanup of the $GIT_DIR doesn't
    +    make sense in such a situation.
    +
         Helped-by: Jean-Noël Avila <jn.avila@free.fr>
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
     
    @@ environment.h
      /*
       * Environment variable used to propagate the --no-advice global option to the
     
    + ## refs.c ##
    +@@ refs.c: int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
    + {
    + 	int ret = refs->be->create_on_disk(refs, flags, err);
    + 
    +-	if (!ret &&
    +-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
    +-		struct strbuf msg = STRBUF_INIT;
    +-
    +-		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
    +-		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
    +-		strbuf_release(&msg);
    ++	if (!ret) {
    ++		/* Creation of stubs for linked worktrees are handled in the worktree code. */
    ++		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
    ++			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
    ++						 "repository uses alternate refs storage");
    ++		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
    ++			struct strbuf msg = STRBUF_INIT;
    ++			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
    ++			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
    ++			strbuf_release(&msg);
    ++		}
    + 	}
    + 
    + 	return ret;
    ++
    + }
    + 
    + int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
    +@@ refs.c: int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
    + 		if (format == REF_STORAGE_FORMAT_FILES)
    + 			return ret;
    + 
    ++		/* Alternate refs backend require stubs in the gitdir. */
    ++		if (refs->repo->ref_storage_payload)
    ++			return ret;
    ++
    + 		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
    + 		if (unlink(sb.buf) < 0) {
    + 			strbuf_addf(err, "could not delete stub HEAD: %s",
    +
      ## setup.c ##
     @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)
      	static struct strbuf cwd = STRBUF_INIT;
    @@ setup.c: const char *setup_git_directory_gently(int *nongit_ok)
     +	 */
     +	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
     +	if (ref_backend_uri) {
    -+		char *backend, *location;
    ++		char *backend, *payload;
     +		enum ref_storage_format format;
     +
    -+		parse_reference_uri(ref_backend_uri, &backend, &location);
    ++		parse_reference_uri(ref_backend_uri, &backend, &payload);
     +		format = ref_storage_format_by_name(backend);
     +		if (format == REF_STORAGE_FORMAT_UNKNOWN)
     +			die(_("unknown ref storage format: '%s'"), backend);
    -+		repo_set_ref_storage_format(the_repository, format, location);
    ++		repo_set_ref_storage_format(the_repository, format, payload);
     +
     +		free(backend);
    -+		free(location);
    ++		free(payload);
     +	}
     +
      	setup_original_cwd();
      
      	strbuf_release(&dir);
    +@@ setup.c: void initialize_repository_version(int hash_algo,
    + 	 * the remote repository's format.
    + 	 */
    + 	if (hash_algo != GIT_HASH_SHA1_LEGACY ||
    +-	    ref_storage_format != REF_STORAGE_FORMAT_FILES)
    ++	    ref_storage_format != REF_STORAGE_FORMAT_FILES ||
    ++	    the_repository->ref_storage_payload)
    + 		target_version = GIT_REPO_VERSION_READ;
    + 
    + 	if (hash_algo != GIT_HASH_SHA1_LEGACY && hash_algo != GIT_HASH_UNKNOWN)
    +@@ setup.c: void initialize_repository_version(int hash_algo,
    + 	else if (reinit)
    + 		repo_config_set_gently(the_repository, "extensions.objectformat", NULL);
    + 
    +-	if (ref_storage_format != REF_STORAGE_FORMAT_FILES)
    ++	if (the_repository->ref_storage_payload) {
    ++		struct strbuf ref_uri = STRBUF_INIT;
    ++
    ++		strbuf_addf(&ref_uri, "%s://%s",
    ++			    ref_storage_format_to_name(ref_storage_format),
    ++			    the_repository->ref_storage_payload);
    ++		repo_config_set(the_repository, "extensions.refstorage", ref_uri.buf);
    ++		strbuf_release(&ref_uri);
    ++	} else if (ref_storage_format != REF_STORAGE_FORMAT_FILES) {
    + 		repo_config_set(the_repository, "extensions.refstorage",
    + 				ref_storage_format_to_name(ref_storage_format));
    +-	else if (reinit)
    ++	} else if (reinit) {
    + 		repo_config_set_gently(the_repository, "extensions.refstorage", NULL);
    ++	}
    + 
    + 	if (reinit) {
    + 		struct strbuf config = STRBUF_INIT;
    +@@ setup.c: static void repository_format_configure(struct repository_format *repo_fmt,
    + 		.ignore_repo = 1,
    + 		.ignore_worktree = 1,
    + 	};
    ++	const char *ref_backend_uri;
    + 	const char *env;
    + 
    + 	config_with_options(read_default_format_config, &cfg, NULL, NULL, &opts);
    +@@ setup.c: static void repository_format_configure(struct repository_format *repo_fmt,
    + 	} else {
    + 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
    + 	}
    ++
    ++
    ++	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
    ++	if (ref_backend_uri) {
    ++		char *backend, *payload;
    ++		enum ref_storage_format format;
    ++
    ++		parse_reference_uri(ref_backend_uri, &backend, &payload);
    ++		format = ref_storage_format_by_name(backend);
    ++		if (format == REF_STORAGE_FORMAT_UNKNOWN)
    ++			die(_("unknown ref storage format: '%s'"), backend);
    ++
    ++		repo_fmt->ref_storage_format = format;
    ++		repo_fmt->ref_storage_payload = payload;
    ++
    ++		free(backend);
    ++	}
    ++
    + 	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
    + 				    repo_fmt->ref_storage_payload);
    + }
     
      ## t/t1423-ref-backend.sh ##
     @@ t/t1423-ref-backend.sh: test_description='Test reference backend URIs'
    @@ t/t1423-ref-backend.sh: run_with_uri() {
      }
      
     -test_expect_success 'URI is invalid' '
    -+methods="config"
    ++methods="config env"
     +for method in $methods
     +do
     +
    @@ t/t1423-ref-backend.sh: run_with_uri() {
      '
      
     @@ t/t1423-ref-backend.sh: do
    - 	for dir in "$(pwd)/repo/.git" "./"
    + 	for dir in "$(pwd)/repo/.git" "."
      	do
      
    --		test_expect_success "$read from $to_format backend, $dir dir" '
    -+		test_expect_success "$method: $read from $to_format backend, $dir dir" '
    +-		test_expect_success "read from $to_format backend, $dir dir" '
    ++		test_expect_success "$method: read from $to_format backend, $dir dir" '
      			test_when_finished "rm -rf repo" &&
      			git init --ref-format=$from_format repo &&
      			(
    @@ t/t1423-ref-backend.sh: do
      			)
      		'
      
    --		test_expect_success "$write to $to_format backend, $dir dir" '
    -+		test_expect_success "$method: $write to $to_format backend, $dir dir" '
    +-		test_expect_success "write to $to_format backend, $dir dir" '
    ++		test_expect_success "$method: write to $to_format backend, $dir dir" '
      			test_when_finished "rm -rf repo" &&
      			git init --ref-format=$from_format repo &&
      			(
    @@ t/t1423-ref-backend.sh: do
      
     -				git config set core.repositoryformatversion 1 &&
     -				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
    +-
    +-				git worktree add ../wt 2
    +-			) &&
     +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
     +					"worktree add ../wt 2" "$method" &&
      
    --				git worktree add ../wt 2
    --			) &&
    +-			git -C repo for-each-ref --include-root-refs >expect &&
    +-			git -C wt for-each-ref --include-root-refs >expect &&
    +-			! test_cmp expect actual &&
     +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
     +					"for-each-ref --include-root-refs" "$method" >actual &&
     +				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
     +					"for-each-ref --include-root-refs" "$method" >expect &&
     +				! test_cmp expect actual &&
      
    --			git -C repo for-each-ref --include-root-refs >expect &&
    --			git -C wt for-each-ref --include-root-refs >expect &&
    --			! test_cmp expect actual &&
    --
     -			git -C wt rev-parse 2 >expect &&
     -			git -C wt rev-parse HEAD >actual &&
     -			test_cmp expect actual
    @@ t/t1423-ref-backend.sh: do
     +			)
      		'
      	done # closes dir
    - done # closes to_format
    --done # closes from_format
    -+done # closes to_format
     +
    -+done # closes method
    ++	test_expect_success "migrating repository to $to_format with alternate refs directory" '
    ++		test_when_finished "rm -rf repo refdir" &&
    ++		mkdir refdir &&
    ++		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
    ++		(
    ++			cd repo &&
    ++
    ++			test_commit 1 &&
    ++			test_commit 2 &&
    ++			test_commit 3 &&
    ++
    ++			git refs migrate --ref-format=$to_format &&
    ++			git refs list >out &&
    ++			test_grep "refs/tags/1"	out &&
    ++			test_grep "refs/tags/2"	out &&
    ++			test_grep "refs/tags/3"	out
    ++		)
    ++	'
    ++
    + done # closes to_format
    + done # closes from_format
      
    ++done # closes method
    ++
    ++test_expect_success 'initializing repository with alt ref directory' '
    ++	test_when_finished "rm -rf repo refdir" &&
    ++	mkdir refdir &&
    ++	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
    ++	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
    ++	(
    ++		cd repo &&
    ++
    ++		git config get extensions.refstorage >expect &&
    ++		echo $BACKEND >actual &&
    ++		test_cmp expect actual &&
    ++
    ++		test_commit 1 &&
    ++		test_commit 2 &&
    ++		test_commit 3 &&
    ++		git refs list >out &&
    ++		test_grep "refs/tags/1"	out &&
    ++		test_grep "refs/tags/2"	out &&
    ++		test_grep "refs/tags/3"	out
    ++	)
    ++'
    ++
    ++test_expect_success 'cloning repository with alt ref directory' '
    ++	test_when_finished "rm -rf source repo refdir" &&
    ++	mkdir refdir &&
    ++
    ++	git init source &&
    ++	test_commit -C source 1 &&
    ++	test_commit -C source 2 &&
    ++	test_commit -C source 3 &&
    ++
    ++	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
    ++	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
    ++
    ++	git -C repo config get extensions.refstorage >expect &&
    ++	echo $BACKEND >actual &&
    ++	test_cmp expect actual &&
    ++
    ++	git -C source for-each-ref refs/tags/ >expect &&
    ++	git -C repo for-each-ref refs/tags/ >actual &&
    ++	test_cmp expect actual
    ++'
    ++
      test_done


base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v6 1/6] setup: don't modify repo in `create_reference_database()`

[Karthik Nayak]

The `create_reference_database()` function is used to create the
reference database during initialization of a repository. The function
calls `repo_set_ref_storage_format()` to set the repositories reference
format. This is an unexpected side-effect of the function. More so
because the function is only called in two locations:

  1. During git-init(1) where the value is propagated from the `struct
     repository_format repo_fmt` value.

  2. During git-clone(1) where the value is propagated from the
     `the_repository` value.

The former is valid, however the flow already calls
`repo_set_ref_storage_format()`, so this effort is simply duplicated.
The latter sets the existing value in `the_repository` back to itself.
While this is okay for now, introduction of more fields in
`repo_set_ref_storage_format()` would cause issues, especially
dynamically allocated strings, where we would free/allocate the same
string back into `the_repostiory`.

To avoid all this confusion, clean up the function to longer take in and
set the repo's reference storage format.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c | 2 +-
 setup.c         | 7 ++-----
 setup.h         | 3 +--
 3 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index b40cee5968..cd43bb5aa2 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1442,7 +1442,7 @@ int cmd_clone(int argc,
 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
 	repo_set_hash_algo(the_repository, hash_algo);
-	create_reference_database(the_repository->ref_storage_format, NULL, 1);
+	create_reference_database(NULL, 1);
 
 	/*
 	 * Before fetching from the remote, download and install bundle
diff --git a/setup.c b/setup.c
index b723f8b339..1fc9ae3872 100644
--- a/setup.c
+++ b/setup.c
@@ -2359,14 +2359,12 @@ static int is_reinit(void)
 	return ret;
 }
 
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet)
+void create_reference_database(const char *initial_branch, int quiet)
 {
 	struct strbuf err = STRBUF_INIT;
 	char *to_free = NULL;
 	int reinit = is_reinit();
 
-	repo_set_ref_storage_format(the_repository, ref_storage_format);
 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -2701,8 +2699,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 				      &repo_fmt, init_shared_repository);
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
-		create_reference_database(repo_fmt.ref_storage_format,
-					  initial_branch, flags & INIT_DB_QUIET);
+		create_reference_database(initial_branch, flags & INIT_DB_QUIET);
 	create_object_directory();
 
 	if (repo_settings_get_shared_repository(the_repository)) {
diff --git a/setup.h b/setup.h
index d55dcc6608..ddb9f6701c 100644
--- a/setup.h
+++ b/setup.h
@@ -240,8 +240,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 void initialize_repository_version(int hash_algo,
 				   enum ref_storage_format ref_storage_format,
 				   int reinit);
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet);
+void create_reference_database(const char *initial_branch, int quiet);
 
 /*
  * NOTE NOTE NOTE!!

-- 
2.52.0

Re: [PATCH v6 1/6] setup: don't modify repo in `create_reference_database()`

[Patrick Steinhardt]

On Sat, Feb 14, 2026 at 11:34:14PM +0100, Karthik Nayak wrote:
> The `create_reference_database()` function is used to create the
> reference database during initialization of a repository. The function
> calls `repo_set_ref_storage_format()` to set the repositories reference
> format. This is an unexpected side-effect of the function. More so
> because the function is only called in two locations:
> 
>   1. During git-init(1) where the value is propagated from the `struct
>      repository_format repo_fmt` value.
> 
>   2. During git-clone(1) where the value is propagated from the
>      `the_repository` value.
> 
> The former is valid, however the flow already calls
> `repo_set_ref_storage_format()`, so this effort is simply duplicated.
> The latter sets the existing value in `the_repository` back to itself.
> While this is okay for now, introduction of more fields in
> `repo_set_ref_storage_format()` would cause issues, especially
> dynamically allocated strings, where we would free/allocate the same
> string back into `the_repostiory`.
> 
> To avoid all this confusion, clean up the function to longer take in and

s/longer/no &/, I assume?

> diff --git a/builtin/clone.c b/builtin/clone.c
> index b40cee5968..cd43bb5aa2 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -1442,7 +1442,7 @@ int cmd_clone(int argc,
>  	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
>  	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
>  	repo_set_hash_algo(the_repository, hash_algo);
> -	create_reference_database(the_repository->ref_storage_format, NULL, 1);
> +	create_reference_database(NULL, 1);
>  
>  	/*
>  	 * Before fetching from the remote, download and install bundle

This is case (2), where we set the ref storage format to itself.

> diff --git a/setup.c b/setup.c
> index b723f8b339..1fc9ae3872 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -2701,8 +2699,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
>  				      &repo_fmt, init_shared_repository);
>  
>  	if (!(flags & INIT_DB_SKIP_REFDB))
> -		create_reference_database(repo_fmt.ref_storage_format,
> -					  initial_branch, flags & INIT_DB_QUIET);
> +		create_reference_database(initial_branch, flags & INIT_DB_QUIET);
>  	create_object_directory();
>  
>  	if (repo_settings_get_shared_repository(the_repository)) {

And this is the second case. We call `repository_format_configure()` a
few lines above, and that function calls `repo_set_ref_storage_format()`
itself.

Looks good to me, and a nice simplification. Thanks!

Patrick

Re: [PATCH v6 1/6] setup: don't modify repo in `create_reference_database()`

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 2703 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Sat, Feb 14, 2026 at 11:34:14PM +0100, Karthik Nayak wrote:
>> The `create_reference_database()` function is used to create the
>> reference database during initialization of a repository. The function
>> calls `repo_set_ref_storage_format()` to set the repositories reference
>> format. This is an unexpected side-effect of the function. More so
>> because the function is only called in two locations:
>>
>>   1. During git-init(1) where the value is propagated from the `struct
>>      repository_format repo_fmt` value.
>>
>>   2. During git-clone(1) where the value is propagated from the
>>      `the_repository` value.
>>
>> The former is valid, however the flow already calls
>> `repo_set_ref_storage_format()`, so this effort is simply duplicated.
>> The latter sets the existing value in `the_repository` back to itself.
>> While this is okay for now, introduction of more fields in
>> `repo_set_ref_storage_format()` would cause issues, especially
>> dynamically allocated strings, where we would free/allocate the same
>> string back into `the_repostiory`.
>>
>> To avoid all this confusion, clean up the function to longer take in and
>
> s/longer/no &/, I assume?
>

Yup.

>> diff --git a/builtin/clone.c b/builtin/clone.c
>> index b40cee5968..cd43bb5aa2 100644
>> --- a/builtin/clone.c
>> +++ b/builtin/clone.c
>> @@ -1442,7 +1442,7 @@ int cmd_clone(int argc,
>>  	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
>>  	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
>>  	repo_set_hash_algo(the_repository, hash_algo);
>> -	create_reference_database(the_repository->ref_storage_format, NULL, 1);
>> +	create_reference_database(NULL, 1);
>>
>>  	/*
>>  	 * Before fetching from the remote, download and install bundle
>
> This is case (2), where we set the ref storage format to itself.
>
>> diff --git a/setup.c b/setup.c
>> index b723f8b339..1fc9ae3872 100644
>> --- a/setup.c
>> +++ b/setup.c
>> @@ -2701,8 +2699,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
>>  				      &repo_fmt, init_shared_repository);
>>
>>  	if (!(flags & INIT_DB_SKIP_REFDB))
>> -		create_reference_database(repo_fmt.ref_storage_format,
>> -					  initial_branch, flags & INIT_DB_QUIET);
>> +		create_reference_database(initial_branch, flags & INIT_DB_QUIET);
>>  	create_object_directory();
>>
>>  	if (repo_settings_get_shared_repository(the_repository)) {
>
> And this is the second case. We call `repository_format_configure()` a
> few lines above, and that function calls `repo_set_ref_storage_format()`
> itself.
>
> Looks good to me, and a nice simplification. Thanks!
>
> Patrick

Thanks!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v6 2/6] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

For Git to recognize a directory as a Git directory, it requires the
directory to contain:

  1. 'HEAD' file
  2. 'objects/' directory
  3. 'refs/' directory

Here, #1 and #3 are part of the reference storage mechanism,
specifically the files backend. Since then, newer backends such as the
reftable backend have moved to using their own path ('reftable/') for
storing references. But to ensure Git still recognizes the directory as
a Git directory, we create stubs.

There are two locations where we create stubs:

- In 'refs/reftable-backend.c' when creating the reftable backend.
- In 'clone.c' before spawning transport helpers.

In a following commit, we'll add another instance. So instead of
repeating the code, let's extract out this code to
`refs_create_refdir_stubs()` and use it.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c         |  7 +------
 refs.c                  | 23 +++++++++++++++++++++++
 refs.h                  | 13 +++++++++++++
 refs/reftable-backend.c | 14 ++------------
 4 files changed, 39 insertions(+), 18 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index cd43bb5aa2..697c5bb5cb 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1225,12 +1225,7 @@ int cmd_clone(int argc,
 	initialize_repository_version(GIT_HASH_UNKNOWN,
 				      the_repository->ref_storage_format, 1);
 
-	strbuf_addf(&buf, "%s/HEAD", git_dir);
-	write_file(buf.buf, "ref: refs/heads/.invalid");
-
-	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/refs", git_dir);
-	safe_create_dir(the_repository, buf.buf, 1);
+	refs_create_refdir_stubs(the_repository, git_dir, NULL);
 
 	/*
 	 * additional config can be injected with -c, make sure it's included
diff --git a/refs.c b/refs.c
index 627b7f8698..77b93d655b 100644
--- a/refs.c
+++ b/refs.c
@@ -2163,6 +2163,29 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 	return NULL;
 }
 
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_content)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	strbuf_addf(&path, "%s/HEAD", refdir);
+	write_file(path.buf, "ref: refs/heads/.invalid");
+	adjust_shared_perm(repo, path.buf);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/refs", refdir);
+	safe_create_dir(repo, path.buf, 1);
+
+	if (refs_heads_content) {
+		strbuf_reset(&path);
+		strbuf_addf(&path, "%s/refs/heads", refdir);
+		write_file(path.buf, "%s", refs_heads_content);
+		adjust_shared_perm(repo, path.buf);
+	}
+
+	strbuf_release(&path);
+}
+
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
diff --git a/refs.h b/refs.h
index f0abfa1d93..9d8890fdff 100644
--- a/refs.h
+++ b/refs.h
@@ -1427,4 +1427,17 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
+/*
+ * Git only recognizes a directory as a repository if it contains:
+ * - HEAD file
+ * - refs/ folder
+ * While it is necessary within the files backend, newer backends may not
+ * follow the same structure. To go around this, we create stubs as necessary.
+ *
+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file
+ * with the provided message.
+ */
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_msg);
+
 #endif /* REFS_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index fe74af73af..d8651fe779 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,18 +491,8 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	write_file(sb.buf, "ref: refs/heads/.invalid");
-	adjust_shared_perm(the_repository, sb.buf);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	safe_create_dir(the_repository, sb.buf, 1);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	write_file(sb.buf, "this repository uses the reftable format");
-	adjust_shared_perm(the_repository, sb.buf);
+	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
+				 "this repository uses the reftable format");
 
 	strbuf_release(&sb);
 	return 0;

-- 
2.52.0

[PATCH v6 3/6] refs: receive and use the reference storage payload

[Karthik Nayak]

An upcoming commit will add support for providing an URI via the
'extensions.refStorage' config. The URI will contain the reference
backend and a corresponding payload. The payload can be then used for
providing an alternate locations for the reference backend.

To prepare for this, modify the existing backends to accept such an
argument when initializing via the 'init()' function. Both the files
and reftable backends will parse the information to be filesystem paths
to store references. Given that no callers pass any payload yet this is
essentially a no-op change for now.

To enable this, provide a 'refs_compute_filesystem_location()' function
which will parse the current 'gitdir' and the 'payload' to provide the
final reference directory and common reference directory (if working in
a linked worktree).

The documentation and tests will be added alongside the extension of the
config variable.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 40 +++++++++++++++++++++++++++++++++++++++-
 refs/files-backend.c    | 17 ++++++++++++-----
 refs/packed-backend.c   |  5 +++++
 refs/packed-backend.h   |  1 +
 refs/refs-internal.h    | 14 ++++++++++++++
 refs/reftable-backend.c | 24 ++++++++++++++----------
 6 files changed, 85 insertions(+), 16 deletions(-)

diff --git a/refs.c b/refs.c
index 77b93d655b..11d028232b 100644
--- a/refs.c
+++ b/refs.c
@@ -5,6 +5,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "abspath.h"
 #include "advice.h"
 #include "config.h"
 #include "environment.h"
@@ -2247,7 +2248,7 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, gitdir, flags);
+	refs = be->init(repo, NULL, gitdir, flags);
 	return refs;
 }
 
@@ -3425,3 +3426,40 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
 		return "unknown failure";
 	}
 }
+
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
+
+	if (!payload) {
+		/*
+		 * We can use the 'gitdir' as the 'refdir' without appending the
+		 * worktree path, as the 'gitdir' here is already the worktree
+		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
+		 */
+		strbuf_addstr(refdir, gitdir);
+		return;
+	}
+
+	if (!is_absolute_path(payload)) {
+		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
+		strbuf_realpath(ref_common_dir, sb.buf, 1);
+	} else {
+		strbuf_realpath(ref_common_dir, payload, 1);
+	}
+
+	strbuf_addbuf(refdir, ref_common_dir);
+
+	if (*is_worktree) {
+		const char *wt_id = strrchr(gitdir, '/');
+		if (!wt_id)
+			BUG("worktree path does not contain slash ");
+		strbuf_addf(refdir, "/worktrees/%s", wt_id + 1);
+	}
+
+	strbuf_release(&sb);
+}
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 240d3c3b26..b40d6feb1f 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
  * set of caches.
  */
 static struct ref_store *files_ref_store_init(struct repository *repo,
+					      const char *payload,
 					      const char *gitdir,
 					      unsigned int flags)
 {
 	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
 	struct ref_store *ref_store = (struct ref_store *)refs;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
+	bool is_worktree;
+
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
 
-	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
+	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
 	refs->store_flags = flags;
-	get_common_dir_noenv(&sb, gitdir);
-	refs->gitcommondir = strbuf_detach(&sb, NULL);
+	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
 	refs->packed_ref_store =
-		packed_ref_store_init(repo, refs->gitcommondir, flags);
+		packed_ref_store_init(repo, NULL, refs->gitcommondir, flags);
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
 	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
 
@@ -126,6 +131,8 @@ static struct ref_store *files_ref_store_init(struct repository *repo,
 	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
 			      &refs->gitcommondir);
 
+	strbuf_release(&refdir);
+
 	return ref_store;
 }
 
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 4ea0c12299..e7bb9f10f9 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -211,7 +211,12 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
 	return snapshot->refs->base.repo->hash_algo->hexsz;
 }
 
+/*
+ * Since packed-refs is only stored in the common dir, don't parse the
+ * payload and rely on the files-backend to set 'gitdir' correctly.
+ */
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload UNUSED,
 					const char *gitdir,
 					unsigned int store_flags)
 {
diff --git a/refs/packed-backend.h b/refs/packed-backend.h
index 9481d5e7c2..2c2377a356 100644
--- a/refs/packed-backend.h
+++ b/refs/packed-backend.h
@@ -14,6 +14,7 @@ struct ref_transaction;
  */
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload,
 					const char *gitdir,
 					unsigned int store_flags);
 
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index c7d2a6e50b..9a635f4e6c 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -389,6 +389,7 @@ struct ref_store;
  * the ref_store and to record the ref_store for later lookup.
  */
 typedef struct ref_store *ref_store_init_fn(struct repository *repo,
+					    const char *payload,
 					    const char *gitdir,
 					    unsigned int flags);
 /*
@@ -666,4 +667,17 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 					  unsigned int initial_transaction,
 					  struct strbuf *err);
 
+/*
+ * Given a gitdir and the reference storage payload provided, retrieve the
+ * 'refdir' and 'ref_common_dir'. The former is where references should be
+ * stored for the current worktree, the latter is the common reference
+ * directory if working with a linked worktree. If working with the main
+ * worktree, both values will be the same.
+ *
+ * This is used by backends that store store files in the repository directly.
+ */
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir);
+
 #endif /* REFS_REFS_INTERNAL_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d8651fe779..964b0b50fc 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -372,18 +372,24 @@ static int reftable_be_fsync(int fd)
 }
 
 static struct ref_store *reftable_be_init(struct repository *repo,
+					  const char *payload,
 					  const char *gitdir,
 					  unsigned int store_flags)
 {
 	struct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
 	struct strbuf path = STRBUF_INIT;
-	int is_worktree;
+	bool is_worktree;
 	mode_t mask;
 
 	mask = umask(0);
 	umask(mask);
 
-	base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
+
+	base_ref_store_init(&refs->base, repo, refdir.buf, &refs_be_reftable);
 	strmap_init(&refs->worktree_backends);
 	refs->store_flags = store_flags;
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
@@ -419,14 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	/*
 	 * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
 	 * This stack contains both the shared and the main worktree refs.
-	 *
-	 * Note that we don't try to resolve the path in case we have a
-	 * worktree because `get_common_dir_noenv()` already does it for us.
 	 */
-	is_worktree = get_common_dir_noenv(&path, gitdir);
+	strbuf_addbuf(&path, &ref_common_dir);
 	if (!is_worktree) {
 		strbuf_reset(&path);
-		strbuf_realpath(&path, gitdir, 0);
+		strbuf_realpath(&path, ref_common_dir.buf, 0);
 	}
 	strbuf_addstr(&path, "/reftable");
 	refs->err = reftable_backend_init(&refs->main_backend, path.buf,
@@ -443,10 +446,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	 * do it efficiently.
 	 */
 	if (is_worktree) {
-		strbuf_reset(&path);
-		strbuf_addf(&path, "%s/reftable", gitdir);
+		strbuf_addstr(&refdir, "/reftable");
 
-		refs->err = reftable_backend_init(&refs->worktree_backend, path.buf,
+		refs->err = reftable_backend_init(&refs->worktree_backend, refdir.buf,
 						  &refs->write_options);
 		if (refs->err)
 			goto done;
@@ -456,6 +458,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 
 done:
 	assert(refs->err != REFTABLE_API_ERROR);
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
 	strbuf_release(&path);
 	return &refs->base;
 }

-- 
2.52.0

Re: [PATCH v6 3/6] refs: receive and use the reference storage payload

[Patrick Steinhardt]

On Sat, Feb 14, 2026 at 11:34:16PM +0100, Karthik Nayak wrote:
> diff --git a/refs.c b/refs.c
> index 77b93d655b..11d028232b 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -3425,3 +3426,40 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
>  		return "unknown failure";
>  	}
>  }
> +
> +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
> +				      bool *is_worktree, struct strbuf *refdir,
> +				      struct strbuf *ref_common_dir)
> +{
> +	struct strbuf sb = STRBUF_INIT;
> +
> +	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
> +
> +	if (!payload) {
> +		/*
> +		 * We can use the 'gitdir' as the 'refdir' without appending the
> +		 * worktree path, as the 'gitdir' here is already the worktree
> +		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
> +		 */
> +		strbuf_addstr(refdir, gitdir);
> +		return;
> +	}
> +
> +	if (!is_absolute_path(payload)) {
> +		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
> +		strbuf_realpath(ref_common_dir, sb.buf, 1);
> +	} else {
> +		strbuf_realpath(ref_common_dir, payload, 1);
> +	}
> +
> +	strbuf_addbuf(refdir, ref_common_dir);
> +
> +	if (*is_worktree) {
> +		const char *wt_id = strrchr(gitdir, '/');
> +		if (!wt_id)
> +			BUG("worktree path does not contain slash ");

There's a trailing space in the error message here.

Patrick

Re: [PATCH v6 3/6] refs: receive and use the reference storage payload

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1484 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Sat, Feb 14, 2026 at 11:34:16PM +0100, Karthik Nayak wrote:
>> diff --git a/refs.c b/refs.c
>> index 77b93d655b..11d028232b 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -3425,3 +3426,40 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
>>  		return "unknown failure";
>>  	}
>>  }
>> +
>> +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
>> +				      bool *is_worktree, struct strbuf *refdir,
>> +				      struct strbuf *ref_common_dir)
>> +{
>> +	struct strbuf sb = STRBUF_INIT;
>> +
>> +	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
>> +
>> +	if (!payload) {
>> +		/*
>> +		 * We can use the 'gitdir' as the 'refdir' without appending the
>> +		 * worktree path, as the 'gitdir' here is already the worktree
>> +		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
>> +		 */
>> +		strbuf_addstr(refdir, gitdir);
>> +		return;
>> +	}
>> +
>> +	if (!is_absolute_path(payload)) {
>> +		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
>> +		strbuf_realpath(ref_common_dir, sb.buf, 1);
>> +	} else {
>> +		strbuf_realpath(ref_common_dir, payload, 1);
>> +	}
>> +
>> +	strbuf_addbuf(refdir, ref_common_dir);
>> +
>> +	if (*is_worktree) {
>> +		const char *wt_id = strrchr(gitdir, '/');
>> +		if (!wt_id)
>> +			BUG("worktree path does not contain slash ");
>
> There's a trailing space in the error message here.
>
> Patrick

Oops, will fix.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v6 4/6] refs: move out stub modification to generic layer

[Karthik Nayak]

When creating the reftable reference backend on disk, we create stubs to
ensure that the directory can be recognized as a Git repository. This is
done by calling `refs_create_refdir_stubs()`. Move this to the generic
layer as this is needed for all backends excluding from the files
backends. In an upcoming commit, we'll also need to extend this logic to
create stubs when using alternate reference directories.

Similarly, move the logic for deletion of stubs to the generic layer.
The files backend recursively calls the remove function of the
'packed-backend', here skip calling the generic function since that
would try to delete stubs.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 51 +++++++++++++++++++++++++++++++++++++++++++++++--
 refs/files-backend.c    |  6 +++++-
 refs/reftable-backend.c | 27 --------------------------
 3 files changed, 54 insertions(+), 30 deletions(-)

diff --git a/refs.c b/refs.c
index 11d028232b..a24602c9bf 100644
--- a/refs.c
+++ b/refs.c
@@ -2190,12 +2190,59 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
-	return refs->be->create_on_disk(refs, flags, err);
+	int ret = refs->be->create_on_disk(refs, flags, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf msg = STRBUF_INIT;
+
+		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+		strbuf_release(&msg);
+	}
+
+	return ret;
 }
 
 int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
-	return refs->be->remove_on_disk(refs, err);
+	int ret = refs->be->remove_on_disk(refs, err);
+
+	if (!ret) {
+		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
+		struct strbuf sb = STRBUF_INIT;
+
+		/* Backends apart from the files backend create stubs. */
+		if (format == REF_STORAGE_FORMAT_FILES)
+			return ret;
+
+		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub HEAD: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs/heads", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub heads: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs", refs->gitdir);
+		if (rmdir(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete refs directory: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+
+		strbuf_release(&sb);
+	}
+
+	return ret;
 }
 
 int repo_resolve_gitlink_ref(struct repository *r,
diff --git a/refs/files-backend.c b/refs/files-backend.c
index b40d6feb1f..9cde3ba724 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -3707,7 +3707,11 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,
 	if (for_each_root_ref(refs, remove_one_root_ref, &data) < 0)
 		ret = -1;
 
-	if (ref_store_remove_on_disk(refs->packed_ref_store, err) < 0)
+	/*
+	 * Directly access the cleanup functions for packed-refs as the generic function
+	 * would try to clear stubs which isn't required for the files backend.
+	 */
+	if (refs->packed_ref_store->be->remove_on_disk(refs->packed_ref_store, err) < 0)
 		ret = -1;
 
 	strbuf_release(&sb);
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 964b0b50fc..0e220d6bb5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -495,9 +495,6 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
-				 "this repository uses the reftable format");
-
 	strbuf_release(&sb);
 	return 0;
 }
@@ -523,30 +520,6 @@ static int reftable_be_remove_on_disk(struct ref_store *ref_store,
 			    strerror(errno));
 		ret = -1;
 	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub HEAD: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub heads: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	if (rmdir(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete refs directory: %s",
-			    strerror(errno));
-		ret = -1;
-	}
 
 	strbuf_release(&sb);
 	return ret;

-- 
2.52.0

Re: [PATCH v6 4/6] refs: move out stub modification to generic layer

[Patrick Steinhardt]

On Sat, Feb 14, 2026 at 11:34:17PM +0100, Karthik Nayak wrote:
> When creating the reftable reference backend on disk, we create stubs to
> ensure that the directory can be recognized as a Git repository. This is
> done by calling `refs_create_refdir_stubs()`. Move this to the generic
> layer as this is needed for all backends excluding from the files
> backends. In an upcoming commit, we'll also need to extend this logic to
> create stubs when using alternate reference directories.
> 
> Similarly, move the logic for deletion of stubs to the generic layer.
> The files backend recursively calls the remove function of the
> 'packed-backend', here skip calling the generic function since that
> would try to delete stubs.

Tiniest nit: it might make sense to reorder patches a bit so that the
creation of `refs_create_refdir_stubs()` and this patch here sit next to
each other.

What's missing a bit in the commit message is the motivation. What does
this step enable us to do that we couldn't do before?

> diff --git a/refs.c b/refs.c
> index 11d028232b..a24602c9bf 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2190,12 +2190,59 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>  /* backend functions */
>  int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
>  {
> -	return refs->be->create_on_disk(refs, flags, err);
> +	int ret = refs->be->create_on_disk(refs, flags, err);
> +
> +	if (!ret &&
> +	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
> +		struct strbuf msg = STRBUF_INIT;
> +
> +		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
> +		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
> +		strbuf_release(&msg);
> +	}
> +
> +	return ret;
>  }

This makes me wonder: if we called `refs_create_refdir_stubs()` before
we call `->create_on_disk()`, could we even do it for the "files"
backend? Just a thought though.

>  int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
>  {
> -	return refs->be->remove_on_disk(refs, err);
> +	int ret = refs->be->remove_on_disk(refs, err);
> +
> +	if (!ret) {
> +		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
> +		struct strbuf sb = STRBUF_INIT;
> +
> +		/* Backends apart from the files backend create stubs. */
> +		if (format == REF_STORAGE_FORMAT_FILES)
> +			return ret;

For symmetry it would be nice to not have an early return here, but also
format the condition for this block in the same way as we have it for
`ref_store_create_on_disk()`.

Patrick

Re: [PATCH v6 4/6] refs: move out stub modification to generic layer

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3427 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Sat, Feb 14, 2026 at 11:34:17PM +0100, Karthik Nayak wrote:
>> When creating the reftable reference backend on disk, we create stubs to
>> ensure that the directory can be recognized as a Git repository. This is
>> done by calling `refs_create_refdir_stubs()`. Move this to the generic
>> layer as this is needed for all backends excluding from the files
>> backends. In an upcoming commit, we'll also need to extend this logic to
>> create stubs when using alternate reference directories.
>>
>> Similarly, move the logic for deletion of stubs to the generic layer.
>> The files backend recursively calls the remove function of the
>> 'packed-backend', here skip calling the generic function since that
>> would try to delete stubs.
>
> Tiniest nit: it might make sense to reorder patches a bit so that the
> creation of `refs_create_refdir_stubs()` and this patch here sit next to
> each other.
>

I think that would be nice, let me do that.

> What's missing a bit in the commit message is the motivation. What does
> this step enable us to do that we couldn't do before?
>

I did add a line

  In an upcoming commit, we'll also need to extend this logic to create
  stubs when using alternate reference directories.

I'll expand a little on that.

>> diff --git a/refs.c b/refs.c
>> index 11d028232b..a24602c9bf 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2190,12 +2190,59 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>>  /* backend functions */
>>  int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
>>  {
>> -	return refs->be->create_on_disk(refs, flags, err);
>> +	int ret = refs->be->create_on_disk(refs, flags, err);
>> +
>> +	if (!ret &&
>> +	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>> +		struct strbuf msg = STRBUF_INIT;
>> +
>> +		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>> +		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>> +		strbuf_release(&msg);
>> +	}
>> +
>> +	return ret;
>>  }
>
> This makes me wonder: if we called `refs_create_refdir_stubs()` before
> we call `->create_on_disk()`, could we even do it for the "files"
> backend? Just a thought though.
>

Well, there is some nuance there

1. 'refs/heads', 'refs/tags' is not created for linked worktrees.
2. 'HEAD' is only created lazily, not in `create_on_disk()`.

Also the intent is totally different, the stubs are for backward
compatibility. So I think its better to let that logic stay within the
files-backend.

>>  int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
>>  {
>> -	return refs->be->remove_on_disk(refs, err);
>> +	int ret = refs->be->remove_on_disk(refs, err);
>> +
>> +	if (!ret) {
>> +		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
>> +		struct strbuf sb = STRBUF_INIT;
>> +
>> +		/* Backends apart from the files backend create stubs. */
>> +		if (format == REF_STORAGE_FORMAT_FILES)
>> +			return ret;
>
> For symmetry it would be nice to not have an early return here, but also
> format the condition for this block in the same way as we have it for
> `ref_store_create_on_disk()`.
>
> Patrick

Yeah sure, we can do that here, in the last commit, we'll have to modify
that anyway back to something like this. But it definitely would be
easier to review this commit. Will add.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v6 4/6] refs: move out stub modification to generic layer

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> Patrick Steinhardt <ps@pks.im> writes:
>
>> On Sat, Feb 14, 2026 at 11:34:17PM +0100, Karthik Nayak wrote:
>>> When creating the reftable reference backend on disk, we create stubs to
>>> ensure that the directory can be recognized as a Git repository. This is
>>> done by calling `refs_create_refdir_stubs()`. Move this to the generic
>>> layer as this is needed for all backends excluding from the files
>>> backends. In an upcoming commit, we'll also need to extend this logic to
>>> create stubs when using alternate reference directories.
>>>
>>> Similarly, move the logic for deletion of stubs to the generic layer.
>>> The files backend recursively calls the remove function of the
>>> 'packed-backend', here skip calling the generic function since that
>>> would try to delete stubs.
>>
>> Tiniest nit: it might make sense to reorder patches a bit so that the
>> creation of `refs_create_refdir_stubs()` and this patch here sit next to
>> each other.
>>
>
> I think that would be nice, let me do that.

Thanks, I was thinking the same, but I wasn't going to comment on that.
Happy to see you've agreed on this already.

>> What's missing a bit in the commit message is the motivation. What does
>> this step enable us to do that we couldn't do before?
>>
>
> I did add a line
>
>   In an upcoming commit, we'll also need to extend this logic to create
>   stubs when using alternate reference directories.
>
> I'll expand a little on that.

<3

>>> diff --git a/refs.c b/refs.c
>>> index 11d028232b..a24602c9bf 100644
>>> --- a/refs.c
>>> +++ b/refs.c
>>> @@ -2190,12 +2190,59 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>>>  /* backend functions */
>>>  int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
>>>  {
>>> -	return refs->be->create_on_disk(refs, flags, err);
>>> +	int ret = refs->be->create_on_disk(refs, flags, err);
>>> +
>>> +	if (!ret &&
>>> +	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>>> +		struct strbuf msg = STRBUF_INIT;
>>> +
>>> +		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>>> +		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>>> +		strbuf_release(&msg);
>>> +	}
>>> +
>>> +	return ret;
>>>  }
>>
>> This makes me wonder: if we called `refs_create_refdir_stubs()` before
>> we call `->create_on_disk()`, could we even do it for the "files"
>> backend? Just a thought though.
>>
>
> Well, there is some nuance there
>
> 1. 'refs/heads', 'refs/tags' is not created for linked worktrees.

I'm a little bit confused what you mean here? Would it be a problem if
it *is* created?

> 2. 'HEAD' is only created lazily, not in `create_on_disk()`.

Okay, seems like a valid argument to me. You don't want to have
`refs/HEAD` created with `ref: refs/heads/.invalid`?

> Also the intent is totally different, the stubs are for backward
> compatibility. So I think its better to let that logic stay within the
> files-backend.

That's mainly because you named the function like this, but it doesn't
have to be named like that.

>> For symmetry it would be nice to not have an early return here, but also
>> format the condition for this block in the same way as we have it for
>> `ref_store_create_on_disk()`.
>>
>> Patrick
>
> Yeah sure, we can do that here, in the last commit, we'll have to modify
> that anyway back to something like this. But it definitely would be
> easier to review this commit. Will add.

:+1:

-- 
Cheers,
Toon

Re: [PATCH v6 4/6] refs: move out stub modification to generic layer

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 4439 bytes --]

Toon Claes <toon@iotcl.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>>> On Sat, Feb 14, 2026 at 11:34:17PM +0100, Karthik Nayak wrote:
>>>> When creating the reftable reference backend on disk, we create stubs to
>>>> ensure that the directory can be recognized as a Git repository. This is
>>>> done by calling `refs_create_refdir_stubs()`. Move this to the generic
>>>> layer as this is needed for all backends excluding from the files
>>>> backends. In an upcoming commit, we'll also need to extend this logic to
>>>> create stubs when using alternate reference directories.
>>>>
>>>> Similarly, move the logic for deletion of stubs to the generic layer.
>>>> The files backend recursively calls the remove function of the
>>>> 'packed-backend', here skip calling the generic function since that
>>>> would try to delete stubs.
>>>
>>> Tiniest nit: it might make sense to reorder patches a bit so that the
>>> creation of `refs_create_refdir_stubs()` and this patch here sit next to
>>> each other.
>>>
>>
>> I think that would be nice, let me do that.
>
> Thanks, I was thinking the same, but I wasn't going to comment on that.
> Happy to see you've agreed on this already.
>
>>> What's missing a bit in the commit message is the motivation. What does
>>> this step enable us to do that we couldn't do before?
>>>
>>
>> I did add a line
>>
>>   In an upcoming commit, we'll also need to extend this logic to create
>>   stubs when using alternate reference directories.
>>
>> I'll expand a little on that.
>
> <3
>
>>>> diff --git a/refs.c b/refs.c
>>>> index 11d028232b..a24602c9bf 100644
>>>> --- a/refs.c
>>>> +++ b/refs.c
>>>> @@ -2190,12 +2190,59 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>>>>  /* backend functions */
>>>>  int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
>>>>  {
>>>> -	return refs->be->create_on_disk(refs, flags, err);
>>>> +	int ret = refs->be->create_on_disk(refs, flags, err);
>>>> +
>>>> +	if (!ret &&
>>>> +	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>>>> +		struct strbuf msg = STRBUF_INIT;
>>>> +
>>>> +		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>>>> +		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>>>> +		strbuf_release(&msg);
>>>> +	}
>>>> +
>>>> +	return ret;
>>>>  }
>>>
>>> This makes me wonder: if we called `refs_create_refdir_stubs()` before
>>> we call `->create_on_disk()`, could we even do it for the "files"
>>> backend? Just a thought though.
>>>
>>
>> Well, there is some nuance there
>>
>> 1. 'refs/heads', 'refs/tags' is not created for linked worktrees.
>
> I'm a little bit confused what you mean here? Would it be a problem if
> it *is* created?
>

Shouldn't be a problem, but I'd rather not create something which isn't
needed.

>> 2. 'HEAD' is only created lazily, not in `create_on_disk()`.
>
> Okay, seems like a valid argument to me. You don't want to have
> `refs/HEAD` created with `ref: refs/heads/.invalid`?
>

We could override it when the actual HEAD ref is created.

>> Also the intent is totally different, the stubs are for backward
>> compatibility. So I think its better to let that logic stay within the
>> files-backend.
>
> That's mainly because you named the function like this, but it doesn't
> have to be named like that.
>

Not really... The difference being that the files backend creates these
files/folders among others since it needs it for its operation.

The other backends create the bare minimum files and folder stubs
because we need to stay backward compatible and ensure the Git folder is
still treated.

I didn't want to mix up that because we could decide to change things in
the files backend, but we can't change how the stubs are created. But I
do agree that we could potentially combine this and allow the files
backend to override it. I don't think I want to get into that in this
patch series.

>>> For symmetry it would be nice to not have an early return here, but also
>>> format the condition for this block in the same way as we have it for
>>> `ref_store_create_on_disk()`.
>>>
>>> Patrick
>>
>> Yeah sure, we can do that here, in the last commit, we'll have to modify
>> that anyway back to something like this. But it definitely would be
>> easier to review this commit. Will add.
>
> :+1:
>
> --
> Cheers,
> Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v6 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

The 'extensions.refStorage' config is used to specify the reference
backend for a given repository. Both the 'files' and 'reftable' backends
utilize the $GIT_DIR as the reference folder by default in
`get_main_ref_store()`.

Since the reference backends are pluggable, this means that they could
work with out-of-tree reference directories too. Extend the 'refStorage'
config to also support taking an URI input, where users can specify the
reference backend and the location.

Add the required changes to obtain and propagate this value to the
individual backends. Add the necessary documentation and tests.

Traditionally, for linked worktrees, references were stored in the
'$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
storage path, it doesn't make sense to store the main worktree
references in the new path, and the linked worktree references in the
$GIT_DIR. So, let's store linked worktree references in
'$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'. To do this, create the
necessary files and folders while also adding stubs in the $GIT_DIR path
to ensure that it is still considered a Git directory.

Ideally, we would want to pass in a `struct worktree *` to individual
backends, instead of passing the `gitdir`. This allows them to handle
worktree specific logic. Currently, that is not possible since the
worktree code is:

  - Tied to using the global `the_repository` variable.

  - Is not setup before the reference database during initialization of
    the repository.

Add a TODO in 'refs.c' to ensure we can eventually make that change.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/config/extensions.adoc |  16 +++-
 builtin/worktree.c                   |  34 ++++++++
 refs.c                               |   6 +-
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  34 +++++++-
 setup.h                              |   1 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
 9 files changed, 259 insertions(+), 9 deletions(-)

diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
index 532456644b..3e51da36d3 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
 `core.repositoryFormatVersion` setting.
 
 refStorage:::
-	Specify the ref storage format to use. The acceptable values are:
+	Specify the ref storage format and a corresponding payload. The value
+	can be either a format name or a URI:
 +
 --
+* A format name alone (e.g., `reftable` or `files`).
+
+* A URI format `<format>://<payload>` explicitly specifies both the
+  format and payload (e.g., `reftable:///foo/bar`).
+
+Supported format names are:
++
 include::../ref-storage-format.adoc[]
++
+The payload is passed directly to the reference backend. For the files and
+reftable backends, this must be a filesystem path where the references will
+be stored. Defaulting to the commondir when no payload is provided. Relative
+paths are resolved relative to the $GIT_DIR. Future backends may support
+other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +
 Note that this setting should only be set by linkgit:git-init[1] or
diff --git a/builtin/worktree.c b/builtin/worktree.c
index fbdaf2eb2e..293e808379 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
 	return run_command(&cp);
 }
 
+/*
+ * References for worktrees are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
+ * But when using alternate reference directories, we want to store the worktree
+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
+ *
+ * Create the necessary folder structure to facilitate the same. But to ensure
+ * that the former path is still considered a Git directory, add stubs.
+ */
+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	char *path;
+
+	path = wt->repo->ref_storage_payload;
+	if (!path)
+		return;
+
+	if (!is_absolute_path(path))
+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
+
+	strbuf_addf(&sb, "%s/worktrees", path);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_addf(&sb, "/%s", wt->id);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_reset(&sb);
+
+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
+		    path, wt->id);
+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
+
+	strbuf_release(&sb);
+}
+
 static int add_worktree(const char *path, const char *refname,
 			const struct add_opts *opts)
 {
@@ -518,6 +551,7 @@ static int add_worktree(const char *path, const char *refname,
 		ret = error(_("could not find created worktree '%s'"), name);
 		goto done;
 	}
+	setup_alternate_ref_dir(wt, sb_repo.buf);
 	wt_refs = get_worktree_ref_store(wt);
 
 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
diff --git a/refs.c b/refs.c
index a24602c9bf..87ef54abd4 100644
--- a/refs.c
+++ b/refs.c
@@ -2295,7 +2295,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, NULL, gitdir, flags);
+	/*
+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
+	 * allow the backend to handle how it wants to deal with worktrees.
+	 */
+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
 	return refs;
 }
 
diff --git a/repository.c b/repository.c
index c7e75215ac..9815f081ef 100644
--- a/repository.c
+++ b/repository.c
@@ -193,9 +193,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
 }
 
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format)
+				 enum ref_storage_format format,
+				 const char *payload)
 {
 	repo->ref_storage_format = format;
+	free(repo->ref_storage_payload);
+	repo->ref_storage_payload = xstrdup_or_null(payload);
 }
 
 /*
@@ -277,7 +280,8 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 	repo_set_compat_hash_algo(repo, format.compat_hash_algo);
-	repo_set_ref_storage_format(repo, format.ref_storage_format);
+	repo_set_ref_storage_format(repo, format.ref_storage_format,
+				    format.ref_storage_payload);
 	repo->repository_format_worktree_config = format.worktree_config;
 	repo->repository_format_relative_worktrees = format.relative_worktrees;
 	repo->repository_format_precious_objects = format.precious_objects;
@@ -369,6 +373,7 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
+	FREE_AND_NULL(repo->ref_storage_payload);
 
 	odb_free(repo->objects);
 	repo->objects = NULL;
diff --git a/repository.h b/repository.h
index 6063c4b846..95e2333bad 100644
--- a/repository.h
+++ b/repository.h
@@ -150,6 +150,11 @@ struct repository {
 
 	/* Repository's reference storage format, as serialized on disk. */
 	enum ref_storage_format ref_storage_format;
+	/*
+	 * Reference storage information as needed for the backend. This contains
+	 * only the payload from the reference URI without the schema.
+	 */
+	char *ref_storage_payload;
 
 	/* A unique-id for tracing purposes. */
 	int trace2_repo_id;
@@ -204,7 +209,8 @@ void repo_set_worktree(struct repository *repo, const char *path);
 void repo_set_hash_algo(struct repository *repo, int algo);
 void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format);
+				 enum ref_storage_format format,
+				 const char *payload);
 void initialize_repository(struct repository *repo);
 RESULT_MUST_BE_USED
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index 1fc9ae3872..d407f3347b 100644
--- a/setup.c
+++ b/setup.c
@@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
 		return EXTENSION_UNKNOWN;
 }
 
+static void parse_reference_uri(const char *value, char **format,
+				char **payload)
+{
+	const char *schema_end;
+
+	schema_end = strstr(value, "://");
+	if (!schema_end) {
+		*format = xstrdup(value);
+		*payload = NULL;
+	} else {
+		*format = xstrndup(value, schema_end - value);
+		*payload = xstrdup_or_null(schema_end + 3);
+	}
+}
+
 /*
  * Record any new extensions in this function.
  */
@@ -674,10 +689,17 @@ static enum extension_result handle_extension(const char *var,
 		return EXTENSION_OK;
 	} else if (!strcmp(ext, "refstorage")) {
 		unsigned int format;
+		char *format_str;
 
 		if (!value)
 			return config_error_nonbool(var);
-		format = ref_storage_format_by_name(value);
+
+		parse_reference_uri(value, &format_str,
+				    &data->ref_storage_payload);
+
+		format = ref_storage_format_by_name(format_str);
+		free(format_str);
+
 		if (format == REF_STORAGE_FORMAT_UNKNOWN)
 			return error(_("invalid value for '%s': '%s'"),
 				     "extensions.refstorage", value);
@@ -850,6 +872,7 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
 	free(format->partial_clone);
+	free(format->ref_storage_payload);
 	init_repository_format(format);
 }
 
@@ -1942,7 +1965,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			repo_set_compat_hash_algo(the_repository,
 						  repo_fmt.compat_hash_algo);
 			repo_set_ref_storage_format(the_repository,
-						    repo_fmt.ref_storage_format);
+						    repo_fmt.ref_storage_format,
+						    repo_fmt.ref_storage_payload);
 			the_repository->repository_format_worktree_config =
 				repo_fmt.worktree_config;
 			the_repository->repository_format_relative_worktrees =
@@ -2042,7 +2066,8 @@ void check_repository_format(struct repository_format *fmt)
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
 	repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo);
 	repo_set_ref_storage_format(the_repository,
-				    fmt->ref_storage_format);
+				    fmt->ref_storage_format,
+				    fmt->ref_storage_payload);
 	the_repository->repository_format_worktree_config =
 		fmt->worktree_config;
 	the_repository->repository_format_relative_worktrees =
@@ -2643,7 +2668,8 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
-	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format);
+	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
+				    repo_fmt->ref_storage_payload);
 }
 
 int init_db(const char *git_dir, const char *real_git_dir,
diff --git a/setup.h b/setup.h
index ddb9f6701c..093af39e84 100644
--- a/setup.h
+++ b/setup.h
@@ -171,6 +171,7 @@ struct repository_format {
 	int hash_algo;
 	int compat_hash_algo;
 	enum ref_storage_format ref_storage_format;
+	char *ref_storage_payload;
 	int sparse_index;
 	char *work_tree;
 	struct string_list unknown_extensions;
diff --git a/t/meson.build b/t/meson.build
index 459c52a489..11fc5a49ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -210,6 +210,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..9912433b8c
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='Test reference backend URIs'
+
+. ./test-lib.sh
+
+# Run a git command with the provided reference storage. Reset the backend
+# post running the command.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <cmd> is the git subcommand to be run in the repository.
+run_with_uri() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	cmd=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1
+	git -C "$repo" config set extensions.refStorage "$uri" &&
+	git -C "$repo" $cmd &&
+	git -C "$repo" config set extensions.refStorage "$backend"
+}
+
+# Test a repository with a given reference storage by running and comparing
+# 'git refs list' before and after setting the new reference backend. If
+# err_msg is set, expect the command to fail and grep for the provided err_msg.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
+test_refs_backend() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	err_msg=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test -n "$err_msg";
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		test_must_fail git -C "$repo" refs list 2>err &&
+		test_grep "$err_msg" err
+	else
+		git -C "$repo" refs list >expect &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		test_cmp expect actual
+	fi
+}
+
+test_expect_success 'URI is invalid' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable@/home/reftable" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'URI ends with colon' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable:" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "db://.git" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+
+for to_format in $ref_formats
+do
+	if test "$from_format" = "$to_format"
+	then
+		continue
+	fi
+
+
+	for dir in "$(pwd)/repo/.git" "."
+	do
+
+		test_expect_success "read from $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
+			)
+		'
+
+		test_expect_success "write to $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+
+				git refs list >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				git refs list | grep -v "refs/tags/1" >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "with worktree and $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo wt" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				git config set core.repositoryformatversion 1 &&
+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+
+				git worktree add ../wt 2
+			) &&
+
+			git -C repo for-each-ref --include-root-refs >expect &&
+			git -C wt for-each-ref --include-root-refs >expect &&
+			! test_cmp expect actual &&
+
+			git -C wt rev-parse 2 >expect &&
+			git -C wt rev-parse HEAD >actual &&
+			test_cmp expect actual
+		'
+	done # closes dir
+done # closes to_format
+done # closes from_format
+
+test_done

-- 
2.52.0

[PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
the previous commit we extended the 'extensions.refStorage' config to
also support an URI input for reference backend with location.

Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
takes in the same input as the config variable. Having an environment
variable allows us to modify the reference backend and location on the
fly for individual Git commands.

The environment variable also allows usage of alternate reference
directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
the repository when created with the environment variable set.

When initializing the repository with an alternate reference folder,
create the required stubs in the repositories $GIT_DIR. The inverse,
i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
since that would render it unusable. Removal of ref store is only used
when migrating between ref formats and cleanup of the $GIT_DIR doesn't
make sense in such a situation.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   5 ++
 environment.h          |   1 +
 refs.c                 |  23 +++++---
 setup.c                |  55 ++++++++++++++++-
 t/t1423-ref-backend.sh | 157 ++++++++++++++++++++++++++++++++++++++-----------
 5 files changed, 198 insertions(+), 43 deletions(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..66442735ea 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REFERENCE_BACKEND`::
+    Specify which reference backend to be used along with its URI.
+    See `extensions.refStorage` option in linkgit:git-config[1] for more
+    details. Overrides the config variable when used.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 27f657af04..540e0a7f6d 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index 87ef54abd4..6b3883a325 100644
--- a/refs.c
+++ b/refs.c
@@ -2192,16 +2192,21 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
 {
 	int ret = refs->be->create_on_disk(refs, flags, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
-		struct strbuf msg = STRBUF_INIT;
-
-		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
-		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
-		strbuf_release(&msg);
+	if (!ret) {
+		/* Creation of stubs for linked worktrees are handled in the worktree code. */
+		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
+			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
+						 "repository uses alternate refs storage");
+		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+			struct strbuf msg = STRBUF_INIT;
+			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+			strbuf_release(&msg);
+		}
 	}
 
 	return ret;
+
 }
 
 int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
@@ -2216,6 +2221,10 @@ int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 		if (format == REF_STORAGE_FORMAT_FILES)
 			return ret;
 
+		/* Alternate refs backend require stubs in the gitdir. */
+		if (refs->repo->ref_storage_payload)
+			return ret;
+
 		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
 		if (unlink(sb.buf) < 0) {
 			strbuf_addf(err, "could not delete stub HEAD: %s",
diff --git a/setup.c b/setup.c
index d407f3347b..90cb9be578 100644
--- a/setup.c
+++ b/setup.c
@@ -1838,6 +1838,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 	static struct strbuf cwd = STRBUF_INIT;
 	struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT;
 	const char *prefix = NULL;
+	const char *ref_backend_uri;
 	struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
 
 	/*
@@ -1995,6 +1996,25 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
+	/*
+	 * The env variable should override the repository config
+	 * for 'extensions.refStorage'.
+	 */
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+		repo_set_ref_storage_format(the_repository, format, payload);
+
+		free(backend);
+		free(payload);
+	}
+
 	setup_original_cwd();
 
 	strbuf_release(&dir);
@@ -2337,7 +2357,8 @@ void initialize_repository_version(int hash_algo,
 	 * the remote repository's format.
 	 */
 	if (hash_algo != GIT_HASH_SHA1_LEGACY ||
-	    ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	    ref_storage_format != REF_STORAGE_FORMAT_FILES ||
+	    the_repository->ref_storage_payload)
 		target_version = GIT_REPO_VERSION_READ;
 
 	if (hash_algo != GIT_HASH_SHA1_LEGACY && hash_algo != GIT_HASH_UNKNOWN)
@@ -2346,11 +2367,20 @@ void initialize_repository_version(int hash_algo,
 	else if (reinit)
 		repo_config_set_gently(the_repository, "extensions.objectformat", NULL);
 
-	if (ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	if (the_repository->ref_storage_payload) {
+		struct strbuf ref_uri = STRBUF_INIT;
+
+		strbuf_addf(&ref_uri, "%s://%s",
+			    ref_storage_format_to_name(ref_storage_format),
+			    the_repository->ref_storage_payload);
+		repo_config_set(the_repository, "extensions.refstorage", ref_uri.buf);
+		strbuf_release(&ref_uri);
+	} else if (ref_storage_format != REF_STORAGE_FORMAT_FILES) {
 		repo_config_set(the_repository, "extensions.refstorage",
 				ref_storage_format_to_name(ref_storage_format));
-	else if (reinit)
+	} else if (reinit) {
 		repo_config_set_gently(the_repository, "extensions.refstorage", NULL);
+	}
 
 	if (reinit) {
 		struct strbuf config = STRBUF_INIT;
@@ -2623,6 +2653,7 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 		.ignore_repo = 1,
 		.ignore_worktree = 1,
 	};
+	const char *ref_backend_uri;
 	const char *env;
 
 	config_with_options(read_default_format_config, &cfg, NULL, NULL, &opts);
@@ -2668,6 +2699,24 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
+
+
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+
+		repo_fmt->ref_storage_format = format;
+		repo_fmt->ref_storage_payload = payload;
+
+		free(backend);
+	}
+
 	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
 				    repo_fmt->ref_storage_payload);
 }
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
index 9912433b8c..b743c03a59 100755
--- a/t/t1423-ref-backend.sh
+++ b/t/t1423-ref-backend.sh
@@ -11,16 +11,25 @@ test_description='Test reference backend URIs'
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
 #   <cmd> is the git subcommand to be run in the repository.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 run_with_uri() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
 	cmd=$4 &&
+	via=$5 &&
 
-	git -C "$repo" config set core.repositoryformatversion 1
-	git -C "$repo" config set extensions.refStorage "$uri" &&
-	git -C "$repo" $cmd &&
-	git -C "$repo" config set extensions.refStorage "$backend"
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test "$via" = "env"
+	then
+		test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd
+	elif test "$via" = "config"
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		git -C "$repo" $cmd &&
+		git -C "$repo" config set extensions.refStorage "$backend"
+	fi
 }
 
 # Test a repository with a given reference storage by running and comparing
@@ -30,44 +39,57 @@ run_with_uri() {
 #   <repo> is the relative path to the repo to run the command in.
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
 test_refs_backend() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
-	err_msg=$4 &&
+	via=$4 &&
+	err_msg=$5 &&
+
 
-	git -C "$repo" config set core.repositoryformatversion 1 &&
 	if test -n "$err_msg";
 	then
-		git -C "$repo" config set extensions.refStorage "$uri" &&
-		test_must_fail git -C "$repo" refs list 2>err &&
-		test_grep "$err_msg" err
+		if test "$via" = "env"
+		then
+			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
+		elif test "$via" = "config"
+		then
+			git -C "$repo" config set extensions.refStorage "$uri" &&
+			test_must_fail git -C "$repo" refs list 2>err &&
+			test_grep "$err_msg" err
+		fi
 	else
 		git -C "$repo" refs list >expect &&
-		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
 		test_cmp expect actual
 	fi
 }
 
-test_expect_success 'URI is invalid' '
+methods="config env"
+for method in $methods
+do
+
+test_expect_success "$method: URI is invalid" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable@/home/reftable" \
+	test_refs_backend repo files "reftable@/home/reftable" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'URI ends with colon' '
+test_expect_success "$method: URI ends with colon" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable:" \
+	test_refs_backend repo files "reftable:" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'unknown reference backend' '
+test_expect_success "$method: unknown reference backend" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "db://.git" \
+	test_refs_backend repo files "db://.git" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
@@ -86,7 +108,7 @@ do
 	for dir in "$(pwd)/repo/.git" "."
 	do
 
-		test_expect_success "read from $to_format backend, $dir dir" '
+		test_expect_success "$method: read from $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -101,7 +123,7 @@ do
 			)
 		'
 
-		test_expect_success "write to $to_format backend, $dir dir" '
+		test_expect_success "$method: write to $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -113,20 +135,22 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" &&
 
 				git refs list >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"tag -d 1" "$method" &&
 				git refs list >actual &&
 				test_cmp expect actual &&
 
 				git refs list | grep -v "refs/tags/1" >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"refs list" "$method" >actual &&
 				test_cmp expect actual
 			)
 		'
 
-		test_expect_success "with worktree and $to_format backend, $dir dir" '
+		test_expect_success "$method: with worktree and $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo wt" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -138,22 +162,89 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				git config set core.repositoryformatversion 1 &&
-				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
-
-				git worktree add ../wt 2
-			) &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"worktree add ../wt 2" "$method" &&
 
-			git -C repo for-each-ref --include-root-refs >expect &&
-			git -C wt for-each-ref --include-root-refs >expect &&
-			! test_cmp expect actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >expect &&
+				! test_cmp expect actual &&
 
-			git -C wt rev-parse 2 >expect &&
-			git -C wt rev-parse HEAD >actual &&
-			test_cmp expect actual
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse 2" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse HEAD" "$method" >expect &&
+				test_cmp expect actual
+			)
 		'
 	done # closes dir
+
+	test_expect_success "migrating repository to $to_format with alternate refs directory" '
+		test_when_finished "rm -rf repo refdir" &&
+		mkdir refdir &&
+		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
+		(
+			cd repo &&
+
+			test_commit 1 &&
+			test_commit 2 &&
+			test_commit 3 &&
+
+			git refs migrate --ref-format=$to_format &&
+			git refs list >out &&
+			test_grep "refs/tags/1"	out &&
+			test_grep "refs/tags/2"	out &&
+			test_grep "refs/tags/3"	out
+		)
+	'
+
 done # closes to_format
 done # closes from_format
 
+done # closes method
+
+test_expect_success 'initializing repository with alt ref directory' '
+	test_when_finished "rm -rf repo refdir" &&
+	mkdir refdir &&
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
+	(
+		cd repo &&
+
+		git config get extensions.refstorage >expect &&
+		echo $BACKEND >actual &&
+		test_cmp expect actual &&
+
+		test_commit 1 &&
+		test_commit 2 &&
+		test_commit 3 &&
+		git refs list >out &&
+		test_grep "refs/tags/1"	out &&
+		test_grep "refs/tags/2"	out &&
+		test_grep "refs/tags/3"	out
+	)
+'
+
+test_expect_success 'cloning repository with alt ref directory' '
+	test_when_finished "rm -rf source repo refdir" &&
+	mkdir refdir &&
+
+	git init source &&
+	test_commit -C source 1 &&
+	test_commit -C source 2 &&
+	test_commit -C source 3 &&
+
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
+
+	git -C repo config get extensions.refstorage >expect &&
+	echo $BACKEND >actual &&
+	test_cmp expect actual &&
+
+	git -C source for-each-ref refs/tags/ >expect &&
+	git -C repo for-each-ref refs/tags/ >actual &&
+	test_cmp expect actual
+'
+
 test_done

-- 
2.52.0

Re: [PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Patrick Steinhardt]

On Sat, Feb 14, 2026 at 11:34:19PM +0100, Karthik Nayak wrote:
> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
> the previous commit we extended the 'extensions.refStorage' config to
> also support an URI input for reference backend with location.
> 
> Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
> takes in the same input as the config variable. Having an environment
> variable allows us to modify the reference backend and location on the
> fly for individual Git commands.
> 
> The environment variable also allows usage of alternate reference
> directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
> the repository when created with the environment variable set.
> 
> When initializing the repository with an alternate reference folder,
> create the required stubs in the repositories $GIT_DIR. The inverse,
> i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
> since that would render it unusable. Removal of ref store is only used
> when migrating between ref formats and cleanup of the $GIT_DIR doesn't
> make sense in such a situation.

Nice.

> diff --git a/refs.c b/refs.c
> index 87ef54abd4..6b3883a325 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2192,16 +2192,21 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
>  {
>  	int ret = refs->be->create_on_disk(refs, flags, err);
>  
> -	if (!ret &&
> -	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
> -		struct strbuf msg = STRBUF_INIT;
> -
> -		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
> -		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
> -		strbuf_release(&msg);
> +	if (!ret) {
> +		/* Creation of stubs for linked worktrees are handled in the worktree code. */
> +		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
> +			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
> +						 "repository uses alternate refs storage");
> +		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
> +			struct strbuf msg = STRBUF_INIT;
> +			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
> +			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
> +			strbuf_release(&msg);
> +		}
>  	}

Okay, this here is the reason why you've moved the stub creation into
the generic parts. Makes sense.

>  
>  	return ret;
> +
>  }

There's a stray newline here.

> diff --git a/setup.c b/setup.c
> index d407f3347b..90cb9be578 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -2337,7 +2357,8 @@ void initialize_repository_version(int hash_algo,
>  	 * the remote repository's format.
>  	 */
>  	if (hash_algo != GIT_HASH_SHA1_LEGACY ||
> -	    ref_storage_format != REF_STORAGE_FORMAT_FILES)
> +	    ref_storage_format != REF_STORAGE_FORMAT_FILES ||
> +	    the_repository->ref_storage_payload)
>  		target_version = GIT_REPO_VERSION_READ;
>  
>  	if (hash_algo != GIT_HASH_SHA1_LEGACY && hash_algo != GIT_HASH_UNKNOWN)

Right. If we've got a payload during init we know that we must use the
repository extension. Makes sense.

> @@ -2346,11 +2367,20 @@ void initialize_repository_version(int hash_algo,
>  	else if (reinit)
>  		repo_config_set_gently(the_repository, "extensions.objectformat", NULL);
>  
> -	if (ref_storage_format != REF_STORAGE_FORMAT_FILES)
> +	if (the_repository->ref_storage_payload) {
> +		struct strbuf ref_uri = STRBUF_INIT;
> +
> +		strbuf_addf(&ref_uri, "%s://%s",
> +			    ref_storage_format_to_name(ref_storage_format),
> +			    the_repository->ref_storage_payload);
> +		repo_config_set(the_repository, "extensions.refstorage", ref_uri.buf);
> +		strbuf_release(&ref_uri);
> +	} else if (ref_storage_format != REF_STORAGE_FORMAT_FILES) {
>  		repo_config_set(the_repository, "extensions.refstorage",
>  				ref_storage_format_to_name(ref_storage_format));
> -	else if (reinit)
> +	} else if (reinit) {
>  		repo_config_set_gently(the_repository, "extensions.refstorage", NULL);
> +	}
>  
>  	if (reinit) {
>  		struct strbuf config = STRBUF_INIT;

And here we write the payload into the configuration, if we've got one.

> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
> index 9912433b8c..b743c03a59 100755
> --- a/t/t1423-ref-backend.sh
> +++ b/t/t1423-ref-backend.sh
[snip]
> +test_expect_success 'initializing repository with alt ref directory' '
> +	test_when_finished "rm -rf repo refdir" &&
> +	mkdir refdir &&
> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
> +	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
> +	(
> +		cd repo &&
> +
> +		git config get extensions.refstorage >expect &&
> +		echo $BACKEND >actual &&
> +		test_cmp expect actual &&
> +
> +		test_commit 1 &&
> +		test_commit 2 &&
> +		test_commit 3 &&
> +		git refs list >out &&
> +		test_grep "refs/tags/1"	out &&
> +		test_grep "refs/tags/2"	out &&
> +		test_grep "refs/tags/3"	out
> +	)
> +'

Should we also verify that the refdir contains the data and that the
repository only contains stubs?

> +test_expect_success 'cloning repository with alt ref directory' '
> +	test_when_finished "rm -rf source repo refdir" &&
> +	mkdir refdir &&
> +
> +	git init source &&
> +	test_commit -C source 1 &&
> +	test_commit -C source 2 &&
> +	test_commit -C source 3 &&
> +
> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
> +	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
> +
> +	git -C repo config get extensions.refstorage >expect &&
> +	echo $BACKEND >actual &&
> +	test_cmp expect actual &&
> +
> +	git -C source for-each-ref refs/tags/ >expect &&
> +	git -C repo for-each-ref refs/tags/ >actual &&
> +	test_cmp expect actual
> +'
> +

Same question here.

Thanks!

Patrick

Re: [PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3288 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

>> diff --git a/refs.c b/refs.c
>> index 87ef54abd4..6b3883a325 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2192,16 +2192,21 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
>>  {
>>  	int ret = refs->be->create_on_disk(refs, flags, err);
>>
>> -	if (!ret &&
>> -	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>> -		struct strbuf msg = STRBUF_INIT;
>> -
>> -		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>> -		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>> -		strbuf_release(&msg);
>> +	if (!ret) {
>> +		/* Creation of stubs for linked worktrees are handled in the worktree code. */
>> +		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
>> +			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
>> +						 "repository uses alternate refs storage");
>> +		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>> +			struct strbuf msg = STRBUF_INIT;
>> +			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>> +			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>> +			strbuf_release(&msg);
>> +		}
>>  	}
>
> Okay, this here is the reason why you've moved the stub creation into
> the generic parts. Makes sense.
>
>>
>>  	return ret;
>> +
>>  }
>
> There's a stray newline here.
>

Will fix.

[snip]

>> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
>> index 9912433b8c..b743c03a59 100755
>> --- a/t/t1423-ref-backend.sh
>> +++ b/t/t1423-ref-backend.sh
> [snip]
>> +test_expect_success 'initializing repository with alt ref directory' '
>> +	test_when_finished "rm -rf repo refdir" &&
>> +	mkdir refdir &&
>> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
>> +	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
>> +	(
>> +		cd repo &&
>> +
>> +		git config get extensions.refstorage >expect &&
>> +		echo $BACKEND >actual &&
>> +		test_cmp expect actual &&
>> +
>> +		test_commit 1 &&
>> +		test_commit 2 &&
>> +		test_commit 3 &&
>> +		git refs list >out &&
>> +		test_grep "refs/tags/1"	out &&
>> +		test_grep "refs/tags/2"	out &&
>> +		test_grep "refs/tags/3"	out
>> +	)
>> +'
>
> Should we also verify that the refdir contains the data and that the
> repository only contains stubs?
>
>> +test_expect_success 'cloning repository with alt ref directory' '
>> +	test_when_finished "rm -rf source repo refdir" &&
>> +	mkdir refdir &&
>> +
>> +	git init source &&
>> +	test_commit -C source 1 &&
>> +	test_commit -C source 2 &&
>> +	test_commit -C source 3 &&
>> +
>> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
>> +	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
>> +
>> +	git -C repo config get extensions.refstorage >expect &&
>> +	echo $BACKEND >actual &&
>> +	test_cmp expect actual &&
>> +
>> +	git -C source for-each-ref refs/tags/ >expect &&
>> +	git -C repo for-each-ref refs/tags/ >actual &&
>> +	test_cmp expect actual
>> +'
>> +
>
> Same question here.
>
> Thanks!
>
> Patrick

We can verify that the repository only contains stubs, but to verify
that the refdir contains the data, we'd have to add backend specific
logic, I'm not sure that is the way to go.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Patrick Steinhardt]

On Tue, Feb 17, 2026 at 03:32:56AM -0600, Karthik Nayak wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> >> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
> >> index 9912433b8c..b743c03a59 100755
> >> --- a/t/t1423-ref-backend.sh
> >> +++ b/t/t1423-ref-backend.sh
> > [snip]
> >> +test_expect_success 'initializing repository with alt ref directory' '
> >> +	test_when_finished "rm -rf repo refdir" &&
> >> +	mkdir refdir &&
> >> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
> >> +	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
> >> +	(
> >> +		cd repo &&
> >> +
> >> +		git config get extensions.refstorage >expect &&
> >> +		echo $BACKEND >actual &&
> >> +		test_cmp expect actual &&
> >> +
> >> +		test_commit 1 &&
> >> +		test_commit 2 &&
> >> +		test_commit 3 &&
> >> +		git refs list >out &&
> >> +		test_grep "refs/tags/1"	out &&
> >> +		test_grep "refs/tags/2"	out &&
> >> +		test_grep "refs/tags/3"	out
> >> +	)
> >> +'
> >
> > Should we also verify that the refdir contains the data and that the
> > repository only contains stubs?
> >
> >> +test_expect_success 'cloning repository with alt ref directory' '
> >> +	test_when_finished "rm -rf source repo refdir" &&
> >> +	mkdir refdir &&
> >> +
> >> +	git init source &&
> >> +	test_commit -C source 1 &&
> >> +	test_commit -C source 2 &&
> >> +	test_commit -C source 3 &&
> >> +
> >> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
> >> +	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
> >> +
> >> +	git -C repo config get extensions.refstorage >expect &&
> >> +	echo $BACKEND >actual &&
> >> +	test_cmp expect actual &&
> >> +
> >> +	git -C source for-each-ref refs/tags/ >expect &&
> >> +	git -C repo for-each-ref refs/tags/ >actual &&
> >> +	test_cmp expect actual
> >> +'
> >> +
> >
> > Same question here.
> >
> > Thanks!
> >
> > Patrick
> 
> We can verify that the repository only contains stubs, but to verify
> that the refdir contains the data, we'd have to add backend specific
> logic, I'm not sure that is the way to go.

I mean all of this very much is backend-specific, but I think that it's
sensible to at least verify some of the basics here. Otherwise it might
silently do the wrong thing, and we'd never notice because we haven't
been testing for it.

Patrick

Re: [PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> Git allows setting a different object directory via
> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
> the previous commit we extended the 'extensions.refStorage' config to
> also support an URI input for reference backend with location.
>
> Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
> takes in the same input as the config variable. Having an environment
> variable allows us to modify the reference backend and location on the
> fly for individual Git commands.
>
> The environment variable also allows usage of alternate reference
> directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
> the repository when created with the environment variable set.
>
> When initializing the repository with an alternate reference folder,
> create the required stubs in the repositories $GIT_DIR. The inverse,
> i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
> since that would render it unusable. Removal of ref store is only used
> when migrating between ref formats and cleanup of the $GIT_DIR doesn't
> make sense in such a situation.
>
> Helped-by: Jean-Noël Avila <jn.avila@free.fr>
> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
>  Documentation/git.adoc |   5 ++
>  environment.h          |   1 +
>  refs.c                 |  23 +++++---
>  setup.c                |  55 ++++++++++++++++-
>  t/t1423-ref-backend.sh | 157 ++++++++++++++++++++++++++++++++++++++-----------
>  5 files changed, 198 insertions(+), 43 deletions(-)
>
> [snip]
>
> diff --git a/refs.c b/refs.c
> index 87ef54abd4..6b3883a325 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2192,16 +2192,21 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
>  {
>  	int ret = refs->be->create_on_disk(refs, flags, err);
>  
> -	if (!ret &&
> -	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
> -		struct strbuf msg = STRBUF_INIT;
> -
> -		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
> -		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
> -		strbuf_release(&msg);
> +	if (!ret) {
> +		/* Creation of stubs for linked worktrees are handled in the worktree code. */
> +		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
> +			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
> +						 "repository uses alternate refs storage");
> +		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
> +			struct strbuf msg = STRBUF_INIT;
> +			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
> +			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
> +			strbuf_release(&msg);
> +		}
>  	}
>  
>  	return ret;
> +
>  }
>  
>  int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
> @@ -2216,6 +2221,10 @@ int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
>  		if (format == REF_STORAGE_FORMAT_FILES)
>  			return ret;
>  
> +		/* Alternate refs backend require stubs in the gitdir. */

I find this comment rather confusing, you say "require stubs" and you do
an early return. I had to read it more than once to understand. What do
you think about:

 +		/* No stubs required in the alternate refs backend,
 +		 * stubs only should be created in the gitdir. */

> +		if (refs->repo->ref_storage_payload)
> +			return ret;
> +

-- 
Cheers,
Toon

Re: [PATCH v6 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 4003 bytes --]

Toon Claes <toon@iotcl.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> Git allows setting a different object directory via
>> 'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
>> the previous commit we extended the 'extensions.refStorage' config to
>> also support an URI input for reference backend with location.
>>
>> Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
>> takes in the same input as the config variable. Having an environment
>> variable allows us to modify the reference backend and location on the
>> fly for individual Git commands.
>>
>> The environment variable also allows usage of alternate reference
>> directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
>> the repository when created with the environment variable set.
>>
>> When initializing the repository with an alternate reference folder,
>> create the required stubs in the repositories $GIT_DIR. The inverse,
>> i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
>> since that would render it unusable. Removal of ref store is only used
>> when migrating between ref formats and cleanup of the $GIT_DIR doesn't
>> make sense in such a situation.
>>
>> Helped-by: Jean-Noël Avila <jn.avila@free.fr>
>> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
>> ---
>>  Documentation/git.adoc |   5 ++
>>  environment.h          |   1 +
>>  refs.c                 |  23 +++++---
>>  setup.c                |  55 ++++++++++++++++-
>>  t/t1423-ref-backend.sh | 157 ++++++++++++++++++++++++++++++++++++++-----------
>>  5 files changed, 198 insertions(+), 43 deletions(-)
>>
>> [snip]
>>
>> diff --git a/refs.c b/refs.c
>> index 87ef54abd4..6b3883a325 100644
>> --- a/refs.c
>> +++ b/refs.c
>> @@ -2192,16 +2192,21 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
>>  {
>>  	int ret = refs->be->create_on_disk(refs, flags, err);
>>
>> -	if (!ret &&
>> -	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>> -		struct strbuf msg = STRBUF_INIT;
>> -
>> -		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>> -		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>> -		strbuf_release(&msg);
>> +	if (!ret) {
>> +		/* Creation of stubs for linked worktrees are handled in the worktree code. */
>> +		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
>> +			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
>> +						 "repository uses alternate refs storage");
>> +		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
>> +			struct strbuf msg = STRBUF_INIT;
>> +			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
>> +			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
>> +			strbuf_release(&msg);
>> +		}
>>  	}
>>
>>  	return ret;
>> +
>>  }
>>
>>  int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
>> @@ -2216,6 +2221,10 @@ int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
>>  		if (format == REF_STORAGE_FORMAT_FILES)
>>  			return ret;
>>
>> +		/* Alternate refs backend require stubs in the gitdir. */
>
> I find this comment rather confusing, you say "require stubs" and you do
> an early return. I had to read it more than once to understand. What do
> you think about:
>
>  +		/* No stubs required in the alternate refs backend,
>  +		 * stubs only should be created in the gitdir. */
>

But that's exactly what it is, this is the `remove_on_disk()` function,
and for general cases it makes sense to remove the stubs, but when using
alternate refs directories, the stubs shouldn't be removed, cause doing
so will mean the directory is no longer recognized as a gitdir.

>> +		if (refs->repo->ref_storage_payload)
>> +			return ret;
>> +
>
> --
> Cheers,
> Toon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v7 0/6] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:

  - A format name alone (e.g., `reftable` or `files`).

  - A URI format `<format>://<payload>` explicitly specifies both the
    format and payload (e.g., `reftable:///foo/bar`).

We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v7:
- Add more details in the commit messages.
- Cleanup some whitespace.
- Reorder the commits to be group related changes together.
- Add checks for stubs in the tests when creating new repos.
- Link to v6: https://patch.msgid.link/20260214-kn-alternate-ref-dir-v6-0-86a82c77cf59@gmail.com

Changes in v6:
- The biggest change in this version is that we now support using the
  environment variable with 'git-clone(1)' and 'git-init(1)'. In such
  situations, the alternate reference directory is created and the
  config is added to the repository.
- Add a new commit which moves stub creation/removal to the generic
  layer.
- Cleanup logic flow in `refs_compute_filesystem_location()`.
- Add more tests for usage with 'git-clone(1)', 'git-init(1)' and
  migration of repositories using alternate refs backend.
- Fixup documentation, commit messages and typos.
- Link to v5: https://patch.msgid.link/20260209-kn-alternate-ref-dir-v5-0-740899834ceb@gmail.com

Changes in v5:
- Moved around the commits, to ensure that the code to handle the config
  in the backend is first. Previously, we added the config first, which
  meant the commit allowed users to provide a URI but it was simply
  ignore.
- Fix typos and grammar and rename variables.
- Clean up the description and documentation to actually specify
  protocol over location.
- Avoid an extra memory allocation by detaching the strbuf value.
- Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
  backends will store worktree references in 'ref_dir/worktrees/wt_id'
  and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
  relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/config/extensions.adoc |  16 +-
 Documentation/git.adoc               |   5 +
 builtin/clone.c                      |   9 +-
 builtin/worktree.c                   |  34 +++++
 environment.h                        |   1 +
 refs.c                               | 126 +++++++++++++++-
 refs.h                               |  13 ++
 refs/files-backend.c                 |  23 ++-
 refs/packed-backend.c                |   5 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  14 ++
 refs/reftable-backend.c              |  61 ++------
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  96 ++++++++++--
 setup.h                              |   4 +-
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 282 +++++++++++++++++++++++++++++++++++
 18 files changed, 627 insertions(+), 81 deletions(-)

Karthik Nayak (6):
      setup: don't modify repo in `create_reference_database()`
      refs: extract out `refs_create_refdir_stubs()`
      refs: move out stub modification to generic layer
      refs: receive and use the reference storage payload
      refs: allow reference location in refstorage config
      refs: add GIT_REFERENCE_BACKEND to specify reference backend

Range-diff versus v6:

1:  bbc8d0678c ! 1:  d958e2597e setup: don't modify repo in `create_reference_database()`
    @@ Commit message
         dynamically allocated strings, where we would free/allocate the same
         string back into `the_repostiory`.
     
    -    To avoid all this confusion, clean up the function to longer take in and
    -    set the repo's reference storage format.
    +    To avoid all this confusion, clean up the function to no longer take in
    +    and set the repo's reference storage format.
     
         Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
     
2:  4f868d2f4c = 2:  f6583f00b1 refs: extract out `refs_create_refdir_stubs()`
4:  6e4c372402 ! 3:  bffb0ede9d refs: move out stub modification to generic layer
    @@ Commit message
         ensure that the directory can be recognized as a Git repository. This is
         done by calling `refs_create_refdir_stubs()`. Move this to the generic
         layer as this is needed for all backends excluding from the files
    -    backends. In an upcoming commit, we'll also need to extend this logic to
    -    create stubs when using alternate reference directories.
    +    backends. In an upcoming commit where we introduce alternate reference
    +    backend locations, we'll have to also create stubs in the $GIT_DIR
    +    irrespective of the backend being used. This commit builds the base to
    +    add that logic.
     
         Similarly, move the logic for deletion of stubs to the generic layer.
         The files backend recursively calls the remove function of the
    @@ refs.c: void refs_create_refdir_stubs(struct repository *repo, const char *refdi
     -	return refs->be->remove_on_disk(refs, err);
     +	int ret = refs->be->remove_on_disk(refs, err);
     +
    -+	if (!ret) {
    -+		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
    ++	if (!ret &&
    ++	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
     +		struct strbuf sb = STRBUF_INIT;
     +
    -+		/* Backends apart from the files backend create stubs. */
    -+		if (format == REF_STORAGE_FORMAT_FILES)
    -+			return ret;
    -+
     +		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
     +		if (unlink(sb.buf) < 0) {
     +			strbuf_addf(err, "could not delete stub HEAD: %s",
3:  363e36875b ! 4:  a7d4c22861 refs: receive and use the reference storage payload
    @@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)
     +	if (*is_worktree) {
     +		const char *wt_id = strrchr(gitdir, '/');
     +		if (!wt_id)
    -+			BUG("worktree path does not contain slash ");
    ++			BUG("worktree path does not contain slash");
     +		strbuf_addf(refdir, "/worktrees/%s", wt_id + 1);
     +	}
     +
5:  da04b77ed6 = 5:  dca8fc6b77 refs: allow reference location in refstorage config
6:  78b9589da7 ! 6:  d515ab78e3 refs: add GIT_REFERENCE_BACKEND to specify reference backend
    @@ refs.c: int ref_store_create_on_disk(struct ref_store *refs, int flags, struct s
      	}
      
      	return ret;
    -+
    - }
    - 
    - int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
     @@ refs.c: int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
    - 		if (format == REF_STORAGE_FORMAT_FILES)
    - 			return ret;
    + {
    + 	int ret = refs->be->remove_on_disk(refs, err);
    + 
    +-	if (!ret &&
    +-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
    ++	if (!ret) {
    ++		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
    + 		struct strbuf sb = STRBUF_INIT;
      
    ++		/* Backends apart from the files backend create stubs. */
    ++		if (format == REF_STORAGE_FORMAT_FILES)
    ++			return ret;
    ++
     +		/* Alternate refs backend require stubs in the gitdir. */
     +		if (refs->repo->ref_storage_payload)
     +			return ret;
    @@ t/t1423-ref-backend.sh: run_with_uri() {
      }
      
     -test_expect_success 'URI is invalid' '
    ++# Verify that the expected files are present in the gitdir and the refsdir.
    ++# Usage: verify_files_exist <gitdir> <refdir>
    ++#   <gitdir> is the path for the gitdir.
    ++#   <refdir> is the path for the refdir.
    ++verify_files_exist() {
    ++	gitdir=$1 &&
    ++	refdir=$2 &&
    ++
    ++	# verify that the stubs were added to the $GITDIR.
    ++	cat $gitdir/refs/heads >actual &&
    ++	echo "repository uses alternate refs storage" >expect &&
    ++	test_cmp expect actual &&
    ++	cat $gitdir/HEAD >actual &&
    ++	echo "ref: refs/heads/.invalid" >expect &&
    ++	test_cmp expect actual
    ++
    ++	# verify that backend specific files exist.
    ++	case "$GIT_DEFAULT_REF_FORMAT" in
    ++	files)
    ++		test_path_is_dir $refdir/refs/heads &&
    ++		test_path_is_file $refdir/HEAD;;
    ++	reftable)
    ++		test_path_is_dir $refdir/reftable &&
    ++		test_path_is_file $refdir/reftable/tables.list;;
    ++	*)
    ++		BUG "unhandled ref format $GIT_DEFAULT_REF_FORMAT";;
    ++	esac
    ++}
    ++
     +methods="config env"
     +for method in $methods
     +do
    @@ t/t1423-ref-backend.sh: do
     +	mkdir refdir &&
     +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
     +	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
    ++	verify_files_exist repo/.git refdir &&
     +	(
     +		cd repo &&
     +
    @@ t/t1423-ref-backend.sh: do
     +	echo $BACKEND >actual &&
     +	test_cmp expect actual &&
     +
    ++	verify_files_exist repo/.git refdir &&
    ++
     +	git -C source for-each-ref refs/tags/ >expect &&
     +	git -C repo for-each-ref refs/tags/ >actual &&
     +	test_cmp expect actual


base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v7 1/6] setup: don't modify repo in `create_reference_database()`

[Karthik Nayak]

The `create_reference_database()` function is used to create the
reference database during initialization of a repository. The function
calls `repo_set_ref_storage_format()` to set the repositories reference
format. This is an unexpected side-effect of the function. More so
because the function is only called in two locations:

  1. During git-init(1) where the value is propagated from the `struct
     repository_format repo_fmt` value.

  2. During git-clone(1) where the value is propagated from the
     `the_repository` value.

The former is valid, however the flow already calls
`repo_set_ref_storage_format()`, so this effort is simply duplicated.
The latter sets the existing value in `the_repository` back to itself.
While this is okay for now, introduction of more fields in
`repo_set_ref_storage_format()` would cause issues, especially
dynamically allocated strings, where we would free/allocate the same
string back into `the_repostiory`.

To avoid all this confusion, clean up the function to no longer take in
and set the repo's reference storage format.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c | 2 +-
 setup.c         | 7 ++-----
 setup.h         | 3 +--
 3 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index b40cee5968..cd43bb5aa2 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1442,7 +1442,7 @@ int cmd_clone(int argc,
 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
 	repo_set_hash_algo(the_repository, hash_algo);
-	create_reference_database(the_repository->ref_storage_format, NULL, 1);
+	create_reference_database(NULL, 1);
 
 	/*
 	 * Before fetching from the remote, download and install bundle
diff --git a/setup.c b/setup.c
index b723f8b339..1fc9ae3872 100644
--- a/setup.c
+++ b/setup.c
@@ -2359,14 +2359,12 @@ static int is_reinit(void)
 	return ret;
 }
 
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet)
+void create_reference_database(const char *initial_branch, int quiet)
 {
 	struct strbuf err = STRBUF_INIT;
 	char *to_free = NULL;
 	int reinit = is_reinit();
 
-	repo_set_ref_storage_format(the_repository, ref_storage_format);
 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -2701,8 +2699,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 				      &repo_fmt, init_shared_repository);
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
-		create_reference_database(repo_fmt.ref_storage_format,
-					  initial_branch, flags & INIT_DB_QUIET);
+		create_reference_database(initial_branch, flags & INIT_DB_QUIET);
 	create_object_directory();
 
 	if (repo_settings_get_shared_repository(the_repository)) {
diff --git a/setup.h b/setup.h
index d55dcc6608..ddb9f6701c 100644
--- a/setup.h
+++ b/setup.h
@@ -240,8 +240,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 void initialize_repository_version(int hash_algo,
 				   enum ref_storage_format ref_storage_format,
 				   int reinit);
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet);
+void create_reference_database(const char *initial_branch, int quiet);
 
 /*
  * NOTE NOTE NOTE!!

-- 
2.53.GIT

[PATCH v7 2/6] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

For Git to recognize a directory as a Git directory, it requires the
directory to contain:

  1. 'HEAD' file
  2. 'objects/' directory
  3. 'refs/' directory

Here, #1 and #3 are part of the reference storage mechanism,
specifically the files backend. Since then, newer backends such as the
reftable backend have moved to using their own path ('reftable/') for
storing references. But to ensure Git still recognizes the directory as
a Git directory, we create stubs.

There are two locations where we create stubs:

- In 'refs/reftable-backend.c' when creating the reftable backend.
- In 'clone.c' before spawning transport helpers.

In a following commit, we'll add another instance. So instead of
repeating the code, let's extract out this code to
`refs_create_refdir_stubs()` and use it.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c         |  7 +------
 refs.c                  | 23 +++++++++++++++++++++++
 refs.h                  | 13 +++++++++++++
 refs/reftable-backend.c | 14 ++------------
 4 files changed, 39 insertions(+), 18 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index cd43bb5aa2..697c5bb5cb 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1225,12 +1225,7 @@ int cmd_clone(int argc,
 	initialize_repository_version(GIT_HASH_UNKNOWN,
 				      the_repository->ref_storage_format, 1);
 
-	strbuf_addf(&buf, "%s/HEAD", git_dir);
-	write_file(buf.buf, "ref: refs/heads/.invalid");
-
-	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/refs", git_dir);
-	safe_create_dir(the_repository, buf.buf, 1);
+	refs_create_refdir_stubs(the_repository, git_dir, NULL);
 
 	/*
 	 * additional config can be injected with -c, make sure it's included
diff --git a/refs.c b/refs.c
index 627b7f8698..77b93d655b 100644
--- a/refs.c
+++ b/refs.c
@@ -2163,6 +2163,29 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 	return NULL;
 }
 
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_content)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	strbuf_addf(&path, "%s/HEAD", refdir);
+	write_file(path.buf, "ref: refs/heads/.invalid");
+	adjust_shared_perm(repo, path.buf);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/refs", refdir);
+	safe_create_dir(repo, path.buf, 1);
+
+	if (refs_heads_content) {
+		strbuf_reset(&path);
+		strbuf_addf(&path, "%s/refs/heads", refdir);
+		write_file(path.buf, "%s", refs_heads_content);
+		adjust_shared_perm(repo, path.buf);
+	}
+
+	strbuf_release(&path);
+}
+
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
diff --git a/refs.h b/refs.h
index f0abfa1d93..9d8890fdff 100644
--- a/refs.h
+++ b/refs.h
@@ -1427,4 +1427,17 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
+/*
+ * Git only recognizes a directory as a repository if it contains:
+ * - HEAD file
+ * - refs/ folder
+ * While it is necessary within the files backend, newer backends may not
+ * follow the same structure. To go around this, we create stubs as necessary.
+ *
+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file
+ * with the provided message.
+ */
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_msg);
+
 #endif /* REFS_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index fe74af73af..d8651fe779 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,18 +491,8 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	write_file(sb.buf, "ref: refs/heads/.invalid");
-	adjust_shared_perm(the_repository, sb.buf);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	safe_create_dir(the_repository, sb.buf, 1);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	write_file(sb.buf, "this repository uses the reftable format");
-	adjust_shared_perm(the_repository, sb.buf);
+	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
+				 "this repository uses the reftable format");
 
 	strbuf_release(&sb);
 	return 0;

-- 
2.53.GIT

[PATCH v7 3/6] refs: move out stub modification to generic layer

[Karthik Nayak]

When creating the reftable reference backend on disk, we create stubs to
ensure that the directory can be recognized as a Git repository. This is
done by calling `refs_create_refdir_stubs()`. Move this to the generic
layer as this is needed for all backends excluding from the files
backends. In an upcoming commit where we introduce alternate reference
backend locations, we'll have to also create stubs in the $GIT_DIR
irrespective of the backend being used. This commit builds the base to
add that logic.

Similarly, move the logic for deletion of stubs to the generic layer.
The files backend recursively calls the remove function of the
'packed-backend', here skip calling the generic function since that
would try to delete stubs.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 47 +++++++++++++++++++++++++++++++++++++++++++++--
 refs/files-backend.c    |  6 +++++-
 refs/reftable-backend.c | 27 ---------------------------
 3 files changed, 50 insertions(+), 30 deletions(-)

diff --git a/refs.c b/refs.c
index 77b93d655b..c83af63dc5 100644
--- a/refs.c
+++ b/refs.c
@@ -2189,12 +2189,55 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
-	return refs->be->create_on_disk(refs, flags, err);
+	int ret = refs->be->create_on_disk(refs, flags, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf msg = STRBUF_INIT;
+
+		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+		strbuf_release(&msg);
+	}
+
+	return ret;
 }
 
 int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
-	return refs->be->remove_on_disk(refs, err);
+	int ret = refs->be->remove_on_disk(refs, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf sb = STRBUF_INIT;
+
+		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub HEAD: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs/heads", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub heads: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs", refs->gitdir);
+		if (rmdir(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete refs directory: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+
+		strbuf_release(&sb);
+	}
+
+	return ret;
 }
 
 int repo_resolve_gitlink_ref(struct repository *r,
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 240d3c3b26..d3f6423261 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -3700,7 +3700,11 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,
 	if (for_each_root_ref(refs, remove_one_root_ref, &data) < 0)
 		ret = -1;
 
-	if (ref_store_remove_on_disk(refs->packed_ref_store, err) < 0)
+	/*
+	 * Directly access the cleanup functions for packed-refs as the generic function
+	 * would try to clear stubs which isn't required for the files backend.
+	 */
+	if (refs->packed_ref_store->be->remove_on_disk(refs->packed_ref_store, err) < 0)
 		ret = -1;
 
 	strbuf_release(&sb);
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d8651fe779..6ce7f9bb8e 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,9 +491,6 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
-				 "this repository uses the reftable format");
-
 	strbuf_release(&sb);
 	return 0;
 }
@@ -519,30 +516,6 @@ static int reftable_be_remove_on_disk(struct ref_store *ref_store,
 			    strerror(errno));
 		ret = -1;
 	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub HEAD: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub heads: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	if (rmdir(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete refs directory: %s",
-			    strerror(errno));
-		ret = -1;
-	}
 
 	strbuf_release(&sb);
 	return ret;

-- 
2.53.GIT

Re: [PATCH v7 3/6] refs: move out stub modification to generic layer

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> When creating the reftable reference backend on disk, we create stubs to
> ensure that the directory can be recognized as a Git repository. This is
> done by calling `refs_create_refdir_stubs()`. Move this to the generic
> layer as this is needed for all backends excluding from the files
> backends. In an upcoming commit where we introduce alternate reference
> backend locations, we'll have to also create stubs in the $GIT_DIR
> irrespective of the backend being used. This commit builds the base to
> add that logic.
>
> Similarly, move the logic for deletion of stubs to the generic layer.
> The files backend recursively calls the remove function of the
> 'packed-backend', here skip calling the generic function since that
> would try to delete stubs.
>
> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
>  refs.c                  | 47 +++++++++++++++++++++++++++++++++++++++++++++--
>  refs/files-backend.c    |  6 +++++-
>  refs/reftable-backend.c | 27 ---------------------------
>  3 files changed, 50 insertions(+), 30 deletions(-)
>
> diff --git a/refs.c b/refs.c
> index 77b93d655b..c83af63dc5 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -2189,12 +2189,55 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
>  /* backend functions */
>  int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
>  {
> -	return refs->be->create_on_disk(refs, flags, err);
> +	int ret = refs->be->create_on_disk(refs, flags, err);
> +
> +	if (!ret &&
> +	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
> +		struct strbuf msg = STRBUF_INIT;
> +
> +		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
> +		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
> +		strbuf_release(&msg);

To me it would feel nicer to have an early return instead of wrapping
this code inside the if block. But because we only need the strbuf
instead this block, having a block to scope it makes more sense.

Looks good to me.

-- 
Cheers,
Toon

[PATCH v7 4/6] refs: receive and use the reference storage payload

[Karthik Nayak]

An upcoming commit will add support for providing an URI via the
'extensions.refStorage' config. The URI will contain the reference
backend and a corresponding payload. The payload can be then used for
providing an alternate locations for the reference backend.

To prepare for this, modify the existing backends to accept such an
argument when initializing via the 'init()' function. Both the files
and reftable backends will parse the information to be filesystem paths
to store references. Given that no callers pass any payload yet this is
essentially a no-op change for now.

To enable this, provide a 'refs_compute_filesystem_location()' function
which will parse the current 'gitdir' and the 'payload' to provide the
final reference directory and common reference directory (if working in
a linked worktree).

The documentation and tests will be added alongside the extension of the
config variable.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 40 +++++++++++++++++++++++++++++++++++++++-
 refs/files-backend.c    | 17 ++++++++++++-----
 refs/packed-backend.c   |  5 +++++
 refs/packed-backend.h   |  1 +
 refs/refs-internal.h    | 14 ++++++++++++++
 refs/reftable-backend.c | 24 ++++++++++++++----------
 6 files changed, 85 insertions(+), 16 deletions(-)

diff --git a/refs.c b/refs.c
index c83af63dc5..ba2573eb7a 100644
--- a/refs.c
+++ b/refs.c
@@ -5,6 +5,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "abspath.h"
 #include "advice.h"
 #include "config.h"
 #include "environment.h"
@@ -2290,7 +2291,7 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, gitdir, flags);
+	refs = be->init(repo, NULL, gitdir, flags);
 	return refs;
 }
 
@@ -3468,3 +3469,40 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
 		return "unknown failure";
 	}
 }
+
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
+
+	if (!payload) {
+		/*
+		 * We can use the 'gitdir' as the 'refdir' without appending the
+		 * worktree path, as the 'gitdir' here is already the worktree
+		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
+		 */
+		strbuf_addstr(refdir, gitdir);
+		return;
+	}
+
+	if (!is_absolute_path(payload)) {
+		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
+		strbuf_realpath(ref_common_dir, sb.buf, 1);
+	} else {
+		strbuf_realpath(ref_common_dir, payload, 1);
+	}
+
+	strbuf_addbuf(refdir, ref_common_dir);
+
+	if (*is_worktree) {
+		const char *wt_id = strrchr(gitdir, '/');
+		if (!wt_id)
+			BUG("worktree path does not contain slash");
+		strbuf_addf(refdir, "/worktrees/%s", wt_id + 1);
+	}
+
+	strbuf_release(&sb);
+}
diff --git a/refs/files-backend.c b/refs/files-backend.c
index d3f6423261..9cde3ba724 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
  * set of caches.
  */
 static struct ref_store *files_ref_store_init(struct repository *repo,
+					      const char *payload,
 					      const char *gitdir,
 					      unsigned int flags)
 {
 	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
 	struct ref_store *ref_store = (struct ref_store *)refs;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
+	bool is_worktree;
+
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
 
-	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
+	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
 	refs->store_flags = flags;
-	get_common_dir_noenv(&sb, gitdir);
-	refs->gitcommondir = strbuf_detach(&sb, NULL);
+	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
 	refs->packed_ref_store =
-		packed_ref_store_init(repo, refs->gitcommondir, flags);
+		packed_ref_store_init(repo, NULL, refs->gitcommondir, flags);
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
 	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
 
@@ -126,6 +131,8 @@ static struct ref_store *files_ref_store_init(struct repository *repo,
 	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
 			      &refs->gitcommondir);
 
+	strbuf_release(&refdir);
+
 	return ref_store;
 }
 
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 4ea0c12299..e7bb9f10f9 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -211,7 +211,12 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
 	return snapshot->refs->base.repo->hash_algo->hexsz;
 }
 
+/*
+ * Since packed-refs is only stored in the common dir, don't parse the
+ * payload and rely on the files-backend to set 'gitdir' correctly.
+ */
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload UNUSED,
 					const char *gitdir,
 					unsigned int store_flags)
 {
diff --git a/refs/packed-backend.h b/refs/packed-backend.h
index 9481d5e7c2..2c2377a356 100644
--- a/refs/packed-backend.h
+++ b/refs/packed-backend.h
@@ -14,6 +14,7 @@ struct ref_transaction;
  */
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload,
 					const char *gitdir,
 					unsigned int store_flags);
 
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index c7d2a6e50b..9a635f4e6c 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -389,6 +389,7 @@ struct ref_store;
  * the ref_store and to record the ref_store for later lookup.
  */
 typedef struct ref_store *ref_store_init_fn(struct repository *repo,
+					    const char *payload,
 					    const char *gitdir,
 					    unsigned int flags);
 /*
@@ -666,4 +667,17 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 					  unsigned int initial_transaction,
 					  struct strbuf *err);
 
+/*
+ * Given a gitdir and the reference storage payload provided, retrieve the
+ * 'refdir' and 'ref_common_dir'. The former is where references should be
+ * stored for the current worktree, the latter is the common reference
+ * directory if working with a linked worktree. If working with the main
+ * worktree, both values will be the same.
+ *
+ * This is used by backends that store store files in the repository directly.
+ */
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir);
+
 #endif /* REFS_REFS_INTERNAL_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 6ce7f9bb8e..0e220d6bb5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -372,18 +372,24 @@ static int reftable_be_fsync(int fd)
 }
 
 static struct ref_store *reftable_be_init(struct repository *repo,
+					  const char *payload,
 					  const char *gitdir,
 					  unsigned int store_flags)
 {
 	struct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
 	struct strbuf path = STRBUF_INIT;
-	int is_worktree;
+	bool is_worktree;
 	mode_t mask;
 
 	mask = umask(0);
 	umask(mask);
 
-	base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
+
+	base_ref_store_init(&refs->base, repo, refdir.buf, &refs_be_reftable);
 	strmap_init(&refs->worktree_backends);
 	refs->store_flags = store_flags;
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
@@ -419,14 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	/*
 	 * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
 	 * This stack contains both the shared and the main worktree refs.
-	 *
-	 * Note that we don't try to resolve the path in case we have a
-	 * worktree because `get_common_dir_noenv()` already does it for us.
 	 */
-	is_worktree = get_common_dir_noenv(&path, gitdir);
+	strbuf_addbuf(&path, &ref_common_dir);
 	if (!is_worktree) {
 		strbuf_reset(&path);
-		strbuf_realpath(&path, gitdir, 0);
+		strbuf_realpath(&path, ref_common_dir.buf, 0);
 	}
 	strbuf_addstr(&path, "/reftable");
 	refs->err = reftable_backend_init(&refs->main_backend, path.buf,
@@ -443,10 +446,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	 * do it efficiently.
 	 */
 	if (is_worktree) {
-		strbuf_reset(&path);
-		strbuf_addf(&path, "%s/reftable", gitdir);
+		strbuf_addstr(&refdir, "/reftable");
 
-		refs->err = reftable_backend_init(&refs->worktree_backend, path.buf,
+		refs->err = reftable_backend_init(&refs->worktree_backend, refdir.buf,
 						  &refs->write_options);
 		if (refs->err)
 			goto done;
@@ -456,6 +458,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 
 done:
 	assert(refs->err != REFTABLE_API_ERROR);
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
 	strbuf_release(&path);
 	return &refs->base;
 }

-- 
2.53.GIT

Re: [PATCH v7 4/6] refs: receive and use the reference storage payload

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> diff --git a/refs/refs-internal.h b/refs/refs-internal.h
> index c7d2a6e50b..9a635f4e6c 100644
> --- a/refs/refs-internal.h
> +++ b/refs/refs-internal.h
> @@ -666,4 +667,17 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
>  					  unsigned int initial_transaction,
>  					  struct strbuf *err);
>  
> +/*
> + * Given a gitdir and the reference storage payload provided, retrieve the
> + * 'refdir' and 'ref_common_dir'. The former is where references should be
> + * stored for the current worktree, the latter is the common reference
> + * directory if working with a linked worktree. If working with the main
> + * worktree, both values will be the same.
> + *
> + * This is used by backends that store store files in the repository directly.

s/store store/store/

Also, shouldn't it be this?

* This is used by backends that store references in the repository directly.

-- 
Cheers,
Toon

Re: [PATCH v7 4/6] refs: receive and use the reference storage payload

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

Toon Claes <toon@iotcl.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> diff --git a/refs/refs-internal.h b/refs/refs-internal.h
>> index c7d2a6e50b..9a635f4e6c 100644
>> --- a/refs/refs-internal.h
>> +++ b/refs/refs-internal.h
>> @@ -666,4 +667,17 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
>>  					  unsigned int initial_transaction,
>>  					  struct strbuf *err);
>>
>> +/*
>> + * Given a gitdir and the reference storage payload provided, retrieve the
>> + * 'refdir' and 'ref_common_dir'. The former is where references should be
>> + * stored for the current worktree, the latter is the common reference
>> + * directory if working with a linked worktree. If working with the main
>> + * worktree, both values will be the same.
>> + *
>> + * This is used by backends that store store files in the repository directly.
>
> s/store store/store/
>
> Also, shouldn't it be this?
>
> * This is used by backends that store references in the repository directly.
>
> --
> Cheers,
> Toon

I mean, it also stores the files themselves in the repository, but yours
is clearer, will change!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v7 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

The 'extensions.refStorage' config is used to specify the reference
backend for a given repository. Both the 'files' and 'reftable' backends
utilize the $GIT_DIR as the reference folder by default in
`get_main_ref_store()`.

Since the reference backends are pluggable, this means that they could
work with out-of-tree reference directories too. Extend the 'refStorage'
config to also support taking an URI input, where users can specify the
reference backend and the location.

Add the required changes to obtain and propagate this value to the
individual backends. Add the necessary documentation and tests.

Traditionally, for linked worktrees, references were stored in the
'$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
storage path, it doesn't make sense to store the main worktree
references in the new path, and the linked worktree references in the
$GIT_DIR. So, let's store linked worktree references in
'$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'. To do this, create the
necessary files and folders while also adding stubs in the $GIT_DIR path
to ensure that it is still considered a Git directory.

Ideally, we would want to pass in a `struct worktree *` to individual
backends, instead of passing the `gitdir`. This allows them to handle
worktree specific logic. Currently, that is not possible since the
worktree code is:

  - Tied to using the global `the_repository` variable.

  - Is not setup before the reference database during initialization of
    the repository.

Add a TODO in 'refs.c' to ensure we can eventually make that change.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/config/extensions.adoc |  16 +++-
 builtin/worktree.c                   |  34 ++++++++
 refs.c                               |   6 +-
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  34 +++++++-
 setup.h                              |   1 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
 9 files changed, 259 insertions(+), 9 deletions(-)

diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
index 532456644b..3e51da36d3 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
 `core.repositoryFormatVersion` setting.
 
 refStorage:::
-	Specify the ref storage format to use. The acceptable values are:
+	Specify the ref storage format and a corresponding payload. The value
+	can be either a format name or a URI:
 +
 --
+* A format name alone (e.g., `reftable` or `files`).
+
+* A URI format `<format>://<payload>` explicitly specifies both the
+  format and payload (e.g., `reftable:///foo/bar`).
+
+Supported format names are:
++
 include::../ref-storage-format.adoc[]
++
+The payload is passed directly to the reference backend. For the files and
+reftable backends, this must be a filesystem path where the references will
+be stored. Defaulting to the commondir when no payload is provided. Relative
+paths are resolved relative to the $GIT_DIR. Future backends may support
+other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +
 Note that this setting should only be set by linkgit:git-init[1] or
diff --git a/builtin/worktree.c b/builtin/worktree.c
index fbdaf2eb2e..293e808379 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
 	return run_command(&cp);
 }
 
+/*
+ * References for worktrees are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
+ * But when using alternate reference directories, we want to store the worktree
+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
+ *
+ * Create the necessary folder structure to facilitate the same. But to ensure
+ * that the former path is still considered a Git directory, add stubs.
+ */
+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	char *path;
+
+	path = wt->repo->ref_storage_payload;
+	if (!path)
+		return;
+
+	if (!is_absolute_path(path))
+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
+
+	strbuf_addf(&sb, "%s/worktrees", path);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_addf(&sb, "/%s", wt->id);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_reset(&sb);
+
+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
+		    path, wt->id);
+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
+
+	strbuf_release(&sb);
+}
+
 static int add_worktree(const char *path, const char *refname,
 			const struct add_opts *opts)
 {
@@ -518,6 +551,7 @@ static int add_worktree(const char *path, const char *refname,
 		ret = error(_("could not find created worktree '%s'"), name);
 		goto done;
 	}
+	setup_alternate_ref_dir(wt, sb_repo.buf);
 	wt_refs = get_worktree_ref_store(wt);
 
 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
diff --git a/refs.c b/refs.c
index ba2573eb7a..ef1902e85c 100644
--- a/refs.c
+++ b/refs.c
@@ -2291,7 +2291,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, NULL, gitdir, flags);
+	/*
+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
+	 * allow the backend to handle how it wants to deal with worktrees.
+	 */
+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
 	return refs;
 }
 
diff --git a/repository.c b/repository.c
index c7e75215ac..9815f081ef 100644
--- a/repository.c
+++ b/repository.c
@@ -193,9 +193,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
 }
 
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format)
+				 enum ref_storage_format format,
+				 const char *payload)
 {
 	repo->ref_storage_format = format;
+	free(repo->ref_storage_payload);
+	repo->ref_storage_payload = xstrdup_or_null(payload);
 }
 
 /*
@@ -277,7 +280,8 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 	repo_set_compat_hash_algo(repo, format.compat_hash_algo);
-	repo_set_ref_storage_format(repo, format.ref_storage_format);
+	repo_set_ref_storage_format(repo, format.ref_storage_format,
+				    format.ref_storage_payload);
 	repo->repository_format_worktree_config = format.worktree_config;
 	repo->repository_format_relative_worktrees = format.relative_worktrees;
 	repo->repository_format_precious_objects = format.precious_objects;
@@ -369,6 +373,7 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
+	FREE_AND_NULL(repo->ref_storage_payload);
 
 	odb_free(repo->objects);
 	repo->objects = NULL;
diff --git a/repository.h b/repository.h
index 6063c4b846..95e2333bad 100644
--- a/repository.h
+++ b/repository.h
@@ -150,6 +150,11 @@ struct repository {
 
 	/* Repository's reference storage format, as serialized on disk. */
 	enum ref_storage_format ref_storage_format;
+	/*
+	 * Reference storage information as needed for the backend. This contains
+	 * only the payload from the reference URI without the schema.
+	 */
+	char *ref_storage_payload;
 
 	/* A unique-id for tracing purposes. */
 	int trace2_repo_id;
@@ -204,7 +209,8 @@ void repo_set_worktree(struct repository *repo, const char *path);
 void repo_set_hash_algo(struct repository *repo, int algo);
 void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format);
+				 enum ref_storage_format format,
+				 const char *payload);
 void initialize_repository(struct repository *repo);
 RESULT_MUST_BE_USED
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index 1fc9ae3872..d407f3347b 100644
--- a/setup.c
+++ b/setup.c
@@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
 		return EXTENSION_UNKNOWN;
 }
 
+static void parse_reference_uri(const char *value, char **format,
+				char **payload)
+{
+	const char *schema_end;
+
+	schema_end = strstr(value, "://");
+	if (!schema_end) {
+		*format = xstrdup(value);
+		*payload = NULL;
+	} else {
+		*format = xstrndup(value, schema_end - value);
+		*payload = xstrdup_or_null(schema_end + 3);
+	}
+}
+
 /*
  * Record any new extensions in this function.
  */
@@ -674,10 +689,17 @@ static enum extension_result handle_extension(const char *var,
 		return EXTENSION_OK;
 	} else if (!strcmp(ext, "refstorage")) {
 		unsigned int format;
+		char *format_str;
 
 		if (!value)
 			return config_error_nonbool(var);
-		format = ref_storage_format_by_name(value);
+
+		parse_reference_uri(value, &format_str,
+				    &data->ref_storage_payload);
+
+		format = ref_storage_format_by_name(format_str);
+		free(format_str);
+
 		if (format == REF_STORAGE_FORMAT_UNKNOWN)
 			return error(_("invalid value for '%s': '%s'"),
 				     "extensions.refstorage", value);
@@ -850,6 +872,7 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
 	free(format->partial_clone);
+	free(format->ref_storage_payload);
 	init_repository_format(format);
 }
 
@@ -1942,7 +1965,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			repo_set_compat_hash_algo(the_repository,
 						  repo_fmt.compat_hash_algo);
 			repo_set_ref_storage_format(the_repository,
-						    repo_fmt.ref_storage_format);
+						    repo_fmt.ref_storage_format,
+						    repo_fmt.ref_storage_payload);
 			the_repository->repository_format_worktree_config =
 				repo_fmt.worktree_config;
 			the_repository->repository_format_relative_worktrees =
@@ -2042,7 +2066,8 @@ void check_repository_format(struct repository_format *fmt)
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
 	repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo);
 	repo_set_ref_storage_format(the_repository,
-				    fmt->ref_storage_format);
+				    fmt->ref_storage_format,
+				    fmt->ref_storage_payload);
 	the_repository->repository_format_worktree_config =
 		fmt->worktree_config;
 	the_repository->repository_format_relative_worktrees =
@@ -2643,7 +2668,8 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
-	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format);
+	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
+				    repo_fmt->ref_storage_payload);
 }
 
 int init_db(const char *git_dir, const char *real_git_dir,
diff --git a/setup.h b/setup.h
index ddb9f6701c..093af39e84 100644
--- a/setup.h
+++ b/setup.h
@@ -171,6 +171,7 @@ struct repository_format {
 	int hash_algo;
 	int compat_hash_algo;
 	enum ref_storage_format ref_storage_format;
+	char *ref_storage_payload;
 	int sparse_index;
 	char *work_tree;
 	struct string_list unknown_extensions;
diff --git a/t/meson.build b/t/meson.build
index 459c52a489..11fc5a49ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -210,6 +210,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..9912433b8c
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='Test reference backend URIs'
+
+. ./test-lib.sh
+
+# Run a git command with the provided reference storage. Reset the backend
+# post running the command.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <cmd> is the git subcommand to be run in the repository.
+run_with_uri() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	cmd=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1
+	git -C "$repo" config set extensions.refStorage "$uri" &&
+	git -C "$repo" $cmd &&
+	git -C "$repo" config set extensions.refStorage "$backend"
+}
+
+# Test a repository with a given reference storage by running and comparing
+# 'git refs list' before and after setting the new reference backend. If
+# err_msg is set, expect the command to fail and grep for the provided err_msg.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
+test_refs_backend() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	err_msg=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test -n "$err_msg";
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		test_must_fail git -C "$repo" refs list 2>err &&
+		test_grep "$err_msg" err
+	else
+		git -C "$repo" refs list >expect &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		test_cmp expect actual
+	fi
+}
+
+test_expect_success 'URI is invalid' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable@/home/reftable" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'URI ends with colon' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable:" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "db://.git" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+
+for to_format in $ref_formats
+do
+	if test "$from_format" = "$to_format"
+	then
+		continue
+	fi
+
+
+	for dir in "$(pwd)/repo/.git" "."
+	do
+
+		test_expect_success "read from $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
+			)
+		'
+
+		test_expect_success "write to $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+
+				git refs list >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				git refs list | grep -v "refs/tags/1" >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "with worktree and $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo wt" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				git config set core.repositoryformatversion 1 &&
+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+
+				git worktree add ../wt 2
+			) &&
+
+			git -C repo for-each-ref --include-root-refs >expect &&
+			git -C wt for-each-ref --include-root-refs >expect &&
+			! test_cmp expect actual &&
+
+			git -C wt rev-parse 2 >expect &&
+			git -C wt rev-parse HEAD >actual &&
+			test_cmp expect actual
+		'
+	done # closes dir
+done # closes to_format
+done # closes from_format
+
+test_done

-- 
2.53.GIT

Re: [PATCH v7 5/6] refs: allow reference location in refstorage config

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> diff --git a/setup.c b/setup.c
> index 1fc9ae3872..d407f3347b 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
>  		return EXTENSION_UNKNOWN;
>  }
>  
> +static void parse_reference_uri(const char *value, char **format,
> +				char **payload)
> +{
> +	const char *schema_end;
> +
> +	schema_end = strstr(value, "://");
> +	if (!schema_end) {
> +		*format = xstrdup(value);
> +		*payload = NULL;
> +	} else {
> +		*format = xstrndup(value, schema_end - value);
> +		*payload = xstrdup_or_null(schema_end + 3);

Also here, why did you put the negated condition in the if clause?

> +	}
> +}
> +

-- 
Cheers,
Toon

Re: [PATCH v7 5/6] refs: allow reference location in refstorage config

[Junio C Hamano]

Toon Claes <toon@iotcl.com> writes:

>> +static void parse_reference_uri(const char *value, char **format,
>> +				char **payload)
>> +{
>> +	const char *schema_end;
>> +
>> +	schema_end = strstr(value, "://");
>> +	if (!schema_end) {
>> +		*format = xstrdup(value);
>> +		*payload = NULL;
>> +	} else {
>> +		*format = xstrndup(value, schema_end - value);
>> +		*payload = xstrdup_or_null(schema_end + 3);
>
> Also here, why did you put the negated condition in the if clause?

Hmph, would it make it easier to follow if you swap them?

	if (schema_end) {
		*format = xstrndup(value, schema_end - value);
		*payload = xstrdup_or_null(schema_end + 3);
	} else {
		*format = xstrdup(value);
		*payload = NULL;
	}

Maybe it is just me, but I often find it easier to follow if the
case that require shorter and/or simpler body, or the case that is
narrower (e.g., error condition), comes first before the main logic.
It is in line with preferring an early return on a more specific
condition.  It frees readers from having to worry about these cases
early and let them concentrate on what is expected to usually happen
in the code.

In this particular case, I do not know which one I would prefer,
though.

Thanks.

Re: [PATCH v7 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1546 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Toon Claes <toon@iotcl.com> writes:
>
>>> +static void parse_reference_uri(const char *value, char **format,
>>> +				char **payload)
>>> +{
>>> +	const char *schema_end;
>>> +
>>> +	schema_end = strstr(value, "://");
>>> +	if (!schema_end) {
>>> +		*format = xstrdup(value);
>>> +		*payload = NULL;
>>> +	} else {
>>> +		*format = xstrndup(value, schema_end - value);
>>> +		*payload = xstrdup_or_null(schema_end + 3);
>>
>> Also here, why did you put the negated condition in the if clause?
>
> Hmph, would it make it easier to follow if you swap them?
>
> 	if (schema_end) {
> 		*format = xstrndup(value, schema_end - value);
> 		*payload = xstrdup_or_null(schema_end + 3);
> 	} else {
> 		*format = xstrdup(value);
> 		*payload = NULL;
> 	}
>
> Maybe it is just me, but I often find it easier to follow if the
> case that require shorter and/or simpler body, or the case that is
> narrower (e.g., error condition), comes first before the main logic.
> It is in line with preferring an early return on a more specific
> condition.  It frees readers from having to worry about these cases
> early and let them concentrate on what is expected to usually happen
> in the code.
>
> In this particular case, I do not know which one I would prefer,
> though.
>
> Thanks.

Kinda similar thought process. Since the URI format is new, the most
likely case here is that 'strstr' will not find a match. That's why the
negative case is first.

But I'd be happy to change, if others feel differently.

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v7 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
the previous commit we extended the 'extensions.refStorage' config to
also support an URI input for reference backend with location.

Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
takes in the same input as the config variable. Having an environment
variable allows us to modify the reference backend and location on the
fly for individual Git commands.

The environment variable also allows usage of alternate reference
directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
the repository when created with the environment variable set.

When initializing the repository with an alternate reference folder,
create the required stubs in the repositories $GIT_DIR. The inverse,
i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
since that would render it unusable. Removal of ref store is only used
when migrating between ref formats and cleanup of the $GIT_DIR doesn't
make sense in such a situation.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   5 ++
 environment.h          |   1 +
 refs.c                 |  30 +++++---
 setup.c                |  55 +++++++++++++-
 t/t1423-ref-backend.sh | 189 ++++++++++++++++++++++++++++++++++++++++---------
 5 files changed, 235 insertions(+), 45 deletions(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..66442735ea 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REFERENCE_BACKEND`::
+    Specify which reference backend to be used along with its URI.
+    See `extensions.refStorage` option in linkgit:git-config[1] for more
+    details. Overrides the config variable when used.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 27f657af04..540e0a7f6d 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index ef1902e85c..a700a66f08 100644
--- a/refs.c
+++ b/refs.c
@@ -2192,13 +2192,17 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
 {
 	int ret = refs->be->create_on_disk(refs, flags, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
-		struct strbuf msg = STRBUF_INIT;
-
-		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
-		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
-		strbuf_release(&msg);
+	if (!ret) {
+		/* Creation of stubs for linked worktrees are handled in the worktree code. */
+		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
+			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
+						 "repository uses alternate refs storage");
+		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+			struct strbuf msg = STRBUF_INIT;
+			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+			strbuf_release(&msg);
+		}
 	}
 
 	return ret;
@@ -2208,10 +2212,18 @@ int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
 	int ret = refs->be->remove_on_disk(refs, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+	if (!ret) {
+		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
 		struct strbuf sb = STRBUF_INIT;
 
+		/* Backends apart from the files backend create stubs. */
+		if (format == REF_STORAGE_FORMAT_FILES)
+			return ret;
+
+		/* Alternate refs backend require stubs in the gitdir. */
+		if (refs->repo->ref_storage_payload)
+			return ret;
+
 		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
 		if (unlink(sb.buf) < 0) {
 			strbuf_addf(err, "could not delete stub HEAD: %s",
diff --git a/setup.c b/setup.c
index d407f3347b..90cb9be578 100644
--- a/setup.c
+++ b/setup.c
@@ -1838,6 +1838,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 	static struct strbuf cwd = STRBUF_INIT;
 	struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT;
 	const char *prefix = NULL;
+	const char *ref_backend_uri;
 	struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
 
 	/*
@@ -1995,6 +1996,25 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
+	/*
+	 * The env variable should override the repository config
+	 * for 'extensions.refStorage'.
+	 */
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+		repo_set_ref_storage_format(the_repository, format, payload);
+
+		free(backend);
+		free(payload);
+	}
+
 	setup_original_cwd();
 
 	strbuf_release(&dir);
@@ -2337,7 +2357,8 @@ void initialize_repository_version(int hash_algo,
 	 * the remote repository's format.
 	 */
 	if (hash_algo != GIT_HASH_SHA1_LEGACY ||
-	    ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	    ref_storage_format != REF_STORAGE_FORMAT_FILES ||
+	    the_repository->ref_storage_payload)
 		target_version = GIT_REPO_VERSION_READ;
 
 	if (hash_algo != GIT_HASH_SHA1_LEGACY && hash_algo != GIT_HASH_UNKNOWN)
@@ -2346,11 +2367,20 @@ void initialize_repository_version(int hash_algo,
 	else if (reinit)
 		repo_config_set_gently(the_repository, "extensions.objectformat", NULL);
 
-	if (ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	if (the_repository->ref_storage_payload) {
+		struct strbuf ref_uri = STRBUF_INIT;
+
+		strbuf_addf(&ref_uri, "%s://%s",
+			    ref_storage_format_to_name(ref_storage_format),
+			    the_repository->ref_storage_payload);
+		repo_config_set(the_repository, "extensions.refstorage", ref_uri.buf);
+		strbuf_release(&ref_uri);
+	} else if (ref_storage_format != REF_STORAGE_FORMAT_FILES) {
 		repo_config_set(the_repository, "extensions.refstorage",
 				ref_storage_format_to_name(ref_storage_format));
-	else if (reinit)
+	} else if (reinit) {
 		repo_config_set_gently(the_repository, "extensions.refstorage", NULL);
+	}
 
 	if (reinit) {
 		struct strbuf config = STRBUF_INIT;
@@ -2623,6 +2653,7 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 		.ignore_repo = 1,
 		.ignore_worktree = 1,
 	};
+	const char *ref_backend_uri;
 	const char *env;
 
 	config_with_options(read_default_format_config, &cfg, NULL, NULL, &opts);
@@ -2668,6 +2699,24 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
+
+
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+
+		repo_fmt->ref_storage_format = format;
+		repo_fmt->ref_storage_payload = payload;
+
+		free(backend);
+	}
+
 	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
 				    repo_fmt->ref_storage_payload);
 }
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
index 9912433b8c..d69aea3f7f 100755
--- a/t/t1423-ref-backend.sh
+++ b/t/t1423-ref-backend.sh
@@ -11,16 +11,25 @@ test_description='Test reference backend URIs'
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
 #   <cmd> is the git subcommand to be run in the repository.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 run_with_uri() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
 	cmd=$4 &&
+	via=$5 &&
 
-	git -C "$repo" config set core.repositoryformatversion 1
-	git -C "$repo" config set extensions.refStorage "$uri" &&
-	git -C "$repo" $cmd &&
-	git -C "$repo" config set extensions.refStorage "$backend"
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test "$via" = "env"
+	then
+		test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd
+	elif test "$via" = "config"
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		git -C "$repo" $cmd &&
+		git -C "$repo" config set extensions.refStorage "$backend"
+	fi
 }
 
 # Test a repository with a given reference storage by running and comparing
@@ -30,44 +39,86 @@ run_with_uri() {
 #   <repo> is the relative path to the repo to run the command in.
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
 test_refs_backend() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
-	err_msg=$4 &&
+	via=$4 &&
+	err_msg=$5 &&
+
 
-	git -C "$repo" config set core.repositoryformatversion 1 &&
 	if test -n "$err_msg";
 	then
-		git -C "$repo" config set extensions.refStorage "$uri" &&
-		test_must_fail git -C "$repo" refs list 2>err &&
-		test_grep "$err_msg" err
+		if test "$via" = "env"
+		then
+			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
+		elif test "$via" = "config"
+		then
+			git -C "$repo" config set extensions.refStorage "$uri" &&
+			test_must_fail git -C "$repo" refs list 2>err &&
+			test_grep "$err_msg" err
+		fi
 	else
 		git -C "$repo" refs list >expect &&
-		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
 		test_cmp expect actual
 	fi
 }
 
-test_expect_success 'URI is invalid' '
+# Verify that the expected files are present in the gitdir and the refsdir.
+# Usage: verify_files_exist <gitdir> <refdir>
+#   <gitdir> is the path for the gitdir.
+#   <refdir> is the path for the refdir.
+verify_files_exist() {
+	gitdir=$1 &&
+	refdir=$2 &&
+
+	# verify that the stubs were added to the $GITDIR.
+	cat $gitdir/refs/heads >actual &&
+	echo "repository uses alternate refs storage" >expect &&
+	test_cmp expect actual &&
+	cat $gitdir/HEAD >actual &&
+	echo "ref: refs/heads/.invalid" >expect &&
+	test_cmp expect actual
+
+	# verify that backend specific files exist.
+	case "$GIT_DEFAULT_REF_FORMAT" in
+	files)
+		test_path_is_dir $refdir/refs/heads &&
+		test_path_is_file $refdir/HEAD;;
+	reftable)
+		test_path_is_dir $refdir/reftable &&
+		test_path_is_file $refdir/reftable/tables.list;;
+	*)
+		BUG "unhandled ref format $GIT_DEFAULT_REF_FORMAT";;
+	esac
+}
+
+methods="config env"
+for method in $methods
+do
+
+test_expect_success "$method: URI is invalid" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable@/home/reftable" \
+	test_refs_backend repo files "reftable@/home/reftable" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'URI ends with colon' '
+test_expect_success "$method: URI ends with colon" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable:" \
+	test_refs_backend repo files "reftable:" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'unknown reference backend' '
+test_expect_success "$method: unknown reference backend" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "db://.git" \
+	test_refs_backend repo files "db://.git" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
@@ -86,7 +137,7 @@ do
 	for dir in "$(pwd)/repo/.git" "."
 	do
 
-		test_expect_success "read from $to_format backend, $dir dir" '
+		test_expect_success "$method: read from $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -101,7 +152,7 @@ do
 			)
 		'
 
-		test_expect_success "write to $to_format backend, $dir dir" '
+		test_expect_success "$method: write to $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -113,20 +164,22 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" &&
 
 				git refs list >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"tag -d 1" "$method" &&
 				git refs list >actual &&
 				test_cmp expect actual &&
 
 				git refs list | grep -v "refs/tags/1" >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"refs list" "$method" >actual &&
 				test_cmp expect actual
 			)
 		'
 
-		test_expect_success "with worktree and $to_format backend, $dir dir" '
+		test_expect_success "$method: with worktree and $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo wt" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -138,22 +191,92 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				git config set core.repositoryformatversion 1 &&
-				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
-
-				git worktree add ../wt 2
-			) &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"worktree add ../wt 2" "$method" &&
 
-			git -C repo for-each-ref --include-root-refs >expect &&
-			git -C wt for-each-ref --include-root-refs >expect &&
-			! test_cmp expect actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >expect &&
+				! test_cmp expect actual &&
 
-			git -C wt rev-parse 2 >expect &&
-			git -C wt rev-parse HEAD >actual &&
-			test_cmp expect actual
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse 2" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse HEAD" "$method" >expect &&
+				test_cmp expect actual
+			)
 		'
 	done # closes dir
+
+	test_expect_success "migrating repository to $to_format with alternate refs directory" '
+		test_when_finished "rm -rf repo refdir" &&
+		mkdir refdir &&
+		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
+		(
+			cd repo &&
+
+			test_commit 1 &&
+			test_commit 2 &&
+			test_commit 3 &&
+
+			git refs migrate --ref-format=$to_format &&
+			git refs list >out &&
+			test_grep "refs/tags/1"	out &&
+			test_grep "refs/tags/2"	out &&
+			test_grep "refs/tags/3"	out
+		)
+	'
+
 done # closes to_format
 done # closes from_format
 
+done # closes method
+
+test_expect_success 'initializing repository with alt ref directory' '
+	test_when_finished "rm -rf repo refdir" &&
+	mkdir refdir &&
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
+	verify_files_exist repo/.git refdir &&
+	(
+		cd repo &&
+
+		git config get extensions.refstorage >expect &&
+		echo $BACKEND >actual &&
+		test_cmp expect actual &&
+
+		test_commit 1 &&
+		test_commit 2 &&
+		test_commit 3 &&
+		git refs list >out &&
+		test_grep "refs/tags/1"	out &&
+		test_grep "refs/tags/2"	out &&
+		test_grep "refs/tags/3"	out
+	)
+'
+
+test_expect_success 'cloning repository with alt ref directory' '
+	test_when_finished "rm -rf source repo refdir" &&
+	mkdir refdir &&
+
+	git init source &&
+	test_commit -C source 1 &&
+	test_commit -C source 2 &&
+	test_commit -C source 3 &&
+
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
+
+	git -C repo config get extensions.refstorage >expect &&
+	echo $BACKEND >actual &&
+	test_cmp expect actual &&
+
+	verify_files_exist repo/.git refdir &&
+
+	git -C source for-each-ref refs/tags/ >expect &&
+	git -C repo for-each-ref refs/tags/ >actual &&
+	test_cmp expect actual
+'
+
 test_done

-- 
2.53.GIT

Re: [PATCH v7 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Patrick Steinhardt]

On Thu, Feb 19, 2026 at 10:38:25AM +0100, Karthik Nayak wrote:
> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
> index 9912433b8c..d69aea3f7f 100755
> --- a/t/t1423-ref-backend.sh
> +++ b/t/t1423-ref-backend.sh
> @@ -30,44 +39,86 @@ run_with_uri() {
>  #   <repo> is the relative path to the repo to run the command in.
>  #   <backend> is the original ref storage of the repo.
>  #   <uri> is the new URI to be set for the ref storage.
> +#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
> +#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
>  #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
>  test_refs_backend() {
>  	repo=$1 &&
>  	backend=$2 &&
>  	uri=$3 &&
> -	err_msg=$4 &&
> +	via=$4 &&
> +	err_msg=$5 &&
> +
>  
> -	git -C "$repo" config set core.repositoryformatversion 1 &&
>  	if test -n "$err_msg";
>  	then
> -		git -C "$repo" config set extensions.refStorage "$uri" &&
> -		test_must_fail git -C "$repo" refs list 2>err &&
> -		test_grep "$err_msg" err
> +		if test "$via" = "env"
> +		then
> +			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
> +		elif test "$via" = "config"
> +		then
> +			git -C "$repo" config set extensions.refStorage "$uri" &&
> +			test_must_fail git -C "$repo" refs list 2>err &&
> +			test_grep "$err_msg" err
> +		fi
>  	else
>  		git -C "$repo" refs list >expect &&
> -		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
> +		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
>  		test_cmp expect actual
>  	fi
>  }
>  
> -test_expect_success 'URI is invalid' '
> +# Verify that the expected files are present in the gitdir and the refsdir.
> +# Usage: verify_files_exist <gitdir> <refdir>
> +#   <gitdir> is the path for the gitdir.
> +#   <refdir> is the path for the refdir.
> +verify_files_exist() {
> +	gitdir=$1 &&
> +	refdir=$2 &&
> +
> +	# verify that the stubs were added to the $GITDIR.
> +	cat $gitdir/refs/heads >actual &&
> +	echo "repository uses alternate refs storage" >expect &&
> +	test_cmp expect actual &&

Tiny nit, not worth addressing on its own: we could simply `test_cmp
expect "$gitdir/refs/heads", without the need to copy that file first.

> +	cat $gitdir/HEAD >actual &&
> +	echo "ref: refs/heads/.invalid" >expect &&
> +	test_cmp expect actual

Same here, no need to copy the file around.

Other than that I'm happy with this patch series now, thanks!

Patrick

Re: [PATCH v7 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1118 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

[snip]

>> -test_expect_success 'URI is invalid' '
>> +# Verify that the expected files are present in the gitdir and the refsdir.
>> +# Usage: verify_files_exist <gitdir> <refdir>
>> +#   <gitdir> is the path for the gitdir.
>> +#   <refdir> is the path for the refdir.
>> +verify_files_exist() {
>> +	gitdir=$1 &&
>> +	refdir=$2 &&
>> +
>> +	# verify that the stubs were added to the $GITDIR.
>> +	cat $gitdir/refs/heads >actual &&
>> +	echo "repository uses alternate refs storage" >expect &&
>> +	test_cmp expect actual &&
>
> Tiny nit, not worth addressing on its own: we could simply `test_cmp
> expect "$gitdir/refs/heads", without the need to copy that file first.
>

Indeed. I'll make this change locally, but hold off on re-rolling for
now.

>> +	cat $gitdir/HEAD >actual &&
>> +	echo "ref: refs/heads/.invalid" >expect &&
>> +	test_cmp expect actual
>
> Same here, no need to copy the file around.
>
> Other than that I'm happy with this patch series now, thanks!
>
> Patrick

I'm also quite happy with how its turned out. Thanks for the continuous
reviews.

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v8 0/6] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:

  - A format name alone (e.g., `reftable` or `files`).

  - A URI format `<format>://<payload>` explicitly specifies both the
    format and payload (e.g., `reftable:///foo/bar`).

We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v8:
- Fix a typo/grammar in commit 4.
- In the final commits tests, avoid creating a file for text
  comparison.
- Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com

Changes in v7:
- Add more details in the commit messages.
- Cleanup some whitespace.
- Reorder the commits to be group related changes together.
- Add checks for stubs in the tests when creating new repos.
- Link to v6: https://patch.msgid.link/20260214-kn-alternate-ref-dir-v6-0-86a82c77cf59@gmail.com

Changes in v6:
- The biggest change in this version is that we now support using the
  environment variable with 'git-clone(1)' and 'git-init(1)'. In such
  situations, the alternate reference directory is created and the
  config is added to the repository.
- Add a new commit which moves stub creation/removal to the generic
  layer.
- Cleanup logic flow in `refs_compute_filesystem_location()`.
- Add more tests for usage with 'git-clone(1)', 'git-init(1)' and
  migration of repositories using alternate refs backend.
- Fixup documentation, commit messages and typos.
- Link to v5: https://patch.msgid.link/20260209-kn-alternate-ref-dir-v5-0-740899834ceb@gmail.com

Changes in v5:
- Moved around the commits, to ensure that the code to handle the config
  in the backend is first. Previously, we added the config first, which
  meant the commit allowed users to provide a URI but it was simply
  ignore.
- Fix typos and grammar and rename variables.
- Clean up the description and documentation to actually specify
  protocol over location.
- Avoid an extra memory allocation by detaching the strbuf value.
- Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
  backends will store worktree references in 'ref_dir/worktrees/wt_id'
  and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
  relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/config/extensions.adoc |  16 +-
 Documentation/git.adoc               |   5 +
 builtin/clone.c                      |   9 +-
 builtin/worktree.c                   |  34 +++++
 environment.h                        |   1 +
 refs.c                               | 126 +++++++++++++++-
 refs.h                               |  13 ++
 refs/files-backend.c                 |  23 ++-
 refs/packed-backend.c                |   5 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  14 ++
 refs/reftable-backend.c              |  61 ++------
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  96 ++++++++++--
 setup.h                              |   4 +-
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 280 +++++++++++++++++++++++++++++++++++
 18 files changed, 625 insertions(+), 81 deletions(-)

Karthik Nayak (6):
      setup: don't modify repo in `create_reference_database()`
      refs: extract out `refs_create_refdir_stubs()`
      refs: move out stub modification to generic layer
      refs: receive and use the reference storage payload
      refs: allow reference location in refstorage config
      refs: add GIT_REFERENCE_BACKEND to specify reference backend

Range-diff versus v7:

1:  0f9fad1145 = 1:  8cc4b88f60 setup: don't modify repo in `create_reference_database()`
2:  cfe28f7464 = 2:  382b7b1964 refs: extract out `refs_create_refdir_stubs()`
3:  7a08fde968 = 3:  ef00e85466 refs: move out stub modification to generic layer
4:  b6988bf969 ! 4:  3e7d7042ef refs: receive and use the reference storage payload
    @@ refs/refs-internal.h: enum ref_transaction_error refs_verify_refnames_available(
     + * directory if working with a linked worktree. If working with the main
     + * worktree, both values will be the same.
     + *
    -+ * This is used by backends that store store files in the repository directly.
    ++ * This is used by backends that store references in the repository directly.
     + */
     +void refs_compute_filesystem_location(const char *gitdir, const char *payload,
     +				      bool *is_worktree, struct strbuf *refdir,
5:  4769fae36f = 5:  5b69104cd9 refs: allow reference location in refstorage config
6:  6bc3f09144 ! 6:  6a007df2ad refs: add GIT_REFERENCE_BACKEND to specify reference backend
    @@ t/t1423-ref-backend.sh: run_with_uri() {
     +	refdir=$2 &&
     +
     +	# verify that the stubs were added to the $GITDIR.
    -+	cat $gitdir/refs/heads >actual &&
     +	echo "repository uses alternate refs storage" >expect &&
    -+	test_cmp expect actual &&
    -+	cat $gitdir/HEAD >actual &&
    ++	test_cmp expect $gitdir/refs/heads &&
     +	echo "ref: refs/heads/.invalid" >expect &&
    -+	test_cmp expect actual
    ++	test_cmp expect $gitdir/HEAD
     +
     +	# verify that backend specific files exist.
     +	case "$GIT_DEFAULT_REF_FORMAT" in


base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v8 1/6] setup: don't modify repo in `create_reference_database()`

[Karthik Nayak]

The `create_reference_database()` function is used to create the
reference database during initialization of a repository. The function
calls `repo_set_ref_storage_format()` to set the repositories reference
format. This is an unexpected side-effect of the function. More so
because the function is only called in two locations:

  1. During git-init(1) where the value is propagated from the `struct
     repository_format repo_fmt` value.

  2. During git-clone(1) where the value is propagated from the
     `the_repository` value.

The former is valid, however the flow already calls
`repo_set_ref_storage_format()`, so this effort is simply duplicated.
The latter sets the existing value in `the_repository` back to itself.
While this is okay for now, introduction of more fields in
`repo_set_ref_storage_format()` would cause issues, especially
dynamically allocated strings, where we would free/allocate the same
string back into `the_repostiory`.

To avoid all this confusion, clean up the function to no longer take in
and set the repo's reference storage format.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c | 2 +-
 setup.c         | 7 ++-----
 setup.h         | 3 +--
 3 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index b40cee5968..cd43bb5aa2 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1442,7 +1442,7 @@ int cmd_clone(int argc,
 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
 	repo_set_hash_algo(the_repository, hash_algo);
-	create_reference_database(the_repository->ref_storage_format, NULL, 1);
+	create_reference_database(NULL, 1);
 
 	/*
 	 * Before fetching from the remote, download and install bundle
diff --git a/setup.c b/setup.c
index b723f8b339..1fc9ae3872 100644
--- a/setup.c
+++ b/setup.c
@@ -2359,14 +2359,12 @@ static int is_reinit(void)
 	return ret;
 }
 
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet)
+void create_reference_database(const char *initial_branch, int quiet)
 {
 	struct strbuf err = STRBUF_INIT;
 	char *to_free = NULL;
 	int reinit = is_reinit();
 
-	repo_set_ref_storage_format(the_repository, ref_storage_format);
 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -2701,8 +2699,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 				      &repo_fmt, init_shared_repository);
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
-		create_reference_database(repo_fmt.ref_storage_format,
-					  initial_branch, flags & INIT_DB_QUIET);
+		create_reference_database(initial_branch, flags & INIT_DB_QUIET);
 	create_object_directory();
 
 	if (repo_settings_get_shared_repository(the_repository)) {
diff --git a/setup.h b/setup.h
index d55dcc6608..ddb9f6701c 100644
--- a/setup.h
+++ b/setup.h
@@ -240,8 +240,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 void initialize_repository_version(int hash_algo,
 				   enum ref_storage_format ref_storage_format,
 				   int reinit);
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet);
+void create_reference_database(const char *initial_branch, int quiet);
 
 /*
  * NOTE NOTE NOTE!!

-- 
2.53.GIT

[PATCH v8 2/6] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

For Git to recognize a directory as a Git directory, it requires the
directory to contain:

  1. 'HEAD' file
  2. 'objects/' directory
  3. 'refs/' directory

Here, #1 and #3 are part of the reference storage mechanism,
specifically the files backend. Since then, newer backends such as the
reftable backend have moved to using their own path ('reftable/') for
storing references. But to ensure Git still recognizes the directory as
a Git directory, we create stubs.

There are two locations where we create stubs:

- In 'refs/reftable-backend.c' when creating the reftable backend.
- In 'clone.c' before spawning transport helpers.

In a following commit, we'll add another instance. So instead of
repeating the code, let's extract out this code to
`refs_create_refdir_stubs()` and use it.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c         |  7 +------
 refs.c                  | 23 +++++++++++++++++++++++
 refs.h                  | 13 +++++++++++++
 refs/reftable-backend.c | 14 ++------------
 4 files changed, 39 insertions(+), 18 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index cd43bb5aa2..697c5bb5cb 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1225,12 +1225,7 @@ int cmd_clone(int argc,
 	initialize_repository_version(GIT_HASH_UNKNOWN,
 				      the_repository->ref_storage_format, 1);
 
-	strbuf_addf(&buf, "%s/HEAD", git_dir);
-	write_file(buf.buf, "ref: refs/heads/.invalid");
-
-	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/refs", git_dir);
-	safe_create_dir(the_repository, buf.buf, 1);
+	refs_create_refdir_stubs(the_repository, git_dir, NULL);
 
 	/*
 	 * additional config can be injected with -c, make sure it's included
diff --git a/refs.c b/refs.c
index 627b7f8698..77b93d655b 100644
--- a/refs.c
+++ b/refs.c
@@ -2163,6 +2163,29 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 	return NULL;
 }
 
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_content)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	strbuf_addf(&path, "%s/HEAD", refdir);
+	write_file(path.buf, "ref: refs/heads/.invalid");
+	adjust_shared_perm(repo, path.buf);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/refs", refdir);
+	safe_create_dir(repo, path.buf, 1);
+
+	if (refs_heads_content) {
+		strbuf_reset(&path);
+		strbuf_addf(&path, "%s/refs/heads", refdir);
+		write_file(path.buf, "%s", refs_heads_content);
+		adjust_shared_perm(repo, path.buf);
+	}
+
+	strbuf_release(&path);
+}
+
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
diff --git a/refs.h b/refs.h
index f0abfa1d93..9d8890fdff 100644
--- a/refs.h
+++ b/refs.h
@@ -1427,4 +1427,17 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
+/*
+ * Git only recognizes a directory as a repository if it contains:
+ * - HEAD file
+ * - refs/ folder
+ * While it is necessary within the files backend, newer backends may not
+ * follow the same structure. To go around this, we create stubs as necessary.
+ *
+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file
+ * with the provided message.
+ */
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_msg);
+
 #endif /* REFS_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index fe74af73af..d8651fe779 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,18 +491,8 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	write_file(sb.buf, "ref: refs/heads/.invalid");
-	adjust_shared_perm(the_repository, sb.buf);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	safe_create_dir(the_repository, sb.buf, 1);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	write_file(sb.buf, "this repository uses the reftable format");
-	adjust_shared_perm(the_repository, sb.buf);
+	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
+				 "this repository uses the reftable format");
 
 	strbuf_release(&sb);
 	return 0;

-- 
2.53.GIT

[PATCH v8 3/6] refs: move out stub modification to generic layer

[Karthik Nayak]

When creating the reftable reference backend on disk, we create stubs to
ensure that the directory can be recognized as a Git repository. This is
done by calling `refs_create_refdir_stubs()`. Move this to the generic
layer as this is needed for all backends excluding from the files
backends. In an upcoming commit where we introduce alternate reference
backend locations, we'll have to also create stubs in the $GIT_DIR
irrespective of the backend being used. This commit builds the base to
add that logic.

Similarly, move the logic for deletion of stubs to the generic layer.
The files backend recursively calls the remove function of the
'packed-backend', here skip calling the generic function since that
would try to delete stubs.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 47 +++++++++++++++++++++++++++++++++++++++++++++--
 refs/files-backend.c    |  6 +++++-
 refs/reftable-backend.c | 27 ---------------------------
 3 files changed, 50 insertions(+), 30 deletions(-)

diff --git a/refs.c b/refs.c
index 77b93d655b..c83af63dc5 100644
--- a/refs.c
+++ b/refs.c
@@ -2189,12 +2189,55 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
-	return refs->be->create_on_disk(refs, flags, err);
+	int ret = refs->be->create_on_disk(refs, flags, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf msg = STRBUF_INIT;
+
+		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+		strbuf_release(&msg);
+	}
+
+	return ret;
 }
 
 int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
-	return refs->be->remove_on_disk(refs, err);
+	int ret = refs->be->remove_on_disk(refs, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf sb = STRBUF_INIT;
+
+		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub HEAD: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs/heads", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub heads: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs", refs->gitdir);
+		if (rmdir(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete refs directory: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+
+		strbuf_release(&sb);
+	}
+
+	return ret;
 }
 
 int repo_resolve_gitlink_ref(struct repository *r,
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 240d3c3b26..d3f6423261 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -3700,7 +3700,11 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,
 	if (for_each_root_ref(refs, remove_one_root_ref, &data) < 0)
 		ret = -1;
 
-	if (ref_store_remove_on_disk(refs->packed_ref_store, err) < 0)
+	/*
+	 * Directly access the cleanup functions for packed-refs as the generic function
+	 * would try to clear stubs which isn't required for the files backend.
+	 */
+	if (refs->packed_ref_store->be->remove_on_disk(refs->packed_ref_store, err) < 0)
 		ret = -1;
 
 	strbuf_release(&sb);
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d8651fe779..6ce7f9bb8e 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,9 +491,6 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
-				 "this repository uses the reftable format");
-
 	strbuf_release(&sb);
 	return 0;
 }
@@ -519,30 +516,6 @@ static int reftable_be_remove_on_disk(struct ref_store *ref_store,
 			    strerror(errno));
 		ret = -1;
 	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub HEAD: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub heads: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	if (rmdir(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete refs directory: %s",
-			    strerror(errno));
-		ret = -1;
-	}
 
 	strbuf_release(&sb);
 	return ret;

-- 
2.53.GIT

[PATCH v8 4/6] refs: receive and use the reference storage payload

[Karthik Nayak]

An upcoming commit will add support for providing an URI via the
'extensions.refStorage' config. The URI will contain the reference
backend and a corresponding payload. The payload can be then used for
providing an alternate locations for the reference backend.

To prepare for this, modify the existing backends to accept such an
argument when initializing via the 'init()' function. Both the files
and reftable backends will parse the information to be filesystem paths
to store references. Given that no callers pass any payload yet this is
essentially a no-op change for now.

To enable this, provide a 'refs_compute_filesystem_location()' function
which will parse the current 'gitdir' and the 'payload' to provide the
final reference directory and common reference directory (if working in
a linked worktree).

The documentation and tests will be added alongside the extension of the
config variable.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 40 +++++++++++++++++++++++++++++++++++++++-
 refs/files-backend.c    | 17 ++++++++++++-----
 refs/packed-backend.c   |  5 +++++
 refs/packed-backend.h   |  1 +
 refs/refs-internal.h    | 14 ++++++++++++++
 refs/reftable-backend.c | 24 ++++++++++++++----------
 6 files changed, 85 insertions(+), 16 deletions(-)

diff --git a/refs.c b/refs.c
index c83af63dc5..ba2573eb7a 100644
--- a/refs.c
+++ b/refs.c
@@ -5,6 +5,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "abspath.h"
 #include "advice.h"
 #include "config.h"
 #include "environment.h"
@@ -2290,7 +2291,7 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, gitdir, flags);
+	refs = be->init(repo, NULL, gitdir, flags);
 	return refs;
 }
 
@@ -3468,3 +3469,40 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
 		return "unknown failure";
 	}
 }
+
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
+
+	if (!payload) {
+		/*
+		 * We can use the 'gitdir' as the 'refdir' without appending the
+		 * worktree path, as the 'gitdir' here is already the worktree
+		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
+		 */
+		strbuf_addstr(refdir, gitdir);
+		return;
+	}
+
+	if (!is_absolute_path(payload)) {
+		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
+		strbuf_realpath(ref_common_dir, sb.buf, 1);
+	} else {
+		strbuf_realpath(ref_common_dir, payload, 1);
+	}
+
+	strbuf_addbuf(refdir, ref_common_dir);
+
+	if (*is_worktree) {
+		const char *wt_id = strrchr(gitdir, '/');
+		if (!wt_id)
+			BUG("worktree path does not contain slash");
+		strbuf_addf(refdir, "/worktrees/%s", wt_id + 1);
+	}
+
+	strbuf_release(&sb);
+}
diff --git a/refs/files-backend.c b/refs/files-backend.c
index d3f6423261..9cde3ba724 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
  * set of caches.
  */
 static struct ref_store *files_ref_store_init(struct repository *repo,
+					      const char *payload,
 					      const char *gitdir,
 					      unsigned int flags)
 {
 	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
 	struct ref_store *ref_store = (struct ref_store *)refs;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
+	bool is_worktree;
+
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
 
-	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
+	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
 	refs->store_flags = flags;
-	get_common_dir_noenv(&sb, gitdir);
-	refs->gitcommondir = strbuf_detach(&sb, NULL);
+	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
 	refs->packed_ref_store =
-		packed_ref_store_init(repo, refs->gitcommondir, flags);
+		packed_ref_store_init(repo, NULL, refs->gitcommondir, flags);
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
 	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
 
@@ -126,6 +131,8 @@ static struct ref_store *files_ref_store_init(struct repository *repo,
 	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
 			      &refs->gitcommondir);
 
+	strbuf_release(&refdir);
+
 	return ref_store;
 }
 
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 4ea0c12299..e7bb9f10f9 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -211,7 +211,12 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
 	return snapshot->refs->base.repo->hash_algo->hexsz;
 }
 
+/*
+ * Since packed-refs is only stored in the common dir, don't parse the
+ * payload and rely on the files-backend to set 'gitdir' correctly.
+ */
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload UNUSED,
 					const char *gitdir,
 					unsigned int store_flags)
 {
diff --git a/refs/packed-backend.h b/refs/packed-backend.h
index 9481d5e7c2..2c2377a356 100644
--- a/refs/packed-backend.h
+++ b/refs/packed-backend.h
@@ -14,6 +14,7 @@ struct ref_transaction;
  */
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload,
 					const char *gitdir,
 					unsigned int store_flags);
 
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index c7d2a6e50b..4fb8fdb872 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -389,6 +389,7 @@ struct ref_store;
  * the ref_store and to record the ref_store for later lookup.
  */
 typedef struct ref_store *ref_store_init_fn(struct repository *repo,
+					    const char *payload,
 					    const char *gitdir,
 					    unsigned int flags);
 /*
@@ -666,4 +667,17 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 					  unsigned int initial_transaction,
 					  struct strbuf *err);
 
+/*
+ * Given a gitdir and the reference storage payload provided, retrieve the
+ * 'refdir' and 'ref_common_dir'. The former is where references should be
+ * stored for the current worktree, the latter is the common reference
+ * directory if working with a linked worktree. If working with the main
+ * worktree, both values will be the same.
+ *
+ * This is used by backends that store references in the repository directly.
+ */
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir);
+
 #endif /* REFS_REFS_INTERNAL_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 6ce7f9bb8e..0e220d6bb5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -372,18 +372,24 @@ static int reftable_be_fsync(int fd)
 }
 
 static struct ref_store *reftable_be_init(struct repository *repo,
+					  const char *payload,
 					  const char *gitdir,
 					  unsigned int store_flags)
 {
 	struct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
 	struct strbuf path = STRBUF_INIT;
-	int is_worktree;
+	bool is_worktree;
 	mode_t mask;
 
 	mask = umask(0);
 	umask(mask);
 
-	base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
+
+	base_ref_store_init(&refs->base, repo, refdir.buf, &refs_be_reftable);
 	strmap_init(&refs->worktree_backends);
 	refs->store_flags = store_flags;
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
@@ -419,14 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	/*
 	 * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
 	 * This stack contains both the shared and the main worktree refs.
-	 *
-	 * Note that we don't try to resolve the path in case we have a
-	 * worktree because `get_common_dir_noenv()` already does it for us.
 	 */
-	is_worktree = get_common_dir_noenv(&path, gitdir);
+	strbuf_addbuf(&path, &ref_common_dir);
 	if (!is_worktree) {
 		strbuf_reset(&path);
-		strbuf_realpath(&path, gitdir, 0);
+		strbuf_realpath(&path, ref_common_dir.buf, 0);
 	}
 	strbuf_addstr(&path, "/reftable");
 	refs->err = reftable_backend_init(&refs->main_backend, path.buf,
@@ -443,10 +446,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	 * do it efficiently.
 	 */
 	if (is_worktree) {
-		strbuf_reset(&path);
-		strbuf_addf(&path, "%s/reftable", gitdir);
+		strbuf_addstr(&refdir, "/reftable");
 
-		refs->err = reftable_backend_init(&refs->worktree_backend, path.buf,
+		refs->err = reftable_backend_init(&refs->worktree_backend, refdir.buf,
 						  &refs->write_options);
 		if (refs->err)
 			goto done;
@@ -456,6 +458,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 
 done:
 	assert(refs->err != REFTABLE_API_ERROR);
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
 	strbuf_release(&path);
 	return &refs->base;
 }

-- 
2.53.GIT

[PATCH v8 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

The 'extensions.refStorage' config is used to specify the reference
backend for a given repository. Both the 'files' and 'reftable' backends
utilize the $GIT_DIR as the reference folder by default in
`get_main_ref_store()`.

Since the reference backends are pluggable, this means that they could
work with out-of-tree reference directories too. Extend the 'refStorage'
config to also support taking an URI input, where users can specify the
reference backend and the location.

Add the required changes to obtain and propagate this value to the
individual backends. Add the necessary documentation and tests.

Traditionally, for linked worktrees, references were stored in the
'$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
storage path, it doesn't make sense to store the main worktree
references in the new path, and the linked worktree references in the
$GIT_DIR. So, let's store linked worktree references in
'$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'. To do this, create the
necessary files and folders while also adding stubs in the $GIT_DIR path
to ensure that it is still considered a Git directory.

Ideally, we would want to pass in a `struct worktree *` to individual
backends, instead of passing the `gitdir`. This allows them to handle
worktree specific logic. Currently, that is not possible since the
worktree code is:

  - Tied to using the global `the_repository` variable.

  - Is not setup before the reference database during initialization of
    the repository.

Add a TODO in 'refs.c' to ensure we can eventually make that change.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/config/extensions.adoc |  16 +++-
 builtin/worktree.c                   |  34 ++++++++
 refs.c                               |   6 +-
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  34 +++++++-
 setup.h                              |   1 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
 9 files changed, 259 insertions(+), 9 deletions(-)

diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
index 532456644b..3e51da36d3 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
 `core.repositoryFormatVersion` setting.
 
 refStorage:::
-	Specify the ref storage format to use. The acceptable values are:
+	Specify the ref storage format and a corresponding payload. The value
+	can be either a format name or a URI:
 +
 --
+* A format name alone (e.g., `reftable` or `files`).
+
+* A URI format `<format>://<payload>` explicitly specifies both the
+  format and payload (e.g., `reftable:///foo/bar`).
+
+Supported format names are:
++
 include::../ref-storage-format.adoc[]
++
+The payload is passed directly to the reference backend. For the files and
+reftable backends, this must be a filesystem path where the references will
+be stored. Defaulting to the commondir when no payload is provided. Relative
+paths are resolved relative to the $GIT_DIR. Future backends may support
+other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +
 Note that this setting should only be set by linkgit:git-init[1] or
diff --git a/builtin/worktree.c b/builtin/worktree.c
index fbdaf2eb2e..293e808379 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
 	return run_command(&cp);
 }
 
+/*
+ * References for worktrees are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
+ * But when using alternate reference directories, we want to store the worktree
+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
+ *
+ * Create the necessary folder structure to facilitate the same. But to ensure
+ * that the former path is still considered a Git directory, add stubs.
+ */
+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	char *path;
+
+	path = wt->repo->ref_storage_payload;
+	if (!path)
+		return;
+
+	if (!is_absolute_path(path))
+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
+
+	strbuf_addf(&sb, "%s/worktrees", path);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_addf(&sb, "/%s", wt->id);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_reset(&sb);
+
+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
+		    path, wt->id);
+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
+
+	strbuf_release(&sb);
+}
+
 static int add_worktree(const char *path, const char *refname,
 			const struct add_opts *opts)
 {
@@ -518,6 +551,7 @@ static int add_worktree(const char *path, const char *refname,
 		ret = error(_("could not find created worktree '%s'"), name);
 		goto done;
 	}
+	setup_alternate_ref_dir(wt, sb_repo.buf);
 	wt_refs = get_worktree_ref_store(wt);
 
 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
diff --git a/refs.c b/refs.c
index ba2573eb7a..ef1902e85c 100644
--- a/refs.c
+++ b/refs.c
@@ -2291,7 +2291,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, NULL, gitdir, flags);
+	/*
+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
+	 * allow the backend to handle how it wants to deal with worktrees.
+	 */
+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
 	return refs;
 }
 
diff --git a/repository.c b/repository.c
index c7e75215ac..9815f081ef 100644
--- a/repository.c
+++ b/repository.c
@@ -193,9 +193,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
 }
 
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format)
+				 enum ref_storage_format format,
+				 const char *payload)
 {
 	repo->ref_storage_format = format;
+	free(repo->ref_storage_payload);
+	repo->ref_storage_payload = xstrdup_or_null(payload);
 }
 
 /*
@@ -277,7 +280,8 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 	repo_set_compat_hash_algo(repo, format.compat_hash_algo);
-	repo_set_ref_storage_format(repo, format.ref_storage_format);
+	repo_set_ref_storage_format(repo, format.ref_storage_format,
+				    format.ref_storage_payload);
 	repo->repository_format_worktree_config = format.worktree_config;
 	repo->repository_format_relative_worktrees = format.relative_worktrees;
 	repo->repository_format_precious_objects = format.precious_objects;
@@ -369,6 +373,7 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
+	FREE_AND_NULL(repo->ref_storage_payload);
 
 	odb_free(repo->objects);
 	repo->objects = NULL;
diff --git a/repository.h b/repository.h
index 6063c4b846..95e2333bad 100644
--- a/repository.h
+++ b/repository.h
@@ -150,6 +150,11 @@ struct repository {
 
 	/* Repository's reference storage format, as serialized on disk. */
 	enum ref_storage_format ref_storage_format;
+	/*
+	 * Reference storage information as needed for the backend. This contains
+	 * only the payload from the reference URI without the schema.
+	 */
+	char *ref_storage_payload;
 
 	/* A unique-id for tracing purposes. */
 	int trace2_repo_id;
@@ -204,7 +209,8 @@ void repo_set_worktree(struct repository *repo, const char *path);
 void repo_set_hash_algo(struct repository *repo, int algo);
 void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format);
+				 enum ref_storage_format format,
+				 const char *payload);
 void initialize_repository(struct repository *repo);
 RESULT_MUST_BE_USED
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index 1fc9ae3872..d407f3347b 100644
--- a/setup.c
+++ b/setup.c
@@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
 		return EXTENSION_UNKNOWN;
 }
 
+static void parse_reference_uri(const char *value, char **format,
+				char **payload)
+{
+	const char *schema_end;
+
+	schema_end = strstr(value, "://");
+	if (!schema_end) {
+		*format = xstrdup(value);
+		*payload = NULL;
+	} else {
+		*format = xstrndup(value, schema_end - value);
+		*payload = xstrdup_or_null(schema_end + 3);
+	}
+}
+
 /*
  * Record any new extensions in this function.
  */
@@ -674,10 +689,17 @@ static enum extension_result handle_extension(const char *var,
 		return EXTENSION_OK;
 	} else if (!strcmp(ext, "refstorage")) {
 		unsigned int format;
+		char *format_str;
 
 		if (!value)
 			return config_error_nonbool(var);
-		format = ref_storage_format_by_name(value);
+
+		parse_reference_uri(value, &format_str,
+				    &data->ref_storage_payload);
+
+		format = ref_storage_format_by_name(format_str);
+		free(format_str);
+
 		if (format == REF_STORAGE_FORMAT_UNKNOWN)
 			return error(_("invalid value for '%s': '%s'"),
 				     "extensions.refstorage", value);
@@ -850,6 +872,7 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
 	free(format->partial_clone);
+	free(format->ref_storage_payload);
 	init_repository_format(format);
 }
 
@@ -1942,7 +1965,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			repo_set_compat_hash_algo(the_repository,
 						  repo_fmt.compat_hash_algo);
 			repo_set_ref_storage_format(the_repository,
-						    repo_fmt.ref_storage_format);
+						    repo_fmt.ref_storage_format,
+						    repo_fmt.ref_storage_payload);
 			the_repository->repository_format_worktree_config =
 				repo_fmt.worktree_config;
 			the_repository->repository_format_relative_worktrees =
@@ -2042,7 +2066,8 @@ void check_repository_format(struct repository_format *fmt)
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
 	repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo);
 	repo_set_ref_storage_format(the_repository,
-				    fmt->ref_storage_format);
+				    fmt->ref_storage_format,
+				    fmt->ref_storage_payload);
 	the_repository->repository_format_worktree_config =
 		fmt->worktree_config;
 	the_repository->repository_format_relative_worktrees =
@@ -2643,7 +2668,8 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
-	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format);
+	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
+				    repo_fmt->ref_storage_payload);
 }
 
 int init_db(const char *git_dir, const char *real_git_dir,
diff --git a/setup.h b/setup.h
index ddb9f6701c..093af39e84 100644
--- a/setup.h
+++ b/setup.h
@@ -171,6 +171,7 @@ struct repository_format {
 	int hash_algo;
 	int compat_hash_algo;
 	enum ref_storage_format ref_storage_format;
+	char *ref_storage_payload;
 	int sparse_index;
 	char *work_tree;
 	struct string_list unknown_extensions;
diff --git a/t/meson.build b/t/meson.build
index 459c52a489..11fc5a49ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -210,6 +210,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..9912433b8c
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='Test reference backend URIs'
+
+. ./test-lib.sh
+
+# Run a git command with the provided reference storage. Reset the backend
+# post running the command.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <cmd> is the git subcommand to be run in the repository.
+run_with_uri() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	cmd=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1
+	git -C "$repo" config set extensions.refStorage "$uri" &&
+	git -C "$repo" $cmd &&
+	git -C "$repo" config set extensions.refStorage "$backend"
+}
+
+# Test a repository with a given reference storage by running and comparing
+# 'git refs list' before and after setting the new reference backend. If
+# err_msg is set, expect the command to fail and grep for the provided err_msg.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
+test_refs_backend() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	err_msg=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test -n "$err_msg";
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		test_must_fail git -C "$repo" refs list 2>err &&
+		test_grep "$err_msg" err
+	else
+		git -C "$repo" refs list >expect &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		test_cmp expect actual
+	fi
+}
+
+test_expect_success 'URI is invalid' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable@/home/reftable" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'URI ends with colon' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable:" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "db://.git" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+
+for to_format in $ref_formats
+do
+	if test "$from_format" = "$to_format"
+	then
+		continue
+	fi
+
+
+	for dir in "$(pwd)/repo/.git" "."
+	do
+
+		test_expect_success "read from $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
+			)
+		'
+
+		test_expect_success "write to $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+
+				git refs list >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				git refs list | grep -v "refs/tags/1" >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "with worktree and $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo wt" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				git config set core.repositoryformatversion 1 &&
+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+
+				git worktree add ../wt 2
+			) &&
+
+			git -C repo for-each-ref --include-root-refs >expect &&
+			git -C wt for-each-ref --include-root-refs >expect &&
+			! test_cmp expect actual &&
+
+			git -C wt rev-parse 2 >expect &&
+			git -C wt rev-parse HEAD >actual &&
+			test_cmp expect actual
+		'
+	done # closes dir
+done # closes to_format
+done # closes from_format
+
+test_done

-- 
2.53.GIT

Re: [PATCH v8 5/6] refs: allow reference location in refstorage config

[Kristoffer Haugsbakk]

On Mon, Feb 23, 2026, at 09:01, Karthik Nayak wrote:
>[snip]
>
> Helped-by: Patrick Steinhardt <ps@pks.im>
> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
> ---
>  Documentation/config/extensions.adoc |  16 +++-
>  builtin/worktree.c                   |  34 ++++++++
>  refs.c                               |   6 +-
>  repository.c                         |   9 +-
>  repository.h                         |   8 +-
>  setup.c                              |  34 +++++++-
>  setup.h                              |   1 +
>  t/meson.build                        |   1 +
>  t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
>  9 files changed, 259 insertions(+), 9 deletions(-)
>
> diff --git a/Documentation/config/extensions.adoc
> b/Documentation/config/extensions.adoc
> index 532456644b..3e51da36d3 100644
> --- a/Documentation/config/extensions.adoc
> +++ b/Documentation/config/extensions.adoc
> @@ -57,10 +57,24 @@ For historical reasons, this extension is respected
> regardless of the
>  `core.repositoryFormatVersion` setting.
>
>  refStorage:::
> -	Specify the ref storage format to use. The acceptable values are:
> +	Specify the ref storage format and a corresponding payload. The value
> +	can be either a format name or a URI:
>  +
>  --
> +* A format name alone (e.g., `reftable` or `files`).
> +
> +* A URI format `<format>://<payload>` explicitly specifies both the
> +  format and payload (e.g., `reftable:///foo/bar`).
> +
> +Supported format names are:
> ++
>  include::../ref-storage-format.adoc[]

It looks like this causes list continuation (+) and the `;;` syntax to
appear in the HTML output of git-config(1).

    + files;; for loose files with packed-refs. ...

    + The payload is passed ...

According to `Documentation/doc-diff master seen`.

It looks like dropping the list continuations fixes it.

    Supported format names are:

    include::../ref-storage-format.adoc[]

    The payload is passed directly to the reference backend. For the files and
    [...]

Maybe because you are inside an open block? I don’t know.

> ++
> +The payload is passed directly to the reference backend. For the files
>[snip]

Re: [PATCH v8 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3349 bytes --]

"Kristoffer Haugsbakk" <kristofferhaugsbakk@fastmail.com> writes:

> On Mon, Feb 23, 2026, at 09:01, Karthik Nayak wrote:
>>[snip]
>>
>> Helped-by: Patrick Steinhardt <ps@pks.im>
>> Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
>> ---
>>  Documentation/config/extensions.adoc |  16 +++-
>>  builtin/worktree.c                   |  34 ++++++++
>>  refs.c                               |   6 +-
>>  repository.c                         |   9 +-
>>  repository.h                         |   8 +-
>>  setup.c                              |  34 +++++++-
>>  setup.h                              |   1 +
>>  t/meson.build                        |   1 +
>>  t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
>>  9 files changed, 259 insertions(+), 9 deletions(-)
>>
>> diff --git a/Documentation/config/extensions.adoc
>> b/Documentation/config/extensions.adoc
>> index 532456644b..3e51da36d3 100644
>> --- a/Documentation/config/extensions.adoc
>> +++ b/Documentation/config/extensions.adoc
>> @@ -57,10 +57,24 @@ For historical reasons, this extension is respected
>> regardless of the
>>  `core.repositoryFormatVersion` setting.
>>
>>  refStorage:::
>> -	Specify the ref storage format to use. The acceptable values are:
>> +	Specify the ref storage format and a corresponding payload. The value
>> +	can be either a format name or a URI:
>>  +
>>  --
>> +* A format name alone (e.g., `reftable` or `files`).
>> +
>> +* A URI format `<format>://<payload>` explicitly specifies both the
>> +  format and payload (e.g., `reftable:///foo/bar`).
>> +
>> +Supported format names are:
>> ++
>>  include::../ref-storage-format.adoc[]
>
> It looks like this causes list continuation (+) and the `;;` syntax to
> appear in the HTML output of git-config(1).
>
>     + files;; for loose files with packed-refs. ...
>
>     + The payload is passed ...
>
> According to `Documentation/doc-diff master seen`.
>
> It looks like dropping the list continuations fixes it.
>
>     Supported format names are:
>
>     include::../ref-storage-format.adoc[]
>
>     The payload is passed directly to the reference backend. For the files and
>     [...]
>
> Maybe because you are inside an open block? I don’t know.
>

I don't know either. But this seems to fix it, let me know if it does
for you too.

--8<--

diff --git a/Documentation/config/extensions.adoc
b/Documentation/config/extensions.adoc
index 3e51da36d3..329d02b3c4 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -67,13 +67,13 @@ refStorage:::
   format and payload (e.g., `reftable:///foo/bar`).

 Supported format names are:
-+
+
 include::../ref-storage-format.adoc[]
-+
+
 The payload is passed directly to the reference backend. For the files and
 reftable backends, this must be a filesystem path where the references will
 be stored. Defaulting to the commondir when no payload is provided. Relative
-paths are resolved relative to the $GIT_DIR. Future backends may support
+paths are resolved relative to the `$GIT_DIR`. Future backends may support
 other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +


>> ++
>> +The payload is passed directly to the reference backend. For the files
>>[snip]

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v8 5/6] refs: allow reference location in refstorage config

[Kristoffer Haugsbakk]

On Tue, Feb 24, 2026, at 14:09, Karthik Nayak wrote:
> "Kristoffer Haugsbakk" <kristofferhaugsbakk@fastmail.com> writes:
>
>> On Mon, Feb 23, 2026, at 09:01, Karthik Nayak wrote:
>>[snip]
>> It looks like dropping the list continuations fixes it.
>>
>>     Supported format names are:
>>
>>     include::../ref-storage-format.adoc[]
>>
>>     The payload is passed directly to the reference backend. For the files and
>>     [...]
>>
>> Maybe because you are inside an open block? I don’t know.
>>
>
> I don't know either. But this seems to fix it, let me know if it does
> for you too.
>
> --8<--
>
> diff --git a/Documentation/config/extensions.adoc
> b/Documentation/config/extensions.adoc
> index 3e51da36d3..329d02b3c4 100644
> --- a/Documentation/config/extensions.adoc
> +++ b/Documentation/config/extensions.adoc
> @@ -67,13 +67,13 @@ refStorage:::
>    format and payload (e.g., `reftable:///foo/bar`).
>
>  Supported format names are:
> -+
> +
>  include::../ref-storage-format.adoc[]
> -+
> +
>  The payload is passed directly to the reference backend. For the files and
>  reftable backends, this must be a filesystem path where the references will
>  be stored. Defaulting to the commondir when no payload is provided. Relative
> -paths are resolved relative to the $GIT_DIR. Future backends may support
> +paths are resolved relative to the `$GIT_DIR`. Future backends may support
>  other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
>  --
>  +
>
>
>>>[snip]

Yeah it does. Thanks!

Re: [PATCH v8 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1722 bytes --]

"Kristoffer Haugsbakk" <kristofferhaugsbakk@fastmail.com> writes:

> On Tue, Feb 24, 2026, at 14:09, Karthik Nayak wrote:
>> "Kristoffer Haugsbakk" <kristofferhaugsbakk@fastmail.com> writes:
>>
>>> On Mon, Feb 23, 2026, at 09:01, Karthik Nayak wrote:
>>>[snip]
>>> It looks like dropping the list continuations fixes it.
>>>
>>>     Supported format names are:
>>>
>>>     include::../ref-storage-format.adoc[]
>>>
>>>     The payload is passed directly to the reference backend. For the files and
>>>     [...]
>>>
>>> Maybe because you are inside an open block? I don’t know.
>>>
>>
>> I don't know either. But this seems to fix it, let me know if it does
>> for you too.
>>
>> --8<--
>>
>> diff --git a/Documentation/config/extensions.adoc
>> b/Documentation/config/extensions.adoc
>> index 3e51da36d3..329d02b3c4 100644
>> --- a/Documentation/config/extensions.adoc
>> +++ b/Documentation/config/extensions.adoc
>> @@ -67,13 +67,13 @@ refStorage:::
>>    format and payload (e.g., `reftable:///foo/bar`).
>>
>>  Supported format names are:
>> -+
>> +
>>  include::../ref-storage-format.adoc[]
>> -+
>> +
>>  The payload is passed directly to the reference backend. For the files and
>>  reftable backends, this must be a filesystem path where the references will
>>  be stored. Defaulting to the commondir when no payload is provided. Relative
>> -paths are resolved relative to the $GIT_DIR. Future backends may support
>> +paths are resolved relative to the `$GIT_DIR`. Future backends may support
>>  other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
>>  --
>>  +
>>
>>
>>>>[snip]
>
> Yeah it does. Thanks!

Awesome. Thanks for noticing.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v8 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
the previous commit we extended the 'extensions.refStorage' config to
also support an URI input for reference backend with location.

Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
takes in the same input as the config variable. Having an environment
variable allows us to modify the reference backend and location on the
fly for individual Git commands.

The environment variable also allows usage of alternate reference
directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
the repository when created with the environment variable set.

When initializing the repository with an alternate reference folder,
create the required stubs in the repositories $GIT_DIR. The inverse,
i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
since that would render it unusable. Removal of ref store is only used
when migrating between ref formats and cleanup of the $GIT_DIR doesn't
make sense in such a situation.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   5 ++
 environment.h          |   1 +
 refs.c                 |  30 +++++---
 setup.c                |  55 ++++++++++++++-
 t/t1423-ref-backend.sh | 187 ++++++++++++++++++++++++++++++++++++++++---------
 5 files changed, 233 insertions(+), 45 deletions(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..66442735ea 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REFERENCE_BACKEND`::
+    Specify which reference backend to be used along with its URI.
+    See `extensions.refStorage` option in linkgit:git-config[1] for more
+    details. Overrides the config variable when used.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 27f657af04..540e0a7f6d 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index ef1902e85c..a700a66f08 100644
--- a/refs.c
+++ b/refs.c
@@ -2192,13 +2192,17 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
 {
 	int ret = refs->be->create_on_disk(refs, flags, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
-		struct strbuf msg = STRBUF_INIT;
-
-		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
-		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
-		strbuf_release(&msg);
+	if (!ret) {
+		/* Creation of stubs for linked worktrees are handled in the worktree code. */
+		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
+			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
+						 "repository uses alternate refs storage");
+		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+			struct strbuf msg = STRBUF_INIT;
+			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+			strbuf_release(&msg);
+		}
 	}
 
 	return ret;
@@ -2208,10 +2212,18 @@ int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
 	int ret = refs->be->remove_on_disk(refs, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+	if (!ret) {
+		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
 		struct strbuf sb = STRBUF_INIT;
 
+		/* Backends apart from the files backend create stubs. */
+		if (format == REF_STORAGE_FORMAT_FILES)
+			return ret;
+
+		/* Alternate refs backend require stubs in the gitdir. */
+		if (refs->repo->ref_storage_payload)
+			return ret;
+
 		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
 		if (unlink(sb.buf) < 0) {
 			strbuf_addf(err, "could not delete stub HEAD: %s",
diff --git a/setup.c b/setup.c
index d407f3347b..90cb9be578 100644
--- a/setup.c
+++ b/setup.c
@@ -1838,6 +1838,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 	static struct strbuf cwd = STRBUF_INIT;
 	struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT;
 	const char *prefix = NULL;
+	const char *ref_backend_uri;
 	struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
 
 	/*
@@ -1995,6 +1996,25 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
+	/*
+	 * The env variable should override the repository config
+	 * for 'extensions.refStorage'.
+	 */
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+		repo_set_ref_storage_format(the_repository, format, payload);
+
+		free(backend);
+		free(payload);
+	}
+
 	setup_original_cwd();
 
 	strbuf_release(&dir);
@@ -2337,7 +2357,8 @@ void initialize_repository_version(int hash_algo,
 	 * the remote repository's format.
 	 */
 	if (hash_algo != GIT_HASH_SHA1_LEGACY ||
-	    ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	    ref_storage_format != REF_STORAGE_FORMAT_FILES ||
+	    the_repository->ref_storage_payload)
 		target_version = GIT_REPO_VERSION_READ;
 
 	if (hash_algo != GIT_HASH_SHA1_LEGACY && hash_algo != GIT_HASH_UNKNOWN)
@@ -2346,11 +2367,20 @@ void initialize_repository_version(int hash_algo,
 	else if (reinit)
 		repo_config_set_gently(the_repository, "extensions.objectformat", NULL);
 
-	if (ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	if (the_repository->ref_storage_payload) {
+		struct strbuf ref_uri = STRBUF_INIT;
+
+		strbuf_addf(&ref_uri, "%s://%s",
+			    ref_storage_format_to_name(ref_storage_format),
+			    the_repository->ref_storage_payload);
+		repo_config_set(the_repository, "extensions.refstorage", ref_uri.buf);
+		strbuf_release(&ref_uri);
+	} else if (ref_storage_format != REF_STORAGE_FORMAT_FILES) {
 		repo_config_set(the_repository, "extensions.refstorage",
 				ref_storage_format_to_name(ref_storage_format));
-	else if (reinit)
+	} else if (reinit) {
 		repo_config_set_gently(the_repository, "extensions.refstorage", NULL);
+	}
 
 	if (reinit) {
 		struct strbuf config = STRBUF_INIT;
@@ -2623,6 +2653,7 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 		.ignore_repo = 1,
 		.ignore_worktree = 1,
 	};
+	const char *ref_backend_uri;
 	const char *env;
 
 	config_with_options(read_default_format_config, &cfg, NULL, NULL, &opts);
@@ -2668,6 +2699,24 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
+
+
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+
+		repo_fmt->ref_storage_format = format;
+		repo_fmt->ref_storage_payload = payload;
+
+		free(backend);
+	}
+
 	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
 				    repo_fmt->ref_storage_payload);
 }
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
index 9912433b8c..1a7b8eadba 100755
--- a/t/t1423-ref-backend.sh
+++ b/t/t1423-ref-backend.sh
@@ -11,16 +11,25 @@ test_description='Test reference backend URIs'
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
 #   <cmd> is the git subcommand to be run in the repository.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 run_with_uri() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
 	cmd=$4 &&
+	via=$5 &&
 
-	git -C "$repo" config set core.repositoryformatversion 1
-	git -C "$repo" config set extensions.refStorage "$uri" &&
-	git -C "$repo" $cmd &&
-	git -C "$repo" config set extensions.refStorage "$backend"
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test "$via" = "env"
+	then
+		test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd
+	elif test "$via" = "config"
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		git -C "$repo" $cmd &&
+		git -C "$repo" config set extensions.refStorage "$backend"
+	fi
 }
 
 # Test a repository with a given reference storage by running and comparing
@@ -30,44 +39,84 @@ run_with_uri() {
 #   <repo> is the relative path to the repo to run the command in.
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
 test_refs_backend() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
-	err_msg=$4 &&
+	via=$4 &&
+	err_msg=$5 &&
+
 
-	git -C "$repo" config set core.repositoryformatversion 1 &&
 	if test -n "$err_msg";
 	then
-		git -C "$repo" config set extensions.refStorage "$uri" &&
-		test_must_fail git -C "$repo" refs list 2>err &&
-		test_grep "$err_msg" err
+		if test "$via" = "env"
+		then
+			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
+		elif test "$via" = "config"
+		then
+			git -C "$repo" config set extensions.refStorage "$uri" &&
+			test_must_fail git -C "$repo" refs list 2>err &&
+			test_grep "$err_msg" err
+		fi
 	else
 		git -C "$repo" refs list >expect &&
-		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
 		test_cmp expect actual
 	fi
 }
 
-test_expect_success 'URI is invalid' '
+# Verify that the expected files are present in the gitdir and the refsdir.
+# Usage: verify_files_exist <gitdir> <refdir>
+#   <gitdir> is the path for the gitdir.
+#   <refdir> is the path for the refdir.
+verify_files_exist() {
+	gitdir=$1 &&
+	refdir=$2 &&
+
+	# verify that the stubs were added to the $GITDIR.
+	echo "repository uses alternate refs storage" >expect &&
+	test_cmp expect $gitdir/refs/heads &&
+	echo "ref: refs/heads/.invalid" >expect &&
+	test_cmp expect $gitdir/HEAD
+
+	# verify that backend specific files exist.
+	case "$GIT_DEFAULT_REF_FORMAT" in
+	files)
+		test_path_is_dir $refdir/refs/heads &&
+		test_path_is_file $refdir/HEAD;;
+	reftable)
+		test_path_is_dir $refdir/reftable &&
+		test_path_is_file $refdir/reftable/tables.list;;
+	*)
+		BUG "unhandled ref format $GIT_DEFAULT_REF_FORMAT";;
+	esac
+}
+
+methods="config env"
+for method in $methods
+do
+
+test_expect_success "$method: URI is invalid" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable@/home/reftable" \
+	test_refs_backend repo files "reftable@/home/reftable" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'URI ends with colon' '
+test_expect_success "$method: URI ends with colon" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable:" \
+	test_refs_backend repo files "reftable:" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'unknown reference backend' '
+test_expect_success "$method: unknown reference backend" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "db://.git" \
+	test_refs_backend repo files "db://.git" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
@@ -86,7 +135,7 @@ do
 	for dir in "$(pwd)/repo/.git" "."
 	do
 
-		test_expect_success "read from $to_format backend, $dir dir" '
+		test_expect_success "$method: read from $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -101,7 +150,7 @@ do
 			)
 		'
 
-		test_expect_success "write to $to_format backend, $dir dir" '
+		test_expect_success "$method: write to $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -113,20 +162,22 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" &&
 
 				git refs list >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"tag -d 1" "$method" &&
 				git refs list >actual &&
 				test_cmp expect actual &&
 
 				git refs list | grep -v "refs/tags/1" >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"refs list" "$method" >actual &&
 				test_cmp expect actual
 			)
 		'
 
-		test_expect_success "with worktree and $to_format backend, $dir dir" '
+		test_expect_success "$method: with worktree and $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo wt" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -138,22 +189,92 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				git config set core.repositoryformatversion 1 &&
-				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
-
-				git worktree add ../wt 2
-			) &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"worktree add ../wt 2" "$method" &&
 
-			git -C repo for-each-ref --include-root-refs >expect &&
-			git -C wt for-each-ref --include-root-refs >expect &&
-			! test_cmp expect actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >expect &&
+				! test_cmp expect actual &&
 
-			git -C wt rev-parse 2 >expect &&
-			git -C wt rev-parse HEAD >actual &&
-			test_cmp expect actual
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse 2" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse HEAD" "$method" >expect &&
+				test_cmp expect actual
+			)
 		'
 	done # closes dir
+
+	test_expect_success "migrating repository to $to_format with alternate refs directory" '
+		test_when_finished "rm -rf repo refdir" &&
+		mkdir refdir &&
+		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
+		(
+			cd repo &&
+
+			test_commit 1 &&
+			test_commit 2 &&
+			test_commit 3 &&
+
+			git refs migrate --ref-format=$to_format &&
+			git refs list >out &&
+			test_grep "refs/tags/1"	out &&
+			test_grep "refs/tags/2"	out &&
+			test_grep "refs/tags/3"	out
+		)
+	'
+
 done # closes to_format
 done # closes from_format
 
+done # closes method
+
+test_expect_success 'initializing repository with alt ref directory' '
+	test_when_finished "rm -rf repo refdir" &&
+	mkdir refdir &&
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
+	verify_files_exist repo/.git refdir &&
+	(
+		cd repo &&
+
+		git config get extensions.refstorage >expect &&
+		echo $BACKEND >actual &&
+		test_cmp expect actual &&
+
+		test_commit 1 &&
+		test_commit 2 &&
+		test_commit 3 &&
+		git refs list >out &&
+		test_grep "refs/tags/1"	out &&
+		test_grep "refs/tags/2"	out &&
+		test_grep "refs/tags/3"	out
+	)
+'
+
+test_expect_success 'cloning repository with alt ref directory' '
+	test_when_finished "rm -rf source repo refdir" &&
+	mkdir refdir &&
+
+	git init source &&
+	test_commit -C source 1 &&
+	test_commit -C source 2 &&
+	test_commit -C source 3 &&
+
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
+
+	git -C repo config get extensions.refstorage >expect &&
+	echo $BACKEND >actual &&
+	test_cmp expect actual &&
+
+	verify_files_exist repo/.git refdir &&
+
+	git -C source for-each-ref refs/tags/ >expect &&
+	git -C repo for-each-ref refs/tags/ >actual &&
+	test_cmp expect actual
+'
+
 test_done

-- 
2.53.GIT

Re: [PATCH v8 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Toon Claes]

Karthik Nayak <karthik.188@gmail.com> writes:

> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
> index 9912433b8c..1a7b8eadba 100755
> --- a/t/t1423-ref-backend.sh
> +++ b/t/t1423-ref-backend.sh
> @@ -138,22 +189,92 @@ do
>  				git refs migrate --dry-run --ref-format=$to_format >out &&
>  				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
>  
> -				git config set core.repositoryformatversion 1 &&
> -				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
> -
> -				git worktree add ../wt 2
> -			) &&
> +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
> +					"worktree add ../wt 2" "$method" &&
>  
> -			git -C repo for-each-ref --include-root-refs >expect &&
> -			git -C wt for-each-ref --include-root-refs >expect &&
> -			! test_cmp expect actual &&
> +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
> +					"for-each-ref --include-root-refs" "$method" >actual &&
> +				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
> +					"for-each-ref --include-root-refs" "$method" >expect &&
> +				! test_cmp expect actual &&
>  
> -			git -C wt rev-parse 2 >expect &&
> -			git -C wt rev-parse HEAD >actual &&
> -			test_cmp expect actual
> +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
> +					"rev-parse 2" "$method" >actual &&
> +				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
> +					"rev-parse HEAD" "$method" >expect &&
> +				test_cmp expect actual
> +			)
>  		'
>  	done # closes dir
> +
> +	test_expect_success "migrating repository to $to_format with alternate refs directory" '
> +		test_when_finished "rm -rf repo refdir" &&
> +		mkdir refdir &&
> +		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
> +		(
> +			cd repo &&
> +
> +			test_commit 1 &&
> +			test_commit 2 &&
> +			test_commit 3 &&
> +
> +			git refs migrate --ref-format=$to_format &&
> +			git refs list >out &&
> +			test_grep "refs/tags/1"	out &&
> +			test_grep "refs/tags/2"	out &&
> +			test_grep "refs/tags/3"	out
> +		)
> +	'
> +
>  done # closes to_format
>  done # closes from_format
>  
> +done # closes method
> +
> +test_expect_success 'initializing repository with alt ref directory' '
> +	test_when_finished "rm -rf repo refdir" &&
> +	mkdir refdir &&
> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
> +	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
> +	verify_files_exist repo/.git refdir &&
> +	(
> +		cd repo &&
> +
> +		git config get extensions.refstorage >expect &&
> +		echo $BACKEND >actual &&

Shouldn't these two be swapped, like:

		git config get extensions.refstorage >actual &&
		echo $BACKEND >expect &&

> +		test_cmp expect actual &&
> +
> +		test_commit 1 &&
> +		test_commit 2 &&
> +		test_commit 3 &&
> +		git refs list >out &&
> +		test_grep "refs/tags/1"	out &&
> +		test_grep "refs/tags/2"	out &&
> +		test_grep "refs/tags/3"	out
> +	)
> +'
> +
> +test_expect_success 'cloning repository with alt ref directory' '
> +	test_when_finished "rm -rf source repo refdir" &&
> +	mkdir refdir &&
> +
> +	git init source &&
> +	test_commit -C source 1 &&
> +	test_commit -C source 2 &&
> +	test_commit -C source 3 &&
> +
> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
> +	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
> +
> +	git -C repo config get extensions.refstorage >expect &&
> +	echo $BACKEND >actual &&
> +	test_cmp expect actual &&

Same here.

> +
> +	verify_files_exist repo/.git refdir &&
> +
> +	git -C source for-each-ref refs/tags/ >expect &&
> +	git -C repo for-each-ref refs/tags/ >actual &&
> +	test_cmp expect actual
> +'
> +
>  test_done
>
> -- 
> 2.53.GIT
>
>

-- 
Cheers,
Toon

Re: [PATCH v8 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3643 bytes --]

Toon Claes <toon@iotcl.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
>> index 9912433b8c..1a7b8eadba 100755
>> --- a/t/t1423-ref-backend.sh
>> +++ b/t/t1423-ref-backend.sh
>> @@ -138,22 +189,92 @@ do
>>  				git refs migrate --dry-run --ref-format=$to_format >out &&
>>  				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
>>
>> -				git config set core.repositoryformatversion 1 &&
>> -				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
>> -
>> -				git worktree add ../wt 2
>> -			) &&
>> +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
>> +					"worktree add ../wt 2" "$method" &&
>>
>> -			git -C repo for-each-ref --include-root-refs >expect &&
>> -			git -C wt for-each-ref --include-root-refs >expect &&
>> -			! test_cmp expect actual &&
>> +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
>> +					"for-each-ref --include-root-refs" "$method" >actual &&
>> +				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
>> +					"for-each-ref --include-root-refs" "$method" >expect &&
>> +				! test_cmp expect actual &&
>>
>> -			git -C wt rev-parse 2 >expect &&
>> -			git -C wt rev-parse HEAD >actual &&
>> -			test_cmp expect actual
>> +				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
>> +					"rev-parse 2" "$method" >actual &&
>> +				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
>> +					"rev-parse HEAD" "$method" >expect &&
>> +				test_cmp expect actual
>> +			)
>>  		'
>>  	done # closes dir
>> +
>> +	test_expect_success "migrating repository to $to_format with alternate refs directory" '
>> +		test_when_finished "rm -rf repo refdir" &&
>> +		mkdir refdir &&
>> +		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
>> +		(
>> +			cd repo &&
>> +
>> +			test_commit 1 &&
>> +			test_commit 2 &&
>> +			test_commit 3 &&
>> +
>> +			git refs migrate --ref-format=$to_format &&
>> +			git refs list >out &&
>> +			test_grep "refs/tags/1"	out &&
>> +			test_grep "refs/tags/2"	out &&
>> +			test_grep "refs/tags/3"	out
>> +		)
>> +	'
>> +
>>  done # closes to_format
>>  done # closes from_format
>>
>> +done # closes method
>> +
>> +test_expect_success 'initializing repository with alt ref directory' '
>> +	test_when_finished "rm -rf repo refdir" &&
>> +	mkdir refdir &&
>> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
>> +	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
>> +	verify_files_exist repo/.git refdir &&
>> +	(
>> +		cd repo &&
>> +
>> +		git config get extensions.refstorage >expect &&
>> +		echo $BACKEND >actual &&
>
> Shouldn't these two be swapped, like:
>
> 		git config get extensions.refstorage >actual &&
> 		echo $BACKEND >expect &&
>
>> +		test_cmp expect actual &&
>> +
>> +		test_commit 1 &&
>> +		test_commit 2 &&
>> +		test_commit 3 &&
>> +		git refs list >out &&
>> +		test_grep "refs/tags/1"	out &&
>> +		test_grep "refs/tags/2"	out &&
>> +		test_grep "refs/tags/3"	out
>> +	)
>> +'
>> +
>> +test_expect_success 'cloning repository with alt ref directory' '
>> +	test_when_finished "rm -rf source repo refdir" &&
>> +	mkdir refdir &&
>> +
>> +	git init source &&
>> +	test_commit -C source 1 &&
>> +	test_commit -C source 2 &&
>> +	test_commit -C source 3 &&
>> +
>> +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
>> +	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
>> +
>> +	git -C repo config get extensions.refstorage >expect &&
>> +	echo $BACKEND >actual &&
>> +	test_cmp expect actual &&
>
> Same here.
>

Makes sense. Will amend.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v8 0/6] refs: allow setting the reference directory

[Patrick Steinhardt]

On Mon, Feb 23, 2026 at 09:01:35AM +0100, Karthik Nayak wrote:
> Changes in v8:
> - Fix a typo/grammar in commit 4.
> - In the final commits tests, avoid creating a file for text
>   comparison.
> - Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com

Thanks, I'm happy now with the current version based on the range-diff.

Patrick

Re: [PATCH v8 0/6] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 454 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 23, 2026 at 09:01:35AM +0100, Karthik Nayak wrote:
>> Changes in v8:
>> - Fix a typo/grammar in commit 4.
>> - In the final commits tests, avoid creating a file for text
>>   comparison.
>> - Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com
>
> Thanks, I'm happy now with the current version based on the range-diff.
>
> Patrick

Thanks for the reviews.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH v8 0/6] refs: allow setting the reference directory

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> Patrick Steinhardt <ps@pks.im> writes:
>
>> On Mon, Feb 23, 2026 at 09:01:35AM +0100, Karthik Nayak wrote:
>>> Changes in v8:
>>> - Fix a typo/grammar in commit 4.
>>> - In the final commits tests, avoid creating a file for text
>>>   comparison.
>>> - Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com
>>
>> Thanks, I'm happy now with the current version based on the range-diff.
>>
>> Patrick
>
> Thanks for the reviews.

Let me replace what I have with this latest and mark the topic for
'next', then.

Thanks, all!

Re: [PATCH v8 0/6] refs: allow setting the reference directory

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

Junio C Hamano <gitster@pobox.com> writes:

> Karthik Nayak <karthik.188@gmail.com> writes:
>
>> Patrick Steinhardt <ps@pks.im> writes:
>>
>>> On Mon, Feb 23, 2026 at 09:01:35AM +0100, Karthik Nayak wrote:
>>>> Changes in v8:
>>>> - Fix a typo/grammar in commit 4.
>>>> - In the final commits tests, avoid creating a file for text
>>>>   comparison.
>>>> - Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com
>>>
>>> Thanks, I'm happy now with the current version based on the range-diff.
>>>
>>> Patrick
>>
>> Thanks for the reviews.
>
> Let me replace what I have with this latest and mark the topic for
> 'next', then.
>
> Thanks, all!

There was an issue with the doc rendering and small nits that Toon
suggested, so I pushed a new version. Hope you can replace it with the
latest for 'next'.

Thanks and sorry for the churn.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v9 0/6] refs: allow setting the reference directory

[Karthik Nayak]

While Git allows users to select different reference backends, unlike
with objects, there is no flexibility in selecting the reference
directory. Currently, the reference format is obtained from the config
of the repository and the reference directory is set to the $GIT_DIR.

This patch extends the config 'extensions.refStorage' to take in
multiple forms of inputs:

  - A format name alone (e.g., `reftable` or `files`).

  - A URI format `<format>://<payload>` explicitly specifies both the
    format and payload (e.g., `reftable:///foo/bar`).

We also add in a new ENV variable GIT_REFERENCE_BACKEND which can be
used to override the config.

One use case for this is migration between different backends. On the
server side, migrating from the files backend to the newly introduced
reftable backend can be achieved by running 'git refs migrate'. However,
for large repositories with millions of references, this migration can
take from seconds to minutes.

For some background, at GitLab, the criteria for our migration was to
reduce the downtime of the migrate ideally to zero. So running 'git refs
migrate --ref-format=reftable' by itself wouldn't work, since it scales
with the number of references and we have repos with millions of
references, so we need to migrate without loosing any information. We
came up with the following plan:

  1. Run git-pack-refs(1) and note timestamp of the generated packed-refs
     file.
  2. Run git refs migrate –dry-run.
  3. If there are no ongoing reference requests (read/write)
     a. Lock the repository by blocking incoming requests (done on a
        layer above git, in Gitaly [1]).
     b. If the timestamp of the packed-refs file has changed, unlock
        the repo and repeat from step 1.
     c. Apply all the loose refs to the dry-run reftable folder (this
        requires support in Git to write refs to arbitrary folder).
     d. Move the reftable dry-run folder into the GIT_DIR.
     e. Swap the repo config
     f. Unlock repo access

Using such a route, scales much better since we only have to worry about
blocking the repository by O(ref written between #1 and #3a) and not
O(refs in repo). But for doing so, we need to be able to write to a
arbitrary reference backend + path. This is to add the missing
references to the dry-run reftable folder. This series, achieves that.

Since there was a long gap between v3 <> v4, the version 4 onward is
based on top of 2258446484 (RelNotes: correct "fast-import" option name,
2026-01-30).

[1]: https://gitlab.com/gitlab-org/gitaly

---
Changes in v9:
- There was a issue with how the docs were parsed, causing the list to
  not be rendered correctly.
- Some small other nits.
- Link to v8: https://patch.msgid.link/20260223-kn-alternate-ref-dir-v8-0-0509c132a203@gmail.com

Changes in v8:
- Fix a typo/grammar in commit 4.
- In the final commits tests, avoid creating a file for text
  comparison.
- Link to v7: https://patch.msgid.link/20260219-kn-alternate-ref-dir-v7-0-16f27860dbdf@gmail.com

Changes in v7:
- Add more details in the commit messages.
- Cleanup some whitespace.
- Reorder the commits to be group related changes together.
- Add checks for stubs in the tests when creating new repos.
- Link to v6: https://patch.msgid.link/20260214-kn-alternate-ref-dir-v6-0-86a82c77cf59@gmail.com

Changes in v6:
- The biggest change in this version is that we now support using the
  environment variable with 'git-clone(1)' and 'git-init(1)'. In such
  situations, the alternate reference directory is created and the
  config is added to the repository.
- Add a new commit which moves stub creation/removal to the generic
  layer.
- Cleanup logic flow in `refs_compute_filesystem_location()`.
- Add more tests for usage with 'git-clone(1)', 'git-init(1)' and
  migration of repositories using alternate refs backend.
- Fixup documentation, commit messages and typos.
- Link to v5: https://patch.msgid.link/20260209-kn-alternate-ref-dir-v5-0-740899834ceb@gmail.com

Changes in v5:
- Moved around the commits, to ensure that the code to handle the config
  in the backend is first. Previously, we added the config first, which
  meant the commit allowed users to provide a URI but it was simply
  ignore.
- Fix typos and grammar and rename variables.
- Clean up the description and documentation to actually specify
  protocol over location.
- Avoid an extra memory allocation by detaching the strbuf value.
- Link to v4: https://patch.msgid.link/20260202-kn-alternate-ref-dir-v4-0-3b30430411e3@gmail.com

Changes in v4:
- Mostly re-wrote the code to also support worktree. Now, the existing
  backends will store worktree references in 'ref_dir/worktrees/wt_id'
  and add corresponding stubs in 'git_dir/worktrees/wt_id'.
- We also support relative paths in the reference directories. These
  relative paths are resolved relative to the GIT_DIR.
- Link to v3: https://patch.msgid.link/20251201-kn-alternate-ref-dir-v3-0-c11b946bc2fa@gmail.com

Changes in v3:
- Cleanup some stale code which wasn't removed.
- Localize strings which will be output to the user.
- Remove additional defensive checks which are not needed.
- Link to v2: https://patch.msgid.link/20251126-kn-alternate-ref-dir-v2-0-8b9f6f18f635@gmail.com

Changes in v2:
- Added more clarification and proper intent in the cover message.
- Changed the format from '<ref_backend>://<path>' to
  `<ref_backend>://<URI-for-resource>` as it much clearer.
- Added logic to check for the '//' in the provided URI and a test for
  the same.
- In the tests:
  - Use test_must_fail() instead of ! git
  - Fix looped tests not using the variables correctly and ensure that
    the test description is correct.
- Link to v1: https://patch.msgid.link/20251119-kn-alternate-ref-dir-v1-0-4cf4a94c8bed@gmail.com

---
 Documentation/config/extensions.adoc |  16 +-
 Documentation/git.adoc               |   5 +
 builtin/clone.c                      |   9 +-
 builtin/worktree.c                   |  34 +++++
 environment.h                        |   1 +
 refs.c                               | 126 +++++++++++++++-
 refs.h                               |  13 ++
 refs/files-backend.c                 |  23 ++-
 refs/packed-backend.c                |   5 +
 refs/packed-backend.h                |   1 +
 refs/refs-internal.h                 |  14 ++
 refs/reftable-backend.c              |  61 ++------
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  96 ++++++++++--
 setup.h                              |   4 +-
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 280 +++++++++++++++++++++++++++++++++++
 18 files changed, 625 insertions(+), 81 deletions(-)

Karthik Nayak (6):
      setup: don't modify repo in `create_reference_database()`
      refs: extract out `refs_create_refdir_stubs()`
      refs: move out stub modification to generic layer
      refs: receive and use the reference storage payload
      refs: allow reference location in refstorage config
      refs: add GIT_REFERENCE_BACKEND to specify reference backend

Range-diff versus v8:

1:  9f1978e991 = 1:  71504213ca setup: don't modify repo in `create_reference_database()`
2:  75013d6874 ! 2:  f372287ec6 refs: extract out `refs_create_refdir_stubs()`
    @@ refs.h: void ref_iterator_free(struct ref_iterator *ref_iterator);
     + * While it is necessary within the files backend, newer backends may not
     + * follow the same structure. To go around this, we create stubs as necessary.
     + *
    -+ * If provided with a 'refs_heads_msg', we create the 'refs/heads/head' file
    ++ * If provided with a 'refs_heads_content', we create the 'refs/heads/head' file
     + * with the provided message.
     + */
     +void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
    -+			      const char *refs_heads_msg);
    ++			      const char *refs_heads_content);
     +
      #endif /* REFS_H */
     
3:  97bef9c5c0 = 3:  b79ac00a3d refs: move out stub modification to generic layer
4:  ed55f79701 = 4:  d0ffa07dfc refs: receive and use the reference storage payload
5:  5902a4588c ! 5:  bf17494952 refs: allow reference location in refstorage config
    @@ Documentation/config/extensions.adoc: For historical reasons, this extension is
     +  format and payload (e.g., `reftable:///foo/bar`).
     +
     +Supported format names are:
    -++
    ++
      include::../ref-storage-format.adoc[]
    -++
    ++
     +The payload is passed directly to the reference backend. For the files and
     +reftable backends, this must be a filesystem path where the references will
     +be stored. Defaulting to the commondir when no payload is provided. Relative
    -+paths are resolved relative to the $GIT_DIR. Future backends may support
    ++paths are resolved relative to the `$GIT_DIR`. Future backends may support
     +other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
      --
      +
6:  7b0f103dbf ! 6:  d1f3323df6 refs: add GIT_REFERENCE_BACKEND to specify reference backend
    @@ t/t1423-ref-backend.sh: do
     +	(
     +		cd repo &&
     +
    -+		git config get extensions.refstorage >expect &&
    -+		echo $BACKEND >actual &&
    ++		git config get extensions.refstorage >actual &&
    ++		echo $BACKEND >expect &&
     +		test_cmp expect actual &&
     +
     +		test_commit 1 &&
    @@ t/t1423-ref-backend.sh: do
     +	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
     +	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
     +
    -+	git -C repo config get extensions.refstorage >expect &&
    -+	echo $BACKEND >actual &&
    ++	git -C repo config get extensions.refstorage >actual &&
    ++	echo $BACKEND >expect &&
     +	test_cmp expect actual &&
     +
     +	verify_files_exist repo/.git refdir &&


base-commit: 22584464849815268419fd9d2eba307362360db1
change-id: 20251105-kn-alternate-ref-dir-3e572e8cd0ef

Thanks
- Karthik

[PATCH v9 1/6] setup: don't modify repo in `create_reference_database()`

[Karthik Nayak]

The `create_reference_database()` function is used to create the
reference database during initialization of a repository. The function
calls `repo_set_ref_storage_format()` to set the repositories reference
format. This is an unexpected side-effect of the function. More so
because the function is only called in two locations:

  1. During git-init(1) where the value is propagated from the `struct
     repository_format repo_fmt` value.

  2. During git-clone(1) where the value is propagated from the
     `the_repository` value.

The former is valid, however the flow already calls
`repo_set_ref_storage_format()`, so this effort is simply duplicated.
The latter sets the existing value in `the_repository` back to itself.
While this is okay for now, introduction of more fields in
`repo_set_ref_storage_format()` would cause issues, especially
dynamically allocated strings, where we would free/allocate the same
string back into `the_repostiory`.

To avoid all this confusion, clean up the function to no longer take in
and set the repo's reference storage format.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c | 2 +-
 setup.c         | 7 ++-----
 setup.h         | 3 +--
 3 files changed, 4 insertions(+), 8 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index b40cee5968..cd43bb5aa2 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1442,7 +1442,7 @@ int cmd_clone(int argc,
 	hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport));
 	initialize_repository_version(hash_algo, the_repository->ref_storage_format, 1);
 	repo_set_hash_algo(the_repository, hash_algo);
-	create_reference_database(the_repository->ref_storage_format, NULL, 1);
+	create_reference_database(NULL, 1);
 
 	/*
 	 * Before fetching from the remote, download and install bundle
diff --git a/setup.c b/setup.c
index b723f8b339..1fc9ae3872 100644
--- a/setup.c
+++ b/setup.c
@@ -2359,14 +2359,12 @@ static int is_reinit(void)
 	return ret;
 }
 
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet)
+void create_reference_database(const char *initial_branch, int quiet)
 {
 	struct strbuf err = STRBUF_INIT;
 	char *to_free = NULL;
 	int reinit = is_reinit();
 
-	repo_set_ref_storage_format(the_repository, ref_storage_format);
 	if (ref_store_create_on_disk(get_main_ref_store(the_repository), 0, &err))
 		die("failed to set up refs db: %s", err.buf);
 
@@ -2701,8 +2699,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 				      &repo_fmt, init_shared_repository);
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
-		create_reference_database(repo_fmt.ref_storage_format,
-					  initial_branch, flags & INIT_DB_QUIET);
+		create_reference_database(initial_branch, flags & INIT_DB_QUIET);
 	create_object_directory();
 
 	if (repo_settings_get_shared_repository(the_repository)) {
diff --git a/setup.h b/setup.h
index d55dcc6608..ddb9f6701c 100644
--- a/setup.h
+++ b/setup.h
@@ -240,8 +240,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 void initialize_repository_version(int hash_algo,
 				   enum ref_storage_format ref_storage_format,
 				   int reinit);
-void create_reference_database(enum ref_storage_format ref_storage_format,
-			       const char *initial_branch, int quiet);
+void create_reference_database(const char *initial_branch, int quiet);
 
 /*
  * NOTE NOTE NOTE!!

-- 
2.53.GIT

[PATCH v9 2/6] refs: extract out `refs_create_refdir_stubs()`

[Karthik Nayak]

For Git to recognize a directory as a Git directory, it requires the
directory to contain:

  1. 'HEAD' file
  2. 'objects/' directory
  3. 'refs/' directory

Here, #1 and #3 are part of the reference storage mechanism,
specifically the files backend. Since then, newer backends such as the
reftable backend have moved to using their own path ('reftable/') for
storing references. But to ensure Git still recognizes the directory as
a Git directory, we create stubs.

There are two locations where we create stubs:

- In 'refs/reftable-backend.c' when creating the reftable backend.
- In 'clone.c' before spawning transport helpers.

In a following commit, we'll add another instance. So instead of
repeating the code, let's extract out this code to
`refs_create_refdir_stubs()` and use it.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 builtin/clone.c         |  7 +------
 refs.c                  | 23 +++++++++++++++++++++++
 refs.h                  | 13 +++++++++++++
 refs/reftable-backend.c | 14 ++------------
 4 files changed, 39 insertions(+), 18 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index cd43bb5aa2..697c5bb5cb 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1225,12 +1225,7 @@ int cmd_clone(int argc,
 	initialize_repository_version(GIT_HASH_UNKNOWN,
 				      the_repository->ref_storage_format, 1);
 
-	strbuf_addf(&buf, "%s/HEAD", git_dir);
-	write_file(buf.buf, "ref: refs/heads/.invalid");
-
-	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/refs", git_dir);
-	safe_create_dir(the_repository, buf.buf, 1);
+	refs_create_refdir_stubs(the_repository, git_dir, NULL);
 
 	/*
 	 * additional config can be injected with -c, make sure it's included
diff --git a/refs.c b/refs.c
index 627b7f8698..77b93d655b 100644
--- a/refs.c
+++ b/refs.c
@@ -2163,6 +2163,29 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 	return NULL;
 }
 
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_content)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	strbuf_addf(&path, "%s/HEAD", refdir);
+	write_file(path.buf, "ref: refs/heads/.invalid");
+	adjust_shared_perm(repo, path.buf);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/refs", refdir);
+	safe_create_dir(repo, path.buf, 1);
+
+	if (refs_heads_content) {
+		strbuf_reset(&path);
+		strbuf_addf(&path, "%s/refs/heads", refdir);
+		write_file(path.buf, "%s", refs_heads_content);
+		adjust_shared_perm(repo, path.buf);
+	}
+
+	strbuf_release(&path);
+}
+
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
diff --git a/refs.h b/refs.h
index f0abfa1d93..a35fdc6642 100644
--- a/refs.h
+++ b/refs.h
@@ -1427,4 +1427,17 @@ void ref_iterator_free(struct ref_iterator *ref_iterator);
 int do_for_each_ref_iterator(struct ref_iterator *iter,
 			     each_ref_fn fn, void *cb_data);
 
+/*
+ * Git only recognizes a directory as a repository if it contains:
+ * - HEAD file
+ * - refs/ folder
+ * While it is necessary within the files backend, newer backends may not
+ * follow the same structure. To go around this, we create stubs as necessary.
+ *
+ * If provided with a 'refs_heads_content', we create the 'refs/heads/head' file
+ * with the provided message.
+ */
+void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
+			      const char *refs_heads_content);
+
 #endif /* REFS_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index fe74af73af..d8651fe779 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,18 +491,8 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	write_file(sb.buf, "ref: refs/heads/.invalid");
-	adjust_shared_perm(the_repository, sb.buf);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	safe_create_dir(the_repository, sb.buf, 1);
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	write_file(sb.buf, "this repository uses the reftable format");
-	adjust_shared_perm(the_repository, sb.buf);
+	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
+				 "this repository uses the reftable format");
 
 	strbuf_release(&sb);
 	return 0;

-- 
2.53.GIT

[PATCH v9 3/6] refs: move out stub modification to generic layer

[Karthik Nayak]

When creating the reftable reference backend on disk, we create stubs to
ensure that the directory can be recognized as a Git repository. This is
done by calling `refs_create_refdir_stubs()`. Move this to the generic
layer as this is needed for all backends excluding from the files
backends. In an upcoming commit where we introduce alternate reference
backend locations, we'll have to also create stubs in the $GIT_DIR
irrespective of the backend being used. This commit builds the base to
add that logic.

Similarly, move the logic for deletion of stubs to the generic layer.
The files backend recursively calls the remove function of the
'packed-backend', here skip calling the generic function since that
would try to delete stubs.

Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 47 +++++++++++++++++++++++++++++++++++++++++++++--
 refs/files-backend.c    |  6 +++++-
 refs/reftable-backend.c | 27 ---------------------------
 3 files changed, 50 insertions(+), 30 deletions(-)

diff --git a/refs.c b/refs.c
index 77b93d655b..c83af63dc5 100644
--- a/refs.c
+++ b/refs.c
@@ -2189,12 +2189,55 @@ void refs_create_refdir_stubs(struct repository *repo, const char *refdir,
 /* backend functions */
 int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *err)
 {
-	return refs->be->create_on_disk(refs, flags, err);
+	int ret = refs->be->create_on_disk(refs, flags, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf msg = STRBUF_INIT;
+
+		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+		strbuf_release(&msg);
+	}
+
+	return ret;
 }
 
 int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
-	return refs->be->remove_on_disk(refs, err);
+	int ret = refs->be->remove_on_disk(refs, err);
+
+	if (!ret &&
+	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+		struct strbuf sb = STRBUF_INIT;
+
+		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub HEAD: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs/heads", refs->gitdir);
+		if (unlink(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete stub heads: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+		strbuf_reset(&sb);
+
+		strbuf_addf(&sb, "%s/refs", refs->gitdir);
+		if (rmdir(sb.buf) < 0) {
+			strbuf_addf(err, "could not delete refs directory: %s",
+				    strerror(errno));
+			ret = -1;
+		}
+
+		strbuf_release(&sb);
+	}
+
+	return ret;
 }
 
 int repo_resolve_gitlink_ref(struct repository *r,
diff --git a/refs/files-backend.c b/refs/files-backend.c
index 240d3c3b26..d3f6423261 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -3700,7 +3700,11 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,
 	if (for_each_root_ref(refs, remove_one_root_ref, &data) < 0)
 		ret = -1;
 
-	if (ref_store_remove_on_disk(refs->packed_ref_store, err) < 0)
+	/*
+	 * Directly access the cleanup functions for packed-refs as the generic function
+	 * would try to clear stubs which isn't required for the files backend.
+	 */
+	if (refs->packed_ref_store->be->remove_on_disk(refs->packed_ref_store, err) < 0)
 		ret = -1;
 
 	strbuf_release(&sb);
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index d8651fe779..6ce7f9bb8e 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -491,9 +491,6 @@ static int reftable_be_create_on_disk(struct ref_store *ref_store,
 	safe_create_dir(the_repository, sb.buf, 1);
 	strbuf_reset(&sb);
 
-	refs_create_refdir_stubs(the_repository, refs->base.gitdir,
-				 "this repository uses the reftable format");
-
 	strbuf_release(&sb);
 	return 0;
 }
@@ -519,30 +516,6 @@ static int reftable_be_remove_on_disk(struct ref_store *ref_store,
 			    strerror(errno));
 		ret = -1;
 	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/HEAD", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub HEAD: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs/heads", refs->base.gitdir);
-	if (unlink(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete stub heads: %s",
-			    strerror(errno));
-		ret = -1;
-	}
-	strbuf_reset(&sb);
-
-	strbuf_addf(&sb, "%s/refs", refs->base.gitdir);
-	if (rmdir(sb.buf) < 0) {
-		strbuf_addf(err, "could not delete refs directory: %s",
-			    strerror(errno));
-		ret = -1;
-	}
 
 	strbuf_release(&sb);
 	return ret;

-- 
2.53.GIT

[PATCH v9 4/6] refs: receive and use the reference storage payload

[Karthik Nayak]

An upcoming commit will add support for providing an URI via the
'extensions.refStorage' config. The URI will contain the reference
backend and a corresponding payload. The payload can be then used for
providing an alternate locations for the reference backend.

To prepare for this, modify the existing backends to accept such an
argument when initializing via the 'init()' function. Both the files
and reftable backends will parse the information to be filesystem paths
to store references. Given that no callers pass any payload yet this is
essentially a no-op change for now.

To enable this, provide a 'refs_compute_filesystem_location()' function
which will parse the current 'gitdir' and the 'payload' to provide the
final reference directory and common reference directory (if working in
a linked worktree).

The documentation and tests will be added alongside the extension of the
config variable.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 refs.c                  | 40 +++++++++++++++++++++++++++++++++++++++-
 refs/files-backend.c    | 17 ++++++++++++-----
 refs/packed-backend.c   |  5 +++++
 refs/packed-backend.h   |  1 +
 refs/refs-internal.h    | 14 ++++++++++++++
 refs/reftable-backend.c | 24 ++++++++++++++----------
 6 files changed, 85 insertions(+), 16 deletions(-)

diff --git a/refs.c b/refs.c
index c83af63dc5..ba2573eb7a 100644
--- a/refs.c
+++ b/refs.c
@@ -5,6 +5,7 @@
 #define USE_THE_REPOSITORY_VARIABLE
 
 #include "git-compat-util.h"
+#include "abspath.h"
 #include "advice.h"
 #include "config.h"
 #include "environment.h"
@@ -2290,7 +2291,7 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, gitdir, flags);
+	refs = be->init(repo, NULL, gitdir, flags);
 	return refs;
 }
 
@@ -3468,3 +3469,40 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)
 		return "unknown failure";
 	}
 }
+
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir)
+{
+	struct strbuf sb = STRBUF_INIT;
+
+	*is_worktree = get_common_dir_noenv(ref_common_dir, gitdir);
+
+	if (!payload) {
+		/*
+		 * We can use the 'gitdir' as the 'refdir' without appending the
+		 * worktree path, as the 'gitdir' here is already the worktree
+		 * path and is different from 'commondir' denoted by 'ref_common_dir'.
+		 */
+		strbuf_addstr(refdir, gitdir);
+		return;
+	}
+
+	if (!is_absolute_path(payload)) {
+		strbuf_addf(&sb, "%s/%s", ref_common_dir->buf, payload);
+		strbuf_realpath(ref_common_dir, sb.buf, 1);
+	} else {
+		strbuf_realpath(ref_common_dir, payload, 1);
+	}
+
+	strbuf_addbuf(refdir, ref_common_dir);
+
+	if (*is_worktree) {
+		const char *wt_id = strrchr(gitdir, '/');
+		if (!wt_id)
+			BUG("worktree path does not contain slash");
+		strbuf_addf(refdir, "/worktrees/%s", wt_id + 1);
+	}
+
+	strbuf_release(&sb);
+}
diff --git a/refs/files-backend.c b/refs/files-backend.c
index d3f6423261..9cde3ba724 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -106,19 +106,24 @@ static void clear_loose_ref_cache(struct files_ref_store *refs)
  * set of caches.
  */
 static struct ref_store *files_ref_store_init(struct repository *repo,
+					      const char *payload,
 					      const char *gitdir,
 					      unsigned int flags)
 {
 	struct files_ref_store *refs = xcalloc(1, sizeof(*refs));
 	struct ref_store *ref_store = (struct ref_store *)refs;
-	struct strbuf sb = STRBUF_INIT;
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
+	bool is_worktree;
+
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
 
-	base_ref_store_init(ref_store, repo, gitdir, &refs_be_files);
+	base_ref_store_init(ref_store, repo, refdir.buf, &refs_be_files);
 	refs->store_flags = flags;
-	get_common_dir_noenv(&sb, gitdir);
-	refs->gitcommondir = strbuf_detach(&sb, NULL);
+	refs->gitcommondir = strbuf_detach(&ref_common_dir, NULL);
 	refs->packed_ref_store =
-		packed_ref_store_init(repo, refs->gitcommondir, flags);
+		packed_ref_store_init(repo, NULL, refs->gitcommondir, flags);
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
 	repo_config_get_bool(repo, "core.prefersymlinkrefs", &refs->prefer_symlink_refs);
 
@@ -126,6 +131,8 @@ static struct ref_store *files_ref_store_init(struct repository *repo,
 	chdir_notify_reparent("files-backend $GIT_COMMONDIR",
 			      &refs->gitcommondir);
 
+	strbuf_release(&refdir);
+
 	return ref_store;
 }
 
diff --git a/refs/packed-backend.c b/refs/packed-backend.c
index 4ea0c12299..e7bb9f10f9 100644
--- a/refs/packed-backend.c
+++ b/refs/packed-backend.c
@@ -211,7 +211,12 @@ static size_t snapshot_hexsz(const struct snapshot *snapshot)
 	return snapshot->refs->base.repo->hash_algo->hexsz;
 }
 
+/*
+ * Since packed-refs is only stored in the common dir, don't parse the
+ * payload and rely on the files-backend to set 'gitdir' correctly.
+ */
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload UNUSED,
 					const char *gitdir,
 					unsigned int store_flags)
 {
diff --git a/refs/packed-backend.h b/refs/packed-backend.h
index 9481d5e7c2..2c2377a356 100644
--- a/refs/packed-backend.h
+++ b/refs/packed-backend.h
@@ -14,6 +14,7 @@ struct ref_transaction;
  */
 
 struct ref_store *packed_ref_store_init(struct repository *repo,
+					const char *payload,
 					const char *gitdir,
 					unsigned int store_flags);
 
diff --git a/refs/refs-internal.h b/refs/refs-internal.h
index c7d2a6e50b..4fb8fdb872 100644
--- a/refs/refs-internal.h
+++ b/refs/refs-internal.h
@@ -389,6 +389,7 @@ struct ref_store;
  * the ref_store and to record the ref_store for later lookup.
  */
 typedef struct ref_store *ref_store_init_fn(struct repository *repo,
+					    const char *payload,
 					    const char *gitdir,
 					    unsigned int flags);
 /*
@@ -666,4 +667,17 @@ enum ref_transaction_error refs_verify_refnames_available(struct ref_store *refs
 					  unsigned int initial_transaction,
 					  struct strbuf *err);
 
+/*
+ * Given a gitdir and the reference storage payload provided, retrieve the
+ * 'refdir' and 'ref_common_dir'. The former is where references should be
+ * stored for the current worktree, the latter is the common reference
+ * directory if working with a linked worktree. If working with the main
+ * worktree, both values will be the same.
+ *
+ * This is used by backends that store references in the repository directly.
+ */
+void refs_compute_filesystem_location(const char *gitdir, const char *payload,
+				      bool *is_worktree, struct strbuf *refdir,
+				      struct strbuf *ref_common_dir);
+
 #endif /* REFS_REFS_INTERNAL_H */
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 6ce7f9bb8e..0e220d6bb5 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -372,18 +372,24 @@ static int reftable_be_fsync(int fd)
 }
 
 static struct ref_store *reftable_be_init(struct repository *repo,
+					  const char *payload,
 					  const char *gitdir,
 					  unsigned int store_flags)
 {
 	struct reftable_ref_store *refs = xcalloc(1, sizeof(*refs));
+	struct strbuf ref_common_dir = STRBUF_INIT;
+	struct strbuf refdir = STRBUF_INIT;
 	struct strbuf path = STRBUF_INIT;
-	int is_worktree;
+	bool is_worktree;
 	mode_t mask;
 
 	mask = umask(0);
 	umask(mask);
 
-	base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
+	refs_compute_filesystem_location(gitdir, payload, &is_worktree, &refdir,
+					 &ref_common_dir);
+
+	base_ref_store_init(&refs->base, repo, refdir.buf, &refs_be_reftable);
 	strmap_init(&refs->worktree_backends);
 	refs->store_flags = store_flags;
 	refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo);
@@ -419,14 +425,11 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	/*
 	 * Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
 	 * This stack contains both the shared and the main worktree refs.
-	 *
-	 * Note that we don't try to resolve the path in case we have a
-	 * worktree because `get_common_dir_noenv()` already does it for us.
 	 */
-	is_worktree = get_common_dir_noenv(&path, gitdir);
+	strbuf_addbuf(&path, &ref_common_dir);
 	if (!is_worktree) {
 		strbuf_reset(&path);
-		strbuf_realpath(&path, gitdir, 0);
+		strbuf_realpath(&path, ref_common_dir.buf, 0);
 	}
 	strbuf_addstr(&path, "/reftable");
 	refs->err = reftable_backend_init(&refs->main_backend, path.buf,
@@ -443,10 +446,9 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 	 * do it efficiently.
 	 */
 	if (is_worktree) {
-		strbuf_reset(&path);
-		strbuf_addf(&path, "%s/reftable", gitdir);
+		strbuf_addstr(&refdir, "/reftable");
 
-		refs->err = reftable_backend_init(&refs->worktree_backend, path.buf,
+		refs->err = reftable_backend_init(&refs->worktree_backend, refdir.buf,
 						  &refs->write_options);
 		if (refs->err)
 			goto done;
@@ -456,6 +458,8 @@ static struct ref_store *reftable_be_init(struct repository *repo,
 
 done:
 	assert(refs->err != REFTABLE_API_ERROR);
+	strbuf_release(&ref_common_dir);
+	strbuf_release(&refdir);
 	strbuf_release(&path);
 	return &refs->base;
 }

-- 
2.53.GIT

[PATCH v9 5/6] refs: allow reference location in refstorage config

[Karthik Nayak]

The 'extensions.refStorage' config is used to specify the reference
backend for a given repository. Both the 'files' and 'reftable' backends
utilize the $GIT_DIR as the reference folder by default in
`get_main_ref_store()`.

Since the reference backends are pluggable, this means that they could
work with out-of-tree reference directories too. Extend the 'refStorage'
config to also support taking an URI input, where users can specify the
reference backend and the location.

Add the required changes to obtain and propagate this value to the
individual backends. Add the necessary documentation and tests.

Traditionally, for linked worktrees, references were stored in the
'$GIT_DIR/worktrees/<wt_id>' path. But when using an alternate reference
storage path, it doesn't make sense to store the main worktree
references in the new path, and the linked worktree references in the
$GIT_DIR. So, let's store linked worktree references in
'$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'. To do this, create the
necessary files and folders while also adding stubs in the $GIT_DIR path
to ensure that it is still considered a Git directory.

Ideally, we would want to pass in a `struct worktree *` to individual
backends, instead of passing the `gitdir`. This allows them to handle
worktree specific logic. Currently, that is not possible since the
worktree code is:

  - Tied to using the global `the_repository` variable.

  - Is not setup before the reference database during initialization of
    the repository.

Add a TODO in 'refs.c' to ensure we can eventually make that change.

Helped-by: Patrick Steinhardt <ps@pks.im>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/config/extensions.adoc |  16 +++-
 builtin/worktree.c                   |  34 ++++++++
 refs.c                               |   6 +-
 repository.c                         |   9 +-
 repository.h                         |   8 +-
 setup.c                              |  34 +++++++-
 setup.h                              |   1 +
 t/meson.build                        |   1 +
 t/t1423-ref-backend.sh               | 159 +++++++++++++++++++++++++++++++++++
 9 files changed, 259 insertions(+), 9 deletions(-)

diff --git a/Documentation/config/extensions.adoc b/Documentation/config/extensions.adoc
index 532456644b..329d02b3c4 100644
--- a/Documentation/config/extensions.adoc
+++ b/Documentation/config/extensions.adoc
@@ -57,10 +57,24 @@ For historical reasons, this extension is respected regardless of the
 `core.repositoryFormatVersion` setting.
 
 refStorage:::
-	Specify the ref storage format to use. The acceptable values are:
+	Specify the ref storage format and a corresponding payload. The value
+	can be either a format name or a URI:
 +
 --
+* A format name alone (e.g., `reftable` or `files`).
+
+* A URI format `<format>://<payload>` explicitly specifies both the
+  format and payload (e.g., `reftable:///foo/bar`).
+
+Supported format names are:
+
 include::../ref-storage-format.adoc[]
+
+The payload is passed directly to the reference backend. For the files and
+reftable backends, this must be a filesystem path where the references will
+be stored. Defaulting to the commondir when no payload is provided. Relative
+paths are resolved relative to the `$GIT_DIR`. Future backends may support
+other payload schemes, e.g., postgres://127.0.0.1:5432?database=myrepo.
 --
 +
 Note that this setting should only be set by linkgit:git-init[1] or
diff --git a/builtin/worktree.c b/builtin/worktree.c
index fbdaf2eb2e..293e808379 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -425,6 +425,39 @@ static int make_worktree_orphan(const char * ref, const struct add_opts *opts,
 	return run_command(&cp);
 }
 
+/*
+ * References for worktrees are generally stored in '$GIT_DIR/worktrees/<wt_id>'.
+ * But when using alternate reference directories, we want to store the worktree
+ * references in '$ALTERNATE_REFERENCE_DIR/worktrees/<wt_id>'.
+ *
+ * Create the necessary folder structure to facilitate the same. But to ensure
+ * that the former path is still considered a Git directory, add stubs.
+ */
+static void setup_alternate_ref_dir(struct worktree *wt, const char *wt_git_path)
+{
+	struct strbuf sb = STRBUF_INIT;
+	char *path;
+
+	path = wt->repo->ref_storage_payload;
+	if (!path)
+		return;
+
+	if (!is_absolute_path(path))
+		strbuf_addf(&sb, "%s/", wt->repo->commondir);
+
+	strbuf_addf(&sb, "%s/worktrees", path);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_addf(&sb, "/%s", wt->id);
+	safe_create_dir(wt->repo, sb.buf, 1);
+	strbuf_reset(&sb);
+
+	strbuf_addf(&sb, "this worktree stores references in %s/worktrees/%s",
+		    path, wt->id);
+	refs_create_refdir_stubs(wt->repo, wt_git_path, sb.buf);
+
+	strbuf_release(&sb);
+}
+
 static int add_worktree(const char *path, const char *refname,
 			const struct add_opts *opts)
 {
@@ -518,6 +551,7 @@ static int add_worktree(const char *path, const char *refname,
 		ret = error(_("could not find created worktree '%s'"), name);
 		goto done;
 	}
+	setup_alternate_ref_dir(wt, sb_repo.buf);
 	wt_refs = get_worktree_ref_store(wt);
 
 	ret = ref_store_create_on_disk(wt_refs, REF_STORE_CREATE_ON_DISK_IS_WORKTREE, &sb);
diff --git a/refs.c b/refs.c
index ba2573eb7a..ef1902e85c 100644
--- a/refs.c
+++ b/refs.c
@@ -2291,7 +2291,11 @@ static struct ref_store *ref_store_init(struct repository *repo,
 	if (!be)
 		BUG("reference backend is unknown");
 
-	refs = be->init(repo, NULL, gitdir, flags);
+	/*
+	 * TODO Send in a 'struct worktree' instead of a 'gitdir', and
+	 * allow the backend to handle how it wants to deal with worktrees.
+	 */
+	refs = be->init(repo, repo->ref_storage_payload, gitdir, flags);
 	return refs;
 }
 
diff --git a/repository.c b/repository.c
index c7e75215ac..9815f081ef 100644
--- a/repository.c
+++ b/repository.c
@@ -193,9 +193,12 @@ void repo_set_compat_hash_algo(struct repository *repo, int algo)
 }
 
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format)
+				 enum ref_storage_format format,
+				 const char *payload)
 {
 	repo->ref_storage_format = format;
+	free(repo->ref_storage_payload);
+	repo->ref_storage_payload = xstrdup_or_null(payload);
 }
 
 /*
@@ -277,7 +280,8 @@ int repo_init(struct repository *repo,
 
 	repo_set_hash_algo(repo, format.hash_algo);
 	repo_set_compat_hash_algo(repo, format.compat_hash_algo);
-	repo_set_ref_storage_format(repo, format.ref_storage_format);
+	repo_set_ref_storage_format(repo, format.ref_storage_format,
+				    format.ref_storage_payload);
 	repo->repository_format_worktree_config = format.worktree_config;
 	repo->repository_format_relative_worktrees = format.relative_worktrees;
 	repo->repository_format_precious_objects = format.precious_objects;
@@ -369,6 +373,7 @@ void repo_clear(struct repository *repo)
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
+	FREE_AND_NULL(repo->ref_storage_payload);
 
 	odb_free(repo->objects);
 	repo->objects = NULL;
diff --git a/repository.h b/repository.h
index 6063c4b846..95e2333bad 100644
--- a/repository.h
+++ b/repository.h
@@ -150,6 +150,11 @@ struct repository {
 
 	/* Repository's reference storage format, as serialized on disk. */
 	enum ref_storage_format ref_storage_format;
+	/*
+	 * Reference storage information as needed for the backend. This contains
+	 * only the payload from the reference URI without the schema.
+	 */
+	char *ref_storage_payload;
 
 	/* A unique-id for tracing purposes. */
 	int trace2_repo_id;
@@ -204,7 +209,8 @@ void repo_set_worktree(struct repository *repo, const char *path);
 void repo_set_hash_algo(struct repository *repo, int algo);
 void repo_set_compat_hash_algo(struct repository *repo, int compat_algo);
 void repo_set_ref_storage_format(struct repository *repo,
-				 enum ref_storage_format format);
+				 enum ref_storage_format format,
+				 const char *payload);
 void initialize_repository(struct repository *repo);
 RESULT_MUST_BE_USED
 int repo_init(struct repository *r, const char *gitdir, const char *worktree);
diff --git a/setup.c b/setup.c
index 1fc9ae3872..d407f3347b 100644
--- a/setup.c
+++ b/setup.c
@@ -632,6 +632,21 @@ static enum extension_result handle_extension_v0(const char *var,
 		return EXTENSION_UNKNOWN;
 }
 
+static void parse_reference_uri(const char *value, char **format,
+				char **payload)
+{
+	const char *schema_end;
+
+	schema_end = strstr(value, "://");
+	if (!schema_end) {
+		*format = xstrdup(value);
+		*payload = NULL;
+	} else {
+		*format = xstrndup(value, schema_end - value);
+		*payload = xstrdup_or_null(schema_end + 3);
+	}
+}
+
 /*
  * Record any new extensions in this function.
  */
@@ -674,10 +689,17 @@ static enum extension_result handle_extension(const char *var,
 		return EXTENSION_OK;
 	} else if (!strcmp(ext, "refstorage")) {
 		unsigned int format;
+		char *format_str;
 
 		if (!value)
 			return config_error_nonbool(var);
-		format = ref_storage_format_by_name(value);
+
+		parse_reference_uri(value, &format_str,
+				    &data->ref_storage_payload);
+
+		format = ref_storage_format_by_name(format_str);
+		free(format_str);
+
 		if (format == REF_STORAGE_FORMAT_UNKNOWN)
 			return error(_("invalid value for '%s': '%s'"),
 				     "extensions.refstorage", value);
@@ -850,6 +872,7 @@ void clear_repository_format(struct repository_format *format)
 	string_list_clear(&format->v1_only_extensions, 0);
 	free(format->work_tree);
 	free(format->partial_clone);
+	free(format->ref_storage_payload);
 	init_repository_format(format);
 }
 
@@ -1942,7 +1965,8 @@ const char *setup_git_directory_gently(int *nongit_ok)
 			repo_set_compat_hash_algo(the_repository,
 						  repo_fmt.compat_hash_algo);
 			repo_set_ref_storage_format(the_repository,
-						    repo_fmt.ref_storage_format);
+						    repo_fmt.ref_storage_format,
+						    repo_fmt.ref_storage_payload);
 			the_repository->repository_format_worktree_config =
 				repo_fmt.worktree_config;
 			the_repository->repository_format_relative_worktrees =
@@ -2042,7 +2066,8 @@ void check_repository_format(struct repository_format *fmt)
 	repo_set_hash_algo(the_repository, fmt->hash_algo);
 	repo_set_compat_hash_algo(the_repository, fmt->compat_hash_algo);
 	repo_set_ref_storage_format(the_repository,
-				    fmt->ref_storage_format);
+				    fmt->ref_storage_format,
+				    fmt->ref_storage_payload);
 	the_repository->repository_format_worktree_config =
 		fmt->worktree_config;
 	the_repository->repository_format_relative_worktrees =
@@ -2643,7 +2668,8 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
-	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format);
+	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
+				    repo_fmt->ref_storage_payload);
 }
 
 int init_db(const char *git_dir, const char *real_git_dir,
diff --git a/setup.h b/setup.h
index ddb9f6701c..093af39e84 100644
--- a/setup.h
+++ b/setup.h
@@ -171,6 +171,7 @@ struct repository_format {
 	int hash_algo;
 	int compat_hash_algo;
 	enum ref_storage_format ref_storage_format;
+	char *ref_storage_payload;
 	int sparse_index;
 	char *work_tree;
 	struct string_list unknown_extensions;
diff --git a/t/meson.build b/t/meson.build
index 459c52a489..11fc5a49ee 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -210,6 +210,7 @@ integration_tests = [
   't1420-lost-found.sh',
   't1421-reflog-write.sh',
   't1422-show-ref-exists.sh',
+  't1423-ref-backend.sh',
   't1430-bad-ref-name.sh',
   't1450-fsck.sh',
   't1451-fsck-buffer.sh',
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
new file mode 100755
index 0000000000..9912433b8c
--- /dev/null
+++ b/t/t1423-ref-backend.sh
@@ -0,0 +1,159 @@
+#!/bin/sh
+
+test_description='Test reference backend URIs'
+
+. ./test-lib.sh
+
+# Run a git command with the provided reference storage. Reset the backend
+# post running the command.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <cmd> is the git subcommand to be run in the repository.
+run_with_uri() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	cmd=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1
+	git -C "$repo" config set extensions.refStorage "$uri" &&
+	git -C "$repo" $cmd &&
+	git -C "$repo" config set extensions.refStorage "$backend"
+}
+
+# Test a repository with a given reference storage by running and comparing
+# 'git refs list' before and after setting the new reference backend. If
+# err_msg is set, expect the command to fail and grep for the provided err_msg.
+# Usage: run_with_uri <repo> <backend> <uri> <cmd>
+#   <repo> is the relative path to the repo to run the command in.
+#   <backend> is the original ref storage of the repo.
+#   <uri> is the new URI to be set for the ref storage.
+#   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
+test_refs_backend() {
+	repo=$1 &&
+	backend=$2 &&
+	uri=$3 &&
+	err_msg=$4 &&
+
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test -n "$err_msg";
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		test_must_fail git -C "$repo" refs list 2>err &&
+		test_grep "$err_msg" err
+	else
+		git -C "$repo" refs list >expect &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		test_cmp expect actual
+	fi
+}
+
+test_expect_success 'URI is invalid' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable@/home/reftable" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'URI ends with colon' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "reftable:" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+test_expect_success 'unknown reference backend' '
+	test_when_finished "rm -rf repo" &&
+	git init repo &&
+	test_refs_backend repo files "db://.git" \
+		"invalid value for ${SQ}extensions.refstorage${SQ}"
+'
+
+ref_formats="files reftable"
+for from_format in $ref_formats
+do
+
+for to_format in $ref_formats
+do
+	if test "$from_format" = "$to_format"
+	then
+		continue
+	fi
+
+
+	for dir in "$(pwd)/repo/.git" "."
+	do
+
+		test_expect_success "read from $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method"
+			)
+		'
+
+		test_expect_success "write to $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+
+				git refs list >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				git refs list >actual &&
+				test_cmp expect actual &&
+
+				git refs list | grep -v "refs/tags/1" >expect &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				test_cmp expect actual
+			)
+		'
+
+		test_expect_success "with worktree and $to_format backend, $dir dir" '
+			test_when_finished "rm -rf repo wt" &&
+			git init --ref-format=$from_format repo &&
+			(
+				cd repo &&
+				test_commit 1 &&
+				test_commit 2 &&
+				test_commit 3 &&
+
+				git refs migrate --dry-run --ref-format=$to_format >out &&
+				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
+
+				git config set core.repositoryformatversion 1 &&
+				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
+
+				git worktree add ../wt 2
+			) &&
+
+			git -C repo for-each-ref --include-root-refs >expect &&
+			git -C wt for-each-ref --include-root-refs >expect &&
+			! test_cmp expect actual &&
+
+			git -C wt rev-parse 2 >expect &&
+			git -C wt rev-parse HEAD >actual &&
+			test_cmp expect actual
+		'
+	done # closes dir
+done # closes to_format
+done # closes from_format
+
+test_done

-- 
2.53.GIT

Re: [PATCH v9 5/6] refs: allow reference location in refstorage config

[Junio C Hamano]

Karthik Nayak <karthik.188@gmail.com> writes:

> +run_with_uri() {
> +test_refs_backend() {

I've been fixing these style violations (and another one in 6/6) for
the past few iterations, but forgot to report it.

Hopefully this will not be rerolled and we can merge the result in
'next', which would mean that I'll do the local touch-up for just
once more.

The changes relative to the previous round all looked sensible.  The
changes to the tests to swap expect and actual so that they correctly
hold expected and actual contents are nice touch.

Thanks.

[PATCH v9 6/6] refs: add GIT_REFERENCE_BACKEND to specify reference backend

[Karthik Nayak]

Git allows setting a different object directory via
'GIT_OBJECT_DIRECTORY', but provides no equivalent for references. In
the previous commit we extended the 'extensions.refStorage' config to
also support an URI input for reference backend with location.

Let's also add a new environment variable 'GIT_REFERENCE_BACKEND' that
takes in the same input as the config variable. Having an environment
variable allows us to modify the reference backend and location on the
fly for individual Git commands.

The environment variable also allows usage of alternate reference
directories during 'git-clone(1)' and 'git-init(1)'. Add the config to
the repository when created with the environment variable set.

When initializing the repository with an alternate reference folder,
create the required stubs in the repositories $GIT_DIR. The inverse,
i.e. removal of the ref store doesn't clean up the stubs in the $GIT_DIR
since that would render it unusable. Removal of ref store is only used
when migrating between ref formats and cleanup of the $GIT_DIR doesn't
make sense in such a situation.

Helped-by: Jean-Noël Avila <jn.avila@free.fr>
Signed-off-by: Karthik Nayak <karthik.188@gmail.com>
---
 Documentation/git.adoc |   5 ++
 environment.h          |   1 +
 refs.c                 |  30 +++++---
 setup.c                |  55 ++++++++++++++-
 t/t1423-ref-backend.sh | 187 ++++++++++++++++++++++++++++++++++++++++---------
 5 files changed, 233 insertions(+), 45 deletions(-)

diff --git a/Documentation/git.adoc b/Documentation/git.adoc
index ce099e78b8..66442735ea 100644
--- a/Documentation/git.adoc
+++ b/Documentation/git.adoc
@@ -584,6 +584,11 @@ double-quotes and respecting backslash escapes. E.g., the value
 	repositories will be set to this value. The default is "files".
 	See `--ref-format` in linkgit:git-init[1].
 
+`GIT_REFERENCE_BACKEND`::
+    Specify which reference backend to be used along with its URI.
+    See `extensions.refStorage` option in linkgit:git-config[1] for more
+    details. Overrides the config variable when used.
+
 Git Commits
 ~~~~~~~~~~~
 `GIT_AUTHOR_NAME`::
diff --git a/environment.h b/environment.h
index 27f657af04..540e0a7f6d 100644
--- a/environment.h
+++ b/environment.h
@@ -42,6 +42,7 @@
 #define GIT_OPTIONAL_LOCKS_ENVIRONMENT "GIT_OPTIONAL_LOCKS"
 #define GIT_TEXT_DOMAIN_DIR_ENVIRONMENT "GIT_TEXTDOMAINDIR"
 #define GIT_ATTR_SOURCE_ENVIRONMENT "GIT_ATTR_SOURCE"
+#define GIT_REFERENCE_BACKEND_ENVIRONMENT "GIT_REFERENCE_BACKEND"
 
 /*
  * Environment variable used to propagate the --no-advice global option to the
diff --git a/refs.c b/refs.c
index ef1902e85c..a700a66f08 100644
--- a/refs.c
+++ b/refs.c
@@ -2192,13 +2192,17 @@ int ref_store_create_on_disk(struct ref_store *refs, int flags, struct strbuf *e
 {
 	int ret = refs->be->create_on_disk(refs, flags, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
-		struct strbuf msg = STRBUF_INIT;
-
-		strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
-		refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
-		strbuf_release(&msg);
+	if (!ret) {
+		/* Creation of stubs for linked worktrees are handled in the worktree code. */
+		if (!(flags & REF_STORE_CREATE_ON_DISK_IS_WORKTREE) && refs->repo->ref_storage_payload) {
+			refs_create_refdir_stubs(refs->repo, refs->repo->gitdir,
+						 "repository uses alternate refs storage");
+		} else if (ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+			struct strbuf msg = STRBUF_INIT;
+			strbuf_addf(&msg, "this repository uses the %s format", refs->be->name);
+			refs_create_refdir_stubs(refs->repo, refs->gitdir, msg.buf);
+			strbuf_release(&msg);
+		}
 	}
 
 	return ret;
@@ -2208,10 +2212,18 @@ int ref_store_remove_on_disk(struct ref_store *refs, struct strbuf *err)
 {
 	int ret = refs->be->remove_on_disk(refs, err);
 
-	if (!ret &&
-	    ref_storage_format_by_name(refs->be->name) != REF_STORAGE_FORMAT_FILES) {
+	if (!ret) {
+		enum ref_storage_format format = ref_storage_format_by_name(refs->be->name);
 		struct strbuf sb = STRBUF_INIT;
 
+		/* Backends apart from the files backend create stubs. */
+		if (format == REF_STORAGE_FORMAT_FILES)
+			return ret;
+
+		/* Alternate refs backend require stubs in the gitdir. */
+		if (refs->repo->ref_storage_payload)
+			return ret;
+
 		strbuf_addf(&sb, "%s/HEAD", refs->gitdir);
 		if (unlink(sb.buf) < 0) {
 			strbuf_addf(err, "could not delete stub HEAD: %s",
diff --git a/setup.c b/setup.c
index d407f3347b..90cb9be578 100644
--- a/setup.c
+++ b/setup.c
@@ -1838,6 +1838,7 @@ const char *setup_git_directory_gently(int *nongit_ok)
 	static struct strbuf cwd = STRBUF_INIT;
 	struct strbuf dir = STRBUF_INIT, gitdir = STRBUF_INIT, report = STRBUF_INIT;
 	const char *prefix = NULL;
+	const char *ref_backend_uri;
 	struct repository_format repo_fmt = REPOSITORY_FORMAT_INIT;
 
 	/*
@@ -1995,6 +1996,25 @@ const char *setup_git_directory_gently(int *nongit_ok)
 		setenv(GIT_PREFIX_ENVIRONMENT, "", 1);
 	}
 
+	/*
+	 * The env variable should override the repository config
+	 * for 'extensions.refStorage'.
+	 */
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+		repo_set_ref_storage_format(the_repository, format, payload);
+
+		free(backend);
+		free(payload);
+	}
+
 	setup_original_cwd();
 
 	strbuf_release(&dir);
@@ -2337,7 +2357,8 @@ void initialize_repository_version(int hash_algo,
 	 * the remote repository's format.
 	 */
 	if (hash_algo != GIT_HASH_SHA1_LEGACY ||
-	    ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	    ref_storage_format != REF_STORAGE_FORMAT_FILES ||
+	    the_repository->ref_storage_payload)
 		target_version = GIT_REPO_VERSION_READ;
 
 	if (hash_algo != GIT_HASH_SHA1_LEGACY && hash_algo != GIT_HASH_UNKNOWN)
@@ -2346,11 +2367,20 @@ void initialize_repository_version(int hash_algo,
 	else if (reinit)
 		repo_config_set_gently(the_repository, "extensions.objectformat", NULL);
 
-	if (ref_storage_format != REF_STORAGE_FORMAT_FILES)
+	if (the_repository->ref_storage_payload) {
+		struct strbuf ref_uri = STRBUF_INIT;
+
+		strbuf_addf(&ref_uri, "%s://%s",
+			    ref_storage_format_to_name(ref_storage_format),
+			    the_repository->ref_storage_payload);
+		repo_config_set(the_repository, "extensions.refstorage", ref_uri.buf);
+		strbuf_release(&ref_uri);
+	} else if (ref_storage_format != REF_STORAGE_FORMAT_FILES) {
 		repo_config_set(the_repository, "extensions.refstorage",
 				ref_storage_format_to_name(ref_storage_format));
-	else if (reinit)
+	} else if (reinit) {
 		repo_config_set_gently(the_repository, "extensions.refstorage", NULL);
+	}
 
 	if (reinit) {
 		struct strbuf config = STRBUF_INIT;
@@ -2623,6 +2653,7 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 		.ignore_repo = 1,
 		.ignore_worktree = 1,
 	};
+	const char *ref_backend_uri;
 	const char *env;
 
 	config_with_options(read_default_format_config, &cfg, NULL, NULL, &opts);
@@ -2668,6 +2699,24 @@ static void repository_format_configure(struct repository_format *repo_fmt,
 	} else {
 		repo_fmt->ref_storage_format = REF_STORAGE_FORMAT_DEFAULT;
 	}
+
+
+	ref_backend_uri = getenv(GIT_REFERENCE_BACKEND_ENVIRONMENT);
+	if (ref_backend_uri) {
+		char *backend, *payload;
+		enum ref_storage_format format;
+
+		parse_reference_uri(ref_backend_uri, &backend, &payload);
+		format = ref_storage_format_by_name(backend);
+		if (format == REF_STORAGE_FORMAT_UNKNOWN)
+			die(_("unknown ref storage format: '%s'"), backend);
+
+		repo_fmt->ref_storage_format = format;
+		repo_fmt->ref_storage_payload = payload;
+
+		free(backend);
+	}
+
 	repo_set_ref_storage_format(the_repository, repo_fmt->ref_storage_format,
 				    repo_fmt->ref_storage_payload);
 }
diff --git a/t/t1423-ref-backend.sh b/t/t1423-ref-backend.sh
index 9912433b8c..9884a07447 100755
--- a/t/t1423-ref-backend.sh
+++ b/t/t1423-ref-backend.sh
@@ -11,16 +11,25 @@ test_description='Test reference backend URIs'
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
 #   <cmd> is the git subcommand to be run in the repository.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 run_with_uri() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
 	cmd=$4 &&
+	via=$5 &&
 
-	git -C "$repo" config set core.repositoryformatversion 1
-	git -C "$repo" config set extensions.refStorage "$uri" &&
-	git -C "$repo" $cmd &&
-	git -C "$repo" config set extensions.refStorage "$backend"
+	git -C "$repo" config set core.repositoryformatversion 1 &&
+	if test "$via" = "env"
+	then
+		test_env GIT_REFERENCE_BACKEND="$uri" git -C "$repo" $cmd
+	elif test "$via" = "config"
+	then
+		git -C "$repo" config set extensions.refStorage "$uri" &&
+		git -C "$repo" $cmd &&
+		git -C "$repo" config set extensions.refStorage "$backend"
+	fi
 }
 
 # Test a repository with a given reference storage by running and comparing
@@ -30,44 +39,84 @@ run_with_uri() {
 #   <repo> is the relative path to the repo to run the command in.
 #   <backend> is the original ref storage of the repo.
 #   <uri> is the new URI to be set for the ref storage.
+#   <via> if 'config', set the backend via the 'extensions.refStorage' config.
+#         if 'env', set the backend via the 'GIT_REFERENCE_BACKEND' env.
 #   <err_msg> (optional) if set, check if 'git-refs(1)' failed with the provided msg.
 test_refs_backend() {
 	repo=$1 &&
 	backend=$2 &&
 	uri=$3 &&
-	err_msg=$4 &&
+	via=$4 &&
+	err_msg=$5 &&
+
 
-	git -C "$repo" config set core.repositoryformatversion 1 &&
 	if test -n "$err_msg";
 	then
-		git -C "$repo" config set extensions.refStorage "$uri" &&
-		test_must_fail git -C "$repo" refs list 2>err &&
-		test_grep "$err_msg" err
+		if test "$via" = "env"
+		then
+			test_env GIT_REFERENCE_BACKEND="$uri" test_must_fail git -C "$repo" refs list 2>err
+		elif test "$via" = "config"
+		then
+			git -C "$repo" config set extensions.refStorage "$uri" &&
+			test_must_fail git -C "$repo" refs list 2>err &&
+			test_grep "$err_msg" err
+		fi
 	else
 		git -C "$repo" refs list >expect &&
-		run_with_uri "$repo" "$backend" "$uri" "refs list" >actual &&
+		run_with_uri "$repo" "$backend" "$uri" "refs list" "$via">actual &&
 		test_cmp expect actual
 	fi
 }
 
-test_expect_success 'URI is invalid' '
+# Verify that the expected files are present in the gitdir and the refsdir.
+# Usage: verify_files_exist <gitdir> <refdir>
+#   <gitdir> is the path for the gitdir.
+#   <refdir> is the path for the refdir.
+verify_files_exist() {
+	gitdir=$1 &&
+	refdir=$2 &&
+
+	# verify that the stubs were added to the $GITDIR.
+	echo "repository uses alternate refs storage" >expect &&
+	test_cmp expect $gitdir/refs/heads &&
+	echo "ref: refs/heads/.invalid" >expect &&
+	test_cmp expect $gitdir/HEAD
+
+	# verify that backend specific files exist.
+	case "$GIT_DEFAULT_REF_FORMAT" in
+	files)
+		test_path_is_dir $refdir/refs/heads &&
+		test_path_is_file $refdir/HEAD;;
+	reftable)
+		test_path_is_dir $refdir/reftable &&
+		test_path_is_file $refdir/reftable/tables.list;;
+	*)
+		BUG "unhandled ref format $GIT_DEFAULT_REF_FORMAT";;
+	esac
+}
+
+methods="config env"
+for method in $methods
+do
+
+test_expect_success "$method: URI is invalid" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable@/home/reftable" \
+	test_refs_backend repo files "reftable@/home/reftable" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'URI ends with colon' '
+test_expect_success "$method: URI ends with colon" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "reftable:" \
+	test_refs_backend repo files "reftable:" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
-test_expect_success 'unknown reference backend' '
+test_expect_success "$method: unknown reference backend" '
 	test_when_finished "rm -rf repo" &&
 	git init repo &&
-	test_refs_backend repo files "db://.git" \
+	test_refs_backend repo files "db://.git" "$method" \
 		"invalid value for ${SQ}extensions.refstorage${SQ}"
 '
 
@@ -86,7 +135,7 @@ do
 	for dir in "$(pwd)/repo/.git" "."
 	do
 
-		test_expect_success "read from $to_format backend, $dir dir" '
+		test_expect_success "$method: read from $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -101,7 +150,7 @@ do
 			)
 		'
 
-		test_expect_success "write to $to_format backend, $dir dir" '
+		test_expect_success "$method: write to $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -113,20 +162,22 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" &&
+				test_refs_backend . $from_format "$to_format://$BACKEND_PATH" "$method" &&
 
 				git refs list >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "tag -d 1" &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"tag -d 1" "$method" &&
 				git refs list >actual &&
 				test_cmp expect actual &&
 
 				git refs list | grep -v "refs/tags/1" >expect &&
-				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" "refs list" >actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"refs list" "$method" >actual &&
 				test_cmp expect actual
 			)
 		'
 
-		test_expect_success "with worktree and $to_format backend, $dir dir" '
+		test_expect_success "$method: with worktree and $to_format backend, $dir dir" '
 			test_when_finished "rm -rf repo wt" &&
 			git init --ref-format=$from_format repo &&
 			(
@@ -138,22 +189,92 @@ do
 				git refs migrate --dry-run --ref-format=$to_format >out &&
 				BACKEND_PATH="$dir/$(sed "s/.* ${SQ}.git\/\(.*\)${SQ}/\1/" out)" &&
 
-				git config set core.repositoryformatversion 1 &&
-				git config set extensions.refStorage "$to_format://$BACKEND_PATH" &&
-
-				git worktree add ../wt 2
-			) &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"worktree add ../wt 2" "$method" &&
 
-			git -C repo for-each-ref --include-root-refs >expect &&
-			git -C wt for-each-ref --include-root-refs >expect &&
-			! test_cmp expect actual &&
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"for-each-ref --include-root-refs" "$method" >expect &&
+				! test_cmp expect actual &&
 
-			git -C wt rev-parse 2 >expect &&
-			git -C wt rev-parse HEAD >actual &&
-			test_cmp expect actual
+				run_with_uri . "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse 2" "$method" >actual &&
+				run_with_uri ../wt "$from_format" "$to_format://$BACKEND_PATH" \
+					"rev-parse HEAD" "$method" >expect &&
+				test_cmp expect actual
+			)
 		'
 	done # closes dir
+
+	test_expect_success "migrating repository to $to_format with alternate refs directory" '
+		test_when_finished "rm -rf repo refdir" &&
+		mkdir refdir &&
+		GIT_REFERENCE_BACKEND="${from_format}://$(pwd)/refdir" git init repo &&
+		(
+			cd repo &&
+
+			test_commit 1 &&
+			test_commit 2 &&
+			test_commit 3 &&
+
+			git refs migrate --ref-format=$to_format &&
+			git refs list >out &&
+			test_grep "refs/tags/1"	out &&
+			test_grep "refs/tags/2"	out &&
+			test_grep "refs/tags/3"	out
+		)
+	'
+
 done # closes to_format
 done # closes from_format
 
+done # closes method
+
+test_expect_success 'initializing repository with alt ref directory' '
+	test_when_finished "rm -rf repo refdir" &&
+	mkdir refdir &&
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git init repo &&
+	verify_files_exist repo/.git refdir &&
+	(
+		cd repo &&
+
+		git config get extensions.refstorage >actual &&
+		echo $BACKEND >expect &&
+		test_cmp expect actual &&
+
+		test_commit 1 &&
+		test_commit 2 &&
+		test_commit 3 &&
+		git refs list >out &&
+		test_grep "refs/tags/1"	out &&
+		test_grep "refs/tags/2"	out &&
+		test_grep "refs/tags/3"	out
+	)
+'
+
+test_expect_success 'cloning repository with alt ref directory' '
+	test_when_finished "rm -rf source repo refdir" &&
+	mkdir refdir &&
+
+	git init source &&
+	test_commit -C source 1 &&
+	test_commit -C source 2 &&
+	test_commit -C source 3 &&
+
+	BACKEND="$(test_detect_ref_format)://$(pwd)/refdir" &&
+	GIT_REFERENCE_BACKEND=$BACKEND git clone source repo &&
+
+	git -C repo config get extensions.refstorage >actual &&
+	echo $BACKEND >expect &&
+	test_cmp expect actual &&
+
+	verify_files_exist repo/.git refdir &&
+
+	git -C source for-each-ref refs/tags/ >expect &&
+	git -C repo for-each-ref refs/tags/ >actual &&
+	test_cmp expect actual
+'
+
 test_done

-- 
2.53.GIT