[PATCH 0/5] odb: make creation of object database pluggable

[PATCH 0/5] odb: make creation of object database pluggable

[Patrick Steinhardt]

Hi,

when creating a new repository we create a couple of on-disk data
structures for the object database. This includes the "objects/"
directory hierarchy with "objects/info" and "objects/pack", which are
specific to the backend.

This patch series makes the creation of the on-disk data structures
pluggable. While we continue to always create "objects/" regardless of
the backend (it's required for a repository to be recognized as such),
the other subdirectories are now created by the backend. This will allow
other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This
detour is required so that we can defer initialization of the object
database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).

Thanks!

Patrick

---
Patrick Steinhardt (5):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable

 loose.c               | 25 ++++++++++----------
 loose.h               |  1 +
 odb/source-files.c    | 19 +++++++++++++++
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.c    |  2 ++
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++
 odb/source.h          | 29 +++++++++++++++++++++++
 repository.c          |  2 --
 setup.c               | 65 +++++++++++++++++++++++++++++++++++----------------
 setup.h               |  9 +++++++
 13 files changed, 149 insertions(+), 38 deletions(-)


---
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

[PATCH 1/5] loose: load loose object map for the correct source

[Patrick Steinhardt]

When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source.

Fix this bug by instead loading the map via the loose source's path.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/loose.c b/loose.c
index bf01d3e42d..9dad75373b 100644
--- a/loose.c
+++ b/loose.c
@@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
+static int load_one_loose_object_map(struct odb_source_loose *loose)
 {
-	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+	struct repository *repo = loose->base.odb->repo;
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
 	FILE *fp;
 	int ret = -1;
 
@@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
 	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
 
-	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
-	fp = fopen(path.buf, "rb");
+	path = xstrfmt("%s/loose-object-idx", loose->base.path);
+	fp = fopen(path, "rb");
 	if (!fp) {
-		strbuf_release(&path);
+		free(path);
 		return 0;
 	}
 
@@ -102,7 +104,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 err:
 	fclose(fp);
 	strbuf_release(&buf);
-	strbuf_release(&path);
+	free(path);
 	return ret;
 }
 
@@ -117,10 +119,10 @@ int repo_read_loose_object_map(struct repository *repo)
 
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(repo, files->loose) < 0) {
+		if (load_one_loose_object_map(files->loose) < 0)
 			return -1;
-		}
 	}
+
 	return 0;
 }
 

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 2/5] setup: detangle loading of loose object maps

[Patrick Steinhardt]

When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.

Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:

  - We assume that the primary object database is the "files" object
    database so that we can extract its "loose" backend. This stops
    working with pluggable object databases.

  - We require the object database to already have been initialized when
    configuring the object database. This means that we must intermix
    configuration of the repository and initialization of its
    sub-structures in a weird way.

Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c            | 11 +++++------
 loose.h            |  1 +
 odb/source-loose.c |  2 ++
 repository.c       |  2 --
 setup.c            |  5 +++--
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/loose.c b/loose.c
index 9dad75373b..a3b2dcedc2 100644
--- a/loose.c
+++ b/loose.c
@@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct odb_source_loose *loose)
+int loose_object_map_load(struct odb_source_loose *loose)
 {
 	struct repository *repo = loose->base.odb->repo;
 	struct strbuf buf = STRBUF_INIT;
@@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
 	FILE *fp;
 	int ret = -1;
 
+	if (!should_use_loose_object_map(repo))
+		return 0;
+
 	if (!loose->map)
 		loose_object_map_init(&loose->map);
 	if (!loose->cache) {
@@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
 {
 	struct odb_source *source;
 
-	if (!should_use_loose_object_map(repo))
-		return 0;
-
 	odb_prepare_alternates(repo->objects);
-
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(files->loose) < 0)
+		if (loose_object_map_load(files->loose) < 0)
 			return -1;
 	}
 
diff --git a/loose.h b/loose.h
index 6c9b3f4571..ed663ac550 100644
--- a/loose.h
+++ b/loose.h
@@ -13,6 +13,7 @@ struct loose_object_map {
 
 void loose_object_map_init(struct loose_object_map **map);
 void loose_object_map_clear(struct loose_object_map **map);
+int loose_object_map_load(struct odb_source_loose *loose);
 int repo_loose_object_map_oid(struct repository *repo,
 			      const struct object_id *src,
 			      const struct git_hash_algo *dest_algo,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3f7d04a56e..812ca1c138 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 	if (!is_absolute_path(loose->base.path))
 		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
 
+	loose_object_map_load(loose);
+
 	return loose;
 }
diff --git a/repository.c b/repository.c
index 2ef0778846..6d633002b4 100644
--- a/repository.c
+++ b/repository.c
@@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
 	if (hash_algo_by_ptr(repo->hash_algo) == algo)
 		BUG("hash_algo and compat_hash_algo match");
 	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
-	if (repo->compat_hash_algo)
-		repo_read_loose_object_map(repo);
 #else
 	if (algo)
 		die(_("compatibility hash algorithm support requires Rust"));
diff --git a/setup.c b/setup.c
index d31808130b..825572f5f1 100644
--- a/setup.c
+++ b/setup.c
@@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
 
 	repo->bare_cfg = format->is_bare;
 	repo_set_hash_algo(repo, format->hash_algo);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
 	repo_set_ref_storage_format(repo,
 				    format->ref_storage_format,
@@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	free(alternate_object_directories);
 	free(object_directory);
 	return 0;

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 3/5] setup: defer object database creation

[Patrick Steinhardt]

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Defer the object database creation so that we handle both steps in the
same function.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 setup.c | 35 +++++++++++++++++++++++++++--------
 setup.h |  9 +++++++++
 2 files changed, 36 insertions(+), 8 deletions(-)

diff --git a/setup.c b/setup.c
index 825572f5f1..a7b1b9eaef 100644
--- a/setup.c
+++ b/setup.c
@@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
 	return result;
 }
 
+static void get_object_directories(char **object_directory,
+				   char **alternate_object_directories)
+{
+	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+}
+
 int apply_repository_format(struct repository *repo,
 			    const struct repository_format *format,
 			    enum apply_repository_format_flags flags,
@@ -1779,8 +1786,9 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+		get_object_directories(&object_directory,
+				       &alternate_object_directories);
+
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,8 +1811,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION))
+		repo->objects = odb_new(repo, object_directory,
+					alternate_object_directories);
 
 	free(alternate_object_directories);
 	free(object_directory);
@@ -2654,11 +2663,16 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
+	char *object_directory, *alternate_object_directories;
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	get_object_directories(&object_directory, &alternate_object_directories);
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2672,6 +2686,8 @@ static void create_object_directory(struct repository *repo)
 	strbuf_addstr(&path, "/info");
 	safe_create_dir(repo, path.buf, 1);
 
+	free(alternate_object_directories);
+	free(object_directory);
 	strbuf_release(&path);
 }
 
@@ -2867,9 +2883,10 @@ int init_db(struct repository *repo,
 	 */
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2885,7 +2902,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];
diff --git a/setup.h b/setup.h
index 654f10e059..e55d647b70 100644
--- a/setup.h
+++ b/setup.h
@@ -241,6 +241,15 @@ enum apply_repository_format_flags {
 	 * relate to the object database.
 	 */
 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
+
+	/*
+	 * Usually, the object database is created after the repository format
+	 * was applied. This step is skipped if this flag is set, which leaves
+	 * us with a partially-working repository.
+	 *
+	 * This is useful when initializing a new repository.
+	 */
+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
 };
 
 /*

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 4/5] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++++++
 odb/source.h          |  6 ++++++
 6 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/odb/source-files.h b/odb/source-files.h
index d7ac3c1c81..6a803afdda 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -28,7 +28,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_FILES)
-		BUG("trying to downcast source of type '%d' to files", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_FILES));
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source-inmemory.h b/odb/source-inmemory.h
index a88fc2e320..adbad23e8b 100644
--- a/odb/source-inmemory.h
+++ b/odb/source-inmemory.h
@@ -26,7 +26,9 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)
 static inline struct odb_source_inmemory *odb_source_inmemory_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_INMEMORY)
-		BUG("trying to downcast source of type '%d' to in-memory", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_INMEMORY));
 	return container_of(source, struct odb_source_inmemory, base);
 }
 
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 6070aaf3ce..3cf2e1f8f1 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -41,7 +41,9 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_LOOSE)
-		BUG("trying to downcast source of type '%d' to loose", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_LOOSE));
 	return container_of(source, struct odb_source_loose, base);
 }
 
diff --git a/odb/source-packed.h b/odb/source-packed.h
index 77309ddd09..a0f6b5096d 100644
--- a/odb/source-packed.h
+++ b/odb/source-packed.h
@@ -78,7 +78,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
 static inline struct odb_source_packed *odb_source_packed_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_PACKED)
-		BUG("trying to downcast source of type '%d' to packed", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_PACKED));
 	return container_of(source, struct odb_source_packed, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index 7993dcbd65..c300e836f6 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -4,6 +4,25 @@
 #include "odb/source.h"
 #include "packfile.h"
 
+static const char * const odb_source_names_by_type[] = {
+	[ODB_SOURCE_UNKNOWN] = "unknown",
+	[ODB_SOURCE_FILES] = "files",
+	[ODB_SOURCE_LOOSE] = "loose",
+	[ODB_SOURCE_PACKED] = "packed",
+	[ODB_SOURCE_INMEMORY] = "inmemory",
+};
+
+const char *odb_source_type_to_name(enum odb_source_type type)
+{
+	const char *name;
+	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
+		type = ODB_SOURCE_UNKNOWN;
+	name = odb_source_names_by_type[type];
+	if (!name)
+		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
+	return name;
+}
+
 struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
diff --git a/odb/source.h b/odb/source.h
index cd63dba91f..ab16d152f4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -25,6 +25,12 @@ enum odb_source_type {
 	ODB_SOURCE_INMEMORY,
 };
 
+/*
+ * Convert between the enum and its name. Returns the equivalent of "unknown"
+ * for unknown types.
+ */
+const char *odb_source_type_to_name(enum odb_source_type type);
+
 struct object_id;
 struct odb_read_stream;
 struct strvec;

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 5/5] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.

Make the creation of on-disk structures pluggable to accommodate for
this.

Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 19 +++++++++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 setup.c            | 35 ++++++++++++++++++++---------------
 3 files changed, 62 insertions(+), 15 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 4138758511..0db6e681fe 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -9,6 +9,7 @@
 #include "odb/source-files.h"
 #include "odb/source-loose.h"
 #include "packfile.h"
+#include "path.h"
 #include "strbuf.h"
 #include "write-or-die.h"
 
@@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
 	odb_source_close(&files->packed->base);
 }
 
+static int odb_source_files_create_on_disk(struct odb_source *source)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	safe_create_dir(source->odb->repo, source->path, 1);
+
+	strbuf_addf(&path, "%s/pack", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/info", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_release(&path);
+	return 0;
+}
+
 static void odb_source_files_prepare(struct odb_source *source,
 				     enum odb_prepare_flags flags)
 {
@@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
+	files->base.create_on_disk = odb_source_files_create_on_disk;
 	files->base.prepare = odb_source_files_prepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
diff --git a/odb/source.h b/odb/source.h
index ab16d152f4..4abc418bdd 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -89,6 +89,18 @@ struct odb_source {
 	 */
 	void (*close)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to create on-disk data structures that are
+	 * required for this source to operate.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 *
+	 * This callback may be NULL in case the source does not need any
+	 * on-disk setup.
+	 */
+	int (*create_on_disk)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to prepare the source so that it becomes
 	 * ready for use. It optionally clears underlying caches of the object
@@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
 	source->close(source);
 }
 
+/*
+ * Create on-disk data structures that are required for this source to operate
+ * correctly. Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_create_on_disk(struct odb_source *source)
+{
+	if (!source->create_on_disk)
+		return 0;
+	return source->create_on_disk(source);
+}
+
 /*
  * Prepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects
diff --git a/setup.c b/setup.c
index a7b1b9eaef..14ef119cb7 100644
--- a/setup.c
+++ b/setup.c
@@ -2666,29 +2666,34 @@ static int create_default_files(struct repository *repo,
 static void create_object_database(struct repository *repo)
 {
 	char *object_directory, *alternate_object_directories;
-	struct strbuf path = STRBUF_INIT;
-	size_t baselen;
 
 	get_object_directories(&object_directory, &alternate_object_directories);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 
-	strbuf_addstr(&path, repo_get_object_directory(repo));
-	baselen = path.len;
-
-	safe_create_dir(repo, path.buf, 1);
+	/*
+	 * Create the "objects" directory in the common directory. This is done
+	 * so that the repository can be discovered regardless of the backend
+	 * used.
+	 *
+	 * Note that we only do this in case the object directory wasn't
+	 * overwritten via an environment variable. If it _is_ being overridden
+	 * then we skip this step, as the repository won't be discoverable
+	 * anyway without the environment variable.
+	 */
+	if (!object_directory) {
+		struct strbuf objects_dir = STRBUF_INIT;
+		repo_common_path_append(repo, &objects_dir, "objects");
+		safe_create_dir(repo, objects_dir.buf, 1);
+		strbuf_release(&objects_dir);
+	}
 
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/pack");
-	safe_create_dir(repo, path.buf, 1);
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
 
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/info");
-	safe_create_dir(repo, path.buf, 1);
+	if (odb_source_create_on_disk(repo->objects->sources) < 0)
+		die("failed creating object database");
 
 	free(alternate_object_directories);
 	free(object_directory);
-	strbuf_release(&path);
 }
 
 static void separate_git_dir(const char *git_dir, const char *git_link)

-- 
2.55.0.407.g700c83d4f3.dirty

Re: [PATCH 1/5] loose: load loose object map for the correct source

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> When loading the loose object map via `load_one_loose_object_map()` we
> pass in both a repository and the corresponding source. We ultimately
> don't really respect the passed-in source though as we instead always
> load the map via the common directory. This doesn't make any sense
> though, as the function is called in a loop through all sources, and as
> such the expectation is that we'll load the map that belongs to the
> given source.
>
> Fix this bug by instead loading the map via the loose source's path.

Makes perfect sense.  We still need access to the 'repo' to learn
the hash algorithm used in the repository along with built-in object
names, but they are now obtained from the repository associated with
the loose object source, which is far more consistent.

Re: [PATCH 2/5] setup: detangle loading of loose object maps

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> When a repository is configured to use a compatibility hash function
> then we load the loose object map when we initialize the repository.
> This object map provides the mappings between the canonical object hash
> and the compatibility object hash.
>
> Loading the object map happens in `repo_set_compat_hash_algo()`, which
> calls `repo_read_loose_object_map()` in case the compatibility object
> hash is non-zero. This setup sequence has two major downsides:
>
>   - We assume that the primary object database is the "files" object
>     database so that we can extract its "loose" backend. This stops
>     working with pluggable object databases.

I am not sure if I understand this sentence, especially "we can
extract its loose backend" part.  Do you mean 'extract the object
map from the loose backend'?  Or something else?

>   - We require the object database to already have been initialized when
>     configuring the object database. This means that we must intermix
>     configuration of the repository and initialization of its
>     sub-structures in a weird way.
>
> Refactor the logic so that we instead load the loose object map via the
> "loose" backend, which fixes both of the above issues.

It does make sense to have loose_object_map_load() that is very much
specific to the loose object odb source to odb_source_loose_new().
That way set_compat_hash_algo() does not have to assume that files
backend is used as the object store.

> @@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
>  {
>  	struct odb_source *source;
>  
> -	if (!should_use_loose_object_map(repo))
> -		return 0;
> -
>  	odb_prepare_alternates(repo->objects);
> -
>  	for (source = repo->objects->sources; source; source = source->next) {
>  		struct odb_source_files *files = odb_source_files_downcast(source);
> -		if (load_one_loose_object_map(files->loose) < 0)
> +		if (loose_object_map_load(files->loose) < 0)
>  			return -1;

If this particular source in the list of sources is not backed by
the files backend, would downcast signal the fact (e.g., by
returning NULL) so that we can skip the next call instead?

Or would the next step in refactoring be to define "load object map"
method that is generic to odb_source so that this part does not have
to do any of these and instead simply do

	for (source = ...) {
		if (odb_source_object_map_load(source))
                	return -1;
        }

or something?

Re: [PATCH 3/5] setup: defer object database creation

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> In a subsequent commit we'll make the creation of the on-disk data
> structures of an object database pluggable. This will lead to an
> in-between state where we have already configured the repository's
> object database, but it's not usable yet until we eventually call
> `create_object_directory()`.
>
> Defer the object database creation so that we handle both steps in the
> same function.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  setup.c | 35 +++++++++++++++++++++++++++--------
>  setup.h |  9 +++++++++
>  2 files changed, 36 insertions(+), 8 deletions(-)
>
> diff --git a/setup.c b/setup.c
> index 825572f5f1..a7b1b9eaef 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
>  	return result;
>  }
>  
> +static void get_object_directories(char **object_directory,
> +				   char **alternate_object_directories)
> +{
> +	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> +	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> +}
> +
>  int apply_repository_format(struct repository *repo,
>  			    const struct repository_format *format,
>  			    enum apply_repository_format_flags flags,
> @@ -1779,8 +1786,9 @@ int apply_repository_format(struct repository *repo,
>  	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
>  		const char *shallow_file;
>  
> -		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> -		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> +		get_object_directories(&object_directory,
> +				       &alternate_object_directories);
> +
>  		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
>  		if (shallow_file)
>  			set_alternate_shallow_file(repo, shallow_file);

HONOR_ENV still means we read the environment variable to learn where
the object directory (which is admittedly a files backend specific
concept) and alternate object directories (ditto) are.

> @@ -1803,8 +1811,9 @@ int apply_repository_format(struct repository *repo,
>  	repo->repository_format_precious_objects =
>  		format->precious_objects;
>  
> -	repo->objects = odb_new(repo, object_directory,
> -				alternate_object_directories);
> +	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION))
> +		repo->objects = odb_new(repo, object_directory,
> +					alternate_object_directories);

And SKIP_ODB_CREATION can tell apply_repository_format() not to
create an odb there.

> -static void create_object_directory(struct repository *repo)
> +static void create_object_database(struct repository *repo)
>  {
> +	char *object_directory, *alternate_object_directories;
>  	struct strbuf path = STRBUF_INIT;
>  	size_t baselen;
>  
> +	get_object_directories(&object_directory, &alternate_object_directories);
> +	repo->objects = odb_new(repo, object_directory,
> +				alternate_object_directories);
> +
>  	strbuf_addstr(&path, repo_get_object_directory(repo));
>  	baselen = path.len;
>  
> @@ -2672,6 +2686,8 @@ static void create_object_directory(struct repository *repo)
>  	strbuf_addstr(&path, "/info");
>  	safe_create_dir(repo, path.buf, 1);
>  
> +	free(alternate_object_directories);
> +	free(object_directory);
>  	strbuf_release(&path);
>  }
>  


> @@ -2867,9 +2883,10 @@ int init_db(struct repository *repo,
>  	 */
>  	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
>  	repository_format_configure(&repo_fmt, hash, ref_storage_format);
> -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
> +	if (apply_repository_format(repo, &repo_fmt,
> +				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
> +				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
>  		die("%s", err.buf);
> -	startup_info->have_repository = 1;

Early in initialization, we no longer recreate the ODB when calling
apply_repository_format(), and we defer declaring that we have a
repository until we call create_object_database().

> @@ -2885,7 +2902,9 @@ int init_db(struct repository *repo,
>  
>  	if (!(flags & INIT_DB_SKIP_REFDB))
>  		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
> -	create_object_directory(repo);
> +	create_object_database(repo);
> +
> +	startup_info->have_repository = 1;

Instead we call create_object_database() rather late, after we
finish creating leading directories and default files and processing
the configuration.  I guess this is a prelude to specifying "no, we
are not doing the files backend but are using this new thing" in the
global configuration?

> diff --git a/setup.h b/setup.h
> index 654f10e059..e55d647b70 100644
> --- a/setup.h
> +++ b/setup.h
> @@ -241,6 +241,15 @@ enum apply_repository_format_flags {
>  	 * relate to the object database.
>  	 */
>  	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
> +
> +	/*
> +	 * Usually, the object database is created after the repository format
> +	 * was applied. This step is skipped if this flag is set, which leaves
> +	 * us with a partially-working repository.
> +	 *
> +	 * This is useful when initializing a new repository.
> +	 */
> +	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
>  };

OK.

Re: [PATCH 4/5] odb/source: introduce function to map source type to name

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Introduce a new function that maps an object source's type to a
> human-readable name. Use the function to provide better human-readable
> error messages for the downcasting functions.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.h    |  4 +++-
>  odb/source-inmemory.h |  4 +++-
>  odb/source-loose.h    |  4 +++-
>  odb/source-packed.h   |  4 +++-
>  odb/source.c          | 19 +++++++++++++++++++
>  odb/source.h          |  6 ++++++
>  6 files changed, 37 insertions(+), 4 deletions(-)

OK.

> +static const char * const odb_source_names_by_type[] = {
> +	[ODB_SOURCE_UNKNOWN] = "unknown",
> +	[ODB_SOURCE_FILES] = "files",
> +	[ODB_SOURCE_LOOSE] = "loose",
> +	[ODB_SOURCE_PACKED] = "packed",
> +	[ODB_SOURCE_INMEMORY] = "inmemory",
> +};

This is a trivially obvious implementation for mapping in either
direction.

'inmemory' should probably be spelled 'in-memory', though.

Thanks.

> +const char *odb_source_type_to_name(enum odb_source_type type)
> +{
> +	const char *name;
> +	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
> +		type = ODB_SOURCE_UNKNOWN;
> +	name = odb_source_names_by_type[type];
> +	if (!name)
> +		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
> +	return name;
> +}
> +
>  struct odb_source *odb_source_new(struct object_database *odb,
>  				  const char *path,
>  				  bool local)
> diff --git a/odb/source.h b/odb/source.h
> index cd63dba91f..ab16d152f4 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -25,6 +25,12 @@ enum odb_source_type {
>  	ODB_SOURCE_INMEMORY,
>  };
>  
> +/*
> + * Convert between the enum and its name. Returns the equivalent of "unknown"
> + * for unknown types.
> + */
> +const char *odb_source_type_to_name(enum odb_source_type type);
> +
>  struct object_id;
>  struct odb_read_stream;
>  struct strvec;

Re: [PATCH 5/5] odb: make creation of on-disk structures pluggable

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> Note that there is one exception though: the "objects" directory must
> exist in a repository regardless of which backend is in use. If it
> doesn't exist then the repository is not treated as a Git repository at
> all. Consequently, we create this directory regardless of the backend.

Very good thing to leave a note in the log message for.

Perhaps in Git 4.0 ;-)

> @@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  
>  	files->base.free = odb_source_files_free;
>  	files->base.close = odb_source_files_close;
> +	files->base.create_on_disk = odb_source_files_create_on_disk;
>  	files->base.prepare = odb_source_files_prepare;
>  	files->base.read_object_info = odb_source_files_read_object_info;
>  	files->base.read_object_stream = odb_source_files_read_object_stream;

If we are going to write a brand new object backing store that does
not use an on-disk filesystem (or a network filesystem, for that
matter) but still requires some sort of "initialization", for
example, an object database in the cloud that needs provisioning
before its first use, would this virtual function be the ideal place
to do so?

I wonder if we can give it a name better suited to its purpose by
moving away from the '_on_disk' suffix.

Re: [PATCH 1/5] loose: load loose object map for the correct source

[Justin Tobler]

On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> When loading the loose object map via `load_one_loose_object_map()` we
> pass in both a repository and the corresponding source. We ultimately
> don't really respect the passed-in source though as we instead always
> load the map via the common directory. This doesn't make any sense
> though, as the function is called in a loop through all sources, and as
> such the expectation is that we'll load the map that belongs to the
> given source.
> 
> Fix this bug by instead loading the map via the loose source's path.

IIUC the primary source is always being used, does this mean that
repositories using a compat hash and alternates are currently broken?

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  loose.c | 18 ++++++++++--------
>  1 file changed, 10 insertions(+), 8 deletions(-)
> 
> diff --git a/loose.c b/loose.c
> index bf01d3e42d..9dad75373b 100644
> --- a/loose.c
> +++ b/loose.c
> @@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
>  	return inserted;
>  }
>  
> -static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
> +static int load_one_loose_object_map(struct odb_source_loose *loose)
>  {
> -	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
> +	struct repository *repo = loose->base.odb->repo;

Ok, we really only need the repository to know the hash algo, but we can
get this from the loose source.

> +	struct strbuf buf = STRBUF_INIT;
> +	char *path;
>  	FILE *fp;
>  	int ret = -1;
>  
> @@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
>  	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
>  	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
>  
> -	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
> -	fp = fopen(path.buf, "rb");
> +	path = xstrfmt("%s/loose-object-idx", loose->base.path);

Now we use the correct path per source. Looks good.

-Justin

Re: [PATCH 2/5] setup: detangle loading of loose object maps

[Justin Tobler]

On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> When a repository is configured to use a compatibility hash function
> then we load the loose object map when we initialize the repository.
> This object map provides the mappings between the canonical object hash
> and the compatibility object hash.
> 
> Loading the object map happens in `repo_set_compat_hash_algo()`, which
> calls `repo_read_loose_object_map()` in case the compatibility object
> hash is non-zero. This setup sequence has two major downsides:
> 
>   - We assume that the primary object database is the "files" object
>     database so that we can extract its "loose" backend. This stops
>     working with pluggable object databases.

So IIUC, does this mean that `repo_set_compat_hash_algo()` is directly
reaching into the loose object source to load the compatibility object
map? I suppose it should be the responsibility of the respective ODB
backend to handle object compatibility.

>   - We require the object database to already have been initialized when
>     configuring the object database. This means that we must intermix
>     configuration of the repository and initialization of its
>     sub-structures in a weird way.

If there any reason we need to eagerly load compatibility object
mappings?

> Refactor the logic so that we instead load the loose object map via the
> "loose" backend, which fixes both of the above issues.

Sounds reasonable.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  loose.c            | 11 +++++------
>  loose.h            |  1 +
>  odb/source-loose.c |  2 ++
>  repository.c       |  2 --
>  setup.c            |  5 +++--
>  5 files changed, 11 insertions(+), 10 deletions(-)
> 
> diff --git a/loose.c b/loose.c
> index 9dad75373b..a3b2dcedc2 100644
> --- a/loose.c
> +++ b/loose.c
> @@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
>  	return inserted;
>  }
>  
> -static int load_one_loose_object_map(struct odb_source_loose *loose)
> +int loose_object_map_load(struct odb_source_loose *loose)
>  {
>  	struct repository *repo = loose->base.odb->repo;
>  	struct strbuf buf = STRBUF_INIT;
> @@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
>  	FILE *fp;
>  	int ret = -1;
>  
> +	if (!should_use_loose_object_map(repo))
> +		return 0;

Previously the above condition has asserted in
`repo_read_loose_object_map()` which calls `loose_object_map_load()` for
each source. Do we expect each source to potentially answer differently
though?

> +
>  	if (!loose->map)
>  		loose_object_map_init(&loose->map);
>  	if (!loose->cache) {
> @@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
>  {
>  	struct odb_source *source;
>  
> -	if (!should_use_loose_object_map(repo))
> -		return 0;
> -
>  	odb_prepare_alternates(repo->objects);
> -
>  	for (source = repo->objects->sources; source; source = source->next) {
>  		struct odb_source_files *files = odb_source_files_downcast(source);
> -		if (load_one_loose_object_map(files->loose) < 0)
> +		if (loose_object_map_load(files->loose) < 0)
>  			return -1;
>  	}
>  
> diff --git a/loose.h b/loose.h
> index 6c9b3f4571..ed663ac550 100644
> --- a/loose.h
> +++ b/loose.h
> @@ -13,6 +13,7 @@ struct loose_object_map {
>  
>  void loose_object_map_init(struct loose_object_map **map);
>  void loose_object_map_clear(struct loose_object_map **map);
> +int loose_object_map_load(struct odb_source_loose *loose);
>  int repo_loose_object_map_oid(struct repository *repo,
>  			      const struct object_id *src,
>  			      const struct git_hash_algo *dest_algo,
> diff --git a/odb/source-loose.c b/odb/source-loose.c
> index 3f7d04a56e..812ca1c138 100644
> --- a/odb/source-loose.c
> +++ b/odb/source-loose.c
> @@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
>  	if (!is_absolute_path(loose->base.path))
>  		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
>  
> +	loose_object_map_load(loose);

Now we load the loose object map for the specific source when its
created.

> +
>  	return loose;
>  }
> diff --git a/repository.c b/repository.c
> index 2ef0778846..6d633002b4 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
>  	if (hash_algo_by_ptr(repo->hash_algo) == algo)
>  		BUG("hash_algo and compat_hash_algo match");
>  	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
> -	if (repo->compat_hash_algo)
> -		repo_read_loose_object_map(repo);

The loose object map is no longer read eagerly.

>  #else
>  	if (algo)
>  		die(_("compatibility hash algorithm support requires Rust"));
> diff --git a/setup.c b/setup.c
> index d31808130b..825572f5f1 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
>  
>  	repo->bare_cfg = format->is_bare;
>  	repo_set_hash_algo(repo, format->hash_algo);
> -	repo->objects = odb_new(repo, object_directory,
> -				alternate_object_directories);
>  	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
>  	repo_set_ref_storage_format(repo,
>  				    format->ref_storage_format,
> @@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
>  	repo->repository_format_precious_objects =
>  		format->precious_objects;
>  
> +	repo->objects = odb_new(repo, object_directory,
> +				alternate_object_directories);

We now defer creating the ODB until after the compat hash is configured.
Makes sense.

-Justin

Re: [PATCH 3/5] setup: defer object database creation

[Justin Tobler]

On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> In a subsequent commit we'll make the creation of the on-disk data
> structures of an object database pluggable. This will lead to an
> in-between state where we have already configured the repository's
> object database, but it's not usable yet until we eventually call
> `create_object_directory()`.
>
> Defer the object database creation so that we handle both steps in the
> same function.

So IIUC, the repository gets configured via `apply_repository_format()`
which invokes `odb_new()`. In this patch a
APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION flag is introduced to allow
the creation of the ODB to be delayed until after source specific
on-disk state has been created.

Naive question: would it be simpler to just require invoking `odb_new()`
explicitly after `apply_repository_format()` in all cases? There doesn't
appear to be too many callsites.

-Justin

Re: [PATCH 5/5] odb: make creation of on-disk structures pluggable

[Justin Tobler]

On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> When creating a new "files" object database source we have to create a
> couple of directories. These directories are of course specific to this
> particular backend, and a different backend may require a setup that is
> completely different.
> 
> Make the creation of on-disk structures pluggable to accommodate for
> this.

Ok.

> Note that there is one exception though: the "objects" directory must
> exist in a repository regardless of which backend is in use. If it
> doesn't exist then the repository is not treated as a Git repository at
> all. Consequently, we create this directory regardless of the backend.

Makes sense.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.c | 19 +++++++++++++++++++
>  odb/source.h       | 23 +++++++++++++++++++++++
>  setup.c            | 35 ++++++++++++++++++++---------------
>  3 files changed, 62 insertions(+), 15 deletions(-)
> 
> diff --git a/odb/source-files.c b/odb/source-files.c
> index 4138758511..0db6e681fe 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -9,6 +9,7 @@
>  #include "odb/source-files.h"
>  #include "odb/source-loose.h"
>  #include "packfile.h"
> +#include "path.h"
>  #include "strbuf.h"
>  #include "write-or-die.h"
>  
> @@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
>  	odb_source_close(&files->packed->base);
>  }
>  
> +static int odb_source_files_create_on_disk(struct odb_source *source)
> +{
> +	struct strbuf path = STRBUF_INIT;
> +
> +	safe_create_dir(source->odb->repo, source->path, 1);
> +
> +	strbuf_addf(&path, "%s/pack", source->path);
> +	safe_create_dir(source->odb->repo, path.buf, 1);
> +
> +	strbuf_reset(&path);
> +	strbuf_addf(&path, "%s/info", source->path);
> +	safe_create_dir(source->odb->repo, path.buf, 1);
> +
> +	strbuf_release(&path);
> +	return 0;
> +}

This is the callback to create on-disk state specific to the "files"
source and matches the current set of created files.

> +
>  static void odb_source_files_prepare(struct odb_source *source,
>  				     enum odb_prepare_flags flags)
>  {
> @@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  
>  	files->base.free = odb_source_files_free;
>  	files->base.close = odb_source_files_close;
> +	files->base.create_on_disk = odb_source_files_create_on_disk;
>  	files->base.prepare = odb_source_files_prepare;
>  	files->base.read_object_info = odb_source_files_read_object_info;
>  	files->base.read_object_stream = odb_source_files_read_object_stream;
> diff --git a/odb/source.h b/odb/source.h
> index ab16d152f4..4abc418bdd 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -89,6 +89,18 @@ struct odb_source {
>  	 */
>  	void (*close)(struct odb_source *source);
>  
> +	/*
> +	 * This callback is expected to create on-disk data structures that are
> +	 * required for this source to operate.
> +	 *
> +	 * The callback is expected to return 0 on success, a negative error
> +	 * code otherwise.
> +	 *
> +	 * This callback may be NULL in case the source does not need any
> +	 * on-disk setup.
> +	 */
> +	int (*create_on_disk)(struct odb_source *source);
> +
>  	/*
>  	 * This callback is expected to prepare the source so that it becomes
>  	 * ready for use. It optionally clears underlying caches of the object
> @@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
>  	source->close(source);
>  }
>  
> +/*
> + * Create on-disk data structures that are required for this source to operate
> + * correctly. Returns 0 on success, a negative error code otherwise.
> + */
> +static inline int odb_source_create_on_disk(struct odb_source *source)
> +{
> +	if (!source->create_on_disk)
> +		return 0;
> +	return source->create_on_disk(source);
> +}
> +
>  /*
>   * Prepare the object database source and clear any caches. Depending on the
>   * backend used this may have the effect that concurrently-written objects
> diff --git a/setup.c b/setup.c
> index a7b1b9eaef..14ef119cb7 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -2666,29 +2666,34 @@ static int create_default_files(struct repository *repo,
>  static void create_object_database(struct repository *repo)
>  {
>  	char *object_directory, *alternate_object_directories;
> -	struct strbuf path = STRBUF_INIT;
> -	size_t baselen;
>  
>  	get_object_directories(&object_directory, &alternate_object_directories);
> -	repo->objects = odb_new(repo, object_directory,
> -				alternate_object_directories);
>  
> -	strbuf_addstr(&path, repo_get_object_directory(repo));
> -	baselen = path.len;
> -
> -	safe_create_dir(repo, path.buf, 1);
> +	/*
> +	 * Create the "objects" directory in the common directory. This is done
> +	 * so that the repository can be discovered regardless of the backend
> +	 * used.
> +	 *
> +	 * Note that we only do this in case the object directory wasn't
> +	 * overwritten via an environment variable. If it _is_ being overridden
> +	 * then we skip this step, as the repository won't be discoverable
> +	 * anyway without the environment variable.
> +	 */
> +	if (!object_directory) {
> +		struct strbuf objects_dir = STRBUF_INIT;
> +		repo_common_path_append(repo, &objects_dir, "objects");
> +		safe_create_dir(repo, objects_dir.buf, 1);
> +		strbuf_release(&objects_dir);
> +	}

Here we always create the objects directory regardless of the backend.
Looks good.

> -	strbuf_setlen(&path, baselen);
> -	strbuf_addstr(&path, "/pack");
> -	safe_create_dir(repo, path.buf, 1);
> +	repo->objects = odb_new(repo, object_directory,
> +				alternate_object_directories);
>  
> -	strbuf_setlen(&path, baselen);
> -	strbuf_addstr(&path, "/info");
> -	safe_create_dir(repo, path.buf, 1);
> +	if (odb_source_create_on_disk(repo->objects->sources) < 0)
> +		die("failed creating object database");

Here we invoke the pluggable callback to create source specific on-disk
state. Part of me does wonder if this would be better to include this
inside of `odb_new()` and enable it with a specific flag, but having it
as a explicit separate step is probably fine too.

-Justin

Re: [PATCH 1/5] loose: load loose object map for the correct source

[Toon Claes]

Justin Tobler <jltobler@gmail.com> writes:

> On 26/07/24 05:48AM, Patrick Steinhardt wrote:
>> When loading the loose object map via `load_one_loose_object_map()` we
>> pass in both a repository and the corresponding source. We ultimately
>> don't really respect the passed-in source though as we instead always
>> load the map via the common directory. This doesn't make any sense
>> though, as the function is called in a loop through all sources, and as
>> such the expectation is that we'll load the map that belongs to the
>> given source.
>> 
>> Fix this bug by instead loading the map via the loose source's path.
>
> IIUC the primary source is always being used, does this mean that
> repositories using a compat hash and alternates are currently broken?

Yeah, the commit message seems to undersell this fix.

I think it wouldn't hurt to add a small test for this:

    test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
    	test_when_finished rm -rf alt borrow &&
    
    	git init --object-format=sha256 alt &&
    	git -C alt config extensions.compatObjectFormat sha1 &&
    	test_commit -C alt A &&
    
    	git init --object-format=sha256 borrow &&
    	git -C borrow config extensions.compatObjectFormat sha1 &&
    	echo "$PWD/alt/.git/objects" >borrow/.git/objects/info/alternates &&
    
    	oid=$(git -C alt rev-parse HEAD) &&
    	git -C alt    rev-parse --output-object-format=sha1 "$oid" >expect &&
    	git -C borrow rev-parse --output-object-format=sha1 "$oid" >actual &&
    	test_cmp expect actual
    '

-- 
Cheers,
Toon

Re: [PATCH 2/5] setup: detangle loading of loose object maps

[Toon Claes]

Justin Tobler <jltobler@gmail.com> writes:

> On 26/07/24 05:48AM, Patrick Steinhardt wrote:
>> When a repository is configured to use a compatibility hash function
>> then we load the loose object map when we initialize the repository.
>> This object map provides the mappings between the canonical object hash
>> and the compatibility object hash.
>> 
>> Loading the object map happens in `repo_set_compat_hash_algo()`, which
>> calls `repo_read_loose_object_map()` in case the compatibility object
>> hash is non-zero. This setup sequence has two major downsides:
>> 
>>   - We assume that the primary object database is the "files" object
>>     database so that we can extract its "loose" backend. This stops
>>     working with pluggable object databases.
>
> So IIUC, does this mean that `repo_set_compat_hash_algo()` is directly
> reaching into the loose object source to load the compatibility object
> map? I suppose it should be the responsibility of the respective ODB
> backend to handle object compatibility.
>
>>   - We require the object database to already have been initialized when
>>     configuring the object database. This means that we must intermix
>>     configuration of the repository and initialization of its
>>     sub-structures in a weird way.
>
> If there any reason we need to eagerly load compatibility object
> mappings?
>
>> Refactor the logic so that we instead load the loose object map via the
>> "loose" backend, which fixes both of the above issues.
>
> Sounds reasonable.
>
>> Signed-off-by: Patrick Steinhardt <ps@pks.im>
>> ---
>>  loose.c            | 11 +++++------
>>  loose.h            |  1 +
>>  odb/source-loose.c |  2 ++
>>  repository.c       |  2 --
>>  setup.c            |  5 +++--
>>  5 files changed, 11 insertions(+), 10 deletions(-)
>> 
>> diff --git a/loose.c b/loose.c
>> index 9dad75373b..a3b2dcedc2 100644
>> --- a/loose.c
>> +++ b/loose.c
>> @@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
>>  	return inserted;
>>  }
>>  
>> -static int load_one_loose_object_map(struct odb_source_loose *loose)
>> +int loose_object_map_load(struct odb_source_loose *loose)
>>  {
>>  	struct repository *repo = loose->base.odb->repo;
>>  	struct strbuf buf = STRBUF_INIT;
>> @@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
>>  	FILE *fp;
>>  	int ret = -1;
>>  
>> +	if (!should_use_loose_object_map(repo))
>> +		return 0;
>
> Previously the above condition has asserted in
> `repo_read_loose_object_map()` which calls `loose_object_map_load()` for
> each source. Do we expect each source to potentially answer differently
> though?

I've been wondering about this as well. The reason for this change is to
also have this guard when odb_source_loose_new(), in source-loose.c (see
further down in the patch), calls this function too.

>> +
>>  	if (!loose->map)
>>  		loose_object_map_init(&loose->map);
>>  	if (!loose->cache) {
>> @@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
>>  {
>>  	struct odb_source *source;
>>  
>> -	if (!should_use_loose_object_map(repo))
>> -		return 0;
>> -
>>  	odb_prepare_alternates(repo->objects);
>> -
>>  	for (source = repo->objects->sources; source; source = source->next) {
>>  		struct odb_source_files *files = odb_source_files_downcast(source);
>> -		if (load_one_loose_object_map(files->loose) < 0)
>> +		if (loose_object_map_load(files->loose) < 0)
>>  			return -1;
>>  	}
>>  
>> diff --git a/loose.h b/loose.h
>> index 6c9b3f4571..ed663ac550 100644
>> --- a/loose.h
>> +++ b/loose.h
>> @@ -13,6 +13,7 @@ struct loose_object_map {
>>  
>>  void loose_object_map_init(struct loose_object_map **map);
>>  void loose_object_map_clear(struct loose_object_map **map);
>> +int loose_object_map_load(struct odb_source_loose *loose);
>>  int repo_loose_object_map_oid(struct repository *repo,
>>  			      const struct object_id *src,
>>  			      const struct git_hash_algo *dest_algo,
>> diff --git a/odb/source-loose.c b/odb/source-loose.c
>> index 3f7d04a56e..812ca1c138 100644
>> --- a/odb/source-loose.c
>> +++ b/odb/source-loose.c
>> @@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
>>  	if (!is_absolute_path(loose->base.path))
>>  		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
>>  
>> +	loose_object_map_load(loose);
>
> Now we load the loose object map for the specific source when its
> created.

Here.

-- 
Cheers,
Toon

Re: [PATCH 1/5] loose: load loose object map for the correct source

[Patrick Steinhardt]

On Thu, Jul 30, 2026 at 02:47:50PM +0200, Toon Claes wrote:
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> >> When loading the loose object map via `load_one_loose_object_map()` we
> >> pass in both a repository and the corresponding source. We ultimately
> >> don't really respect the passed-in source though as we instead always
> >> load the map via the common directory. This doesn't make any sense
> >> though, as the function is called in a loop through all sources, and as
> >> such the expectation is that we'll load the map that belongs to the
> >> given source.
> >> 
> >> Fix this bug by instead loading the map via the loose source's path.
> >
> > IIUC the primary source is always being used, does this mean that
> > repositories using a compat hash and alternates are currently broken?
> 
> Yeah, the commit message seems to undersell this fix.
> 
> I think it wouldn't hurt to add a small test for this:
> 
>     test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
>     	test_when_finished rm -rf alt borrow &&
>     
>     	git init --object-format=sha256 alt &&
>     	git -C alt config extensions.compatObjectFormat sha1 &&
>     	test_commit -C alt A &&
>     
>     	git init --object-format=sha256 borrow &&
>     	git -C borrow config extensions.compatObjectFormat sha1 &&
>     	echo "$PWD/alt/.git/objects" >borrow/.git/objects/info/alternates &&
>     
>     	oid=$(git -C alt rev-parse HEAD) &&
>     	git -C alt    rev-parse --output-object-format=sha1 "$oid" >expect &&
>     	git -C borrow rev-parse --output-object-format=sha1 "$oid" >actual &&
>     	test_cmp expect actual
>     '

Good idea indeed, will do. Thanks!

Patrick

Re: [PATCH 5/5] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

On Sun, Jul 26, 2026 at 01:42:09PM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > @@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
> >  
> >  	files->base.free = odb_source_files_free;
> >  	files->base.close = odb_source_files_close;
> > +	files->base.create_on_disk = odb_source_files_create_on_disk;
> >  	files->base.prepare = odb_source_files_prepare;
> >  	files->base.read_object_info = odb_source_files_read_object_info;
> >  	files->base.read_object_stream = odb_source_files_read_object_stream;
> 
> If we are going to write a brand new object backing store that does
> not use an on-disk filesystem (or a network filesystem, for that
> matter) but still requires some sort of "initialization", for
> example, an object database in the cloud that needs provisioning
> before its first use, would this virtual function be the ideal place
> to do so?

It would, even though...

> I wonder if we can give it a name better suited to its purpose by
> moving away from the '_on_disk' suffix.

... the name is admittedly a bit misleading. I couldn't really come up
with a better name though, and the `on_disk()` suffix is what we already
use in the reference subsystem, too (see `ref_store_create_on_disk()`).
So I'm inclined to leave the name as-is for the sake of consistency.

Patrick

Re: [PATCH 4/5] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

On Sun, Jul 26, 2026 at 01:34:17PM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Introduce a new function that maps an object source's type to a
> > human-readable name. Use the function to provide better human-readable
> > error messages for the downcasting functions.
> >
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  odb/source-files.h    |  4 +++-
> >  odb/source-inmemory.h |  4 +++-
> >  odb/source-loose.h    |  4 +++-
> >  odb/source-packed.h   |  4 +++-
> >  odb/source.c          | 19 +++++++++++++++++++
> >  odb/source.h          |  6 ++++++
> >  6 files changed, 37 insertions(+), 4 deletions(-)
> 
> OK.
> 
> > +static const char * const odb_source_names_by_type[] = {
> > +	[ODB_SOURCE_UNKNOWN] = "unknown",
> > +	[ODB_SOURCE_FILES] = "files",
> > +	[ODB_SOURCE_LOOSE] = "loose",
> > +	[ODB_SOURCE_PACKED] = "packed",
> > +	[ODB_SOURCE_INMEMORY] = "inmemory",
> > +};
> 
> This is a trivially obvious implementation for mapping in either
> direction.
> 
> 'inmemory' should probably be spelled 'in-memory', though.

Fair, that reads better indeed. Will adapt.

Patrick

Re: [PATCH 2/5] setup: detangle loading of loose object maps

[Patrick Steinhardt]

On Fri, Jul 24, 2026 at 11:41:41AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > When a repository is configured to use a compatibility hash function
> > then we load the loose object map when we initialize the repository.
> > This object map provides the mappings between the canonical object hash
> > and the compatibility object hash.
> >
> > Loading the object map happens in `repo_set_compat_hash_algo()`, which
> > calls `repo_read_loose_object_map()` in case the compatibility object
> > hash is non-zero. This setup sequence has two major downsides:
> >
> >   - We assume that the primary object database is the "files" object
> >     database so that we can extract its "loose" backend. This stops
> >     working with pluggable object databases.
> 
> I am not sure if I understand this sentence, especially "we can
> extract its loose backend" part.  Do you mean 'extract the object
> map from the loose backend'?  Or something else?

Yeah, this is a bit awkward. Rewritten like this:

  - We assume that the primary object database is the "files" object
    database and unconditionally downcast it. This will BUG in case a
    different object database type was used together with a compat hash
    algorithm.

> > @@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
> >  {
> >  	struct odb_source *source;
> >  
> > -	if (!should_use_loose_object_map(repo))
> > -		return 0;
> > -
> >  	odb_prepare_alternates(repo->objects);
> > -
> >  	for (source = repo->objects->sources; source; source = source->next) {
> >  		struct odb_source_files *files = odb_source_files_downcast(source);
> > -		if (load_one_loose_object_map(files->loose) < 0)
> > +		if (loose_object_map_load(files->loose) < 0)
> >  			return -1;
> 
> If this particular source in the list of sources is not backed by
> the files backend, would downcast signal the fact (e.g., by
> returning NULL) so that we can skip the next call instead?

No, the downcast will BUG in case it's not the "files" backend.

> Or would the next step in refactoring be to define "load object map"
> method that is generic to odb_source so that this part does not have
> to do any of these and instead simply do
> 
> 	for (source = ...) {
> 		if (odb_source_object_map_load(source))
>                 	return -1;
>         }
> 
> or something?

This patch series is rather moving into the direction of making the
object map an internal implementation detail. Ideally, callers shouldn't
even have to be aware that such an object map exists. And by making the
loose object source load it automatically we get closer to that state.

There's only one more caller that calls `repo_read_loose_object_map()`
directly, in "object-file-convert.c", and that caller only calls it to
reload the map in case a concurrent process may have rewritten it. If we
make the backends handle this via `odb_source_prepare(FLUSH_CACHES)`
then we could also get rid of that caller.

Patrick

Re: [PATCH 2/5] setup: detangle loading of loose object maps

[Patrick Steinhardt]

On Tue, Jul 28, 2026 at 03:32:27PM -0500, Justin Tobler wrote:
> On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> > When a repository is configured to use a compatibility hash function
> > then we load the loose object map when we initialize the repository.
> > This object map provides the mappings between the canonical object hash
> > and the compatibility object hash.
> > 
> > Loading the object map happens in `repo_set_compat_hash_algo()`, which
> > calls `repo_read_loose_object_map()` in case the compatibility object
> > hash is non-zero. This setup sequence has two major downsides:
> > 
> >   - We assume that the primary object database is the "files" object
> >     database so that we can extract its "loose" backend. This stops
> >     working with pluggable object databases.
> 
> So IIUC, does this mean that `repo_set_compat_hash_algo()` is directly
> reaching into the loose object source to load the compatibility object
> map? I suppose it should be the responsibility of the respective ODB
> backend to handle object compatibility.
> 
> >   - We require the object database to already have been initialized when
> >     configuring the object database. This means that we must intermix
> >     configuration of the repository and initialization of its
> >     sub-structures in a weird way.
> 
> If there any reason we need to eagerly load compatibility object
> mappings?

I'm not familiar enough with the compatibility mappings to really be
able to say. Naively I'd say "no", but I'm rather erring on the side of
caution and want to leave this as-is.

> > diff --git a/loose.c b/loose.c
> > index 9dad75373b..a3b2dcedc2 100644
> > --- a/loose.c
> > +++ b/loose.c
> > @@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
> >  	FILE *fp;
> >  	int ret = -1;
> >  
> > +	if (!should_use_loose_object_map(repo))
> > +		return 0;
> 
> Previously the above condition has asserted in
> `repo_read_loose_object_map()` which calls `loose_object_map_load()` for
> each source. Do we expect each source to potentially answer differently
> though?

Not really, no. But as it's now the source that loads the object map it
has to verify for itself whether it should or should not load it.

Patrick

Re: [PATCH 3/5] setup: defer object database creation

[Patrick Steinhardt]

On Tue, Jul 28, 2026 at 04:13:42PM -0500, Justin Tobler wrote:
> On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> > In a subsequent commit we'll make the creation of the on-disk data
> > structures of an object database pluggable. This will lead to an
> > in-between state where we have already configured the repository's
> > object database, but it's not usable yet until we eventually call
> > `create_object_directory()`.
> >
> > Defer the object database creation so that we handle both steps in the
> > same function.
> 
> So IIUC, the repository gets configured via `apply_repository_format()`
> which invokes `odb_new()`. In this patch a
> APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION flag is introduced to allow
> the creation of the ODB to be delayed until after source specific
> on-disk state has been created.
> 
> Naive question: would it be simpler to just require invoking `odb_new()`
> explicitly after `apply_repository_format()` in all cases? There doesn't
> appear to be too many callsites.

I don't think it would, mostly because the logic to figure out the
object directory and the alternate object directory requires a bunch of
logic.

I think it'll ultimately become simpler though once we move into the
direction of what we've discussed in [1], where we said that we want to
move handling of those environment variables into the "files" backend,
too. And then it might make sense to revisit this.

Patrick

Re: [PATCH 3/5] setup: defer object database creation

[Patrick Steinhardt]

On Fri, Jul 24, 2026 at 11:50:38AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/setup.c b/setup.c
> > index 825572f5f1..a7b1b9eaef 100644
> > --- a/setup.c
> > +++ b/setup.c
> > @@ -2885,7 +2902,9 @@ int init_db(struct repository *repo,
> >  
> >  	if (!(flags & INIT_DB_SKIP_REFDB))
> >  		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
> > -	create_object_directory(repo);
> > +	create_object_database(repo);
> > +
> > +	startup_info->have_repository = 1;
> 
> Instead we call create_object_database() rather late, after we
> finish creating leading directories and default files and processing
> the configuration.  I guess this is a prelude to specifying "no, we
> are not doing the files backend but are using this new thing" in the
> global configuration?

Yes, exactly. Many of the refactorings I'm doing in "setup.c" ultimately
have the goal to detangle the setup and configuration of repository
extensions. It's been painful back when I introduced the "refStorage"
extension, and it's still painful now with the planned "objectStorage"
extension. So this time around I decided to detangle the logic before
introducing the extension to make the infra easier to understand going
forward.

Patrick

Re: [PATCH 3/5] setup: defer object database creation

[Patrick Steinhardt]

On Tue, Aug 04, 2026 at 09:21:45AM +0200, Patrick Steinhardt wrote:
> On Tue, Jul 28, 2026 at 04:13:42PM -0500, Justin Tobler wrote:
> > On 26/07/24 05:48AM, Patrick Steinhardt wrote:
> > > In a subsequent commit we'll make the creation of the on-disk data
> > > structures of an object database pluggable. This will lead to an
> > > in-between state where we have already configured the repository's
> > > object database, but it's not usable yet until we eventually call
> > > `create_object_directory()`.
> > >
> > > Defer the object database creation so that we handle both steps in the
> > > same function.
> > 
> > So IIUC, the repository gets configured via `apply_repository_format()`
> > which invokes `odb_new()`. In this patch a
> > APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION flag is introduced to allow
> > the creation of the ODB to be delayed until after source specific
> > on-disk state has been created.
> > 
> > Naive question: would it be simpler to just require invoking `odb_new()`
> > explicitly after `apply_repository_format()` in all cases? There doesn't
> > appear to be too many callsites.
> 
> I don't think it would, mostly because the logic to figure out the
> object directory and the alternate object directory requires a bunch of
> logic.
> 
> I think it'll ultimately become simpler though once we move into the
> direction of what we've discussed in [1], where we said that we want to
> move handling of those environment variables into the "files" backend,
> too. And then it might make sense to revisit this.
> 
> Patrick

[1]: <amLgMqkqxR8mKIbT@pks.im>

[PATCH v2 0/5] odb: make creation of object database pluggable

[Patrick Steinhardt]

Hi,

when creating a new repository we create a couple of on-disk data
structures for the object database. This includes the "objects/"
directory hierarchy with "objects/info" and "objects/pack", which are
specific to the backend.

This patch series makes the creation of the on-disk data structures
pluggable. While we continue to always create "objects/" regardless of
the backend (it's required for a repository to be recognized as such),
the other subdirectories are now created by the backend. This will allow
other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This
detour is required so that we can defer initialization of the object
database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).

Changes in v2:
  - Add a testcase that demonstrates the bug fixed with alternate loose
    object maps.
  - Rename the "inmemory" bakcend to "in-memory".
  - Clarify some commit messages.
  - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (5):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable

 loose.c                       | 25 +++++++++--------
 loose.h                       |  1 +
 odb/source-files.c            | 19 +++++++++++++
 odb/source-files.h            |  4 ++-
 odb/source-inmemory.h         |  4 ++-
 odb/source-loose.c            |  2 ++
 odb/source-loose.h            |  4 ++-
 odb/source-packed.h           |  4 ++-
 odb/source.c                  | 19 +++++++++++++
 odb/source.h                  | 29 +++++++++++++++++++
 repository.c                  |  2 --
 setup.c                       | 65 ++++++++++++++++++++++++++++++-------------
 setup.h                       |  9 ++++++
 t/t1016-compatObjectFormat.sh | 18 ++++++++++++
 14 files changed, 167 insertions(+), 38 deletions(-)

Range-diff versus v1:

1:  c126882da3 ! 1:  087bbd9fa7 loose: load loose object map for the correct source
    @@ Commit message
         load the map via the common directory. This doesn't make any sense
         though, as the function is called in a loop through all sources, and as
         such the expectation is that we'll load the map that belongs to the
    -    given source.
    +    given source. The consequence is that we'll ignore loose object maps of
    +    any configured alternates.
     
         Fix this bug by instead loading the map via the loose source's path.
     
    +    Helped-by: Toon Claes <toon@iotcl.com>
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## loose.c ##
    @@ loose.c: int repo_read_loose_object_map(struct repository *repo)
      	return 0;
      }
      
    +
    + ## t/t1016-compatObjectFormat.sh ##
    +@@ t/t1016-compatObjectFormat.sh: do
    + 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
    + 		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
    + 	'
    ++
    ++	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
    ++		for repo in alt borrow
    ++		do
    ++			test_when_finished "rm -rf $repo" &&
    ++			git init --object-format=$hash $repo &&
    ++			git -C $repo config set core.repositoryformatversion 1 &&
    ++			git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
    ++		done &&
    ++
    ++		git -C alt commit --allow-empty --message A &&
    ++		echo "$(pwd)/alt/.git/objects" >borrow/.git/objects/info/alternates &&
    ++
    ++		oid=$(git -C alt rev-parse HEAD) &&
    ++		git -C alt    rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
    ++		git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
    ++		test_cmp expect actual
    ++	'
    + done
    + cd "$base"
    + 
2:  6e06a82905 ! 2:  00a693dd72 setup: detangle loading of loose object maps
    @@ Commit message
         hash is non-zero. This setup sequence has two major downsides:
     
           - We assume that the primary object database is the "files" object
    -        database so that we can extract its "loose" backend. This stops
    -        working with pluggable object databases.
    +        database and unconditionally downcast it. This will cause us to BUG
    +        in case a different object database type was used together with a
    +        compat hash algorithm.
     
           - We require the object database to already have been initialized when
             configuring the object database. This means that we must intermix
3:  183ed0f34c = 3:  1dc1f83d73 setup: defer object database creation
4:  eb997d22d7 ! 4:  de1555ee1f odb/source: introduce function to map source type to name
    @@ odb/source.c
     +	[ODB_SOURCE_FILES] = "files",
     +	[ODB_SOURCE_LOOSE] = "loose",
     +	[ODB_SOURCE_PACKED] = "packed",
    -+	[ODB_SOURCE_INMEMORY] = "inmemory",
    ++	[ODB_SOURCE_INMEMORY] = "in-memory",
     +};
     +
     +const char *odb_source_type_to_name(enum odb_source_type type)
5:  3303124a7d = 5:  cadf131e70 odb: make creation of on-disk structures pluggable

---
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

[PATCH v2 1/5] loose: load loose object map for the correct source

[Patrick Steinhardt]

When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source. The consequence is that we'll ignore loose object maps of
any configured alternates.

Fix this bug by instead loading the map via the loose source's path.

Helped-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c                       | 18 ++++++++++--------
 t/t1016-compatObjectFormat.sh | 18 ++++++++++++++++++
 2 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/loose.c b/loose.c
index bf01d3e42d..9dad75373b 100644
--- a/loose.c
+++ b/loose.c
@@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
+static int load_one_loose_object_map(struct odb_source_loose *loose)
 {
-	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+	struct repository *repo = loose->base.odb->repo;
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
 	FILE *fp;
 	int ret = -1;
 
@@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
 	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
 
-	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
-	fp = fopen(path.buf, "rb");
+	path = xstrfmt("%s/loose-object-idx", loose->base.path);
+	fp = fopen(path, "rb");
 	if (!fp) {
-		strbuf_release(&path);
+		free(path);
 		return 0;
 	}
 
@@ -102,7 +104,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 err:
 	fclose(fp);
 	strbuf_release(&buf);
-	strbuf_release(&path);
+	free(path);
 	return ret;
 }
 
@@ -117,10 +119,10 @@ int repo_read_loose_object_map(struct repository *repo)
 
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(repo, files->loose) < 0) {
+		if (load_one_loose_object_map(files->loose) < 0)
 			return -1;
-		}
 	}
+
 	return 0;
 }
 
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 92d48b96a1..9cafcee509 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -187,6 +187,24 @@ do
 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
 		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
 	'
+
+	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
+		for repo in alt borrow
+		do
+			test_when_finished "rm -rf $repo" &&
+			git init --object-format=$hash $repo &&
+			git -C $repo config set core.repositoryformatversion 1 &&
+			git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
+		done &&
+
+		git -C alt commit --allow-empty --message A &&
+		echo "$(pwd)/alt/.git/objects" >borrow/.git/objects/info/alternates &&
+
+		oid=$(git -C alt rev-parse HEAD) &&
+		git -C alt    rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
+		git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+		test_cmp expect actual
+	'
 done
 cd "$base"
 

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v2 2/5] setup: detangle loading of loose object maps

[Patrick Steinhardt]

When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.

Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:

  - We assume that the primary object database is the "files" object
    database and unconditionally downcast it. This will cause us to BUG
    in case a different object database type was used together with a
    compat hash algorithm.

  - We require the object database to already have been initialized when
    configuring the object database. This means that we must intermix
    configuration of the repository and initialization of its
    sub-structures in a weird way.

Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c            | 11 +++++------
 loose.h            |  1 +
 odb/source-loose.c |  2 ++
 repository.c       |  2 --
 setup.c            |  5 +++--
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/loose.c b/loose.c
index 9dad75373b..a3b2dcedc2 100644
--- a/loose.c
+++ b/loose.c
@@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct odb_source_loose *loose)
+int loose_object_map_load(struct odb_source_loose *loose)
 {
 	struct repository *repo = loose->base.odb->repo;
 	struct strbuf buf = STRBUF_INIT;
@@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
 	FILE *fp;
 	int ret = -1;
 
+	if (!should_use_loose_object_map(repo))
+		return 0;
+
 	if (!loose->map)
 		loose_object_map_init(&loose->map);
 	if (!loose->cache) {
@@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
 {
 	struct odb_source *source;
 
-	if (!should_use_loose_object_map(repo))
-		return 0;
-
 	odb_prepare_alternates(repo->objects);
-
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(files->loose) < 0)
+		if (loose_object_map_load(files->loose) < 0)
 			return -1;
 	}
 
diff --git a/loose.h b/loose.h
index 6c9b3f4571..ed663ac550 100644
--- a/loose.h
+++ b/loose.h
@@ -13,6 +13,7 @@ struct loose_object_map {
 
 void loose_object_map_init(struct loose_object_map **map);
 void loose_object_map_clear(struct loose_object_map **map);
+int loose_object_map_load(struct odb_source_loose *loose);
 int repo_loose_object_map_oid(struct repository *repo,
 			      const struct object_id *src,
 			      const struct git_hash_algo *dest_algo,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3f7d04a56e..812ca1c138 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 	if (!is_absolute_path(loose->base.path))
 		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
 
+	loose_object_map_load(loose);
+
 	return loose;
 }
diff --git a/repository.c b/repository.c
index 2ef0778846..6d633002b4 100644
--- a/repository.c
+++ b/repository.c
@@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
 	if (hash_algo_by_ptr(repo->hash_algo) == algo)
 		BUG("hash_algo and compat_hash_algo match");
 	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
-	if (repo->compat_hash_algo)
-		repo_read_loose_object_map(repo);
 #else
 	if (algo)
 		die(_("compatibility hash algorithm support requires Rust"));
diff --git a/setup.c b/setup.c
index d31808130b..825572f5f1 100644
--- a/setup.c
+++ b/setup.c
@@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
 
 	repo->bare_cfg = format->is_bare;
 	repo_set_hash_algo(repo, format->hash_algo);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
 	repo_set_ref_storage_format(repo,
 				    format->ref_storage_format,
@@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	free(alternate_object_directories);
 	free(object_directory);
 	return 0;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v2 3/5] setup: defer object database creation

[Patrick Steinhardt]

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Defer the object database creation so that we handle both steps in the
same function.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 setup.c | 35 +++++++++++++++++++++++++++--------
 setup.h |  9 +++++++++
 2 files changed, 36 insertions(+), 8 deletions(-)

diff --git a/setup.c b/setup.c
index 825572f5f1..a7b1b9eaef 100644
--- a/setup.c
+++ b/setup.c
@@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
 	return result;
 }
 
+static void get_object_directories(char **object_directory,
+				   char **alternate_object_directories)
+{
+	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+}
+
 int apply_repository_format(struct repository *repo,
 			    const struct repository_format *format,
 			    enum apply_repository_format_flags flags,
@@ -1779,8 +1786,9 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+		get_object_directories(&object_directory,
+				       &alternate_object_directories);
+
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,8 +1811,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION))
+		repo->objects = odb_new(repo, object_directory,
+					alternate_object_directories);
 
 	free(alternate_object_directories);
 	free(object_directory);
@@ -2654,11 +2663,16 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
+	char *object_directory, *alternate_object_directories;
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	get_object_directories(&object_directory, &alternate_object_directories);
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2672,6 +2686,8 @@ static void create_object_directory(struct repository *repo)
 	strbuf_addstr(&path, "/info");
 	safe_create_dir(repo, path.buf, 1);
 
+	free(alternate_object_directories);
+	free(object_directory);
 	strbuf_release(&path);
 }
 
@@ -2867,9 +2883,10 @@ int init_db(struct repository *repo,
 	 */
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2885,7 +2902,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];
diff --git a/setup.h b/setup.h
index 654f10e059..e55d647b70 100644
--- a/setup.h
+++ b/setup.h
@@ -241,6 +241,15 @@ enum apply_repository_format_flags {
 	 * relate to the object database.
 	 */
 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
+
+	/*
+	 * Usually, the object database is created after the repository format
+	 * was applied. This step is skipped if this flag is set, which leaves
+	 * us with a partially-working repository.
+	 *
+	 * This is useful when initializing a new repository.
+	 */
+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
 };
 
 /*

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v2 4/5] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++++++
 odb/source.h          |  6 ++++++
 6 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/odb/source-files.h b/odb/source-files.h
index d7ac3c1c81..6a803afdda 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -28,7 +28,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_FILES)
-		BUG("trying to downcast source of type '%d' to files", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_FILES));
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source-inmemory.h b/odb/source-inmemory.h
index a88fc2e320..adbad23e8b 100644
--- a/odb/source-inmemory.h
+++ b/odb/source-inmemory.h
@@ -26,7 +26,9 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)
 static inline struct odb_source_inmemory *odb_source_inmemory_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_INMEMORY)
-		BUG("trying to downcast source of type '%d' to in-memory", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_INMEMORY));
 	return container_of(source, struct odb_source_inmemory, base);
 }
 
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 6070aaf3ce..3cf2e1f8f1 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -41,7 +41,9 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_LOOSE)
-		BUG("trying to downcast source of type '%d' to loose", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_LOOSE));
 	return container_of(source, struct odb_source_loose, base);
 }
 
diff --git a/odb/source-packed.h b/odb/source-packed.h
index 77309ddd09..a0f6b5096d 100644
--- a/odb/source-packed.h
+++ b/odb/source-packed.h
@@ -78,7 +78,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
 static inline struct odb_source_packed *odb_source_packed_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_PACKED)
-		BUG("trying to downcast source of type '%d' to packed", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_PACKED));
 	return container_of(source, struct odb_source_packed, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index 7993dcbd65..30188b806d 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -4,6 +4,25 @@
 #include "odb/source.h"
 #include "packfile.h"
 
+static const char * const odb_source_names_by_type[] = {
+	[ODB_SOURCE_UNKNOWN] = "unknown",
+	[ODB_SOURCE_FILES] = "files",
+	[ODB_SOURCE_LOOSE] = "loose",
+	[ODB_SOURCE_PACKED] = "packed",
+	[ODB_SOURCE_INMEMORY] = "in-memory",
+};
+
+const char *odb_source_type_to_name(enum odb_source_type type)
+{
+	const char *name;
+	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
+		type = ODB_SOURCE_UNKNOWN;
+	name = odb_source_names_by_type[type];
+	if (!name)
+		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
+	return name;
+}
+
 struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
diff --git a/odb/source.h b/odb/source.h
index cd63dba91f..ab16d152f4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -25,6 +25,12 @@ enum odb_source_type {
 	ODB_SOURCE_INMEMORY,
 };
 
+/*
+ * Convert between the enum and its name. Returns the equivalent of "unknown"
+ * for unknown types.
+ */
+const char *odb_source_type_to_name(enum odb_source_type type);
+
 struct object_id;
 struct odb_read_stream;
 struct strvec;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v2 5/5] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.

Make the creation of on-disk structures pluggable to accommodate for
this.

Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 19 +++++++++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 setup.c            | 35 ++++++++++++++++++++---------------
 3 files changed, 62 insertions(+), 15 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 4138758511..0db6e681fe 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -9,6 +9,7 @@
 #include "odb/source-files.h"
 #include "odb/source-loose.h"
 #include "packfile.h"
+#include "path.h"
 #include "strbuf.h"
 #include "write-or-die.h"
 
@@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
 	odb_source_close(&files->packed->base);
 }
 
+static int odb_source_files_create_on_disk(struct odb_source *source)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	safe_create_dir(source->odb->repo, source->path, 1);
+
+	strbuf_addf(&path, "%s/pack", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/info", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_release(&path);
+	return 0;
+}
+
 static void odb_source_files_prepare(struct odb_source *source,
 				     enum odb_prepare_flags flags)
 {
@@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
+	files->base.create_on_disk = odb_source_files_create_on_disk;
 	files->base.prepare = odb_source_files_prepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
diff --git a/odb/source.h b/odb/source.h
index ab16d152f4..4abc418bdd 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -89,6 +89,18 @@ struct odb_source {
 	 */
 	void (*close)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to create on-disk data structures that are
+	 * required for this source to operate.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 *
+	 * This callback may be NULL in case the source does not need any
+	 * on-disk setup.
+	 */
+	int (*create_on_disk)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to prepare the source so that it becomes
 	 * ready for use. It optionally clears underlying caches of the object
@@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
 	source->close(source);
 }
 
+/*
+ * Create on-disk data structures that are required for this source to operate
+ * correctly. Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_create_on_disk(struct odb_source *source)
+{
+	if (!source->create_on_disk)
+		return 0;
+	return source->create_on_disk(source);
+}
+
 /*
  * Prepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects
diff --git a/setup.c b/setup.c
index a7b1b9eaef..14ef119cb7 100644
--- a/setup.c
+++ b/setup.c
@@ -2666,29 +2666,34 @@ static int create_default_files(struct repository *repo,
 static void create_object_database(struct repository *repo)
 {
 	char *object_directory, *alternate_object_directories;
-	struct strbuf path = STRBUF_INIT;
-	size_t baselen;
 
 	get_object_directories(&object_directory, &alternate_object_directories);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 
-	strbuf_addstr(&path, repo_get_object_directory(repo));
-	baselen = path.len;
-
-	safe_create_dir(repo, path.buf, 1);
+	/*
+	 * Create the "objects" directory in the common directory. This is done
+	 * so that the repository can be discovered regardless of the backend
+	 * used.
+	 *
+	 * Note that we only do this in case the object directory wasn't
+	 * overwritten via an environment variable. If it _is_ being overridden
+	 * then we skip this step, as the repository won't be discoverable
+	 * anyway without the environment variable.
+	 */
+	if (!object_directory) {
+		struct strbuf objects_dir = STRBUF_INIT;
+		repo_common_path_append(repo, &objects_dir, "objects");
+		safe_create_dir(repo, objects_dir.buf, 1);
+		strbuf_release(&objects_dir);
+	}
 
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/pack");
-	safe_create_dir(repo, path.buf, 1);
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
 
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/info");
-	safe_create_dir(repo, path.buf, 1);
+	if (odb_source_create_on_disk(repo->objects->sources) < 0)
+		die("failed creating object database");
 
 	free(alternate_object_directories);
 	free(object_directory);
-	strbuf_release(&path);
 }
 
 static void separate_git_dir(const char *git_dir, const char *git_link)

-- 
2.55.0.679.g6767b8d81c.dirty

Re: [PATCH v2 0/5] odb: make creation of object database pluggable

[Justin Tobler]

On 26/08/04 10:29AM, Patrick Steinhardt wrote:
> Changes in v2:
>   - Add a testcase that demonstrates the bug fixed with alternate loose
>     object maps.
>   - Rename the "inmemory" bakcend to "in-memory".
>   - Clarify some commit messages.
>   - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im

From the range-diff, this version of the series looks good to me.

-Justin

Re: [PATCH v2 3/5] setup: defer object database creation

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> In a subsequent commit we'll make the creation of the on-disk data
> structures of an object database pluggable. This will lead to an
> in-between state where we have already configured the repository's
> object database, but it's not usable yet until we eventually call
> `create_object_directory()`.
>
> Defer the object database creation so that we handle both steps in the
> same function.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  setup.c | 35 +++++++++++++++++++++++++++--------
>  setup.h |  9 +++++++++
>  2 files changed, 36 insertions(+), 8 deletions(-)
>
> diff --git a/setup.c b/setup.c
> index 825572f5f1..a7b1b9eaef 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
>  	return result;
>  }
>  
> +static void get_object_directories(char **object_directory,
> +				   char **alternate_object_directories)
> +{
> +	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> +	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> +}

Would it make sense to wrap these in a APPLY_REPOSITORY_FORMAT_HONOR_ENV
guard?

I mean, below we call this function *only* when flags has that bit set.
But the return values of that function are used at the bottom of
apply_repository_format(), that's a bit awkard.

So can I suggest the following patch instead? That would remove the
weird double pointer passing around, which feels a bit unneeded.


--- >8 ---
Subject: [PATCH] setup: defer object database creation

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Defer the object database creation so that we handle both steps in the
same function.

Signed-off-by: Toon Claes <toon@iotcl.com>
---
 setup.c | 35 +++++++++++++++++++++++++++--------
 setup.h |  9 +++++++++
 2 files changed, 36 insertions(+), 8 deletions(-)

diff --git a/setup.c b/setup.c
index 825572f5f1..2e9bc92481 100644
--- a/setup.c
+++ b/setup.c
@@ -1760,13 +1760,28 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
 	return result;
 }
 
+static void setup_objects_odb_new(struct repository *repo,
+				  bool from_env)
+{
+	char *object_directory = NULL, *alternate_object_directories = NULL;
+
+	if (from_env) {
+		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+	}
+
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
+	free(alternate_object_directories);
+	free(object_directory);
+}
+
 int apply_repository_format(struct repository *repo,
 			    const struct repository_format *format,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	char *object_directory = NULL, *alternate_object_directories = NULL;
-
 	if (verify_repository_format(format, err) < 0)
 		return -1;
 
@@ -1779,8 +1794,6 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,11 +1816,11 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)
+		return 0;
+
+	setup_objects_odb_new(repo, flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV);
 
-	free(alternate_object_directories);
-	free(object_directory);
 	return 0;
 }
 
@@ -2654,11 +2667,13 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	setup_objects_odb_new(repo, true);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2867,9 +2882,10 @@ int init_db(struct repository *repo,
 	 */
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2885,7 +2901,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];
diff --git a/setup.h b/setup.h
index 654f10e059..e55d647b70 100644
--- a/setup.h
+++ b/setup.h
@@ -241,6 +241,15 @@ enum apply_repository_format_flags {
 	 * relate to the object database.
 	 */
 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
+
+	/*
+	 * Usually, the object database is created after the repository format
+	 * was applied. This step is skipped if this flag is set, which leaves
+	 * us with a partially-working repository.
+	 *
+	 * This is useful when initializing a new repository.
+	 */
+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
 };
 
 /*
-- 
2.55.0.629.g250fe7f194

Re: [PATCH v2 3/5] setup: defer object database creation

[Patrick Steinhardt]

On Tue, Aug 04, 2026 at 08:48:42PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > In a subsequent commit we'll make the creation of the on-disk data
> > structures of an object database pluggable. This will lead to an
> > in-between state where we have already configured the repository's
> > object database, but it's not usable yet until we eventually call
> > `create_object_directory()`.
> >
> > Defer the object database creation so that we handle both steps in the
> > same function.
> >
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  setup.c | 35 +++++++++++++++++++++++++++--------
> >  setup.h |  9 +++++++++
> >  2 files changed, 36 insertions(+), 8 deletions(-)
> >
> > diff --git a/setup.c b/setup.c
> > index 825572f5f1..a7b1b9eaef 100644
> > --- a/setup.c
> > +++ b/setup.c
> > @@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
> >  	return result;
> >  }
> >  
> > +static void get_object_directories(char **object_directory,
> > +				   char **alternate_object_directories)
> > +{
> > +	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> > +	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> > +}
> 
> Would it make sense to wrap these in a APPLY_REPOSITORY_FORMAT_HONOR_ENV
> guard?
> 
> I mean, below we call this function *only* when flags has that bit set.
> But the return values of that function are used at the bottom of
> apply_repository_format(), that's a bit awkard.
> 
> So can I suggest the following patch instead? That would remove the
> weird double pointer passing around, which feels a bit unneeded.

You're right, this is somewhat awkward. I have a different proposal
though: instead of creating a separate function, we can move handling of
environment variables into `odb_new()` itself. This also paves the way
for moving handling of these environment variables into the backend,
which is something I want to do soonish [1].

Patrick

[1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/

[PATCH v3 0/6] odb: make creation of object database pluggable

[Patrick Steinhardt]

Hi,

when creating a new repository we create a couple of on-disk data
structures for the object database. This includes the "objects/"
directory hierarchy with "objects/info" and "objects/pack", which are
specific to the backend.

This patch series makes the creation of the on-disk data structures
pluggable. While we continue to always create "objects/" regardless of
the backend (it's required for a repository to be recognized as such),
the other subdirectories are now created by the backend. This will allow
other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This
detour is required so that we can defer initialization of the object
database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).

Changes in v3:
  - Move handling of GIT_OBJECT_DIRECTORY and
    GIT_ALTERNATE_OBJECT_DIRECTORIES into `odb_new()` itself. This
    deduplicates some of the logic and also preps us for a future where
    alternates are handled in the "files" backend itself.
  - Link to v2: https://patch.msgid.link/20260804-pks-odb-create-on-disk-v2-0-ddf8b59bd207@pks.im

Changes in v2:
  - Add a testcase that demonstrates the bug fixed with alternate loose
    object maps.
  - Rename the "inmemory" bakcend to "in-memory".
  - Clarify some commit messages.
  - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (6):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: handle ODB-related environment variables in `odb_new()`
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable

 loose.c                       | 25 +++++++++---------
 loose.h                       |  1 +
 odb.c                         | 20 +++++++++------
 odb.h                         | 17 ++++++++++--
 odb/source-files.c            | 19 ++++++++++++++
 odb/source-files.h            |  4 ++-
 odb/source-inmemory.h         |  4 ++-
 odb/source-loose.c            |  2 ++
 odb/source-loose.h            |  4 ++-
 odb/source-packed.h           |  4 ++-
 odb/source.c                  | 19 ++++++++++++++
 odb/source.h                  | 29 +++++++++++++++++++++
 repository.c                  |  2 --
 setup.c                       | 60 ++++++++++++++++++++++++-------------------
 setup.h                       |  9 +++++++
 t/t1016-compatObjectFormat.sh | 18 +++++++++++++
 t/unit-tests/u-odb-inmemory.c |  2 +-
 17 files changed, 183 insertions(+), 56 deletions(-)

Range-diff versus v2:

1:  b0beb61a74 = 1:  d384dd0635 loose: load loose object map for the correct source
2:  097bdcad14 = 2:  0ee1b3c032 setup: detangle loading of loose object maps
-:  ---------- > 3:  f52992b9bd setup: handle ODB-related environment variables in `odb_new()`
3:  06645224ef ! 4:  4524fc5ec4 setup: defer object database creation
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## setup.c ##
    -@@ setup.c: enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
    - 	return result;
    - }
    - 
    -+static void get_object_directories(char **object_directory,
    -+				   char **alternate_object_directories)
    -+{
    -+	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
    -+	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
    -+}
    -+
    - int apply_repository_format(struct repository *repo,
    - 			    const struct repository_format *format,
    - 			    enum apply_repository_format_flags flags,
     @@ setup.c: int apply_repository_format(struct repository *repo,
    - 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
    - 		const char *shallow_file;
    + 			    enum apply_repository_format_flags flags,
    + 			    struct strbuf *err)
    + {
    +-	enum odb_new_flags odb_new_flags = 0;
    +-
    + 	if (verify_repository_format(format, err) < 0)
    + 		return -1;
      
    --		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
    --		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
    -+		get_object_directories(&object_directory,
    -+				       &alternate_object_directories);
    -+
    - 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
    - 		if (shallow_file)
    - 			set_alternate_shallow_file(repo, shallow_file);
     @@ setup.c: int apply_repository_format(struct repository *repo,
      	repo->repository_format_precious_objects =
      		format->precious_objects;
      
    --	repo->objects = odb_new(repo, object_directory,
    --				alternate_object_directories);
    -+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION))
    -+		repo->objects = odb_new(repo, object_directory,
    -+					alternate_object_directories);
    +-	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
    +-		odb_new_flags |= ODB_NEW_HONOR_ENV;
    +-	repo->objects = odb_new(repo, odb_new_flags);
    ++	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)) {
    ++		enum odb_new_flags odb_new_flags = 0;
    ++		if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
    ++			odb_new_flags |= ODB_NEW_HONOR_ENV;
    ++		repo->objects = odb_new(repo, odb_new_flags);
    ++	}
      
    - 	free(alternate_object_directories);
    - 	free(object_directory);
    + 	return 0;
    + }
     @@ setup.c: static int create_default_files(struct repository *repo,
      	return reinit;
      }
    @@ setup.c: static int create_default_files(struct repository *repo,
     -static void create_object_directory(struct repository *repo)
     +static void create_object_database(struct repository *repo)
      {
    -+	char *object_directory, *alternate_object_directories;
      	struct strbuf path = STRBUF_INIT;
      	size_t baselen;
      
    -+	get_object_directories(&object_directory, &alternate_object_directories);
    -+	repo->objects = odb_new(repo, object_directory,
    -+				alternate_object_directories);
    ++	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
     +
      	strbuf_addstr(&path, repo_get_object_directory(repo));
      	baselen = path.len;
      
    -@@ setup.c: static void create_object_directory(struct repository *repo)
    - 	strbuf_addstr(&path, "/info");
    - 	safe_create_dir(repo, path.buf, 1);
    - 
    -+	free(alternate_object_directories);
    -+	free(object_directory);
    - 	strbuf_release(&path);
    - }
    - 
     @@ setup.c: int init_db(struct repository *repo,
      	 */
      	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
4:  46ad0386bb = 5:  c526fd526b odb/source: introduce function to map source type to name
5:  3063325cf9 ! 6:  d752e48eba odb: make creation of on-disk structures pluggable
    @@ odb/source.h: static inline void odb_source_close(struct odb_source *source)
     
      ## setup.c ##
     @@ setup.c: static int create_default_files(struct repository *repo,
    + 
      static void create_object_database(struct repository *repo)
      {
    - 	char *object_directory, *alternate_object_directories;
     -	struct strbuf path = STRBUF_INIT;
     -	size_t baselen;
    - 
    - 	get_object_directories(&object_directory, &alternate_object_directories);
    --	repo->objects = odb_new(repo, object_directory,
    --				alternate_object_directories);
    - 
    --	strbuf_addstr(&path, repo_get_object_directory(repo));
    --	baselen = path.len;
    --
    --	safe_create_dir(repo, path.buf, 1);
     +	/*
     +	 * Create the "objects" directory in the common directory. This is done
     +	 * so that the repository can be discovered regardless of the backend
    @@ setup.c: static int create_default_files(struct repository *repo,
     +	 * then we skip this step, as the repository won't be discoverable
     +	 * anyway without the environment variable.
     +	 */
    -+	if (!object_directory) {
    ++	if (!getenv(DB_ENVIRONMENT)) {
     +		struct strbuf objects_dir = STRBUF_INIT;
     +		repo_common_path_append(repo, &objects_dir, "objects");
     +		safe_create_dir(repo, objects_dir.buf, 1);
     +		strbuf_release(&objects_dir);
     +	}
      
    + 	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
    + 
    +-	strbuf_addstr(&path, repo_get_object_directory(repo));
    +-	baselen = path.len;
    +-
    +-	safe_create_dir(repo, path.buf, 1);
    +-
     -	strbuf_setlen(&path, baselen);
     -	strbuf_addstr(&path, "/pack");
     -	safe_create_dir(repo, path.buf, 1);
    -+	repo->objects = odb_new(repo, object_directory,
    -+				alternate_object_directories);
    - 
    +-
     -	strbuf_setlen(&path, baselen);
     -	strbuf_addstr(&path, "/info");
     -	safe_create_dir(repo, path.buf, 1);
    +-
    +-	strbuf_release(&path);
     +	if (odb_source_create_on_disk(repo->objects->sources) < 0)
     +		die("failed creating object database");
    - 
    - 	free(alternate_object_directories);
    - 	free(object_directory);
    --	strbuf_release(&path);
      }
      
      static void separate_git_dir(const char *git_dir, const char *git_link)

---
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

[PATCH v3 1/6] loose: load loose object map for the correct source

[Patrick Steinhardt]

When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source. The consequence is that we'll ignore loose object maps of
any configured alternates.

Fix this bug by instead loading the map via the loose source's path.

Helped-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c                       | 18 ++++++++++--------
 t/t1016-compatObjectFormat.sh | 18 ++++++++++++++++++
 2 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/loose.c b/loose.c
index bf01d3e42d..9dad75373b 100644
--- a/loose.c
+++ b/loose.c
@@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
+static int load_one_loose_object_map(struct odb_source_loose *loose)
 {
-	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+	struct repository *repo = loose->base.odb->repo;
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
 	FILE *fp;
 	int ret = -1;
 
@@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
 	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
 
-	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
-	fp = fopen(path.buf, "rb");
+	path = xstrfmt("%s/loose-object-idx", loose->base.path);
+	fp = fopen(path, "rb");
 	if (!fp) {
-		strbuf_release(&path);
+		free(path);
 		return 0;
 	}
 
@@ -102,7 +104,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 err:
 	fclose(fp);
 	strbuf_release(&buf);
-	strbuf_release(&path);
+	free(path);
 	return ret;
 }
 
@@ -117,10 +119,10 @@ int repo_read_loose_object_map(struct repository *repo)
 
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(repo, files->loose) < 0) {
+		if (load_one_loose_object_map(files->loose) < 0)
 			return -1;
-		}
 	}
+
 	return 0;
 }
 
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 92d48b96a1..9cafcee509 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -187,6 +187,24 @@ do
 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
 		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
 	'
+
+	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
+		for repo in alt borrow
+		do
+			test_when_finished "rm -rf $repo" &&
+			git init --object-format=$hash $repo &&
+			git -C $repo config set core.repositoryformatversion 1 &&
+			git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
+		done &&
+
+		git -C alt commit --allow-empty --message A &&
+		echo "$(pwd)/alt/.git/objects" >borrow/.git/objects/info/alternates &&
+
+		oid=$(git -C alt rev-parse HEAD) &&
+		git -C alt    rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
+		git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+		test_cmp expect actual
+	'
 done
 cd "$base"
 

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v3 2/6] setup: detangle loading of loose object maps

[Patrick Steinhardt]

When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.

Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:

  - We assume that the primary object database is the "files" object
    database and unconditionally downcast it. This will cause us to BUG
    in case a different object database type was used together with a
    compat hash algorithm.

  - We require the object database to already have been initialized when
    configuring the object database. This means that we must intermix
    configuration of the repository and initialization of its
    sub-structures in a weird way.

Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c            | 11 +++++------
 loose.h            |  1 +
 odb/source-loose.c |  2 ++
 repository.c       |  2 --
 setup.c            |  5 +++--
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/loose.c b/loose.c
index 9dad75373b..a3b2dcedc2 100644
--- a/loose.c
+++ b/loose.c
@@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct odb_source_loose *loose)
+int loose_object_map_load(struct odb_source_loose *loose)
 {
 	struct repository *repo = loose->base.odb->repo;
 	struct strbuf buf = STRBUF_INIT;
@@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
 	FILE *fp;
 	int ret = -1;
 
+	if (!should_use_loose_object_map(repo))
+		return 0;
+
 	if (!loose->map)
 		loose_object_map_init(&loose->map);
 	if (!loose->cache) {
@@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
 {
 	struct odb_source *source;
 
-	if (!should_use_loose_object_map(repo))
-		return 0;
-
 	odb_prepare_alternates(repo->objects);
-
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(files->loose) < 0)
+		if (loose_object_map_load(files->loose) < 0)
 			return -1;
 	}
 
diff --git a/loose.h b/loose.h
index 6c9b3f4571..ed663ac550 100644
--- a/loose.h
+++ b/loose.h
@@ -13,6 +13,7 @@ struct loose_object_map {
 
 void loose_object_map_init(struct loose_object_map **map);
 void loose_object_map_clear(struct loose_object_map **map);
+int loose_object_map_load(struct odb_source_loose *loose);
 int repo_loose_object_map_oid(struct repository *repo,
 			      const struct object_id *src,
 			      const struct git_hash_algo *dest_algo,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3f7d04a56e..812ca1c138 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 	if (!is_absolute_path(loose->base.path))
 		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
 
+	loose_object_map_load(loose);
+
 	return loose;
 }
diff --git a/repository.c b/repository.c
index 2ef0778846..6d633002b4 100644
--- a/repository.c
+++ b/repository.c
@@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
 	if (hash_algo_by_ptr(repo->hash_algo) == algo)
 		BUG("hash_algo and compat_hash_algo match");
 	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
-	if (repo->compat_hash_algo)
-		repo_read_loose_object_map(repo);
 #else
 	if (algo)
 		die(_("compatibility hash algorithm support requires Rust"));
diff --git a/setup.c b/setup.c
index d31808130b..825572f5f1 100644
--- a/setup.c
+++ b/setup.c
@@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
 
 	repo->bare_cfg = format->is_bare;
 	repo_set_hash_algo(repo, format->hash_algo);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
 	repo_set_ref_storage_format(repo,
 				    format->ref_storage_format,
@@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	free(alternate_object_directories);
 	free(object_directory);
 	return 0;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v3 3/6] setup: handle ODB-related environment variables in `odb_new()`

[Patrick Steinhardt]

When initializing a repository's object database we have to respect the
GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment
variables, which can be set by the user to override the default location
of where we write objects to and read objects from.

This is handled in `apply_repository_format()`, which is fine. But in a
subsequent commit we'll have to defer constructing the object database
to a later point in some cases, and that will require a second site
where we call `odb_new()`. And of course, that second site would have to
handle those environment variables, as well.

It would be somewhat awkward to duplicate the logic though. But there's
a better alternative: instead of handling this logic in "setup.c", we
can easily handle environment variables in `odb_new()` itself. This
ensures that object database creation is neatly self-contained, and we
don't have to duplicate any of the logic.

Another benefit is that in a future patch series we plan to move
handling of alternates into the backends themselves [1], and that will
require us to also handle those environment variables in the "files"
backend itself. So moving the logic into the ODB level already gets us
one step closer to that goal.

Refactor the logic accordingly.

[1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c                         | 20 ++++++++++++--------
 odb.h                         | 17 +++++++++++++++--
 setup.c                       | 11 ++++-------
 t/unit-tests/u-odb-inmemory.c |  2 +-
 4 files changed, 32 insertions(+), 18 deletions(-)

diff --git a/odb.c b/odb.c
index cf6e7938c0..b463afa072 100644
--- a/odb.c
+++ b/odb.c
@@ -1004,26 +1004,30 @@ int odb_write_object_stream(struct object_database *odb,
 }
 
 struct object_database *odb_new(struct repository *repo,
-				const char *primary_source,
-				const char *secondary_sources)
+				enum odb_new_flags flags)
 {
-	struct object_database *o = xmalloc(sizeof(*o));
-	char *to_free = NULL;
+	char *primary_source = NULL, *secondary_sources = NULL;
+	struct object_database *o;
 
-	memset(o, 0, sizeof(*o));
+	CALLOC_ARRAY(o, 1);
 	o->repo = repo;
 	pthread_mutex_init(&o->replace_mutex, NULL);
 	string_list_init_dup(&o->submodule_source_paths);
 
+	if (flags & ODB_NEW_HONOR_ENV) {
+		primary_source = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+		secondary_sources = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+	}
 	if (!primary_source)
-		primary_source = to_free = xstrfmt("%s/objects", repo->commondir);
+		primary_source = xstrfmt("%s/objects", repo->commondir);
+
 	o->sources = odb_source_new(o, primary_source, true);
 	o->sources_tail = &o->sources->next;
 	o->alternate_db = xstrdup_or_null(secondary_sources);
 	o->inmemory_objects = &odb_source_inmemory_new(o)->base;
 
-	free(to_free);
-
+	free(secondary_sources);
+	free(primary_source);
 	return o;
 }
 
diff --git a/odb.h b/odb.h
index 7995bed97b..8ec335c7f7 100644
--- a/odb.h
+++ b/odb.h
@@ -100,6 +100,20 @@ struct object_database {
 	struct string_list submodule_source_paths;
 };
 
+enum odb_new_flags {
+	/*
+	 * Honor environment variables when constructing the object database
+	 * sources. This makes us respect the following environment variables:
+	 *
+	 *   - GIT_OBJECT_DIRECTORY to override the primary object directory.
+	 *
+	 *   - GIT_ALTERNATE_OBJECT_DIRECTORIES to override alternates.
+	 *
+	 * Environment variables may be backend-specific.
+	 */
+	ODB_NEW_HONOR_ENV = (1 << 0),
+};
+
 /*
  * Create a new object database for the given repository.
  *
@@ -112,8 +126,7 @@ struct object_database {
  * Returns the newly created object database.
  */
 struct object_database *odb_new(struct repository *repo,
-				const char *primary_source,
-				const char *alternate_sources);
+				enum odb_new_flags flags);
 
 /* Free the object database and release all resources. */
 void odb_free(struct object_database *o);
diff --git a/setup.c b/setup.c
index 825572f5f1..5dfab3e79e 100644
--- a/setup.c
+++ b/setup.c
@@ -1765,7 +1765,7 @@ int apply_repository_format(struct repository *repo,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	char *object_directory = NULL, *alternate_object_directories = NULL;
+	enum odb_new_flags odb_new_flags = 0;
 
 	if (verify_repository_format(format, err) < 0)
 		return -1;
@@ -1779,8 +1779,6 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,11 +1801,10 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
+		odb_new_flags |= ODB_NEW_HONOR_ENV;
+	repo->objects = odb_new(repo, odb_new_flags);
 
-	free(alternate_object_directories);
-	free(object_directory);
 	return 0;
 }
 
diff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c
index 6844bfc37c..db323e10fd 100644
--- a/t/unit-tests/u-odb-inmemory.c
+++ b/t/unit-tests/u-odb-inmemory.c
@@ -38,7 +38,7 @@ static void cl_assert_object_info(struct odb_source_inmemory *source,
 
 void test_odb_inmemory__initialize(void)
 {
-	odb = odb_new(&repo, "", "");
+	odb = odb_new(&repo, 0);
 }
 
 void test_odb_inmemory__cleanup(void)

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v3 4/6] setup: defer object database creation

[Patrick Steinhardt]

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Defer the object database creation so that we handle both steps in the
same function.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 setup.c | 24 +++++++++++++++---------
 setup.h |  9 +++++++++
 2 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/setup.c b/setup.c
index 5dfab3e79e..d85171f3b6 100644
--- a/setup.c
+++ b/setup.c
@@ -1765,8 +1765,6 @@ int apply_repository_format(struct repository *repo,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	enum odb_new_flags odb_new_flags = 0;
-
 	if (verify_repository_format(format, err) < 0)
 		return -1;
 
@@ -1801,9 +1799,12 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
-		odb_new_flags |= ODB_NEW_HONOR_ENV;
-	repo->objects = odb_new(repo, odb_new_flags);
+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)) {
+		enum odb_new_flags odb_new_flags = 0;
+		if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
+			odb_new_flags |= ODB_NEW_HONOR_ENV;
+		repo->objects = odb_new(repo, odb_new_flags);
+	}
 
 	return 0;
 }
@@ -2651,11 +2652,13 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2864,9 +2867,10 @@ int init_db(struct repository *repo,
 	 */
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2882,7 +2886,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];
diff --git a/setup.h b/setup.h
index 654f10e059..e55d647b70 100644
--- a/setup.h
+++ b/setup.h
@@ -241,6 +241,15 @@ enum apply_repository_format_flags {
 	 * relate to the object database.
 	 */
 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
+
+	/*
+	 * Usually, the object database is created after the repository format
+	 * was applied. This step is skipped if this flag is set, which leaves
+	 * us with a partially-working repository.
+	 *
+	 * This is useful when initializing a new repository.
+	 */
+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
 };
 
 /*

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v3 5/6] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++++++
 odb/source.h          |  6 ++++++
 6 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/odb/source-files.h b/odb/source-files.h
index d7ac3c1c81..6a803afdda 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -28,7 +28,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_FILES)
-		BUG("trying to downcast source of type '%d' to files", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_FILES));
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source-inmemory.h b/odb/source-inmemory.h
index a88fc2e320..adbad23e8b 100644
--- a/odb/source-inmemory.h
+++ b/odb/source-inmemory.h
@@ -26,7 +26,9 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)
 static inline struct odb_source_inmemory *odb_source_inmemory_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_INMEMORY)
-		BUG("trying to downcast source of type '%d' to in-memory", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_INMEMORY));
 	return container_of(source, struct odb_source_inmemory, base);
 }
 
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 6070aaf3ce..3cf2e1f8f1 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -41,7 +41,9 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_LOOSE)
-		BUG("trying to downcast source of type '%d' to loose", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_LOOSE));
 	return container_of(source, struct odb_source_loose, base);
 }
 
diff --git a/odb/source-packed.h b/odb/source-packed.h
index 77309ddd09..a0f6b5096d 100644
--- a/odb/source-packed.h
+++ b/odb/source-packed.h
@@ -78,7 +78,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
 static inline struct odb_source_packed *odb_source_packed_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_PACKED)
-		BUG("trying to downcast source of type '%d' to packed", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_PACKED));
 	return container_of(source, struct odb_source_packed, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index 7993dcbd65..30188b806d 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -4,6 +4,25 @@
 #include "odb/source.h"
 #include "packfile.h"
 
+static const char * const odb_source_names_by_type[] = {
+	[ODB_SOURCE_UNKNOWN] = "unknown",
+	[ODB_SOURCE_FILES] = "files",
+	[ODB_SOURCE_LOOSE] = "loose",
+	[ODB_SOURCE_PACKED] = "packed",
+	[ODB_SOURCE_INMEMORY] = "in-memory",
+};
+
+const char *odb_source_type_to_name(enum odb_source_type type)
+{
+	const char *name;
+	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
+		type = ODB_SOURCE_UNKNOWN;
+	name = odb_source_names_by_type[type];
+	if (!name)
+		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
+	return name;
+}
+
 struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
diff --git a/odb/source.h b/odb/source.h
index cd63dba91f..ab16d152f4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -25,6 +25,12 @@ enum odb_source_type {
 	ODB_SOURCE_INMEMORY,
 };
 
+/*
+ * Convert between the enum and its name. Returns the equivalent of "unknown"
+ * for unknown types.
+ */
+const char *odb_source_type_to_name(enum odb_source_type type);
+
 struct object_id;
 struct odb_read_stream;
 struct strvec;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v3 6/6] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.

Make the creation of on-disk structures pluggable to accommodate for
this.

Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 19 +++++++++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 setup.c            | 34 ++++++++++++++++++----------------
 3 files changed, 60 insertions(+), 16 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 4138758511..0db6e681fe 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -9,6 +9,7 @@
 #include "odb/source-files.h"
 #include "odb/source-loose.h"
 #include "packfile.h"
+#include "path.h"
 #include "strbuf.h"
 #include "write-or-die.h"
 
@@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
 	odb_source_close(&files->packed->base);
 }
 
+static int odb_source_files_create_on_disk(struct odb_source *source)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	safe_create_dir(source->odb->repo, source->path, 1);
+
+	strbuf_addf(&path, "%s/pack", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/info", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_release(&path);
+	return 0;
+}
+
 static void odb_source_files_prepare(struct odb_source *source,
 				     enum odb_prepare_flags flags)
 {
@@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
+	files->base.create_on_disk = odb_source_files_create_on_disk;
 	files->base.prepare = odb_source_files_prepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
diff --git a/odb/source.h b/odb/source.h
index ab16d152f4..4abc418bdd 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -89,6 +89,18 @@ struct odb_source {
 	 */
 	void (*close)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to create on-disk data structures that are
+	 * required for this source to operate.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 *
+	 * This callback may be NULL in case the source does not need any
+	 * on-disk setup.
+	 */
+	int (*create_on_disk)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to prepare the source so that it becomes
 	 * ready for use. It optionally clears underlying caches of the object
@@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
 	source->close(source);
 }
 
+/*
+ * Create on-disk data structures that are required for this source to operate
+ * correctly. Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_create_on_disk(struct odb_source *source)
+{
+	if (!source->create_on_disk)
+		return 0;
+	return source->create_on_disk(source);
+}
+
 /*
  * Prepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects
diff --git a/setup.c b/setup.c
index d85171f3b6..af02cd965c 100644
--- a/setup.c
+++ b/setup.c
@@ -2654,25 +2654,27 @@ static int create_default_files(struct repository *repo,
 
 static void create_object_database(struct repository *repo)
 {
-	struct strbuf path = STRBUF_INIT;
-	size_t baselen;
+	/*
+	 * Create the "objects" directory in the common directory. This is done
+	 * so that the repository can be discovered regardless of the backend
+	 * used.
+	 *
+	 * Note that we only do this in case the object directory wasn't
+	 * overwritten via an environment variable. If it _is_ being overridden
+	 * then we skip this step, as the repository won't be discoverable
+	 * anyway without the environment variable.
+	 */
+	if (!getenv(DB_ENVIRONMENT)) {
+		struct strbuf objects_dir = STRBUF_INIT;
+		repo_common_path_append(repo, &objects_dir, "objects");
+		safe_create_dir(repo, objects_dir.buf, 1);
+		strbuf_release(&objects_dir);
+	}
 
 	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 
-	strbuf_addstr(&path, repo_get_object_directory(repo));
-	baselen = path.len;
-
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/pack");
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/info");
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_release(&path);
+	if (odb_source_create_on_disk(repo->objects->sources) < 0)
+		die("failed creating object database");
 }
 
 static void separate_git_dir(const char *git_dir, const char *git_link)

-- 
2.55.0.679.g6767b8d81c.dirty

Re: [PATCH v3 3/6] setup: handle ODB-related environment variables in `odb_new()`

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> When initializing a repository's object database we have to respect the
> GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment
> variables, which can be set by the user to override the default location
> of where we write objects to and read objects from.
>
> This is handled in `apply_repository_format()`, which is fine. But in a
> subsequent commit we'll have to defer constructing the object database
> to a later point in some cases, and that will require a second site
> where we call `odb_new()`. And of course, that second site would have to
> handle those environment variables, as well.
>
> It would be somewhat awkward to duplicate the logic though. But there's
> a better alternative: instead of handling this logic in "setup.c", we
> can easily handle environment variables in `odb_new()` itself. This
> ensures that object database creation is neatly self-contained, and we
> don't have to duplicate any of the logic.
>
> Another benefit is that in a future patch series we plan to move
> handling of alternates into the backends themselves [1], and that will
> require us to also handle those environment variables in the "files"
> backend itself. So moving the logic into the ODB level already gets us
> one step closer to that goal.
>
> Refactor the logic accordingly.

I like this!

>
> [1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb.c                         | 20 ++++++++++++--------
>  odb.h                         | 17 +++++++++++++++--
>  setup.c                       | 11 ++++-------
>  t/unit-tests/u-odb-inmemory.c |  2 +-
>  4 files changed, 32 insertions(+), 18 deletions(-)
>
> diff --git a/odb.c b/odb.c
> index cf6e7938c0..b463afa072 100644
> --- a/odb.c
> +++ b/odb.c
> @@ -1004,26 +1004,30 @@ int odb_write_object_stream(struct object_database *odb,
>  }
>  
>  struct object_database *odb_new(struct repository *repo,
> -				const char *primary_source,
> -				const char *secondary_sources)
> +				enum odb_new_flags flags)
>  {
> -	struct object_database *o = xmalloc(sizeof(*o));
> -	char *to_free = NULL;
> +	char *primary_source = NULL, *secondary_sources = NULL;
> +	struct object_database *o;
>  
> -	memset(o, 0, sizeof(*o));
> +	CALLOC_ARRAY(o, 1);
>  	o->repo = repo;
>  	pthread_mutex_init(&o->replace_mutex, NULL);
>  	string_list_init_dup(&o->submodule_source_paths);
>  
> +	if (flags & ODB_NEW_HONOR_ENV) {
> +		primary_source = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> +		secondary_sources = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> +	}
>  	if (!primary_source)
> -		primary_source = to_free = xstrfmt("%s/objects", repo->commondir);
> +		primary_source = xstrfmt("%s/objects", repo->commondir);
> +
>  	o->sources = odb_source_new(o, primary_source, true);
>  	o->sources_tail = &o->sources->next;
>  	o->alternate_db = xstrdup_or_null(secondary_sources);

I'd say this xstrdup_or_null() is not needed no more, and so is the
free() of that variable below.

>  	o->inmemory_objects = &odb_source_inmemory_new(o)->base;
>  
> -	free(to_free);
> -
> +	free(secondary_sources);
> +	free(primary_source);
>  	return o;
>  }

-- 
Cheers,
Toon

Re: [PATCH v3 4/6] setup: defer object database creation

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> In a subsequent commit we'll make the creation of the on-disk data
> structures of an object database pluggable. This will lead to an
> in-between state where we have already configured the repository's
> object database, but it's not usable yet until we eventually call
> `create_object_directory()`.
>
> Defer the object database creation so that we handle both steps in the
> same function.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  setup.c | 24 +++++++++++++++---------
>  setup.h |  9 +++++++++
>  2 files changed, 24 insertions(+), 9 deletions(-)
>
> diff --git a/setup.c b/setup.c
> index 5dfab3e79e..d85171f3b6 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1765,8 +1765,6 @@ int apply_repository_format(struct repository *repo,
>  			    enum apply_repository_format_flags flags,
>  			    struct strbuf *err)
>  {
> -	enum odb_new_flags odb_new_flags = 0;
> -
>  	if (verify_repository_format(format, err) < 0)
>  		return -1;
>  
> @@ -1801,9 +1799,12 @@ int apply_repository_format(struct repository *repo,
>  	repo->repository_format_precious_objects =
>  		format->precious_objects;
>  
> -	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
> -		odb_new_flags |= ODB_NEW_HONOR_ENV;
> -	repo->objects = odb_new(repo, odb_new_flags);
> +	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)) {
> +		enum odb_new_flags odb_new_flags = 0;
> +		if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
> +			odb_new_flags |= ODB_NEW_HONOR_ENV;
> +		repo->objects = odb_new(repo, odb_new_flags);
> +	}
>  
>  	return 0;
>  }
> @@ -2651,11 +2652,13 @@ static int create_default_files(struct repository *repo,
>  	return reinit;
>  }
>  
> -static void create_object_directory(struct repository *repo)
> +static void create_object_database(struct repository *repo)
>  {
>  	struct strbuf path = STRBUF_INIT;
>  	size_t baselen;
>  
> +	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
> +
>  	strbuf_addstr(&path, repo_get_object_directory(repo));
>  	baselen = path.len;
>  
> @@ -2864,9 +2867,10 @@ int init_db(struct repository *repo,
>  	 */
>  	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
>  	repository_format_configure(&repo_fmt, hash, ref_storage_format);
> -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
> +	if (apply_repository_format(repo, &repo_fmt,
> +				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
> +				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
>  		die("%s", err.buf);
> -	startup_info->have_repository = 1;
>  
>  	/*
>  	 * Ensure `core.hidedotfiles` is processed. This must happen after we
> @@ -2882,7 +2886,9 @@ int init_db(struct repository *repo,
>  
>  	if (!(flags & INIT_DB_SKIP_REFDB))
>  		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
> -	create_object_directory(repo);
> +	create_object_database(repo);
> +
> +	startup_info->have_repository = 1;
>  
>  	if (repo_settings_get_shared_repository(repo)) {
>  		char buf[10];
> diff --git a/setup.h b/setup.h
> index 654f10e059..e55d647b70 100644
> --- a/setup.h
> +++ b/setup.h
> @@ -241,6 +241,15 @@ enum apply_repository_format_flags {
>  	 * relate to the object database.
>  	 */
>  	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
> +
> +	/*
> +	 * Usually, the object database is created after the repository format
> +	 * was applied. This step is skipped if this flag is set, which leaves
> +	 * us with a partially-working repository.
> +	 *
> +	 * This is useful when initializing a new repository.
> +	 */
> +	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
>  };
>  
>  /*
>
> -- 
> 2.55.0.679.g6767b8d81c.dirty
>

With [PATCH v3 3/6], Justin's objection[1] is stronger now:

> Naive question: would it be simpler to just require invoking `odb_new()`
> explicitly after `apply_repository_format()` in all cases? There doesn't
> appear to be too many callsites.

As a matter of fact, I've given this a try and see these changes on top
of this series below.

[1]: <amkXcmwzbBYsMgjc@denethor>

--- >8 ---

diff --git a/repository.c b/repository.c
index 6d633002b4..9eee74113c 100644
--- a/repository.c
+++ b/repository.c
@@ -295,6 +295,8 @@ int repo_init(struct repository *repo,
 		goto error;
 	}
 
+	repo->objects = odb_new(repo, 0);
+
 	if (worktree)
 		repo_set_worktree(repo, worktree);
 
diff --git a/setup.c b/setup.c
index af02cd965c..1106f38bb0 100644
--- a/setup.c
+++ b/setup.c
@@ -1799,13 +1799,6 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)) {
-		enum odb_new_flags odb_new_flags = 0;
-		if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
-			odb_new_flags |= ODB_NEW_HONOR_ENV;
-		repo->objects = odb_new(repo, odb_new_flags);
-	}
-
 	return 0;
 }
 
@@ -1889,6 +1882,7 @@ const char *enter_repo(struct repository *repo, const char *path, unsigned flags
 		read_and_verify_repository_format(&fmt, ".", NULL);
 		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 			die("%s", err.buf);
+		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 		startup_info->have_repository = 1;
 
 		clear_repository_format(&fmt);
@@ -2092,6 +2086,8 @@ const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
 						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 				die("%s", err.buf);
 
+			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
+
 			clear_repository_format(&discovery.format);
 			strbuf_release(&err);
 		}
@@ -2870,8 +2866,7 @@ int init_db(struct repository *repo,
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
 	if (apply_repository_format(repo, &repo_fmt,
-				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
-				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 		die("%s", err.buf);
 
 	/*
diff --git a/setup.h b/setup.h
index e55d647b70..654f10e059 100644
--- a/setup.h
+++ b/setup.h
@@ -241,15 +241,6 @@ enum apply_repository_format_flags {
 	 * relate to the object database.
 	 */
 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
-
-	/*
-	 * Usually, the object database is created after the repository format
-	 * was applied. This step is skipped if this flag is set, which leaves
-	 * us with a partially-working repository.
-	 *
-	 * This is useful when initializing a new repository.
-	 */
-	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
 };
 
 /*

Re: [PATCH v3 6/6] odb: make creation of on-disk structures pluggable

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> When creating a new "files" object database source we have to create a
> couple of directories. These directories are of course specific to this
> particular backend, and a different backend may require a setup that is
> completely different.
>
> Make the creation of on-disk structures pluggable to accommodate for
> this.
>
> Note that there is one exception though: the "objects" directory must
> exist in a repository regardless of which backend is in use. If it
> doesn't exist then the repository is not treated as a Git repository at
> all. Consequently, we create this directory regardless of the backend.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.c | 19 +++++++++++++++++++
>  odb/source.h       | 23 +++++++++++++++++++++++
>  setup.c            | 34 ++++++++++++++++++----------------
>  3 files changed, 60 insertions(+), 16 deletions(-)
>
> diff --git a/odb/source-files.c b/odb/source-files.c
> index 4138758511..0db6e681fe 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -9,6 +9,7 @@
>  #include "odb/source-files.h"
>  #include "odb/source-loose.h"
>  #include "packfile.h"
> +#include "path.h"
>  #include "strbuf.h"
>  #include "write-or-die.h"
>  
> @@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
>  	odb_source_close(&files->packed->base);
>  }
>  
> +static int odb_source_files_create_on_disk(struct odb_source *source)
> +{
> +	struct strbuf path = STRBUF_INIT;
> +
> +	safe_create_dir(source->odb->repo, source->path, 1);
> +
> +	strbuf_addf(&path, "%s/pack", source->path);
> +	safe_create_dir(source->odb->repo, path.buf, 1);
> +
> +	strbuf_reset(&path);
> +	strbuf_addf(&path, "%s/info", source->path);
> +	safe_create_dir(source->odb->repo, path.buf, 1);
> +
> +	strbuf_release(&path);
> +	return 0;
> +}
> +
>  static void odb_source_files_prepare(struct odb_source *source,
>  				     enum odb_prepare_flags flags)
>  {
> @@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  
>  	files->base.free = odb_source_files_free;
>  	files->base.close = odb_source_files_close;
> +	files->base.create_on_disk = odb_source_files_create_on_disk;
>  	files->base.prepare = odb_source_files_prepare;
>  	files->base.read_object_info = odb_source_files_read_object_info;
>  	files->base.read_object_stream = odb_source_files_read_object_stream;
> diff --git a/odb/source.h b/odb/source.h
> index ab16d152f4..4abc418bdd 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -89,6 +89,18 @@ struct odb_source {
>  	 */
>  	void (*close)(struct odb_source *source);
>  
> +	/*
> +	 * This callback is expected to create on-disk data structures that are
> +	 * required for this source to operate.
> +	 *
> +	 * The callback is expected to return 0 on success, a negative error
> +	 * code otherwise.
> +	 *
> +	 * This callback may be NULL in case the source does not need any
> +	 * on-disk setup.
> +	 */
> +	int (*create_on_disk)(struct odb_source *source);
> +
>  	/*
>  	 * This callback is expected to prepare the source so that it becomes
>  	 * ready for use. It optionally clears underlying caches of the object
> @@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
>  	source->close(source);
>  }
>  
> +/*
> + * Create on-disk data structures that are required for this source to operate
> + * correctly. Returns 0 on success, a negative error code otherwise.
> + */
> +static inline int odb_source_create_on_disk(struct odb_source *source)
> +{
> +	if (!source->create_on_disk)
> +		return 0;
> +	return source->create_on_disk(source);
> +}
> +
>  /*
>   * Prepare the object database source and clear any caches. Depending on the
>   * backend used this may have the effect that concurrently-written objects
> diff --git a/setup.c b/setup.c
> index d85171f3b6..af02cd965c 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -2654,25 +2654,27 @@ static int create_default_files(struct repository *repo,
>  
>  static void create_object_database(struct repository *repo)
>  {
> -	struct strbuf path = STRBUF_INIT;
> -	size_t baselen;
> +	/*
> +	 * Create the "objects" directory in the common directory. This is done
> +	 * so that the repository can be discovered regardless of the backend
> +	 * used.
> +	 *
> +	 * Note that we only do this in case the object directory wasn't
> +	 * overwritten via an environment variable. If it _is_ being overridden
> +	 * then we skip this step, as the repository won't be discoverable
> +	 * anyway without the environment variable.
> +	 */
> +	if (!getenv(DB_ENVIRONMENT)) {

It's a bit sad that [PATCH 3/6] removed the use of DB_ENVIRONMENT from
this file, and now we're re-adding it. Although, I don't see how else we
can do this.

> +		struct strbuf objects_dir = STRBUF_INIT;
> +		repo_common_path_append(repo, &objects_dir, "objects");
> +		safe_create_dir(repo, objects_dir.buf, 1);
> +		strbuf_release(&objects_dir);
> +	}
>  
>  	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
>  
> -	strbuf_addstr(&path, repo_get_object_directory(repo));
> -	baselen = path.len;
> -
> -	safe_create_dir(repo, path.buf, 1);
> -
> -	strbuf_setlen(&path, baselen);
> -	strbuf_addstr(&path, "/pack");
> -	safe_create_dir(repo, path.buf, 1);
> -
> -	strbuf_setlen(&path, baselen);
> -	strbuf_addstr(&path, "/info");
> -	safe_create_dir(repo, path.buf, 1);
> -
> -	strbuf_release(&path);
> +	if (odb_source_create_on_disk(repo->objects->sources) < 0)
> +		die("failed creating object database");

This error isn't translatable.

>  }
>  
>  static void separate_git_dir(const char *git_dir, const char *git_link)
>
> -- 
> 2.55.0.679.g6767b8d81c.dirty
>

-- 
Cheers,
Toon

Re: [PATCH v3 4/6] setup: defer object database creation

[Patrick Steinhardt]

On Wed, Aug 05, 2026 at 04:21:39PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > In a subsequent commit we'll make the creation of the on-disk data
> > structures of an object database pluggable. This will lead to an
> > in-between state where we have already configured the repository's
> > object database, but it's not usable yet until we eventually call
> > `create_object_directory()`.
> >
> > Defer the object database creation so that we handle both steps in the
> > same function.
> 
> With [PATCH v3 3/6], Justin's objection[1] is stronger now:
> 
> > Naive question: would it be simpler to just require invoking `odb_new()`
> > explicitly after `apply_repository_format()` in all cases? There doesn't
> > appear to be too many callsites.
> 
> As a matter of fact, I've given this a try and see these changes on top
> of this series below.

The reason I was hesitant to do this is that I want to move
`apply_repository_format()` into `repo_init()` eventually. But I guess
moving the call to `odb_new()` out of it doesn't really prevent that.
So... fine, I'll do it.

Patrick

Re: [PATCH v3 3/6] setup: handle ODB-related environment variables in `odb_new()`

[Patrick Steinhardt]

On Wed, Aug 05, 2026 at 03:29:21PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/odb.c b/odb.c
> > index cf6e7938c0..b463afa072 100644
> > --- a/odb.c
> > +++ b/odb.c
> > @@ -1004,26 +1004,30 @@ int odb_write_object_stream(struct object_database *odb,
> >  }
> >  
> >  struct object_database *odb_new(struct repository *repo,
> > -				const char *primary_source,
> > -				const char *secondary_sources)
> > +				enum odb_new_flags flags)
> >  {
> > -	struct object_database *o = xmalloc(sizeof(*o));
> > -	char *to_free = NULL;
> > +	char *primary_source = NULL, *secondary_sources = NULL;
> > +	struct object_database *o;
> >  
> > -	memset(o, 0, sizeof(*o));
> > +	CALLOC_ARRAY(o, 1);
> >  	o->repo = repo;
> >  	pthread_mutex_init(&o->replace_mutex, NULL);
> >  	string_list_init_dup(&o->submodule_source_paths);
> >  
> > +	if (flags & ODB_NEW_HONOR_ENV) {
> > +		primary_source = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> > +		secondary_sources = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> > +	}
> >  	if (!primary_source)
> > -		primary_source = to_free = xstrfmt("%s/objects", repo->commondir);
> > +		primary_source = xstrfmt("%s/objects", repo->commondir);
> > +
> >  	o->sources = odb_source_new(o, primary_source, true);
> >  	o->sources_tail = &o->sources->next;
> >  	o->alternate_db = xstrdup_or_null(secondary_sources);
> 
> I'd say this xstrdup_or_null() is not needed no more, and so is the
> free() of that variable below.

True indeed.

Patrick

[PATCH v4 0/6] odb: make creation of object database pluggable

[Patrick Steinhardt]

Hi,

when creating a new repository we create a couple of on-disk data
structures for the object database. This includes the "objects/"
directory hierarchy with "objects/info" and "objects/pack", which are
specific to the backend.

This patch series makes the creation of the on-disk data structures
pluggable. While we continue to always create "objects/" regardless of
the backend (it's required for a repository to be recognized as such),
the other subdirectories are now created by the backend. This will allow
other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This
detour is required so that we can defer initialization of the object
database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).

Changes in v4:
  - Drop `APPLY_REPOSITOY_FORMAT_SKIP_ODB_CREATION` in favor of explicit
    calls to `odb_new()`.
  - Remove a useless call to `xstrdup()`.
  - Mark a string as translatable.
  - Link to v3: https://patch.msgid.link/20260805-pks-odb-create-on-disk-v3-0-c0ee3ac5141f@pks.im

Changes in v3:
  - Move handling of GIT_OBJECT_DIRECTORY and
    GIT_ALTERNATE_OBJECT_DIRECTORIES into `odb_new()` itself. This
    deduplicates some of the logic and also preps us for a future where
    alternates are handled in the "files" backend itself.
  - Link to v2: https://patch.msgid.link/20260804-pks-odb-create-on-disk-v2-0-ddf8b59bd207@pks.im

Changes in v2:
  - Add a testcase that demonstrates the bug fixed with alternate loose
    object maps.
  - Rename the "inmemory" bakcend to "in-memory".
  - Clarify some commit messages.
  - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (6):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: handle ODB-related environment variables in `odb_new()`
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable

 loose.c                       | 25 ++++++++++----------
 loose.h                       |  1 +
 odb.c                         | 21 +++++++++--------
 odb.h                         | 17 ++++++++++++--
 odb/source-files.c            | 19 +++++++++++++++
 odb/source-files.h            |  4 +++-
 odb/source-inmemory.h         |  4 +++-
 odb/source-loose.c            |  2 ++
 odb/source-loose.h            |  4 +++-
 odb/source-packed.h           |  4 +++-
 odb/source.c                  | 19 +++++++++++++++
 odb/source.h                  | 29 +++++++++++++++++++++++
 repository.c                  |  3 +--
 setup.c                       | 54 +++++++++++++++++++++----------------------
 t/t1016-compatObjectFormat.sh | 18 +++++++++++++++
 t/unit-tests/u-odb-inmemory.c |  2 +-
 16 files changed, 169 insertions(+), 57 deletions(-)

Range-diff versus v3:

1:  e1a585a3f7 = 1:  6dd8d575c6 loose: load loose object map for the correct source
2:  1f1200f7ba = 2:  1e7adada64 setup: detangle loading of loose object maps
3:  af02e520a2 ! 3:  2265f38695 setup: handle ODB-related environment variables in `odb_new()`
    @@ odb.c: int odb_write_object_stream(struct object_database *odb,
     +
      	o->sources = odb_source_new(o, primary_source, true);
      	o->sources_tail = &o->sources->next;
    - 	o->alternate_db = xstrdup_or_null(secondary_sources);
    +-	o->alternate_db = xstrdup_or_null(secondary_sources);
    ++	o->alternate_db = secondary_sources;
      	o->inmemory_objects = &odb_source_inmemory_new(o)->base;
      
     -	free(to_free);
     -
    -+	free(secondary_sources);
     +	free(primary_source);
      	return o;
      }
4:  2c794be101 ! 4:  5274ee6bab setup: defer object database creation
    @@ Commit message
         object database, but it's not usable yet until we eventually call
         `create_object_directory()`.
     
    -    Defer the object database creation so that we handle both steps in the
    -    same function.
    +    Lift the call to `odb_new()` out of `apply_repository_format()` so that
    +    callers have more wiggle room with when exactly they call it, and adapt
    +    them accordingly. The only exception is `init_db()`, where we now defer
    +    creating the object database until we call `create_object_database()`.
    +
    +    With this change, initializing and creating the object database on disk
    +    is now neatly encapsulated in a single function, which will make it
    +    easier for a subsequent commit to move creation of the on-disk data
    +    structures into the `struct odb_source` backends.
     
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
    + ## repository.c ##
    +@@ repository.c: int repo_init(struct repository *repo,
    + 		warning("%s", err.buf);
    + 		goto error;
    + 	}
    ++	repo->objects = odb_new(repo, 0);
    + 
    + 	if (worktree)
    + 		repo_set_worktree(repo, worktree);
    +
      ## setup.c ##
     @@ setup.c: int apply_repository_format(struct repository *repo,
      			    enum apply_repository_format_flags flags,
    @@ setup.c: int apply_repository_format(struct repository *repo,
     -	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
     -		odb_new_flags |= ODB_NEW_HONOR_ENV;
     -	repo->objects = odb_new(repo, odb_new_flags);
    -+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION)) {
    -+		enum odb_new_flags odb_new_flags = 0;
    -+		if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
    -+			odb_new_flags |= ODB_NEW_HONOR_ENV;
    -+		repo->objects = odb_new(repo, odb_new_flags);
    -+	}
    - 
    +-
      	return 0;
      }
    + 
    +@@ setup.c: const char *enter_repo(struct repository *repo, const char *path, unsigned flags
    + 		read_and_verify_repository_format(&fmt, ".", NULL);
    + 		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
    + 			die("%s", err.buf);
    ++		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
    + 		startup_info->have_repository = 1;
    + 
    + 		clear_repository_format(&fmt);
    +@@ setup.c: const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
    + 			if (apply_repository_format(repo, &discovery.format,
    + 						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
    + 				die("%s", err.buf);
    ++			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
    + 
    + 			clear_repository_format(&discovery.format);
    + 			strbuf_release(&err);
     @@ setup.c: static int create_default_files(struct repository *repo,
      	return reinit;
      }
    @@ setup.c: int init_db(struct repository *repo,
      	repository_format_configure(&repo_fmt, hash, ref_storage_format);
     -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
     +	if (apply_repository_format(repo, &repo_fmt,
    -+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
    -+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
    ++				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
      		die("%s", err.buf);
     -	startup_info->have_repository = 1;
      
    @@ setup.c: int init_db(struct repository *repo,
      
      	if (repo_settings_get_shared_repository(repo)) {
      		char buf[10];
    -
    - ## setup.h ##
    -@@ setup.h: enum apply_repository_format_flags {
    - 	 * relate to the object database.
    - 	 */
    - 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
    -+
    -+	/*
    -+	 * Usually, the object database is created after the repository format
    -+	 * was applied. This step is skipped if this flag is set, which leaves
    -+	 * us with a partially-working repository.
    -+	 *
    -+	 * This is useful when initializing a new repository.
    -+	 */
    -+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
    - };
    - 
    - /*
5:  7397c760df = 5:  b444a314a6 odb/source: introduce function to map source type to name
6:  7049e41a73 ! 6:  acb48f1072 odb: make creation of on-disk structures pluggable
    @@ setup.c: static int create_default_files(struct repository *repo,
     -
     -	strbuf_release(&path);
     +	if (odb_source_create_on_disk(repo->objects->sources) < 0)
    -+		die("failed creating object database");
    ++		die(_("failed creating object database"));
      }
      
      static void separate_git_dir(const char *git_dir, const char *git_link)

---
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

[PATCH v4 1/6] loose: load loose object map for the correct source

[Patrick Steinhardt]

When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source. The consequence is that we'll ignore loose object maps of
any configured alternates.

Fix this bug by instead loading the map via the loose source's path.

Helped-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c                       | 18 ++++++++++--------
 t/t1016-compatObjectFormat.sh | 18 ++++++++++++++++++
 2 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/loose.c b/loose.c
index bf01d3e42d..9dad75373b 100644
--- a/loose.c
+++ b/loose.c
@@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
+static int load_one_loose_object_map(struct odb_source_loose *loose)
 {
-	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+	struct repository *repo = loose->base.odb->repo;
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
 	FILE *fp;
 	int ret = -1;
 
@@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
 	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
 
-	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
-	fp = fopen(path.buf, "rb");
+	path = xstrfmt("%s/loose-object-idx", loose->base.path);
+	fp = fopen(path, "rb");
 	if (!fp) {
-		strbuf_release(&path);
+		free(path);
 		return 0;
 	}
 
@@ -102,7 +104,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 err:
 	fclose(fp);
 	strbuf_release(&buf);
-	strbuf_release(&path);
+	free(path);
 	return ret;
 }
 
@@ -117,10 +119,10 @@ int repo_read_loose_object_map(struct repository *repo)
 
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(repo, files->loose) < 0) {
+		if (load_one_loose_object_map(files->loose) < 0)
 			return -1;
-		}
 	}
+
 	return 0;
 }
 
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 92d48b96a1..9cafcee509 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -187,6 +187,24 @@ do
 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
 		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
 	'
+
+	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
+		for repo in alt borrow
+		do
+			test_when_finished "rm -rf $repo" &&
+			git init --object-format=$hash $repo &&
+			git -C $repo config set core.repositoryformatversion 1 &&
+			git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
+		done &&
+
+		git -C alt commit --allow-empty --message A &&
+		echo "$(pwd)/alt/.git/objects" >borrow/.git/objects/info/alternates &&
+
+		oid=$(git -C alt rev-parse HEAD) &&
+		git -C alt    rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
+		git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+		test_cmp expect actual
+	'
 done
 cd "$base"
 

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v4 2/6] setup: detangle loading of loose object maps

[Patrick Steinhardt]

When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.

Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:

  - We assume that the primary object database is the "files" object
    database and unconditionally downcast it. This will cause us to BUG
    in case a different object database type was used together with a
    compat hash algorithm.

  - We require the object database to already have been initialized when
    configuring the object database. This means that we must intermix
    configuration of the repository and initialization of its
    sub-structures in a weird way.

Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c            | 11 +++++------
 loose.h            |  1 +
 odb/source-loose.c |  2 ++
 repository.c       |  2 --
 setup.c            |  5 +++--
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/loose.c b/loose.c
index 9dad75373b..a3b2dcedc2 100644
--- a/loose.c
+++ b/loose.c
@@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct odb_source_loose *loose)
+int loose_object_map_load(struct odb_source_loose *loose)
 {
 	struct repository *repo = loose->base.odb->repo;
 	struct strbuf buf = STRBUF_INIT;
@@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
 	FILE *fp;
 	int ret = -1;
 
+	if (!should_use_loose_object_map(repo))
+		return 0;
+
 	if (!loose->map)
 		loose_object_map_init(&loose->map);
 	if (!loose->cache) {
@@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
 {
 	struct odb_source *source;
 
-	if (!should_use_loose_object_map(repo))
-		return 0;
-
 	odb_prepare_alternates(repo->objects);
-
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(files->loose) < 0)
+		if (loose_object_map_load(files->loose) < 0)
 			return -1;
 	}
 
diff --git a/loose.h b/loose.h
index 6c9b3f4571..ed663ac550 100644
--- a/loose.h
+++ b/loose.h
@@ -13,6 +13,7 @@ struct loose_object_map {
 
 void loose_object_map_init(struct loose_object_map **map);
 void loose_object_map_clear(struct loose_object_map **map);
+int loose_object_map_load(struct odb_source_loose *loose);
 int repo_loose_object_map_oid(struct repository *repo,
 			      const struct object_id *src,
 			      const struct git_hash_algo *dest_algo,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3f7d04a56e..812ca1c138 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 	if (!is_absolute_path(loose->base.path))
 		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
 
+	loose_object_map_load(loose);
+
 	return loose;
 }
diff --git a/repository.c b/repository.c
index 2ef0778846..6d633002b4 100644
--- a/repository.c
+++ b/repository.c
@@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
 	if (hash_algo_by_ptr(repo->hash_algo) == algo)
 		BUG("hash_algo and compat_hash_algo match");
 	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
-	if (repo->compat_hash_algo)
-		repo_read_loose_object_map(repo);
 #else
 	if (algo)
 		die(_("compatibility hash algorithm support requires Rust"));
diff --git a/setup.c b/setup.c
index d31808130b..825572f5f1 100644
--- a/setup.c
+++ b/setup.c
@@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
 
 	repo->bare_cfg = format->is_bare;
 	repo_set_hash_algo(repo, format->hash_algo);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
 	repo_set_ref_storage_format(repo,
 				    format->ref_storage_format,
@@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	free(alternate_object_directories);
 	free(object_directory);
 	return 0;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v4 3/6] setup: handle ODB-related environment variables in `odb_new()`

[Patrick Steinhardt]

When initializing a repository's object database we have to respect the
GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment
variables, which can be set by the user to override the default location
of where we write objects to and read objects from.

This is handled in `apply_repository_format()`, which is fine. But in a
subsequent commit we'll have to defer constructing the object database
to a later point in some cases, and that will require a second site
where we call `odb_new()`. And of course, that second site would have to
handle those environment variables, as well.

It would be somewhat awkward to duplicate the logic though. But there's
a better alternative: instead of handling this logic in "setup.c", we
can easily handle environment variables in `odb_new()` itself. This
ensures that object database creation is neatly self-contained, and we
don't have to duplicate any of the logic.

Another benefit is that in a future patch series we plan to move
handling of alternates into the backends themselves [1], and that will
require us to also handle those environment variables in the "files"
backend itself. So moving the logic into the ODB level already gets us
one step closer to that goal.

Refactor the logic accordingly.

[1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c                         | 21 ++++++++++++---------
 odb.h                         | 17 +++++++++++++++--
 setup.c                       | 11 ++++-------
 t/unit-tests/u-odb-inmemory.c |  2 +-
 4 files changed, 32 insertions(+), 19 deletions(-)

diff --git a/odb.c b/odb.c
index cf6e7938c0..ed1d63f4bd 100644
--- a/odb.c
+++ b/odb.c
@@ -1004,26 +1004,29 @@ int odb_write_object_stream(struct object_database *odb,
 }
 
 struct object_database *odb_new(struct repository *repo,
-				const char *primary_source,
-				const char *secondary_sources)
+				enum odb_new_flags flags)
 {
-	struct object_database *o = xmalloc(sizeof(*o));
-	char *to_free = NULL;
+	char *primary_source = NULL, *secondary_sources = NULL;
+	struct object_database *o;
 
-	memset(o, 0, sizeof(*o));
+	CALLOC_ARRAY(o, 1);
 	o->repo = repo;
 	pthread_mutex_init(&o->replace_mutex, NULL);
 	string_list_init_dup(&o->submodule_source_paths);
 
+	if (flags & ODB_NEW_HONOR_ENV) {
+		primary_source = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+		secondary_sources = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+	}
 	if (!primary_source)
-		primary_source = to_free = xstrfmt("%s/objects", repo->commondir);
+		primary_source = xstrfmt("%s/objects", repo->commondir);
+
 	o->sources = odb_source_new(o, primary_source, true);
 	o->sources_tail = &o->sources->next;
-	o->alternate_db = xstrdup_or_null(secondary_sources);
+	o->alternate_db = secondary_sources;
 	o->inmemory_objects = &odb_source_inmemory_new(o)->base;
 
-	free(to_free);
-
+	free(primary_source);
 	return o;
 }
 
diff --git a/odb.h b/odb.h
index 7995bed97b..8ec335c7f7 100644
--- a/odb.h
+++ b/odb.h
@@ -100,6 +100,20 @@ struct object_database {
 	struct string_list submodule_source_paths;
 };
 
+enum odb_new_flags {
+	/*
+	 * Honor environment variables when constructing the object database
+	 * sources. This makes us respect the following environment variables:
+	 *
+	 *   - GIT_OBJECT_DIRECTORY to override the primary object directory.
+	 *
+	 *   - GIT_ALTERNATE_OBJECT_DIRECTORIES to override alternates.
+	 *
+	 * Environment variables may be backend-specific.
+	 */
+	ODB_NEW_HONOR_ENV = (1 << 0),
+};
+
 /*
  * Create a new object database for the given repository.
  *
@@ -112,8 +126,7 @@ struct object_database {
  * Returns the newly created object database.
  */
 struct object_database *odb_new(struct repository *repo,
-				const char *primary_source,
-				const char *alternate_sources);
+				enum odb_new_flags flags);
 
 /* Free the object database and release all resources. */
 void odb_free(struct object_database *o);
diff --git a/setup.c b/setup.c
index 825572f5f1..5dfab3e79e 100644
--- a/setup.c
+++ b/setup.c
@@ -1765,7 +1765,7 @@ int apply_repository_format(struct repository *repo,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	char *object_directory = NULL, *alternate_object_directories = NULL;
+	enum odb_new_flags odb_new_flags = 0;
 
 	if (verify_repository_format(format, err) < 0)
 		return -1;
@@ -1779,8 +1779,6 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,11 +1801,10 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
+		odb_new_flags |= ODB_NEW_HONOR_ENV;
+	repo->objects = odb_new(repo, odb_new_flags);
 
-	free(alternate_object_directories);
-	free(object_directory);
 	return 0;
 }
 
diff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c
index 6844bfc37c..db323e10fd 100644
--- a/t/unit-tests/u-odb-inmemory.c
+++ b/t/unit-tests/u-odb-inmemory.c
@@ -38,7 +38,7 @@ static void cl_assert_object_info(struct odb_source_inmemory *source,
 
 void test_odb_inmemory__initialize(void)
 {
-	odb = odb_new(&repo, "", "");
+	odb = odb_new(&repo, 0);
 }
 
 void test_odb_inmemory__cleanup(void)

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v4 4/6] setup: defer object database creation

[Patrick Steinhardt]

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Lift the call to `odb_new()` out of `apply_repository_format()` so that
callers have more wiggle room with when exactly they call it, and adapt
them accordingly. The only exception is `init_db()`, where we now defer
creating the object database until we call `create_object_database()`.

With this change, initializing and creating the object database on disk
is now neatly encapsulated in a single function, which will make it
easier for a subsequent commit to move creation of the on-disk data
structures into the `struct odb_source` backends.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 repository.c |  1 +
 setup.c      | 20 ++++++++++----------
 2 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/repository.c b/repository.c
index 6d633002b4..5ec264e607 100644
--- a/repository.c
+++ b/repository.c
@@ -294,6 +294,7 @@ int repo_init(struct repository *repo,
 		warning("%s", err.buf);
 		goto error;
 	}
+	repo->objects = odb_new(repo, 0);
 
 	if (worktree)
 		repo_set_worktree(repo, worktree);
diff --git a/setup.c b/setup.c
index 5dfab3e79e..e39a1646bb 100644
--- a/setup.c
+++ b/setup.c
@@ -1765,8 +1765,6 @@ int apply_repository_format(struct repository *repo,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	enum odb_new_flags odb_new_flags = 0;
-
 	if (verify_repository_format(format, err) < 0)
 		return -1;
 
@@ -1801,10 +1799,6 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
-		odb_new_flags |= ODB_NEW_HONOR_ENV;
-	repo->objects = odb_new(repo, odb_new_flags);
-
 	return 0;
 }
 
@@ -1888,6 +1882,7 @@ const char *enter_repo(struct repository *repo, const char *path, unsigned flags
 		read_and_verify_repository_format(&fmt, ".", NULL);
 		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 			die("%s", err.buf);
+		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 		startup_info->have_repository = 1;
 
 		clear_repository_format(&fmt);
@@ -2090,6 +2085,7 @@ const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
 			if (apply_repository_format(repo, &discovery.format,
 						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 				die("%s", err.buf);
+			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 
 			clear_repository_format(&discovery.format);
 			strbuf_release(&err);
@@ -2651,11 +2647,13 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2864,9 +2862,9 @@ int init_db(struct repository *repo,
 	 */
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2882,7 +2880,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v4 5/6] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++++++
 odb/source.h          |  6 ++++++
 6 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/odb/source-files.h b/odb/source-files.h
index d7ac3c1c81..6a803afdda 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -28,7 +28,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_FILES)
-		BUG("trying to downcast source of type '%d' to files", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_FILES));
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source-inmemory.h b/odb/source-inmemory.h
index a88fc2e320..adbad23e8b 100644
--- a/odb/source-inmemory.h
+++ b/odb/source-inmemory.h
@@ -26,7 +26,9 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)
 static inline struct odb_source_inmemory *odb_source_inmemory_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_INMEMORY)
-		BUG("trying to downcast source of type '%d' to in-memory", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_INMEMORY));
 	return container_of(source, struct odb_source_inmemory, base);
 }
 
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 6070aaf3ce..3cf2e1f8f1 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -41,7 +41,9 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_LOOSE)
-		BUG("trying to downcast source of type '%d' to loose", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_LOOSE));
 	return container_of(source, struct odb_source_loose, base);
 }
 
diff --git a/odb/source-packed.h b/odb/source-packed.h
index 77309ddd09..a0f6b5096d 100644
--- a/odb/source-packed.h
+++ b/odb/source-packed.h
@@ -78,7 +78,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
 static inline struct odb_source_packed *odb_source_packed_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_PACKED)
-		BUG("trying to downcast source of type '%d' to packed", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_PACKED));
 	return container_of(source, struct odb_source_packed, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index 7993dcbd65..30188b806d 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -4,6 +4,25 @@
 #include "odb/source.h"
 #include "packfile.h"
 
+static const char * const odb_source_names_by_type[] = {
+	[ODB_SOURCE_UNKNOWN] = "unknown",
+	[ODB_SOURCE_FILES] = "files",
+	[ODB_SOURCE_LOOSE] = "loose",
+	[ODB_SOURCE_PACKED] = "packed",
+	[ODB_SOURCE_INMEMORY] = "in-memory",
+};
+
+const char *odb_source_type_to_name(enum odb_source_type type)
+{
+	const char *name;
+	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
+		type = ODB_SOURCE_UNKNOWN;
+	name = odb_source_names_by_type[type];
+	if (!name)
+		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
+	return name;
+}
+
 struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
diff --git a/odb/source.h b/odb/source.h
index cd63dba91f..ab16d152f4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -25,6 +25,12 @@ enum odb_source_type {
 	ODB_SOURCE_INMEMORY,
 };
 
+/*
+ * Convert between the enum and its name. Returns the equivalent of "unknown"
+ * for unknown types.
+ */
+const char *odb_source_type_to_name(enum odb_source_type type);
+
 struct object_id;
 struct odb_read_stream;
 struct strvec;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v4 6/6] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.

Make the creation of on-disk structures pluggable to accommodate for
this.

Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 19 +++++++++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 setup.c            | 34 ++++++++++++++++++----------------
 3 files changed, 60 insertions(+), 16 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 4138758511..0db6e681fe 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -9,6 +9,7 @@
 #include "odb/source-files.h"
 #include "odb/source-loose.h"
 #include "packfile.h"
+#include "path.h"
 #include "strbuf.h"
 #include "write-or-die.h"
 
@@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
 	odb_source_close(&files->packed->base);
 }
 
+static int odb_source_files_create_on_disk(struct odb_source *source)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	safe_create_dir(source->odb->repo, source->path, 1);
+
+	strbuf_addf(&path, "%s/pack", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/info", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_release(&path);
+	return 0;
+}
+
 static void odb_source_files_prepare(struct odb_source *source,
 				     enum odb_prepare_flags flags)
 {
@@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
+	files->base.create_on_disk = odb_source_files_create_on_disk;
 	files->base.prepare = odb_source_files_prepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
diff --git a/odb/source.h b/odb/source.h
index ab16d152f4..4abc418bdd 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -89,6 +89,18 @@ struct odb_source {
 	 */
 	void (*close)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to create on-disk data structures that are
+	 * required for this source to operate.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 *
+	 * This callback may be NULL in case the source does not need any
+	 * on-disk setup.
+	 */
+	int (*create_on_disk)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to prepare the source so that it becomes
 	 * ready for use. It optionally clears underlying caches of the object
@@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
 	source->close(source);
 }
 
+/*
+ * Create on-disk data structures that are required for this source to operate
+ * correctly. Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_create_on_disk(struct odb_source *source)
+{
+	if (!source->create_on_disk)
+		return 0;
+	return source->create_on_disk(source);
+}
+
 /*
  * Prepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects
diff --git a/setup.c b/setup.c
index e39a1646bb..1f65f69534 100644
--- a/setup.c
+++ b/setup.c
@@ -2649,25 +2649,27 @@ static int create_default_files(struct repository *repo,
 
 static void create_object_database(struct repository *repo)
 {
-	struct strbuf path = STRBUF_INIT;
-	size_t baselen;
+	/*
+	 * Create the "objects" directory in the common directory. This is done
+	 * so that the repository can be discovered regardless of the backend
+	 * used.
+	 *
+	 * Note that we only do this in case the object directory wasn't
+	 * overwritten via an environment variable. If it _is_ being overridden
+	 * then we skip this step, as the repository won't be discoverable
+	 * anyway without the environment variable.
+	 */
+	if (!getenv(DB_ENVIRONMENT)) {
+		struct strbuf objects_dir = STRBUF_INIT;
+		repo_common_path_append(repo, &objects_dir, "objects");
+		safe_create_dir(repo, objects_dir.buf, 1);
+		strbuf_release(&objects_dir);
+	}
 
 	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 
-	strbuf_addstr(&path, repo_get_object_directory(repo));
-	baselen = path.len;
-
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/pack");
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/info");
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_release(&path);
+	if (odb_source_create_on_disk(repo->objects->sources) < 0)
+		die(_("failed creating object database"));
 }
 
 static void separate_git_dir(const char *git_dir, const char *git_link)

-- 
2.55.0.679.g6767b8d81c.dirty

Re: [PATCH v4 4/6] setup: defer object database creation

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> In a subsequent commit we'll make the creation of the on-disk data
> structures of an object database pluggable. This will lead to an
> in-between state where we have already configured the repository's
> object database, but it's not usable yet until we eventually call
> `create_object_directory()`.
>
> Lift the call to `odb_new()` out of `apply_repository_format()` so that
> callers have more wiggle room with when exactly they call it, and adapt
> them accordingly. The only exception is `init_db()`, where we now defer
> creating the object database until we call `create_object_database()`.
>
> With this change, initializing and creating the object database on disk
> is now neatly encapsulated in a single function, which will make it
> easier for a subsequent commit to move creation of the on-disk data
> structures into the `struct odb_source` backends.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  repository.c |  1 +
>  setup.c      | 20 ++++++++++----------
>  2 files changed, 11 insertions(+), 10 deletions(-)
>
> diff --git a/repository.c b/repository.c
> index 6d633002b4..5ec264e607 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -294,6 +294,7 @@ int repo_init(struct repository *repo,
>  		warning("%s", err.buf);
>  		goto error;
>  	}
> +	repo->objects = odb_new(repo, 0);
>  
>  	if (worktree)
>  		repo_set_worktree(repo, worktree);
> diff --git a/setup.c b/setup.c
> index 5dfab3e79e..e39a1646bb 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1765,8 +1765,6 @@ int apply_repository_format(struct repository *repo,
>  			    enum apply_repository_format_flags flags,
>  			    struct strbuf *err)

I've noticed the docs in setup.h say:

    /*
     * Apply the given repository format to the repo. This initializes extensions
     * and basic data structures required for normal operation. Returns 0 on
     * success, a negative error code when the format is not valid as determined by
     * `verify_repository_format()`.
     */

I'm not sure that's still applicable, now odb_new() isn't called no
more.

>  {
> -	enum odb_new_flags odb_new_flags = 0;
> -
>  	if (verify_repository_format(format, err) < 0)
>  		return -1;
>  
> @@ -1801,10 +1799,6 @@ int apply_repository_format(struct repository *repo,
>  	repo->repository_format_precious_objects =
>  		format->precious_objects;
>  
> -	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
> -		odb_new_flags |= ODB_NEW_HONOR_ENV;
> -	repo->objects = odb_new(repo, odb_new_flags);
> -
>  	return 0;
>  }
>  
> @@ -1888,6 +1882,7 @@ const char *enter_repo(struct repository *repo, const char *path, unsigned flags
>  		read_and_verify_repository_format(&fmt, ".", NULL);
>  		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
>  			die("%s", err.buf);
> +		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
>  		startup_info->have_repository = 1;
>  
>  		clear_repository_format(&fmt);
> @@ -2090,6 +2085,7 @@ const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
>  			if (apply_repository_format(repo, &discovery.format,
>  						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
>  				die("%s", err.buf);
> +			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
>  
>  			clear_repository_format(&discovery.format);
>  			strbuf_release(&err);
> @@ -2651,11 +2647,13 @@ static int create_default_files(struct repository *repo,
>  	return reinit;
>  }
>  
> -static void create_object_directory(struct repository *repo)
> +static void create_object_database(struct repository *repo)
>  {
>  	struct strbuf path = STRBUF_INIT;
>  	size_t baselen;
>  
> +	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
> +
>  	strbuf_addstr(&path, repo_get_object_directory(repo));
>  	baselen = path.len;
>  
> @@ -2864,9 +2862,9 @@ int init_db(struct repository *repo,
>  	 */
>  	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
>  	repository_format_configure(&repo_fmt, hash, ref_storage_format);
> -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
> +	if (apply_repository_format(repo, &repo_fmt,
> +				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)

Nit: Not sure why this formatting change was needed. I would have
assumed to have all apply_repository_format() calls formatted the same,
but I've noticed at line 1883 in enter_repo() it's still a single-line
call.

>  		die("%s", err.buf);
> -	startup_info->have_repository = 1;
>  
>  	/*
>  	 * Ensure `core.hidedotfiles` is processed. This must happen after we
> @@ -2882,7 +2880,9 @@ int init_db(struct repository *repo,
>  
>  	if (!(flags & INIT_DB_SKIP_REFDB))
>  		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
> -	create_object_directory(repo);
> +	create_object_database(repo);
> +
> +	startup_info->have_repository = 1;
>  
>  	if (repo_settings_get_shared_repository(repo)) {
>  		char buf[10];
>
> -- 
> 2.55.0.679.g6767b8d81c.dirty
>

-- 
Cheers,
Toon

Re: [PATCH v4 0/6] odb: make creation of object database pluggable

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> Hi,
>
> when creating a new repository we create a couple of on-disk data
> structures for the object database. This includes the "objects/"
> directory hierarchy with "objects/info" and "objects/pack", which are
> specific to the backend.
>
> This patch series makes the creation of the on-disk data structures
> pluggable. While we continue to always create "objects/" regardless of
> the backend (it's required for a repository to be recognized as such),
> the other subdirectories are now created by the backend. This will allow
> other backends to plug in their own logic.
>
> The series starts with a small detour into the loose-object map. This
> detour is required so that we can defer initialization of the object
> database itself to a later point in time.
>
> The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).
>
> Changes in v4:
>   - Drop `APPLY_REPOSITOY_FORMAT_SKIP_ODB_CREATION` in favor of explicit
>     calls to `odb_new()`.
>   - Remove a useless call to `xstrdup()`.
>   - Mark a string as translatable.
>   - Link to v3: https://patch.msgid.link/20260805-pks-odb-create-on-disk-v3-0-c0ee3ac5141f@pks.im

Structurally I'm very happy about this version. Only had some nits about
comments and formatting, but overall this version looks good to me.

Thanks!

-- 
Cheers,
Toon

Re: [PATCH v4 4/6] setup: defer object database creation

[Patrick Steinhardt]

On Thu, Aug 06, 2026 at 04:23:17PM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/setup.c b/setup.c
> > index 5dfab3e79e..e39a1646bb 100644
> > --- a/setup.c
> > +++ b/setup.c
> > @@ -1765,8 +1765,6 @@ int apply_repository_format(struct repository *repo,
> >  			    enum apply_repository_format_flags flags,
> >  			    struct strbuf *err)
> 
> I've noticed the docs in setup.h say:
> 
>     /*
>      * Apply the given repository format to the repo. This initializes extensions
>      * and basic data structures required for normal operation. Returns 0 on
>      * success, a negative error code when the format is not valid as determined by
>      * `verify_repository_format()`.
>      */
> 
> I'm not sure that's still applicable, now odb_new() isn't called no
> more.

Fair enough.

> > @@ -2864,9 +2862,9 @@ int init_db(struct repository *repo,
> >  	 */
> >  	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
> >  	repository_format_configure(&repo_fmt, hash, ref_storage_format);
> > -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
> > +	if (apply_repository_format(repo, &repo_fmt,
> > +				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
> 
> Nit: Not sure why this formatting change was needed. I would have
> assumed to have all apply_repository_format() calls formatted the same,
> but I've noticed at line 1883 in enter_repo() it's still a single-line
> call.

It's an artifact from previous versions.

I'll send a (hopefully last) reroll in a bit. Thanks!

Patrick

Re: [PATCH v4 4/6] setup: defer object database creation

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> It's an artifact from previous versions.
>
> I'll send a (hopefully last) reroll in a bit. Thanks!
>
> Patrick

With Toon's <87qzkb495s.fsf@emacs.iotcl.com> and this message, I'll
mark the topic as "Expecting a (hopefully small and final) reroll."

Thanks.

[PATCH v5 0/6] odb: make creation of object database pluggable

[Patrick Steinhardt]

Hi,

when creating a new repository we create a couple of on-disk data
structures for the object database. This includes the "objects/"
directory hierarchy with "objects/info" and "objects/pack", which are
specific to the backend.

This patch series makes the creation of the on-disk data structures
pluggable. While we continue to always create "objects/" regardless of
the backend (it's required for a repository to be recognized as such),
the other subdirectories are now created by the backend. This will allow
other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This
detour is required so that we can defer initialization of the object
database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).

Changes in v5:
  - Remove a leftover formatting change.
  - Fix a stale comment.
  - Link to v4: https://patch.msgid.link/20260806-pks-odb-create-on-disk-v4-0-ba8b4fdd2e3c@pks.im

Changes in v4:
  - Drop `APPLY_REPOSITOY_FORMAT_SKIP_ODB_CREATION` in favor of explicit
    calls to `odb_new()`.
  - Remove a useless call to `xstrdup()`.
  - Mark a string as translatable.
  - Link to v3: https://patch.msgid.link/20260805-pks-odb-create-on-disk-v3-0-c0ee3ac5141f@pks.im

Changes in v3:
  - Move handling of GIT_OBJECT_DIRECTORY and
    GIT_ALTERNATE_OBJECT_DIRECTORIES into `odb_new()` itself. This
    deduplicates some of the logic and also preps us for a future where
    alternates are handled in the "files" backend itself.
  - Link to v2: https://patch.msgid.link/20260804-pks-odb-create-on-disk-v2-0-ddf8b59bd207@pks.im

Changes in v2:
  - Add a testcase that demonstrates the bug fixed with alternate loose
    object maps.
  - Rename the "inmemory" bakcend to "in-memory".
  - Clarify some commit messages.
  - Link to v1: https://patch.msgid.link/20260724-pks-odb-create-on-disk-v1-0-3b3d265d979b@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (6):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: handle ODB-related environment variables in `odb_new()`
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable

 loose.c                       | 25 +++++++++++----------
 loose.h                       |  1 +
 odb.c                         | 21 ++++++++++--------
 odb.h                         | 17 +++++++++++++--
 odb/source-files.c            | 19 ++++++++++++++++
 odb/source-files.h            |  4 +++-
 odb/source-inmemory.h         |  4 +++-
 odb/source-loose.c            |  2 ++
 odb/source-loose.h            |  4 +++-
 odb/source-packed.h           |  4 +++-
 odb/source.c                  | 19 ++++++++++++++++
 odb/source.h                  | 29 ++++++++++++++++++++++++
 repository.c                  |  3 +--
 setup.c                       | 51 +++++++++++++++++++++----------------------
 setup.h                       |  4 ++--
 t/t1016-compatObjectFormat.sh | 18 +++++++++++++++
 t/unit-tests/u-odb-inmemory.c |  2 +-
 17 files changed, 169 insertions(+), 58 deletions(-)

Range-diff versus v4:

1:  40ca0d1345 = 1:  3a0fbf9498 loose: load loose object map for the correct source
2:  d18ddec5dd = 2:  7ba250f4d7 setup: detangle loading of loose object maps
3:  9b6fbc510f = 3:  fbe755388b setup: handle ODB-related environment variables in `odb_new()`
4:  f27f8d45a4 ! 4:  4d7a12e3cb setup: defer object database creation
    @@ setup.c: static int create_default_files(struct repository *repo,
      	baselen = path.len;
      
     @@ setup.c: int init_db(struct repository *repo,
    - 	 */
    - 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
      	repository_format_configure(&repo_fmt, hash, ref_storage_format);
    --	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
    -+	if (apply_repository_format(repo, &repo_fmt,
    -+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
    + 	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
      		die("%s", err.buf);
     -	startup_info->have_repository = 1;
      
    @@ setup.c: int init_db(struct repository *repo,
      
      	if (repo_settings_get_shared_repository(repo)) {
      		char buf[10];
    +
    + ## setup.h ##
    +@@ setup.h: enum apply_repository_format_flags {
    + 
    + /*
    +  * Apply the given repository format to the repo. This initializes extensions
    +- * and basic data structures required for normal operation. Returns 0 on
    +- * success, a negative error code when the format is not valid as determined by
    ++ * required for normal operation. Returns 0 on success, a negative error code
    ++ * when the format is not valid as determined by
    +  * `verify_repository_format()`.
    +  */
    + int apply_repository_format(struct repository *repo,
5:  1c0afb893f = 5:  6bb4ecc76d odb/source: introduce function to map source type to name
6:  387fe6e204 = 6:  806f399c63 odb: make creation of on-disk structures pluggable

---
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

[PATCH v5 1/6] loose: load loose object map for the correct source

[Patrick Steinhardt]

When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source. The consequence is that we'll ignore loose object maps of
any configured alternates.

Fix this bug by instead loading the map via the loose source's path.

Helped-by: Toon Claes <toon@iotcl.com>
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c                       | 18 ++++++++++--------
 t/t1016-compatObjectFormat.sh | 18 ++++++++++++++++++
 2 files changed, 28 insertions(+), 8 deletions(-)

diff --git a/loose.c b/loose.c
index bf01d3e42d..9dad75373b 100644
--- a/loose.c
+++ b/loose.c
@@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
+static int load_one_loose_object_map(struct odb_source_loose *loose)
 {
-	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+	struct repository *repo = loose->base.odb->repo;
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
 	FILE *fp;
 	int ret = -1;
 
@@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
 	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
 
-	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
-	fp = fopen(path.buf, "rb");
+	path = xstrfmt("%s/loose-object-idx", loose->base.path);
+	fp = fopen(path, "rb");
 	if (!fp) {
-		strbuf_release(&path);
+		free(path);
 		return 0;
 	}
 
@@ -102,7 +104,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 err:
 	fclose(fp);
 	strbuf_release(&buf);
-	strbuf_release(&path);
+	free(path);
 	return ret;
 }
 
@@ -117,10 +119,10 @@ int repo_read_loose_object_map(struct repository *repo)
 
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(repo, files->loose) < 0) {
+		if (load_one_loose_object_map(files->loose) < 0)
 			return -1;
-		}
 	}
+
 	return 0;
 }
 
diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh
index 92d48b96a1..9cafcee509 100755
--- a/t/t1016-compatObjectFormat.sh
+++ b/t/t1016-compatObjectFormat.sh
@@ -187,6 +187,24 @@ do
 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
 		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
 	'
+
+	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
+		for repo in alt borrow
+		do
+			test_when_finished "rm -rf $repo" &&
+			git init --object-format=$hash $repo &&
+			git -C $repo config set core.repositoryformatversion 1 &&
+			git -C $repo config set extensions.compatObjectFormat $(compat_hash $hash) || exit 1
+		done &&
+
+		git -C alt commit --allow-empty --message A &&
+		echo "$(pwd)/alt/.git/objects" >borrow/.git/objects/info/alternates &&
+
+		oid=$(git -C alt rev-parse HEAD) &&
+		git -C alt    rev-parse --output-object-format=$(compat_hash $hash) "$oid" >expect &&
+		git -C borrow rev-parse --output-object-format=$(compat_hash $hash) "$oid" >actual &&
+		test_cmp expect actual
+	'
 done
 cd "$base"
 

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v5 2/6] setup: detangle loading of loose object maps

[Patrick Steinhardt]

When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.

Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:

  - We assume that the primary object database is the "files" object
    database and unconditionally downcast it. This will cause us to BUG
    in case a different object database type was used together with a
    compat hash algorithm.

  - We require the object database to already have been initialized when
    configuring the object database. This means that we must intermix
    configuration of the repository and initialization of its
    sub-structures in a weird way.

Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c            | 11 +++++------
 loose.h            |  1 +
 odb/source-loose.c |  2 ++
 repository.c       |  2 --
 setup.c            |  5 +++--
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/loose.c b/loose.c
index 9dad75373b..a3b2dcedc2 100644
--- a/loose.c
+++ b/loose.c
@@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct odb_source_loose *loose)
+int loose_object_map_load(struct odb_source_loose *loose)
 {
 	struct repository *repo = loose->base.odb->repo;
 	struct strbuf buf = STRBUF_INIT;
@@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
 	FILE *fp;
 	int ret = -1;
 
+	if (!should_use_loose_object_map(repo))
+		return 0;
+
 	if (!loose->map)
 		loose_object_map_init(&loose->map);
 	if (!loose->cache) {
@@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
 {
 	struct odb_source *source;
 
-	if (!should_use_loose_object_map(repo))
-		return 0;
-
 	odb_prepare_alternates(repo->objects);
-
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(files->loose) < 0)
+		if (loose_object_map_load(files->loose) < 0)
 			return -1;
 	}
 
diff --git a/loose.h b/loose.h
index 6c9b3f4571..ed663ac550 100644
--- a/loose.h
+++ b/loose.h
@@ -13,6 +13,7 @@ struct loose_object_map {
 
 void loose_object_map_init(struct loose_object_map **map);
 void loose_object_map_clear(struct loose_object_map **map);
+int loose_object_map_load(struct odb_source_loose *loose);
 int repo_loose_object_map_oid(struct repository *repo,
 			      const struct object_id *src,
 			      const struct git_hash_algo *dest_algo,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3f7d04a56e..812ca1c138 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 	if (!is_absolute_path(loose->base.path))
 		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
 
+	loose_object_map_load(loose);
+
 	return loose;
 }
diff --git a/repository.c b/repository.c
index 2ef0778846..6d633002b4 100644
--- a/repository.c
+++ b/repository.c
@@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
 	if (hash_algo_by_ptr(repo->hash_algo) == algo)
 		BUG("hash_algo and compat_hash_algo match");
 	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
-	if (repo->compat_hash_algo)
-		repo_read_loose_object_map(repo);
 #else
 	if (algo)
 		die(_("compatibility hash algorithm support requires Rust"));
diff --git a/setup.c b/setup.c
index d31808130b..825572f5f1 100644
--- a/setup.c
+++ b/setup.c
@@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
 
 	repo->bare_cfg = format->is_bare;
 	repo_set_hash_algo(repo, format->hash_algo);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
 	repo_set_ref_storage_format(repo,
 				    format->ref_storage_format,
@@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	free(alternate_object_directories);
 	free(object_directory);
 	return 0;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v5 3/6] setup: handle ODB-related environment variables in `odb_new()`

[Patrick Steinhardt]

When initializing a repository's object database we have to respect the
GIT_OBJECT_DIRECTORY and GIT_ALTERNATE_OBJECT_DIRECTORIES environment
variables, which can be set by the user to override the default location
of where we write objects to and read objects from.

This is handled in `apply_repository_format()`, which is fine. But in a
subsequent commit we'll have to defer constructing the object database
to a later point in some cases, and that will require a second site
where we call `odb_new()`. And of course, that second site would have to
handle those environment variables, as well.

It would be somewhat awkward to duplicate the logic though. But there's
a better alternative: instead of handling this logic in "setup.c", we
can easily handle environment variables in `odb_new()` itself. This
ensures that object database creation is neatly self-contained, and we
don't have to duplicate any of the logic.

Another benefit is that in a future patch series we plan to move
handling of alternates into the backends themselves [1], and that will
require us to also handle those environment variables in the "files"
backend itself. So moving the logic into the ODB level already gets us
one step closer to that goal.

Refactor the logic accordingly.

[1]: https://lore.kernel.org/git/amLgMqkqxR8mKIbT@pks.im/

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c                         | 21 ++++++++++++---------
 odb.h                         | 17 +++++++++++++++--
 setup.c                       | 11 ++++-------
 t/unit-tests/u-odb-inmemory.c |  2 +-
 4 files changed, 32 insertions(+), 19 deletions(-)

diff --git a/odb.c b/odb.c
index cf6e7938c0..ed1d63f4bd 100644
--- a/odb.c
+++ b/odb.c
@@ -1004,26 +1004,29 @@ int odb_write_object_stream(struct object_database *odb,
 }
 
 struct object_database *odb_new(struct repository *repo,
-				const char *primary_source,
-				const char *secondary_sources)
+				enum odb_new_flags flags)
 {
-	struct object_database *o = xmalloc(sizeof(*o));
-	char *to_free = NULL;
+	char *primary_source = NULL, *secondary_sources = NULL;
+	struct object_database *o;
 
-	memset(o, 0, sizeof(*o));
+	CALLOC_ARRAY(o, 1);
 	o->repo = repo;
 	pthread_mutex_init(&o->replace_mutex, NULL);
 	string_list_init_dup(&o->submodule_source_paths);
 
+	if (flags & ODB_NEW_HONOR_ENV) {
+		primary_source = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+		secondary_sources = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+	}
 	if (!primary_source)
-		primary_source = to_free = xstrfmt("%s/objects", repo->commondir);
+		primary_source = xstrfmt("%s/objects", repo->commondir);
+
 	o->sources = odb_source_new(o, primary_source, true);
 	o->sources_tail = &o->sources->next;
-	o->alternate_db = xstrdup_or_null(secondary_sources);
+	o->alternate_db = secondary_sources;
 	o->inmemory_objects = &odb_source_inmemory_new(o)->base;
 
-	free(to_free);
-
+	free(primary_source);
 	return o;
 }
 
diff --git a/odb.h b/odb.h
index 7995bed97b..8ec335c7f7 100644
--- a/odb.h
+++ b/odb.h
@@ -100,6 +100,20 @@ struct object_database {
 	struct string_list submodule_source_paths;
 };
 
+enum odb_new_flags {
+	/*
+	 * Honor environment variables when constructing the object database
+	 * sources. This makes us respect the following environment variables:
+	 *
+	 *   - GIT_OBJECT_DIRECTORY to override the primary object directory.
+	 *
+	 *   - GIT_ALTERNATE_OBJECT_DIRECTORIES to override alternates.
+	 *
+	 * Environment variables may be backend-specific.
+	 */
+	ODB_NEW_HONOR_ENV = (1 << 0),
+};
+
 /*
  * Create a new object database for the given repository.
  *
@@ -112,8 +126,7 @@ struct object_database {
  * Returns the newly created object database.
  */
 struct object_database *odb_new(struct repository *repo,
-				const char *primary_source,
-				const char *alternate_sources);
+				enum odb_new_flags flags);
 
 /* Free the object database and release all resources. */
 void odb_free(struct object_database *o);
diff --git a/setup.c b/setup.c
index 825572f5f1..5dfab3e79e 100644
--- a/setup.c
+++ b/setup.c
@@ -1765,7 +1765,7 @@ int apply_repository_format(struct repository *repo,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	char *object_directory = NULL, *alternate_object_directories = NULL;
+	enum odb_new_flags odb_new_flags = 0;
 
 	if (verify_repository_format(format, err) < 0)
 		return -1;
@@ -1779,8 +1779,6 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,11 +1801,10 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
+		odb_new_flags |= ODB_NEW_HONOR_ENV;
+	repo->objects = odb_new(repo, odb_new_flags);
 
-	free(alternate_object_directories);
-	free(object_directory);
 	return 0;
 }
 
diff --git a/t/unit-tests/u-odb-inmemory.c b/t/unit-tests/u-odb-inmemory.c
index 6844bfc37c..db323e10fd 100644
--- a/t/unit-tests/u-odb-inmemory.c
+++ b/t/unit-tests/u-odb-inmemory.c
@@ -38,7 +38,7 @@ static void cl_assert_object_info(struct odb_source_inmemory *source,
 
 void test_odb_inmemory__initialize(void)
 {
-	odb = odb_new(&repo, "", "");
+	odb = odb_new(&repo, 0);
 }
 
 void test_odb_inmemory__cleanup(void)

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v5 4/6] setup: defer object database creation

[Patrick Steinhardt]

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Lift the call to `odb_new()` out of `apply_repository_format()` so that
callers have more wiggle room with when exactly they call it, and adapt
them accordingly. The only exception is `init_db()`, where we now defer
creating the object database until we call `create_object_database()`.

With this change, initializing and creating the object database on disk
is now neatly encapsulated in a single function, which will make it
easier for a subsequent commit to move creation of the on-disk data
structures into the `struct odb_source` backends.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 repository.c |  1 +
 setup.c      | 17 ++++++++---------
 setup.h      |  4 ++--
 3 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/repository.c b/repository.c
index 6d633002b4..5ec264e607 100644
--- a/repository.c
+++ b/repository.c
@@ -294,6 +294,7 @@ int repo_init(struct repository *repo,
 		warning("%s", err.buf);
 		goto error;
 	}
+	repo->objects = odb_new(repo, 0);
 
 	if (worktree)
 		repo_set_worktree(repo, worktree);
diff --git a/setup.c b/setup.c
index 5dfab3e79e..97338cbc51 100644
--- a/setup.c
+++ b/setup.c
@@ -1765,8 +1765,6 @@ int apply_repository_format(struct repository *repo,
 			    enum apply_repository_format_flags flags,
 			    struct strbuf *err)
 {
-	enum odb_new_flags odb_new_flags = 0;
-
 	if (verify_repository_format(format, err) < 0)
 		return -1;
 
@@ -1801,10 +1799,6 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV)
-		odb_new_flags |= ODB_NEW_HONOR_ENV;
-	repo->objects = odb_new(repo, odb_new_flags);
-
 	return 0;
 }
 
@@ -1888,6 +1882,7 @@ const char *enter_repo(struct repository *repo, const char *path, unsigned flags
 		read_and_verify_repository_format(&fmt, ".", NULL);
 		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 			die("%s", err.buf);
+		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 		startup_info->have_repository = 1;
 
 		clear_repository_format(&fmt);
@@ -2090,6 +2085,7 @@ const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
 			if (apply_repository_format(repo, &discovery.format,
 						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 				die("%s", err.buf);
+			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 
 			clear_repository_format(&discovery.format);
 			strbuf_release(&err);
@@ -2651,11 +2647,13 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2866,7 +2864,6 @@ int init_db(struct repository *repo,
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
 	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2882,7 +2879,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];
diff --git a/setup.h b/setup.h
index 654f10e059..763fd384e8 100644
--- a/setup.h
+++ b/setup.h
@@ -245,8 +245,8 @@ enum apply_repository_format_flags {
 
 /*
  * Apply the given repository format to the repo. This initializes extensions
- * and basic data structures required for normal operation. Returns 0 on
- * success, a negative error code when the format is not valid as determined by
+ * required for normal operation. Returns 0 on success, a negative error code
+ * when the format is not valid as determined by
  * `verify_repository_format()`.
  */
 int apply_repository_format(struct repository *repo,

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v5 5/6] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++++++
 odb/source.h          |  6 ++++++
 6 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/odb/source-files.h b/odb/source-files.h
index d7ac3c1c81..6a803afdda 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -28,7 +28,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_FILES)
-		BUG("trying to downcast source of type '%d' to files", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_FILES));
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source-inmemory.h b/odb/source-inmemory.h
index a88fc2e320..adbad23e8b 100644
--- a/odb/source-inmemory.h
+++ b/odb/source-inmemory.h
@@ -26,7 +26,9 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)
 static inline struct odb_source_inmemory *odb_source_inmemory_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_INMEMORY)
-		BUG("trying to downcast source of type '%d' to in-memory", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_INMEMORY));
 	return container_of(source, struct odb_source_inmemory, base);
 }
 
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 6070aaf3ce..3cf2e1f8f1 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -41,7 +41,9 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_LOOSE)
-		BUG("trying to downcast source of type '%d' to loose", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_LOOSE));
 	return container_of(source, struct odb_source_loose, base);
 }
 
diff --git a/odb/source-packed.h b/odb/source-packed.h
index 77309ddd09..a0f6b5096d 100644
--- a/odb/source-packed.h
+++ b/odb/source-packed.h
@@ -78,7 +78,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
 static inline struct odb_source_packed *odb_source_packed_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_PACKED)
-		BUG("trying to downcast source of type '%d' to packed", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_PACKED));
 	return container_of(source, struct odb_source_packed, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index 7993dcbd65..30188b806d 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -4,6 +4,25 @@
 #include "odb/source.h"
 #include "packfile.h"
 
+static const char * const odb_source_names_by_type[] = {
+	[ODB_SOURCE_UNKNOWN] = "unknown",
+	[ODB_SOURCE_FILES] = "files",
+	[ODB_SOURCE_LOOSE] = "loose",
+	[ODB_SOURCE_PACKED] = "packed",
+	[ODB_SOURCE_INMEMORY] = "in-memory",
+};
+
+const char *odb_source_type_to_name(enum odb_source_type type)
+{
+	const char *name;
+	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
+		type = ODB_SOURCE_UNKNOWN;
+	name = odb_source_names_by_type[type];
+	if (!name)
+		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
+	return name;
+}
+
 struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
diff --git a/odb/source.h b/odb/source.h
index cd63dba91f..ab16d152f4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -25,6 +25,12 @@ enum odb_source_type {
 	ODB_SOURCE_INMEMORY,
 };
 
+/*
+ * Convert between the enum and its name. Returns the equivalent of "unknown"
+ * for unknown types.
+ */
+const char *odb_source_type_to_name(enum odb_source_type type);
+
 struct object_id;
 struct odb_read_stream;
 struct strvec;

-- 
2.55.0.679.g6767b8d81c.dirty

[PATCH v5 6/6] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.

Make the creation of on-disk structures pluggable to accommodate for
this.

Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 19 +++++++++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 setup.c            | 34 ++++++++++++++++++----------------
 3 files changed, 60 insertions(+), 16 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 4138758511..0db6e681fe 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -9,6 +9,7 @@
 #include "odb/source-files.h"
 #include "odb/source-loose.h"
 #include "packfile.h"
+#include "path.h"
 #include "strbuf.h"
 #include "write-or-die.h"
 
@@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
 	odb_source_close(&files->packed->base);
 }
 
+static int odb_source_files_create_on_disk(struct odb_source *source)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	safe_create_dir(source->odb->repo, source->path, 1);
+
+	strbuf_addf(&path, "%s/pack", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/info", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_release(&path);
+	return 0;
+}
+
 static void odb_source_files_prepare(struct odb_source *source,
 				     enum odb_prepare_flags flags)
 {
@@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
+	files->base.create_on_disk = odb_source_files_create_on_disk;
 	files->base.prepare = odb_source_files_prepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
diff --git a/odb/source.h b/odb/source.h
index ab16d152f4..4abc418bdd 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -89,6 +89,18 @@ struct odb_source {
 	 */
 	void (*close)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to create on-disk data structures that are
+	 * required for this source to operate.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 *
+	 * This callback may be NULL in case the source does not need any
+	 * on-disk setup.
+	 */
+	int (*create_on_disk)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to prepare the source so that it becomes
 	 * ready for use. It optionally clears underlying caches of the object
@@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
 	source->close(source);
 }
 
+/*
+ * Create on-disk data structures that are required for this source to operate
+ * correctly. Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_create_on_disk(struct odb_source *source)
+{
+	if (!source->create_on_disk)
+		return 0;
+	return source->create_on_disk(source);
+}
+
 /*
  * Prepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects
diff --git a/setup.c b/setup.c
index 97338cbc51..ace3c59d18 100644
--- a/setup.c
+++ b/setup.c
@@ -2649,25 +2649,27 @@ static int create_default_files(struct repository *repo,
 
 static void create_object_database(struct repository *repo)
 {
-	struct strbuf path = STRBUF_INIT;
-	size_t baselen;
+	/*
+	 * Create the "objects" directory in the common directory. This is done
+	 * so that the repository can be discovered regardless of the backend
+	 * used.
+	 *
+	 * Note that we only do this in case the object directory wasn't
+	 * overwritten via an environment variable. If it _is_ being overridden
+	 * then we skip this step, as the repository won't be discoverable
+	 * anyway without the environment variable.
+	 */
+	if (!getenv(DB_ENVIRONMENT)) {
+		struct strbuf objects_dir = STRBUF_INIT;
+		repo_common_path_append(repo, &objects_dir, "objects");
+		safe_create_dir(repo, objects_dir.buf, 1);
+		strbuf_release(&objects_dir);
+	}
 
 	repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
 
-	strbuf_addstr(&path, repo_get_object_directory(repo));
-	baselen = path.len;
-
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/pack");
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/info");
-	safe_create_dir(repo, path.buf, 1);
-
-	strbuf_release(&path);
+	if (odb_source_create_on_disk(repo->objects->sources) < 0)
+		die(_("failed creating object database"));
 }
 
 static void separate_git_dir(const char *git_dir, const char *git_link)

-- 
2.55.0.679.g6767b8d81c.dirty

Re: [PATCH v5 4/6] setup: defer object database creation

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> diff --git a/setup.c b/setup.c
> index 5dfab3e79e..97338cbc51 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1888,6 +1882,7 @@ const char *enter_repo(struct repository *repo, const char *path, unsigned flags
>  		read_and_verify_repository_format(&fmt, ".", NULL);
>  		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
>  			die("%s", err.buf);
> +		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
>  		startup_info->have_repository = 1;
>  
>  		clear_repository_format(&fmt);

The previous round corrected the overly long line while at it, but
it is no longer done here.

Which is OK either way.

> @@ -2090,6 +2085,7 @@ const char *setup_git_directory_gently(struct repository *repo, int *nongit_ok)
>  			if (apply_repository_format(repo, &discovery.format,
>  						    APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
>  				die("%s", err.buf);
> +			repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);

Looks like the differences since the last round is truly minimum ;-)

Re: [PATCH v5 4/6] setup: defer object database creation

[Toon Claes]

Junio C Hamano <gitster@pobox.com> writes:

> Patrick Steinhardt <ps@pks.im> writes:
>
>> diff --git a/setup.c b/setup.c
>> index 5dfab3e79e..97338cbc51 100644
>> --- a/setup.c
>> +++ b/setup.c
>> @@ -1888,6 +1882,7 @@ const char *enter_repo(struct repository *repo, const char *path, unsigned flags
>>  		read_and_verify_repository_format(&fmt, ".", NULL);
>>  		if (apply_repository_format(repo, &fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
>>  			die("%s", err.buf);
>> +		repo->objects = odb_new(repo, ODB_NEW_HONOR_ENV);
>>  		startup_info->have_repository = 1;
>>  
>>  		clear_repository_format(&fmt);
>
> The previous round corrected the overly long line while at it, but
> it is no longer done here.

Yeah, I've asked about this. In [PATCH v3 4/6] this change existed:

-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)

But adding APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION was reverted in v4,
but that version still had the reformatting change (fixing the overly
long line).

There are multiple occurrences of this overly long line, but only this
one was changed in v4. So Patrick reverted changing the overly long line
in v5, which I think is better.

-- 
Cheers,
Toon

Re: [PATCH v5 0/6] odb: make creation of object database pluggable

[Toon Claes]

Patrick Steinhardt <ps@pks.im> writes:

> Hi,
>
> when creating a new repository we create a couple of on-disk data
> structures for the object database. This includes the "objects/"
> directory hierarchy with "objects/info" and "objects/pack", which are
> specific to the backend.
>
> This patch series makes the creation of the on-disk data structures
> pluggable. While we continue to always create "objects/" regardless of
> the backend (it's required for a repository to be recognized as such),
> the other subdirectories are now created by the backend. This will allow
> other backends to plug in their own logic.
>
> The series starts with a small detour into the loose-object map. This
> detour is required so that we can defer initialization of the object
> database itself to a later point in time.
>
> The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).
>
> Changes in v5:
>   - Remove a leftover formatting change.
>   - Fix a stale comment.
>   - Link to v4: https://patch.msgid.link/20260806-pks-odb-create-on-disk-v4-0-ba8b4fdd2e3c@pks.im

I'm completely happy with this version, thanks for bearing with me.

-- 
Cheers,
Toon

Re: [PATCH v5 0/6] odb: make creation of object database pluggable

[Patrick Steinhardt]

On Fri, Aug 07, 2026 at 09:17:25AM +0200, Toon Claes wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > Hi,
> >
> > when creating a new repository we create a couple of on-disk data
> > structures for the object database. This includes the "objects/"
> > directory hierarchy with "objects/info" and "objects/pack", which are
> > specific to the backend.
> >
> > This patch series makes the creation of the on-disk data structures
> > pluggable. While we continue to always create "objects/" regardless of
> > the backend (it's required for a repository to be recognized as such),
> > the other subdirectories are now created by the backend. This will allow
> > other backends to plug in their own logic.
> >
> > The series starts with a small detour into the loose-object map. This
> > detour is required so that we can defer initialization of the object
> > database itself to a later point in time.
> >
> > The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).
> >
> > Changes in v5:
> >   - Remove a leftover formatting change.
> >   - Fix a stale comment.
> >   - Link to v4: https://patch.msgid.link/20260806-pks-odb-create-on-disk-v4-0-ba8b4fdd2e3c@pks.im
> 
> I'm completely happy with this version, thanks for bearing with me.

Thanks for your reviews!

Patrick