[PATCH 0/5] odb: make creation of object database pluggable

[PATCH 0/5] odb: make creation of object database pluggable

[Patrick Steinhardt]

Hi,

when creating a new repository we create a couple of on-disk data
structures for the object database. This includes the "objects/"
directory hierarchy with "objects/info" and "objects/pack", which are
specific to the backend.

This patch series makes the creation of the on-disk data structures
pluggable. While we continue to always create "objects/" regardless of
the backend (it's required for a repository to be recognized as such),
the other subdirectories are now created by the backend. This will allow
other backends to plug in their own logic.

The series starts with a small detour into the loose-object map. This
detour is required so that we can defer initialization of the object
database itself to a later point in time.

The series is based on 9a0c4701dc (The 7th batch, 2026-07-22).

Thanks!

Patrick

---
Patrick Steinhardt (5):
      loose: load loose object map for the correct source
      setup: detangle loading of loose object maps
      setup: defer object database creation
      odb/source: introduce function to map source type to name
      odb: make creation of on-disk structures pluggable

 loose.c               | 25 ++++++++++----------
 loose.h               |  1 +
 odb/source-files.c    | 19 +++++++++++++++
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.c    |  2 ++
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++
 odb/source.h          | 29 +++++++++++++++++++++++
 repository.c          |  2 --
 setup.c               | 65 +++++++++++++++++++++++++++++++++++----------------
 setup.h               |  9 +++++++
 13 files changed, 149 insertions(+), 38 deletions(-)


---
base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca
change-id: 20260710-pks-odb-create-on-disk-ae8757861c69

[PATCH 1/5] loose: load loose object map for the correct source

[Patrick Steinhardt]

When loading the loose object map via `load_one_loose_object_map()` we
pass in both a repository and the corresponding source. We ultimately
don't really respect the passed-in source though as we instead always
load the map via the common directory. This doesn't make any sense
though, as the function is called in a loop through all sources, and as
such the expectation is that we'll load the map that belongs to the
given source.

Fix this bug by instead loading the map via the loose source's path.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c | 18 ++++++++++--------
 1 file changed, 10 insertions(+), 8 deletions(-)

diff --git a/loose.c b/loose.c
index bf01d3e42d..9dad75373b 100644
--- a/loose.c
+++ b/loose.c
@@ -61,9 +61,11 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
+static int load_one_loose_object_map(struct odb_source_loose *loose)
 {
-	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
+	struct repository *repo = loose->base.odb->repo;
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
 	FILE *fp;
 	int ret = -1;
 
@@ -78,10 +80,10 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 	insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
 	insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
 
-	repo_common_path_replace(repo, &path, "objects/loose-object-idx");
-	fp = fopen(path.buf, "rb");
+	path = xstrfmt("%s/loose-object-idx", loose->base.path);
+	fp = fopen(path, "rb");
 	if (!fp) {
-		strbuf_release(&path);
+		free(path);
 		return 0;
 	}
 
@@ -102,7 +104,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source_
 err:
 	fclose(fp);
 	strbuf_release(&buf);
-	strbuf_release(&path);
+	free(path);
 	return ret;
 }
 
@@ -117,10 +119,10 @@ int repo_read_loose_object_map(struct repository *repo)
 
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(repo, files->loose) < 0) {
+		if (load_one_loose_object_map(files->loose) < 0)
 			return -1;
-		}
 	}
+
 	return 0;
 }
 

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 2/5] setup: detangle loading of loose object maps

[Patrick Steinhardt]

When a repository is configured to use a compatibility hash function
then we load the loose object map when we initialize the repository.
This object map provides the mappings between the canonical object hash
and the compatibility object hash.

Loading the object map happens in `repo_set_compat_hash_algo()`, which
calls `repo_read_loose_object_map()` in case the compatibility object
hash is non-zero. This setup sequence has two major downsides:

  - We assume that the primary object database is the "files" object
    database so that we can extract its "loose" backend. This stops
    working with pluggable object databases.

  - We require the object database to already have been initialized when
    configuring the object database. This means that we must intermix
    configuration of the repository and initialization of its
    sub-structures in a weird way.

Refactor the logic so that we instead load the loose object map via the
"loose" backend, which fixes both of the above issues.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 loose.c            | 11 +++++------
 loose.h            |  1 +
 odb/source-loose.c |  2 ++
 repository.c       |  2 --
 setup.c            |  5 +++--
 5 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/loose.c b/loose.c
index 9dad75373b..a3b2dcedc2 100644
--- a/loose.c
+++ b/loose.c
@@ -61,7 +61,7 @@ static int insert_loose_map(struct odb_source_loose *loose,
 	return inserted;
 }
 
-static int load_one_loose_object_map(struct odb_source_loose *loose)
+int loose_object_map_load(struct odb_source_loose *loose)
 {
 	struct repository *repo = loose->base.odb->repo;
 	struct strbuf buf = STRBUF_INIT;
@@ -69,6 +69,9 @@ static int load_one_loose_object_map(struct odb_source_loose *loose)
 	FILE *fp;
 	int ret = -1;
 
+	if (!should_use_loose_object_map(repo))
+		return 0;
+
 	if (!loose->map)
 		loose_object_map_init(&loose->map);
 	if (!loose->cache) {
@@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
 {
 	struct odb_source *source;
 
-	if (!should_use_loose_object_map(repo))
-		return 0;
-
 	odb_prepare_alternates(repo->objects);
-
 	for (source = repo->objects->sources; source; source = source->next) {
 		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (load_one_loose_object_map(files->loose) < 0)
+		if (loose_object_map_load(files->loose) < 0)
 			return -1;
 	}
 
diff --git a/loose.h b/loose.h
index 6c9b3f4571..ed663ac550 100644
--- a/loose.h
+++ b/loose.h
@@ -13,6 +13,7 @@ struct loose_object_map {
 
 void loose_object_map_init(struct loose_object_map **map);
 void loose_object_map_clear(struct loose_object_map **map);
+int loose_object_map_load(struct odb_source_loose *loose);
 int repo_loose_object_map_oid(struct repository *repo,
 			      const struct object_id *src,
 			      const struct git_hash_algo *dest_algo,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 3f7d04a56e..812ca1c138 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -727,5 +727,7 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 	if (!is_absolute_path(loose->base.path))
 		chdir_notify_register(NULL, odb_source_loose_reparent, loose);
 
+	loose_object_map_load(loose);
+
 	return loose;
 }
diff --git a/repository.c b/repository.c
index 2ef0778846..6d633002b4 100644
--- a/repository.c
+++ b/repository.c
@@ -201,8 +201,6 @@ void repo_set_compat_hash_algo(struct repository *repo MAYBE_UNUSED, uint32_t al
 	if (hash_algo_by_ptr(repo->hash_algo) == algo)
 		BUG("hash_algo and compat_hash_algo match");
 	repo->compat_hash_algo = algo ? &hash_algos[algo] : NULL;
-	if (repo->compat_hash_algo)
-		repo_read_loose_object_map(repo);
 #else
 	if (algo)
 		die(_("compatibility hash algorithm support requires Rust"));
diff --git a/setup.c b/setup.c
index d31808130b..825572f5f1 100644
--- a/setup.c
+++ b/setup.c
@@ -1788,8 +1788,6 @@ int apply_repository_format(struct repository *repo,
 
 	repo->bare_cfg = format->is_bare;
 	repo_set_hash_algo(repo, format->hash_algo);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 	repo_set_compat_hash_algo(repo, format->compat_hash_algo);
 	repo_set_ref_storage_format(repo,
 				    format->ref_storage_format,
@@ -1805,6 +1803,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	free(alternate_object_directories);
 	free(object_directory);
 	return 0;

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 3/5] setup: defer object database creation

[Patrick Steinhardt]

In a subsequent commit we'll make the creation of the on-disk data
structures of an object database pluggable. This will lead to an
in-between state where we have already configured the repository's
object database, but it's not usable yet until we eventually call
`create_object_directory()`.

Defer the object database creation so that we handle both steps in the
same function.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 setup.c | 35 +++++++++++++++++++++++++++--------
 setup.h |  9 +++++++++
 2 files changed, 36 insertions(+), 8 deletions(-)

diff --git a/setup.c b/setup.c
index 825572f5f1..a7b1b9eaef 100644
--- a/setup.c
+++ b/setup.c
@@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
 	return result;
 }
 
+static void get_object_directories(char **object_directory,
+				   char **alternate_object_directories)
+{
+	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
+	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+}
+
 int apply_repository_format(struct repository *repo,
 			    const struct repository_format *format,
 			    enum apply_repository_format_flags flags,
@@ -1779,8 +1786,9 @@ int apply_repository_format(struct repository *repo,
 	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
 		const char *shallow_file;
 
-		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
-		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
+		get_object_directories(&object_directory,
+				       &alternate_object_directories);
+
 		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
 		if (shallow_file)
 			set_alternate_shallow_file(repo, shallow_file);
@@ -1803,8 +1811,9 @@ int apply_repository_format(struct repository *repo,
 	repo->repository_format_precious_objects =
 		format->precious_objects;
 
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
+	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION))
+		repo->objects = odb_new(repo, object_directory,
+					alternate_object_directories);
 
 	free(alternate_object_directories);
 	free(object_directory);
@@ -2654,11 +2663,16 @@ static int create_default_files(struct repository *repo,
 	return reinit;
 }
 
-static void create_object_directory(struct repository *repo)
+static void create_object_database(struct repository *repo)
 {
+	char *object_directory, *alternate_object_directories;
 	struct strbuf path = STRBUF_INIT;
 	size_t baselen;
 
+	get_object_directories(&object_directory, &alternate_object_directories);
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
+
 	strbuf_addstr(&path, repo_get_object_directory(repo));
 	baselen = path.len;
 
@@ -2672,6 +2686,8 @@ static void create_object_directory(struct repository *repo)
 	strbuf_addstr(&path, "/info");
 	safe_create_dir(repo, path.buf, 1);
 
+	free(alternate_object_directories);
+	free(object_directory);
 	strbuf_release(&path);
 }
 
@@ -2867,9 +2883,10 @@ int init_db(struct repository *repo,
 	 */
 	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
 	repository_format_configure(&repo_fmt, hash, ref_storage_format);
-	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
+	if (apply_repository_format(repo, &repo_fmt,
+				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
+				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
 		die("%s", err.buf);
-	startup_info->have_repository = 1;
 
 	/*
 	 * Ensure `core.hidedotfiles` is processed. This must happen after we
@@ -2885,7 +2902,9 @@ int init_db(struct repository *repo,
 
 	if (!(flags & INIT_DB_SKIP_REFDB))
 		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
-	create_object_directory(repo);
+	create_object_database(repo);
+
+	startup_info->have_repository = 1;
 
 	if (repo_settings_get_shared_repository(repo)) {
 		char buf[10];
diff --git a/setup.h b/setup.h
index 654f10e059..e55d647b70 100644
--- a/setup.h
+++ b/setup.h
@@ -241,6 +241,15 @@ enum apply_repository_format_flags {
 	 * relate to the object database.
 	 */
 	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
+
+	/*
+	 * Usually, the object database is created after the repository format
+	 * was applied. This step is skipped if this flag is set, which leaves
+	 * us with a partially-working repository.
+	 *
+	 * This is useful when initializing a new repository.
+	 */
+	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
 };
 
 /*

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 4/5] odb/source: introduce function to map source type to name

[Patrick Steinhardt]

Introduce a new function that maps an object source's type to a
human-readable name. Use the function to provide better human-readable
error messages for the downcasting functions.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.h    |  4 +++-
 odb/source-inmemory.h |  4 +++-
 odb/source-loose.h    |  4 +++-
 odb/source-packed.h   |  4 +++-
 odb/source.c          | 19 +++++++++++++++++++
 odb/source.h          |  6 ++++++
 6 files changed, 37 insertions(+), 4 deletions(-)

diff --git a/odb/source-files.h b/odb/source-files.h
index d7ac3c1c81..6a803afdda 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -28,7 +28,9 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_FILES)
-		BUG("trying to downcast source of type '%d' to files", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_FILES));
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source-inmemory.h b/odb/source-inmemory.h
index a88fc2e320..adbad23e8b 100644
--- a/odb/source-inmemory.h
+++ b/odb/source-inmemory.h
@@ -26,7 +26,9 @@ struct odb_source_inmemory *odb_source_inmemory_new(struct object_database *odb)
 static inline struct odb_source_inmemory *odb_source_inmemory_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_INMEMORY)
-		BUG("trying to downcast source of type '%d' to in-memory", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_INMEMORY));
 	return container_of(source, struct odb_source_inmemory, base);
 }
 
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 6070aaf3ce..3cf2e1f8f1 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -41,7 +41,9 @@ struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
 static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_LOOSE)
-		BUG("trying to downcast source of type '%d' to loose", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_LOOSE));
 	return container_of(source, struct odb_source_loose, base);
 }
 
diff --git a/odb/source-packed.h b/odb/source-packed.h
index 77309ddd09..a0f6b5096d 100644
--- a/odb/source-packed.h
+++ b/odb/source-packed.h
@@ -78,7 +78,9 @@ struct odb_source_packed *odb_source_packed_new(struct object_database *odb,
 static inline struct odb_source_packed *odb_source_packed_downcast(struct odb_source *source)
 {
 	if (source->type != ODB_SOURCE_PACKED)
-		BUG("trying to downcast source of type '%d' to packed", source->type);
+		BUG("trying to downcast source of type '%s' to '%s'",
+		    odb_source_type_to_name(source->type),
+		    odb_source_type_to_name(ODB_SOURCE_PACKED));
 	return container_of(source, struct odb_source_packed, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index 7993dcbd65..c300e836f6 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -4,6 +4,25 @@
 #include "odb/source.h"
 #include "packfile.h"
 
+static const char * const odb_source_names_by_type[] = {
+	[ODB_SOURCE_UNKNOWN] = "unknown",
+	[ODB_SOURCE_FILES] = "files",
+	[ODB_SOURCE_LOOSE] = "loose",
+	[ODB_SOURCE_PACKED] = "packed",
+	[ODB_SOURCE_INMEMORY] = "inmemory",
+};
+
+const char *odb_source_type_to_name(enum odb_source_type type)
+{
+	const char *name;
+	if (type < 0 || type >= ARRAY_SIZE(odb_source_names_by_type))
+		type = ODB_SOURCE_UNKNOWN;
+	name = odb_source_names_by_type[type];
+	if (!name)
+		BUG("name missing in `odb_source_names_by_type` for '%d'", type);
+	return name;
+}
+
 struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
diff --git a/odb/source.h b/odb/source.h
index cd63dba91f..ab16d152f4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -25,6 +25,12 @@ enum odb_source_type {
 	ODB_SOURCE_INMEMORY,
 };
 
+/*
+ * Convert between the enum and its name. Returns the equivalent of "unknown"
+ * for unknown types.
+ */
+const char *odb_source_type_to_name(enum odb_source_type type);
+
 struct object_id;
 struct odb_read_stream;
 struct strvec;

-- 
2.55.0.407.g700c83d4f3.dirty

[PATCH 5/5] odb: make creation of on-disk structures pluggable

[Patrick Steinhardt]

When creating a new "files" object database source we have to create a
couple of directories. These directories are of course specific to this
particular backend, and a different backend may require a setup that is
completely different.

Make the creation of on-disk structures pluggable to accommodate for
this.

Note that there is one exception though: the "objects" directory must
exist in a repository regardless of which backend is in use. If it
doesn't exist then the repository is not treated as a Git repository at
all. Consequently, we create this directory regardless of the backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 19 +++++++++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 setup.c            | 35 ++++++++++++++++++++---------------
 3 files changed, 62 insertions(+), 15 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 4138758511..0db6e681fe 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -9,6 +9,7 @@
 #include "odb/source-files.h"
 #include "odb/source-loose.h"
 #include "packfile.h"
+#include "path.h"
 #include "strbuf.h"
 #include "write-or-die.h"
 
@@ -41,6 +42,23 @@ static void odb_source_files_close(struct odb_source *source)
 	odb_source_close(&files->packed->base);
 }
 
+static int odb_source_files_create_on_disk(struct odb_source *source)
+{
+	struct strbuf path = STRBUF_INIT;
+
+	safe_create_dir(source->odb->repo, source->path, 1);
+
+	strbuf_addf(&path, "%s/pack", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_reset(&path);
+	strbuf_addf(&path, "%s/info", source->path);
+	safe_create_dir(source->odb->repo, path.buf, 1);
+
+	strbuf_release(&path);
+	return 0;
+}
+
 static void odb_source_files_prepare(struct odb_source *source,
 				     enum odb_prepare_flags flags)
 {
@@ -271,6 +289,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
+	files->base.create_on_disk = odb_source_files_create_on_disk;
 	files->base.prepare = odb_source_files_prepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
diff --git a/odb/source.h b/odb/source.h
index ab16d152f4..4abc418bdd 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -89,6 +89,18 @@ struct odb_source {
 	 */
 	void (*close)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to create on-disk data structures that are
+	 * required for this source to operate.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 *
+	 * This callback may be NULL in case the source does not need any
+	 * on-disk setup.
+	 */
+	int (*create_on_disk)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to prepare the source so that it becomes
 	 * ready for use. It optionally clears underlying caches of the object
@@ -316,6 +328,17 @@ static inline void odb_source_close(struct odb_source *source)
 	source->close(source);
 }
 
+/*
+ * Create on-disk data structures that are required for this source to operate
+ * correctly. Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_create_on_disk(struct odb_source *source)
+{
+	if (!source->create_on_disk)
+		return 0;
+	return source->create_on_disk(source);
+}
+
 /*
  * Prepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects
diff --git a/setup.c b/setup.c
index a7b1b9eaef..14ef119cb7 100644
--- a/setup.c
+++ b/setup.c
@@ -2666,29 +2666,34 @@ static int create_default_files(struct repository *repo,
 static void create_object_database(struct repository *repo)
 {
 	char *object_directory, *alternate_object_directories;
-	struct strbuf path = STRBUF_INIT;
-	size_t baselen;
 
 	get_object_directories(&object_directory, &alternate_object_directories);
-	repo->objects = odb_new(repo, object_directory,
-				alternate_object_directories);
 
-	strbuf_addstr(&path, repo_get_object_directory(repo));
-	baselen = path.len;
-
-	safe_create_dir(repo, path.buf, 1);
+	/*
+	 * Create the "objects" directory in the common directory. This is done
+	 * so that the repository can be discovered regardless of the backend
+	 * used.
+	 *
+	 * Note that we only do this in case the object directory wasn't
+	 * overwritten via an environment variable. If it _is_ being overridden
+	 * then we skip this step, as the repository won't be discoverable
+	 * anyway without the environment variable.
+	 */
+	if (!object_directory) {
+		struct strbuf objects_dir = STRBUF_INIT;
+		repo_common_path_append(repo, &objects_dir, "objects");
+		safe_create_dir(repo, objects_dir.buf, 1);
+		strbuf_release(&objects_dir);
+	}
 
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/pack");
-	safe_create_dir(repo, path.buf, 1);
+	repo->objects = odb_new(repo, object_directory,
+				alternate_object_directories);
 
-	strbuf_setlen(&path, baselen);
-	strbuf_addstr(&path, "/info");
-	safe_create_dir(repo, path.buf, 1);
+	if (odb_source_create_on_disk(repo->objects->sources) < 0)
+		die("failed creating object database");
 
 	free(alternate_object_directories);
 	free(object_directory);
-	strbuf_release(&path);
 }
 
 static void separate_git_dir(const char *git_dir, const char *git_link)

-- 
2.55.0.407.g700c83d4f3.dirty

Re: [PATCH 1/5] loose: load loose object map for the correct source

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> When loading the loose object map via `load_one_loose_object_map()` we
> pass in both a repository and the corresponding source. We ultimately
> don't really respect the passed-in source though as we instead always
> load the map via the common directory. This doesn't make any sense
> though, as the function is called in a loop through all sources, and as
> such the expectation is that we'll load the map that belongs to the
> given source.
>
> Fix this bug by instead loading the map via the loose source's path.

Makes perfect sense.  We still need access to the 'repo' to learn
the hash algorithm used in the repository along with built-in object
names, but they are now obtained from the repository associated with
the loose object source, which is far more consistent.

Re: [PATCH 2/5] setup: detangle loading of loose object maps

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> When a repository is configured to use a compatibility hash function
> then we load the loose object map when we initialize the repository.
> This object map provides the mappings between the canonical object hash
> and the compatibility object hash.
>
> Loading the object map happens in `repo_set_compat_hash_algo()`, which
> calls `repo_read_loose_object_map()` in case the compatibility object
> hash is non-zero. This setup sequence has two major downsides:
>
>   - We assume that the primary object database is the "files" object
>     database so that we can extract its "loose" backend. This stops
>     working with pluggable object databases.

I am not sure if I understand this sentence, especially "we can
extract its loose backend" part.  Do you mean 'extract the object
map from the loose backend'?  Or something else?

>   - We require the object database to already have been initialized when
>     configuring the object database. This means that we must intermix
>     configuration of the repository and initialization of its
>     sub-structures in a weird way.
>
> Refactor the logic so that we instead load the loose object map via the
> "loose" backend, which fixes both of the above issues.

It does make sense to have loose_object_map_load() that is very much
specific to the loose object odb source to odb_source_loose_new().
That way set_compat_hash_algo() does not have to assume that files
backend is used as the object store.

> @@ -112,14 +115,10 @@ int repo_read_loose_object_map(struct repository *repo)
>  {
>  	struct odb_source *source;
>  
> -	if (!should_use_loose_object_map(repo))
> -		return 0;
> -
>  	odb_prepare_alternates(repo->objects);
> -
>  	for (source = repo->objects->sources; source; source = source->next) {
>  		struct odb_source_files *files = odb_source_files_downcast(source);
> -		if (load_one_loose_object_map(files->loose) < 0)
> +		if (loose_object_map_load(files->loose) < 0)
>  			return -1;

If this particular source in the list of sources is not backed by
the files backend, would downcast signal the fact (e.g., by
returning NULL) so that we can skip the next call instead?

Or would the next step in refactoring be to define "load object map"
method that is generic to odb_source so that this part does not have
to do any of these and instead simply do

	for (source = ...) {
		if (odb_source_object_map_load(source))
                	return -1;
        }

or something?

Re: [PATCH 3/5] setup: defer object database creation

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> In a subsequent commit we'll make the creation of the on-disk data
> structures of an object database pluggable. This will lead to an
> in-between state where we have already configured the repository's
> object database, but it's not usable yet until we eventually call
> `create_object_directory()`.
>
> Defer the object database creation so that we handle both steps in the
> same function.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  setup.c | 35 +++++++++++++++++++++++++++--------
>  setup.h |  9 +++++++++
>  2 files changed, 36 insertions(+), 8 deletions(-)
>
> diff --git a/setup.c b/setup.c
> index 825572f5f1..a7b1b9eaef 100644
> --- a/setup.c
> +++ b/setup.c
> @@ -1760,6 +1760,13 @@ enum discovery_result discover_git_directory_reason(struct strbuf *commondir,
>  	return result;
>  }
>  
> +static void get_object_directories(char **object_directory,
> +				   char **alternate_object_directories)
> +{
> +	*object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> +	*alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> +}
> +
>  int apply_repository_format(struct repository *repo,
>  			    const struct repository_format *format,
>  			    enum apply_repository_format_flags flags,
> @@ -1779,8 +1786,9 @@ int apply_repository_format(struct repository *repo,
>  	if (flags & APPLY_REPOSITORY_FORMAT_HONOR_ENV) {
>  		const char *shallow_file;
>  
> -		object_directory = xstrdup_or_null(getenv(DB_ENVIRONMENT));
> -		alternate_object_directories = xstrdup_or_null(getenv(ALTERNATE_DB_ENVIRONMENT));
> +		get_object_directories(&object_directory,
> +				       &alternate_object_directories);
> +
>  		shallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);
>  		if (shallow_file)
>  			set_alternate_shallow_file(repo, shallow_file);

HONOR_ENV still means we read the environment variable to learn where
the object directory (which is admittedly a files backend specific
concept) and alternate object directories (ditto) are.

> @@ -1803,8 +1811,9 @@ int apply_repository_format(struct repository *repo,
>  	repo->repository_format_precious_objects =
>  		format->precious_objects;
>  
> -	repo->objects = odb_new(repo, object_directory,
> -				alternate_object_directories);
> +	if (!(flags & APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION))
> +		repo->objects = odb_new(repo, object_directory,
> +					alternate_object_directories);

And SKIP_ODB_CREATION can tell apply_repository_format() not to
create an odb there.

> -static void create_object_directory(struct repository *repo)
> +static void create_object_database(struct repository *repo)
>  {
> +	char *object_directory, *alternate_object_directories;
>  	struct strbuf path = STRBUF_INIT;
>  	size_t baselen;
>  
> +	get_object_directories(&object_directory, &alternate_object_directories);
> +	repo->objects = odb_new(repo, object_directory,
> +				alternate_object_directories);
> +
>  	strbuf_addstr(&path, repo_get_object_directory(repo));
>  	baselen = path.len;
>  
> @@ -2672,6 +2686,8 @@ static void create_object_directory(struct repository *repo)
>  	strbuf_addstr(&path, "/info");
>  	safe_create_dir(repo, path.buf, 1);
>  
> +	free(alternate_object_directories);
> +	free(object_directory);
>  	strbuf_release(&path);
>  }
>  


> @@ -2867,9 +2883,10 @@ int init_db(struct repository *repo,
>  	 */
>  	read_and_verify_repository_format(&repo_fmt, repo_get_git_dir(repo), NULL);
>  	repository_format_configure(&repo_fmt, hash, ref_storage_format);
> -	if (apply_repository_format(repo, &repo_fmt, APPLY_REPOSITORY_FORMAT_HONOR_ENV, &err) < 0)
> +	if (apply_repository_format(repo, &repo_fmt,
> +				    APPLY_REPOSITORY_FORMAT_HONOR_ENV |
> +				    APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION, &err) < 0)
>  		die("%s", err.buf);
> -	startup_info->have_repository = 1;

Early in initialization, we no longer recreate the ODB when calling
apply_repository_format(), and we defer declaring that we have a
repository until we call create_object_database().

> @@ -2885,7 +2902,9 @@ int init_db(struct repository *repo,
>  
>  	if (!(flags & INIT_DB_SKIP_REFDB))
>  		create_reference_database(repo, initial_branch, flags & INIT_DB_QUIET);
> -	create_object_directory(repo);
> +	create_object_database(repo);
> +
> +	startup_info->have_repository = 1;

Instead we call create_object_database() rather late, after we
finish creating leading directories and default files and processing
the configuration.  I guess this is a prelude to specifying "no, we
are not doing the files backend but are using this new thing" in the
global configuration?

> diff --git a/setup.h b/setup.h
> index 654f10e059..e55d647b70 100644
> --- a/setup.h
> +++ b/setup.h
> @@ -241,6 +241,15 @@ enum apply_repository_format_flags {
>  	 * relate to the object database.
>  	 */
>  	APPLY_REPOSITORY_FORMAT_HONOR_ENV = (1 << 0),
> +
> +	/*
> +	 * Usually, the object database is created after the repository format
> +	 * was applied. This step is skipped if this flag is set, which leaves
> +	 * us with a partially-working repository.
> +	 *
> +	 * This is useful when initializing a new repository.
> +	 */
> +	APPLY_REPOSITORY_FORMAT_SKIP_ODB_CREATION = (1 << 1),
>  };

OK.