[PATCH 00/17] odb: make object database sources pluggable

[PATCH 00/17] odb: make object database sources pluggable

[Patrick Steinhardt]

Hi,

this patch series finally makes the object database source pluggable.
This is done by moving backend-specific logics into callback functions
that are part of `struct odb_source` and providing thin wrappers that
call those functions.

To set expectations: this is only a start, there is still functionality
missing that needs to be made pluggable. Most importantly:

  - Counting of objects.

  - Abbreviating object IDs and finding ambiguous objects.

  - Consistency checks.

  - Optimizing the object database.

  - Generating packfiles.

These will all happen in later patch series. That being said, with this
patch series one already gets a lot of the basic functionality, and it's
almost possible to do local workflows. Only "almost" though because we
rely on abbreviating object IDs in a lot of places, but once that part
is implemented in a subsequent patch series you can indeed work locally
with an alternate backend.

Furthermore, what I didn't include as part of this patch series just yet
is the introduction of the "objectStorage" extension. I mostly wanted to
focus on the mostly-trivial parts without introducing any change in
behaviour.

Thanks!

Patrick

---
Patrick Steinhardt (17):
      odb: split `struct odb_source` into separate header
      odb: introduce "files" source
      odb: embed base source in the "files" backend
      odb: move reparenting logic into respective subsystems
      odb/source: introduce source type for robustness
      odb/source: make `free()` function pluggable
      odb/source: make `reprepare()` function pluggable
      odb/source: make `close()` function pluggable
      odb/source: make `read_object_info()` function pluggable
      odb/source: make `read_object_stream()` function pluggable
      odb/source: make `for_each_object()` function pluggable
      odb/source: make `freshen_object()` function pluggable
      odb/source: make `write_object()` function pluggable
      odb/source: make `write_object_stream()` function pluggable
      odb/source: make `read_alternates()` function pluggable
      odb/source: make `write_alternate()` function pluggable
      odb/source: make `begin_transaction()` function pluggable

 Makefile               |   2 +
 builtin/cat-file.c     |   3 +-
 builtin/fast-import.c  |  12 +-
 builtin/grep.c         |   6 +-
 builtin/index-pack.c   |   8 +-
 builtin/pack-objects.c |  13 +-
 commit-graph.c         |   6 +-
 http.c                 |   3 +-
 loose.c                |  23 ++-
 meson.build            |   2 +
 midx.c                 |  26 +--
 object-file.c          |  40 +++--
 odb.c                  | 191 +++-----------------
 odb.h                  |  86 +--------
 odb/source-files.c     | 239 +++++++++++++++++++++++++
 odb/source-files.h     |  35 ++++
 odb/source.c           |  38 ++++
 odb/source.h           | 464 +++++++++++++++++++++++++++++++++++++++++++++++++
 odb/streaming.c        |   8 +-
 packfile.c             |  36 ++--
 packfile.h             |   7 +-
 tmp-objdir.c           |  42 ++---
 tmp-objdir.h           |  15 --
 23 files changed, 950 insertions(+), 355 deletions(-)


---
base-commit: 197ce3527e423304844fef02ea067a85c0c75e70
change-id: 20260120-b4-pks-odb-source-pluggable-5c724250b3c8

[PATCH 01/17] odb: split `struct odb_source` into separate header

[Patrick Steinhardt]

Subsequent commits will expand the `struct odb_source` to become a
generic interface for accessing an object database source. As part of
these refactorings we'll add a set of function pointers that will
significantly expand the structure overall.

Prepare for this by splitting out the `struct odb_source` into a
separate header. This keeps the high-level object database interface
detached from the low-level object database sources.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Makefile     |  1 +
 meson.build  |  1 +
 odb.c        | 25 -------------------------
 odb.h        | 45 +--------------------------------------------
 odb/source.c | 28 ++++++++++++++++++++++++++++
 odb/source.h | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 91 insertions(+), 69 deletions(-)

diff --git a/Makefile b/Makefile
index 47ed9fa7fd..116358e484 100644
--- a/Makefile
+++ b/Makefile
@@ -1214,6 +1214,7 @@ LIB_OBJS += object-file.o
 LIB_OBJS += object-name.o
 LIB_OBJS += object.o
 LIB_OBJS += odb.o
+LIB_OBJS += odb/source.o
 LIB_OBJS += odb/streaming.o
 LIB_OBJS += oid-array.o
 LIB_OBJS += oidmap.o
diff --git a/meson.build b/meson.build
index 3a1d12caa4..1018af17c3 100644
--- a/meson.build
+++ b/meson.build
@@ -397,6 +397,7 @@ libgit_sources = [
   'object-name.c',
   'object.c',
   'odb.c',
+  'odb/source.c',
   'odb/streaming.c',
   'oid-array.c',
   'oidmap.c',
diff --git a/odb.c b/odb.c
index 776de5356c..d318482d47 100644
--- a/odb.c
+++ b/odb.c
@@ -217,23 +217,6 @@ static void odb_source_read_alternates(struct odb_source *source,
 	free(path);
 }
 
-
-static struct odb_source *odb_source_new(struct object_database *odb,
-					 const char *path,
-					 bool local)
-{
-	struct odb_source *source;
-
-	CALLOC_ARRAY(source, 1);
-	source->odb = odb;
-	source->local = local;
-	source->path = xstrdup(path);
-	source->loose = odb_source_loose_new(source);
-	source->packfiles = packfile_store_new(source);
-
-	return source;
-}
-
 static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,
 							const char *source,
 							int depth)
@@ -373,14 +356,6 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,
 	return source->next;
 }
 
-static void odb_source_free(struct odb_source *source)
-{
-	free(source->path);
-	odb_source_loose_free(source->loose);
-	packfile_store_free(source->packfiles);
-	free(source);
-}
-
 void odb_restore_primary_source(struct object_database *odb,
 				struct odb_source *restore_source,
 				const char *old_path)
diff --git a/odb.h b/odb.h
index 68b8ec2289..e13b5b7c44 100644
--- a/odb.h
+++ b/odb.h
@@ -3,6 +3,7 @@
 
 #include "hashmap.h"
 #include "object.h"
+#include "odb/source.h"
 #include "oidset.h"
 #include "oidmap.h"
 #include "string-list.h"
@@ -30,50 +31,6 @@ extern int fetch_if_missing;
  */
 char *compute_alternate_path(const char *path, struct strbuf *err);
 
-/*
- * The source is the part of the object database that stores the actual
- * objects. It thus encapsulates the logic to read and write the specific
- * on-disk format. An object database can have multiple sources:
- *
- *   - The primary source, which is typically located in "$GIT_DIR/objects".
- *     This is where new objects are usually written to.
- *
- *   - Alternate sources, which are configured via "objects/info/alternates" or
- *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These
- *     alternate sources are only used to read objects.
- */
-struct odb_source {
-	struct odb_source *next;
-
-	/* Object database that owns this object source. */
-	struct object_database *odb;
-
-	/* Private state for loose objects. */
-	struct odb_source_loose *loose;
-
-	/* Should only be accessed directly by packfile.c and midx.c. */
-	struct packfile_store *packfiles;
-
-	/*
-	 * Figure out whether this is the local source of the owning
-	 * repository, which would typically be its ".git/objects" directory.
-	 * This local object directory is usually where objects would be
-	 * written to.
-	 */
-	bool local;
-
-	/*
-	 * This object store is ephemeral, so there is no need to fsync.
-	 */
-	int will_destroy;
-
-	/*
-	 * Path to the source. If this is a relative path, it is relative to
-	 * the current working directory.
-	 */
-	char *path;
-};
-
 struct packed_git;
 struct packfile_store;
 struct cached_object_entry;
diff --git a/odb/source.c b/odb/source.c
new file mode 100644
index 0000000000..7fc89806f9
--- /dev/null
+++ b/odb/source.c
@@ -0,0 +1,28 @@
+#include "git-compat-util.h"
+#include "object-file.h"
+#include "odb/source.h"
+#include "packfile.h"
+
+struct odb_source *odb_source_new(struct object_database *odb,
+				  const char *path,
+				  bool local)
+{
+	struct odb_source *source;
+
+	CALLOC_ARRAY(source, 1);
+	source->odb = odb;
+	source->local = local;
+	source->path = xstrdup(path);
+	source->loose = odb_source_loose_new(source);
+	source->packfiles = packfile_store_new(source);
+
+	return source;
+}
+
+void odb_source_free(struct odb_source *source)
+{
+	free(source->path);
+	odb_source_loose_free(source->loose);
+	packfile_store_free(source->packfiles);
+	free(source);
+}
diff --git a/odb/source.h b/odb/source.h
new file mode 100644
index 0000000000..391d6d1e38
--- /dev/null
+++ b/odb/source.h
@@ -0,0 +1,60 @@
+#ifndef ODB_SOURCE_H
+#define ODB_SOURCE_H
+
+/*
+ * The source is the part of the object database that stores the actual
+ * objects. It thus encapsulates the logic to read and write the specific
+ * on-disk format. An object database can have multiple sources:
+ *
+ *   - The primary source, which is typically located in "$GIT_DIR/objects".
+ *     This is where new objects are usually written to.
+ *
+ *   - Alternate sources, which are configured via "objects/info/alternates" or
+ *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These
+ *     alternate sources are only used to read objects.
+ */
+struct odb_source {
+	struct odb_source *next;
+
+	/* Object database that owns this object source. */
+	struct object_database *odb;
+
+	/* Private state for loose objects. */
+	struct odb_source_loose *loose;
+
+	/* Should only be accessed directly by packfile.c and midx.c. */
+	struct packfile_store *packfiles;
+
+	/*
+	 * Figure out whether this is the local source of the owning
+	 * repository, which would typically be its ".git/objects" directory.
+	 * This local object directory is usually where objects would be
+	 * written to.
+	 */
+	bool local;
+
+	/*
+	 * This object store is ephemeral, so there is no need to fsync.
+	 */
+	int will_destroy;
+
+	/*
+	 * Path to the source. If this is a relative path, it is relative to
+	 * the current working directory.
+	 */
+	char *path;
+};
+
+/*
+ * Allocate and initialize a new source for the given object database located
+ * at `path`. `local` indicates whether or not the source is the local and thus
+ * primary object source of the object database.
+ */
+struct odb_source *odb_source_new(struct object_database *odb,
+				  const char *path,
+				  bool local);
+
+/* Free the object database source, releasing all associated resources. */
+void odb_source_free(struct odb_source *source);
+
+#endif

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 01/17] odb: split `struct odb_source` into separate header

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> Subsequent commits will expand the `struct odb_source` to become a
> generic interface for accessing an object database source. As part of
> these refactorings we'll add a set of function pointers that will
> significantly expand the structure overall.
> 
> Prepare for this by splitting out the `struct odb_source` into a
> separate header. This keeps the high-level object database interface
> detached from the low-level object database sources.

This certainly seems sensible to me. I've been thinking about also
splitting out ODB transactions into a separate header. I may do
something similar in the future.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> diff --git a/odb.h b/odb.h
> index 68b8ec2289..e13b5b7c44 100644
> --- a/odb.h
> +++ b/odb.h
> @@ -3,6 +3,7 @@
>  
>  #include "hashmap.h"
>  #include "object.h"
> +#include "odb/source.h"

Out of curiousity, since we include the header here, it is transitively
included wherever we are using `struct odb_source`. Ideally should we be
explicit or would it be best to just rely on this transitively?

The rest of this patch looks good.

-Justin

Re: [PATCH 01/17] odb: split `struct odb_source` into separate header

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 09:55:11AM -0600, Justin Tobler wrote:
> > diff --git a/odb.h b/odb.h
> > index 68b8ec2289..e13b5b7c44 100644
> > --- a/odb.h
> > +++ b/odb.h
> > @@ -3,6 +3,7 @@
> >  
> >  #include "hashmap.h"
> >  #include "object.h"
> > +#include "odb/source.h"
> 
> Out of curiousity, since we include the header here, it is transitively
> included wherever we are using `struct odb_source`. Ideally should we be
> explicit or would it be best to just rely on this transitively?

Hum, dunno. I think it's fine to just be pragmatic here and only include
"odb.h"?

Patrick

Re: [PATCH 01/17] odb: split `struct odb_source` into separate header

[Justin Tobler]

On 26/03/05 02:23PM, Patrick Steinhardt wrote:
> On Wed, Mar 04, 2026 at 09:55:11AM -0600, Justin Tobler wrote:
> > > diff --git a/odb.h b/odb.h
> > > index 68b8ec2289..e13b5b7c44 100644
> > > --- a/odb.h
> > > +++ b/odb.h
> > > @@ -3,6 +3,7 @@
> > >  
> > >  #include "hashmap.h"
> > >  #include "object.h"
> > > +#include "odb/source.h"
> > 
> > Out of curiousity, since we include the header here, it is transitively
> > included wherever we are using `struct odb_source`. Ideally should we be
> > explicit or would it be best to just rely on this transitively?
> 
> Hum, dunno. I think it's fine to just be pragmatic here and only include
> "odb.h"?

Ya sounds completely fair. I was mostly curious if we intended "odb.h"
to server as the entry point here and expected "odb/source.h" to be
"internal". This is certainly fine though.

-Justin

[PATCH 02/17] odb: introduce "files" source

[Patrick Steinhardt]

Introduce a new "files" object database source. This source encapsulates
access to both loose object files and the packfile store, similar to how
the "files" backend for refs encapsulates access to loose refs and the
packed-refs file.

Note that for now the "files" source is still a direct member of a
`struct odb_source`. This architecture will be reversed in the next
commit so that the files source contains a `struct odb_source`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Makefile               |  1 +
 builtin/cat-file.c     |  2 +-
 builtin/fast-import.c  |  6 +++---
 builtin/grep.c         |  2 +-
 builtin/index-pack.c   |  2 +-
 builtin/pack-objects.c |  8 ++++----
 commit-graph.c         |  2 +-
 http.c                 |  2 +-
 loose.c                | 18 +++++++++---------
 meson.build            |  1 +
 midx.c                 | 18 +++++++++---------
 object-file.c          | 24 ++++++++++++------------
 odb.c                  | 12 ++++++------
 odb/source-files.c     | 23 +++++++++++++++++++++++
 odb/source-files.h     | 24 ++++++++++++++++++++++++
 odb/source.c           |  6 ++----
 odb/source.h           |  9 ++++-----
 odb/streaming.c        |  2 +-
 packfile.c             | 16 ++++++++--------
 packfile.h             |  4 ++--
 20 files changed, 114 insertions(+), 68 deletions(-)

diff --git a/Makefile b/Makefile
index 116358e484..c05285399c 100644
--- a/Makefile
+++ b/Makefile
@@ -1215,6 +1215,7 @@ LIB_OBJS += object-name.o
 LIB_OBJS += object.o
 LIB_OBJS += odb.o
 LIB_OBJS += odb/source.o
+LIB_OBJS += odb/source-files.o
 LIB_OBJS += odb/streaming.o
 LIB_OBJS += oid-array.o
 LIB_OBJS += oidmap.o
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 53ffe80c79..01a53f3f29 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -882,7 +882,7 @@ static void batch_each_object(struct batch_options *opt,
 		struct object_info oi = { 0 };
 
 		for (source = the_repository->objects->sources; source; source = source->next) {
-			int ret = packfile_store_for_each_object(source->packfiles, &oi,
+			int ret = packfile_store_for_each_object(source->files->packed, &oi,
 								 batch_one_object_oi, &payload, flags);
 			if (ret)
 				break;
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index b8a7757cfd..627dcbf4f3 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -900,7 +900,7 @@ static void end_packfile(void)
 		idx_name = keep_pack(create_index());
 
 		/* Register the packfile with core git's machinery. */
-		new_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,
+		new_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,
 						 idx_name, 1);
 		if (!new_p)
 			die(_("core Git rejected index %s"), idx_name);
@@ -982,7 +982,7 @@ static int store_object(
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))
+		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
 			continue;
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
@@ -1187,7 +1187,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))
+		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
 			continue;
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
diff --git a/builtin/grep.c b/builtin/grep.c
index 5b8b87b1ac..c8d0e51415 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -1219,7 +1219,7 @@ int cmd_grep(int argc,
 
 			odb_prepare_alternates(the_repository->objects);
 			for (source = the_repository->objects->sources; source; source = source->next)
-				packfile_store_prepare(source->packfiles);
+				packfile_store_prepare(source->files->packed);
 		}
 
 		start_threads(&opt);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index b67fb0256c..f0cce534b2 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
 			    hash, "idx", 1);
 
 	if (do_fsck_object && startup_info->have_repository)
-		packfile_store_load_pack(the_repository->objects->sources->packfiles,
+		packfile_store_load_pack(the_repository->objects->sources->files->packed,
 					 final_index_name, 0);
 
 	if (!from_stdin) {
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 242d1c68f0..0c3c01cdc9 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1531,7 +1531,7 @@ static int want_cruft_object_mtime(struct repository *r,
 	struct odb_source *source;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		struct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);
+		struct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
@@ -1753,11 +1753,11 @@ static int want_object_in_pack_mtime(const struct object_id *oid,
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		for (e = source->packfiles->packs.head; e; e = e->next) {
+		for (e = source->files->packed->packs.head; e; e = e->next) {
 			struct packed_git *p = e->pack;
 			want = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);
 			if (!exclude && want > 0)
-				packfile_list_prepend(&source->packfiles->packs, p);
+				packfile_list_prepend(&source->files->packed->packs, p);
 			if (want != -1)
 				return want;
 		}
@@ -4340,7 +4340,7 @@ static void add_objects_in_unpacked_packs(void)
 		if (!source->local)
 			continue;
 
-		if (packfile_store_for_each_object(source->packfiles, &oi,
+		if (packfile_store_for_each_object(source->files->packed, &oi,
 						   add_object_in_unpacked_pack, NULL,
 						   ODB_FOR_EACH_OBJECT_PACK_ORDER |
 						   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |
diff --git a/commit-graph.c b/commit-graph.c
index d250a729b1..967eb77047 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1981,7 +1981,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)
 
 	odb_prepare_alternates(ctx->r->objects);
 	for (source = ctx->r->objects->sources; source; source = source->next)
-		packfile_store_for_each_object(source->packfiles, &oi, add_packed_commits_oi,
+		packfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,
 					       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);
 
 	if (ctx->progress_done < ctx->approx_nr_objects)
diff --git a/http.c b/http.c
index 7815f144de..b44f493919 100644
--- a/http.c
+++ b/http.c
@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,
 			   struct packfile_list *list_to_remove_from)
 {
 	packfile_list_remove(list_to_remove_from, p);
-	packfile_store_add_pack(the_repository->objects->sources->packfiles, p);
+	packfile_store_add_pack(the_repository->objects->sources->files->packed, p);
 }
 
 struct http_pack_request *new_http_pack_request(
diff --git a/loose.c b/loose.c
index 56cf64b648..c921d46b94 100644
--- a/loose.c
+++ b/loose.c
@@ -49,13 +49,13 @@ static int insert_loose_map(struct odb_source *source,
 			    const struct object_id *oid,
 			    const struct object_id *compat_oid)
 {
-	struct loose_object_map *map = source->loose->map;
+	struct loose_object_map *map = source->files->loose->map;
 	int inserted = 0;
 
 	inserted |= insert_oid_pair(map->to_compat, oid, compat_oid);
 	inserted |= insert_oid_pair(map->to_storage, compat_oid, oid);
 	if (inserted)
-		oidtree_insert(source->loose->cache, compat_oid);
+		oidtree_insert(source->files->loose->cache, compat_oid);
 
 	return inserted;
 }
@@ -65,11 +65,11 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source
 	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
 	FILE *fp;
 
-	if (!source->loose->map)
-		loose_object_map_init(&source->loose->map);
-	if (!source->loose->cache) {
-		ALLOC_ARRAY(source->loose->cache, 1);
-		oidtree_init(source->loose->cache);
+	if (!source->files->loose->map)
+		loose_object_map_init(&source->files->loose->map);
+	if (!source->files->loose->cache) {
+		ALLOC_ARRAY(source->files->loose->cache, 1);
+		oidtree_init(source->files->loose->cache);
 	}
 
 	insert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)
 
 int repo_write_loose_object_map(struct repository *repo)
 {
-	kh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;
+	kh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;
 	struct lock_file lock;
 	int fd;
 	khiter_t iter;
@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,
 	khiter_t pos;
 
 	for (source = repo->objects->sources; source; source = source->next) {
-		struct loose_object_map *loose_map = source->loose->map;
+		struct loose_object_map *loose_map = source->files->loose->map;
 		if (!loose_map)
 			continue;
 		map = (to == repo->compat_hash_algo) ?
diff --git a/meson.build b/meson.build
index 1018af17c3..8e1125a585 100644
--- a/meson.build
+++ b/meson.build
@@ -398,6 +398,7 @@ libgit_sources = [
   'object.c',
   'odb.c',
   'odb/source.c',
+  'odb/source-files.c',
   'odb/streaming.c',
   'oid-array.c',
   'oidmap.c',
diff --git a/midx.c b/midx.c
index a75ea99a0d..698d10a1c6 100644
--- a/midx.c
+++ b/midx.c
@@ -95,8 +95,8 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,
 
 struct multi_pack_index *get_multi_pack_index(struct odb_source *source)
 {
-	packfile_store_prepare(source->packfiles);
-	return source->packfiles->midx;
+	packfile_store_prepare(source->files->packed);
+	return source->files->packed->midx;
 }
 
 static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,
@@ -459,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,
 
 	strbuf_addf(&pack_name, "%s/pack/%s", m->source->path,
 		    m->pack_names[pack_int_id]);
-	p = packfile_store_load_pack(m->source->packfiles,
+	p = packfile_store_load_pack(m->source->files->packed,
 				     pack_name.buf, m->source->local);
 	strbuf_release(&pack_name);
 
@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)
 	if (!r->settings.core_multi_pack_index)
 		return 0;
 
-	if (source->packfiles->midx)
+	if (source->files->packed->midx)
 		return 1;
 
-	source->packfiles->midx = load_multi_pack_index(source);
+	source->files->packed->midx = load_multi_pack_index(source);
 
-	return !!source->packfiles->midx;
+	return !!source->files->packed->midx;
 }
 
 int midx_checksum_valid(struct multi_pack_index *m)
@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)
 		struct odb_source *source;
 
 		for (source = r->objects->sources; source; source = source->next) {
-			if (source->packfiles->midx)
-				close_midx(source->packfiles->midx);
-			source->packfiles->midx = NULL;
+			if (source->files->packed->midx)
+				close_midx(source->files->packed->midx);
+			source->files->packed->midx = NULL;
 		}
 	}
 
diff --git a/object-file.c b/object-file.c
index a72048dfdc..ec04d3572a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -220,7 +220,7 @@ static void *odb_source_loose_map_object(struct odb_source *source,
 					 unsigned long *size)
 {
 	const char *p;
-	int fd = open_loose_object(source->loose, oid, &p);
+	int fd = open_loose_object(source->files->loose, oid, &p);
 
 	if (fd < 0)
 		return NULL;
@@ -423,7 +423,7 @@ static int read_object_info_from_path(struct odb_source *source,
 		struct stat st;
 
 		if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
-			ret = quick_has_loose(source->loose, oid) ? 0 : -1;
+			ret = quick_has_loose(source->files->loose, oid) ? 0 : -1;
 			goto out;
 		}
 
@@ -1868,31 +1868,31 @@ struct oidtree *odb_source_loose_cache(struct odb_source *source,
 {
 	int subdir_nr = oid->hash[0];
 	struct strbuf buf = STRBUF_INIT;
-	size_t word_bits = bitsizeof(source->loose->subdir_seen[0]);
+	size_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);
 	size_t word_index = subdir_nr / word_bits;
 	size_t mask = (size_t)1u << (subdir_nr % word_bits);
 	uint32_t *bitmap;
 
 	if (subdir_nr < 0 ||
-	    (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))
+	    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))
 		BUG("subdir_nr out of range");
 
-	bitmap = &source->loose->subdir_seen[word_index];
+	bitmap = &source->files->loose->subdir_seen[word_index];
 	if (*bitmap & mask)
-		return source->loose->cache;
-	if (!source->loose->cache) {
-		ALLOC_ARRAY(source->loose->cache, 1);
-		oidtree_init(source->loose->cache);
+		return source->files->loose->cache;
+	if (!source->files->loose->cache) {
+		ALLOC_ARRAY(source->files->loose->cache, 1);
+		oidtree_init(source->files->loose->cache);
 	}
 	strbuf_addstr(&buf, source->path);
 	for_each_file_in_obj_subdir(subdir_nr, &buf,
 				    source->odb->repo->hash_algo,
 				    append_loose_object,
 				    NULL, NULL,
-				    source->loose->cache);
+				    source->files->loose->cache);
 	*bitmap |= mask;
 	strbuf_release(&buf);
-	return source->loose->cache;
+	return source->files->loose->cache;
 }
 
 static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
@@ -1905,7 +1905,7 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
 
 void odb_source_loose_reprepare(struct odb_source *source)
 {
-	odb_source_loose_clear_cache(source->loose);
+	odb_source_loose_clear_cache(source->files->loose);
 }
 
 static int check_stream_oid(git_zstream *stream,
diff --git a/odb.c b/odb.c
index d318482d47..c9ebc7e741 100644
--- a/odb.c
+++ b/odb.c
@@ -691,7 +691,7 @@ static int do_oid_object_info_extended(struct object_database *odb,
 
 		/* Most likely it's a loose object. */
 		for (source = odb->sources; source; source = source->next) {
-			if (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||
+			if (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||
 			    !odb_source_loose_read_object_info(source, real, oi, flags))
 				return 0;
 		}
@@ -700,7 +700,7 @@ static int do_oid_object_info_extended(struct object_database *odb,
 		if (!(flags & OBJECT_INFO_QUICK)) {
 			odb_reprepare(odb->repo->objects);
 			for (source = odb->sources; source; source = source->next)
-				if (!packfile_store_read_object_info(source->packfiles, real, oi, flags))
+				if (!packfile_store_read_object_info(source->files->packed, real, oi, flags))
 					return 0;
 		}
 
@@ -962,7 +962,7 @@ int odb_freshen_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (packfile_store_freshen_object(source->packfiles, oid))
+		if (packfile_store_freshen_object(source->files->packed, oid))
 			return 1;
 
 		if (odb_source_loose_freshen_object(source, oid))
@@ -992,7 +992,7 @@ int odb_for_each_object(struct object_database *odb,
 				return ret;
 		}
 
-		ret = packfile_store_for_each_object(source->packfiles, request,
+		ret = packfile_store_for_each_object(source->files->packed, request,
 						     cb, cb_data, flags);
 		if (ret)
 			return ret;
@@ -1091,7 +1091,7 @@ void odb_close(struct object_database *o)
 {
 	struct odb_source *source;
 	for (source = o->sources; source; source = source->next)
-		packfile_store_close(source->packfiles);
+		packfile_store_close(source->files->packed);
 	close_commit_graph(o);
 }
 
@@ -1149,7 +1149,7 @@ void odb_reprepare(struct object_database *o)
 
 	for (source = o->sources; source; source = source->next) {
 		odb_source_loose_reprepare(source);
-		packfile_store_reprepare(source->packfiles);
+		packfile_store_reprepare(source->files->packed);
 	}
 
 	o->approximate_object_count_valid = 0;
diff --git a/odb/source-files.c b/odb/source-files.c
new file mode 100644
index 0000000000..cbdaa6850f
--- /dev/null
+++ b/odb/source-files.c
@@ -0,0 +1,23 @@
+#include "git-compat-util.h"
+#include "object-file.h"
+#include "odb/source-files.h"
+#include "packfile.h"
+
+void odb_source_files_free(struct odb_source_files *files)
+{
+	if (!files)
+		return;
+	odb_source_loose_free(files->loose);
+	packfile_store_free(files->packed);
+	free(files);
+}
+
+struct odb_source_files *odb_source_files_new(struct odb_source *source)
+{
+	struct odb_source_files *files;
+	CALLOC_ARRAY(files, 1);
+	files->source = source;
+	files->loose = odb_source_loose_new(source);
+	files->packed = packfile_store_new(source);
+	return files;
+}
diff --git a/odb/source-files.h b/odb/source-files.h
new file mode 100644
index 0000000000..0b8bf773ca
--- /dev/null
+++ b/odb/source-files.h
@@ -0,0 +1,24 @@
+#ifndef ODB_SOURCE_FILES_H
+#define ODB_SOURCE_FILES_H
+
+struct odb_source_loose;
+struct odb_source;
+struct packfile_store;
+
+/*
+ * The files object database source uses a combination of loose objects and
+ * packfiles. It is the default backend used by Git to store objects.
+ */
+struct odb_source_files {
+	struct odb_source *source;
+	struct odb_source_loose *loose;
+	struct packfile_store *packed;
+};
+
+/* Allocate and initialize a new object source. */
+struct odb_source_files *odb_source_files_new(struct odb_source *source);
+
+/* Free the object source and release all associated resources. */
+void odb_source_files_free(struct odb_source_files *files);
+
+#endif
diff --git a/odb/source.c b/odb/source.c
index 7fc89806f9..9d7fd19f45 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -13,8 +13,7 @@ struct odb_source *odb_source_new(struct object_database *odb,
 	source->odb = odb;
 	source->local = local;
 	source->path = xstrdup(path);
-	source->loose = odb_source_loose_new(source);
-	source->packfiles = packfile_store_new(source);
+	source->files = odb_source_files_new(source);
 
 	return source;
 }
@@ -22,7 +21,6 @@ struct odb_source *odb_source_new(struct object_database *odb,
 void odb_source_free(struct odb_source *source)
 {
 	free(source->path);
-	odb_source_loose_free(source->loose);
-	packfile_store_free(source->packfiles);
+	odb_source_files_free(source->files);
 	free(source);
 }
diff --git a/odb/source.h b/odb/source.h
index 391d6d1e38..1c34265189 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,6 +1,8 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
+#include "odb/source-files.h"
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -19,11 +21,8 @@ struct odb_source {
 	/* Object database that owns this object source. */
 	struct object_database *odb;
 
-	/* Private state for loose objects. */
-	struct odb_source_loose *loose;
-
-	/* Should only be accessed directly by packfile.c and midx.c. */
-	struct packfile_store *packfiles;
+	/* The backend used to store objects. */
+	struct odb_source_files *files;
 
 	/*
 	 * Figure out whether this is the local source of the owning
diff --git a/odb/streaming.c b/odb/streaming.c
index 4a4474f891..26b0a1a0f5 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -187,7 +187,7 @@ static int istream_source(struct odb_read_stream **out,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (!packfile_store_read_object_stream(out, source->packfiles, oid) ||
+		if (!packfile_store_read_object_stream(out, source->files->packed, oid) ||
 		    !odb_source_loose_read_object_stream(out, source, oid))
 			return 0;
 	}
diff --git a/packfile.c b/packfile.c
index ce837f852a..4e1f6087ed 100644
--- a/packfile.c
+++ b/packfile.c
@@ -363,7 +363,7 @@ static int unuse_one_window(struct object_database *odb)
 	struct pack_window *lru_w = NULL, *lru_l = NULL;
 
 	for (source = odb->sources; source; source = source->next)
-		for (e = source->packfiles->packs.head; e; e = e->next)
+		for (e = source->files->packed->packs.head; e; e = e->next)
 			scan_windows(e->pack, &lru_p, &lru_w, &lru_l);
 
 	if (lru_p) {
@@ -537,7 +537,7 @@ static int close_one_pack(struct repository *r)
 	int accept_windows_inuse = 1;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		for (e = source->packfiles->packs.head; e; e = e->next) {
+		for (e = source->files->packed->packs.head; e; e = e->next) {
 			if (e->pack->pack_fd == -1)
 				continue;
 			find_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);
@@ -990,10 +990,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	size_t base_len = full_name_len;
 
 	if (strip_suffix_mem(full_name, &base_len, ".idx") &&
-	    !(data->source->packfiles->midx &&
-	      midx_contains_pack(data->source->packfiles->midx, file_name))) {
+	    !(data->source->files->packed->midx &&
+	      midx_contains_pack(data->source->files->packed->midx, file_name))) {
 		char *trimmed_path = xstrndup(full_name, full_name_len);
-		packfile_store_load_pack(data->source->packfiles,
+		packfile_store_load_pack(data->source->files->packed,
 					 trimmed_path, data->source->local);
 		free(trimmed_path);
 	}
@@ -1248,7 +1248,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,
 
 	for (source = r->objects->sources; source; source = source->next) {
 		struct packfile_list_entry *e;
-		for (e = source->packfiles->packs.head; e; e = e->next)
+		for (e = source->files->packed->packs.head; e; e = e->next)
 			if (oidset_contains(&e->pack->bad_objects, oid))
 				return e->pack;
 	}
@@ -2254,7 +2254,7 @@ int has_object_pack(struct repository *r, const struct object_id *oid)
 
 	odb_prepare_alternates(r->objects);
 	for (source = r->objects->sources; source; source = source->next) {
-		int ret = find_pack_entry(source->packfiles, oid, &e);
+		int ret = find_pack_entry(source->files->packed, oid, &e);
 		if (ret)
 			return ret;
 	}
@@ -2271,7 +2271,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,
 	for (source = r->objects->sources; source; source = source->next) {
 		struct packed_git **cache;
 
-		cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);
+		cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
diff --git a/packfile.h b/packfile.h
index 224142fd34..e8de06ee86 100644
--- a/packfile.h
+++ b/packfile.h
@@ -192,7 +192,7 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct
 	odb_prepare_alternates(repo->objects);
 
 	for (struct odb_source *source = repo->objects->sources; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);
+		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
 		if (!entry)
 			continue;
 		data.source = source;
@@ -212,7 +212,7 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *
 		return;
 
 	for (source = data->source->next; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);
+		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
 		if (!entry)
 			continue;
 		data->source = source;

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 02/17] odb: introduce "files" source

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> Introduce a new "files" object database source. This source encapsulates
> access to both loose object files and the packfile store, similar to how
> the "files" backend for refs encapsulates access to loose refs and the
> packed-refs file.

Makes sense.

> Note that for now the "files" source is still a direct member of a
> `struct odb_source`. This architecture will be reversed in the next
> commit so that the files source contains a `struct odb_source`.

Ok so for now all ODB operations are going to reach directly into the
contained "files" source.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> diff --git a/odb/source-files.h b/odb/source-files.h
> new file mode 100644
> index 0000000000..0b8bf773ca
> --- /dev/null
> +++ b/odb/source-files.h
> @@ -0,0 +1,24 @@
> +#ifndef ODB_SOURCE_FILES_H
> +#define ODB_SOURCE_FILES_H
> +
> +struct odb_source_loose;
> +struct odb_source;
> +struct packfile_store;
> +
> +/*
> + * The files object database source uses a combination of loose objects and
> + * packfiles. It is the default backend used by Git to store objects.
> + */
> +struct odb_source_files {
> +	struct odb_source *source;

I don't think we use this anywhere yet, but I suspect this is the
placeholder for the "base" ODB source.

> +	struct odb_source_loose *loose;
> +	struct packfile_store *packed;

So with this patch we are really just moving odb_source_loose and
packfile_store into `struct odb_source_files`. Most of the other changes
are just fallout from this structural change.

> +};
> +
> +/* Allocate and initialize a new object source. */
> +struct odb_source_files *odb_source_files_new(struct odb_source *source);
> +
> +/* Free the object source and release all associated resources. */
> +void odb_source_files_free(struct odb_source_files *files);
> +
> +#endif
[snip]
> diff --git a/odb/source.h b/odb/source.h
> index 391d6d1e38..1c34265189 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -1,6 +1,8 @@
>  #ifndef ODB_SOURCE_H
>  #define ODB_SOURCE_H
>  
> +#include "odb/source-files.h"
> +
>  /*
>   * The source is the part of the object database that stores the actual
>   * objects. It thus encapsulates the logic to read and write the specific
> @@ -19,11 +21,8 @@ struct odb_source {
>  	/* Object database that owns this object source. */
>  	struct object_database *odb;
>  
> -	/* Private state for loose objects. */
> -	struct odb_source_loose *loose;
> -
> -	/* Should only be accessed directly by packfile.c and midx.c. */

Is there any value to keeping this comment around?

> -	struct packfile_store *packfiles;
> +	/* The backend used to store objects. */
> +	struct odb_source_files *files;

For now we store a direct reference to the "files" ODB source, but I
assume in the future this won't be the case and instead will cast the
"base" ODB source into its concrete type as needed.

-Justin

Re: [PATCH 02/17] odb: introduce "files" source

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 10:57:26AM -0600, Justin Tobler wrote:
> On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > diff --git a/odb/source.h b/odb/source.h
> > index 391d6d1e38..1c34265189 100644
> > --- a/odb/source.h
> > +++ b/odb/source.h
> > @@ -19,11 +21,8 @@ struct odb_source {
> >  	/* Object database that owns this object source. */
> >  	struct object_database *odb;
> >  
> > -	/* Private state for loose objects. */
> > -	struct odb_source_loose *loose;
> > -
> > -	/* Should only be accessed directly by packfile.c and midx.c. */
> 
> Is there any value to keeping this comment around?

I don't think so. With this series it becomes clear that all of the info
in the sources become private implementation details, and future patch
series will double down on that even further.

Patrick

Re: [PATCH 02/17] odb: introduce "files" source

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> Introduce a new "files" object database source. This source encapsulates
> access to both loose object files and the packfile store, similar to how
> the "files" backend for refs encapsulates access to loose refs and the
> packed-refs file.
>
> Note that for now the "files" source is still a direct member of a
> `struct odb_source`. This architecture will be reversed in the next
> commit so that the files source contains a `struct odb_source`.
>

Okay, so peeking ahead, we will follow the same format as in the refs
DB, but this is an intermediate step in that direction.


> diff --git a/odb/source-files.c b/odb/source-files.c
> new file mode 100644
> index 0000000000..cbdaa6850f
> --- /dev/null
> +++ b/odb/source-files.c
> @@ -0,0 +1,23 @@
> +#include "git-compat-util.h"
> +#include "object-file.h"
> +#include "odb/source-files.h"
> +#include "packfile.h"
> +
> +void odb_source_files_free(struct odb_source_files *files)
> +{
> +	if (!files)
> +		return;
> +	odb_source_loose_free(files->loose);
> +	packfile_store_free(files->packed);
> +	free(files);
> +}
> +
> +struct odb_source_files *odb_source_files_new(struct odb_source *source)
> +{
> +	struct odb_source_files *files;
> +	CALLOC_ARRAY(files, 1);
> +	files->source = source;
> +	files->loose = odb_source_loose_new(source);
> +	files->packed = packfile_store_new(source);

Instead of defining `loose` and `packed` as part of the `obd_source`, we
move it specifically to the `obd_source->files`.

> +	return files;
> +}

The rest of the patch is just variable swapping, makes sense!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH 03/17] odb: embed base source in the "files" backend

[Patrick Steinhardt]

The "files" backend is implemented as a pointer in the `struct
odb_source`. This contradicts our typical pattern for pluggable backends
like we use it for example in the ref store or for object database
streams, where we typically embed the generic base structure in the
specialized implementation. This pattern has a couple of small benefits:

  - We avoid an extra allocation.

  - We hide implementation details in the generic structure.

  - We can easily downcast from a generic backend to the specialized
    structure and vice versa because the offsets are known at compile
    time.

  - It becomes trivial to identify locations where we depend on backend
    specific logic because the cast needs to be explicit.

Refactor our "files" object database source to do the same and embed the
`struct odb_source` in the `struct odb_source_files`.

There are still a bunch of sites in our code base where we do have to
access internals of the "files" backend. The intent is that those will
go away over time, but this will certainly take a while. Meanwhile,
provide a `odb_source_files_downcast()` function that can convert a
generic source into a "files" source.

As we only have a single source the downcast succeeds unconditionally
for now. Eventually though the intent is to make the cast `BUG()` in
case the caller requests to downcast a non-"files" backend to a "files"
backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/cat-file.c     |  3 ++-
 builtin/fast-import.c  | 12 ++++++++----
 builtin/grep.c         |  6 ++++--
 builtin/index-pack.c   |  8 +++++---
 builtin/pack-objects.c | 13 +++++++++----
 commit-graph.c         |  6 ++++--
 http.c                 |  3 ++-
 loose.c                | 23 ++++++++++++++---------
 midx.c                 | 26 +++++++++++++++-----------
 object-file.c          | 28 ++++++++++++++++------------
 odb.c                  | 26 ++++++++++++++++++--------
 odb/source-files.c     | 14 ++++++++++----
 odb/source-files.h     | 18 +++++++++++++++---
 odb/source.c           | 26 +++++++++++++++++++-------
 odb/source.h           | 31 +++++++++++++++++++++++++------
 odb/streaming.c        |  3 ++-
 packfile.c             | 26 +++++++++++++++++---------
 packfile.h             |  7 +++++--
 18 files changed, 190 insertions(+), 89 deletions(-)

diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 01a53f3f29..0c68d61b91 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -882,7 +882,8 @@ static void batch_each_object(struct batch_options *opt,
 		struct object_info oi = { 0 };
 
 		for (source = the_repository->objects->sources; source; source = source->next) {
-			int ret = packfile_store_for_each_object(source->files->packed, &oi,
+			struct odb_source_files *files = odb_source_files_downcast(source);
+			int ret = packfile_store_for_each_object(files->packed, &oi,
 								 batch_one_object_oi, &payload, flags);
 			if (ret)
 				break;
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 627dcbf4f3..a41f95191e 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -875,6 +875,7 @@ static void end_packfile(void)
 	running = 1;
 	clear_delta_base_cache();
 	if (object_count) {
+		struct odb_source_files *files = odb_source_files_downcast(pack_data->repo->objects->sources);
 		struct packed_git *new_p;
 		struct object_id cur_pack_oid;
 		char *idx_name;
@@ -900,8 +901,7 @@ static void end_packfile(void)
 		idx_name = keep_pack(create_index());
 
 		/* Register the packfile with core git's machinery. */
-		new_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,
-						 idx_name, 1);
+		new_p = packfile_store_load_pack(files->packed, idx_name, 1);
 		if (!new_p)
 			die(_("core Git rejected index %s"), idx_name);
 		all_packs[pack_id] = new_p;
@@ -982,7 +982,9 @@ static int store_object(
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		if (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))
 			continue;
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
@@ -1187,7 +1189,9 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		if (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))
 			continue;
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
diff --git a/builtin/grep.c b/builtin/grep.c
index c8d0e51415..61379909b8 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -1218,8 +1218,10 @@ int cmd_grep(int argc,
 			struct odb_source *source;
 
 			odb_prepare_alternates(the_repository->objects);
-			for (source = the_repository->objects->sources; source; source = source->next)
-				packfile_store_prepare(source->files->packed);
+			for (source = the_repository->objects->sources; source; source = source->next) {
+				struct odb_source_files *files = odb_source_files_downcast(source);
+				packfile_store_prepare(files->packed);
+			}
 		}
 
 		start_threads(&opt);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index f0cce534b2..d1e47279a8 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1637,9 +1637,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
 	rename_tmp_packfile(&final_index_name, curr_index_name, &index_name,
 			    hash, "idx", 1);
 
-	if (do_fsck_object && startup_info->have_repository)
-		packfile_store_load_pack(the_repository->objects->sources->files->packed,
-					 final_index_name, 0);
+	if (do_fsck_object && startup_info->have_repository) {
+		struct odb_source_files *files =
+			odb_source_files_downcast(the_repository->objects->sources);
+		packfile_store_load_pack(files->packed, final_index_name, 0);
+	}
 
 	if (!from_stdin) {
 		printf("%s\n", hash_to_hex(hash));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 0c3c01cdc9..63fea80b08 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1531,7 +1531,8 @@ static int want_cruft_object_mtime(struct repository *r,
 	struct odb_source *source;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		struct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct packed_git **cache = packfile_store_get_kept_pack_cache(files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
@@ -1753,11 +1754,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		for (e = source->files->packed->packs.head; e; e = e->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		for (e = files->packed->packs.head; e; e = e->next) {
 			struct packed_git *p = e->pack;
 			want = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);
 			if (!exclude && want > 0)
-				packfile_list_prepend(&source->files->packed->packs, p);
+				packfile_list_prepend(&files->packed->packs, p);
 			if (want != -1)
 				return want;
 		}
@@ -4337,10 +4340,12 @@ static void add_objects_in_unpacked_packs(void)
 
 	odb_prepare_alternates(to_pack.repo->objects);
 	for (source = to_pack.repo->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
 		if (!source->local)
 			continue;
 
-		if (packfile_store_for_each_object(source->files->packed, &oi,
+		if (packfile_store_for_each_object(files->packed, &oi,
 						   add_object_in_unpacked_pack, NULL,
 						   ODB_FOR_EACH_OBJECT_PACK_ORDER |
 						   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |
diff --git a/commit-graph.c b/commit-graph.c
index 967eb77047..f8e24145a5 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1980,9 +1980,11 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)
 			ctx->approx_nr_objects);
 
 	odb_prepare_alternates(ctx->r->objects);
-	for (source = ctx->r->objects->sources; source; source = source->next)
-		packfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,
+	for (source = ctx->r->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		packfile_store_for_each_object(files->packed, &oi, add_packed_commits_oi,
 					       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);
+	}
 
 	if (ctx->progress_done < ctx->approx_nr_objects)
 		display_progress(ctx->progress, ctx->approx_nr_objects);
diff --git a/http.c b/http.c
index b44f493919..8ea1b9d1f6 100644
--- a/http.c
+++ b/http.c
@@ -2543,8 +2543,9 @@ int finish_http_pack_request(struct http_pack_request *preq)
 void http_install_packfile(struct packed_git *p,
 			   struct packfile_list *list_to_remove_from)
 {
+	struct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);
 	packfile_list_remove(list_to_remove_from, p);
-	packfile_store_add_pack(the_repository->objects->sources->files->packed, p);
+	packfile_store_add_pack(files->packed, p);
 }
 
 struct http_pack_request *new_http_pack_request(
diff --git a/loose.c b/loose.c
index c921d46b94..07333be696 100644
--- a/loose.c
+++ b/loose.c
@@ -3,6 +3,7 @@
 #include "path.h"
 #include "object-file.h"
 #include "odb.h"
+#include "odb/source-files.h"
 #include "hex.h"
 #include "repository.h"
 #include "wrapper.h"
@@ -49,27 +50,29 @@ static int insert_loose_map(struct odb_source *source,
 			    const struct object_id *oid,
 			    const struct object_id *compat_oid)
 {
-	struct loose_object_map *map = source->files->loose->map;
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	struct loose_object_map *map = files->loose->map;
 	int inserted = 0;
 
 	inserted |= insert_oid_pair(map->to_compat, oid, compat_oid);
 	inserted |= insert_oid_pair(map->to_storage, compat_oid, oid);
 	if (inserted)
-		oidtree_insert(source->files->loose->cache, compat_oid);
+		oidtree_insert(files->loose->cache, compat_oid);
 
 	return inserted;
 }
 
 static int load_one_loose_object_map(struct repository *repo, struct odb_source *source)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
 	FILE *fp;
 
-	if (!source->files->loose->map)
-		loose_object_map_init(&source->files->loose->map);
-	if (!source->files->loose->cache) {
-		ALLOC_ARRAY(source->files->loose->cache, 1);
-		oidtree_init(source->files->loose->cache);
+	if (!files->loose->map)
+		loose_object_map_init(&files->loose->map);
+	if (!files->loose->cache) {
+		ALLOC_ARRAY(files->loose->cache, 1);
+		oidtree_init(files->loose->cache);
 	}
 
 	insert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
@@ -125,7 +128,8 @@ int repo_read_loose_object_map(struct repository *repo)
 
 int repo_write_loose_object_map(struct repository *repo)
 {
-	kh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;
+	struct odb_source_files *files = odb_source_files_downcast(repo->objects->sources);
+	kh_oid_map_t *map = files->loose->map->to_compat;
 	struct lock_file lock;
 	int fd;
 	khiter_t iter;
@@ -231,7 +235,8 @@ int repo_loose_object_map_oid(struct repository *repo,
 	khiter_t pos;
 
 	for (source = repo->objects->sources; source; source = source->next) {
-		struct loose_object_map *loose_map = source->files->loose->map;
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct loose_object_map *loose_map = files->loose->map;
 		if (!loose_map)
 			continue;
 		map = (to == repo->compat_hash_algo) ?
diff --git a/midx.c b/midx.c
index 698d10a1c6..ab8e2611d1 100644
--- a/midx.c
+++ b/midx.c
@@ -95,8 +95,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,
 
 struct multi_pack_index *get_multi_pack_index(struct odb_source *source)
 {
-	packfile_store_prepare(source->files->packed);
-	return source->files->packed->midx;
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	packfile_store_prepare(files->packed);
+	return files->packed->midx;
 }
 
 static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,
@@ -447,6 +448,7 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,
 int prepare_midx_pack(struct multi_pack_index *m,
 		      uint32_t pack_int_id)
 {
+	struct odb_source_files *files = odb_source_files_downcast(m->source);
 	struct strbuf pack_name = STRBUF_INIT;
 	struct packed_git *p;
 
@@ -457,10 +459,10 @@ int prepare_midx_pack(struct multi_pack_index *m,
 	if (m->packs[pack_int_id])
 		return 0;
 
-	strbuf_addf(&pack_name, "%s/pack/%s", m->source->path,
+	strbuf_addf(&pack_name, "%s/pack/%s", files->base.path,
 		    m->pack_names[pack_int_id]);
-	p = packfile_store_load_pack(m->source->files->packed,
-				     pack_name.buf, m->source->local);
+	p = packfile_store_load_pack(files->packed,
+				     pack_name.buf, files->base.local);
 	strbuf_release(&pack_name);
 
 	if (!p) {
@@ -703,18 +705,19 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)
 
 int prepare_multi_pack_index_one(struct odb_source *source)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	struct repository *r = source->odb->repo;
 
 	prepare_repo_settings(r);
 	if (!r->settings.core_multi_pack_index)
 		return 0;
 
-	if (source->files->packed->midx)
+	if (files->packed->midx)
 		return 1;
 
-	source->files->packed->midx = load_multi_pack_index(source);
+	files->packed->midx = load_multi_pack_index(source);
 
-	return !!source->files->packed->midx;
+	return !!files->packed->midx;
 }
 
 int midx_checksum_valid(struct multi_pack_index *m)
@@ -803,9 +806,10 @@ void clear_midx_file(struct repository *r)
 		struct odb_source *source;
 
 		for (source = r->objects->sources; source; source = source->next) {
-			if (source->files->packed->midx)
-				close_midx(source->files->packed->midx);
-			source->files->packed->midx = NULL;
+			struct odb_source_files *files = odb_source_files_downcast(source);
+			if (files->packed->midx)
+				close_midx(files->packed->midx);
+			files->packed->midx = NULL;
 		}
 	}
 
diff --git a/object-file.c b/object-file.c
index ec04d3572a..25c1146849 100644
--- a/object-file.c
+++ b/object-file.c
@@ -219,8 +219,9 @@ static void *odb_source_loose_map_object(struct odb_source *source,
 					 const struct object_id *oid,
 					 unsigned long *size)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	const char *p;
-	int fd = open_loose_object(source->files->loose, oid, &p);
+	int fd = open_loose_object(files->loose, oid, &p);
 
 	if (fd < 0)
 		return NULL;
@@ -401,6 +402,7 @@ static int read_object_info_from_path(struct odb_source *source,
 				      struct object_info *oi,
 				      enum object_info_flags flags)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	int ret;
 	int fd;
 	unsigned long mapsize;
@@ -423,7 +425,7 @@ static int read_object_info_from_path(struct odb_source *source,
 		struct stat st;
 
 		if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
-			ret = quick_has_loose(source->files->loose, oid) ? 0 : -1;
+			ret = quick_has_loose(files->loose, oid) ? 0 : -1;
 			goto out;
 		}
 
@@ -1866,33 +1868,34 @@ static int append_loose_object(const struct object_id *oid,
 struct oidtree *odb_source_loose_cache(struct odb_source *source,
 				       const struct object_id *oid)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	int subdir_nr = oid->hash[0];
 	struct strbuf buf = STRBUF_INIT;
-	size_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);
+	size_t word_bits = bitsizeof(files->loose->subdir_seen[0]);
 	size_t word_index = subdir_nr / word_bits;
 	size_t mask = (size_t)1u << (subdir_nr % word_bits);
 	uint32_t *bitmap;
 
 	if (subdir_nr < 0 ||
-	    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))
+	    (size_t) subdir_nr >= bitsizeof(files->loose->subdir_seen))
 		BUG("subdir_nr out of range");
 
-	bitmap = &source->files->loose->subdir_seen[word_index];
+	bitmap = &files->loose->subdir_seen[word_index];
 	if (*bitmap & mask)
-		return source->files->loose->cache;
-	if (!source->files->loose->cache) {
-		ALLOC_ARRAY(source->files->loose->cache, 1);
-		oidtree_init(source->files->loose->cache);
+		return files->loose->cache;
+	if (!files->loose->cache) {
+		ALLOC_ARRAY(files->loose->cache, 1);
+		oidtree_init(files->loose->cache);
 	}
 	strbuf_addstr(&buf, source->path);
 	for_each_file_in_obj_subdir(subdir_nr, &buf,
 				    source->odb->repo->hash_algo,
 				    append_loose_object,
 				    NULL, NULL,
-				    source->files->loose->cache);
+				    files->loose->cache);
 	*bitmap |= mask;
 	strbuf_release(&buf);
-	return source->files->loose->cache;
+	return files->loose->cache;
 }
 
 static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
@@ -1905,7 +1908,8 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
 
 void odb_source_loose_reprepare(struct odb_source *source)
 {
-	odb_source_loose_clear_cache(source->files->loose);
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	odb_source_loose_clear_cache(files->loose);
 }
 
 static int check_stream_oid(git_zstream *stream,
diff --git a/odb.c b/odb.c
index c9ebc7e741..e5aa8deb88 100644
--- a/odb.c
+++ b/odb.c
@@ -691,7 +691,8 @@ static int do_oid_object_info_extended(struct object_database *odb,
 
 		/* Most likely it's a loose object. */
 		for (source = odb->sources; source; source = source->next) {
-			if (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||
+			struct odb_source_files *files = odb_source_files_downcast(source);
+			if (!packfile_store_read_object_info(files->packed, real, oi, flags) ||
 			    !odb_source_loose_read_object_info(source, real, oi, flags))
 				return 0;
 		}
@@ -699,9 +700,11 @@ static int do_oid_object_info_extended(struct object_database *odb,
 		/* Not a loose object; someone else may have just packed it. */
 		if (!(flags & OBJECT_INFO_QUICK)) {
 			odb_reprepare(odb->repo->objects);
-			for (source = odb->sources; source; source = source->next)
-				if (!packfile_store_read_object_info(source->files->packed, real, oi, flags))
+			for (source = odb->sources; source; source = source->next) {
+				struct odb_source_files *files = odb_source_files_downcast(source);
+				if (!packfile_store_read_object_info(files->packed, real, oi, flags))
 					return 0;
+			}
 		}
 
 		/*
@@ -962,7 +965,9 @@ int odb_freshen_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (packfile_store_freshen_object(source->files->packed, oid))
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		if (packfile_store_freshen_object(files->packed, oid))
 			return 1;
 
 		if (odb_source_loose_freshen_object(source, oid))
@@ -982,6 +987,8 @@ int odb_for_each_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (struct odb_source *source = odb->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
 		if (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)
 			continue;
 
@@ -992,7 +999,7 @@ int odb_for_each_object(struct object_database *odb,
 				return ret;
 		}
 
-		ret = packfile_store_for_each_object(source->files->packed, request,
+		ret = packfile_store_for_each_object(files->packed, request,
 						     cb, cb_data, flags);
 		if (ret)
 			return ret;
@@ -1090,8 +1097,10 @@ struct object_database *odb_new(struct repository *repo,
 void odb_close(struct object_database *o)
 {
 	struct odb_source *source;
-	for (source = o->sources; source; source = source->next)
-		packfile_store_close(source->files->packed);
+	for (source = o->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		packfile_store_close(files->packed);
+	}
 	close_commit_graph(o);
 }
 
@@ -1148,8 +1157,9 @@ void odb_reprepare(struct object_database *o)
 	odb_prepare_alternates(o);
 
 	for (source = o->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
 		odb_source_loose_reprepare(source);
-		packfile_store_reprepare(source->files->packed);
+		packfile_store_reprepare(files->packed);
 	}
 
 	o->approximate_object_count_valid = 0;
diff --git a/odb/source-files.c b/odb/source-files.c
index cbdaa6850f..a43a197157 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -1,5 +1,6 @@
 #include "git-compat-util.h"
 #include "object-file.h"
+#include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
 
@@ -9,15 +10,20 @@ void odb_source_files_free(struct odb_source_files *files)
 		return;
 	odb_source_loose_free(files->loose);
 	packfile_store_free(files->packed);
+	odb_source_release(&files->base);
 	free(files);
 }
 
-struct odb_source_files *odb_source_files_new(struct odb_source *source)
+struct odb_source_files *odb_source_files_new(struct object_database *odb,
+					      const char *path,
+					      bool local)
 {
 	struct odb_source_files *files;
+
 	CALLOC_ARRAY(files, 1);
-	files->source = source;
-	files->loose = odb_source_loose_new(source);
-	files->packed = packfile_store_new(source);
+	odb_source_init(&files->base, odb, path, local);
+	files->loose = odb_source_loose_new(&files->base);
+	files->packed = packfile_store_new(&files->base);
+
 	return files;
 }
diff --git a/odb/source-files.h b/odb/source-files.h
index 0b8bf773ca..58753d40de 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -1,8 +1,9 @@
 #ifndef ODB_SOURCE_FILES_H
 #define ODB_SOURCE_FILES_H
 
+#include "odb/source.h"
+
 struct odb_source_loose;
-struct odb_source;
 struct packfile_store;
 
 /*
@@ -10,15 +11,26 @@ struct packfile_store;
  * packfiles. It is the default backend used by Git to store objects.
  */
 struct odb_source_files {
-	struct odb_source *source;
+	struct odb_source base;
 	struct odb_source_loose *loose;
 	struct packfile_store *packed;
 };
 
 /* Allocate and initialize a new object source. */
-struct odb_source_files *odb_source_files_new(struct odb_source *source);
+struct odb_source_files *odb_source_files_new(struct object_database *odb,
+					      const char *path,
+					      bool local);
 
 /* Free the object source and release all associated resources. */
 void odb_source_files_free(struct odb_source_files *files);
 
+/*
+ * Cast the given object database source to the files backend. This will cause
+ * a BUG in case the source doesn't use this backend.
+ */
+static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
+{
+	return container_of(source, struct odb_source_files, base);
+}
+
 #endif
diff --git a/odb/source.c b/odb/source.c
index 9d7fd19f45..d8b2176a94 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -1,5 +1,6 @@
 #include "git-compat-util.h"
 #include "object-file.h"
+#include "odb/source-files.h"
 #include "odb/source.h"
 #include "packfile.h"
 
@@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
 {
-	struct odb_source *source;
+	return &odb_source_files_new(odb, path, local)->base;
+}
 
-	CALLOC_ARRAY(source, 1);
+void odb_source_init(struct odb_source *source,
+		     struct object_database *odb,
+		     const char *path,
+		     bool local)
+{
 	source->odb = odb;
 	source->local = local;
 	source->path = xstrdup(path);
-	source->files = odb_source_files_new(source);
-
-	return source;
 }
 
 void odb_source_free(struct odb_source *source)
 {
+	struct odb_source_files *files;
+	if (!source)
+		return;
+	files = odb_source_files_downcast(source);
+	odb_source_files_free(files);
+}
+
+void odb_source_release(struct odb_source *source)
+{
+	if (!source)
+		return;
 	free(source->path);
-	odb_source_files_free(source->files);
-	free(source);
 }
diff --git a/odb/source.h b/odb/source.h
index 1c34265189..e6698b73a3 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,8 +1,6 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
-#include "odb/source-files.h"
-
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -21,9 +19,6 @@ struct odb_source {
 	/* Object database that owns this object source. */
 	struct object_database *odb;
 
-	/* The backend used to store objects. */
-	struct odb_source_files *files;
-
 	/*
 	 * Figure out whether this is the local source of the owning
 	 * repository, which would typically be its ".git/objects" directory.
@@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local);
 
-/* Free the object database source, releasing all associated resources. */
+/*
+ * Initialize the source for the given object database located at `path`.
+ * `local` indicates whether or not the source is the local and thus primary
+ * object source of the object database.
+ *
+ * This function is only supposed to be called by specific object source
+ * implementations.
+ */
+void odb_source_init(struct odb_source *source,
+		     struct object_database *odb,
+		     const char *path,
+		     bool local);
+
+/*
+ * Free the object database source, releasing all associated resources and
+ * freeing the structure itself.
+ */
 void odb_source_free(struct odb_source *source);
 
+/*
+ * Release the object database source, releasing all associated resources.
+ *
+ * This function is only supposed to be called by specific object source
+ * implementations.
+ */
+void odb_source_release(struct odb_source *source);
+
 #endif
diff --git a/odb/streaming.c b/odb/streaming.c
index 26b0a1a0f5..19cda9407d 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -187,7 +187,8 @@ static int istream_source(struct odb_read_stream **out,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (!packfile_store_read_object_stream(out, source->files->packed, oid) ||
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		if (!packfile_store_read_object_stream(out, files->packed, oid) ||
 		    !odb_source_loose_read_object_stream(out, source, oid))
 			return 0;
 	}
diff --git a/packfile.c b/packfile.c
index 4e1f6087ed..da1c0dfa39 100644
--- a/packfile.c
+++ b/packfile.c
@@ -362,9 +362,11 @@ static int unuse_one_window(struct object_database *odb)
 	struct packed_git *lru_p = NULL;
 	struct pack_window *lru_w = NULL, *lru_l = NULL;
 
-	for (source = odb->sources; source; source = source->next)
-		for (e = source->files->packed->packs.head; e; e = e->next)
+	for (source = odb->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		for (e = files->packed->packs.head; e; e = e->next)
 			scan_windows(e->pack, &lru_p, &lru_w, &lru_l);
+	}
 
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -537,7 +539,8 @@ static int close_one_pack(struct repository *r)
 	int accept_windows_inuse = 1;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		for (e = source->files->packed->packs.head; e; e = e->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		for (e = files->packed->packs.head; e; e = e->next) {
 			if (e->pack->pack_fd == -1)
 				continue;
 			find_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);
@@ -987,13 +990,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 			 const char *file_name, void *_data)
 {
 	struct prepare_pack_data *data = (struct prepare_pack_data *)_data;
+	struct odb_source_files *files = odb_source_files_downcast(data->source);
 	size_t base_len = full_name_len;
 
 	if (strip_suffix_mem(full_name, &base_len, ".idx") &&
-	    !(data->source->files->packed->midx &&
-	      midx_contains_pack(data->source->files->packed->midx, file_name))) {
+	    !(files->packed->midx &&
+	      midx_contains_pack(files->packed->midx, file_name))) {
 		char *trimmed_path = xstrndup(full_name, full_name_len);
-		packfile_store_load_pack(data->source->files->packed,
+		packfile_store_load_pack(files->packed,
 					 trimmed_path, data->source->local);
 		free(trimmed_path);
 	}
@@ -1247,8 +1251,10 @@ const struct packed_git *has_packed_and_bad(struct repository *r,
 	struct odb_source *source;
 
 	for (source = r->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
 		struct packfile_list_entry *e;
-		for (e = source->files->packed->packs.head; e; e = e->next)
+
+		for (e = files->packed->packs.head; e; e = e->next)
 			if (oidset_contains(&e->pack->bad_objects, oid))
 				return e->pack;
 	}
@@ -2254,7 +2260,8 @@ int has_object_pack(struct repository *r, const struct object_id *oid)
 
 	odb_prepare_alternates(r->objects);
 	for (source = r->objects->sources; source; source = source->next) {
-		int ret = find_pack_entry(source->files->packed, oid, &e);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		int ret = find_pack_entry(files->packed, oid, &e);
 		if (ret)
 			return ret;
 	}
@@ -2269,9 +2276,10 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,
 	struct pack_entry e;
 
 	for (source = r->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
 		struct packed_git **cache;
 
-		cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
+		cache = packfile_store_get_kept_pack_cache(files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
diff --git a/packfile.h b/packfile.h
index e8de06ee86..64a31738c0 100644
--- a/packfile.h
+++ b/packfile.h
@@ -4,6 +4,7 @@
 #include "list.h"
 #include "object.h"
 #include "odb.h"
+#include "odb/source-files.h"
 #include "oidset.h"
 #include "repository.h"
 #include "strmap.h"
@@ -192,7 +193,8 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct
 	odb_prepare_alternates(repo->objects);
 
 	for (struct odb_source *source = repo->objects->sources; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct packfile_list_entry *entry = packfile_store_get_packs(files->packed);
 		if (!entry)
 			continue;
 		data.source = source;
@@ -212,7 +214,8 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *
 		return;
 
 	for (source = data->source->next; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct packfile_list_entry *entry = packfile_store_get_packs(files->packed);
 		if (!entry)
 			continue;
 		data->source = source;

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 03/17] odb: embed base source in the "files" backend

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> The "files" backend is implemented as a pointer in the `struct
> odb_source`. This contradicts our typical pattern for pluggable backends
> like we use it for example in the ref store or for object database
> streams, where we typically embed the generic base structure in the
> specialized implementation. This pattern has a couple of small benefits:
> 
>   - We avoid an extra allocation.
> 
>   - We hide implementation details in the generic structure.
> 
>   - We can easily downcast from a generic backend to the specialized
>     structure and vice versa because the offsets are known at compile
>     time.
> 
>   - It becomes trivial to identify locations where we depend on backend
>     specific logic because the cast needs to be explicit.
> 
> Refactor our "files" object database source to do the same and embed the
> `struct odb_source` in the `struct odb_source_files`.

Makes sense.
 
> There are still a bunch of sites in our code base where we do have to
> access internals of the "files" backend. The intent is that those will
> go away over time, but this will certainly take a while. Meanwhile,
> provide a `odb_source_files_downcast()` function that can convert a
> generic source into a "files" source.
> 
> As we only have a single source the downcast succeeds unconditionally
> for now. Eventually though the intent is to make the cast `BUG()` in
> case the caller requests to downcast a non-"files" backend to a "files"
> backend.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> diff --git a/odb/source-files.c b/odb/source-files.c
> index cbdaa6850f..a43a197157 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -1,5 +1,6 @@
>  #include "git-compat-util.h"
>  #include "object-file.h"
> +#include "odb/source.h"
>  #include "odb/source-files.h"
>  #include "packfile.h"
>  
> @@ -9,15 +10,20 @@ void odb_source_files_free(struct odb_source_files *files)
>  		return;
>  	odb_source_loose_free(files->loose);
>  	packfile_store_free(files->packed);
> +	odb_source_release(&files->base);
>  	free(files);
>  }
>  
> -struct odb_source_files *odb_source_files_new(struct odb_source *source)
> +struct odb_source_files *odb_source_files_new(struct object_database *odb,
> +					      const char *path,
> +					      bool local)
>  {
>  	struct odb_source_files *files;
> +
>  	CALLOC_ARRAY(files, 1);
> -	files->source = source;
> -	files->loose = odb_source_loose_new(source);
> -	files->packed = packfile_store_new(source);
> +	odb_source_init(&files->base, odb, path, local);
> +	files->loose = odb_source_loose_new(&files->base);
> +	files->packed = packfile_store_new(&files->base);

When creating the files ODB source, it is now responsible for also
creating the embedded base ODB souce. Makes sense.

> +
>  	return files;
>  }
> diff --git a/odb/source-files.h b/odb/source-files.h
> index 0b8bf773ca..58753d40de 100644
> --- a/odb/source-files.h
> +++ b/odb/source-files.h
> @@ -1,8 +1,9 @@
>  #ifndef ODB_SOURCE_FILES_H
>  #define ODB_SOURCE_FILES_H
>  
> +#include "odb/source.h"
> +
>  struct odb_source_loose;
> -struct odb_source;
>  struct packfile_store;
>  
>  /*
> @@ -10,15 +11,26 @@ struct packfile_store;
>   * packfiles. It is the default backend used by Git to store objects.
>   */
>  struct odb_source_files {
> -	struct odb_source *source;
> +	struct odb_source base;

Out of curiousity, was there any reason to the reference ODB source in
the prior patch? Seems like we could have just added it here.

>  	struct odb_source_loose *loose;
>  	struct packfile_store *packed;
>  };
>  
>  /* Allocate and initialize a new object source. */
> -struct odb_source_files *odb_source_files_new(struct odb_source *source);
> +struct odb_source_files *odb_source_files_new(struct object_database *odb,
> +					      const char *path,
> +					      bool local);
>  
>  /* Free the object source and release all associated resources. */
>  void odb_source_files_free(struct odb_source_files *files);
>  
> +/*
> + * Cast the given object database source to the files backend. This will cause
> + * a BUG in case the source doesn't use this backend.
> + */

In the commit message you mention that eventually
`odb_source_files_downcast()` will BUG() if the source doesn't use the
backend. But, it doesn't appear to do this yet. Should we still have
this comment?

> +static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
> +{
> +	return container_of(source, struct odb_source_files, base);
> +}
> +
>  #endif
[snip]
> diff --git a/odb/source.h b/odb/source.h
> index 1c34265189..e6698b73a3 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -1,8 +1,6 @@
>  #ifndef ODB_SOURCE_H
>  #define ODB_SOURCE_H
>  
> -#include "odb/source-files.h"
> -
>  /*
>   * The source is the part of the object database that stores the actual
>   * objects. It thus encapsulates the logic to read and write the specific
> @@ -21,9 +19,6 @@ struct odb_source {
>  	/* Object database that owns this object source. */
>  	struct object_database *odb;
>  
> -	/* The backend used to store objects. */
> -	struct odb_source_files *files;

Now that the base ODB source is embedded in `struct odb_source_files`,
it is accessed via downcasting and the direct reference is no longer
needed. This is responsible for most of the structural change fallout in
this patch.

> -
>  	/*
>  	 * Figure out whether this is the local source of the owning
>  	 * repository, which would typically be its ".git/objects" directory.
> @@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
>  				  const char *path,
>  				  bool local);
>  
> -/* Free the object database source, releasing all associated resources. */
> +/*
> + * Initialize the source for the given object database located at `path`.
> + * `local` indicates whether or not the source is the local and thus primary
> + * object source of the object database.
> + *
> + * This function is only supposed to be called by specific object source
> + * implementations.
> + */
> +void odb_source_init(struct odb_source *source,
> +		     struct object_database *odb,
> +		     const char *path,
> +		     bool local);
> +
> +/*
> + * Free the object database source, releasing all associated resources and
> + * freeing the structure itself.
> + */
>  void odb_source_free(struct odb_source *source);
>  
> +/*
> + * Release the object database source, releasing all associated resources.
> + *
> + * This function is only supposed to be called by specific object source
> + * implementations.
> + */
> +void odb_source_release(struct odb_source *source);

From a naming perspective, I do find the odb_source_new() vs
odb_source_init() and odb_source_free() vs odb_source_release()
interfaces to be tad bit confusing. I understand that odb_source_init()
and odb_source_release() and only intended for use by the concrete ODB
source implementations to facilitate initializing/freeing the base ODB
source. The comments also do help clarify this, but I think it is still
rather easy to get them mixed up when reading.

Maybe we could rename them to odb_base_source_init() and
odb_base_source_free()?

> +
>  #endif
> diff --git a/odb/streaming.c b/odb/streaming.c
> index 26b0a1a0f5..19cda9407d 100644
> --- a/odb/streaming.c
> +++ b/odb/streaming.c
> @@ -187,7 +187,8 @@ static int istream_source(struct odb_read_stream **out,
>  
>  	odb_prepare_alternates(odb);
>  	for (source = odb->sources; source; source = source->next) {
> -		if (!packfile_store_read_object_stream(out, source->files->packed, oid) ||
> +		struct odb_source_files *files = odb_source_files_downcast(source);
> +		if (!packfile_store_read_object_stream(out, files->packed, oid) ||
>  		    !odb_source_loose_read_object_stream(out, source, oid))
>  			return 0;
>  	}

Overall this patch looks good.

-Justin

Re: [PATCH 03/17] odb: embed base source in the "files" backend

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 11:40:47AM -0600, Justin Tobler wrote:
> On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > diff --git a/odb/source-files.h b/odb/source-files.h
> > index 0b8bf773ca..58753d40de 100644
> > --- a/odb/source-files.h
> > +++ b/odb/source-files.h
> > @@ -10,15 +11,26 @@ struct packfile_store;
> >   * packfiles. It is the default backend used by Git to store objects.
> >   */
> >  struct odb_source_files {
> > -	struct odb_source *source;
> > +	struct odb_source base;
> 
> Out of curiousity, was there any reason to the reference ODB source in
> the prior patch? Seems like we could have just added it here.

Good question. The reason why I stored this pointer in the preceding
commit is mostly to demonstrate that we're actually using the source
that's passed to `db_source_files_new()`. I didn't want to have to
change the signature of that function in this commit again.

So the field was unused indeed, but intentionally so.

> >  	struct odb_source_loose *loose;
> >  	struct packfile_store *packed;
> >  };
> >  
> >  /* Allocate and initialize a new object source. */
> > -struct odb_source_files *odb_source_files_new(struct odb_source *source);
> > +struct odb_source_files *odb_source_files_new(struct object_database *odb,
> > +					      const char *path,
> > +					      bool local);
> >  
> >  /* Free the object source and release all associated resources. */
> >  void odb_source_files_free(struct odb_source_files *files);
> >  
> > +/*
> > + * Cast the given object database source to the files backend. This will cause
> > + * a BUG in case the source doesn't use this backend.
> > + */
> 
> In the commit message you mention that eventually
> `odb_source_files_downcast()` will BUG() if the source doesn't use the
> backend. But, it doesn't appear to do this yet. Should we still have
> this comment?

Good point, let me move this into the patch that introduces this.

> > diff --git a/odb/source.h b/odb/source.h
> > index 1c34265189..e6698b73a3 100644
> > --- a/odb/source.h
> > +++ b/odb/source.h
> > @@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
> >  				  const char *path,
> >  				  bool local);
> >  
> > -/* Free the object database source, releasing all associated resources. */
> > +/*
> > + * Initialize the source for the given object database located at `path`.
> > + * `local` indicates whether or not the source is the local and thus primary
> > + * object source of the object database.
> > + *
> > + * This function is only supposed to be called by specific object source
> > + * implementations.
> > + */
> > +void odb_source_init(struct odb_source *source,
> > +		     struct object_database *odb,
> > +		     const char *path,
> > +		     bool local);
> > +
> > +/*
> > + * Free the object database source, releasing all associated resources and
> > + * freeing the structure itself.
> > + */
> >  void odb_source_free(struct odb_source *source);
> >  
> > +/*
> > + * Release the object database source, releasing all associated resources.
> > + *
> > + * This function is only supposed to be called by specific object source
> > + * implementations.
> > + */
> > +void odb_source_release(struct odb_source *source);
> 
> From a naming perspective, I do find the odb_source_new() vs
> odb_source_init() and odb_source_free() vs odb_source_release()
> interfaces to be tad bit confusing. I understand that odb_source_init()
> and odb_source_release() and only intended for use by the concrete ODB
> source implementations to facilitate initializing/freeing the base ODB
> source. The comments also do help clarify this, but I think it is still
> rather easy to get them mixed up when reading.
> 
> Maybe we could rename them to odb_base_source_init() and
> odb_base_source_free()?

I think for `odb_source_free()` it's a definitive no. This will be the
way to free any source, not only the base, and this will become clear in
a subsequent patch.

For `odb_source_init()` you have a better point though, as it really
only cares about initializing the base object. But I think it's still
sensible to keep the name as it _does_ act on `struct odb_source`, and
it would be the only instance where we have the "base" infix.

Patrick

Re: [PATCH 03/17] odb: embed base source in the "files" backend

[Justin Tobler]

On 26/03/05 02:23PM, Patrick Steinhardt wrote:
> On Wed, Mar 04, 2026 at 11:40:47AM -0600, Justin Tobler wrote:
> > On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > > diff --git a/odb/source-files.h b/odb/source-files.h
> > > index 0b8bf773ca..58753d40de 100644
> > > --- a/odb/source-files.h
> > > +++ b/odb/source-files.h
> > > @@ -10,15 +11,26 @@ struct packfile_store;
> > >   * packfiles. It is the default backend used by Git to store objects.
> > >   */
> > >  struct odb_source_files {
> > > -	struct odb_source *source;
> > > +	struct odb_source base;
> > 
> > Out of curiousity, was there any reason to the reference ODB source in
> > the prior patch? Seems like we could have just added it here.
> 
> Good question. The reason why I stored this pointer in the preceding
> commit is mostly to demonstrate that we're actually using the source
> that's passed to `db_source_files_new()`. I didn't want to have to
> change the signature of that function in this commit again.
> 
> So the field was unused indeed, but intentionally so.

That's fair. I did find it mildly confusing to see its introduction
without any uses, only to be renamed here. But it's not really a big
deal either way.

> > From a naming perspective, I do find the odb_source_new() vs
> > odb_source_init() and odb_source_free() vs odb_source_release()
> > interfaces to be tad bit confusing. I understand that odb_source_init()
> > and odb_source_release() and only intended for use by the concrete ODB
> > source implementations to facilitate initializing/freeing the base ODB
> > source. The comments also do help clarify this, but I think it is still
> > rather easy to get them mixed up when reading.
> > 
> > Maybe we could rename them to odb_base_source_init() and
> > odb_base_source_free()?
> 
> I think for `odb_source_free()` it's a definitive no. This will be the
> way to free any source, not only the base, and this will become clear in
> a subsequent patch.

Fair.

> For `odb_source_init()` you have a better point though, as it really
> only cares about initializing the base object. But I think it's still
> sensible to keep the name as it _does_ act on `struct odb_source`, and
> it would be the only instance where we have the "base" infix.

Ya it does still act on the `struct odb_source`, but IMO the name fails
to properly differentiant it's usecase which it a tad bit confusing.
Naming is hard though and I don't have really a better suggestion so it
is probably fine as-is. At least the comments do a reasonable job of
explaining the intent here. :)

-Justin

Re: [PATCH 03/17] odb: embed base source in the "files" backend

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> The "files" backend is implemented as a pointer in the `struct
> odb_source`. This contradicts our typical pattern for pluggable backends
> like we use it for example in the ref store or for object database
> streams, where we typically embed the generic base structure in the
> specialized implementation. This pattern has a couple of small benefits:
>
>   - We avoid an extra allocation.
>

Because currently we allocate `obd_source` and also its `files` variable
independently. With the change, the `odb_source_files` will embed the
`obd_source` and be allocated together in one call. Makes sense.

>   - We hide implementation details in the generic structure.
>
>   - We can easily downcast from a generic backend to the specialized
>     structure and vice versa because the offsets are known at compile
>     time.
>
>   - It becomes trivial to identify locations where we depend on backend
>     specific logic because the cast needs to be explicit.
>

Indeed, also makes it easier to move generic logic out of individual
backends into the generic layer.

> Refactor our "files" object database source to do the same and embed the
> `struct odb_source` in the `struct odb_source_files`.
>
> There are still a bunch of sites in our code base where we do have to
> access internals of the "files" backend. The intent is that those will
> go away over time, but this will certainly take a while. Meanwhile,
> provide a `odb_source_files_downcast()` function that can convert a
> generic source into a "files" source.
>
> As we only have a single source the downcast succeeds unconditionally
> for now. Eventually though the intent is to make the cast `BUG()` in
> case the caller requests to downcast a non-"files" backend to a "files"
> backend.
>

Do we also plan to add read/write permissions check within the downcast
logic? Similar to the refs DB? Doesn't have to be in this patch, just
curious if that is something we plan to include.

> diff --git a/odb/source.c b/odb/source.c
> index 9d7fd19f45..d8b2176a94 100644
> --- a/odb/source.c
> +++ b/odb/source.c
> @@ -1,5 +1,6 @@
>  #include "git-compat-util.h"
>  #include "object-file.h"
> +#include "odb/source-files.h"
>  #include "odb/source.h"
>  #include "packfile.h"
>
> @@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
>  				  const char *path,
>  				  bool local)
>  {
> -	struct odb_source *source;
> +	return &odb_source_files_new(odb, path, local)->base;
> +}
>

Since we only have one source right now (files), we directly call the
internals of that source, I guess once we add more this would be more
modular.

> -	CALLOC_ARRAY(source, 1);
> +void odb_source_init(struct odb_source *source,
> +		     struct object_database *odb,
> +		     const char *path,
> +		     bool local)
> +{
>  	source->odb = odb;
>  	source->local = local;
>  	source->path = xstrdup(path);
> -	source->files = odb_source_files_new(source);
> -
> -	return source;
>  }
>
>  void odb_source_free(struct odb_source *source)
>  {
> +	struct odb_source_files *files;
> +	if (!source)
> +		return;
> +	files = odb_source_files_downcast(source);
> +	odb_source_files_free(files);
> +}
> +
> +void odb_source_release(struct odb_source *source)
> +{
> +	if (!source)
> +		return;
>  	free(source->path);
> -	odb_source_files_free(source->files);
> -	free(source);
>  }

The patch looks good.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 03/17] odb: embed base source in the "files" backend

[Patrick Steinhardt]

On Thu, Mar 05, 2026 at 10:45:07AM +0000, Karthik Nayak wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > Refactor our "files" object database source to do the same and embed the
> > `struct odb_source` in the `struct odb_source_files`.
> >
> > There are still a bunch of sites in our code base where we do have to
> > access internals of the "files" backend. The intent is that those will
> > go away over time, but this will certainly take a while. Meanwhile,
> > provide a `odb_source_files_downcast()` function that can convert a
> > generic source into a "files" source.
> >
> > As we only have a single source the downcast succeeds unconditionally
> > for now. Eventually though the intent is to make the cast `BUG()` in
> > case the caller requests to downcast a non-"files" backend to a "files"
> > backend.
> >
> 
> Do we also plan to add read/write permissions check within the downcast
> logic? Similar to the refs DB? Doesn't have to be in this patch, just
> curious if that is something we plan to include.

I didn't plan to. I guess we could have such a check eventually though
to for example keep somebody from writing to secondary ODB sources. I
don't have anything cooking here though.

> > diff --git a/odb/source.c b/odb/source.c
> > index 9d7fd19f45..d8b2176a94 100644
> > --- a/odb/source.c
> > +++ b/odb/source.c
> > @@ -1,5 +1,6 @@
> >  #include "git-compat-util.h"
> >  #include "object-file.h"
> > +#include "odb/source-files.h"
> >  #include "odb/source.h"
> >  #include "packfile.h"
> >
> > @@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
> >  				  const char *path,
> >  				  bool local)
> >  {
> > -	struct odb_source *source;
> > +	return &odb_source_files_new(odb, path, local)->base;
> > +}
> >
> 
> Since we only have one source right now (files), we directly call the
> internals of that source, I guess once we add more this would be more
> modular.

Yeah. This will eventually be handled via a new object storage
extension, similar to how we do this for the reference backends.

Patrick

[PATCH 04/17] odb: move reparenting logic into respective subsystems

[Patrick Steinhardt]

The primary object database source may be initialized with a relative
path. When reparenting the process to a different working directory we
thus have to update this path and have it point to the same path, but
relative to the new working directory.

This logic is handled in the object database layer. It consists of three
steps:

  1. We undo any potential temporary object directory, which are used
     for transactions. This is done so that we don't end up modifying
     the temporary object database source that got applied for the
     transaction.

  2. We then iterate through the non-transactional sources and reparent
     their respective paths.

  3. We reapply the temporary object directory, but update its path.

All of this logic is heavily tied to how the object database source
handles paths in the first place. It's an internal implementation
detail, and as sources may not even use an on-disk path at all it is not
a mechanism that applies to all potential sources.

Refactor the code so that the logic to reparent the sources is hosted by
the "files" source and the temporary object directory subsystems,
respectively. This logic is easier to reason about, but it also ensures
that this logic is handled at the correct level.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 37 -------------------------------------
 odb/source-files.c | 23 +++++++++++++++++++++++
 tmp-objdir.c       | 42 +++++++++++++++++++-----------------------
 tmp-objdir.h       | 15 ---------------
 4 files changed, 42 insertions(+), 75 deletions(-)

diff --git a/odb.c b/odb.c
index e5aa8deb88..86f7cf70a8 100644
--- a/odb.c
+++ b/odb.c
@@ -1,6 +1,5 @@
 #include "git-compat-util.h"
 #include "abspath.h"
-#include "chdir-notify.h"
 #include "commit-graph.h"
 #include "config.h"
 #include "dir.h"
@@ -1037,38 +1036,6 @@ int odb_write_object_stream(struct object_database *odb,
 	return odb_source_loose_write_stream(odb->sources, stream, len, oid);
 }
 
-static void odb_update_commondir(const char *name UNUSED,
-				 const char *old_cwd,
-				 const char *new_cwd,
-				 void *cb_data)
-{
-	struct object_database *odb = cb_data;
-	struct tmp_objdir *tmp_objdir;
-	struct odb_source *source;
-
-	tmp_objdir = tmp_objdir_unapply_primary_odb();
-
-	/*
-	 * In theory, we only have to do this for the primary object source, as
-	 * alternates' paths are always resolved to an absolute path.
-	 */
-	for (source = odb->sources; source; source = source->next) {
-		char *path;
-
-		if (is_absolute_path(source->path))
-			continue;
-
-		path = reparent_relative_path(old_cwd, new_cwd,
-					      source->path);
-
-		free(source->path);
-		source->path = path;
-	}
-
-	if (tmp_objdir)
-		tmp_objdir_reapply_primary_odb(tmp_objdir, old_cwd, new_cwd);
-}
-
 struct object_database *odb_new(struct repository *repo,
 				const char *primary_source,
 				const char *secondary_sources)
@@ -1089,8 +1056,6 @@ struct object_database *odb_new(struct repository *repo,
 
 	free(to_free);
 
-	chdir_notify_register(NULL, odb_update_commondir, o);
-
 	return o;
 }
 
@@ -1136,8 +1101,6 @@ void odb_free(struct object_database *o)
 
 	string_list_clear(&o->submodule_source_paths, 0);
 
-	chdir_notify_unregister(NULL, odb_update_commondir, o);
-
 	free(o);
 }
 
diff --git a/odb/source-files.c b/odb/source-files.c
index a43a197157..df0ea9ee62 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -1,13 +1,28 @@
 #include "git-compat-util.h"
+#include "abspath.h"
+#include "chdir-notify.h"
 #include "object-file.h"
 #include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
 
+static void odb_source_files_reparent(const char *name UNUSED,
+				      const char *old_cwd,
+				      const char *new_cwd,
+				      void *cb_data)
+{
+	struct odb_source_files *files = cb_data;
+	char *path = reparent_relative_path(old_cwd, new_cwd,
+					    files->base.path);
+	free(files->base.path);
+	files->base.path = path;
+}
+
 void odb_source_files_free(struct odb_source_files *files)
 {
 	if (!files)
 		return;
+	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
 	odb_source_loose_free(files->loose);
 	packfile_store_free(files->packed);
 	odb_source_release(&files->base);
@@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->loose = odb_source_loose_new(&files->base);
 	files->packed = packfile_store_new(&files->base);
 
+	/*
+	 * Ideally, we would only ever store absolute paths in the source. This
+	 * is not (yet) possible though because we access and assume relative
+	 * paths in the primary ODB source in some user-facing functionality.
+	 */
+	if (!is_absolute_path(path))
+		chdir_notify_register(NULL, odb_source_files_reparent, files);
+
 	return files;
 }
diff --git a/tmp-objdir.c b/tmp-objdir.c
index 9f5a1788cd..e436eed07e 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -36,6 +36,21 @@ static void tmp_objdir_free(struct tmp_objdir *t)
 	free(t);
 }
 
+static void tmp_objdir_reparent(const char *name UNUSED,
+				const char *old_cwd,
+				const char *new_cwd,
+				void *cb_data)
+{
+	struct tmp_objdir *t = cb_data;
+	char *path;
+
+	path = reparent_relative_path(old_cwd, new_cwd,
+				      t->path.buf);
+	strbuf_reset(&t->path);
+	strbuf_addstr(&t->path, path);
+	free(path);
+}
+
 int tmp_objdir_destroy(struct tmp_objdir *t)
 {
 	int err;
@@ -51,6 +66,7 @@ int tmp_objdir_destroy(struct tmp_objdir *t)
 
 	err = remove_dir_recursively(&t->path, 0);
 
+	chdir_notify_unregister(NULL, tmp_objdir_reparent, t);
 	tmp_objdir_free(t);
 
 	return err;
@@ -137,6 +153,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	strbuf_addf(&t->path, "%s/tmp_objdir-%s-XXXXXX",
 		    repo_get_object_directory(r), prefix);
 
+	if (!is_absolute_path(t->path.buf))
+		chdir_notify_register(NULL, tmp_objdir_reparent, t);
+
 	if (!mkdtemp(t->path.buf)) {
 		/* free, not destroy, as we never touched the filesystem */
 		tmp_objdir_free(t);
@@ -315,26 +334,3 @@ void tmp_objdir_replace_primary_odb(struct tmp_objdir *t, int will_destroy)
 							  t->path.buf, will_destroy);
 	t->will_destroy = will_destroy;
 }
-
-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void)
-{
-	if (!the_tmp_objdir || !the_tmp_objdir->prev_source)
-		return NULL;
-
-	odb_restore_primary_source(the_tmp_objdir->repo->objects,
-				   the_tmp_objdir->prev_source, the_tmp_objdir->path.buf);
-	the_tmp_objdir->prev_source = NULL;
-	return the_tmp_objdir;
-}
-
-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *t, const char *old_cwd,
-		const char *new_cwd)
-{
-	char *path;
-
-	path = reparent_relative_path(old_cwd, new_cwd, t->path.buf);
-	strbuf_reset(&t->path);
-	strbuf_addstr(&t->path, path);
-	free(path);
-	tmp_objdir_replace_primary_odb(t, t->will_destroy);
-}
diff --git a/tmp-objdir.h b/tmp-objdir.h
index fceda14979..ccf800faa7 100644
--- a/tmp-objdir.h
+++ b/tmp-objdir.h
@@ -68,19 +68,4 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *);
  */
 void tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy);
 
-/*
- * If the primary object database was replaced by a temporary object directory,
- * restore it to its original value while keeping the directory contents around.
- * Returns NULL if the primary object database was not replaced.
- */
-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void);
-
-/*
- * Reapplies the former primary temporary object database, after potentially
- * changing its relative path.
- */
-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *, const char *old_cwd,
-		const char *new_cwd);
-
-
 #endif /* TMP_OBJDIR_H */

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 04/17] odb: move reparenting logic into respective subsystems

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> The primary object database source may be initialized with a relative
> path. When reparenting the process to a different working directory we

I find the wording here a bit confusing. Maybe something like this would
be a bit clearer:

  When the process changes its current working directory...

> thus have to update this path and have it point to the same path, but
> relative to the new working directory.
> 
> This logic is handled in the object database layer. It consists of three
> steps:
> 
>   1. We undo any potential temporary object directory, which are used
>      for transactions. This is done so that we don't end up modifying
>      the temporary object database source that got applied for the
>      transaction.
> 
>   2. We then iterate through the non-transactional sources and reparent
>      their respective paths.
> 
>   3. We reapply the temporary object directory, but update its path.
> 
> All of this logic is heavily tied to how the object database source
> handles paths in the first place. It's an internal implementation
> detail, and as sources may not even use an on-disk path at all it is not
> a mechanism that applies to all potential sources.

Indeed this mechanism is directly coupled to how the "files" backend
operates.

> Refactor the code so that the logic to reparent the sources is hosted by
> the "files" source and the temporary object directory subsystems,
> respectively. This logic is easier to reason about, but it also ensures
> that this logic is handled at the correct level.

Makes sense.
 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> diff --git a/odb/source-files.c b/odb/source-files.c
> index a43a197157..df0ea9ee62 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -1,13 +1,28 @@
>  #include "git-compat-util.h"
> +#include "abspath.h"
> +#include "chdir-notify.h"
>  #include "object-file.h"
>  #include "odb/source.h"
>  #include "odb/source-files.h"
>  #include "packfile.h"
>  
> +static void odb_source_files_reparent(const char *name UNUSED,
> +				      const char *old_cwd,
> +				      const char *new_cwd,
> +				      void *cb_data)
> +{
> +	struct odb_source_files *files = cb_data;
> +	char *path = reparent_relative_path(old_cwd, new_cwd,
> +					    files->base.path);
> +	free(files->base.path);
> +	files->base.path = path;

I do find it a bit curious that we consider the "path" to be specific to
the "files" backend, but still track it as part of the "base" ODB
source. I suspect this will eventually change though?

> +}
> +
>  void odb_source_files_free(struct odb_source_files *files)
>  {
>  	if (!files)
>  		return;
> +	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
>  	odb_source_loose_free(files->loose);
>  	packfile_store_free(files->packed);
>  	odb_source_release(&files->base);
> @@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  	files->loose = odb_source_loose_new(&files->base);
>  	files->packed = packfile_store_new(&files->base);
>  
> +	/*
> +	 * Ideally, we would only ever store absolute paths in the source. This
> +	 * is not (yet) possible though because we access and assume relative
> +	 * paths in the primary ODB source in some user-facing functionality.
> +	 */

Should this be a NEEDSWORK comment? Or do we expect it to remain this
way for the forseeable future?

> +	if (!is_absolute_path(path))
> +		chdir_notify_register(NULL, odb_source_files_reparent, files);

Ok so now a callback to reparent the path is set up for the "files"
source when it is created. If there are multiple "files" sources
created, each source will be handled separately.

> +
>  	return files;
>  }
> diff --git a/tmp-objdir.c b/tmp-objdir.c
> index 9f5a1788cd..e436eed07e 100644
> --- a/tmp-objdir.c
> +++ b/tmp-objdir.c
> @@ -36,6 +36,21 @@ static void tmp_objdir_free(struct tmp_objdir *t)
>  	free(t);
>  }
>  
> +static void tmp_objdir_reparent(const char *name UNUSED,
> +				const char *old_cwd,
> +				const char *new_cwd,
> +				void *cb_data)
> +{
> +	struct tmp_objdir *t = cb_data;
> +	char *path;
> +
> +	path = reparent_relative_path(old_cwd, new_cwd,
> +				      t->path.buf);
> +	strbuf_reset(&t->path);
> +	strbuf_addstr(&t->path, path);
> +	free(path);
> +}

Ok, at first I was a bit confused as to why we needed this logic for the
tmpdir as well. I thought reparenting as only applied to the primary
ODB, but it looks like the tmpdir was also reparented via
tmp_objdir_reapply_primary_odb().

> +
>  int tmp_objdir_destroy(struct tmp_objdir *t)
>  {
>  	int err;
> @@ -51,6 +66,7 @@ int tmp_objdir_destroy(struct tmp_objdir *t)
>  
>  	err = remove_dir_recursively(&t->path, 0);
>  
> +	chdir_notify_unregister(NULL, tmp_objdir_reparent, t);
>  	tmp_objdir_free(t);
>  
>  	return err;
> @@ -137,6 +153,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
>  	strbuf_addf(&t->path, "%s/tmp_objdir-%s-XXXXXX",
>  		    repo_get_object_directory(r), prefix);
>  
> +	if (!is_absolute_path(t->path.buf))
> +		chdir_notify_register(NULL, tmp_objdir_reparent, t);
> +
>  	if (!mkdtemp(t->path.buf)) {
>  		/* free, not destroy, as we never touched the filesystem */
>  		tmp_objdir_free(t);
[snip]
> diff --git a/tmp-objdir.h b/tmp-objdir.h
> index fceda14979..ccf800faa7 100644
> --- a/tmp-objdir.h
> +++ b/tmp-objdir.h
> @@ -68,19 +68,4 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *);
>   */
>  void tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy);
>  
> -/*
> - * If the primary object database was replaced by a temporary object directory,
> - * restore it to its original value while keeping the directory contents around.
> - * Returns NULL if the primary object database was not replaced.
> - */
> -struct tmp_objdir *tmp_objdir_unapply_primary_odb(void);
> -
> -/*
> - * Reapplies the former primary temporary object database, after potentially
> - * changing its relative path.
> - */
> -void tmp_objdir_reapply_primary_odb(struct tmp_objdir *, const char *old_cwd,
> -		const char *new_cwd);

These functions are no longer needed because each of the sources have
their paths updated directly via separate registered callbacks. Makes
sense.

-Justin

Re: [PATCH 04/17] odb: move reparenting logic into respective subsystems

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 02:39:57PM -0600, Justin Tobler wrote:
> On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > The primary object database source may be initialized with a relative
> > path. When reparenting the process to a different working directory we
> 
> I find the wording here a bit confusing. Maybe something like this would
> be a bit clearer:
> 
>   When the process changes its current working directory...

Yup, this reads clearer indeed.

> > diff --git a/odb/source-files.c b/odb/source-files.c
> > index a43a197157..df0ea9ee62 100644
> > --- a/odb/source-files.c
> > +++ b/odb/source-files.c
> > @@ -1,13 +1,28 @@
> >  #include "git-compat-util.h"
> > +#include "abspath.h"
> > +#include "chdir-notify.h"
> >  #include "object-file.h"
> >  #include "odb/source.h"
> >  #include "odb/source-files.h"
> >  #include "packfile.h"
> >  
> > +static void odb_source_files_reparent(const char *name UNUSED,
> > +				      const char *old_cwd,
> > +				      const char *new_cwd,
> > +				      void *cb_data)
> > +{
> > +	struct odb_source_files *files = cb_data;
> > +	char *path = reparent_relative_path(old_cwd, new_cwd,
> > +					    files->base.path);
> > +	free(files->base.path);
> > +	files->base.path = path;
> 
> I do find it a bit curious that we consider the "path" to be specific to
> the "files" backend, but still track it as part of the "base" ODB
> source. I suspect this will eventually change though?

Yeah, this will change eventually, but it's going to take a while to get
there. I plan to drop the "path" pointer from the base completely, as
other sources may not even have a path in the first place. But that
first requires us to address all instances where we directly access the
path

> > +}
> > +
> >  void odb_source_files_free(struct odb_source_files *files)
> >  {
> >  	if (!files)
> >  		return;
> > +	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
> >  	odb_source_loose_free(files->loose);
> >  	packfile_store_free(files->packed);
> >  	odb_source_release(&files->base);
> > @@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
> >  	files->loose = odb_source_loose_new(&files->base);
> >  	files->packed = packfile_store_new(&files->base);
> >  
> > +	/*
> > +	 * Ideally, we would only ever store absolute paths in the source. This
> > +	 * is not (yet) possible though because we access and assume relative
> > +	 * paths in the primary ODB source in some user-facing functionality.
> > +	 */
> 
> Should this be a NEEDSWORK comment? Or do we expect it to remain this
> way for the forseeable future?

Once we are able to drop the `struct odb_source::path` field it should
become feasible. So I don't think we should add a NEEDSWORK comment now,
as it might mislead fellow developers to think it's already doable and
can be worked on right away.

Patrick

[PATCH 05/17] odb/source: introduce source type for robustness

[Patrick Steinhardt]

When a caller holds a `struct odb_source`, they have no way of telling
what type the source is. This doesn't really cause any problems in the
current status quo as we only have a single type anyway, "files". But
going forward we expect to add more types, and if so it will become
necessary to tell the sources apart.

Introduce a new enum to cover this use case and assert that the given
source actually matches the target source when performing the downcast.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c |  2 +-
 odb/source-files.h |  2 ++
 odb/source.c       |  2 ++
 odb/source.h       | 16 ++++++++++++++++
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index df0ea9ee62..7496e1d9f8 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -36,7 +36,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	struct odb_source_files *files;
 
 	CALLOC_ARRAY(files, 1);
-	odb_source_init(&files->base, odb, path, local);
+	odb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);
 	files->loose = odb_source_loose_new(&files->base);
 	files->packed = packfile_store_new(&files->base);
 
diff --git a/odb/source-files.h b/odb/source-files.h
index 58753d40de..803fa995fb 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -30,6 +30,8 @@ void odb_source_files_free(struct odb_source_files *files);
  */
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
+	if (source->type != ODB_SOURCE_FILES)
+		BUG("trying to downcast source of type '%d' to files", source->type);
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index d8b2176a94..c7dcc528f6 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -13,10 +13,12 @@ struct odb_source *odb_source_new(struct object_database *odb,
 
 void odb_source_init(struct odb_source *source,
 		     struct object_database *odb,
+		     enum odb_source_type type,
 		     const char *path,
 		     bool local)
 {
 	source->odb = odb;
+	source->type = type;
 	source->local = local;
 	source->path = xstrdup(path);
 }
diff --git a/odb/source.h b/odb/source.h
index e6698b73a3..a1f2f8fdb1 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,6 +1,18 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
+enum odb_source_type {
+	/*
+	 * The "unknown" type, which should never be in use. This is type
+	 * mostly exists to catch cases where the type field remains zeroed
+	 * out.
+	 */
+	ODB_SOURCE_UNKNOWN,
+
+	/* The "files" backend that uses loose objects and packfiles. */
+	ODB_SOURCE_FILES,
+};
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -19,6 +31,9 @@ struct odb_source {
 	/* Object database that owns this object source. */
 	struct object_database *odb;
 
+	/* The type used by this source. */
+	enum odb_source_type type;
+
 	/*
 	 * Figure out whether this is the local source of the owning
 	 * repository, which would typically be its ".git/objects" directory.
@@ -58,6 +73,7 @@ struct odb_source *odb_source_new(struct object_database *odb,
  */
 void odb_source_init(struct odb_source *source,
 		     struct object_database *odb,
+		     enum odb_source_type type,
 		     const char *path,
 		     bool local);
 

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 05/17] odb/source: introduce source type for robustness

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> When a caller holds a `struct odb_source`, they have no way of telling
> what type the source is. This doesn't really cause any problems in the
> current status quo as we only have a single type anyway, "files". But
> going forward we expect to add more types, and if so it will become
> necessary to tell the sources apart.

In this patch, it looks like are only using the ODB source "type" to
know to properly BUG() out when downcasting. Do we anticipate other uses
here?

> Introduce a new enum to cover this use case and assert that the given
> source actually matches the target source when performing the downcast.

Does these mean all future source types would be required to have their
own enum value defined?

-Justin

Re: [PATCH 05/17] odb/source: introduce source type for robustness

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 02:46:38PM -0600, Justin Tobler wrote:
> On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > When a caller holds a `struct odb_source`, they have no way of telling
> > what type the source is. This doesn't really cause any problems in the
> > current status quo as we only have a single type anyway, "files". But
> > going forward we expect to add more types, and if so it will become
> > necessary to tell the sources apart.
> 
> In this patch, it looks like are only using the ODB source "type" to
> know to properly BUG() out when downcasting. Do we anticipate other uses
> here?

Yup. There are sites in Git that simply need to know the type of the
source because of functionality that is deeply entangled with the files
backend. And in such cases we'll have to determine the type of a
specific ODB source so that we can act accordingly.

The number of such callsites should be low, and they should decrease
over time. But some simply won't go away.

> > Introduce a new enum to cover this use case and assert that the given
> > source actually matches the target source when performing the downcast.
> 
> Does these mean all future source types would be required to have their
> own enum value defined?

Yes. In an integration branch I have four different backends: "files",
"loose", "packed" and "inmemory".

Patrick

Re: [PATCH 05/17] odb/source: introduce source type for robustness

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3751 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> When a caller holds a `struct odb_source`, they have no way of telling
> what type the source is. This doesn't really cause any problems in the
> current status quo as we only have a single type anyway, "files". But
> going forward we expect to add more types, and if so it will become
> necessary to tell the sources apart.
>
> Introduce a new enum to cover this use case and assert that the given
> source actually matches the target source when performing the downcast.
>

So this is what I was talking about in a previous commit, nice to see.

> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.c |  2 +-
>  odb/source-files.h |  2 ++
>  odb/source.c       |  2 ++
>  odb/source.h       | 16 ++++++++++++++++
>  4 files changed, 21 insertions(+), 1 deletion(-)
>
> diff --git a/odb/source-files.c b/odb/source-files.c
> index df0ea9ee62..7496e1d9f8 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -36,7 +36,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  	struct odb_source_files *files;
>
>  	CALLOC_ARRAY(files, 1);
> -	odb_source_init(&files->base, odb, path, local);
> +	odb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);
>  	files->loose = odb_source_loose_new(&files->base);
>  	files->packed = packfile_store_new(&files->base);
>
> diff --git a/odb/source-files.h b/odb/source-files.h
> index 58753d40de..803fa995fb 100644
> --- a/odb/source-files.h
> +++ b/odb/source-files.h
> @@ -30,6 +30,8 @@ void odb_source_files_free(struct odb_source_files *files);
>   */
>  static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
>  {
> +	if (source->type != ODB_SOURCE_FILES)
> +		BUG("trying to downcast source of type '%d' to files", source->type);
>  	return container_of(source, struct odb_source_files, base);
>  }
>
> diff --git a/odb/source.c b/odb/source.c
> index d8b2176a94..c7dcc528f6 100644
> --- a/odb/source.c
> +++ b/odb/source.c
> @@ -13,10 +13,12 @@ struct odb_source *odb_source_new(struct object_database *odb,
>
>  void odb_source_init(struct odb_source *source,
>  		     struct object_database *odb,
> +		     enum odb_source_type type,
>  		     const char *path,
>  		     bool local)
>  {
>  	source->odb = odb;
> +	source->type = type;
>  	source->local = local;
>  	source->path = xstrdup(path);
>  }
> diff --git a/odb/source.h b/odb/source.h
> index e6698b73a3..a1f2f8fdb1 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -1,6 +1,18 @@
>  #ifndef ODB_SOURCE_H
>  #define ODB_SOURCE_H
>
> +enum odb_source_type {
> +	/*
> +	 * The "unknown" type, which should never be in use. This is type

Nit: s/is//

> +	 * mostly exists to catch cases where the type field remains zeroed
> +	 * out.
> +	 */
> +	ODB_SOURCE_UNKNOWN,
> +
> +	/* The "files" backend that uses loose objects and packfiles. */
> +	ODB_SOURCE_FILES,
> +};
> +
>  /*
>   * The source is the part of the object database that stores the actual
>   * objects. It thus encapsulates the logic to read and write the specific
> @@ -19,6 +31,9 @@ struct odb_source {
>  	/* Object database that owns this object source. */
>  	struct object_database *odb;
>
> +	/* The type used by this source. */
> +	enum odb_source_type type;
> +
>  	/*
>  	 * Figure out whether this is the local source of the owning
>  	 * repository, which would typically be its ".git/objects" directory.
> @@ -58,6 +73,7 @@ struct odb_source *odb_source_new(struct object_database *odb,
>   */
>  void odb_source_init(struct odb_source *source,
>  		     struct object_database *odb,
> +		     enum odb_source_type type,
>  		     const char *path,
>  		     bool local);
>
>
> --
> 2.53.0.536.g309c995771.dirty

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH 06/17] odb/source: make `free()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 7 ++++---
 odb/source-files.h | 3 ---
 odb/source.c       | 4 +---
 odb/source.h       | 6 ++++++
 4 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 7496e1d9f8..65d7805c5a 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -18,10 +18,9 @@ static void odb_source_files_reparent(const char *name UNUSED,
 	files->base.path = path;
 }
 
-void odb_source_files_free(struct odb_source_files *files)
+static void odb_source_files_free(struct odb_source *source)
 {
-	if (!files)
-		return;
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
 	odb_source_loose_free(files->loose);
 	packfile_store_free(files->packed);
@@ -40,6 +39,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->loose = odb_source_loose_new(&files->base);
 	files->packed = packfile_store_new(&files->base);
 
+	files->base.free = odb_source_files_free;
+
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
 	 * is not (yet) possible though because we access and assume relative
diff --git a/odb/source-files.h b/odb/source-files.h
index 803fa995fb..23a3b4e04b 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -21,9 +21,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local);
 
-/* Free the object source and release all associated resources. */
-void odb_source_files_free(struct odb_source_files *files);
-
 /*
  * Cast the given object database source to the files backend. This will cause
  * a BUG in case the source doesn't use this backend.
diff --git a/odb/source.c b/odb/source.c
index c7dcc528f6..7993dcbd65 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -25,11 +25,9 @@ void odb_source_init(struct odb_source *source,
 
 void odb_source_free(struct odb_source *source)
 {
-	struct odb_source_files *files;
 	if (!source)
 		return;
-	files = odb_source_files_downcast(source);
-	odb_source_files_free(files);
+	source->free(source);
 }
 
 void odb_source_release(struct odb_source *source)
diff --git a/odb/source.h b/odb/source.h
index a1f2f8fdb1..f84da59ef0 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -52,6 +52,12 @@ struct odb_source {
 	 * the current working directory.
 	 */
 	char *path;
+
+	/*
+	 * This callback is expected to free the underlying object database source and
+	 * all associated resources. The function will never be called with a NULL pointer.
+	 */
+	void (*free)(struct odb_source *source);
 };
 
 /*

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 06/17] odb/source: make `free()` function pluggable

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.c | 7 ++++---
>  odb/source-files.h | 3 ---
>  odb/source.c       | 4 +---
>  odb/source.h       | 6 ++++++
>  4 files changed, 11 insertions(+), 9 deletions(-)
> 
> diff --git a/odb/source-files.c b/odb/source-files.c
> index 7496e1d9f8..65d7805c5a 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -18,10 +18,9 @@ static void odb_source_files_reparent(const char *name UNUSED,
>  	files->base.path = path;
>  }
>  
> -void odb_source_files_free(struct odb_source_files *files)
> +static void odb_source_files_free(struct odb_source *source)
>  {
> -	if (!files)
> -		return;
> +	struct odb_source_files *files = odb_source_files_downcast(source);

Now each callback will be responsible for downcasting to the concrete
type. Looks good.

>  	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
>  	odb_source_loose_free(files->loose);
>  	packfile_store_free(files->packed);
> @@ -40,6 +39,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  	files->loose = odb_source_loose_new(&files->base);
>  	files->packed = packfile_store_new(&files->base);
>  
> +	files->base.free = odb_source_files_free;

Callback is registered.

> +
>  	/*
>  	 * Ideally, we would only ever store absolute paths in the source. This
>  	 * is not (yet) possible though because we access and assume relative
> diff --git a/odb/source-files.h b/odb/source-files.h
> index 803fa995fb..23a3b4e04b 100644
> --- a/odb/source-files.h
> +++ b/odb/source-files.h
> @@ -21,9 +21,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
>  					      const char *path,
>  					      bool local);
>  
> -/* Free the object source and release all associated resources. */
> -void odb_source_files_free(struct odb_source_files *files);

The forward header is no longer needed as this function becomes an
internal detail of how the "files" source is impelmented.

> -
>  /*
>   * Cast the given object database source to the files backend. This will cause
>   * a BUG in case the source doesn't use this backend.
> diff --git a/odb/source.c b/odb/source.c
> index c7dcc528f6..7993dcbd65 100644
> --- a/odb/source.c
> +++ b/odb/source.c
> @@ -25,11 +25,9 @@ void odb_source_init(struct odb_source *source,
>  
>  void odb_source_free(struct odb_source *source)
>  {
> -	struct odb_source_files *files;
>  	if (!source)
>  		return;
> -	files = odb_source_files_downcast(source);
> -	odb_source_files_free(files);
> +	source->free(source);

Freeing the source can now be gone generically for different sources.
Nice. :)

>  }
>  
>  void odb_source_release(struct odb_source *source)
> diff --git a/odb/source.h b/odb/source.h
> index a1f2f8fdb1..f84da59ef0 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -52,6 +52,12 @@ struct odb_source {
>  	 * the current working directory.
>  	 */
>  	char *path;
> +
> +	/*
> +	 * This callback is expected to free the underlying object database source and
> +	 * all associated resources. The function will never be called with a NULL pointer.
> +	 */
> +	void (*free)(struct odb_source *source);

Looks good.

-Justin

[PATCH 07/17] odb/source: make `reprepare()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  7 ++-----
 odb/source-files.c |  8 ++++++++
 odb/source.h       | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/odb.c b/odb.c
index 86f7cf70a8..2cf6a53dc3 100644
--- a/odb.c
+++ b/odb.c
@@ -1119,11 +1119,8 @@ void odb_reprepare(struct object_database *o)
 	o->loaded_alternates = 0;
 	odb_prepare_alternates(o);
 
-	for (source = o->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-		odb_source_loose_reprepare(source);
-		packfile_store_reprepare(files->packed);
-	}
+	for (source = o->sources; source; source = source->next)
+		odb_source_reprepare(source);
 
 	o->approximate_object_count_valid = 0;
 
diff --git a/odb/source-files.c b/odb/source-files.c
index 65d7805c5a..d0f7ee072e 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -28,6 +28,13 @@ static void odb_source_files_free(struct odb_source *source)
 	free(files);
 }
 
+static void odb_source_files_reprepare(struct odb_source *source)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	odb_source_loose_reprepare(&files->base);
+	packfile_store_reprepare(files->packed);
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -40,6 +47,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->packed = packfile_store_new(&files->base);
 
 	files->base.free = odb_source_files_free;
+	files->base.reprepare = odb_source_files_reprepare;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index f84da59ef0..2f8132f9e1 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -58,6 +58,13 @@ struct odb_source {
 	 * all associated resources. The function will never be called with a NULL pointer.
 	 */
 	void (*free)(struct odb_source *source);
+
+	/*
+	 * This callback is expected to clear underlying caches of the object
+	 * database source. The function is called when the repository has for
+	 * example just been repacked so that new objects will become visible.
+	 */
+	void (*reprepare)(struct odb_source *source);
 };
 
 /*
@@ -97,4 +104,14 @@ void odb_source_free(struct odb_source *source);
  */
 void odb_source_release(struct odb_source *source);
 
+/*
+ * Reprepare the object database source and clear any caches. Depending on the
+ * backend used this may have the effect that concurrently-written objects
+ * become visible.
+ */
+static inline void odb_source_reprepare(struct odb_source *source)
+{
+	source->reprepare(source);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 07/17] odb/source: make `reprepare()` function pluggable

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> diff --git a/odb/source.h b/odb/source.h
> index f84da59ef0..2f8132f9e1 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -58,6 +58,13 @@ struct odb_source {
>  	 * all associated resources. The function will never be called with a NULL pointer.
>  	 */
>  	void (*free)(struct odb_source *source);
> +
> +	/*
> +	 * This callback is expected to clear underlying caches of the object
> +	 * database source. The function is called when the repository has for
> +	 * example just been repacked so that new objects will become visible.
> +	 */
> +	void (*reprepare)(struct odb_source *source);

Naive question: does repreparing a source still make sense outside of
the "files" ODB source? I almost sounds like it should be an internal
detail of the source when reading objects.

-Justin

Re: [PATCH 07/17] odb/source: make `reprepare()` function pluggable

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 03:08:16PM -0600, Justin Tobler wrote:
> On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > Introduce a new callback function in `struct odb_source` to make the
> > function pluggable.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> [snip]
> > diff --git a/odb/source.h b/odb/source.h
> > index f84da59ef0..2f8132f9e1 100644
> > --- a/odb/source.h
> > +++ b/odb/source.h
> > @@ -58,6 +58,13 @@ struct odb_source {
> >  	 * all associated resources. The function will never be called with a NULL pointer.
> >  	 */
> >  	void (*free)(struct odb_source *source);
> > +
> > +	/*
> > +	 * This callback is expected to clear underlying caches of the object
> > +	 * database source. The function is called when the repository has for
> > +	 * example just been repacked so that new objects will become visible.
> > +	 */
> > +	void (*reprepare)(struct odb_source *source);
> 
> Naive question: does repreparing a source still make sense outside of
> the "files" ODB source? I almost sounds like it should be an internal
> detail of the source when reading objects.

Ideally it would be, and I agree that repreparing is a detail that we
should in the best case never have to handle. In fact, I have plans to
eventually refactor this to a `prepare()` function as we have some sites
that want to ensure that the backends have been loaded before doing any
operation.

In that case, we'd likely add a `force` flag or something like that to
cover the repreparing use case. But ideally I agree with you that such
uses should be reduced over time.

Patrick

[PATCH 08/17] odb/source: make `close()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  6 ++----
 odb/source-files.c |  7 +++++++
 odb/source.h       | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/odb.c b/odb.c
index 2cf6a53dc3..f7487eb0df 100644
--- a/odb.c
+++ b/odb.c
@@ -1062,10 +1062,8 @@ struct object_database *odb_new(struct repository *repo,
 void odb_close(struct object_database *o)
 {
 	struct odb_source *source;
-	for (source = o->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-		packfile_store_close(files->packed);
-	}
+	for (source = o->sources; source; source = source->next)
+		odb_source_close(source);
 	close_commit_graph(o);
 }
 
diff --git a/odb/source-files.c b/odb/source-files.c
index d0f7ee072e..20a24f524a 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -28,6 +28,12 @@ static void odb_source_files_free(struct odb_source *source)
 	free(files);
 }
 
+static void odb_source_files_close(struct odb_source *source)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	packfile_store_close(files->packed);
+}
+
 static void odb_source_files_reprepare(struct odb_source *source)
 {
 	struct odb_source_files *files = odb_source_files_downcast(source);
@@ -47,6 +53,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->packed = packfile_store_new(&files->base);
 
 	files->base.free = odb_source_files_free;
+	files->base.close = odb_source_files_close;
 	files->base.reprepare = odb_source_files_reprepare;
 
 	/*
diff --git a/odb/source.h b/odb/source.h
index 2f8132f9e1..7af4900ab4 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -59,6 +59,14 @@ struct odb_source {
 	 */
 	void (*free)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to close any open resources, like for
+	 * example file descriptors or connections. The source is expected to
+	 * still be usable after it has been closed. Closed resources may need
+	 * to be reopened in that case.
+	 */
+	void (*close)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to clear underlying caches of the object
 	 * database source. The function is called when the repository has for
@@ -104,6 +112,16 @@ void odb_source_free(struct odb_source *source);
  */
 void odb_source_release(struct odb_source *source);
 
+/*
+ * Close the object database source without releasing he underlying data. The
+ * source can still be used going forward, but it first needs to be reopened.
+ * This can be useful to reduce resource usage.
+ */
+static inline void odb_source_close(struct odb_source *source)
+{
+	source->close(source);
+}
+
 /*
  * Reprepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 08/17] odb/source: make `close()` function pluggable

[Justin Tobler]

On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> +/*
> + * Close the object database source without releasing he underlying data. The
> + * source can still be used going forward, but it first needs to be reopened.
> + * This can be useful to reduce resource usage.
> + */
> +static inline void odb_source_close(struct odb_source *source)
> +{
> +	source->close(source);
> +}

Just to be safe, should we BUG()/ASSERT() in case the provide source is
NULL? Or do we expect the calling pattern to always provide an actual
source?

-Justin

Re: [PATCH 08/17] odb/source: make `close()` function pluggable

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 03:03:26PM -0600, Justin Tobler wrote:
> On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > Introduce a new callback function in `struct odb_source` to make the
> > function pluggable.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> [snip]
> > +/*
> > + * Close the object database source without releasing he underlying data. The
> > + * source can still be used going forward, but it first needs to be reopened.
> > + * This can be useful to reduce resource usage.
> > + */
> > +static inline void odb_source_close(struct odb_source *source)
> > +{
> > +	source->close(source);
> > +}
> 
> Just to be safe, should we BUG()/ASSERT() in case the provide source is
> NULL? Or do we expect the calling pattern to always provide an actual
> source?

We don't do that for any of the other wrappers either, so I'm not quite
sure why closing would be special. If this was the free function I might
agree, but otherwise I don't quite see the value.

Patrick

Re: [PATCH 08/17] odb/source: make `close()` function pluggable

[Justin Tobler]

On 26/03/05 02:23PM, Patrick Steinhardt wrote:
> On Wed, Mar 04, 2026 at 03:03:26PM -0600, Justin Tobler wrote:
> > On 26/02/23 05:17PM, Patrick Steinhardt wrote:
> > > Introduce a new callback function in `struct odb_source` to make the
> > > function pluggable.
> > > 
> > > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > > ---
> > [snip]
> > > +/*
> > > + * Close the object database source without releasing he underlying data. The
> > > + * source can still be used going forward, but it first needs to be reopened.
> > > + * This can be useful to reduce resource usage.
> > > + */
> > > +static inline void odb_source_close(struct odb_source *source)
> > > +{
> > > +	source->close(source);
> > > +}
> > 
> > Just to be safe, should we BUG()/ASSERT() in case the provide source is
> > NULL? Or do we expect the calling pattern to always provide an actual
> > source?
> 
> We don't do that for any of the other wrappers either, so I'm not quite
> sure why closing would be special. If this was the free function I might
> agree, but otherwise I don't quite see the value.

Fair, I noticed that we did it in the free function, so I was wondering
if we wanted to apply it to the other functions as well. But thinking
about it some more, there is proabably no/little value.

-Justin

Re: [PATCH 08/17] odb/source: make `close()` function pluggable

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1679 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> diff --git a/odb/source.h b/odb/source.h
> index 2f8132f9e1..7af4900ab4 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -59,6 +59,14 @@ struct odb_source {
>  	 */
>  	void (*free)(struct odb_source *source);
>
> +	/*
> +	 * This callback is expected to close any open resources, like for
> +	 * example file descriptors or connections. The source is expected to
> +	 * still be usable after it has been closed. Closed resources may need
> +	 * to be reopened in that case.
> +	 */

Nit: here we say 'may' need to be reopened...

> +	void (*close)(struct odb_source *source);
> +
>  	/*
>  	 * This callback is expected to clear underlying caches of the object
>  	 * database source. The function is called when the repository has for
> @@ -104,6 +112,16 @@ void odb_source_free(struct odb_source *source);
>   */
>  void odb_source_release(struct odb_source *source);
>
> +/*
> + * Close the object database source without releasing he underlying data. The
> + * source can still be used going forward, but it first needs to be reopened.
> + * This can be useful to reduce resource usage.
> + */

Here, we're more explicit that it does need to be reopened. I like the
latter better, this way, sources which don't need to be re-opened can
simply do a no-op. But this makes the expectation on the user side more clear.

> +static inline void odb_source_close(struct odb_source *source)
> +{
> +	source->close(source);
> +}
> +
>  /*
>   * Reprepare the object database source and clear any caches. Depending on the
>   * backend used this may have the effect that concurrently-written objects
>
> --
> 2.53.0.536.g309c995771.dirty

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 08/17] odb/source: make `close()` function pluggable

[Patrick Steinhardt]

On Thu, Mar 05, 2026 at 10:58:32AM +0000, Karthik Nayak wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > diff --git a/odb/source.h b/odb/source.h
> > index 2f8132f9e1..7af4900ab4 100644
> > --- a/odb/source.h
> > +++ b/odb/source.h
> > @@ -59,6 +59,14 @@ struct odb_source {
> >  	 */
> >  	void (*free)(struct odb_source *source);
> >
> > +	/*
> > +	 * This callback is expected to close any open resources, like for
> > +	 * example file descriptors or connections. The source is expected to
> > +	 * still be usable after it has been closed. Closed resources may need
> > +	 * to be reopened in that case.
> > +	 */
> 
> Nit: here we say 'may' need to be reopened...
> 
> > +	void (*close)(struct odb_source *source);
> > +
> >  	/*
> >  	 * This callback is expected to clear underlying caches of the object
> >  	 * database source. The function is called when the repository has for
> > @@ -104,6 +112,16 @@ void odb_source_free(struct odb_source *source);
> >   */
> >  void odb_source_release(struct odb_source *source);
> >
> > +/*
> > + * Close the object database source without releasing he underlying data. The
> > + * source can still be used going forward, but it first needs to be reopened.
> > + * This can be useful to reduce resource usage.
> > + */
> 
> Here, we're more explicit that it does need to be reopened. I like the
> latter better, this way, sources which don't need to be re-opened can
> simply do a no-op. But this makes the expectation on the user side more clear.

I consider the first comment to be catered towards the developer of a
backend, whereas the second comment is catered towards the user of these
interfaces. So I'm intentionally being a bit more lose on the first one
as we cannot assume how exactly the backend is implemented, and whteher
it even needs to open anything. For the end user though they should
treat this as if we were always reopening.

Patrick

[PATCH 09/17] odb/source: make `read_object_info()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Note that this function is a bit less straight-forward to convert
compared to the other functions. The reason here is that the logic to
read an object is:

  1. We try to read the object. If it exists we return it.

  2. If the object does not exist we reprepare the object database
     source.

  3. We then try reading the object info a second time in case the
     reprepare caused it to appear.

The second read is only supposed to happen for the packfile store
though, as reading loose objects is not impacted by repreparing the
object database.

Ideally, we'd just move this whole logic into the ODB source. But that's
not easily possible because we try to avoid the reprepare unless really
required, which is after we have found out that no other ODB source
contains the object, either. So the logic spans across multiple ODB
sources, and consequently we cannot move it into an individual source.

Instead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the
backend that we already tried to look up the object once, and that this
time around the ODB source should try to find any new objects that may
have surfaced due to an on-disk change.

With this flag, the "files" backend can trivially skip trying to re-read
the object as a loose object. Furthermore, as we know that we only try
the second read via the packfile store, we can skip repreparing loose
objects and only reprepare the packfile store.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 object-file.c      | 12 ++++++++-
 odb.c              | 22 +++++++--------
 odb.h              | 24 -----------------
 odb/source-files.c | 15 +++++++++++
 odb/source.h       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.c         | 10 ++++++-
 6 files changed, 123 insertions(+), 38 deletions(-)

diff --git a/object-file.c b/object-file.c
index 25c1146849..eefde72c7d 100644
--- a/object-file.c
+++ b/object-file.c
@@ -543,9 +543,19 @@ static int read_object_info_from_path(struct odb_source *source,
 int odb_source_loose_read_object_info(struct odb_source *source,
 				      const struct object_id *oid,
 				      struct object_info *oi,
-				      unsigned flags)
+				      enum object_info_flags flags)
 {
 	static struct strbuf buf = STRBUF_INIT;
+
+	/*
+	 * The second read shouldn't cause new loose objects to show up, unless
+	 * there was a race condition with a secondary process. We don't care
+	 * about this case though, so we simply skip reading loose objects a
+	 * second time.
+	 */
+	if (flags & OBJECT_INFO_SECOND_READ)
+		return -1;
+
 	odb_loose_path(source, &buf, oid);
 	return read_object_info_from_path(source, buf.buf, oid, oi, flags);
 }
diff --git a/odb.c b/odb.c
index f7487eb0df..c0b8cd062b 100644
--- a/odb.c
+++ b/odb.c
@@ -688,22 +688,20 @@ static int do_oid_object_info_extended(struct object_database *odb,
 	while (1) {
 		struct odb_source *source;
 
-		/* Most likely it's a loose object. */
-		for (source = odb->sources; source; source = source->next) {
-			struct odb_source_files *files = odb_source_files_downcast(source);
-			if (!packfile_store_read_object_info(files->packed, real, oi, flags) ||
-			    !odb_source_loose_read_object_info(source, real, oi, flags))
+		for (source = odb->sources; source; source = source->next)
+			if (!odb_source_read_object_info(source, real, oi, flags))
 				return 0;
-		}
 
-		/* Not a loose object; someone else may have just packed it. */
+		/*
+		 * When the object hasn't been found we try a second read and
+		 * tell the sources so. This may cause them to invalidate
+		 * caches or reload on-disk state.
+		 */
 		if (!(flags & OBJECT_INFO_QUICK)) {
-			odb_reprepare(odb->repo->objects);
-			for (source = odb->sources; source; source = source->next) {
-				struct odb_source_files *files = odb_source_files_downcast(source);
-				if (!packfile_store_read_object_info(files->packed, real, oi, flags))
+			for (source = odb->sources; source; source = source->next)
+				if (!odb_source_read_object_info(source, real, oi,
+								 flags | OBJECT_INFO_SECOND_READ))
 					return 0;
-			}
 		}
 
 		/*
diff --git a/odb.h b/odb.h
index e13b5b7c44..70ffb033f9 100644
--- a/odb.h
+++ b/odb.h
@@ -339,30 +339,6 @@ struct object_info {
  */
 #define OBJECT_INFO_INIT { 0 }
 
-/* Flags that can be passed to `odb_read_object_info_extended()`. */
-enum object_info_flags {
-	/* Invoke lookup_replace_object() on the given hash. */
-	OBJECT_INFO_LOOKUP_REPLACE = (1 << 0),
-
-	/* Do not reprepare object sources when the first lookup has failed. */
-	OBJECT_INFO_QUICK = (1 << 1),
-
-	/*
-	 * Do not attempt to fetch the object if missing (even if fetch_is_missing is
-	 * nonzero).
-	 */
-	OBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),
-
-	/* Die if object corruption (not just an object being missing) was detected. */
-	OBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),
-
-	/*
-	 * This is meant for bulk prefetching of missing blobs in a partial
-	 * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.
-	 */
-	OBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),
-};
-
 /*
  * Read object info from the object database and populate the `object_info`
  * structure. Returns 0 on success, a negative error code otherwise.
diff --git a/odb/source-files.c b/odb/source-files.c
index 20a24f524a..f2969a1214 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -41,6 +41,20 @@ static void odb_source_files_reprepare(struct odb_source *source)
 	packfile_store_reprepare(files->packed);
 }
 
+static int odb_source_files_read_object_info(struct odb_source *source,
+					     const struct object_id *oid,
+					     struct object_info *oi,
+					     enum object_info_flags flags)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+
+	if (!packfile_store_read_object_info(files->packed, oid, oi, flags) ||
+	    !odb_source_loose_read_object_info(source, oid, oi, flags))
+		return 0;
+
+	return -1;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -55,6 +69,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
 	files->base.reprepare = odb_source_files_reprepare;
+	files->base.read_object_info = odb_source_files_read_object_info;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 7af4900ab4..45563de61e 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -13,6 +13,45 @@ enum odb_source_type {
 	ODB_SOURCE_FILES,
 };
 
+/* Flags that can be passed to `odb_read_object_info_extended()`. */
+enum object_info_flags {
+	/* Invoke lookup_replace_object() on the given hash. */
+	OBJECT_INFO_LOOKUP_REPLACE = (1 << 0),
+
+	/* Do not reprepare object sources when the first lookup has failed. */
+	OBJECT_INFO_QUICK = (1 << 1),
+
+	/*
+	 * Do not attempt to fetch the object if missing (even if fetch_is_missing is
+	 * nonzero).
+	 */
+	OBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),
+
+	/* Die if object corruption (not just an object being missing) was detected. */
+	OBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),
+
+	/*
+	 * We have already tried reading the object, but it couldn't be found
+	 * via any of the attached sources, and are now doing a second read.
+	 * This second read asks the individual sources to also evaluate
+	 * whether any on-disk state may have changed that may have caused the
+	 * object to appear.
+	 *
+	 * This flag is for internal use, only. The second read only occurs
+	 * when `OBJECT_INFO_QUICK` was not passed.
+	 */
+	OBJECT_INFO_SECOND_READ = (1 << 4),
+
+	/*
+	 * This is meant for bulk prefetching of missing blobs in a partial
+	 * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.
+	 */
+	OBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),
+};
+
+struct object_id;
+struct object_info;
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -73,6 +112,33 @@ struct odb_source {
 	 * example just been repacked so that new objects will become visible.
 	 */
 	void (*reprepare)(struct odb_source *source);
+
+	/*
+	 * This callback is expected to read object information from the object
+	 * database source. The object info will be partially populated with
+	 * pointers for each bit of information that was requested by the
+	 * caller.
+	 *
+	 * The flags field is a combination of `OBJECT_INFO` flags. Only the
+	 * following fields need to be handled by the backend:
+	 *
+	 *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without
+	 *     re-verifying the data.
+	 *
+	 *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object
+	 *     lookup has failed and that the object sources should check
+	 *     whether any of its on-disk state has changed that may have
+	 *     caused the object to appear. Sources are free to ignore the
+	 *     second read in case they know that the first read would have
+	 *     already surfaced the object without reloading any on-disk state.
+	 *
+	 * The callback is expected to return a negative error code in case
+	 * reading the object has failed, 0 otherwise.
+	 */
+	int (*read_object_info)(struct odb_source *source,
+				const struct object_id *oid,
+				struct object_info *oi,
+				enum object_info_flags flags);
 };
 
 /*
@@ -132,4 +198,16 @@ static inline void odb_source_reprepare(struct odb_source *source)
 	source->reprepare(source);
 }
 
+/*
+ * Read an object from the object database source identified by its object ID.
+ * Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_read_object_info(struct odb_source *source,
+					      const struct object_id *oid,
+					      struct object_info *oi,
+					      enum object_info_flags flags)
+{
+	return source->read_object_info(source, oid, oi, flags);
+}
+
 #endif
diff --git a/packfile.c b/packfile.c
index da1c0dfa39..71db10e7c6 100644
--- a/packfile.c
+++ b/packfile.c
@@ -2181,11 +2181,19 @@ int packfile_store_freshen_object(struct packfile_store *store,
 int packfile_store_read_object_info(struct packfile_store *store,
 				    const struct object_id *oid,
 				    struct object_info *oi,
-				    enum object_info_flags flags UNUSED)
+				    enum object_info_flags flags)
 {
 	struct pack_entry e;
 	int ret;
 
+	/*
+	 * In case the first read didn't surface the object, we have to reload
+	 * packfiles. This may cause us to discover new packfiles that have
+	 * been added since the last time we have prepared the packfile store.
+	 */
+	if (flags & OBJECT_INFO_SECOND_READ)
+		packfile_store_reprepare(store);
+
 	if (!find_pack_entry(store, oid, &e))
 		return 1;
 

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 09/17] odb/source: make `read_object_info()` function pluggable

[Justin Tobler]

On 26/02/23 05:18PM, Patrick Steinhardt wrote:
> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
> 
> Note that this function is a bit less straight-forward to convert
> compared to the other functions. The reason here is that the logic to
> read an object is:
> 
>   1. We try to read the object. If it exists we return it.
> 
>   2. If the object does not exist we reprepare the object database
>      source.
> 
>   3. We then try reading the object info a second time in case the
>      reprepare caused it to appear.
> 
> The second read is only supposed to happen for the packfile store
> though, as reading loose objects is not impacted by repreparing the
> object database.
> 
> Ideally, we'd just move this whole logic into the ODB source. But that's
> not easily possible because we try to avoid the reprepare unless really
> required, which is after we have found out that no other ODB source
> contains the object, either. So the logic spans across multiple ODB
> sources, and consequently we cannot move it into an individual source.

Ok, I think gives a bit more context around one of my question in a
previous patch. So IIUC, when reading objects, that the object could
have been repacked and thus no longer discoverable from the current Git
process. We could just reprepare the ODB source immediately, but it
could be that the object exists in another ODB source so we should check
other sources first. Only if the object can't be found in other sources,
then we should attempt to reprepare the ODB sources in search of the
object.

> Instead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the
> backend that we already tried to look up the object once, and that this
> time around the ODB source should try to find any new objects that may
> have surfaced due to an on-disk change.

Ok, now that the "files" ODB source combines the loose and packed
sources, we need a way to differentiate between first and second time
reads to void reading loose objects again. Makes sense.

-Justin

[PATCH 10/17] odb/source: make `read_object_stream()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 12 ++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 odb/streaming.c    |  9 ++-------
 3 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index f2969a1214..b50a1f5492 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,
 	return -1;
 }
 
+static int odb_source_files_read_object_stream(struct odb_read_stream **out,
+					       struct odb_source *source,
+					       const struct object_id *oid)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	if (!packfile_store_read_object_stream(out, files->packed, oid) ||
+	    !odb_source_loose_read_object_stream(out, source, oid))
+		return 0;
+	return -1;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -70,6 +81,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.close = odb_source_files_close;
 	files->base.reprepare = odb_source_files_reprepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
+	files->base.read_object_stream = odb_source_files_read_object_stream;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 45563de61e..edb425fdef 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -51,6 +51,7 @@ enum object_info_flags {
 
 struct object_id;
 struct object_info;
+struct odb_read_stream;
 
 /*
  * The source is the part of the object database that stores the actual
@@ -139,6 +140,17 @@ struct odb_source {
 				const struct object_id *oid,
 				struct object_info *oi,
 				enum object_info_flags flags);
+
+	/*
+	 * This callback is expected to create a new read stream that can be
+	 * used to stream the object identified by the given ID.
+	 *
+	 * The callback is expected to return a negative error code in case
+	 * creating the object stream has failed, 0 otherwise.
+	 */
+	int (*read_object_stream)(struct odb_read_stream **out,
+				  struct odb_source *source,
+				  const struct object_id *oid);
 };
 
 /*
@@ -210,4 +222,15 @@ static inline int odb_source_read_object_info(struct odb_source *source,
 	return source->read_object_info(source, oid, oi, flags);
 }
 
+/*
+ * Create a new read stream for the given object ID. Returns 0 on success, a
+ * negative error code otherwise.
+ */
+static inline int odb_source_read_object_stream(struct odb_read_stream **out,
+						struct odb_source *source,
+						const struct object_id *oid)
+{
+	return source->read_object_stream(out, source, oid);
+}
+
 #endif
diff --git a/odb/streaming.c b/odb/streaming.c
index 19cda9407d..a4355cd245 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -6,11 +6,9 @@
 #include "convert.h"
 #include "environment.h"
 #include "repository.h"
-#include "object-file.h"
 #include "odb.h"
 #include "odb/streaming.h"
 #include "replace-object.h"
-#include "packfile.h"
 
 #define FILTER_BUFFER (1024*16)
 
@@ -186,12 +184,9 @@ static int istream_source(struct odb_read_stream **out,
 	struct odb_source *source;
 
 	odb_prepare_alternates(odb);
-	for (source = odb->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (!packfile_store_read_object_stream(out, files->packed, oid) ||
-		    !odb_source_loose_read_object_stream(out, source, oid))
+	for (source = odb->sources; source; source = source->next)
+		if (!odb_source_read_object_stream(out, source, oid))
 			return 0;
-	}
 
 	return open_istream_incore(out, odb, oid);
 }

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 10/17] odb/source: make `read_object_stream()` function pluggable

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.c | 12 ++++++++++++
>  odb/source.h       | 23 +++++++++++++++++++++++
>  odb/streaming.c    |  9 ++-------
>  3 files changed, 37 insertions(+), 7 deletions(-)
>
> diff --git a/odb/source-files.c b/odb/source-files.c
> index f2969a1214..b50a1f5492 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,
>  	return -1;
>  }
>
> +static int odb_source_files_read_object_stream(struct odb_read_stream **out,
> +					       struct odb_source *source,
> +					       const struct object_id *oid)
> +{
> +	struct odb_source_files *files = odb_source_files_downcast(source);
> +	if (!packfile_store_read_object_stream(out, files->packed, oid) ||
> +	    !odb_source_loose_read_object_stream(out, source, oid))
> +		return 0;
> +	return -1;

Same issue here regarding loss of error code propagation.

[snip]

The patch looks good otherwise.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 10/17] odb/source: make `read_object_stream()` function pluggable

[Patrick Steinhardt]

On Thu, Mar 05, 2026 at 11:13:24AM +0000, Karthik Nayak wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> > diff --git a/odb/source-files.c b/odb/source-files.c
> > index f2969a1214..b50a1f5492 100644
> > --- a/odb/source-files.c
> > +++ b/odb/source-files.c
> > @@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,
> >  	return -1;
> >  }
> >
> > +static int odb_source_files_read_object_stream(struct odb_read_stream **out,
> > +					       struct odb_source *source,
> > +					       const struct object_id *oid)
> > +{
> > +	struct odb_source_files *files = odb_source_files_downcast(source);
> > +	if (!packfile_store_read_object_stream(out, files->packed, oid) ||
> > +	    !odb_source_loose_read_object_stream(out, source, oid))
> > +		return 0;
> > +	return -1;
> 
> Same issue here regarding loss of error code propagation.

That's fair, but we didn't propagate the exact error code beforehand,
either. Furthermore, it's not even specified what different error codes
would mean, so returning `-1` seems good enough to me. It would also
mean that we only ever propagate error codes from reading loose objects,
not from the packfiles, which would be leaking an implementation detail.

So overall I think this is okay as-is, but let me know in case you
disagree.

Patrick

[PATCH 11/17] odb/source: make `for_each_object()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 12 +----------
 odb.h              | 12 -----------
 odb/source-files.c | 23 +++++++++++++++++++++
 odb/source.h       | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 83 insertions(+), 23 deletions(-)

diff --git a/odb.c b/odb.c
index c0b8cd062b..494a3273cf 100644
--- a/odb.c
+++ b/odb.c
@@ -984,20 +984,10 @@ int odb_for_each_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (struct odb_source *source = odb->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-
 		if (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)
 			continue;
 
-		if (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {
-			ret = odb_source_loose_for_each_object(source, request,
-							       cb, cb_data, flags);
-			if (ret)
-				return ret;
-		}
-
-		ret = packfile_store_for_each_object(files->packed, request,
-						     cb, cb_data, flags);
+		ret = odb_source_for_each_object(source, request, cb, cb_data, flags);
 		if (ret)
 			return ret;
 	}
diff --git a/odb.h b/odb.h
index 70ffb033f9..692d9029ef 100644
--- a/odb.h
+++ b/odb.h
@@ -432,18 +432,6 @@ enum odb_for_each_object_flags {
 	ODB_FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS = (1<<4),
 };
 
-/*
- * A callback function that can be used to iterate through objects. If given,
- * the optional `oi` parameter will be populated the same as if you would call
- * `odb_read_object_info()`.
- *
- * Returning a non-zero error code will cause iteration to abort. The error
- * code will be propagated.
- */
-typedef int (*odb_for_each_object_cb)(const struct object_id *oid,
-				      struct object_info *oi,
-				      void *cb_data);
-
 /*
  * Iterate through all objects contained in the object database. Note that
  * objects may be iterated over multiple times in case they are either stored
diff --git a/odb/source-files.c b/odb/source-files.c
index b50a1f5492..d8ef1d8237 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -66,6 +66,28 @@ static int odb_source_files_read_object_stream(struct odb_read_stream **out,
 	return -1;
 }
 
+static int odb_source_files_for_each_object(struct odb_source *source,
+					    const struct object_info *request,
+					    odb_for_each_object_cb cb,
+					    void *cb_data,
+					    unsigned flags)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	int ret;
+
+	if (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {
+		ret = odb_source_loose_for_each_object(source, request, cb, cb_data, flags);
+		if (ret)
+			return ret;
+	}
+
+	ret = packfile_store_for_each_object(files->packed, request, cb, cb_data, flags);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -82,6 +104,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.reprepare = odb_source_files_reprepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
+	files->base.for_each_object = odb_source_files_for_each_object;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index edb425fdef..35aa78e140 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -53,6 +53,18 @@ struct object_id;
 struct object_info;
 struct odb_read_stream;
 
+/*
+ * A callback function that can be used to iterate through objects. If given,
+ * the optional `oi` parameter will be populated the same as if you would call
+ * `odb_read_object_info()`.
+ *
+ * Returning a non-zero error code will cause iteration to abort. The error
+ * code will be propagated.
+ */
+typedef int (*odb_for_each_object_cb)(const struct object_id *oid,
+				      struct object_info *oi,
+				      void *cb_data);
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -151,6 +163,27 @@ struct odb_source {
 	int (*read_object_stream)(struct odb_read_stream **out,
 				  struct odb_source *source,
 				  const struct object_id *oid);
+
+	/*
+	 * This callback is expected to iterate over all objects stored in this
+	 * source and invoke the callback function for each of them. It is
+	 * valid to yield the same object multiple time. A non-zero exit code
+	 * from the object callback shall abort iteration.
+	 *
+	 * The optional `oi` structure shall be populated similar to how an individual
+	 * call to `odb_source_read_object_info()` would have behaved. If the caller
+	 * passes a `NULL` pointer then the object itself shall not be read.
+	 *
+	 * The callback is expected to return a negative error code in case the
+	 * iteration has failed to read all objects, 0 otherwise. When the
+	 * callback function returns a non-zero error code then that error code
+	 * should be returned.
+	 */
+	int (*for_each_object)(struct odb_source *source,
+			       const struct object_info *request,
+			       odb_for_each_object_cb cb,
+			       void *cb_data,
+			       unsigned flags);
 };
 
 /*
@@ -233,4 +266,30 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,
 	return source->read_object_stream(out, source, oid);
 }
 
+/*
+ * Iterate through all objects contained in the given source and invoke the
+ * callback function for each of them. Returning a non-zero code from the
+ * callback function aborts iteration. There is no guarantee that objects
+ * are only iterated over once.
+ *
+ * The optional `oi` structure shall be populated similar to how an individual
+ * call to `odb_source_read_object_info()` would have behaved. If the caller
+ * passes a `NULL` pointer then the object itself shall not be read.
+ *
+ * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may
+ * apply to a specific backend, so whether or not they are honored is defined
+ * by the implementation.
+ *
+ * Returns 0 when all objects have been iterated over, a negative error code in
+ * case iteration has failed, or a non-zero value returned from the callback.
+ */
+static inline int odb_source_for_each_object(struct odb_source *source,
+					     const struct object_info *request,
+					     odb_for_each_object_cb cb,
+					     void *cb_data,
+					     unsigned flags)
+{
+	return source->for_each_object(source, request, cb, cb_data, flags);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 11/17] odb/source: make `for_each_object()` function pluggable

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3060 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb.c              | 12 +----------
>  odb.h              | 12 -----------
>  odb/source-files.c | 23 +++++++++++++++++++++
>  odb/source.h       | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 83 insertions(+), 23 deletions(-)
>

[snip]

> @@ -151,6 +163,27 @@ struct odb_source {
>  	int (*read_object_stream)(struct odb_read_stream **out,
>  				  struct odb_source *source,
>  				  const struct object_id *oid);
> +
> +	/*
> +	 * This callback is expected to iterate over all objects stored in this
> +	 * source and invoke the callback function for each of them. It is
> +	 * valid to yield the same object multiple time. A non-zero exit code
> +	 * from the object callback shall abort iteration.
> +	 *
> +	 * The optional `oi` structure shall be populated similar to how an individual
> +	 * call to `odb_source_read_object_info()` would have behaved. If the caller
> +	 * passes a `NULL` pointer then the object itself shall not be read.
> +	 *
> +	 * The callback is expected to return a negative error code in case the
> +	 * iteration has failed to read all objects, 0 otherwise. When the
> +	 * callback function returns a non-zero error code then that error code
> +	 * should be returned.
> +	 */
> +	int (*for_each_object)(struct odb_source *source,
> +			       const struct object_info *request,
> +			       odb_for_each_object_cb cb,
> +			       void *cb_data,
> +			       unsigned flags);
>  };
>
>  /*
> @@ -233,4 +266,30 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,
>  	return source->read_object_stream(out, source, oid);
>  }
>
> +/*
> + * Iterate through all objects contained in the given source and invoke the
> + * callback function for each of them. Returning a non-zero code from the
> + * callback function aborts iteration. There is no guarantee that objects
> + * are only iterated over once.
> + *
> + * The optional `oi` structure shall be populated similar to how an individual
> + * call to `odb_source_read_object_info()` would have behaved. If the caller
> + * passes a `NULL` pointer then the object itself shall not be read.
> + *
> + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may
> + * apply to a specific backend, so whether or not they are honored is defined
> + * by the implementation.
> + *
> + * Returns 0 when all objects have been iterated over, a negative error code in
> + * case iteration has failed, or a non-zero value returned from the callback.
> + */
> +static inline int odb_source_for_each_object(struct odb_source *source,
> +					     const struct object_info *request,
> +					     odb_for_each_object_cb cb,
> +					     void *cb_data,
> +					     unsigned flags)
> +{
> +	return source->for_each_object(source, request, cb, cb_data, flags);
> +}
> +
>  #endif
>
> --
> 2.53.0.536.g309c995771.dirty

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 11/17] odb/source: make `for_each_object()` function pluggable

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 3649 bytes --]

[snip]

> diff --git a/odb/source.h b/odb/source.h
> index edb425fdef..35aa78e140 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -53,6 +53,18 @@ struct object_id;
>  struct object_info;
>  struct odb_read_stream;
>
> +/*
> + * A callback function that can be used to iterate through objects. If given,
> + * the optional `oi` parameter will be populated the same as if you would call
> + * `odb_read_object_info()`.
> + *
> + * Returning a non-zero error code will cause iteration to abort. The error
> + * code will be propagated.
> + */
> +typedef int (*odb_for_each_object_cb)(const struct object_id *oid,
> +				      struct object_info *oi,
> +				      void *cb_data);
> +
>  /*
>   * The source is the part of the object database that stores the actual
>   * objects. It thus encapsulates the logic to read and write the specific
> @@ -151,6 +163,27 @@ struct odb_source {
>  	int (*read_object_stream)(struct odb_read_stream **out,
>  				  struct odb_source *source,
>  				  const struct object_id *oid);
> +
> +	/*
> +	 * This callback is expected to iterate over all objects stored in this

This isn't a callback though, this is a function which calls the
callback, right?

> +	 * source and invoke the callback function for each of them. It is
> +	 * valid to yield the same object multiple time. A non-zero exit code
> +	 * from the object callback shall abort iteration.
> +	 *
> +	 * The optional `oi` structure shall be populated similar to how an individual
> +	 * call to `odb_source_read_object_info()` would have behaved. If the caller
> +	 * passes a `NULL` pointer then the object itself shall not be read.
> +	 *

Nit: here and below, we talk about the `oi` structure, but that's in the
callback function, maybe we should clarify that.

> +	 * The callback is expected to return a negative error code in case the
> +	 * iteration has failed to read all objects, 0 otherwise. When the
> +	 * callback function returns a non-zero error code then that error code
> +	 * should be returned.
> +	 */
> +	int (*for_each_object)(struct odb_source *source,
> +			       const struct object_info *request,
> +			       odb_for_each_object_cb cb,
> +			       void *cb_data,
> +			       unsigned flags);
>  };
>
>  /*
> @@ -233,4 +266,30 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,
>  	return source->read_object_stream(out, source, oid);
>  }
>
> +/*
> + * Iterate through all objects contained in the given source and invoke the
> + * callback function for each of them. Returning a non-zero code from the
> + * callback function aborts iteration. There is no guarantee that objects
> + * are only iterated over once.
> + *
> + * The optional `oi` structure shall be populated similar to how an individual
> + * call to `odb_source_read_object_info()` would have behaved. If the caller
> + * passes a `NULL` pointer then the object itself shall not be read.
> + *
> + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may
> + * apply to a specific backend, so whether or not they are honored is defined
> + * by the implementation.
> + *
> + * Returns 0 when all objects have been iterated over, a negative error code in
> + * case iteration has failed, or a non-zero value returned from the callback.
> + */
> +static inline int odb_source_for_each_object(struct odb_source *source,
> +					     const struct object_info *request,
> +					     odb_for_each_object_cb cb,
> +					     void *cb_data,
> +					     unsigned flags)
> +{
> +	return source->for_each_object(source, request, cb, cb_data, flags);
> +}
> +
>  #endif
>
> --
> 2.53.0.536.g309c995771.dirty

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

Re: [PATCH 11/17] odb/source: make `for_each_object()` function pluggable

[Patrick Steinhardt]

On Thu, Mar 05, 2026 at 01:07:15PM +0000, Karthik Nayak wrote:
> [snip]
> 
> > diff --git a/odb/source.h b/odb/source.h
> > index edb425fdef..35aa78e140 100644
> > --- a/odb/source.h
> > +++ b/odb/source.h
> > @@ -151,6 +163,27 @@ struct odb_source {
> >  	int (*read_object_stream)(struct odb_read_stream **out,
> >  				  struct odb_source *source,
> >  				  const struct object_id *oid);
> > +
> > +	/*
> > +	 * This callback is expected to iterate over all objects stored in this
> 
> This isn't a callback though, this is a function which calls the
> callback, right?

No, this is the callback function in the `struct odb_source`. That
callback in turn ends up invoking another callback though :)

> > +	 * source and invoke the callback function for each of them. It is
> > +	 * valid to yield the same object multiple time. A non-zero exit code
> > +	 * from the object callback shall abort iteration.
> > +	 *
> > +	 * The optional `oi` structure shall be populated similar to how an individual
> > +	 * call to `odb_source_read_object_info()` would have behaved. If the caller
> > +	 * passes a `NULL` pointer then the object itself shall not be read.
> > +	 *
> 
> Nit: here and below, we talk about the `oi` structure, but that's in the
> callback function, maybe we should clarify that.

Ah, this is still somewhat stale from an earlier iteration where `oi`
and `request` were the same thing, and `request` was non-const. Will
fix.

Patrick

[PATCH 12/17] odb/source: make `freshen_object()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 12 ++----------
 odb/source-files.c | 11 +++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 3 files changed, 36 insertions(+), 10 deletions(-)

diff --git a/odb.c b/odb.c
index 494a3273cf..c9f42c5afd 100644
--- a/odb.c
+++ b/odb.c
@@ -959,18 +959,10 @@ int odb_freshen_object(struct object_database *odb,
 		       const struct object_id *oid)
 {
 	struct odb_source *source;
-
 	odb_prepare_alternates(odb);
-	for (source = odb->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-
-		if (packfile_store_freshen_object(files->packed, oid))
+	for (source = odb->sources; source; source = source->next)
+		if (odb_source_freshen_object(source, oid))
 			return 1;
-
-		if (odb_source_loose_freshen_object(source, oid))
-			return 1;
-	}
-
 	return 0;
 }
 
diff --git a/odb/source-files.c b/odb/source-files.c
index d8ef1d8237..a6447909e0 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -88,6 +88,16 @@ static int odb_source_files_for_each_object(struct odb_source *source,
 	return 0;
 }
 
+static int odb_source_files_freshen_object(struct odb_source *source,
+					   const struct object_id *oid)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	if (packfile_store_freshen_object(files->packed, oid) ||
+	    odb_source_loose_freshen_object(source, oid))
+		return 1;
+	return 0;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -105,6 +115,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
 	files->base.for_each_object = odb_source_files_for_each_object;
+	files->base.freshen_object = odb_source_files_freshen_object;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 35aa78e140..9324fce2ba 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -184,6 +184,18 @@ struct odb_source {
 			       odb_for_each_object_cb cb,
 			       void *cb_data,
 			       unsigned flags);
+
+	/*
+	 * This callback is expected to freshen the given object so that its
+	 * last access time is set to the current time. This is used to ensure
+	 * that objects that are recent will not get garbage collected even if
+	 * they were unreachable.
+	 *
+	 * Returns 0 in case the object does not exist, 1 in case the object
+	 * has been freshened.
+	 */
+	int (*freshen_object)(struct odb_source *source,
+			      const struct object_id *oid);
 };
 
 /*
@@ -292,4 +304,15 @@ static inline int odb_source_for_each_object(struct odb_source *source,
 	return source->for_each_object(source, request, cb, cb_data, flags);
 }
 
+/*
+ * Freshen an object in the object database by updating its timestamp.
+ * Returns 1 in case the object has been freshened, 0 in case the object does
+ * not exist.
+ */
+static inline int odb_source_freshen_object(struct odb_source *source,
+					    const struct object_id *oid)
+{
+	return source->freshen_object(source, oid);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

[PATCH 13/17] odb/source: make `write_object()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  4 ++--
 odb/source-files.c | 12 ++++++++++++
 odb/source.h       | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/odb.c b/odb.c
index c9f42c5afd..5eb60063dc 100644
--- a/odb.c
+++ b/odb.c
@@ -1005,8 +1005,8 @@ int odb_write_object_ext(struct object_database *odb,
 			 struct object_id *compat_oid,
 			 unsigned flags)
 {
-	return odb_source_loose_write_object(odb->sources, buf, len, type,
-					     oid, compat_oid, flags);
+	return odb_source_write_object(odb->sources, buf, len, type,
+				       oid, compat_oid, flags);
 }
 
 int odb_write_object_stream(struct object_database *odb,
diff --git a/odb/source-files.c b/odb/source-files.c
index a6447909e0..67c2aff659 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -98,6 +98,17 @@ static int odb_source_files_freshen_object(struct odb_source *source,
 	return 0;
 }
 
+static int odb_source_files_write_object(struct odb_source *source,
+					 const void *buf, unsigned long len,
+					 enum object_type type,
+					 struct object_id *oid,
+					 struct object_id *compat_oid,
+					 unsigned flags)
+{
+	return odb_source_loose_write_object(source, buf, len, type,
+					     oid, compat_oid, flags);
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -116,6 +127,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.read_object_stream = odb_source_files_read_object_stream;
 	files->base.for_each_object = odb_source_files_for_each_object;
 	files->base.freshen_object = odb_source_files_freshen_object;
+	files->base.write_object = odb_source_files_write_object;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 9324fce2ba..a6ef7f782c 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,6 +1,8 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
+#include "object.h"
+
 enum odb_source_type {
 	/*
 	 * The "unknown" type, which should never be in use. This is type
@@ -196,6 +198,24 @@ struct odb_source {
 	 */
 	int (*freshen_object)(struct odb_source *source,
 			      const struct object_id *oid);
+
+	/*
+	 * This callback is expected to persist the given object into the
+	 * object source. In case the object already exists it shall be
+	 * freshened.
+	 *
+	 * The flags field is a combination of `WRITE_OBJECT` flags.
+	 *
+	 * The resulting object ID (and optionally the compatibility object ID)
+	 * shall be written into the out pointers. The callback is expected to
+	 * return 0 on success, a negative error code otherwise.
+	 */
+	int (*write_object)(struct odb_source *source,
+			    const void *buf, unsigned long len,
+			    enum object_type type,
+			    struct object_id *oid,
+			    struct object_id *compat_oid,
+			    unsigned flags);
 };
 
 /*
@@ -315,4 +335,20 @@ static inline int odb_source_freshen_object(struct odb_source *source,
 	return source->freshen_object(source, oid);
 }
 
+/*
+ * Write an object into the object database source. Returns 0 on success, a
+ * negative error code otherwise. Populates the given out pointers for the
+ * object ID and the compatibility object ID, if non-NULL.
+ */
+static inline int odb_source_write_object(struct odb_source *source,
+					  const void *buf, unsigned long len,
+					  enum object_type type,
+					  struct object_id *oid,
+					  struct object_id *compat_oid,
+					  unsigned flags)
+{
+	return source->write_object(source, buf, len, type, oid,
+				    compat_oid, flags);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

[PATCH 14/17] odb/source: make `write_object_stream()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  2 +-
 odb/source-files.c |  9 +++++++++
 odb/source.h       | 28 ++++++++++++++++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/odb.c b/odb.c
index 5eb60063dc..f439de9db2 100644
--- a/odb.c
+++ b/odb.c
@@ -1013,7 +1013,7 @@ int odb_write_object_stream(struct object_database *odb,
 			    struct odb_write_stream *stream, size_t len,
 			    struct object_id *oid)
 {
-	return odb_source_loose_write_stream(odb->sources, stream, len, oid);
+	return odb_source_write_object_stream(odb->sources, stream, len, oid);
 }
 
 struct object_database *odb_new(struct repository *repo,
diff --git a/odb/source-files.c b/odb/source-files.c
index 67c2aff659..b8844f11b7 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -109,6 +109,14 @@ static int odb_source_files_write_object(struct odb_source *source,
 					     oid, compat_oid, flags);
 }
 
+static int odb_source_files_write_object_stream(struct odb_source *source,
+						struct odb_write_stream *stream,
+						size_t len,
+						struct object_id *oid)
+{
+	return odb_source_loose_write_stream(source, stream, len, oid);
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -128,6 +136,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.for_each_object = odb_source_files_for_each_object;
 	files->base.freshen_object = odb_source_files_freshen_object;
 	files->base.write_object = odb_source_files_write_object;
+	files->base.write_object_stream = odb_source_files_write_object_stream;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index a6ef7f782c..ddce43eb20 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -54,6 +54,7 @@ enum object_info_flags {
 struct object_id;
 struct object_info;
 struct odb_read_stream;
+struct odb_write_stream;
 
 /*
  * A callback function that can be used to iterate through objects. If given,
@@ -216,6 +217,18 @@ struct odb_source {
 			    struct object_id *oid,
 			    struct object_id *compat_oid,
 			    unsigned flags);
+
+	/*
+	 * This callback is expected to persist the given object stream into
+	 * the object source.
+	 *
+	 * The resulting object ID shall be written into the out pointer. The
+	 * callback is expected to return 0 on success, a negative error code
+	 * otherwise.
+	 */
+	int (*write_object_stream)(struct odb_source *source,
+				   struct odb_write_stream *stream, size_t len,
+				   struct object_id *oid);
 };
 
 /*
@@ -351,4 +364,19 @@ static inline int odb_source_write_object(struct odb_source *source,
 				    compat_oid, flags);
 }
 
+/*
+ * Write an object into the object database source via a stream. The overall
+ * length of the object must be known in advance.
+ *
+ * Return 0 on success, a negative error code otherwise. Populates the given
+ * out pointer for the object ID.
+ */
+static inline int odb_source_write_object_stream(struct odb_source *source,
+						 struct odb_write_stream *stream,
+						 size_t len,
+						 struct object_id *oid)
+{
+	return source->write_object_stream(source, stream, len, oid);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

[PATCH 15/17] odb/source: make `read_alternates()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 26 ++++----------------------
 odb.h              |  5 +++++
 odb/source-files.c | 22 ++++++++++++++++++++++
 odb/source.h       | 29 +++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 22 deletions(-)

diff --git a/odb.c b/odb.c
index f439de9db2..d9424cdfd0 100644
--- a/odb.c
+++ b/odb.c
@@ -131,10 +131,10 @@ static bool odb_is_source_usable(struct object_database *o, const char *path)
 	return usable;
 }
 
-static void parse_alternates(const char *string,
-			     int sep,
-			     const char *relative_base,
-			     struct strvec *out)
+void parse_alternates(const char *string,
+		      int sep,
+		      const char *relative_base,
+		      struct strvec *out)
 {
 	struct strbuf pathbuf = STRBUF_INIT;
 	struct strbuf buf = STRBUF_INIT;
@@ -198,24 +198,6 @@ static void parse_alternates(const char *string,
 	strbuf_release(&buf);
 }
 
-static void odb_source_read_alternates(struct odb_source *source,
-				       struct strvec *out)
-{
-	struct strbuf buf = STRBUF_INIT;
-	char *path;
-
-	path = xstrfmt("%s/info/alternates", source->path);
-	if (strbuf_read_file(&buf, path, 1024) < 0) {
-		warn_on_fopen_errors(path);
-		free(path);
-		return;
-	}
-	parse_alternates(buf.buf, '\n', source->path, out);
-
-	strbuf_release(&buf);
-	free(path);
-}
-
 static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,
 							const char *source,
 							int depth)
diff --git a/odb.h b/odb.h
index 692d9029ef..86e0365c24 100644
--- a/odb.h
+++ b/odb.h
@@ -500,4 +500,9 @@ int odb_write_object_stream(struct object_database *odb,
 			    struct odb_write_stream *stream, size_t len,
 			    struct object_id *oid);
 
+void parse_alternates(const char *string,
+		      int sep,
+		      const char *relative_base,
+		      struct strvec *out);
+
 #endif /* ODB_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index b8844f11b7..199c55cfa4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -2,9 +2,11 @@
 #include "abspath.h"
 #include "chdir-notify.h"
 #include "object-file.h"
+#include "odb.h"
 #include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
+#include "strbuf.h"
 
 static void odb_source_files_reparent(const char *name UNUSED,
 				      const char *old_cwd,
@@ -117,6 +119,25 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 	return odb_source_loose_write_stream(source, stream, len, oid);
 }
 
+static int odb_source_files_read_alternates(struct odb_source *source,
+					    struct strvec *out)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
+
+	path = xstrfmt("%s/info/alternates", source->path);
+	if (strbuf_read_file(&buf, path, 1024) < 0) {
+		warn_on_fopen_errors(path);
+		free(path);
+		return 0;
+	}
+	parse_alternates(buf.buf, '\n', source->path, out);
+
+	strbuf_release(&buf);
+	free(path);
+	return 0;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -137,6 +158,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.freshen_object = odb_source_files_freshen_object;
 	files->base.write_object = odb_source_files_write_object;
 	files->base.write_object_stream = odb_source_files_write_object_stream;
+	files->base.read_alternates = odb_source_files_read_alternates;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index ddce43eb20..14f5d56f68 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -55,6 +55,7 @@ struct object_id;
 struct object_info;
 struct odb_read_stream;
 struct odb_write_stream;
+struct strvec;
 
 /*
  * A callback function that can be used to iterate through objects. If given,
@@ -229,6 +230,20 @@ struct odb_source {
 	int (*write_object_stream)(struct odb_source *source,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to read the list of alternate object
+	 * database sources connected to it and write them into the `strvec`.
+	 *
+	 * The format is expected to follow the "objectStorage" extension
+	 * format with `(backend://)?payload` syntax. If the payload contains
+	 * paths, these paths must be resolved to absolute paths.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 */
+	int (*read_alternates)(struct odb_source *source,
+			       struct strvec *out);
 };
 
 /*
@@ -379,4 +394,18 @@ static inline int odb_source_write_object_stream(struct odb_source *source,
 	return source->write_object_stream(source, stream, len, oid);
 }
 
+/*
+ * Read the list of alternative object database sources from the given backend
+ * and populate the `strvec` with them. The listing is not recursive -- that
+ * is, if any of the yielded alternate sources has alternates itself, those
+ * will not be yielded as part of this function call.
+ *
+ * Return 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_read_alternates(struct odb_source *source,
+					     struct strvec *out)
+{
+	return source->read_alternates(source, out);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 15/17] odb/source: make `read_alternates()` function pluggable

[Justin Tobler]

On 26/02/23 05:18PM, Patrick Steinhardt wrote:
> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
[snip]
> diff --git a/odb/source.h b/odb/source.h
> index ddce43eb20..14f5d56f68 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -55,6 +55,7 @@ struct object_id;
>  struct object_info;
>  struct odb_read_stream;
>  struct odb_write_stream;
> +struct strvec;
>  
>  /*
>   * A callback function that can be used to iterate through objects. If given,
> @@ -229,6 +230,20 @@ struct odb_source {
>  	int (*write_object_stream)(struct odb_source *source,
>  				   struct odb_write_stream *stream, size_t len,
>  				   struct object_id *oid);
> +
> +	/*
> +	 * This callback is expected to read the list of alternate object
> +	 * database sources connected to it and write them into the `strvec`.
> +	 *
> +	 * The format is expected to follow the "objectStorage" extension
> +	 * format with `(backend://)?payload` syntax. If the payload contains
> +	 * paths, these paths must be resolved to absolute paths.

This seems sensible, but also sounds like a change that might be worth
explaining in the commit message. Does this mean we should expect an
alternates file containing list prefixed with "files://" to start
working? If so, this doesn't appear to be implemented yet.

-Justin

Re: [PATCH 15/17] odb/source: make `read_alternates()` function pluggable

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 03:49:01PM -0600, Justin Tobler wrote:
> On 26/02/23 05:18PM, Patrick Steinhardt wrote:
> > diff --git a/odb/source.h b/odb/source.h
> > index ddce43eb20..14f5d56f68 100644
> > --- a/odb/source.h
> > +++ b/odb/source.h
> > @@ -229,6 +230,20 @@ struct odb_source {
> >  	int (*write_object_stream)(struct odb_source *source,
> >  				   struct odb_write_stream *stream, size_t len,
> >  				   struct object_id *oid);
> > +
> > +	/*
> > +	 * This callback is expected to read the list of alternate object
> > +	 * database sources connected to it and write them into the `strvec`.
> > +	 *
> > +	 * The format is expected to follow the "objectStorage" extension
> > +	 * format with `(backend://)?payload` syntax. If the payload contains
> > +	 * paths, these paths must be resolved to absolute paths.
> 
> This seems sensible, but also sounds like a change that might be worth
> explaining in the commit message. Does this mean we should expect an
> alternates file containing list prefixed with "files://" to start
> working? If so, this doesn't appear to be implemented yet.

Fair, none of this is implemented yet. Let me adapt the comment.

Patrick

[PATCH 16/17] odb/source: make `write_alternate()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 52 --------------------------------------------------
 odb/source-files.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 odb/source.h       | 26 +++++++++++++++++++++++++
 3 files changed, 82 insertions(+), 52 deletions(-)

diff --git a/odb.c b/odb.c
index d9424cdfd0..84a31084d3 100644
--- a/odb.c
+++ b/odb.c
@@ -236,58 +236,6 @@ static struct odb_source *odb_add_alternate_recursively(struct object_database *
 	return alternate;
 }
 
-static int odb_source_write_alternate(struct odb_source *source,
-				      const char *alternate)
-{
-	struct lock_file lock = LOCK_INIT;
-	char *path = xstrfmt("%s/%s", source->path, "info/alternates");
-	FILE *in, *out;
-	int found = 0;
-	int ret;
-
-	hold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);
-	out = fdopen_lock_file(&lock, "w");
-	if (!out) {
-		ret = error_errno(_("unable to fdopen alternates lockfile"));
-		goto out;
-	}
-
-	in = fopen(path, "r");
-	if (in) {
-		struct strbuf line = STRBUF_INIT;
-
-		while (strbuf_getline(&line, in) != EOF) {
-			if (!strcmp(alternate, line.buf)) {
-				found = 1;
-				break;
-			}
-			fprintf_or_die(out, "%s\n", line.buf);
-		}
-
-		strbuf_release(&line);
-		fclose(in);
-	} else if (errno != ENOENT) {
-		ret = error_errno(_("unable to read alternates file"));
-		goto out;
-	}
-
-	if (found) {
-		rollback_lock_file(&lock);
-	} else {
-		fprintf_or_die(out, "%s\n", alternate);
-		if (commit_lock_file(&lock)) {
-			ret = error_errno(_("unable to move new alternates file into place"));
-			goto out;
-		}
-	}
-
-	ret = 0;
-
-out:
-	free(path);
-	return ret;
-}
-
 void odb_add_to_alternates_file(struct object_database *odb,
 				const char *dir)
 {
diff --git a/odb/source-files.c b/odb/source-files.c
index 199c55cfa4..c32cd67b26 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -1,12 +1,15 @@
 #include "git-compat-util.h"
 #include "abspath.h"
 #include "chdir-notify.h"
+#include "gettext.h"
+#include "lockfile.h"
 #include "object-file.h"
 #include "odb.h"
 #include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
 #include "strbuf.h"
+#include "write-or-die.h"
 
 static void odb_source_files_reparent(const char *name UNUSED,
 				      const char *old_cwd,
@@ -138,6 +141,58 @@ static int odb_source_files_read_alternates(struct odb_source *source,
 	return 0;
 }
 
+static int odb_source_files_write_alternate(struct odb_source *source,
+					    const char *alternate)
+{
+	struct lock_file lock = LOCK_INIT;
+	char *path = xstrfmt("%s/%s", source->path, "info/alternates");
+	FILE *in, *out;
+	int found = 0;
+	int ret;
+
+	hold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);
+	out = fdopen_lock_file(&lock, "w");
+	if (!out) {
+		ret = error_errno(_("unable to fdopen alternates lockfile"));
+		goto out;
+	}
+
+	in = fopen(path, "r");
+	if (in) {
+		struct strbuf line = STRBUF_INIT;
+
+		while (strbuf_getline(&line, in) != EOF) {
+			if (!strcmp(alternate, line.buf)) {
+				found = 1;
+				break;
+			}
+			fprintf_or_die(out, "%s\n", line.buf);
+		}
+
+		strbuf_release(&line);
+		fclose(in);
+	} else if (errno != ENOENT) {
+		ret = error_errno(_("unable to read alternates file"));
+		goto out;
+	}
+
+	if (found) {
+		rollback_lock_file(&lock);
+	} else {
+		fprintf_or_die(out, "%s\n", alternate);
+		if (commit_lock_file(&lock)) {
+			ret = error_errno(_("unable to move new alternates file into place"));
+			goto out;
+		}
+	}
+
+	ret = 0;
+
+out:
+	free(path);
+	return ret;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -159,6 +214,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.write_object = odb_source_files_write_object;
 	files->base.write_object_stream = odb_source_files_write_object_stream;
 	files->base.read_alternates = odb_source_files_read_alternates;
+	files->base.write_alternate = odb_source_files_write_alternate;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 14f5d56f68..cf301679da 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -244,6 +244,19 @@ struct odb_source {
 	 */
 	int (*read_alternates)(struct odb_source *source,
 			       struct strvec *out);
+
+	/*
+	 * This callback is expected to persist the singular alternate passed
+	 * to it into its list of alternates. Any pre-existing alternates are
+	 * expected to remain active. Subsequent calls to `read_alternates` are
+	 * thus expected to yield the pre-existing list of alternates plus the
+	 * newly added alternate appended to its end.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 */
+	int (*write_alternate)(struct odb_source *source,
+			       const char *alternate);
 };
 
 /*
@@ -408,4 +421,17 @@ static inline int odb_source_read_alternates(struct odb_source *source,
 	return source->read_alternates(source, out);
 }
 
+/*
+ * Write and persist a new alternate object database source for the given
+ * source. Any preexisting alternates are expected to stay valid, and the new
+ * alternate shall be appended to the end of the list.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_write_alternate(struct odb_source *source,
+					      const char *alternate)
+{
+	return source->write_alternate(source, alternate);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

[PATCH 17/17] odb/source: make `begin_transaction()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 11 +++++++++++
 odb/source.h       | 27 +++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/odb/source-files.c b/odb/source-files.c
index c32cd67b26..14cb9adeca 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 	return odb_source_loose_write_stream(source, stream, len, oid);
 }
 
+static int odb_source_files_begin_transaction(struct odb_source *source,
+					      struct odb_transaction **out)
+{
+	struct odb_transaction *tx = odb_transaction_files_begin(source);
+	if (!tx)
+		return -1;
+	*out = tx;
+	return 0;
+}
+
 static int odb_source_files_read_alternates(struct odb_source *source,
 					    struct strvec *out)
 {
@@ -213,6 +223,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.freshen_object = odb_source_files_freshen_object;
 	files->base.write_object = odb_source_files_write_object;
 	files->base.write_object_stream = odb_source_files_write_object_stream;
+	files->base.begin_transaction = odb_source_files_begin_transaction;
 	files->base.read_alternates = odb_source_files_read_alternates;
 	files->base.write_alternate = odb_source_files_write_alternate;
 
diff --git a/odb/source.h b/odb/source.h
index cf301679da..0e99052e08 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -54,6 +54,7 @@ enum object_info_flags {
 struct object_id;
 struct object_info;
 struct odb_read_stream;
+struct odb_transaction;
 struct odb_write_stream;
 struct strvec;
 
@@ -231,6 +232,19 @@ struct odb_source {
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
 
+	/*
+	 * This callback is expected to create a new transaction that can be
+	 * used to write objects to. The objects shall only be persisted into
+	 * the object database when the transcation's commit function is
+	 * called. Otherwise, the objects shall be discarded.
+	 *
+	 * Returns 0 on success, in which case the `*out` pointer will have
+	 * been populated with the object database transaction. Returns a
+	 * negative error code otherwise.
+	 */
+	int (*begin_transaction)(struct odb_source *source,
+				 struct odb_transaction **out);
+
 	/*
 	 * This callback is expected to read the list of alternate object
 	 * database sources connected to it and write them into the `strvec`.
@@ -434,4 +448,17 @@ static inline int odb_source_write_alternate(struct odb_source *source,
 	return source->write_alternate(source, alternate);
 }
 
+/*
+ * Create a new transaction that can be used to write objects into a temporary
+ * staging area. The objects will only be persisted when the transaction is
+ * committed.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_begin_transaction(struct odb_source *source,
+					       struct odb_transaction **out)
+{
+	return source->begin_transaction(source, out);
+}
+
 #endif

-- 
2.53.0.536.g309c995771.dirty

Re: [PATCH 17/17] odb/source: make `begin_transaction()` function pluggable

[Justin Tobler]

On 26/02/23 05:18PM, Patrick Steinhardt wrote:
> Introduce a new callback function in `struct odb_source` to make the
> function pluggable.
> 
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
>  odb/source-files.c | 11 +++++++++++
>  odb/source.h       | 27 +++++++++++++++++++++++++++
>  2 files changed, 38 insertions(+)
> 
> diff --git a/odb/source-files.c b/odb/source-files.c
> index c32cd67b26..14cb9adeca 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
>  	return odb_source_loose_write_stream(source, stream, len, oid);
>  }
>  
> +static int odb_source_files_begin_transaction(struct odb_source *source,
> +					      struct odb_transaction **out)
> +{
> +	struct odb_transaction *tx = odb_transaction_files_begin(source);

For a given ODB source, I would always expect that the resulting
transaction would always be of the same source type. This makes me think
that the underlying logic to handle transactions should also live along
side the concrete ODB source implementation. Doesn't have to be a part
of this series, but maybe in the future we should just merge
odb_transaction_files_begin() into here.

-Justin

Re: [PATCH 17/17] odb/source: make `begin_transaction()` function pluggable

[Patrick Steinhardt]

On Wed, Mar 04, 2026 at 04:01:32PM -0600, Justin Tobler wrote:
> On 26/02/23 05:18PM, Patrick Steinhardt wrote:
> > Introduce a new callback function in `struct odb_source` to make the
> > function pluggable.
> > 
> > Signed-off-by: Patrick Steinhardt <ps@pks.im>
> > ---
> >  odb/source-files.c | 11 +++++++++++
> >  odb/source.h       | 27 +++++++++++++++++++++++++++
> >  2 files changed, 38 insertions(+)
> > 
> > diff --git a/odb/source-files.c b/odb/source-files.c
> > index c32cd67b26..14cb9adeca 100644
> > --- a/odb/source-files.c
> > +++ b/odb/source-files.c
> > @@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
> >  	return odb_source_loose_write_stream(source, stream, len, oid);
> >  }
> >  
> > +static int odb_source_files_begin_transaction(struct odb_source *source,
> > +					      struct odb_transaction **out)
> > +{
> > +	struct odb_transaction *tx = odb_transaction_files_begin(source);
> 
> For a given ODB source, I would always expect that the resulting
> transaction would always be of the same source type. This makes me think
> that the underlying logic to handle transactions should also live along
> side the concrete ODB source implementation. Doesn't have to be a part
> of this series, but maybe in the future we should just merge
> odb_transaction_files_begin() into here.

I'm not quite sure. The current transaction mechanism we have has two
different modes: it either writes loose objects, or it writes all
objects into packfiles. So arguably, we should split up this transaction
so that we implement it on the respective sub-types of the "files"
source and then have the "files" source route requests to the correct
backend depending on the current use case.

But this area definitely needs more work, agreed.

Patrick

Re: [PATCH 00/17] odb: make object database sources pluggable

[Patrick Steinhardt]

On Mon, Feb 23, 2026 at 05:17:51PM +0100, Patrick Steinhardt wrote:
> Hi,
> 
> this patch series finally makes the object database source pluggable.
> This is done by moving backend-specific logics into callback functions
> that are part of `struct odb_source` and providing thin wrappers that
> call those functions.
> 
> To set expectations: this is only a start, there is still functionality
> missing that needs to be made pluggable. Most importantly:
> 
>   - Counting of objects.
> 
>   - Abbreviating object IDs and finding ambiguous objects.
> 
>   - Consistency checks.
> 
>   - Optimizing the object database.
> 
>   - Generating packfiles.
> 
> These will all happen in later patch series. That being said, with this
> patch series one already gets a lot of the basic functionality, and it's
> almost possible to do local workflows. Only "almost" though because we
> rely on abbreviating object IDs in a lot of places, but once that part
> is implemented in a subsequent patch series you can indeed work locally
> with an alternate backend.
> 
> Furthermore, what I didn't include as part of this patch series just yet
> is the introduction of the "objectStorage" extension. I mostly wanted to
> focus on the mostly-trivial parts without introducing any change in
> behaviour.

I forgot to note that this series is based on top of 7c02d39fc2 (The 6th
batch, 2026-02-20) with the following two series merged into it:

  - ps/odb-for-each-object at 3565faf28c (odb: drop unused
    `for_each_{loose,packed}_object()` functions, 2026-01-26)

  - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert
    `odb_has_object()` flags into an enum, 2026-02-12)

Thanks!

Patrick

Re: [PATCH 00/17] odb: make object database sources pluggable

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> I forgot to note that this series is based on top of 7c02d39fc2 (The 6th
> batch, 2026-02-20) with the following two series merged into it:
>
>   - ps/odb-for-each-object at 3565faf28c (odb: drop unused
>     `for_each_{loose,packed}_object()` functions, 2026-01-26)
>
>   - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert
>     `odb_has_object()` flags into an enum, 2026-02-12)

With the above base, [09/17] fails to apply, as the function
signature of odb_source_loose_read_object_info() no longer has
"unsigned flags" after "int flags" turns into "enum
object_info_flags flags" in f6516a5241 (odb: convert object info
flags into an enum, 2026-02-12).

+++ b/object-file.c
@@ -543,9 +543,19 @@ static int read_object_info_from_path(struct odb_source *source,
 int odb_source_loose_read_object_info(struct odb_source *source,
 				      const struct object_id *oid,
 				      struct object_info *oi,
-				      unsigned flags)
+				      enum object_info_flags flags)

Tweaking the patch (e.g., "unsigned" -> "enum object_info_flags") to
make it apply was trivial, so there is no need to resend.  Hopefully
there is no semantic conflicts due to confused bases (the result
compiled and linked fine).

Thanks.

Re: [PATCH 00/17] odb: make object database sources pluggable

[Patrick Steinhardt]

On Mon, Feb 23, 2026 at 01:59:51PM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > I forgot to note that this series is based on top of 7c02d39fc2 (The 6th
> > batch, 2026-02-20) with the following two series merged into it:
> >
> >   - ps/odb-for-each-object at 3565faf28c (odb: drop unused
> >     `for_each_{loose,packed}_object()` functions, 2026-01-26)
> >
> >   - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert
> >     `odb_has_object()` flags into an enum, 2026-02-12)
> 
> With the above base, [09/17] fails to apply, as the function
> signature of odb_source_loose_read_object_info() no longer has
> "unsigned flags" after "int flags" turns into "enum
> object_info_flags flags" in f6516a5241 (odb: convert object info
> flags into an enum, 2026-02-12).

Indeed. It seems like I mis-resolved the conflict that happens when
those two patch series are merged together. I properly resolved it in
the header, but not in the implementation.

The fun part is that this compiles cleanly with Clang 20. I would have
expected a warning here that the function signatures are different. I
tried to play around with -Weverything, but couldn't get it to produce
the expected warning. Oh, well...

> +++ b/object-file.c
> @@ -543,9 +543,19 @@ static int read_object_info_from_path(struct odb_source *source,
>  int odb_source_loose_read_object_info(struct odb_source *source,
>  				      const struct object_id *oid,
>  				      struct object_info *oi,
> -				      unsigned flags)
> +				      enum object_info_flags flags)
> 
> Tweaking the patch (e.g., "unsigned" -> "enum object_info_flags") to
> make it apply was trivial, so there is no need to resend.  Hopefully
> there is no semantic conflicts due to confused bases (the result
> compiled and linked fine).

Yeah. I'll rebuild my patch series on top of the base that you have
constructed. Thanks!

Patrick

Re: [PATCH 00/17] odb: make object database sources pluggable

[Karthik Nayak]

[-- Attachment #1: Type: text/plain, Size: 1849 bytes --]

Patrick Steinhardt <ps@pks.im> writes:

> On Mon, Feb 23, 2026 at 05:17:51PM +0100, Patrick Steinhardt wrote:
>> Hi,
>>
>> this patch series finally makes the object database source pluggable.
>> This is done by moving backend-specific logics into callback functions
>> that are part of `struct odb_source` and providing thin wrappers that
>> call those functions.
>>
>> To set expectations: this is only a start, there is still functionality
>> missing that needs to be made pluggable. Most importantly:
>>
>>   - Counting of objects.
>>
>>   - Abbreviating object IDs and finding ambiguous objects.
>>
>>   - Consistency checks.
>>
>>   - Optimizing the object database.
>>
>>   - Generating packfiles.
>>
>> These will all happen in later patch series. That being said, with this
>> patch series one already gets a lot of the basic functionality, and it's
>> almost possible to do local workflows. Only "almost" though because we
>> rely on abbreviating object IDs in a lot of places, but once that part
>> is implemented in a subsequent patch series you can indeed work locally
>> with an alternate backend.
>>
>> Furthermore, what I didn't include as part of this patch series just yet
>> is the introduction of the "objectStorage" extension. I mostly wanted to
>> focus on the mostly-trivial parts without introducing any change in
>> behaviour.
>
> I forgot to note that this series is based on top of 7c02d39fc2 (The 6th
> batch, 2026-02-20) with the following two series merged into it:
>
>   - ps/odb-for-each-object at 3565faf28c (odb: drop unused
>     `for_each_{loose,packed}_object()` functions, 2026-01-26)
>
>   - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert
>     `odb_has_object()` flags into an enum, 2026-02-12)
>
> Thanks!
>
> Patrick

Apart from some comments/nits, I think the series already looks great.

Karthik

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 690 bytes --]

[PATCH v2 00/17] odb: make object database sources pluggable

[Patrick Steinhardt]

Hi,

this patch series finally makes the object database source pluggable.
This is done by moving backend-specific logics into callback functions
that are part of `struct odb_source` and providing thin wrappers that
call those functions.

To set expectations: this is only a start, there is still functionality
missing that needs to be made pluggable. Most importantly:

  - Counting of objects.

  - Abbreviating object IDs and finding ambiguous objects.

  - Consistency checks.

  - Optimizing the object database.

  - Generating packfiles.

These will all happen in later patch series. That being said, with this
patch series one already gets a lot of the basic functionality, and it's
almost possible to do local workflows. Only "almost" though because we
rely on abbreviating object IDs in a lot of places, but once that part
is implemented in a subsequent patch series you can indeed work locally
with an alternate backend.

Furthermore, what I didn't include as part of this patch series just yet
is the introduction of the "objectStorage" extension. I mostly wanted to
focus on the mostly-trivial parts without introducing any change in
behaviour.

This series is based on top of 7c02d39fc2 (The 6th batch, 2026-02-20)
with the following two series merged into it:

  - ps/odb-for-each-object at 3565faf28c (odb: drop unused
    `for_each_{loose,packed}_object()` functions, 2026-01-26)

  - ps/object-info-bits-cleanup at 732ec9b17b (odb: convert
    `odb_has_object()` flags into an enum, 2026-02-12)

Changes in v2:
  - Fix mismerge in the base of this patch series.
  - Adjust several comments and improve commit messages a bit.
  - Link to v1: https://lore.kernel.org/r/20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im

Thanks!

Patrick

---
Patrick Steinhardt (17):
      odb: split `struct odb_source` into separate header
      odb: introduce "files" source
      odb: embed base source in the "files" backend
      odb: move reparenting logic into respective subsystems
      odb/source: introduce source type for robustness
      odb/source: make `free()` function pluggable
      odb/source: make `reprepare()` function pluggable
      odb/source: make `close()` function pluggable
      odb/source: make `read_object_info()` function pluggable
      odb/source: make `read_object_stream()` function pluggable
      odb/source: make `for_each_object()` function pluggable
      odb/source: make `freshen_object()` function pluggable
      odb/source: make `write_object()` function pluggable
      odb/source: make `write_object_stream()` function pluggable
      odb/source: make `read_alternates()` function pluggable
      odb/source: make `write_alternate()` function pluggable
      odb/source: make `begin_transaction()` function pluggable

 Makefile               |   2 +
 builtin/cat-file.c     |   3 +-
 builtin/fast-import.c  |  12 +-
 builtin/grep.c         |   6 +-
 builtin/index-pack.c   |   8 +-
 builtin/pack-objects.c |  13 +-
 commit-graph.c         |   6 +-
 http.c                 |   3 +-
 loose.c                |  23 ++-
 meson.build            |   2 +
 midx.c                 |  26 +--
 object-file.c          |  38 ++--
 odb.c                  | 191 +++-----------------
 odb.h                  |  86 +--------
 odb/source-files.c     | 239 +++++++++++++++++++++++++
 odb/source-files.h     |  35 ++++
 odb/source.c           |  38 ++++
 odb/source.h           | 468 +++++++++++++++++++++++++++++++++++++++++++++++++
 odb/streaming.c        |   8 +-
 packfile.c             |  36 ++--
 packfile.h             |   7 +-
 tmp-objdir.c           |  42 ++---
 tmp-objdir.h           |  15 --
 23 files changed, 953 insertions(+), 354 deletions(-)

Range-diff versus v1:

 1:  28258657d5 =  1:  6dd89d5721 odb: split `struct odb_source` into separate header
 2:  38fa6650e7 =  2:  aaf6175ad7 odb: introduce "files" source
 3:  bbdfe087d3 !  3:  1188bc969a odb: embed base source in the "files" backend
    @@ odb/source-files.h: struct packfile_store;
      void odb_source_files_free(struct odb_source_files *files);
      
     +/*
    -+ * Cast the given object database source to the files backend. This will cause
    -+ * a BUG in case the source doesn't use this backend.
    ++ * Cast the given object database source to the files backend.
     + */
     +static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
     +{
 4:  1f545a0b28 !  4:  a5deca0da9 odb: move reparenting logic into respective subsystems
    @@ Commit message
         odb: move reparenting logic into respective subsystems
     
         The primary object database source may be initialized with a relative
    -    path. When reparenting the process to a different working directory we
    -    thus have to update this path and have it point to the same path, but
    +    path. When the process changes its current working directory we thus
    +    have to update this path and have it point to the same path, but
         relative to the new working directory.
     
         This logic is handled in the object database layer. It consists of three
 5:  f3f0f3daeb !  5:  defb03a1b9 odb/source: introduce source type for robustness
    @@ odb/source-files.c: struct odb_source_files *odb_source_files_new(struct object_
      
     
      ## odb/source-files.h ##
    -@@ odb/source-files.h: void odb_source_files_free(struct odb_source_files *files);
    +@@ odb/source-files.h: struct odb_source_files *odb_source_files_new(struct object_database *odb,
    + void odb_source_files_free(struct odb_source_files *files);
    + 
    + /*
    +- * Cast the given object database source to the files backend.
    ++ * Cast the given object database source to the files backend. This will cause
    ++ * a BUG in case the source doesn't use this backend.
       */
      static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
      {
    @@ odb/source.h
      
     +enum odb_source_type {
     +	/*
    -+	 * The "unknown" type, which should never be in use. This is type
    -+	 * mostly exists to catch cases where the type field remains zeroed
    -+	 * out.
    ++	 * The "unknown" type, which should never be in use. This type mostly
    ++	 * exists to catch cases where the type field remains zeroed out.
     +	 */
     +	ODB_SOURCE_UNKNOWN,
     +
 6:  c86a03bf7c =  6:  df5c9e7584 odb/source: make `free()` function pluggable
 7:  b1645d0de0 =  7:  6787995a2c odb/source: make `reprepare()` function pluggable
 8:  e873c4f32c =  8:  9942876dbe odb/source: make `close()` function pluggable
 9:  0ccf994441 !  9:  9902f4561b odb/source: make `read_object_info()` function pluggable
    @@ Commit message
         Signed-off-by: Patrick Steinhardt <ps@pks.im>
     
      ## object-file.c ##
    -@@ object-file.c: static int read_object_info_from_path(struct odb_source *source,
    - int odb_source_loose_read_object_info(struct odb_source *source,
    - 				      const struct object_id *oid,
    - 				      struct object_info *oi,
    --				      unsigned flags)
    -+				      enum object_info_flags flags)
    +@@ object-file.c: int odb_source_loose_read_object_info(struct odb_source *source,
    + 				      enum object_info_flags flags)
      {
      	static struct strbuf buf = STRBUF_INIT;
     +
10:  f98a8adfed = 10:  99299ed03e odb/source: make `read_object_stream()` function pluggable
11:  b8a9b9fe16 ! 11:  274a6020ab odb/source: make `for_each_object()` function pluggable
    @@ odb/source.h: struct odb_source {
     +	 * valid to yield the same object multiple time. A non-zero exit code
     +	 * from the object callback shall abort iteration.
     +	 *
    -+	 * The optional `oi` structure shall be populated similar to how an individual
    -+	 * call to `odb_source_read_object_info()` would have behaved. If the caller
    -+	 * passes a `NULL` pointer then the object itself shall not be read.
    ++	 * The optional `request` structure should serve as a template for
    ++	 * looking up object info for every individual iterated object. It
    ++	 * should not be modified directly and should instead be copied into a
    ++	 * separate `struct object_info` that gets passed to the callback. If
    ++	 * the caller passes a `NULL` pointer then the object itself shall not
    ++	 * be read.
     +	 *
     +	 * The callback is expected to return a negative error code in case the
     +	 * iteration has failed to read all objects, 0 otherwise. When the
    @@ odb/source.h: static inline int odb_source_read_object_stream(struct odb_read_st
     + * callback function aborts iteration. There is no guarantee that objects
     + * are only iterated over once.
     + *
    -+ * The optional `oi` structure shall be populated similar to how an individual
    -+ * call to `odb_source_read_object_info()` would have behaved. If the caller
    -+ * passes a `NULL` pointer then the object itself shall not be read.
    ++ * The optional `request` structure serves as a template for retrieving the
    ++ * object info for each indvidual iterated object and will be populated as if
    ++ * `odb_source_read_object_info()` was called on the object. It will not be
    ++ * modified, the callback will instead be invoked with a separate `struct
    ++ * object_info` for every object. Object info will not be read when passing a
    ++ * `NULL` pointer.
     + *
     + * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may
     + * apply to a specific backend, so whether or not they are honored is defined
12:  406826905d = 12:  abc1bc6f81 odb/source: make `freshen_object()` function pluggable
13:  59a3678799 ! 13:  9a995ff455 odb/source: make `write_object()` function pluggable
    @@ odb/source.h
     +
      enum odb_source_type {
      	/*
    - 	 * The "unknown" type, which should never be in use. This is type
    + 	 * The "unknown" type, which should never be in use. This type mostly
     @@ odb/source.h: struct odb_source {
      	 */
      	int (*freshen_object)(struct odb_source *source,
14:  e5c47518ef = 14:  8c938de272 odb/source: make `write_object_stream()` function pluggable
15:  ca0e6dfb1a ! 15:  16a826e24c odb/source: make `read_alternates()` function pluggable
    @@ odb/source.h: struct odb_source {
     +	 * This callback is expected to read the list of alternate object
     +	 * database sources connected to it and write them into the `strvec`.
     +	 *
    -+	 * The format is expected to follow the "objectStorage" extension
    -+	 * format with `(backend://)?payload` syntax. If the payload contains
    -+	 * paths, these paths must be resolved to absolute paths.
    ++	 * The result is expected to be paths to the alternates. All paths must
    ++	 * be resolved to absolute paths.
     +	 *
     +	 * The callback is expected to return 0 on success, a negative error
     +	 * code otherwise.
16:  7e36a7ec8f = 16:  2f6bf3aedc odb/source: make `write_alternate()` function pluggable
17:  dc918d3fc5 = 17:  118b442202 odb/source: make `begin_transaction()` function pluggable

---
base-commit: b1af291b4adf1c433ad2b79f0390f7d6b516a964
change-id: 20260120-b4-pks-odb-source-pluggable-5c724250b3c8

[PATCH v2 01/17] odb: split `struct odb_source` into separate header

[Patrick Steinhardt]

Subsequent commits will expand the `struct odb_source` to become a
generic interface for accessing an object database source. As part of
these refactorings we'll add a set of function pointers that will
significantly expand the structure overall.

Prepare for this by splitting out the `struct odb_source` into a
separate header. This keeps the high-level object database interface
detached from the low-level object database sources.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Makefile     |  1 +
 meson.build  |  1 +
 odb.c        | 25 -------------------------
 odb.h        | 45 +--------------------------------------------
 odb/source.c | 28 ++++++++++++++++++++++++++++
 odb/source.h | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 91 insertions(+), 69 deletions(-)

diff --git a/Makefile b/Makefile
index 47ed9fa7fd..116358e484 100644
--- a/Makefile
+++ b/Makefile
@@ -1214,6 +1214,7 @@ LIB_OBJS += object-file.o
 LIB_OBJS += object-name.o
 LIB_OBJS += object.o
 LIB_OBJS += odb.o
+LIB_OBJS += odb/source.o
 LIB_OBJS += odb/streaming.o
 LIB_OBJS += oid-array.o
 LIB_OBJS += oidmap.o
diff --git a/meson.build b/meson.build
index 3a1d12caa4..1018af17c3 100644
--- a/meson.build
+++ b/meson.build
@@ -397,6 +397,7 @@ libgit_sources = [
   'object-name.c',
   'object.c',
   'odb.c',
+  'odb/source.c',
   'odb/streaming.c',
   'oid-array.c',
   'oidmap.c',
diff --git a/odb.c b/odb.c
index 776de5356c..d318482d47 100644
--- a/odb.c
+++ b/odb.c
@@ -217,23 +217,6 @@ static void odb_source_read_alternates(struct odb_source *source,
 	free(path);
 }
 
-
-static struct odb_source *odb_source_new(struct object_database *odb,
-					 const char *path,
-					 bool local)
-{
-	struct odb_source *source;
-
-	CALLOC_ARRAY(source, 1);
-	source->odb = odb;
-	source->local = local;
-	source->path = xstrdup(path);
-	source->loose = odb_source_loose_new(source);
-	source->packfiles = packfile_store_new(source);
-
-	return source;
-}
-
 static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,
 							const char *source,
 							int depth)
@@ -373,14 +356,6 @@ struct odb_source *odb_set_temporary_primary_source(struct object_database *odb,
 	return source->next;
 }
 
-static void odb_source_free(struct odb_source *source)
-{
-	free(source->path);
-	odb_source_loose_free(source->loose);
-	packfile_store_free(source->packfiles);
-	free(source);
-}
-
 void odb_restore_primary_source(struct object_database *odb,
 				struct odb_source *restore_source,
 				const char *old_path)
diff --git a/odb.h b/odb.h
index 68b8ec2289..e13b5b7c44 100644
--- a/odb.h
+++ b/odb.h
@@ -3,6 +3,7 @@
 
 #include "hashmap.h"
 #include "object.h"
+#include "odb/source.h"
 #include "oidset.h"
 #include "oidmap.h"
 #include "string-list.h"
@@ -30,50 +31,6 @@ extern int fetch_if_missing;
  */
 char *compute_alternate_path(const char *path, struct strbuf *err);
 
-/*
- * The source is the part of the object database that stores the actual
- * objects. It thus encapsulates the logic to read and write the specific
- * on-disk format. An object database can have multiple sources:
- *
- *   - The primary source, which is typically located in "$GIT_DIR/objects".
- *     This is where new objects are usually written to.
- *
- *   - Alternate sources, which are configured via "objects/info/alternates" or
- *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These
- *     alternate sources are only used to read objects.
- */
-struct odb_source {
-	struct odb_source *next;
-
-	/* Object database that owns this object source. */
-	struct object_database *odb;
-
-	/* Private state for loose objects. */
-	struct odb_source_loose *loose;
-
-	/* Should only be accessed directly by packfile.c and midx.c. */
-	struct packfile_store *packfiles;
-
-	/*
-	 * Figure out whether this is the local source of the owning
-	 * repository, which would typically be its ".git/objects" directory.
-	 * This local object directory is usually where objects would be
-	 * written to.
-	 */
-	bool local;
-
-	/*
-	 * This object store is ephemeral, so there is no need to fsync.
-	 */
-	int will_destroy;
-
-	/*
-	 * Path to the source. If this is a relative path, it is relative to
-	 * the current working directory.
-	 */
-	char *path;
-};
-
 struct packed_git;
 struct packfile_store;
 struct cached_object_entry;
diff --git a/odb/source.c b/odb/source.c
new file mode 100644
index 0000000000..7fc89806f9
--- /dev/null
+++ b/odb/source.c
@@ -0,0 +1,28 @@
+#include "git-compat-util.h"
+#include "object-file.h"
+#include "odb/source.h"
+#include "packfile.h"
+
+struct odb_source *odb_source_new(struct object_database *odb,
+				  const char *path,
+				  bool local)
+{
+	struct odb_source *source;
+
+	CALLOC_ARRAY(source, 1);
+	source->odb = odb;
+	source->local = local;
+	source->path = xstrdup(path);
+	source->loose = odb_source_loose_new(source);
+	source->packfiles = packfile_store_new(source);
+
+	return source;
+}
+
+void odb_source_free(struct odb_source *source)
+{
+	free(source->path);
+	odb_source_loose_free(source->loose);
+	packfile_store_free(source->packfiles);
+	free(source);
+}
diff --git a/odb/source.h b/odb/source.h
new file mode 100644
index 0000000000..391d6d1e38
--- /dev/null
+++ b/odb/source.h
@@ -0,0 +1,60 @@
+#ifndef ODB_SOURCE_H
+#define ODB_SOURCE_H
+
+/*
+ * The source is the part of the object database that stores the actual
+ * objects. It thus encapsulates the logic to read and write the specific
+ * on-disk format. An object database can have multiple sources:
+ *
+ *   - The primary source, which is typically located in "$GIT_DIR/objects".
+ *     This is where new objects are usually written to.
+ *
+ *   - Alternate sources, which are configured via "objects/info/alternates" or
+ *     via the GIT_ALTERNATE_OBJECT_DIRECTORIES environment variable. These
+ *     alternate sources are only used to read objects.
+ */
+struct odb_source {
+	struct odb_source *next;
+
+	/* Object database that owns this object source. */
+	struct object_database *odb;
+
+	/* Private state for loose objects. */
+	struct odb_source_loose *loose;
+
+	/* Should only be accessed directly by packfile.c and midx.c. */
+	struct packfile_store *packfiles;
+
+	/*
+	 * Figure out whether this is the local source of the owning
+	 * repository, which would typically be its ".git/objects" directory.
+	 * This local object directory is usually where objects would be
+	 * written to.
+	 */
+	bool local;
+
+	/*
+	 * This object store is ephemeral, so there is no need to fsync.
+	 */
+	int will_destroy;
+
+	/*
+	 * Path to the source. If this is a relative path, it is relative to
+	 * the current working directory.
+	 */
+	char *path;
+};
+
+/*
+ * Allocate and initialize a new source for the given object database located
+ * at `path`. `local` indicates whether or not the source is the local and thus
+ * primary object source of the object database.
+ */
+struct odb_source *odb_source_new(struct object_database *odb,
+				  const char *path,
+				  bool local);
+
+/* Free the object database source, releasing all associated resources. */
+void odb_source_free(struct odb_source *source);
+
+#endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 02/17] odb: introduce "files" source

[Patrick Steinhardt]

Introduce a new "files" object database source. This source encapsulates
access to both loose object files and the packfile store, similar to how
the "files" backend for refs encapsulates access to loose refs and the
packed-refs file.

Note that for now the "files" source is still a direct member of a
`struct odb_source`. This architecture will be reversed in the next
commit so that the files source contains a `struct odb_source`.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 Makefile               |  1 +
 builtin/cat-file.c     |  2 +-
 builtin/fast-import.c  |  6 +++---
 builtin/grep.c         |  2 +-
 builtin/index-pack.c   |  2 +-
 builtin/pack-objects.c |  8 ++++----
 commit-graph.c         |  2 +-
 http.c                 |  2 +-
 loose.c                | 18 +++++++++---------
 meson.build            |  1 +
 midx.c                 | 18 +++++++++---------
 object-file.c          | 24 ++++++++++++------------
 odb.c                  | 12 ++++++------
 odb/source-files.c     | 23 +++++++++++++++++++++++
 odb/source-files.h     | 24 ++++++++++++++++++++++++
 odb/source.c           |  6 ++----
 odb/source.h           |  9 ++++-----
 odb/streaming.c        |  2 +-
 packfile.c             | 16 ++++++++--------
 packfile.h             |  4 ++--
 20 files changed, 114 insertions(+), 68 deletions(-)

diff --git a/Makefile b/Makefile
index 116358e484..c05285399c 100644
--- a/Makefile
+++ b/Makefile
@@ -1215,6 +1215,7 @@ LIB_OBJS += object-name.o
 LIB_OBJS += object.o
 LIB_OBJS += odb.o
 LIB_OBJS += odb/source.o
+LIB_OBJS += odb/source-files.o
 LIB_OBJS += odb/streaming.o
 LIB_OBJS += oid-array.o
 LIB_OBJS += oidmap.o
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 53ffe80c79..01a53f3f29 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -882,7 +882,7 @@ static void batch_each_object(struct batch_options *opt,
 		struct object_info oi = { 0 };
 
 		for (source = the_repository->objects->sources; source; source = source->next) {
-			int ret = packfile_store_for_each_object(source->packfiles, &oi,
+			int ret = packfile_store_for_each_object(source->files->packed, &oi,
 								 batch_one_object_oi, &payload, flags);
 			if (ret)
 				break;
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index b8a7757cfd..627dcbf4f3 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -900,7 +900,7 @@ static void end_packfile(void)
 		idx_name = keep_pack(create_index());
 
 		/* Register the packfile with core git's machinery. */
-		new_p = packfile_store_load_pack(pack_data->repo->objects->sources->packfiles,
+		new_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,
 						 idx_name, 1);
 		if (!new_p)
 			die(_("core Git rejected index %s"), idx_name);
@@ -982,7 +982,7 @@ static int store_object(
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))
+		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
 			continue;
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
@@ -1187,7 +1187,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->packfiles), &oid))
+		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
 			continue;
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
diff --git a/builtin/grep.c b/builtin/grep.c
index 5b8b87b1ac..c8d0e51415 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -1219,7 +1219,7 @@ int cmd_grep(int argc,
 
 			odb_prepare_alternates(the_repository->objects);
 			for (source = the_repository->objects->sources; source; source = source->next)
-				packfile_store_prepare(source->packfiles);
+				packfile_store_prepare(source->files->packed);
 		}
 
 		start_threads(&opt);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index b67fb0256c..f0cce534b2 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1638,7 +1638,7 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
 			    hash, "idx", 1);
 
 	if (do_fsck_object && startup_info->have_repository)
-		packfile_store_load_pack(the_repository->objects->sources->packfiles,
+		packfile_store_load_pack(the_repository->objects->sources->files->packed,
 					 final_index_name, 0);
 
 	if (!from_stdin) {
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 242d1c68f0..0c3c01cdc9 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1531,7 +1531,7 @@ static int want_cruft_object_mtime(struct repository *r,
 	struct odb_source *source;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		struct packed_git **cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);
+		struct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
@@ -1753,11 +1753,11 @@ static int want_object_in_pack_mtime(const struct object_id *oid,
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		for (e = source->packfiles->packs.head; e; e = e->next) {
+		for (e = source->files->packed->packs.head; e; e = e->next) {
 			struct packed_git *p = e->pack;
 			want = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);
 			if (!exclude && want > 0)
-				packfile_list_prepend(&source->packfiles->packs, p);
+				packfile_list_prepend(&source->files->packed->packs, p);
 			if (want != -1)
 				return want;
 		}
@@ -4340,7 +4340,7 @@ static void add_objects_in_unpacked_packs(void)
 		if (!source->local)
 			continue;
 
-		if (packfile_store_for_each_object(source->packfiles, &oi,
+		if (packfile_store_for_each_object(source->files->packed, &oi,
 						   add_object_in_unpacked_pack, NULL,
 						   ODB_FOR_EACH_OBJECT_PACK_ORDER |
 						   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |
diff --git a/commit-graph.c b/commit-graph.c
index d250a729b1..967eb77047 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1981,7 +1981,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)
 
 	odb_prepare_alternates(ctx->r->objects);
 	for (source = ctx->r->objects->sources; source; source = source->next)
-		packfile_store_for_each_object(source->packfiles, &oi, add_packed_commits_oi,
+		packfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,
 					       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);
 
 	if (ctx->progress_done < ctx->approx_nr_objects)
diff --git a/http.c b/http.c
index 7815f144de..b44f493919 100644
--- a/http.c
+++ b/http.c
@@ -2544,7 +2544,7 @@ void http_install_packfile(struct packed_git *p,
 			   struct packfile_list *list_to_remove_from)
 {
 	packfile_list_remove(list_to_remove_from, p);
-	packfile_store_add_pack(the_repository->objects->sources->packfiles, p);
+	packfile_store_add_pack(the_repository->objects->sources->files->packed, p);
 }
 
 struct http_pack_request *new_http_pack_request(
diff --git a/loose.c b/loose.c
index 56cf64b648..c921d46b94 100644
--- a/loose.c
+++ b/loose.c
@@ -49,13 +49,13 @@ static int insert_loose_map(struct odb_source *source,
 			    const struct object_id *oid,
 			    const struct object_id *compat_oid)
 {
-	struct loose_object_map *map = source->loose->map;
+	struct loose_object_map *map = source->files->loose->map;
 	int inserted = 0;
 
 	inserted |= insert_oid_pair(map->to_compat, oid, compat_oid);
 	inserted |= insert_oid_pair(map->to_storage, compat_oid, oid);
 	if (inserted)
-		oidtree_insert(source->loose->cache, compat_oid);
+		oidtree_insert(source->files->loose->cache, compat_oid);
 
 	return inserted;
 }
@@ -65,11 +65,11 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source
 	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
 	FILE *fp;
 
-	if (!source->loose->map)
-		loose_object_map_init(&source->loose->map);
-	if (!source->loose->cache) {
-		ALLOC_ARRAY(source->loose->cache, 1);
-		oidtree_init(source->loose->cache);
+	if (!source->files->loose->map)
+		loose_object_map_init(&source->files->loose->map);
+	if (!source->files->loose->cache) {
+		ALLOC_ARRAY(source->files->loose->cache, 1);
+		oidtree_init(source->files->loose->cache);
 	}
 
 	insert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
@@ -125,7 +125,7 @@ int repo_read_loose_object_map(struct repository *repo)
 
 int repo_write_loose_object_map(struct repository *repo)
 {
-	kh_oid_map_t *map = repo->objects->sources->loose->map->to_compat;
+	kh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;
 	struct lock_file lock;
 	int fd;
 	khiter_t iter;
@@ -231,7 +231,7 @@ int repo_loose_object_map_oid(struct repository *repo,
 	khiter_t pos;
 
 	for (source = repo->objects->sources; source; source = source->next) {
-		struct loose_object_map *loose_map = source->loose->map;
+		struct loose_object_map *loose_map = source->files->loose->map;
 		if (!loose_map)
 			continue;
 		map = (to == repo->compat_hash_algo) ?
diff --git a/meson.build b/meson.build
index 1018af17c3..8e1125a585 100644
--- a/meson.build
+++ b/meson.build
@@ -398,6 +398,7 @@ libgit_sources = [
   'object.c',
   'odb.c',
   'odb/source.c',
+  'odb/source-files.c',
   'odb/streaming.c',
   'oid-array.c',
   'oidmap.c',
diff --git a/midx.c b/midx.c
index a75ea99a0d..698d10a1c6 100644
--- a/midx.c
+++ b/midx.c
@@ -95,8 +95,8 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,
 
 struct multi_pack_index *get_multi_pack_index(struct odb_source *source)
 {
-	packfile_store_prepare(source->packfiles);
-	return source->packfiles->midx;
+	packfile_store_prepare(source->files->packed);
+	return source->files->packed->midx;
 }
 
 static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,
@@ -459,7 +459,7 @@ int prepare_midx_pack(struct multi_pack_index *m,
 
 	strbuf_addf(&pack_name, "%s/pack/%s", m->source->path,
 		    m->pack_names[pack_int_id]);
-	p = packfile_store_load_pack(m->source->packfiles,
+	p = packfile_store_load_pack(m->source->files->packed,
 				     pack_name.buf, m->source->local);
 	strbuf_release(&pack_name);
 
@@ -709,12 +709,12 @@ int prepare_multi_pack_index_one(struct odb_source *source)
 	if (!r->settings.core_multi_pack_index)
 		return 0;
 
-	if (source->packfiles->midx)
+	if (source->files->packed->midx)
 		return 1;
 
-	source->packfiles->midx = load_multi_pack_index(source);
+	source->files->packed->midx = load_multi_pack_index(source);
 
-	return !!source->packfiles->midx;
+	return !!source->files->packed->midx;
 }
 
 int midx_checksum_valid(struct multi_pack_index *m)
@@ -803,9 +803,9 @@ void clear_midx_file(struct repository *r)
 		struct odb_source *source;
 
 		for (source = r->objects->sources; source; source = source->next) {
-			if (source->packfiles->midx)
-				close_midx(source->packfiles->midx);
-			source->packfiles->midx = NULL;
+			if (source->files->packed->midx)
+				close_midx(source->files->packed->midx);
+			source->files->packed->midx = NULL;
 		}
 	}
 
diff --git a/object-file.c b/object-file.c
index 098b0541ab..db66ae5ebe 100644
--- a/object-file.c
+++ b/object-file.c
@@ -220,7 +220,7 @@ static void *odb_source_loose_map_object(struct odb_source *source,
 					 unsigned long *size)
 {
 	const char *p;
-	int fd = open_loose_object(source->loose, oid, &p);
+	int fd = open_loose_object(source->files->loose, oid, &p);
 
 	if (fd < 0)
 		return NULL;
@@ -423,7 +423,7 @@ static int read_object_info_from_path(struct odb_source *source,
 		struct stat st;
 
 		if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
-			ret = quick_has_loose(source->loose, oid) ? 0 : -1;
+			ret = quick_has_loose(source->files->loose, oid) ? 0 : -1;
 			goto out;
 		}
 
@@ -1868,31 +1868,31 @@ struct oidtree *odb_source_loose_cache(struct odb_source *source,
 {
 	int subdir_nr = oid->hash[0];
 	struct strbuf buf = STRBUF_INIT;
-	size_t word_bits = bitsizeof(source->loose->subdir_seen[0]);
+	size_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);
 	size_t word_index = subdir_nr / word_bits;
 	size_t mask = (size_t)1u << (subdir_nr % word_bits);
 	uint32_t *bitmap;
 
 	if (subdir_nr < 0 ||
-	    (size_t) subdir_nr >= bitsizeof(source->loose->subdir_seen))
+	    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))
 		BUG("subdir_nr out of range");
 
-	bitmap = &source->loose->subdir_seen[word_index];
+	bitmap = &source->files->loose->subdir_seen[word_index];
 	if (*bitmap & mask)
-		return source->loose->cache;
-	if (!source->loose->cache) {
-		ALLOC_ARRAY(source->loose->cache, 1);
-		oidtree_init(source->loose->cache);
+		return source->files->loose->cache;
+	if (!source->files->loose->cache) {
+		ALLOC_ARRAY(source->files->loose->cache, 1);
+		oidtree_init(source->files->loose->cache);
 	}
 	strbuf_addstr(&buf, source->path);
 	for_each_file_in_obj_subdir(subdir_nr, &buf,
 				    source->odb->repo->hash_algo,
 				    append_loose_object,
 				    NULL, NULL,
-				    source->loose->cache);
+				    source->files->loose->cache);
 	*bitmap |= mask;
 	strbuf_release(&buf);
-	return source->loose->cache;
+	return source->files->loose->cache;
 }
 
 static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
@@ -1905,7 +1905,7 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
 
 void odb_source_loose_reprepare(struct odb_source *source)
 {
-	odb_source_loose_clear_cache(source->loose);
+	odb_source_loose_clear_cache(source->files->loose);
 }
 
 static int check_stream_oid(git_zstream *stream,
diff --git a/odb.c b/odb.c
index d318482d47..c9ebc7e741 100644
--- a/odb.c
+++ b/odb.c
@@ -691,7 +691,7 @@ static int do_oid_object_info_extended(struct object_database *odb,
 
 		/* Most likely it's a loose object. */
 		for (source = odb->sources; source; source = source->next) {
-			if (!packfile_store_read_object_info(source->packfiles, real, oi, flags) ||
+			if (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||
 			    !odb_source_loose_read_object_info(source, real, oi, flags))
 				return 0;
 		}
@@ -700,7 +700,7 @@ static int do_oid_object_info_extended(struct object_database *odb,
 		if (!(flags & OBJECT_INFO_QUICK)) {
 			odb_reprepare(odb->repo->objects);
 			for (source = odb->sources; source; source = source->next)
-				if (!packfile_store_read_object_info(source->packfiles, real, oi, flags))
+				if (!packfile_store_read_object_info(source->files->packed, real, oi, flags))
 					return 0;
 		}
 
@@ -962,7 +962,7 @@ int odb_freshen_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (packfile_store_freshen_object(source->packfiles, oid))
+		if (packfile_store_freshen_object(source->files->packed, oid))
 			return 1;
 
 		if (odb_source_loose_freshen_object(source, oid))
@@ -992,7 +992,7 @@ int odb_for_each_object(struct object_database *odb,
 				return ret;
 		}
 
-		ret = packfile_store_for_each_object(source->packfiles, request,
+		ret = packfile_store_for_each_object(source->files->packed, request,
 						     cb, cb_data, flags);
 		if (ret)
 			return ret;
@@ -1091,7 +1091,7 @@ void odb_close(struct object_database *o)
 {
 	struct odb_source *source;
 	for (source = o->sources; source; source = source->next)
-		packfile_store_close(source->packfiles);
+		packfile_store_close(source->files->packed);
 	close_commit_graph(o);
 }
 
@@ -1149,7 +1149,7 @@ void odb_reprepare(struct object_database *o)
 
 	for (source = o->sources; source; source = source->next) {
 		odb_source_loose_reprepare(source);
-		packfile_store_reprepare(source->packfiles);
+		packfile_store_reprepare(source->files->packed);
 	}
 
 	o->approximate_object_count_valid = 0;
diff --git a/odb/source-files.c b/odb/source-files.c
new file mode 100644
index 0000000000..cbdaa6850f
--- /dev/null
+++ b/odb/source-files.c
@@ -0,0 +1,23 @@
+#include "git-compat-util.h"
+#include "object-file.h"
+#include "odb/source-files.h"
+#include "packfile.h"
+
+void odb_source_files_free(struct odb_source_files *files)
+{
+	if (!files)
+		return;
+	odb_source_loose_free(files->loose);
+	packfile_store_free(files->packed);
+	free(files);
+}
+
+struct odb_source_files *odb_source_files_new(struct odb_source *source)
+{
+	struct odb_source_files *files;
+	CALLOC_ARRAY(files, 1);
+	files->source = source;
+	files->loose = odb_source_loose_new(source);
+	files->packed = packfile_store_new(source);
+	return files;
+}
diff --git a/odb/source-files.h b/odb/source-files.h
new file mode 100644
index 0000000000..0b8bf773ca
--- /dev/null
+++ b/odb/source-files.h
@@ -0,0 +1,24 @@
+#ifndef ODB_SOURCE_FILES_H
+#define ODB_SOURCE_FILES_H
+
+struct odb_source_loose;
+struct odb_source;
+struct packfile_store;
+
+/*
+ * The files object database source uses a combination of loose objects and
+ * packfiles. It is the default backend used by Git to store objects.
+ */
+struct odb_source_files {
+	struct odb_source *source;
+	struct odb_source_loose *loose;
+	struct packfile_store *packed;
+};
+
+/* Allocate and initialize a new object source. */
+struct odb_source_files *odb_source_files_new(struct odb_source *source);
+
+/* Free the object source and release all associated resources. */
+void odb_source_files_free(struct odb_source_files *files);
+
+#endif
diff --git a/odb/source.c b/odb/source.c
index 7fc89806f9..9d7fd19f45 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -13,8 +13,7 @@ struct odb_source *odb_source_new(struct object_database *odb,
 	source->odb = odb;
 	source->local = local;
 	source->path = xstrdup(path);
-	source->loose = odb_source_loose_new(source);
-	source->packfiles = packfile_store_new(source);
+	source->files = odb_source_files_new(source);
 
 	return source;
 }
@@ -22,7 +21,6 @@ struct odb_source *odb_source_new(struct object_database *odb,
 void odb_source_free(struct odb_source *source)
 {
 	free(source->path);
-	odb_source_loose_free(source->loose);
-	packfile_store_free(source->packfiles);
+	odb_source_files_free(source->files);
 	free(source);
 }
diff --git a/odb/source.h b/odb/source.h
index 391d6d1e38..1c34265189 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,6 +1,8 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
+#include "odb/source-files.h"
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -19,11 +21,8 @@ struct odb_source {
 	/* Object database that owns this object source. */
 	struct object_database *odb;
 
-	/* Private state for loose objects. */
-	struct odb_source_loose *loose;
-
-	/* Should only be accessed directly by packfile.c and midx.c. */
-	struct packfile_store *packfiles;
+	/* The backend used to store objects. */
+	struct odb_source_files *files;
 
 	/*
 	 * Figure out whether this is the local source of the owning
diff --git a/odb/streaming.c b/odb/streaming.c
index 4a4474f891..26b0a1a0f5 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -187,7 +187,7 @@ static int istream_source(struct odb_read_stream **out,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (!packfile_store_read_object_stream(out, source->packfiles, oid) ||
+		if (!packfile_store_read_object_stream(out, source->files->packed, oid) ||
 		    !odb_source_loose_read_object_stream(out, source, oid))
 			return 0;
 	}
diff --git a/packfile.c b/packfile.c
index ce837f852a..4e1f6087ed 100644
--- a/packfile.c
+++ b/packfile.c
@@ -363,7 +363,7 @@ static int unuse_one_window(struct object_database *odb)
 	struct pack_window *lru_w = NULL, *lru_l = NULL;
 
 	for (source = odb->sources; source; source = source->next)
-		for (e = source->packfiles->packs.head; e; e = e->next)
+		for (e = source->files->packed->packs.head; e; e = e->next)
 			scan_windows(e->pack, &lru_p, &lru_w, &lru_l);
 
 	if (lru_p) {
@@ -537,7 +537,7 @@ static int close_one_pack(struct repository *r)
 	int accept_windows_inuse = 1;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		for (e = source->packfiles->packs.head; e; e = e->next) {
+		for (e = source->files->packed->packs.head; e; e = e->next) {
 			if (e->pack->pack_fd == -1)
 				continue;
 			find_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);
@@ -990,10 +990,10 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 	size_t base_len = full_name_len;
 
 	if (strip_suffix_mem(full_name, &base_len, ".idx") &&
-	    !(data->source->packfiles->midx &&
-	      midx_contains_pack(data->source->packfiles->midx, file_name))) {
+	    !(data->source->files->packed->midx &&
+	      midx_contains_pack(data->source->files->packed->midx, file_name))) {
 		char *trimmed_path = xstrndup(full_name, full_name_len);
-		packfile_store_load_pack(data->source->packfiles,
+		packfile_store_load_pack(data->source->files->packed,
 					 trimmed_path, data->source->local);
 		free(trimmed_path);
 	}
@@ -1248,7 +1248,7 @@ const struct packed_git *has_packed_and_bad(struct repository *r,
 
 	for (source = r->objects->sources; source; source = source->next) {
 		struct packfile_list_entry *e;
-		for (e = source->packfiles->packs.head; e; e = e->next)
+		for (e = source->files->packed->packs.head; e; e = e->next)
 			if (oidset_contains(&e->pack->bad_objects, oid))
 				return e->pack;
 	}
@@ -2254,7 +2254,7 @@ int has_object_pack(struct repository *r, const struct object_id *oid)
 
 	odb_prepare_alternates(r->objects);
 	for (source = r->objects->sources; source; source = source->next) {
-		int ret = find_pack_entry(source->packfiles, oid, &e);
+		int ret = find_pack_entry(source->files->packed, oid, &e);
 		if (ret)
 			return ret;
 	}
@@ -2271,7 +2271,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,
 	for (source = r->objects->sources; source; source = source->next) {
 		struct packed_git **cache;
 
-		cache = packfile_store_get_kept_pack_cache(source->packfiles, flags);
+		cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
diff --git a/packfile.h b/packfile.h
index 224142fd34..e8de06ee86 100644
--- a/packfile.h
+++ b/packfile.h
@@ -192,7 +192,7 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct
 	odb_prepare_alternates(repo->objects);
 
 	for (struct odb_source *source = repo->objects->sources; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);
+		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
 		if (!entry)
 			continue;
 		data.source = source;
@@ -212,7 +212,7 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *
 		return;
 
 	for (source = data->source->next; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->packfiles);
+		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
 		if (!entry)
 			continue;
 		data->source = source;

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 03/17] odb: embed base source in the "files" backend

[Patrick Steinhardt]

The "files" backend is implemented as a pointer in the `struct
odb_source`. This contradicts our typical pattern for pluggable backends
like we use it for example in the ref store or for object database
streams, where we typically embed the generic base structure in the
specialized implementation. This pattern has a couple of small benefits:

  - We avoid an extra allocation.

  - We hide implementation details in the generic structure.

  - We can easily downcast from a generic backend to the specialized
    structure and vice versa because the offsets are known at compile
    time.

  - It becomes trivial to identify locations where we depend on backend
    specific logic because the cast needs to be explicit.

Refactor our "files" object database source to do the same and embed the
`struct odb_source` in the `struct odb_source_files`.

There are still a bunch of sites in our code base where we do have to
access internals of the "files" backend. The intent is that those will
go away over time, but this will certainly take a while. Meanwhile,
provide a `odb_source_files_downcast()` function that can convert a
generic source into a "files" source.

As we only have a single source the downcast succeeds unconditionally
for now. Eventually though the intent is to make the cast `BUG()` in
case the caller requests to downcast a non-"files" backend to a "files"
backend.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/cat-file.c     |  3 ++-
 builtin/fast-import.c  | 12 ++++++++----
 builtin/grep.c         |  6 ++++--
 builtin/index-pack.c   |  8 +++++---
 builtin/pack-objects.c | 13 +++++++++----
 commit-graph.c         |  6 ++++--
 http.c                 |  3 ++-
 loose.c                | 23 ++++++++++++++---------
 midx.c                 | 26 +++++++++++++++-----------
 object-file.c          | 28 ++++++++++++++++------------
 odb.c                  | 26 ++++++++++++++++++--------
 odb/source-files.c     | 14 ++++++++++----
 odb/source-files.h     | 17 ++++++++++++++---
 odb/source.c           | 26 +++++++++++++++++++-------
 odb/source.h           | 31 +++++++++++++++++++++++++------
 odb/streaming.c        |  3 ++-
 packfile.c             | 26 +++++++++++++++++---------
 packfile.h             |  7 +++++--
 18 files changed, 189 insertions(+), 89 deletions(-)

diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 01a53f3f29..0c68d61b91 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -882,7 +882,8 @@ static void batch_each_object(struct batch_options *opt,
 		struct object_info oi = { 0 };
 
 		for (source = the_repository->objects->sources; source; source = source->next) {
-			int ret = packfile_store_for_each_object(source->files->packed, &oi,
+			struct odb_source_files *files = odb_source_files_downcast(source);
+			int ret = packfile_store_for_each_object(files->packed, &oi,
 								 batch_one_object_oi, &payload, flags);
 			if (ret)
 				break;
diff --git a/builtin/fast-import.c b/builtin/fast-import.c
index 627dcbf4f3..a41f95191e 100644
--- a/builtin/fast-import.c
+++ b/builtin/fast-import.c
@@ -875,6 +875,7 @@ static void end_packfile(void)
 	running = 1;
 	clear_delta_base_cache();
 	if (object_count) {
+		struct odb_source_files *files = odb_source_files_downcast(pack_data->repo->objects->sources);
 		struct packed_git *new_p;
 		struct object_id cur_pack_oid;
 		char *idx_name;
@@ -900,8 +901,7 @@ static void end_packfile(void)
 		idx_name = keep_pack(create_index());
 
 		/* Register the packfile with core git's machinery. */
-		new_p = packfile_store_load_pack(pack_data->repo->objects->sources->files->packed,
-						 idx_name, 1);
+		new_p = packfile_store_load_pack(files->packed, idx_name, 1);
 		if (!new_p)
 			die(_("core Git rejected index %s"), idx_name);
 		all_packs[pack_id] = new_p;
@@ -982,7 +982,9 @@ static int store_object(
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		if (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))
 			continue;
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
@@ -1187,7 +1189,9 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		if (!packfile_list_find_oid(packfile_store_get_packs(source->files->packed), &oid))
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		if (!packfile_list_find_oid(packfile_store_get_packs(files->packed), &oid))
 			continue;
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
diff --git a/builtin/grep.c b/builtin/grep.c
index c8d0e51415..61379909b8 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -1218,8 +1218,10 @@ int cmd_grep(int argc,
 			struct odb_source *source;
 
 			odb_prepare_alternates(the_repository->objects);
-			for (source = the_repository->objects->sources; source; source = source->next)
-				packfile_store_prepare(source->files->packed);
+			for (source = the_repository->objects->sources; source; source = source->next) {
+				struct odb_source_files *files = odb_source_files_downcast(source);
+				packfile_store_prepare(files->packed);
+			}
 		}
 
 		start_threads(&opt);
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index f0cce534b2..d1e47279a8 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1637,9 +1637,11 @@ static void final(const char *final_pack_name, const char *curr_pack_name,
 	rename_tmp_packfile(&final_index_name, curr_index_name, &index_name,
 			    hash, "idx", 1);
 
-	if (do_fsck_object && startup_info->have_repository)
-		packfile_store_load_pack(the_repository->objects->sources->files->packed,
-					 final_index_name, 0);
+	if (do_fsck_object && startup_info->have_repository) {
+		struct odb_source_files *files =
+			odb_source_files_downcast(the_repository->objects->sources);
+		packfile_store_load_pack(files->packed, final_index_name, 0);
+	}
 
 	if (!from_stdin) {
 		printf("%s\n", hash_to_hex(hash));
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 0c3c01cdc9..63fea80b08 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1531,7 +1531,8 @@ static int want_cruft_object_mtime(struct repository *r,
 	struct odb_source *source;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		struct packed_git **cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct packed_git **cache = packfile_store_get_kept_pack_cache(files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
@@ -1753,11 +1754,13 @@ static int want_object_in_pack_mtime(const struct object_id *oid,
 	}
 
 	for (source = the_repository->objects->sources; source; source = source->next) {
-		for (e = source->files->packed->packs.head; e; e = e->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		for (e = files->packed->packs.head; e; e = e->next) {
 			struct packed_git *p = e->pack;
 			want = want_object_in_pack_one(p, oid, exclude, found_pack, found_offset, found_mtime);
 			if (!exclude && want > 0)
-				packfile_list_prepend(&source->files->packed->packs, p);
+				packfile_list_prepend(&files->packed->packs, p);
 			if (want != -1)
 				return want;
 		}
@@ -4337,10 +4340,12 @@ static void add_objects_in_unpacked_packs(void)
 
 	odb_prepare_alternates(to_pack.repo->objects);
 	for (source = to_pack.repo->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
 		if (!source->local)
 			continue;
 
-		if (packfile_store_for_each_object(source->files->packed, &oi,
+		if (packfile_store_for_each_object(files->packed, &oi,
 						   add_object_in_unpacked_pack, NULL,
 						   ODB_FOR_EACH_OBJECT_PACK_ORDER |
 						   ODB_FOR_EACH_OBJECT_LOCAL_ONLY |
diff --git a/commit-graph.c b/commit-graph.c
index 967eb77047..f8e24145a5 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -1980,9 +1980,11 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx)
 			ctx->approx_nr_objects);
 
 	odb_prepare_alternates(ctx->r->objects);
-	for (source = ctx->r->objects->sources; source; source = source->next)
-		packfile_store_for_each_object(source->files->packed, &oi, add_packed_commits_oi,
+	for (source = ctx->r->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		packfile_store_for_each_object(files->packed, &oi, add_packed_commits_oi,
 					       ctx, ODB_FOR_EACH_OBJECT_PACK_ORDER);
+	}
 
 	if (ctx->progress_done < ctx->approx_nr_objects)
 		display_progress(ctx->progress, ctx->approx_nr_objects);
diff --git a/http.c b/http.c
index b44f493919..8ea1b9d1f6 100644
--- a/http.c
+++ b/http.c
@@ -2543,8 +2543,9 @@ int finish_http_pack_request(struct http_pack_request *preq)
 void http_install_packfile(struct packed_git *p,
 			   struct packfile_list *list_to_remove_from)
 {
+	struct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);
 	packfile_list_remove(list_to_remove_from, p);
-	packfile_store_add_pack(the_repository->objects->sources->files->packed, p);
+	packfile_store_add_pack(files->packed, p);
 }
 
 struct http_pack_request *new_http_pack_request(
diff --git a/loose.c b/loose.c
index c921d46b94..07333be696 100644
--- a/loose.c
+++ b/loose.c
@@ -3,6 +3,7 @@
 #include "path.h"
 #include "object-file.h"
 #include "odb.h"
+#include "odb/source-files.h"
 #include "hex.h"
 #include "repository.h"
 #include "wrapper.h"
@@ -49,27 +50,29 @@ static int insert_loose_map(struct odb_source *source,
 			    const struct object_id *oid,
 			    const struct object_id *compat_oid)
 {
-	struct loose_object_map *map = source->files->loose->map;
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	struct loose_object_map *map = files->loose->map;
 	int inserted = 0;
 
 	inserted |= insert_oid_pair(map->to_compat, oid, compat_oid);
 	inserted |= insert_oid_pair(map->to_storage, compat_oid, oid);
 	if (inserted)
-		oidtree_insert(source->files->loose->cache, compat_oid);
+		oidtree_insert(files->loose->cache, compat_oid);
 
 	return inserted;
 }
 
 static int load_one_loose_object_map(struct repository *repo, struct odb_source *source)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
 	FILE *fp;
 
-	if (!source->files->loose->map)
-		loose_object_map_init(&source->files->loose->map);
-	if (!source->files->loose->cache) {
-		ALLOC_ARRAY(source->files->loose->cache, 1);
-		oidtree_init(source->files->loose->cache);
+	if (!files->loose->map)
+		loose_object_map_init(&files->loose->map);
+	if (!files->loose->cache) {
+		ALLOC_ARRAY(files->loose->cache, 1);
+		oidtree_init(files->loose->cache);
 	}
 
 	insert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
@@ -125,7 +128,8 @@ int repo_read_loose_object_map(struct repository *repo)
 
 int repo_write_loose_object_map(struct repository *repo)
 {
-	kh_oid_map_t *map = repo->objects->sources->files->loose->map->to_compat;
+	struct odb_source_files *files = odb_source_files_downcast(repo->objects->sources);
+	kh_oid_map_t *map = files->loose->map->to_compat;
 	struct lock_file lock;
 	int fd;
 	khiter_t iter;
@@ -231,7 +235,8 @@ int repo_loose_object_map_oid(struct repository *repo,
 	khiter_t pos;
 
 	for (source = repo->objects->sources; source; source = source->next) {
-		struct loose_object_map *loose_map = source->files->loose->map;
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct loose_object_map *loose_map = files->loose->map;
 		if (!loose_map)
 			continue;
 		map = (to == repo->compat_hash_algo) ?
diff --git a/midx.c b/midx.c
index 698d10a1c6..ab8e2611d1 100644
--- a/midx.c
+++ b/midx.c
@@ -95,8 +95,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start,
 
 struct multi_pack_index *get_multi_pack_index(struct odb_source *source)
 {
-	packfile_store_prepare(source->files->packed);
-	return source->files->packed->midx;
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	packfile_store_prepare(files->packed);
+	return files->packed->midx;
 }
 
 static struct multi_pack_index *load_multi_pack_index_one(struct odb_source *source,
@@ -447,6 +448,7 @@ static uint32_t midx_for_pack(struct multi_pack_index **_m,
 int prepare_midx_pack(struct multi_pack_index *m,
 		      uint32_t pack_int_id)
 {
+	struct odb_source_files *files = odb_source_files_downcast(m->source);
 	struct strbuf pack_name = STRBUF_INIT;
 	struct packed_git *p;
 
@@ -457,10 +459,10 @@ int prepare_midx_pack(struct multi_pack_index *m,
 	if (m->packs[pack_int_id])
 		return 0;
 
-	strbuf_addf(&pack_name, "%s/pack/%s", m->source->path,
+	strbuf_addf(&pack_name, "%s/pack/%s", files->base.path,
 		    m->pack_names[pack_int_id]);
-	p = packfile_store_load_pack(m->source->files->packed,
-				     pack_name.buf, m->source->local);
+	p = packfile_store_load_pack(files->packed,
+				     pack_name.buf, files->base.local);
 	strbuf_release(&pack_name);
 
 	if (!p) {
@@ -703,18 +705,19 @@ int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id)
 
 int prepare_multi_pack_index_one(struct odb_source *source)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	struct repository *r = source->odb->repo;
 
 	prepare_repo_settings(r);
 	if (!r->settings.core_multi_pack_index)
 		return 0;
 
-	if (source->files->packed->midx)
+	if (files->packed->midx)
 		return 1;
 
-	source->files->packed->midx = load_multi_pack_index(source);
+	files->packed->midx = load_multi_pack_index(source);
 
-	return !!source->files->packed->midx;
+	return !!files->packed->midx;
 }
 
 int midx_checksum_valid(struct multi_pack_index *m)
@@ -803,9 +806,10 @@ void clear_midx_file(struct repository *r)
 		struct odb_source *source;
 
 		for (source = r->objects->sources; source; source = source->next) {
-			if (source->files->packed->midx)
-				close_midx(source->files->packed->midx);
-			source->files->packed->midx = NULL;
+			struct odb_source_files *files = odb_source_files_downcast(source);
+			if (files->packed->midx)
+				close_midx(files->packed->midx);
+			files->packed->midx = NULL;
 		}
 	}
 
diff --git a/object-file.c b/object-file.c
index db66ae5ebe..7ef8291a48 100644
--- a/object-file.c
+++ b/object-file.c
@@ -219,8 +219,9 @@ static void *odb_source_loose_map_object(struct odb_source *source,
 					 const struct object_id *oid,
 					 unsigned long *size)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	const char *p;
-	int fd = open_loose_object(source->files->loose, oid, &p);
+	int fd = open_loose_object(files->loose, oid, &p);
 
 	if (fd < 0)
 		return NULL;
@@ -401,6 +402,7 @@ static int read_object_info_from_path(struct odb_source *source,
 				      struct object_info *oi,
 				      enum object_info_flags flags)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	int ret;
 	int fd;
 	unsigned long mapsize;
@@ -423,7 +425,7 @@ static int read_object_info_from_path(struct odb_source *source,
 		struct stat st;
 
 		if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
-			ret = quick_has_loose(source->files->loose, oid) ? 0 : -1;
+			ret = quick_has_loose(files->loose, oid) ? 0 : -1;
 			goto out;
 		}
 
@@ -1866,33 +1868,34 @@ static int append_loose_object(const struct object_id *oid,
 struct oidtree *odb_source_loose_cache(struct odb_source *source,
 				       const struct object_id *oid)
 {
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	int subdir_nr = oid->hash[0];
 	struct strbuf buf = STRBUF_INIT;
-	size_t word_bits = bitsizeof(source->files->loose->subdir_seen[0]);
+	size_t word_bits = bitsizeof(files->loose->subdir_seen[0]);
 	size_t word_index = subdir_nr / word_bits;
 	size_t mask = (size_t)1u << (subdir_nr % word_bits);
 	uint32_t *bitmap;
 
 	if (subdir_nr < 0 ||
-	    (size_t) subdir_nr >= bitsizeof(source->files->loose->subdir_seen))
+	    (size_t) subdir_nr >= bitsizeof(files->loose->subdir_seen))
 		BUG("subdir_nr out of range");
 
-	bitmap = &source->files->loose->subdir_seen[word_index];
+	bitmap = &files->loose->subdir_seen[word_index];
 	if (*bitmap & mask)
-		return source->files->loose->cache;
-	if (!source->files->loose->cache) {
-		ALLOC_ARRAY(source->files->loose->cache, 1);
-		oidtree_init(source->files->loose->cache);
+		return files->loose->cache;
+	if (!files->loose->cache) {
+		ALLOC_ARRAY(files->loose->cache, 1);
+		oidtree_init(files->loose->cache);
 	}
 	strbuf_addstr(&buf, source->path);
 	for_each_file_in_obj_subdir(subdir_nr, &buf,
 				    source->odb->repo->hash_algo,
 				    append_loose_object,
 				    NULL, NULL,
-				    source->files->loose->cache);
+				    files->loose->cache);
 	*bitmap |= mask;
 	strbuf_release(&buf);
-	return source->files->loose->cache;
+	return files->loose->cache;
 }
 
 static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
@@ -1905,7 +1908,8 @@ static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
 
 void odb_source_loose_reprepare(struct odb_source *source)
 {
-	odb_source_loose_clear_cache(source->files->loose);
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	odb_source_loose_clear_cache(files->loose);
 }
 
 static int check_stream_oid(git_zstream *stream,
diff --git a/odb.c b/odb.c
index c9ebc7e741..e5aa8deb88 100644
--- a/odb.c
+++ b/odb.c
@@ -691,7 +691,8 @@ static int do_oid_object_info_extended(struct object_database *odb,
 
 		/* Most likely it's a loose object. */
 		for (source = odb->sources; source; source = source->next) {
-			if (!packfile_store_read_object_info(source->files->packed, real, oi, flags) ||
+			struct odb_source_files *files = odb_source_files_downcast(source);
+			if (!packfile_store_read_object_info(files->packed, real, oi, flags) ||
 			    !odb_source_loose_read_object_info(source, real, oi, flags))
 				return 0;
 		}
@@ -699,9 +700,11 @@ static int do_oid_object_info_extended(struct object_database *odb,
 		/* Not a loose object; someone else may have just packed it. */
 		if (!(flags & OBJECT_INFO_QUICK)) {
 			odb_reprepare(odb->repo->objects);
-			for (source = odb->sources; source; source = source->next)
-				if (!packfile_store_read_object_info(source->files->packed, real, oi, flags))
+			for (source = odb->sources; source; source = source->next) {
+				struct odb_source_files *files = odb_source_files_downcast(source);
+				if (!packfile_store_read_object_info(files->packed, real, oi, flags))
 					return 0;
+			}
 		}
 
 		/*
@@ -962,7 +965,9 @@ int odb_freshen_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (packfile_store_freshen_object(source->files->packed, oid))
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
+		if (packfile_store_freshen_object(files->packed, oid))
 			return 1;
 
 		if (odb_source_loose_freshen_object(source, oid))
@@ -982,6 +987,8 @@ int odb_for_each_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (struct odb_source *source = odb->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+
 		if (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)
 			continue;
 
@@ -992,7 +999,7 @@ int odb_for_each_object(struct object_database *odb,
 				return ret;
 		}
 
-		ret = packfile_store_for_each_object(source->files->packed, request,
+		ret = packfile_store_for_each_object(files->packed, request,
 						     cb, cb_data, flags);
 		if (ret)
 			return ret;
@@ -1090,8 +1097,10 @@ struct object_database *odb_new(struct repository *repo,
 void odb_close(struct object_database *o)
 {
 	struct odb_source *source;
-	for (source = o->sources; source; source = source->next)
-		packfile_store_close(source->files->packed);
+	for (source = o->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		packfile_store_close(files->packed);
+	}
 	close_commit_graph(o);
 }
 
@@ -1148,8 +1157,9 @@ void odb_reprepare(struct object_database *o)
 	odb_prepare_alternates(o);
 
 	for (source = o->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
 		odb_source_loose_reprepare(source);
-		packfile_store_reprepare(source->files->packed);
+		packfile_store_reprepare(files->packed);
 	}
 
 	o->approximate_object_count_valid = 0;
diff --git a/odb/source-files.c b/odb/source-files.c
index cbdaa6850f..a43a197157 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -1,5 +1,6 @@
 #include "git-compat-util.h"
 #include "object-file.h"
+#include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
 
@@ -9,15 +10,20 @@ void odb_source_files_free(struct odb_source_files *files)
 		return;
 	odb_source_loose_free(files->loose);
 	packfile_store_free(files->packed);
+	odb_source_release(&files->base);
 	free(files);
 }
 
-struct odb_source_files *odb_source_files_new(struct odb_source *source)
+struct odb_source_files *odb_source_files_new(struct object_database *odb,
+					      const char *path,
+					      bool local)
 {
 	struct odb_source_files *files;
+
 	CALLOC_ARRAY(files, 1);
-	files->source = source;
-	files->loose = odb_source_loose_new(source);
-	files->packed = packfile_store_new(source);
+	odb_source_init(&files->base, odb, path, local);
+	files->loose = odb_source_loose_new(&files->base);
+	files->packed = packfile_store_new(&files->base);
+
 	return files;
 }
diff --git a/odb/source-files.h b/odb/source-files.h
index 0b8bf773ca..859a8f518a 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -1,8 +1,9 @@
 #ifndef ODB_SOURCE_FILES_H
 #define ODB_SOURCE_FILES_H
 
+#include "odb/source.h"
+
 struct odb_source_loose;
-struct odb_source;
 struct packfile_store;
 
 /*
@@ -10,15 +11,25 @@ struct packfile_store;
  * packfiles. It is the default backend used by Git to store objects.
  */
 struct odb_source_files {
-	struct odb_source *source;
+	struct odb_source base;
 	struct odb_source_loose *loose;
 	struct packfile_store *packed;
 };
 
 /* Allocate and initialize a new object source. */
-struct odb_source_files *odb_source_files_new(struct odb_source *source);
+struct odb_source_files *odb_source_files_new(struct object_database *odb,
+					      const char *path,
+					      bool local);
 
 /* Free the object source and release all associated resources. */
 void odb_source_files_free(struct odb_source_files *files);
 
+/*
+ * Cast the given object database source to the files backend.
+ */
+static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
+{
+	return container_of(source, struct odb_source_files, base);
+}
+
 #endif
diff --git a/odb/source.c b/odb/source.c
index 9d7fd19f45..d8b2176a94 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -1,5 +1,6 @@
 #include "git-compat-util.h"
 #include "object-file.h"
+#include "odb/source-files.h"
 #include "odb/source.h"
 #include "packfile.h"
 
@@ -7,20 +8,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local)
 {
-	struct odb_source *source;
+	return &odb_source_files_new(odb, path, local)->base;
+}
 
-	CALLOC_ARRAY(source, 1);
+void odb_source_init(struct odb_source *source,
+		     struct object_database *odb,
+		     const char *path,
+		     bool local)
+{
 	source->odb = odb;
 	source->local = local;
 	source->path = xstrdup(path);
-	source->files = odb_source_files_new(source);
-
-	return source;
 }
 
 void odb_source_free(struct odb_source *source)
 {
+	struct odb_source_files *files;
+	if (!source)
+		return;
+	files = odb_source_files_downcast(source);
+	odb_source_files_free(files);
+}
+
+void odb_source_release(struct odb_source *source)
+{
+	if (!source)
+		return;
 	free(source->path);
-	odb_source_files_free(source->files);
-	free(source);
 }
diff --git a/odb/source.h b/odb/source.h
index 1c34265189..e6698b73a3 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,8 +1,6 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
-#include "odb/source-files.h"
-
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -21,9 +19,6 @@ struct odb_source {
 	/* Object database that owns this object source. */
 	struct object_database *odb;
 
-	/* The backend used to store objects. */
-	struct odb_source_files *files;
-
 	/*
 	 * Figure out whether this is the local source of the owning
 	 * repository, which would typically be its ".git/objects" directory.
@@ -53,7 +48,31 @@ struct odb_source *odb_source_new(struct object_database *odb,
 				  const char *path,
 				  bool local);
 
-/* Free the object database source, releasing all associated resources. */
+/*
+ * Initialize the source for the given object database located at `path`.
+ * `local` indicates whether or not the source is the local and thus primary
+ * object source of the object database.
+ *
+ * This function is only supposed to be called by specific object source
+ * implementations.
+ */
+void odb_source_init(struct odb_source *source,
+		     struct object_database *odb,
+		     const char *path,
+		     bool local);
+
+/*
+ * Free the object database source, releasing all associated resources and
+ * freeing the structure itself.
+ */
 void odb_source_free(struct odb_source *source);
 
+/*
+ * Release the object database source, releasing all associated resources.
+ *
+ * This function is only supposed to be called by specific object source
+ * implementations.
+ */
+void odb_source_release(struct odb_source *source);
+
 #endif
diff --git a/odb/streaming.c b/odb/streaming.c
index 26b0a1a0f5..19cda9407d 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -187,7 +187,8 @@ static int istream_source(struct odb_read_stream **out,
 
 	odb_prepare_alternates(odb);
 	for (source = odb->sources; source; source = source->next) {
-		if (!packfile_store_read_object_stream(out, source->files->packed, oid) ||
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		if (!packfile_store_read_object_stream(out, files->packed, oid) ||
 		    !odb_source_loose_read_object_stream(out, source, oid))
 			return 0;
 	}
diff --git a/packfile.c b/packfile.c
index 4e1f6087ed..da1c0dfa39 100644
--- a/packfile.c
+++ b/packfile.c
@@ -362,9 +362,11 @@ static int unuse_one_window(struct object_database *odb)
 	struct packed_git *lru_p = NULL;
 	struct pack_window *lru_w = NULL, *lru_l = NULL;
 
-	for (source = odb->sources; source; source = source->next)
-		for (e = source->files->packed->packs.head; e; e = e->next)
+	for (source = odb->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		for (e = files->packed->packs.head; e; e = e->next)
 			scan_windows(e->pack, &lru_p, &lru_w, &lru_l);
+	}
 
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -537,7 +539,8 @@ static int close_one_pack(struct repository *r)
 	int accept_windows_inuse = 1;
 
 	for (source = r->objects->sources; source; source = source->next) {
-		for (e = source->files->packed->packs.head; e; e = e->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		for (e = files->packed->packs.head; e; e = e->next) {
 			if (e->pack->pack_fd == -1)
 				continue;
 			find_lru_pack(e->pack, &lru_p, &mru_w, &accept_windows_inuse);
@@ -987,13 +990,14 @@ static void prepare_pack(const char *full_name, size_t full_name_len,
 			 const char *file_name, void *_data)
 {
 	struct prepare_pack_data *data = (struct prepare_pack_data *)_data;
+	struct odb_source_files *files = odb_source_files_downcast(data->source);
 	size_t base_len = full_name_len;
 
 	if (strip_suffix_mem(full_name, &base_len, ".idx") &&
-	    !(data->source->files->packed->midx &&
-	      midx_contains_pack(data->source->files->packed->midx, file_name))) {
+	    !(files->packed->midx &&
+	      midx_contains_pack(files->packed->midx, file_name))) {
 		char *trimmed_path = xstrndup(full_name, full_name_len);
-		packfile_store_load_pack(data->source->files->packed,
+		packfile_store_load_pack(files->packed,
 					 trimmed_path, data->source->local);
 		free(trimmed_path);
 	}
@@ -1247,8 +1251,10 @@ const struct packed_git *has_packed_and_bad(struct repository *r,
 	struct odb_source *source;
 
 	for (source = r->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
 		struct packfile_list_entry *e;
-		for (e = source->files->packed->packs.head; e; e = e->next)
+
+		for (e = files->packed->packs.head; e; e = e->next)
 			if (oidset_contains(&e->pack->bad_objects, oid))
 				return e->pack;
 	}
@@ -2254,7 +2260,8 @@ int has_object_pack(struct repository *r, const struct object_id *oid)
 
 	odb_prepare_alternates(r->objects);
 	for (source = r->objects->sources; source; source = source->next) {
-		int ret = find_pack_entry(source->files->packed, oid, &e);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		int ret = find_pack_entry(files->packed, oid, &e);
 		if (ret)
 			return ret;
 	}
@@ -2269,9 +2276,10 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,
 	struct pack_entry e;
 
 	for (source = r->objects->sources; source; source = source->next) {
+		struct odb_source_files *files = odb_source_files_downcast(source);
 		struct packed_git **cache;
 
-		cache = packfile_store_get_kept_pack_cache(source->files->packed, flags);
+		cache = packfile_store_get_kept_pack_cache(files->packed, flags);
 
 		for (; *cache; cache++) {
 			struct packed_git *p = *cache;
diff --git a/packfile.h b/packfile.h
index e8de06ee86..64a31738c0 100644
--- a/packfile.h
+++ b/packfile.h
@@ -4,6 +4,7 @@
 #include "list.h"
 #include "object.h"
 #include "odb.h"
+#include "odb/source-files.h"
 #include "oidset.h"
 #include "repository.h"
 #include "strmap.h"
@@ -192,7 +193,8 @@ static inline struct repo_for_each_pack_data repo_for_eack_pack_data_init(struct
 	odb_prepare_alternates(repo->objects);
 
 	for (struct odb_source *source = repo->objects->sources; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct packfile_list_entry *entry = packfile_store_get_packs(files->packed);
 		if (!entry)
 			continue;
 		data.source = source;
@@ -212,7 +214,8 @@ static inline void repo_for_each_pack_data_next(struct repo_for_each_pack_data *
 		return;
 
 	for (source = data->source->next; source; source = source->next) {
-		struct packfile_list_entry *entry = packfile_store_get_packs(source->files->packed);
+		struct odb_source_files *files = odb_source_files_downcast(source);
+		struct packfile_list_entry *entry = packfile_store_get_packs(files->packed);
 		if (!entry)
 			continue;
 		data->source = source;

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 04/17] odb: move reparenting logic into respective subsystems

[Patrick Steinhardt]

The primary object database source may be initialized with a relative
path. When the process changes its current working directory we thus
have to update this path and have it point to the same path, but
relative to the new working directory.

This logic is handled in the object database layer. It consists of three
steps:

  1. We undo any potential temporary object directory, which are used
     for transactions. This is done so that we don't end up modifying
     the temporary object database source that got applied for the
     transaction.

  2. We then iterate through the non-transactional sources and reparent
     their respective paths.

  3. We reapply the temporary object directory, but update its path.

All of this logic is heavily tied to how the object database source
handles paths in the first place. It's an internal implementation
detail, and as sources may not even use an on-disk path at all it is not
a mechanism that applies to all potential sources.

Refactor the code so that the logic to reparent the sources is hosted by
the "files" source and the temporary object directory subsystems,
respectively. This logic is easier to reason about, but it also ensures
that this logic is handled at the correct level.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 37 -------------------------------------
 odb/source-files.c | 23 +++++++++++++++++++++++
 tmp-objdir.c       | 42 +++++++++++++++++++-----------------------
 tmp-objdir.h       | 15 ---------------
 4 files changed, 42 insertions(+), 75 deletions(-)

diff --git a/odb.c b/odb.c
index e5aa8deb88..86f7cf70a8 100644
--- a/odb.c
+++ b/odb.c
@@ -1,6 +1,5 @@
 #include "git-compat-util.h"
 #include "abspath.h"
-#include "chdir-notify.h"
 #include "commit-graph.h"
 #include "config.h"
 #include "dir.h"
@@ -1037,38 +1036,6 @@ int odb_write_object_stream(struct object_database *odb,
 	return odb_source_loose_write_stream(odb->sources, stream, len, oid);
 }
 
-static void odb_update_commondir(const char *name UNUSED,
-				 const char *old_cwd,
-				 const char *new_cwd,
-				 void *cb_data)
-{
-	struct object_database *odb = cb_data;
-	struct tmp_objdir *tmp_objdir;
-	struct odb_source *source;
-
-	tmp_objdir = tmp_objdir_unapply_primary_odb();
-
-	/*
-	 * In theory, we only have to do this for the primary object source, as
-	 * alternates' paths are always resolved to an absolute path.
-	 */
-	for (source = odb->sources; source; source = source->next) {
-		char *path;
-
-		if (is_absolute_path(source->path))
-			continue;
-
-		path = reparent_relative_path(old_cwd, new_cwd,
-					      source->path);
-
-		free(source->path);
-		source->path = path;
-	}
-
-	if (tmp_objdir)
-		tmp_objdir_reapply_primary_odb(tmp_objdir, old_cwd, new_cwd);
-}
-
 struct object_database *odb_new(struct repository *repo,
 				const char *primary_source,
 				const char *secondary_sources)
@@ -1089,8 +1056,6 @@ struct object_database *odb_new(struct repository *repo,
 
 	free(to_free);
 
-	chdir_notify_register(NULL, odb_update_commondir, o);
-
 	return o;
 }
 
@@ -1136,8 +1101,6 @@ void odb_free(struct object_database *o)
 
 	string_list_clear(&o->submodule_source_paths, 0);
 
-	chdir_notify_unregister(NULL, odb_update_commondir, o);
-
 	free(o);
 }
 
diff --git a/odb/source-files.c b/odb/source-files.c
index a43a197157..df0ea9ee62 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -1,13 +1,28 @@
 #include "git-compat-util.h"
+#include "abspath.h"
+#include "chdir-notify.h"
 #include "object-file.h"
 #include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
 
+static void odb_source_files_reparent(const char *name UNUSED,
+				      const char *old_cwd,
+				      const char *new_cwd,
+				      void *cb_data)
+{
+	struct odb_source_files *files = cb_data;
+	char *path = reparent_relative_path(old_cwd, new_cwd,
+					    files->base.path);
+	free(files->base.path);
+	files->base.path = path;
+}
+
 void odb_source_files_free(struct odb_source_files *files)
 {
 	if (!files)
 		return;
+	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
 	odb_source_loose_free(files->loose);
 	packfile_store_free(files->packed);
 	odb_source_release(&files->base);
@@ -25,5 +40,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->loose = odb_source_loose_new(&files->base);
 	files->packed = packfile_store_new(&files->base);
 
+	/*
+	 * Ideally, we would only ever store absolute paths in the source. This
+	 * is not (yet) possible though because we access and assume relative
+	 * paths in the primary ODB source in some user-facing functionality.
+	 */
+	if (!is_absolute_path(path))
+		chdir_notify_register(NULL, odb_source_files_reparent, files);
+
 	return files;
 }
diff --git a/tmp-objdir.c b/tmp-objdir.c
index 9f5a1788cd..e436eed07e 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -36,6 +36,21 @@ static void tmp_objdir_free(struct tmp_objdir *t)
 	free(t);
 }
 
+static void tmp_objdir_reparent(const char *name UNUSED,
+				const char *old_cwd,
+				const char *new_cwd,
+				void *cb_data)
+{
+	struct tmp_objdir *t = cb_data;
+	char *path;
+
+	path = reparent_relative_path(old_cwd, new_cwd,
+				      t->path.buf);
+	strbuf_reset(&t->path);
+	strbuf_addstr(&t->path, path);
+	free(path);
+}
+
 int tmp_objdir_destroy(struct tmp_objdir *t)
 {
 	int err;
@@ -51,6 +66,7 @@ int tmp_objdir_destroy(struct tmp_objdir *t)
 
 	err = remove_dir_recursively(&t->path, 0);
 
+	chdir_notify_unregister(NULL, tmp_objdir_reparent, t);
 	tmp_objdir_free(t);
 
 	return err;
@@ -137,6 +153,9 @@ struct tmp_objdir *tmp_objdir_create(struct repository *r,
 	strbuf_addf(&t->path, "%s/tmp_objdir-%s-XXXXXX",
 		    repo_get_object_directory(r), prefix);
 
+	if (!is_absolute_path(t->path.buf))
+		chdir_notify_register(NULL, tmp_objdir_reparent, t);
+
 	if (!mkdtemp(t->path.buf)) {
 		/* free, not destroy, as we never touched the filesystem */
 		tmp_objdir_free(t);
@@ -315,26 +334,3 @@ void tmp_objdir_replace_primary_odb(struct tmp_objdir *t, int will_destroy)
 							  t->path.buf, will_destroy);
 	t->will_destroy = will_destroy;
 }
-
-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void)
-{
-	if (!the_tmp_objdir || !the_tmp_objdir->prev_source)
-		return NULL;
-
-	odb_restore_primary_source(the_tmp_objdir->repo->objects,
-				   the_tmp_objdir->prev_source, the_tmp_objdir->path.buf);
-	the_tmp_objdir->prev_source = NULL;
-	return the_tmp_objdir;
-}
-
-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *t, const char *old_cwd,
-		const char *new_cwd)
-{
-	char *path;
-
-	path = reparent_relative_path(old_cwd, new_cwd, t->path.buf);
-	strbuf_reset(&t->path);
-	strbuf_addstr(&t->path, path);
-	free(path);
-	tmp_objdir_replace_primary_odb(t, t->will_destroy);
-}
diff --git a/tmp-objdir.h b/tmp-objdir.h
index fceda14979..ccf800faa7 100644
--- a/tmp-objdir.h
+++ b/tmp-objdir.h
@@ -68,19 +68,4 @@ void tmp_objdir_add_as_alternate(const struct tmp_objdir *);
  */
 void tmp_objdir_replace_primary_odb(struct tmp_objdir *, int will_destroy);
 
-/*
- * If the primary object database was replaced by a temporary object directory,
- * restore it to its original value while keeping the directory contents around.
- * Returns NULL if the primary object database was not replaced.
- */
-struct tmp_objdir *tmp_objdir_unapply_primary_odb(void);
-
-/*
- * Reapplies the former primary temporary object database, after potentially
- * changing its relative path.
- */
-void tmp_objdir_reapply_primary_odb(struct tmp_objdir *, const char *old_cwd,
-		const char *new_cwd);
-
-
 #endif /* TMP_OBJDIR_H */

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 05/17] odb/source: introduce source type for robustness

[Patrick Steinhardt]

When a caller holds a `struct odb_source`, they have no way of telling
what type the source is. This doesn't really cause any problems in the
current status quo as we only have a single type anyway, "files". But
going forward we expect to add more types, and if so it will become
necessary to tell the sources apart.

Introduce a new enum to cover this use case and assert that the given
source actually matches the target source when performing the downcast.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c |  2 +-
 odb/source-files.h |  5 ++++-
 odb/source.c       |  2 ++
 odb/source.h       | 15 +++++++++++++++
 4 files changed, 22 insertions(+), 2 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index df0ea9ee62..7496e1d9f8 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -36,7 +36,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	struct odb_source_files *files;
 
 	CALLOC_ARRAY(files, 1);
-	odb_source_init(&files->base, odb, path, local);
+	odb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);
 	files->loose = odb_source_loose_new(&files->base);
 	files->packed = packfile_store_new(&files->base);
 
diff --git a/odb/source-files.h b/odb/source-files.h
index 859a8f518a..803fa995fb 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -25,10 +25,13 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 void odb_source_files_free(struct odb_source_files *files);
 
 /*
- * Cast the given object database source to the files backend.
+ * Cast the given object database source to the files backend. This will cause
+ * a BUG in case the source doesn't use this backend.
  */
 static inline struct odb_source_files *odb_source_files_downcast(struct odb_source *source)
 {
+	if (source->type != ODB_SOURCE_FILES)
+		BUG("trying to downcast source of type '%d' to files", source->type);
 	return container_of(source, struct odb_source_files, base);
 }
 
diff --git a/odb/source.c b/odb/source.c
index d8b2176a94..c7dcc528f6 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -13,10 +13,12 @@ struct odb_source *odb_source_new(struct object_database *odb,
 
 void odb_source_init(struct odb_source *source,
 		     struct object_database *odb,
+		     enum odb_source_type type,
 		     const char *path,
 		     bool local)
 {
 	source->odb = odb;
+	source->type = type;
 	source->local = local;
 	source->path = xstrdup(path);
 }
diff --git a/odb/source.h b/odb/source.h
index e6698b73a3..45b72b81a0 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,6 +1,17 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
+enum odb_source_type {
+	/*
+	 * The "unknown" type, which should never be in use. This type mostly
+	 * exists to catch cases where the type field remains zeroed out.
+	 */
+	ODB_SOURCE_UNKNOWN,
+
+	/* The "files" backend that uses loose objects and packfiles. */
+	ODB_SOURCE_FILES,
+};
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -19,6 +30,9 @@ struct odb_source {
 	/* Object database that owns this object source. */
 	struct object_database *odb;
 
+	/* The type used by this source. */
+	enum odb_source_type type;
+
 	/*
 	 * Figure out whether this is the local source of the owning
 	 * repository, which would typically be its ".git/objects" directory.
@@ -58,6 +72,7 @@ struct odb_source *odb_source_new(struct object_database *odb,
  */
 void odb_source_init(struct odb_source *source,
 		     struct object_database *odb,
+		     enum odb_source_type type,
 		     const char *path,
 		     bool local);
 

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 06/17] odb/source: make `free()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 7 ++++---
 odb/source-files.h | 3 ---
 odb/source.c       | 4 +---
 odb/source.h       | 6 ++++++
 4 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index 7496e1d9f8..65d7805c5a 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -18,10 +18,9 @@ static void odb_source_files_reparent(const char *name UNUSED,
 	files->base.path = path;
 }
 
-void odb_source_files_free(struct odb_source_files *files)
+static void odb_source_files_free(struct odb_source *source)
 {
-	if (!files)
-		return;
+	struct odb_source_files *files = odb_source_files_downcast(source);
 	chdir_notify_unregister(NULL, odb_source_files_reparent, files);
 	odb_source_loose_free(files->loose);
 	packfile_store_free(files->packed);
@@ -40,6 +39,8 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->loose = odb_source_loose_new(&files->base);
 	files->packed = packfile_store_new(&files->base);
 
+	files->base.free = odb_source_files_free;
+
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
 	 * is not (yet) possible though because we access and assume relative
diff --git a/odb/source-files.h b/odb/source-files.h
index 803fa995fb..23a3b4e04b 100644
--- a/odb/source-files.h
+++ b/odb/source-files.h
@@ -21,9 +21,6 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local);
 
-/* Free the object source and release all associated resources. */
-void odb_source_files_free(struct odb_source_files *files);
-
 /*
  * Cast the given object database source to the files backend. This will cause
  * a BUG in case the source doesn't use this backend.
diff --git a/odb/source.c b/odb/source.c
index c7dcc528f6..7993dcbd65 100644
--- a/odb/source.c
+++ b/odb/source.c
@@ -25,11 +25,9 @@ void odb_source_init(struct odb_source *source,
 
 void odb_source_free(struct odb_source *source)
 {
-	struct odb_source_files *files;
 	if (!source)
 		return;
-	files = odb_source_files_downcast(source);
-	odb_source_files_free(files);
+	source->free(source);
 }
 
 void odb_source_release(struct odb_source *source)
diff --git a/odb/source.h b/odb/source.h
index 45b72b81a0..4973fb4251 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -51,6 +51,12 @@ struct odb_source {
 	 * the current working directory.
 	 */
 	char *path;
+
+	/*
+	 * This callback is expected to free the underlying object database source and
+	 * all associated resources. The function will never be called with a NULL pointer.
+	 */
+	void (*free)(struct odb_source *source);
 };
 
 /*

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 07/17] odb/source: make `reprepare()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  7 ++-----
 odb/source-files.c |  8 ++++++++
 odb/source.h       | 17 +++++++++++++++++
 3 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/odb.c b/odb.c
index 86f7cf70a8..2cf6a53dc3 100644
--- a/odb.c
+++ b/odb.c
@@ -1119,11 +1119,8 @@ void odb_reprepare(struct object_database *o)
 	o->loaded_alternates = 0;
 	odb_prepare_alternates(o);
 
-	for (source = o->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-		odb_source_loose_reprepare(source);
-		packfile_store_reprepare(files->packed);
-	}
+	for (source = o->sources; source; source = source->next)
+		odb_source_reprepare(source);
 
 	o->approximate_object_count_valid = 0;
 
diff --git a/odb/source-files.c b/odb/source-files.c
index 65d7805c5a..d0f7ee072e 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -28,6 +28,13 @@ static void odb_source_files_free(struct odb_source *source)
 	free(files);
 }
 
+static void odb_source_files_reprepare(struct odb_source *source)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	odb_source_loose_reprepare(&files->base);
+	packfile_store_reprepare(files->packed);
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -40,6 +47,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->packed = packfile_store_new(&files->base);
 
 	files->base.free = odb_source_files_free;
+	files->base.reprepare = odb_source_files_reprepare;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 4973fb4251..09cca839fe 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -57,6 +57,13 @@ struct odb_source {
 	 * all associated resources. The function will never be called with a NULL pointer.
 	 */
 	void (*free)(struct odb_source *source);
+
+	/*
+	 * This callback is expected to clear underlying caches of the object
+	 * database source. The function is called when the repository has for
+	 * example just been repacked so that new objects will become visible.
+	 */
+	void (*reprepare)(struct odb_source *source);
 };
 
 /*
@@ -96,4 +103,14 @@ void odb_source_free(struct odb_source *source);
  */
 void odb_source_release(struct odb_source *source);
 
+/*
+ * Reprepare the object database source and clear any caches. Depending on the
+ * backend used this may have the effect that concurrently-written objects
+ * become visible.
+ */
+static inline void odb_source_reprepare(struct odb_source *source)
+{
+	source->reprepare(source);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 08/17] odb/source: make `close()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  6 ++----
 odb/source-files.c |  7 +++++++
 odb/source.h       | 18 ++++++++++++++++++
 3 files changed, 27 insertions(+), 4 deletions(-)

diff --git a/odb.c b/odb.c
index 2cf6a53dc3..f7487eb0df 100644
--- a/odb.c
+++ b/odb.c
@@ -1062,10 +1062,8 @@ struct object_database *odb_new(struct repository *repo,
 void odb_close(struct object_database *o)
 {
 	struct odb_source *source;
-	for (source = o->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-		packfile_store_close(files->packed);
-	}
+	for (source = o->sources; source; source = source->next)
+		odb_source_close(source);
 	close_commit_graph(o);
 }
 
diff --git a/odb/source-files.c b/odb/source-files.c
index d0f7ee072e..20a24f524a 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -28,6 +28,12 @@ static void odb_source_files_free(struct odb_source *source)
 	free(files);
 }
 
+static void odb_source_files_close(struct odb_source *source)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	packfile_store_close(files->packed);
+}
+
 static void odb_source_files_reprepare(struct odb_source *source)
 {
 	struct odb_source_files *files = odb_source_files_downcast(source);
@@ -47,6 +53,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->packed = packfile_store_new(&files->base);
 
 	files->base.free = odb_source_files_free;
+	files->base.close = odb_source_files_close;
 	files->base.reprepare = odb_source_files_reprepare;
 
 	/*
diff --git a/odb/source.h b/odb/source.h
index 09cca839fe..0e6c6abdb1 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -58,6 +58,14 @@ struct odb_source {
 	 */
 	void (*free)(struct odb_source *source);
 
+	/*
+	 * This callback is expected to close any open resources, like for
+	 * example file descriptors or connections. The source is expected to
+	 * still be usable after it has been closed. Closed resources may need
+	 * to be reopened in that case.
+	 */
+	void (*close)(struct odb_source *source);
+
 	/*
 	 * This callback is expected to clear underlying caches of the object
 	 * database source. The function is called when the repository has for
@@ -103,6 +111,16 @@ void odb_source_free(struct odb_source *source);
  */
 void odb_source_release(struct odb_source *source);
 
+/*
+ * Close the object database source without releasing he underlying data. The
+ * source can still be used going forward, but it first needs to be reopened.
+ * This can be useful to reduce resource usage.
+ */
+static inline void odb_source_close(struct odb_source *source)
+{
+	source->close(source);
+}
+
 /*
  * Reprepare the object database source and clear any caches. Depending on the
  * backend used this may have the effect that concurrently-written objects

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 09/17] odb/source: make `read_object_info()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Note that this function is a bit less straight-forward to convert
compared to the other functions. The reason here is that the logic to
read an object is:

  1. We try to read the object. If it exists we return it.

  2. If the object does not exist we reprepare the object database
     source.

  3. We then try reading the object info a second time in case the
     reprepare caused it to appear.

The second read is only supposed to happen for the packfile store
though, as reading loose objects is not impacted by repreparing the
object database.

Ideally, we'd just move this whole logic into the ODB source. But that's
not easily possible because we try to avoid the reprepare unless really
required, which is after we have found out that no other ODB source
contains the object, either. So the logic spans across multiple ODB
sources, and consequently we cannot move it into an individual source.

Instead, introduce a new flag `OBJECT_INFO_SECOND_READ` that tells the
backend that we already tried to look up the object once, and that this
time around the ODB source should try to find any new objects that may
have surfaced due to an on-disk change.

With this flag, the "files" backend can trivially skip trying to re-read
the object as a loose object. Furthermore, as we know that we only try
the second read via the packfile store, we can skip repreparing loose
objects and only reprepare the packfile store.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 object-file.c      | 10 +++++++
 odb.c              | 22 +++++++--------
 odb.h              | 24 -----------------
 odb/source-files.c | 15 +++++++++++
 odb/source.h       | 78 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.c         | 10 ++++++-
 6 files changed, 122 insertions(+), 37 deletions(-)

diff --git a/object-file.c b/object-file.c
index 7ef8291a48..eefde72c7d 100644
--- a/object-file.c
+++ b/object-file.c
@@ -546,6 +546,16 @@ int odb_source_loose_read_object_info(struct odb_source *source,
 				      enum object_info_flags flags)
 {
 	static struct strbuf buf = STRBUF_INIT;
+
+	/*
+	 * The second read shouldn't cause new loose objects to show up, unless
+	 * there was a race condition with a secondary process. We don't care
+	 * about this case though, so we simply skip reading loose objects a
+	 * second time.
+	 */
+	if (flags & OBJECT_INFO_SECOND_READ)
+		return -1;
+
 	odb_loose_path(source, &buf, oid);
 	return read_object_info_from_path(source, buf.buf, oid, oi, flags);
 }
diff --git a/odb.c b/odb.c
index f7487eb0df..c0b8cd062b 100644
--- a/odb.c
+++ b/odb.c
@@ -688,22 +688,20 @@ static int do_oid_object_info_extended(struct object_database *odb,
 	while (1) {
 		struct odb_source *source;
 
-		/* Most likely it's a loose object. */
-		for (source = odb->sources; source; source = source->next) {
-			struct odb_source_files *files = odb_source_files_downcast(source);
-			if (!packfile_store_read_object_info(files->packed, real, oi, flags) ||
-			    !odb_source_loose_read_object_info(source, real, oi, flags))
+		for (source = odb->sources; source; source = source->next)
+			if (!odb_source_read_object_info(source, real, oi, flags))
 				return 0;
-		}
 
-		/* Not a loose object; someone else may have just packed it. */
+		/*
+		 * When the object hasn't been found we try a second read and
+		 * tell the sources so. This may cause them to invalidate
+		 * caches or reload on-disk state.
+		 */
 		if (!(flags & OBJECT_INFO_QUICK)) {
-			odb_reprepare(odb->repo->objects);
-			for (source = odb->sources; source; source = source->next) {
-				struct odb_source_files *files = odb_source_files_downcast(source);
-				if (!packfile_store_read_object_info(files->packed, real, oi, flags))
+			for (source = odb->sources; source; source = source->next)
+				if (!odb_source_read_object_info(source, real, oi,
+								 flags | OBJECT_INFO_SECOND_READ))
 					return 0;
-			}
 		}
 
 		/*
diff --git a/odb.h b/odb.h
index e13b5b7c44..70ffb033f9 100644
--- a/odb.h
+++ b/odb.h
@@ -339,30 +339,6 @@ struct object_info {
  */
 #define OBJECT_INFO_INIT { 0 }
 
-/* Flags that can be passed to `odb_read_object_info_extended()`. */
-enum object_info_flags {
-	/* Invoke lookup_replace_object() on the given hash. */
-	OBJECT_INFO_LOOKUP_REPLACE = (1 << 0),
-
-	/* Do not reprepare object sources when the first lookup has failed. */
-	OBJECT_INFO_QUICK = (1 << 1),
-
-	/*
-	 * Do not attempt to fetch the object if missing (even if fetch_is_missing is
-	 * nonzero).
-	 */
-	OBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),
-
-	/* Die if object corruption (not just an object being missing) was detected. */
-	OBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),
-
-	/*
-	 * This is meant for bulk prefetching of missing blobs in a partial
-	 * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.
-	 */
-	OBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),
-};
-
 /*
  * Read object info from the object database and populate the `object_info`
  * structure. Returns 0 on success, a negative error code otherwise.
diff --git a/odb/source-files.c b/odb/source-files.c
index 20a24f524a..f2969a1214 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -41,6 +41,20 @@ static void odb_source_files_reprepare(struct odb_source *source)
 	packfile_store_reprepare(files->packed);
 }
 
+static int odb_source_files_read_object_info(struct odb_source *source,
+					     const struct object_id *oid,
+					     struct object_info *oi,
+					     enum object_info_flags flags)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+
+	if (!packfile_store_read_object_info(files->packed, oid, oi, flags) ||
+	    !odb_source_loose_read_object_info(source, oid, oi, flags))
+		return 0;
+
+	return -1;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -55,6 +69,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.free = odb_source_files_free;
 	files->base.close = odb_source_files_close;
 	files->base.reprepare = odb_source_files_reprepare;
+	files->base.read_object_info = odb_source_files_read_object_info;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 0e6c6abdb1..150becafe6 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -12,6 +12,45 @@ enum odb_source_type {
 	ODB_SOURCE_FILES,
 };
 
+/* Flags that can be passed to `odb_read_object_info_extended()`. */
+enum object_info_flags {
+	/* Invoke lookup_replace_object() on the given hash. */
+	OBJECT_INFO_LOOKUP_REPLACE = (1 << 0),
+
+	/* Do not reprepare object sources when the first lookup has failed. */
+	OBJECT_INFO_QUICK = (1 << 1),
+
+	/*
+	 * Do not attempt to fetch the object if missing (even if fetch_is_missing is
+	 * nonzero).
+	 */
+	OBJECT_INFO_SKIP_FETCH_OBJECT = (1 << 2),
+
+	/* Die if object corruption (not just an object being missing) was detected. */
+	OBJECT_INFO_DIE_IF_CORRUPT = (1 << 3),
+
+	/*
+	 * We have already tried reading the object, but it couldn't be found
+	 * via any of the attached sources, and are now doing a second read.
+	 * This second read asks the individual sources to also evaluate
+	 * whether any on-disk state may have changed that may have caused the
+	 * object to appear.
+	 *
+	 * This flag is for internal use, only. The second read only occurs
+	 * when `OBJECT_INFO_QUICK` was not passed.
+	 */
+	OBJECT_INFO_SECOND_READ = (1 << 4),
+
+	/*
+	 * This is meant for bulk prefetching of missing blobs in a partial
+	 * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK.
+	 */
+	OBJECT_INFO_FOR_PREFETCH = (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK),
+};
+
+struct object_id;
+struct object_info;
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -72,6 +111,33 @@ struct odb_source {
 	 * example just been repacked so that new objects will become visible.
 	 */
 	void (*reprepare)(struct odb_source *source);
+
+	/*
+	 * This callback is expected to read object information from the object
+	 * database source. The object info will be partially populated with
+	 * pointers for each bit of information that was requested by the
+	 * caller.
+	 *
+	 * The flags field is a combination of `OBJECT_INFO` flags. Only the
+	 * following fields need to be handled by the backend:
+	 *
+	 *   - `OBJECT_INFO_QUICK` indicates it is fine to use caches without
+	 *     re-verifying the data.
+	 *
+	 *   - `OBJECT_INFO_SECOND_READ` indicates that the initial object
+	 *     lookup has failed and that the object sources should check
+	 *     whether any of its on-disk state has changed that may have
+	 *     caused the object to appear. Sources are free to ignore the
+	 *     second read in case they know that the first read would have
+	 *     already surfaced the object without reloading any on-disk state.
+	 *
+	 * The callback is expected to return a negative error code in case
+	 * reading the object has failed, 0 otherwise.
+	 */
+	int (*read_object_info)(struct odb_source *source,
+				const struct object_id *oid,
+				struct object_info *oi,
+				enum object_info_flags flags);
 };
 
 /*
@@ -131,4 +197,16 @@ static inline void odb_source_reprepare(struct odb_source *source)
 	source->reprepare(source);
 }
 
+/*
+ * Read an object from the object database source identified by its object ID.
+ * Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_read_object_info(struct odb_source *source,
+					      const struct object_id *oid,
+					      struct object_info *oi,
+					      enum object_info_flags flags)
+{
+	return source->read_object_info(source, oid, oi, flags);
+}
+
 #endif
diff --git a/packfile.c b/packfile.c
index da1c0dfa39..71db10e7c6 100644
--- a/packfile.c
+++ b/packfile.c
@@ -2181,11 +2181,19 @@ int packfile_store_freshen_object(struct packfile_store *store,
 int packfile_store_read_object_info(struct packfile_store *store,
 				    const struct object_id *oid,
 				    struct object_info *oi,
-				    enum object_info_flags flags UNUSED)
+				    enum object_info_flags flags)
 {
 	struct pack_entry e;
 	int ret;
 
+	/*
+	 * In case the first read didn't surface the object, we have to reload
+	 * packfiles. This may cause us to discover new packfiles that have
+	 * been added since the last time we have prepared the packfile store.
+	 */
+	if (flags & OBJECT_INFO_SECOND_READ)
+		packfile_store_reprepare(store);
+
 	if (!find_pack_entry(store, oid, &e))
 		return 1;
 

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 10/17] odb/source: make `read_object_stream()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 12 ++++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 odb/streaming.c    |  9 ++-------
 3 files changed, 37 insertions(+), 7 deletions(-)

diff --git a/odb/source-files.c b/odb/source-files.c
index f2969a1214..b50a1f5492 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -55,6 +55,17 @@ static int odb_source_files_read_object_info(struct odb_source *source,
 	return -1;
 }
 
+static int odb_source_files_read_object_stream(struct odb_read_stream **out,
+					       struct odb_source *source,
+					       const struct object_id *oid)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	if (!packfile_store_read_object_stream(out, files->packed, oid) ||
+	    !odb_source_loose_read_object_stream(out, source, oid))
+		return 0;
+	return -1;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -70,6 +81,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.close = odb_source_files_close;
 	files->base.reprepare = odb_source_files_reprepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
+	files->base.read_object_stream = odb_source_files_read_object_stream;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 150becafe6..4397cada27 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -50,6 +50,7 @@ enum object_info_flags {
 
 struct object_id;
 struct object_info;
+struct odb_read_stream;
 
 /*
  * The source is the part of the object database that stores the actual
@@ -138,6 +139,17 @@ struct odb_source {
 				const struct object_id *oid,
 				struct object_info *oi,
 				enum object_info_flags flags);
+
+	/*
+	 * This callback is expected to create a new read stream that can be
+	 * used to stream the object identified by the given ID.
+	 *
+	 * The callback is expected to return a negative error code in case
+	 * creating the object stream has failed, 0 otherwise.
+	 */
+	int (*read_object_stream)(struct odb_read_stream **out,
+				  struct odb_source *source,
+				  const struct object_id *oid);
 };
 
 /*
@@ -209,4 +221,15 @@ static inline int odb_source_read_object_info(struct odb_source *source,
 	return source->read_object_info(source, oid, oi, flags);
 }
 
+/*
+ * Create a new read stream for the given object ID. Returns 0 on success, a
+ * negative error code otherwise.
+ */
+static inline int odb_source_read_object_stream(struct odb_read_stream **out,
+						struct odb_source *source,
+						const struct object_id *oid)
+{
+	return source->read_object_stream(out, source, oid);
+}
+
 #endif
diff --git a/odb/streaming.c b/odb/streaming.c
index 19cda9407d..a4355cd245 100644
--- a/odb/streaming.c
+++ b/odb/streaming.c
@@ -6,11 +6,9 @@
 #include "convert.h"
 #include "environment.h"
 #include "repository.h"
-#include "object-file.h"
 #include "odb.h"
 #include "odb/streaming.h"
 #include "replace-object.h"
-#include "packfile.h"
 
 #define FILTER_BUFFER (1024*16)
 
@@ -186,12 +184,9 @@ static int istream_source(struct odb_read_stream **out,
 	struct odb_source *source;
 
 	odb_prepare_alternates(odb);
-	for (source = odb->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-		if (!packfile_store_read_object_stream(out, files->packed, oid) ||
-		    !odb_source_loose_read_object_stream(out, source, oid))
+	for (source = odb->sources; source; source = source->next)
+		if (!odb_source_read_object_stream(out, source, oid))
 			return 0;
-	}
 
 	return open_istream_incore(out, odb, oid);
 }

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 11/17] odb/source: make `for_each_object()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 12 +---------
 odb.h              | 12 ----------
 odb/source-files.c | 23 +++++++++++++++++++
 odb/source.h       | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 89 insertions(+), 23 deletions(-)

diff --git a/odb.c b/odb.c
index c0b8cd062b..494a3273cf 100644
--- a/odb.c
+++ b/odb.c
@@ -984,20 +984,10 @@ int odb_for_each_object(struct object_database *odb,
 
 	odb_prepare_alternates(odb);
 	for (struct odb_source *source = odb->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-
 		if (flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY && !source->local)
 			continue;
 
-		if (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {
-			ret = odb_source_loose_for_each_object(source, request,
-							       cb, cb_data, flags);
-			if (ret)
-				return ret;
-		}
-
-		ret = packfile_store_for_each_object(files->packed, request,
-						     cb, cb_data, flags);
+		ret = odb_source_for_each_object(source, request, cb, cb_data, flags);
 		if (ret)
 			return ret;
 	}
diff --git a/odb.h b/odb.h
index 70ffb033f9..692d9029ef 100644
--- a/odb.h
+++ b/odb.h
@@ -432,18 +432,6 @@ enum odb_for_each_object_flags {
 	ODB_FOR_EACH_OBJECT_SKIP_ON_DISK_KEPT_PACKS = (1<<4),
 };
 
-/*
- * A callback function that can be used to iterate through objects. If given,
- * the optional `oi` parameter will be populated the same as if you would call
- * `odb_read_object_info()`.
- *
- * Returning a non-zero error code will cause iteration to abort. The error
- * code will be propagated.
- */
-typedef int (*odb_for_each_object_cb)(const struct object_id *oid,
-				      struct object_info *oi,
-				      void *cb_data);
-
 /*
  * Iterate through all objects contained in the object database. Note that
  * objects may be iterated over multiple times in case they are either stored
diff --git a/odb/source-files.c b/odb/source-files.c
index b50a1f5492..d8ef1d8237 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -66,6 +66,28 @@ static int odb_source_files_read_object_stream(struct odb_read_stream **out,
 	return -1;
 }
 
+static int odb_source_files_for_each_object(struct odb_source *source,
+					    const struct object_info *request,
+					    odb_for_each_object_cb cb,
+					    void *cb_data,
+					    unsigned flags)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	int ret;
+
+	if (!(flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {
+		ret = odb_source_loose_for_each_object(source, request, cb, cb_data, flags);
+		if (ret)
+			return ret;
+	}
+
+	ret = packfile_store_for_each_object(files->packed, request, cb, cb_data, flags);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -82,6 +104,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.reprepare = odb_source_files_reprepare;
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
+	files->base.for_each_object = odb_source_files_for_each_object;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 4397cada27..be56995389 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -52,6 +52,18 @@ struct object_id;
 struct object_info;
 struct odb_read_stream;
 
+/*
+ * A callback function that can be used to iterate through objects. If given,
+ * the optional `oi` parameter will be populated the same as if you would call
+ * `odb_read_object_info()`.
+ *
+ * Returning a non-zero error code will cause iteration to abort. The error
+ * code will be propagated.
+ */
+typedef int (*odb_for_each_object_cb)(const struct object_id *oid,
+				      struct object_info *oi,
+				      void *cb_data);
+
 /*
  * The source is the part of the object database that stores the actual
  * objects. It thus encapsulates the logic to read and write the specific
@@ -150,6 +162,30 @@ struct odb_source {
 	int (*read_object_stream)(struct odb_read_stream **out,
 				  struct odb_source *source,
 				  const struct object_id *oid);
+
+	/*
+	 * This callback is expected to iterate over all objects stored in this
+	 * source and invoke the callback function for each of them. It is
+	 * valid to yield the same object multiple time. A non-zero exit code
+	 * from the object callback shall abort iteration.
+	 *
+	 * The optional `request` structure should serve as a template for
+	 * looking up object info for every individual iterated object. It
+	 * should not be modified directly and should instead be copied into a
+	 * separate `struct object_info` that gets passed to the callback. If
+	 * the caller passes a `NULL` pointer then the object itself shall not
+	 * be read.
+	 *
+	 * The callback is expected to return a negative error code in case the
+	 * iteration has failed to read all objects, 0 otherwise. When the
+	 * callback function returns a non-zero error code then that error code
+	 * should be returned.
+	 */
+	int (*for_each_object)(struct odb_source *source,
+			       const struct object_info *request,
+			       odb_for_each_object_cb cb,
+			       void *cb_data,
+			       unsigned flags);
 };
 
 /*
@@ -232,4 +268,33 @@ static inline int odb_source_read_object_stream(struct odb_read_stream **out,
 	return source->read_object_stream(out, source, oid);
 }
 
+/*
+ * Iterate through all objects contained in the given source and invoke the
+ * callback function for each of them. Returning a non-zero code from the
+ * callback function aborts iteration. There is no guarantee that objects
+ * are only iterated over once.
+ *
+ * The optional `request` structure serves as a template for retrieving the
+ * object info for each indvidual iterated object and will be populated as if
+ * `odb_source_read_object_info()` was called on the object. It will not be
+ * modified, the callback will instead be invoked with a separate `struct
+ * object_info` for every object. Object info will not be read when passing a
+ * `NULL` pointer.
+ *
+ * The flags is a bitfield of `ODB_FOR_EACH_OBJECT_*` flags. Not all flags may
+ * apply to a specific backend, so whether or not they are honored is defined
+ * by the implementation.
+ *
+ * Returns 0 when all objects have been iterated over, a negative error code in
+ * case iteration has failed, or a non-zero value returned from the callback.
+ */
+static inline int odb_source_for_each_object(struct odb_source *source,
+					     const struct object_info *request,
+					     odb_for_each_object_cb cb,
+					     void *cb_data,
+					     unsigned flags)
+{
+	return source->for_each_object(source, request, cb, cb_data, flags);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 12/17] odb/source: make `freshen_object()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 12 ++----------
 odb/source-files.c | 11 +++++++++++
 odb/source.h       | 23 +++++++++++++++++++++++
 3 files changed, 36 insertions(+), 10 deletions(-)

diff --git a/odb.c b/odb.c
index 494a3273cf..c9f42c5afd 100644
--- a/odb.c
+++ b/odb.c
@@ -959,18 +959,10 @@ int odb_freshen_object(struct object_database *odb,
 		       const struct object_id *oid)
 {
 	struct odb_source *source;
-
 	odb_prepare_alternates(odb);
-	for (source = odb->sources; source; source = source->next) {
-		struct odb_source_files *files = odb_source_files_downcast(source);
-
-		if (packfile_store_freshen_object(files->packed, oid))
+	for (source = odb->sources; source; source = source->next)
+		if (odb_source_freshen_object(source, oid))
 			return 1;
-
-		if (odb_source_loose_freshen_object(source, oid))
-			return 1;
-	}
-
 	return 0;
 }
 
diff --git a/odb/source-files.c b/odb/source-files.c
index d8ef1d8237..a6447909e0 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -88,6 +88,16 @@ static int odb_source_files_for_each_object(struct odb_source *source,
 	return 0;
 }
 
+static int odb_source_files_freshen_object(struct odb_source *source,
+					   const struct object_id *oid)
+{
+	struct odb_source_files *files = odb_source_files_downcast(source);
+	if (packfile_store_freshen_object(files->packed, oid) ||
+	    odb_source_loose_freshen_object(source, oid))
+		return 1;
+	return 0;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -105,6 +115,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.read_object_info = odb_source_files_read_object_info;
 	files->base.read_object_stream = odb_source_files_read_object_stream;
 	files->base.for_each_object = odb_source_files_for_each_object;
+	files->base.freshen_object = odb_source_files_freshen_object;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index be56995389..7f2ecf420b 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -186,6 +186,18 @@ struct odb_source {
 			       odb_for_each_object_cb cb,
 			       void *cb_data,
 			       unsigned flags);
+
+	/*
+	 * This callback is expected to freshen the given object so that its
+	 * last access time is set to the current time. This is used to ensure
+	 * that objects that are recent will not get garbage collected even if
+	 * they were unreachable.
+	 *
+	 * Returns 0 in case the object does not exist, 1 in case the object
+	 * has been freshened.
+	 */
+	int (*freshen_object)(struct odb_source *source,
+			      const struct object_id *oid);
 };
 
 /*
@@ -297,4 +309,15 @@ static inline int odb_source_for_each_object(struct odb_source *source,
 	return source->for_each_object(source, request, cb, cb_data, flags);
 }
 
+/*
+ * Freshen an object in the object database by updating its timestamp.
+ * Returns 1 in case the object has been freshened, 0 in case the object does
+ * not exist.
+ */
+static inline int odb_source_freshen_object(struct odb_source *source,
+					    const struct object_id *oid)
+{
+	return source->freshen_object(source, oid);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 13/17] odb/source: make `write_object()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  4 ++--
 odb/source-files.c | 12 ++++++++++++
 odb/source.h       | 36 ++++++++++++++++++++++++++++++++++++
 3 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/odb.c b/odb.c
index c9f42c5afd..5eb60063dc 100644
--- a/odb.c
+++ b/odb.c
@@ -1005,8 +1005,8 @@ int odb_write_object_ext(struct object_database *odb,
 			 struct object_id *compat_oid,
 			 unsigned flags)
 {
-	return odb_source_loose_write_object(odb->sources, buf, len, type,
-					     oid, compat_oid, flags);
+	return odb_source_write_object(odb->sources, buf, len, type,
+				       oid, compat_oid, flags);
 }
 
 int odb_write_object_stream(struct object_database *odb,
diff --git a/odb/source-files.c b/odb/source-files.c
index a6447909e0..67c2aff659 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -98,6 +98,17 @@ static int odb_source_files_freshen_object(struct odb_source *source,
 	return 0;
 }
 
+static int odb_source_files_write_object(struct odb_source *source,
+					 const void *buf, unsigned long len,
+					 enum object_type type,
+					 struct object_id *oid,
+					 struct object_id *compat_oid,
+					 unsigned flags)
+{
+	return odb_source_loose_write_object(source, buf, len, type,
+					     oid, compat_oid, flags);
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -116,6 +127,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.read_object_stream = odb_source_files_read_object_stream;
 	files->base.for_each_object = odb_source_files_for_each_object;
 	files->base.freshen_object = odb_source_files_freshen_object;
+	files->base.write_object = odb_source_files_write_object;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 7f2ecf420b..c959e962f6 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -1,6 +1,8 @@
 #ifndef ODB_SOURCE_H
 #define ODB_SOURCE_H
 
+#include "object.h"
+
 enum odb_source_type {
 	/*
 	 * The "unknown" type, which should never be in use. This type mostly
@@ -198,6 +200,24 @@ struct odb_source {
 	 */
 	int (*freshen_object)(struct odb_source *source,
 			      const struct object_id *oid);
+
+	/*
+	 * This callback is expected to persist the given object into the
+	 * object source. In case the object already exists it shall be
+	 * freshened.
+	 *
+	 * The flags field is a combination of `WRITE_OBJECT` flags.
+	 *
+	 * The resulting object ID (and optionally the compatibility object ID)
+	 * shall be written into the out pointers. The callback is expected to
+	 * return 0 on success, a negative error code otherwise.
+	 */
+	int (*write_object)(struct odb_source *source,
+			    const void *buf, unsigned long len,
+			    enum object_type type,
+			    struct object_id *oid,
+			    struct object_id *compat_oid,
+			    unsigned flags);
 };
 
 /*
@@ -320,4 +340,20 @@ static inline int odb_source_freshen_object(struct odb_source *source,
 	return source->freshen_object(source, oid);
 }
 
+/*
+ * Write an object into the object database source. Returns 0 on success, a
+ * negative error code otherwise. Populates the given out pointers for the
+ * object ID and the compatibility object ID, if non-NULL.
+ */
+static inline int odb_source_write_object(struct odb_source *source,
+					  const void *buf, unsigned long len,
+					  enum object_type type,
+					  struct object_id *oid,
+					  struct object_id *compat_oid,
+					  unsigned flags)
+{
+	return source->write_object(source, buf, len, type, oid,
+				    compat_oid, flags);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 14/17] odb/source: make `write_object_stream()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              |  2 +-
 odb/source-files.c |  9 +++++++++
 odb/source.h       | 28 ++++++++++++++++++++++++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/odb.c b/odb.c
index 5eb60063dc..f439de9db2 100644
--- a/odb.c
+++ b/odb.c
@@ -1013,7 +1013,7 @@ int odb_write_object_stream(struct object_database *odb,
 			    struct odb_write_stream *stream, size_t len,
 			    struct object_id *oid)
 {
-	return odb_source_loose_write_stream(odb->sources, stream, len, oid);
+	return odb_source_write_object_stream(odb->sources, stream, len, oid);
 }
 
 struct object_database *odb_new(struct repository *repo,
diff --git a/odb/source-files.c b/odb/source-files.c
index 67c2aff659..b8844f11b7 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -109,6 +109,14 @@ static int odb_source_files_write_object(struct odb_source *source,
 					     oid, compat_oid, flags);
 }
 
+static int odb_source_files_write_object_stream(struct odb_source *source,
+						struct odb_write_stream *stream,
+						size_t len,
+						struct object_id *oid)
+{
+	return odb_source_loose_write_stream(source, stream, len, oid);
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -128,6 +136,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.for_each_object = odb_source_files_for_each_object;
 	files->base.freshen_object = odb_source_files_freshen_object;
 	files->base.write_object = odb_source_files_write_object;
+	files->base.write_object_stream = odb_source_files_write_object_stream;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index c959e962f6..6c8bec1912 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -53,6 +53,7 @@ enum object_info_flags {
 struct object_id;
 struct object_info;
 struct odb_read_stream;
+struct odb_write_stream;
 
 /*
  * A callback function that can be used to iterate through objects. If given,
@@ -218,6 +219,18 @@ struct odb_source {
 			    struct object_id *oid,
 			    struct object_id *compat_oid,
 			    unsigned flags);
+
+	/*
+	 * This callback is expected to persist the given object stream into
+	 * the object source.
+	 *
+	 * The resulting object ID shall be written into the out pointer. The
+	 * callback is expected to return 0 on success, a negative error code
+	 * otherwise.
+	 */
+	int (*write_object_stream)(struct odb_source *source,
+				   struct odb_write_stream *stream, size_t len,
+				   struct object_id *oid);
 };
 
 /*
@@ -356,4 +369,19 @@ static inline int odb_source_write_object(struct odb_source *source,
 				    compat_oid, flags);
 }
 
+/*
+ * Write an object into the object database source via a stream. The overall
+ * length of the object must be known in advance.
+ *
+ * Return 0 on success, a negative error code otherwise. Populates the given
+ * out pointer for the object ID.
+ */
+static inline int odb_source_write_object_stream(struct odb_source *source,
+						 struct odb_write_stream *stream,
+						 size_t len,
+						 struct object_id *oid)
+{
+	return source->write_object_stream(source, stream, len, oid);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 15/17] odb/source: make `read_alternates()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 26 ++++----------------------
 odb.h              |  5 +++++
 odb/source-files.c | 22 ++++++++++++++++++++++
 odb/source.h       | 28 ++++++++++++++++++++++++++++
 4 files changed, 59 insertions(+), 22 deletions(-)

diff --git a/odb.c b/odb.c
index f439de9db2..d9424cdfd0 100644
--- a/odb.c
+++ b/odb.c
@@ -131,10 +131,10 @@ static bool odb_is_source_usable(struct object_database *o, const char *path)
 	return usable;
 }
 
-static void parse_alternates(const char *string,
-			     int sep,
-			     const char *relative_base,
-			     struct strvec *out)
+void parse_alternates(const char *string,
+		      int sep,
+		      const char *relative_base,
+		      struct strvec *out)
 {
 	struct strbuf pathbuf = STRBUF_INIT;
 	struct strbuf buf = STRBUF_INIT;
@@ -198,24 +198,6 @@ static void parse_alternates(const char *string,
 	strbuf_release(&buf);
 }
 
-static void odb_source_read_alternates(struct odb_source *source,
-				       struct strvec *out)
-{
-	struct strbuf buf = STRBUF_INIT;
-	char *path;
-
-	path = xstrfmt("%s/info/alternates", source->path);
-	if (strbuf_read_file(&buf, path, 1024) < 0) {
-		warn_on_fopen_errors(path);
-		free(path);
-		return;
-	}
-	parse_alternates(buf.buf, '\n', source->path, out);
-
-	strbuf_release(&buf);
-	free(path);
-}
-
 static struct odb_source *odb_add_alternate_recursively(struct object_database *odb,
 							const char *source,
 							int depth)
diff --git a/odb.h b/odb.h
index 692d9029ef..86e0365c24 100644
--- a/odb.h
+++ b/odb.h
@@ -500,4 +500,9 @@ int odb_write_object_stream(struct object_database *odb,
 			    struct odb_write_stream *stream, size_t len,
 			    struct object_id *oid);
 
+void parse_alternates(const char *string,
+		      int sep,
+		      const char *relative_base,
+		      struct strvec *out);
+
 #endif /* ODB_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index b8844f11b7..199c55cfa4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -2,9 +2,11 @@
 #include "abspath.h"
 #include "chdir-notify.h"
 #include "object-file.h"
+#include "odb.h"
 #include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
+#include "strbuf.h"
 
 static void odb_source_files_reparent(const char *name UNUSED,
 				      const char *old_cwd,
@@ -117,6 +119,25 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 	return odb_source_loose_write_stream(source, stream, len, oid);
 }
 
+static int odb_source_files_read_alternates(struct odb_source *source,
+					    struct strvec *out)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *path;
+
+	path = xstrfmt("%s/info/alternates", source->path);
+	if (strbuf_read_file(&buf, path, 1024) < 0) {
+		warn_on_fopen_errors(path);
+		free(path);
+		return 0;
+	}
+	parse_alternates(buf.buf, '\n', source->path, out);
+
+	strbuf_release(&buf);
+	free(path);
+	return 0;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -137,6 +158,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.freshen_object = odb_source_files_freshen_object;
 	files->base.write_object = odb_source_files_write_object;
 	files->base.write_object_stream = odb_source_files_write_object_stream;
+	files->base.read_alternates = odb_source_files_read_alternates;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index 6c8bec1912..fbdddcb2eb 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -54,6 +54,7 @@ struct object_id;
 struct object_info;
 struct odb_read_stream;
 struct odb_write_stream;
+struct strvec;
 
 /*
  * A callback function that can be used to iterate through objects. If given,
@@ -231,6 +232,19 @@ struct odb_source {
 	int (*write_object_stream)(struct odb_source *source,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to read the list of alternate object
+	 * database sources connected to it and write them into the `strvec`.
+	 *
+	 * The result is expected to be paths to the alternates. All paths must
+	 * be resolved to absolute paths.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 */
+	int (*read_alternates)(struct odb_source *source,
+			       struct strvec *out);
 };
 
 /*
@@ -384,4 +398,18 @@ static inline int odb_source_write_object_stream(struct odb_source *source,
 	return source->write_object_stream(source, stream, len, oid);
 }
 
+/*
+ * Read the list of alternative object database sources from the given backend
+ * and populate the `strvec` with them. The listing is not recursive -- that
+ * is, if any of the yielded alternate sources has alternates itself, those
+ * will not be yielded as part of this function call.
+ *
+ * Return 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_read_alternates(struct odb_source *source,
+					     struct strvec *out)
+{
+	return source->read_alternates(source, out);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 16/17] odb/source: make `write_alternate()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb.c              | 52 --------------------------------------------------
 odb/source-files.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 odb/source.h       | 26 +++++++++++++++++++++++++
 3 files changed, 82 insertions(+), 52 deletions(-)

diff --git a/odb.c b/odb.c
index d9424cdfd0..84a31084d3 100644
--- a/odb.c
+++ b/odb.c
@@ -236,58 +236,6 @@ static struct odb_source *odb_add_alternate_recursively(struct object_database *
 	return alternate;
 }
 
-static int odb_source_write_alternate(struct odb_source *source,
-				      const char *alternate)
-{
-	struct lock_file lock = LOCK_INIT;
-	char *path = xstrfmt("%s/%s", source->path, "info/alternates");
-	FILE *in, *out;
-	int found = 0;
-	int ret;
-
-	hold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);
-	out = fdopen_lock_file(&lock, "w");
-	if (!out) {
-		ret = error_errno(_("unable to fdopen alternates lockfile"));
-		goto out;
-	}
-
-	in = fopen(path, "r");
-	if (in) {
-		struct strbuf line = STRBUF_INIT;
-
-		while (strbuf_getline(&line, in) != EOF) {
-			if (!strcmp(alternate, line.buf)) {
-				found = 1;
-				break;
-			}
-			fprintf_or_die(out, "%s\n", line.buf);
-		}
-
-		strbuf_release(&line);
-		fclose(in);
-	} else if (errno != ENOENT) {
-		ret = error_errno(_("unable to read alternates file"));
-		goto out;
-	}
-
-	if (found) {
-		rollback_lock_file(&lock);
-	} else {
-		fprintf_or_die(out, "%s\n", alternate);
-		if (commit_lock_file(&lock)) {
-			ret = error_errno(_("unable to move new alternates file into place"));
-			goto out;
-		}
-	}
-
-	ret = 0;
-
-out:
-	free(path);
-	return ret;
-}
-
 void odb_add_to_alternates_file(struct object_database *odb,
 				const char *dir)
 {
diff --git a/odb/source-files.c b/odb/source-files.c
index 199c55cfa4..c32cd67b26 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -1,12 +1,15 @@
 #include "git-compat-util.h"
 #include "abspath.h"
 #include "chdir-notify.h"
+#include "gettext.h"
+#include "lockfile.h"
 #include "object-file.h"
 #include "odb.h"
 #include "odb/source.h"
 #include "odb/source-files.h"
 #include "packfile.h"
 #include "strbuf.h"
+#include "write-or-die.h"
 
 static void odb_source_files_reparent(const char *name UNUSED,
 				      const char *old_cwd,
@@ -138,6 +141,58 @@ static int odb_source_files_read_alternates(struct odb_source *source,
 	return 0;
 }
 
+static int odb_source_files_write_alternate(struct odb_source *source,
+					    const char *alternate)
+{
+	struct lock_file lock = LOCK_INIT;
+	char *path = xstrfmt("%s/%s", source->path, "info/alternates");
+	FILE *in, *out;
+	int found = 0;
+	int ret;
+
+	hold_lock_file_for_update(&lock, path, LOCK_DIE_ON_ERROR);
+	out = fdopen_lock_file(&lock, "w");
+	if (!out) {
+		ret = error_errno(_("unable to fdopen alternates lockfile"));
+		goto out;
+	}
+
+	in = fopen(path, "r");
+	if (in) {
+		struct strbuf line = STRBUF_INIT;
+
+		while (strbuf_getline(&line, in) != EOF) {
+			if (!strcmp(alternate, line.buf)) {
+				found = 1;
+				break;
+			}
+			fprintf_or_die(out, "%s\n", line.buf);
+		}
+
+		strbuf_release(&line);
+		fclose(in);
+	} else if (errno != ENOENT) {
+		ret = error_errno(_("unable to read alternates file"));
+		goto out;
+	}
+
+	if (found) {
+		rollback_lock_file(&lock);
+	} else {
+		fprintf_or_die(out, "%s\n", alternate);
+		if (commit_lock_file(&lock)) {
+			ret = error_errno(_("unable to move new alternates file into place"));
+			goto out;
+		}
+	}
+
+	ret = 0;
+
+out:
+	free(path);
+	return ret;
+}
+
 struct odb_source_files *odb_source_files_new(struct object_database *odb,
 					      const char *path,
 					      bool local)
@@ -159,6 +214,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.write_object = odb_source_files_write_object;
 	files->base.write_object_stream = odb_source_files_write_object_stream;
 	files->base.read_alternates = odb_source_files_read_alternates;
+	files->base.write_alternate = odb_source_files_write_alternate;
 
 	/*
 	 * Ideally, we would only ever store absolute paths in the source. This
diff --git a/odb/source.h b/odb/source.h
index fbdddcb2eb..ee540630d2 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -245,6 +245,19 @@ struct odb_source {
 	 */
 	int (*read_alternates)(struct odb_source *source,
 			       struct strvec *out);
+
+	/*
+	 * This callback is expected to persist the singular alternate passed
+	 * to it into its list of alternates. Any pre-existing alternates are
+	 * expected to remain active. Subsequent calls to `read_alternates` are
+	 * thus expected to yield the pre-existing list of alternates plus the
+	 * newly added alternate appended to its end.
+	 *
+	 * The callback is expected to return 0 on success, a negative error
+	 * code otherwise.
+	 */
+	int (*write_alternate)(struct odb_source *source,
+			       const char *alternate);
 };
 
 /*
@@ -412,4 +425,17 @@ static inline int odb_source_read_alternates(struct odb_source *source,
 	return source->read_alternates(source, out);
 }
 
+/*
+ * Write and persist a new alternate object database source for the given
+ * source. Any preexisting alternates are expected to stay valid, and the new
+ * alternate shall be appended to the end of the list.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_write_alternate(struct odb_source *source,
+					      const char *alternate)
+{
+	return source->write_alternate(source, alternate);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

[PATCH v2 17/17] odb/source: make `begin_transaction()` function pluggable

[Patrick Steinhardt]

Introduce a new callback function in `struct odb_source` to make the
function pluggable.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 odb/source-files.c | 11 +++++++++++
 odb/source.h       | 27 +++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/odb/source-files.c b/odb/source-files.c
index c32cd67b26..14cb9adeca 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -122,6 +122,16 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 	return odb_source_loose_write_stream(source, stream, len, oid);
 }
 
+static int odb_source_files_begin_transaction(struct odb_source *source,
+					      struct odb_transaction **out)
+{
+	struct odb_transaction *tx = odb_transaction_files_begin(source);
+	if (!tx)
+		return -1;
+	*out = tx;
+	return 0;
+}
+
 static int odb_source_files_read_alternates(struct odb_source *source,
 					    struct strvec *out)
 {
@@ -213,6 +223,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
 	files->base.freshen_object = odb_source_files_freshen_object;
 	files->base.write_object = odb_source_files_write_object;
 	files->base.write_object_stream = odb_source_files_write_object_stream;
+	files->base.begin_transaction = odb_source_files_begin_transaction;
 	files->base.read_alternates = odb_source_files_read_alternates;
 	files->base.write_alternate = odb_source_files_write_alternate;
 
diff --git a/odb/source.h b/odb/source.h
index ee540630d2..caac558149 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -53,6 +53,7 @@ enum object_info_flags {
 struct object_id;
 struct object_info;
 struct odb_read_stream;
+struct odb_transaction;
 struct odb_write_stream;
 struct strvec;
 
@@ -233,6 +234,19 @@ struct odb_source {
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
 
+	/*
+	 * This callback is expected to create a new transaction that can be
+	 * used to write objects to. The objects shall only be persisted into
+	 * the object database when the transcation's commit function is
+	 * called. Otherwise, the objects shall be discarded.
+	 *
+	 * Returns 0 on success, in which case the `*out` pointer will have
+	 * been populated with the object database transaction. Returns a
+	 * negative error code otherwise.
+	 */
+	int (*begin_transaction)(struct odb_source *source,
+				 struct odb_transaction **out);
+
 	/*
 	 * This callback is expected to read the list of alternate object
 	 * database sources connected to it and write them into the `strvec`.
@@ -438,4 +452,17 @@ static inline int odb_source_write_alternate(struct odb_source *source,
 	return source->write_alternate(source, alternate);
 }
 
+/*
+ * Create a new transaction that can be used to write objects into a temporary
+ * staging area. The objects will only be persisted when the transaction is
+ * committed.
+ *
+ * Returns 0 on success, a negative error code otherwise.
+ */
+static inline int odb_source_begin_transaction(struct odb_source *source,
+					       struct odb_transaction **out)
+{
+	return source->begin_transaction(source, out);
+}
+
 #endif

-- 
2.53.0.797.g7842e34a66.dirty

Re: [PATCH v2 00/17] odb: make object database sources pluggable

[Justin Tobler]

On 26/03/05 03:19PM, Patrick Steinhardt wrote:
> Changes in v2:
>   - Fix mismerge in the base of this patch series.
>   - Adjust several comments and improve commit messages a bit.
>   - Link to v1: https://lore.kernel.org/r/20260223-b4-pks-odb-source-pluggable-v1-0-253bac1db598@pks.im

The changes in this version addressed my previous comments. This version
looks good to me. Thanks.

-Justin

Re: [PATCH v2 00/17] odb: make object database sources pluggable

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> To set expectations: this is only a start, there is still functionality
> missing that needs to be made pluggable. Most importantly:
>
>   - Counting of objects.
>
>   - Abbreviating object IDs and finding ambiguous objects.
>
>   - Consistency checks.
>
>   - Optimizing the object database.
>
>   - Generating packfiles.
>
> These will all happen in later patch series. That being said, with this
> patch series one already gets a lot of the basic functionality, and it's
> almost possible to do local workflows. Only "almost" though because we
> rely on abbreviating object IDs in a lot of places, but once that part
> is implemented in a subsequent patch series you can indeed work locally
> with an alternate backend.

I've been looking over this series, and the transition to a pluggable
interface for ODB sources is very clean and follows the patterns we've
established for refs and streams quite well.

One thing I am puzzled on the design, specifically starting with
patch 09 and onward, is the lack of documentation regarding which of
the new callbacks in `struct odb_source` are mandatory and which are
optional.

In `odb/source.h`, the static inline wrapper functions dereference the
backend's function pointers directly. For example:

+static inline int odb_source_read_object_info(struct odb_source *source,
+					      const struct object_id *oid,
+					      struct object_info *oi,
+					      enum object_info_flags flags)
+{
+	return source->read_object_info(source, oid, oi, flags);
+}

If a future backend (say, a read-only network proxy) doesn't implement
some of the write-related functions or the iteration functions, the
current wrappers will cause a segmentation fault.

Do we want to

  - Document in `struct odb_source` which callbacks must be implemented
    by every backend.

  - Have the wrapper functions check for NULL. If a mandatory function
    is missing, a `BUG()` would be appropriate. If it's truly optional,
    the wrapper could return a suitable error code (like -1 or
    `GIT_ENOTSUP`).

Given that the "files" backend implements the full set, it's easy to
miss, but as we add more specialized backends, a clearly defined
interface contract may become important.

What are your thoughts on which of these should be considered the
"minimal viable" set for an ODB source?

Re: [PATCH v2 00/17] odb: make object database sources pluggable

[Patrick Steinhardt]

On Thu, Mar 05, 2026 at 12:42:19PM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > To set expectations: this is only a start, there is still functionality
> > missing that needs to be made pluggable. Most importantly:
> >
> >   - Counting of objects.
> >
> >   - Abbreviating object IDs and finding ambiguous objects.
> >
> >   - Consistency checks.
> >
> >   - Optimizing the object database.
> >
> >   - Generating packfiles.
> >
> > These will all happen in later patch series. That being said, with this
> > patch series one already gets a lot of the basic functionality, and it's
> > almost possible to do local workflows. Only "almost" though because we
> > rely on abbreviating object IDs in a lot of places, but once that part
> > is implemented in a subsequent patch series you can indeed work locally
> > with an alternate backend.
> 
> I've been looking over this series, and the transition to a pluggable
> interface for ODB sources is very clean and follows the patterns we've
> established for refs and streams quite well.
> 
> One thing I am puzzled on the design, specifically starting with
> patch 09 and onward, is the lack of documentation regarding which of
> the new callbacks in `struct odb_source` are mandatory and which are
> optional.

That's mostly explained by the fact that all of them are mandatory for
now. :)

> In `odb/source.h`, the static inline wrapper functions dereference the
> backend's function pointers directly. For example:
> 
> +static inline int odb_source_read_object_info(struct odb_source *source,
> +					      const struct object_id *oid,
> +					      struct object_info *oi,
> +					      enum object_info_flags flags)
> +{
> +	return source->read_object_info(source, oid, oi, flags);
> +}
> 
> If a future backend (say, a read-only network proxy) doesn't implement
> some of the write-related functions or the iteration functions, the
> current wrappers will cause a segmentation fault.
> 
> Do we want to
> 
>   - Document in `struct odb_source` which callbacks must be implemented
>     by every backend.
> 
>   - Have the wrapper functions check for NULL. If a mandatory function
>     is missing, a `BUG()` would be appropriate. If it's truly optional,
>     the wrapper could return a suitable error code (like -1 or
>     `GIT_ENOTSUP`).
> 
> Given that the "files" backend implements the full set, it's easy to
> miss, but as we add more specialized backends, a clearly defined
> interface contract may become important.
> 
> What are your thoughts on which of these should be considered the
> "minimal viable" set for an ODB source?

I guess this'll become more interesting once we have additional ODB
sources -- and that'll likely happen sooner rather than later. I've got
a couple of patch series pending that'll convert our existing sources
that we've already got into "proper" sources. And with those it may make
sense to document this better.

I see that this series already got merged to "next", but I'll keep it in
mind going forward that we'll want to eventually do this.

Thanks!

Patrick