* [PATCH 00/18] odb: make loose object source a proper `struct odb_source`
@ 2026-05-21 8:22 Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 01/18] odb/source-loose: move loose source into "odb/" subsystem Patrick Steinhardt
` (17 more replies)
0 siblings, 18 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Hi,
this patch series converts the loose object source into a proper `struct
odb_source` so that it can be used via our generic interfaces.
The patch series is relatively straight-forward, as the source basically
already exists as such and the interfaces already match. So for most of
the part we are just moving around some code and converting functions
that were previously called directly into callbacks.
I guess the only part that needs some attention is that there is some
confusion at first with the `struct odb_source_loose::source` parent
pointer that initially points at the owning `struct odb_source_files`.
This relationship doesn't make much sense, as a loose source can totally
exist standalone without the files source.
We're thus getting rid of this relationship in this series, too. I found
it quite hard to reason about which pointer one is holding at any point
in time though, doubly so because the parent pointer was named "source",
which is rather generic. The second commit thus renames the pointer to
`files` and converts it into `struct odb_source_files` to make the
transition cleaner, but the whole pointer will be dropped at the end of
this series.
The series is built on top of aec3f58750 (Sync with 'maint', 2026-05-21)
with ps/odb-in-memory at d2902a4549 (t/unit-tests: add tests for the
in-memory object source, 2026-04-10) merged into it.
Thanks!
Patrick
---
Patrick Steinhardt (18):
odb/source-loose: move loose source into "odb/" subsystem
odb/source-loose: store pointer to "files" instead of generic source
odb/source-loose: start converting to a proper `struct odb_source`
odb/source-loose: wire up `reprepare()` callback
odb/source-loose: wire up `close()` callback
odb/source-loose: wire up `read_object_info()` callback
odb/source-loose: wire up `read_object_stream()` callback
odb/source-loose: wire up `for_each_object()` callback
odb/source-loose: wire up `find_abbrev_len()` callback
odb/source-loose: wire up `count_objects()` callback
odb/source-loose: drop `odb_source_loose_has_object()`
odb/source-loose: wire up `freshen_object()` callback
loose: refactor object map to operate on `struct odb_source_loose`
odb/source-loose: wire up `write_object()` callback
object-file: refactor writing objects to use loose source
odb/source-loose: wire up `write_object_stream()` callback
odb/source-loose: stub out remaining callbacks
odb/source-loose: drop pointer to the "files" source
Makefile | 1 +
builtin/cat-file.c | 5 +-
builtin/gc.c | 6 +-
builtin/pack-objects.c | 12 +-
http-walker.c | 3 +-
http.c | 6 +-
loose.c | 45 ++-
loose.h | 4 +-
meson.build | 1 +
object-file.c | 796 ++++---------------------------------------------
object-file.h | 149 ++++-----
odb/source-files.c | 28 +-
odb/source-loose.c | 736 +++++++++++++++++++++++++++++++++++++++++++++
odb/source-loose.h | 48 +++
odb/source.h | 3 +
15 files changed, 973 insertions(+), 870 deletions(-)
---
base-commit: 072edab49f312c80561b2899f03f361f74fc38e4
change-id: 20260413-b4-pks-odb-source-loose-4900c8ca91db
^ permalink raw reply [flat|nested] 22+ messages in thread
* [PATCH 01/18] odb/source-loose: move loose source into "odb/" subsystem
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 02/18] odb/source-loose: store pointer to "files" instead of generic source Patrick Steinhardt
` (16 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
In subsequent patches we'll be turning `struct odb_source_loose` into a
proper `struct odb_source`. As a first step towards this goal, move its
struct out of "object-file.c" and into "odb/source-loose.c".
This detaches the implementation of the loose object source from the
generic object file code, following the same convention already used by
the "files" and "in-memory" sources.
No functional changes are intended.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Makefile | 1 +
meson.build | 1 +
object-file.c | 8 --------
object-file.h | 21 +--------------------
odb/source-loose.c | 10 ++++++++++
odb/source-loose.h | 34 ++++++++++++++++++++++++++++++++++
6 files changed, 47 insertions(+), 28 deletions(-)
diff --git a/Makefile b/Makefile
index a43b8ee067..01356235c3 100644
--- a/Makefile
+++ b/Makefile
@@ -1217,6 +1217,7 @@ LIB_OBJS += odb.o
LIB_OBJS += odb/source.o
LIB_OBJS += odb/source-files.o
LIB_OBJS += odb/source-inmemory.o
+LIB_OBJS += odb/source-loose.o
LIB_OBJS += odb/streaming.o
LIB_OBJS += odb/transaction.o
LIB_OBJS += oid-array.o
diff --git a/meson.build b/meson.build
index 664d831329..c85e598835 100644
--- a/meson.build
+++ b/meson.build
@@ -405,6 +405,7 @@ libgit_sources = [
'odb/source.c',
'odb/source-files.c',
'odb/source-inmemory.c',
+ 'odb/source-loose.c',
'odb/streaming.c',
'odb/transaction.c',
'oid-array.c',
diff --git a/object-file.c b/object-file.c
index 90f995d000..641bd9c079 100644
--- a/object-file.c
+++ b/object-file.c
@@ -2205,14 +2205,6 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
return &transaction->base;
}
-struct odb_source_loose *odb_source_loose_new(struct odb_source *source)
-{
- struct odb_source_loose *loose;
- CALLOC_ARRAY(loose, 1);
- loose->source = source;
- return loose;
-}
-
void odb_source_loose_free(struct odb_source_loose *loose)
{
if (!loose)
diff --git a/object-file.h b/object-file.h
index 5241b8dd5c..1d8312cf7f 100644
--- a/object-file.h
+++ b/object-file.h
@@ -4,6 +4,7 @@
#include "git-zlib.h"
#include "object.h"
#include "odb.h"
+#include "odb/source-loose.h"
struct index_state;
@@ -20,26 +21,6 @@ struct object_info;
struct odb_read_stream;
struct odb_source;
-struct odb_source_loose {
- struct odb_source *source;
-
- /*
- * Used to store the results of readdir(3) calls when we are OK
- * sacrificing accuracy due to races for speed. That includes
- * object existence with OBJECT_INFO_QUICK, as well as
- * our search for unique abbreviated hashes. Don't use it for tasks
- * requiring greater accuracy!
- *
- * Be sure to call odb_load_loose_cache() before using.
- */
- uint32_t subdir_seen[8]; /* 256 bits */
- struct oidtree *cache;
-
- /* Map between object IDs for loose objects. */
- struct loose_object_map *map;
-};
-
-struct odb_source_loose *odb_source_loose_new(struct odb_source *source);
void odb_source_loose_free(struct odb_source_loose *loose);
/* Reprepare the loose source by emptying the loose object cache. */
diff --git a/odb/source-loose.c b/odb/source-loose.c
new file mode 100644
index 0000000000..b944d21813
--- /dev/null
+++ b/odb/source-loose.c
@@ -0,0 +1,10 @@
+#include "git-compat-util.h"
+#include "odb/source-loose.h"
+
+struct odb_source_loose *odb_source_loose_new(struct odb_source *source)
+{
+ struct odb_source_loose *loose;
+ CALLOC_ARRAY(loose, 1);
+ loose->source = source;
+ return loose;
+}
diff --git a/odb/source-loose.h b/odb/source-loose.h
new file mode 100644
index 0000000000..8b4bac77ea
--- /dev/null
+++ b/odb/source-loose.h
@@ -0,0 +1,34 @@
+#ifndef ODB_SOURCE_LOOSE_H
+#define ODB_SOURCE_LOOSE_H
+
+#include "odb/source.h"
+
+struct object_database;
+struct oidtree;
+
+/*
+ * An object database source that stores its objects in loose format, one
+ * file per object. This source is part of the files source.
+ */
+struct odb_source_loose {
+ struct odb_source *source;
+
+ /*
+ * Used to store the results of readdir(3) calls when we are OK
+ * sacrificing accuracy due to races for speed. That includes
+ * object existence with OBJECT_INFO_QUICK, as well as
+ * our search for unique abbreviated hashes. Don't use it for tasks
+ * requiring greater accuracy!
+ *
+ * Be sure to call odb_load_loose_cache() before using.
+ */
+ uint32_t subdir_seen[8]; /* 256 bits */
+ struct oidtree *cache;
+
+ /* Map between object IDs for loose objects. */
+ struct loose_object_map *map;
+};
+
+struct odb_source_loose *odb_source_loose_new(struct odb_source *source);
+
+#endif
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 02/18] odb/source-loose: store pointer to "files" instead of generic source
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 01/18] odb/source-loose: move loose source into "odb/" subsystem Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 03/18] odb/source-loose: start converting to a proper `struct odb_source` Patrick Steinhardt
` (15 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
The `struct odb_source_loose` holds a pointer to its owning parent
source. The way that Git is currently structured, this parent is always
the "files" source. In subsequent commits we're going to detangle that
so that the "loose" source doesn't have any owning parent source at all
so that it can be used as a completely standalone source.
Detangling this mess is somewhat intricate though, and is made even more
intricate because it's not always clear which kind of source one is
holding at a specific point in time -- either the parent "files" source,
or the child "loose" source.
Make this relationship more explicit by storing a pointer to the "files"
source instead of storing a pointer to a generic `struct odb_source`.
This will help make subsequent steps a bit clearer.
Note that this is a temporary step, only. At the end of this series
we will have dropped the parent pointer completely.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 4 ++--
odb/source-files.c | 2 +-
odb/source-loose.c | 4 ++--
odb/source-loose.h | 5 +++--
4 files changed, 8 insertions(+), 7 deletions(-)
diff --git a/object-file.c b/object-file.c
index 641bd9c079..7a1908bfc0 100644
--- a/object-file.c
+++ b/object-file.c
@@ -178,7 +178,7 @@ static int open_loose_object(struct odb_source_loose *loose,
static struct strbuf buf = STRBUF_INIT;
int fd;
- *path = odb_loose_path(loose->source, &buf, oid);
+ *path = odb_loose_path(&loose->files->base, &buf, oid);
fd = git_open(*path);
if (fd >= 0)
return fd;
@@ -189,7 +189,7 @@ static int open_loose_object(struct odb_source_loose *loose,
static int quick_has_loose(struct odb_source_loose *loose,
const struct object_id *oid)
{
- return !!oidtree_contains(odb_source_loose_cache(loose->source, oid), oid);
+ return !!oidtree_contains(odb_source_loose_cache(&loose->files->base, oid), oid);
}
/*
diff --git a/odb/source-files.c b/odb/source-files.c
index b5abd20e97..185cc6903e 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -264,7 +264,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
CALLOC_ARRAY(files, 1);
odb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);
- files->loose = odb_source_loose_new(&files->base);
+ files->loose = odb_source_loose_new(files);
files->packed = packfile_store_new(&files->base);
files->base.free = odb_source_files_free;
diff --git a/odb/source-loose.c b/odb/source-loose.c
index b944d21813..c9e7414814 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -1,10 +1,10 @@
#include "git-compat-util.h"
#include "odb/source-loose.h"
-struct odb_source_loose *odb_source_loose_new(struct odb_source *source)
+struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
{
struct odb_source_loose *loose;
CALLOC_ARRAY(loose, 1);
- loose->source = source;
+ loose->files = files;
return loose;
}
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 8b4bac77ea..bf61e767c8 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -3,6 +3,7 @@
#include "odb/source.h"
+struct odb_source_files;
struct object_database;
struct oidtree;
@@ -11,7 +12,7 @@ struct oidtree;
* file per object. This source is part of the files source.
*/
struct odb_source_loose {
- struct odb_source *source;
+ struct odb_source_files *files;
/*
* Used to store the results of readdir(3) calls when we are OK
@@ -29,6 +30,6 @@ struct odb_source_loose {
struct loose_object_map *map;
};
-struct odb_source_loose *odb_source_loose_new(struct odb_source *source);
+struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files);
#endif
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 03/18] odb/source-loose: start converting to a proper `struct odb_source`
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 01/18] odb/source-loose: move loose source into "odb/" subsystem Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 02/18] odb/source-loose: store pointer to "files" instead of generic source Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 15:49 ` Junio C Hamano
2026-05-21 8:22 ` [PATCH 04/18] odb/source-loose: wire up `reprepare()` callback Patrick Steinhardt
` (14 subsequent siblings)
17 siblings, 1 reply; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Start converting `struct odb_source_loose` into a proper pluggable
`struct odb_source` by embedding the base struct and assigning it the
new `ODB_SOURCE_LOOSE` type. Furthermore, wire up lifecycle management
of this source by implementing the `free` callback and taking ownership
of the chdir notifications.
Note that the loose source is not yet functional as a standalone `struct
odb_source`, as it's missing all of the callback implementations. These
will be wired up in subsequent commits.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 17 -----------------
object-file.h | 2 --
odb/source-files.c | 2 +-
odb/source-loose.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
odb/source-loose.h | 14 ++++++++++++++
odb/source.h | 3 +++
6 files changed, 63 insertions(+), 20 deletions(-)
diff --git a/object-file.c b/object-file.c
index 7a1908bfc0..977d959d33 100644
--- a/object-file.c
+++ b/object-file.c
@@ -2041,14 +2041,6 @@ static struct oidtree *odb_source_loose_cache(struct odb_source *source,
return files->loose->cache;
}
-static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
-{
- oidtree_clear(loose->cache);
- FREE_AND_NULL(loose->cache);
- memset(&loose->subdir_seen, 0,
- sizeof(loose->subdir_seen));
-}
-
void odb_source_loose_reprepare(struct odb_source *source)
{
struct odb_source_files *files = odb_source_files_downcast(source);
@@ -2205,15 +2197,6 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
return &transaction->base;
}
-void odb_source_loose_free(struct odb_source_loose *loose)
-{
- if (!loose)
- return;
- odb_source_loose_clear_cache(loose);
- loose_object_map_clear(&loose->map);
- free(loose);
-}
-
struct odb_loose_read_stream {
struct odb_read_stream base;
git_zstream z;
diff --git a/object-file.h b/object-file.h
index 1d8312cf7f..02c9680980 100644
--- a/object-file.h
+++ b/object-file.h
@@ -21,8 +21,6 @@ struct object_info;
struct odb_read_stream;
struct odb_source;
-void odb_source_loose_free(struct odb_source_loose *loose);
-
/* Reprepare the loose source by emptying the loose object cache. */
void odb_source_loose_reprepare(struct odb_source *source);
diff --git a/odb/source-files.c b/odb/source-files.c
index 185cc6903e..ccc637311b 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -27,7 +27,7 @@ static void odb_source_files_free(struct odb_source *source)
{
struct odb_source_files *files = odb_source_files_downcast(source);
chdir_notify_unregister(NULL, odb_source_files_reparent, files);
- odb_source_loose_free(files->loose);
+ odb_source_free(&files->loose->base);
packfile_store_free(files->packed);
odb_source_release(&files->base);
free(files);
diff --git a/odb/source-loose.c b/odb/source-loose.c
index c9e7414814..92e18f5adb 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -1,10 +1,55 @@
#include "git-compat-util.h"
+#include "abspath.h"
+#include "chdir-notify.h"
+#include "loose.h"
+#include "odb.h"
+#include "odb/source-files.h"
#include "odb/source-loose.h"
+#include "oidtree.h"
+
+void odb_source_loose_clear_cache(struct odb_source_loose *loose)
+{
+ oidtree_clear(loose->cache);
+ FREE_AND_NULL(loose->cache);
+ memset(&loose->subdir_seen, 0,
+ sizeof(loose->subdir_seen));
+}
+
+static void odb_source_loose_reparent(const char *name UNUSED,
+ const char *old_cwd,
+ const char *new_cwd,
+ void *cb_data)
+{
+ struct odb_source_loose *loose = cb_data;
+ char *path = reparent_relative_path(old_cwd, new_cwd,
+ loose->base.path);
+ free(loose->base.path);
+ loose->base.path = path;
+}
+
+static void odb_source_loose_free(struct odb_source *source)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ odb_source_loose_clear_cache(loose);
+ loose_object_map_clear(&loose->map);
+ chdir_notify_unregister(NULL, odb_source_loose_reparent, loose);
+ odb_source_release(&loose->base);
+ free(loose);
+}
struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
{
struct odb_source_loose *loose;
+
CALLOC_ARRAY(loose, 1);
+ odb_source_init(&loose->base, files->base.odb, ODB_SOURCE_LOOSE,
+ files->base.path, files->base.local);
loose->files = files;
+
+ loose->base.free = odb_source_loose_free;
+
+ if (!is_absolute_path(loose->base.path))
+ chdir_notify_register(NULL, odb_source_loose_reparent, loose);
+
return loose;
}
diff --git a/odb/source-loose.h b/odb/source-loose.h
index bf61e767c8..441da9e418 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -12,6 +12,7 @@ struct oidtree;
* file per object. This source is part of the files source.
*/
struct odb_source_loose {
+ struct odb_source base;
struct odb_source_files *files;
/*
@@ -32,4 +33,17 @@ struct odb_source_loose {
struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files);
+/*
+ * Cast the given object database source to the loose backend. This will cause
+ * a BUG in case the source uses doesn't use this backend.
+ */
+static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
+{
+ if (source->type != ODB_SOURCE_LOOSE)
+ BUG("trying to downcast source of type '%d' to loose", source->type);
+ return container_of(source, struct odb_source_loose, base);
+}
+
+void odb_source_loose_clear_cache(struct odb_source_loose *loose);
+
#endif
diff --git a/odb/source.h b/odb/source.h
index 0a440884e4..8bcb67787e 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -14,6 +14,9 @@ enum odb_source_type {
/* The "files" backend that uses loose objects and packfiles. */
ODB_SOURCE_FILES,
+ /* The "loose" backend that uses loose objects, only. */
+ ODB_SOURCE_LOOSE,
+
/* The "in-memory" backend that stores objects in memory. */
ODB_SOURCE_INMEMORY,
};
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 04/18] odb/source-loose: wire up `reprepare()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (2 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 03/18] odb/source-loose: start converting to a proper `struct odb_source` Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 05/18] odb/source-loose: wire up `close()` callback Patrick Steinhardt
` (13 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_reprepare()` from "object-file.c" into
"odb/source-loose.c" and wire it up as the `reprepare()` callback of the
loose source.
While at it, make `odb_source_loose_clear_cache()` static, as it is no
longer needed outside of its file.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 6 ------
object-file.h | 3 ---
odb/source-files.c | 2 +-
odb/source-loose.c | 9 ++++++++-
odb/source-loose.h | 2 --
5 files changed, 9 insertions(+), 13 deletions(-)
diff --git a/object-file.c b/object-file.c
index 977d959d33..0f4f1e7bdc 100644
--- a/object-file.c
+++ b/object-file.c
@@ -2041,12 +2041,6 @@ static struct oidtree *odb_source_loose_cache(struct odb_source *source,
return files->loose->cache;
}
-void odb_source_loose_reprepare(struct odb_source *source)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- odb_source_loose_clear_cache(files->loose);
-}
-
static int check_stream_oid(git_zstream *stream,
const char *hdr,
unsigned long size,
diff --git a/object-file.h b/object-file.h
index 02c9680980..420a0fff2e 100644
--- a/object-file.h
+++ b/object-file.h
@@ -21,9 +21,6 @@ struct object_info;
struct odb_read_stream;
struct odb_source;
-/* Reprepare the loose source by emptying the loose object cache. */
-void odb_source_loose_reprepare(struct odb_source *source);
-
int odb_source_loose_read_object_info(struct odb_source *source,
const struct object_id *oid,
struct object_info *oi,
diff --git a/odb/source-files.c b/odb/source-files.c
index ccc637311b..10832e81e4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -42,7 +42,7 @@ static void odb_source_files_close(struct odb_source *source)
static void odb_source_files_reprepare(struct odb_source *source)
{
struct odb_source_files *files = odb_source_files_downcast(source);
- odb_source_loose_reprepare(&files->base);
+ odb_source_reprepare(&files->loose->base);
packfile_store_reprepare(files->packed);
}
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 92e18f5adb..e0fe0d513d 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -7,7 +7,7 @@
#include "odb/source-loose.h"
#include "oidtree.h"
-void odb_source_loose_clear_cache(struct odb_source_loose *loose)
+static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
FREE_AND_NULL(loose->cache);
@@ -15,6 +15,12 @@ void odb_source_loose_clear_cache(struct odb_source_loose *loose)
sizeof(loose->subdir_seen));
}
+static void odb_source_loose_reprepare(struct odb_source *source)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ odb_source_loose_clear_cache(loose);
+}
+
static void odb_source_loose_reparent(const char *name UNUSED,
const char *old_cwd,
const char *new_cwd,
@@ -47,6 +53,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->files = files;
loose->base.free = odb_source_loose_free;
+ loose->base.reprepare = odb_source_loose_reprepare;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 441da9e418..825e703072 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -44,6 +44,4 @@ static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_sour
return container_of(source, struct odb_source_loose, base);
}
-void odb_source_loose_clear_cache(struct odb_source_loose *loose);
-
#endif
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 05/18] odb/source-loose: wire up `close()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (3 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 04/18] odb/source-loose: wire up `reprepare()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 06/18] odb/source-loose: wire up `read_object_info()` callback Patrick Steinhardt
` (12 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Wire up a new `close()` callback for the loose source and call it from
the "files" source via the generic `odb_source_close()` interface. The
callback itself is a no-op as the loose source has no resources that
need to be released on close.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
odb/source-files.c | 1 +
odb/source-loose.c | 6 ++++++
2 files changed, 7 insertions(+)
diff --git a/odb/source-files.c b/odb/source-files.c
index 10832e81e4..59e3a70d80 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -36,6 +36,7 @@ static void odb_source_files_free(struct odb_source *source)
static void odb_source_files_close(struct odb_source *source)
{
struct odb_source_files *files = odb_source_files_downcast(source);
+ odb_source_close(&files->loose->base);
packfile_store_close(files->packed);
}
diff --git a/odb/source-loose.c b/odb/source-loose.c
index e0fe0d513d..65c1076659 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -21,6 +21,11 @@ static void odb_source_loose_reprepare(struct odb_source *source)
odb_source_loose_clear_cache(loose);
}
+static void odb_source_loose_close(struct odb_source *source UNUSED)
+{
+ /* Nothing to do. */
+}
+
static void odb_source_loose_reparent(const char *name UNUSED,
const char *old_cwd,
const char *new_cwd,
@@ -53,6 +58,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->files = files;
loose->base.free = odb_source_loose_free;
+ loose->base.close = odb_source_loose_close;
loose->base.reprepare = odb_source_loose_reprepare;
if (!is_absolute_path(loose->base.path))
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 06/18] odb/source-loose: wire up `read_object_info()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (4 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 05/18] odb/source-loose: wire up `close()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 07/18] odb/source-loose: wire up `read_object_stream()` callback Patrick Steinhardt
` (11 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_read_object_info()` from "object-file.c" into
"odb/source-loose.c" and wire it up as the `read_object_info()` callback
of the loose source. Callers that previously invoked it directly now go
through the generic `odb_source_read_object_info()` interface instead.
The function `read_object_info_from_path()` cannot be moved along with
it because it is still called by `for_each_object_wrapper_cb()`. It is
therefore kept in place, but adjusted to take a loose source to clarify
that it's always operating on this structure.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 46 +++++++++++++---------------------------------
object-file.h | 11 ++++++-----
odb/source-files.c | 2 +-
odb/source-loose.c | 24 ++++++++++++++++++++++++
4 files changed, 44 insertions(+), 39 deletions(-)
diff --git a/object-file.c b/object-file.c
index 0f4f1e7bdc..fa174512a4 100644
--- a/object-file.c
+++ b/object-file.c
@@ -396,13 +396,12 @@ static int parse_loose_header(const char *hdr, struct object_info *oi)
return 0;
}
-static int read_object_info_from_path(struct odb_source *source,
- const char *path,
- const struct object_id *oid,
- struct object_info *oi,
- enum object_info_flags flags)
+int read_object_info_from_path(struct odb_source_loose *loose,
+ const char *path,
+ const struct object_id *oid,
+ struct object_info *oi,
+ enum object_info_flags flags)
{
- struct odb_source_files *files = odb_source_files_downcast(source);
int ret;
int fd;
unsigned long mapsize;
@@ -425,7 +424,7 @@ static int read_object_info_from_path(struct odb_source *source,
struct stat st;
if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
- ret = quick_has_loose(files->loose, oid) ? 0 : -1;
+ ret = quick_has_loose(loose, oid) ? 0 : -1;
goto out;
}
@@ -532,7 +531,7 @@ static int read_object_info_from_path(struct odb_source *source,
if (oi->typep == &type_scratch)
oi->typep = NULL;
if (oi->delta_base_oid)
- oidclr(oi->delta_base_oid, source->odb->repo->hash_algo);
+ oidclr(oi->delta_base_oid, loose->base.odb->repo->hash_algo);
if (!ret)
oi->whence = OI_LOOSE;
}
@@ -540,26 +539,6 @@ static int read_object_info_from_path(struct odb_source *source,
return ret;
}
-int odb_source_loose_read_object_info(struct odb_source *source,
- const struct object_id *oid,
- struct object_info *oi,
- enum object_info_flags flags)
-{
- static struct strbuf buf = STRBUF_INIT;
-
- /*
- * The second read shouldn't cause new loose objects to show up, unless
- * there was a race condition with a secondary process. We don't care
- * about this case though, so we simply skip reading loose objects a
- * second time.
- */
- if (flags & OBJECT_INFO_SECOND_READ)
- return -1;
-
- odb_loose_path(source, &buf, oid);
- return read_object_info_from_path(source, buf.buf, oid, oi, flags);
-}
-
static void hash_object_body(const struct git_hash_algo *algo, struct git_hash_ctx *c,
const void *buf, unsigned long len,
struct object_id *oid,
@@ -1833,7 +1812,7 @@ int for_each_loose_file_in_source(struct odb_source *source,
}
struct for_each_object_wrapper_data {
- struct odb_source *source;
+ struct odb_source_loose *loose;
const struct object_info *request;
odb_for_each_object_cb cb;
void *cb_data;
@@ -1848,7 +1827,7 @@ static int for_each_object_wrapper_cb(const struct object_id *oid,
if (data->request) {
struct object_info oi = *data->request;
- if (read_object_info_from_path(data->source, path, oid, &oi, 0) < 0)
+ if (read_object_info_from_path(data->loose, path, oid, &oi, 0) < 0)
return -1;
return data->cb(oid, &oi, data->cb_data);
@@ -1865,8 +1844,8 @@ static int for_each_prefixed_object_wrapper_cb(const struct object_id *oid,
if (data->request) {
struct object_info oi = *data->request;
- if (odb_source_loose_read_object_info(data->source,
- oid, &oi, 0) < 0)
+ if (odb_source_read_object_info(&data->loose->base,
+ oid, &oi, 0) < 0)
return -1;
return data->cb(oid, &oi, data->cb_data);
@@ -1881,8 +1860,9 @@ int odb_source_loose_for_each_object(struct odb_source *source,
void *cb_data,
const struct odb_for_each_object_options *opts)
{
+ struct odb_source_files *files = odb_source_files_downcast(source);
struct for_each_object_wrapper_data data = {
- .source = source,
+ .loose = files->loose,
.request = request,
.cb = cb,
.cb_data = cb_data,
diff --git a/object-file.h b/object-file.h
index 420a0fff2e..8ac2832dac 100644
--- a/object-file.h
+++ b/object-file.h
@@ -21,11 +21,6 @@ struct object_info;
struct odb_read_stream;
struct odb_source;
-int odb_source_loose_read_object_info(struct odb_source *source,
- const struct object_id *oid,
- struct object_info *oi,
- enum object_info_flags flags);
-
int odb_source_loose_read_object_stream(struct odb_read_stream **out,
struct odb_source *source,
const struct object_id *oid);
@@ -198,6 +193,12 @@ int read_loose_object(struct repository *repo,
void **contents,
struct object_info *oi);
+int read_object_info_from_path(struct odb_source_loose *loose,
+ const char *path,
+ const struct object_id *oid,
+ struct object_info *oi,
+ enum object_info_flags flags);
+
struct odb_transaction;
/*
diff --git a/odb/source-files.c b/odb/source-files.c
index 59e3a70d80..8d6924755f 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -55,7 +55,7 @@ static int odb_source_files_read_object_info(struct odb_source *source,
struct odb_source_files *files = odb_source_files_downcast(source);
if (!packfile_store_read_object_info(files->packed, oid, oi, flags) ||
- !odb_source_loose_read_object_info(source, oid, oi, flags))
+ !odb_source_read_object_info(&files->loose->base, oid, oi, flags))
return 0;
return -1;
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 65c1076659..50f387ecf3 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -2,10 +2,33 @@
#include "abspath.h"
#include "chdir-notify.h"
#include "loose.h"
+#include "object-file.h"
#include "odb.h"
#include "odb/source-files.h"
#include "odb/source-loose.h"
#include "oidtree.h"
+#include "strbuf.h"
+
+static int odb_source_loose_read_object_info(struct odb_source *source,
+ const struct object_id *oid,
+ struct object_info *oi,
+ enum object_info_flags flags)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ static struct strbuf buf = STRBUF_INIT;
+
+ /*
+ * The second read shouldn't cause new loose objects to show up, unless
+ * there was a race condition with a secondary process. We don't care
+ * about this case though, so we simply skip reading loose objects a
+ * second time.
+ */
+ if (flags & OBJECT_INFO_SECOND_READ)
+ return -1;
+
+ odb_loose_path(source, &buf, oid);
+ return read_object_info_from_path(loose, buf.buf, oid, oi, flags);
+}
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
@@ -60,6 +83,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.free = odb_source_loose_free;
loose->base.close = odb_source_loose_close;
loose->base.reprepare = odb_source_loose_reprepare;
+ loose->base.read_object_info = odb_source_loose_read_object_info;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 07/18] odb/source-loose: wire up `read_object_stream()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (5 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 06/18] odb/source-loose: wire up `read_object_info()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 08/18] odb/source-loose: wire up `for_each_object()` callback Patrick Steinhardt
` (10 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_read_object_stream()` and its associated helpers
from "object-file.c" into "odb/source-loose.c" and wire it up as the
`read_object_stream()` callback of the loose source.
As part of the move we are also forced to expose a couple of functions
from "object-file.h" that parse object headers in a somewhat-generic
way, as those functions are now used by both subsystems.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 200 ++---------------------------------------------------
object-file.h | 31 +++++++--
odb/source-files.c | 2 +-
odb/source-loose.c | 189 ++++++++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 222 insertions(+), 200 deletions(-)
diff --git a/object-file.c b/object-file.c
index fa174512a4..adfb672493 100644
--- a/object-file.c
+++ b/object-file.c
@@ -164,28 +164,6 @@ int stream_object_signature(struct repository *r,
return !oideq(oid, &real_oid) ? -1 : 0;
}
-/*
- * Find "oid" as a loose object in given source, open the object and return its
- * file descriptor. Returns the file descriptor on success, negative on failure.
- *
- * The "path" out-parameter will give the path of the object we found (if any).
- * Note that it may point to static storage and is only valid until another
- * call to stat_loose_object().
- */
-static int open_loose_object(struct odb_source_loose *loose,
- const struct object_id *oid, const char **path)
-{
- static struct strbuf buf = STRBUF_INIT;
- int fd;
-
- *path = odb_loose_path(&loose->files->base, &buf, oid);
- fd = git_open(*path);
- if (fd >= 0)
- return fd;
-
- return -1;
-}
-
static int quick_has_loose(struct odb_source_loose *loose,
const struct object_id *oid)
{
@@ -215,42 +193,11 @@ static void *map_fd(int fd, const char *path, unsigned long *size)
return map;
}
-static void *odb_source_loose_map_object(struct odb_source *source,
- const struct object_id *oid,
- unsigned long *size)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- const char *p;
- int fd = open_loose_object(files->loose, oid, &p);
-
- if (fd < 0)
- return NULL;
- return map_fd(fd, p, size);
-}
-
-enum unpack_loose_header_result {
- ULHR_OK,
- ULHR_BAD,
- ULHR_TOO_LONG,
-};
-
-/**
- * unpack_loose_header() initializes the data stream needed to unpack
- * a loose object header.
- *
- * Returns:
- *
- * - ULHR_OK on success
- * - ULHR_BAD on error
- * - ULHR_TOO_LONG if the header was too long
- *
- * It will only parse up to MAX_HEADER_LEN bytes.
- */
-static enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
- unsigned char *map,
- unsigned long mapsize,
- void *buffer,
- unsigned long bufsiz)
+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
+ unsigned char *map,
+ unsigned long mapsize,
+ void *buffer,
+ unsigned long bufsiz)
{
int status;
@@ -340,7 +287,7 @@ static void *unpack_loose_rest(git_zstream *stream,
* too permissive for what we want to check. So do an anal
* object header parse by hand.
*/
-static int parse_loose_header(const char *hdr, struct object_info *oi)
+int parse_loose_header(const char *hdr, struct object_info *oi)
{
const char *type_buf = hdr;
size_t size;
@@ -2170,138 +2117,3 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
return &transaction->base;
}
-
-struct odb_loose_read_stream {
- struct odb_read_stream base;
- git_zstream z;
- enum {
- ODB_LOOSE_READ_STREAM_INUSE,
- ODB_LOOSE_READ_STREAM_DONE,
- ODB_LOOSE_READ_STREAM_ERROR,
- } z_state;
- void *mapped;
- unsigned long mapsize;
- char hdr[32];
- int hdr_avail;
- int hdr_used;
-};
-
-static ssize_t read_istream_loose(struct odb_read_stream *_st, char *buf, size_t sz)
-{
- struct odb_loose_read_stream *st =
- container_of(_st, struct odb_loose_read_stream, base);
- size_t total_read = 0;
-
- switch (st->z_state) {
- case ODB_LOOSE_READ_STREAM_DONE:
- return 0;
- case ODB_LOOSE_READ_STREAM_ERROR:
- return -1;
- default:
- break;
- }
-
- if (st->hdr_used < st->hdr_avail) {
- size_t to_copy = st->hdr_avail - st->hdr_used;
- if (sz < to_copy)
- to_copy = sz;
- memcpy(buf, st->hdr + st->hdr_used, to_copy);
- st->hdr_used += to_copy;
- total_read += to_copy;
- }
-
- while (total_read < sz) {
- int status;
-
- st->z.next_out = (unsigned char *)buf + total_read;
- st->z.avail_out = sz - total_read;
- status = git_inflate(&st->z, Z_FINISH);
-
- total_read = st->z.next_out - (unsigned char *)buf;
-
- if (status == Z_STREAM_END) {
- git_inflate_end(&st->z);
- st->z_state = ODB_LOOSE_READ_STREAM_DONE;
- break;
- }
- if (status != Z_OK && (status != Z_BUF_ERROR || total_read < sz)) {
- git_inflate_end(&st->z);
- st->z_state = ODB_LOOSE_READ_STREAM_ERROR;
- return -1;
- }
- }
- return total_read;
-}
-
-static int close_istream_loose(struct odb_read_stream *_st)
-{
- struct odb_loose_read_stream *st =
- container_of(_st, struct odb_loose_read_stream, base);
-
- if (st->z_state == ODB_LOOSE_READ_STREAM_INUSE)
- git_inflate_end(&st->z);
- munmap(st->mapped, st->mapsize);
- return 0;
-}
-
-int odb_source_loose_read_object_stream(struct odb_read_stream **out,
- struct odb_source *source,
- const struct object_id *oid)
-{
- struct object_info oi = OBJECT_INFO_INIT;
- struct odb_loose_read_stream *st;
- unsigned long mapsize;
- unsigned long size_ul;
- void *mapped;
-
- mapped = odb_source_loose_map_object(source, oid, &mapsize);
- if (!mapped)
- return -1;
-
- /*
- * Note: we must allocate this structure early even though we may still
- * fail. This is because we need to initialize the zlib stream, and it
- * is not possible to copy the stream around after the fact because it
- * has self-referencing pointers.
- */
- CALLOC_ARRAY(st, 1);
-
- switch (unpack_loose_header(&st->z, mapped, mapsize, st->hdr,
- sizeof(st->hdr))) {
- case ULHR_OK:
- break;
- case ULHR_BAD:
- case ULHR_TOO_LONG:
- goto error;
- }
-
- /*
- * object_info.sizep is unsigned long* (32-bit on Windows), but
- * st->base.size is size_t (64-bit). Use temporary variable.
- * Note: loose objects >4GB would still truncate here, but such
- * large loose objects are uncommon (they'd normally be packed).
- */
- oi.sizep = &size_ul;
- oi.typep = &st->base.type;
-
- if (parse_loose_header(st->hdr, &oi) < 0 || st->base.type < 0)
- goto error;
- st->base.size = size_ul;
-
- st->mapped = mapped;
- st->mapsize = mapsize;
- st->hdr_used = strlen(st->hdr) + 1;
- st->hdr_avail = st->z.total_out;
- st->z_state = ODB_LOOSE_READ_STREAM_INUSE;
- st->base.close = close_istream_loose;
- st->base.read = read_istream_loose;
-
- *out = &st->base;
-
- return 0;
-error:
- git_inflate_end(&st->z);
- munmap(mapped, mapsize);
- free(st);
- return -1;
-}
diff --git a/object-file.h b/object-file.h
index 8ac2832dac..d93b7ffad7 100644
--- a/object-file.h
+++ b/object-file.h
@@ -18,13 +18,8 @@ int index_fd(struct index_state *istate, struct object_id *oid, int fd, struct s
int index_path(struct index_state *istate, struct object_id *oid, const char *path, struct stat *st, unsigned flags);
struct object_info;
-struct odb_read_stream;
struct odb_source;
-int odb_source_loose_read_object_stream(struct odb_read_stream **out,
- struct odb_source *source,
- const struct object_id *oid);
-
/*
* Return true iff an object database source has a loose object
* with the specified name. This function does not respect replace
@@ -199,6 +194,32 @@ int read_object_info_from_path(struct odb_source_loose *loose,
struct object_info *oi,
enum object_info_flags flags);
+enum unpack_loose_header_result {
+ ULHR_OK,
+ ULHR_BAD,
+ ULHR_TOO_LONG,
+};
+
+/**
+ * unpack_loose_header() initializes the data stream needed to unpack
+ * a loose object header.
+ *
+ * Returns:
+ *
+ * - ULHR_OK on success
+ * - ULHR_BAD on error
+ * - ULHR_TOO_LONG if the header was too long
+ *
+ * It will only parse up to MAX_HEADER_LEN bytes.
+ */
+enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
+ unsigned char *map,
+ unsigned long mapsize,
+ void *buffer,
+ unsigned long bufsiz);
+
+int parse_loose_header(const char *hdr, struct object_info *oi);
+
struct odb_transaction;
/*
diff --git a/odb/source-files.c b/odb/source-files.c
index 8d6924755f..90806ddf86 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -67,7 +67,7 @@ static int odb_source_files_read_object_stream(struct odb_read_stream **out,
{
struct odb_source_files *files = odb_source_files_downcast(source);
if (!packfile_store_read_object_stream(out, files->packed, oid) ||
- !odb_source_loose_read_object_stream(out, source, oid))
+ !odb_source_read_object_stream(out, &files->loose->base, oid))
return 0;
return -1;
}
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 50f387ecf3..4b82c6f316 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -1,11 +1,13 @@
#include "git-compat-util.h"
#include "abspath.h"
#include "chdir-notify.h"
+#include "gettext.h"
#include "loose.h"
#include "object-file.h"
#include "odb.h"
#include "odb/source-files.h"
#include "odb/source-loose.h"
+#include "odb/streaming.h"
#include "oidtree.h"
#include "strbuf.h"
@@ -30,6 +32,192 @@ static int odb_source_loose_read_object_info(struct odb_source *source,
return read_object_info_from_path(loose, buf.buf, oid, oi, flags);
}
+/*
+ * Find "oid" as a loose object in given source, open the object and return its
+ * file descriptor. Returns the file descriptor on success, negative on failure.
+ *
+ * The "path" out-parameter will give the path of the object we found (if any).
+ * Note that it may point to static storage and is only valid until another
+ * call to open_loose_object().
+ */
+static int open_loose_object(struct odb_source_loose *loose,
+ const struct object_id *oid, const char **path)
+{
+ static struct strbuf buf = STRBUF_INIT;
+ int fd;
+
+ *path = odb_loose_path(&loose->base, &buf, oid);
+ fd = git_open(*path);
+ if (fd >= 0)
+ return fd;
+
+ return -1;
+}
+
+static void *odb_source_loose_map_object(struct odb_source_loose *loose,
+ const struct object_id *oid,
+ unsigned long *size)
+{
+ const char *p;
+ int fd = open_loose_object(loose, oid, &p);
+ void *map = NULL;
+ struct stat st;
+
+ if (fd < 0)
+ return NULL;
+
+ if (!fstat(fd, &st)) {
+ *size = xsize_t(st.st_size);
+ if (!*size) {
+ /* mmap() is forbidden on empty files */
+ error(_("object file %s is empty"), p);
+ goto out;
+ }
+
+ map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
+ }
+
+out:
+ close(fd);
+ return map;
+}
+
+struct odb_loose_read_stream {
+ struct odb_read_stream base;
+ git_zstream z;
+ enum {
+ ODB_LOOSE_READ_STREAM_INUSE,
+ ODB_LOOSE_READ_STREAM_DONE,
+ ODB_LOOSE_READ_STREAM_ERROR,
+ } z_state;
+ void *mapped;
+ unsigned long mapsize;
+ char hdr[32];
+ int hdr_avail;
+ int hdr_used;
+};
+
+static ssize_t read_istream_loose(struct odb_read_stream *_st, char *buf, size_t sz)
+{
+ struct odb_loose_read_stream *st =
+ container_of(_st, struct odb_loose_read_stream, base);
+ size_t total_read = 0;
+
+ switch (st->z_state) {
+ case ODB_LOOSE_READ_STREAM_DONE:
+ return 0;
+ case ODB_LOOSE_READ_STREAM_ERROR:
+ return -1;
+ default:
+ break;
+ }
+
+ if (st->hdr_used < st->hdr_avail) {
+ size_t to_copy = st->hdr_avail - st->hdr_used;
+ if (sz < to_copy)
+ to_copy = sz;
+ memcpy(buf, st->hdr + st->hdr_used, to_copy);
+ st->hdr_used += to_copy;
+ total_read += to_copy;
+ }
+
+ while (total_read < sz) {
+ int status;
+
+ st->z.next_out = (unsigned char *)buf + total_read;
+ st->z.avail_out = sz - total_read;
+ status = git_inflate(&st->z, Z_FINISH);
+
+ total_read = st->z.next_out - (unsigned char *)buf;
+
+ if (status == Z_STREAM_END) {
+ git_inflate_end(&st->z);
+ st->z_state = ODB_LOOSE_READ_STREAM_DONE;
+ break;
+ }
+ if (status != Z_OK && (status != Z_BUF_ERROR || total_read < sz)) {
+ git_inflate_end(&st->z);
+ st->z_state = ODB_LOOSE_READ_STREAM_ERROR;
+ return -1;
+ }
+ }
+ return total_read;
+}
+
+static int close_istream_loose(struct odb_read_stream *_st)
+{
+ struct odb_loose_read_stream *st =
+ container_of(_st, struct odb_loose_read_stream, base);
+
+ if (st->z_state == ODB_LOOSE_READ_STREAM_INUSE)
+ git_inflate_end(&st->z);
+ munmap(st->mapped, st->mapsize);
+ return 0;
+}
+
+static int odb_source_loose_read_object_stream(struct odb_read_stream **out,
+ struct odb_source *source,
+ const struct object_id *oid)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ struct object_info oi = OBJECT_INFO_INIT;
+ struct odb_loose_read_stream *st;
+ unsigned long mapsize;
+ unsigned long size_ul;
+ void *mapped;
+
+ mapped = odb_source_loose_map_object(loose, oid, &mapsize);
+ if (!mapped)
+ return -1;
+
+ /*
+ * Note: we must allocate this structure early even though we may still
+ * fail. This is because we need to initialize the zlib stream, and it
+ * is not possible to copy the stream around after the fact because it
+ * has self-referencing pointers.
+ */
+ CALLOC_ARRAY(st, 1);
+
+ switch (unpack_loose_header(&st->z, mapped, mapsize, st->hdr,
+ sizeof(st->hdr))) {
+ case ULHR_OK:
+ break;
+ case ULHR_BAD:
+ case ULHR_TOO_LONG:
+ goto error;
+ }
+
+ /*
+ * object_info.sizep is unsigned long* (32-bit on Windows), but
+ * st->base.size is size_t (64-bit). Use temporary variable.
+ * Note: loose objects >4GB would still truncate here, but such
+ * large loose objects are uncommon (they'd normally be packed).
+ */
+ oi.sizep = &size_ul;
+ oi.typep = &st->base.type;
+
+ if (parse_loose_header(st->hdr, &oi) < 0 || st->base.type < 0)
+ goto error;
+ st->base.size = size_ul;
+
+ st->mapped = mapped;
+ st->mapsize = mapsize;
+ st->hdr_used = strlen(st->hdr) + 1;
+ st->hdr_avail = st->z.total_out;
+ st->z_state = ODB_LOOSE_READ_STREAM_INUSE;
+ st->base.close = close_istream_loose;
+ st->base.read = read_istream_loose;
+
+ *out = &st->base;
+
+ return 0;
+error:
+ git_inflate_end(&st->z);
+ munmap(mapped, mapsize);
+ free(st);
+ return -1;
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -84,6 +272,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.close = odb_source_loose_close;
loose->base.reprepare = odb_source_loose_reprepare;
loose->base.read_object_info = odb_source_loose_read_object_info;
+ loose->base.read_object_stream = odb_source_loose_read_object_stream;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 08/18] odb/source-loose: wire up `for_each_object()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (6 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 07/18] odb/source-loose: wire up `read_object_stream()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 09/18] odb/source-loose: wire up `find_abbrev_len()` callback Patrick Steinhardt
` (9 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_for_each_object()` and its associated helpers
from "object-file.c" into "odb/source-loose.c" and wire it up as the
`for_each_object()` callback of the loose source.
Again, as in the preceding commit, we are forced to expose a couple of
functions from "object-file.c" that are now used by both subsystems.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
builtin/cat-file.c | 5 +-
object-file.c | 299 +++--------------------------------------------------
object-file.h | 32 +++---
odb/source-files.c | 2 +-
odb/source-loose.c | 264 ++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 297 insertions(+), 305 deletions(-)
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index d9fbad5358..2958fc5357 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -862,8 +862,9 @@ static void batch_each_object(struct batch_options *opt,
*/
odb_prepare_alternates(the_repository->objects);
for (source = the_repository->objects->sources; source; source = source->next) {
- int ret = odb_source_loose_for_each_object(source, NULL, batch_one_object_oi,
- &payload, &opts);
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ int ret = odb_source_for_each_object(&files->loose->base, NULL, batch_one_object_oi,
+ &payload, &opts);
if (ret)
break;
}
diff --git a/object-file.c b/object-file.c
index adfb672493..157ecad3ea 100644
--- a/object-file.c
+++ b/object-file.c
@@ -22,7 +22,6 @@
#include "odb.h"
#include "odb/streaming.h"
#include "odb/transaction.h"
-#include "oidtree.h"
#include "pack.h"
#include "packfile.h"
#include "path.h"
@@ -31,12 +30,6 @@
#include "tempfile.h"
#include "tmp-objdir.h"
-/* The maximum size for an object header. */
-#define MAX_HEADER_LEN 32
-
-static struct oidtree *odb_source_loose_cache(struct odb_source *source,
- const struct object_id *oid);
-
static int get_conv_flags(unsigned flags)
{
if (flags & INDEX_RENORMALIZE)
@@ -164,12 +157,6 @@ int stream_object_signature(struct repository *r,
return !oideq(oid, &real_oid) ? -1 : 0;
}
-static int quick_has_loose(struct odb_source_loose *loose,
- const struct object_id *oid)
-{
- return !!oidtree_contains(odb_source_loose_cache(&loose->files->base, oid), oid);
-}
-
/*
* Map and close the given loose object fd. The path argument is used for
* error reporting.
@@ -227,9 +214,9 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
return ULHR_TOO_LONG;
}
-static void *unpack_loose_rest(git_zstream *stream,
- void *buffer, unsigned long size,
- const struct object_id *oid)
+void *unpack_loose_rest(git_zstream *stream,
+ void *buffer, unsigned long size,
+ const struct object_id *oid)
{
size_t bytes = strlen(buffer) + 1, n;
unsigned char *buf = xmallocz(size);
@@ -343,149 +330,6 @@ int parse_loose_header(const char *hdr, struct object_info *oi)
return 0;
}
-int read_object_info_from_path(struct odb_source_loose *loose,
- const char *path,
- const struct object_id *oid,
- struct object_info *oi,
- enum object_info_flags flags)
-{
- int ret;
- int fd;
- unsigned long mapsize;
- void *map = NULL;
- git_zstream stream, *stream_to_end = NULL;
- char hdr[MAX_HEADER_LEN];
- unsigned long size_scratch;
- enum object_type type_scratch;
- struct stat st;
-
- /*
- * If we don't care about type or size, then we don't
- * need to look inside the object at all. Note that we
- * do not optimize out the stat call, even if the
- * caller doesn't care about the disk-size, since our
- * return value implicitly indicates whether the
- * object even exists.
- */
- if (!oi || (!oi->typep && !oi->sizep && !oi->contentp)) {
- struct stat st;
-
- if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
- ret = quick_has_loose(loose, oid) ? 0 : -1;
- goto out;
- }
-
- if (lstat(path, &st) < 0) {
- ret = -1;
- goto out;
- }
-
- if (oi) {
- if (oi->disk_sizep)
- *oi->disk_sizep = st.st_size;
- if (oi->mtimep)
- *oi->mtimep = st.st_mtime;
- }
-
- ret = 0;
- goto out;
- }
-
- fd = git_open(path);
- if (fd < 0) {
- if (errno != ENOENT)
- error_errno(_("unable to open loose object %s"), oid_to_hex(oid));
- ret = -1;
- goto out;
- }
-
- if (fstat(fd, &st)) {
- close(fd);
- ret = -1;
- goto out;
- }
-
- mapsize = xsize_t(st.st_size);
- if (!mapsize) {
- close(fd);
- ret = error(_("object file %s is empty"), path);
- goto out;
- }
-
- map = xmmap(NULL, mapsize, PROT_READ, MAP_PRIVATE, fd, 0);
- close(fd);
- if (!map) {
- ret = -1;
- goto out;
- }
-
- if (oi->disk_sizep)
- *oi->disk_sizep = mapsize;
- if (oi->mtimep)
- *oi->mtimep = st.st_mtime;
-
- stream_to_end = &stream;
-
- switch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr))) {
- case ULHR_OK:
- if (!oi->sizep)
- oi->sizep = &size_scratch;
- if (!oi->typep)
- oi->typep = &type_scratch;
-
- if (parse_loose_header(hdr, oi) < 0) {
- ret = error(_("unable to parse %s header"), oid_to_hex(oid));
- goto corrupt;
- }
-
- if (*oi->typep < 0)
- die(_("invalid object type"));
-
- if (oi->contentp) {
- *oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);
- if (!*oi->contentp) {
- ret = -1;
- goto corrupt;
- }
- }
-
- break;
- case ULHR_BAD:
- ret = error(_("unable to unpack %s header"),
- oid_to_hex(oid));
- goto corrupt;
- case ULHR_TOO_LONG:
- ret = error(_("header for %s too long, exceeds %d bytes"),
- oid_to_hex(oid), MAX_HEADER_LEN);
- goto corrupt;
- }
-
- ret = 0;
-
-corrupt:
- if (ret && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
- die(_("loose object %s (stored in %s) is corrupt"),
- oid_to_hex(oid), path);
-
-out:
- if (stream_to_end)
- git_inflate_end(stream_to_end);
- if (map)
- munmap(map, mapsize);
- if (oi) {
- if (oi->sizep == &size_scratch)
- oi->sizep = NULL;
- if (oi->typep == &type_scratch)
- oi->typep = NULL;
- if (oi->delta_base_oid)
- oidclr(oi->delta_base_oid, loose->base.odb->repo->hash_algo);
- if (!ret)
- oi->whence = OI_LOOSE;
- }
-
- return ret;
-}
-
static void hash_object_body(const struct git_hash_algo *algo, struct git_hash_ctx *c,
const void *buf, unsigned long len,
struct object_id *oid,
@@ -1667,13 +1511,13 @@ int read_pack_header(int fd, struct pack_header *header)
return 0;
}
-static int for_each_file_in_obj_subdir(unsigned int subdir_nr,
- struct strbuf *path,
- const struct git_hash_algo *algop,
- each_loose_object_fn obj_cb,
- each_loose_cruft_fn cruft_cb,
- each_loose_subdir_fn subdir_cb,
- void *data)
+int for_each_file_in_obj_subdir(unsigned int subdir_nr,
+ struct strbuf *path,
+ const struct git_hash_algo *algop,
+ each_loose_object_fn obj_cb,
+ each_loose_cruft_fn cruft_cb,
+ each_loose_subdir_fn subdir_cb,
+ void *data)
{
size_t origlen, baselen;
DIR *dir;
@@ -1758,78 +1602,6 @@ int for_each_loose_file_in_source(struct odb_source *source,
return r;
}
-struct for_each_object_wrapper_data {
- struct odb_source_loose *loose;
- const struct object_info *request;
- odb_for_each_object_cb cb;
- void *cb_data;
-};
-
-static int for_each_object_wrapper_cb(const struct object_id *oid,
- const char *path,
- void *cb_data)
-{
- struct for_each_object_wrapper_data *data = cb_data;
-
- if (data->request) {
- struct object_info oi = *data->request;
-
- if (read_object_info_from_path(data->loose, path, oid, &oi, 0) < 0)
- return -1;
-
- return data->cb(oid, &oi, data->cb_data);
- } else {
- return data->cb(oid, NULL, data->cb_data);
- }
-}
-
-static int for_each_prefixed_object_wrapper_cb(const struct object_id *oid,
- void *node_data UNUSED,
- void *cb_data)
-{
- struct for_each_object_wrapper_data *data = cb_data;
- if (data->request) {
- struct object_info oi = *data->request;
-
- if (odb_source_read_object_info(&data->loose->base,
- oid, &oi, 0) < 0)
- return -1;
-
- return data->cb(oid, &oi, data->cb_data);
- } else {
- return data->cb(oid, NULL, data->cb_data);
- }
-}
-
-int odb_source_loose_for_each_object(struct odb_source *source,
- const struct object_info *request,
- odb_for_each_object_cb cb,
- void *cb_data,
- const struct odb_for_each_object_options *opts)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- struct for_each_object_wrapper_data data = {
- .loose = files->loose,
- .request = request,
- .cb = cb,
- .cb_data = cb_data,
- };
-
- /* There are no loose promisor objects, so we can return immediately. */
- if ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY))
- return 0;
- if ((opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY) && !source->local)
- return 0;
-
- if (opts->prefix)
- return oidtree_each(odb_source_loose_cache(source, opts->prefix),
- opts->prefix, opts->prefix_hex_len,
- for_each_prefixed_object_wrapper_cb, &data);
-
- return for_each_loose_file_in_source(source, for_each_object_wrapper_cb,
- NULL, NULL, &data);
-}
-
static int count_loose_object(const struct object_id *oid UNUSED,
struct object_info *oi UNUSED,
void *payload)
@@ -1843,6 +1615,7 @@ int odb_source_loose_count_objects(struct odb_source *source,
enum odb_count_objects_flags flags,
unsigned long *out)
{
+ struct odb_source_files *files = odb_source_files_downcast(source);
const unsigned hexsz = source->odb->repo->hash_algo->hexsz - 2;
char *path = NULL;
DIR *dir = NULL;
@@ -1878,8 +1651,8 @@ int odb_source_loose_count_objects(struct odb_source *source,
} else {
struct odb_for_each_object_options opts = { 0 };
*out = 0;
- ret = odb_source_loose_for_each_object(source, NULL, count_loose_object,
- out, &opts);
+ ret = odb_source_for_each_object(&files->loose->base, NULL, count_loose_object,
+ out, &opts);
}
out:
@@ -1910,6 +1683,7 @@ int odb_source_loose_find_abbrev_len(struct odb_source *source,
unsigned min_len,
unsigned *out)
{
+ struct odb_source_files *files = odb_source_files_downcast(source);
struct odb_for_each_object_options opts = {
.prefix = oid,
.prefix_hex_len = min_len,
@@ -1920,54 +1694,13 @@ int odb_source_loose_find_abbrev_len(struct odb_source *source,
};
int ret;
- ret = odb_source_loose_for_each_object(source, NULL, find_abbrev_len_cb,
- &data, &opts);
+ ret = odb_source_for_each_object(&files->loose->base, NULL, find_abbrev_len_cb,
+ &data, &opts);
*out = data.len;
return ret;
}
-static int append_loose_object(const struct object_id *oid,
- const char *path UNUSED,
- void *data)
-{
- oidtree_insert(data, oid, NULL);
- return 0;
-}
-
-static struct oidtree *odb_source_loose_cache(struct odb_source *source,
- const struct object_id *oid)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- int subdir_nr = oid->hash[0];
- struct strbuf buf = STRBUF_INIT;
- size_t word_bits = bitsizeof(files->loose->subdir_seen[0]);
- size_t word_index = subdir_nr / word_bits;
- size_t mask = (size_t)1u << (subdir_nr % word_bits);
- uint32_t *bitmap;
-
- if (subdir_nr < 0 ||
- (size_t) subdir_nr >= bitsizeof(files->loose->subdir_seen))
- BUG("subdir_nr out of range");
-
- bitmap = &files->loose->subdir_seen[word_index];
- if (*bitmap & mask)
- return files->loose->cache;
- if (!files->loose->cache) {
- ALLOC_ARRAY(files->loose->cache, 1);
- oidtree_init(files->loose->cache);
- }
- strbuf_addstr(&buf, source->path);
- for_each_file_in_obj_subdir(subdir_nr, &buf,
- source->odb->repo->hash_algo,
- append_loose_object,
- NULL, NULL,
- files->loose->cache);
- *bitmap |= mask;
- strbuf_release(&buf);
- return files->loose->cache;
-}
-
static int check_stream_oid(git_zstream *stream,
const char *hdr,
unsigned long size,
diff --git a/object-file.h b/object-file.h
index d93b7ffad7..9ee5649220 100644
--- a/object-file.h
+++ b/object-file.h
@@ -6,6 +6,9 @@
#include "odb.h"
#include "odb/source-loose.h"
+/* The maximum size for an object header. */
+#define MAX_HEADER_LEN 32
+
struct index_state;
enum {
@@ -85,19 +88,13 @@ int for_each_loose_file_in_source(struct odb_source *source,
each_loose_cruft_fn cruft_cb,
each_loose_subdir_fn subdir_cb,
void *data);
-
-/*
- * Iterate through all loose objects in the given object database source and
- * invoke the callback function for each of them. If an object info request is
- * given, then the object info will be read for every individual object and
- * passed to the callback as if `odb_source_loose_read_object_info()` was
- * called for the object.
- */
-int odb_source_loose_for_each_object(struct odb_source *source,
- const struct object_info *request,
- odb_for_each_object_cb cb,
- void *cb_data,
- const struct odb_for_each_object_options *opts);
+int for_each_file_in_obj_subdir(unsigned int subdir_nr,
+ struct strbuf *path,
+ const struct git_hash_algo *algop,
+ each_loose_object_fn obj_cb,
+ each_loose_cruft_fn cruft_cb,
+ each_loose_subdir_fn subdir_cb,
+ void *data);
/*
* Count the number of loose objects in this source.
@@ -188,12 +185,6 @@ int read_loose_object(struct repository *repo,
void **contents,
struct object_info *oi);
-int read_object_info_from_path(struct odb_source_loose *loose,
- const char *path,
- const struct object_id *oid,
- struct object_info *oi,
- enum object_info_flags flags);
-
enum unpack_loose_header_result {
ULHR_OK,
ULHR_BAD,
@@ -217,6 +208,9 @@ enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
unsigned long mapsize,
void *buffer,
unsigned long bufsiz);
+void *unpack_loose_rest(git_zstream *stream,
+ void *buffer, unsigned long size,
+ const struct object_id *oid);
int parse_loose_header(const char *hdr, struct object_info *oi);
diff --git a/odb/source-files.c b/odb/source-files.c
index 90806ddf86..676a641739 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -82,7 +82,7 @@ static int odb_source_files_for_each_object(struct odb_source *source,
int ret;
if (!(opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)) {
- ret = odb_source_loose_for_each_object(source, request, cb, cb_data, opts);
+ ret = odb_source_for_each_object(&files->loose->base, request, cb, cb_data, opts);
if (ret)
return ret;
}
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 4b82c6f316..4e8b923498 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -2,6 +2,7 @@
#include "abspath.h"
#include "chdir-notify.h"
#include "gettext.h"
+#include "hex.h"
#include "loose.h"
#include "object-file.h"
#include "odb.h"
@@ -9,8 +10,198 @@
#include "odb/source-loose.h"
#include "odb/streaming.h"
#include "oidtree.h"
+#include "repository.h"
#include "strbuf.h"
+static int append_loose_object(const struct object_id *oid,
+ const char *path UNUSED,
+ void *data)
+{
+ oidtree_insert(data, oid, NULL);
+ return 0;
+}
+
+static struct oidtree *odb_source_loose_cache(struct odb_source_loose *loose,
+ const struct object_id *oid)
+{
+ int subdir_nr = oid->hash[0];
+ struct strbuf buf = STRBUF_INIT;
+ size_t word_bits = bitsizeof(loose->subdir_seen[0]);
+ size_t word_index = subdir_nr / word_bits;
+ size_t mask = (size_t)1u << (subdir_nr % word_bits);
+ uint32_t *bitmap;
+
+ if (subdir_nr < 0 ||
+ (size_t) subdir_nr >= bitsizeof(loose->subdir_seen))
+ BUG("subdir_nr out of range");
+
+ bitmap = &loose->subdir_seen[word_index];
+ if (*bitmap & mask)
+ return loose->cache;
+ if (!loose->cache) {
+ ALLOC_ARRAY(loose->cache, 1);
+ oidtree_init(loose->cache);
+ }
+ strbuf_addstr(&buf, loose->base.path);
+ for_each_file_in_obj_subdir(subdir_nr, &buf,
+ loose->base.odb->repo->hash_algo,
+ append_loose_object,
+ NULL, NULL,
+ loose->cache);
+ *bitmap |= mask;
+ strbuf_release(&buf);
+ return loose->cache;
+}
+
+static int quick_has_loose(struct odb_source_loose *loose,
+ const struct object_id *oid)
+{
+ return !!oidtree_contains(odb_source_loose_cache(loose, oid), oid);
+}
+
+static int read_object_info_from_path(struct odb_source_loose *loose,
+ const char *path,
+ const struct object_id *oid,
+ struct object_info *oi,
+ enum object_info_flags flags)
+{
+ int ret;
+ int fd;
+ unsigned long mapsize;
+ void *map = NULL;
+ git_zstream stream, *stream_to_end = NULL;
+ char hdr[MAX_HEADER_LEN];
+ unsigned long size_scratch;
+ enum object_type type_scratch;
+ struct stat st;
+
+ /*
+ * If we don't care about type or size, then we don't
+ * need to look inside the object at all. Note that we
+ * do not optimize out the stat call, even if the
+ * caller doesn't care about the disk-size, since our
+ * return value implicitly indicates whether the
+ * object even exists.
+ */
+ if (!oi || (!oi->typep && !oi->sizep && !oi->contentp)) {
+ struct stat st;
+
+ if ((!oi || (!oi->disk_sizep && !oi->mtimep)) && (flags & OBJECT_INFO_QUICK)) {
+ ret = quick_has_loose(loose, oid) ? 0 : -1;
+ goto out;
+ }
+
+ if (lstat(path, &st) < 0) {
+ ret = -1;
+ goto out;
+ }
+
+ if (oi) {
+ if (oi->disk_sizep)
+ *oi->disk_sizep = st.st_size;
+ if (oi->mtimep)
+ *oi->mtimep = st.st_mtime;
+ }
+
+ ret = 0;
+ goto out;
+ }
+
+ fd = git_open(path);
+ if (fd < 0) {
+ if (errno != ENOENT)
+ error_errno(_("unable to open loose object %s"), oid_to_hex(oid));
+ ret = -1;
+ goto out;
+ }
+
+ if (fstat(fd, &st)) {
+ close(fd);
+ ret = -1;
+ goto out;
+ }
+
+ mapsize = xsize_t(st.st_size);
+ if (!mapsize) {
+ close(fd);
+ ret = error(_("object file %s is empty"), path);
+ goto out;
+ }
+
+ map = xmmap(NULL, mapsize, PROT_READ, MAP_PRIVATE, fd, 0);
+ close(fd);
+ if (!map) {
+ ret = -1;
+ goto out;
+ }
+
+ if (oi->disk_sizep)
+ *oi->disk_sizep = mapsize;
+ if (oi->mtimep)
+ *oi->mtimep = st.st_mtime;
+
+ stream_to_end = &stream;
+
+ switch (unpack_loose_header(&stream, map, mapsize, hdr, sizeof(hdr))) {
+ case ULHR_OK:
+ if (!oi->sizep)
+ oi->sizep = &size_scratch;
+ if (!oi->typep)
+ oi->typep = &type_scratch;
+
+ if (parse_loose_header(hdr, oi) < 0) {
+ ret = error(_("unable to parse %s header"), oid_to_hex(oid));
+ goto corrupt;
+ }
+
+ if (*oi->typep < 0)
+ die(_("invalid object type"));
+
+ if (oi->contentp) {
+ *oi->contentp = unpack_loose_rest(&stream, hdr, *oi->sizep, oid);
+ if (!*oi->contentp) {
+ ret = -1;
+ goto corrupt;
+ }
+ }
+
+ break;
+ case ULHR_BAD:
+ ret = error(_("unable to unpack %s header"),
+ oid_to_hex(oid));
+ goto corrupt;
+ case ULHR_TOO_LONG:
+ ret = error(_("header for %s too long, exceeds %d bytes"),
+ oid_to_hex(oid), MAX_HEADER_LEN);
+ goto corrupt;
+ }
+
+ ret = 0;
+
+corrupt:
+ if (ret && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+ die(_("loose object %s (stored in %s) is corrupt"),
+ oid_to_hex(oid), path);
+
+out:
+ if (stream_to_end)
+ git_inflate_end(stream_to_end);
+ if (map)
+ munmap(map, mapsize);
+ if (oi) {
+ if (oi->sizep == &size_scratch)
+ oi->sizep = NULL;
+ if (oi->typep == &type_scratch)
+ oi->typep = NULL;
+ if (oi->delta_base_oid)
+ oidclr(oi->delta_base_oid, loose->base.odb->repo->hash_algo);
+ if (!ret)
+ oi->whence = OI_LOOSE;
+ }
+
+ return ret;
+}
+
static int odb_source_loose_read_object_info(struct odb_source *source,
const struct object_id *oid,
struct object_info *oi,
@@ -218,6 +409,78 @@ static int odb_source_loose_read_object_stream(struct odb_read_stream **out,
return -1;
}
+struct for_each_object_wrapper_data {
+ struct odb_source_loose *loose;
+ const struct object_info *request;
+ odb_for_each_object_cb cb;
+ void *cb_data;
+};
+
+static int for_each_object_wrapper_cb(const struct object_id *oid,
+ const char *path,
+ void *cb_data)
+{
+ struct for_each_object_wrapper_data *data = cb_data;
+
+ if (data->request) {
+ struct object_info oi = *data->request;
+
+ if (read_object_info_from_path(data->loose, path, oid, &oi, 0) < 0)
+ return -1;
+
+ return data->cb(oid, &oi, data->cb_data);
+ } else {
+ return data->cb(oid, NULL, data->cb_data);
+ }
+}
+
+static int for_each_prefixed_object_wrapper_cb(const struct object_id *oid,
+ void *node_data UNUSED,
+ void *cb_data)
+{
+ struct for_each_object_wrapper_data *data = cb_data;
+ if (data->request) {
+ struct object_info oi = *data->request;
+
+ if (odb_source_read_object_info(&data->loose->base,
+ oid, &oi, 0) < 0)
+ return -1;
+
+ return data->cb(oid, &oi, data->cb_data);
+ } else {
+ return data->cb(oid, NULL, data->cb_data);
+ }
+}
+
+static int odb_source_loose_for_each_object(struct odb_source *source,
+ const struct object_info *request,
+ odb_for_each_object_cb cb,
+ void *cb_data,
+ const struct odb_for_each_object_options *opts)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ struct for_each_object_wrapper_data data = {
+ .loose = loose,
+ .request = request,
+ .cb = cb,
+ .cb_data = cb_data,
+ };
+
+ /* There are no loose promisor objects, so we can return immediately. */
+ if ((opts->flags & ODB_FOR_EACH_OBJECT_PROMISOR_ONLY))
+ return 0;
+ if ((opts->flags & ODB_FOR_EACH_OBJECT_LOCAL_ONLY) && !source->local)
+ return 0;
+
+ if (opts->prefix)
+ return oidtree_each(odb_source_loose_cache(loose, opts->prefix),
+ opts->prefix, opts->prefix_hex_len,
+ for_each_prefixed_object_wrapper_cb, &data);
+
+ return for_each_loose_file_in_source(source, for_each_object_wrapper_cb,
+ NULL, NULL, &data);
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -273,6 +536,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.reprepare = odb_source_loose_reprepare;
loose->base.read_object_info = odb_source_loose_read_object_info;
loose->base.read_object_stream = odb_source_loose_read_object_stream;
+ loose->base.for_each_object = odb_source_loose_for_each_object;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 09/18] odb/source-loose: wire up `find_abbrev_len()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (7 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 08/18] odb/source-loose: wire up `for_each_object()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 10/18] odb/source-loose: wire up `count_objects()` callback Patrick Steinhardt
` (8 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_find_abbrev_len()` and its associated helpers
from "object-file.c" into "odb/source-loose.c" and wire it up as the
`find_abbrev_len` callback of the loose source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 39 ---------------------------------------
object-file.h | 12 ------------
odb/source-files.c | 2 +-
odb/source-loose.c | 40 ++++++++++++++++++++++++++++++++++++++++
4 files changed, 41 insertions(+), 52 deletions(-)
diff --git a/object-file.c b/object-file.c
index 157ecad3ea..11957aa44f 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1662,45 +1662,6 @@ int odb_source_loose_count_objects(struct odb_source *source,
return ret;
}
-struct find_abbrev_len_data {
- const struct object_id *oid;
- unsigned len;
-};
-
-static int find_abbrev_len_cb(const struct object_id *oid,
- struct object_info *oi UNUSED,
- void *cb_data)
-{
- struct find_abbrev_len_data *data = cb_data;
- unsigned len = oid_common_prefix_hexlen(oid, data->oid);
- if (len != hash_algos[oid->algo].hexsz && len >= data->len)
- data->len = len + 1;
- return 0;
-}
-
-int odb_source_loose_find_abbrev_len(struct odb_source *source,
- const struct object_id *oid,
- unsigned min_len,
- unsigned *out)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- struct odb_for_each_object_options opts = {
- .prefix = oid,
- .prefix_hex_len = min_len,
- };
- struct find_abbrev_len_data data = {
- .oid = oid,
- .len = min_len,
- };
- int ret;
-
- ret = odb_source_for_each_object(&files->loose->base, NULL, find_abbrev_len_cb,
- &data, &opts);
- *out = data.len;
-
- return ret;
-}
-
static int check_stream_oid(git_zstream *stream,
const char *hdr,
unsigned long size,
diff --git a/object-file.h b/object-file.h
index 9ee5649220..96760db0e1 100644
--- a/object-file.h
+++ b/object-file.h
@@ -110,18 +110,6 @@ int odb_source_loose_count_objects(struct odb_source *source,
enum odb_count_objects_flags flags,
unsigned long *out);
-/*
- * Find the shortest unique prefix for the given object ID, where `min_len` is
- * the minimum length that the prefix should have.
- *
- * Returns 0 on success, in which case the computed length will be written to
- * `out`. Otherwise, a negative error code is returned.
- */
-int odb_source_loose_find_abbrev_len(struct odb_source *source,
- const struct object_id *oid,
- unsigned min_len,
- unsigned *out);
-
/**
* format_object_header() is a thin wrapper around s xsnprintf() that
* writes the initial "<type> <obj-len>" part of the loose object
diff --git a/odb/source-files.c b/odb/source-files.c
index 676a641739..4a54b10e4a 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -136,7 +136,7 @@ static int odb_source_files_find_abbrev_len(struct odb_source *source,
if (ret < 0)
goto out;
- ret = odb_source_loose_find_abbrev_len(source, oid, len, &len);
+ ret = odb_source_find_abbrev_len(&files->loose->base, oid, len, &len);
if (ret < 0)
goto out;
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 4e8b923498..4b8d10bc87 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -481,6 +481,45 @@ static int odb_source_loose_for_each_object(struct odb_source *source,
NULL, NULL, &data);
}
+struct find_abbrev_len_data {
+ const struct object_id *oid;
+ unsigned len;
+};
+
+static int find_abbrev_len_cb(const struct object_id *oid,
+ struct object_info *oi UNUSED,
+ void *cb_data)
+{
+ struct find_abbrev_len_data *data = cb_data;
+ unsigned len = oid_common_prefix_hexlen(oid, data->oid);
+ if (len != hash_algos[oid->algo].hexsz && len >= data->len)
+ data->len = len + 1;
+ return 0;
+}
+
+static int odb_source_loose_find_abbrev_len(struct odb_source *source,
+ const struct object_id *oid,
+ unsigned min_len,
+ unsigned *out)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ struct odb_for_each_object_options opts = {
+ .prefix = oid,
+ .prefix_hex_len = min_len,
+ };
+ struct find_abbrev_len_data data = {
+ .oid = oid,
+ .len = min_len,
+ };
+ int ret;
+
+ ret = odb_source_for_each_object(&loose->base, NULL, find_abbrev_len_cb,
+ &data, &opts);
+ *out = data.len;
+
+ return ret;
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -537,6 +576,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.read_object_info = odb_source_loose_read_object_info;
loose->base.read_object_stream = odb_source_loose_read_object_stream;
loose->base.for_each_object = odb_source_loose_for_each_object;
+ loose->base.find_abbrev_len = odb_source_loose_find_abbrev_len;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 10/18] odb/source-loose: wire up `count_objects()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (8 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 09/18] odb/source-loose: wire up `find_abbrev_len()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 11/18] odb/source-loose: drop `odb_source_loose_has_object()` Patrick Steinhardt
` (7 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_count_objects()` and its associated helpers from
"object-file.c" into "odb/source-loose.c" and wire it up as the
`count_objects()` callback of the loose source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
builtin/gc.c | 6 +++---
object-file.c | 60 -----------------------------------------------------
object-file.h | 14 -------------
odb/source-files.c | 2 +-
odb/source-loose.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 65 insertions(+), 78 deletions(-)
diff --git a/builtin/gc.c b/builtin/gc.c
index 84a66d3240..c26c93ee0f 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -466,6 +466,7 @@ static int rerere_gc_condition(struct gc_config *cfg UNUSED)
static int too_many_loose_objects(int limit)
{
+ struct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);
/*
* This is weird, but stems from legacy behaviour: the GC auto
* threshold was always essentially interpreted as if it was rounded up
@@ -474,9 +475,8 @@ static int too_many_loose_objects(int limit)
int auto_threshold = DIV_ROUND_UP(limit, 256) * 256;
unsigned long loose_count;
- if (odb_source_loose_count_objects(the_repository->objects->sources,
- ODB_COUNT_OBJECTS_APPROXIMATE,
- &loose_count) < 0)
+ if (odb_source_count_objects(&files->loose->base, ODB_COUNT_OBJECTS_APPROXIMATE,
+ &loose_count) < 0)
return 0;
return loose_count > auto_threshold;
diff --git a/object-file.c b/object-file.c
index 11957aa44f..9b2044de37 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1602,66 +1602,6 @@ int for_each_loose_file_in_source(struct odb_source *source,
return r;
}
-static int count_loose_object(const struct object_id *oid UNUSED,
- struct object_info *oi UNUSED,
- void *payload)
-{
- unsigned long *count = payload;
- (*count)++;
- return 0;
-}
-
-int odb_source_loose_count_objects(struct odb_source *source,
- enum odb_count_objects_flags flags,
- unsigned long *out)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- const unsigned hexsz = source->odb->repo->hash_algo->hexsz - 2;
- char *path = NULL;
- DIR *dir = NULL;
- int ret;
-
- if (flags & ODB_COUNT_OBJECTS_APPROXIMATE) {
- unsigned long count = 0;
- struct dirent *ent;
-
- path = xstrfmt("%s/17", source->path);
-
- dir = opendir(path);
- if (!dir) {
- if (errno == ENOENT) {
- *out = 0;
- ret = 0;
- goto out;
- }
-
- ret = error_errno("cannot open object shard '%s'", path);
- goto out;
- }
-
- while ((ent = readdir(dir)) != NULL) {
- if (strspn(ent->d_name, "0123456789abcdef") != hexsz ||
- ent->d_name[hexsz] != '\0')
- continue;
- count++;
- }
-
- *out = count * 256;
- ret = 0;
- } else {
- struct odb_for_each_object_options opts = { 0 };
- *out = 0;
- ret = odb_source_for_each_object(&files->loose->base, NULL, count_loose_object,
- out, &opts);
- }
-
-out:
- if (dir)
- closedir(dir);
- free(path);
- return ret;
-}
-
static int check_stream_oid(git_zstream *stream,
const char *hdr,
unsigned long size,
diff --git a/object-file.h b/object-file.h
index 96760db0e1..bc72d89f54 100644
--- a/object-file.h
+++ b/object-file.h
@@ -96,20 +96,6 @@ int for_each_file_in_obj_subdir(unsigned int subdir_nr,
each_loose_subdir_fn subdir_cb,
void *data);
-/*
- * Count the number of loose objects in this source.
- *
- * The object count is approximated by opening a single sharding directory for
- * loose objects and scanning its contents. The result is then extrapolated by
- * 256. This should generally work as a reasonable estimate given that the
- * object hash is supposed to be indistinguishable from random.
- *
- * Returns 0 on success, a negative error code otherwise.
- */
-int odb_source_loose_count_objects(struct odb_source *source,
- enum odb_count_objects_flags flags,
- unsigned long *out);
-
/**
* format_object_header() is a thin wrapper around s xsnprintf() that
* writes the initial "<type> <obj-len>" part of the loose object
diff --git a/odb/source-files.c b/odb/source-files.c
index 4a54b10e4a..d5454e170d 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -109,7 +109,7 @@ static int odb_source_files_count_objects(struct odb_source *source,
if (!(flags & ODB_COUNT_OBJECTS_APPROXIMATE)) {
unsigned long loose_count;
- ret = odb_source_loose_count_objects(source, flags, &loose_count);
+ ret = odb_source_count_objects(&files->loose->base, flags, &loose_count);
if (ret < 0)
goto out;
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 4b8d10bc87..27be066327 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -520,6 +520,66 @@ static int odb_source_loose_find_abbrev_len(struct odb_source *source,
return ret;
}
+static int count_loose_object(const struct object_id *oid UNUSED,
+ struct object_info *oi UNUSED,
+ void *payload)
+{
+ unsigned long *count = payload;
+ (*count)++;
+ return 0;
+}
+
+static int odb_source_loose_count_objects(struct odb_source *source,
+ enum odb_count_objects_flags flags,
+ unsigned long *out)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ const unsigned hexsz = source->odb->repo->hash_algo->hexsz - 2;
+ char *path = NULL;
+ DIR *dir = NULL;
+ int ret;
+
+ if (flags & ODB_COUNT_OBJECTS_APPROXIMATE) {
+ unsigned long count = 0;
+ struct dirent *ent;
+
+ path = xstrfmt("%s/17", source->path);
+
+ dir = opendir(path);
+ if (!dir) {
+ if (errno == ENOENT) {
+ *out = 0;
+ ret = 0;
+ goto out;
+ }
+
+ ret = error_errno("cannot open object shard '%s'", path);
+ goto out;
+ }
+
+ while ((ent = readdir(dir)) != NULL) {
+ if (strspn(ent->d_name, "0123456789abcdef") != hexsz ||
+ ent->d_name[hexsz] != '\0')
+ continue;
+ count++;
+ }
+
+ *out = count * 256;
+ ret = 0;
+ } else {
+ struct odb_for_each_object_options opts = { 0 };
+ *out = 0;
+ ret = odb_source_for_each_object(&loose->base, NULL, count_loose_object,
+ out, &opts);
+ }
+
+out:
+ if (dir)
+ closedir(dir);
+ free(path);
+ return ret;
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -577,6 +637,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.read_object_stream = odb_source_loose_read_object_stream;
loose->base.for_each_object = odb_source_loose_for_each_object;
loose->base.find_abbrev_len = odb_source_loose_find_abbrev_len;
+ loose->base.count_objects = odb_source_loose_count_objects;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 11/18] odb/source-loose: drop `odb_source_loose_has_object()`
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (9 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 10/18] odb/source-loose: wire up `count_objects()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 12/18] odb/source-loose: wire up `freshen_object()` callback Patrick Steinhardt
` (6 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
The function `odb_source_loose_has_object()` checks whether a specific
object exists as a loose object on disk by using lstat(3p). This
interface is somewhat redundant, as we typically check for object
existence in a generic way via `odb_source_read_object_info()`.
In fact, these two calls are redundant in case the latter is called in a
specific way: when called without an object info request and without the
`OBJECT_INFO_QUICK` flag, then we will end up doing the same call to
lstat(3p) in `read_object_info_from_path()`.
Drop the function and adapt callers to instead use the generic
interface so that its calling conventions align with that of other
sources.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
builtin/pack-objects.c | 12 ++++++++----
object-file.c | 12 ++++--------
object-file.h | 8 --------
3 files changed, 12 insertions(+), 20 deletions(-)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 480cc0bd8c..a6be3d659f 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1750,9 +1750,11 @@ static int want_object_in_pack_mtime(const struct object_id *oid,
* skip the local object source.
*/
struct odb_source *source = the_repository->objects->sources->next;
- for (; source; source = source->next)
- if (odb_source_loose_has_object(source, oid))
+ for (; source; source = source->next) {
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ if (!odb_source_read_object_info(&files->loose->base, oid, NULL, 0))
return 0;
+ }
}
/*
@@ -4135,9 +4137,11 @@ static void add_cruft_object_entry(const struct object_id *oid, enum object_type
struct odb_source *source = the_repository->objects->sources;
int found = 0;
- for (; !found && source; source = source->next)
- if (odb_source_loose_has_object(source, oid))
+ for (; !found && source; source = source->next) {
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ if (!odb_source_read_object_info(&files->loose->base, oid, NULL, 0))
found = 1;
+ }
/*
* If a traversed tree has a missing blob then we want
diff --git a/object-file.c b/object-file.c
index 9b2044de37..c83136cf70 100644
--- a/object-file.c
+++ b/object-file.c
@@ -96,12 +96,6 @@ static int check_and_freshen_source(struct odb_source *source,
return check_and_freshen_file(path.buf, freshen);
}
-int odb_source_loose_has_object(struct odb_source *source,
- const struct object_id *oid)
-{
- return check_and_freshen_source(source, oid, 0);
-}
-
int format_object_header(char *str, size_t size, enum object_type type,
size_t objsize)
{
@@ -1000,9 +994,11 @@ int force_object_loose(struct odb_source *source,
int hdrlen;
int ret;
- for (struct odb_source *s = source->odb->sources; s; s = s->next)
- if (odb_source_loose_has_object(s, oid))
+ for (struct odb_source *s = source->odb->sources; s; s = s->next) {
+ struct odb_source_files *files = odb_source_files_downcast(s);
+ if (!odb_source_read_object_info(&files->loose->base, oid, NULL, 0))
return 0;
+ }
oi.typep = &type;
oi.sizep = &len;
diff --git a/object-file.h b/object-file.h
index bc72d89f54..506ca6be40 100644
--- a/object-file.h
+++ b/object-file.h
@@ -23,14 +23,6 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa
struct object_info;
struct odb_source;
-/*
- * Return true iff an object database source has a loose object
- * with the specified name. This function does not respect replace
- * references.
- */
-int odb_source_loose_has_object(struct odb_source *source,
- const struct object_id *oid);
-
int odb_source_loose_freshen_object(struct odb_source *source,
const struct object_id *oid);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 12/18] odb/source-loose: wire up `freshen_object()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (10 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 11/18] odb/source-loose: drop `odb_source_loose_has_object()` Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 13/18] loose: refactor object map to operate on `struct odb_source_loose` Patrick Steinhardt
` (5 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_freshen_object()` from "object-file.c" into
"odb/source-loose.c" and wire it up as the `freshen_object()` callback
of the loose source.
As part of the move, `check_and_freshen_source()` is inlined into the
callback function, as it has no other callers anymore.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 15 ---------------
object-file.h | 3 ---
odb/source-files.c | 2 +-
odb/source-loose.c | 9 +++++++++
4 files changed, 10 insertions(+), 19 deletions(-)
diff --git a/object-file.c b/object-file.c
index c83136cf70..0689a4e67b 100644
--- a/object-file.c
+++ b/object-file.c
@@ -87,15 +87,6 @@ int check_and_freshen_file(const char *fn, int freshen)
return 1;
}
-static int check_and_freshen_source(struct odb_source *source,
- const struct object_id *oid,
- int freshen)
-{
- static struct strbuf path = STRBUF_INIT;
- odb_loose_path(source, &path, oid);
- return check_and_freshen_file(path.buf, freshen);
-}
-
int format_object_header(char *str, size_t size, enum object_type type,
size_t objsize)
{
@@ -815,12 +806,6 @@ static int write_loose_object(struct odb_source *source,
FOF_SKIP_COLLISION_CHECK);
}
-int odb_source_loose_freshen_object(struct odb_source *source,
- const struct object_id *oid)
-{
- return !!check_and_freshen_source(source, oid, 1);
-}
-
int odb_source_loose_write_stream(struct odb_source *source,
struct odb_write_stream *in_stream, size_t len,
struct object_id *oid)
diff --git a/object-file.h b/object-file.h
index 506ca6be40..1d90df9d98 100644
--- a/object-file.h
+++ b/object-file.h
@@ -23,9 +23,6 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa
struct object_info;
struct odb_source;
-int odb_source_loose_freshen_object(struct odb_source *source,
- const struct object_id *oid);
-
int odb_source_loose_write_object(struct odb_source *source,
const void *buf, unsigned long len,
enum object_type type, struct object_id *oid,
diff --git a/odb/source-files.c b/odb/source-files.c
index d5454e170d..ef548e6fe6 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -152,7 +152,7 @@ static int odb_source_files_freshen_object(struct odb_source *source,
{
struct odb_source_files *files = odb_source_files_downcast(source);
if (packfile_store_freshen_object(files->packed, oid) ||
- odb_source_loose_freshen_object(source, oid))
+ odb_source_freshen_object(&files->loose->base, oid))
return 1;
return 0;
}
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 27be066327..e519365d23 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -580,6 +580,14 @@ static int odb_source_loose_count_objects(struct odb_source *source,
return ret;
}
+static int odb_source_loose_freshen_object(struct odb_source *source,
+ const struct object_id *oid)
+{
+ static struct strbuf path = STRBUF_INIT;
+ odb_loose_path(source, &path, oid);
+ return !!check_and_freshen_file(path.buf, 1);
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -638,6 +646,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.for_each_object = odb_source_loose_for_each_object;
loose->base.find_abbrev_len = odb_source_loose_find_abbrev_len;
loose->base.count_objects = odb_source_loose_count_objects;
+ loose->base.freshen_object = odb_source_loose_freshen_object;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 13/18] loose: refactor object map to operate on `struct odb_source_loose`
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (11 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 12/18] odb/source-loose: wire up `freshen_object()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 14/18] odb/source-loose: wire up `write_object()` callback Patrick Steinhardt
` (4 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
While the loose object map functions in "loose.c" accept a generic
`struct odb_source *`, they always expect this to be the "files"
backend. Furthermore, the subsystem doesn't even care about the "files"
backend, but only uses it as a stepping stone to get to the "loose"
backend.
This assumption is implicit and thus not immediately obvious. Refactor
the interfaces to instead operate on a `struct odb_source_loose`
instead, which eliminates the implicit dependency and unnecessary detour
via the "files" source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
loose.c | 45 ++++++++++++++++++++++-----------------------
loose.h | 4 ++--
object-file.c | 9 ++++++---
3 files changed, 30 insertions(+), 28 deletions(-)
diff --git a/loose.c b/loose.c
index f7a3dd1a72..0b626c1b85 100644
--- a/loose.c
+++ b/loose.c
@@ -46,38 +46,36 @@ static int insert_oid_pair(kh_oid_map_t *map, const struct object_id *key, const
return 1;
}
-static int insert_loose_map(struct odb_source *source,
+static int insert_loose_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid)
{
- struct odb_source_files *files = odb_source_files_downcast(source);
- struct loose_object_map *map = files->loose->map;
+ struct loose_object_map *map = loose->map;
int inserted = 0;
inserted |= insert_oid_pair(map->to_compat, oid, compat_oid);
inserted |= insert_oid_pair(map->to_storage, compat_oid, oid);
if (inserted)
- oidtree_insert(files->loose->cache, compat_oid, NULL);
+ oidtree_insert(loose->cache, compat_oid, NULL);
return inserted;
}
-static int load_one_loose_object_map(struct repository *repo, struct odb_source *source)
+static int load_one_loose_object_map(struct repository *repo, struct odb_source_loose *loose)
{
- struct odb_source_files *files = odb_source_files_downcast(source);
struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
FILE *fp;
- if (!files->loose->map)
- loose_object_map_init(&files->loose->map);
- if (!files->loose->cache) {
- ALLOC_ARRAY(files->loose->cache, 1);
- oidtree_init(files->loose->cache);
+ if (!loose->map)
+ loose_object_map_init(&loose->map);
+ if (!loose->cache) {
+ ALLOC_ARRAY(loose->cache, 1);
+ oidtree_init(loose->cache);
}
- insert_loose_map(source, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
- insert_loose_map(source, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
- insert_loose_map(source, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
+ insert_loose_map(loose, repo->hash_algo->empty_tree, repo->compat_hash_algo->empty_tree);
+ insert_loose_map(loose, repo->hash_algo->empty_blob, repo->compat_hash_algo->empty_blob);
+ insert_loose_map(loose, repo->hash_algo->null_oid, repo->compat_hash_algo->null_oid);
repo_common_path_replace(repo, &path, "objects/loose-object-idx");
fp = fopen(path.buf, "rb");
@@ -97,7 +95,7 @@ static int load_one_loose_object_map(struct repository *repo, struct odb_source
parse_oid_hex_algop(p, &compat_oid, &p, repo->compat_hash_algo) ||
p != buf.buf + buf.len)
goto err;
- insert_loose_map(source, &oid, &compat_oid);
+ insert_loose_map(loose, &oid, &compat_oid);
}
strbuf_release(&buf);
@@ -119,7 +117,8 @@ int repo_read_loose_object_map(struct repository *repo)
odb_prepare_alternates(repo->objects);
for (source = repo->objects->sources; source; source = source->next) {
- if (load_one_loose_object_map(repo, source) < 0) {
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ if (load_one_loose_object_map(repo, files->loose) < 0) {
return -1;
}
}
@@ -171,7 +170,7 @@ int repo_write_loose_object_map(struct repository *repo)
return -1;
}
-static int write_one_object(struct odb_source *source,
+static int write_one_object(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid)
{
@@ -180,7 +179,7 @@ static int write_one_object(struct odb_source *source,
struct stat st;
struct strbuf buf = STRBUF_INIT, path = STRBUF_INIT;
- strbuf_addf(&path, "%s/loose-object-idx", source->path);
+ strbuf_addf(&path, "%s/loose-object-idx", loose->base.path);
hold_lock_file_for_update_timeout(&lock, path.buf, LOCK_DIE_ON_ERROR, -1);
fd = open(path.buf, O_WRONLY | O_CREAT | O_APPEND, 0666);
@@ -196,7 +195,7 @@ static int write_one_object(struct odb_source *source,
goto errout;
if (close(fd))
goto errout;
- adjust_shared_perm(source->odb->repo, path.buf);
+ adjust_shared_perm(loose->base.odb->repo, path.buf);
rollback_lock_file(&lock);
strbuf_release(&buf);
strbuf_release(&path);
@@ -210,18 +209,18 @@ static int write_one_object(struct odb_source *source,
return -1;
}
-int repo_add_loose_object_map(struct odb_source *source,
+int repo_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid)
{
int inserted = 0;
- if (!should_use_loose_object_map(source->odb->repo))
+ if (!should_use_loose_object_map(loose->base.odb->repo))
return 0;
- inserted = insert_loose_map(source, oid, compat_oid);
+ inserted = insert_loose_map(loose, oid, compat_oid);
if (inserted)
- return write_one_object(source, oid, compat_oid);
+ return write_one_object(loose, oid, compat_oid);
return 0;
}
diff --git a/loose.h b/loose.h
index 6af1702973..6c9b3f4571 100644
--- a/loose.h
+++ b/loose.h
@@ -4,7 +4,7 @@
#include "khash.h"
struct repository;
-struct odb_source;
+struct odb_source_loose;
struct loose_object_map {
kh_oid_map_t *to_compat;
@@ -17,7 +17,7 @@ int repo_loose_object_map_oid(struct repository *repo,
const struct object_id *src,
const struct git_hash_algo *dest_algo,
struct object_id *dest);
-int repo_add_loose_object_map(struct odb_source *source,
+int repo_add_loose_object_map(struct odb_source_loose *loose,
const struct object_id *oid,
const struct object_id *compat_oid);
int repo_read_loose_object_map(struct repository *repo);
diff --git a/object-file.c b/object-file.c
index 0689a4e67b..fe24f00d1b 100644
--- a/object-file.c
+++ b/object-file.c
@@ -810,6 +810,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
struct odb_write_stream *in_stream, size_t len,
struct object_id *oid)
{
+ struct odb_source_files *files = odb_source_files_downcast(source);
const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
struct object_id compat_oid;
int fd, ret, err = 0, flush = 0;
@@ -918,7 +919,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
err = finalize_object_file_flags(source->odb->repo, tmp_file.buf, filename.buf,
FOF_SKIP_COLLISION_CHECK);
if (!err && compat)
- err = repo_add_loose_object_map(source, oid, &compat_oid);
+ err = repo_add_loose_object_map(files->loose, oid, &compat_oid);
cleanup:
strbuf_release(&tmp_file);
strbuf_release(&filename);
@@ -931,6 +932,7 @@ int odb_source_loose_write_object(struct odb_source *source,
struct object_id *compat_oid_in,
enum odb_write_object_flags flags)
{
+ struct odb_source_files *files = odb_source_files_downcast(source);
const struct git_hash_algo *algo = source->odb->repo->hash_algo;
const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
struct object_id compat_oid;
@@ -962,13 +964,14 @@ int odb_source_loose_write_object(struct odb_source *source,
if (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))
return -1;
if (compat)
- return repo_add_loose_object_map(source, oid, &compat_oid);
+ return repo_add_loose_object_map(files->loose, oid, &compat_oid);
return 0;
}
int force_object_loose(struct odb_source *source,
const struct object_id *oid, time_t mtime)
{
+ struct odb_source_files *files = odb_source_files_downcast(source);
const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
void *buf;
unsigned long len;
@@ -998,7 +1001,7 @@ int force_object_loose(struct odb_source *source,
hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
ret = write_loose_object(source, oid, hdr, hdrlen, buf, len, mtime, 0);
if (!ret && compat)
- ret = repo_add_loose_object_map(source, oid, &compat_oid);
+ ret = repo_add_loose_object_map(files->loose, oid, &compat_oid);
free(buf);
return ret;
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 14/18] odb/source-loose: wire up `write_object()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (12 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 13/18] loose: refactor object map to operate on `struct odb_source_loose` Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 15/18] object-file: refactor writing objects to use loose source Patrick Steinhardt
` (3 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Move `odb_source_loose_write_object()` from "object-file.c" into
"odb/source-loose.c" and wire it up as the `write_object()` callback of
the loose source.
As in preceding commits, this requires us to expose a couple of generic
functions from "object-file.c" as they are used in both subsystems now.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.c | 58 ++++++++----------------------------------------------
object-file.h | 14 +++++++------
odb/source-files.c | 5 +++--
odb/source-loose.c | 44 +++++++++++++++++++++++++++++++++++++++++
4 files changed, 63 insertions(+), 58 deletions(-)
diff --git a/object-file.c b/object-file.c
index fe24f00d1b..7bb5b31bca 100644
--- a/object-file.c
+++ b/object-file.c
@@ -326,10 +326,10 @@ static void hash_object_body(const struct git_hash_algo *algo, struct git_hash_c
git_hash_final_oid(oid, c);
}
-static void write_object_file_prepare(const struct git_hash_algo *algo,
- const void *buf, unsigned long len,
- enum object_type type, struct object_id *oid,
- char *hdr, int *hdrlen)
+void write_object_file_prepare(const struct git_hash_algo *algo,
+ const void *buf, unsigned long len,
+ enum object_type type, struct object_id *oid,
+ char *hdr, int *hdrlen)
{
struct git_hash_ctx c;
@@ -746,10 +746,10 @@ static int end_loose_object_common(struct odb_source *source,
return Z_OK;
}
-static int write_loose_object(struct odb_source *source,
- const struct object_id *oid, char *hdr,
- int hdrlen, const void *buf, unsigned long len,
- time_t mtime, unsigned flags)
+int write_loose_object(struct odb_source *source,
+ const struct object_id *oid, char *hdr,
+ int hdrlen, const void *buf, unsigned long len,
+ time_t mtime, unsigned flags)
{
int fd, ret;
unsigned char compressed[4096];
@@ -926,48 +926,6 @@ int odb_source_loose_write_stream(struct odb_source *source,
return err;
}
-int odb_source_loose_write_object(struct odb_source *source,
- const void *buf, unsigned long len,
- enum object_type type, struct object_id *oid,
- struct object_id *compat_oid_in,
- enum odb_write_object_flags flags)
-{
- struct odb_source_files *files = odb_source_files_downcast(source);
- const struct git_hash_algo *algo = source->odb->repo->hash_algo;
- const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
- struct object_id compat_oid;
- char hdr[MAX_HEADER_LEN];
- int hdrlen = sizeof(hdr);
-
- /* Generate compat_oid */
- if (compat) {
- if (compat_oid_in)
- oidcpy(&compat_oid, compat_oid_in);
- else if (type == OBJ_BLOB)
- hash_object_file(compat, buf, len, type, &compat_oid);
- else {
- struct strbuf converted = STRBUF_INIT;
- convert_object_file(source->odb->repo, &converted, algo, compat,
- buf, len, type, 0);
- hash_object_file(compat, converted.buf, converted.len,
- type, &compat_oid);
- strbuf_release(&converted);
- }
- }
-
- /* Normally if we have it in the pack then we do not bother writing
- * it out into .git/objects/??/?{38} file.
- */
- write_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);
- if (odb_freshen_object(source->odb, oid))
- return 0;
- if (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))
- return -1;
- if (compat)
- return repo_add_loose_object_map(files->loose, oid, &compat_oid);
- return 0;
-}
-
int force_object_loose(struct odb_source *source,
const struct object_id *oid, time_t mtime)
{
diff --git a/object-file.h b/object-file.h
index 1d90df9d98..2b32592de1 100644
--- a/object-file.h
+++ b/object-file.h
@@ -23,12 +23,6 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa
struct object_info;
struct odb_source;
-int odb_source_loose_write_object(struct odb_source *source,
- const void *buf, unsigned long len,
- enum object_type type, struct object_id *oid,
- struct object_id *compat_oid_in,
- enum odb_write_object_flags flags);
-
int odb_source_loose_write_stream(struct odb_source *source,
struct odb_write_stream *stream, size_t len,
struct object_id *oid);
@@ -129,6 +123,14 @@ int finalize_object_file_flags(struct repository *repo,
void hash_object_file(const struct git_hash_algo *algo, const void *buf,
unsigned long len, enum object_type type,
struct object_id *oid);
+void write_object_file_prepare(const struct git_hash_algo *algo,
+ const void *buf, unsigned long len,
+ enum object_type type, struct object_id *oid,
+ char *hdr, int *hdrlen);
+int write_loose_object(struct odb_source *source,
+ const struct object_id *oid, char *hdr,
+ int hdrlen, const void *buf, unsigned long len,
+ time_t mtime, unsigned flags);
/* Helper to check and "touch" a file */
int check_and_freshen_file(const char *fn, int freshen);
diff --git a/odb/source-files.c b/odb/source-files.c
index ef548e6fe6..52ba04237a 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -164,8 +164,9 @@ static int odb_source_files_write_object(struct odb_source *source,
struct object_id *compat_oid,
enum odb_write_object_flags flags)
{
- return odb_source_loose_write_object(source, buf, len, type,
- oid, compat_oid, flags);
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ return odb_source_write_object(&files->loose->base, buf, len, type,
+ oid, compat_oid, flags);
}
static int odb_source_files_write_object_stream(struct odb_source *source,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index e519365d23..c91018109e 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -5,6 +5,7 @@
#include "hex.h"
#include "loose.h"
#include "object-file.h"
+#include "object-file-convert.h"
#include "odb.h"
#include "odb/source-files.h"
#include "odb/source-loose.h"
@@ -588,6 +589,48 @@ static int odb_source_loose_freshen_object(struct odb_source *source,
return !!check_and_freshen_file(path.buf, 1);
}
+static int odb_source_loose_write_object(struct odb_source *source,
+ const void *buf, unsigned long len,
+ enum object_type type, struct object_id *oid,
+ struct object_id *compat_oid_in,
+ enum odb_write_object_flags flags)
+{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ const struct git_hash_algo *algo = source->odb->repo->hash_algo;
+ const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
+ struct object_id compat_oid;
+ char hdr[MAX_HEADER_LEN];
+ int hdrlen = sizeof(hdr);
+
+ /* Generate compat_oid */
+ if (compat) {
+ if (compat_oid_in)
+ oidcpy(&compat_oid, compat_oid_in);
+ else if (type == OBJ_BLOB)
+ hash_object_file(compat, buf, len, type, &compat_oid);
+ else {
+ struct strbuf converted = STRBUF_INIT;
+ convert_object_file(source->odb->repo, &converted, algo, compat,
+ buf, len, type, 0);
+ hash_object_file(compat, converted.buf, converted.len,
+ type, &compat_oid);
+ strbuf_release(&converted);
+ }
+ }
+
+ /* Normally if we have it in the pack then we do not bother writing
+ * it out into .git/objects/??/?{38} file.
+ */
+ write_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);
+ if (odb_freshen_object(source->odb, oid))
+ return 0;
+ if (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))
+ return -1;
+ if (compat)
+ return repo_add_loose_object_map(loose, oid, &compat_oid);
+ return 0;
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -647,6 +690,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.find_abbrev_len = odb_source_loose_find_abbrev_len;
loose->base.count_objects = odb_source_loose_count_objects;
loose->base.freshen_object = odb_source_loose_freshen_object;
+ loose->base.write_object = odb_source_loose_write_object;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 15/18] object-file: refactor writing objects to use loose source
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (13 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 14/18] odb/source-loose: wire up `write_object()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback Patrick Steinhardt
` (2 subsequent siblings)
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
The "object-file" subsystem still hosts the majority of logic used to
write loose objects. Eventually, we'll want to move this logic into
"odb/source-loose.c", but this isn't yet easily possible because a lot
of the writing logic is still being shared with `force_object_loose()`.
We will eventually detangle this logic so that we can indeed move all of
it into the "loose" source. Meanwhile though, refactor the code so that
it operates on a `struct odb_source_loose` directly to already make the
dependency explicit.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
http-walker.c | 3 ++-
http.c | 6 +++--
object-file.c | 75 +++++++++++++++++++++++++++---------------------------
object-file.h | 6 ++---
odb/source-files.c | 3 ++-
odb/source-loose.c | 9 ++++---
6 files changed, 53 insertions(+), 49 deletions(-)
diff --git a/http-walker.c b/http-walker.c
index 1b6d496548..435a726540 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -539,8 +539,9 @@ static int fetch_object(struct walker *walker, const struct object_id *oid)
} else if (!oideq(&obj_req->oid, &req->real_oid)) {
ret = error("File %s has bad hash", hex);
} else if (req->rename < 0) {
+ struct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);
struct strbuf buf = STRBUF_INIT;
- odb_loose_path(the_repository->objects->sources, &buf, &req->oid);
+ odb_loose_path(files->loose, &buf, &req->oid);
ret = error("unable to write sha1 filename %s", buf.buf);
strbuf_release(&buf);
}
diff --git a/http.c b/http.c
index ea9b16861b..3fcc012233 100644
--- a/http.c
+++ b/http.c
@@ -2826,6 +2826,7 @@ static size_t fwrite_sha1_file(char *ptr, size_t eltsize, size_t nmemb,
struct http_object_request *new_http_object_request(const char *base_url,
const struct object_id *oid)
{
+ struct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);
char *hex = oid_to_hex(oid);
struct strbuf filename = STRBUF_INIT;
struct strbuf prevfile = STRBUF_INIT;
@@ -2840,7 +2841,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
oidcpy(&freq->oid, oid);
freq->localfile = -1;
- odb_loose_path(the_repository->objects->sources, &filename, oid);
+ odb_loose_path(files->loose, &filename, oid);
strbuf_addf(&freq->tmpfile, "%s.temp", filename.buf);
strbuf_addf(&prevfile, "%s.prev", filename.buf);
@@ -2966,6 +2967,7 @@ void process_http_object_request(struct http_object_request *freq)
int finish_http_object_request(struct http_object_request *freq)
{
+ struct odb_source_files *files = odb_source_files_downcast(the_repository->objects->sources);
struct stat st;
struct strbuf filename = STRBUF_INIT;
@@ -2992,7 +2994,7 @@ int finish_http_object_request(struct http_object_request *freq)
unlink_or_warn(freq->tmpfile.buf);
return -1;
}
- odb_loose_path(the_repository->objects->sources, &filename, &freq->oid);
+ odb_loose_path(files->loose, &filename, &freq->oid);
freq->rename = finalize_object_file(the_repository, freq->tmpfile.buf, filename.buf);
strbuf_release(&filename);
diff --git a/object-file.c b/object-file.c
index 7bb5b31bca..bce941874e 100644
--- a/object-file.c
+++ b/object-file.c
@@ -54,14 +54,14 @@ static void fill_loose_path(struct strbuf *buf,
}
}
-const char *odb_loose_path(struct odb_source *source,
+const char *odb_loose_path(struct odb_source_loose *loose,
struct strbuf *buf,
const struct object_id *oid)
{
strbuf_reset(buf);
- strbuf_addstr(buf, source->path);
+ strbuf_addstr(buf, loose->base.path);
strbuf_addch(buf, '/');
- fill_loose_path(buf, oid, source->odb->repo->hash_algo);
+ fill_loose_path(buf, oid, loose->base.odb->repo->hash_algo);
return buf->buf;
}
@@ -575,14 +575,14 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
}
/* Finalize a file on disk, and close it. */
-static void close_loose_object(struct odb_source *source,
+static void close_loose_object(struct odb_source_loose *loose,
int fd, const char *filename)
{
- if (source->will_destroy)
+ if (loose->base.will_destroy)
goto out;
if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
- fsync_loose_object_transaction(source->odb->transaction, fd, filename);
+ fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
else if (fsync_object_files > 0)
fsync_or_die(fd, filename);
else
@@ -651,7 +651,7 @@ static int create_tmpfile(struct repository *repo,
* Returns a "fd", which should later be provided to
* end_loose_object_common().
*/
-static int start_loose_object_common(struct odb_source *source,
+static int start_loose_object_common(struct odb_source_loose *loose,
struct strbuf *tmp_file,
const char *filename, unsigned flags,
git_zstream *stream,
@@ -659,18 +659,18 @@ static int start_loose_object_common(struct odb_source *source,
struct git_hash_ctx *c, struct git_hash_ctx *compat_c,
char *hdr, int hdrlen)
{
- const struct git_hash_algo *algo = source->odb->repo->hash_algo;
- const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
+ const struct git_hash_algo *algo = loose->base.odb->repo->hash_algo;
+ const struct git_hash_algo *compat = loose->base.odb->repo->compat_hash_algo;
int fd;
- fd = create_tmpfile(source->odb->repo, tmp_file, filename);
+ fd = create_tmpfile(loose->base.odb->repo, tmp_file, filename);
if (fd < 0) {
if (flags & ODB_WRITE_OBJECT_SILENT)
return -1;
else if (errno == EACCES)
return error(_("insufficient permission for adding "
"an object to repository database %s"),
- source->path);
+ loose->base.path);
else
return error_errno(
_("unable to create temporary file"));
@@ -700,14 +700,14 @@ static int start_loose_object_common(struct odb_source *source,
* Common steps for the inner git_deflate() loop for writing loose
* objects. Returns what git_deflate() returns.
*/
-static int write_loose_object_common(struct odb_source *source,
+static int write_loose_object_common(struct odb_source_loose *loose,
struct git_hash_ctx *c, struct git_hash_ctx *compat_c,
git_zstream *stream, const int flush,
unsigned char *in0, const int fd,
unsigned char *compressed,
const size_t compressed_len)
{
- const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
+ const struct git_hash_algo *compat = loose->base.odb->repo->compat_hash_algo;
int ret;
ret = git_deflate(stream, flush ? Z_FINISH : 0);
@@ -728,12 +728,12 @@ static int write_loose_object_common(struct odb_source *source,
* - End the compression of zlib stream.
* - Get the calculated oid to "oid".
*/
-static int end_loose_object_common(struct odb_source *source,
+static int end_loose_object_common(struct odb_source_loose *loose,
struct git_hash_ctx *c, struct git_hash_ctx *compat_c,
git_zstream *stream, struct object_id *oid,
struct object_id *compat_oid)
{
- const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
+ const struct git_hash_algo *compat = loose->base.odb->repo->compat_hash_algo;
int ret;
ret = git_deflate_end_gently(stream);
@@ -746,7 +746,7 @@ static int end_loose_object_common(struct odb_source *source,
return Z_OK;
}
-int write_loose_object(struct odb_source *source,
+int write_loose_object(struct odb_source_loose *loose,
const struct object_id *oid, char *hdr,
int hdrlen, const void *buf, unsigned long len,
time_t mtime, unsigned flags)
@@ -760,11 +760,11 @@ int write_loose_object(struct odb_source *source,
static struct strbuf filename = STRBUF_INIT;
if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
- prepare_loose_object_transaction(source->odb->transaction);
+ prepare_loose_object_transaction(loose->base.odb->transaction);
- odb_loose_path(source, &filename, oid);
+ odb_loose_path(loose, &filename, oid);
- fd = start_loose_object_common(source, &tmp_file, filename.buf, flags,
+ fd = start_loose_object_common(loose, &tmp_file, filename.buf, flags,
&stream, compressed, sizeof(compressed),
&c, NULL, hdr, hdrlen);
if (fd < 0)
@@ -776,14 +776,14 @@ int write_loose_object(struct odb_source *source,
do {
unsigned char *in0 = stream.next_in;
- ret = write_loose_object_common(source, &c, NULL, &stream, 1, in0, fd,
+ ret = write_loose_object_common(loose, &c, NULL, &stream, 1, in0, fd,
compressed, sizeof(compressed));
} while (ret == Z_OK);
if (ret != Z_STREAM_END)
die(_("unable to deflate new object %s (%d)"), oid_to_hex(oid),
ret);
- ret = end_loose_object_common(source, &c, NULL, &stream, ¶no_oid, NULL);
+ ret = end_loose_object_common(loose, &c, NULL, &stream, ¶no_oid, NULL);
if (ret != Z_OK)
die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
ret);
@@ -791,7 +791,7 @@ int write_loose_object(struct odb_source *source,
die(_("confused by unstable object source data for %s"),
oid_to_hex(oid));
- close_loose_object(source, fd, tmp_file.buf);
+ close_loose_object(loose, fd, tmp_file.buf);
if (mtime) {
struct utimbuf utb;
@@ -802,16 +802,15 @@ int write_loose_object(struct odb_source *source,
warning_errno(_("failed utime() on %s"), tmp_file.buf);
}
- return finalize_object_file_flags(source->odb->repo, tmp_file.buf, filename.buf,
+ return finalize_object_file_flags(loose->base.odb->repo, tmp_file.buf, filename.buf,
FOF_SKIP_COLLISION_CHECK);
}
-int odb_source_loose_write_stream(struct odb_source *source,
+int odb_source_loose_write_stream(struct odb_source_loose *loose,
struct odb_write_stream *in_stream, size_t len,
struct object_id *oid)
{
- struct odb_source_files *files = odb_source_files_downcast(source);
- const struct git_hash_algo *compat = source->odb->repo->compat_hash_algo;
+ const struct git_hash_algo *compat = loose->base.odb->repo->compat_hash_algo;
struct object_id compat_oid;
int fd, ret, err = 0, flush = 0;
unsigned char compressed[4096];
@@ -825,10 +824,10 @@ int odb_source_loose_write_stream(struct odb_source *source,
int hdrlen;
if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
- prepare_loose_object_transaction(source->odb->transaction);
+ prepare_loose_object_transaction(loose->base.odb->transaction);
/* Since oid is not determined, save tmp file to odb path. */
- strbuf_addf(&filename, "%s/", source->path);
+ strbuf_addf(&filename, "%s/", loose->base.path);
hdrlen = format_object_header(hdr, sizeof(hdr), OBJ_BLOB, len);
/*
@@ -839,7 +838,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
* - Setup zlib stream for compression.
* - Start to feed header to zlib stream.
*/
- fd = start_loose_object_common(source, &tmp_file, filename.buf, 0,
+ fd = start_loose_object_common(loose, &tmp_file, filename.buf, 0,
&stream, compressed, sizeof(compressed),
&c, &compat_c, hdr, hdrlen);
if (fd < 0) {
@@ -867,7 +866,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
if (in_stream->is_finished)
flush = 1;
}
- ret = write_loose_object_common(source, &c, &compat_c, &stream, flush, in0, fd,
+ ret = write_loose_object_common(loose, &c, &compat_c, &stream, flush, in0, fd,
compressed, sizeof(compressed));
/*
* Unlike write_loose_object(), we do not have the entire
@@ -890,16 +889,16 @@ int odb_source_loose_write_stream(struct odb_source *source,
*/
if (ret != Z_STREAM_END)
die(_("unable to stream deflate new object (%d)"), ret);
- ret = end_loose_object_common(source, &c, &compat_c, &stream, oid, &compat_oid);
+ ret = end_loose_object_common(loose, &c, &compat_c, &stream, oid, &compat_oid);
if (ret != Z_OK)
die(_("deflateEnd on stream object failed (%d)"), ret);
- close_loose_object(source, fd, tmp_file.buf);
+ close_loose_object(loose, fd, tmp_file.buf);
- if (odb_freshen_object(source->odb, oid)) {
+ if (odb_freshen_object(loose->base.odb, oid)) {
unlink_or_warn(tmp_file.buf);
goto cleanup;
}
- odb_loose_path(source, &filename, oid);
+ odb_loose_path(loose, &filename, oid);
/* We finally know the object path, and create the missing dir. */
dirlen = directory_size(filename.buf);
@@ -907,7 +906,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
struct strbuf dir = STRBUF_INIT;
strbuf_add(&dir, filename.buf, dirlen);
- if (safe_create_dir_in_gitdir(source->odb->repo, dir.buf) &&
+ if (safe_create_dir_in_gitdir(loose->base.odb->repo, dir.buf) &&
errno != EEXIST) {
err = error_errno(_("unable to create directory %s"), dir.buf);
strbuf_release(&dir);
@@ -916,10 +915,10 @@ int odb_source_loose_write_stream(struct odb_source *source,
strbuf_release(&dir);
}
- err = finalize_object_file_flags(source->odb->repo, tmp_file.buf, filename.buf,
+ err = finalize_object_file_flags(loose->base.odb->repo, tmp_file.buf, filename.buf,
FOF_SKIP_COLLISION_CHECK);
if (!err && compat)
- err = repo_add_loose_object_map(files->loose, oid, &compat_oid);
+ err = repo_add_loose_object_map(loose, oid, &compat_oid);
cleanup:
strbuf_release(&tmp_file);
strbuf_release(&filename);
@@ -957,7 +956,7 @@ int force_object_loose(struct odb_source *source,
oid_to_hex(oid), compat->name);
}
hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
- ret = write_loose_object(source, oid, hdr, hdrlen, buf, len, mtime, 0);
+ ret = write_loose_object(files->loose, oid, hdr, hdrlen, buf, len, mtime, 0);
if (!ret && compat)
ret = repo_add_loose_object_map(files->loose, oid, &compat_oid);
free(buf);
diff --git a/object-file.h b/object-file.h
index 2b32592de1..d30f1b10b2 100644
--- a/object-file.h
+++ b/object-file.h
@@ -23,7 +23,7 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa
struct object_info;
struct odb_source;
-int odb_source_loose_write_stream(struct odb_source *source,
+int odb_source_loose_write_stream(struct odb_source_loose *loose,
struct odb_write_stream *stream, size_t len,
struct object_id *oid);
@@ -31,7 +31,7 @@ int odb_source_loose_write_stream(struct odb_source *source,
* Put in `buf` the name of the file in the local object database that
* would be used to store a loose object with the specified oid.
*/
-const char *odb_loose_path(struct odb_source *source,
+const char *odb_loose_path(struct odb_source_loose *source,
struct strbuf *buf,
const struct object_id *oid);
@@ -127,7 +127,7 @@ void write_object_file_prepare(const struct git_hash_algo *algo,
const void *buf, unsigned long len,
enum object_type type, struct object_id *oid,
char *hdr, int *hdrlen);
-int write_loose_object(struct odb_source *source,
+int write_loose_object(struct odb_source_loose *loose,
const struct object_id *oid, char *hdr,
int hdrlen, const void *buf, unsigned long len,
time_t mtime, unsigned flags);
diff --git a/odb/source-files.c b/odb/source-files.c
index 52ba04237a..2ba1def776 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -174,7 +174,8 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
size_t len,
struct object_id *oid)
{
- return odb_source_loose_write_stream(source, stream, len, oid);
+ struct odb_source_files *files = odb_source_files_downcast(source);
+ return odb_source_loose_write_stream(files->loose, stream, len, oid);
}
static int odb_source_files_begin_transaction(struct odb_source *source,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index c91018109e..da8a60dba1 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -220,7 +220,7 @@ static int odb_source_loose_read_object_info(struct odb_source *source,
if (flags & OBJECT_INFO_SECOND_READ)
return -1;
- odb_loose_path(source, &buf, oid);
+ odb_loose_path(loose, &buf, oid);
return read_object_info_from_path(loose, buf.buf, oid, oi, flags);
}
@@ -238,7 +238,7 @@ static int open_loose_object(struct odb_source_loose *loose,
static struct strbuf buf = STRBUF_INIT;
int fd;
- *path = odb_loose_path(&loose->base, &buf, oid);
+ *path = odb_loose_path(loose, &buf, oid);
fd = git_open(*path);
if (fd >= 0)
return fd;
@@ -584,8 +584,9 @@ static int odb_source_loose_count_objects(struct odb_source *source,
static int odb_source_loose_freshen_object(struct odb_source *source,
const struct object_id *oid)
{
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
static struct strbuf path = STRBUF_INIT;
- odb_loose_path(source, &path, oid);
+ odb_loose_path(loose, &path, oid);
return !!check_and_freshen_file(path.buf, 1);
}
@@ -624,7 +625,7 @@ static int odb_source_loose_write_object(struct odb_source *source,
write_object_file_prepare(algo, buf, len, type, oid, hdr, &hdrlen);
if (odb_freshen_object(source->odb, oid))
return 0;
- if (write_loose_object(source, oid, hdr, hdrlen, buf, len, 0, flags))
+ if (write_loose_object(loose, oid, hdr, hdrlen, buf, len, 0, flags))
return -1;
if (compat)
return repo_add_loose_object_map(loose, oid, &compat_oid);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (14 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 15/18] object-file: refactor writing objects to use loose source Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 17:49 ` Junio C Hamano
2026-05-21 8:22 ` [PATCH 17/18] odb/source-loose: stub out remaining callbacks Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 18/18] odb/source-loose: drop pointer to the "files" source Patrick Steinhardt
17 siblings, 1 reply; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Wire up the `write_object_stream()` callback.
Note that we don't move the implementation into "odb/source-loose.c".
This is because most of the logic to write loose objects is still
contained in "object-file.c", and detangling that requires us to do some
refactorings as explained in the preceding commit. So for now, the
implementation of writing an object stream is still located in
"object-file.c".
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
object-file.h | 12 +++++++++++-
odb/source-files.c | 3 ++-
odb/source-loose.c | 14 ++++++++++++++
3 files changed, 27 insertions(+), 2 deletions(-)
diff --git a/object-file.h b/object-file.h
index d30f1b10b2..b864351372 100644
--- a/object-file.h
+++ b/object-file.h
@@ -23,7 +23,17 @@ int index_path(struct index_state *istate, struct object_id *oid, const char *pa
struct object_info;
struct odb_source;
-int odb_source_loose_write_stream(struct odb_source_loose *loose,
+/*
+ * Write the given stream into the loose object source. The only difference to
+ * the generic implementation of this function is that we don't perform an
+ * object existence check here.
+ *
+ * TODO: We should stop exposing this function altogether and move it into
+ * "odb/source-loose.c". This requires a couple of refactorings though to make
+ * `force_object_loose()` generic and is thus postponed to a later point in
+ * time.
+ */
+int odb_source_loose_write_stream(struct odb_source_loose *source,
struct odb_write_stream *stream, size_t len,
struct object_id *oid);
diff --git a/odb/source-files.c b/odb/source-files.c
index 2ba1def776..83f8066c67 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -7,6 +7,7 @@
#include "odb.h"
#include "odb/source.h"
#include "odb/source-files.h"
+#include "odb/source-loose.h"
#include "packfile.h"
#include "strbuf.h"
#include "write-or-die.h"
@@ -175,7 +176,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
struct object_id *oid)
{
struct odb_source_files *files = odb_source_files_downcast(source);
- return odb_source_loose_write_stream(files->loose, stream, len, oid);
+ return odb_source_write_object_stream(&files->loose->base, stream, len, oid);
}
static int odb_source_files_begin_transaction(struct odb_source *source,
diff --git a/odb/source-loose.c b/odb/source-loose.c
index da8a60dba1..e52fc289a2 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -632,6 +632,19 @@ static int odb_source_loose_write_object(struct odb_source *source,
return 0;
}
+static int odb_source_loose_write_object_stream(struct odb_source *source,
+ struct odb_write_stream *in_stream,
+ size_t len,
+ struct object_id *oid)
+{
+ /*
+ * TODO: the implementation should be moved here, see the comment on
+ * the called function in "object-file.h".
+ */
+ struct odb_source_loose *loose = odb_source_loose_downcast(source);
+ return odb_source_loose_write_stream(loose, in_stream, len, oid);
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -692,6 +705,7 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.count_objects = odb_source_loose_count_objects;
loose->base.freshen_object = odb_source_loose_freshen_object;
loose->base.write_object = odb_source_loose_write_object;
+ loose->base.write_object_stream = odb_source_loose_write_object_stream;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 17/18] odb/source-loose: stub out remaining callbacks
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (15 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 18/18] odb/source-loose: drop pointer to the "files" source Patrick Steinhardt
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Stub out remaining callback functions for the "loose" backend.
Note that we also stub out transactions for loose objects. In fact, we
already have the infrastructure in place for those, and we could in
theory implement those, as well. But there are separate efforts ongoing
to polish up transactional interfaces, and doing so now would likely
result in some messiness. This omission will thus be worked on in a
subsequent patch series, once the dust has settled.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
odb/source-loose.c | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)
diff --git a/odb/source-loose.c b/odb/source-loose.c
index e52fc289a2..e174941318 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -645,6 +645,25 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
return odb_source_loose_write_stream(loose, in_stream, len, oid);
}
+static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
+ struct odb_transaction **out UNUSED)
+{
+ /* TODO: this is a known omission that we'll want to address eventually. */
+ return error("loose source does not support transactions");
+}
+
+static int odb_source_loose_read_alternates(struct odb_source *source UNUSED,
+ struct strvec *out UNUSED)
+{
+ return 0;
+}
+
+static int odb_source_loose_write_alternate(struct odb_source *source UNUSED,
+ const char *alternate UNUSED)
+{
+ return error("loose source does not support alternates");
+}
+
static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
{
oidtree_clear(loose->cache);
@@ -706,6 +725,9 @@ struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
loose->base.freshen_object = odb_source_loose_freshen_object;
loose->base.write_object = odb_source_loose_write_object;
loose->base.write_object_stream = odb_source_loose_write_object_stream;
+ loose->base.begin_transaction = odb_source_loose_begin_transaction;
+ loose->base.read_alternates = odb_source_loose_read_alternates;
+ loose->base.write_alternate = odb_source_loose_write_alternate;
if (!is_absolute_path(loose->base.path))
chdir_notify_register(NULL, odb_source_loose_reparent, loose);
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* [PATCH 18/18] odb/source-loose: drop pointer to the "files" source
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
` (16 preceding siblings ...)
2026-05-21 8:22 ` [PATCH 17/18] odb/source-loose: stub out remaining callbacks Patrick Steinhardt
@ 2026-05-21 8:22 ` Patrick Steinhardt
17 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-21 8:22 UTC (permalink / raw)
To: git
Now that all callbacks of the loose source operate on `struct
odb_source_loose` directly we no longer have to reach into the "files"
source at all.
Drop this field and update `odb_source_loose_new()` to instead accept
all parameters required to initialize itself. This ensures that the
"loose" backend is a fully standalone source.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
odb/source-files.c | 2 +-
odb/source-loose.c | 8 ++++----
odb/source-loose.h | 7 ++++---
3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/odb/source-files.c b/odb/source-files.c
index 83f8066c67..5bdd042922 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -268,7 +268,7 @@ struct odb_source_files *odb_source_files_new(struct object_database *odb,
CALLOC_ARRAY(files, 1);
odb_source_init(&files->base, odb, ODB_SOURCE_FILES, path, local);
- files->loose = odb_source_loose_new(files);
+ files->loose = odb_source_loose_new(odb, path, local);
files->packed = packfile_store_new(&files->base);
files->base.free = odb_source_files_free;
diff --git a/odb/source-loose.c b/odb/source-loose.c
index e174941318..7d7ea2fb84 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -705,14 +705,14 @@ static void odb_source_loose_free(struct odb_source *source)
free(loose);
}
-struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
+struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
+ const char *path,
+ bool local)
{
struct odb_source_loose *loose;
CALLOC_ARRAY(loose, 1);
- odb_source_init(&loose->base, files->base.odb, ODB_SOURCE_LOOSE,
- files->base.path, files->base.local);
- loose->files = files;
+ odb_source_init(&loose->base, odb, ODB_SOURCE_LOOSE, path, local);
loose->base.free = odb_source_loose_free;
loose->base.close = odb_source_loose_close;
diff --git a/odb/source-loose.h b/odb/source-loose.h
index 825e703072..fb75e3bbff 100644
--- a/odb/source-loose.h
+++ b/odb/source-loose.h
@@ -9,11 +9,10 @@ struct oidtree;
/*
* An object database source that stores its objects in loose format, one
- * file per object. This source is part of the files source.
+ * file per object.
*/
struct odb_source_loose {
struct odb_source base;
- struct odb_source_files *files;
/*
* Used to store the results of readdir(3) calls when we are OK
@@ -31,7 +30,9 @@ struct odb_source_loose {
struct loose_object_map *map;
};
-struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files);
+struct odb_source_loose *odb_source_loose_new(struct object_database *odb,
+ const char *path,
+ bool local);
/*
* Cast the given object database source to the loose backend. This will cause
--
2.54.0.926.g75ba10bac6.dirty
^ permalink raw reply related [flat|nested] 22+ messages in thread
* Re: [PATCH 03/18] odb/source-loose: start converting to a proper `struct odb_source`
2026-05-21 8:22 ` [PATCH 03/18] odb/source-loose: start converting to a proper `struct odb_source` Patrick Steinhardt
@ 2026-05-21 15:49 ` Junio C Hamano
0 siblings, 0 replies; 22+ messages in thread
From: Junio C Hamano @ 2026-05-21 15:49 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: git
Patrick Steinhardt <ps@pks.im> writes:
> Start converting `struct odb_source_loose` into a proper pluggable
> `struct odb_source` by embedding the base struct and assigning it the
> new `ODB_SOURCE_LOOSE` type. Furthermore, wire up lifecycle management
> of this source by implementing the `free` callback and taking ownership
> of the chdir notifications.
>
> Note that the loose source is not yet functional as a standalone `struct
> odb_source`, as it's missing all of the callback implementations. These
> will be wired up in subsequent commits.
>
> Signed-off-by: Patrick Steinhardt <ps@pks.im>
> ---
> object-file.c | 17 -----------------
> object-file.h | 2 --
> odb/source-files.c | 2 +-
> odb/source-loose.c | 45 +++++++++++++++++++++++++++++++++++++++++++++
> odb/source-loose.h | 14 ++++++++++++++
> odb/source.h | 3 +++
> 6 files changed, 63 insertions(+), 20 deletions(-)
>
> diff --git a/object-file.c b/object-file.c
> index 7a1908bfc0..977d959d33 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -2041,14 +2041,6 @@ static struct oidtree *odb_source_loose_cache(struct odb_source *source,
> return files->loose->cache;
> }
>
> -static void odb_source_loose_clear_cache(struct odb_source_loose *loose)
> -{
> - oidtree_clear(loose->cache);
> - FREE_AND_NULL(loose->cache);
> - memset(&loose->subdir_seen, 0,
> - sizeof(loose->subdir_seen));
> -}
> -
> void odb_source_loose_reprepare(struct odb_source *source)
> {
> struct odb_source_files *files = odb_source_files_downcast(source);
> @@ -2205,15 +2197,6 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
> return &transaction->base;
> }
>
> -void odb_source_loose_free(struct odb_source_loose *loose)
> -{
> - if (!loose)
> - return;
> - odb_source_loose_clear_cache(loose);
> - loose_object_map_clear(&loose->map);
> - free(loose);
> -}
> -
> struct odb_loose_read_stream {
> struct odb_read_stream base;
> git_zstream z;
> diff --git a/object-file.h b/object-file.h
> index 1d8312cf7f..02c9680980 100644
> --- a/object-file.h
> +++ b/object-file.h
> @@ -21,8 +21,6 @@ struct object_info;
> struct odb_read_stream;
> struct odb_source;
>
> -void odb_source_loose_free(struct odb_source_loose *loose);
> -
> /* Reprepare the loose source by emptying the loose object cache. */
> void odb_source_loose_reprepare(struct odb_source *source);
>
> diff --git a/odb/source-files.c b/odb/source-files.c
> index 185cc6903e..ccc637311b 100644
> --- a/odb/source-files.c
> +++ b/odb/source-files.c
> @@ -27,7 +27,7 @@ static void odb_source_files_free(struct odb_source *source)
> {
> struct odb_source_files *files = odb_source_files_downcast(source);
> chdir_notify_unregister(NULL, odb_source_files_reparent, files);
> - odb_source_loose_free(files->loose);
> + odb_source_free(&files->loose->base);
> packfile_store_free(files->packed);
> odb_source_release(&files->base);
> free(files);
> diff --git a/odb/source-loose.c b/odb/source-loose.c
> index c9e7414814..92e18f5adb 100644
> --- a/odb/source-loose.c
> +++ b/odb/source-loose.c
> @@ -1,10 +1,55 @@
> #include "git-compat-util.h"
> +#include "abspath.h"
> +#include "chdir-notify.h"
> +#include "loose.h"
> +#include "odb.h"
> +#include "odb/source-files.h"
> #include "odb/source-loose.h"
> +#include "oidtree.h"
> +
> +void odb_source_loose_clear_cache(struct odb_source_loose *loose)
> +{
> + oidtree_clear(loose->cache);
> + FREE_AND_NULL(loose->cache);
> + memset(&loose->subdir_seen, 0,
> + sizeof(loose->subdir_seen));
> +}
> +
> +static void odb_source_loose_reparent(const char *name UNUSED,
> + const char *old_cwd,
> + const char *new_cwd,
> + void *cb_data)
> +{
> + struct odb_source_loose *loose = cb_data;
> + char *path = reparent_relative_path(old_cwd, new_cwd,
> + loose->base.path);
> + free(loose->base.path);
> + loose->base.path = path;
> +}
> +
> +static void odb_source_loose_free(struct odb_source *source)
> +{
> + struct odb_source_loose *loose = odb_source_loose_downcast(source);
> + odb_source_loose_clear_cache(loose);
> + loose_object_map_clear(&loose->map);
> + chdir_notify_unregister(NULL, odb_source_loose_reparent, loose);
> + odb_source_release(&loose->base);
> + free(loose);
> +}
>
> struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files)
> {
> struct odb_source_loose *loose;
> +
> CALLOC_ARRAY(loose, 1);
> + odb_source_init(&loose->base, files->base.odb, ODB_SOURCE_LOOSE,
> + files->base.path, files->base.local);
> loose->files = files;
> +
> + loose->base.free = odb_source_loose_free;
> +
> + if (!is_absolute_path(loose->base.path))
> + chdir_notify_register(NULL, odb_source_loose_reparent, loose);
> +
> return loose;
> }
> diff --git a/odb/source-loose.h b/odb/source-loose.h
> index bf61e767c8..441da9e418 100644
> --- a/odb/source-loose.h
> +++ b/odb/source-loose.h
> @@ -12,6 +12,7 @@ struct oidtree;
> * file per object. This source is part of the files source.
> */
> struct odb_source_loose {
> + struct odb_source base;
> struct odb_source_files *files;
>
> /*
> @@ -32,4 +33,17 @@ struct odb_source_loose {
>
> struct odb_source_loose *odb_source_loose_new(struct odb_source_files *files);
>
> +/*
> + * Cast the given object database source to the loose backend. This will cause
> + * a BUG in case the source uses doesn't use this backend.
"uses doesn't use"???
> + */
> +static inline struct odb_source_loose *odb_source_loose_downcast(struct odb_source *source)
> +{
> + if (source->type != ODB_SOURCE_LOOSE)
> + BUG("trying to downcast source of type '%d' to loose", source->type);
> + return container_of(source, struct odb_source_loose, base);
> +}
> +
> +void odb_source_loose_clear_cache(struct odb_source_loose *loose);
> +
> #endif
> diff --git a/odb/source.h b/odb/source.h
> index 0a440884e4..8bcb67787e 100644
> --- a/odb/source.h
> +++ b/odb/source.h
> @@ -14,6 +14,9 @@ enum odb_source_type {
> /* The "files" backend that uses loose objects and packfiles. */
> ODB_SOURCE_FILES,
>
> + /* The "loose" backend that uses loose objects, only. */
> + ODB_SOURCE_LOOSE,
> +
> /* The "in-memory" backend that stores objects in memory. */
> ODB_SOURCE_INMEMORY,
> };
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback
2026-05-21 8:22 ` [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback Patrick Steinhardt
@ 2026-05-21 17:49 ` Junio C Hamano
2026-05-22 6:12 ` Patrick Steinhardt
0 siblings, 1 reply; 22+ messages in thread
From: Junio C Hamano @ 2026-05-21 17:49 UTC (permalink / raw)
To: Patrick Steinhardt; +Cc: git
Patrick Steinhardt <ps@pks.im> writes:
> -int odb_source_loose_write_stream(struct odb_source_loose *loose,
> +/*
> + * Write the given stream into the loose object source. The only difference to
> + * the generic implementation of this function is that we don't perform an
"difference to" -> "difference from"???
^ permalink raw reply [flat|nested] 22+ messages in thread
* Re: [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback
2026-05-21 17:49 ` Junio C Hamano
@ 2026-05-22 6:12 ` Patrick Steinhardt
0 siblings, 0 replies; 22+ messages in thread
From: Patrick Steinhardt @ 2026-05-22 6:12 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Fri, May 22, 2026 at 02:49:24AM +0900, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
>
> > -int odb_source_loose_write_stream(struct odb_source_loose *loose,
> > +/*
> > + * Write the given stream into the loose object source. The only difference to
> > + * the generic implementation of this function is that we don't perform an
>
> "difference to" -> "difference from"???
I guess this is a difference between American and British English. "to"
is more popular in British English, but basically not used at all in
American English. Will adapt, thanks.
Patrick
^ permalink raw reply [flat|nested] 22+ messages in thread
end of thread, other threads:[~2026-05-22 6:12 UTC | newest]
Thread overview: 22+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-05-21 8:22 [PATCH 00/18] odb: make loose object source a proper `struct odb_source` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 01/18] odb/source-loose: move loose source into "odb/" subsystem Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 02/18] odb/source-loose: store pointer to "files" instead of generic source Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 03/18] odb/source-loose: start converting to a proper `struct odb_source` Patrick Steinhardt
2026-05-21 15:49 ` Junio C Hamano
2026-05-21 8:22 ` [PATCH 04/18] odb/source-loose: wire up `reprepare()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 05/18] odb/source-loose: wire up `close()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 06/18] odb/source-loose: wire up `read_object_info()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 07/18] odb/source-loose: wire up `read_object_stream()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 08/18] odb/source-loose: wire up `for_each_object()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 09/18] odb/source-loose: wire up `find_abbrev_len()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 10/18] odb/source-loose: wire up `count_objects()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 11/18] odb/source-loose: drop `odb_source_loose_has_object()` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 12/18] odb/source-loose: wire up `freshen_object()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 13/18] loose: refactor object map to operate on `struct odb_source_loose` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 14/18] odb/source-loose: wire up `write_object()` callback Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 15/18] object-file: refactor writing objects to use loose source Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 16/18] odb/source-loose: wire up `write_object_stream()` callback Patrick Steinhardt
2026-05-21 17:49 ` Junio C Hamano
2026-05-22 6:12 ` Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 17/18] odb/source-loose: stub out remaining callbacks Patrick Steinhardt
2026-05-21 8:22 ` [PATCH 18/18] odb/source-loose: drop pointer to the "files" source Patrick Steinhardt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox