[PATCH 0/6] receive-pack: use ODB transactions to stage object writes

[PATCH 0/6] receive-pack: use ODB transactions to stage object writes

[Justin Tobler]

Greetings,

This patch series replaces direct usage of the `tmp_objdir` interfaces
in git-receive-pack(1) to instead use the `odb_transaction` interfaces
to create/manage a staging area to write objects to. The purpose of this
change is to get git-receive-pack(1) one step closer to being ODB
backend agnostic. For now, the object writes themselves are still
"files" backend specific due to being handled by the git-index-pack(1)
and git-unpack-objects(1) child processes. This will be tackled in a
separate series though.

Thanks,
-Justin

Justin Tobler (6):
  object-file: rename files transaction prepare function
  object-file: propagate files transaction errors
  odb/transaction: propagate begin errors
  odb/transaction: propagate commit errors
  odb/transaction: add transaction env interface
  builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |  2 +-
 builtin/receive-pack.c   | 46 ++++++++++--------------
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 77 +++++++++++++++++++++++++++++++---------
 object-file.h            |  7 ++--
 odb/source-files.c       |  9 ++---
 odb/source-inmemory.c    |  3 +-
 odb/source-loose.c       |  3 +-
 odb/source.h             |  9 +++--
 odb/transaction.c        | 38 +++++++++++++++-----
 odb/transaction.h        | 49 +++++++++++++++++++++----
 read-cache.c             |  2 +-
 14 files changed, 173 insertions(+), 78 deletions(-)


base-commit: ab776a62a78576513ee121424adb19597fbb7613
-- 
2.54.0.105.g59ff4886a5

[PATCH 1/6] object-file: rename files transaction prepare function

[Justin Tobler]

The "files" ODB transaction backend lazily creates a temporary object
directory when the first loose object is written to the transaction via
`prepare_loose_object_transaction()`. In a subsequent commit, the
temporary directory is used to also write packfiles to.

Rename the function to `odb_transaction_files_prepare()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/object-file.c b/object-file.c
index e3d92bbda2..a3eb8d71dd 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
-- 
2.54.0.105.g59ff4886a5

[PATCH 2/6] object-file: propagate files transaction errors

[Justin Tobler]

The "files" transaction backend may encounter errors related to managing
the temporary directory used to stage objects, but silently ignores
these errors. Instead return errors encountered in the
`odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
callers to handle as needed.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c      | 41 ++++++++++++++++++++++++++++-------------
 object-file.h      |  5 +++--
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  2 +-
 4 files changed, 33 insertions(+), 21 deletions(-)

diff --git a/object-file.c b/object-file.c
index a3eb8d71dd..18c2df75fb 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void fsync_loose_object_transaction(struct odb_transaction *base,
@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 /*
  * Cleanup after batch-mode fsync_object_files.
  */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
 {
 	struct strbuf temp_path = STRBUF_INIT;
 	struct tempfile *temp;
 
 	if (!transaction->objdir)
-		return;
+		return 0;
 
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
@@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	 * Make the object files visible in the primary ODB after their data is
 	 * fully durable.
 	 */
-	tmp_objdir_migrate(transaction->objdir);
+	if (tmp_objdir_migrate(transaction->objdir))
+		return -1;
+
 	transaction->objdir = NULL;
+
+	return 0;
 }
 
 /* Finalize a file on disk, and close it. */
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
 
-	if (odb->transaction)
-		return NULL;
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
diff --git a/object-file.h b/object-file.h
index 528c4e6e69..ac927fec07 100644
--- a/object-file.h
+++ b/object-file.h
@@ -195,8 +195,9 @@ struct odb_transaction;
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * pending, out is set to NULL.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 5bdd042922..2545bd81d4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/transaction.h b/odb/transaction.h
index 854fda06f5..f4c1ebfaaa 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -17,7 +17,7 @@ struct odb_transaction {
 	struct odb_source *source;
 
 	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
-- 
2.54.0.105.g59ff4886a5

[PATCH 3/6] odb/transaction: propagate begin errors

[Justin Tobler]

When `odb_transaction_begin()` is invoked, the function returns the
transaction pointer directly. There is no way for the backend to
signal that it failed to set up its state, such as when creating the
temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB
transactions and needs to be able to report such failures rather
than silently ignore them. Refactor `odb_transaction_begin()` to
return an int error code and write the resulting transaction into an
out parameter. Also introduce `odb_transaction_begin_or_die()` as a
convenience for callsites that do not need to handle errors
explicitly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            |  3 ++-
 odb/transaction.c        | 16 +++++++++++-----
 odb/transaction.h        | 19 +++++++++++++++----
 read-cache.c             |  2 +-
 8 files changed, 33 insertions(+), 15 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index c859f66519..3d5d9cfdb9 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index f3849bb654..d0136cdd99 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 3d6646c318..17f3ea284c 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 184f7e2635..1a7dfed9cf 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -490,7 +490,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	odb_transaction_commit(transaction);
diff --git a/object-file.c b/object-file.c
index 18c2df75fb..696f05dc2d 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1389,8 +1389,9 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction;
 
+			odb_transaction_begin_or_die(odb, &transaction);
 			ret = odb_transaction_write_object_stream(odb->transaction,
 								  &stream,
 								  xsize_t(st->st_size),
diff --git a/odb/transaction.c b/odb/transaction.c
index b16e07aebf..d3de01db50 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -2,14 +2,20 @@
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
-	if (odb->transaction)
-		return NULL;
+	int ret;
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}
 
-	return odb->transaction;
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;
+
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
diff --git a/odb/transaction.h b/odb/transaction.h
index f4c1ebfaaa..cd6d50f2e5 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -1,6 +1,8 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "git-compat-util.h"
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -33,11 +35,20 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
diff --git a/read-cache.c b/read-cache.c
index 21ca58beea..db0bfa60fe 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4042,7 +4042,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	odb_transaction_commit(transaction);
 
-- 
2.54.0.105.g59ff4886a5

[PATCH 4/6] odb/transaction: propagate commit errors

[Justin Tobler]

When `odb_transaction_commit()` is invoked, the return value of the
backend commit callback is silently discarded. A backend has no way
to signal that committing failed, such as when the "files" backend
cannot migrate its temporary object directory into the permanent
ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction
to stage objects and consequently cares about such failures so it can
handle the error appropriately. Change the commit callback signature to
return an int error code and have `odb_transaction_commit()` forward it
accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 odb/transaction.c | 13 ++++++++++---
 odb/transaction.h |  2 +-
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/odb/transaction.c b/odb/transaction.c
index d3de01db50..b20d6a16f8 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -18,19 +18,26 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
+	if (ret)
+		return ret;
+
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return 0;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
diff --git a/odb/transaction.h b/odb/transaction.h
index cd6d50f2e5..7898770071 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
  * Commits an ODB transaction making the written objects visible. If the
  * specified transaction is NULL, the function is a no-op.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
-- 
2.54.0.105.g59ff4886a5

[PATCH 5/6] odb/transaction: add transaction env interface

[Justin Tobler]

The ODB transaction backend is responsible for creating/managing its own
staging area for writing objects. Other child processes spawned by Git
may need to access to uncommitted objects or write new objects in the
staging area though.

Introduce `odb_transaction_env()` which is expected to provide the set
of environment variables needed by a child process to access the
transaction staging area.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c     | 11 +++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 19 +++++++++++++++++++
 3 files changed, 38 insertions(+)

diff --git a/object-file.c b/object-file.c
index 696f05dc2d..14064d188a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1691,6 +1691,16 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static const char **odb_transaction_files_env(struct odb_transaction *base)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+
+	return tmp_objdir_env(transaction->objdir);
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1706,6 +1716,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
diff --git a/odb/transaction.c b/odb/transaction.c
index b20d6a16f8..20d3f43f54 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -46,3 +46,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+const char **odb_transaction_env(struct odb_transaction *transaction)
+{
+	if (!transaction)
+		return NULL;
+
+	return transaction->env(transaction);
+}
diff --git a/odb/transaction.h b/odb/transaction.h
index 7898770071..536458297b 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -32,6 +32,16 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to return a NULL-terminated array of
+	 * environment variables that a child process should inherit so
+	 * that its object writes participate in the transaction. The
+	 * returned array is owned by the backend and remains valid until
+	 * the transaction ends. May return NULL when the backend does not
+	 * need to expose any state to child processes.
+	 */
+	const char **(*env)(struct odb_transaction *transaction);
 };
 
 /*
@@ -65,4 +75,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Returns a NULL-terminated array of environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for passing via child_process.env. Returns NULL if
+ * the transaction is NULL or the backend does not expose any state to child
+ * processes.
+ */
+const char **odb_transaction_env(struct odb_transaction *transaction);
+
 #endif
-- 
2.54.0.105.g59ff4886a5

[PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

[Justin Tobler]

Objects received by git-receive-pack(1) are quarantined in a temporary
"incoming" directory and migrated into the object database prior to the
reference updates. The quarantine is currently managed through
`tmp_objdir` directly. In a pluggable ODB future, how exactly an object
gets written to a transaction may vary for a given ODB source. Refactor
git-receive-pack(1) to use the ODB transaction interfaces to manage the
object staging area in a more agnostic manner accordingly.

Note that the temporary directory created for git-receive-pack(1) is
eagerly created and uses a different prefix name. This behavior is
special cased in the "files" backend by having `odb_transaction_begin()`
callers that require this behavior provide an `ODB_TRANSACTION_RECEIVE`
flag.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/receive-pack.c   | 46 ++++++++++++++++------------------------
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 22 ++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 +++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 13 ++++++++----
 read-cache.c             |  2 +-
 14 files changed, 70 insertions(+), 50 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 3d5d9cfdb9..60ffbede2b 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 19eb6a1b61..ee8e03e2ab 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -112,8 +112,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (the_repository->objects->transaction)
+		strvec_pushv(&opt.env, odb_transaction_env(the_repository->objects->transaction));
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1363,7 +1361,7 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
+	opt.env = odb_transaction_env(the_repository->objects->transaction);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
@@ -1802,7 +1800,7 @@ static void set_connectivity_errors(struct command *commands,
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		opt.env = odb_transaction_env(the_repository->objects->transaction);
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
@@ -2057,7 +2055,7 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		opt.env = odb_transaction_env(the_repository->objects->transaction);
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
@@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(the_repository->objects->transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	strvec_pushv(&child.env, odb_transaction_env(transaction));
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2431,13 +2416,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2446,7 +2432,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2623,6 +2609,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start ODB transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index d0136cdd99..c3d0fc7507 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 17f3ea284c..bf6ea60ef4 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 1a7dfed9cf..ed05acc4c7 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -490,7 +490,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	odb_transaction_commit(transaction);
diff --git a/object-file.c b/object-file.c
index 14064d188a..e7958753ec 100644
--- a/object-file.c
+++ b/object-file.c
@@ -497,6 +497,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -513,7 +514,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return -1;
 
@@ -1391,7 +1392,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			struct object_database *odb = the_repository->objects;
 			struct odb_transaction *transaction;
 
-			odb_transaction_begin_or_die(odb, &transaction);
+			odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(odb->transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base)
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different prefix.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
diff --git a/object-file.h b/object-file.h
index ac927fec07..fe098d54cb 100644
--- a/object-file.h
+++ b/object-file.h
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -198,6 +199,7 @@ struct odb_transaction;
  * pending, out is set to NULL.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 2545bd81d4..534f48aad9 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c
index e004566d76..9644d9d474 100644
--- a/odb/source-inmemory.c
+++ b/odb/source-inmemory.c
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 66e6bb8d3f..57c91986b4 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
diff --git a/odb/source.h b/odb/source.h
index 2192a101b8..3790d03ff2 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
diff --git a/odb/transaction.c b/odb/transaction.c
index 20d3f43f54..34c212020c 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -3,7 +3,8 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
@@ -12,7 +13,7 @@ int odb_transaction_begin(struct object_database *odb,
 		return 0;
 	}
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	odb->transaction = *out;
 
 	return ret;
diff --git a/odb/transaction.h b/odb/transaction.h
index 536458297b..78392ff13d 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -4,7 +4,6 @@
 #include "git-compat-util.h"
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,10 @@ struct odb_transaction {
 	const char **(*env)(struct odb_transaction *transaction);
 };
 
+enum odb_transaction_flags {
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -51,12 +54,14 @@ struct odb_transaction {
  * error. If the ODB already has a pending transaction, `out` is set to NULL.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
diff --git a/read-cache.c b/read-cache.c
index db0bfa60fe..35bfb25576 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4042,7 +4042,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	odb_transaction_commit(transaction);
 
-- 
2.54.0.105.g59ff4886a5

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Patrick Steinhardt]

On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> The "files" transaction backend may encounter errors related to managing
> the temporary directory used to stage objects, but silently ignores
> these errors. Instead return errors encountered in the
> `odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
> callers to handle as needed.

Missing a then? "to handle as needed" -> "to handle them as needed"

Makes sense. It always felt a bit off that those functions didn't have a
way to signal errors to the caller.

> diff --git a/object-file.c b/object-file.c
> index a3eb8d71dd..18c2df75fb 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -499,7 +499,7 @@ struct odb_transaction_files {
>  	struct transaction_packfile packfile;
>  };
>  
> -static void odb_transaction_files_prepare(struct odb_transaction *base)
> +static int odb_transaction_files_prepare(struct odb_transaction *base)
>  {
>  	struct odb_transaction_files *transaction =
>  		container_of_or_null(base, struct odb_transaction_files, base);

By the way, is there any reason why those functions are still hosted in
"object-file.c" instead of in "odb/source-files.c"? I should probably
know, but I forgot.

> @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
>  	 * added at the time they call odb_transaction_files_begin.
>  	 */
>  	if (!transaction || transaction->objdir)
> -		return;
> +		return 0;
>  
>  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> -	if (transaction->objdir)
> -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> +	if (!transaction->objdir)
> +		return -1;

Huh. So previously we just didn't handle this error at all and just
continued to tag along? Did that result in anything sensible or was this
just YOLOing it?

> @@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
>  /*
>   * Cleanup after batch-mode fsync_object_files.
>   */
> -static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
> +static int flush_loose_object_transaction(struct odb_transaction_files *transaction)

Feels like this function should've been renamed in the preceding commit,
as well.

>  {
>  	struct strbuf temp_path = STRBUF_INIT;
>  	struct tempfile *temp;
>  
>  	if (!transaction->objdir)
> -		return;
> +		return 0;
>  
>  	/*
>  	 * Issue a full hardware flush against a temporary file to ensure
> @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac

There is a call to `xmks_tempfile()` hidden that can fail, but that
failure is already handled in that function itself by dying.

>  	 * Make the object files visible in the primary ODB after their data is
>  	 * fully durable.
>  	 */
> -	tmp_objdir_migrate(transaction->objdir);
> +	if (tmp_objdir_migrate(transaction->objdir))
> +		return -1;

Feels like another case of YOLOing it. The migration could have failed,
but we just ignored that failure and never told the user about it. The
result may be silent corruption, I assume?

> @@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
>  	return ret;
>  }
>  
> -static void odb_transaction_files_commit(struct odb_transaction *base)
> +static int odb_transaction_files_commit(struct odb_transaction *base)
>  {
>  	struct odb_transaction_files *transaction =
>  		container_of(base, struct odb_transaction_files, base);
>  
> -	flush_loose_object_transaction(transaction);
> +	if (flush_loose_object_transaction(transaction))
> +		return -1;
>  	flush_packfile_transaction(transaction);
> +
> +	return 0;
>  }
>  
> -struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
> +int odb_transaction_files_begin(struct odb_source *source,
> +				struct odb_transaction **out)
>  {
>  	struct odb_transaction_files *transaction;
>  	struct object_database *odb = source->odb;
>  
> -	if (odb->transaction)
> -		return NULL;
> +	if (odb->transaction) {
> +		*out = NULL;
> +		return 0;
> +	}
>  
>  	transaction = xcalloc(1, sizeof(*transaction));
>  	transaction->base.source = source;
>  	transaction->base.commit = odb_transaction_files_commit;
>  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
> +	*out = &transaction->base;
>  
> -	return &transaction->base;
> +	return 0;
>  }

It's still somewhat fishy that we have this ODB-level transaction, but
that's a preexisting issue and thus outside the scope of this patch
series. Ideally though, it would be possible for there to be multiple
transactions, and it would be the caller's responsibility for juggling
these transactions. Just as it happens with reference transactions.

> diff --git a/odb/transaction.h b/odb/transaction.h
> index 854fda06f5..f4c1ebfaaa 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -17,7 +17,7 @@ struct odb_transaction {
>  	struct odb_source *source;
>  
>  	/* The ODB source specific callback invoked to commit a transaction. */
> -	void (*commit)(struct odb_transaction *transaction);
> +	int (*commit)(struct odb_transaction *transaction);

We might want to document the returned error code here.

Patrick

Re: [PATCH 3/6] odb/transaction: propagate begin errors

[Patrick Steinhardt]

On Tue, Jun 23, 2026 at 11:19:17PM -0500, Justin Tobler wrote:
> diff --git a/odb/transaction.c b/odb/transaction.c
> index b16e07aebf..d3de01db50 100644
> --- a/odb/transaction.c
> +++ b/odb/transaction.c
> @@ -2,14 +2,20 @@
>  #include "odb/source.h"
>  #include "odb/transaction.h"
>  
> -struct odb_transaction *odb_transaction_begin(struct object_database *odb)
> +int odb_transaction_begin(struct object_database *odb,
> +			  struct odb_transaction **out)
>  {
> -	if (odb->transaction)
> -		return NULL;
> +	int ret;
>  
> -	odb_source_begin_transaction(odb->sources, &odb->transaction);
> +	if (odb->transaction) {
> +		*out = NULL;
> +		return 0;
> +	}

Hm. So we may return successful, but not set the `out` pointer to a
transaction. And...

> diff --git a/odb/transaction.h b/odb/transaction.h
> index f4c1ebfaaa..cd6d50f2e5 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -33,11 +35,20 @@ struct odb_transaction {
>  };
>  
>  /*
> - * Starts an ODB transaction. Subsequent objects are written to the transaction
> - * and not committed until odb_transaction_commit() is invoked on the
> - * transaction. If the ODB already has a pending transaction, NULL is returned.
> + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
> + * written to the transaction and not committed until odb_transaction_commit()
> + * is invoked on the transaction. Returns 0 on success and a negative value on
> + * error. If the ODB already has a pending transaction, `out` is set to NULL.
>   */
> -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
> +int odb_transaction_begin(struct object_database *odb,
> +			  struct odb_transaction **out);
> +
> +static inline void odb_transaction_begin_or_die(struct object_database *odb,
> +						struct odb_transaction **out)
> +{
> +	if (odb_transaction_begin(odb, out))
> +		die(_("failed to start ODB transaction"));
> +}

... we don't special-case that here, either. So a caller may invoke the
function, not die, but it might still not have a valid transaction. That
feels wrong to me.

Patrick

Re: [PATCH 4/6] odb/transaction: propagate commit errors

[Patrick Steinhardt]

On Tue, Jun 23, 2026 at 11:19:18PM -0500, Justin Tobler wrote:
> diff --git a/odb/transaction.h b/odb/transaction.h
> index cd6d50f2e5..7898770071 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
>   * Commits an ODB transaction making the written objects visible. If the
>   * specified transaction is NULL, the function is a no-op.
>   */
> -void odb_transaction_commit(struct odb_transaction *transaction);
> +int odb_transaction_commit(struct odb_transaction *transaction);

Should the function comment be amended, as well? We should definitely
point out that calling this with a NULL transaction also returns
success.

Patrick

Re: [PATCH 5/6] odb/transaction: add transaction env interface

[Patrick Steinhardt]

On Tue, Jun 23, 2026 at 11:19:19PM -0500, Justin Tobler wrote:
> The ODB transaction backend is responsible for creating/managing its own
> staging area for writing objects. Other child processes spawned by Git
> may need to access to uncommitted objects or write new objects in the

s/may need to access to/may need access to/

> staging area though.
> 
> Introduce `odb_transaction_env()` which is expected to provide the set
> of environment variables needed by a child process to access the
> transaction staging area.

Possessive s is missing, I think.

> diff --git a/object-file.c b/object-file.c
> index 696f05dc2d..14064d188a 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1691,6 +1691,16 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
>  	return 0;
>  }
>  
> +static const char **odb_transaction_files_env(struct odb_transaction *base)
> +{
> +	struct odb_transaction_files *transaction =
> +		container_of(base, struct odb_transaction_files, base);
> +
> +	odb_transaction_files_prepare(&transaction->base);
> +
> +	return tmp_objdir_env(transaction->objdir);
> +}
> +
>  int odb_transaction_files_begin(struct odb_source *source,
>  				struct odb_transaction **out)
>  {

Makes sense. Transactions may have a different way to quarantine the
write than using a quarantine directory. So making this functionality
pluggable so that backends may expose a separate set of environment
variables feels sensible.

> diff --git a/odb/transaction.h b/odb/transaction.h
> index 7898770071..536458297b 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -32,6 +32,16 @@ struct odb_transaction {
>  	int (*write_object_stream)(struct odb_transaction *transaction,
>  				   struct odb_write_stream *stream, size_t len,
>  				   struct object_id *oid);
> +
> +	/*
> +	 * This callback is expected to return a NULL-terminated array of
> +	 * environment variables that a child process should inherit so
> +	 * that its object writes participate in the transaction. The
> +	 * returned array is owned by the backend and remains valid until
> +	 * the transaction ends. May return NULL when the backend does not
> +	 * need to expose any state to child processes.
> +	 */
> +	const char **(*env)(struct odb_transaction *transaction);

Would it make more sense to adapt this function so that:

  - It receives a `struct strvec` as input that the environment
    variables are to be amended to.

  - It returns a normal error code to indicate errors?

Patrick

Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

[Patrick Steinhardt]

On Tue, Jun 23, 2026 at 11:19:20PM -0500, Justin Tobler wrote:
> Objects received by git-receive-pack(1) are quarantined in a temporary
> "incoming" directory and migrated into the object database prior to the
> reference updates. The quarantine is currently managed through
> `tmp_objdir` directly. In a pluggable ODB future, how exactly an object
> gets written to a transaction may vary for a given ODB source. Refactor
> git-receive-pack(1) to use the ODB transaction interfaces to manage the
> object staging area in a more agnostic manner accordingly.
> 
> Note that the temporary directory created for git-receive-pack(1) is
> eagerly created and uses a different prefix name. This behavior is

A different prefix name compared to what?

> special cased in the "files" backend by having `odb_transaction_begin()`
> callers that require this behavior provide an `ODB_TRANSACTION_RECEIVE`
> flag.

Okay. I guess this is to retain existing behaviour where the temporary
directory is created lazily everywhere else. Makes me wonder whether we
should eventually change this to just unconditionally create the
directory in all cases so that we can drop this new flag.

It might've also made sense to split this commit up into two: one to
introduce the flag parameter, and then one to do the changes to
git-receive-pack(1).

> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index 19eb6a1b61..ee8e03e2ab 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -112,8 +112,6 @@ static enum {
>  } use_keepalive;
>  static int keepalive_in_sec = 5;
>  
> -static struct tmp_objdir *tmp_objdir;
> -
>  static struct proc_receive_ref {
>  	unsigned int want_add:1,
>  		     want_delete:1,

I assume the goal is that we convert all other users of the tmp-objdir
subsystem to also use transactions eventually, so that this becomes an
implementation detail fo the files transaction?

> @@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,
>  	 * Now we'll start writing out refs, which means the objects need
>  	 * to be in their final positions so that other processes can see them.
>  	 */
> -	if (tmp_objdir_migrate(tmp_objdir) < 0) {
> +	if (odb_transaction_commit(the_repository->objects->transaction)) {
>  		for (cmd = commands; cmd; cmd = cmd->next) {
>  			if (!cmd->error_string)
>  				cmd->error_string = "unable to migrate objects to permanent storage";
>  		}
>  		return;
>  	}
> -	tmp_objdir = NULL;

We don't need to unset the transaction because that's what
`odb_transaction_commit()` already does for us, I assume?

> @@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
>  		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
>  }
>  
> -static const char *unpack(int err_fd, struct shallow_info *si)
> +static const char *unpack(int err_fd, struct shallow_info *si,
> +			  struct odb_transaction *transaction)
>  {
>  	struct pack_header hdr;
>  	const char *hdr_err;

It feels a bit weird that we sometimes pass the transaction as
parameter, whereas othertimes we access it via `the_repository`.

> @@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
>  		strvec_push(&child.args, alt_shallow_file);
>  	}
>  
> -	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
> -	if (!tmp_objdir) {
> -		if (err_fd > 0)
> -			close(err_fd);
> -		return "unable to create temporary object directory";
> -	}
> -	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
> -
> -	/*
> -	 * Normally we just pass the tmp_objdir environment to the child
> -	 * processes that do the heavy lifting, but we may need to see these
> -	 * objects ourselves to set up shallow information.
> -	 */
> -	tmp_objdir_add_as_alternate(tmp_objdir);
> +	strvec_pushv(&child.env, odb_transaction_env(transaction));

Interesting, this here seems like a change in behaviour. Previously we
added the transactions as an alternate, but now we only propagate it via
the environment. I didn't see this mentioned in the commit message.

> @@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,
>  		if (!si.nr_ours && !si.nr_theirs)
>  			shallow_update = 0;
>  		if (!delete_only(commands)) {
> -			unpack_status = unpack_with_sideband(&si);
> +			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
> +				unpack_status = "unable to start ODB transaction";

s/ODB/object/

This may be visible to the user, and "ODB" may mean nothing to them.

> diff --git a/object-file.c b/object-file.c
> index 14064d188a..e7958753ec 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base)
>  }
>  
>  int odb_transaction_files_begin(struct odb_source *source,
> -				struct odb_transaction **out)
> +				struct odb_transaction **out,
> +				enum odb_transaction_flags flags)
>  {
>  	struct odb_transaction_files *transaction;
>  	struct object_database *odb = source->odb;
> @@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source,
>  	transaction->base.commit = odb_transaction_files_commit;
>  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
>  	transaction->base.env = odb_transaction_files_env;
> +
> +	transaction->prefix = "bulk-fsync";
> +	if (flags & ODB_TRANSACTION_RECEIVE) {
> +		/*
> +		 * ODB transactions for git-receive-pack(1) eagerly create a
> +		 * temporary directory and use a different prefix.
> +		 */
> +		transaction->prefix = "incoming";
> +		if (odb_transaction_files_prepare(&transaction->base)) {
> +			free(transaction);
> +			return -1;
> +		}
> +	}
> +

Okay, makes sense. I really wonder whether we need to insist this much
on the exact name used by this, but better be safe than sorry for now I
guess.

And as mentioned before, I also wonder whether it really makes sense to
have the lazy creation of the tmp-objdir. Maybe add a NEEDSWORK item
here that mentions we want to investigate whether this is even needed at
all?

> diff --git a/odb/transaction.h b/odb/transaction.h
> index 536458297b..78392ff13d 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -44,6 +43,10 @@ struct odb_transaction {
>  	const char **(*env)(struct odb_transaction *transaction);
>  };
>  
> +enum odb_transaction_flags {
> +	ODB_TRANSACTION_RECEIVE = (1 << 0),
> +};

It's not clear at all what this flag does based on its name, so we
should have documentation for it.

Patrick

Re: [PATCH 0/6] receive-pack: use ODB transactions to stage object writes

[Patrick Steinhardt]

On Tue, Jun 23, 2026 at 11:19:14PM -0500, Justin Tobler wrote:
> Greetings,
> 
> This patch series replaces direct usage of the `tmp_objdir` interfaces
> in git-receive-pack(1) to instead use the `odb_transaction` interfaces
> to create/manage a staging area to write objects to. The purpose of this
> change is to get git-receive-pack(1) one step closer to being ODB
> backend agnostic. For now, the object writes themselves are still
> "files" backend specific due to being handled by the git-index-pack(1)
> and git-unpack-objects(1) child processes. This will be tackled in a
> separate series though.

Thanks, this was a pleasant read. I've got a bunch of comments, but
overall I really like the direction of this patch series.

Patrick

Re: [PATCH 1/6] object-file: rename files transaction prepare function

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> The "files" ODB transaction backend lazily creates a temporary object
> directory when the first loose object is written to the transaction via
> `prepare_loose_object_transaction()`. In a subsequent commit, the
> temporary directory is used to also write packfiles to.
>
> Rename the function to `odb_transaction_files_prepare()` accordingly.

Taken by itself this renaming does make sense, but there are many
other function that follow the historical naming convention, like
{fsync,flush}_loose_object_transaction().  Should we rename them for
consistency with the new naming scheme, not necessarily as part of
this series but with a todo comment to do so once the dust settles,
or something?

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> The "files" transaction backend may encounter errors related to managing
> the temporary directory used to stage objects, but silently ignores
> these errors. Instead return errors encountered in the
> `odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
> callers to handle as needed.

"handle them as needed", perhaps.

> -static void odb_transaction_files_prepare(struct odb_transaction *base)
> +static int odb_transaction_files_prepare(struct odb_transaction *base)
>  {
>  	struct odb_transaction_files *transaction =
>  		container_of_or_null(base, struct odb_transaction_files, base);
> @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
>  	 * added at the time they call odb_transaction_files_begin.
>  	 */
>  	if (!transaction || transaction->objdir)
> -		return;
> +		return 0;
>  
>  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");

If this fails, NULL is returned, and ...

> -	if (transaction->objdir)
> -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> +	if (!transaction->objdir)
> +		return -1;

... we return -1 from here to signal an error now.

But callers of this function in write_loose_object(), and
odb_source_loose_write_stream() are not prepared to react to such an
error.

I guess this is nothing new.  The callers ignored such an error from here
in the original and proceeded writing the primary ODB anyway, and we
continue to do so after this step.

> @@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
>  /*
>   * Cleanup after batch-mode fsync_object_files.
>   */
> -static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
> +static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
>  {
>  	struct strbuf temp_path = STRBUF_INIT;
>  	struct tempfile *temp;
>  
>  	if (!transaction->objdir)
> -		return;
> +		return 0;
>  
>  	/*
>  	 * Issue a full hardware flush against a temporary file to ensure
> @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
>  	 * Make the object files visible in the primary ODB after their data is
>  	 * fully durable.
>  	 */
> -	tmp_objdir_migrate(transaction->objdir);
> +	if (tmp_objdir_migrate(transaction->objdir))
> +		return -1;
> +
>  	transaction->objdir = NULL;
> +
> +	return 0;
>  }

The caller of this function does react to a failure of it, ...

> @@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
>  	return ret;
>  }
>  
> -static void odb_transaction_files_commit(struct odb_transaction *base)
> +static int odb_transaction_files_commit(struct odb_transaction *base)
>  {
>  	struct odb_transaction_files *transaction =
>  		container_of(base, struct odb_transaction_files, base);
>  
> -	flush_loose_object_transaction(transaction);
> +	if (flush_loose_object_transaction(transaction))
> +		return -1;
>  	flush_packfile_transaction(transaction);
> +
> +	return 0;
>  }

... like this, which is good.  Do we need an explicit "abort-transaction",
or is that implicit?

Thanks.

Re: [PATCH 0/6] receive-pack: use ODB transactions to stage object writes

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> Greetings,
>
> This patch series replaces direct usage of the `tmp_objdir` interfaces
> in git-receive-pack(1) to instead use the `odb_transaction` interfaces
> to create/manage a staging area to write objects to. The purpose of this
> change is to get git-receive-pack(1) one step closer to being ODB
> backend agnostic. For now, the object writes themselves are still
> "files" backend specific due to being handled by the git-index-pack(1)
> and git-unpack-objects(1) child processes. This will be tackled in a
> separate series though.
>
> Thanks,
> -Justin

The integration cycle this morning was somehow more painful than
other cycles.  

FYI, this topic had some interactions with ps/odb-source-packed and
ps/refs-avoid-chdir-notify-reparent and needed the following
evil-merge fix to make it build.

commit 721c15c1d5ef8f8aab8b9f50463e979e890b1ab6
Author: Junio C Hamano <gitster@pobox.com>
Date:   Wed Jun 24 12:45:35 2026 -0700

    merge-fix/jt/receive-pack-use-odb-transactions
    
    conflict with ps/odb-source-packed and ps/refs-avoid-chdir-notify-reparent

diff --git a/odb/source-packed.c b/odb/source-packed.c
index c3c26fb53e..b9231a8da0 100644
--- a/odb/source-packed.c
+++ b/odb/source-packed.c
@@ -545,7 +545,8 @@ static int odb_source_packed_write_object_stream(struct odb_source *source UNUSE
 }
 
 static int odb_source_packed_begin_transaction(struct odb_source *source UNUSED,
-					       struct odb_transaction **out UNUSED)
+					       struct odb_transaction **out UNUSED,
+					       enum odb_transaction_flags flags UNUSED)
 {
 	return error("packed backend cannot begin transactions");
 }
diff --git a/t/unit-tests/u-reftable-table.c b/t/unit-tests/u-reftable-table.c
index a35bbc8004..dd5db06534 100644
--- a/t/unit-tests/u-reftable-table.c
+++ b/t/unit-tests/u-reftable-table.c
@@ -235,7 +235,8 @@ void test_reftable_table__seek_invalid_log_offset(void)
 	uint8_t *footer;
 
 	cl_reftable_write_to_buf(&buf, refs, ARRAY_SIZE(refs),
-				 logs, ARRAY_SIZE(logs), NULL);
+				 logs, ARRAY_SIZE(logs),
+				 REFTABLE_HASH_SHA1, NULL);
 
 	/*
 	 * Corrupt the log section offset stored in the footer so that it points
@@ -278,7 +279,8 @@ void test_reftable_table__new_with_truncated_table(void)
 	struct reftable_table *table;
 	struct reftable_buf buf = REFTABLE_BUF_INIT;
 
-	cl_reftable_write_to_buf(&buf, refs, ARRAY_SIZE(refs), NULL, 0, NULL);
+	cl_reftable_write_to_buf(&buf, refs, ARRAY_SIZE(refs), NULL, 0,
+				 REFTABLE_HASH_SHA1, NULL);
 
 	/*
 	 * Truncate the table so that it is large enough to read the header, but

Re: [PATCH 1/6] object-file: rename files transaction prepare function

[Justin Tobler]

On 26/06/24 11:26AM, Junio C Hamano wrote:
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > The "files" ODB transaction backend lazily creates a temporary object
> > directory when the first loose object is written to the transaction via
> > `prepare_loose_object_transaction()`. In a subsequent commit, the
> > temporary directory is used to also write packfiles to.
> >
> > Rename the function to `odb_transaction_files_prepare()` accordingly.
> 
> Taken by itself this renaming does make sense, but there are many
> other function that follow the historical naming convention, like
> {fsync,flush}_loose_object_transaction().  Should we rename them for
> consistency with the new naming scheme, not necessarily as part of
> this series but with a todo comment to do so once the dust settles,
> or something?

Ya, both {fsync,flush}_loose_object_transaction() are probably good
candidates to be renamed to odb_transaction_files_{flush,flush} also. In
the next version, I'll probably add another patch to do so accordingly. 

-Justin

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Justin Tobler]

On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> > The "files" transaction backend may encounter errors related to managing
> > the temporary directory used to stage objects, but silently ignores
> > these errors. Instead return errors encountered in the
> > `odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
> > callers to handle as needed.
> 
> Missing a then? "to handle as needed" -> "to handle them as needed"

Will fix.

[snip]
> > diff --git a/object-file.c b/object-file.c
> > index a3eb8d71dd..18c2df75fb 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -499,7 +499,7 @@ struct odb_transaction_files {
> >  	struct transaction_packfile packfile;
> >  };
> >  
> > -static void odb_transaction_files_prepare(struct odb_transaction *base)
> > +static int odb_transaction_files_prepare(struct odb_transaction *base)
> >  {
> >  	struct odb_transaction_files *transaction =
> >  		container_of_or_null(base, struct odb_transaction_files, base);
> 
> By the way, is there any reason why those functions are still hosted in
> "object-file.c" instead of in "odb/source-files.c"? I should probably
> know, but I forgot.

There are currently a couple spots in the "files" object write path in
"object-file.c" that still reach into some of these transaction function
that are not part of the generic ODB transaction interface. I'm hoping
in a future followup series to detangle this a bit and be able to get
all the "files" ODB transaction related code moved into
"odb/source-files.c".

> > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> >  	 * added at the time they call odb_transaction_files_begin.
> >  	 */
> >  	if (!transaction || transaction->objdir)
> > -		return;
> > +		return 0;
> >  
> >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > -	if (transaction->objdir)
> > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > +	if (!transaction->objdir)
> > +		return -1;
> 
> Huh. So previously we just didn't handle this error at all and just
> continued to tag along? Did that result in anything sensible or was this
> just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up
creating any tmpdir and would instead continue to use the primary ODB to
write objects in. This change would make it a hard error if we fail to
create the temp dir. This matches the behavior that git-receive-pack(1)
expects, but I didn't consider that the existing callers could
transparently handle there being no temp dir.

I suspect we may want existing ODB transaction users to continue being
resilient in the same manner. In the next version, I'll maintain the
same behavior.

> > @@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
> >  /*
> >   * Cleanup after batch-mode fsync_object_files.
> >   */
> > -static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
> > +static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
> 
> Feels like this function should've been renamed in the preceding commit,
> as well.

Ya I agree, will address in the next version.

> >  {
> >  	struct strbuf temp_path = STRBUF_INIT;
> >  	struct tempfile *temp;
> >  
> >  	if (!transaction->objdir)
> > -		return;
> > +		return 0;
> >  
> >  	/*
> >  	 * Issue a full hardware flush against a temporary file to ensure
> > @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
> 
> There is a call to `xmks_tempfile()` hidden that can fail, but that
> failure is already handled in that function itself by dying.
> 
> >  	 * Make the object files visible in the primary ODB after their data is
> >  	 * fully durable.
> >  	 */
> > -	tmp_objdir_migrate(transaction->objdir);
> > +	if (tmp_objdir_migrate(transaction->objdir))
> > +		return -1;
> 
> Feels like another case of YOLOing it. The migration could have failed,
> but we just ignored that failure and never told the user about it. The
> result may be silent corruption, I assume?

Ya in this case, if the migration failed, we would just continue on
which would likely result in silent corruption.

> 
> > @@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
> >  	return ret;
> >  }
> >  
> > -static void odb_transaction_files_commit(struct odb_transaction *base)
> > +static int odb_transaction_files_commit(struct odb_transaction *base)
> >  {
> >  	struct odb_transaction_files *transaction =
> >  		container_of(base, struct odb_transaction_files, base);
> >  
> > -	flush_loose_object_transaction(transaction);
> > +	if (flush_loose_object_transaction(transaction))
> > +		return -1;
> >  	flush_packfile_transaction(transaction);
> > +
> > +	return 0;
> >  }
> >  
> > -struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
> > +int odb_transaction_files_begin(struct odb_source *source,
> > +				struct odb_transaction **out)
> >  {
> >  	struct odb_transaction_files *transaction;
> >  	struct object_database *odb = source->odb;
> >  
> > -	if (odb->transaction)
> > -		return NULL;
> > +	if (odb->transaction) {
> > +		*out = NULL;
> > +		return 0;
> > +	}
> >  
> >  	transaction = xcalloc(1, sizeof(*transaction));
> >  	transaction->base.source = source;
> >  	transaction->base.commit = odb_transaction_files_commit;
> >  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
> > +	*out = &transaction->base;
> >  
> > -	return &transaction->base;
> > +	return 0;
> >  }
> 
> It's still somewhat fishy that we have this ODB-level transaction, but
> that's a preexisting issue and thus outside the scope of this patch
> series. Ideally though, it would be possible for there to be multiple
> transactions, and it would be the caller's responsibility for juggling
> these transactions. Just as it happens with reference transactions.

I completely agree. One of the current problems preventing this is that
only a single instance of tmp_objdir is allowed and stored globally.
This is done to keep atexit() cleanup simple.

My plan is to eventually convert all existing tmp_objdir callsites to
use ODB transactions which should hopefully allow us to remove the need
for a separate tmp_objdir system. At that point, we can also work to
change how temp dir cleanup is handled at exit.

Another problem is that there are also a couple of ODB transaction
callsites that require to know if there is already a pending transaction
for the repository and the transaction has not been wired down to these
callsites. My hope is that this can be addressed though as we expand ODB
transaction usage for object writes.

> > diff --git a/odb/transaction.h b/odb/transaction.h
> > index 854fda06f5..f4c1ebfaaa 100644
> > --- a/odb/transaction.h
> > +++ b/odb/transaction.h
> > @@ -17,7 +17,7 @@ struct odb_transaction {
> >  	struct odb_source *source;
> >  
> >  	/* The ODB source specific callback invoked to commit a transaction. */
> > -	void (*commit)(struct odb_transaction *transaction);
> > +	int (*commit)(struct odb_transaction *transaction);
> 
> We might want to document the returned error code here.

Will do in the next version.

Thanks,
-Justin

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Justin Tobler]

On 26/06/29 01:58PM, Justin Tobler wrote:
> On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> > On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> > > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> > >  	 * added at the time they call odb_transaction_files_begin.
> > >  	 */
> > >  	if (!transaction || transaction->objdir)
> > > -		return;
> > > +		return 0;
> > >  
> > >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > > -	if (transaction->objdir)
> > > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > > +	if (!transaction->objdir)
> > > +		return -1;
> > 
> > Huh. So previously we just didn't handle this error at all and just
> > continued to tag along? Did that result in anything sensible or was this
> > just YOLOing it?
> 
> Good question. Previously if there was an error, we wouldn't end up
> creating any tmpdir and would instead continue to use the primary ODB to
> write objects in. This change would make it a hard error if we fail to
> create the temp dir. This matches the behavior that git-receive-pack(1)
> expects, but I didn't consider that the existing callers could
> transparently handle there being no temp dir.
> 
> I suspect we may want existing ODB transaction users to continue being
> resilient in the same manner. In the next version, I'll maintain the
> same behavior.

I think I got a bit ahead of myself. The existing callers of
odb_transaction_files_prepare() still continue to ignore this error. So
the behavior already does remain the same here.

-Justin

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Justin Tobler]

On 26/06/24 11:35AM, Junio C Hamano wrote:
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > The "files" transaction backend may encounter errors related to managing
> > the temporary directory used to stage objects, but silently ignores
> > these errors. Instead return errors encountered in the
> > `odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
> > callers to handle as needed.
> 
> "handle them as needed", perhaps.

Will fix, thanks

[snip]
> The caller of this function does react to a failure of it, ...
> 
> > @@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
> >  	return ret;
> >  }
> >  
> > -static void odb_transaction_files_commit(struct odb_transaction *base)
> > +static int odb_transaction_files_commit(struct odb_transaction *base)
> >  {
> >  	struct odb_transaction_files *transaction =
> >  		container_of(base, struct odb_transaction_files, base);
> >  
> > -	flush_loose_object_transaction(transaction);
> > +	if (flush_loose_object_transaction(transaction))
> > +		return -1;
> >  	flush_packfile_transaction(transaction);
> > +
> > +	return 0;
> >  }
> 
> ... like this, which is good.  Do we need an explicit "abort-transaction",
> or is that implicit?

So this is currently handled implicitly via
`tmp-objdir.c:remove_tmp_objdir()` which gets registered as an atexit()
handler. As long as the tmp_objdir global remains set, it will
automatically get cleaned up.

In a subsequent series, I do plan to add `odb_transaction_abort()` to
the transaction interface. It may make sense to also use that here to
make the cleanup a bit more explicit though.

-Justin

Re: [PATCH 3/6] odb/transaction: propagate begin errors

[Justin Tobler]

On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> On Tue, Jun 23, 2026 at 11:19:17PM -0500, Justin Tobler wrote:
> > diff --git a/odb/transaction.c b/odb/transaction.c
> > index b16e07aebf..d3de01db50 100644
> > --- a/odb/transaction.c
> > +++ b/odb/transaction.c
> > @@ -2,14 +2,20 @@
> >  #include "odb/source.h"
> >  #include "odb/transaction.h"
> >  
> > -struct odb_transaction *odb_transaction_begin(struct object_database *odb)
> > +int odb_transaction_begin(struct object_database *odb,
> > +			  struct odb_transaction **out)
> >  {
> > -	if (odb->transaction)
> > -		return NULL;
> > +	int ret;
> >  
> > -	odb_source_begin_transaction(odb->sources, &odb->transaction);
> > +	if (odb->transaction) {
> > +		*out = NULL;
> > +		return 0;
> > +	}
> 
> Hm. So we may return successful, but not set the `out` pointer to a
> transaction. And...
> 
> > diff --git a/odb/transaction.h b/odb/transaction.h
> > index f4c1ebfaaa..cd6d50f2e5 100644
> > --- a/odb/transaction.h
> > +++ b/odb/transaction.h
> > @@ -33,11 +35,20 @@ struct odb_transaction {
> >  };
> >  
> >  /*
> > - * Starts an ODB transaction. Subsequent objects are written to the transaction
> > - * and not committed until odb_transaction_commit() is invoked on the
> > - * transaction. If the ODB already has a pending transaction, NULL is returned.
> > + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
> > + * written to the transaction and not committed until odb_transaction_commit()
> > + * is invoked on the transaction. Returns 0 on success and a negative value on
> > + * error. If the ODB already has a pending transaction, `out` is set to NULL.
> >   */
> > -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
> > +int odb_transaction_begin(struct object_database *odb,
> > +			  struct odb_transaction **out);
> > +
> > +static inline void odb_transaction_begin_or_die(struct object_database *odb,
> > +						struct odb_transaction **out)
> > +{
> > +	if (odb_transaction_begin(odb, out))
> > +		die(_("failed to start ODB transaction"));
> > +}
> 
> ... we don't special-case that here, either. So a caller may invoke the
> function, not die, but it might still not have a valid transaction. That
> feels wrong to me.

Ya, the original idea for a NULL transaction to signal that there is
already a transaction in flight and nested ODB transaction users to use
that to determine whether they needed to start a new transaction or not.
I completely agree thought that this is rather ackward.

Instead we could just make the callers that are worried about
potentially nested transactions handle this explicitly. I'll do that in
the next version.

-Justin

Re: [PATCH 4/6] odb/transaction: propagate commit errors

[Justin Tobler]

On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> On Tue, Jun 23, 2026 at 11:19:18PM -0500, Justin Tobler wrote:
> > diff --git a/odb/transaction.h b/odb/transaction.h
> > index cd6d50f2e5..7898770071 100644
> > --- a/odb/transaction.h
> > +++ b/odb/transaction.h
> > @@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
> >   * Commits an ODB transaction making the written objects visible. If the
> >   * specified transaction is NULL, the function is a no-op.
> >   */
> > -void odb_transaction_commit(struct odb_transaction *transaction);
> > +int odb_transaction_commit(struct odb_transaction *transaction);
> 
> Should the function comment be amended, as well? We should definitely
> point out that calling this with a NULL transaction also returns
> success.

Will do in the next version.

Thanks,
-Justin

Re: [PATCH 5/6] odb/transaction: add transaction env interface

[Justin Tobler]

On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> On Tue, Jun 23, 2026 at 11:19:19PM -0500, Justin Tobler wrote:
> > The ODB transaction backend is responsible for creating/managing its own
> > staging area for writing objects. Other child processes spawned by Git
> > may need to access to uncommitted objects or write new objects in the
> 
> s/may need to access to/may need access to/

Will fix.

> > staging area though.
> > 
> > Introduce `odb_transaction_env()` which is expected to provide the set
> > of environment variables needed by a child process to access the
> > transaction staging area.
> 
> Possessive s is missing, I think.

Yes. :)

[snip]
> > +
> > +	/*
> > +	 * This callback is expected to return a NULL-terminated array of
> > +	 * environment variables that a child process should inherit so
> > +	 * that its object writes participate in the transaction. The
> > +	 * returned array is owned by the backend and remains valid until
> > +	 * the transaction ends. May return NULL when the backend does not
> > +	 * need to expose any state to child processes.
> > +	 */
> > +	const char **(*env)(struct odb_transaction *transaction);
> 
> Would it make more sense to adapt this function so that:
> 
>   - It receives a `struct strvec` as input that the environment
>     variables are to be amended to.
> 
>   - It returns a normal error code to indicate errors?

Ya, that is probably a more sensible interface as it would be nice to be
able to signal an error. Will do in the next version.

Thanks,
-Justin

Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

[Justin Tobler]

On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> On Tue, Jun 23, 2026 at 11:19:20PM -0500, Justin Tobler wrote:
> > Objects received by git-receive-pack(1) are quarantined in a temporary
> > "incoming" directory and migrated into the object database prior to the
> > reference updates. The quarantine is currently managed through
> > `tmp_objdir` directly. In a pluggable ODB future, how exactly an object
> > gets written to a transaction may vary for a given ODB source. Refactor
> > git-receive-pack(1) to use the ODB transaction interfaces to manage the
> > object staging area in a more agnostic manner accordingly.
> > 
> > Note that the temporary directory created for git-receive-pack(1) is
> > eagerly created and uses a different prefix name. This behavior is
> 
> A different prefix name compared to what?

Currently the temporary directories created for ODB transactions all use
the prefix "bulk-fsync". The temp dir created by git-receive-pack(1) is
expected to have the prefix "incoming".

> > special cased in the "files" backend by having `odb_transaction_begin()`
> > callers that require this behavior provide an `ODB_TRANSACTION_RECEIVE`
> > flag.
> 
> Okay. I guess this is to retain existing behaviour where the temporary
> directory is created lazily everywhere else. Makes me wonder whether we
> should eventually change this to just unconditionally create the
> directory in all cases so that we can drop this new flag.

It would be nice to not have to have a flag here, but if we want to also
keep the existing temp dir prefixes, we would also need to keep the
flags.

> It might've also made sense to split this commit up into two: one to
> introduce the flag parameter, and then one to do the changes to
> git-receive-pack(1).

Ya, I think you are right. I actually originally had it this way but
squashed these two commits together for some reason. I'll split them
back up in the next version.

> > diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> > index 19eb6a1b61..ee8e03e2ab 100644
> > --- a/builtin/receive-pack.c
> > +++ b/builtin/receive-pack.c
> > @@ -112,8 +112,6 @@ static enum {
> >  } use_keepalive;
> >  static int keepalive_in_sec = 5;
> >  
> > -static struct tmp_objdir *tmp_objdir;
> > -
> >  static struct proc_receive_ref {
> >  	unsigned int want_add:1,
> >  		     want_delete:1,
> 
> I assume the goal is that we convert all other users of the tmp-objdir
> subsystem to also use transactions eventually, so that this becomes an
> implementation detail fo the files transaction?

Yes, that is exactly my plan :)

> > @@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,
> >  	 * Now we'll start writing out refs, which means the objects need
> >  	 * to be in their final positions so that other processes can see them.
> >  	 */
> > -	if (tmp_objdir_migrate(tmp_objdir) < 0) {
> > +	if (odb_transaction_commit(the_repository->objects->transaction)) {
> >  		for (cmd = commands; cmd; cmd = cmd->next) {
> >  			if (!cmd->error_string)
> >  				cmd->error_string = "unable to migrate objects to permanent storage";
> >  		}
> >  		return;
> >  	}
> > -	tmp_objdir = NULL;
> 
> We don't need to unset the transaction because that's what
> `odb_transaction_commit()` already does for us, I assume?

That is correct. The tmp_objdir global is actually removed and now
competely managed by the ODB transaction. We can probably actuall remove
the "tmp-objdir" include now too. I'll do that in the next version.

> > @@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
> >  		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
> >  }
> >  
> > -static const char *unpack(int err_fd, struct shallow_info *si)
> > +static const char *unpack(int err_fd, struct shallow_info *si,
> > +			  struct odb_transaction *transaction)
> >  {
> >  	struct pack_header hdr;
> >  	const char *hdr_err;
> 
> It feels a bit weird that we sometimes pass the transaction as
> parameter, whereas othertimes we access it via `the_repository`.

That's fair. I was trying to avoid the churn of wiring to all its
callsites, but it's probably best to be consistent. Maybe it would be
fine to just create a transaction global like we do for the reference
transaction?

> > @@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
> >  		strvec_push(&child.args, alt_shallow_file);
> >  	}
> >  
> > -	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
> > -	if (!tmp_objdir) {
> > -		if (err_fd > 0)
> > -			close(err_fd);
> > -		return "unable to create temporary object directory";
> > -	}
> > -	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
> > -
> > -	/*
> > -	 * Normally we just pass the tmp_objdir environment to the child
> > -	 * processes that do the heavy lifting, but we may need to see these
> > -	 * objects ourselves to set up shallow information.
> > -	 */
> > -	tmp_objdir_add_as_alternate(tmp_objdir);
> > +	strvec_pushv(&child.env, odb_transaction_env(transaction));
> 
> Interesting, this here seems like a change in behaviour. Previously we
> added the transactions as an alternate, but now we only propagate it via
> the environment. I didn't see this mentioned in the commit message.

That's not quite correct. By using the ODB transaction interface, the
underlying tmp_objdir is managed transparently. When we begin the ODB
transaction, the temp directory that gets created is automatically
created and applied as the primary ODB. This ultimately produces an
equivalent result, its just now set up when the transaction is started
(which happens a little bit earlier).

The only behavior change here is that the temp directory is being
applied as the primary instead of an alternate in the main
git-receive-pack(1) process. Since all writes though are handled by the
child processes that get spawned, this shouldn't really matter though.

This is certainly rather confusing though, and should be mentioned in
the commit message. I'll do this in the next version.

> > @@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,
> >  		if (!si.nr_ours && !si.nr_theirs)
> >  			shallow_update = 0;
> >  		if (!delete_only(commands)) {
> > -			unpack_status = unpack_with_sideband(&si);
> > +			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
> > +				unpack_status = "unable to start ODB transaction";
> 
> s/ODB/object/
> 
> This may be visible to the user, and "ODB" may mean nothing to them.

Will change in the next version.

> > diff --git a/object-file.c b/object-file.c
> > index 14064d188a..e7958753ec 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base)
> >  }
> >  
> >  int odb_transaction_files_begin(struct odb_source *source,
> > -				struct odb_transaction **out)
> > +				struct odb_transaction **out,
> > +				enum odb_transaction_flags flags)
> >  {
> >  	struct odb_transaction_files *transaction;
> >  	struct object_database *odb = source->odb;
> > @@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source,
> >  	transaction->base.commit = odb_transaction_files_commit;
> >  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
> >  	transaction->base.env = odb_transaction_files_env;
> > +
> > +	transaction->prefix = "bulk-fsync";
> > +	if (flags & ODB_TRANSACTION_RECEIVE) {
> > +		/*
> > +		 * ODB transactions for git-receive-pack(1) eagerly create a
> > +		 * temporary directory and use a different prefix.
> > +		 */
> > +		transaction->prefix = "incoming";
> > +		if (odb_transaction_files_prepare(&transaction->base)) {
> > +			free(transaction);
> > +			return -1;
> > +		}
> > +	}
> > +
> 
> Okay, makes sense. I really wonder whether we need to insist this much
> on the exact name used by this, but better be safe than sorry for now I
> guess.
> 
> And as mentioned before, I also wonder whether it really makes sense to
> have the lazy creation of the tmp-objdir. Maybe add a NEEDSWORK item
> here that mentions we want to investigate whether this is even needed at
> all?

Ya, I was worried that there would be some reason the temp dir prefix
should remain the same or there would be some performance reason to
lazily create directories. I didn't investigate too deeply though. I'll
add a NEEDSWORK comment to make note of this.

> > diff --git a/odb/transaction.h b/odb/transaction.h
> > index 536458297b..78392ff13d 100644
> > --- a/odb/transaction.h
> > +++ b/odb/transaction.h
> > @@ -44,6 +43,10 @@ struct odb_transaction {
> >  	const char **(*env)(struct odb_transaction *transaction);
> >  };
> >  
> > +enum odb_transaction_flags {
> > +	ODB_TRANSACTION_RECEIVE = (1 << 0),
> > +};
> 
> It's not clear at all what this flag does based on its name, so we
> should have documentation for it.

Will update.

Thanks,
-Justin

Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

[Patrick Steinhardt]

On Mon, Jun 29, 2026 at 03:25:18PM -0500, Justin Tobler wrote:
> On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> > On Tue, Jun 23, 2026 at 11:19:20PM -0500, Justin Tobler wrote:
> > > Objects received by git-receive-pack(1) are quarantined in a temporary
> > > "incoming" directory and migrated into the object database prior to the
> > > reference updates. The quarantine is currently managed through
> > > `tmp_objdir` directly. In a pluggable ODB future, how exactly an object
> > > gets written to a transaction may vary for a given ODB source. Refactor
> > > git-receive-pack(1) to use the ODB transaction interfaces to manage the
> > > object staging area in a more agnostic manner accordingly.
> > > 
> > > Note that the temporary directory created for git-receive-pack(1) is
> > > eagerly created and uses a different prefix name. This behavior is
> > 
> > A different prefix name compared to what?
> 
> Currently the temporary directories created for ODB transactions all use
> the prefix "bulk-fsync". The temp dir created by git-receive-pack(1) is
> expected to have the prefix "incoming".
> 
> > > special cased in the "files" backend by having `odb_transaction_begin()`
> > > callers that require this behavior provide an `ODB_TRANSACTION_RECEIVE`
> > > flag.
> > 
> > Okay. I guess this is to retain existing behaviour where the temporary
> > directory is created lazily everywhere else. Makes me wonder whether we
> > should eventually change this to just unconditionally create the
> > directory in all cases so that we can drop this new flag.
> 
> It would be nice to not have to have a flag here, but if we want to also
> keep the existing temp dir prefixes, we would also need to keep the
> flags.

Fair. Makes me wonder whether we really need to keep the exact same
naming for this temporary directory. This is so deep into internals that
I'm not sure whether we really need to treat this as part of our
interface. I'm rather inclined to say it's not necessary.

In any case, I think it's fine to defer that discussion and keep this
as-is for now. But we might keep it in the back of our minds and maybe
simplify this in a subsequent patch series.

> > > diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> > > index 19eb6a1b61..ee8e03e2ab 100644
> > > --- a/builtin/receive-pack.c
> > > +++ b/builtin/receive-pack.c
> > > @@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
> > >  		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
> > >  }
> > >  
> > > -static const char *unpack(int err_fd, struct shallow_info *si)
> > > +static const char *unpack(int err_fd, struct shallow_info *si,
> > > +			  struct odb_transaction *transaction)
> > >  {
> > >  	struct pack_header hdr;
> > >  	const char *hdr_err;
> > 
> > It feels a bit weird that we sometimes pass the transaction as
> > parameter, whereas othertimes we access it via `the_repository`.
> 
> That's fair. I was trying to avoid the churn of wiring to all its
> callsites, but it's probably best to be consistent. Maybe it would be
> fine to just create a transaction global like we do for the reference
> transaction?

My first kneejerk reaction was "no", but then I noticed that the global
variable you're talking about is local to "builtin/receive-pack.c". So
that might be an okayish solution.

Thanks!

Patrick

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Patrick Steinhardt]

On Mon, Jun 29, 2026 at 01:58:54PM -0500, Justin Tobler wrote:
> On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> > On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> > > diff --git a/object-file.c b/object-file.c
> > > index a3eb8d71dd..18c2df75fb 100644
> > > --- a/object-file.c
> > > +++ b/object-file.c
> > > @@ -499,7 +499,7 @@ struct odb_transaction_files {
> > >  	struct transaction_packfile packfile;
> > >  };
> > >  
> > > -static void odb_transaction_files_prepare(struct odb_transaction *base)
> > > +static int odb_transaction_files_prepare(struct odb_transaction *base)
> > >  {
> > >  	struct odb_transaction_files *transaction =
> > >  		container_of_or_null(base, struct odb_transaction_files, base);
> > 
> > By the way, is there any reason why those functions are still hosted in
> > "object-file.c" instead of in "odb/source-files.c"? I should probably
> > know, but I forgot.
> 
> There are currently a couple spots in the "files" object write path in
> "object-file.c" that still reach into some of these transaction function
> that are not part of the generic ODB transaction interface. I'm hoping
> in a future followup series to detangle this a bit and be able to get
> all the "files" ODB transaction related code moved into
> "odb/source-files.c".

Makes sense. I also revisited that code a couple days ago, and the
answer is "it's messy right now". Hopefully this will become easier to
detangle as we make progress on pluggifying transactions.

> > > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> > >  	 * added at the time they call odb_transaction_files_begin.
> > >  	 */
> > >  	if (!transaction || transaction->objdir)
> > > -		return;
> > > +		return 0;
> > >  
> > >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > > -	if (transaction->objdir)
> > > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > > +	if (!transaction->objdir)
> > > +		return -1;
> > 
> > Huh. So previously we just didn't handle this error at all and just
> > continued to tag along? Did that result in anything sensible or was this
> > just YOLOing it?
> 
> Good question. Previously if there was an error, we wouldn't end up
> creating any tmpdir and would instead continue to use the primary ODB to
> write objects in. This change would make it a hard error if we fail to
> create the temp dir. This matches the behavior that git-receive-pack(1)
> expects, but I didn't consider that the existing callers could
> transparently handle there being no temp dir.
> 
> I suspect we may want existing ODB transaction users to continue being
> resilient in the same manner. In the next version, I'll maintain the
> same behavior.

Honestly I'd say that the change is a good one. I cannot think of a
single reason to just blindly not create the transaction and proceed.
But it certainly is something that should be documented as part of the
commit message.

> > > @@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
> > >  	return ret;
> > >  }
> > >  
> > > -static void odb_transaction_files_commit(struct odb_transaction *base)
> > > +static int odb_transaction_files_commit(struct odb_transaction *base)
> > >  {
> > >  	struct odb_transaction_files *transaction =
> > >  		container_of(base, struct odb_transaction_files, base);
> > >  
> > > -	flush_loose_object_transaction(transaction);
> > > +	if (flush_loose_object_transaction(transaction))
> > > +		return -1;
> > >  	flush_packfile_transaction(transaction);
> > > +
> > > +	return 0;
> > >  }
> > >  
> > > -struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
> > > +int odb_transaction_files_begin(struct odb_source *source,
> > > +				struct odb_transaction **out)
> > >  {
> > >  	struct odb_transaction_files *transaction;
> > >  	struct object_database *odb = source->odb;
> > >  
> > > -	if (odb->transaction)
> > > -		return NULL;
> > > +	if (odb->transaction) {
> > > +		*out = NULL;
> > > +		return 0;
> > > +	}
> > >  
> > >  	transaction = xcalloc(1, sizeof(*transaction));
> > >  	transaction->base.source = source;
> > >  	transaction->base.commit = odb_transaction_files_commit;
> > >  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
> > > +	*out = &transaction->base;
> > >  
> > > -	return &transaction->base;
> > > +	return 0;
> > >  }
> > 
> > It's still somewhat fishy that we have this ODB-level transaction, but
> > that's a preexisting issue and thus outside the scope of this patch
> > series. Ideally though, it would be possible for there to be multiple
> > transactions, and it would be the caller's responsibility for juggling
> > these transactions. Just as it happens with reference transactions.
> 
> I completely agree. One of the current problems preventing this is that
> only a single instance of tmp_objdir is allowed and stored globally.
> This is done to keep atexit() cleanup simple.
> 
> My plan is to eventually convert all existing tmp_objdir callsites to
> use ODB transactions which should hopefully allow us to remove the need
> for a separate tmp_objdir system. At that point, we can also work to
> change how temp dir cleanup is handled at exit.

Great.

> Another problem is that there are also a couple of ODB transaction
> callsites that require to know if there is already a pending transaction
> for the repository and the transaction has not been wired down to these
> callsites. My hope is that this can be addressed though as we expand ODB
> transaction usage for object writes.

Yeah, agreed. Making the use of transactions explicit feels sensible to
me.

Patrick

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Patrick Steinhardt]

On Mon, Jun 29, 2026 at 02:04:08PM -0500, Justin Tobler wrote:
> On 26/06/29 01:58PM, Justin Tobler wrote:
> > On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> > > On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> > > > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> > > >  	 * added at the time they call odb_transaction_files_begin.
> > > >  	 */
> > > >  	if (!transaction || transaction->objdir)
> > > > -		return;
> > > > +		return 0;
> > > >  
> > > >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > > > -	if (transaction->objdir)
> > > > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > > > +	if (!transaction->objdir)
> > > > +		return -1;
> > > 
> > > Huh. So previously we just didn't handle this error at all and just
> > > continued to tag along? Did that result in anything sensible or was this
> > > just YOLOing it?
> > 
> > Good question. Previously if there was an error, we wouldn't end up
> > creating any tmpdir and would instead continue to use the primary ODB to
> > write objects in. This change would make it a hard error if we fail to
> > create the temp dir. This matches the behavior that git-receive-pack(1)
> > expects, but I didn't consider that the existing callers could
> > transparently handle there being no temp dir.
> > 
> > I suspect we may want existing ODB transaction users to continue being
> > resilient in the same manner. In the next version, I'll maintain the
> > same behavior.
> 
> I think I got a bit ahead of myself. The existing callers of
> odb_transaction_files_prepare() still continue to ignore this error. So
> the behavior already does remain the same here.

Oh, well, okay. I think this behaviour is plain bad -- if the caller
wants to have a transaction, then we should bail in case we cannot
create one. But this doesn't need to be fixed in this patch series.

Patrick

Re: [PATCH 2/6] object-file: propagate files transaction errors

[Justin Tobler]

On 26/06/30 10:45AM, Patrick Steinhardt wrote:
> On Mon, Jun 29, 2026 at 02:04:08PM -0500, Justin Tobler wrote:
> > On 26/06/29 01:58PM, Justin Tobler wrote:
> > > On 26/06/24 01:26PM, Patrick Steinhardt wrote:
> > > > On Tue, Jun 23, 2026 at 11:19:16PM -0500, Justin Tobler wrote:
> > > > > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> > > > >  	 * added at the time they call odb_transaction_files_begin.
> > > > >  	 */
> > > > >  	if (!transaction || transaction->objdir)
> > > > > -		return;
> > > > > +		return 0;
> > > > >  
> > > > >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > > > > -	if (transaction->objdir)
> > > > > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > > > > +	if (!transaction->objdir)
> > > > > +		return -1;
> > > > 
> > > > Huh. So previously we just didn't handle this error at all and just
> > > > continued to tag along? Did that result in anything sensible or was this
> > > > just YOLOing it?
> > > 
> > > Good question. Previously if there was an error, we wouldn't end up
> > > creating any tmpdir and would instead continue to use the primary ODB to
> > > write objects in. This change would make it a hard error if we fail to
> > > create the temp dir. This matches the behavior that git-receive-pack(1)
> > > expects, but I didn't consider that the existing callers could
> > > transparently handle there being no temp dir.
> > > 
> > > I suspect we may want existing ODB transaction users to continue being
> > > resilient in the same manner. In the next version, I'll maintain the
> > > same behavior.
> > 
> > I think I got a bit ahead of myself. The existing callers of
> > odb_transaction_files_prepare() still continue to ignore this error. So
> > the behavior already does remain the same here.
> 
> Oh, well, okay. I think this behaviour is plain bad -- if the caller
> wants to have a transaction, then we should bail in case we cannot
> create one. But this doesn't need to be fixed in this patch series.

Ya, I tend to agree. The problem here is that
odb_transaction_files_prepare() is being invoked lazily during the
object write. This would be another argument against lazily creating the
temporary directory though. In a followup series I'll explore removing
this and investigate if it has any meaninful performance implications.

-Justin

[PATCH v2 00/11] receive-pack: use ODB transactions to stage object writes

[Justin Tobler]

Greetings,

This patch series replaces direct usage of the `tmp_objdir` interfaces
in git-receive-pack(1) to instead use the `odb_transaction` interfaces
to create/manage a staging area to write objects to. The purpose of this
change is to get git-receive-pack(1) one step closer to being ODB
backend agnostic. For now, the object writes themselves are still
"files" backend specific due to being handled by the git-index-pack(1)
and git-unpack-objects(1) child processes. This will be tackled in a
separate series though.

Changes since V1:

  - Adapted other "file" ODB transaction helpers to be more consistent
    with current naming scheme.
  - Removed redundant NULL transaction handling from
    `odb_transaction_files_begin()`.
  - `odb_transaction_begin()` now returns an error if there is already
    an inflight transaction pending instead of setting the `out` pointer
    to NULL.
  - Updated `odb_transaction_env()` to return an error code and append
    environment variables to a strvec provided as an argument.
  - Removed redundant setting of tmpdir environment variables for child
    processes after tmpdir has been migrated.
  - Split changes adding ODB transaction flags into a separate commit.
  - Consistently wire the ODB transaction throughout git-receive-pack
    code instead of reading it from `the_repository`.
  - Updated user facing error message.
  - Updated some comments to better document functions/flags.
  - Clarified some commit messages.
  - Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
  object-file: rename files transaction prepare function
  object-file: rename files transaction fsync function
  object-file: embed transaction flush logic in commit function
  object-file: drop check for inflight transactions
  object-file: propagate files transaction errors
  odb/transaction: propagate begin errors
  odb/transaction: propagate commit errors
  odb/transaction: add transaction env interface
  odb/transaction: introduce ODB transaction flags
  builtin/receive-pack: drop redundant tmpdir env
  builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 159 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  34 +++++++--
 odb/transaction.h        |  60 ++++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 244 insertions(+), 130 deletions(-)

Range-diff against v1:
 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 -:  ---------- >  2:  5703a9e93b object-file: rename files transaction fsync function
 -:  ---------- >  3:  4c37398ac8 object-file: embed transaction flush logic in commit function
 -:  ---------- >  4:  623c6b02ea object-file: drop check for inflight transactions
 2:  201f543692 !  5:  ca59176657 object-file: propagate files transaction errors
    @@ Commit message
         the temporary directory used to stage objects, but silently ignores
         these errors. Instead return errors encountered in the
         `odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
    -    callers to handle as needed.
    +    callers to handle them as needed.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
    @@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
     +	return 0;
      }
      
    - static void fsync_loose_object_transaction(struct odb_transaction *base,
    -@@ object-file.c: static void fsync_loose_object_transaction(struct odb_transaction *base,
    - /*
    -  * Cleanup after batch-mode fsync_object_files.
    -  */
    --static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
    -+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
    - {
    - 	struct strbuf temp_path = STRBUF_INIT;
    - 	struct tempfile *temp;
    - 
    - 	if (!transaction->objdir)
    --		return;
    -+		return 0;
    - 
    - 	/*
    - 	 * Issue a full hardware flush against a temporary file to ensure
    -@@ object-file.c: static void flush_loose_object_transaction(struct odb_transaction_files *transac
    - 	 * Make the object files visible in the primary ODB after their data is
    - 	 * fully durable.
    - 	 */
    --	tmp_objdir_migrate(transaction->objdir);
    -+	if (tmp_objdir_migrate(transaction->objdir))
    -+		return -1;
    -+
    - 	transaction->objdir = NULL;
    -+
    -+	return 0;
    - }
    - 
    - /* Finalize a file on disk, and close it. */
    + static void odb_transaction_files_fsync(struct odb_transaction *base,
     @@ object-file.c: int read_loose_object(struct repository *repo,
      	return ret;
      }
    @@ object-file.c: int read_loose_object(struct repository *repo,
      {
      	struct odb_transaction_files *transaction =
      		container_of(base, struct odb_transaction_files, base);
    +@@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *base)
    + 		 * Make the object files visible in the primary ODB after their data is
    + 		 * fully durable.
    + 		 */
    +-		tmp_objdir_migrate(transaction->objdir);
    ++		if (tmp_objdir_migrate(transaction->objdir))
    ++			return -1;
    ++
    + 		transaction->objdir = NULL;
    + 	}
      
    --	flush_loose_object_transaction(transaction);
    -+	if (flush_loose_object_transaction(transaction))
    -+		return -1;
      	flush_packfile_transaction(transaction);
     +
     +	return 0;
    @@ object-file.c: int read_loose_object(struct repository *repo,
     +				struct odb_transaction **out)
      {
      	struct odb_transaction_files *transaction;
    - 	struct object_database *odb = source->odb;
    - 
    --	if (odb->transaction)
    --		return NULL;
    -+	if (odb->transaction) {
    -+		*out = NULL;
    -+		return 0;
    -+	}
      
    - 	transaction = xcalloc(1, sizeof(*transaction));
    +@@ object-file.c: struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
      	transaction->base.source = source;
      	transaction->base.commit = odb_transaction_files_commit;
      	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
    @@ object-file.c: int read_loose_object(struct repository *repo,
     
      ## object-file.h ##
     @@ object-file.h: struct odb_transaction;
    -  * Tell the object database to optimize for adding
       * multiple objects. odb_transaction_files_commit must be called
    -  * to make new objects visible. If a transaction is already
    -- * pending, NULL is returned.
    -+ * pending, out is set to NULL.
    +  * to make new objects visible.
       */
     -struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
     +int odb_transaction_files_begin(struct odb_source *source,
    @@ odb/source-files.c: static int odb_source_files_write_object_stream(struct odb_s
     
      ## odb/transaction.h ##
     @@ odb/transaction.h: struct odb_transaction {
    + 	/* The ODB source the transaction is opened against. */
      	struct odb_source *source;
      
    - 	/* The ODB source specific callback invoked to commit a transaction. */
    +-	/* The ODB source specific callback invoked to commit a transaction. */
     -	void (*commit)(struct odb_transaction *transaction);
    ++	/*
    ++	 * The ODB source specific callback invoked to commit a transaction.
    ++	 * Returns 0 on success, a negative error code otherwise.
    ++	 */
     +	int (*commit)(struct odb_transaction *transaction);
      
      	/*
 3:  68cdd88ab5 !  6:  717a1ce9a7 odb/transaction: propagate begin errors
    @@ Commit message
         convenience for callsites that do not need to handle errors
         explicitly.
     
    +    Note that `odb_transaction_begin()` now returns an error when the ODB
    +    already has an inflight transaction pending. ODB transaction call sites
    +    that may encounter an inflight transaction are updated to explicitly
    +    handle this case.
    +
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## builtin/add.c ##
    @@ builtin/update-index.c: int cmd_update_index(int argc,
      			parseopt_state = parse_options_step(&ctx, options,
     
      ## cache-tree.c ##
    +@@ cache-tree.c: static int update_one(struct cache_tree *it,
    + 
    + int cache_tree_update(struct index_state *istate, int flags)
    + {
    ++	int inflight = !!the_repository->objects->transaction;
    + 	struct odb_transaction *transaction;
    + 	int skip, i;
    + 
     @@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)
      
      	trace_performance_enter();
      	trace2_region_enter("cache_tree", "update", istate->repo);
     -	transaction = odb_transaction_begin(the_repository->objects);
    -+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
    ++	if (!inflight)
    ++		odb_transaction_begin_or_die(the_repository->objects, &transaction);
      	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
      		       "", 0, &skip, flags);
    - 	odb_transaction_commit(transaction);
    +-	odb_transaction_commit(transaction);
    ++	if (!inflight)
    ++		odb_transaction_commit(transaction);
    + 	trace2_region_leave("cache_tree", "update", istate->repo);
    + 	trace_performance_leave("cache_tree_update");
    + 	if (i < 0)
     
      ## object-file.c ##
     @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
    @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
      		if (flags & INDEX_WRITE_OBJECT) {
      			struct object_database *odb = the_repository->objects;
     -			struct odb_transaction *transaction = odb_transaction_begin(odb);
    -+			struct odb_transaction *transaction;
    ++			struct odb_transaction *transaction = odb->transaction;
    ++			int inflight = !!transaction;
      
    -+			odb_transaction_begin_or_die(odb, &transaction);
    - 			ret = odb_transaction_write_object_stream(odb->transaction,
    +-			ret = odb_transaction_write_object_stream(odb->transaction,
    ++			if (!inflight)
    ++				odb_transaction_begin_or_die(odb, &transaction);
    ++			ret = odb_transaction_write_object_stream(transaction,
      								  &stream,
      								  xsize_t(st->st_size),
    + 								  oid);
    +-			odb_transaction_commit(transaction);
    ++			if (!inflight)
    ++				odb_transaction_commit(transaction);
    + 		} else {
    + 			ret = hash_blob_stream(&stream,
    + 					       the_repository->hash_algo, oid,
     
      ## odb/transaction.c ##
     @@
    @@ odb/transaction.c
     +int odb_transaction_begin(struct object_database *odb,
     +			  struct odb_transaction **out)
      {
    --	if (odb->transaction)
    --		return NULL;
     +	int ret;
    ++
    + 	if (odb->transaction)
    +-		return NULL;
    ++		return -1;
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
    -+	if (odb->transaction) {
    -+		*out = NULL;
    -+		return 0;
    -+	}
    - 
    --	return odb->transaction;
     +	ret = odb_source_begin_transaction(odb->sources, out);
     +	odb->transaction = *out;
    -+
    + 
    +-	return odb->transaction;
     +	return ret;
      }
      
    @@ odb/transaction.h: struct odb_transaction {
     + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
     + * written to the transaction and not committed until odb_transaction_commit()
     + * is invoked on the transaction. Returns 0 on success and a negative value on
    -+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
    ++ * error. Note that it is considered an error to start a new transaction if the
    ++ * ODB already has an inflight transaction pending.
       */
     -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
     +int odb_transaction_begin(struct object_database *odb,
    @@ odb/transaction.h: struct odb_transaction {
       * Commits an ODB transaction making the written objects visible. If the
     
      ## read-cache.c ##
    +@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
    + 		       const struct pathspec *pathspec, char *ps_matched,
    + 		       int include_sparse, int flags, int ignored_too )
    + {
    ++	int inflight = !!repo->objects->transaction;
    + 	struct odb_transaction *transaction;
    + 	struct update_callback_data data;
    + 	struct rev_info rev;
     @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
      	 * This function is invoked from commands other than 'add', which
      	 * may not have their own transaction active.
      	 */
     -	transaction = odb_transaction_begin(repo->objects);
    -+	odb_transaction_begin_or_die(repo->objects, &transaction);
    ++	if (!inflight)
    ++		odb_transaction_begin_or_die(repo->objects, &transaction);
      	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
    - 	odb_transaction_commit(transaction);
    +-	odb_transaction_commit(transaction);
    ++	if (!inflight)
    ++		odb_transaction_commit(transaction);
      
    + 	release_revisions(&rev);
    + 	return !!data.add_errors;
 -:  ---------- >  7:  ff8e133965 odb/transaction: propagate commit errors
 5:  82302db9f0 !  8:  264ba94b83 odb/transaction: add transaction env interface
    @@ Commit message
     
         The ODB transaction backend is responsible for creating/managing its own
         staging area for writing objects. Other child processes spawned by Git
    -    may need to access to uncommitted objects or write new objects in the
    +    may need access to uncommitted objects or write new objects in the
         staging area though.
     
         Introduce `odb_transaction_env()` which is expected to provide the set
         of environment variables needed by a child process to access the
    -    transaction staging area.
    +    transaction's staging area.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## object-file.c ##
    +@@
    + #include "path.h"
    + #include "read-cache-ll.h"
    + #include "setup.h"
    ++#include "strvec.h"
    + #include "tempfile.h"
    + #include "tmp-objdir.h"
    + 
     @@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *base)
      	return 0;
      }
      
    -+static const char **odb_transaction_files_env(struct odb_transaction *base)
    ++static int odb_transaction_files_env(struct odb_transaction *base,
    ++				     struct strvec *env)
     +{
     +	struct odb_transaction_files *transaction =
     +		container_of(base, struct odb_transaction_files, base);
     +
     +	odb_transaction_files_prepare(&transaction->base);
    ++	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
     +
    -+	return tmp_objdir_env(transaction->objdir);
    ++	return 0;
     +}
     +
      int odb_transaction_files_begin(struct odb_source *source,
    @@ odb/transaction.c: int odb_transaction_write_object_stream(struct odb_transactio
      	return transaction->write_object_stream(transaction, stream, len, oid);
      }
     +
    -+const char **odb_transaction_env(struct odb_transaction *transaction)
    ++int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
     +{
     +	if (!transaction)
    -+		return NULL;
    ++		return 0;
     +
    -+	return transaction->env(transaction);
    ++	return transaction->env(transaction, env);
     +}
     
      ## odb/transaction.h ##
    @@ odb/transaction.h: struct odb_transaction {
      				   struct object_id *oid);
     +
     +	/*
    -+	 * This callback is expected to return a NULL-terminated array of
    -+	 * environment variables that a child process should inherit so
    -+	 * that its object writes participate in the transaction. The
    -+	 * returned array is owned by the backend and remains valid until
    -+	 * the transaction ends. May return NULL when the backend does not
    -+	 * need to expose any state to child processes.
    ++	 * This callback is expected to populate the provided strvec with the
    ++	 * environment variables that a child process should inherit so that its
    ++	 * object writes participate in the transaction. Returns 0 on success, a
    ++	 * negative error code otherwise.
     +	 */
    -+	const char **(*env)(struct odb_transaction *transaction);
    ++	int (*env)(struct odb_transaction *transaction, struct strvec *env);
      };
      
      /*
    @@ odb/transaction.h: int odb_transaction_write_object_stream(struct odb_transactio
      					size_t len, struct object_id *oid);
      
     +/*
    -+ * Returns a NULL-terminated array of environment variables that a child
    ++ * Populates the provided strvec with the environment variables that a child
     + * process should inherit so that its object writes participate in the
    -+ * transaction, suitable for passing via child_process.env. Returns NULL if
    -+ * the transaction is NULL or the backend does not expose any state to child
    -+ * processes.
    ++ * transaction, suitable for using via child_process.env. Returns 0 on success,
    ++ * a negative error code otherwise. Note that, if the specified transaction is
    ++ * NULL, the function is a no-op and no error is returned.
     + */
    -+const char **odb_transaction_env(struct odb_transaction *transaction);
    ++int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
     +
      #endif
 6:  0674e8a3d3 !  9:  1e0a491ef2 builtin/receive-pack: stage incoming objects via ODB transactions
    @@ Metadata
     Author: Justin Tobler <jltobler@gmail.com>
     
      ## Commit message ##
    -    builtin/receive-pack: stage incoming objects via ODB transactions
    +    odb/transaction: introduce ODB transaction flags
     
    -    Objects received by git-receive-pack(1) are quarantined in a temporary
    -    "incoming" directory and migrated into the object database prior to the
    -    reference updates. The quarantine is currently managed through
    -    `tmp_objdir` directly. In a pluggable ODB future, how exactly an object
    -    gets written to a transaction may vary for a given ODB source. Refactor
    -    git-receive-pack(1) to use the ODB transaction interfaces to manage the
    -    object staging area in a more agnostic manner accordingly.
    +    The temporary directory used by git-receive-pack(1) to write objects is
    +    managed slightly differently than how it is done via ODB transactions:
     
    -    Note that the temporary directory created for git-receive-pack(1) is
    -    eagerly created and uses a different prefix name. This behavior is
    -    special cased in the "files" backend by having `odb_transaction_begin()`
    -    callers that require this behavior provide an `ODB_TRANSACTION_RECEIVE`
    -    flag.
    +      - The temporary directory is eagerly created upfront, instead of
    +        waiting for the first object write.
    +
    +      - The prefix name of the temporary directory is "incoming" instead of
    +        "bulk-fsync".
    +
    +    In a subsequent commit, git-receive-pack(1) will use ODB transactions
    +    instead of `tmp_objdir` directly. To provide a means to configure the
    +    same transaction behavior, introduce `enum odb_transaction_flags` and
    +    the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
    +    transactions using the "files" backend to be set up for
    +    git-receive-pack(1). Transaction call sites are updated accordingly to
    +    provide the required flag parameter.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
    @@ builtin/add.c: int cmd_add(int argc,
      	ps_matched = xcalloc(pathspec.nr, 1);
      	if (add_renormalize)
     
    - ## builtin/receive-pack.c ##
    -@@ builtin/receive-pack.c: static enum {
    - } use_keepalive;
    - static int keepalive_in_sec = 5;
    - 
    --static struct tmp_objdir *tmp_objdir;
    --
    - static struct proc_receive_ref {
    - 	unsigned int want_add:1,
    - 		     want_delete:1,
    -@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
    - 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
    - 	}
    - 
    --	if (tmp_objdir)
    --		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
    -+	if (the_repository->objects->transaction)
    -+		strvec_pushv(&opt.env, odb_transaction_env(the_repository->objects->transaction));
    - 
    - 	prepare_push_cert_sha1(&opt);
    - 
    -@@ builtin/receive-pack.c: static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
    - 		    !delayed_reachability_test(si, i))
    - 			oid_array_append(&extra, &si->shallow->oid[i]);
    - 
    --	opt.env = tmp_objdir_env(tmp_objdir);
    -+	opt.env = odb_transaction_env(the_repository->objects->transaction);
    - 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
    - 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
    - 		rollback_shallow_file(the_repository, &shallow_lock);
    -@@ builtin/receive-pack.c: static void set_connectivity_errors(struct command *commands,
    - 			/* to be checked in update_shallow_ref() */
    - 			continue;
    - 
    --		opt.env = tmp_objdir_env(tmp_objdir);
    -+		opt.env = odb_transaction_env(the_repository->objects->transaction);
    - 		if (!check_connected(command_singleton_iterator, &singleton,
    - 				     &opt))
    - 			continue;
    -@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
    - 		data.si = si;
    - 		opt.err_fd = err_fd;
    - 		opt.progress = err_fd && !quiet;
    --		opt.env = tmp_objdir_env(tmp_objdir);
    -+		opt.env = odb_transaction_env(the_repository->objects->transaction);
    - 		opt.exclude_hidden_refs_section = "receive";
    - 
    - 		if (check_connected(iterate_receive_command_list, &data, &opt))
    -@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
    - 	 * Now we'll start writing out refs, which means the objects need
    - 	 * to be in their final positions so that other processes can see them.
    - 	 */
    --	if (tmp_objdir_migrate(tmp_objdir) < 0) {
    -+	if (odb_transaction_commit(the_repository->objects->transaction)) {
    - 		for (cmd = commands; cmd; cmd = cmd->next) {
    - 			if (!cmd->error_string)
    - 				cmd->error_string = "unable to migrate objects to permanent storage";
    - 		}
    - 		return;
    - 	}
    --	tmp_objdir = NULL;
    - 
    - 	check_aliased_updates(commands);
    - 
    -@@ builtin/receive-pack.c: static void push_header_arg(struct strvec *args, struct pack_header *hdr)
    - 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
    - }
    - 
    --static const char *unpack(int err_fd, struct shallow_info *si)
    -+static const char *unpack(int err_fd, struct shallow_info *si,
    -+			  struct odb_transaction *transaction)
    - {
    - 	struct pack_header hdr;
    - 	const char *hdr_err;
    -@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
    - 		strvec_push(&child.args, alt_shallow_file);
    - 	}
    - 
    --	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
    --	if (!tmp_objdir) {
    --		if (err_fd > 0)
    --			close(err_fd);
    --		return "unable to create temporary object directory";
    --	}
    --	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
    --
    --	/*
    --	 * Normally we just pass the tmp_objdir environment to the child
    --	 * processes that do the heavy lifting, but we may need to see these
    --	 * objects ourselves to set up shallow information.
    --	 */
    --	tmp_objdir_add_as_alternate(tmp_objdir);
    -+	strvec_pushv(&child.env, odb_transaction_env(transaction));
    - 
    - 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
    - 		strvec_push(&child.args, "unpack-objects");
    -@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
    - 	return NULL;
    - }
    - 
    --static const char *unpack_with_sideband(struct shallow_info *si)
    -+static const char *unpack_with_sideband(struct shallow_info *si,
    -+					struct odb_transaction *transaction)
    - {
    - 	struct async muxer;
    - 	const char *ret;
    - 
    - 	if (!use_sideband)
    --		return unpack(0, si);
    -+		return unpack(0, si, transaction);
    - 
    - 	use_keepalive = KEEPALIVE_AFTER_NUL;
    - 	memset(&muxer, 0, sizeof(muxer));
    -@@ builtin/receive-pack.c: static const char *unpack_with_sideband(struct shallow_info *si)
    - 	if (start_async(&muxer))
    - 		return NULL;
    - 
    --	ret = unpack(muxer.in, si);
    -+	ret = unpack(muxer.in, si, transaction);
    - 
    - 	finish_async(&muxer);
    - 	return ret;
    -@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
    - 	struct oid_array ref = OID_ARRAY_INIT;
    - 	struct shallow_info si;
    - 	struct packet_reader reader;
    -+	struct odb_transaction *transaction = NULL;
    - 
    - 	struct option options[] = {
    - 		OPT__QUIET(&quiet, N_("quiet")),
    -@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
    - 		if (!si.nr_ours && !si.nr_theirs)
    - 			shallow_update = 0;
    - 		if (!delete_only(commands)) {
    --			unpack_status = unpack_with_sideband(&si);
    -+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
    -+				unpack_status = "unable to start ODB transaction";
    -+			else
    -+				unpack_status = unpack_with_sideband(&si, transaction);
    - 			update_shallow_info(commands, &si, &ref);
    - 		}
    - 		use_keepalive = KEEPALIVE_ALWAYS;
    -
      ## builtin/unpack-objects.c ##
     @@ builtin/unpack-objects.c: static void unpack_all(void)
      		progress = start_progress(the_repository,
    @@ builtin/update-index.c: int cmd_update_index(int argc,
     
      ## cache-tree.c ##
     @@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)
    - 
      	trace_performance_enter();
      	trace2_region_enter("cache_tree", "update", istate->repo);
    --	odb_transaction_begin_or_die(the_repository->objects, &transaction);
    -+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
    + 	if (!inflight)
    +-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
    ++		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
      	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
      		       "", 0, &skip, flags);
    - 	odb_transaction_commit(transaction);
    + 	if (!inflight)
     
      ## object-file.c ##
     @@ object-file.c: struct odb_transaction_files {
    @@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
      		return -1;
      
     @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
    - 			struct object_database *odb = the_repository->objects;
    - 			struct odb_transaction *transaction;
    + 			int inflight = !!transaction;
      
    --			odb_transaction_begin_or_die(odb, &transaction);
    -+			odb_transaction_begin_or_die(odb, &transaction, 0);
    - 			ret = odb_transaction_write_object_stream(odb->transaction,
    + 			if (!inflight)
    +-				odb_transaction_begin_or_die(odb, &transaction);
    ++				odb_transaction_begin_or_die(odb, &transaction, 0);
    + 			ret = odb_transaction_write_object_stream(transaction,
      								  &stream,
      								  xsize_t(st->st_size),
    -@@ object-file.c: static const char **odb_transaction_files_env(struct odb_transaction *base)
    +@@ object-file.c: static int odb_transaction_files_env(struct odb_transaction *base,
      }
      
      int odb_transaction_files_begin(struct odb_source *source,
    @@ object-file.c: static const char **odb_transaction_files_env(struct odb_transact
     +				enum odb_transaction_flags flags)
      {
      	struct odb_transaction_files *transaction;
    - 	struct object_database *odb = source->odb;
    + 
     @@ object-file.c: int odb_transaction_files_begin(struct odb_source *source,
      	transaction->base.commit = odb_transaction_files_commit;
      	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
    @@ object-file.c: int odb_transaction_files_begin(struct odb_source *source,
     +	if (flags & ODB_TRANSACTION_RECEIVE) {
     +		/*
     +		 * ODB transactions for git-receive-pack(1) eagerly create a
    -+		 * temporary directory and use a different prefix.
    ++		 * temporary directory and use a different temporary directory
    ++		 * prefix.
    ++		 *
    ++		 * NEEDSWORK: This transaction flag is only used by the "files"
    ++		 * backend to special case temporary directory set up and
    ++		 * handling. Ideally transaction users should not have to care
    ++		 * though. To avoid this, we could eagerly create the temporary
    ++		 * directory and use the same prefix name for all transactions.
     +		 */
     +		transaction->prefix = "incoming";
     +		if (odb_transaction_files_prepare(&transaction->base)) {
    @@ object-file.h
      /* The maximum size for an object header. */
      #define MAX_HEADER_LEN 32
     @@ object-file.h: struct odb_transaction;
    -  * pending, out is set to NULL.
    +  * to make new objects visible.
       */
      int odb_transaction_files_begin(struct odb_source *source,
     -				struct odb_transaction **out);
    @@ odb/transaction.c
      {
      	int ret;
      
    -@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
    - 		return 0;
    - 	}
    + 	if (odb->transaction)
    + 		return -1;
      
     -	ret = odb_source_begin_transaction(odb->sources, out);
     +	ret = odb_source_begin_transaction(odb->sources, out, flags);
    @@ odb/transaction.h
      /*
       * A transaction may be started for an object database prior to writing new
     @@ odb/transaction.h: struct odb_transaction {
    - 	const char **(*env)(struct odb_transaction *transaction);
    + 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
      };
      
    ++/* Flags used to configure an ODB transaction. */
     +enum odb_transaction_flags {
    ++	/* Configures the transaction for use with git-receive-pack(1). */
     +	ODB_TRANSACTION_RECEIVE = (1 << 0),
     +};
     +
    @@ odb/transaction.h: struct odb_transaction {
       * Starts an ODB transaction and returns it via `out`. Subsequent objects are
       * written to the transaction and not committed until odb_transaction_commit()
     @@ odb/transaction.h: struct odb_transaction {
    -  * error. If the ODB already has a pending transaction, `out` is set to NULL.
    +  * ODB already has an inflight transaction pending.
       */
      int odb_transaction_begin(struct object_database *odb,
     -			  struct odb_transaction **out);
    @@ odb/transaction.h: struct odb_transaction {
     
      ## read-cache.c ##
     @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
    - 	 * This function is invoked from commands other than 'add', which
      	 * may not have their own transaction active.
      	 */
    --	odb_transaction_begin_or_die(repo->objects, &transaction);
    -+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
    + 	if (!inflight)
    +-		odb_transaction_begin_or_die(repo->objects, &transaction);
    ++		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
      	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
    - 	odb_transaction_commit(transaction);
    - 
    + 	if (!inflight)
    + 		odb_transaction_commit(transaction);
 -:  ---------- > 10:  6c8d878349 builtin/receive-pack: drop redundant tmpdir env
 4:  e9303f9b08 ! 11:  8db95fef56 odb/transaction: propagate commit errors
    @@ Metadata
     Author: Justin Tobler <jltobler@gmail.com>
     
      ## Commit message ##
    -    odb/transaction: propagate commit errors
    +    builtin/receive-pack: stage incoming objects via ODB transactions
     
    -    When `odb_transaction_commit()` is invoked, the return value of the
    -    backend commit callback is silently discarded. A backend has no way
    -    to signal that committing failed, such as when the "files" backend
    -    cannot migrate its temporary object directory into the permanent
    -    ODB.
    +    Objects received by git-receive-pack(1) are quarantined in a temporary
    +    "incoming" directory and migrated into the object database prior to the
    +    reference updates. The quarantine is currently managed through
    +    `tmp_objdir` directly. In a pluggable ODB future, how exactly an object
    +    gets written to a transaction may vary for a given ODB source. Refactor
    +    git-receive-pack(1) to use the ODB transaction interfaces to manage the
    +    object staging area in a more agnostic manner accordingly.
     
    -    In a subsequent commit, git-receive-pack(1) starts using ODB transaction
    -    to stage objects and consequently cares about such failures so it can
    -    handle the error appropriately. Change the commit callback signature to
    -    return an int error code and have `odb_transaction_commit()` forward it
    -    accordingly.
    +    Note that the ODB transaction is now responsible for managing the
    +    primary and alternate ODBs for the repository. One small change as a
    +    result is that the temporary directory is now applied as the primary ODB
    +    in the main process instead of an alternate. This does not change
    +    anything for git-receive-pack(1) though because it only needs access to
    +    the newly written objects and doesn't care how exactly it is set up.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
    - ## odb/transaction.c ##
    -@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
    - 	return ret;
    + ## builtin/receive-pack.c ##
    +@@
    + #include "sigchain.h"
    + #include "string-list.h"
    + #include "strvec.h"
    +-#include "tmp-objdir.h"
    + #include "trace.h"
    + #include "trace2.h"
    + #include "version.h"
    +@@ builtin/receive-pack.c: static enum {
    + } use_keepalive;
    + static int keepalive_in_sec = 5;
    + 
    +-static struct tmp_objdir *tmp_objdir;
    +-
    + static struct proc_receive_ref {
    + 	unsigned int want_add:1,
    + 		     want_delete:1,
    +@@ builtin/receive-pack.c: static void receive_hook_feed_state_free(void *data)
    + static int run_receive_hook(struct command *commands,
    + 			    const char *hook_name,
    + 			    int skip_broken,
    ++			    struct odb_transaction *transaction,
    + 			    const struct string_list *push_options)
    + {
    + 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
    +@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
    + 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
    + 	}
    + 
    +-	if (tmp_objdir)
    +-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
    ++	if (transaction)
    ++		odb_transaction_env(transaction, &opt.env);
    + 
    + 	prepare_push_cert_sha1(&opt);
    + 
    +@@ builtin/receive-pack.c: static const struct object_id *command_singleton_iterator(void *cb_data)
      }
      
    --void odb_transaction_commit(struct odb_transaction *transaction)
    -+int odb_transaction_commit(struct odb_transaction *transaction)
    + static void set_connectivity_errors(struct command *commands,
    +-				    struct shallow_info *si)
    ++				    struct shallow_info *si,
    ++				    struct odb_transaction *transaction)
      {
    -+	int ret;
    + 	struct command *cmd;
    + 
    + 	for (cmd = commands; cmd; cmd = cmd->next) {
    + 		struct command *singleton = cmd;
    + 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
    ++		struct strvec env = STRVEC_INIT;
    + 
    + 		if (shallow_update && si->shallow_ref[cmd->index])
    + 			/* to be checked in update_shallow_ref() */
    + 			continue;
    + 
    +-		opt.env = tmp_objdir_env(tmp_objdir);
    ++		odb_transaction_env(transaction, &env);
    ++		opt.env = env.v;
     +
    - 	if (!transaction)
    --		return;
    -+		return 0;
    + 		if (!check_connected(command_singleton_iterator, &singleton,
    + 				     &opt))
    + 			continue;
      
    - 	/*
    - 	 * Ensure the transaction ending matches the pending transaction.
    - 	 */
    - 	ASSERT(transaction == transaction->source->odb->transaction);
    + 		cmd->error_string = "missing necessary objects";
    ++
    ++		strvec_clear(&env);
    + 	}
    + }
    + 
    +@@ builtin/receive-pack.c: static void execute_commands_atomic(struct command *commands,
    + static void execute_commands(struct command *commands,
    + 			     const char *unpacker_error,
    + 			     struct shallow_info *si,
    ++			     struct odb_transaction *transaction,
    + 			     const struct string_list *push_options)
    + {
    + 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
    +@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
    + 	}
      
    --	transaction->commit(transaction);
    -+	ret = transaction->commit(transaction);
    -+	if (ret)
    -+		return ret;
    + 	if (!skip_connectivity_check) {
    ++		struct strvec env = STRVEC_INIT;
     +
    - 	transaction->source->odb->transaction = NULL;
    - 	free(transaction);
    + 		if (use_sideband) {
    + 			memset(&muxer, 0, sizeof(muxer));
    + 			muxer.proc = copy_to_sideband;
    +@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
    + 		data.si = si;
    + 		opt.err_fd = err_fd;
    + 		opt.progress = err_fd && !quiet;
    +-		opt.env = tmp_objdir_env(tmp_objdir);
    ++		odb_transaction_env(transaction, &env);
    ++		opt.env = env.v;
    + 		opt.exclude_hidden_refs_section = "receive";
    + 
    + 		if (check_connected(iterate_receive_command_list, &data, &opt))
    +-			set_connectivity_errors(commands, si);
    ++			set_connectivity_errors(commands, si, transaction);
    + 
    + 		if (use_sideband)
    + 			finish_async(&muxer);
     +
    -+	return 0;
    ++		strvec_clear(&env);
    + 	}
    + 
    + 	reject_updates_to_hidden(commands);
    +@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
    + 		}
    + 	}
    + 
    +-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
    ++	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
    + 		for (cmd = commands; cmd; cmd = cmd->next) {
    + 			if (!cmd->error_string)
    + 				cmd->error_string = "pre-receive hook declined";
    +@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
    + 	 * Now we'll start writing out refs, which means the objects need
    + 	 * to be in their final positions so that other processes can see them.
    + 	 */
    +-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
    ++	if (odb_transaction_commit(transaction)) {
    + 		for (cmd = commands; cmd; cmd = cmd->next) {
    + 			if (!cmd->error_string)
    + 				cmd->error_string = "unable to migrate objects to permanent storage";
    + 		}
    + 		return;
    + 	}
    +-	tmp_objdir = NULL;
    + 
    + 	check_aliased_updates(commands);
    + 
    +@@ builtin/receive-pack.c: static void push_header_arg(struct strvec *args, struct pack_header *hdr)
    + 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
      }
      
    - int odb_transaction_write_object_stream(struct odb_transaction *transaction,
    -
    - ## odb/transaction.h ##
    -@@ odb/transaction.h: static inline void odb_transaction_begin_or_die(struct object_database *odb,
    -  * Commits an ODB transaction making the written objects visible. If the
    -  * specified transaction is NULL, the function is a no-op.
    -  */
    --void odb_transaction_commit(struct odb_transaction *transaction);
    -+int odb_transaction_commit(struct odb_transaction *transaction);
    - 
    - /*
    -  * Writes the object in the provided stream into the transaction. The resulting
    +-static const char *unpack(int err_fd, struct shallow_info *si)
    ++static const char *unpack(int err_fd, struct shallow_info *si,
    ++			  struct odb_transaction *transaction)
    + {
    + 	struct pack_header hdr;
    + 	const char *hdr_err;
    +@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
    + 		strvec_push(&child.args, alt_shallow_file);
    + 	}
    + 
    +-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
    +-	if (!tmp_objdir) {
    +-		if (err_fd > 0)
    +-			close(err_fd);
    +-		return "unable to create temporary object directory";
    +-	}
    +-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
    +-
    +-	/*
    +-	 * Normally we just pass the tmp_objdir environment to the child
    +-	 * processes that do the heavy lifting, but we may need to see these
    +-	 * objects ourselves to set up shallow information.
    +-	 */
    +-	tmp_objdir_add_as_alternate(tmp_objdir);
    ++	odb_transaction_env(transaction, &child.env);
    + 
    + 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
    + 		strvec_push(&child.args, "unpack-objects");
    +@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
    + 	return NULL;
    + }
    + 
    +-static const char *unpack_with_sideband(struct shallow_info *si)
    ++static const char *unpack_with_sideband(struct shallow_info *si,
    ++					struct odb_transaction *transaction)
    + {
    + 	struct async muxer;
    + 	const char *ret;
    + 
    + 	if (!use_sideband)
    +-		return unpack(0, si);
    ++		return unpack(0, si, transaction);
    + 
    + 	use_keepalive = KEEPALIVE_AFTER_NUL;
    + 	memset(&muxer, 0, sizeof(muxer));
    +@@ builtin/receive-pack.c: static const char *unpack_with_sideband(struct shallow_info *si)
    + 	if (start_async(&muxer))
    + 		return NULL;
    + 
    +-	ret = unpack(muxer.in, si);
    ++	ret = unpack(muxer.in, si, transaction);
    + 
    + 	finish_async(&muxer);
    + 	return ret;
    +@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
    + 	struct oid_array ref = OID_ARRAY_INIT;
    + 	struct shallow_info si;
    + 	struct packet_reader reader;
    ++	struct odb_transaction *transaction = NULL;
    + 
    + 	struct option options[] = {
    + 		OPT__QUIET(&quiet, N_("quiet")),
    +@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
    + 		if (!si.nr_ours && !si.nr_theirs)
    + 			shallow_update = 0;
    + 		if (!delete_only(commands)) {
    +-			unpack_status = unpack_with_sideband(&si);
    ++			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
    ++				unpack_status = "unable to start object transaction";
    ++			else
    ++				unpack_status = unpack_with_sideband(&si, transaction);
    + 			update_shallow_info(commands, &si, &ref);
    + 		}
    + 		use_keepalive = KEEPALIVE_ALWAYS;
    +-		execute_commands(commands, unpack_status, &si,
    ++		execute_commands(commands, unpack_status, &si, transaction,
    + 				 &push_options);
    + 		delete_tempfile(&pack_lockfile);
    + 		sigchain_push(SIGPIPE, SIG_IGN);
    +@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
    + 		else if (report_status)
    + 			report(commands, unpack_status);
    + 		sigchain_pop(SIGPIPE);
    +-		run_receive_hook(commands, "post-receive", 1,
    ++		run_receive_hook(commands, "post-receive", 1, NULL,
    + 				 &push_options);
    + 		run_update_post_hook(commands);
    + 		free_commands(commands);

base-commit: ab776a62a78576513ee121424adb19597fbb7613
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 01/11] object-file: rename files transaction prepare function

[Justin Tobler]

The "files" ODB transaction backend lazily creates a temporary object
directory when the first loose object is written to the transaction via
`prepare_loose_object_transaction()`. In a subsequent commit, the
temporary directory is used to also write packfiles to.

Rename the function to `odb_transaction_files_prepare()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/object-file.c b/object-file.c
index e3d92bbda2..a3eb8d71dd 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 02/11] object-file: rename files transaction fsync function

[Justin Tobler]

When writing an object to a "files" ODB transaction, a full hardware
flush is not initially performed during the fsync in
`fsync_loose_object_transaction()` and instead delayed until the
transaction is later committed.

To be more consistent with other "files" ODB transaction helpers, rename
the function to `odb_transaction_files_fsync()` accordingly. The
conditional in the helper is also slightly restructured to improve
clarity to readers.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/object-file.c b/object-file.c
index a3eb8d71dd..d68824bb44 100644
--- a/object-file.c
+++ b/object-file.c
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}
+
 	/*
 	 * If we have an active ODB transaction, we issue a call that
 	 * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
 	 * that all objects are durable before any renames occur. The code in
-	 * fsync_loose_object_transaction has already issued a writeout
+	 * odb_transaction_files_fsync has already issued a writeout
 	 * request, but it has not flushed any writeback cache in the storage
 	 * hardware or any filesystem logs. This fsync call acts as a barrier
 	 * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose,
 		goto out;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
+		odb_transaction_files_fsync(loose->base.odb->transaction, fd, filename);
 	else if (fsync_object_files > 0)
 		fsync_or_die(fd, filename);
 	else
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 03/11] object-file: embed transaction flush logic in commit function

[Justin Tobler]

When a "files" transaction is committed,
`flush_loose_object_transaction()` is invoked to handle performing a
hardware flush along with migrating the temporary object directory into
the primary. In a subsequent commit, the temporary directory is also
used to write packfiles.

Instead of maintaining a separate helper function, embed the logic to
flush and migrate the temporary directory directly into
`odb_transaction_files_commit()`.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)

diff --git a/object-file.c b/object-file.c
index d68824bb44..33bd6c6810 100644
--- a/object-file.c
+++ b/object-file.c
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base,
 	}
 }
 
-/*
- * Cleanup after batch-mode fsync_object_files.
- */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
-{
-	struct strbuf temp_path = STRBUF_INIT;
-	struct tempfile *temp;
-
-	if (!transaction->objdir)
-		return;
-
-	/*
-	 * Issue a full hardware flush against a temporary file to ensure
-	 * that all objects are durable before any renames occur. The code in
-	 * odb_transaction_files_fsync has already issued a writeout
-	 * request, but it has not flushed any writeback cache in the storage
-	 * hardware or any filesystem logs. This fsync call acts as a barrier
-	 * to ensure that the data in each new object file is durable before
-	 * the final name is visible.
-	 */
-	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-		    repo_get_object_directory(transaction->base.source->odb->repo));
-	temp = xmks_tempfile(temp_path.buf);
-	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-	delete_tempfile(&temp);
-	strbuf_release(&temp_path);
-
-	/*
-	 * Make the object files visible in the primary ODB after their data is
-	 * fully durable.
-	 */
-	tmp_objdir_migrate(transaction->objdir);
-	transaction->objdir = NULL;
-}
-
 /* Finalize a file on disk, and close it. */
 static void close_loose_object(struct odb_source_loose *loose,
 			       int fd, const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (transaction->objdir) {
+		struct strbuf temp_path = STRBUF_INIT;
+		struct tempfile *temp;
+
+		/*
+		 * Issue a full hardware flush against a temporary file to ensure
+		 * that all objects are durable before any renames occur. The code in
+		 * odb_transaction_files_fsync has already issued a writeout
+		 * request, but it has not flushed any writeback cache in the storage
+		 * hardware or any filesystem logs. This fsync call acts as a barrier
+		 * to ensure that the data in each new object file is durable before
+		 * the final name is visible.
+		 */
+		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+			    repo_get_object_directory(transaction->base.source->odb->repo));
+		temp = xmks_tempfile(temp_path.buf);
+		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+		delete_tempfile(&temp);
+		strbuf_release(&temp_path);
+
+		/*
+		 * Make the object files visible in the primary ODB after their data is
+		 * fully durable.
+		 */
+		tmp_objdir_migrate(transaction->objdir);
+		transaction->objdir = NULL;
+	}
+
 	flush_packfile_transaction(transaction);
 }
 
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 04/11] object-file: drop check for inflight transactions

[Justin Tobler]

ODB transactions are started via `odb_transaction_begin()` and contain
validation to avoid starting multiple transactions at the same time. The
"files" backend also has the same logic, but is redundant due to the
generic layer already handling it. Drop this validation from the "files"
backend accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 4 ----
 object-file.h | 3 +--
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/object-file.c b/object-file.c
index 33bd6c6810..e51389833a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 {
 	struct odb_transaction_files *transaction;
-	struct object_database *odb = source->odb;
-
-	if (odb->transaction)
-		return NULL;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
diff --git a/object-file.h b/object-file.h
index 528c4e6e69..ea43d818f0 100644
--- a/object-file.h
+++ b/object-file.h
@@ -194,8 +194,7 @@ struct odb_transaction;
 /*
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * to make new objects visible.
  */
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
 
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 05/11] object-file: propagate files transaction errors

[Justin Tobler]

The "files" transaction backend may encounter errors related to managing
the temporary directory used to stage objects, but silently ignores
these errors. Instead return errors encountered in the
`odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
callers to handle them as needed.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c      | 26 ++++++++++++++++++--------
 object-file.h      |  3 ++-
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  7 +++++--
 4 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/object-file.c b/object-file.c
index e51389833a..64cb874fe7 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return -1;
+
 		transaction->objdir = NULL;
 	}
 
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
diff --git a/object-file.h b/object-file.h
index ea43d818f0..1a023226ac 100644
--- a/object-file.h
+++ b/object-file.h
@@ -196,6 +196,7 @@ struct odb_transaction;
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 5bdd042922..2545bd81d4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/transaction.h b/odb/transaction.h
index 854fda06f5..d52f0533ce 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -16,8 +16,11 @@ struct odb_transaction {
 	/* The ODB source the transaction is opened against. */
 	struct odb_source *source;
 
-	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	/*
+	 * The ODB source specific callback invoked to commit a transaction.
+	 * Returns 0 on success, a negative error code otherwise.
+	 */
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 06/11] odb/transaction: propagate begin errors

[Justin Tobler]

When `odb_transaction_begin()` is invoked, the function returns the
transaction pointer directly. There is no way for the backend to
signal that it failed to set up its state, such as when creating the
temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB
transactions and needs to be able to report such failures rather
than silently ignore them. Refactor `odb_transaction_begin()` to
return an int error code and write the resulting transaction into an
out parameter. Also introduce `odb_transaction_begin_or_die()` as a
convenience for callsites that do not need to handle errors
explicitly.

Note that `odb_transaction_begin()` now returns an error when the ODB
already has an inflight transaction pending. ODB transaction call sites
that may encounter an inflight transaction are updated to explicitly
handle this case.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  7 +++++--
 object-file.c            | 10 +++++++---
 odb/transaction.c        | 12 ++++++++----
 odb/transaction.h        | 20 ++++++++++++++++----
 read-cache.c             |  7 +++++--
 8 files changed, 44 insertions(+), 18 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index c859f66519..3d5d9cfdb9 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index f3849bb654..d0136cdd99 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 3d6646c318..17f3ea284c 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 184f7e2635..8eec1d4d52 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
diff --git a/object-file.c b/object-file.c
index 64cb874fe7..cd1aa36462 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction = odb->transaction;
+			int inflight = !!transaction;
 
-			ret = odb_transaction_write_object_stream(odb->transaction,
+			if (!inflight)
+				odb_transaction_begin_or_die(odb, &transaction);
+			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
 								  oid);
-			odb_transaction_commit(transaction);
+			if (!inflight)
+				odb_transaction_commit(transaction);
 		} else {
 			ret = hash_blob_stream(&stream,
 					       the_repository->hash_algo, oid,
diff --git a/odb/transaction.c b/odb/transaction.c
index b16e07aebf..df4275151b 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -2,14 +2,18 @@
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return -1;
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;
 
-	return odb->transaction;
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
diff --git a/odb/transaction.h b/odb/transaction.h
index d52f0533ce..36032a5365 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -1,6 +1,8 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "git-compat-util.h"
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -36,11 +38,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
diff --git a/read-cache.c b/read-cache.c
index 21ca58beea..d511d25834 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 		       const struct pathspec *pathspec, char *ps_matched,
 		       int include_sparse, int flags, int ignored_too )
 {
+	int inflight = !!repo->objects->transaction;
 	struct odb_transaction *transaction;
 	struct update_callback_data data;
 	struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 
 	release_revisions(&rev);
 	return !!data.add_errors;
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 07/11] odb/transaction: propagate commit errors

[Justin Tobler]

When `odb_transaction_commit()` is invoked, the return value of the
backend commit callback is silently discarded. A backend has no way
to signal that committing failed, such as when the "files" backend
cannot migrate its temporary object directory into the permanent
ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction
to stage objects and consequently cares about such failures so it can
handle the error appropriately. Change the commit callback signature to
return an int error code and have `odb_transaction_commit()` forward it
accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 odb/transaction.c | 13 ++++++++++---
 odb/transaction.h |  7 ++++---
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/odb/transaction.c b/odb/transaction.c
index df4275151b..51af2c9a61 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
+	if (ret)
+		return ret;
+
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return 0;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
diff --git a/odb/transaction.h b/odb/transaction.h
index 36032a5365..9557ee0fd2 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -55,10 +55,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 }
 
 /*
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
+ * Commits an ODB transaction making the written objects visible. Returns 0 on
+ * success, a negative error code otherwise. Note that, if the specified
+ * transaction is NULL, the function is a no-op and no error is returned.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 08/11] odb/transaction: add transaction env interface

[Justin Tobler]

The ODB transaction backend is responsible for creating/managing its own
staging area for writing objects. Other child processes spawned by Git
may need access to uncommitted objects or write new objects in the
staging area though.

Introduce `odb_transaction_env()` which is expected to provide the set
of environment variables needed by a child process to access the
transaction's staging area.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c     | 14 ++++++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)

diff --git a/object-file.c b/object-file.c
index cd1aa36462..9b8ee6f36c 100644
--- a/object-file.c
+++ b/object-file.c
@@ -27,6 +27,7 @@
 #include "path.h"
 #include "read-cache-ll.h"
 #include "setup.h"
+#include "strvec.h"
 #include "tempfile.h"
 #include "tmp-objdir.h"
 
@@ -1687,6 +1688,18 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
+	return 0;
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1696,6 +1709,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
diff --git a/odb/transaction.c b/odb/transaction.c
index 51af2c9a61..acb1c967e7 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -44,3 +44,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+	if (!transaction)
+		return 0;
+
+	return transaction->env(transaction, env);
+}
diff --git a/odb/transaction.h b/odb/transaction.h
index 9557ee0fd2..1c6c97a53e 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -35,6 +35,14 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to populate the provided strvec with the
+	 * environment variables that a child process should inherit so that its
+	 * object writes participate in the transaction. Returns 0 on success, a
+	 * negative error code otherwise.
+	 */
+	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
 /*
@@ -70,4 +78,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for using via child_process.env. Returns 0 on success,
+ * a negative error code otherwise. Note that, if the specified transaction is
+ * NULL, the function is a no-op and no error is returned.
+ */
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
 #endif
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 09/11] odb/transaction: introduce ODB transaction flags

[Justin Tobler]

The temporary directory used by git-receive-pack(1) to write objects is
managed slightly differently than how it is done via ODB transactions:

  - The temporary directory is eagerly created upfront, instead of
    waiting for the first object write.

  - The prefix name of the temporary directory is "incoming" instead of
    "bulk-fsync".

In a subsequent commit, git-receive-pack(1) will use ODB transactions
instead of `tmp_objdir` directly. To provide a means to configure the
same transaction behavior, introduce `enum odb_transaction_flags` and
the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
transactions using the "files" backend to be set up for
git-receive-pack(1). Transaction call sites are updated accordingly to
provide the required flag parameter.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 29 ++++++++++++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 ++++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 15 +++++++++++----
 read-cache.c             |  2 +-
 13 files changed, 61 insertions(+), 22 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 3d5d9cfdb9..60ffbede2b 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index d0136cdd99..c3d0fc7507 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 17f3ea284c..bf6ea60ef4 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 8eec1d4d52..99c6a0a7d0 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
 	if (!inflight)
-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
+		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	if (!inflight)
diff --git a/object-file.c b/object-file.c
index 9b8ee6f36c..d95bdabba5 100644
--- a/object-file.c
+++ b/object-file.c
@@ -498,6 +498,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return -1;
 
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			int inflight = !!transaction;
 
 			if (!inflight)
-				odb_transaction_begin_or_die(odb, &transaction);
+				odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1701,7 +1702,8 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
diff --git a/object-file.h b/object-file.h
index 1a023226ac..bdd2d67a2e 100644
--- a/object-file.h
+++ b/object-file.h
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;
  * to make new objects visible.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 2545bd81d4..534f48aad9 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c
index e004566d76..9644d9d474 100644
--- a/odb/source-inmemory.c
+++ b/odb/source-inmemory.c
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 66e6bb8d3f..57c91986b4 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
diff --git a/odb/source.h b/odb/source.h
index 2192a101b8..3790d03ff2 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
diff --git a/odb/transaction.c b/odb/transaction.c
index acb1c967e7..007ab73c0c 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -3,14 +3,15 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
 	if (odb->transaction)
 		return -1;
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	odb->transaction = *out;
 
 	return ret;
diff --git a/odb/transaction.h b/odb/transaction.h
index 1c6c97a53e..b19f180aee 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -4,7 +4,6 @@
 #include "git-compat-util.h"
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -45,6 +44,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -53,12 +58,14 @@ struct odb_transaction {
  * ODB already has an inflight transaction pending.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
diff --git a/read-cache.c b/read-cache.c
index d511d25834..50e2320c8d 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * may not have their own transaction active.
 	 */
 	if (!inflight)
-		odb_transaction_begin_or_die(repo->objects, &transaction);
+		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	if (!inflight)
 		odb_transaction_commit(transaction);
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 10/11] builtin/receive-pack: drop redundant tmpdir env

[Justin Tobler]

When performing the connectivity checks for a shallow ref in
`update_shallow_ref()`, the child process environment variables are
populated via `tmp_objdir_env()`. This is unnecessary though as
`update_shallow_ref()` is only reached after `tmp_objdir_migrate()` has
been performed which means there is no longer a temporary directory that
needs to be shared with child processes.

Drop the call to `tmp_objdir_env()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/receive-pack.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 19eb6a1b61..50bc05c70c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
-- 
2.55.0.122.gf85a7e6620

[PATCH v2 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

[Justin Tobler]

Objects received by git-receive-pack(1) are quarantined in a temporary
"incoming" directory and migrated into the object database prior to the
reference updates. The quarantine is currently managed through
`tmp_objdir` directly. In a pluggable ODB future, how exactly an object
gets written to a transaction may vary for a given ODB source. Refactor
git-receive-pack(1) to use the ODB transaction interfaces to manage the
object staging area in a more agnostic manner accordingly.

Note that the ODB transaction is now responsible for managing the
primary and alternate ODBs for the repository. One small change as a
result is that the temporary directory is now applied as the primary ODB
in the main process instead of an alternate. This does not change
anything for git-receive-pack(1) though because it only needs access to
the newly written objects and doesn't care how exactly it is set up.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/receive-pack.c | 68 ++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 33 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 50bc05c70c..8b8c20dc1a 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -37,7 +37,6 @@
 #include "sigchain.h"
 #include "string-list.h"
 #include "strvec.h"
-#include "tmp-objdir.h"
 #include "trace.h"
 #include "trace2.h"
 #include "version.h"
@@ -112,8 +111,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
+			    struct odb_transaction *transaction,
 			    const struct string_list *push_options)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (transaction)
+		odb_transaction_env(transaction, &opt.env);
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data)
 }
 
 static void set_connectivity_errors(struct command *commands,
-				    struct shallow_info *si)
+				    struct shallow_info *si,
+				    struct odb_transaction *transaction)
 {
 	struct command *cmd;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		struct command *singleton = cmd;
 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
+		struct strvec env = STRVEC_INIT;
 
 		if (shallow_update && si->shallow_ref[cmd->index])
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
+
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
 
 		cmd->error_string = "missing necessary objects";
+
+		strvec_clear(&env);
 	}
 }
 
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands,
 	}
 
 	if (!skip_connectivity_check) {
+		struct strvec env = STRVEC_INIT;
+
 		if (use_sideband) {
 			memset(&muxer, 0, sizeof(muxer));
 			muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
-			set_connectivity_errors(commands, si);
+			set_connectivity_errors(commands, si, transaction);
 
 		if (use_sideband)
 			finish_async(&muxer);
+
+		strvec_clear(&env);
 	}
 
 	reject_updates_to_hidden(commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands,
 		}
 	}
 
-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
+	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "pre-receive hook declined";
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	odb_transaction_env(transaction, &child.env);
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
-- 
2.55.0.122.gf85a7e6620

Re: [PATCH v2 02/11] object-file: rename files transaction fsync function

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:03PM -0500, Justin Tobler wrote:
> diff --git a/object-file.c b/object-file.c
> index a3eb8d71dd..d68824bb44 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
>  		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
>  }
>  
> -static void fsync_loose_object_transaction(struct odb_transaction *base,
> -					   int fd, const char *filename)
> +static void odb_transaction_files_fsync(struct odb_transaction *base,
> +					int fd, const char *filename)
>  {
>  	struct odb_transaction_files *transaction =
>  		container_of_or_null(base, struct odb_transaction_files, base);
>  
> +	if (!transaction || !transaction->objdir) {
> +		fsync_or_die(fd, filename);
> +		return;
> +	}

The change results in a tiny bit of duplication, but I agree that it's
easier to reason about.

> @@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,

Somewhat funny that the diff renderer decided to put the hunk header
here instead of showing the single line that's now missing from the
diff.

>  	 * before renaming the objects to their final names as part of
>  	 * flush_batch_fsync.
>  	 */
> -	if (!transaction || !transaction->objdir ||
> -	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
> +	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
>  		if (errno == ENOSYS)
>  			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
>  		fsync_or_die(fd, filename);

Thanks!

Patrick

Re: [PATCH v2 03/11] object-file: embed transaction flush logic in commit function

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:04PM -0500, Justin Tobler wrote:
> When a "files" transaction is committed,
> `flush_loose_object_transaction()` is invoked to handle performing a
> hardware flush along with migrating the temporary object directory into
> the primary. In a subsequent commit, the temporary directory is also
> used to write packfiles.
> 
> Instead of maintaining a separate helper function, embed the logic to
> flush and migrate the temporary directory directly into
> `odb_transaction_files_commit()`.

The change itself looks simple enough, but this makes me wonder why we
want to change this. Like, what subsequent step does this enable that
would otherwise be harder to do?

Maybe this will be answered by a subsequent commit.

Patrick

Re: [PATCH v2 04/11] object-file: drop check for inflight transactions

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:05PM -0500, Justin Tobler wrote:
> ODB transactions are started via `odb_transaction_begin()` and contain
> validation to avoid starting multiple transactions at the same time. The
> "files" backend also has the same logic, but is redundant due to the
> generic layer already handling it. Drop this validation from the "files"
> backend accordingly.

Makes sense, and it fixes a layering violation: in the best case, a
source only has to care about itself and not about the owning object
database. Managing object-database-level state should be done by the
object database itself.

Patrick

Re: [PATCH v2 05/11] object-file: propagate files transaction errors

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:06PM -0500, Justin Tobler wrote:
> diff --git a/object-file.c b/object-file.c
> index e51389833a..64cb874fe7 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
>  	 * added at the time they call odb_transaction_files_begin.
>  	 */
>  	if (!transaction || transaction->objdir)
> -		return;
> +		return 0;
>  
>  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> -	if (transaction->objdir)
> -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> +	if (!transaction->objdir)
> +		return -1;

As far as I can see we don't report any errors as part of
`tmp_objdir_create()`, so we should probably print an error here.

> @@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
>  		 * Make the object files visible in the primary ODB after their data is
>  		 * fully durable.
>  		 */
> -		tmp_objdir_migrate(transaction->objdir);
> +		if (tmp_objdir_migrate(transaction->objdir))
> +			return -1;
> +

Likewise.

Patrick

Re: [PATCH v2 06/11] odb/transaction: propagate begin errors

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:07PM -0500, Justin Tobler wrote:
> When `odb_transaction_begin()` is invoked, the function returns the
> transaction pointer directly. There is no way for the backend to
> signal that it failed to set up its state, such as when creating the
> temporary object directory backing the transaction.
> 
> In a subsequent commit, git-receive-pack(1) starts using ODB
> transactions and needs to be able to report such failures rather
> than silently ignore them. Refactor `odb_transaction_begin()` to
> return an int error code and write the resulting transaction into an
> out parameter. Also introduce `odb_transaction_begin_or_die()` as a
> convenience for callsites that do not need to handle errors
> explicitly.
> 
> Note that `odb_transaction_begin()` now returns an error when the ODB
> already has an inflight transaction pending. ODB transaction call sites
> that may encounter an inflight transaction are updated to explicitly
> handle this case.

Yeah, this change is very much welcome and results in much saner
behaviour with less surprises. Thanks for making the change.

> diff --git a/cache-tree.c b/cache-tree.c
> index 184f7e2635..8eec1d4d52 100644
> --- a/cache-tree.c
> +++ b/cache-tree.c
> @@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
>  
>  int cache_tree_update(struct index_state *istate, int flags)
>  {
> +	int inflight = !!the_repository->objects->transaction;
>  	struct odb_transaction *transaction;
>  	int skip, i;
>  
> @@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
>  
>  	trace_performance_enter();
>  	trace2_region_enter("cache_tree", "update", istate->repo);
> -	transaction = odb_transaction_begin(the_repository->objects);
> +	if (!inflight)
> +		odb_transaction_begin_or_die(the_repository->objects, &transaction);
>  	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
>  		       "", 0, &skip, flags);
> -	odb_transaction_commit(transaction);
> +	if (!inflight)
> +		odb_transaction_commit(transaction);
>  	trace2_region_leave("cache_tree", "update", istate->repo);
>  	trace_performance_leave("cache_tree_update");
>  	if (i < 0)

Callsites like this really make me wonder why we even care to create
a transaction in the first place if we basically just commit it
immediately anyway. And while it's a bit sad that we have so many sites
where we don't really know whether we even have a transaction, I think
it's a good change that we have now annotated them clearly. A subsequent
patch series may then eventually refactor those sites so that we stop
depending on `odb->transaction` and inject the transaction via a
parameter.

> diff --git a/odb/transaction.h b/odb/transaction.h
> index d52f0533ce..36032a5365 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -1,6 +1,8 @@
>  #ifndef ODB_TRANSACTION_H
>  #define ODB_TRANSACTION_H
>  
> +#include "git-compat-util.h"

We typically don't include "git-compat-util.h" in header files.

> @@ -36,11 +38,21 @@ struct odb_transaction {
>  };
>  
>  /*
> - * Starts an ODB transaction. Subsequent objects are written to the transaction
> - * and not committed until odb_transaction_commit() is invoked on the
> - * transaction. If the ODB already has a pending transaction, NULL is returned.
> + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
> + * written to the transaction and not committed until odb_transaction_commit()
> + * is invoked on the transaction. Returns 0 on success and a negative value on
> + * error. Note that it is considered an error to start a new transaction if the
> + * ODB already has an inflight transaction pending.
>   */
> -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
> +int odb_transaction_begin(struct object_database *odb,
> +			  struct odb_transaction **out);
> +
> +static inline void odb_transaction_begin_or_die(struct object_database *odb,
> +						struct odb_transaction **out)
> +{
> +	if (odb_transaction_begin(odb, out))
> +		die(_("failed to start ODB transaction"));
> +}

We could make it a bit simpler to use this function by continuing to
return the transaction directly. But on the other hand this results in a
more consistent interface.

Patrick

Re: [PATCH v2 07/11] odb/transaction: propagate commit errors

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:08PM -0500, Justin Tobler wrote:
> diff --git a/odb/transaction.c b/odb/transaction.c
> index df4275151b..51af2c9a61 100644
> --- a/odb/transaction.c
> +++ b/odb/transaction.c
> @@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,
>  	return ret;
>  }
>  
> -void odb_transaction_commit(struct odb_transaction *transaction)
> +int odb_transaction_commit(struct odb_transaction *transaction)
>  {
> +	int ret;
> +
>  	if (!transaction)
> -		return;
> +		return 0;
>  
>  	/*
>  	 * Ensure the transaction ending matches the pending transaction.
>  	 */
>  	ASSERT(transaction == transaction->source->odb->transaction);
>  
> -	transaction->commit(transaction);
> +	ret = transaction->commit(transaction);
> +	if (ret)
> +		return ret;
> +
>  	transaction->source->odb->transaction = NULL;
>  	free(transaction);
> +
> +	return 0;
>  }

Doesn't this cause a leak now?

I think this interface here is doing the same mistake that our reference
transactions did, where we automatically released the transaction on
commit. That caused multiple lifetime issues with references all over
the place.

This isn't an issue introduced by this patch series though, so it's fine
to ignore this for now.

Patrick

Re: [PATCH v2 09/11] odb/transaction: introduce ODB transaction flags

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:10PM -0500, Justin Tobler wrote:
> diff --git a/object-file.c b/object-file.c
> index 9b8ee6f36c..d95bdabba5 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source,
>  	transaction->base.commit = odb_transaction_files_commit;
>  	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
>  	transaction->base.env = odb_transaction_files_env;
> +
> +	transaction->prefix = "bulk-fsync";
> +	if (flags & ODB_TRANSACTION_RECEIVE) {
> +		/*
> +		 * ODB transactions for git-receive-pack(1) eagerly create a
> +		 * temporary directory and use a different temporary directory
> +		 * prefix.
> +		 *
> +		 * NEEDSWORK: This transaction flag is only used by the "files"
> +		 * backend to special case temporary directory set up and
> +		 * handling. Ideally transaction users should not have to care
> +		 * though. To avoid this, we could eagerly create the temporary
> +		 * directory and use the same prefix name for all transactions.
> +		 */

Yup, agreed, thanks for noting this here.

> diff --git a/odb/transaction.h b/odb/transaction.h
> index 1c6c97a53e..b19f180aee 100644
> --- a/odb/transaction.h
> +++ b/odb/transaction.h
> @@ -4,7 +4,6 @@
>  #include "git-compat-util.h"
>  #include "gettext.h"
>  #include "odb.h"
> -#include "odb/source.h"

This is curious, and likely a result of you adding "odb/transaction.h"
to "odb/source.h".

> @@ -45,6 +44,12 @@ struct odb_transaction {
>  	int (*env)(struct odb_transaction *transaction, struct strvec *env);
>  };
>  
> +/* Flags used to configure an ODB transaction. */
> +enum odb_transaction_flags {
> +	/* Configures the transaction for use with git-receive-pack(1). */
> +	ODB_TRANSACTION_RECEIVE = (1 << 0),
> +};
> +
>  /*
>   * Starts an ODB transaction and returns it via `out`. Subsequent objects are
>   * written to the transaction and not committed until odb_transaction_commit()

And this is the reason you have to add the include, so that the flags
are visible in both "odb/source.h" and in "odb/transaction.h".

This makes me wonder whether there's really much value in having this
header here be split out of "odb/source.h".

Patrick

Re: [PATCH v2 10/11] builtin/receive-pack: drop redundant tmpdir env

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:11PM -0500, Justin Tobler wrote:
> When performing the connectivity checks for a shallow ref in
> `update_shallow_ref()`, the child process environment variables are
> populated via `tmp_objdir_env()`. This is unnecessary though as
> `update_shallow_ref()` is only reached after `tmp_objdir_migrate()` has
> been performed which means there is no longer a temporary directory that
> needs to be shared with child processes.

Right. We call it transitively via either `execute_commands_atomic()` or
`execute_commands_not_atomic()`, both of which are called after
`tmp_objdir_migrate()`.

Patrick

Re: [PATCH v2 00/11] receive-pack: use ODB transactions to stage object writes

[Patrick Steinhardt]

On Tue, Jul 07, 2026 at 11:14:01PM -0500, Justin Tobler wrote:
> Changes since V1:
> 
>   - Adapted other "file" ODB transaction helpers to be more consistent
>     with current naming scheme.
>   - Removed redundant NULL transaction handling from
>     `odb_transaction_files_begin()`.
>   - `odb_transaction_begin()` now returns an error if there is already
>     an inflight transaction pending instead of setting the `out` pointer
>     to NULL.
>   - Updated `odb_transaction_env()` to return an error code and append
>     environment variables to a strvec provided as an argument.
>   - Removed redundant setting of tmpdir environment variables for child
>     processes after tmpdir has been migrated.
>   - Split changes adding ODB transaction flags into a separate commit.
>   - Consistently wire the ODB transaction throughout git-receive-pack
>     code instead of reading it from `the_repository`.
>   - Updated user facing error message.
>   - Updated some comments to better document functions/flags.
>   - Clarified some commit messages.
>   - Fixed typos.

I've got a couple smaller nits, but overall I'm quite happy with the
shape of this series now. Thanks!

Patrick

Re: [PATCH v2 03/11] object-file: embed transaction flush logic in commit function

[Justin Tobler]

On 26/07/08 08:41AM, Patrick Steinhardt wrote:
> On Tue, Jul 07, 2026 at 11:14:04PM -0500, Justin Tobler wrote:
> > When a "files" transaction is committed,
> > `flush_loose_object_transaction()` is invoked to handle performing a
> > hardware flush along with migrating the temporary object directory into
> > the primary. In a subsequent commit, the temporary directory is also
> > used to write packfiles.
> > 
> > Instead of maintaining a separate helper function, embed the logic to
> > flush and migrate the temporary directory directly into
> > `odb_transaction_files_commit()`.
> 
> The change itself looks simple enough, but this makes me wonder why we
> want to change this. Like, what subsequent step does this enable that
> would otherwise be harder to do?

Originally, I was planning to rename both
`{fsync,flush}_loose_object_transaction()` to
`odb_transaction_files_{fsync,flush}()` respectively. For the flush
helper though, it's doing more than just "flushing" the temporary
directory files; it's also migrating the objects to the primary ODB and
reconfiguring the repository ODB sources accordingly.

This is really what you think of happening during the commit phase.
Consequently, it made more sense to me organizationally to just embed
the logic `odb_transaction_files_commit()`.

> Maybe this will be answered by a subsequent commit.

Will update the commit message to properly explain this.

-Justin

Re: [PATCH v2 05/11] object-file: propagate files transaction errors

[Justin Tobler]

On 26/07/08 08:41AM, Patrick Steinhardt wrote:
> On Tue, Jul 07, 2026 at 11:14:06PM -0500, Justin Tobler wrote:
> > diff --git a/object-file.c b/object-file.c
> > index e51389833a..64cb874fe7 100644
> > --- a/object-file.c
> > +++ b/object-file.c
> > @@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
> >  	 * added at the time they call odb_transaction_files_begin.
> >  	 */
> >  	if (!transaction || transaction->objdir)
> > -		return;
> > +		return 0;
> >  
> >  	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
> > -	if (transaction->objdir)
> > -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
> > +	if (!transaction->objdir)
> > +		return -1;
> 
> As far as I can see we don't report any errors as part of
> `tmp_objdir_create()`, so we should probably print an error here.
> 
> > @@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
> >  		 * Make the object files visible in the primary ODB after their data is
> >  		 * fully durable.
> >  		 */
> > -		tmp_objdir_migrate(transaction->objdir);
> > +		if (tmp_objdir_migrate(transaction->objdir))
> > +			return -1;
> > +
> 
> Likewise.

Ya, printing some error messages seems like a good idea here. Will do so
in the next version.

-Justin

Re: [PATCH v2 06/11] odb/transaction: propagate begin errors

[Justin Tobler]

On 26/07/08 08:41AM, Patrick Steinhardt wrote:
> On Tue, Jul 07, 2026 at 11:14:07PM -0500, Justin Tobler wrote:
> > @@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
> >  
> >  	trace_performance_enter();
> >  	trace2_region_enter("cache_tree", "update", istate->repo);
> > -	transaction = odb_transaction_begin(the_repository->objects);
> > +	if (!inflight)
> > +		odb_transaction_begin_or_die(the_repository->objects, &transaction);
> >  	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
> >  		       "", 0, &skip, flags);
> > -	odb_transaction_commit(transaction);
> > +	if (!inflight)
> > +		odb_transaction_commit(transaction);
> >  	trace2_region_leave("cache_tree", "update", istate->repo);
> >  	trace_performance_leave("cache_tree_update");
> >  	if (i < 0)
> 
> Callsites like this really make me wonder why we even care to create
> a transaction in the first place if we basically just commit it
> immediately anyway. And while it's a bit sad that we have so many sites
> where we don't really know whether we even have a transaction, I think
> it's a good change that we have now annotated them clearly. A subsequent
> patch series may then eventually refactor those sites so that we stop
> depending on `odb->transaction` and inject the transaction via a
> parameter.

Call sites like the one mentioned above are using ODB transactions as an
optimization to batch the full fsyncs in bulk. In cases where the is not
already a transaction, they start one to take advantage of it.

I fully agree though that an ODB transaction should ideally be started
at a higher layer and wired down to these call sites. I have a couple of
patches in my tree that start to tackle this which I plan to send in
another series. :)

> > diff --git a/odb/transaction.h b/odb/transaction.h
> > index d52f0533ce..36032a5365 100644
> > --- a/odb/transaction.h
> > +++ b/odb/transaction.h
> > @@ -1,6 +1,8 @@
> >  #ifndef ODB_TRANSACTION_H
> >  #define ODB_TRANSACTION_H
> >  
> > +#include "git-compat-util.h"
> 
> We typically don't include "git-compat-util.h" in header files.

Will remove. Thanks

> > @@ -36,11 +38,21 @@ struct odb_transaction {
> >  };
> >  
> >  /*
> > - * Starts an ODB transaction. Subsequent objects are written to the transaction
> > - * and not committed until odb_transaction_commit() is invoked on the
> > - * transaction. If the ODB already has a pending transaction, NULL is returned.
> > + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
> > + * written to the transaction and not committed until odb_transaction_commit()
> > + * is invoked on the transaction. Returns 0 on success and a negative value on
> > + * error. Note that it is considered an error to start a new transaction if the
> > + * ODB already has an inflight transaction pending.
> >   */
> > -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
> > +int odb_transaction_begin(struct object_database *odb,
> > +			  struct odb_transaction **out);
> > +
> > +static inline void odb_transaction_begin_or_die(struct object_database *odb,
> > +						struct odb_transaction **out)
> > +{
> > +	if (odb_transaction_begin(odb, out))
> > +		die(_("failed to start ODB transaction"));
> > +}
> 
> We could make it a bit simpler to use this function by continuing to
> return the transaction directly. But on the other hand this results in a
> more consistent interface.

Ya, I was a bit back and forth about this myself. I ultimately landed on
keeping a more consistent interface though. Happy to change if others
feel differently though.

-Justin

Re: [PATCH v2 07/11] odb/transaction: propagate commit errors

[Justin Tobler]

On 26/07/08 08:41AM, Patrick Steinhardt wrote:
> On Tue, Jul 07, 2026 at 11:14:08PM -0500, Justin Tobler wrote:
> > diff --git a/odb/transaction.c b/odb/transaction.c
> > index df4275151b..51af2c9a61 100644
> > --- a/odb/transaction.c
> > +++ b/odb/transaction.c
> > @@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,
> >  	return ret;
> >  }
> >  
> > -void odb_transaction_commit(struct odb_transaction *transaction)
> > +int odb_transaction_commit(struct odb_transaction *transaction)
> >  {
> > +	int ret;
> > +
> >  	if (!transaction)
> > -		return;
> > +		return 0;
> >  
> >  	/*
> >  	 * Ensure the transaction ending matches the pending transaction.
> >  	 */
> >  	ASSERT(transaction == transaction->source->odb->transaction);
> >  
> > -	transaction->commit(transaction);
> > +	ret = transaction->commit(transaction);
> > +	if (ret)
> > +		return ret;
> > +
> >  	transaction->source->odb->transaction = NULL;
> >  	free(transaction);
> > +
> > +	return 0;
> >  }
> 
> Doesn't this cause a leak now?

Good call. Ya, if odb_transaction_commit() fails, we don't free the
transaction. In the next version I'll go ahead and clear the transaction
if we fail.

> I think this interface here is doing the same mistake that our reference
> transactions did, where we automatically released the transaction on
> commit. That caused multiple lifetime issues with references all over
> the place.

Ya, it probaby makes sense to introduce a separate
`odb_transaction_release()` function to make this explicit and update
callers accordingly. I have another series I working on that introduces
`odb_transaction_abort()`. This might be a good place to add it in too.

-Justin

Re: [PATCH v2 09/11] odb/transaction: introduce ODB transaction flags

[Justin Tobler]

On 26/07/08 08:41AM, Patrick Steinhardt wrote:
> On Tue, Jul 07, 2026 at 11:14:10PM -0500, Justin Tobler wrote:
> > +/* Flags used to configure an ODB transaction. */
> > +enum odb_transaction_flags {
> > +	/* Configures the transaction for use with git-receive-pack(1). */
> > +	ODB_TRANSACTION_RECEIVE = (1 << 0),
> > +};
> > +
> >  /*
> >   * Starts an ODB transaction and returns it via `out`. Subsequent objects are
> >   * written to the transaction and not committed until odb_transaction_commit()
> 
> And this is the reason you have to add the include, so that the flags
> are visible in both "odb/source.h" and in "odb/transaction.h".
> 
> This makes me wonder whether there's really much value in having this
> header here be split out of "odb/source.h".

Ya, I've started wondering the same thing. A transaction implementation
is always going to be tightly coupled to the ODB source it pertains too.
It probably makes sense to merge "odb/transaction.{c,h}" with
"odb/source.{c,h}". I'll leave it as-is for now and likely explore this
is a future series though.

-Justin

Re: [PATCH v2 00/11] receive-pack: use ODB transactions to stage object writes

[Justin Tobler]

On 26/07/08 08:42AM, Patrick Steinhardt wrote:
> On Tue, Jul 07, 2026 at 11:14:01PM -0500, Justin Tobler wrote:
> > Changes since V1:
> > 
> >   - Adapted other "file" ODB transaction helpers to be more consistent
> >     with current naming scheme.
> >   - Removed redundant NULL transaction handling from
> >     `odb_transaction_files_begin()`.
> >   - `odb_transaction_begin()` now returns an error if there is already
> >     an inflight transaction pending instead of setting the `out` pointer
> >     to NULL.
> >   - Updated `odb_transaction_env()` to return an error code and append
> >     environment variables to a strvec provided as an argument.
> >   - Removed redundant setting of tmpdir environment variables for child
> >     processes after tmpdir has been migrated.
> >   - Split changes adding ODB transaction flags into a separate commit.
> >   - Consistently wire the ODB transaction throughout git-receive-pack
> >     code instead of reading it from `the_repository`.
> >   - Updated user facing error message.
> >   - Updated some comments to better document functions/flags.
> >   - Clarified some commit messages.
> >   - Fixed typos.
> 
> I've got a couple smaller nits, but overall I'm quite happy with the
> shape of this series now. Thanks!

Thanks for the review! I'll send another version later today. :)

-Justin

[PATCH v3 00/11] receive-pack: use ODB transactions to stage object writes

[Justin Tobler]

Greetings,

This patch series replaces direct usage of the `tmp_objdir` interfaces
in git-receive-pack(1) to instead use the `odb_transaction` interfaces
to create/manage a staging area to write objects to. The purpose of this
change is to get git-receive-pack(1) one step closer to being ODB
backend agnostic. For now, the object writes themselves are still
"files" backend specific due to being handled by the git-index-pack(1)
and git-unpack-objects(1) child processes. This will be tackled in a
separate series though.

Changes since V2:
  - Clarified commit log reasoning for embedding
    `flush_loose_object_transaction()` logic in commit function.
  - Started printed some error messages on transaction errors.
  - Removed include statement.
  - Fixed transaction leak on `odb_transaction_commit()` error.

Changes since V1:
  - Adapted other "file" ODB transaction helpers to be more consistent
    with current naming scheme.
  - Removed redundant NULL transaction handling from
    `odb_transaction_files_begin()`.
  - `odb_transaction_begin()` now returns an error if there is already
    an inflight transaction pending instead of setting the `out` pointer
    to NULL.
  - Updated `odb_transaction_env()` to return an error code and append
    environment variables to a strvec provided as an argument.
  - Removed redundant setting of tmpdir environment variables for child
    processes after tmpdir has been migrated.
  - Split changes adding ODB transaction flags into a separate commit.
  - Consistently wire the ODB transaction throughout git-receive-pack
    code instead of reading it from `the_repository`.
  - Updated user facing error message.
  - Updated some comments to better document functions/flags.
  - Clarified some commit messages.
  - Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
  object-file: rename files transaction prepare function
  object-file: rename files transaction fsync function
  object-file: embed transaction flush logic in commit function
  object-file: drop check for inflight transactions
  object-file: propagate files transaction errors
  odb/transaction: propagate begin errors
  odb/transaction: propagate commit errors
  odb/transaction: add transaction env interface
  odb/transaction: introduce ODB transaction flags
  builtin/receive-pack: drop redundant tmpdir env
  builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 159 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  32 ++++++--
 odb/transaction.h        |  59 ++++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 241 insertions(+), 130 deletions(-)

Range-diff against v2:
 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 2:  5703a9e93b =  2:  5703a9e93b object-file: rename files transaction fsync function
 3:  4c37398ac8 !  3:  76204847f2 object-file: embed transaction flush logic in commit function
    @@ Commit message
         When a "files" transaction is committed,
         `flush_loose_object_transaction()` is invoked to handle performing a
         hardware flush along with migrating the temporary object directory into
    -    the primary. In a subsequent commit, the temporary directory is also
    -    used to write packfiles.
    +    the primary and configuring the repository ODB source accordingly. The
    +    function name here is a bit misleading because the helper is doing a bit
    +    more than just "flushing" the transaction contents. Also, in a
    +    subsequent commit, the transaction temporary directory is used to stage
    +    packfiles and not just loose objects anymore.
     
    -    Instead of maintaining a separate helper function, embed the logic to
    -    flush and migrate the temporary directory directly into
    -    `odb_transaction_files_commit()`.
    +    Lift the helper function logic directly into
    +    `odb_transaction_files_commit()` to more accurately signal to readers
    +    the operation being performed.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
 4:  623c6b02ea =  4:  c97eb7763f object-file: drop check for inflight transactions
 5:  ca59176657 !  5:  1f3a1f7714 object-file: propagate files transaction errors
    @@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
     -	if (transaction->objdir)
     -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
     +	if (!transaction->objdir)
    -+		return -1;
    ++		return error(_("unable to create temporary object directory"));
     +
     +	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
     +
    @@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *
      		 */
     -		tmp_objdir_migrate(transaction->objdir);
     +		if (tmp_objdir_migrate(transaction->objdir))
    -+			return -1;
    ++			return error(_("unable to migrate temporary objects"));
     +
      		transaction->objdir = NULL;
      	}
 6:  717a1ce9a7 !  6:  09d13272d5 odb/transaction: propagate begin errors
    @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
     
      ## odb/transaction.c ##
     @@
    + #include "git-compat-util.h"
    ++#include "gettext.h"
      #include "odb/source.h"
      #include "odb/transaction.h"
      
    @@ odb/transaction.c
     +
      	if (odb->transaction)
     -		return NULL;
    -+		return -1;
    ++		return error(_("object database transaction already pending"));
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
     +	ret = odb_source_begin_transaction(odb->sources, out);
    @@ odb/transaction.h
      #ifndef ODB_TRANSACTION_H
      #define ODB_TRANSACTION_H
      
    -+#include "git-compat-util.h"
     +#include "gettext.h"
      #include "odb.h"
      #include "odb/source.h"
 7:  ff8e133965 !  7:  12833d6773 odb/transaction: propagate commit errors
    @@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
      
     -	transaction->commit(transaction);
     +	ret = transaction->commit(transaction);
    -+	if (ret)
    -+		return ret;
    -+
      	transaction->source->odb->transaction = NULL;
      	free(transaction);
     +
    -+	return 0;
    ++	return ret;
      }
      
      int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 8:  264ba94b83 =  8:  f2586f2f34 odb/transaction: add transaction env interface
 9:  1e0a491ef2 !  9:  9d082b5e47 odb/transaction: introduce ODB transaction flags
    @@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
     -	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
     +	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
      	if (!transaction->objdir)
    - 		return -1;
    + 		return error(_("unable to create temporary object directory"));
      
     @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
      			int inflight = !!transaction;
    @@ odb/transaction.c
      	int ret;
      
      	if (odb->transaction)
    - 		return -1;
    + 		return error(_("object database transaction already pending"));
      
     -	ret = odb_source_begin_transaction(odb->sources, out);
     +	ret = odb_source_begin_transaction(odb->sources, out, flags);
    @@ odb/transaction.c
     
      ## odb/transaction.h ##
     @@
    - #include "git-compat-util.h"
    + 
      #include "gettext.h"
      #include "odb.h"
     -#include "odb/source.h"
10:  6c8d878349 = 10:  e11d8a6676 builtin/receive-pack: drop redundant tmpdir env
11:  8db95fef56 = 11:  fee57c2817 builtin/receive-pack: stage incoming objects via ODB transactions

base-commit: ab776a62a78576513ee121424adb19597fbb7613
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 01/11] object-file: rename files transaction prepare function

[Justin Tobler]

The "files" ODB transaction backend lazily creates a temporary object
directory when the first loose object is written to the transaction via
`prepare_loose_object_transaction()`. In a subsequent commit, the
temporary directory is used to also write packfiles to.

Rename the function to `odb_transaction_files_prepare()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/object-file.c b/object-file.c
index e3d92bbda2..a3eb8d71dd 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 02/11] object-file: rename files transaction fsync function

[Justin Tobler]

When writing an object to a "files" ODB transaction, a full hardware
flush is not initially performed during the fsync in
`fsync_loose_object_transaction()` and instead delayed until the
transaction is later committed.

To be more consistent with other "files" ODB transaction helpers, rename
the function to `odb_transaction_files_fsync()` accordingly. The
conditional in the helper is also slightly restructured to improve
clarity to readers.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/object-file.c b/object-file.c
index a3eb8d71dd..d68824bb44 100644
--- a/object-file.c
+++ b/object-file.c
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}
+
 	/*
 	 * If we have an active ODB transaction, we issue a call that
 	 * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
 	 * that all objects are durable before any renames occur. The code in
-	 * fsync_loose_object_transaction has already issued a writeout
+	 * odb_transaction_files_fsync has already issued a writeout
 	 * request, but it has not flushed any writeback cache in the storage
 	 * hardware or any filesystem logs. This fsync call acts as a barrier
 	 * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose,
 		goto out;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
+		odb_transaction_files_fsync(loose->base.odb->transaction, fd, filename);
 	else if (fsync_object_files > 0)
 		fsync_or_die(fd, filename);
 	else
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 03/11] object-file: embed transaction flush logic in commit function

[Justin Tobler]

When a "files" transaction is committed,
`flush_loose_object_transaction()` is invoked to handle performing a
hardware flush along with migrating the temporary object directory into
the primary and configuring the repository ODB source accordingly. The
function name here is a bit misleading because the helper is doing a bit
more than just "flushing" the transaction contents. Also, in a
subsequent commit, the transaction temporary directory is used to stage
packfiles and not just loose objects anymore.

Lift the helper function logic directly into
`odb_transaction_files_commit()` to more accurately signal to readers
the operation being performed.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)

diff --git a/object-file.c b/object-file.c
index d68824bb44..33bd6c6810 100644
--- a/object-file.c
+++ b/object-file.c
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base,
 	}
 }
 
-/*
- * Cleanup after batch-mode fsync_object_files.
- */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
-{
-	struct strbuf temp_path = STRBUF_INIT;
-	struct tempfile *temp;
-
-	if (!transaction->objdir)
-		return;
-
-	/*
-	 * Issue a full hardware flush against a temporary file to ensure
-	 * that all objects are durable before any renames occur. The code in
-	 * odb_transaction_files_fsync has already issued a writeout
-	 * request, but it has not flushed any writeback cache in the storage
-	 * hardware or any filesystem logs. This fsync call acts as a barrier
-	 * to ensure that the data in each new object file is durable before
-	 * the final name is visible.
-	 */
-	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-		    repo_get_object_directory(transaction->base.source->odb->repo));
-	temp = xmks_tempfile(temp_path.buf);
-	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-	delete_tempfile(&temp);
-	strbuf_release(&temp_path);
-
-	/*
-	 * Make the object files visible in the primary ODB after their data is
-	 * fully durable.
-	 */
-	tmp_objdir_migrate(transaction->objdir);
-	transaction->objdir = NULL;
-}
-
 /* Finalize a file on disk, and close it. */
 static void close_loose_object(struct odb_source_loose *loose,
 			       int fd, const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (transaction->objdir) {
+		struct strbuf temp_path = STRBUF_INIT;
+		struct tempfile *temp;
+
+		/*
+		 * Issue a full hardware flush against a temporary file to ensure
+		 * that all objects are durable before any renames occur. The code in
+		 * odb_transaction_files_fsync has already issued a writeout
+		 * request, but it has not flushed any writeback cache in the storage
+		 * hardware or any filesystem logs. This fsync call acts as a barrier
+		 * to ensure that the data in each new object file is durable before
+		 * the final name is visible.
+		 */
+		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+			    repo_get_object_directory(transaction->base.source->odb->repo));
+		temp = xmks_tempfile(temp_path.buf);
+		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+		delete_tempfile(&temp);
+		strbuf_release(&temp_path);
+
+		/*
+		 * Make the object files visible in the primary ODB after their data is
+		 * fully durable.
+		 */
+		tmp_objdir_migrate(transaction->objdir);
+		transaction->objdir = NULL;
+	}
+
 	flush_packfile_transaction(transaction);
 }
 
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 04/11] object-file: drop check for inflight transactions

[Justin Tobler]

ODB transactions are started via `odb_transaction_begin()` and contain
validation to avoid starting multiple transactions at the same time. The
"files" backend also has the same logic, but is redundant due to the
generic layer already handling it. Drop this validation from the "files"
backend accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 4 ----
 object-file.h | 3 +--
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/object-file.c b/object-file.c
index 33bd6c6810..e51389833a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 {
 	struct odb_transaction_files *transaction;
-	struct object_database *odb = source->odb;
-
-	if (odb->transaction)
-		return NULL;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
diff --git a/object-file.h b/object-file.h
index 528c4e6e69..ea43d818f0 100644
--- a/object-file.h
+++ b/object-file.h
@@ -194,8 +194,7 @@ struct odb_transaction;
 /*
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * to make new objects visible.
  */
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
 
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 05/11] object-file: propagate files transaction errors

[Justin Tobler]

The "files" transaction backend may encounter errors related to managing
the temporary directory used to stage objects, but silently ignores
these errors. Instead return errors encountered in the
`odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
callers to handle them as needed.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c      | 26 ++++++++++++++++++--------
 object-file.h      |  3 ++-
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  7 +++++--
 4 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/object-file.c b/object-file.c
index e51389833a..3651605ea2 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return error(_("unable to create temporary object directory"));
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return error(_("unable to migrate temporary objects"));
+
 		transaction->objdir = NULL;
 	}
 
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
diff --git a/object-file.h b/object-file.h
index ea43d818f0..1a023226ac 100644
--- a/object-file.h
+++ b/object-file.h
@@ -196,6 +196,7 @@ struct odb_transaction;
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 5bdd042922..2545bd81d4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/transaction.h b/odb/transaction.h
index 854fda06f5..d52f0533ce 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -16,8 +16,11 @@ struct odb_transaction {
 	/* The ODB source the transaction is opened against. */
 	struct odb_source *source;
 
-	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	/*
+	 * The ODB source specific callback invoked to commit a transaction.
+	 * Returns 0 on success, a negative error code otherwise.
+	 */
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 06/11] odb/transaction: propagate begin errors

[Justin Tobler]

When `odb_transaction_begin()` is invoked, the function returns the
transaction pointer directly. There is no way for the backend to
signal that it failed to set up its state, such as when creating the
temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB
transactions and needs to be able to report such failures rather
than silently ignore them. Refactor `odb_transaction_begin()` to
return an int error code and write the resulting transaction into an
out parameter. Also introduce `odb_transaction_begin_or_die()` as a
convenience for callsites that do not need to handle errors
explicitly.

Note that `odb_transaction_begin()` now returns an error when the ODB
already has an inflight transaction pending. ODB transaction call sites
that may encounter an inflight transaction are updated to explicitly
handle this case.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  7 +++++--
 object-file.c            | 10 +++++++---
 odb/transaction.c        | 13 +++++++++----
 odb/transaction.h        | 19 +++++++++++++++----
 read-cache.c             |  7 +++++--
 8 files changed, 44 insertions(+), 18 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index c859f66519..3d5d9cfdb9 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index f3849bb654..d0136cdd99 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 3d6646c318..17f3ea284c 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 184f7e2635..8eec1d4d52 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
diff --git a/object-file.c b/object-file.c
index 3651605ea2..358684beae 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction = odb->transaction;
+			int inflight = !!transaction;
 
-			ret = odb_transaction_write_object_stream(odb->transaction,
+			if (!inflight)
+				odb_transaction_begin_or_die(odb, &transaction);
+			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
 								  oid);
-			odb_transaction_commit(transaction);
+			if (!inflight)
+				odb_transaction_commit(transaction);
 		} else {
 			ret = hash_blob_stream(&stream,
 					       the_repository->hash_algo, oid,
diff --git a/odb/transaction.c b/odb/transaction.c
index b16e07aebf..a5fba7f908 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -1,15 +1,20 @@
 #include "git-compat-util.h"
+#include "gettext.h"
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return error(_("object database transaction already pending"));
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;
 
-	return odb->transaction;
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
diff --git a/odb/transaction.h b/odb/transaction.h
index d52f0533ce..f5c43187c9 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -1,6 +1,7 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -36,11 +37,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
diff --git a/read-cache.c b/read-cache.c
index 21ca58beea..d511d25834 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 		       const struct pathspec *pathspec, char *ps_matched,
 		       int include_sparse, int flags, int ignored_too )
 {
+	int inflight = !!repo->objects->transaction;
 	struct odb_transaction *transaction;
 	struct update_callback_data data;
 	struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 
 	release_revisions(&rev);
 	return !!data.add_errors;
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 07/11] odb/transaction: propagate commit errors

[Justin Tobler]

When `odb_transaction_commit()` is invoked, the return value of the
backend commit callback is silently discarded. A backend has no way
to signal that committing failed, such as when the "files" backend
cannot migrate its temporary object directory into the permanent
ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction
to stage objects and consequently cares about such failures so it can
handle the error appropriately. Change the commit callback signature to
return an int error code and have `odb_transaction_commit()` forward it
accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 odb/transaction.c | 10 +++++++---
 odb/transaction.h |  7 ++++---
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/odb/transaction.c b/odb/transaction.c
index a5fba7f908..0a924e73f7 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -17,19 +17,23 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return ret;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
diff --git a/odb/transaction.h b/odb/transaction.h
index f5c43187c9..3b0a5a78e5 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -54,10 +54,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 }
 
 /*
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
+ * Commits an ODB transaction making the written objects visible. Returns 0 on
+ * success, a negative error code otherwise. Note that, if the specified
+ * transaction is NULL, the function is a no-op and no error is returned.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 08/11] odb/transaction: add transaction env interface

[Justin Tobler]

The ODB transaction backend is responsible for creating/managing its own
staging area for writing objects. Other child processes spawned by Git
may need access to uncommitted objects or write new objects in the
staging area though.

Introduce `odb_transaction_env()` which is expected to provide the set
of environment variables needed by a child process to access the
transaction's staging area.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c     | 14 ++++++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)

diff --git a/object-file.c b/object-file.c
index 358684beae..f0b066798a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -27,6 +27,7 @@
 #include "path.h"
 #include "read-cache-ll.h"
 #include "setup.h"
+#include "strvec.h"
 #include "tempfile.h"
 #include "tmp-objdir.h"
 
@@ -1687,6 +1688,18 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
+	return 0;
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1696,6 +1709,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
diff --git a/odb/transaction.c b/odb/transaction.c
index 0a924e73f7..7f1b30945d 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -42,3 +42,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+	if (!transaction)
+		return 0;
+
+	return transaction->env(transaction, env);
+}
diff --git a/odb/transaction.h b/odb/transaction.h
index 3b0a5a78e5..5e51ce5ca4 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -34,6 +34,14 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to populate the provided strvec with the
+	 * environment variables that a child process should inherit so that its
+	 * object writes participate in the transaction. Returns 0 on success, a
+	 * negative error code otherwise.
+	 */
+	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
 /*
@@ -69,4 +77,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for using via child_process.env. Returns 0 on success,
+ * a negative error code otherwise. Note that, if the specified transaction is
+ * NULL, the function is a no-op and no error is returned.
+ */
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
 #endif
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 09/11] odb/transaction: introduce ODB transaction flags

[Justin Tobler]

The temporary directory used by git-receive-pack(1) to write objects is
managed slightly differently than how it is done via ODB transactions:

  - The temporary directory is eagerly created upfront, instead of
    waiting for the first object write.

  - The prefix name of the temporary directory is "incoming" instead of
    "bulk-fsync".

In a subsequent commit, git-receive-pack(1) will use ODB transactions
instead of `tmp_objdir` directly. To provide a means to configure the
same transaction behavior, introduce `enum odb_transaction_flags` and
the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
transactions using the "files" backend to be set up for
git-receive-pack(1). Transaction call sites are updated accordingly to
provide the required flag parameter.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 29 ++++++++++++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 ++++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 15 +++++++++++----
 read-cache.c             |  2 +-
 13 files changed, 61 insertions(+), 22 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 3d5d9cfdb9..60ffbede2b 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index d0136cdd99..c3d0fc7507 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 17f3ea284c..bf6ea60ef4 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 8eec1d4d52..99c6a0a7d0 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
 	if (!inflight)
-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
+		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	if (!inflight)
diff --git a/object-file.c b/object-file.c
index f0b066798a..d2508d148f 100644
--- a/object-file.c
+++ b/object-file.c
@@ -498,6 +498,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return error(_("unable to create temporary object directory"));
 
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			int inflight = !!transaction;
 
 			if (!inflight)
-				odb_transaction_begin_or_die(odb, &transaction);
+				odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1701,7 +1702,8 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
diff --git a/object-file.h b/object-file.h
index 1a023226ac..bdd2d67a2e 100644
--- a/object-file.h
+++ b/object-file.h
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;
  * to make new objects visible.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 2545bd81d4..534f48aad9 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c
index e004566d76..9644d9d474 100644
--- a/odb/source-inmemory.c
+++ b/odb/source-inmemory.c
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 66e6bb8d3f..57c91986b4 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
diff --git a/odb/source.h b/odb/source.h
index 2192a101b8..3790d03ff2 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
diff --git a/odb/transaction.c b/odb/transaction.c
index 7f1b30945d..edf5488c81 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -4,14 +4,15 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
 	if (odb->transaction)
 		return error(_("object database transaction already pending"));
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	odb->transaction = *out;
 
 	return ret;
diff --git a/odb/transaction.h b/odb/transaction.h
index 5e51ce5ca4..4cb2eafcbf 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -3,7 +3,6 @@
 
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -52,12 +57,14 @@ struct odb_transaction {
  * ODB already has an inflight transaction pending.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
diff --git a/read-cache.c b/read-cache.c
index d511d25834..50e2320c8d 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * may not have their own transaction active.
 	 */
 	if (!inflight)
-		odb_transaction_begin_or_die(repo->objects, &transaction);
+		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	if (!inflight)
 		odb_transaction_commit(transaction);
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 10/11] builtin/receive-pack: drop redundant tmpdir env

[Justin Tobler]

When performing the connectivity checks for a shallow ref in
`update_shallow_ref()`, the child process environment variables are
populated via `tmp_objdir_env()`. This is unnecessary though as
`update_shallow_ref()` is only reached after `tmp_objdir_migrate()` has
been performed which means there is no longer a temporary directory that
needs to be shared with child processes.

Drop the call to `tmp_objdir_env()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/receive-pack.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 19eb6a1b61..50bc05c70c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
-- 
2.55.0.122.gf85a7e6620

[PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

[Justin Tobler]

Objects received by git-receive-pack(1) are quarantined in a temporary
"incoming" directory and migrated into the object database prior to the
reference updates. The quarantine is currently managed through
`tmp_objdir` directly. In a pluggable ODB future, how exactly an object
gets written to a transaction may vary for a given ODB source. Refactor
git-receive-pack(1) to use the ODB transaction interfaces to manage the
object staging area in a more agnostic manner accordingly.

Note that the ODB transaction is now responsible for managing the
primary and alternate ODBs for the repository. One small change as a
result is that the temporary directory is now applied as the primary ODB
in the main process instead of an alternate. This does not change
anything for git-receive-pack(1) though because it only needs access to
the newly written objects and doesn't care how exactly it is set up.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/receive-pack.c | 68 ++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 33 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 50bc05c70c..8b8c20dc1a 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -37,7 +37,6 @@
 #include "sigchain.h"
 #include "string-list.h"
 #include "strvec.h"
-#include "tmp-objdir.h"
 #include "trace.h"
 #include "trace2.h"
 #include "version.h"
@@ -112,8 +111,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
+			    struct odb_transaction *transaction,
 			    const struct string_list *push_options)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (transaction)
+		odb_transaction_env(transaction, &opt.env);
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data)
 }
 
 static void set_connectivity_errors(struct command *commands,
-				    struct shallow_info *si)
+				    struct shallow_info *si,
+				    struct odb_transaction *transaction)
 {
 	struct command *cmd;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		struct command *singleton = cmd;
 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
+		struct strvec env = STRVEC_INIT;
 
 		if (shallow_update && si->shallow_ref[cmd->index])
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
+
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
 
 		cmd->error_string = "missing necessary objects";
+
+		strvec_clear(&env);
 	}
 }
 
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands,
 	}
 
 	if (!skip_connectivity_check) {
+		struct strvec env = STRVEC_INIT;
+
 		if (use_sideband) {
 			memset(&muxer, 0, sizeof(muxer));
 			muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
-			set_connectivity_errors(commands, si);
+			set_connectivity_errors(commands, si, transaction);
 
 		if (use_sideband)
 			finish_async(&muxer);
+
+		strvec_clear(&env);
 	}
 
 	reject_updates_to_hidden(commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands,
 		}
 	}
 
-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
+	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "pre-receive hook declined";
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	odb_transaction_env(transaction, &child.env);
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
-- 
2.55.0.122.gf85a7e6620

Re: [PATCH v3 06/11] odb/transaction: propagate begin errors

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> When `odb_transaction_begin()` is invoked, the function returns the
> transaction pointer directly. There is no way for the backend to
> signal that it failed to set up its state, such as when creating the
> temporary object directory backing the transaction.
>
> In a subsequent commit, git-receive-pack(1) starts using ODB
> transactions and needs to be able to report such failures rather
> than silently ignore them. Refactor `odb_transaction_begin()` to
> return an int error code and write the resulting transaction into an
> out parameter. Also introduce `odb_transaction_begin_or_die()` as a
> convenience for callsites that do not need to handle errors
> explicitly.
>
> Note that `odb_transaction_begin()` now returns an error when the ODB
> already has an inflight transaction pending. ODB transaction call sites
> that may encounter an inflight transaction are updated to explicitly
> handle this case.
>
> Signed-off-by: Justin Tobler <jltobler@gmail.com>
> ---
> ...
> diff --git a/odb/transaction.c b/odb/transaction.c
> index b16e07aebf..a5fba7f908 100644
> --- a/odb/transaction.c
> +++ b/odb/transaction.c
> @@ -1,15 +1,20 @@
>  #include "git-compat-util.h"
> +#include "gettext.h"
>  #include "odb/source.h"
>  #include "odb/transaction.h"
>  
> -struct odb_transaction *odb_transaction_begin(struct object_database *odb)
> +int odb_transaction_begin(struct object_database *odb,
> +			  struct odb_transaction **out)
>  {
> +	int ret;
> +
>  	if (odb->transaction)
> -		return NULL;
> +		return error(_("object database transaction already pending"));
>  
> -	odb_source_begin_transaction(odb->sources, &odb->transaction);
> +	ret = odb_source_begin_transaction(odb->sources, out);
> +	odb->transaction = *out;

Can odb_source_begin_transaction() ever fail?  If so, and when it
fails, would *out be left untouched?  I am wondering if we want

	if (!(ret = odb_source_begin_transaction(odb->sources, out)))
        	odb->transaction = *out;

or something like that.

Re: [PATCH v3 08/11] odb/transaction: add transaction env interface

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> +static int odb_transaction_files_env(struct odb_transaction *base,
> +				     struct strvec *env)
> +{
> +	struct odb_transaction_files *transaction =
> +		container_of(base, struct odb_transaction_files, base);
> +
> +	odb_transaction_files_prepare(&transaction->base);

Can this fail?  The caller of us would not notice that something
went wrong, and ...

> +	strvec_pushv(env, tmp_objdir_env(transaction->objdir));

... happily ends up using transaction->objdir that may not be
appropriate for it to use if it fails, no?

> +	return 0;
> +}

Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> @@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
>  static void execute_commands(struct command *commands,
>  			     const char *unpacker_error,
>  			     struct shallow_info *si,
> +			     struct odb_transaction *transaction,
>  			     const struct string_list *push_options)
>  {
>  	struct check_connected_options opt = CHECK_CONNECTED_INIT;
> ...

Hidden in the context early in this function is an error return.
When unpacker_error string is non NULL, we mark all the commands in
the linked commands list as failed, and return early from this
function.

> @@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
>  	 * Now we'll start writing out refs, which means the objects need
>  	 * to be in their final positions so that other processes can see them.
>  	 */
> -	if (tmp_objdir_migrate(tmp_objdir) < 0) {
> +	if (odb_transaction_commit(transaction)) {
>  		for (cmd = commands; cmd; cmd = cmd->next) {
>  			if (!cmd->error_string)
>  				cmd->error_string = "unable to migrate objects to permanent storage";
>  		}
>  		return;
>  	}
> -	tmp_objdir = NULL;
>  
>  	check_aliased_updates(commands);

In the "happy case", execute_commands() would commit the transaction
before going on to do the execute_commands_{atomic,nonatomic}() that
appears later in it.

> @@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
>  		if (!si.nr_ours && !si.nr_theirs)
>  			shallow_update = 0;
>  		if (!delete_only(commands)) {
> -			unpack_status = unpack_with_sideband(&si);
> +			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))

In the "main" program, we start a transaction here, and

> +				unpack_status = "unable to start object transaction";
> +			else
> +				unpack_status = unpack_with_sideband(&si, transaction);

then call unpack_with_sideband().  It may fail.

>  			update_shallow_info(commands, &si, &ref);
>  		}
>  		use_keepalive = KEEPALIVE_ALWAYS;
> -		execute_commands(commands, unpack_status, &si,
> +		execute_commands(commands, unpack_status, &si, transaction,
>  				 &push_options);

And in such a case, execute_commands() returns without committing
the transaction.  Is there a need to add and make an
odb_transaction_abort() call or something in such a case?
Everything should be cleaned up upon process exit, and on file based
backends, we probably let the tempfile/lockfile API do their thing
to clean up, but are there other things we may want to clean up?

>  		delete_tempfile(&pack_lockfile);
>  		sigchain_push(SIGPIPE, SIG_IGN);
> @@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
>  		else if (report_status)
>  			report(commands, unpack_status);
>  		sigchain_pop(SIGPIPE);
> -		run_receive_hook(commands, "post-receive", 1,
> +		run_receive_hook(commands, "post-receive", 1, NULL,
>  				 &push_options);
>  		run_update_post_hook(commands);
>  		free_commands(commands);

Re: [PATCH v3 03/11] object-file: embed transaction flush logic in commit function

[Patrick Steinhardt]

On Wed, Jul 08, 2026 at 06:59:17PM -0500, Justin Tobler wrote:
> When a "files" transaction is committed,
> `flush_loose_object_transaction()` is invoked to handle performing a
> hardware flush along with migrating the temporary object directory into
> the primary and configuring the repository ODB source accordingly. The
> function name here is a bit misleading because the helper is doing a bit
> more than just "flushing" the transaction contents. Also, in a
> subsequent commit, the transaction temporary directory is used to stage
> packfiles and not just loose objects anymore.
> 
> Lift the helper function logic directly into
> `odb_transaction_files_commit()` to more accurately signal to readers
> the operation being performed.

This line break makes my eyes bleed.

Patrick

Re: [PATCH v2 06/11] odb/transaction: propagate begin errors

[Patrick Steinhardt]

On Wed, Jul 08, 2026 at 11:56:43AM -0500, Justin Tobler wrote:
> On 26/07/08 08:41AM, Patrick Steinhardt wrote:
> > On Tue, Jul 07, 2026 at 11:14:07PM -0500, Justin Tobler wrote:
> > > @@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
> > >  
> > >  	trace_performance_enter();
> > >  	trace2_region_enter("cache_tree", "update", istate->repo);
> > > -	transaction = odb_transaction_begin(the_repository->objects);
> > > +	if (!inflight)
> > > +		odb_transaction_begin_or_die(the_repository->objects, &transaction);
> > >  	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
> > >  		       "", 0, &skip, flags);
> > > -	odb_transaction_commit(transaction);
> > > +	if (!inflight)
> > > +		odb_transaction_commit(transaction);
> > >  	trace2_region_leave("cache_tree", "update", istate->repo);
> > >  	trace_performance_leave("cache_tree_update");
> > >  	if (i < 0)
> > 
> > Callsites like this really make me wonder why we even care to create
> > a transaction in the first place if we basically just commit it
> > immediately anyway. And while it's a bit sad that we have so many sites
> > where we don't really know whether we even have a transaction, I think
> > it's a good change that we have now annotated them clearly. A subsequent
> > patch series may then eventually refactor those sites so that we stop
> > depending on `odb->transaction` and inject the transaction via a
> > parameter.
> 
> Call sites like the one mentioned above are using ODB transactions as an
> optimization to batch the full fsyncs in bulk. In cases where the is not
> already a transaction, they start one to take advantage of it.

I know. But in the case where we don't want to batch we create the
transaction anyway as far as I can see, and then we commit it
immediately. So arguably, we could've just `odb_write_object()` and call
it a day.

> I fully agree though that an ODB transaction should ideally be started
> at a higher layer and wired down to these call sites. I have a couple of
> patches in my tree that start to tackle this which I plan to send in
> another series. :)

Yeah, let's not worry about that too much for now then. One step at a
time :)

> > > @@ -36,11 +38,21 @@ struct odb_transaction {
> > >  };
> > >  
> > >  /*
> > > - * Starts an ODB transaction. Subsequent objects are written to the transaction
> > > - * and not committed until odb_transaction_commit() is invoked on the
> > > - * transaction. If the ODB already has a pending transaction, NULL is returned.
> > > + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
> > > + * written to the transaction and not committed until odb_transaction_commit()
> > > + * is invoked on the transaction. Returns 0 on success and a negative value on
> > > + * error. Note that it is considered an error to start a new transaction if the
> > > + * ODB already has an inflight transaction pending.
> > >   */
> > > -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
> > > +int odb_transaction_begin(struct object_database *odb,
> > > +			  struct odb_transaction **out);
> > > +
> > > +static inline void odb_transaction_begin_or_die(struct object_database *odb,
> > > +						struct odb_transaction **out)
> > > +{
> > > +	if (odb_transaction_begin(odb, out))
> > > +		die(_("failed to start ODB transaction"));
> > > +}
> > 
> > We could make it a bit simpler to use this function by continuing to
> > return the transaction directly. But on the other hand this results in a
> > more consistent interface.
> 
> Ya, I was a bit back and forth about this myself. I ultimately landed on
> keeping a more consistent interface though. Happy to change if others
> feel differently though.

As said, I can see both arguments. And ultimately, I don't care too
much, so it's fine if this is just kept as-is.

Patrick

Re: [PATCH v3 00/11] receive-pack: use ODB transactions to stage object writes

[Patrick Steinhardt]

On Wed, Jul 08, 2026 at 06:59:14PM -0500, Justin Tobler wrote:
> Changes since V2:
>   - Clarified commit log reasoning for embedding
>     `flush_loose_object_transaction()` logic in commit function.
>   - Started printed some error messages on transaction errors.
>   - Removed include statement.
>   - Fixed transaction leak on `odb_transaction_commit()` error.

Thanks, the changes all look good to me and I don't have anything else
to add.

Patrick

Re: [PATCH v3 06/11] odb/transaction: propagate begin errors

[Justin Tobler]

On 26/07/08 08:32PM, Junio C Hamano wrote:
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > When `odb_transaction_begin()` is invoked, the function returns the
> > transaction pointer directly. There is no way for the backend to
> > signal that it failed to set up its state, such as when creating the
> > temporary object directory backing the transaction.
> >
> > In a subsequent commit, git-receive-pack(1) starts using ODB
> > transactions and needs to be able to report such failures rather
> > than silently ignore them. Refactor `odb_transaction_begin()` to
> > return an int error code and write the resulting transaction into an
> > out parameter. Also introduce `odb_transaction_begin_or_die()` as a
> > convenience for callsites that do not need to handle errors
> > explicitly.
> >
> > Note that `odb_transaction_begin()` now returns an error when the ODB
> > already has an inflight transaction pending. ODB transaction call sites
> > that may encounter an inflight transaction are updated to explicitly
> > handle this case.
> >
> > Signed-off-by: Justin Tobler <jltobler@gmail.com>
> > ---
> > ...
> > diff --git a/odb/transaction.c b/odb/transaction.c
> > index b16e07aebf..a5fba7f908 100644
> > --- a/odb/transaction.c
> > +++ b/odb/transaction.c
> > @@ -1,15 +1,20 @@
> >  #include "git-compat-util.h"
> > +#include "gettext.h"
> >  #include "odb/source.h"
> >  #include "odb/transaction.h"
> >  
> > -struct odb_transaction *odb_transaction_begin(struct object_database *odb)
> > +int odb_transaction_begin(struct object_database *odb,
> > +			  struct odb_transaction **out)
> >  {
> > +	int ret;
> > +
> >  	if (odb->transaction)
> > -		return NULL;
> > +		return error(_("object database transaction already pending"));
> >  
> > -	odb_source_begin_transaction(odb->sources, &odb->transaction);
> > +	ret = odb_source_begin_transaction(odb->sources, out);
> > +	odb->transaction = *out;
> 
> Can odb_source_begin_transaction() ever fail?  If so, and when it
> fails, would *out be left untouched?  

In this patch there it not yet a way for it to fail, but it can return
an error later in the series. When an error is encountered though, *out
_is_ left untouched.

> I am wondering if we want
> 
> 	if (!(ret = odb_source_begin_transaction(odb->sources, out)))
>         	odb->transaction = *out;
> 
> or something like that.

I think it is a good idea to for `odb_transaction_begin()` to ensure the
repository transaction is only set on success though. Will update in the
next version. Thanks

-Justin

Re: [PATCH v3 08/11] odb/transaction: add transaction env interface

[Justin Tobler]

On 26/07/08 08:36PM, Junio C Hamano wrote:
> Justin Tobler <jltobler@gmail.com> writes:
> 
> > +static int odb_transaction_files_env(struct odb_transaction *base,
> > +				     struct strvec *env)
> > +{
> > +	struct odb_transaction_files *transaction =
> > +		container_of(base, struct odb_transaction_files, base);
> > +
> > +	odb_transaction_files_prepare(&transaction->base);
> 
> Can this fail?  The caller of us would not notice that something
> went wrong, and ...
> 
> > +	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
> 
> ... happily ends up using transaction->objdir that may not be
> appropriate for it to use if it fails, no?

Ya, `odb_transaction_files_prepare()` can fail here. In practice,
failure results in no temporary directory being created which
`tmp_objdir_env()` does handle gracefully, but we should ideally still
be reported the failure back to callers. Will update in the next
version.

-Justin

Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

[Justin Tobler]

On 26/07/08 08:49PM, Junio C Hamano wrote:
> Justin Tobler <jltobler@gmail.com> writes:
> >  			update_shallow_info(commands, &si, &ref);
> >  		}
> >  		use_keepalive = KEEPALIVE_ALWAYS;
> > -		execute_commands(commands, unpack_status, &si,
> > +		execute_commands(commands, unpack_status, &si, transaction,
> >  				 &push_options);
> 
> And in such a case, execute_commands() returns without committing
> the transaction.  Is there a need to add and make an
> odb_transaction_abort() call or something in such a case?
> Everything should be cleaned up upon process exit, and on file based
> backends, we probably let the tempfile/lockfile API do their thing
> to clean up, but are there other things we may want to clean up?

As you mentioned, if we exit before committing the ODB transaction, the
temporary directory will get cleaned up when the process exits. I don't
think there is anything else we need to cleanup that wouldn't be handled
at exit though. Regardless, I do plan to add `odb_transaction_abort()`
in a followup series and I think it would be nice to have an explicit
"abort" here when we know that we are not going to commit anyways. I
would like to defer this to my next series though.

-Justin

[PATCH v4 00/11] receive-pack: use ODB transactions to stage object writes

[Justin Tobler]

Greetings,

This patch series replaces direct usage of the `tmp_objdir` interfaces
in git-receive-pack(1) to instead use the `odb_transaction` interfaces
to create/manage a staging area to write objects to. The purpose of this
change is to get git-receive-pack(1) one step closer to being ODB
backend agnostic. For now, the object writes themselves are still
"files" backend specific due to being handled by the git-index-pack(1)
and git-unpack-objects(1) child processes. This will be tackled in a
separate series though.

Changes since V3:
  - Removed ugly line break in commit message to prevent eye strain.
  - `odb_transaction_begin()` now only sets the repository transaction
    on success.
  - `odb_transaction_env()` now bubbles up error when failing to create
    the temporary directory.

Changes since V2:
  - Clarified commit log reasoning for embedding
    `flush_loose_object_transaction()` logic in commit function.
  - Started printed some error messages on transaction errors.
  - Removed include statement.
  - Fixed transaction leak on `odb_transaction_commit()` error.

Changes since V1:
  - Adapted other "file" ODB transaction helpers to be more consistent
    with current naming scheme.
  - Removed redundant NULL transaction handling from
    `odb_transaction_files_begin()`.
  - `odb_transaction_begin()` now returns an error if there is already
    an inflight transaction pending instead of setting the `out` pointer
    to NULL.
  - Updated `odb_transaction_env()` to return an error code and append
    environment variables to a strvec provided as an argument.
  - Removed redundant setting of tmpdir environment variables for child
    processes after tmpdir has been migrated.
  - Split changes adding ODB transaction flags into a separate commit.
  - Consistently wire the ODB transaction throughout git-receive-pack
    code instead of reading it from `the_repository`.
  - Updated user facing error message.
  - Updated some comments to better document functions/flags.
  - Clarified some commit messages.
  - Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
  object-file: rename files transaction prepare function
  object-file: rename files transaction fsync function
  object-file: embed transaction flush logic in commit function
  object-file: drop check for inflight transactions
  object-file: propagate files transaction errors
  odb/transaction: propagate begin errors
  odb/transaction: propagate commit errors
  odb/transaction: add transaction env interface
  odb/transaction: introduce ODB transaction flags
  builtin/receive-pack: drop redundant tmpdir env
  builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 161 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  33 ++++++--
 odb/transaction.h        |  59 +++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 244 insertions(+), 130 deletions(-)

Range-diff against v3:
 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 2:  5703a9e93b =  2:  5703a9e93b object-file: rename files transaction fsync function
 3:  76204847f2 !  3:  70267741b0 object-file: embed transaction flush logic in commit function
    @@ Commit message
         subsequent commit, the transaction temporary directory is used to stage
         packfiles and not just loose objects anymore.
     
    -    Lift the helper function logic directly into
    -    `odb_transaction_files_commit()` to more accurately signal to readers
    -    the operation being performed.
    +    Lift the helper function logic into `odb_transaction_files_commit()` to
    +    more accurately signal to readers the operation being performed.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
 4:  c97eb7763f =  4:  34cd3822c5 object-file: drop check for inflight transactions
 5:  1f3a1f7714 =  5:  240aa3475f object-file: propagate files transaction errors
 6:  09d13272d5 !  6:  0d91310fac odb/transaction: propagate begin errors
    @@ odb/transaction.c
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
     +	ret = odb_source_begin_transaction(odb->sources, out);
    -+	odb->transaction = *out;
    ++	if (!ret)
    ++		odb->transaction = *out;
      
     -	return odb->transaction;
     +	return ret;
 7:  12833d6773 =  7:  5e4680ed75 odb/transaction: propagate commit errors
 8:  f2586f2f34 !  8:  babcf6b156 odb/transaction: add transaction env interface
    @@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *b
     +{
     +	struct odb_transaction_files *transaction =
     +		container_of(base, struct odb_transaction_files, base);
    ++	int ret;
     +
    -+	odb_transaction_files_prepare(&transaction->base);
    -+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
    ++	ret = odb_transaction_files_prepare(&transaction->base);
    ++	if (!ret)
    ++		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
     +
    -+	return 0;
    ++	return ret;
     +}
     +
      int odb_transaction_files_begin(struct odb_source *source,
 9:  9d082b5e47 !  9:  96f2a21eec odb/transaction: introduce ODB transaction flags
    @@ odb/transaction.c
      
     -	ret = odb_source_begin_transaction(odb->sources, out);
     +	ret = odb_source_begin_transaction(odb->sources, out, flags);
    - 	odb->transaction = *out;
    + 	if (!ret)
    + 		odb->transaction = *out;
      
    - 	return ret;
     
      ## odb/transaction.h ##
     @@
10:  e11d8a6676 = 10:  56718f1190 builtin/receive-pack: drop redundant tmpdir env
11:  fee57c2817 = 11:  5197a19fbf builtin/receive-pack: stage incoming objects via ODB transactions

base-commit: ab776a62a78576513ee121424adb19597fbb7613
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 01/11] object-file: rename files transaction prepare function

[Justin Tobler]

The "files" ODB transaction backend lazily creates a temporary object
directory when the first loose object is written to the transaction via
`prepare_loose_object_transaction()`. In a subsequent commit, the
temporary directory is used to also write packfiles to.

Rename the function to `odb_transaction_files_prepare()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/object-file.c b/object-file.c
index e3d92bbda2..a3eb8d71dd 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 02/11] object-file: rename files transaction fsync function

[Justin Tobler]

When writing an object to a "files" ODB transaction, a full hardware
flush is not initially performed during the fsync in
`fsync_loose_object_transaction()` and instead delayed until the
transaction is later committed.

To be more consistent with other "files" ODB transaction helpers, rename
the function to `odb_transaction_files_fsync()` accordingly. The
conditional in the helper is also slightly restructured to improve
clarity to readers.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)

diff --git a/object-file.c b/object-file.c
index a3eb8d71dd..d68824bb44 100644
--- a/object-file.c
+++ b/object-file.c
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}
+
 	/*
 	 * If we have an active ODB transaction, we issue a call that
 	 * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
 	 * that all objects are durable before any renames occur. The code in
-	 * fsync_loose_object_transaction has already issued a writeout
+	 * odb_transaction_files_fsync has already issued a writeout
 	 * request, but it has not flushed any writeback cache in the storage
 	 * hardware or any filesystem logs. This fsync call acts as a barrier
 	 * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose,
 		goto out;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
+		odb_transaction_files_fsync(loose->base.odb->transaction, fd, filename);
 	else if (fsync_object_files > 0)
 		fsync_or_die(fd, filename);
 	else
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 03/11] object-file: embed transaction flush logic in commit function

[Justin Tobler]

When a "files" transaction is committed,
`flush_loose_object_transaction()` is invoked to handle performing a
hardware flush along with migrating the temporary object directory into
the primary and configuring the repository ODB source accordingly. The
function name here is a bit misleading because the helper is doing a bit
more than just "flushing" the transaction contents. Also, in a
subsequent commit, the transaction temporary directory is used to stage
packfiles and not just loose objects anymore.

Lift the helper function logic into `odb_transaction_files_commit()` to
more accurately signal to readers the operation being performed.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)

diff --git a/object-file.c b/object-file.c
index d68824bb44..33bd6c6810 100644
--- a/object-file.c
+++ b/object-file.c
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base,
 	}
 }
 
-/*
- * Cleanup after batch-mode fsync_object_files.
- */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
-{
-	struct strbuf temp_path = STRBUF_INIT;
-	struct tempfile *temp;
-
-	if (!transaction->objdir)
-		return;
-
-	/*
-	 * Issue a full hardware flush against a temporary file to ensure
-	 * that all objects are durable before any renames occur. The code in
-	 * odb_transaction_files_fsync has already issued a writeout
-	 * request, but it has not flushed any writeback cache in the storage
-	 * hardware or any filesystem logs. This fsync call acts as a barrier
-	 * to ensure that the data in each new object file is durable before
-	 * the final name is visible.
-	 */
-	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-		    repo_get_object_directory(transaction->base.source->odb->repo));
-	temp = xmks_tempfile(temp_path.buf);
-	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-	delete_tempfile(&temp);
-	strbuf_release(&temp_path);
-
-	/*
-	 * Make the object files visible in the primary ODB after their data is
-	 * fully durable.
-	 */
-	tmp_objdir_migrate(transaction->objdir);
-	transaction->objdir = NULL;
-}
-
 /* Finalize a file on disk, and close it. */
 static void close_loose_object(struct odb_source_loose *loose,
 			       int fd, const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (transaction->objdir) {
+		struct strbuf temp_path = STRBUF_INIT;
+		struct tempfile *temp;
+
+		/*
+		 * Issue a full hardware flush against a temporary file to ensure
+		 * that all objects are durable before any renames occur. The code in
+		 * odb_transaction_files_fsync has already issued a writeout
+		 * request, but it has not flushed any writeback cache in the storage
+		 * hardware or any filesystem logs. This fsync call acts as a barrier
+		 * to ensure that the data in each new object file is durable before
+		 * the final name is visible.
+		 */
+		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+			    repo_get_object_directory(transaction->base.source->odb->repo));
+		temp = xmks_tempfile(temp_path.buf);
+		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+		delete_tempfile(&temp);
+		strbuf_release(&temp_path);
+
+		/*
+		 * Make the object files visible in the primary ODB after their data is
+		 * fully durable.
+		 */
+		tmp_objdir_migrate(transaction->objdir);
+		transaction->objdir = NULL;
+	}
+
 	flush_packfile_transaction(transaction);
 }
 
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 04/11] object-file: drop check for inflight transactions

[Justin Tobler]

ODB transactions are started via `odb_transaction_begin()` and contain
validation to avoid starting multiple transactions at the same time. The
"files" backend also has the same logic, but is redundant due to the
generic layer already handling it. Drop this validation from the "files"
backend accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c | 4 ----
 object-file.h | 3 +--
 2 files changed, 1 insertion(+), 6 deletions(-)

diff --git a/object-file.c b/object-file.c
index 33bd6c6810..e51389833a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 {
 	struct odb_transaction_files *transaction;
-	struct object_database *odb = source->odb;
-
-	if (odb->transaction)
-		return NULL;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
diff --git a/object-file.h b/object-file.h
index 528c4e6e69..ea43d818f0 100644
--- a/object-file.h
+++ b/object-file.h
@@ -194,8 +194,7 @@ struct odb_transaction;
 /*
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * to make new objects visible.
  */
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
 
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 05/11] object-file: propagate files transaction errors

[Justin Tobler]

The "files" transaction backend may encounter errors related to managing
the temporary directory used to stage objects, but silently ignores
these errors. Instead return errors encountered in the
`odb_transaction_files_{prepare,begin,commit}()` interfaces to allow
callers to handle them as needed.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c      | 26 ++++++++++++++++++--------
 object-file.h      |  3 ++-
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  7 +++++--
 4 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/object-file.c b/object-file.c
index e51389833a..3651605ea2 100644
--- a/object-file.c
+++ b/object-file.c
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return error(_("unable to create temporary object directory"));
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return error(_("unable to migrate temporary objects"));
+
 		transaction->objdir = NULL;
 	}
 
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
diff --git a/object-file.h b/object-file.h
index ea43d818f0..1a023226ac 100644
--- a/object-file.h
+++ b/object-file.h
@@ -196,6 +196,7 @@ struct odb_transaction;
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 5bdd042922..2545bd81d4 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/transaction.h b/odb/transaction.h
index 854fda06f5..d52f0533ce 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -16,8 +16,11 @@ struct odb_transaction {
 	/* The ODB source the transaction is opened against. */
 	struct odb_source *source;
 
-	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	/*
+	 * The ODB source specific callback invoked to commit a transaction.
+	 * Returns 0 on success, a negative error code otherwise.
+	 */
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 06/11] odb/transaction: propagate begin errors

[Justin Tobler]

When `odb_transaction_begin()` is invoked, the function returns the
transaction pointer directly. There is no way for the backend to
signal that it failed to set up its state, such as when creating the
temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB
transactions and needs to be able to report such failures rather
than silently ignore them. Refactor `odb_transaction_begin()` to
return an int error code and write the resulting transaction into an
out parameter. Also introduce `odb_transaction_begin_or_die()` as a
convenience for callsites that do not need to handle errors
explicitly.

Note that `odb_transaction_begin()` now returns an error when the ODB
already has an inflight transaction pending. ODB transaction call sites
that may encounter an inflight transaction are updated to explicitly
handle this case.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  7 +++++--
 object-file.c            | 10 +++++++---
 odb/transaction.c        | 14 ++++++++++----
 odb/transaction.h        | 19 +++++++++++++++----
 read-cache.c             |  7 +++++--
 8 files changed, 45 insertions(+), 18 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index c859f66519..3d5d9cfdb9 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index f3849bb654..d0136cdd99 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 3d6646c318..17f3ea284c 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 184f7e2635..8eec1d4d52 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
diff --git a/object-file.c b/object-file.c
index 3651605ea2..358684beae 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction = odb->transaction;
+			int inflight = !!transaction;
 
-			ret = odb_transaction_write_object_stream(odb->transaction,
+			if (!inflight)
+				odb_transaction_begin_or_die(odb, &transaction);
+			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
 								  oid);
-			odb_transaction_commit(transaction);
+			if (!inflight)
+				odb_transaction_commit(transaction);
 		} else {
 			ret = hash_blob_stream(&stream,
 					       the_repository->hash_algo, oid,
diff --git a/odb/transaction.c b/odb/transaction.c
index b16e07aebf..b6da4a3942 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -1,15 +1,21 @@
 #include "git-compat-util.h"
+#include "gettext.h"
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return error(_("object database transaction already pending"));
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	if (!ret)
+		odb->transaction = *out;
 
-	return odb->transaction;
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
diff --git a/odb/transaction.h b/odb/transaction.h
index d52f0533ce..f5c43187c9 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -1,6 +1,7 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -36,11 +37,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
diff --git a/read-cache.c b/read-cache.c
index 21ca58beea..d511d25834 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 		       const struct pathspec *pathspec, char *ps_matched,
 		       int include_sparse, int flags, int ignored_too )
 {
+	int inflight = !!repo->objects->transaction;
 	struct odb_transaction *transaction;
 	struct update_callback_data data;
 	struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 
 	release_revisions(&rev);
 	return !!data.add_errors;
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 07/11] odb/transaction: propagate commit errors

[Justin Tobler]

When `odb_transaction_commit()` is invoked, the return value of the
backend commit callback is silently discarded. A backend has no way
to signal that committing failed, such as when the "files" backend
cannot migrate its temporary object directory into the permanent
ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction
to stage objects and consequently cares about such failures so it can
handle the error appropriately. Change the commit callback signature to
return an int error code and have `odb_transaction_commit()` forward it
accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 odb/transaction.c | 10 +++++++---
 odb/transaction.h |  7 ++++---
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/odb/transaction.c b/odb/transaction.c
index b6da4a3942..249ef4d9b7 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -18,19 +18,23 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return ret;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
diff --git a/odb/transaction.h b/odb/transaction.h
index f5c43187c9..3b0a5a78e5 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -54,10 +54,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 }
 
 /*
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
+ * Commits an ODB transaction making the written objects visible. Returns 0 on
+ * success, a negative error code otherwise. Note that, if the specified
+ * transaction is NULL, the function is a no-op and no error is returned.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 08/11] odb/transaction: add transaction env interface

[Justin Tobler]

The ODB transaction backend is responsible for creating/managing its own
staging area for writing objects. Other child processes spawned by Git
may need access to uncommitted objects or write new objects in the
staging area though.

Introduce `odb_transaction_env()` which is expected to provide the set
of environment variables needed by a child process to access the
transaction's staging area.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 object-file.c     | 16 ++++++++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 17 +++++++++++++++++
 3 files changed, 41 insertions(+)

diff --git a/object-file.c b/object-file.c
index 358684beae..39b92e275c 100644
--- a/object-file.c
+++ b/object-file.c
@@ -27,6 +27,7 @@
 #include "path.h"
 #include "read-cache-ll.h"
 #include "setup.h"
+#include "strvec.h"
 #include "tempfile.h"
 #include "tmp-objdir.h"
 
@@ -1687,6 +1688,20 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+	int ret;
+
+	ret = odb_transaction_files_prepare(&transaction->base);
+	if (!ret)
+		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
+	return ret;
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1696,6 +1711,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
diff --git a/odb/transaction.c b/odb/transaction.c
index 249ef4d9b7..92ec8786a1 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -43,3 +43,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+	if (!transaction)
+		return 0;
+
+	return transaction->env(transaction, env);
+}
diff --git a/odb/transaction.h b/odb/transaction.h
index 3b0a5a78e5..5e51ce5ca4 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -34,6 +34,14 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to populate the provided strvec with the
+	 * environment variables that a child process should inherit so that its
+	 * object writes participate in the transaction. Returns 0 on success, a
+	 * negative error code otherwise.
+	 */
+	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
 /*
@@ -69,4 +77,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for using via child_process.env. Returns 0 on success,
+ * a negative error code otherwise. Note that, if the specified transaction is
+ * NULL, the function is a no-op and no error is returned.
+ */
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
 #endif
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 09/11] odb/transaction: introduce ODB transaction flags

[Justin Tobler]

The temporary directory used by git-receive-pack(1) to write objects is
managed slightly differently than how it is done via ODB transactions:

  - The temporary directory is eagerly created upfront, instead of
    waiting for the first object write.

  - The prefix name of the temporary directory is "incoming" instead of
    "bulk-fsync".

In a subsequent commit, git-receive-pack(1) will use ODB transactions
instead of `tmp_objdir` directly. To provide a means to configure the
same transaction behavior, introduce `enum odb_transaction_flags` and
the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
transactions using the "files" backend to be set up for
git-receive-pack(1). Transaction call sites are updated accordingly to
provide the required flag parameter.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 29 ++++++++++++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 ++++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 15 +++++++++++----
 read-cache.c             |  2 +-
 13 files changed, 61 insertions(+), 22 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index 3d5d9cfdb9..60ffbede2b 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index d0136cdd99..c3d0fc7507 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
diff --git a/builtin/update-index.c b/builtin/update-index.c
index 17f3ea284c..bf6ea60ef4 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
diff --git a/cache-tree.c b/cache-tree.c
index 8eec1d4d52..99c6a0a7d0 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
 	if (!inflight)
-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
+		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	if (!inflight)
diff --git a/object-file.c b/object-file.c
index 39b92e275c..0640a22009 100644
--- a/object-file.c
+++ b/object-file.c
@@ -498,6 +498,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return error(_("unable to create temporary object directory"));
 
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			int inflight = !!transaction;
 
 			if (!inflight)
-				odb_transaction_begin_or_die(odb, &transaction);
+				odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1703,7 +1704,8 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1712,6 +1714,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
diff --git a/object-file.h b/object-file.h
index 1a023226ac..bdd2d67a2e 100644
--- a/object-file.h
+++ b/object-file.h
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;
  * to make new objects visible.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
diff --git a/odb/source-files.c b/odb/source-files.c
index 2545bd81d4..534f48aad9 100644
--- a/odb/source-files.c
+++ b/odb/source-files.c
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
diff --git a/odb/source-inmemory.c b/odb/source-inmemory.c
index e004566d76..9644d9d474 100644
--- a/odb/source-inmemory.c
+++ b/odb/source-inmemory.c
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
diff --git a/odb/source-loose.c b/odb/source-loose.c
index 66e6bb8d3f..57c91986b4 100644
--- a/odb/source-loose.c
+++ b/odb/source-loose.c
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
diff --git a/odb/source.h b/odb/source.h
index 2192a101b8..3790d03ff2 100644
--- a/odb/source.h
+++ b/odb/source.h
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
diff --git a/odb/transaction.c b/odb/transaction.c
index 92ec8786a1..dab7da6a9a 100644
--- a/odb/transaction.c
+++ b/odb/transaction.c
@@ -4,14 +4,15 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
 	if (odb->transaction)
 		return error(_("object database transaction already pending"));
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	if (!ret)
 		odb->transaction = *out;
 
diff --git a/odb/transaction.h b/odb/transaction.h
index 5e51ce5ca4..4cb2eafcbf 100644
--- a/odb/transaction.h
+++ b/odb/transaction.h
@@ -3,7 +3,6 @@
 
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -52,12 +57,14 @@ struct odb_transaction {
  * ODB already has an inflight transaction pending.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
diff --git a/read-cache.c b/read-cache.c
index d511d25834..50e2320c8d 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * may not have their own transaction active.
 	 */
 	if (!inflight)
-		odb_transaction_begin_or_die(repo->objects, &transaction);
+		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	if (!inflight)
 		odb_transaction_commit(transaction);
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 10/11] builtin/receive-pack: drop redundant tmpdir env

[Justin Tobler]

When performing the connectivity checks for a shallow ref in
`update_shallow_ref()`, the child process environment variables are
populated via `tmp_objdir_env()`. This is unnecessary though as
`update_shallow_ref()` is only reached after `tmp_objdir_migrate()` has
been performed which means there is no longer a temporary directory that
needs to be shared with child processes.

Drop the call to `tmp_objdir_env()` accordingly.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/receive-pack.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 19eb6a1b61..50bc05c70c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
-- 
2.55.0.122.gf85a7e6620

[PATCH v4 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

[Justin Tobler]

Objects received by git-receive-pack(1) are quarantined in a temporary
"incoming" directory and migrated into the object database prior to the
reference updates. The quarantine is currently managed through
`tmp_objdir` directly. In a pluggable ODB future, how exactly an object
gets written to a transaction may vary for a given ODB source. Refactor
git-receive-pack(1) to use the ODB transaction interfaces to manage the
object staging area in a more agnostic manner accordingly.

Note that the ODB transaction is now responsible for managing the
primary and alternate ODBs for the repository. One small change as a
result is that the temporary directory is now applied as the primary ODB
in the main process instead of an alternate. This does not change
anything for git-receive-pack(1) though because it only needs access to
the newly written objects and doesn't care how exactly it is set up.

Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
 builtin/receive-pack.c | 68 ++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 33 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 50bc05c70c..8b8c20dc1a 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -37,7 +37,6 @@
 #include "sigchain.h"
 #include "string-list.h"
 #include "strvec.h"
-#include "tmp-objdir.h"
 #include "trace.h"
 #include "trace2.h"
 #include "version.h"
@@ -112,8 +111,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
+			    struct odb_transaction *transaction,
 			    const struct string_list *push_options)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (transaction)
+		odb_transaction_env(transaction, &opt.env);
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data)
 }
 
 static void set_connectivity_errors(struct command *commands,
-				    struct shallow_info *si)
+				    struct shallow_info *si,
+				    struct odb_transaction *transaction)
 {
 	struct command *cmd;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		struct command *singleton = cmd;
 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
+		struct strvec env = STRVEC_INIT;
 
 		if (shallow_update && si->shallow_ref[cmd->index])
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
+
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
 
 		cmd->error_string = "missing necessary objects";
+
+		strvec_clear(&env);
 	}
 }
 
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands,
 	}
 
 	if (!skip_connectivity_check) {
+		struct strvec env = STRVEC_INIT;
+
 		if (use_sideband) {
 			memset(&muxer, 0, sizeof(muxer));
 			muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
-			set_connectivity_errors(commands, si);
+			set_connectivity_errors(commands, si, transaction);
 
 		if (use_sideband)
 			finish_async(&muxer);
+
+		strvec_clear(&env);
 	}
 
 	reject_updates_to_hidden(commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands,
 		}
 	}
 
-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
+	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "pre-receive hook declined";
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	odb_transaction_env(transaction, &child.env);
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
-- 
2.55.0.122.gf85a7e6620

Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

[Junio C Hamano]

Justin Tobler <jltobler@gmail.com> writes:

> On 26/07/08 08:49PM, Junio C Hamano wrote:
>> Justin Tobler <jltobler@gmail.com> writes:
>> >  			update_shallow_info(commands, &si, &ref);
>> >  		}
>> >  		use_keepalive = KEEPALIVE_ALWAYS;
>> > -		execute_commands(commands, unpack_status, &si,
>> > +		execute_commands(commands, unpack_status, &si, transaction,
>> >  				 &push_options);
>> 
>> And in such a case, execute_commands() returns without committing
>> the transaction.  Is there a need to add and make an
>> odb_transaction_abort() call or something in such a case?
>> Everything should be cleaned up upon process exit, and on file based
>> backends, we probably let the tempfile/lockfile API do their thing
>> to clean up, but are there other things we may want to clean up?
>
> As you mentioned, if we exit before committing the ODB transaction, the
> temporary directory will get cleaned up when the process exits. I don't
> think there is anything else we need to cleanup that wouldn't be handled
> at exit though. Regardless, I do plan to add `odb_transaction_abort()`
> in a followup series and I think it would be nice to have an explicit
> "abort" here when we know that we are not going to commit anyways. I
> would like to defer this to my next series though.

Sounds good.  We cannot trigger receive-pack as a subroutine call in
a long running daemon until that happens, but that is OK for now.
One step at a time.

Re: [PATCH v4 00/11] receive-pack: use ODB transactions to stage object writes

[Patrick Steinhardt]

On Fri, Jul 10, 2026 at 11:37:11AM -0500, Justin Tobler wrote:
> Changes since V3:
>   - Removed ugly line break in commit message to prevent eye strain.
>   - `odb_transaction_begin()` now only sets the repository transaction
>     on success.
>   - `odb_transaction_env()` now bubbles up error when failing to create
>     the temporary directory.

Thanks, all the changes here look good to me and I'm happy with the
state of this patch series.

Patrick