Git development
 help / color / mirror / Atom feed
From: Justin Tobler <jltobler@gmail.com>
To: git@vger.kernel.org
Cc: ps@pks.im, karthik.188@gmail.com, Justin Tobler <jltobler@gmail.com>
Subject: [PATCH v2 0/6] object-file: eagerly create ODB transaction temporary directory
Date: Wed, 18 Nov 2026 18:20:37 -0600	[thread overview]
Message-ID: <20261119002043.2108654-1-jltobler@gmail.com> (raw)
In-Reply-To: <20261110002251.2007617-1-jltobler@gmail.com>

Greetings,

This series continues work for the "files" ODB transactions and makes it
so the underlying transaction temporary directory is created eagerly.
Previously, by default, it would wait for the first object write to
lazily create the temporary object directory, and only did so if batched
fsync was configured. This lazy setup is awkward for a couple of
reasons. Writing loose objects has to know about transaction internals
to trigger the creation of the temporary directory, and whether objects
written in a transaction become visible at once on commit depends on the
configured fsync method. Creating the temporary directory eagerly
removes both of these special cases and ensures that objects are written
to the temporary directory for all transactional object writes.

The first five patches consist of some minor preparatory refactors. The
final patch swaps to creating the temporary directory eagerly and drops
the lazy setup.

Changes since v1:
  - Moved the comment for `flush_batched_fsync()` above the function and
    documented that batch fsync only applies to loose objects.
  - Added a new patch that inlines `setup_tmp_objdir()` into
    `tmp_objdir_create()`, which now restores `errno` ina  single exit path.
  - Added a new patch that avoids the "ODB" term in user-facing error
    messages.
  - Clarified some commit messages.

This series is based on 3c60ef1e38 (The 17th batch, 2026-11-08) with
jt/tmp-objdir-loose-idx-merge merged into it.

Thanks,
-Justin

Justin Tobler (6):
  object-file: rename default transaction prefix
  object-file: only flush batched fsync writes when enabled
  tmp-objdir: inline `setup_tmp_objdir()`
  tmp-objdir: preserve errno when directory creation fails
  odb/transaction: avoid "ODB" in error messages
  object-file: create ODB transaction temporary directory eagerly

 object-file.c         | 98 ++++++++++++++++---------------------------
 object-file.h         |  1 -
 odb/source-loose.c    |  6 ---
 odb/transaction.h     |  6 +--
 t/t0004-unwritable.sh | 19 ++++-----
 tmp-objdir.c          | 30 ++++++-------
 tmp-objdir.h          |  3 +-
 7 files changed, 63 insertions(+), 100 deletions(-)

Range-diff against v1:
1:  eb3679a4ea = 1:  eb3679a4ea object-file: rename default transaction prefix
2:  392893af4e ! 2:  e2fb4b4833 object-file: only flush batched fsync writes when enabled
    @@ Commit message
         unnecessary. Only issue the flush when batched fsync is enabled
         accordingly.
     
    +    Note that batched fsync only applies to loose objects. Other files
    +    written as part of the transaction, such as packfiles, are fully synced
    +    when written or not synced at all, and thus do not rely on this flush.
    +
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## object-file.c ##
    @@ object-file.c: static int write_pack_lockfiles(struct odb_transaction_files *tra
      	return ret;
      }
      
    ++/*
    ++ * Issue a full hardware flush against a temporary file to ensure that all
    ++ * objects are durable before any renames occur. The code in
    ++ * odb_transaction_files_fsync has already issued a writeout request, but it has
    ++ * not flushed any writeback cache in the storage hardware or any filesystem
    ++ * logs. This fsync call acts as a barrier to ensure that the data in each new
    ++ * object file is durable before the final name is visible.
    ++ */
     +static void flush_batched_fsync(struct repository *repo)
     +{
     +	struct strbuf temp_path = STRBUF_INIT;
     +	struct tempfile *temp;
     +
    -+	/*
    -+	 * Issue a full hardware flush against a temporary file to ensure
    -+	 * that all objects are durable before any renames occur. The code in
    -+	 * odb_transaction_files_fsync has already issued a writeout
    -+	 * request, but it has not flushed any writeback cache in the storage
    -+	 * hardware or any filesystem logs. This fsync call acts as a barrier
    -+	 * to ensure that the data in each new object file is durable before
    -+	 * the final name is visible.
    -+	 */
     +	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
     +		    repo_get_object_directory(repo));
     +	temp = xmks_tempfile(temp_path.buf);
-:  ---------- > 3:  ae6548cd8f tmp-objdir: inline `setup_tmp_objdir()`
3:  ede4bc8aa1 ! 4:  f013767e55 tmp-objdir: preserve errno when directory creation fails
    @@ Commit message
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
      ## tmp-objdir.c ##
    -@@ tmp-objdir.c: static int setup_tmp_objdir(const char *root)
    - {
    - 	char *path;
    - 	int ret = 0;
    -+	int saved_errno;
    - 
    - 	path = xstrfmt("%s/pack", root);
    - 	ret = mkdir(path, 0777);
    -+	saved_errno = errno;
    - 	free(path);
    -+	errno = saved_errno;
    - 
    - 	return ret;
    - }
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
      	struct odb_source_files *files = odb_source_files_downcast(r->objects->source);
      	static int installed_handlers;
      	struct tmp_objdir *t;
     +	int saved_errno;
    + 	char *path;
    + 	int ret;
      
    - 	if (the_tmp_objdir)
    - 		BUG("only one tmp_objdir can be used at a time");
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
      		chdir_notify_register(tmp_objdir_reparent, t);
      
    @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
     +		saved_errno = errno;
      		/* free, not destroy, as we never touched the filesystem */
      		tmp_objdir_free(t);
    -+		errno = saved_errno;
    - 		return NULL;
    +-		return NULL;
    ++		goto error;
      	}
      
    + 	the_tmp_objdir = t;
     @@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
    - 	}
      
    - 	if (setup_tmp_objdir(t->path.buf)) {
    -+		saved_errno = errno;
    + 	path = xstrfmt("%s/pack", t->path.buf);
    + 	ret = mkdir(path, 0777);
    ++	saved_errno = errno;
    + 	free(path);
    + 	if (ret) {
      		tmp_objdir_destroy(t);
    -+		errno = saved_errno;
    - 		return NULL;
    +-		return NULL;
    ++		goto error;
      	}
      
    + 	env_append(&t->env, ALTERNATE_DB_ENVIRONMENT,
    +@@ tmp-objdir.c: struct tmp_objdir *tmp_objdir_create(struct repository *r,
    + 	t->repo->disable_ref_updates = true;
    + 
    + 	return t;
    ++
    ++error:
    ++	errno = saved_errno;
    ++	return NULL;
    + }
    + 
    + /*
     
      ## tmp-objdir.h ##
     @@ tmp-objdir.h: struct tmp_objdir;
-:  ---------- > 5:  66eaf498e9 odb/transaction: avoid "ODB" in error messages
4:  dd36ba0fb7 ! 6:  6d2266ad20 object-file: create ODB transaction temporary directory eagerly
    @@ Commit message
             git add . (rev = HEAD~) ran
               1.07 ± 0.04 times faster than git add . (rev = HEAD)
     
    -    Typically large numbers of loose objects are not expected to be written
    -    into an ODB transaction so the overhead here should be marginal.
    +    The overhead is a small fixed cost per object for migrating it into the
    +    primary object directory, which is minor compared to the cost of writing
    +    the object in the first place. It is most visible when adding many small
    +    files at once, as in the benchmark above. Otherwise, transactions
    +    typically write only a few loose objects, for example when unpacking a
    +    pushed packfile with fewer objects than the unpack limit. So the
    +    overhead is not expected to be noticeable in most cases.
     
         Signed-off-by: Justin Tobler <jltobler@gmail.com>
     
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'write-tree should n
     -	error: insufficient permission for adding an object to repository database .git/objects
     -	fatal: git-write-tree: error building trees
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.write-tree
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'commit should notic
     -	error: insufficient permission for adding an object to repository database .git/objects
     -	error: Error building trees
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.commit
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'update-index should
     -	error: file: failed to insert into database
     -	fatal: Unable to process path file
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.update-index
      '
    @@ t/t0004-unwritable.sh: test_expect_success POSIXPERM,SANITY 'add should notice u
     -	error: unable to index file '\''file'\''
     -	fatal: updating files failed
     +	error: unable to create temporary object directory in .git/objects: Permission denied
    -+	fatal: failed to start ODB transaction
    ++	fatal: failed to start object transaction
      	EOF
      	test_cmp expect out.add
      '
-- 
2.56.0.173.gf57c9a7388


  parent reply	other threads:[~2026-11-19  0:20 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-10  0:22 [PATCH 0/4] object-file: eagerly create ODB transaction temporary directory Justin Tobler
2026-11-10  0:22 ` [PATCH 1/4] object-file: rename default transaction prefix Justin Tobler
2026-11-11 13:15   ` Patrick Steinhardt
2026-11-11 14:36   ` Karthik Nayak
2026-11-10  0:22 ` [PATCH 2/4] object-file: only flush batched fsync writes when enabled Justin Tobler
2026-11-11 13:16   ` Patrick Steinhardt
2026-11-17 23:35     ` Justin Tobler
2026-11-11 14:48   ` Karthik Nayak
2026-11-17 23:42     ` Justin Tobler
2026-11-10  0:22 ` [PATCH 3/4] tmp-objdir: preserve errno when directory creation fails Justin Tobler
2026-11-11 13:16   ` Patrick Steinhardt
2026-11-11 16:44   ` Karthik Nayak
2026-11-17 23:46     ` Justin Tobler
2026-11-10  0:22 ` [PATCH 4/4] object-file: create ODB transaction temporary directory eagerly Justin Tobler
2026-11-11 13:15   ` Patrick Steinhardt
2026-11-17 23:48     ` Justin Tobler
2026-11-11 16:48   ` Karthik Nayak
2026-11-17 23:57     ` Justin Tobler
2026-11-11 14:33 ` [PATCH 0/4] object-file: eagerly create ODB transaction temporary directory Karthik Nayak
2026-11-19  0:20 ` Justin Tobler [this message]
2026-11-19  0:20   ` [PATCH v2 1/6] object-file: rename default transaction prefix Justin Tobler
2026-11-19  0:20   ` [PATCH v2 2/6] object-file: only flush batched fsync writes when enabled Justin Tobler
2026-11-19  0:20   ` [PATCH v2 3/6] tmp-objdir: inline `setup_tmp_objdir()` Justin Tobler
2026-11-19  0:20   ` [PATCH v2 4/6] tmp-objdir: preserve errno when directory creation fails Justin Tobler
2026-11-19  0:20   ` [PATCH v2 5/6] odb/transaction: avoid "ODB" in error messages Justin Tobler
2026-11-19  0:20   ` [PATCH v2 6/6] object-file: create ODB transaction temporary directory eagerly Justin Tobler

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261119002043.2108654-1-jltobler@gmail.com \
    --to=jltobler@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=karthik.188@gmail.com \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox