* [PATCH 1/3] pack-write: lift and expose index-pack report parsing
2026-10-16 2:28 [PATCH 0/3] object-file: fix ".keep" lockfile ownership in ODB transactions Justin Tobler
@ 2026-10-16 2:28 ` Justin Tobler
2026-10-16 6:58 ` Patrick Steinhardt
2026-10-16 2:29 ` [PATCH 2/3] object-file: explicitly write ".keep" lockfile Justin Tobler
2026-10-16 2:29 ` [PATCH 3/3] object-file: write ".keep" lockfiles during transaction commit Justin Tobler
2 siblings, 1 reply; 7+ messages in thread
From: Justin Tobler @ 2026-10-16 2:28 UTC (permalink / raw)
To: git; +Cc: Justin Tobler
Once git-index-pack(1) finishes writing a packfile, it reports the
hexadecimal name of that packfile on stdout prefixed with either "pack"
or "keep" depending on whether a ".keep" file was also created alongside
it. Callers that are required to manage the lifecycle of ".keep" files
parse this report via `index_pack_lockfile()`, which provides the path
to the temporary ".keep" file.
In a subsequent commit, the "files" ODB transaction begins staging
packfiles in a transaction without eagerly creating a ".keep" file and
must instead read git-index-pack(1) "pack" output to get the packfile
name. Extract the generic parsing logic for the git-index-pack(1) output
into `read_index_pack_report()` in preparation accordingly.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
pack-write.c | 69 +++++++++++++++++++++++++++++++++++++++-------------
pack.h | 10 ++++++++
2 files changed, 62 insertions(+), 17 deletions(-)
diff --git a/pack-write.c b/pack-write.c
index 85674e4b72..5f5bb894a5 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -469,33 +469,68 @@ void fixup_pack_header_footer(const struct git_hash_algo *hash_algo,
fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
}
-char *index_pack_lockfile(struct odb_source *source, int ip_out,
- int *is_well_formed)
+int read_index_pack_report(struct repository *repo, int ip_out,
+ struct strbuf *pack_name, int *is_keep)
{
+ const int len = repo->hash_algo->hexsz + 6;
char packname[GIT_MAX_HEXSZ + 6];
- const int len = source->odb->repo->hash_algo->hexsz + 6;
+ const char *name;
+ int keep;
/*
- * The first thing we expect from index-pack's output
- * is "pack\t%40s\n" or "keep\t%40s\n" (46 bytes) where
- * %40s is the newly created pack SHA1 name. In the "keep"
- * case, we need it to remove the corresponding .keep file
- * later on. If we don't get that then tough luck with it.
+ * The first thing we expect from index-pack's output is
+ * "pack\t<name>\n" or "keep\t<name>\n", where <name> is the hexadecimal
+ * name of the packfile it has just written. The line is therefore five
+ * bytes of prefix, `hexsz` bytes of name, and a trailing newline. A
+ * "keep" prefix tells us that index-pack has also created the
+ * corresponding ".keep" file.
*/
- if (read_in_full(ip_out, packname, len) == len && packname[len-1] == '\n') {
- const char *name;
+ if (read_in_full(ip_out, packname, len) != len || packname[len-1] != '\n')
+ return -1;
+
+ packname[len-1] = 0;
+ if (skip_prefix(packname, "pack\t", &name))
+ keep = 0;
+ else if (skip_prefix(packname, "keep\t", &name))
+ keep = 1;
+ else
+ return -1;
+
+ strbuf_addstr(pack_name, name);
+ if (is_keep)
+ *is_keep = keep;
+
+ return 0;
+}
+
+char *index_pack_lockfile(struct odb_source *source, int ip_out,
+ int *is_well_formed)
+{
+ struct strbuf pack_name = STRBUF_INIT;
+ char *lockfile = NULL;
+ int is_keep = 0;
+ if (read_index_pack_report(source->odb->repo, ip_out, &pack_name, &is_keep)) {
if (is_well_formed)
- *is_well_formed = 1;
- packname[len-1] = 0;
- if (skip_prefix(packname, "keep\t", &name))
- return xstrfmt("%s/pack/pack-%s.keep",
- source->path, name);
+ *is_well_formed = 0;
return NULL;
}
+
if (is_well_formed)
- *is_well_formed = 0;
- return NULL;
+ *is_well_formed = 1;
+
+ /*
+ * If there is no ".keep" file, then there is no lockfile to return.
+ */
+ if (!is_keep) {
+ strbuf_release(&pack_name);
+ return NULL;
+ }
+
+ lockfile = xstrfmt("%s/pack/pack-%s.keep", source->path, pack_name.buf);
+
+ strbuf_release(&pack_name);
+ return lockfile;
}
/*
diff --git a/pack.h b/pack.h
index ada506b5c5..ddf2d52934 100644
--- a/pack.h
+++ b/pack.h
@@ -106,6 +106,16 @@ off_t write_pack_header(struct hashfile *f, uint32_t);
void fixup_pack_header_footer(const struct git_hash_algo *, int,
unsigned char *, const char *, uint32_t,
unsigned char *, off_t);
+
+/*
+ * Read the report that git-index-pack(1) writes to its standard output after
+ * the packfile is written. The hexadecimal name of the packfile is appended to
+ * `pack_name` and, if `is_keep` is non-NULL, set it to indicate whether a
+ * ".keep" file was written. Return 0 on success and a negative error code on
+ * failure.
+ */
+int read_index_pack_report(struct repository *repo, int ip_out,
+ struct strbuf *pack_name, int *is_keep);
char *index_pack_lockfile(struct odb_source *source, int fd,
int *is_well_formed);
--
2.55.0.424.g13c7afec21
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH 1/3] pack-write: lift and expose index-pack report parsing
2026-10-16 2:28 ` [PATCH 1/3] pack-write: lift and expose index-pack report parsing Justin Tobler
@ 2026-10-16 6:58 ` Patrick Steinhardt
0 siblings, 0 replies; 7+ messages in thread
From: Patrick Steinhardt @ 2026-10-16 6:58 UTC (permalink / raw)
To: Justin Tobler; +Cc: git
On Thu, Oct 15, 2026 at 09:28:59PM -0500, Justin Tobler wrote:
> Once git-index-pack(1) finishes writing a packfile, it reports the
> hexadecimal name of that packfile on stdout prefixed with either "pack"
> or "keep" depending on whether a ".keep" file was also created alongside
> it. Callers that are required to manage the lifecycle of ".keep" files
> parse this report via `index_pack_lockfile()`, which provides the path
> to the temporary ".keep" file.
>
> In a subsequent commit, the "files" ODB transaction begins staging
> packfiles in a transaction without eagerly creating a ".keep" file and
> must instead read git-index-pack(1) "pack" output to get the packfile
> name. Extract the generic parsing logic for the git-index-pack(1) output
> into `read_index_pack_report()` in preparation accordingly.
So if I understand correctly, the intent is to eventually stop creating
the ".keep" file in the quarantine directory? I assume that instead,
you'll then manually create that file in the final location before
moving around the pack itself.
If so that makes sense -- it's kind of pointless to create ".keep" files
in quarantines anyway, nobody but the current process should be touching
it anyway.
> diff --git a/pack-write.c b/pack-write.c
> index 85674e4b72..5f5bb894a5 100644
> --- a/pack-write.c
> +++ b/pack-write.c
> @@ -469,33 +469,68 @@ void fixup_pack_header_footer(const struct git_hash_algo *hash_algo,
> fsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);
> }
>
> -char *index_pack_lockfile(struct odb_source *source, int ip_out,
> - int *is_well_formed)
> +int read_index_pack_report(struct repository *repo, int ip_out,
> + struct strbuf *pack_name, int *is_keep)
> {
> + const int len = repo->hash_algo->hexsz + 6;
> char packname[GIT_MAX_HEXSZ + 6];
> - const int len = source->odb->repo->hash_algo->hexsz + 6;
> + const char *name;
> + int keep;
Suggestion, please feel free to ignore:
int is_keep_ignored;
if (!is_keep)
is_keep = &is_keep_ignored;
Then you can unconditionally use `is_keep` further down, which I think
is a bit less awkward.
> /*
> - * The first thing we expect from index-pack's output
> - * is "pack\t%40s\n" or "keep\t%40s\n" (46 bytes) where
> - * %40s is the newly created pack SHA1 name. In the "keep"
> - * case, we need it to remove the corresponding .keep file
> - * later on. If we don't get that then tough luck with it.
> + * The first thing we expect from index-pack's output is
> + * "pack\t<name>\n" or "keep\t<name>\n", where <name> is the hexadecimal
> + * name of the packfile it has just written. The line is therefore five
> + * bytes of prefix, `hexsz` bytes of name, and a trailing newline. A
> + * "keep" prefix tells us that index-pack has also created the
> + * corresponding ".keep" file.
> */
Okay. I originally thought you only reflow this comment, but you fix it
to no longer assume that packfile hashes use SHA1. I would have either
pointed this out in the commit message or done this as a preparatory
commit.
> - if (read_in_full(ip_out, packname, len) == len && packname[len-1] == '\n') {
> - const char *name;
> + if (read_in_full(ip_out, packname, len) != len || packname[len-1] != '\n')
> + return -1;
> +
> + packname[len-1] = 0;
Missing spaces around `-`.
> + if (skip_prefix(packname, "pack\t", &name))
> + keep = 0;
> + else if (skip_prefix(packname, "keep\t", &name))
> + keep = 1;
> + else
> + return -1;
> +
> + strbuf_addstr(pack_name, name);
> + if (is_keep)
> + *is_keep = keep;
> +
> + return 0;
> +}
> +
> +char *index_pack_lockfile(struct odb_source *source, int ip_out,
> + int *is_well_formed)
> +{
> + struct strbuf pack_name = STRBUF_INIT;
> + char *lockfile = NULL;
> + int is_keep = 0;
>
> + if (read_index_pack_report(source->odb->repo, ip_out, &pack_name, &is_keep)) {
> if (is_well_formed)
> - *is_well_formed = 1;
> - packname[len-1] = 0;
> - if (skip_prefix(packname, "keep\t", &name))
> - return xstrfmt("%s/pack/pack-%s.keep",
> - source->path, name);
> + *is_well_formed = 0;
> return NULL;
> }
> +
> if (is_well_formed)
> - *is_well_formed = 0;
> - return NULL;
> + *is_well_formed = 1;
> +
> + /*
> + * If there is no ".keep" file, then there is no lockfile to return.
> + */
> + if (!is_keep) {
> + strbuf_release(&pack_name);
> + return NULL;
> + }
Can we maybe have a single exit path responsible for releasing
`pack_name`?
/* If there is no ".keep" file, then there is no lockfile to return. */
if (is_keep)
lockfile = xstrfmt("%s/pack/pack-%s.keep", source->path, pack_name.buf);
strbuf_release(&pack_name);
return lockfile;
> diff --git a/pack.h b/pack.h
> index ada506b5c5..ddf2d52934 100644
> --- a/pack.h
> +++ b/pack.h
> @@ -106,6 +106,16 @@ off_t write_pack_header(struct hashfile *f, uint32_t);
> void fixup_pack_header_footer(const struct git_hash_algo *, int,
> unsigned char *, const char *, uint32_t,
> unsigned char *, off_t);
> +
> +/*
> + * Read the report that git-index-pack(1) writes to its standard output after
> + * the packfile is written. The hexadecimal name of the packfile is appended to
> + * `pack_name` and, if `is_keep` is non-NULL, set it to indicate whether a
s/set it to/it is set to/
> + * ".keep" file was written. Return 0 on success and a negative error code on
> + * failure.
> + */
> +int read_index_pack_report(struct repository *repo, int ip_out,
> + struct strbuf *pack_name, int *is_keep);
> char *index_pack_lockfile(struct odb_source *source, int fd,
> int *is_well_formed);
Patrick
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 2/3] object-file: explicitly write ".keep" lockfile
2026-10-16 2:28 [PATCH 0/3] object-file: fix ".keep" lockfile ownership in ODB transactions Justin Tobler
2026-10-16 2:28 ` [PATCH 1/3] pack-write: lift and expose index-pack report parsing Justin Tobler
@ 2026-10-16 2:29 ` Justin Tobler
2026-10-16 6:58 ` Patrick Steinhardt
2026-10-16 2:29 ` [PATCH 3/3] object-file: write ".keep" lockfiles during transaction commit Justin Tobler
2 siblings, 1 reply; 7+ messages in thread
From: Justin Tobler @ 2026-10-16 2:29 UTC (permalink / raw)
To: git; +Cc: Justin Tobler
The `odb_transaction_write_pack()` interface for the "files" backend
relies on git-index-pack(1) to write the specified packfile into an ODB
transaction as a packfile. To prevent a concurrent housekeeping
operation from deleting the newly written packfile prior to the
references being updated, a ".keep" lockfile is also generated by
git-index-pack(1) via the `--keep` option. Consequently, the ".keep"
lockfile is written into the ODB transaction alongside the packfile and
later migrated together to the primary ODB during transaction commit.
In a subsequent commit, writing the lockfile is deferred until
transaction commit and thus cannot rely on git-index-pack(1) to write
it. In preparation for this, update `odb_transaction_files_write_pack()`
to explicitly generate the same ".keep" lockfile itself.
Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
object-file.c | 99 ++++++++++++++++++++++++++++++++++++++++-----------
1 file changed, 79 insertions(+), 20 deletions(-)
diff --git a/object-file.c b/object-file.c
index 210984f825..ebe46d35de 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1256,6 +1256,43 @@ int read_loose_object(struct repository *repo,
return ret;
}
+static int write_pack_lock(struct odb_source *source, const char *pack_name,
+ const char *keep_msg)
+{
+ struct strbuf lock_path = STRBUF_INIT;
+ struct strbuf contents = STRBUF_INIT;
+ int ret = -1;
+ int fd;
+
+ strbuf_addf(&lock_path, "%s/pack/pack-%s.keep",
+ source->path, pack_name);
+ strbuf_addf(&contents, "%s\n", keep_msg);
+
+ fd = safe_create_file_with_leading_directories(source->odb->repo,
+ lock_path.buf);
+ if (fd < 0)
+ goto out;
+
+ if (write_in_full(fd, contents.buf, contents.len) < 0) {
+ close(fd);
+ unlink(lock_path.buf);
+ goto out;
+ }
+
+ if (close(fd)) {
+ unlink(lock_path.buf);
+ goto out;
+ }
+
+ ret = 0;
+
+out:
+ strbuf_release(&lock_path);
+ strbuf_release(&contents);
+
+ return ret;
+}
+
static int odb_transaction_files_commit(struct odb_transaction *base)
{
struct odb_transaction_files *transaction =
@@ -1389,18 +1426,17 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
return -1;
}
} else {
+ struct strbuf final_path = STRBUF_INIT;
+ struct strbuf pack_name = STRBUF_INIT;
+ struct strbuf keep_msg = STRBUF_INIT;
char hostname[HOST_NAME_MAX + 1];
- char *lockfile;
+ int ret;
strvec_pushl(&child.args, "index-pack", "--stdin", NULL);
push_header_arg(&child.args, &hdr);
if (xgethostname(hostname, sizeof(hostname)))
xsnprintf(hostname, sizeof(hostname), "localhost");
- strvec_pushf(&child.args,
- "--keep=receive-pack %"PRIuMAX" on %s",
- (uintmax_t)getpid(),
- hostname);
if (!opts->quiet && err_fd)
strvec_push(&child.args, "--show-resolving-progress");
@@ -1424,31 +1460,54 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
return -1;
}
- /*
- * The lockfile filepath is expected to be the final location of
- * the ".keep" file after being migrated to the main ODB source.
- * This ensures the lockfile can be found and removed later
- * after the ODB transaction has been committed.
- */
- lockfile = index_pack_lockfile(base->source, child.out, NULL);
- if (lockfile) {
- ALLOC_GROW(transaction->pack_lockfiles,
- transaction->pack_lockfiles_nr + 1,
- transaction->pack_lockfiles_alloc);
- transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] =
- register_tempfile(lockfile);
- free(lockfile);
- }
+ ret = read_index_pack_report(repo, child.out, &pack_name, NULL);
+
close(child.out);
status = finish_command(&child);
if (status) {
+ strbuf_release(&pack_name);
strbuf_addstr(err_msg, "index-pack abnormal exit");
return -1;
}
+ if (ret) {
+ strbuf_release(&pack_name);
+ strbuf_addstr(err_msg, "index-pack failed to report pack name");
+ return -1;
+ }
+
+ strbuf_addf(&keep_msg, "receive-pack %"PRIuMAX" on %s",
+ (uintmax_t)getpid(), hostname);
+
+ if (write_pack_lock(transaction->quarantine, pack_name.buf, keep_msg.buf)) {
+ strbuf_release(&pack_name);
+ strbuf_release(&keep_msg);
+ strbuf_addstr(err_msg, "lockfile write failed");
+ return -1;
+ }
+
+ ALLOC_GROW(transaction->pack_lockfiles,
+ transaction->pack_lockfiles_nr + 1,
+ transaction->pack_lockfiles_alloc);
+
+ /*
+ * The lockfile filepath is expected to be the final location of
+ * the ".keep" file after being migrated to the main ODB source.
+ * This ensures the lockfile can be found and removed later
+ * after the ODB transaction has been committed.
+ */
+ strbuf_addf(&final_path, "%s/pack/pack-%s.keep",
+ base->source->path, pack_name.buf);
+ transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] =
+ register_tempfile(final_path.buf);
+
odb_source_prepare(transaction->quarantine,
ODB_PREPARE_FLUSH_CACHES);
+
+ strbuf_release(&final_path);
+ strbuf_release(&pack_name);
+ strbuf_release(&keep_msg);
}
return 0;
--
2.55.0.424.g13c7afec21
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH 2/3] object-file: explicitly write ".keep" lockfile
2026-10-16 2:29 ` [PATCH 2/3] object-file: explicitly write ".keep" lockfile Justin Tobler
@ 2026-10-16 6:58 ` Patrick Steinhardt
0 siblings, 0 replies; 7+ messages in thread
From: Patrick Steinhardt @ 2026-10-16 6:58 UTC (permalink / raw)
To: Justin Tobler; +Cc: git
On Thu, Oct 15, 2026 at 09:29:00PM -0500, Justin Tobler wrote:
> diff --git a/object-file.c b/object-file.c
> index 210984f825..ebe46d35de 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1256,6 +1256,43 @@ int read_loose_object(struct repository *repo,
> return ret;
> }
>
> +static int write_pack_lock(struct odb_source *source, const char *pack_name,
> + const char *keep_msg)
Nit: I'd have called this `write_pack_keep()`, but I see that we refer
to the files as "lock" in `index_pack_lockfile(), too. Should we align
those two functions and call this `write_pack_lockfile()`?
> +{
> + struct strbuf lock_path = STRBUF_INIT;
> + struct strbuf contents = STRBUF_INIT;
> + int ret = -1;
> + int fd;
> +
> + strbuf_addf(&lock_path, "%s/pack/pack-%s.keep",
> + source->path, pack_name);
> + strbuf_addf(&contents, "%s\n", keep_msg);
> +
> + fd = safe_create_file_with_leading_directories(source->odb->repo,
> + lock_path.buf);
The command has just written its packfile, so can't we assume that
leading directories have already been created?
> + if (fd < 0)
> + goto out;
> +
> + if (write_in_full(fd, contents.buf, contents.len) < 0) {
> + close(fd);
> + unlink(lock_path.buf);
> + goto out;
> + }
> +
> + if (close(fd)) {
`if (close(fd) < 0)`, just to be consistent.
> @@ -1389,18 +1426,17 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
> return -1;
> }
> } else {
> + struct strbuf final_path = STRBUF_INIT;
> + struct strbuf pack_name = STRBUF_INIT;
> + struct strbuf keep_msg = STRBUF_INIT;
> char hostname[HOST_NAME_MAX + 1];
> - char *lockfile;
> + int ret;
>
> strvec_pushl(&child.args, "index-pack", "--stdin", NULL);
> push_header_arg(&child.args, &hdr);
>
> if (xgethostname(hostname, sizeof(hostname)))
> xsnprintf(hostname, sizeof(hostname), "localhost");
> - strvec_pushf(&child.args,
> - "--keep=receive-pack %"PRIuMAX" on %s",
> - (uintmax_t)getpid(),
> - hostname);
Okay. So instead of asking git-index-pack(1) to write the ".keep" file,
we now...
> @@ -1424,31 +1460,54 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
> return -1;
> }
>
> - /*
> - * The lockfile filepath is expected to be the final location of
> - * the ".keep" file after being migrated to the main ODB source.
> - * This ensures the lockfile can be found and removed later
> - * after the ODB transaction has been committed.
> - */
> - lockfile = index_pack_lockfile(base->source, child.out, NULL);
> - if (lockfile) {
> - ALLOC_GROW(transaction->pack_lockfiles,
> - transaction->pack_lockfiles_nr + 1,
> - transaction->pack_lockfiles_alloc);
> - transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] =
> - register_tempfile(lockfile);
> - free(lockfile);
> - }
> + ret = read_index_pack_report(repo, child.out, &pack_name, NULL);
> +
... read its stdout to figure out where the packfile was written to
and...
> status = finish_command(&child);
> if (status) {
> + strbuf_release(&pack_name);
> strbuf_addstr(err_msg, "index-pack abnormal exit");
> return -1;
> }
>
> + if (ret) {
> + strbuf_release(&pack_name);
> + strbuf_addstr(err_msg, "index-pack failed to report pack name");
> + return -1;
> + }
> +
> + strbuf_addf(&keep_msg, "receive-pack %"PRIuMAX" on %s",
> + (uintmax_t)getpid(), hostname);
> +
> + if (write_pack_lock(transaction->quarantine, pack_name.buf, keep_msg.buf)) {
> + strbuf_release(&pack_name);
> + strbuf_release(&keep_msg);
> + strbuf_addstr(err_msg, "lockfile write failed");
> + return -1;
> + }
... then write the lockfile ourselves.
By the way, the error handling here starts to become unwieldy. Can we
maybe have a common exit path that releases all resources for us?
> + ALLOC_GROW(transaction->pack_lockfiles,
> + transaction->pack_lockfiles_nr + 1,
> + transaction->pack_lockfiles_alloc);
> +
> + /*
> + * The lockfile filepath is expected to be the final location of
> + * the ".keep" file after being migrated to the main ODB source.
> + * This ensures the lockfile can be found and removed later
> + * after the ODB transaction has been committed.
> + */
> + strbuf_addf(&final_path, "%s/pack/pack-%s.keep",
> + base->source->path, pack_name.buf);
> + transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] =
> + register_tempfile(final_path.buf);
So here we retain the status quo, which is to record the lockfile in the
final location even though it doesn't exist there yet. And that can lead
to us pruning a pack lockfile from a different process.
Patrick
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 3/3] object-file: write ".keep" lockfiles during transaction commit
2026-10-16 2:28 [PATCH 0/3] object-file: fix ".keep" lockfile ownership in ODB transactions Justin Tobler
2026-10-16 2:28 ` [PATCH 1/3] pack-write: lift and expose index-pack report parsing Justin Tobler
2026-10-16 2:29 ` [PATCH 2/3] object-file: explicitly write ".keep" lockfile Justin Tobler
@ 2026-10-16 2:29 ` Justin Tobler
2026-10-16 6:58 ` Patrick Steinhardt
2 siblings, 1 reply; 7+ messages in thread
From: Justin Tobler @ 2026-10-16 2:29 UTC (permalink / raw)
To: git; +Cc: Justin Tobler, Qin ShiCheng
The "files" ODB backend writes a ".keep" lockfile into the transaction
quarantine directory via `odb_transaction_files_write_pack()`. When the
ODB transaction is committed, the ".keep" lockfile is migrated to the
primary ODB alongside its packfile. Writing the temporary lockfile in
this manner has a couple of downsides:
- Failing to migrate the lockfile during transaction commit due to a
concurrent process already acquiring the same lock still allows the
rest of the transaction to be migrated into the primary ODB. This
dirties the primary ODB with new objects even though no references
were updated and the operation failed.
- Even more concerning, the ".keep" tempfile atexit(3) handler tries
to remove any lockfile of the same name regardless of whether it was
created by the same process. If the ".keep" lockfile of the same
name for a concurrent process gets deleted prematurely, its packfile
becomes at risk of being deleted by concurrent housekeeping.
Stop creating/registering the temporary lockfiles when writing the
packfile to the transaction and instead do so when committing the
transaction. This ensures the lockfile is properly acquired prior to
migrating objects and that only the owning process can delete its own
lockfiles.
Reported-by: Qin ShiCheng <qeesung@live.com>
Signed-off-by: Justin Tobler <jltobler@gmail.com>
---
object-file.c | 76 +++++++++++++++++++++-----------------
t/t5547-push-quarantine.sh | 29 +++++++++++++--
2 files changed, 69 insertions(+), 36 deletions(-)
diff --git a/object-file.c b/object-file.c
index ebe46d35de..39e6299be1 100644
--- a/object-file.c
+++ b/object-file.c
@@ -492,6 +492,7 @@ struct odb_transaction_files {
struct transaction_packfile packfile;
const char *prefix;
+ struct strvec pack_names;
struct tempfile **pack_lockfiles;
size_t pack_lockfiles_nr;
size_t pack_lockfiles_alloc;
@@ -1257,7 +1258,7 @@ int read_loose_object(struct repository *repo,
}
static int write_pack_lock(struct odb_source *source, const char *pack_name,
- const char *keep_msg)
+ const char *keep_msg, struct tempfile **out)
{
struct strbuf lock_path = STRBUF_INIT;
struct strbuf contents = STRBUF_INIT;
@@ -1284,6 +1285,7 @@ static int write_pack_lock(struct odb_source *source, const char *pack_name,
goto out;
}
+ *out = register_tempfile(lock_path.buf);
ret = 0;
out:
@@ -1293,6 +1295,40 @@ static int write_pack_lock(struct odb_source *source, const char *pack_name,
return ret;
}
+static int write_pack_locks(struct odb_transaction_files *transaction)
+{
+ struct strbuf keep_msg = STRBUF_INIT;
+ char hostname[HOST_NAME_MAX + 1];
+ int ret = 0;
+
+ if (!transaction->pack_names.nr)
+ return 0;
+
+ if (xgethostname(hostname, sizeof(hostname)))
+ xsnprintf(hostname, sizeof(hostname), "localhost");
+ strbuf_addf(&keep_msg, "receive-pack %"PRIuMAX" on %s",
+ (uintmax_t)getpid(), hostname);
+
+ for (size_t i = 0; i < transaction->pack_names.nr; i++) {
+ struct tempfile *lock;
+
+ ret = write_pack_lock(transaction->base.source,
+ transaction->pack_names.v[i],
+ keep_msg.buf, &lock);
+ if (ret < 0)
+ break;
+
+ ALLOC_GROW(transaction->pack_lockfiles,
+ transaction->pack_lockfiles_nr + 1,
+ transaction->pack_lockfiles_alloc);
+ transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] = lock;
+ }
+
+ strbuf_release(&keep_msg);
+
+ return ret;
+}
+
static int odb_transaction_files_commit(struct odb_transaction *base)
{
struct odb_transaction_files *transaction =
@@ -1321,6 +1357,9 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
delete_tempfile(&temp);
strbuf_release(&temp_path);
+ if (write_pack_locks(transaction))
+ return error(_("unable to lock packfiles"));
+
/*
* Make the object files visible in the primary ODB after their data is
* fully durable.
@@ -1426,18 +1465,12 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
return -1;
}
} else {
- struct strbuf final_path = STRBUF_INIT;
struct strbuf pack_name = STRBUF_INIT;
- struct strbuf keep_msg = STRBUF_INIT;
- char hostname[HOST_NAME_MAX + 1];
int ret;
strvec_pushl(&child.args, "index-pack", "--stdin", NULL);
push_header_arg(&child.args, &hdr);
- if (xgethostname(hostname, sizeof(hostname)))
- xsnprintf(hostname, sizeof(hostname), "localhost");
-
if (!opts->quiet && err_fd)
strvec_push(&child.args, "--show-resolving-progress");
if (err_fd)
@@ -1477,37 +1510,12 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
return -1;
}
- strbuf_addf(&keep_msg, "receive-pack %"PRIuMAX" on %s",
- (uintmax_t)getpid(), hostname);
-
- if (write_pack_lock(transaction->quarantine, pack_name.buf, keep_msg.buf)) {
- strbuf_release(&pack_name);
- strbuf_release(&keep_msg);
- strbuf_addstr(err_msg, "lockfile write failed");
- return -1;
- }
-
- ALLOC_GROW(transaction->pack_lockfiles,
- transaction->pack_lockfiles_nr + 1,
- transaction->pack_lockfiles_alloc);
-
- /*
- * The lockfile filepath is expected to be the final location of
- * the ".keep" file after being migrated to the main ODB source.
- * This ensures the lockfile can be found and removed later
- * after the ODB transaction has been committed.
- */
- strbuf_addf(&final_path, "%s/pack/pack-%s.keep",
- base->source->path, pack_name.buf);
- transaction->pack_lockfiles[transaction->pack_lockfiles_nr++] =
- register_tempfile(final_path.buf);
+ strvec_push(&transaction->pack_names, pack_name.buf);
odb_source_prepare(transaction->quarantine,
ODB_PREPARE_FLUSH_CACHES);
- strbuf_release(&final_path);
strbuf_release(&pack_name);
- strbuf_release(&keep_msg);
}
return 0;
@@ -1523,6 +1531,7 @@ static int odb_transaction_files_finalize(struct odb_transaction *base)
ret |= delete_tempfile(&transaction->pack_lockfiles[i]);
free(transaction->pack_lockfiles);
+ strvec_clear(&transaction->pack_names);
return ret;
}
@@ -1555,6 +1564,7 @@ int odb_transaction_files_begin(struct odb_source *source,
transaction->base.write_pack = odb_transaction_files_write_pack;
transaction->base.env = odb_transaction_files_env;
transaction->flags = flags;
+ strvec_init(&transaction->pack_names);
transaction->prefix = "bulk-fsync";
if (flags & ODB_TRANSACTION_RECEIVE) {
diff --git a/t/t5547-push-quarantine.sh b/t/t5547-push-quarantine.sh
index 1b7097179e..cf349eee2c 100755
--- a/t/t5547-push-quarantine.sh
+++ b/t/t5547-push-quarantine.sh
@@ -77,10 +77,9 @@ test_expect_success '.keep file is removed after push' '
git -C keep.git config set receive.unpackLimit 0 &&
# While incoming objects are still quarantined, validate that the
- # ".keep" lockfile is present in the quarantine directory.
+ # ".keep" lockfile is not present in the quarantine directory.
test_hook -C keep.git pre-receive <<-\EOF &&
- keep="$(ls "$GIT_QUARANTINE_PATH"/pack/pack-*.keep)" &&
- test -f "$keep"
+ test ! -e "$GIT_QUARANTINE_PATH"/pack/pack-*.keep
EOF
# After quarantined objects are migrated, validate that the ".keep"
@@ -101,4 +100,28 @@ test_expect_success '.keep file is removed after push' '
test_path_is_missing "$keep"
'
+test_expect_success '.keep file created outside of process is not removed' '
+ test_when_finished rm -rf keep.git &&
+ git init --bare keep.git &&
+
+ git -C keep.git config set receive.unpackLimit 0 &&
+
+ # Create a ".keep" lockfile matching the packfile to simulate a
+ # concurrent git-receive-pack(1) that was already written its lockfile.
+ test_hook -C keep.git pre-receive <<-\EOF &&
+ pack="$(ls "$GIT_QUARANTINE_PATH"/pack/pack-*.pack)" &&
+ pack="${pack##*/}" &&
+ echo "baz" >"objects/pack/${pack%.pack}.keep"
+ EOF
+
+ test_commit bar &&
+ test_must_fail git push keep.git HEAD &&
+
+ # The ".keep" lockfile of the concurrent git-receive-pack(1) should be
+ # left in its place.
+ keep="$(ls keep.git/objects/pack/pack-*.keep)" &&
+ test_path_is_file "$keep" &&
+ test_grep "baz" "$keep"
+'
+
test_done
--
2.55.0.424.g13c7afec21
^ permalink raw reply related [flat|nested] 7+ messages in thread* Re: [PATCH 3/3] object-file: write ".keep" lockfiles during transaction commit
2026-10-16 2:29 ` [PATCH 3/3] object-file: write ".keep" lockfiles during transaction commit Justin Tobler
@ 2026-10-16 6:58 ` Patrick Steinhardt
0 siblings, 0 replies; 7+ messages in thread
From: Patrick Steinhardt @ 2026-10-16 6:58 UTC (permalink / raw)
To: Justin Tobler; +Cc: git, Qin ShiCheng
On Thu, Oct 15, 2026 at 09:29:01PM -0500, Justin Tobler wrote:
> The "files" ODB backend writes a ".keep" lockfile into the transaction
> quarantine directory via `odb_transaction_files_write_pack()`. When the
> ODB transaction is committed, the ".keep" lockfile is migrated to the
> primary ODB alongside its packfile. Writing the temporary lockfile in
> this manner has a couple of downsides:
>
> - Failing to migrate the lockfile during transaction commit due to a
> concurrent process already acquiring the same lock still allows the
> rest of the transaction to be migrated into the primary ODB. This
> dirties the primary ODB with new objects even though no references
> were updated and the operation failed.
I'm not fully sure I understand this downside. Are you saying that we
fail to migrate the lockfile, then continue migrating everything else
only to then fail the whole transaction?
... skipping ahead a bit...
Oh, I get it now. You propose to first acquire all lockfiles before
migrating anything so that we cannot conflict with a concurrent process
that aims to claim the same locks.
That will of course mean that we will fail the transaction for any
concurrent process that happens to write the same packfiles. But based
on what you claim that was already the case anyway, but now we notice
early and thus won't commit parts of the transaction. Makes sense.
> - Even more concerning, the ".keep" tempfile atexit(3) handler tries
> to remove any lockfile of the same name regardless of whether it was
> created by the same process. If the ".keep" lockfile of the same
> name for a concurrent process gets deleted prematurely, its packfile
> becomes at risk of being deleted by concurrent housekeeping.
And this is the other issue that you've already been hinting at.
> diff --git a/object-file.c b/object-file.c
> index ebe46d35de..39e6299be1 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1321,6 +1357,9 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
> delete_tempfile(&temp);
> strbuf_release(&temp_path);
>
> + if (write_pack_locks(transaction))
> + return error(_("unable to lock packfiles"));
Okay, we'll now write lockfiles before migrating any of the data. So if
any of the files already happens to exist, we'll bail out here. It would
have been great to maybe also provide a proper failure message via errno
for example, but okay.
> @@ -1477,37 +1510,12 @@ static int odb_transaction_files_write_pack(struct odb_transaction *base,
> return -1;
> }
>
> - strbuf_addf(&keep_msg, "receive-pack %"PRIuMAX" on %s",
> - (uintmax_t)getpid(), hostname);
> -
> - if (write_pack_lock(transaction->quarantine, pack_name.buf, keep_msg.buf)) {
> - strbuf_release(&pack_name);
> - strbuf_release(&keep_msg);
> - strbuf_addstr(err_msg, "lockfile write failed");
> - return -1;
> - }
And here we stop writing lockfiles when writing the pack.
> diff --git a/t/t5547-push-quarantine.sh b/t/t5547-push-quarantine.sh
> index 1b7097179e..cf349eee2c 100755
> --- a/t/t5547-push-quarantine.sh
> +++ b/t/t5547-push-quarantine.sh
> @@ -77,10 +77,9 @@ test_expect_success '.keep file is removed after push' '
> git -C keep.git config set receive.unpackLimit 0 &&
>
> # While incoming objects are still quarantined, validate that the
> - # ".keep" lockfile is present in the quarantine directory.
> + # ".keep" lockfile is not present in the quarantine directory.
> test_hook -C keep.git pre-receive <<-\EOF &&
> - keep="$(ls "$GIT_QUARANTINE_PATH"/pack/pack-*.keep)" &&
> - test -f "$keep"
> + test ! -e "$GIT_QUARANTINE_PATH"/pack/pack-*.keep
> EOF
>
> # After quarantined objects are migrated, validate that the ".keep"
> @@ -101,4 +100,28 @@ test_expect_success '.keep file is removed after push' '
> test_path_is_missing "$keep"
> '
>
> +test_expect_success '.keep file created outside of process is not removed' '
> + test_when_finished rm -rf keep.git &&
> + git init --bare keep.git &&
> +
> + git -C keep.git config set receive.unpackLimit 0 &&
> +
> + # Create a ".keep" lockfile matching the packfile to simulate a
> + # concurrent git-receive-pack(1) that was already written its lockfile.
> + test_hook -C keep.git pre-receive <<-\EOF &&
> + pack="$(ls "$GIT_QUARANTINE_PATH"/pack/pack-*.pack)" &&
> + pack="${pack##*/}" &&
> + echo "baz" >"objects/pack/${pack%.pack}.keep"
> + EOF
> +
> + test_commit bar &&
> + test_must_fail git push keep.git HEAD &&
> +
> + # The ".keep" lockfile of the concurrent git-receive-pack(1) should be
> + # left in its place.
> + keep="$(ls keep.git/objects/pack/pack-*.keep)" &&
> + test_path_is_file "$keep" &&
> + test_grep "baz" "$keep"
Also, there should be no corresponding ".pack" file, right? Do we want
to assert that, too?
Thanks!
Patrick
^ permalink raw reply [flat|nested] 7+ messages in thread