[PATCH 0/2] packfile URIs: support concurrent downloads

[PATCH 0/2] packfile URIs: support concurrent downloads

[Ted Nyman]

Packfile URI downloads currently stage a pack at
objects/pack/pack-<hash>.pack.temp. Two Git processes fetching the same
pack into one object database can append to that file concurrently,
which can corrupt the temporary pack or cause a resume request at EOF.

The first patch gives each direct packfile URI download a private
temporary file. Ordinary dumb HTTP pack requests retain their existing
resumable staging behavior. A later packfile URI retry starts a new
download.

The second patch handles the related .keep race. When another process
has already created the keep file, index-pack reports "pack<TAB><hash>"
instead of "keep<TAB><hash>". Accept both successful forms and remove
only keep files created by the current process.

Each patch adds a regression test for its respective race.

Ted Nyman (2):
  http: use unique tempfiles for packfile URI downloads
  fetch-pack: accept "pack" output for packfile URIs

 Documentation/git-http-fetch.adoc |  5 +-
 fetch-pack.c                      | 36 ++++++++-------
 http.c                            | 77 +++++++++++++++++++++----------
 http.h                            |  1 +
 t/t5550-http-fetch-dumb.sh        | 72 ++++++++++++++++++++++++++++-
 t/t5702-protocol-v2.sh            | 31 +++++++++++++
 6 files changed, 177 insertions(+), 45 deletions(-)


base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
-- 
2.55.0

[PATCH 0/2] packfile URIs: support concurrent downloads

[Ted Nyman]

Packfile URI downloads currently stage a pack at
objects/pack/pack-<hash>.pack.temp. Two Git processes fetching the same
pack into one object database can append to that file concurrently,
which can corrupt the temporary pack or cause a resume request at EOF.

The first patch gives each direct packfile URI download a private
temporary file. Ordinary dumb HTTP pack requests retain their existing
resumable staging behavior. A later packfile URI retry starts a new
download.

The second patch handles the related .keep race. When another process
has already created the keep file, index-pack reports "pack<TAB><hash>"
instead of "keep<TAB><hash>". Accept both successful forms and remove
only keep files created by the current process.

Each patch adds a regression test for its respective race.

Ted Nyman (2):
  http: use unique tempfiles for packfile URI downloads
  fetch-pack: accept "pack" output for packfile URIs

 Documentation/git-http-fetch.adoc |  5 +-
 fetch-pack.c                      | 36 ++++++++-------
 http.c                            | 77 +++++++++++++++++++++----------
 http.h                            |  1 +
 t/t5550-http-fetch-dumb.sh        | 72 ++++++++++++++++++++++++++++-
 t/t5702-protocol-v2.sh            | 31 +++++++++++++
 6 files changed, 177 insertions(+), 45 deletions(-)


base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
-- 
2.55.0

[PATCH 0/2] packfile URIs: support concurrent downloads

[Ted Nyman]

Packfile URI downloads currently stage a pack at
objects/pack/pack-<hash>.pack.temp. Two Git processes fetching the same
pack into one object database can append to that file concurrently,
which can corrupt the temporary pack or cause a resume request at EOF.

The first patch gives each direct packfile URI download a private
temporary file. Ordinary dumb HTTP pack requests retain their existing
resumable staging behavior. A later packfile URI retry starts a new
download.

The second patch handles the related .keep race. When another process
has already created the keep file, index-pack reports "pack<TAB><hash>"
instead of "keep<TAB><hash>". Accept both successful forms and remove
only keep files created by the current process.

Each patch adds a regression test for its respective race.

Ted Nyman (2):
  http: use unique tempfiles for packfile URI downloads
  fetch-pack: accept "pack" output for packfile URIs

 Documentation/git-http-fetch.adoc |  5 +-
 fetch-pack.c                      | 36 ++++++++-------
 http.c                            | 77 +++++++++++++++++++++----------
 http.h                            |  1 +
 t/t5550-http-fetch-dumb.sh        | 72 ++++++++++++++++++++++++++++-
 t/t5702-protocol-v2.sh            | 31 +++++++++++++
 6 files changed, 177 insertions(+), 45 deletions(-)


base-commit: e9019fcafe0040228b8631c30f97ae1adb61bcdc
-- 
2.55.0

Re: [PATCH 0/2] packfile URIs: support concurrent downloads

[Junio C Hamano]

Ted Nyman <tnyman@openai.com> writes:

> Packfile URI downloads currently stage a pack at
> objects/pack/pack-<hash>.pack.temp. Two Git processes fetching the same
> pack into one object database can append to that file concurrently,
> which can corrupt the temporary pack or cause a resume request at EOF.
>
> The first patch gives each direct packfile URI download a private
> temporary file. Ordinary dumb HTTP pack requests retain their existing
> resumable staging behavior. A later packfile URI retry starts a new
> download.
>
> The second patch handles the related .keep race. When another process
> has already created the keep file, index-pack reports "pack<TAB><hash>"
> instead of "keep<TAB><hash>". Accept both successful forms and remove
> only keep files created by the current process.
>
> Each patch adds a regression test for its respective race.
>
> Ted Nyman (2):
>   http: use unique tempfiles for packfile URI downloads
>   fetch-pack: accept "pack" output for packfile URIs

This cover letter has

    Message-ID: <alVn7UWvdWRAG-Vv@com-76773>

but in the header of [PATCH 1/2] has

    Message-ID: <alVn-QmK3K91_tkH@com-76773>
    References: <cover.1783982021.git.tnyman@openai.com>
    In-Reply-To: <cover.1783982021.git.tnyman@openai.com>

Similarly, [PATCH 2/2] has

    Message-ID: <alVoA5-fDDPwKPZZ@com-76773>
    References: <cover.1783982021.git.tnyman@openai.com>
    In-Reply-To: <cover.1783982021.git.tnyman@openai.com>

And "b4 am" seems to be having problem grabbing the patchset X-<.

Re: [PATCH 0/2] packfile URIs: support concurrent downloads

[Ted Nyman]

> And "b4 am" seems to be having problem grabbing the patchset X-<.

Sorry for the noise -- Mutt rewrote the original cover-letter
Message-ID. The patches reference the corrected cover:

  https://lore.kernel.org/git/cover.1783982021.git.tnyman@openai.com/

I confirmed that this retrieves both patches:

  b4 am cover.1783982021.git.tnyman@openai.com

Thanks,
Ted

Re: [PATCH 0/2] packfile URIs: support concurrent downloads

[Taylor Blau]

On Mon, Jul 13, 2026 at 03:55:28PM -0700, Ted Nyman wrote:
> > And "b4 am" seems to be having problem grabbing the patchset X-<.
>
> Sorry for the noise -- Mutt rewrote the original cover-letter
> Message-ID. The patches reference the corrected cover:
>
>   https://lore.kernel.org/git/cover.1783982021.git.tnyman@openai.com/
>
> I confirmed that this retrieves both patches:
>
>   b4 am cover.1783982021.git.tnyman@openai.com

This is a mistake on my end as I was porting over some of the scripts
for sending patches to the mailing list to OpenAI's infrastructure.

The short version of this e-mail is that the issue is fixed. But the
longer version is funny (at least to me), so I figured I would share.

As some background, my workflow for sending patches to the mailing list
is to use a script called 'git mail' that effectively runs format-patch
to build an *.mbox and then opens Mutt in that directory. I then review
the patches one last time before sending, and then run a macro I have
bound to 'B', which (effectively) runs <resend-message>.

For reasons that I cannot quite recall, I chose this workflow many years
ago when it would likely have been more appropriate to use `mutt -H`,
which does *not* rewrite Message-ID headers when resending.

To work around this, I wrote a patch that I applied to the version of
Mutt I used both on my old work laptop as well as the Linux workstation
where I did the majority of my work. The patch is fairly small, and is
effectively:

--- 8< ---
diff --git a/postpone.c b/postpone.c
index f557976d..accbb4f6 100644
--- a/postpone.c
+++ b/postpone.c
@@ -607,13 +607,9 @@ int mutt_prepare_template (FILE *fp, CONTEXT *ctx, HEADER *newhdr, HEADER *hdr,
   newhdr->content->length = hdr->content->length;
   mutt_parse_part (fp, newhdr->content);

-  /* If resending a message, don't keep message_id or mail_followup_to.
-   * Otherwise, we are resuming a postponed message, and want to keep those
-   * headers if they exist.
-   */
+  /* If resending a message, don't keep mail_followup_to. */
   if (resend)
   {
-    FREE (&newhdr->env->message_id);
     FREE (&newhdr->env->mail_followup_to);
   }

--
2.26.0.106.g9fadedd637
--- >8 ---

(The Git version this patch was prepared with should give you some sense
of how ancient this part of my workflow is ;-).)

When looking at this yesterday after sending the 'no-ref-delta' patches
to the list, I could not figure out quite why my Mutt client was
rewriting Message-ID headers until I remembered the aforementioned
patch.

The fix is somewhat OpenAI-specific, and so not interesting to share
with the list, but effectively relies on piping messages to 'mutt -H -'
to send the message without dropping (and thus rewriting) the Message-ID
header.

(As an alternative, I could have continued to carry that patch to
'postpone.c', but in retrospect it seems gross^W unnecessary, so I
ditched it.)

Thanks,
Taylor

Re: [PATCH 0/2] packfile URIs: support concurrent downloads

[Taylor Blau]

On Mon, Jul 13, 2026 at 03:37:58PM -0700, Ted Nyman wrote:
> Ted Nyman (2):
>   http: use unique tempfiles for packfile URI downloads
>   fetch-pack: accept "pack" output for packfile URIs

I left one pretty minor style-nit on the first patch, but otherwise this
looks good to me.

Thanks,
Taylor

Re: [PATCH 0/2] packfile URIs: support concurrent downloads

[Jeff King]

On Mon, Jul 13, 2026 at 07:42:25PM -0700, Taylor Blau wrote:

> As some background, my workflow for sending patches to the mailing list
> is to use a script called 'git mail' that effectively runs format-patch
> to build an *.mbox and then opens Mutt in that directory. I then review
> the patches one last time before sending, and then run a macro I have
> bound to 'B', which (effectively) runs <resend-message>.
> 
> For reasons that I cannot quite recall, I chose this workflow many years
> ago when it would likely have been more appropriate to use `mutt -H`,
> which does *not* rewrite Message-ID headers when resending.

You might have inherited the <resend-message> thing from me. I thought I
used it exactly because "-H" insisted on rewriting the message-id, but
it doesn't seem to now. So either I've completely forgotten the reason,
or perhaps the behavior used to be different.

I also like that <resend-message> lets me open the whole mbox and send
each message within a single session. But I never run into the issue
you're mentioning because I write my cover letter separately as a normal
email, and then generate the actual patches as in-reply-to (with some
script magic to pull the message-id from my sent folder).

So possibly my fault for leading you in the wrong direction many years
ago, or your fault for not following my sage advice to the letter. ;)

-Peff