[PATCH 0/4] send-pack: introduce a `no-ref-delta` capability

[PATCH 0/4] send-pack: introduce a `no-ref-delta` capability

[Taylor Blau]

This series teaches 'send-pack' to avoid writing `REF_DELTA` entries
when the receiving end asks it to.

Some 'receive-pack' implementations may wish to retain the incoming pack
without first building an object ID index, in which case requiring delta
bases to appear earlier in the same pack makes them easier to locate.

The new `no-ref-delta` capability is deliberately independent of
`ofs-delta`, and thus asking the sender not to write `REF_DELTA` entries
does not by itself mean that the receiver understands `OFS_DELTA`
entries. The corresponding `pack-objects` option therefore controls
`REF_DELTA` without changing whether `OFS_DELTA` is allowed.

The main complication is reuse. Ordinary delta reuse reuses the
compressed delta instructions, but rewrites the entry header and base
reference. It can therefore write an existing `REF_DELTA` as an
`OFS_DELTA` when `--delta-base-offset` is in effect and the base has
already been written in the output pack. Deltas against preferred or
external thin-pack bases cannot be reused in this way, since those bases
do not appear in the output at all.

Bitmap pack reuse is different, since it copies entries directly from
an existing pack. Under `--no-ref-delta`, it must inspect candidate
objects individually, omit `REF_DELTA` entries from direct pack reuse,
and leave them to the normal object-writing path.

The patches are organized as follows:

 - The first patch teaches 'test-tool pack-deltas' to list each delta's
   representation and base. I originally wrote the series without this,
   but found that writing tests demonstrating which specific *kind* of
   delta representation was chosen to be awkward without having a
   dedicated test helper.

 - The second patch introduces the `--no-ref-delta` option in
   'pack-objects', though initially with delta- and bitmap-reuse
   disabled for the sake of simplicity.

 - The third patch re-enables ordinary delta- and bitmap-reuse where it
   is safe to do so.

 - The final patch advertises and consumes the new `no-ref-delta`
   capability.

Thanks in advance for your review!

Taylor Blau (4):
  t/helper: teach pack-deltas to list delta entries
  pack-objects: introduce `--no-ref-delta`
  pack-objects: support reuse with `--no-ref-delta`
  send-pack: honor `no-ref-delta` capability

 Documentation/git-pack-objects.adoc         |  8 ++-
 Documentation/gitprotocol-capabilities.adoc | 17 ++++-
 builtin/pack-objects.c                      | 29 ++++++--
 builtin/receive-pack.c                      |  5 ++
 pack-bitmap.c                               | 30 +++++---
 pack-bitmap.h                               |  3 +-
 send-pack.c                                 |  4 ++
 send-pack.h                                 |  1 +
 t/helper/test-pack-deltas.c                 | 69 ++++++++++++++++++
 t/t5300-pack-object.sh                      | 79 ++++++++++++++++++++-
 t/t5332-multi-pack-reuse.sh                 | 16 +++++
 t/t5516-fetch-push.sh                       | 14 ++++
 12 files changed, 252 insertions(+), 23 deletions(-)

-- 
2.55.0

Re: [PATCH 0/4] send-pack: introduce a `no-ref-delta` capability

[Taylor Blau]

On Sun, Jul 12, 2026 at 06:11:47PM -0700, Taylor Blau wrote:
> This series teaches 'send-pack' to avoid writing `REF_DELTA` entries
> when the receiving end asks it to.

Hmmph. ISTM that my scripts for sending patches to the list somehow
broke the Message-ID of the cover letter, so the patches themselves are
not properly connected to this thread.

I'll investigate why that is separately, but in the meantime, the actual
patches may be found beginning here:

    https://lore.kernel.org/git/alQ7WKITYDXfiVn9@com-79390/T/#meaec3602fcf2e3c6d05f7248239c1b167a1e6ddf

Thanks,
Taylor

Re: [PATCH 0/4] send-pack: introduce a `no-ref-delta` capability

[Jeff King]

On Sun, Jul 12, 2026 at 06:11:47PM -0700, Taylor Blau wrote:

> Some 'receive-pack' implementations may wish to retain the incoming pack
> without first building an object ID index, in which case requiring delta
> bases to appear earlier in the same pack makes them easier to locate.

This explanation puzzles me. OK, I can see why you might want to take in
the incoming pack and then sit on it for a bit. But surely you are not
going to update refs without seeing what's in the pack, right? Otherwise
any pushing client can corrupt your repo.

And the only way to know what's in the pack is to index it. At which
point resolving REF_DELTAs is the least of your worries there.

So I have the feeling that there's some ulterior motive, or that this is
part of a larger system, but I don't quite understand what it is. And so
it's hard to say whether this is a sensible approach.

> Bitmap pack reuse is different, since it copies entries directly from
> an existing pack. Under `--no-ref-delta`, it must inspect candidate
> objects individually, omit `REF_DELTA` entries from direct pack reuse,
> and leave them to the normal object-writing path.

Hmm. We wouldn't normally expect verbatim pack-reuse to kick in, since
this is about the client sending to the server. But OK, we certainly
need to make sure that path remains correct.

>  - The final patch advertises and consumes the new `no-ref-delta`
>    capability.

What about thin packs? They'll result in REF_DELTAs on the server once
the pack is completed/indexed. I guess we have the "no-thin" capability,
but I don't think our receive-pack implementation support sending it. I
also wouldn't be terribly surprised if not every client implementation
supports it (it was added in 2013 I think to support libgit2). But I
guess that is also true of your new no-ref-delta; only updated clients
will respect it.

What will/should a server do when they get a ref delta anyway? That
again goes back to the question of: why don't we want ref deltas?

-Peff

Re: [PATCH 0/4] send-pack: introduce a `no-ref-delta` capability

[Taylor Blau]

On Tue, Jul 14, 2026 at 03:45:06AM -0400, Jeff King wrote:
> On Sun, Jul 12, 2026 at 06:11:47PM -0700, Taylor Blau wrote:
>
> > Some 'receive-pack' implementations may wish to retain the incoming pack
> > without first building an object ID index, in which case requiring delta
> > bases to appear earlier in the same pack makes them easier to locate.
>
> This explanation puzzles me. OK, I can see why you might want to take in
> the incoming pack and then sit on it for a bit. But surely you are not
> going to update refs without seeing what's in the pack, right? Otherwise
> any pushing client can corrupt your repo.
>
> And the only way to know what's in the pack is to index it. At which
> point resolving REF_DELTAs is the least of your worries there.
>
> So I have the feeling that there's some ulterior motive, or that this is
> part of a larger system, but I don't quite understand what it is. And so
> it's hard to say whether this is a sensible approach.

The implementation motivating this is write-through in the sense that it
first parses and spools the incoming pack, then replays those exact
bytes together with the same ref commands to an upstream receive-pack.

The packfile contents and pending transaction may be staged before that
upstream request finishes, but no local ref update is published unless
the upstream accepts the push. So the usual receive-pack connectivity
checks still happen before the update becomes visible locally.

(Apologies for all of the hand-waving here, BTW. I'm trying to describe
the system in generic terms to make clear my motivations here, but I am
somewhat limited in what I can discuss.)

In retrospect, I don't think the cover letter distinguishes this well.
The pack that we receive over the wire is stored byte-for-byte as an
immutable artifact, and the per-object physical index is derived
asynchronously. That indexer is designed to operate in a single pass
forward over the pack.

(Supporting REF_DELTA there during the indexing process is possible in
theory, but requires keeping an OID lookup around, delaying resolution,
taking another pass, or rewriting the retained pack. This design avoids
all of those.)

> > Bitmap pack reuse is different, since it copies entries directly from
> > an existing pack. Under `--no-ref-delta`, it must inspect candidate
> > objects individually, omit `REF_DELTA` entries from direct pack reuse,
> > and leave them to the normal object-writing path.
>
> Hmm. We wouldn't normally expect verbatim pack-reuse to kick in, since
> this is about the client sending to the server. But OK, we certainly
> need to make sure that path remains correct.

That is an edge case rather than part of the motivation. It is only
there so that `pack-objects --no-ref-delta` means what it says even
if/when it performs verbatim pack-reuse.

> >  - The final patch advertises and consumes the new `no-ref-delta`
> >    capability.
>
> What about thin packs? They'll result in REF_DELTAs on the server once
> the pack is completed/indexed. I guess we have the "no-thin" capability,
> but I don't think our receive-pack implementation support sending it. I
> also wouldn't be terribly surprised if not every client implementation
> supports it (it was added in 2013 I think to support libgit2). But I
> guess that is also true of your new no-ref-delta; only updated clients
> will respect it.
>
> What will/should a server do when they get a ref delta anyway? That
> again goes back to the question of: why don't we want ref deltas?

The implementation in question already advertises 'no-thin',

A sender honoring `no-ref-delta` cannot send a thin pack in the first
place, since an external base must be encoded as REF_DELTA. `send-pack`
may still invoke `pack-objects` with both `--thin` and `--no-ref-delta`,
but the latter causes it to skip excluded bases.

Older clients may ignore `no-ref-delta` and still get rejected. That is
already the receiver's behavior. The capability just lets updated
clients avoid sending a pack which will be rejected.

If a REF_DELTA arrives anyway, the receiver rejects the pack before
publishing the ref.

Thanks,
Taylor

Re: [PATCH 0/4] send-pack: introduce a `no-ref-delta` capability

[Jeff King]

On Tue, Jul 14, 2026 at 02:58:36PM -0700, Taylor Blau wrote:

> > So I have the feeling that there's some ulterior motive, or that this is
> > part of a larger system, but I don't quite understand what it is. And so
> > it's hard to say whether this is a sensible approach.
> 
> The implementation motivating this is write-through in the sense that it
> first parses and spools the incoming pack, then replays those exact
> bytes together with the same ref commands to an upstream receive-pack.

OK, that kind-of makes sense. But is that intermediate layer not
indexing at all? As in, it does not know which OIDs are present in the
incoming pack?

It sounds like you _do_ index it based on this:

> In retrospect, I don't think the cover letter distinguishes this well.
> The pack that we receive over the wire is stored byte-for-byte as an
> immutable artifact, and the per-object physical index is derived
> asynchronously. That indexer is designed to operate in a single pass
> forward over the pack.

So I can see how REF_DELTA makes a single streaming pass harder. But can
you actually do a single pass even with OFS_DELTA? You'll need to look
at earlier bytes in order to resolve each new object. And possibly
recursively. So you must maintain random access to the new pack data,
and you must maintain some kind of in-memory structure, at least for
each delta family (especially if you want to avoid re-generating deltas
over and over as you satisfy a long chain).

> (Supporting REF_DELTA there during the indexing process is possible in
> theory, but requires keeping an OID lookup around, delaying resolution,
> taking another pass, or rewriting the retained pack. This design avoids
> all of those.)

Right, for each new OID you discover, you need to ask "is this an OID I
was looking to resolve". But the size of that table scales with the
number of objects in the pack. And if you are generating a pack index
with one OID per object in the pack, doesn't that also scale?

I.e., I am having trouble understanding what makes REF_DELTA more
expensive than OFS_DELTA, either in terms of random-access to the pack
or in terms of indexing memory. It's more _complicated_ for sure,
though.

I guess maybe you are also trying to send the resulting .idx to each of
the child receive-packs, and you want to be able to stream it to them as
you go rather than waiting for final REF_DELTA resolution at the end?
You can't send a literal .idx as you generate it (because it's sorted),
but presumably you're mostly trying to offload the hash computation, and
it's OK if the receivers still have to sort the set of (oid,offset)
pairs themselves.

That doesn't seem all _that_ compelling to me. I think the more
interesting thing is that in the worst case, a single REF_DELTA can kill
your pipeline entirely (assuming you are trying to do as much CPU-heavy
resolving work as possible while the pack is transferring). Imagine a
pack like this:

  0: REF_DELTA(abcd)
  1: OFS_DELTA(offsetof(0))
  2: OFS_DELTA(offsetof(1))
  3: OFS_DELTA(offsetof(2))
  ...
  N: base object with hash "abcd"

You can't resolve a single delta until you hit object N, after which you
must then resolve each of 0..N-1 sequentially because they all depend on
each other.

The problem there is not REF_DELTA itself, but the fact that REF_DELTA
allows you to place a base after the delta which depends on it.

If _that_ is your main concern, would it be worth a tighter capability
advertisement that insists that bases come before their deltas (if they
are in the pack at all)? We already generate packs that way by default,
and it would really just give the server a license to reject these
non-standard packs.

But now I'm about 3 levels deep in guessing at your real issues, so I'll
stop for now and see how close I got. ;)

-Peff