Re: [PATCH 1/9] docs: update pack index v3 format

["brian m. carlson" <sandals@crustytoothpaste.net>]

[-- Attachment #1: Type: text/plain, Size: 3615 bytes --]

On 2025-09-24 at 07:55:29, Patrick Steinhardt wrote:
> On Fri, Sep 19, 2025 at 01:09:03AM +0000, brian m. carlson wrote:
> > Our current pack index v3 format uses 4-byte integers to find the
> > trailer of the file.  This effectively means that the file cannot be
> > much larger than 2^32.  While this might at first seem to be okay, we
> > expect that each object will have at least 64 bytes worth of data, which
> > means that no more than about 67 million objects can be stored.
> > 
> > Again, this might seem fine, but unfortunately, we know of many users
> > who attempt to create repos with extremely large numbers of commits to
> > get a "high score," and we've already seen repositories with at least 55
> > million commits.  In the interests of gracefully handling repositories
> > even for these well-intentioned but ultimately misguided users, let's
> > change these lengths to 8 bytes.
> 
> Yeah, this makes sense. We can only assume that repositories will
> continue to grow, so it makes sense to future proof.
> 
> We also have the 4-byte number of objects contained in the pack. But as
> you explain, it's nothing we should need to worry about given that this
> is a mere counter, and not an offset into the file. I doubt that there's
> repositories out there that'll have more than 4 billion objects anytime
> soon.

There are certainly some users who try to do that at $DAYJOB, but they
come to our attention (because our maintenance job fails due to taking
too long) before they get there.  I am not, however, aware of any
actually legitimate and productive uses of repositories that threaten to
break that limit, which is what I think what we should really care
about.

In the event we start seeing those kinds of problems, it should be easy
to implement pack v5 with a corresponding index, just with a larger
number of objects.

> For now we only have SHA256 and SHA1. But thinking about the future,
> there will be a time when SHA256 will be considered broken. I wonder
> whether we should safeguard against that and also specify the trailer
> hash to be agile? That is, instead of hardcoding the hash function, we
> add something like a "primary" hash to the packfile and then use the
> full output of that hash as checksum.
> 
> In any case, please feel free to say "no" to the above thought. It's
> just something that popped into my mind upon reading this.

It is actually that it's the main hash algorithm in use.  So if we add a
third algorithm which is SHA-3-512, then the trailer checksum will be
SHA-3-512 when that's the main algorithm.

Technically, it's also SHA-1 if we're in a SHA-1 repository with SHA-256
compatibility.  That's not a use case I really encourage, but it is a
use case I'm testing because it exposes bugs in our codebase and I
expect people will want to do in-place conversion from SHA-1 only to
SHA-1 with SHA-256 at some point.

I'll fix that for v2.

> I guess one thing that should be explicitly pointed out in the commit
> message is that there are no implementations of the v3 format yet, so
> this is basically updating our envisioned design, only. Otherwise one
> might wonder why we can update the spec just so.

That isn't completely true.  There is an implementation, but it is not
yet on the list, and it follows the spec written here.  I will provide
documentation with the rest of the pack index code when index v3 comes
in, but I wanted to update this in case people are trying to add it in
other implementations as well.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]