Re: [PATCH 09/17] midx: do not require packs to be sorted in lexicographic order

[Taylor Blau <me@ttaylorr.com>]

On Mon, Dec 08, 2025 at 09:07:33PM -0500, Taylor Blau wrote:
> On Mon, Dec 08, 2025 at 07:26:53PM +0100, Patrick Steinhardt wrote:
> > On Sat, Dec 06, 2025 at 03:31:25PM -0500, Taylor Blau wrote:
> > > Note that this produces MIDXs which may be incompatible with earlier
> > > versions of Git that have stricter requirements on the layout of packs
> > > within a MIDX. This patch does *not* modify the version number of the
> > > MIDX format, since existing versions of Git already know to gracefully
> > > ignore a MIDX with packs that appear out-of-order.
> >
> > Interesting. Did you verify how other implementations of Git behave if
> > we start to relax this requirement? It seems like a somewhat dangerous
> > assumption to me that this will just continue to work.
>
> That's a great point. It looks like current libgit2 assumes[1] that the
> list is sorted and complains loudly if it is not. Presumably other
> implementations behave similarly.
>
> I think that is a compelling enough argument to swing us towards
> bumping the version number to avoid compatibility issues.

I had another thought about how we might work around this without
forcing a compatibility issue, but it's a non-starter. I wanted to share
it on the list for posterity regardless.

I was going to add that we could instead consider adding a new chunk to
the MIDX format that lists the pack names in the order that they should
appear in the pseudo-pack order. Absent of that chunk, the pseudo-pack
order would be defined by the lexicographic order of pack names. If the
chunk exists, it would supersede that ordering.

But that just kicks the can down the road, since implementations like
libgit2 would think that they could read a *.midx file, but then they'd
produce all sorts of errors when trying to read its corresponding
*.bitmap file by permuting its bits out-of-order.

(I'm not sure off-hand whether or not libgit2 supports reading MIDX
bitmaps to begin with. Regardless, we should not introduce the
possibility for such a breakage in clients that *do* support reading
MIDX bitmaps, whether or not libgit2 is such a client.)

Thanks,
Taylor