Re: Efficiently storing SHA-1 ↔ SHA-256 mappings in compatibility mode

[Junio C Hamano <gitster@pobox.com>]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

I do not know if you want my input (as I wasn't CC'ed), but anyway...

> ...  We can store them in the
> `loose-object-idx`, but since it's not sorted or easily searchable, it's
> going to perform really terribly when we store enough of them.  Right
> now, we read the entire file into two hashmaps (one in each direction)
> and we sometimes need to re-read it when other processes add items, so
> it won't take much to make it be slow and take a lot of memory.
>
> For these reasons, I think we need a different datastore for this and
> I'd like to solicit opinions on what that should look like.  Here are
> some things that come to mind:

I do not see why loose-object-idx is not sorted in the first place,
but to account for new objects getting into the object store, it
would not be a viable way forward to maintain a single sorted file.
We obviously do not want to keep rewriting it in its entirety all
the time,

> Some rough ideas of what this could look like:
>
> * We could repurpose the top-bit of the pack order value in pack index
>   v3 to indicate an object that's not in the pack (this would limit us
>   to 2^31 items per pack).

Nice to see an effort to see if we can do with a small incremental
change, but would a single bit be sufficient to cover all the needs?

I suspect that the answer is no, in which case the v3 pack .idx
format would need to be further tweaked, but in that case we do not
have to resort to such a trick of stealing a single bit from here
and abusing it for other purposes.  We should just make sure that
the new .idx file format can have extensions, unlike older format
that has fixed sections in fixed order.

If there aren't any radically novel idea, I would imagine that our
design would default to have a big base file that is optimized for
reading and searching, plus another format that is easier and
quicker to write that would overlay, possibly in a way similar to
packed and loose refs work?

> * We could write some sort of quadratic rollup format like reftable.

The mapping between two hash formats is stable and once computed can
be cast in stone.  Other attributes like the type of each object may
fall into the same category.  Multi-level roll-up may be overkill
for such static data items, especially if consolidation would be a
simple "merge two sorted files into one sorted file" operation.

As there are some objects for which we need to carry dynamic
information, e.g. "we expect not to have this in our object store
and that is fine", which may be set for objects immediately behind
the shallow-clone boundary, may need to be cleared when the depth of
shallowness changes.  Would it make sense to store these auxiliary
pieces of information in separate place(s)?  I suspect that the
objects that need these extra bits of information form a small
subset of all objects that we need to have the conversion data, so a
separate table that is indexed into using the order in the main
table may not be a bad way to go.