[NOTES 05/11] Pluggable object databases

[Taylor Blau <me@ttaylorr.com>]

Topic: Pluggable object databases
Leader: Patrick Steinhardt

* Already working towards, since git 2.50.
* Allow innovation on the server side on large binary.
* The design will soon be up for discussion.
* Allow migration between different object format, and allow to be picked later
	by the implementer.
* The planned work is to make the new db more pluggable, right now the work is
	still about refactoring. 2.53 will have a proof of concept. Might take into
	the second half of 2026 to be done.
* Blocker1: The current db format is still not clear. Particularly latency perf
	related issues.
	 * Might be using content chunking hashing, might be using existing db impl
		 like cassandra.
* Blocker2: Second problem is how to generate the packfile.
* Taylor wonder whether we can reuse the current object db, but patrick thinks
	the current impl is too large/complex to adopt. The current refactoring effort
	with better abstraction might speed up future changes.
* Gitster wonders whether we can just use the hash of the chunks' hashes.
* Taylor also thinks a new obj db might become just as complex.
* Patrick thinks the new obj db can be more maintainable. Starting off with a
	brand new abstraction allows faster iteration.
* Rewriting obj db in a new world might be challenging because the pack obj is
	so intimate to so many usage and optimizations (e.g. bitmap), also the need to
	identify big binary obj over the wire.
* Taylor thinks maybe we don't need to rewrite pack obj, but abstracting the
	packfile could make it worse and more verbose.
* Patrick mentions there's already many other adjacent projects abstract away
	from the pack format; e.g. jgit, libgit2. Jgit initially already identified
	Casadra's perf would never work due to latency overhead.
* Taylor suggests we identify a proof of concept with comparable latency to
	existing obj db before doing additional refactoring.
* Ezekiel is refocusing the discussion on targeting large binary files. Maybe
	with large binary files, latency degradation is not as important.
* In git, we already have a divergent code path for large binary files, we just
	chose to store them in the packfile, technically people can change the storage
	selection without refactoring.
* Patrick still thinks having sub-system abstraction would make code more
	maintainable.
* Taylor is supportive about some objects can use the current db vs only have
	the large binary files to use the new db; at least we don't impose the
	overhead over all objects.
* The obj chunk design Patrick proposing is meant to benefit both client side
	storage and server side.
* We should resume this discussion with more concrete usage, right now we are
	still talking about potential scenarios.
* The premisor feature from server side cannot satisfy all clients, since some
	clients don't want to use premisor, so the server side might still be expected
	to have the large binary files on disk.
* The packfile url might still be the main direction we can use to fix the large
	binary issue without doing exploding obj chunking.
* Another benefit of obj chunking is to reduce hash time for large binary files.
	Gerrit currently sees 50% of clone time is due to hashing. Parallel hashing is
	also possible with obj chunking.