[PATCH 00/10] reftable: fix reload with active iterators

[Patrick Steinhardt <ps@pks.im>]

Hi,

as reported by Peff in [1], the reftable backend fails in t5616 in a
racy manner. The cause of this is that git-maintenance(1) processes leak
into subsequent tests and perform concurrent compaction of the stack.
The reftable backend is expected to handle this gracefully, but doesn't.

The issue can surface whenever reloading the reftable stack while an
iterator is active. In case some of the tables got compacted, we will
end up closing them and thus the iterator now tries to use those closed
tables.

This patch series fixes that issue by starting to refcount the readers.
Each iterator will bump the refcount, thus avoiding the problem. While
at it, I also found a second issue where we segfault when reloading a
table fails while reusing one of the table readers. In this scenario, we
would end up releasing the reader of the stack itself, even though it
would still be used by it.

This patch series addresses those issues by making the reftable reader
refcounted. The first 6 patches do some simplifications of code which is
close. The remaining 4 ones introduce refcounting and wire it up as
required.

With this, the following command now passes:

    make -C .. -j20 SANITIZE=address && GIT_TEST_DEFAULT_REF_FORMAT=reftable ./t5616-partial-clone.sh --run=1-16 --stress

The patch series builds on top of Junio's ps/reftable-drop-generic at
1a3c3870ee (reftable/generic: drop interface, 2024-08-14). It's only on
seen right now, but would otherise cause a bunch of conflicts.

Thanks!

[1]: <20240817121424.GA2439299@coredump.intra.peff.net>

Patrick Steinhardt (10):
  reftable/blocksource: drop malloc block source
  reftable/stack: inline `stack_compact_range_stats()`
  reftable/reader: rename `reftable_new_reader()`
  reftable/reader: inline `init_reader()`
  reftable/reader: inline `reader_close()`
  reftable/stack: fix broken refnames in `write_n_ref_tables()`
  reftable/reader: introduce refcounting
  reftable/reader: keep readers alive during iteration
  reftable/stack: reorder swapping in the reloaded stack contents
  reftable/stack: fix segfault when reload with reused readers fails

 reftable/block_test.c            |   3 +-
 reftable/blocksource.c           |  20 -----
 reftable/blocksource.h           |   2 -
 reftable/reader.c                | 149 ++++++++++++++++---------------
 reftable/reader.h                |   5 +-
 reftable/readwrite_test.c        |  85 +++++++++---------
 reftable/reftable-reader.h       |  19 ++--
 reftable/stack.c                 |  90 +++++++++++--------
 reftable/stack_test.c            | 115 +++++++++++++++++++++++-
 t/helper/test-reftable.c         |   4 +-
 t/unit-tests/t-reftable-merged.c |  10 +--
 11 files changed, 311 insertions(+), 191 deletions(-)


base-commit: 1a3c3870ee1a65b0579ccbac6b18e22b8c44c5b4
-- 
2.46.0.164.g477ce5ccd6.dirty