[PATCH v5 0/2] refs: fix some bugs with batched-updates

[Karthik Nayak <karthik.188@gmail.com>]

In 23fc8e4f61 (refs: implement batch reference update support,
2025-04-08) we introduced a mechanism to batch reference updates. The
idea being that we wanted to batch updates similar to a transaction, but
allow certain updates to fail. This would help reference backends
optimize such operations and also remove the overhead of processing
updates individually. Especially for the reftable backend, where
batching updates would ensure that the autocompaction would only occur
at the end of the batch instead of after every reference update.

As of 9d2962a7c4 (receive-pack: use batched reference updates,
2025-05-19) we also updated the 'git-fetch(1)' and 'git-receive-pack(1)'
command to use batched reference updates. This series fixes some bugs
that we found at GitLab by running our Gitaly service with the `next`
build of Git.

The first being in the files backend, which missed skipping over failed
updates in certain flows. When certain individual updates fail, we mark
them as such. However, we missed skipping over such failed updates,
which can cause a SEGFAULT.

The other is in the git-receive-pack(1) implementation when a user
pushes multiple branches such as:

  delete refs/heads/branch/conflict
  create refs/heads/branch

Before using batched updates, the references would be applied
sequentially and hence no conflicts would arise. With batched updates,
while the first update applies, the second fails due to F/D conflict. A
similar issue was present in 'git-fetch(1)' and was fixed by using
separating out reference pruning into a separate transaction. Apply a
similar mechanism for 'git-receive-pack(1)' and separate out reference
deletions into its own batch.

This is based off master 7014b55638 (A bit more topics for -rc1,
2025-05-30), with the changes from kn/fetch-push-bulk-ref-update merged
in.

---
Changes in v5:
- Modify the commit message in patch 2 to also mention the commit
  which adds multiple transactions to 'git-fetch(1)'.
- Also add a NEEDSWORK to the existing comment to mention how conflict
  resolution within transactions can remove the need for multiple 
  transactions.
- Link to v4: https://lore.kernel.org/r/20250613-6769-address-test-failures-in-the-next-branch-caused-by-batched-reference-updates-v4-0-ebf53edb9795@gmail.com

Changes in v4:
- Swapped out F/D for D/F in the second commit, since we are talking
  about conflicts between a directory and a file, also D/F is more
  consistent.
- Fixed some typos in the second commit.
- Changed comment to single line.
- Link to v3: https://lore.kernel.org/r/20250606-6769-address-test-failures-in-the-next-branch-caused-by-batched-reference-updates-v3-0-e1c41693bd35@gmail.com

Changes in v3:
- Cleanup the second commit message and remove stale comments.
- In the second commit, only create the transaction if needed.
- Link to v2: https://lore.kernel.org/r/20250605-6769-address-test-failures-in-the-next-branch-caused-by-batched-reference-updates-v2-0-26cd05b8a79e@gmail.com

Changes in v2:
- Modify the test in the first commit to no longer do a quiet grep,
  and some more tests.
- Remove the second commit as it was unnecessary.
- Modify the commit message in the last commit, to also talk about how
  we now use 2 transactions at minimum but this is still better than
  before.
- Modify the logic in the last commit to no longer use an XOR and
  instead loop over the two different scenarios (deletion updates, other
  updates).
- Link to v1: https://lore.kernel.org/r/20250602-6769-address-test-failures-in-the-next-branch-caused-by-batched-reference-updates-v1-0-903d1db3f10e@gmail.com

---
 builtin/receive-pack.c | 102 ++++++++++++++++++++++++++++++++-----------------
 refs/files-backend.c   |   7 ++++
 t/t1400-update-ref.sh  |  45 ++++++++++++++++++++++
 t/t5516-fetch-push.sh  |  17 +++++++--
 4 files changed, 133 insertions(+), 38 deletions(-)

Karthik Nayak (2):
      refs/files: skip updates with errors in batched updates
      receive-pack: handle reference deletions separately

Range-diff versus v4:

1:  0ab0890bf1 = 1:  9248bfd474 refs/files: skip updates with errors in batched updates
2:  e45ec3139b ! 2:  00bf816b2d receive-pack: handle reference deletions separately
    @@ Commit message
         sequentially and hence no conflicts would arise. With batched updates,
         while the first update applies, the second fails due to D/F conflict. A
         similar issue was present in 'git-fetch(1)' and was fixed by separating
    -    out reference pruning into a separate transaction. Apply a similar
    -    mechanism for 'git-receive-pack(1)' and separate out reference deletions
    -    into its own batch.
    +    out reference pruning into a separate transaction in the commit 'fetch:
    +    use batched reference updates'. Apply a similar mechanism for
    +    'git-receive-pack(1)' and separate out reference deletions into its own
    +    batch.
     
         This means 'git-receive-pack(1)' will now use up to two transactions,
         whereas before using batched updates it would use _at least_ two
    @@ builtin/receive-pack.c: static void execute_commands_non_atomic(struct command *
     +	 * are treated as conflicts in batched updates. This is because
     +	 * we don't do conflict resolution inside a transaction. To
     +	 * mitigate this, delete references in a separate batch.
    ++	 *
    ++	 * NEEDSWORK: Add conflict resolution between deletion and creation
    ++	 * of reference updates within a transaction. With that, we can
    ++	 * combine the two phases.
     +	 */
     +	enum processing_phase {
     +		PHASE_DELETIONS,


base-commit: 931c39f05e078e0df968a439379cb04b5c4666ef
change-id: 20250528-6769-address-test-failures-in-the-next-branch-caused-by-batched-reference-updates-24ff53680144

Thanks
- Karthik