git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH 0/3] fetch/receive: use batched reference updates
@ 2025-05-14  9:03 Karthik Nayak
  2025-05-14  9:03 ` [PATCH 1/3] fetch: " Karthik Nayak
                   ` (4 more replies)
  0 siblings, 5 replies; 41+ messages in thread
From: Karthik Nayak @ 2025-05-14  9:03 UTC (permalink / raw)
  To: git; +Cc: toon, ps, Karthik Nayak

The git-fetch(1) and git-receive-pack(1) commands update references as
part of the flow. Each reference update is treated as a single entity
and a transaction is created for each update.

This can be really slow, specifically in reference backends where there
are optimizations which ensure a single transaction with 'N' reference
update perform much faster than 'N' individual transactions. Also having
'N' individual transactions has buildup and teardown costs. These costs
add up in repositories with a large number of references.

Also specifically in the reftable backend, 'N' individual transactions
would also trigger auto-compaction for every transaction.

The reasoning for using individual transactions is because we want to
allow partial updates of references in these commands. Using a single
transaction would be an all-or-nothing scenario.

Recently we introduced an in-between solution called batched reference
updates in 23fc8e4f61 (refs: implement batch reference update support,
2025-04-08). This allows us to batch a set of reference updates, where
individual updates can pass/fail without affecting the batch.

This patch series, modifies both 'git-fetch(1)' and
'git-receive-pack(1)' to use this mechanism. With this, we see a
significant performance boost:

+---------------------+---------------+------------------+
|                     | files backend | reftable backend |
+---------------------+---------------+------------------+
| git-fetch(1)        | 1.25x         | 22x              |
| git-receive-pack(1) | 1.21x         | 18x              |
+---------------------+---------------+------------------+

The first and third patch handle the changes for 'git-fetch(1)' and
'git-receive-pack(1)' respectively. The second patch fixes a small
memory leak I encountered while working on this series.

This is based on top of master: 7a1d2bd0a5 (Merge branch 'master' of
https://github.com/j6t/gitk, 2025-05-09). There were no conflicts 
observed with next or seen.

---
 builtin/fetch.c                  | 119 +++++++++++++++++++++------------------
 builtin/receive-pack.c           |  88 +++++++++++++++++++++++------
 send-pack.c                      |   7 +++
 t/t1416-ref-transaction-hooks.sh |   2 -
 t/t5408-send-pack-stdin.sh       |  14 ++++-
 5 files changed, 155 insertions(+), 75 deletions(-)

Karthik Nayak (3):
      fetch: use batched reference updates
      send-pack: fix memory leak around duplicate refs
      receive-pack: use batched reference updates



base-commit: 7a1d2bd0a596f42a8a7a68d55577967bb454fec0
change-id: 20250423-501-update-git-fetch-1-to-use-partial-transactions-7fb66339fe79

Thanks
- Karthik


^ permalink raw reply	[flat|nested] 41+ messages in thread

end of thread, other threads:[~2025-05-22 15:31 UTC | newest]

Thread overview: 41+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-05-14  9:03 [PATCH 0/3] fetch/receive: use batched reference updates Karthik Nayak
2025-05-14  9:03 ` [PATCH 1/3] fetch: " Karthik Nayak
2025-05-14 12:31   ` Patrick Steinhardt
2025-05-15 11:13     ` Karthik Nayak
2025-05-15 11:30       ` Patrick Steinhardt
2025-05-15 11:36         ` Karthik Nayak
2025-05-14 17:36   ` Junio C Hamano
2025-05-14  9:03 ` [PATCH 2/3] send-pack: fix memory leak around duplicate refs Karthik Nayak
2025-05-14 17:46   ` Junio C Hamano
2025-05-15 11:23     ` Karthik Nayak
2025-05-14  9:03 ` [PATCH 3/3] receive-pack: use batched reference updates Karthik Nayak
2025-05-14 12:31   ` Patrick Steinhardt
2025-05-14 19:00     ` Junio C Hamano
2025-05-15 11:30     ` Karthik Nayak
2025-05-15 14:07 ` [PATCH v2 0/4] fetch/receive: " Karthik Nayak
2025-05-15 14:07   ` [PATCH v2 1/4] refs: add function to translate errors to strings Karthik Nayak
2025-05-15 19:11     ` Jeff King
2025-05-16  9:11       ` Karthik Nayak
2025-05-15 20:26     ` Junio C Hamano
2025-05-16  9:12       ` Karthik Nayak
2025-05-15 14:07   ` [PATCH v2 2/4] fetch: use batched reference updates Karthik Nayak
2025-05-16  5:40     ` Patrick Steinhardt
2025-05-16  9:53       ` Karthik Nayak
2025-05-16 10:00         ` Patrick Steinhardt
2025-05-18 11:30           ` Karthik Nayak
2025-05-15 14:07   ` [PATCH v2 3/4] send-pack: fix memory leak around duplicate refs Karthik Nayak
2025-05-15 14:07   ` [PATCH v2 4/4] receive-pack: use batched reference updates Karthik Nayak
2025-05-15 18:55     ` Jeff King
2025-05-15 19:09       ` Jeff King
2025-05-16 19:49         ` Karthik Nayak
2025-05-19  9:58 ` [PATCH v3 0/4] fetch/receive: " Karthik Nayak
2025-05-19  9:58   ` [PATCH v3 1/4] refs: add function to translate errors to strings Karthik Nayak
2025-05-19  9:58   ` [PATCH v3 2/4] fetch: use batched reference updates Karthik Nayak
2025-05-19  9:58   ` [PATCH v3 3/4] send-pack: fix memory leak around duplicate refs Karthik Nayak
2025-05-19  9:58   ` [PATCH v3 4/4] receive-pack: use batched reference updates Karthik Nayak
2025-05-19 18:14   ` [PATCH v3 0/4] fetch/receive: " Junio C Hamano
2025-05-20  9:05     ` Karthik Nayak
2025-05-21 13:14       ` Junio C Hamano
2025-05-22  6:00       ` Jeff King
2025-05-22  8:50         ` Karthik Nayak
2025-05-22 15:31           ` Jeff King

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).