git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: Karthik Nayak <karthik.188@gmail.com>,
	 "brian m. carlson" <sandals@crustytoothpaste.net>,
	 Jeff King <peff@peff.net>, Junio C Hamano <gitster@pobox.com>,
	 Christian Couder <chriscool@tuxfamily.org>
Subject: [PATCH 14/14] refs: reuse iterators when determining refname availability
Date: Mon, 17 Feb 2025 16:50:28 +0100	[thread overview]
Message-ID: <20250217-pks-update-ref-optimization-v1-14-a2b6d87a24af@pks.im> (raw)
In-Reply-To: <20250217-pks-update-ref-optimization-v1-0-a2b6d87a24af@pks.im>

When verifying whether refnames are available we have to verify whether
any reference exists that is nested under the current reference. E.g.
given a reference "refs/heads/foo", we must make sure that there is no
other reference "refs/heads/foo/*".

This check is performed using a ref iterator with the prefix set to the
nested reference namespace. Until now it used to not be possible to
reseek iterators, so we always had to reallocate the iterator for every
single reference we're about to check. This keeps us from reusing state
that the iterator may have and that may make it work more efficiently.

Refactor the logic to reseek iterators. This leads to a speedup with the
reftable backend, which is the only backend that knows to batch refname
availability checks:

    Benchmark 1: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)
      Time (mean ± σ):      39.8 ms ±   0.9 ms    [User: 29.7 ms, System: 9.8 ms]
      Range (min … max):    38.4 ms …  42.0 ms    62 runs

    Benchmark 2: update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD)
      Time (mean ± σ):      31.9 ms ±   1.1 ms    [User: 27.0 ms, System: 4.5 ms]
      Range (min … max):    29.8 ms …  34.3 ms    74 runs

    Summary
      update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD) ran
        1.25 ± 0.05 times faster than update-ref: create many refs (refformat = reftable, preexisting = 100000, new = 10000, revision = HEAD~)

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 refs.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/refs.c b/refs.c
index 8eff60a2186..6cbb9decdb0 100644
--- a/refs.c
+++ b/refs.c
@@ -2555,8 +2555,13 @@ int refs_verify_refnames_available(struct ref_store *refs,
 		if (!initial_transaction) {
 			int ok;
 
-			iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
-						       DO_FOR_EACH_INCLUDE_BROKEN);
+			if (!iter) {
+				iter = refs_ref_iterator_begin(refs, dirname.buf, NULL, 0,
+							       DO_FOR_EACH_INCLUDE_BROKEN);
+			} else if (ref_iterator_seek(iter, dirname.buf) < 0) {
+				goto cleanup;
+			}
+
 			while ((ok = ref_iterator_advance(iter)) == ITER_OK) {
 				if (skip &&
 				    string_list_has_string(skip, iter->refname))
@@ -2569,9 +2574,6 @@ int refs_verify_refnames_available(struct ref_store *refs,
 
 			if (ok != ITER_DONE)
 				BUG("error while iterating over references");
-
-			ref_iterator_free(iter);
-			iter = NULL;
 		}
 
 		extra_refname = find_descendant_ref(dirname.buf, extras, skip);

-- 
2.48.1.666.gff9fcf71b7.dirty


  parent reply	other threads:[~2025-02-17 15:50 UTC|newest]

Thread overview: 169+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-02-17 15:50 [PATCH 00/14] refs: batch refname availability checks Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 01/14] object-name: introduce `repo_get_oid_with_flags()` Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 02/14] object-name: allow skipping ambiguity checks in `get_oid()` family Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 03/14] builtin/update-ref: skip ambiguity checks when parsing object IDs Patrick Steinhardt
2025-02-18 16:04   ` Karthik Nayak
2025-02-17 15:50 ` [PATCH 04/14] refs: introduce function to batch refname availability checks Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 05/14] refs/reftable: start using `refs_verify_refnames_available()` Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 06/14] refs: stop re-verifying common prefixes for availability Patrick Steinhardt
2025-02-18 16:12   ` Karthik Nayak
2025-02-19 11:52     ` Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 07/14] refs/iterator: separate lifecycle from iteration Patrick Steinhardt
2025-02-18 16:52   ` shejialuo
2025-02-19 11:52     ` Patrick Steinhardt
2025-02-19 12:41       ` shejialuo
2025-02-19 12:59         ` Patrick Steinhardt
2025-02-19 13:06           ` shejialuo
2025-02-19 13:17             ` Patrick Steinhardt
2025-02-19 13:20               ` Patrick Steinhardt
2025-02-19 13:23                 ` shejialuo
2025-02-18 17:13   ` Karthik Nayak
2025-02-19 11:52     ` Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 08/14] refs/iterator: provide infrastructure to re-seek iterators Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 09/14] refs/iterator: implement seeking for merged iterators Patrick Steinhardt
2025-02-19 20:10   ` Karthik Nayak
2025-02-17 15:50 ` [PATCH 10/14] refs/iterator: implement seeking for reftable iterators Patrick Steinhardt
2025-02-19 20:13   ` Karthik Nayak
2025-02-17 15:50 ` [PATCH 11/14] refs/iterator: implement seeking for ref-cache iterators Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 12/14] refs/iterator: implement seeking for `packed-ref` iterators Patrick Steinhardt
2025-02-17 15:50 ` [PATCH 13/14] refs/iterator: implement seeking for "files" iterators Patrick Steinhardt
2025-02-17 15:50 ` Patrick Steinhardt [this message]
2025-02-18 17:10 ` [PATCH 00/14] refs: batch refname availability checks brian m. carlson
2025-02-19 13:23 ` [PATCH v2 00/16] " Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 01/16] object-name: introduce `repo_get_oid_with_flags()` Patrick Steinhardt
2025-02-19 17:02     ` Justin Tobler
2025-02-19 13:23   ` [PATCH v2 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family Patrick Steinhardt
2025-02-21  8:00     ` Jeff King
2025-02-21  8:36       ` Patrick Steinhardt
2025-02-21  9:06         ` Jeff King
2025-02-19 13:23   ` [PATCH v2 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs Patrick Steinhardt
2025-02-19 18:21     ` Justin Tobler
2025-02-20  8:05       ` Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 04/16] refs: introduce function to batch refname availability checks Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 05/16] refs/reftable: " Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 06/16] refs/files: batch refname availability checks for normal transactions Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 07/16] refs/files: batch refname availability checks for initial transactions Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 08/16] refs: stop re-verifying common prefixes for availability Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 09/16] refs/iterator: separate lifecycle from iteration Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 10/16] refs/iterator: provide infrastructure to re-seek iterators Patrick Steinhardt
2025-02-24 13:08     ` shejialuo
2025-02-25  7:39       ` Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 11/16] refs/iterator: implement seeking for merged iterators Patrick Steinhardt
2025-02-24 13:37     ` shejialuo
2025-02-25  7:39       ` Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 12/16] refs/iterator: implement seeking for reftable iterators Patrick Steinhardt
2025-02-24 14:00     ` shejialuo
2025-02-25  7:39       ` Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 13/16] refs/iterator: implement seeking for ref-cache iterators Patrick Steinhardt
2025-02-24 14:49     ` shejialuo
2025-02-25  7:39       ` Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 14/16] refs/iterator: implement seeking for `packed-ref` iterators Patrick Steinhardt
2025-02-24 15:09     ` shejialuo
2025-02-25  7:39       ` Patrick Steinhardt
2025-02-25 12:07         ` shejialuo
2025-02-19 13:23   ` [PATCH v2 15/16] refs/iterator: implement seeking for "files" iterators Patrick Steinhardt
2025-02-19 13:23   ` [PATCH v2 16/16] refs: reuse iterators when determining refname availability Patrick Steinhardt
2025-02-24 15:14     ` shejialuo
2025-02-24 15:18   ` [PATCH v2 00/16] refs: batch refname availability checks shejialuo
2025-02-25  7:39     ` Patrick Steinhardt
2025-02-25  8:55 ` [PATCH v3 " Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 01/16] object-name: introduce `repo_get_oid_with_flags()` Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs Patrick Steinhardt
2025-02-26 22:26     ` Junio C Hamano
2025-02-27 11:57       ` Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 04/16] refs: introduce function to batch refname availability checks Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 05/16] refs/reftable: " Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 06/16] refs/files: batch refname availability checks for normal transactions Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 07/16] refs/files: batch refname availability checks for initial transactions Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 08/16] refs: stop re-verifying common prefixes for availability Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 09/16] refs/iterator: separate lifecycle from iteration Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 10/16] refs/iterator: provide infrastructure to re-seek iterators Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 11/16] refs/iterator: implement seeking for merged iterators Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 12/16] refs/iterator: implement seeking for reftable iterators Patrick Steinhardt
2025-02-25  8:55   ` [PATCH v3 13/16] refs/iterator: implement seeking for ref-cache iterators Patrick Steinhardt
2025-02-25  8:56   ` [PATCH v3 14/16] refs/iterator: implement seeking for packed-ref iterators Patrick Steinhardt
2025-02-25  8:56   ` [PATCH v3 15/16] refs/iterator: implement seeking for files iterators Patrick Steinhardt
2025-02-25  8:56   ` [PATCH v3 16/16] refs: reuse iterators when determining refname availability Patrick Steinhardt
2025-02-28  9:26 ` [PATCH v4 00/16] refs: batch refname availability checks Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 01/16] object-name: introduce `repo_get_oid_with_flags()` Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family Patrick Steinhardt
2025-03-06 13:21     ` Karthik Nayak
2025-02-28  9:26   ` [PATCH v4 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 04/16] refs: introduce function to batch refname availability checks Patrick Steinhardt
2025-03-06 13:47     ` Karthik Nayak
2025-02-28  9:26   ` [PATCH v4 05/16] refs/reftable: " Patrick Steinhardt
2025-03-06 14:00     ` Karthik Nayak
2025-03-06 14:12       ` Karthik Nayak
2025-03-06 15:13         ` Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 06/16] refs/files: batch refname availability checks for normal transactions Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 07/16] refs/files: batch refname availability checks for initial transactions Patrick Steinhardt
2025-03-06 14:10     ` Karthik Nayak
2025-02-28  9:26   ` [PATCH v4 08/16] refs: stop re-verifying common prefixes for availability Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 09/16] refs/iterator: separate lifecycle from iteration Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 10/16] refs/iterator: provide infrastructure to re-seek iterators Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 11/16] refs/iterator: implement seeking for merged iterators Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 12/16] refs/iterator: implement seeking for reftable iterators Patrick Steinhardt
2025-03-06 14:16     ` Karthik Nayak
2025-02-28  9:26   ` [PATCH v4 13/16] refs/iterator: implement seeking for ref-cache iterators Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 14/16] refs/iterator: implement seeking for packed-ref iterators Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 15/16] refs/iterator: implement seeking for files iterators Patrick Steinhardt
2025-02-28  9:26   ` [PATCH v4 16/16] refs: reuse iterators when determining refname availability Patrick Steinhardt
2025-03-06 14:20   ` [PATCH v4 00/16] refs: batch refname availability checks Karthik Nayak
2025-03-06 15:08 ` [PATCH v5 " Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 01/16] object-name: introduce `repo_get_oid_with_flags()` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family Patrick Steinhardt
2025-03-12 12:12     ` shejialuo
2025-03-06 15:08   ` [PATCH v5 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 04/16] refs: introduce function to batch refname availability checks Patrick Steinhardt
2025-03-12 12:36     ` shejialuo
2025-03-12 12:44       ` shejialuo
2025-03-12 15:36       ` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 05/16] refs/reftable: " Patrick Steinhardt
2025-03-12 12:54     ` shejialuo
2025-03-12 15:36       ` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 06/16] refs/files: batch refname availability checks for normal transactions Patrick Steinhardt
2025-03-12 12:58     ` shejialuo
2025-03-12 15:36       ` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 07/16] refs/files: batch refname availability checks for initial transactions Patrick Steinhardt
2025-03-12 13:06     ` shejialuo
2025-03-12 15:36       ` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 08/16] refs: stop re-verifying common prefixes for availability Patrick Steinhardt
2025-03-12 13:22     ` shejialuo
2025-03-12 15:36       ` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 09/16] refs/iterator: separate lifecycle from iteration Patrick Steinhardt
2025-03-12 13:45     ` shejialuo
2025-03-12 15:36       ` Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 10/16] refs/iterator: provide infrastructure to re-seek iterators Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 11/16] refs/iterator: implement seeking for merged iterators Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 12/16] refs/iterator: implement seeking for reftable iterators Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 13/16] refs/iterator: implement seeking for ref-cache iterators Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 14/16] refs/iterator: implement seeking for packed-ref iterators Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 15/16] refs/iterator: implement seeking for files iterators Patrick Steinhardt
2025-03-06 15:08   ` [PATCH v5 16/16] refs: reuse iterators when determining refname availability Patrick Steinhardt
2025-03-06 15:32   ` [PATCH v5 00/16] refs: batch refname availability checks Karthik Nayak
2025-03-12 14:03   ` shejialuo
2025-03-12 15:56 ` [PATCH v6 " Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 01/16] object-name: introduce `repo_get_oid_with_flags()` Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 02/16] object-name: allow skipping ambiguity checks in `get_oid()` family Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 03/16] builtin/update-ref: skip ambiguity checks when parsing object IDs Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 04/16] refs: introduce function to batch refname availability checks Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 05/16] refs/reftable: " Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 06/16] refs/files: batch refname availability checks for normal transactions Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 07/16] refs/files: batch refname availability checks for initial transactions Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 08/16] refs: stop re-verifying common prefixes for availability Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 09/16] refs/iterator: separate lifecycle from iteration Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 10/16] refs/iterator: provide infrastructure to re-seek iterators Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 11/16] refs/iterator: implement seeking for merged iterators Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 12/16] refs/iterator: implement seeking for reftable iterators Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 13/16] refs/iterator: implement seeking for ref-cache iterators Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 14/16] refs/iterator: implement seeking for packed-ref iterators Patrick Steinhardt
2025-04-03 19:56     ` Elijah Newren
2025-04-03 22:18       ` brian m. carlson
2025-04-04  7:18         ` shejialuo
2025-04-04 10:00       ` Patrick Steinhardt
2025-04-04 10:05         ` Patrick Steinhardt
2025-04-04 10:59           ` Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 15/16] refs/iterator: implement seeking for files iterators Patrick Steinhardt
2025-03-12 15:56   ` [PATCH v6 16/16] refs: reuse iterators when determining refname availability Patrick Steinhardt
2025-03-13  2:57   ` [PATCH v6 00/16] refs: batch refname availability checks shejialuo

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250217-pks-update-ref-optimization-v1-14-a2b6d87a24af@pks.im \
    --to=ps@pks.im \
    --cc=chriscool@tuxfamily.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=karthik.188@gmail.com \
    --cc=peff@peff.net \
    --cc=sandals@crustytoothpaste.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).