Git development
 help / color / mirror / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: Toon Claes <toon@iotcl.com>
Cc: git@vger.kernel.org, Gusted <gusted@codeberg.org>,
	Jeff King <peff@peff.net>, Taylor Blau <me@ttaylorr.com>
Subject: Re: [PATCH v4 5/6] last-modified: check pathspec against Bloom filter first
Date: Thu, 10 Sep 2026 09:04:15 +0200	[thread overview]
Message-ID: <aqJWb1kq81A8AJWI@pks.im> (raw)
In-Reply-To: <20260901-toon-speed-up-last-modified-v4-5-a09949800404@iotcl.com>

On Tue, Sep 01, 2026 at 11:10:25AM +0200, Toon Claes wrote:
> When git-last-modified(1) starts, it builds a list of all the paths
> matching the pathspec it needs to find the last modifying commit for.
> For example, every file and subdirectory listed by:
> 
>     $ git last-modified -t --max-depth=0 -- src/
> 
> As it resolves a commit for each path during the revision walk, it drops
> that path from the list.
> 
> To avoid diffing trees for every commit, Bloom filters are used when
> available. For each remaining path, the commit's Bloom filter is checked
> to see whether the commit changed that path. The Bloom filter says
> either "no" or "maybe", and only in the latter case is the diff
> calculated.
> 
> git-log(1) does this differently. It does not expand the pathspec but
> checks the Bloom filter against the pathspec itself. This way, commits
> not touching any path matching the pathspec can be discarded as a whole.
> 
> Apply this same check to git-last-modified(1). In a previous commit the
> function revs_maybe_changed_in_bloom(), used by git-log(1), was made
> public. Use this as a pre-filter in git-last-modified(1). After this
> pre-filter, paths are still checked one-by-one to only find those which
> don't have a "last commit" yet.

So in theory, we _might_ now do some of the checks multiple times. But
the expectation is that the number of pathspecs is typically much lower
than the number of expanded paths to check against, so in most cases it
should be faster to do this pre-filtering?

It'll probably be possible to craft edge cases where the new logic is
slower because we now do more work in the matching case. But overall I
think this is a sensible tradeoff. After all, we use the same tradeoff
in git-log(1).

> With `--show-trees` the list holds more than the paths matching the
> pathspec. It also holds each parent tree entry, up to the root. Each of
> those can resolve to a different commit. Thus for the pathspec "a/b/c",
> the list will also hold "a" and "a/b".
> 
> When a commit touches "a/other", that commit could be the last commit
> for "a", but revs_maybe_changed_in_bloom() would discard it, because it
> doesn't match the full pathspec.
> 
> Instead, when `--show-trees` is given, use
> revs_maybe_changed_in_bloom_with_parents(), which indicates the commit
> maybe changed any of the paths leading up to the path in the pathspec.

You explain what we do and why it's safe, which is good. But what's
missing is the "why". As far as I understand the reason is performance,
but if so I'd have expected a benchmark demonstrating the benefit.

Patrick

  reply	other threads:[~2026-09-10  7:04 UTC|newest]

Thread overview: 52+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-17 15:46 [PATCH 0/4] last-modified: use the pathspec's Bloom key to pre-filter commits Toon Claes
2026-07-17 15:46 ` [PATCH 1/4] revision: move bloom keyvec precondition into function Toon Claes
2026-07-18  7:57   ` Jeff King
2026-08-05 19:16     ` Toon Claes
2026-08-05 20:32       ` Jeff King
2026-07-17 15:47 ` [PATCH 2/4] revision: expose check for paths maybe changed in Bloom filter Toon Claes
2026-07-17 20:47   ` Junio C Hamano
2026-07-17 23:26     ` Taylor Blau
2026-07-17 15:47 ` [PATCH 3/4] last-modified: check pathspec against Bloom filter first Toon Claes
2026-07-17 23:05   ` Taylor Blau
2026-07-18  8:37     ` Jeff King
2026-07-18 21:22       ` Taylor Blau
2026-07-20  9:42         ` Jeff King
2026-07-17 15:47 ` [PATCH 4/4] last-modified: keep per-path Bloom filters for wildcard pathspecs Toon Claes
2026-07-17 19:16   ` Toon Claes
2026-07-18  8:14     ` Jeff King
2026-08-04 22:19       ` Junio C Hamano
2026-08-05  0:43         ` Taylor Blau
2026-08-05 16:01           ` Junio C Hamano
2026-08-05  1:18         ` Jeff King
2026-07-17 23:18   ` Taylor Blau
2026-07-17 19:13 ` [PATCH 0/4] last-modified: use the pathspec's Bloom key to pre-filter commits Toon Claes
2026-08-07 18:26 ` [PATCH v2 0/6] " Toon Claes
2026-08-07 18:26   ` [PATCH v2 1/6] revision: move bloom keyvec precondition into function Toon Claes
2026-08-07 18:26   ` [PATCH v2 2/6] revision: expose check for paths maybe changed in Bloom filter Toon Claes
2026-08-07 18:26   ` [PATCH v2 3/6] bloom: add helper to check if any key in a vector is present Toon Claes
2026-08-07 18:26   ` [PATCH v2 4/6] revision: add Bloom check that includes parent directories Toon Claes
2026-08-07 18:26   ` [PATCH v2 5/6] last-modified: check pathspec against Bloom filter first Toon Claes
2026-08-07 18:26   ` [PATCH v2 6/6] last-modified: keep per-path Bloom filters for wildcard pathspecs Toon Claes
2026-08-08 17:07     ` Junio C Hamano
2026-08-31 15:18   ` [PATCH v3 0/6] last-modified: use the pathspec's Bloom key to pre-filter commits Toon Claes
2026-08-31 15:18     ` [PATCH v3 1/6] revision: move bloom keyvec precondition into function Toon Claes
2026-08-31 15:18     ` [PATCH v3 2/6] revision: expose check for paths maybe changed in Bloom filter Toon Claes
2026-08-31 15:18     ` [PATCH v3 3/6] bloom: add helper to check if any key in a vector is present Toon Claes
2026-08-31 15:18     ` [PATCH v3 4/6] revision: add Bloom check that includes parent directories Toon Claes
2026-08-31 15:18     ` [PATCH v3 5/6] last-modified: check pathspec against Bloom filter first Toon Claes
2026-08-31 15:18     ` [PATCH v3 6/6] last-modified: keep per-path Bloom filters for wildcard pathspecs Toon Claes
2026-09-01  4:19       ` Junio C Hamano
2026-09-01  9:14         ` Toon Claes
2026-09-01 13:47           ` Junio C Hamano
2026-08-31 21:19     ` [PATCH v3 0/6] last-modified: use the pathspec's Bloom key to pre-filter commits Junio C Hamano
2026-09-01  9:10     ` [PATCH v4 " Toon Claes
2026-09-01  9:10       ` [PATCH v4 1/6] revision: move bloom keyvec precondition into function Toon Claes
2026-09-01  9:10       ` [PATCH v4 2/6] revision: expose check for paths maybe changed in Bloom filter Toon Claes
2026-09-01  9:10       ` [PATCH v4 3/6] bloom: add helper to check if any key in a vector is present Toon Claes
2026-09-10  7:03         ` Patrick Steinhardt
2026-09-01  9:10       ` [PATCH v4 4/6] revision: add Bloom check that includes parent directories Toon Claes
2026-09-10  7:04         ` Patrick Steinhardt
2026-09-01  9:10       ` [PATCH v4 5/6] last-modified: check pathspec against Bloom filter first Toon Claes
2026-09-10  7:04         ` Patrick Steinhardt [this message]
2026-09-01  9:10       ` [PATCH v4 6/6] last-modified: keep per-path Bloom filters for wildcard pathspecs Toon Claes
2026-09-10  7:04         ` Patrick Steinhardt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=aqJWb1kq81A8AJWI@pks.im \
    --to=ps@pks.im \
    --cc=git@vger.kernel.org \
    --cc=gusted@codeberg.org \
    --cc=me@ttaylorr.com \
    --cc=peff@peff.net \
    --cc=toon@iotcl.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox