[PATCH] rev-list: skip bitmap traversal for --left-right

[PATCH] rev-list: skip bitmap traversal for --left-right

[Jeff King]

Running:

  git rev-list --left-right --use-bitmap-index one...two

will produce output without any left-right markers, since the bitmap
traversal returns only a single set of reachable commits. Instead we
should refuse to use bitmaps here and produce the correct output using a
traditional traversal.

This is probably not the only remaining option that misbehaves with
bitmaps, but it's particularly egregious in that it feels like it
_could_ work. Doing two separate traversals for the left/right sides and
then taking the symmetric set differences should yield the correct
answer, but our traversal code doesn't know how to do that.

It's not clear if naively doing two separate traversals would always be
a performance win. A traditional traversal only needs to walk down to
the merge base, but bitmaps always fill out the full reachability set.
So depending on your bitmap coverage, we could end up walking old bits
of history twice to fill out the same uninteresting bits on both sides.
We'd also of course end up with a very large --boundary set, if the user
asked for that.

So this might or might not be something worth implementing later. But
for now, let's make sure we don't produce the wrong answer if somebody
tries it.

The test covers this, but also the same thing with "--count" (which is
what I originally tried in a real-world case). Ironically the
try_bitmap_count() code already realizes that "--left-right" won't work
there. But that just causes us to fall back to the regular bitmap
traversal code, which itself doesn't handle counting (we produce a list
of objects rather than a count).

Signed-off-by: Jeff King <peff@peff.net>
---
 builtin/rev-list.c      |  7 +++++++
 t/t5310-pack-bitmaps.sh | 12 ++++++++++++
 2 files changed, 19 insertions(+)

diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index f62bcbf2b1..3078787115 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -485,6 +485,13 @@ static int try_bitmap_traversal(struct rev_info *revs,
 	if (revs->max_count >= 0)
 		return -1;
 
+	/*
+	 * We can't know which commits were left/right in a single traversal,
+	 * and we don't yet know how to traverse them separately.
+	 */
+	if (revs->left_right)
+		return -1;
+
 	bitmap_git = prepare_bitmap_walk(revs, filter_provided_objects);
 	if (!bitmap_git)
 		return -1;
diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh
index 7044c7d7c6..6bcbea64cc 100755
--- a/t/t5310-pack-bitmaps.sh
+++ b/t/t5310-pack-bitmaps.sh
@@ -503,6 +503,18 @@ test_expect_success 'boundary-based traversal is used when requested' '
 	done
 '
 
+test_expect_success 'left-right not confused by bitmap index' '
+	git rev-list --left-right other...HEAD >expect &&
+	git rev-list --use-bitmap-index --left-right other...HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'left-right count not confused by bitmap-index' '
+	git rev-list --left-right --count other...HEAD >expect &&
+	git rev-list --use-bitmap-index --left-right --count other...HEAD >actual &&
+	test_cmp expect actual
+'
+
 test_bitmap_cases "pack.writeBitmapLookupTable"
 
 test_expect_success 'verify writing bitmap lookup table when enabled' '
-- 
2.47.0.441.g1a09955689

Re: [PATCH] rev-list: skip bitmap traversal for --left-right

[Taylor Blau]

On Fri, Nov 01, 2024 at 08:16:06AM -0400, Jeff King wrote:
>  builtin/rev-list.c      |  7 +++++++
>  t/t5310-pack-bitmaps.sh | 12 ++++++++++++
>  2 files changed, 19 insertions(+)

Nice find, and well explained (not just why it doesn't work today but
could in the future, but why making it work in the future with bitmaps
does not necessarily a clear performance improvement).

Probably you and I should think more about other rev-list options that
*don't* work with --use-bitmap-index. I share your feeling there that
there probably exist such options which silently do nothing (or the
wrong thing) in the presence of '--use-bitmap-index'.

Let's merge this one down.

Thanks,
Taylor

Re: [PATCH] rev-list: skip bitmap traversal for --left-right

[Jeff King]

On Fri, Nov 01, 2024 at 10:40:41AM -0400, Taylor Blau wrote:

> Nice find, and well explained (not just why it doesn't work today but
> could in the future, but why making it work in the future with bitmaps
> does not necessarily a clear performance improvement).
> 
> Probably you and I should think more about other rev-list options that
> *don't* work with --use-bitmap-index. I share your feeling there that
> there probably exist such options which silently do nothing (or the
> wrong thing) in the presence of '--use-bitmap-index'.

I'm pretty sure --cherry-pick and --cherry-mark are examples, but I
suspect it's the tip of the iceberg. I don't know if it's worth spending
much effort on this. It's certainly a wart, but there's a certain amount
of "if it hurts, don't do it". The --left-right one bugged me so much
just because I thought it _would_ work. ;)

-Peff

Re: [PATCH] rev-list: skip bitmap traversal for --left-right

[Jeff King]

On Mon, Nov 04, 2024 at 12:41:27PM -0500, Jeff King wrote:

> On Fri, Nov 01, 2024 at 10:40:41AM -0400, Taylor Blau wrote:
> 
> > Nice find, and well explained (not just why it doesn't work today but
> > could in the future, but why making it work in the future with bitmaps
> > does not necessarily a clear performance improvement).
> > 
> > Probably you and I should think more about other rev-list options that
> > *don't* work with --use-bitmap-index. I share your feeling there that
> > there probably exist such options which silently do nothing (or the
> > wrong thing) in the presence of '--use-bitmap-index'.
> 
> I'm pretty sure --cherry-pick and --cherry-mark are examples, but I
> suspect it's the tip of the iceberg. I don't know if it's worth spending
> much effort on this. It's certainly a wart, but there's a certain amount
> of "if it hurts, don't do it". The --left-right one bugged me so much
> just because I thought it _would_ work. ;)

Actually, I wonder if this could be an interesting #leftoverbits or
intern mini-project:

  1. Look at the set of rev-list options and pick one.

  2. Repack a repo (say, git.git) with reachability bitmaps using "git
     repack -adb".

  3. Compare output of "rev-list --your-option" with and without
     --use-bitmap-index. If the outputs do not reasonably match, add
     code to try_bitmap_traversal() to forbid it and bail to the
     non-bitmap path. Maybe also other try_bitmap_*() functions, as
     well.

It's a little more involved than other miniprojects just because there
may be some judgement needed for "reasonably match". Being byte-for-byte
identical is good, but in some cases the output will be different (e.g.,
bitmap output is in a different order, lacks path annotations, etc). So
it would help in step (1) if you understand what the option is supposed
to do and how you'd expect it to work with bitmaps. ;)

-Peff

Re: [PATCH] rev-list: skip bitmap traversal for --left-right

[Taylor Blau]

On Mon, Nov 04, 2024 at 12:46:28PM -0500, Jeff King wrote:
> On Mon, Nov 04, 2024 at 12:41:27PM -0500, Jeff King wrote:
>
> > On Fri, Nov 01, 2024 at 10:40:41AM -0400, Taylor Blau wrote:
> >
> > > Nice find, and well explained (not just why it doesn't work today but
> > > could in the future, but why making it work in the future with bitmaps
> > > does not necessarily a clear performance improvement).
> > >
> > > Probably you and I should think more about other rev-list options that
> > > *don't* work with --use-bitmap-index. I share your feeling there that
> > > there probably exist such options which silently do nothing (or the
> > > wrong thing) in the presence of '--use-bitmap-index'.
> >
> > I'm pretty sure --cherry-pick and --cherry-mark are examples, but I
> > suspect it's the tip of the iceberg. I don't know if it's worth spending
> > much effort on this. It's certainly a wart, but there's a certain amount
> > of "if it hurts, don't do it". The --left-right one bugged me so much
> > just because I thought it _would_ work. ;)
>
> Actually, I wonder if this could be an interesting #leftoverbits or
> intern mini-project:

I agree; that would be a good project for interns. Great idea!

Thanks,
Taylor

Re: [PATCH] rev-list: skip bitmap traversal for --left-right

[Ayush Chandekar]

Hey,
I found this issue quite interesting. Would you like to discuss a bit more,
maybe about how to implement it?

I am looking forward to solving this.

Regards,
Ayush