Git development
 help / color / mirror / Atom feed
From: Christian Couder <christian.couder@gmail.com>
To: git@vger.kernel.org
Cc: Junio C Hamano <gitster@pobox.com>,
	Patrick Steinhardt <ps@pks.im>, Matt Cote <mcote@nvidia.com>,
	Christian Couder <christian.couder@gmail.com>
Subject: [PATCH v2 7/8] fetch-pack: ensure fetch sends a promisor-remote reply
Date: Fri, 20 Nov 2026 17:48:20 +0100	[thread overview]
Message-ID: <20261120164812.707946-17-christian.couder@gmail.com> (raw)
In-Reply-To: <20261120164812.707946-10-christian.couder@gmail.com>

In protocol v2, when the client accepts promisor remotes, the
`promisor-remote=...` reply is only sent with the `ls-refs` and
`bundle-uri` protocol commands, by get_remote_refs() and
get_remote_bundle_uri() in "connect.c".

The request for the `fetch` protocol command is built by
send_fetch_request() in "fetch-pack.c" which doesn't send that reply.
do_fetch_pack_v2() calls promisor_remote_reply(), but only to mark the
accepted remotes, as e15a6b2f8b (promisor-remote: change
promisor_remote_reply()'s signature, 2026-02-16) assumed that
fetch-pack would not need to send a reply.

With file:// and ssh:// URLs on a Git server, one upload-pack process
handles both `ls-refs` and `fetch`. The acceptance recorded during
`ls-refs` by promisor_remote_receive() in "serve.c" is still there at
`fetch` time, so the clone succeeds. But it fails when each request is
handled on its own, for example with HTTP where different upload-pack
processes are used.

It also fails when only object IDs are fetched, as no `ls-refs`
command is then sent. In these cases, the server doesn't know that the
client accepted a promisor remote, so it doesn't allow promisor
objects to be missing from the pack it sends, and it fails with "could
not fetch <oid> from promisor remote".

Let's fix that by building the list of capabilities for the `fetch`
requests once in do_fetch_pack_v2(), using add_server_options() and
add_promisor_remote_reply(), and by passing that list to
send_fetch_request(). This way the `promisor-remote=...` reply is
always properly sent with the `fetch` requests, and the accepted
remotes are still marked at the same point as before.

The `fetch` requests sent by negotiate_using_fetch() don't ask for a
pack, so they don't need that reply.

Let's also add tests in t5710 to check that the reply is sent with the
`fetch` request, and that fetching only an object ID works. For HTTP,
let's add a new t5711 test script, copying the setup from t5710.

Reported-by: Matt Cote <mcote@nvidia.com>
Signed-off-by: Christian Couder <christian.couder@gmail.com>
---
 fetch-pack.c                               |  18 +--
 t/meson.build                              |   1 +
 t/t5710-promisor-remote-capability.sh      |  40 +++++++
 t/t5711-promisor-remote-capability-http.sh | 126 +++++++++++++++++++++
 4 files changed, 176 insertions(+), 9 deletions(-)
 create mode 100755 t/t5711-promisor-remote-capability-http.sh

diff --git a/fetch-pack.c b/fetch-pack.c
index 176d625936..3d6d3e33a2 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -1380,16 +1380,15 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,
 			      const struct ref *wants, struct oidset *common,
 			      int *haves_to_send, int *in_vain,
 			      int sideband_all, int seen_ack,
-			      struct oidset *negotiation_include_oids)
+			      struct oidset *negotiation_include_oids,
+			      const struct strvec *capabilities)
 {
 	int haves_added;
 	int done_sent = 0;
 	struct strbuf req_buf = STRBUF_INIT;
-	struct strvec capabilities = STRVEC_INIT;
 
-	add_server_options(&capabilities, args->server_options);
 	write_command_and_capabilities(&req_buf, "fetch",
-				       &capabilities, NULL);
+				       capabilities, NULL);
 
 	if (args->use_thin_pack)
 		packet_buf_write(&req_buf, "thin-pack");
@@ -1454,7 +1453,6 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out,
 	if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0)
 		die_errno(_("unable to write request to remote"));
 
-	strvec_clear(&capabilities);
 	strbuf_release(&req_buf);
 	return done_sent;
 }
@@ -1698,12 +1696,12 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	struct string_list packfile_uris = STRING_LIST_INIT_DUP;
 	int i;
 	struct strvec index_pack_args = STRVEC_INIT;
-	const char *promisor_remote_config;
+	struct strvec capabilities = STRVEC_INIT;
 
 	fsck_options_init(&fsck_options, the_repository, FSCK_OPTIONS_MISSING_GITMODULES);
 
-	if (server_feature_v2("promisor-remote", &promisor_remote_config))
-		promisor_remote_reply(promisor_remote_config, NULL);
+	add_server_options(&capabilities, args->server_options);
+	add_promisor_remote_reply(&capabilities);
 
 	if (args->filter_options.choice == LOFC_AUTO) {
 		struct strbuf errbuf = STRBUF_INIT;
@@ -1781,7 +1779,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 					       &haves_to_send, &in_vain,
 					       reader.use_sideband,
 					       seen_ack,
-					       &negotiation_include_oids)) {
+					       &negotiation_include_oids,
+					       &capabilities)) {
 				trace2_region_leave_printf("negotiation_v2", "round",
 							   the_repository, "%d",
 							   negotiation_round);
@@ -1924,6 +1923,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 
 	string_list_clear(&packfile_uris, 0);
 	strvec_clear(&index_pack_args);
+	strvec_clear(&capabilities);
 
 	if (fsck_finish(&fsck_options))
 		die("fsck failed");
diff --git a/t/meson.build b/t/meson.build
index 1cf1d95f70..84644703f1 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -766,6 +766,7 @@ integration_tests = [
   't5704-protocol-violations.sh',
   't5705-session-id-in-capabilities.sh',
   't5710-promisor-remote-capability.sh',
+  't5711-promisor-remote-capability-http.sh',
   't5730-protocol-v2-bundle-uri-file.sh',
   't5731-protocol-v2-bundle-uri-git.sh',
   't5732-protocol-v2-bundle-uri-http.sh',
diff --git a/t/t5710-promisor-remote-capability.sh b/t/t5710-promisor-remote-capability.sh
index 463e9e00b0..e5d6f9d1c6 100755
--- a/t/t5710-promisor-remote-capability.sh
+++ b/t/t5710-promisor-remote-capability.sh
@@ -999,6 +999,46 @@ test_expect_success "clone and fetch with --filter=auto" '
 	check_missing_objects server 1 "$oid"
 '
 
+test_expect_success "clone sends the promisor-remote reply with fetch" '
+	git -C server config promisor.advertise true &&
+	test_when_finished "rm -rf client trace" &&
+
+	GIT_TRACE_PACKET="$(pwd)/trace" git clone \
+		-c remote.lop.promisor=true \
+		-c remote.lop.url="$TRASH_DIRECTORY_URL/lop" \
+		-c promisor.acceptfromserver=All \
+		--no-local --filter="blob:limit=5k" server client &&
+
+	# Check that the reply is part of the fetch request
+	sed -n "/clone> command=fetch/,/clone> 0001/p" trace >request &&
+	test_grep "clone> promisor-remote=lop" request &&
+
+	# Check that the largest object is still missing on the server
+	check_missing_objects server 1 "$oid"
+'
+
+test_expect_success "fetch by object ID sends the promisor-remote reply" '
+	git -C server config promisor.advertise true &&
+	test_when_finished "rm -rf client trace" &&
+
+	git init client &&
+	git -C client remote add origin "$TRASH_DIRECTORY_URL/server" &&
+	git -C client config remote.lop.promisor true &&
+	git -C client config remote.lop.url "$TRASH_DIRECTORY_URL/lop" &&
+	git -C client config promisor.acceptfromserver All &&
+
+	# Fetching only an object ID skips the "ls-refs" command, so the
+	# reply can only be sent with "fetch"
+	head=$(git -C server rev-parse HEAD) &&
+	GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch --no-tags \
+		--filter="blob:limit=5k" origin "$head" &&
+	test_grep ! "fetch> command=ls-refs" trace &&
+	test_grep "fetch> promisor-remote=lop" trace &&
+
+	# Check that the largest object is still missing on the server
+	check_missing_objects server 1 "$oid"
+'
+
 test_expect_success "clone with promisor.advertise set to 'true' but don't delete the client" '
 	git -C server config promisor.advertise true &&
 
diff --git a/t/t5711-promisor-remote-capability-http.sh b/t/t5711-promisor-remote-capability-http.sh
new file mode 100755
index 0000000000..c08cca3e51
--- /dev/null
+++ b/t/t5711-promisor-remote-capability-http.sh
@@ -0,0 +1,126 @@
+#!/bin/sh
+
+test_description='handling of promisor remote advertisement over HTTP'
+
+. ./test-lib.sh
+
+if ! test_have_prereq PERL_TEST_HELPERS
+then
+	skip_all='skipping promisor remote capabilities tests; Perl not available'
+	test_done
+fi
+
+. "$TEST_DIRECTORY"/lib-httpd.sh
+start_httpd
+
+GIT_TEST_MULTI_PACK_INDEX=0
+GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0
+
+# The setup and helpers below are copied from t5710, except that the
+# server repository is in the HTTP document root.
+
+SERVER="$HTTPD_DOCUMENT_ROOT_PATH/server.git"
+
+check_missing_objects () {
+	git -C "$1" rev-list --objects --all --missing=print > all.txt &&
+	perl -ne 'print if s/^[?]//' all.txt >missing.txt &&
+	test_line_count = "$2" missing.txt &&
+	test "$3" = "$(cat missing.txt)"
+}
+
+initialize_server () {
+	# Repack everything first
+	git -C "$SERVER" -c repack.writebitmaps=false repack -a -d &&
+
+	# Repack without the largest object and create a promisor pack on server
+	git -C "$SERVER" -c repack.writebitmaps=false repack -a -d \
+	    --filter=blob:limit=5k --filter-to="$(pwd)/pack" &&
+	promisor_file=$(ls "$SERVER"/objects/pack/*.pack | sed "s/\.pack/.promisor/") &&
+	>"$promisor_file" &&
+
+	# Check objects missing on the server
+	check_missing_objects "$SERVER" 1 "$1"
+}
+
+copy_to_lop () {
+	oid_path="$(test_oid_to_path $1)" &&
+	path="$SERVER/objects/$oid_path" &&
+	path2="lop/objects/$oid_path" &&
+	mkdir -p $(dirname "$path2") &&
+	cp "$path" "$path2"
+}
+
+# On Windows, `pwd` returns a path like 'D:/foo/bar'. Prepend '/' to turn
+# it into '/D:/foo/bar', which is what git expects in file:// URLs on Windows.
+# On Unix, the path already starts with '/', so this is a no-op.
+pwd_path=$(pwd)
+case "$pwd_path" in
+[a-zA-Z]:*) pwd_path="/$pwd_path" ;;
+esac
+
+TRASH_DIRECTORY_URL="file://$pwd_path"
+
+test_expect_success 'setup: create "template" repository' '
+	git init template &&
+	test_commit -C template 1 &&
+	test_commit -C template 2 &&
+	test_commit -C template 3 &&
+	test-tool genrandom foo 10k >template/foo &&
+	git -C template add foo &&
+	git -C template commit -m foo
+'
+
+# A bare repo will act as a server repo with unpacked objects.
+test_expect_success 'setup: create bare "server" repository' '
+	git clone --bare --no-local template "$SERVER" &&
+	mv "$SERVER"/objects/pack/pack-* . &&
+	packfile=$(ls pack-*.pack) &&
+	git -C "$SERVER" unpack-objects --strict <"$packfile"
+'
+
+test_expect_success "setup for testing promisor remote advertisement" '
+	# Create another bare repo called "lop" (for Large Object Promisor)
+	git init --bare lop &&
+
+	# Copy the largest object from server to lop
+	obj="HEAD:foo" &&
+	oid="$(git -C "$SERVER" rev-parse $obj)" &&
+	copy_to_lop "$oid" &&
+
+	initialize_server "$oid" &&
+
+	# Configure lop as promisor remote for server
+	git -C "$SERVER" remote add lop "$TRASH_DIRECTORY_URL/lop" &&
+	git -C "$SERVER" config remote.lop.promisor true &&
+
+	git -C lop config uploadpack.allowFilter true &&
+	git -C lop config uploadpack.allowAnySHA1InWant true &&
+	git -C "$SERVER" config uploadpack.allowFilter true &&
+	git -C "$SERVER" config uploadpack.allowAnySHA1InWant true &&
+	git -C "$SERVER" config promisor.advertise true
+'
+
+# Over HTTP, each request is handled by a separate upload-pack process,
+# so the server knows that the client accepted "lop" only if the reply
+# is sent with the "fetch" request itself.
+test_expect_success "clone over HTTP with an accepted promisor remote" '
+	test_when_finished "rm -rf client trace" &&
+
+	GIT_TRACE_PACKET="$(pwd)/trace" git -c protocol.version=2 clone \
+		-c remote.lop.promisor=true \
+		-c remote.lop.url="$TRASH_DIRECTORY_URL/lop" \
+		-c promisor.acceptfromserver=All \
+		--filter="blob:limit=5k" "$HTTPD_URL/smart/server.git" client &&
+
+	# Check that the reply is part of the fetch request
+	sed -n "/clone> command=fetch/,/clone> 0001/p" trace >request &&
+	test_grep "clone> promisor-remote=lop" request &&
+
+	# Check that the largest object was lazily fetched from lop
+	test_cmp template/foo client/foo &&
+
+	# Check that the largest object is still missing on the server
+	check_missing_objects "$SERVER" 1 "$oid"
+'
+
+test_done
-- 
2.56.0


  parent reply	other threads:[~2026-11-20 16:48 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 19:58 [PATCH 0/3] Fix promisor-remote reply missing from fetch Christian Couder
2026-11-12 19:59 ` [PATCH 1/3] connect: let write_command_and_capabilities() send a promisor-remote reply Christian Couder
2026-11-12 22:18   ` Junio C Hamano
2026-11-12 22:34     ` Junio C Hamano
2026-11-13 13:10       ` Christian Couder
2026-11-13 19:11         ` Junio C Hamano
2026-11-20 16:57           ` Christian Couder
2026-11-12 19:59 ` [PATCH 2/3] fetch-pack: ensure fetch sends " Christian Couder
2026-11-12 19:59 ` [PATCH 3/3] doc: gitprotocol-v2: send the promisor-remote reply with fetch Christian Couder
2026-11-20 16:48 ` [PATCH v2 0/8] Fix promisor-remote reply missing from fetch Christian Couder
2026-11-20 16:48   ` [PATCH v2 1/8] connect: pass capability lines to write_command_and_capabilities() Christian Couder
2026-11-20 16:48   ` [PATCH v2 2/8] connect: send session-id with ls-refs and bundle-uri Christian Couder
2026-11-20 16:48   ` [PATCH v2 3/8] connect: pass a reader to write_command_and_capabilities() Christian Couder
2026-11-20 16:48   ` [PATCH v2 4/8] connect: buffer the ls-refs and bundle-uri requests Christian Couder
2026-11-20 16:48   ` [PATCH v2 5/8] connect: use write_command_and_capabilities() in get_remote_refs() Christian Couder
2026-11-20 16:48   ` [PATCH v2 6/8] connect: use write_command_and_capabilities() in get_remote_bundle_uri() Christian Couder
2026-11-20 16:48   ` Christian Couder [this message]
2026-11-20 16:48   ` [PATCH v2 8/8] doc: gitprotocol-v2: send the promisor-remote reply with fetch Christian Couder

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261120164812.707946-17-christian.couder@gmail.com \
    --to=christian.couder@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=mcote@nvidia.com \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox