[PATCH 0/2] Annotated tags pointing to missing but promised blobs

[PATCH 0/2] Annotated tags pointing to missing but promised blobs

[Jonathan Tan]

These are based on jt/partial-clone-fsck-connectivity.

The patches in jt/partial-clone-fsck-connectivity were motivated by bugs
I discovered in partial clones when refs pointed to blobs directly.
While continuing to work on this, I noticed issues with annotated tags -
that is, refs pointing to tags pointing to blobs. Here are some fixes.

Jonathan Tan (2):
  revision: tolerate promised targets of tags
  tag: don't warn if target is missing but promised

 revision.c               |  3 +++
 t/t5616-partial-clone.sh | 44 ++++++++++++++++++++++++++++++++++++++++
 tag.c                    | 13 +++++++++---
 3 files changed, 57 insertions(+), 3 deletions(-)

-- 
2.18.0.203.gfac676dfb9-goog

[PATCH 1/2] revision: tolerate promised targets of tags

[Jonathan Tan]

In handle_commit(), it is fatal for an annotated tag to point to a
non-existent object. --exclude-promisor-objects should relax this rule
and allow non-existent objects that are promisor objects, but this is
not the case. Update handle_commit() to tolerate this situation.

This was observed when cloning from a repository with an annotated tag
pointing to a blob. The test included in this patch demonstrates this
case.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
---
 revision.c               |  3 +++
 t/t5616-partial-clone.sh | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 42 insertions(+)

diff --git a/revision.c b/revision.c
index 1b37da988..95546e6d4 100644
--- a/revision.c
+++ b/revision.c
@@ -248,6 +248,9 @@ static struct commit *handle_commit(struct rev_info *revs,
 		if (!object) {
 			if (revs->ignore_missing_links || (flags & UNINTERESTING))
 				return NULL;
+			if (revs->exclude_promisor_objects &&
+			    is_promisor_object(&tag->tagged->oid))
+				return NULL;
 			die("bad object %s", oid_to_hex(&tag->tagged->oid));
 		}
 		object->flags |= flags;
diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
index 44d8e8017..e8dfeafe7 100755
--- a/t/t5616-partial-clone.sh
+++ b/t/t5616-partial-clone.sh
@@ -216,6 +216,45 @@ test_expect_success 'upon cloning, check that all refs point to objects' '
 	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
 '
 
+test_expect_success 'when partial cloning, tolerate server not sending target of tag' '
+	SERVER="$HTTPD_DOCUMENT_ROOT_PATH/server" &&
+	rm -rf "$SERVER" repo &&
+	test_create_repo "$SERVER" &&
+	test_commit -C "$SERVER" foo &&
+	test_config -C "$SERVER" uploadpack.allowfilter 1 &&
+	test_config -C "$SERVER" uploadpack.allowanysha1inwant 1 &&
+
+	# Create an annotated tag pointing to a blob.
+	BLOB=$(echo blob-contents | git -C "$SERVER" hash-object --stdin -w) &&
+	git -C "$SERVER" tag -m message -a myblob "$BLOB" &&
+
+	# Craft a packfile including the tag, but not the blob it points to.
+	printf "%s\n%s\n--not\n%s\n" \
+		$(git -C "$SERVER" rev-parse HEAD) \
+		$(git -C "$SERVER" rev-parse myblob) \
+		$(git -C "$SERVER" rev-parse myblob^{blob}) |
+		git -C "$SERVER" pack-objects --thin --stdout >incomplete.pack &&
+
+	# Replace the existing packfile with the crafted one. The protocol
+	# requires that the packfile be sent in sideband 1, hence the extra
+	# \x01 byte at the beginning.
+	printf "1,/packfile/!c %04x\\\\x01%s0000" \
+		"$(($(wc -c <incomplete.pack) + 5))" \
+		"$(sed_escape <incomplete.pack)" \
+		>"$HTTPD_ROOT_PATH/one-time-sed" &&
+
+	# Use protocol v2 because the sed command looks for the "packfile"
+	# section header.
+	test_config -C "$SERVER" protocol.version 2 &&
+
+	# Exercise to make sure it works.
+	git -c protocol.version=2 clone \
+		--filter=blob:none $HTTPD_URL/one_time_sed/server repo &&
+
+	# Ensure that the one-time-sed script was used.
+	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
+'
+
 stop_httpd
 
 test_done
-- 
2.18.0.203.gfac676dfb9-goog

[PATCH 2/2] tag: don't warn if target is missing but promised

[Jonathan Tan]

deref_tag() prints a warning if the object that a tag refers to does not
exist. However, when a partial clone has an annotated tag from its
promisor remote, but not the object that it refers to, printing a
warning on such a tag is incorrect.

This occurs, for example, when the checkout that happens after a partial
clone causes some objects to be fetched - and as part of the fetch, all
local refs are read. The test included in this patch demonstrates this
situation.

Therefore, do not print a warning in this case.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
---
 t/t5616-partial-clone.sh |  9 +++++++--
 tag.c                    | 13 ++++++++++---
 2 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh
index e8dfeafe7..bbbe7537d 100755
--- a/t/t5616-partial-clone.sh
+++ b/t/t5616-partial-clone.sh
@@ -229,9 +229,13 @@ test_expect_success 'when partial cloning, tolerate server not sending target of
 	git -C "$SERVER" tag -m message -a myblob "$BLOB" &&
 
 	# Craft a packfile including the tag, but not the blob it points to.
-	printf "%s\n%s\n--not\n%s\n" \
+	# Also, omit objects referenced from HEAD in order to force a second
+	# fetch (to fetch missing objects) upon the automatic checkout that
+	# happens after a clone.
+	printf "%s\n%s\n--not\n%s\n%s\n" \
 		$(git -C "$SERVER" rev-parse HEAD) \
 		$(git -C "$SERVER" rev-parse myblob) \
+		$(git -C "$SERVER" rev-parse HEAD^{tree}) \
 		$(git -C "$SERVER" rev-parse myblob^{blob}) |
 		git -C "$SERVER" pack-objects --thin --stdout >incomplete.pack &&
 
@@ -249,7 +253,8 @@ test_expect_success 'when partial cloning, tolerate server not sending target of
 
 	# Exercise to make sure it works.
 	git -c protocol.version=2 clone \
-		--filter=blob:none $HTTPD_URL/one_time_sed/server repo &&
+		--filter=blob:none $HTTPD_URL/one_time_sed/server repo 2> err &&
+	! grep "missing object referenced by" err &&
 
 	# Ensure that the one-time-sed script was used.
 	! test -e "$HTTPD_ROOT_PATH/one-time-sed"
diff --git a/tag.c b/tag.c
index 3d37c1bd2..1110e3643 100644
--- a/tag.c
+++ b/tag.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "blob.h"
 #include "gpg-interface.h"
+#include "packfile.h"
 
 const char *tag_type = "tag";
 
@@ -64,12 +65,18 @@ int gpg_verify_tag(const struct object_id *oid, const char *name_to_report,
 
 struct object *deref_tag(struct object *o, const char *warn, int warnlen)
 {
+	struct object_id *last_oid = NULL;
 	while (o && o->type == OBJ_TAG)
-		if (((struct tag *)o)->tagged)
-			o = parse_object(&((struct tag *)o)->tagged->oid);
-		else
+		if (((struct tag *)o)->tagged) {
+			last_oid = &((struct tag *)o)->tagged->oid;
+			o = parse_object(last_oid);
+		} else {
+			last_oid = NULL;
 			o = NULL;
+		}
 	if (!o && warn) {
+		if (last_oid && is_promisor_object(last_oid))
+			return NULL;
 		if (!warnlen)
 			warnlen = strlen(warn);
 		error("missing object referenced by '%.*s'", warnlen, warn);
-- 
2.18.0.203.gfac676dfb9-goog