[PATCH v2] upload-pack: ignore 'shallow' lines with unknown obj-ids

[PATCH v2] upload-pack: ignore 'shallow' lines with unknown obj-ids

[Michael Heemskerk]

When the client sends a 'shallow' line for an object that the server does
not have, the server currently dies with the error: "did not find object
for shallow <obj-id>".  The client may have truncated the history at
the commit by fetching shallowly from a different server, or the commit
may have been garbage collected by the server. In either case, this
unknown commit is not relevant for calculating the pack that is to be
sent and can be safely ignored, and it is not used when recomputing where
the updated history of the client is cauterised.

The documentation in technical/pack-protocol.txt has been updated to
remove the restriction that "Clients MUST NOT mention an obj-id which it
does not know exists on the server". This requirement is not realistic
because clients cannot know whether an object has been garbage collected
by the server.

Signed-off-by: Michael Heemskerk <mheemskerk@atlassian.com>
Reviewed-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/technical/pack-protocol.txt |  3 +--
 t/t5500-fetch-pack.sh                     | 13 +++++++++++++
 upload-pack.c                             |  2 +-
 3 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index f1a51ed..b898e97 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -228,8 +228,7 @@ obtained through ref discovery.
 The client MUST write all obj-ids which it only has shallow copies
 of (meaning that it does not have the parents of a commit) as
 'shallow' lines so that the server is aware of the limitations of
-the client's history. Clients MUST NOT mention an obj-id which
-it does not know exists on the server.
+the client's history.
 
 The client now sends the maximum commit history depth it wants for
 this transaction, which is the number of commits it wants from the
diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index d574085..3f0ca10 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -373,6 +373,19 @@ test_expect_success 'clone shallow with packed refs' '
 	test_cmp count8.expected count8.actual
 '
 
+test_expect_success 'fetch in shallow repo unreachable shallow objects' '
+	(
+		git clone --bare --branch B --single-branch "file://$(pwd)/." no-reflog &&
+		git clone --depth 1 "file://$(pwd)/no-reflog" shallow9 &&
+		cd no-reflog &&
+		git tag -d TAGB1 TAGB2 &&
+		git update-ref refs/heads/B B~~ &&
+		git gc --prune=now &&
+		cd ../shallow9 &&
+		git fetch origin
+	)
+'
+
 test_expect_success 'setup tests for the --stdin parameter' '
 	for head in C D E F
 	do
diff --git a/upload-pack.c b/upload-pack.c
index bfa6279..127e59a 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -592,7 +592,7 @@ static void receive_needs(void)
 				die("invalid shallow line: %s", line);
 			object = parse_object(sha1);
 			if (!object)
-				die("did not find object for %s", line);
+				continue;
 			if (object->type != OBJ_COMMIT)
 				die("invalid shallow object %s", sha1_to_hex(sha1));
 			if (!(object->flags & CLIENT_SHALLOW)) {
-- 
1.8.0.2

Re: [PATCH v2] upload-pack: ignore 'shallow' lines with unknown obj-ids

[Junio C Hamano]

Michael Heemskerk <mheemskerk@atlassian.com> writes:

> When the client sends a 'shallow' line for an object that the server does
> not have, the server currently dies with the error: "did not find object
> ...
>  Documentation/technical/pack-protocol.txt |  3 +--
>  t/t5500-fetch-pack.sh                     | 13 +++++++++++++
>  upload-pack.c                             |  2 +-
>  3 files changed, 15 insertions(+), 3 deletions(-)

Thanks.

The previous one (without the test) is already in 'next' so I'll
pick only the test part and queue it as an update to the series.

> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
> index d574085..3f0ca10 100755
> --- a/t/t5500-fetch-pack.sh
> +++ b/t/t5500-fetch-pack.sh
> @@ -373,6 +373,19 @@ test_expect_success 'clone shallow with packed refs' '
>  	test_cmp count8.expected count8.actual
>  '
>  
> +test_expect_success 'fetch in shallow repo unreachable shallow objects' '
> +	(
> +		git clone --bare --branch B --single-branch "file://$(pwd)/." no-reflog &&
> +		git clone --depth 1 "file://$(pwd)/no-reflog" shallow9 &&
> +		cd no-reflog &&
> +		git tag -d TAGB1 TAGB2 &&
> +		git update-ref refs/heads/B B~~ &&
> +		git gc --prune=now &&
> +		cd ../shallow9 &&
> +		git fetch origin
> +	)
> +'
> +
>  test_expect_success 'setup tests for the --stdin parameter' '
>  	for head in C D E F
>  	do

Re: [PATCH v2] upload-pack: ignore 'shallow' lines with unknown obj-ids

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Michael Heemskerk <mheemskerk@atlassian.com> writes:
>
>> When the client sends a 'shallow' line for an object that the server does
>> not have, the server currently dies with the error: "did not find object
>> ...
>>  Documentation/technical/pack-protocol.txt |  3 +--
>>  t/t5500-fetch-pack.sh                     | 13 +++++++++++++
>>  upload-pack.c                             |  2 +-
>>  3 files changed, 15 insertions(+), 3 deletions(-)
>
> Thanks.
>
> The previous one (without the test) is already in 'next' so I'll
> pick only the test part and queue it as an update to the series.
>
>> diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
>> index d574085..3f0ca10 100755
>> --- a/t/t5500-fetch-pack.sh
>> +++ b/t/t5500-fetch-pack.sh
>> @@ -373,6 +373,19 @@ test_expect_success 'clone shallow with packed refs' '
>>  	test_cmp count8.expected count8.actual
>>  '
>>  
>> +test_expect_success 'fetch in shallow repo unreachable shallow objects' '
>> +	(
>> +		git clone --bare --branch B --single-branch "file://$(pwd)/." no-reflog &&
>> +		git clone --depth 1 "file://$(pwd)/no-reflog" shallow9 &&
>> +		cd no-reflog &&
>> +		git tag -d TAGB1 TAGB2 &&
>> +		git update-ref refs/heads/B B~~ &&
>> +		git gc --prune=now &&
>> +		cd ../shallow9 &&
>> +		git fetch origin
>> +	)
>> +'

It would also be good to make sure that shallow9 does not lose the
shallow history boundary that the origin does not know about after
this fetch, so I'll further squash this in.

 t/t5500-fetch-pack.sh | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 3f0ca10..6133d9e 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -382,7 +382,8 @@ test_expect_success 'fetch in shallow repo unreachable shallow objects' '
 		git update-ref refs/heads/B B~~ &&
 		git gc --prune=now &&
 		cd ../shallow9 &&
-		git fetch origin
+		git fetch origin &&
+		git fsck --no-dangling
 	)
 '