git clone --depth: shallow clone problems

git clone --depth: shallow clone problems

[Michal Pomorski]

tl: skip to the second paragraph

So here is what I just experienced:
We had an emergency error in an application at work and as the
responsible developer was unavailable, I was asked to check it out and
look into it.
We are a small branch of a bigger company and our connection to the
company's source servers is really slow, so just to quickly start it
up, I decided to take a shallow clone (that's what it is for, right?).
After a while we realized, the clone I have made was not sufficient
and was missing some newest work done on the project.
I did "git fetch --unshallow" which finished surprisingly quickly, and
it did not bring any newer commits.
Unaware of the fine print hiding in the documentation of git clone, I
blamed the repo (and in extension the person who provided me the
address to it). After coming to a realization, a process which cost me
time and embarrassment, I understood what is supposed to be the
"correct" behaviour:
"--single-branch
Clone only the history leading to the tip of a single branch, either
specified by the --branch option or the primary branch remote's HEAD
points at. When creating a shallow clone with the --depth option, this
is the default, unless --no-single-branch is given to fetch the
histories near the tips of all branches."
Of course, the new commits were put on a custom branch, and I knew
that all the time, but I expected the branch to show up eventually, at
least after git fetch --unshallow.


I hope you can see the faults in the usability of the commands I was exposed to:
1) git clone --depth  should:
   -warn about only fetching the current HEAD (name it by a real name
if applicable)
and/or
   -require the --branch option so that it is not left to chance
(current HEAD could be anything; is it really meaningful to talk about
the current HEAD on a server?)
and/or
   -make the --no-single-branch the default...
and maybe
   -have the option to clone the most recent commits.

2) git fetch --unshallow should convert the clone into an actual
equivalent of a normal, not shallow clone.

3) The documentation should be improved. The behaviour of --shallow is
described partly in the description of --no-single-branch. This is the
documentation equivalent of the infamous "come from" control flow
structure.

Regards,
Michal Pomorski

Re: git clone --depth: shallow clone problems

[Duy Nguyen]

On Thu, Apr 23, 2015 at 04:53:04PM +0200, Michal Pomorski wrote:
> tl: skip to the second paragraph
> 
> So here is what I just experienced:
> We had an emergency error in an application at work and as the
> responsible developer was unavailable, I was asked to check it out and
> look into it.
> We are a small branch of a bigger company and our connection to the
> company's source servers is really slow, so just to quickly start it
> up, I decided to take a shallow clone (that's what it is for, right?).
> After a while we realized, the clone I have made was not sufficient
> and was missing some newest work done on the project.
> I did "git fetch --unshallow" which finished surprisingly quickly, and
> it did not bring any newer commits.
> Unaware of the fine print hiding in the documentation of git clone, I
> blamed the repo (and in extension the person who provided me the
> address to it). After coming to a realization, a process which cost me
> time and embarrassment, I understood what is supposed to be the
> "correct" behaviour:
> "--single-branch
> Clone only the history leading to the tip of a single branch, either
> specified by the --branch option or the primary branch remote's HEAD
> points at. When creating a shallow clone with the --depth option, this
> is the default, unless --no-single-branch is given to fetch the
> histories near the tips of all branches."
> Of course, the new commits were put on a custom branch, and I knew
> that all the time, but I expected the branch to show up eventually, at
> least after git fetch --unshallow.
> 
>
> I hope you can see the faults in the usability of the commands I was
> exposed to:
> 1) git clone --depth  should:
>    -warn about only fetching the current HEAD (name it by a real name
> if applicable)
> and/or
>    -require the --branch option so that it is not left to chance
> (current HEAD could be anything; is it really meaningful to talk about
> the current HEAD on a server?)
> and/or
>    -make the --no-single-branch the default...
> and maybe
>    -have the option to clone the most recent commits.

--single-branch being the default was because we (at least I, stilll)
believed it was the common case, so I don't think we should revert
it. But I can see --unshallow documentation is misleading. How about
this? I think it's better than what we have.

-- 8< --
Subject: [PATCH] fetch-options.txt: clarify what --unshallow does

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/fetch-options.txt | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt
index 45583d8..63d3452 100644
--- a/Documentation/fetch-options.txt
+++ b/Documentation/fetch-options.txt
@@ -14,12 +14,17 @@
 	branch history. Tags for the deepened commits are not fetched.
 
 --unshallow::
-	If the source repository is complete, convert a shallow
-	repository to a complete one, removing all the limitations
+	If the source repository is complete, convert all shallow
+	refs to complete ones, removing all the limitations
 	imposed by shallow repositories.
 +
 If the source repository is shallow, fetch as much as possible so that
-the current repository has the same history as the source repository.
+the all existing refs of current repository have the same history as in
+the source repository.
++
+Note that if the repository is created with --single-branch, which is
+default for a shallow clone, only one ref is completed. `--unshallow`
+does not fetch all remaining refs from source repository.
 
 --update-shallow::
 	By default when fetching from a shallow repository,
-- 
2.3.0.rc1.137.g477eb31
-- 8< --

> 2) git fetch --unshallow should convert the clone into an actual
> equivalent of a normal, not shallow clone.

I was thinking of some way to undo --single-branch too. I don't think
it should be the job of --unshallow, maybe a new option, but I can't
think of a name better than --really-unshallow :P

It's been a while and no one responds to this, perhaps people are not
interested in such an option?

> 3) The documentation should be improved. The behaviour of --shallow is
> described partly in the description of --no-single-branch. This is the
> documentation equivalent of the infamous "come from" control flow
> structure.

Yes. Like this?

-- 8< --
Subject: [PATCH] clone.txt: mention --single-branch in --depth

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-clone.txt | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index f1f2a3f..9c320da 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -185,7 +185,9 @@ objects from the source repository into a pack in the cloned repository.
 
 --depth <depth>::
 	Create a 'shallow' clone with a history truncated to the
-	specified number of revisions.
+	specified number of revisions. --single-branch is
+	automatically specified unless the user overrides it with
+	--no-single-branch
 
 --[no-]single-branch::
 	Clone only the history leading to the tip of a single branch,
-- 
2.3.0.rc1.137.g477eb31

-- 8< --

Re: git clone --depth: shallow clone problems

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

>  --unshallow::
> -	If the source repository is complete, convert a shallow
> -	repository to a complete one, removing all the limitations
> +	If the source repository is complete, convert all shallow
> +	refs to complete ones, removing all the limitations

Define "shallow ref", or better yet explain without introducing such
a new term (I do not think shallow-ness is a property of a ref, by
the way---I think you meant all refs that can reach the points the
history is cauterised by .git/shallow, but we shouldn't assume that
the target audience of this paragraph to know Git as well as I do).

>  	imposed by shallow repositories.
>  +
>  If the source repository is shallow, fetch as much as possible so that
> -the current repository has the same history as the source repository.
> +the all existing refs of current repository have the same history as in
> +the source repository.

Makes sense, modulo "so that the all existing refs" sounds strange
to my ears ("all the existing refs" perhaps).

> ++
> +Note that if the repository is created with --single-branch, which is
> +default for a shallow clone, only one ref is completed. `--unshallow`
> +does not fetch all remaining refs from source repository.

I do not think this "Note" is being as helpful as it could be.

 - If the repository was created with --single-branch but if the
   user later fetched and started tracking other branches, the
   statement "only one ref is completed" is untrue; the true version
   is "only the existing refs are completed", which leads to a more
   important point.  It says the same thing as "all existing refs"
   above and does not add any useful information.

 - The last sentence is less than useful but merely frustrating---it
   says what you cannot do with this option, strongly hints that the
   writer of the sentence knows what the reader wants to achieve,
   but without saying what other way the reader go to achieve it.

Perhaps replace that Note paragraph with something along this line?

    +
    By fetching and tracking refs that you haven't been tracking,
    you can add these new refs to "all refs" to be unshallowed.

>> 2) git fetch --unshallow should convert the clone into an actual
>> equivalent of a normal, not shallow clone.
>
> I was thinking of some way to undo --single-branch too. I don't think
> it should be the job of --unshallow, maybe a new option, but I can't
> think of a name better than --really-unshallow :P

Isn't that just the matter of updating remote.origin.fetch?  I do
not think it belongs to "clone" (or "fetch").  Perhaps it belongs to
"remote", where it already shows with "git remote -v" what is
fetched and pushed.

>  --depth <depth>::
>  	Create a 'shallow' clone with a history truncated to the
> -	specified number of revisions.
> +	specified number of revisions. --single-branch is
> +	automatically specified unless the user overrides it with
> +	--no-single-branch

Yeah, something like that would be a definite improvement.

By the way, while composing this message, I scratched my head after
typing

    $ git clone --shallow=4 --no-local ./git.git ./playpen

Is it just me or do we want to add such a synonym?

Thanks.

Re: git clone --depth: shallow clone problems

[Duy Nguyen]

On Tue, May 26, 2015 at 9:54 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Duy Nguyen <pclouds@gmail.com> writes:
>
>>  --unshallow::
>> -     If the source repository is complete, convert a shallow
>> -     repository to a complete one, removing all the limitations
>> +     If the source repository is complete, convert all shallow
>> +     refs to complete ones, removing all the limitations
>
> Define "shallow ref", or better yet explain without introducing such
> a new term (I do not think shallow-ness is a property of a ref, by
> the way---I think you meant all refs that can reach the points the
> history is cauterised by .git/shallow, but we shouldn't assume that
> the target audience of this paragraph to know Git as well as I do).

OK maybe

Make sure all existing refs have the same history as the ones from the
source repository. If the source repository is complete, this removes
all limitations imposed by shallow repositories.

?

>>       imposed by shallow repositories.
>>  +
>>  If the source repository is shallow, fetch as much as possible so that
>> -the current repository has the same history as the source repository.
>> +the all existing refs of current repository have the same history as in
>> +the source repository.
>
> Makes sense, modulo "so that the all existing refs" sounds strange
> to my ears ("all the existing refs" perhaps).

Will fix. Although I think the new paragraph above already covers this case too.

>> ++
>> +Note that if the repository is created with --single-branch, which is
>> +default for a shallow clone, only one ref is completed. `--unshallow`
>> +does not fetch all remaining refs from source repository.
>
> I do not think this "Note" is being as helpful as it could be.
>
>  - If the repository was created with --single-branch but if the
>    user later fetched and started tracking other branches, the
>    statement "only one ref is completed" is untrue; the true version
>    is "only the existing refs are completed", which leads to a more
>    important point.  It says the same thing as "all existing refs"
>    above and does not add any useful information.
>
>  - The last sentence is less than useful but merely frustrating---it
>    says what you cannot do with this option, strongly hints that the
>    writer of the sentence knows what the reader wants to achieve,
>    but without saying what other way the reader go to achieve it.
>
> Perhaps replace that Note paragraph with something along this line?
>
>     +
>     By fetching and tracking refs that you haven't been tracking,
>     you can add these new refs to "all refs" to be unshallowed.

It was meant to emphasize --unshallow would not create a true clone,
if people miss the keywords "all existing" refs.. Maybe..

You may need to fetch and track other refs from the source repository
if you want to make full clone.

?

>
>>> 2) git fetch --unshallow should convert the clone into an actual
>>> equivalent of a normal, not shallow clone.
>>
>> I was thinking of some way to undo --single-branch too. I don't think
>> it should be the job of --unshallow, maybe a new option, but I can't
>> think of a name better than --really-unshallow :P
>
> Isn't that just the matter of updating remote.origin.fetch?  I do
> not think it belongs to "clone" (or "fetch").  Perhaps it belongs to
> "remote", where it already shows with "git remote -v" what is
> fetched and pushed.

Yeah, perhaps showing an advice about git-remote is enough if the
current ref spec does not fetch everything.

>>  --depth <depth>::
>>       Create a 'shallow' clone with a history truncated to the
>> -     specified number of revisions.
>> +     specified number of revisions. --single-branch is
>> +     automatically specified unless the user overrides it with
>> +     --no-single-branch
>
> Yeah, something like that would be a definite improvement.
>
> By the way, while composing this message, I scratched my head after
> typing
>
>     $ git clone --shallow=4 --no-local ./git.git ./playpen
>
> Is it just me or do we want to add such a synonym?

I have a series to define shallow boundaries by tags or dates, in
addition to history depth. It may take a few more months until I'm
finished with my ongoing topics. Then you could think about what
--shallow means exactly ;-)
-- 
Duy

Re: git clone --depth: shallow clone problems

[Junio C Hamano]

Duy Nguyen <pclouds@gmail.com> writes:

> On Tue, May 26, 2015 at 9:54 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Duy Nguyen <pclouds@gmail.com> writes:
>>
>>>  --unshallow::
>>> -     If the source repository is complete, convert a shallow
>>> -     repository to a complete one, removing all the limitations
>>> +     If the source repository is complete, convert all shallow
>>> +     refs to complete ones, removing all the limitations
>>
>> Define "shallow ref", or better yet explain without introducing such
>> a new term (I do not think shallow-ness is a property of a ref, by
>> the way---I think you meant all refs that can reach the points the
>> history is cauterised by .git/shallow, but we shouldn't assume that
>> the target audience of this paragraph to know Git as well as I do).
>
> OK maybe
>
> Make sure all existing refs have the same history as the ones from the
> source repository. If the source repository is complete, this removes
> all limitations imposed by shallow repositories.

OK, I can understand that description a lot easier than the original
one.

>>> ++
>>> +Note that if the repository is created with --single-branch, which is
>>> +default for a shallow clone, only one ref is completed. `--unshallow`
>>> +does not fetch all remaining refs from source repository.
>>
>> I do not think this "Note" is being as helpful as it could be.
>>
>>  - If the repository was created with --single-branch but if the
>>    user later fetched and started tracking other branches, the
>>    statement "only one ref is completed" is untrue; the true version
>>    is "only the existing refs are completed", which leads to a more
>>    important point.  It says the same thing as "all existing refs"
>>    above and does not add any useful information.
>>
>>  - The last sentence is less than useful but merely frustrating---it
>>    says what you cannot do with this option, strongly hints that the
>>    writer of the sentence knows what the reader wants to achieve,
>>    but without saying what other way the reader go to achieve it.
>>
>> Perhaps replace that Note paragraph with something along this line?
>>
>>     +
>>     By fetching and tracking refs that you haven't been tracking,
>>     you can add these new refs to "all refs" to be unshallowed.
>
> It was meant to emphasize --unshallow would not create a true clone,
> if people miss the keywords "all existing" refs.. Maybe..

I know; it is not helpful to describe what it does *NOT* do, without
saying what the reader is likely trying to find out.

> You may need to fetch and track other refs from the source repository
> if you want to make full clone.

That is much better.