shallow clone not very shallow due to tags

shallow clone not very shallow due to tags

[Shawn Pearce]

$ git clone --depth 1 git://repo.or.cz/alt-git.git tmp_cgit_5
Cloning into tmp_cgit_5...
remote: Counting objects: 20234, done.
remote: Compressing objects: 100% (9454/9454), done.
remote: Total 20234 (delta 16355), reused 13587 (delta 10421)
Receiving objects: 100% (20234/20234), 9.15 MiB | 1.17 MiB/s, done.
Resolving deltas: 100% (16355/16355), done.

$ cd tmp_cgit_5
$ git tag -l | grep 0.99
v0.99

Uhm. That is not a very shallow clone. The clone copied 20234 objects
at 9.15 MiB... so its ~20 MiB lighter than a full clone. But nearly
all of the tags exist, because the clone client is declaring want
lines for them, making the server generate up to 1 commit back from
the wanted tag. I know shallow support is the feature nobody wants to
think about, but this just seems broken to me. Clients performing a
shallow clone shouldn't be asking for tags... but they should be using
the include-tag protocol option so that if they do happen to receive a
tagged commit, the tag object will also be sent.

-- 
Shawn.

Re: shallow clone not very shallow due to tags

[Nguyen Thai Ngoc Duy]

On Mon, Aug 15, 2011 at 6:58 AM, Shawn Pearce <spearce@spearce.org> wrote:
> Uhm. That is not a very shallow clone. The clone copied 20234 objects
> at 9.15 MiB... so its ~20 MiB lighter than a full clone. But nearly
> all of the tags exist, because the clone client is declaring want
> lines for them, making the server generate up to 1 commit back from
> the wanted tag. I know shallow support is the feature nobody wants to
> think about, but this just seems broken to me. Clients performing a
> shallow clone shouldn't be asking for tags... but they should be using
> the include-tag protocol option so that if they do happen to receive a
> tagged commit, the tag object will also be sent.

The same would apply if the repo in question has many branches. Should
we fetch only master (or a user-specified set of refs) in shallow
clone?
-- 
Duy

Re: shallow clone not very shallow due to tags

[Shawn Pearce]

On Mon, Aug 15, 2011 at 07:03, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On Mon, Aug 15, 2011 at 6:58 AM, Shawn Pearce <spearce@spearce.org> wrote:
>> Uhm. That is not a very shallow clone. The clone copied 20234 objects
>> at 9.15 MiB... so its ~20 MiB lighter than a full clone. But nearly
>> all of the tags exist, because the clone client is declaring want
>> lines for them, making the server generate up to 1 commit back from
>> the wanted tag. I know shallow support is the feature nobody wants to
>> think about, but this just seems broken to me. Clients performing a
>> shallow clone shouldn't be asking for tags... but they should be using
>> the include-tag protocol option so that if they do happen to receive a
>> tagged commit, the tag object will also be sent.
>
> The same would apply if the repo in question has many branches. Should
> we fetch only master (or a user-specified set of refs) in shallow
> clone?

Yes, I think a user who is making a shallow clone should only get the
HEAD branch to the depth specified, nothing else. If they want
additional branches, they should either pass them on the command line
to a new flag for clone, or modify their fetch configuration
after-the-fact and fetch those separately.

>From what I can gather from some users I have talked to, the primary
usage of shallow clone is to try and (sort of) quickly grab a branch,
make a change, and post that change to the maintainers for review and
acceptance. E.g. correcting a spelling typo. Relatively simple changes
that can be built on a specific branch, and don't really require all
of the history.

And if a repository does have more than one branch, but is shallow
cloned at say depth of 1, the user probably doesn't get the merge
bases between them, so the value of the other branches is greatly
reduced. You can't make a merge between these, and a new developer
getting involved in the project cannot see how the branches relate to
each other. So there isn't a lot of value in sucking down those
additional branches during clone.

-- 
Shawn.

Re: shallow clone not very shallow due to tags

[Philip Oakley]

Monday, August 15, 2011 6:16 PM "Shawn Pearce" <spearce@spearce.org>
> On Mon, Aug 15, 2011 at 07:03, Nguyen Thai Ngoc Duy <pclouds@gmail.com> 
> wrote:
>> On Mon, Aug 15, 2011 at 6:58 AM, Shawn Pearce <spearce@spearce.org> 
>> wrote:
>>> Uhm. That is not a very shallow clone. The clone copied 20234 objects
>>> at 9.15 MiB... so its ~20 MiB lighter than a full clone. But nearly
>>> all of the tags exist, because the clone client is declaring want
>>> lines for them, making the server generate up to 1 commit back from
>>> the wanted tag. I know shallow support is the feature nobody wants to
>>> think about, but this just seems broken to me. Clients performing a
>>> shallow clone shouldn't be asking for tags... but they should be using
>>> the include-tag protocol option so that if they do happen to receive a
>>> tagged commit, the tag object will also be sent.
>>
>> The same would apply if the repo in question has many branches. Should
>> we fetch only master (or a user-specified set of refs) in shallow
>> clone?
>
> Yes, I think a user who is making a shallow clone should only get the
> HEAD branch to the depth specified, nothing else. If they want
> additional branches, they should either pass them on the command line
> to a new flag for clone, or modify their fetch configuration
> after-the-fact and fetch those separately.
>
> From what I can gather from some users I have talked to, the primary
> usage of shallow clone is to try and (sort of) quickly grab a branch,
> make a change, and post that change to the maintainers for review and
> acceptance. E.g. correcting a spelling typo. Relatively simple changes
> that can be built on a specific branch, and don't really require all
> of the history.
>
> And if a repository does have more than one branch, but is shallow
> cloned at say depth of 1, the user probably doesn't get the merge
> bases between them, so the value of the other branches is greatly
> reduced. You can't make a merge between these, and a new developer
> getting involved in the project cannot see how the branches relate to
> each other. So there isn't a lot of value in sucking down those
> additional branches during clone.
>
The shallow clone problem came up a few times recently on StackOverflow. One 
usage is to create an orphan branch, but the manual doesn't point the user 
to that option. When I first read the man page I expected that any clone 
would be from a fixed point in history, rather than a point that may move 
(as extra commits are made to the remote). It maybe that what is needed is 
an option that will clone from a fixed commit point, so that there is no 
confusion as to the 'current depth'. 

Re: shallow clone not very shallow due to tags

[Nguyen Thai Ngoc Duy]

On Tue, Aug 16, 2011 at 12:16 AM, Shawn Pearce > Yes, I think a user
who is making a shallow clone should only get the
> HEAD branch to the depth specified, nothing else. If they want
> additional branches, they should either pass them on the command line
> to a new flag for clone, or modify their fetch configuration
> after-the-fact and fetch those separately.

I'd like maintain all refs of the repo, even though we though we only
have objects for one branch. If I do a shallow clone and want to
change branch, I could just list available branch locally instead of
going to gitweb.somewhere for the branch name.

Do we have someway to represent a ref placeholder?
-- 
Duy

Re: shallow clone not very shallow due to tags

[Shawn Pearce]

On Tue, Aug 16, 2011 at 19:49, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> On Tue, Aug 16, 2011 at 12:16 AM, Shawn Pearce > Yes, I think a user
> who is making a shallow clone should only get the
>> HEAD branch to the depth specified, nothing else. If they want
>> additional branches, they should either pass them on the command line
>> to a new flag for clone, or modify their fetch configuration
>> after-the-fact and fetch those separately.
>
> I'd like maintain all refs of the repo, even though we though we only
> have objects for one branch. If I do a shallow clone and want to
> change branch, I could just list available branch locally instead of
> going to gitweb.somewhere for the branch name.
>
> Do we have someway to represent a ref placeholder?

Not currently, no. I haven't checked refs.c, but you may be able to
create an empty (0 byte) ref file for each remote reference we didn't
obtain. These should be ignored since they aren't a valid reference..
Though they might cause warnings to be sent to stderr on every
command, since they exist in the refs directory but aren't a valid
reference.

Of course a placeholder ref cannot be used as a starting point to
checkout something, can it? It has no commit object available. So if
you did decide to checkout a placeholder, you would need to first
fetch that placeholder. So why not use git ls-remote to list the
remotely available branches?

-- 
Shawn.

Re: shallow clone not very shallow due to tags

[Nguyen Thai Ngoc Duy]

On Wed, Aug 17, 2011 at 9:19 PM, Shawn Pearce <spearce@spearce.org> wrote:
> Of course a placeholder ref cannot be used as a starting point to
> checkout something, can it? It has no commit object available. So if
> you did decide to checkout a placeholder, you would need to first
> fetch that placeholder. So why not use git ls-remote to list the
> remotely available branches?

Ah.. ls-remote is the answer. Thanks, placeholders are gone (for me).
-- 
Duy