[PATCH] Do not fetch tags on new shallow clones

[PATCH] Do not fetch tags on new shallow clones

[Nguyễn Thái Ngọc Duy]

The main purpose of shallow clones is to reduce download. Fetching
tags likely defeats this purpose because old-enough repos tend to have
a lot of tags, spreading across history, which may increase the number
of objects to download significantly.

For example, "git clone --depth=10 git://.../git.git" without changes
fetches ~16M (50k objects). The same command with changes fetches
~6.5M (10k objects).

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 This could also be applied for normal clones. But I don't think
 there are many use cases for it, enough to deserve new --no-tags
 option.

 We should also fetch a single branch, but because branches are
 usually less crowded and stay close the tip, they do not produce too
 many extra objects. Let's leave it until somebody yells up.

 We should also fetch tags that reference to downloaded objects. But I
 don't know how fetch does that magic, so for now users have to do
 "git fetch" after cloning for tags. I have only gone as far as
 fetching tags along by setting TRANS_OPT_FOLLOWTAGS? Help?

 builtin/clone.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 86db954..abd8578 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -428,7 +428,7 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	struct ref **tail = head ? &head->next : &local_refs;
 
 	get_fetch_map(refs, refspec, &tail, 0);
-	if (!option_mirror)
+	if (!option_mirror && !option_depth)
 		get_fetch_map(refs, tag_refspec, &tail, 0);
 
 	return local_refs;
-- 
1.7.8.36.g69ee2

Re: [PATCH] Do not fetch tags on new shallow clones

[Shawn Pearce]

2012/1/4 Nguyễn Thái Ngọc Duy <pclouds@gmail.com>:
> The main purpose of shallow clones is to reduce download. Fetching
> tags likely defeats this purpose because old-enough repos tend to have
> a lot of tags, spreading across history, which may increase the number
> of objects to download significantly.

Thank you for looking at this. I complained about it to Junio many
weeks ago, but never took the time myself to fix it. :-)

>  We should also fetch a single branch, but because branches are
>  usually less crowded and stay close the tip, they do not produce too
>  many extra objects. Let's leave it until somebody yells up.

Depends on the project. In git.git maint stays relatively close to
master, but its still not really that close. In other projects, there
are certainly huge differences between two active branches, sometimes
spanning years. Consider any product with a multiple year support
contract on an older version, where the support contract demands
patches for the older version to fix bugs.  :-)

I agree this can be looked at later with a different change, but there
should be a way to specify exactly which branches you want to clone,
especially in the shallow case.

>  We should also fetch tags that reference to downloaded objects. But I
>  don't know how fetch does that magic,

If the remote advertises the capability "include-tag", and the client
wants tags, it asks for that include-tag capability in its request.
This is handled by the fetch_pack args field include_tag being set to
1. When the remote side sees the client requesting include-tag and it
packs the thing a tag points at, the tag is also packed, even though
it wasn't explicitly requested by the client.

> so for now users have to do
>  "git fetch" after cloning for tags. I have only gone as far as
>  fetching tags along by setting TRANS_OPT_FOLLOWTAGS? Help?

Right. Set TRANS_OPT_FOLLOWTAGS in the transport structure to fetch
only tags that are pointing at things already being sent. The delta
increase in transfer is 1 object (the tag) and whatever that tag takes
up on disk.

> diff --git a/builtin/clone.c b/builtin/clone.c
> index 86db954..abd8578 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -428,7 +428,7 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
>        struct ref **tail = head ? &head->next : &local_refs;
>
>        get_fetch_map(refs, refspec, &tail, 0);
> -       if (!option_mirror)
> +       if (!option_mirror && !option_depth)
>                get_fetch_map(refs, tag_refspec, &tail, 0);
>
>        return local_refs;

I think if you just add this into your patch, you get the auto follow
tag feature enabled:

diff --git a/builtin/clone.c b/builtin/clone.c
index efe8b6c..ecaafdb 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -641,6 +641,7 @@ int cmd_clone(int argc, const char **argv, const char *prefi
                        die(_("Don't know how to clone %s"), transport->url);

                transport_set_option(transport, TRANS_OPT_KEEP, "yes");
+               transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, "1");

                if (option_depth)
                        transport_set_option(transport, TRANS_OPT_DEPTH,

totally untested (didn't even compile). This only works for the
"remote" cases where the native Git protocol is used. A local clone
using the hardlink or copy objects path, or a dumb HTTP or rsync clone
will ignore the option and not supply you the tags.

Annnddddd..... it doesn't appear to work.

You need to copy a block of code from fetch. The problem is the object
was copied locally by the transport, but the transport doesn't tell
you what extra objects came along. Clone has to loop back through the
advertised reference map from the transport, checking each tag to see
if has_sha1_file() says the object exists locally. If it does, then
clone needs to add that reference update to the set of things it will
store (and print to the terminal).

I think this loop is the find_non_local_tags() in builtin/fetch.c. Its
been a long time since I hacked on this code. The JGit version of
looking for these extra objects post transfer is more clearly
documented. *sigh*

Re: [PATCH] Do not fetch tags on new shallow clones

[Junio C Hamano]

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:

>  This could also be applied for normal clones. But I don't think
>  there are many use cases for it, enough to deserve new --no-tags
>  option.

Correct.

>  We should also fetch a single branch, but because branches are
>  usually less crowded and stay close the tip, they do not produce too
>  many extra objects. Let's leave it until somebody yells up.

Probably this is needed.

>  We should also fetch tags that reference to downloaded objects.

I do not think this has much merit. The usual tag-following rules AFAIK
assumes that if you have the commit C then you ought to have all the
ancestors of C, which does not apply at all for the shallow hack to begin
with, and if you make the rule apply for the shallow hack, you would end
up slurping the objects that are needed only for ancient versions, which
would defeat the objective of this patch, no?

It also is my understanding that the shallow hack is almost always used
with a depth of one, not a range, like "git archive | tar xf -", so if
anything, I would say a single-branch cloning has much higher priority
than following tags.

Will queue on 'pu'. Thanks.

Re: [PATCH] Do not fetch tags on new shallow clones

[Shawn Pearce]

2012/1/4 Junio C Hamano <gitster@pobox.com>:
> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
>>  We should also fetch tags that reference to downloaded objects.
>
> I do not think this has much merit.

I disagree. Its useful because cloning a branch immediately after it
has been tagged for a release should have `git describe` provide back
the name of the release from the tag (assuming of course no new
commits were made since the tag).

> The usual tag-following rules AFAIK
> assumes that if you have the commit C then you ought to have all the
> ancestors of C, which does not apply at all for the shallow hack to begin
> with, and if you make the rule apply for the shallow hack, you would end
> up slurping the objects that are needed only for ancient versions, which
> would defeat the objective of this patch, no?

We aren't talking about fetching the ancient history tags. We are
talking about fetching a tag that *directly* points at one of the
commits we did download the complete tree of.

> It also is my understanding that the shallow hack is almost always used
> with a depth of one, not a range, like "git archive | tar xf -", so if
> anything, I would say a single-branch cloning has much higher priority
> than following tags.

I think I agree with you on priority of work effort. But I lack time
to make good on that by writing the code myself.  :-)

Re: [PATCH] Do not fetch tags on new shallow clones

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> 2012/1/4 Junio C Hamano <gitster@pobox.com>:
>> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:
>>>  We should also fetch tags that reference to downloaded objects.
>>
>> I do not think this has much merit.
>
> I disagree. ...

No, you don't.  Because...

> ... Its useful because cloning a branch immediately after it
> has been tagged for a release should have `git describe` provide back
> the name of the release from the tag (assuming of course no new
> commits were made since the tag).

... this is a natural extension of ...

>> It also is my understanding that the shallow hack is almost always used
>> with a depth of one, not a range, like "git archive | tar xf -", so if
>> anything, I would say a single-branch cloning has much higher priority
>> than following tags.
>
> I think I agree with you on priority of work effort. ...

... this thing, once you have a "single ref only" stuff working.  After
Linus announces that he released 3.2, you would do the poor emulation of
"git archive | tar xf -" with something like:

    git clone --single=v3.2 --shallow $there linux-3.2

and your "git describe" should fall out as a natural consequence out of
everything else, without the usual "tag following" semantics, no?

Re: [PATCH] Do not fetch tags on new shallow clones

[Shawn Pearce]

2012/1/4 Junio C Hamano <gitster@pobox.com>:
> Shawn Pearce <spearce@spearce.org> writes:
>> ... Its useful because cloning a branch immediately after it
>> has been tagged for a release should have `git describe` provide back
>> the name of the release from the tag (assuming of course no new
>> commits were made since the tag).
...
> ... this thing, once you have a "single ref only" stuff working.  After
> Linus announces that he released 3.2, you would do the poor emulation of
> "git archive | tar xf -" with something like:
>
>    git clone --single=v3.2 --shallow $there linux-3.2
>
> and your "git describe" should fall out as a natural consequence out of
> everything else, without the usual "tag following" semantics, no?

I said "branch" not "tag". Of course a single ref clone might be able
to be used on a tag.

If my project maintainer tags a release from "maint" and announces
that, I should also be able to shallow clone maint and pick up the tag
automatically if it is within the depth I have asked for from the
server.

Consider this case, a client clones shallow with a depth of 1. Then
does normal `git fetch` to keep up-to-date with the project. When the
project places a new tag on a branch, our shallow follower will
automatically get that tag during her next `git fetch`, because auto
following tags is enabled in fetch and the tag's referent was included
in the pack. Why is this case permitted to get a tag, and shallow
clone is not?

Actually, I think you might find that a shallow client with depth of 1
will automatically pick up a missing tag at the branch head on its
next `git fetch`. It will see the tag's ^{} line advertise an object
it has, and ask for the tag.

We really should support auto-following tags within the history space
the client already has. Its mostly done for us with the include-tag
capability, the client just needs to make sure it asks for it from the
server, and check to see if any tag reference points to an object it
has.

> you would do the poor emulation of
> "git archive | tar xf -" with something like:
>
>    git clone --single=v3.2 --shallow $there linux-3.2

Is it really that poor of an emulation? Like tar, we get only one copy
of each file (assuming depth 1). Assuming --format=tar.gz, both are
compressed. I wonder how much better or worse the Git cross-object
delta compression is than the libz rolling window. I could see how Git
might be able to compress something like C source code smaller than
tar | gzip by using delta compression on related files (e.g. Makefile
in every directory, or *.h and *.c files pairing by type). An added
advantage of the shallow clone is you can incrementally update that
stream, as its easy to fetch a v3.2.1 patch release, or apply a patch
and record it on top.

[PATCH v2] Limit refs to fetch to minimum in shallow clones

[Nguyễn Thái Ngọc Duy]

The main purpose of shallow clones is to reduce download by only
fetching objects up to a certain depth from the given refs. The number
of objects depends on how many refs to follow. So:

 - Only fetch HEAD or the ref specified by --branch
 - Only fetch tags that point to downloaded objects

More tags/branches can be fetched later using git-fetch as usual.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Only lightly tested, but seems to work. I'll check later if --branch
 works with a tag like Junio's example --single=v3.2

 Documentation/git-clone.txt |   14 ++++++----
 builtin/clone.c             |   54 +++++++++++++++++++++++++++++++++++++++----
 2 files changed, 57 insertions(+), 11 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 4b8b26b..d69f440 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -172,12 +172,14 @@ objects from the source repository into a pack in the cloned repository.
 
 --depth <depth>::
 	Create a 'shallow' clone with a history truncated to the
-	specified number of revisions.  A shallow repository has a
-	number of limitations (you cannot clone or fetch from
-	it, nor push from nor into it), but is adequate if you
-	are only interested in the recent history of a large project
-	with a long history, and would want to send in fixes
-	as patches.
+	specified number of revisions. Only one branch (either HEAD
+	or specified by --branch) is fetched. Tags that point
+	outside truncated history are not fetched.
++
+A shallow repository has a number of limitations (you cannot clone or
+fetch from it, nor push from nor into it), but is adequate if you are
+only interested in the recent history of a large project with a long
+history, and would want to send in fixes as patches.
 
 --recursive::
 --recurse-submodules::
diff --git a/builtin/clone.c b/builtin/clone.c
index efe8b6c..8de9248 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -48,6 +48,7 @@ static int option_verbosity;
 static int option_progress;
 static struct string_list option_config;
 static struct string_list option_reference;
+static char *src_ref_prefix = "refs/heads/";
 
 static int opt_parse_reference(const struct option *opt, const char *arg, int unset)
 {
@@ -427,9 +428,27 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	struct ref *local_refs = head;
 	struct ref **tail = head ? &head->next : &local_refs;
 
-	get_fetch_map(refs, refspec, &tail, 0);
-	if (!option_mirror)
-		get_fetch_map(refs, tag_refspec, &tail, 0);
+	if (option_depth) {
+		struct ref *remote_head = NULL;
+
+		if (!option_branch)
+			remote_head = guess_remote_head(head, refs, 0);
+		else {
+			struct strbuf sb = STRBUF_INIT;
+			strbuf_addstr(&sb, src_ref_prefix);
+			strbuf_addstr(&sb, option_branch);
+			remote_head = find_ref_by_name(refs, sb.buf);
+			strbuf_release(&sb);
+		}
+
+		if (remote_head)
+			get_fetch_map(remote_head, refspec, &tail, 0);
+	}
+	else {
+		get_fetch_map(refs, refspec, &tail, 0);
+		if (!option_mirror)
+			get_fetch_map(refs, tag_refspec, &tail, 0);
+	}
 
 	return local_refs;
 }
@@ -448,6 +467,27 @@ static void write_remote_refs(const struct ref *local_refs)
 	clear_extra_refs();
 }
 
+static void write_followtags(const struct ref *refs)
+{
+	struct ref_lock *lock;
+	const struct ref *ref;
+
+	for (ref = refs; ref; ref = ref->next) {
+		if (prefixcmp(ref->name, "refs/tags"))
+			continue;
+		if (!suffixcmp(ref->name, "^{}"))
+			continue;
+		if (!has_sha1_file(ref->old_sha1))
+			continue;
+
+		lock = lock_any_ref_for_update(ref->name, NULL, 0);
+		if (!lock)
+			die_errno(_("unable to lock %s for writing"), ref->name);
+		if (write_ref_sha1(lock, ref->old_sha1, "storing tag") < 0)
+			die_errno(_("unable to write %s"), ref->name);
+	}
+}
+
 static int write_one_config(const char *key, const char *value, void *data)
 {
 	return git_config_set_multivar(key, value ? value : "true", "^$", 0);
@@ -478,7 +518,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	struct strbuf key = STRBUF_INIT, value = STRBUF_INIT;
 	struct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;
 	struct transport *transport = NULL;
-	char *src_ref_prefix = "refs/heads/";
 	int err = 0;
 
 	struct refspec *refspec;
@@ -642,9 +681,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 		transport_set_option(transport, TRANS_OPT_KEEP, "yes");
 
-		if (option_depth)
+		if (option_depth) {
 			transport_set_option(transport, TRANS_OPT_DEPTH,
 					     option_depth);
+			transport_set_option(transport, TRANS_OPT_FOLLOWTAGS,
+					     "1");
+		}
 
 		transport_set_verbosity(transport, option_verbosity, option_progress);
 
@@ -663,6 +705,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		clear_extra_refs();
 
 		write_remote_refs(mapped_refs);
+		if (option_depth)
+			write_followtags(refs);
 
 		remote_head = find_ref_by_name(refs, "HEAD");
 		remote_head_points_at =
-- 
1.7.3.1.256.g2539c.dirty

Re: [PATCH v2] Limit refs to fetch to minimum in shallow clones

[Junio C Hamano]

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:

> The main purpose of shallow clones is to reduce download by only
> fetching objects up to a certain depth from the given refs. The number
> of objects depends on how many refs to follow. So:
>
>  - Only fetch HEAD or the ref specified by --branch
>  - Only fetch tags that point to downloaded objects
>
> More tags/branches can be fetched later using git-fetch as usual.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  Only lightly tested, but seems to work.

Thanks.

Perhaps you would want to add tests so that you do not have to say
"lightly tested"?

> diff --git a/builtin/clone.c b/builtin/clone.c
> index efe8b6c..8de9248 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -48,6 +48,7 @@ static int option_verbosity;
>  static int option_progress;
>  static struct string_list option_config;
>  static struct string_list option_reference;
> +static char *src_ref_prefix = "refs/heads/";

Would this be const?

>  static int opt_parse_reference(const struct option *opt, const char *arg, int unset)
>  {
> @@ -427,9 +428,27 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
>  	struct ref *local_refs = head;
>  	struct ref **tail = head ? &head->next : &local_refs;
>  
> -	get_fetch_map(refs, refspec, &tail, 0);
> -	if (!option_mirror)
> -		get_fetch_map(refs, tag_refspec, &tail, 0);
> +	if (option_depth) {
> +		struct ref *remote_head = NULL;
> +
> +		if (!option_branch)
> +			remote_head = guess_remote_head(head, refs, 0);
> +		else {
> +			struct strbuf sb = STRBUF_INIT;
> +			strbuf_addstr(&sb, src_ref_prefix);
> +			strbuf_addstr(&sb, option_branch);
> +			remote_head = find_ref_by_name(refs, sb.buf);
> +			strbuf_release(&sb);
> +		}
> +
> +		if (remote_head)
> +			get_fetch_map(remote_head, refspec, &tail, 0);

What happens when we fail to find any remote_head and make no call to
get_fetch_map() here?  I am wondering if that should trigger an error
here.

Also this breaks 5500 for rather obvious reasons, as the point of this
patch is to reduce the object transferred when a shallow clone is made.

Perhaps there should be an option to give users the historical "all
branches equally shallow" behaviour?

[PATCH v3] Limit refs to fetch to minimum in shallow clones

[Nguyễn Thái Ngọc Duy]

The main purpose of shallow clones is to reduce download by only
fetching objects up to a certain depth from the given refs. The number
of objects depend on how many refs to follow. So:

 - Only fetch HEAD or the ref specified by --branch
 - Only fetch tags that reference to downloaded objects

More tags/branches can be fetched later using git-fetch as usual.

The old behaviour can still be called with --no-single-branch

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
  - add --no-single-branch so t5500 works without big changes.
  - die() if we cannot find suitable branch to fetch (and suggest --no-single-branch)
  - and a bit more tests to exercise new code

  --branch=<tag> (or something similar) has to wait until my other
  patch gets in a good shape (or gets dropped)

 There may be something slightly wrong with shallow code. I expect it
 to fetch only 3 objects (1 commit, 1 tree, 1 blob) with --depth=1
 in my new test but it fetches 6 (one more commit).

 Documentation/git-clone.txt |   12 ++++++++-
 builtin/clone.c             |   54 ++++++++++++++++++++++++++++++++++++++----
 t/t5500-fetch-pack.sh       |   34 ++++++++++++++++++++++++++-
 3 files changed, 92 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 4b8b26b..58f21d6 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -13,7 +13,8 @@ SYNOPSIS
 	  [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]
 	  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
 	  [--separate-git-dir <git dir>]
-	  [--depth <depth>] [--recursive|--recurse-submodules] [--] <repository>
+	  [--depth <depth> [--[no-]single-branch]]
+	  [--recursive|--recurse-submodules] [--] <repository>
 	  [<directory>]
 
 DESCRIPTION
@@ -179,6 +180,15 @@ objects from the source repository into a pack in the cloned repository.
 	with a long history, and would want to send in fixes
 	as patches.
 
+--single-branch::
+--no-single-branch::
+	These options are only valid when --depth is given.
+	 `--single-branch` only fetches one branch (either HEAD or
+	specified by --branch) and tags that point to the downloaded
+	history. `--no-single-branch` fetches all branches and tags
+	like in normal clones. `--single-branch` is implied by
+	default.
+
 --recursive::
 --recurse-submodules::
 	After the clone is created, initialize all submodules within,
diff --git a/builtin/clone.c b/builtin/clone.c
index efe8b6c..3424e1c 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -37,7 +37,7 @@ static const char * const builtin_clone_usage[] = {
 	NULL
 };
 
-static int option_no_checkout, option_bare, option_mirror;
+static int option_no_checkout, option_bare, option_mirror, option_single_branch = 1;
 static int option_local, option_no_hardlinks, option_shared, option_recursive;
 static char *option_template, *option_depth;
 static char *option_origin = NULL;
@@ -48,6 +48,7 @@ static int option_verbosity;
 static int option_progress;
 static struct string_list option_config;
 static struct string_list option_reference;
+static const char *src_ref_prefix = "refs/heads/";
 
 static int opt_parse_reference(const struct option *opt, const char *arg, int unset)
 {
@@ -92,6 +93,8 @@ static struct option builtin_clone_options[] = {
 		   "path to git-upload-pack on the remote"),
 	OPT_STRING(0, "depth", &option_depth, "depth",
 		    "create a shallow clone of that depth"),
+	OPT_BOOL(0, "single-branch", &option_single_branch,
+		    "do not limit fetched refs in shallow clones"),
 	OPT_STRING(0, "separate-git-dir", &real_git_dir, "gitdir",
 		   "separate git dir from working tree"),
 	OPT_STRING_LIST('c', "config", &option_config, "key=value",
@@ -427,9 +430,29 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	struct ref *local_refs = head;
 	struct ref **tail = head ? &head->next : &local_refs;
 
-	get_fetch_map(refs, refspec, &tail, 0);
-	if (!option_mirror)
-		get_fetch_map(refs, tag_refspec, &tail, 0);
+	if (!(option_depth && option_single_branch)) {
+		get_fetch_map(refs, refspec, &tail, 0);
+		if (!option_mirror)
+			get_fetch_map(refs, tag_refspec, &tail, 0);
+	} else {
+		struct ref *remote_head = NULL;
+
+		if (!option_branch)
+			remote_head = guess_remote_head(head, refs, 0);
+		else {
+			struct strbuf sb = STRBUF_INIT;
+			strbuf_addstr(&sb, src_ref_prefix);
+			strbuf_addstr(&sb, option_branch);
+			remote_head = find_ref_by_name(refs, sb.buf);
+			strbuf_release(&sb);
+		}
+
+		if (!remote_head)
+			die(_("Remote branch \"%s\" not found. Nothing to clone.\n"
+			      "Try --no-single-branch to fetch all refs."),
+			    option_branch ? option_branch : "HEAD");
+		get_fetch_map(remote_head, refspec, &tail, 0);
+	}
 
 	return local_refs;
 }
@@ -448,6 +471,21 @@ static void write_remote_refs(const struct ref *local_refs)
 	clear_extra_refs();
 }
 
+static void write_followtags(const struct ref *refs, const char *msg)
+{
+	const struct ref *ref;
+	for (ref = refs; ref; ref = ref->next) {
+		if (prefixcmp(ref->name, "refs/tags/"))
+			continue;
+		if (!suffixcmp(ref->name, "^{}"))
+			continue;
+		if (!has_sha1_file(ref->old_sha1))
+			continue;
+		update_ref(msg, ref->name, ref->old_sha1,
+			   NULL, 0, DIE_ON_ERR);
+	}
+}
+
 static int write_one_config(const char *key, const char *value, void *data)
 {
 	return git_config_set_multivar(key, value ? value : "true", "^$", 0);
@@ -478,7 +516,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	struct strbuf key = STRBUF_INIT, value = STRBUF_INIT;
 	struct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;
 	struct transport *transport = NULL;
-	char *src_ref_prefix = "refs/heads/";
 	int err = 0;
 
 	struct refspec *refspec;
@@ -642,9 +679,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 
 		transport_set_option(transport, TRANS_OPT_KEEP, "yes");
 
-		if (option_depth)
+		if (option_depth) {
 			transport_set_option(transport, TRANS_OPT_DEPTH,
 					     option_depth);
+			transport_set_option(transport, TRANS_OPT_FOLLOWTAGS,
+					     option_single_branch ? "1" : NULL);
+		}
 
 		transport_set_verbosity(transport, option_verbosity, option_progress);
 
@@ -663,6 +703,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		clear_extra_refs();
 
 		write_remote_refs(mapped_refs);
+		if (option_depth && option_single_branch)
+			write_followtags(refs, reflog_msg.buf);
 
 		remote_head = find_ref_by_name(refs, "HEAD");
 		remote_head_points_at =
diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index bafcca7..c76a53b 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -115,7 +115,7 @@ pull_to_client 2nd "B" $((64*3))
 pull_to_client 3rd "A" $((1*3))
 
 test_expect_success 'clone shallow' '
-	git clone --depth 2 "file://$(pwd)/." shallow
+	git clone --no-single-branch --depth 2 "file://$(pwd)/." shallow
 '
 
 test_expect_success 'clone shallow object count' '
@@ -248,4 +248,36 @@ test_expect_success 'clone shallow object count' '
 	grep "^count: 52" count.shallow
 '
 
+test_expect_success 'clone shallow without --no-single-branch' '
+	git clone --depth 1 "file://$(pwd)/." shallow2
+'
+
+test_expect_success 'clone shallow object count' '
+	(
+		cd shallow2 &&
+		git count-objects -v
+	) > count.shallow2 &&
+	grep "^in-pack: 6" count.shallow2
+'
+
+test_expect_success 'shallow clone pulling tags' '
+	git tag -a -m A TAGA1 A &&
+	git tag -a -m B TAGB1 B &&
+	git tag TAGA2 A &&
+	git tag TAGB2 B &&
+	git clone --depth 1 "file://$(pwd)/." shallow3 &&
+
+	cat >taglist.expected <<\EOF &&
+TAGB1
+TAGB2
+EOF
+	GIT_DIR=shallow3/.git git tag -l >taglist.actual &&
+	test_cmp taglist.expected taglist.actual &&
+
+	echo "in-pack: 7" > count3.expected &&
+	GIT_DIR=shallow3/.git git count-objects -v |
+		grep "^in-pack" > count3.actual &&
+	test_cmp count3.expected count3.actual
+'
+
 test_done
-- 
1.7.3.1.256.g2539c.dirty

Re: [PATCH v3] Limit refs to fetch to minimum in shallow clones

[Junio C Hamano]

Nguyễn Thái Ngọc Duy <pclouds@gmail.com> writes:

> The main purpose of shallow clones is to reduce download by only
> fetching objects up to a certain depth from the given refs. The number
> of objects depend on how many refs to follow. So:
>
>  - Only fetch HEAD or the ref specified by --branch
>  - Only fetch tags that reference to downloaded objects
>
> More tags/branches can be fetched later using git-fetch as usual.
>
> The old behaviour can still be called with --no-single-branch
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---

Thanks.

> @@ -179,6 +180,15 @@ objects from the source repository into a pack in the cloned repository.
>  	with a long history, and would want to send in fixes
>  	as patches.
>  
> +--single-branch::
> +--no-single-branch::
> +	These options are only valid when --depth is given.
> +	 `--single-branch` only fetches one branch (either HEAD or
> +	specified by --branch) and tags that point to the downloaded
> +	history. `--no-single-branch` fetches all branches and tags
> +	like in normal clones. `--single-branch` is implied by
> +	default.
> +

My first reaction after reading "is implied by default" was "Huh? didn't
we just read these kick in only when --depth is given?" and I had to read
it again. Here is my attempt to rephrase it.

        Clone only the history leading to the tip of a single branch,
        either specified by the `--branch` option or the primary branch
        remote's `HEAD` points at. When creating a shallow clone with the
        `--depth` option, this is the default, unless `--no-single-branch`
        is given to fetch the histories near the tips of all branches.

        Currently this option only works when creating a shallow clone and
        does not have any effect without the `--depth` option.

We might want to later enhance this to work also with a full-depth clone
that tracks only one branch, and the above phrasing would make it clear.

> +		if (!option_branch)
> +			remote_head = guess_remote_head(head, refs, 0);
> +		else {
> +			struct strbuf sb = STRBUF_INIT;
> +			strbuf_addstr(&sb, src_ref_prefix);
> +			strbuf_addstr(&sb, option_branch);
> +			remote_head = find_ref_by_name(refs, sb.buf);
> +			strbuf_release(&sb);
> +		}
> +
> +		if (!remote_head)
> +			die(_("Remote branch \"%s\" not found. Nothing to clone.\n"
> +			      "Try --no-single-branch to fetch all refs."),
> +			    option_branch ? option_branch : "HEAD");

Switching upon option_branch to tweak the message is a good idea, but
strictly speaking, we would hit this die() when guess_remote_head() does
not find where HEAD points at because it is detached, and in that case,
the error is not "Nothing to clone", but "We couldn't tell which branch
you meant to limit this cloning to".

> @@ -642,9 +679,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>  
>  		transport_set_option(transport, TRANS_OPT_KEEP, "yes");
>  
> -		if (option_depth)
> +		if (option_depth) {
>  			transport_set_option(transport, TRANS_OPT_DEPTH,
>  					     option_depth);
> +			transport_set_option(transport, TRANS_OPT_FOLLOWTAGS,
> +					     option_single_branch ? "1" : NULL);

Curious. Does anybody set FOLLOWTAGS to the transport by default becore we
come here (just asking)?

Re: [PATCH v3] Limit refs to fetch to minimum in shallow clones

[Nguyen Thai Ngoc Duy]

2012/1/7 Junio C Hamano <gitster@pobox.com>:
> My first reaction after reading "is implied by default" was "Huh? didn't
> we just read these kick in only when --depth is given?" and I had to read
> it again. Here is my attempt to rephrase it.
>
>        Clone only the history leading to the tip of a single branch,
>        either specified by the `--branch` option or the primary branch
>        remote's `HEAD` points at. When creating a shallow clone with the
>        `--depth` option, this is the default, unless `--no-single-branch`
>        is given to fetch the histories near the tips of all branches.
>
>        Currently this option only works when creating a shallow clone and
>        does not have any effect without the `--depth` option.
>
> We might want to later enhance this to work also with a full-depth clone
> that tracks only one branch, and the above phrasing would make it clear.

Interesting. Yes that's another possibility.

>> +             if (!option_branch)
>> +                     remote_head = guess_remote_head(head, refs, 0);
>> +             else {
>> +                     struct strbuf sb = STRBUF_INIT;
>> +                     strbuf_addstr(&sb, src_ref_prefix);
>> +                     strbuf_addstr(&sb, option_branch);
>> +                     remote_head = find_ref_by_name(refs, sb.buf);
>> +                     strbuf_release(&sb);
>> +             }
>> +
>> +             if (!remote_head)
>> +                     die(_("Remote branch \"%s\" not found. Nothing to clone.\n"
>> +                           "Try --no-single-branch to fetch all refs."),
>> +                         option_branch ? option_branch : "HEAD");
>
> Switching upon option_branch to tweak the message is a good idea, but
> strictly speaking, we would hit this die() when guess_remote_head() does
> not find where HEAD points at because it is detached, and in that case,
> the error is not "Nothing to clone", but "We couldn't tell which branch
> you meant to limit this cloning to".

Yeah. And in detached case without --branch, we probably should not
say anything.

>> @@ -642,9 +679,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>>
>>               transport_set_option(transport, TRANS_OPT_KEEP, "yes");
>>
>> -             if (option_depth)
>> +             if (option_depth) {
>>                       transport_set_option(transport, TRANS_OPT_DEPTH,
>>                                            option_depth);
>> +                     transport_set_option(transport, TRANS_OPT_FOLLOWTAGS,
>> +                                          option_single_branch ? "1" : NULL);
>
> Curious. Does anybody set FOLLOWTAGS to the transport by default becore we
> come here (just asking)?

No, I just hate another "if (option_single_branch)", which indents the
call one more time. Always setting it to "1" in this case should
probably be OK too. I'm just not sure if upload-pack realizes all tags
are requested so include-tag extension means nothing, or it does extra
work for no gain.
-- 
Duy

[PATCH v4] clone: add --single-branch to fetch only one branch

[Nguyễn Thái Ngọc Duy]

When --single-branch is given, only one branch, either HEAD or one
specified by --branch, will be fetched. Also only tags that point to
the downloaded history are fetched.

This helps most in shallow clones, where it can reduce the download to
minimum and that is why it is enabled by default when --depth is given.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 The fourth round does not limit --single-branch to shallow clones any
 more. I did not see that the patch finally comes close to what Carlos
 tried to do [1], the patch that reminded me of the tag issue in shallow
 clone Shawn mentioned a while ago.

 [1] http://thread.gmane.org/gmane.comp.version-control.git/187639

 Documentation/git-clone.txt |   11 ++++++-
 builtin/clone.c             |   52 ++++++++++++++++++++++++++++--
 t/t5500-fetch-pack.sh       |   72 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 129 insertions(+), 6 deletions(-)

diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt
index 4b8b26b..0931a3e 100644
--- a/Documentation/git-clone.txt
+++ b/Documentation/git-clone.txt
@@ -13,7 +13,8 @@ SYNOPSIS
 	  [-l] [-s] [--no-hardlinks] [-q] [-n] [--bare] [--mirror]
 	  [-o <name>] [-b <name>] [-u <upload-pack>] [--reference <repository>]
 	  [--separate-git-dir <git dir>]
-	  [--depth <depth>] [--recursive|--recurse-submodules] [--] <repository>
+	  [--depth <depth>] [--[no-]single-branch]
+	  [--recursive|--recurse-submodules] [--] <repository>
 	  [<directory>]
 
 DESCRIPTION
@@ -179,6 +180,14 @@ objects from the source repository into a pack in the cloned repository.
 	with a long history, and would want to send in fixes
 	as patches.
 
+--single-branch::
+	Clone only the history leading to the tip of a single branch,
+	either specified by the `--branch` option or the primary
+	branch remote's `HEAD` points at. When creating a shallow
+	clone with the `--depth` option, this is the default, unless
+	`--no-single-branch` is given to fetch the histories near the
+	tips of all branches.
+
 --recursive::
 --recurse-submodules::
 	After the clone is created, initialize all submodules within,
diff --git a/builtin/clone.c b/builtin/clone.c
index 86db954..9dcc5fe 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -37,7 +37,7 @@ static const char * const builtin_clone_usage[] = {
 	NULL
 };
 
-static int option_no_checkout, option_bare, option_mirror;
+static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;
 static int option_local, option_no_hardlinks, option_shared, option_recursive;
 static char *option_template, *option_depth;
 static char *option_origin = NULL;
@@ -48,6 +48,7 @@ static int option_verbosity;
 static int option_progress;
 static struct string_list option_config;
 static struct string_list option_reference;
+static const char *src_ref_prefix = "refs/heads/";
 
 static int opt_parse_reference(const struct option *opt, const char *arg, int unset)
 {
@@ -92,6 +93,8 @@ static struct option builtin_clone_options[] = {
 		   "path to git-upload-pack on the remote"),
 	OPT_STRING(0, "depth", &option_depth, "depth",
 		    "create a shallow clone of that depth"),
+	OPT_BOOL(0, "single-branch", &option_single_branch,
+		    "clone only one branch, HEAD or --branch"),
 	OPT_STRING(0, "separate-git-dir", &real_git_dir, "gitdir",
 		   "separate git dir from working tree"),
 	OPT_STRING_LIST('c', "config", &option_config, "key=value",
@@ -427,8 +430,28 @@ static struct ref *wanted_peer_refs(const struct ref *refs,
 	struct ref *local_refs = head;
 	struct ref **tail = head ? &head->next : &local_refs;
 
-	get_fetch_map(refs, refspec, &tail, 0);
-	if (!option_mirror)
+	if (option_single_branch) {
+		struct ref *remote_head = NULL;
+
+		if (!option_branch)
+			remote_head = guess_remote_head(head, refs, 0);
+		else {
+			struct strbuf sb = STRBUF_INIT;
+			strbuf_addstr(&sb, src_ref_prefix);
+			strbuf_addstr(&sb, option_branch);
+			remote_head = find_ref_by_name(refs, sb.buf);
+			strbuf_release(&sb);
+		}
+
+		if (!remote_head && option_branch)
+			warning(_("Could not find remote branch %s to clone."),
+				option_branch);
+		else
+			get_fetch_map(remote_head, refspec, &tail, 0);
+	} else
+		get_fetch_map(refs, refspec, &tail, 0);
+
+	if (!option_mirror && !option_single_branch)
 		get_fetch_map(refs, tag_refspec, &tail, 0);
 
 	return local_refs;
@@ -448,6 +471,21 @@ static void write_remote_refs(const struct ref *local_refs)
 	clear_extra_refs();
 }
 
+static void write_followtags(const struct ref *refs, const char *msg)
+{
+	const struct ref *ref;
+	for (ref = refs; ref; ref = ref->next) {
+		if (prefixcmp(ref->name, "refs/tags/"))
+			continue;
+		if (!suffixcmp(ref->name, "^{}"))
+			continue;
+		if (!has_sha1_file(ref->old_sha1))
+			continue;
+		update_ref(msg, ref->name, ref->old_sha1,
+			   NULL, 0, DIE_ON_ERR);
+	}
+}
+
 static int write_one_config(const char *key, const char *value, void *data)
 {
 	return git_config_set_multivar(key, value ? value : "true", "^$", 0);
@@ -478,7 +516,6 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	struct strbuf key = STRBUF_INIT, value = STRBUF_INIT;
 	struct strbuf branch_top = STRBUF_INIT, reflog_msg = STRBUF_INIT;
 	struct transport *transport = NULL;
-	char *src_ref_prefix = "refs/heads/";
 	int err = 0;
 
 	struct refspec *refspec;
@@ -498,6 +535,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		usage_msg_opt(_("You must specify a repository to clone."),
 			builtin_clone_usage, builtin_clone_options);
 
+	if (option_single_branch == -1)
+		option_single_branch = option_depth ? 1 : 0;
+
 	if (option_mirror)
 		option_bare = 1;
 
@@ -645,6 +685,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		if (option_depth)
 			transport_set_option(transport, TRANS_OPT_DEPTH,
 					     option_depth);
+		if (option_single_branch)
+			transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, "1");
 
 		transport_set_verbosity(transport, option_verbosity, option_progress);
 
@@ -663,6 +705,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 		clear_extra_refs();
 
 		write_remote_refs(mapped_refs);
+		if (option_single_branch)
+			write_followtags(refs, reflog_msg.buf);
 
 		remote_head = find_ref_by_name(refs, "HEAD");
 		remote_head_points_at =
diff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh
index 9bf69e9..7e85c71 100755
--- a/t/t5500-fetch-pack.sh
+++ b/t/t5500-fetch-pack.sh
@@ -114,8 +114,19 @@ pull_to_client 2nd "refs/heads/B" $((64*3))
 
 pull_to_client 3rd "refs/heads/A" $((1*3))
 
+test_expect_success 'single branch clone' '
+	git clone --single-branch "file://$(pwd)/." singlebranch
+'
+
+test_expect_success 'single branch object count' '
+	GIT_DIR=singlebranch/.git git count-objects -v |
+		grep "^in-pack:" > count.singlebranch &&
+	echo "in-pack: 198" >expected &&
+	test_cmp expected count.singlebranch
+'
+
 test_expect_success 'clone shallow' '
-	git clone --depth 2 "file://$(pwd)/." shallow
+	git clone --no-single-branch --depth 2 "file://$(pwd)/." shallow
 '
 
 test_expect_success 'clone shallow object count' '
@@ -248,4 +259,63 @@ test_expect_success 'clone shallow object count' '
 	grep "^count: 52" count.shallow
 '
 
+test_expect_success 'clone shallow without --no-single-branch' '
+	git clone --depth 1 "file://$(pwd)/." shallow2
+'
+
+test_expect_success 'clone shallow object count' '
+	(
+		cd shallow2 &&
+		git count-objects -v
+	) > count.shallow2 &&
+	grep "^in-pack: 6" count.shallow2
+'
+
+test_expect_success 'clone shallow with --branch' '
+	git clone --depth 1 --branch A "file://$(pwd)/." shallow3
+'
+
+test_expect_success 'clone shallow object count' '
+	echo "in-pack: 12" > count3.expected &&
+	GIT_DIR=shallow3/.git git count-objects -v |
+		grep "^in-pack" > count3.actual &&
+	test_cmp count3.expected count3.actual
+'
+
+test_expect_success 'clone shallow with nonexistent --branch' '
+	git clone --depth 1 --branch Z "file://$(pwd)/." shallow4 &&
+	GIT_DIR=shallow4/.git git rev-parse HEAD >actual &&
+	git rev-parse HEAD >expected &&
+	test_cmp expected actual
+'
+
+test_expect_success 'clone shallow with detached HEAD' '
+	git checkout HEAD^ &&
+	git clone --depth 1 "file://$(pwd)/." shallow5 &&
+	git checkout - &&
+	GIT_DIR=shallow5/.git git rev-parse HEAD >actual &&
+	git rev-parse HEAD^ >expected &&
+	test_cmp expected actual
+'
+
+test_expect_success 'shallow clone pulling tags' '
+	git tag -a -m A TAGA1 A &&
+	git tag -a -m B TAGB1 B &&
+	git tag TAGA2 A &&
+	git tag TAGB2 B &&
+	git clone --depth 1 "file://$(pwd)/." shallow6 &&
+
+	cat >taglist.expected <<\EOF &&
+TAGB1
+TAGB2
+EOF
+	GIT_DIR=shallow6/.git git tag -l >taglist.actual &&
+	test_cmp taglist.expected taglist.actual &&
+
+	echo "in-pack: 7" > count6.expected &&
+	GIT_DIR=shallow6/.git git count-objects -v |
+		grep "^in-pack" > count6.actual &&
+	test_cmp count6.expected count6.actual
+'
+
 test_done
-- 
1.7.8.36.g69ee2