[PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Shawn O. Pearce]

If autofollowing of tags is enabled, we see a new tag on the remote
that we don't have, and we already have the SHA-1 object that the
tag is peeled to, then we can fetch the tag while we are fetching
the other objects on the first connection.

This is a slight optimization for projects that have a habit of
tagging a release commit after most users have already seen and
downloaded that commit object through a prior fetch session. In
such cases the users may still find new objects in branch heads,
but the new tag will now also be part of the first pack transfer
and the subsequent connection to autofollow tags is not required.

Currently git.git does not benefit from this optimization as any
release usually gets a new commit at the same time that it gets a
new release tag, however git-gui.git and many other projects are
in the habit of tagging fairly old commits.

Users who did not already have the tagged commit still require
opening a second connection to autofollow the tag, as we are unable
to determine on the client side if $tag^{} will be sent to the
client during the first transfer or not.  Such computation must be
performed on the remote side of the connection and is deferred to
another series of changes.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 builtin-fetch.c |    6 ++++++
 1 files changed, 6 insertions(+), 0 deletions(-)

diff --git a/builtin-fetch.c b/builtin-fetch.c
index 834fbc6..a80f95b 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -101,6 +101,10 @@ static void add_merge_config(struct ref **head,
 	}
 }
 
+static void find_non_local_tags(struct transport *transport,
+			struct ref **head,
+			struct ref ***tail);
+
 static struct ref *get_ref_map(struct transport *transport,
 			       struct refspec *refs, int ref_count, int tags,
 			       int *autotags)
@@ -159,6 +163,8 @@ static struct ref *get_ref_map(struct transport *transport,
 			ref_map->merge = 1;
 		}
 	}
+	if (tags == TAGS_DEFAULT && autotags)
+		find_non_local_tags(transport, &ref_map, &tail);
 	ref_remove_duplicates(ref_map);
 
 	return ref_map;
-- 
1.5.4.3.393.g5540

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Junio C Hamano]

This is cute.  Obviously some tests need to be adjusted for this
change, though.

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Shawn O. Pearce]

Junio C Hamano <gitster@pobox.com> wrote:
> This is cute.  Obviously some tests need to be adjusted for this
> change, though.

I'll take a look at the current tests this weekend and see what
needs to be adjusted, if anything.  I'd also like to get a few
tests written for this, so we are certain the optimizations are
kicking in when they are supposed to be.

-- 
Shawn.

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Daniel Barkalow]

On Fri, 29 Feb 2008, Shawn O. Pearce wrote:

> Junio C Hamano <gitster@pobox.com> wrote:
> > This is cute.  Obviously some tests need to be adjusted for this
> > change, though.
> 
> I'll take a look at the current tests this weekend and see what
> needs to be adjusted, if anything.  I'd also like to get a few
> tests written for this, so we are certain the optimizations are
> kicking in when they are supposed to be.

I'd be really grateful if you came up with a good general strategy for 
testing that we're not doing too much work in fetching, because clone has 
optimizations that I need to test in a similar way, and I haven't been 
able to think of anything not horribly intrusive.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Shawn O. Pearce]

Daniel Barkalow <barkalow@iabervon.org> wrote:
> On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
> 
> > Junio C Hamano <gitster@pobox.com> wrote:
> > > This is cute.  Obviously some tests need to be adjusted for this
> > > change, though.
> > 
> > I'll take a look at the current tests this weekend and see what
> > needs to be adjusted, if anything.  I'd also like to get a few
> > tests written for this, so we are certain the optimizations are
> > kicking in when they are supposed to be.
> 
> I'd be really grateful if you came up with a good general strategy for 
> testing that we're not doing too much work in fetching, because clone has 
> optimizations that I need to test in a similar way, and I haven't been 
> able to think of anything not horribly intrusive.

Is this "horribly intrusive" ?

We can test it with something like this:

  $ GIT_DEBUG_SEND_PACK=1 git fetch 3>UPLOAD_LOG
  $ cat UPLOAD_LOG
  #S
  want 8e10cf4e007ad7e003463c30c34b1050b039db78 multi_ack side-band-64k thin-pack ofs-delta
  want ddfa4a33562179aca1ace2bcc662244a17d0b503
  #E
  #S
  want 3253df4d1cf6fb138b52b1938473bcfec1483223 multi_ack side-band-64k thin-pack ofs-delta
  #E

Notice we made two connections, wanting 2 things and then later
only wanting 1 thing.  Tag auto-following.  :-)

diff --git a/upload-pack.c b/upload-pack.c
index b26d053..4e14020 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -444,7 +444,10 @@ static void receive_needs(void)
 	struct object_array shallows = {0, 0, NULL};
 	static char line[1000];
 	int len, depth = 0;
+	int debug_needs = !!getenv("GIT_DEBUG_SEND_PACK");
 
+	if (debug_needs)
+		write_in_full(3, "#S\n", 3);
 	for (;;) {
 		struct object *o;
 		unsigned char sha1_buf[20];
@@ -452,6 +455,8 @@ static void receive_needs(void)
 		reset_timeout();
 		if (!len)
 			break;
+		if (debug_needs)
+			write_in_full(3, line, len);
 
 		if (!prefixcmp(line, "shallow ")) {
 			unsigned char sha1[20];
@@ -507,6 +512,8 @@ static void receive_needs(void)
 			add_object_array(o, NULL, &want_obj);
 		}
 	}
+	if (debug_needs)
+		write_in_full(3, "#E\n", 3);
 	if (depth == 0 && shallows.nr == 0)
 		return;
 	if (depth > 0) {

-- 
Shawn.

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Daniel Barkalow]

On Fri, 29 Feb 2008, Shawn O. Pearce wrote:

> Daniel Barkalow <barkalow@iabervon.org> wrote:
> > On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
> > 
> > > Junio C Hamano <gitster@pobox.com> wrote:
> > > > This is cute.  Obviously some tests need to be adjusted for this
> > > > change, though.
> > > 
> > > I'll take a look at the current tests this weekend and see what
> > > needs to be adjusted, if anything.  I'd also like to get a few
> > > tests written for this, so we are certain the optimizations are
> > > kicking in when they are supposed to be.
> > 
> > I'd be really grateful if you came up with a good general strategy for 
> > testing that we're not doing too much work in fetching, because clone has 
> > optimizations that I need to test in a similar way, and I haven't been 
> > able to think of anything not horribly intrusive.
> 
> Is this "horribly intrusive" ?
> 
> We can test it with something like this:

That looks suitable to me (although maybe the environment variable should 
contain the fd to write to?). I also want to know the "have" info, so I 
can verify that the client is listing things appropriately, but that's an 
obvious addition.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 3/4] Make git-fetch follow tags we already have objects for sooner

[Shawn O. Pearce]

Daniel Barkalow <barkalow@iabervon.org> wrote:
> On Fri, 29 Feb 2008, Shawn O. Pearce wrote:
> > 
> > Is this "horribly intrusive" ?
> > 
> > We can test it with something like this:
> 
> That looks suitable to me (although maybe the environment variable should 
> contain the fd to write to?). I also want to know the "have" info, so I 
> can verify that the client is listing things appropriately, but that's an 
> obvious addition.

Yea, I was thinking about that last night myself.  I think the patch
is still in `pu` so I will work up something for Junio to squash into
it that uses the variable to specify the fd, rather than assume 3.

-- 
Shawn.