[PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Shawn O. Pearce]

The new option "--auto-follow-tags" allows the caller to request that
any annotated tag be included into the packfile if the object the tag
references was also included as part of the packfile.

This option can be useful on the server side of a native git transport,
where the server knows what commits it is including into a packfile to
update the client.  If new annotated tags have been introduced then we
can also include them in the packfile, saving the client from needing
to request them through a second connection.

This change only introduces the backend option and provides a test.
Protocol extensions to make this useful in fetch-pack/upload-pack
are still necessary to activate the logic during transport.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 Documentation/git-pack-objects.txt |    5 ++
 builtin-pack-objects.c             |   24 +++++++++-
 t/t5305-autofollow-tag.sh          |   84 ++++++++++++++++++++++++++++++++++++
 3 files changed, 111 insertions(+), 2 deletions(-)
 create mode 100755 t/t5305-autofollow-tag.sh

diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt
index 5c1bd3b..0ba9cbe 100644
--- a/Documentation/git-pack-objects.txt
+++ b/Documentation/git-pack-objects.txt
@@ -170,6 +170,11 @@ base-name::
 	length, this option typically shrinks the resulting
 	packfile by 3-5 per-cent.
 
+--auto-follow-tags::
+	Automatically include annotated tags if the object they
+	reference was included in the resulting packfile.  This
+	can be useful to send new tags to native git clients.
+
 --threads=<n>::
 	Specifies the number of threads to spawn when searching for best
 	delta matches.  This requires that pack-objects be compiled with
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 2799e68..8b5d90a 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -15,6 +15,7 @@
 #include "revision.h"
 #include "list-objects.h"
 #include "progress.h"
+#include "refs.h"
 
 #ifdef THREADED_DELTA_SEARCH
 #include "thread-utils.h"
@@ -27,7 +28,8 @@ git-pack-objects [{ -q | --progress | --all-progress }] \n\
 	[--window=N] [--window-memory=N] [--depth=N] \n\
 	[--no-reuse-delta] [--no-reuse-object] [--delta-base-offset] \n\
 	[--threads=N] [--non-empty] [--revs [--unpacked | --all]*] [--reflog] \n\
-	[--stdout | base-name] [--keep-unreachable] [<ref-list | <object-list]";
+	[--stdout | base-name] [--auto-follow-tags] [--keep-unreachable] \n\
+	[<ref-list | <object-list]";
 
 struct object_entry {
 	struct pack_idx_entry idx;
@@ -63,7 +65,7 @@ static struct pack_idx_entry **written_list;
 static uint32_t nr_objects, nr_alloc, nr_result, nr_written;
 
 static int non_empty;
-static int no_reuse_delta, no_reuse_object, keep_unreachable;
+static int no_reuse_delta, no_reuse_object, keep_unreachable, auto_tags;
 static int local;
 static int incremental;
 static int allow_ofs_delta;
@@ -1630,6 +1632,18 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,
 #define ll_find_deltas(l, s, w, d, p)	find_deltas(l, &s, w, d, p)
 #endif
 
+static int add_ref_tag(const char *path, const unsigned char *sha1, int flag, void *cb_data)
+{
+	unsigned char peeled[20];
+
+	if (!prefixcmp(path, "refs/tags/") && /* is a tag? */
+	    !peel_ref(path, peeled)        && /* peelable? */
+	    !is_null_sha1(peeled)          && /* annotated tag? */
+	    locate_object_entry(peeled))      /* object packed? */
+		add_object_entry(sha1, OBJ_TAG, NULL, 0);
+	return 0;
+}
+
 static void prepare_pack(int window, int depth)
 {
 	struct object_entry **delta_list;
@@ -2033,6 +2047,10 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 			keep_unreachable = 1;
 			continue;
 		}
+		if (!strcmp("--auto-follow-tags", arg)) {
+			auto_tags = 1;
+			continue;
+		}
 		if (!strcmp("--unpacked", arg) ||
 		    !prefixcmp(arg, "--unpacked=") ||
 		    !strcmp("--reflog", arg) ||
@@ -2109,6 +2127,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		rp_av[rp_ac] = NULL;
 		get_object_list(rp_ac, rp_av);
 	}
+	if (auto_tags && nr_result)
+		for_each_ref(add_ref_tag, NULL);
 	stop_progress(&progress_state);
 
 	if (non_empty && !nr_result)
diff --git a/t/t5305-autofollow-tag.sh b/t/t5305-autofollow-tag.sh
new file mode 100755
index 0000000..3d327ee
--- /dev/null
+++ b/t/t5305-autofollow-tag.sh
@@ -0,0 +1,84 @@
+#!/bin/sh
+
+test_description='git-pack-object --auto-follow-tags'
+. ./test-lib.sh
+
+TRASH=`pwd`
+
+test_expect_success setup '
+	echo c >d &&
+	git update-index --add d &&
+	tree=`git write-tree` &&
+	commit=`git commit-tree $tree </dev/null` &&
+	echo "object $commit" >sig &&
+	echo "type commit" >>sig &&
+	echo "tag mytag" >>sig &&
+	echo "tagger $(git var GIT_COMMITTER_IDENT)" >>sig &&
+	echo >>sig &&
+	echo "our test tag" >>sig &&
+	tag=`git mktag <sig` &&
+	rm d sig &&
+	git update-ref refs/tags/mytag $tag && {
+		echo $tree &&
+		echo $commit &&
+		git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\)	.*/\\1/"
+	} >obj-list
+'
+
+rm -rf clone.git
+test_expect_success 'pack without --auto-follow-tags' '
+	packname_1=$(git pack-objects \
+		--window=0 \
+		test-1 <obj-list)
+'
+
+test_expect_success 'unpack objects' '
+	(
+		GIT_DIR=clone.git &&
+		export GIT_DIR &&
+		git init &&
+		git unpack-objects -n <test-1-${packname_1}.pack &&
+		git unpack-objects <test-1-${packname_1}.pack
+	)
+'
+
+test_expect_success 'check unpacked result (have commit, no tag)' '
+	git rev-list --objects $commit >list.expect &&
+	(
+		GIT_DIR=clone.git &&
+		export GIT_DIR &&
+		test_must_fail git cat-file -e $tag &&
+		git rev-list --objects $commit
+	) >list.actual &&
+	git diff list.expect list.actual
+'
+
+rm -rf clone.git
+test_expect_success 'pack with --auto-follow-tags' '
+	packname_1=$(git pack-objects \
+		--window=0 \
+		--auto-follow-tags \
+		test-2 <obj-list)
+'
+
+test_expect_success 'unpack objects' '
+	(
+		GIT_DIR=clone.git &&
+		export GIT_DIR &&
+		git init &&
+		git unpack-objects -n <test-2-${packname_1}.pack &&
+		git unpack-objects <test-2-${packname_1}.pack
+	)
+'
+
+test_expect_success 'check unpacked result (have commit, have tag)' '
+	git rev-list --objects mytag >list.expect &&
+	(
+		GIT_DIR=clone.git &&
+		export GIT_DIR &&
+		git rev-list --objects $tag
+	) >list.actual &&
+	git diff list.expect list.actual
+'
+
+test_done
-- 
1.5.4.3.529.gb25fb

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Nicolas Pitre]

On Mon, 3 Mar 2008, Shawn O. Pearce wrote:

> The new option "--auto-follow-tags" allows the caller to request that
> any annotated tag be included into the packfile if the object the tag
> references was also included as part of the packfile.

Wouldn't "auto-include-tag" a better name for this option?


Nicolas

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Daniel Barkalow]

On Mon, 3 Mar 2008, Shawn O. Pearce wrote:

> +	if (auto_tags && nr_result)
> +		for_each_ref(add_ref_tag, NULL);

There's for_each_tag_ref() that does the path-based filter internally, in 
a possibly more efficient way, and avoids open-coding the test for whether 
this is a tag sort of ref.

Aside from that, it looks good to me.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Shawn O. Pearce]

Daniel Barkalow <barkalow@iabervon.org> wrote:
> On Mon, 3 Mar 2008, Shawn O. Pearce wrote:
> 
> > +	if (auto_tags && nr_result)
> > +		for_each_ref(add_ref_tag, NULL);
> 
> There's for_each_tag_ref() that does the path-based filter internally, in 
> a possibly more efficient way, and avoids open-coding the test for whether 
> this is a tag sort of ref.

Yea, I know about that call, but I didn't use it because I was
trying to use peel_ref() for its optimized peeled caching in
a packed-refs file.

To use peel_ref() you have to pass the full ref name ("refs/tags/v1.0")
and not a partial ref name ("v1.0") such as for_each_tag_ref() will
return to the callback function.  So I'd have to actually do extra
work to reconstruct a ref name, and possibly an invalid ref name at
that if for_each_tag_ref() ever looked outside of the refs/tags/
namespace.

Today there is no performance benefit to for_each_tag_ref() over
for_each_ref(), but using for_each_ref() lets me use peel_ref(),
which is a performance improvement if the ref in question came from
the packed-refs file.

So.... do we keep things as-is or try to optimize peel_ref() (or
some new variant of it?!?) for use in for_each_tag_ref() just on
the off chance that we somehow can optimize for_each_tag_ref to
avoid scanning refs/heads/?  Note that I did look into trying to
do that optimization to for_each_tag_ref() earlier last week and
concluded it wasn't easily possible right now, and wasn't likely
to be worth the development costs.

-- 
Shawn.

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Shawn O. Pearce]

Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 3 Mar 2008, Shawn O. Pearce wrote:
> 
> > The new option "--auto-follow-tags" allows the caller to request that
> > any annotated tag be included into the packfile if the object the tag
> > references was also included as part of the packfile.
> 
> Wouldn't "auto-include-tag" a better name for this option?

Ooooh.  Indeed it would!

I shall respin this series based around this better name.

-- 
Shawn.

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Daniel Barkalow]

On Mon, 3 Mar 2008, Shawn O. Pearce wrote:

> Daniel Barkalow <barkalow@iabervon.org> wrote:
> > On Mon, 3 Mar 2008, Shawn O. Pearce wrote:
> > 
> > > +	if (auto_tags && nr_result)
> > > +		for_each_ref(add_ref_tag, NULL);
> > 
> > There's for_each_tag_ref() that does the path-based filter internally, in 
> > a possibly more efficient way, and avoids open-coding the test for whether 
> > this is a tag sort of ref.
> 
> Yea, I know about that call, but I didn't use it because I was
> trying to use peel_ref() for its optimized peeled caching in
> a packed-refs file.

Ah, okay. I hadn't noticed that it didn't pass on the full name. Using 
for_each_ref() is better, then, I think.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Johannes Sixt]

Shawn O. Pearce schrieb:
> Nicolas Pitre <nico@cam.org> wrote:
>> On Mon, 3 Mar 2008, Shawn O. Pearce wrote:
>>
>>> The new option "--auto-follow-tags" allows the caller to request that
>>> any annotated tag be included into the packfile if the object the tag
>>> references was also included as part of the packfile.
>> Wouldn't "auto-include-tag" a better name for this option?
> 
> Ooooh.  Indeed it would!
> 
> I shall respin this series based around this better name.

More bikeshedding: Why are you calling it "auto-"? If I tell the program
to "--include-tags", I expect it to happen automatically.

-- Hannes

Re: [PATCH 1/3] git-pack-objects: Automatically pack annotated tags if object was packed

[Shawn O. Pearce]

Johannes Sixt <j.sixt@viscovery.net> wrote:
> Shawn O. Pearce schrieb:
> > Nicolas Pitre <nico@cam.org> wrote:
> >> On Mon, 3 Mar 2008, Shawn O. Pearce wrote:
> >>
> >>> The new option "--auto-follow-tags" allows the caller to request that
> >>> any annotated tag be included into the packfile if the object the tag
> >>> references was also included as part of the packfile.
> >> Wouldn't "auto-include-tag" a better name for this option?
> > 
> > Ooooh.  Indeed it would!
> > 
> > I shall respin this series based around this better name.
> 
> More bikeshedding: Why are you calling it "auto-"? If I tell the program
> to "--include-tags", I expect it to happen automatically.

Because its more automatic than manually including tags?
I don't know.

Junio, if you haven't already merged this topic into someplace that
I can't rewrite the series for a _3rd_ time, I might be willing to
respin it without the "auto-" prefix.  We are unlikely to ever have
a reason to support "--manually-include-tags".  Stone tablet based
computing stopped being fun back in the '50s.

-- 
Shawn.