[PATCH 0/4] Fix gc failure when a remote HEAD goes stale

[PATCH 0/4] Fix gc failure when a remote HEAD goes stale

[Johannes Schindelin]

There has been a report in the Git for Windows project that gc fails
sometimes: https://github.com/git-for-windows/git/issues/423

It turns out that there are cases when a remote HEAD can go stale and
it is not the user's fault at all. It can happen, for example, if the
active branch in the remote repository gets renamed.

Git's garbage collector should handle this gracefully. The best this
developer could come up with, is to simply ignore and delete the
now-broken refs.


Johannes Schindelin (4):
  gc: demonstrate failure with stale remote HEAD
  pack-objects: do not get distracted by stale refs
  mark_reachable_objects(): optionally collect broken refs
  gc: remove broken refs

 builtin/pack-objects.c |  1 +
 builtin/prune.c        | 12 +++++++++++-
 builtin/reflog.c       |  2 +-
 reachable.c            | 26 ++++++++++++++++++++------
 reachable.h            |  3 ++-
 t/t6500-gc.sh          | 15 +++++++++++++++
 6 files changed, 50 insertions(+), 9 deletions(-)

-- 
2.5.2.windows.2

[PATCH 1/4] gc: demonstrate failure with stale remote HEAD

[Johannes Schindelin]

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 63194d8..b736774 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,4 +30,19 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
+test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+	git init remote &&
+	(
+		cd remote &&
+		test_commit initial &&
+		git clone . ../client &&
+		git branch -m develop &&
+		cd ../client &&
+		git fetch --prune &&
+		git gc &&
+		git branch --list -r origin/HEAD >actual &&
+		test_line_count = 0 actual
+	)
+'
+
 test_done
-- 
2.5.2.windows.2

[PATCH 2/4] pack-objects: do not get distracted by stale refs

[Johannes Schindelin]

It is quite possible for, say, a remote HEAD to become stale, e.g. when
the default branch was renamed.

We should still be able to pack our objects when such a thing happens;
simply ignore invalid refs (because they cannot matter for the packing
process anyway).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/pack-objects.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 1c63f8f..ef2f794 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -2499,6 +2499,7 @@ static void get_object_list(int ac, const char **av)
 	int flags = 0;
 
 	init_revisions(&revs, NULL);
+	revs.ignore_missing = 1;
 	save_commit_buffer = 0;
 	setup_revisions(ac, av, &revs, NULL);
 
-- 
2.5.2.windows.2

Re: [PATCH 2/4] pack-objects: do not get distracted by stale refs

[Jeff King]

On Thu, Sep 24, 2015 at 11:13:44AM +0200, Johannes Schindelin wrote:

> It is quite possible for, say, a remote HEAD to become stale, e.g. when
> the default branch was renamed.
> 
> We should still be able to pack our objects when such a thing happens;
> simply ignore invalid refs (because they cannot matter for the packing
> process anyway).
> [...]
>  	init_revisions(&revs, NULL);
> +	revs.ignore_missing = 1;

I think this is dangerous, because a repack may drop unreferenced
history. Imagine you have a corrupted repository where you are missing
the commit at the tip of "master", but still have "master^" and the rest
of the history, and "git gc --auto" triggers.

Right now, pack-objects will barf and refuse to pack. Your repo is still
corrupted, but you can use git-fsck to recover the history minus the
single commit. With your patch, the repack will delete the entire
history. Of course there are complications like other branches
referencing the history, reflogs if it is a non-bare repo, and the
2-week expiration time. But there's definitely a potential to make
things worse. I think commands that drop objects (like repack and prune)
need to be more careful than other commands about refusing to run when
there is something fishy in the repository.

I think we would want to introduce a flag to ignore dangling
symbolic refs when running for_each_ref, as those should be harmless. In
fact, I think we could omit symrefs completely for reachability
analysis; we would hit the ref they refer to anyway during the rest of
the traversal.

-Peff

[PATCH 3/4] mark_reachable_objects(): optionally collect broken refs

[Johannes Schindelin]

The behavior of `mark_reachable_objects()` without this patch is that it
dies if it encounters a broken ref. This is sometimes undesirable, e.g.
when garbage collecting in a repository with a stale remote HEAD.

So let's introduce an optional parameter to collect such broken refs. The
behavior of the function is unchanged if that parameter is `NULL`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c  |  2 +-
 builtin/reflog.c |  2 +-
 reachable.c      | 26 ++++++++++++++++++++------
 reachable.h      |  3 ++-
 4 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/builtin/prune.c b/builtin/prune.c
index 10b03d3..d6f664f 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -136,7 +136,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress);
+	mark_reachable_objects(&revs, 1, expire, progress, NULL);
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/builtin/reflog.c b/builtin/reflog.c
index f96ca2a..cb8758a 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -583,7 +583,7 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)
 		init_revisions(&cb.cmd.revs, prefix);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			printf("Marking reachable objects...");
-		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL);
+		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL, NULL);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			putchar('\n');
 	}
diff --git a/reachable.c b/reachable.c
index 9cff25b..1fc7ada 100644
--- a/reachable.c
+++ b/reachable.c
@@ -15,6 +15,11 @@ struct connectivity_progress {
 	unsigned long count;
 };
 
+struct add_one_data {
+	struct rev_info *revs;
+	struct string_list *broken_refs;
+};
+
 static void update_progress(struct connectivity_progress *cp)
 {
 	cp->count++;
@@ -25,10 +30,14 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct object *object = parse_object_or_die(oid->hash, path);
-	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct add_one_data *data = (struct add_one_data *)cb_data;
+	struct object *object = data->broken_refs ? parse_object(oid->hash) :
+		parse_object_or_die(oid->hash, path);
 
-	add_pending_object(revs, object, "");
+	if (!object)
+		string_list_append(data->broken_refs, path);
+	else
+		add_pending_object(data->revs, object, "");
 
 	return 0;
 }
@@ -153,9 +162,11 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
-			    struct progress *progress)
+			    struct progress *progress,
+			    struct string_list *broken_refs)
 {
 	struct connectivity_progress cp;
+	struct add_one_data data;
 
 	/*
 	 * Set up revision parsing, and mark us as being interested
@@ -168,11 +179,14 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	/* Add all refs from the index file */
 	add_index_objects_to_pending(revs, 0);
 
+	data.revs = revs;
+	data.broken_refs = broken_refs;
+
 	/* Add all external refs */
-	for_each_ref(add_one_ref, revs);
+	for_each_ref(add_one_ref, &data);
 
 	/* detached HEAD is not included in the list above */
-	head_ref(add_one_ref, revs);
+	head_ref(add_one_ref, &data);
 
 	/* Add all reflog info */
 	if (mark_reflog)
diff --git a/reachable.h b/reachable.h
index d23efc3..39de1c7 100644
--- a/reachable.h
+++ b/reachable.h
@@ -5,6 +5,7 @@ struct progress;
 extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
-				   unsigned long mark_recent, struct progress *);
+				   unsigned long mark_recent, struct progress *,
+				   struct string_list *broken_refs);
 
 #endif
-- 
2.5.2.windows.2

Re: [PATCH 3/4] mark_reachable_objects(): optionally collect broken refs

[Jeff King]

On Thu, Sep 24, 2015 at 11:13:52AM +0200, Johannes Schindelin wrote:

> The behavior of `mark_reachable_objects()` without this patch is that it
> dies if it encounters a broken ref. This is sometimes undesirable, e.g.
> when garbage collecting in a repository with a stale remote HEAD.
> 
> So let's introduce an optional parameter to collect such broken refs. The
> behavior of the function is unchanged if that parameter is `NULL`.

Similar comment to the last one. :)

I suspect the issues you are seeing are largely new due to the
ref-paranoia work I did (merged in 05e816e37). We used to ignore broken
refs at the for_each_ref() level, but now we feed them to the calling
code (which generally chokes).

So in that sense, a simpler fix than your series would be to simply
revert 8d42299 and ff4056bbc. :)

But I think those new checks are valuable, and we really just need to
gracefully ignore the dangling-symref case (which is _a_ breakage, but
not a dangerous one).

-Peff

[PATCH 4/4] gc: remove broken refs

[Johannes Schindelin]

When encountering broken refs, such as a stale remote HEAD (which can
happen if the active branch was renamed in the remote), it is more
helpful to remove those refs than to exit with an error.

This fixes https://github.com/git-for-windows/git/issues/423

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c | 12 +++++++++++-
 t/t6500-gc.sh   |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/builtin/prune.c b/builtin/prune.c
index d6f664f..1a30f65 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,6 +6,7 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
+#include "refs.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
@@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
+	struct string_list broken_refs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 	char *s;
+	int i;
 
 	expire = ULONG_MAX;
 	save_commit_buffer = 0;
@@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress, NULL);
+	revs.ignore_missing = 1;
+	mark_reachable_objects(&revs, 1, expire, progress, &broken_refs);
+	for (i = 0; i < broken_refs.nr; i++) {
+		char *path = broken_refs.items[i].string;
+		printf("Removing stale ref %s\n", path);
+		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
+			die("Could not remove stale ref %s", path);
+	}
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index b736774..0ae4271 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
 	git init remote &&
 	(
 		cd remote &&
-- 
2.5.2.windows.2

Re: [PATCH 4/4] gc: remove broken refs

[Jeff King]

On Thu, Sep 24, 2015 at 11:14:02AM +0200, Johannes Schindelin wrote:

> When encountering broken refs, such as a stale remote HEAD (which can
> happen if the active branch was renamed in the remote), it is more
> helpful to remove those refs than to exit with an error.

For the same reasons as in my earlier responses, I think it's dangerous
to remove broken refs (it makes a small corruption much worse). It seems
reasonably sane to remove a dangling symref, though if we teach
for_each_ref to gracefully skip them, then leaving them in place isn't a
problem.

-Peff

Re: [PATCH 4/4] gc: remove broken refs

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> For the same reasons as in my earlier responses, I think it's dangerous
> to remove broken refs (it makes a small corruption much worse). It seems
> reasonably sane to remove a dangling symref, though if we teach
> for_each_ref to gracefully skip them, then leaving them in place isn't a
> problem.

One thing I wondered was if we can reliably tell between a ref that
wanted to be a real ref that records a broken object name and a ref
that wanted to be a symbolic ref that points a bogus thing, and if
we can't, should we worry about it too much.  The former is more
serious, as the history behind the commit it wanted to but failed to
record is at risk of being pruned.

One case that is clearly safe is "ref: refs/heads/gone"; it is not
likely to be the result of attempting to write a real object name
gone bad by whatever filesystem corruption.  On the other hand, an
obviously problematic case is an empty file.  We cannot tell if the
"broken" ref used to anchor the tip of a real history (which is
about to be lost with Dscho's patch 1/4) or was merely pointing at
another ref (which will not harm the object database if ignored).

So the rule should be

    If resolve_ref_unsafe_1() says it is a symbolic ref, if
    check_ref_format() is OK with the ref it points at, and if that
    pointee is missing, then it is safe to skip.

All other funnies should trigger the safety.

The collection of "broken and can be removed" refs introduced by 3/4
may also have to take that into account, I think.

Re: [PATCH 4/4] gc: remove broken refs

[Jeff King]

On Thu, Sep 24, 2015 at 05:08:41PM -0700, Junio C Hamano wrote:

> One thing I wondered was if we can reliably tell between a ref that
> wanted to be a real ref that records a broken object name and a ref
> that wanted to be a symbolic ref that points a bogus thing, and if
> we can't, should we worry about it too much.  The former is more
> serious, as the history behind the commit it wanted to but failed to
> record is at risk of being pruned.
> 
> One case that is clearly safe is "ref: refs/heads/gone"; it is not
> likely to be the result of attempting to write a real object name
> gone bad by whatever filesystem corruption.  On the other hand, an
> obviously problematic case is an empty file.  We cannot tell if the
> "broken" ref used to anchor the tip of a real history (which is
> about to be lost with Dscho's patch 1/4) or was merely pointing at
> another ref (which will not harm the object database if ignored).
> 
> So the rule should be
> 
>     If resolve_ref_unsafe_1() says it is a symbolic ref, if
>     check_ref_format() is OK with the ref it points at, and if that
>     pointee is missing, then it is safe to skip.
> 
> All other funnies should trigger the safety.

Right, I agree with that rule. If we don't know what it is, we should
err on the conservative side (and that _shouldn't_ happen, because does
not generate files in .git/refs that it cannot itself understand). But
"ref: " is clearly something we understand.

-Peff

[PATCH v2 0/4] Fix gc failure when a remote HEAD goes stale

[Johannes Schindelin]

There has been a report in the Git for Windows project that gc fails
sometimes: https://github.com/git-for-windows/git/issues/423

It turns out that there are cases when a remote HEAD can go stale and
it is not the user's fault at all. It can happen, for example, if the
active branch in the remote repository gets renamed.

Git's garbage collector should handle this gracefully. The best this
developer could come up with, is to simply warn and delete broken
symrefs.

Thanks to Junio and Peff for their really valuable sanity check.

Interdiff vs v1 below diffstat.

Johannes Schindelin (4):
  gc: demonstrate failure with stale remote HEAD
  pack-objects: do not get distracted by broken symrefs
  mark_reachable_objects(): optionally collect broken symrefs
  gc: remove broken refs

 builtin/prune.c  | 12 +++++++++++-
 builtin/reflog.c |  2 +-
 reachable.c      | 31 +++++++++++++++++++++++++------
 reachable.h      |  3 ++-
 t/t6500-gc.sh    | 15 +++++++++++++++
 5 files changed, 54 insertions(+), 9 deletions(-)

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ef2f794..1c63f8f 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -2499,7 +2499,6 @@ static void get_object_list(int ac, const char **av)
 	int flags = 0;
 
 	init_revisions(&revs, NULL);
-	revs.ignore_missing = 1;
 	save_commit_buffer = 0;
 	setup_revisions(ac, av, &revs, NULL);
 
diff --git a/builtin/prune.c b/builtin/prune.c
index 1a30f65..337b12a 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -101,7 +101,7 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
-	struct string_list broken_refs = STRING_LIST_INIT_DUP;
+	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -140,9 +140,9 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
 	revs.ignore_missing = 1;
-	mark_reachable_objects(&revs, 1, expire, progress, &broken_refs);
-	for (i = 0; i < broken_refs.nr; i++) {
-		char *path = broken_refs.items[i].string;
+	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
+	for (i = 0; i < broken_symrefs.nr; i++) {
+		char *path = broken_symrefs.items[i].string;
 		printf("Removing stale ref %s\n", path);
 		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
 			die("Could not remove stale ref %s", path);
diff --git a/reachable.c b/reachable.c
index 1fc7ada..25c4932 100644
--- a/reachable.c
+++ b/reachable.c
@@ -17,7 +17,7 @@ struct connectivity_progress {
 
 struct add_one_data {
 	struct rev_info *revs;
-	struct string_list *broken_refs;
+	struct string_list *broken_symrefs;
 };
 
 static void update_progress(struct connectivity_progress *cp)
@@ -31,13 +31,18 @@ static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
 	struct add_one_data *data = (struct add_one_data *)cb_data;
-	struct object *object = data->broken_refs ? parse_object(oid->hash) :
-		parse_object_or_die(oid->hash, path);
+	struct object *object;
 
-	if (!object)
-		string_list_append(data->broken_refs, path);
-	else
-		add_pending_object(data->revs, object, "");
+	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
+		if (data->broken_symrefs)
+			string_list_append(data->broken_symrefs, path);
+		else
+			warning("ref is broken: %s", path);
+		return 0;
+	}
+
+	object = parse_object_or_die(oid->hash, path);
+	add_pending_object(data->revs, object, "");
 
 	return 0;
 }
@@ -163,7 +168,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
 			    struct progress *progress,
-			    struct string_list *broken_refs)
+			    struct string_list *broken_symrefs)
 {
 	struct connectivity_progress cp;
 	struct add_one_data data;
@@ -180,7 +185,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	add_index_objects_to_pending(revs, 0);
 
 	data.revs = revs;
-	data.broken_refs = broken_refs;
+	data.broken_symrefs = broken_symrefs;
 
 	/* Add all external refs */
 	for_each_ref(add_one_ref, &data);
diff --git a/reachable.h b/reachable.h
index 39de1c7..06f1400 100644
--- a/reachable.h
+++ b/reachable.h
@@ -6,6 +6,6 @@ extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 				   unsigned long mark_recent, struct progress *,
-				   struct string_list *broken_refs);
+				   struct string_list *broken_symrefs);
 
 #endif
-- 
2.5.3.windows.1.3.gc322723

[PATCH v2 1/4] gc: demonstrate failure with stale remote HEAD

[Johannes Schindelin]

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 63194d8..b736774 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,4 +30,19 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
+test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+	git init remote &&
+	(
+		cd remote &&
+		test_commit initial &&
+		git clone . ../client &&
+		git branch -m develop &&
+		cd ../client &&
+		git fetch --prune &&
+		git gc &&
+		git branch --list -r origin/HEAD >actual &&
+		test_line_count = 0 actual
+	)
+'
+
 test_done
-- 
2.5.3.windows.1.3.gc322723

[PATCH v2 2/4] pack-objects: do not get distracted by broken symrefs

[Johannes Schindelin]

It is quite possible for, say, a remote HEAD to become broken, e.g.
when the default branch was renamed.

We should still be able to pack our objects when such a thing happens;
simply ignore broken symrefs (because they cannot matter for the packing
process anyway).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 reachable.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/reachable.c b/reachable.c
index 9cff25b..64289f3 100644
--- a/reachable.c
+++ b/reachable.c
@@ -25,9 +25,15 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct object *object = parse_object_or_die(oid->hash, path);
 	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct object *object;
 
+	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
+		warning("ref is broken: %s", path);
+		return 0;
+	}
+
+	object = parse_object_or_die(oid->hash, path);
 	add_pending_object(revs, object, "");
 
 	return 0;
-- 
2.5.3.windows.1.3.gc322723

[PATCH v2 3/4] mark_reachable_objects(): optionally collect broken symrefs

[Johannes Schindelin]

The behavior of `mark_reachable_objects()` without this patch is that
it warns if it encounters a broken symref. Sometimes the caller wants
to act upon broken symrefs, e.g. when garbage collecting in a repository
with a broken remote HEAD.

So let's introduce an optional parameter to collect broken symrefs. The
behavior of the function is unchanged if that parameter is `NULL`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c  |  2 +-
 builtin/reflog.c |  2 +-
 reachable.c      | 25 +++++++++++++++++++------
 reachable.h      |  3 ++-
 4 files changed, 23 insertions(+), 9 deletions(-)

diff --git a/builtin/prune.c b/builtin/prune.c
index 10b03d3..d6f664f 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -136,7 +136,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress);
+	mark_reachable_objects(&revs, 1, expire, progress, NULL);
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/builtin/reflog.c b/builtin/reflog.c
index f96ca2a..cb8758a 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -583,7 +583,7 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)
 		init_revisions(&cb.cmd.revs, prefix);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			printf("Marking reachable objects...");
-		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL);
+		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL, NULL);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			putchar('\n');
 	}
diff --git a/reachable.c b/reachable.c
index 64289f3..25c4932 100644
--- a/reachable.c
+++ b/reachable.c
@@ -15,6 +15,11 @@ struct connectivity_progress {
 	unsigned long count;
 };
 
+struct add_one_data {
+	struct rev_info *revs;
+	struct string_list *broken_symrefs;
+};
+
 static void update_progress(struct connectivity_progress *cp)
 {
 	cp->count++;
@@ -25,16 +30,19 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct add_one_data *data = (struct add_one_data *)cb_data;
 	struct object *object;
 
 	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
-		warning("ref is broken: %s", path);
+		if (data->broken_symrefs)
+			string_list_append(data->broken_symrefs, path);
+		else
+			warning("ref is broken: %s", path);
 		return 0;
 	}
 
 	object = parse_object_or_die(oid->hash, path);
-	add_pending_object(revs, object, "");
+	add_pending_object(data->revs, object, "");
 
 	return 0;
 }
@@ -159,9 +167,11 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
-			    struct progress *progress)
+			    struct progress *progress,
+			    struct string_list *broken_symrefs)
 {
 	struct connectivity_progress cp;
+	struct add_one_data data;
 
 	/*
 	 * Set up revision parsing, and mark us as being interested
@@ -174,11 +184,14 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	/* Add all refs from the index file */
 	add_index_objects_to_pending(revs, 0);
 
+	data.revs = revs;
+	data.broken_symrefs = broken_symrefs;
+
 	/* Add all external refs */
-	for_each_ref(add_one_ref, revs);
+	for_each_ref(add_one_ref, &data);
 
 	/* detached HEAD is not included in the list above */
-	head_ref(add_one_ref, revs);
+	head_ref(add_one_ref, &data);
 
 	/* Add all reflog info */
 	if (mark_reflog)
diff --git a/reachable.h b/reachable.h
index d23efc3..06f1400 100644
--- a/reachable.h
+++ b/reachable.h
@@ -5,6 +5,7 @@ struct progress;
 extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
-				   unsigned long mark_recent, struct progress *);
+				   unsigned long mark_recent, struct progress *,
+				   struct string_list *broken_symrefs);
 
 #endif
-- 
2.5.3.windows.1.3.gc322723

[PATCH v2 4/4] gc: remove broken symrefs

[Johannes Schindelin]

When encountering broken symrefs, such as a stale remote HEAD (which can
happen if the active branch was renamed in the remote), it is more
helpful to remove those symrefs than to exit with an error.

This fixes https://github.com/git-for-windows/git/issues/423

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c | 12 +++++++++++-
 t/t6500-gc.sh   |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/builtin/prune.c b/builtin/prune.c
index d6f664f..337b12a 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,6 +6,7 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
+#include "refs.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
@@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
+	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 	char *s;
+	int i;
 
 	expire = ULONG_MAX;
 	save_commit_buffer = 0;
@@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress, NULL);
+	revs.ignore_missing = 1;
+	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
+	for (i = 0; i < broken_symrefs.nr; i++) {
+		char *path = broken_symrefs.items[i].string;
+		printf("Removing stale ref %s\n", path);
+		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
+			die("Could not remove stale ref %s", path);
+	}
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index b736774..0ae4271 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
 	git init remote &&
 	(
 		cd remote &&
-- 
2.5.3.windows.1.3.gc322723

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Junio C Hamano]

Johannes Schindelin <johannes.schindelin@gmx.de> writes:

> When encountering broken symrefs, such as a stale remote HEAD (which can
> happen if the active branch was renamed in the remote), it is more
> helpful to remove those symrefs than to exit with an error.

I think this depends on the perspective.  One side of me says that a
remote HEAD that points at refs/remotes/origin/topic that no longer
exists is still giving me a valuable information and it should take
a conscious action by the user to remove it, or evne better to
repoint it to a more useful place.  And from that point of view,
removing is not all that helpful.  Keeping them and not allowing
them to exit with an error would be a real improvement.

On the other hand, I can certainly understand a view that considers
that such a dangling symbolic ref is merely a cruft like any other
cruft, and "gc" is all about removing cruft.

It just feels to me that this is a bit more valuable than other
kinds of cruft, but maybe it is just me.



>
> This fixes https://github.com/git-for-windows/git/issues/423
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  builtin/prune.c | 12 +++++++++++-
>  t/t6500-gc.sh   |  2 +-
>  2 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/builtin/prune.c b/builtin/prune.c
> index d6f664f..337b12a 100644
> --- a/builtin/prune.c
> +++ b/builtin/prune.c
> @@ -6,6 +6,7 @@
>  #include "reachable.h"
>  #include "parse-options.h"
>  #include "progress.h"
> +#include "refs.h"
>  
>  static const char * const prune_usage[] = {
>  	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
> @@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
>  int cmd_prune(int argc, const char **argv, const char *prefix)
>  {
>  	struct rev_info revs;
> +	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
>  	struct progress *progress = NULL;
>  	const struct option options[] = {
>  		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
> @@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>  		OPT_END()
>  	};
>  	char *s;
> +	int i;
>  
>  	expire = ULONG_MAX;
>  	save_commit_buffer = 0;
> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>  	if (show_progress)
>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>  
> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
> +	revs.ignore_missing = 1;
> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
> +	for (i = 0; i < broken_symrefs.nr; i++) {
> +		char *path = broken_symrefs.items[i].string;
> +		printf("Removing stale ref %s\n", path);
> +		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
> +			die("Could not remove stale ref %s", path);
> +	}
>  	stop_progress(&progress);
>  	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
>  				      prune_cruft, prune_subdir, NULL);
> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
> index b736774..0ae4271 100755
> --- a/t/t6500-gc.sh
> +++ b/t/t6500-gc.sh
> @@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
>  	test_i18ngrep "[Uu]sage" broken/usage
>  '
>  
> -test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
> +test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
>  	git init remote &&
>  	(
>  		cd remote &&

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
>
>> When encountering broken symrefs, such as a stale remote HEAD (which can
>> happen if the active branch was renamed in the remote), it is more
>> helpful to remove those symrefs than to exit with an error.
>
> I think this depends on the perspective.  One side of me says that a
> remote HEAD that points at refs/remotes/origin/topic that no longer
> exists is still giving me a valuable information and it should take
> a conscious action by the user to remove it, or evne better to
> repoint it to a more useful place.  And from that point of view,
> removing is not all that helpful.  Keeping them and not allowing
> them to exit with an error would be a real improvement.
>
> On the other hand, I can certainly understand a view that considers
> that such a dangling symbolic ref is merely a cruft like any other
> cruft, and "gc" is all about removing cruft.
>
> It just feels to me that this is a bit more valuable than other
> kinds of cruft, but maybe it is just me.

Sorry, it is a bad habit of me to send out without concluding
remark, leaving the recipient hanging without knowing what the next
step should be.

I meant to say that I plan to, and I indeed did, queue these 4
without changes.  I am not opposed to the removal so strongly to
reject [4/4].

The above comment was that I just do not know if this is the right
thing to do, or it will be hurting users.

Thanks.

>
>>
>> This fixes https://github.com/git-for-windows/git/issues/423
>>
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>> ---
>>  builtin/prune.c | 12 +++++++++++-
>>  t/t6500-gc.sh   |  2 +-
>>  2 files changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/builtin/prune.c b/builtin/prune.c
>> index d6f664f..337b12a 100644
>> --- a/builtin/prune.c
>> +++ b/builtin/prune.c
>> @@ -6,6 +6,7 @@
>>  #include "reachable.h"
>>  #include "parse-options.h"
>>  #include "progress.h"
>> +#include "refs.h"
>>  
>>  static const char * const prune_usage[] = {
>>  	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
>> @@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
>>  int cmd_prune(int argc, const char **argv, const char *prefix)
>>  {
>>  	struct rev_info revs;
>> +	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
>>  	struct progress *progress = NULL;
>>  	const struct option options[] = {
>>  		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
>> @@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>>  		OPT_END()
>>  	};
>>  	char *s;
>> +	int i;
>>  
>>  	expire = ULONG_MAX;
>>  	save_commit_buffer = 0;
>> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>>  	if (show_progress)
>>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>>  
>> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
>> +	revs.ignore_missing = 1;
>> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
>> +	for (i = 0; i < broken_symrefs.nr; i++) {
>> +		char *path = broken_symrefs.items[i].string;
>> +		printf("Removing stale ref %s\n", path);
>> +		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
>> +			die("Could not remove stale ref %s", path);
>> +	}
>>  	stop_progress(&progress);
>>  	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
>>  				      prune_cruft, prune_subdir, NULL);
>> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
>> index b736774..0ae4271 100755
>> --- a/t/t6500-gc.sh
>> +++ b/t/t6500-gc.sh
>> @@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
>>  	test_i18ngrep "[Uu]sage" broken/usage
>>  '
>>  
>> -test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
>> +test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
>>  	git init remote &&
>>  	(
>>  		cd remote &&

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Johannes Schindelin]

Hi Junio,

On 2015-09-28 20:49, Junio C Hamano wrote:
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
>>
>>> When encountering broken symrefs, such as a stale remote HEAD (which can
>>> happen if the active branch was renamed in the remote), it is more
>>> helpful to remove those symrefs than to exit with an error.
>>
>> I think this depends on the perspective.  One side of me says that a
>> remote HEAD that points at refs/remotes/origin/topic that no longer
>> exists is still giving me a valuable information and it should take
>> a conscious action by the user to remove it, or evne better to
>> repoint it to a more useful place.  And from that point of view,
>> removing is not all that helpful.  Keeping them and not allowing
>> them to exit with an error would be a real improvement.
>>
>> On the other hand, I can certainly understand a view that considers
>> that such a dangling symbolic ref is merely a cruft like any other
>> cruft, and "gc" is all about removing cruft.
>>
>> It just feels to me that this is a bit more valuable than other
>> kinds of cruft, but maybe it is just me.
> 
> Sorry, it is a bad habit of me to send out without concluding
> remark, leaving the recipient hanging without knowing what the next
> step should be.
> 
> I meant to say that I plan to, and I indeed did, queue these 4
> without changes.  I am not opposed to the removal so strongly to
> reject [4/4].
> 
> The above comment was that I just do not know if this is the right
> thing to do, or it will be hurting users.

Oh, I appreciate your feedback. I am actually not all *that* certain that removing the broken symref is the correct thing. It is this sort of fruitful exchange that allows me to throw out an idea and be relatively certain that something better will come out of v3 or v8 of the patch series than what I had in mind.

To be honest, the most important outcome is probably 2/4 -- which should be enough to fix the issue reported by the Git for Windows user. I could adjust the test so that it no longer insists that `origin/HEAD` be deleted, but still requires that `git gc` succeeds.

I would have no problem to let this sit for a couple of days until the final verdict.

Ciao,
Dscho

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Junio C Hamano]

Johannes Schindelin <johannes.schindelin@gmx.de> writes:

> Oh, I appreciate your feedback. I am actually not all *that* certain
> that removing the broken symref is the correct thing. It is this sort
> of fruitful exchange that allows me to throw out an idea and be
> relatively certain that something better will come out of v3 or v8 of
> the patch series than what I had in mind.
>
> To be honest, the most important outcome is probably 2/4 -- which
> should be enough to fix the issue reported by the Git for Windows
> user. I could adjust the test so that it no longer insists that
> origin/HEAD` be deleted, but still requires that `git gc` succeeds.
>
> I would have no problem to let this sit for a couple of days until
> the final verdict.

... and a few days have passed.  I am tempted to do the easy bits
first, discarding the parts that both of us feel iffy for now
without prejudice, keeping the first two commits with a bit of
tweak.

The primary tweak is to t6500 in the first patch, which I retitled
below (and the patch shows s/failure/success/ but in the first step
that only adds a failing test, it would of course expect a failure).

And with "pack-objects: do not get distracted", the test will start
passing.

diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index b736774..5d7d414 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc is not aborted due to a stale symref' '
 	git init remote &&
 	(
 		cd remote &&
@@ -39,9 +39,7 @@ test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
 		git branch -m develop &&
 		cd ../client &&
 		git fetch --prune &&
-		git gc &&
-		git branch --list -r origin/HEAD >actual &&
-		test_line_count = 0 actual
+		git gc
 	)
 '
 
diff --git a/reachable.c b/reachable.c
index 6356ae8..4cfd0de 100644
--- a/reachable.c
+++ b/reachable.c
@@ -28,7 +28,7 @@ static int add_one_ref(const char *path, const unsigned char *sha1, int flag, vo
 	struct object *object;
 
 	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
-		warning("ref is broken: %s", path);
+		warning("symbolic ref is dangling: %s", path);
 		return 0;
 	}
 

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Jeff King]

On Mon, Sep 28, 2015 at 04:02:08PM +0200, Johannes Schindelin wrote:

> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>  	if (show_progress)
>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>  
> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
> +	revs.ignore_missing = 1;
> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);

You should not need this ignore_missing anymore, right?

It is the dangerous thing I mentioned earlier, but I am puzzled why it
does not cause t5312 to fail.

-Peff

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Johannes Schindelin]

Hi Peff,

On 2015-09-28 21:03, Jeff King wrote:
> On Mon, Sep 28, 2015 at 04:02:08PM +0200, Johannes Schindelin wrote:
> 
>> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>>  	if (show_progress)
>>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>>
>> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
>> +	revs.ignore_missing = 1;
>> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
> 
> You should not need this ignore_missing anymore, right?
> 
> It is the dangerous thing I mentioned earlier, but I am puzzled why it
> does not cause t5312 to fail.

Gaaaaaah! I edited this out at least twice, yet it creeps back in. Bah!

Junio, would you mind fixing this up locally? If you want me to submit v3, just let me know, I will take care of it tomorrow.

Ciao,
Dscho

[PATCH v3 0/2] Fix gc failure when a remote HEAD goes stale

[Johannes Schindelin]

There has been a report in the Git for Windows project that gc fails
sometimes: https://github.com/git-for-windows/git/issues/423

It turns out that there are cases when a remote HEAD can go stale and
it is not the user's fault at all. It can happen, for example, if the
active branch in the remote repository gets renamed.

Thanks to Junio and Peff for their really valuable sanity check.

Interdiff re: v2 after the diffstat (3/4 and 4/4 were dropped, I am
no longer removing the broken symrefs).


Johannes Schindelin (2):
  gc: demonstrate failure with stale remote HEAD
  pack-objects: do not get distracted by broken symrefs

 reachable.c   |  8 +++++++-
 t/t6500-gc.sh | 13 +++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/builtin/prune.c b/builtin/prune.c
index 337b12a..10b03d3 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,7 +6,6 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
-#include "refs.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
@@ -101,7 +100,6 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
-	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -112,7 +110,6 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 	char *s;
-	int i;
 
 	expire = ULONG_MAX;
 	save_commit_buffer = 0;
@@ -139,14 +136,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	revs.ignore_missing = 1;
-	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
-	for (i = 0; i < broken_symrefs.nr; i++) {
-		char *path = broken_symrefs.items[i].string;
-		printf("Removing stale ref %s\n", path);
-		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
-			die("Could not remove stale ref %s", path);
-	}
+	mark_reachable_objects(&revs, 1, expire, progress);
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/builtin/reflog.c b/builtin/reflog.c
index cb8758a..f96ca2a 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -583,7 +583,7 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)
 		init_revisions(&cb.cmd.revs, prefix);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			printf("Marking reachable objects...");
-		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL, NULL);
+		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			putchar('\n');
 	}
diff --git a/reachable.c b/reachable.c
index 25c4932..43616d4 100644
--- a/reachable.c
+++ b/reachable.c
@@ -15,11 +15,6 @@ struct connectivity_progress {
 	unsigned long count;
 };
 
-struct add_one_data {
-	struct rev_info *revs;
-	struct string_list *broken_symrefs;
-};
-
 static void update_progress(struct connectivity_progress *cp)
 {
 	cp->count++;
@@ -30,19 +25,16 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct add_one_data *data = (struct add_one_data *)cb_data;
+	struct rev_info *revs = (struct rev_info *)cb_data;
 	struct object *object;
 
 	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
-		if (data->broken_symrefs)
-			string_list_append(data->broken_symrefs, path);
-		else
-			warning("ref is broken: %s", path);
+		warning("symbolic ref is dangling: %s", path);
 		return 0;
 	}
 
 	object = parse_object_or_die(oid->hash, path);
-	add_pending_object(data->revs, object, "");
+	add_pending_object(revs, object, "");
 
 	return 0;
 }
@@ -167,11 +159,9 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
-			    struct progress *progress,
-			    struct string_list *broken_symrefs)
+			    struct progress *progress)
 {
 	struct connectivity_progress cp;
-	struct add_one_data data;
 
 	/*
 	 * Set up revision parsing, and mark us as being interested
@@ -184,14 +174,11 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	/* Add all refs from the index file */
 	add_index_objects_to_pending(revs, 0);
 
-	data.revs = revs;
-	data.broken_symrefs = broken_symrefs;
-
 	/* Add all external refs */
-	for_each_ref(add_one_ref, &data);
+	for_each_ref(add_one_ref, revs);
 
 	/* detached HEAD is not included in the list above */
-	head_ref(add_one_ref, &data);
+	head_ref(add_one_ref, revs);
 
 	/* Add all reflog info */
 	if (mark_reflog)
diff --git a/reachable.h b/reachable.h
index 06f1400..d23efc3 100644
--- a/reachable.h
+++ b/reachable.h
@@ -5,7 +5,6 @@ struct progress;
 extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
-				   unsigned long mark_recent, struct progress *,
-				   struct string_list *broken_symrefs);
+				   unsigned long mark_recent, struct progress *);
 
 #endif
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 0ae4271..5d7d414 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc is not aborted due to a stale symref' '
 	git init remote &&
 	(
 		cd remote &&
@@ -39,9 +39,7 @@ test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
 		git branch -m develop &&
 		cd ../client &&
 		git fetch --prune &&
-		git gc &&
-		git branch --list -r origin/HEAD >actual &&
-		test_line_count = 0 actual
+		git gc
 	)
 '
 
-- 
2.6.1.windows.1

[PATCH v3 1/2] gc: demonstrate failure with stale remote HEAD

[Johannes Schindelin]

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 63194d8..9a3a285 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,4 +30,17 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
+test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+	git init remote &&
+	(
+		cd remote &&
+		test_commit initial &&
+		git clone . ../client &&
+		git branch -m develop &&
+		cd ../client &&
+		git fetch --prune &&
+		git gc
+	)
+'
+
 test_done
-- 
2.6.1.windows.1

[PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

[Johannes Schindelin]

It is quite possible for, say, a remote HEAD to become broken, e.g.
when the default branch was renamed.

We should still be able to pack our objects when such a thing happens;
simply ignore broken symrefs (because they cannot matter for the packing
process anyway).

This fixes https://github.com/git-for-windows/git/issues/423

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 reachable.c   | 8 +++++++-
 t/t6500-gc.sh | 2 +-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/reachable.c b/reachable.c
index 9cff25b..43616d4 100644
--- a/reachable.c
+++ b/reachable.c
@@ -25,9 +25,15 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct object *object = parse_object_or_die(oid->hash, path);
 	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct object *object;
 
+	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
+		warning("symbolic ref is dangling: %s", path);
+		return 0;
+	}
+
+	object = parse_object_or_die(oid->hash, path);
 	add_pending_object(revs, object, "");
 
 	return 0;
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 9a3a285..5d7d414 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc is not aborted due to a stale symref' '
 	git init remote &&
 	(
 		cd remote &&
-- 
2.6.1.windows.1

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

[Junio C Hamano]

Johannes Schindelin <johannes.schindelin@gmx.de> writes:

> It is quite possible for, say, a remote HEAD to become broken, e.g.
> when the default branch was renamed.
>
> We should still be able to pack our objects when such a thing happens;
> simply ignore broken symrefs (because they cannot matter for the packing
> process anyway).
>
> This fixes https://github.com/git-for-windows/git/issues/423
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---

It seems that the result of applying these two patches and log
messages of them are the same with what I queued on 'pu', except
that the first of these two patches adds a test with a wrong name
and then this one does "oops, that was misnamed".  So I'll keep what
is already queued.

Thanks.

>  reachable.c   | 8 +++++++-
>  t/t6500-gc.sh | 2 +-
>  2 files changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/reachable.c b/reachable.c
> index 9cff25b..43616d4 100644
> --- a/reachable.c
> +++ b/reachable.c
> @@ -25,9 +25,15 @@ static void update_progress(struct connectivity_progress *cp)
>  static int add_one_ref(const char *path, const struct object_id *oid,
>  		       int flag, void *cb_data)
>  {
> -	struct object *object = parse_object_or_die(oid->hash, path);
>  	struct rev_info *revs = (struct rev_info *)cb_data;
> +	struct object *object;
>  
> +	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
> +		warning("symbolic ref is dangling: %s", path);
> +		return 0;
> +	}
> +
> +	object = parse_object_or_die(oid->hash, path);
>  	add_pending_object(revs, object, "");
>  
>  	return 0;
> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
> index 9a3a285..5d7d414 100755
> --- a/t/t6500-gc.sh
> +++ b/t/t6500-gc.sh
> @@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
>  	test_i18ngrep "[Uu]sage" broken/usage
>  '
>  
> -test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
> +test_expect_success 'gc is not aborted due to a stale symref' '
>  	git init remote &&
>  	(
>  		cd remote &&

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

[Johannes Schindelin]

Hi Junio,

On 2015-10-07 19:45, Junio C Hamano wrote:
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> 
>> It is quite possible for, say, a remote HEAD to become broken, e.g.
>> when the default branch was renamed.
>>
>> We should still be able to pack our objects when such a thing happens;
>> simply ignore broken symrefs (because they cannot matter for the packing
>> process anyway).
>>
>> This fixes https://github.com/git-for-windows/git/issues/423
>>
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>> ---
> 
> It seems that the result of applying these two patches and log
> messages of them are the same with what I queued on 'pu', except
> that the first of these two patches adds a test with a wrong name
> and then this one does "oops, that was misnamed".  So I'll keep what
> is already queued.

Sorry for fixing up the wrong commit. I honestly meant to fix up the first one. And thank you for fixing it up in `pu` already; I should have known better and check first whether you fixed it.

However, there was one more change I made: I wanted to have that link to https://github.com/git-for-windows/git/issues/423 in the commit message to better link the original report with the commit.

Would yo kindly add the line

    This fixes https://github.com/git-for-windows/git/issues/423

before the Signed-off-by lines?

Thanks,
Dscho

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

[Junio C Hamano]

Johannes Schindelin <johannes.schindelin@gmx.de> writes:

> Would yo kindly add the line
>
>     This fixes https://github.com/git-for-windows/git/issues/423
>
> before the Signed-off-by lines?

Oh, sorry, I missed that one.  Fixed.

Thanks.

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

[Johannes Schindelin]

Hi Junio,

On 2015-10-08 21:42, Junio C Hamano wrote:
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> 
>> Would yo kindly add the line
>>
>>     This fixes https://github.com/git-for-windows/git/issues/423
>>
>> before the Signed-off-by lines?
> 
> Oh, sorry, I missed that one.  Fixed.
> 
> Thanks.

Thank you!
Dscho

Re: [PATCH v2 4/4] gc: remove broken symrefs

[Johannes Schindelin]

Hi Junio,

On 2015-10-06 00:06, Junio C Hamano wrote:
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> 
>> Oh, I appreciate your feedback. I am actually not all *that* certain
>> that removing the broken symref is the correct thing. It is this sort
>> of fruitful exchange that allows me to throw out an idea and be
>> relatively certain that something better will come out of v3 or v8 of
>> the patch series than what I had in mind.
>>
>> To be honest, the most important outcome is probably 2/4 -- which
>> should be enough to fix the issue reported by the Git for Windows
>> user. I could adjust the test so that it no longer insists that
>> origin/HEAD` be deleted, but still requires that `git gc` succeeds.
>>
>> I would have no problem to let this sit for a couple of days until
>> the final verdict.
> 
> ... and a few days have passed.  I am tempted to do the easy bits
> first, discarding the parts that both of us feel iffy for now
> without prejudice, keeping the first two commits with a bit of
> tweak.

I agree. And I just sent out v3 of the patch series.

Thanks,
Dscho