[PATCH/RFC] remote: support --all for the prune-subcommand

[PATCH/RFC] remote: support --all for the prune-subcommand

[Erik Faye-Lund]

While we're at it, wrap a long line to fit on a 80 char terminal.

Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
---

I recently needed to prune remote branches in a repo with a lot
of remotes, and to my surprise "git remote prune" didn't support
the --all option. So I added it. Perhaps this is useful for other
people as well?

 Documentation/git-remote.txt |    2 +-
 builtin/remote.c             |   27 ++++++++++++++++++++-------
 2 files changed, 21 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-remote.txt b/Documentation/git-remote.txt
index 5a8c506..856cc7f 100644
--- a/Documentation/git-remote.txt
+++ b/Documentation/git-remote.txt
@@ -19,7 +19,7 @@ SYNOPSIS
 'git remote set-url --add' [--push] <name> <newurl>
 'git remote set-url --delete' [--push] <name> <url>
 'git remote' [-v | --verbose] 'show' [-n] <name>
-'git remote prune' [-n | --dry-run] <name>
+'git remote prune' [-n | --dry-run] (--all | <name>...)
 'git remote' [-v | --verbose] 'update' [-p | --prune] [(<group> | <remote>)...]
 
 DESCRIPTION
diff --git a/builtin/remote.c b/builtin/remote.c
index f2a9c26..2e8407d 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -14,7 +14,7 @@ static const char * const builtin_remote_usage[] = {
 	"git remote rm <name>",
 	"git remote set-head <name> (-a | -d | <branch>)",
 	"git remote [-v | --verbose] show [-n] <name>",
-	"git remote prune [-n | --dry-run] <name>",
+	"git remote prune [-n | --dry-run] (--all | <name>)",
 	"git remote [-v | --verbose] update [-p | --prune] [(<group> | <remote>)...]",
 	"git remote set-branches <name> [--add] <branch>...",
 	"git remote set-url <name> <newurl> [<oldurl>]",
@@ -1222,22 +1222,35 @@ static int set_head(int argc, const char **argv)
 	return result;
 }
 
+static int add_one_remote(struct remote *remote, void *remotes)
+{
+	string_list_append(remotes, remote->name);
+	return 0;
+}
+
 static int prune(int argc, const char **argv)
 {
-	int dry_run = 0, result = 0;
+	struct string_list remotes = STRING_LIST_INIT_NODUP;
+	int dry_run = 0, result = 0, all = 0, i;
 	struct option options[] = {
+		OPT_BOOLEAN(0, "all", &all, "prune all remotes"),
 		OPT__DRY_RUN(&dry_run, "dry run"),
 		OPT_END()
 	};
 
-	argc = parse_options(argc, argv, NULL, options, builtin_remote_prune_usage,
-			     0);
+	argc = parse_options(argc, argv, NULL, options,
+	                     builtin_remote_prune_usage, 0);
 
-	if (argc < 1)
+	if (all)
+		for_each_remote(add_one_remote, &remotes);
+	else if (argc < 1)
 		usage_with_options(builtin_remote_prune_usage, options);
+	else
+		for (; argc; argc--, argv++)
+			string_list_append(&remotes, *argv);
 
-	for (; argc; argc--, argv++)
-		result |= prune_remote(*argv, dry_run);
+	for (i = 0; i < remotes.nr; ++i)
+		result |= prune_remote(remotes.items[i].string, dry_run);
 
 	return result;
 }
-- 
1.7.6.msysgit.0.579.ga3d6f

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Jacob Helwig]

[-- Attachment #1: Type: text/plain, Size: 676 bytes --]

On Mon, 03 Oct 2011 14:16:08 +0200, Erik Faye-Lund wrote:
> 
> While we're at it, wrap a long line to fit on a 80 char terminal.
> 
> Signed-off-by: Erik Faye-Lund <kusmabite@gmail.com>
> ---
> 
> I recently needed to prune remote branches in a repo with a lot
> of remotes, and to my surprise "git remote prune" didn't support
> the --all option. So I added it. Perhaps this is useful for other
> people as well?
> 

Can't really comment on the implementation (especially since I didn't
actually look at it), but having "git remote prune --all" work would be
_tremendously_ helpful to me.  Thanks for doing this!

-- 
Jacob Helwig
http://about.me/jhelwig

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 665 bytes --]

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Jeff King]

On Mon, Oct 03, 2011 at 02:16:08PM +0200, Erik Faye-Lund wrote:

> I recently needed to prune remote branches in a repo with a lot
> of remotes, and to my surprise "git remote prune" didn't support
> the --all option. So I added it. Perhaps this is useful for other
> people as well?

You could do:

  git remote update --prune

But I thought we were trying to get away from remote doing fetch-like
things in the long term. Isn't the "right" way to do this these days:

  git fetch --all --prune

?

-Peff

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Erik Faye-Lund]

On Tue, Oct 4, 2011 at 9:00 AM, Jeff King <peff@peff.net> wrote:
> On Mon, Oct 03, 2011 at 02:16:08PM +0200, Erik Faye-Lund wrote:
>
>> I recently needed to prune remote branches in a repo with a lot
>> of remotes, and to my surprise "git remote prune" didn't support
>> the --all option. So I added it. Perhaps this is useful for other
>> people as well?
>
> You could do:
>
>  git remote update --prune
>
> But I thought we were trying to get away from remote doing fetch-like
> things in the long term. Isn't the "right" way to do this these days:
>
>  git fetch --all --prune
>
> ?
>
> -Peff
>

I wasn't aware that fetch could prune, but yeah, that seems much
better to me. Perhaps a mention of this in the "git remote prune"
documentation could steer other users in the right direction?

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Jeff King]

On Tue, Oct 04, 2011 at 09:10:40AM +0200, Erik Faye-Lund wrote:

> >  git fetch --all --prune
> >
> I wasn't aware that fetch could prune, but yeah, that seems much
> better to me. Perhaps a mention of this in the "git remote prune"
> documentation could steer other users in the right direction?

Yeah, that makes sense.

There is one slight difference: I think "git remote prune" will _just_
prune, and not fetch into existing refs at all. I'm not sure exactly
why you would want that, though. Presumably you run "prune" after you
just fetched, anyway. Combining the two steps saves an extra network
connection.

-Peff

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Erik Faye-Lund]

On Tue, Oct 4, 2011 at 9:13 AM, Jeff King <peff@peff.net> wrote:
> On Tue, Oct 04, 2011 at 09:10:40AM +0200, Erik Faye-Lund wrote:
>
>> >  git fetch --all --prune
>> >
>> I wasn't aware that fetch could prune, but yeah, that seems much
>> better to me. Perhaps a mention of this in the "git remote prune"
>> documentation could steer other users in the right direction?
>
> Yeah, that makes sense.
>
> There is one slight difference: I think "git remote prune" will _just_
> prune, and not fetch into existing refs at all. I'm not sure exactly
> why you would want that, though.

Hmm, you might want to do that on, say, a mobile network to save
bandwidth; i.e throw away the stale branches, but not yet update the
non-stale ones because downloading the objects might take a long time
(and/or be expensive).

So with that in mind, I actually think my patch makes sense in some
cases, but it certainly is less useful that I originally though ;)

A mention in the documentation seems like a good move no matter what, though.

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Erik Faye-Lund]

On Tue, Oct 4, 2011 at 9:18 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> On Tue, Oct 4, 2011 at 9:13 AM, Jeff King <peff@peff.net> wrote:
>> On Tue, Oct 04, 2011 at 09:10:40AM +0200, Erik Faye-Lund wrote:
>>
>>> >  git fetch --all --prune
>>> >
>>> I wasn't aware that fetch could prune, but yeah, that seems much
>>> better to me. Perhaps a mention of this in the "git remote prune"
>>> documentation could steer other users in the right direction?
>>
>> Yeah, that makes sense.
>>
>> There is one slight difference: I think "git remote prune" will _just_
>> prune, and not fetch into existing refs at all. I'm not sure exactly
>> why you would want that, though.
>
> Hmm, you might want to do that on, say, a mobile network to save
> bandwidth; i.e throw away the stale branches, but not yet update the
> non-stale ones because downloading the objects might take a long time
> (and/or be expensive).
>
> So with that in mind, I actually think my patch makes sense in some
> cases, but it certainly is less useful that I originally though ;)

Strike that part; I hadn't had my morning coffee yet. It might make
sense to have similar _functionality_, but having this as a flag to
"git fetch" instead of "git remote prune" strikes me as the only sane
approach.

In fact, I'm not sure I understand why we simply do not always prune
by default. My guess would be backward compatibility, but this strikes
me as one of these things where we should introduce a config variable
(there's already one for git-gui: gui.pruneduringfetch), add a warning
if unset, and flip the default at some future major release. After
all, a remote branch isn't the user's branch - it's a cache/mirror
some other user's branch. If a user wants to keep another user's
branch, surely the most sane thing would be to make a local branch of
it?

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Jeff King]

On Tue, Oct 04, 2011 at 09:40:22AM +0200, Erik Faye-Lund wrote:

> > Hmm, you might want to do that on, say, a mobile network to save
> > bandwidth; i.e throw away the stale branches, but not yet update the
> > non-stale ones because downloading the objects might take a long time
> > (and/or be expensive).
> >
> > So with that in mind, I actually think my patch makes sense in some
> > cases, but it certainly is less useful that I originally though ;)
> 
> Strike that part; I hadn't had my morning coffee yet. It might make
> sense to have similar _functionality_, but having this as a flag to
> "git fetch" instead of "git remote prune" strikes me as the only sane
> approach.

I agree that "git fetch --prune-only" (or something similar) would be a
natural way to do it.

> In fact, I'm not sure I understand why we simply do not always prune
> by default.

I think the original rationale was that we didn't want fetch to be
"lossy". That is, if I were using upstream's "foo" branch as part of my
work (to diff against, or whatever), then doing a "git fetch" to update
should not suddenly make it hard to do my work. And not just hard as in
"I notice that it's gone and I adapt my workflow". But that you no
longer have _any_ record of where upstream's "foo" branch used to point,
so even doing something like:

  git rebase --onto new-foo foo my-topic

is impossible.

These days we have reflogs, so you would hope to do something like:

  git rebase --onto new-foo foo@{1} my-topic

But ref deletion also deletes the reflog completely, so that doesn't
work.

The right solution, IMHO, is that ref deletion should actually keep the
reflog around in a graveyard of some sort. Entries would expire
naturally over time, as they do in regular reflogs. And then it becomes
a lot safer to prune on every fetch, because you still have 90 days look
at the reflog.

There is still one sticky point, which is that your branch config may
refer to an upstream branch that gets pruned. That will break some
operations (as well it should, as the branch is gone, and the user needs
to adapt their config appropriately). It might be nice if we noticed
when accessing a ref that it doesn't exist but has a deleted reflog, so
we can give the user better advice.

> If a user wants to keep another user's branch, surely the most sane
> thing would be to make a local branch of it?

Unfortunately there are some management problems there. How do I keep my
local branch up to date with what I fetch? I have to keep checking out
and merging on every fetch (or use some plumbing), which is a pain. But
if I don't, then when the upstream branch goes away, I still have no
clue where its tip was right before it got pruned.

-Peff

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Erik Faye-Lund]

On Tue, Oct 4, 2011 at 9:56 AM, Jeff King <peff@peff.net> wrote:
> On Tue, Oct 04, 2011 at 09:40:22AM +0200, Erik Faye-Lund wrote:
>> In fact, I'm not sure I understand why we simply do not always prune
>> by default.
>
> I think the original rationale was that we didn't want fetch to be
> "lossy". That is, if I were using upstream's "foo" branch as part of my
> work (to diff against, or whatever), then doing a "git fetch" to update
> should not suddenly make it hard to do my work. And not just hard as in
> "I notice that it's gone and I adapt my workflow". But that you no
> longer have _any_ record of where upstream's "foo" branch used to point,
> so even doing something like:
>
>  git rebase --onto new-foo foo my-topic
>
> is impossible.
>

Following that logic, a user cannot _ever_ safely prune a remote if he
wants to work on some of the branches. Doing something like "git
remote foo -n" to check if the branch would get pruned before doing a
proper prune is prone to a race-condition; the branch could be deleted
on the remote between the dry-run and the actual pruning.

Besides, the owner of the repo can just as easily have deleted the
branch and created a new one with the same name, causing the contents
of the branch to be lost. This happens all the time with
"for-upstream"-kind of branches, no?

> These days we have reflogs, so you would hope to do something like:
>
>  git rebase --onto new-foo foo@{1} my-topic
>
> But ref deletion also deletes the reflog completely, so that doesn't
> work.
>

...and this just makes the problem I pointed out above a lot worse.

So surely, the only sane thing is to make a local branch of what
you're interested in to be safe?

> The right solution, IMHO, is that ref deletion should actually keep the
> reflog around in a graveyard of some sort. Entries would expire
> naturally over time, as they do in regular reflogs. And then it becomes
> a lot safer to prune on every fetch, because you still have 90 days look
> at the reflog.
>

Fixing the reflog to expire for ref deletion rather than completely
deleting it sounds like a good move, indeed.

>> If a user wants to keep another user's branch, surely the most sane
>> thing would be to make a local branch of it?
>
> Unfortunately there are some management problems there. How do I keep my
> local branch up to date with what I fetch? I have to keep checking out
> and merging on every fetch (or use some plumbing), which is a pain. But
> if I don't, then when the upstream branch goes away, I still have no
> clue where its tip was right before it got pruned.

Hmm, good point. I tend to just do the dirty work every now and then
myself. But I only tend to track upstream and stale
development-branches that I intend to pick up, so I'm probably not the
best user-example.

While we're on the subject, an additional argument to change "git
fetch" to always prune is that it's much much easier for user to grok
"last known state of <remote>'s branches" than "the union of all the
branches that were ever pulled from <remote>, unless --prune was
specified". But that's not a technical one, and surely there's issues
to resolve with the proposal before going in that direction.

Re: [PATCH/RFC] remote: support --all for the prune-subcommand

[Jeff King]

On Tue, Oct 04, 2011 at 10:22:35AM +0200, Erik Faye-Lund wrote:

> > I think the original rationale was that we didn't want fetch to be
> > "lossy". That is, if I were using upstream's "foo" branch as part of my
> > work (to diff against, or whatever), then doing a "git fetch" to update
> > should not suddenly make it hard to do my work. And not just hard as in
> > "I notice that it's gone and I adapt my workflow". But that you no
> > longer have _any_ record of where upstream's "foo" branch used to point,
> > so even doing something like:
> >
> >  git rebase --onto new-foo foo my-topic
> >
> > is impossible.
> 
> Following that logic, a user cannot _ever_ safely prune a remote if he
> wants to work on some of the branches. Doing something like "git
> remote foo -n" to check if the branch would get pruned before doing a
> proper prune is prone to a race-condition; the branch could be deleted
> on the remote between the dry-run and the actual pruning.

Right. And that's why we don't prune by default. In practice, it tends
to be safe if you pick a reasonable time to prune, and the upstream is
reasonable about their branches. But turning it on all the time takes
away the "pick a reasonable time".

> Besides, the owner of the repo can just as easily have deleted the
> branch and created a new one with the same name, causing the contents
> of the branch to be lost. This happens all the time with
> "for-upstream"-kind of branches, no?

They can do that, but on the local side, you will just see a jump in
history. But because we didn't _delete_ the ref on the local side, you
will retain your reflog.

IOW, the reflog can save us from anything the upstream will do. And
that's what makes deletion so special: we delete the local reflog.

> > The right solution, IMHO, is that ref deletion should actually keep the
> > reflog around in a graveyard of some sort. Entries would expire
> > naturally over time, as they do in regular reflogs. And then it becomes
> > a lot safer to prune on every fetch, because you still have 90 days look
> > at the reflog.
> >
> Fixing the reflog to expire for ref deletion rather than completely
> deleting it sounds like a good move, indeed.

This is on my long-term todo list, but if somebody gets around to it
before me, I won't be upset. :)

> While we're on the subject, an additional argument to change "git
> fetch" to always prune is that it's much much easier for user to grok
> "last known state of <remote>'s branches" than "the union of all the
> branches that were ever pulled from <remote>, unless --prune was
> specified". But that's not a technical one, and surely there's issues
> to resolve with the proposal before going in that direction.

Agreed. Really, everything argument points towards auto-prune except the
reflog-safety thing. I think once that is fixed, turning on pruning by
default becomes a no-brainer.

-Peff