[PATCH] Documentation: pruning recipe for destructive filter-branch

[PATCH] Documentation: pruning recipe for destructive filter-branch

[Thomas Rast]

Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.

This comes up every week or so on IRC, and the commands required to
handle every case are not very newbie-friendly, so hopefully writing
them down somewhere leads to fewer questions.

Thanks to doener (Björn Steinbrink) for comments and corrections.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Or we could just add an option --prune-everything-I-have-a-backup-I-promise
to git-filter-branch, so that users can get the same result with less effort.


 Documentation/git-filter-branch.txt |   36 +++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 1fbbbb4..737c555 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -339,6 +339,42 @@ git filter-branch --index-filter \
 ---------------------------------------------------------------
 
 
+
+Checklist for Shrinking a Repository
+------------------------------------
+
+git-filter-branch is often used to get rid of a subset of files,
+usually with some combination of `\--index-filter` and
+`\--subdirectory-filter`.  If you want to physically shrink the
+repository afterwards, you have some choices:
+
+* Clone it with `git clone file:///path/to/repo`.  The clone will not
+  have the removed objects.  See linkgit:git-clone[1].  (Note that
+  cloning with a plain path just hardlinks everything!)
+
+If you really don't want to clone it, for whatever reasons, check the
+following points (in this order).  This is a very destructive
+approach, so *make a backup* or go back to cloning it.  You have been
+warned.
+
+* Make sure you really removed all variants of a filename, if a blob
+  was moved over its lifetime.  `git log \--follow \--all \-- foo` can
+  help you find renames.
+
+* Make sure you really filtered all refs: use `\--tag-name-filter cat
+  \-- \--all` when calling git-filter-branch.
+
+* Make sure you remove the original refs backed up by
+  git-filter-branch: say `git for-each-ref \--format="%(refname)"
+  refs/original/ | xargs -n 1 git update-ref -d`.
+
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.
+
+* Repack to remove packed objects with `git repack -ad`.
+
+* Prune unpacked unreferenced objects with `git prune \--expire=now`.
+
+
 Author
 ------
 Written by Petr "Pasky" Baudis <pasky@suse.cz>,
-- 
1.6.2.rc0.274.g97213

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
> index 1fbbbb4..737c555 100644
> --- a/Documentation/git-filter-branch.txt
> +++ b/Documentation/git-filter-branch.txt
> @@ -339,6 +339,42 @@ git filter-branch --index-filter \
>  ---------------------------------------------------------------
>  
>  
> +
> +Checklist for Shrinking a Repository
> +------------------------------------
> +
> +git-filter-branch is often used to get rid of a subset of files,
> +usually with some combination of `\--index-filter` and
> +`\--subdirectory-filter`.  If you want to physically shrink the
> +repository afterwards, you have some choices:
> +
> +* Clone it with `git clone file:///path/to/repo`.  The clone will not
> +  have the removed objects.  See linkgit:git-clone[1].  (Note that
> +  cloning with a plain path just hardlinks everything!)
> +
> +If you really don't want to clone it, for whatever reasons, check the
> +following points (in this order).  This is a very destructive
> +approach, so *make a backup* or go back to cloning it.  You have been
> +warned.

Sounds sane so far and even somewhat amusing.

> +* Make sure you really removed all variants of a filename, if a blob
> +  was moved over its lifetime.  `git log \--follow \--all \-- foo` can
> +  help you find renames.
> +
> +* Make sure you really filtered all refs: use `\--tag-name-filter cat
> +  \-- \--all` when calling git-filter-branch.

Don't these two apply equally to "git clone" approach?  Only the remaining
four are specific to in-place pruning, if I am reading this correctly.

> +* Make sure you remove the original refs backed up by
> +  git-filter-branch: say `git for-each-ref \--format="%(refname)"
> +  refs/original/ | xargs -n 1 git update-ref -d`.
> +
> +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> +
> +* Repack to remove packed objects with `git repack -ad`.
> +
> +* Prune unpacked unreferenced objects with `git prune \--expire=now`.
> +
> +
>  Author
>  ------
>  Written by Petr "Pasky" Baudis <pasky@suse.cz>,
> -- 
> 1.6.2.rc0.274.g97213
>
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

[Jan Krüger]

Hi,

On Fri, 13 Feb 2009 18:14:39 +0100, Thomas Rast <trast@student.ethz.ch>
wrote:

> Add a section about how to shrink a repository's size after running
> git-filter-branch to remove large blobs from history.

Good idea.

> +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> +
> +* Repack to remove packed objects with `git repack -ad`.
> +
> +* Prune unpacked unreferenced objects with `git prune \--expire=now`.

Is the --expire=now actually needed for prune? The way I read it
git-prune(1), it defaults to pruning everything anyway.

-Jan

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1040 bytes --]

Hi,

On Sat, 14 Feb 2009, Jan Krüger wrote:

> On Fri, 13 Feb 2009 18:14:39 +0100, Thomas Rast <trast@student.ethz.ch>
> wrote:
> 
> > Add a section about how to shrink a repository's size after running
> > git-filter-branch to remove large blobs from history.
> 
> Good idea.

Indeed.  It is funny when some people ask, but reading it several times a 
week in #git makes that fun go away real quick.

> > +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> > +
> > +* Repack to remove packed objects with `git repack -ad`.
> > +
> > +* Prune unpacked unreferenced objects with `git prune \--expire=now`.
> 
> Is the --expire=now actually needed for prune? The way I read it
> git-prune(1), it defaults to pruning everything anyway.

I hope not.  IIRC I set the default to 2 weeks.

*goes to read the source* Ah, that default is only set in "git gc", and 
"git prune" still prunes everything.  So you're absolutely correct.

But maybe we do not want to direct the user to use "prune" here, but 
rather "gc"?

Ciao,
Dscho

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

[Jan Krüger]

Hi,

On Sat, 14 Feb 2009 06:46:18 +0100 (CET), Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:

> > Is the --expire=now actually needed for prune? The way I read it
> > git-prune(1), it defaults to pruning everything anyway.
> 
> [...] Ah, that default is only set in "git gc", and "git prune" still
> prunes everything. [...] But maybe we do not want to direct the user
> to use "prune" here, but rather "gc"?

I just read the source code of gc and noticed that the only way to
influence its expiration limit for prune is to set it in the config file
(gc.pruneExpire). Thus there is no immediate way to tell gc to prune
*all* unreachable objects and we need to stick to the instructions that
are in the patch (minus the unnecessary --expire=now for prune). The
alternative would be to add a command line option to gc that
corresponds to gc.pruneExpire. I don't really care about that, but a
patch for it is below.

-Jan

-----8<-----

From 42226c6c542be4e9fff0817662115f180033e75a Mon Sep 17 00:00:00 2001
From: =?utf-8?q?Jan=20Kr=C3=BCger?= <jk@jk.gs>
Date: Sat, 14 Feb 2009 07:39:23 +0100
Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

In semi-regular intervals, people tend to want to get rid of huge
objects they accidentally created in a repository. Of course it
shouldn't be too easy to do but also not too hard. Thus we allow them to
get rid of unreferenced objects more easily once they have managed to
make the objects unreferenced, by allowing them to override the prune
expiration limit.

Signed-off-by: Jan Krüger <jk@jk.gs>
---
 Documentation/git-gc.txt |    5 +++++
 builtin-gc.c             |    5 +++--
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 7086eea..a4d35fb 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -59,6 +59,11 @@ are consolidated into a single pack by using the `-A` option of
 'git-repack'. Setting `gc.autopacklimit` to 0 disables
 automatic consolidation of packs.
 
+--prune-expire <time>::
+	Expire unreachable objects older than <time>, rather than
+	considering the configuration variable or the default explained in
+	the next section.
+
 --quiet::
 	Suppress all progress reports.
 
diff --git a/builtin-gc.c b/builtin-gc.c
index a201438..a62c762 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -179,16 +179,17 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	int quiet = 0;
 	char buf[80];
 
+	git_config(gc_config, NULL);
+
 	struct option builtin_gc_options[] = {
 		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects (deprecated)"),
+		OPT_STRING(0, "prune-expire", &prune_expire, "time", "expire unreferenced objects older than <time>"),
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
 		OPT_BOOLEAN('q', "quiet", &quiet, "suppress progress reports"),
 		OPT_END()
 	};
 
-	git_config(gc_config, NULL);
-
 	if (pack_refs < 0)
 		pack_refs = !is_bare_repository();
 
-- 
1.6.2.rc0.61.g5cd12.dirty

Re: [PATCH] Documentation: pruning recipe for destructive filter-branch

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 222 bytes --]

Hi,

On Sat, 14 Feb 2009, Jan Krüger wrote:

> Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire

I like it, and if nobody beats me to it, I will add a test for this 
option tonight.

Ciao,
Dscho

[PATCH v2] Documentation: pruning recipe for destructive filter-branch

[Thomas Rast]

Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.

This comes up every week or so on IRC, and the commands required to
handle every case are not very newbie-friendly, so hopefully writing
them down somewhere leads to fewer questions.

It may seem contradictory to document fallbacks for older Gits in
newer docs, but we want to point people at this as a FAQ answer, and
they will frequently not have the newest version installed.

Thanks to Björn Steinbrink and Junio C Hamano for comments and
corrections.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Johannes Schindelin wrote:
> On Sat, 14 Feb 2009, Jan Krüger wrote:
> 
> > Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
> 
> I like it, and if nobody beats me to it, I will add a test for this 
> option tonight.

Cool, then let's use that in the docs.


 Documentation/git-filter-branch.txt |   36 +++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 1fbbbb4..d148e58 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -339,6 +339,42 @@ git filter-branch --index-filter \
 ---------------------------------------------------------------
 
 
+
+Checklist for Shrinking a Repository
+------------------------------------
+
+git-filter-branch is often used to get rid of a subset of files,
+usually with some combination of `\--index-filter` and
+`\--subdirectory-filter`.  If you want to physically shrink the
+repository afterwards, you have some choices:
+
+* Make sure you really removed all variants of a filename, if a blob
+  was moved over its lifetime.  `git log \--name-only \--follow \--all
+  \-- filename` can help you find renames.
+
+* Make sure you really filtered all refs: use `\--tag-name-filter cat
+  \-- \--all` when calling git-filter-branch.
+
+* Clone it with `git clone file:///path/to/repo`.  The clone will not
+  have the removed objects.  See linkgit:git-clone[1].  (Note that
+  cloning with a plain path just hardlinks everything!)
+
+If you really don't want to clone it, for whatever reasons, check the
+following points instead (in this order).  This is a very destructive
+approach, so *make a backup* or go back to cloning it.  You have been
+warned.
+
+* Remove the original refs backed up by git-filter-branch: say `git
+  for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git
+  update-ref -d`.
+
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.
+
+* Garbage collect all unreferenced objects with `git gc
+  \--prune-expire=now` (or if your git-gc is not new enough to have
+  that option, `git repack -ad; git prune`).
+
+
 Author
 ------
 Written by Petr "Pasky" Baudis <pasky@suse.cz>,
-- 
1.6.2.rc0.283.g181d4

[PATCH] gc: make --prune useful again by accepting an optional parameter

[Johannes Schindelin]

With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---

	Actually, when trying to add tests to Jan's patch, I realized that 
	I already have a patch in my personal tree, which makes --prune 
	accept an optional parameter...

	Jan, what do you think?  If you prefer --prune-expire, can you 
	please adapt my test?

 Documentation/git-gc.txt |    9 ++++++++-
 builtin-gc.c             |   19 ++++++++++++-------
 t/t5304-prune.sh         |   20 ++++++++++++++++++++
 3 files changed, 40 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 7086eea..fcef5fb 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository
 
 SYNOPSIS
 --------
-'git gc' [--aggressive] [--auto] [--quiet]
+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]
 
 DESCRIPTION
 -----------
@@ -59,6 +59,13 @@ are consolidated into a single pack by using the `-A` option of
 'git-repack'. Setting `gc.autopacklimit` to 0 disables
 automatic consolidation of packs.
 
+--prune=<date>::
+	Prune loose objects older than date (default is 2 weeks ago).
+	This option is on by default.
+
+--no-prune::
+	Do not prune any loose objects.
+
 --quiet::
 	Suppress all progress reports.
 
diff --git a/builtin-gc.c b/builtin-gc.c
index a201438..a962f38 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -161,7 +161,8 @@ static int need_to_gc(void)
 	 */
 	if (too_many_packs())
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now") ?
+			      "-a" : "-A",
 			      MAX_ADD);
 	else if (!too_many_loose_objects())
 		return 0;
@@ -173,14 +174,15 @@ static int need_to_gc(void)
 
 int cmd_gc(int argc, const char **argv, const char *prefix)
 {
-	int prune = 0;
 	int aggressive = 0;
 	int auto_gc = 0;
 	int quiet = 0;
 	char buf[80];
 
 	struct option builtin_gc_options[] = {
-		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects (deprecated)"),
+		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
+			"prune unreferenced objects (deprecated)",
+			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
 		OPT_BOOLEAN('q', "quiet", &quiet, "suppress progress reports"),
@@ -218,7 +220,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 			"\"git help gc\" for more information.\n");
 	} else
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now")
+			      ? "-a" : "-A",
 			      MAX_ADD);
 
 	if (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))
@@ -230,9 +233,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	if (run_command_v_opt(argv_repack, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_repack[0]);
 
-	argv_prune[2] = prune_expire;
-	if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
-		return error(FAILED_RUN, argv_prune[0]);
+	if (prune_expire) {
+		argv_prune[2] = prune_expire;
+		if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
+			return error(FAILED_RUN, argv_prune[0]);
+	}
 
 	if (run_command_v_opt(argv_rerere, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_rerere[0]);
diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
index 771c0a0..2e9c4a9 100755
--- a/t/t5304-prune.sh
+++ b/t/t5304-prune.sh
@@ -112,4 +112,24 @@ test_expect_success 'prune: do not prune heads listed as an argument' '
 
 '
 
+test_expect_success 'gc --no-prune && gc --prune=<date>' '
+
+	before=$(git count-objects | sed "s/ .*//") &&
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git gc --no-prune &&
+	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	git gc --prune=5002.days.ago &&
+	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	git gc --prune=5000.days.ago &&
+	test 0 = $(git count-objects | sed "s/ .*//") &&
+	test ! -f $BLOB_FILE
+
+'
+
 test_done
-- 
1.6.2.rc0.367.g7f9a5

Re: [PATCH v2] Documentation: pruning recipe for destructive filter-branch

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 994 bytes --]

Hi,

On Sat, 14 Feb 2009, Thomas Rast wrote:

> Johannes Schindelin wrote:
> > On Sat, 14 Feb 2009, Jan Krüger wrote:
> > 
> > > Subject: [PATCH] gc: add --prune-expire option equivalent to gc.pruneExpire
> > 
> > I like it, and if nobody beats me to it, I will add a test for this 
> > option tonight.
> 
> Cool, then let's use that in the docs.

This would be needed in addition, incase that my --prune=<date> patch is 
taken:

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index d148e58..a8ad884 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -371,7 +371,7 @@ warned.
 * Expire all reflogs with `git reflog expire \--expire=now \--all`.
 
 * Garbage collect all unreferenced objects with `git gc
-  \--prune-expire=now` (or if your git-gc is not new enough to have
+  \--prune=now` (or if your git-gc is not new enough to have
   that option, `git repack -ad; git prune`).
 
 
-- 
1.6.2.rc0.367.g7f9a5

Re: [PATCH v2] Documentation: pruning recipe for destructive filter-branch

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> +git-filter-branch is often used to get rid of a subset of files,
> +usually with some combination of `\--index-filter` and
> +`\--subdirectory-filter`.  If you want to physically shrink the
> +repository afterwards, you have some choices:
> +
> +* Make sure you really removed all variants of a filename, if a blob
> +  was moved over its lifetime.  `git log \--name-only \--follow \--all
> +  \-- filename` can help you find renames.
> +
> +* Make sure you really filtered all refs: use `\--tag-name-filter cat
> +  \-- \--all` when calling git-filter-branch.
> +
> +* Clone it with `git clone file:///path/to/repo`.  The clone will not
> +  have the removed objects.  See linkgit:git-clone[1].  (Note that
> +  cloning with a plain path just hardlinks everything!)
> +
> +If you really don't want to clone it, for whatever reasons, check the
> +following points instead (in this order).  This is a very destructive
> +approach, so *make a backup* or go back to cloning it.  You have been
> +warned.
> +
> +* Remove the original refs backed up by git-filter-branch: say `git
> +  for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git
> +  update-ref -d`.
> +
> +* Expire all reflogs with `git reflog expire \--expire=now \--all`.
> +
> +* Garbage collect all unreferenced objects with `git gc
> +  \--prune-expire=now` (or if your git-gc is not new enough to have
> +  that option, `git repack -ad; git prune`).

I am sorry, but the above does not flow well.  The first two common
preparatory steps appear as if it is only for the "clone" method.

	... often used to get rid of ... and people expect the resulting
        repository to be smaller than the original, but you need a few
        more steps to actually make it smaller, because git tries hard 
        not to lose your objects until you tell it to.  First make sure
	that:

	* you really removed all...

	* you reall filtered all...

	Then, there are two ways to get a smaller repository.  A safer way
	is to clone, that keeps your original intact.

	* Clone it with ...

	If you do not want to ...

	* Remove the original refs...

	* Expire all...

[PATCH v3] Documentation: pruning recipe for destructive filter-branch

[Thomas Rast]

Add a section about how to shrink a repository's size after running
git-filter-branch to remove large blobs from history.

This comes up every week or so on IRC, and the commands required to
handle every case are not very newbie-friendly, so hopefully writing
them down somewhere leads to fewer questions.

It may seem contradictory to document fallbacks for older Gits in
newer docs, but we want to point people at this as a FAQ answer, and
they will frequently not have the newest version installed.

Thanks to Björn Steinbrink and Junio C Hamano for comments and
corrections.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Junio C Hamano wrote:
> I am sorry, but the above does not flow well.  The first two common
> preparatory steps appear as if it is only for the "clone" method.

You're absolutely right.  I should have re-read it more thoroughly
after the rearrangement...

I also changed the --prune-expire to --prune, and changed the
surrounding wording since it's now not a new option any more, just new
semantics for an old one.

Furthermore, I noticed that asciidoc tries to turn file:///... into a
link, mangling the entire paragraph's formatting in the process.
Apparently +++quoting+++ helps.


 Documentation/git-filter-branch.txt |   41 +++++++++++++++++++++++++++++++++++
 1 files changed, 41 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 1fbbbb4..7ffe03f 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -339,6 +339,47 @@ git filter-branch --index-filter \
 ---------------------------------------------------------------
 
 
+
+Checklist for Shrinking a Repository
+------------------------------------
+
+git-filter-branch is often used to get rid of a subset of files,
+usually with some combination of `\--index-filter` and
+`\--subdirectory-filter`.  People expect the resulting repository to
+be smaller than the original, but you need a few more steps to
+actually make it smaller, because git tries hard not to lose your
+objects until you tell it to.  First make sure that:
+
+* You really removed all variants of a filename, if a blob was moved
+  over its lifetime.  `git log \--name-only \--follow \--all \--
+  filename` can help you find renames.
+
+* You really filtered all refs: use `\--tag-name-filter cat \--
+  \--all` when calling git-filter-branch.
+
+Then there are two ways to get a smaller repository.  A safer way is
+to clone, that keeps your original intact.
+
+* Clone it with `git clone +++file:///path/to/repo+++`.  The clone
+  will not have the removed objects.  See linkgit:git-clone[1].  (Note
+  that cloning with a plain path just hardlinks everything!)
+
+If you really don't want to clone it, for whatever reasons, check the
+following points instead (in this order).  This is a very destructive
+approach, so *make a backup* or go back to cloning it.  You have been
+warned.
+
+* Remove the original refs backed up by git-filter-branch: say `git
+  for-each-ref \--format="%(refname)" refs/original/ | xargs -n 1 git
+  update-ref -d`.
+
+* Expire all reflogs with `git reflog expire \--expire=now \--all`.
+
+* Garbage collect all unreferenced objects with `git gc \--prune=now`
+  (or if your git-gc is not new enough to support arguments to
+  `\--prune`, use `git repack -ad; git prune` instead).
+
+
 Author
 ------
 Written by Petr "Pasky" Baudis <pasky@suse.cz>,
-- 
1.6.2.rc0.296.ge2122

Re: [PATCH] gc: make --prune useful again by accepting an optional parameter

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

Johannes Schindelin wrote:
> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.

Prompted by discussion on IRC, I just spent some time staring at the
code, and for the code part and FWIW,

  Acked-by: Thomas Rast <trast@student.ethz.ch>

Meaning that for all I can see, it handles every combination of
config, --prune and --no-prune correctly.

I have some non-code nits however:

> +--prune=<date>::
> +	Prune loose objects older than date (default is 2 weeks ago).
> +	This option is on by default.

This isn't really true; to the user, it defaults to the value of
'gc.pruneExpire'.  That this is done by defaulting the state of
--prune to 2 weeks ago is an implementation detail.

> +		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
> +			"prune unreferenced objects (deprecated)",
> +			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },

You still call the option "deprecated" :-)

> diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
[...]
> +test_expect_success 'gc --no-prune && gc --prune=<date>' '
> +
> +	before=$(git count-objects | sed "s/ .*//") &&
> +	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
> +	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
> +	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
> +	test -f $BLOB_FILE &&
> +	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
> +	git gc --no-prune &&
> +	test 1 = $(git count-objects | sed "s/ .*//") &&
> +	test -f $BLOB_FILE &&
> +	git gc --prune=5002.days.ago &&
> +	test 1 = $(git count-objects | sed "s/ .*//") &&
> +	test -f $BLOB_FILE &&
> +	git gc --prune=5000.days.ago &&
> +	test 0 = $(git count-objects | sed "s/ .*//") &&
> +	test ! -f $BLOB_FILE
> +
> +'
> +

I'm not a huge fan of such contiguous blocks, as a failure is harder
to pin to any of the several tests inside, especially with 'test' not
showing any output when it fails.

Also, it does test the interaction between having gc.pruneExpire set,
but passing --no-prune: the tests above leave gc.pruneExpire at
"2.days.ago".  This wasn't immediately obvious however, and you should
probably set the config again just in case someone decides to insert a
test with a different value.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH] gc: make --prune useful again by accepting an optional parameter

[Jan Krüger]

Hi,

> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.

I like it, so disregard my patch.

> diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
> index 7086eea..fcef5fb 100644
> --- a/Documentation/git-gc.txt
> +++ b/Documentation/git-gc.txt
> @@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local
> repository
>
>  SYNOPSIS
>  --------
> -'git gc' [--aggressive] [--auto] [--quiet]
> +'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]

Maybe add --no-prune too.

>
>  DESCRIPTION
>  -----------
> @@ -59,6 +59,13 @@ are consolidated into a single pack by using the `-A`
> option of
>  'git-repack'. Setting `gc.autopacklimit` to 0 disables
>  automatic consolidation of packs.
>
> +--prune=<date>::
> +	Prune loose objects older than date (default is 2 weeks ago).
> +	This option is on by default.

Maybe refer to the config option at least briefly.

> +
> +--no-prune::
> +	Do not prune any loose objects.
> +
>  --quiet::
>  	Suppress all progress reports.
>
> diff --git a/builtin-gc.c b/builtin-gc.c
> index a201438..a962f38 100644
> --- a/builtin-gc.c
> +++ b/builtin-gc.c
> [snip]
> @@ -173,14 +174,15 @@ static int need_to_gc(void)
>
>  int cmd_gc(int argc, const char **argv, const char *prefix)
>  {
> -	int prune = 0;
>  	int aggressive = 0;
>  	int auto_gc = 0;
>  	int quiet = 0;
>  	char buf[80];
>
>  	struct option builtin_gc_options[] = {
> -		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects
> (deprecated)"),
> +		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
> +			"prune unreferenced objects (deprecated)",

If you actually make it useful again shouldn't we stop marking it as
deprecated?

-Jan

Re: [PATCH] gc: make --prune useful again by accepting an optional parameter

[Johannes Schindelin]

Thanks for both of your sets of comments.

This is the interdiff.

---
 Documentation/git-gc.txt |    7 ++++---
 builtin-gc.c             |    2 +-
 t/t5304-prune.sh         |   24 +++++++++++++++++++++---
 3 files changed, 26 insertions(+), 7 deletions(-)

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index fcef5fb..b292e98 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository
 
 SYNOPSIS
 --------
-'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date>]
+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune]
 
 DESCRIPTION
 -----------
@@ -60,8 +60,9 @@ are consolidated into a single pack by using the `-A` option of
 automatic consolidation of packs.
 
 --prune=<date>::
-	Prune loose objects older than date (default is 2 weeks ago).
-	This option is on by default.
+	Prune loose objects older than date (default is 2 weeks ago,
+	overrideable by the config variable `gc.pruneExpire`).  This
+	option is on by default.
 
 --no-prune::
 	Do not prune any loose objects.
diff --git a/builtin-gc.c b/builtin-gc.c
index 06bc83c..fc556ed 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -181,7 +181,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 
 	struct option builtin_gc_options[] = {
 		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
-			"prune unreferenced objects (deprecated)",
+			"prune unreferenced objects",
 			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
index 2e9c4a9..55ed7c7 100755
--- a/t/t5304-prune.sh
+++ b/t/t5304-prune.sh
@@ -112,7 +112,7 @@ test_expect_success 'prune: do not prune heads listed as an argument' '
 
 '
 
-test_expect_success 'gc --no-prune && gc --prune=<date>' '
+test_expect_success 'gc --no-prune' '
 
 	before=$(git count-objects | sed "s/ .*//") &&
 	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
@@ -120,14 +120,32 @@ test_expect_success 'gc --no-prune && gc --prune=<date>' '
 	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
 	test -f $BLOB_FILE &&
 	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git config gc.pruneExpire 2.days.ago &&
 	git gc --no-prune &&
 	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc respects gc.pruneExpire' '
+
+	git config gc.pruneExpire 5002.days.ago &&
+	git gc &&
 	test -f $BLOB_FILE &&
+	git config gc.pruneExpire 5000.days.ago &&
+	git gc &&
+	test ! -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc --prune=<date>' '
+
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
 	git gc --prune=5002.days.ago &&
-	test 1 = $(git count-objects | sed "s/ .*//") &&
 	test -f $BLOB_FILE &&
 	git gc --prune=5000.days.ago &&
-	test 0 = $(git count-objects | sed "s/ .*//") &&
 	test ! -f $BLOB_FILE
 
 '
-- 
1.6.2.rc0.367.g7f9a5

[PATCH v2] gc: make --prune useful again by accepting an optional parameter

[Johannes Schindelin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4743 bytes --]

With this patch, "git gc --no-prune" will not prune any loose (and
dangling) object, and "git gc --prune=5.minutes.ago" will prune
all loose objects older than 5 minutes.

This patch benefitted from suggestions by Thomas Rast and Jan Krüger.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Documentation/git-gc.txt |   10 +++++++++-
 builtin-gc.c             |   19 ++++++++++++-------
 t/t5304-prune.sh         |   38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 59 insertions(+), 8 deletions(-)

diff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt
index 7086eea..b292e98 100644
--- a/Documentation/git-gc.txt
+++ b/Documentation/git-gc.txt
@@ -8,7 +8,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository
 
 SYNOPSIS
 --------
-'git gc' [--aggressive] [--auto] [--quiet]
+'git gc' [--aggressive] [--auto] [--quiet] [--prune=<date> | --no-prune]
 
 DESCRIPTION
 -----------
@@ -59,6 +59,14 @@ are consolidated into a single pack by using the `-A` option of
 'git-repack'. Setting `gc.autopacklimit` to 0 disables
 automatic consolidation of packs.
 
+--prune=<date>::
+	Prune loose objects older than date (default is 2 weeks ago,
+	overrideable by the config variable `gc.pruneExpire`).  This
+	option is on by default.
+
+--no-prune::
+	Do not prune any loose objects.
+
 --quiet::
 	Suppress all progress reports.
 
diff --git a/builtin-gc.c b/builtin-gc.c
index a201438..8d990ed 100644
--- a/builtin-gc.c
+++ b/builtin-gc.c
@@ -161,7 +161,8 @@ static int need_to_gc(void)
 	 */
 	if (too_many_packs())
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now") ?
+			      "-a" : "-A",
 			      MAX_ADD);
 	else if (!too_many_loose_objects())
 		return 0;
@@ -173,14 +174,15 @@ static int need_to_gc(void)
 
 int cmd_gc(int argc, const char **argv, const char *prefix)
 {
-	int prune = 0;
 	int aggressive = 0;
 	int auto_gc = 0;
 	int quiet = 0;
 	char buf[80];
 
 	struct option builtin_gc_options[] = {
-		OPT_BOOLEAN(0, "prune", &prune, "prune unreferenced objects (deprecated)"),
+		{ OPTION_STRING, 0, "prune", &prune_expire, "date",
+			"prune unreferenced objects",
+			PARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },
 		OPT_BOOLEAN(0, "aggressive", &aggressive, "be more thorough (increased runtime)"),
 		OPT_BOOLEAN(0, "auto", &auto_gc, "enable auto-gc mode"),
 		OPT_BOOLEAN('q', "quiet", &quiet, "suppress progress reports"),
@@ -218,7 +220,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 			"\"git help gc\" for more information.\n");
 	} else
 		append_option(argv_repack,
-			      !strcmp(prune_expire, "now") ? "-a" : "-A",
+			      prune_expire && !strcmp(prune_expire, "now")
+			      ? "-a" : "-A",
 			      MAX_ADD);
 
 	if (pack_refs && run_command_v_opt(argv_pack_refs, RUN_GIT_CMD))
@@ -230,9 +233,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 	if (run_command_v_opt(argv_repack, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_repack[0]);
 
-	argv_prune[2] = prune_expire;
-	if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
-		return error(FAILED_RUN, argv_prune[0]);
+	if (prune_expire) {
+		argv_prune[2] = prune_expire;
+		if (run_command_v_opt(argv_prune, RUN_GIT_CMD))
+			return error(FAILED_RUN, argv_prune[0]);
+	}
 
 	if (run_command_v_opt(argv_rerere, RUN_GIT_CMD))
 		return error(FAILED_RUN, argv_rerere[0]);
diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh
index 771c0a0..55ed7c7 100755
--- a/t/t5304-prune.sh
+++ b/t/t5304-prune.sh
@@ -112,4 +112,42 @@ test_expect_success 'prune: do not prune heads listed as an argument' '
 
 '
 
+test_expect_success 'gc --no-prune' '
+
+	before=$(git count-objects | sed "s/ .*//") &&
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test $((1 + $before)) = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git config gc.pruneExpire 2.days.ago &&
+	git gc --no-prune &&
+	test 1 = $(git count-objects | sed "s/ .*//") &&
+	test -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc respects gc.pruneExpire' '
+
+	git config gc.pruneExpire 5002.days.ago &&
+	git gc &&
+	test -f $BLOB_FILE &&
+	git config gc.pruneExpire 5000.days.ago &&
+	git gc &&
+	test ! -f $BLOB_FILE
+
+'
+
+test_expect_success 'gc --prune=<date>' '
+
+	BLOB=$(echo aleph_0 | git hash-object -w --stdin) &&
+	BLOB_FILE=.git/objects/$(echo $BLOB | sed "s/^../&\//") &&
+	test-chmtime =-$((86400*5001)) $BLOB_FILE &&
+	git gc --prune=5002.days.ago &&
+	test -f $BLOB_FILE &&
+	git gc --prune=5000.days.ago &&
+	test ! -f $BLOB_FILE
+
+'
+
 test_done
-- 
1.6.2.rc0.367.g7f9a5

Re: [PATCH v2] gc: make --prune useful again by accepting an optional parameter

[Thomas Rast]

[-- Attachment #1: Type: text/plain, Size: 652 bytes --]

Johannes Schindelin wrote:
> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.
> 
> This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
> 
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  Documentation/git-gc.txt |   10 +++++++++-
>  builtin-gc.c             |   19 ++++++++++++-------
>  t/t5304-prune.sh         |   38 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 59 insertions(+), 8 deletions(-)

Ack.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [PATCH v2] gc: make --prune useful again by accepting an optional parameter

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> With this patch, "git gc --no-prune" will not prune any loose (and
> dangling) object, and "git gc --prune=5.minutes.ago" will prune
> all loose objects older than 5 minutes.
>
> This patch benefitted from suggestions by Thomas Rast and Jan Krüger.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Thanks.

Being able to say --no-$opt to an OPT_STRING to NULLify a variable that
has non-NULL default value is somewhat amusing ;-)