[PATCH 1/3] Add --blob-filter option to filter-branch.

[PATCH 1/3] Add --blob-filter option to filter-branch.

[Avery Pennarun]

From: Jeff King <peff@peff.net>

On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:

> Do you think git would benefit from having a generalized version of
> this script?  Basically, the user provides a "munge" script on the
> command line, and there's a git-filter-branch mode for auto-munging
> (with a cache) every file in every checkin.  Even if it's *only* ever
> used for CRLF, I can imagine this being useful to a lot of people.

It was easy enough to work up the patch below, which allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible,
but running a tight loop like this is just painful. I suspect
filter-branch in something like perl would be a lot faster and just as
flexible (you could even do it in C, but you'd probably have to invent a
little domain-specific scripting language).

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Lots of system time in both. I'm sure we spend a fair bit of time
hitting our very large map and blob-cache directories, which would be
much more nicely implemented as associative arrays in memory (if we were
using a more featureful language).

Anyway, here is the patch. I don't know if it is even worth applying,
since it is still painfully slow.

Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 git-filter-branch.sh |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ea59015..980c431 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -54,6 +54,23 @@ EOF
 
 eval "$functions"
 
+munge_blobs() {
+	while read mode sha1 stage path
+	do
+		if ! test -r "$workdir/../blob-cache/$sha1"
+		then
+			new=`git cat-file blob $sha1 |
+			     eval "$filter_blob" |
+			     git hash-object -w --stdin`
+			printf $new >$workdir/../blob-cache/$sha1
+		fi
+		printf "%s %s\t%s\n" \
+			"$mode" \
+			$(cat "$workdir/../blob-cache/$sha1") \
+			"$path"
+	done
+}
+
 # When piped a commit, output a script to set the ident of either
 # "author" or "committer
 
@@ -105,6 +122,7 @@ tempdir=.git-rewrite
 filter_env=
 filter_tree=
 filter_index=
+filter_blob=
 filter_parent=
 filter_msg=cat
 filter_commit='git commit-tree "$@"'
@@ -150,6 +168,9 @@ do
 	--index-filter)
 		filter_index="$OPTARG"
 		;;
+	--blob-filter)
+		filter_blob="$OPTARG"
+		;;
 	--parent-filter)
 		filter_parent="$OPTARG"
 		;;
@@ -227,6 +248,9 @@ ret=0
 # map old->new commit ids for rewriting parents
 mkdir ../map || die "Could not create map/ directory"
 
+# cache rewritten blobs for blob filter
+mkdir ../blob-cache || die "Could not create blob-cache/ directory"
+
 case "$filter_subdir" in
 "")
 	git rev-list --reverse --topo-order --default HEAD \
@@ -295,6 +319,12 @@ while read commit parents; do
 	eval "$filter_index" < /dev/null ||
 		die "index filter failed: $filter_index"
 
+	if test -n "$filter_blob"; then
+		git ls-files --stage |
+		munge_blobs |
+		git update-index --index-info
+	fi
+
 	parentstr=
 	for parent in $parents; do
 		for reparent in $(map "$parent"); do
-- 
1.5.4.3

[PATCH 2/3] Make filter-branch --glob-filter much faster by not calling 'cat'

[Avery Pennarun]

The main loop of munge_blobs() had to fork-exec "cat" every time through the
loop, even when a blob was already cached.  Let's use the sh builtin 'read'
instead for a huge speedup.

cd git
time git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD

(original --blob-filter)
real    3m58.569s
user    0m22.900s
sys     3m32.030s

(with 'cat' calls removed)
real	1m11.931s
user	0m8.520s
sys	1m2.900s

(with 'cat' calls removed and blob cache already filled)
real	0m19.660s
user	0m3.930s
sys	0m15.720s

Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
---
 git-filter-branch.sh |   18 +++++++++++-------
 1 files changed, 11 insertions(+), 7 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 980c431..37ac99d 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -57,16 +57,18 @@ eval "$functions"
 munge_blobs() {
 	while read mode sha1 stage path
 	do
-		if ! test -r "$workdir/../blob-cache/$sha1"
+		if ! test -r "$cachedir/$sha1"
 		then
-			new=`git cat-file blob $sha1 |
-			     eval "$filter_blob" |
-			     git hash-object -w --stdin`
-			printf $new >$workdir/../blob-cache/$sha1
+			new=$(git cat-file blob $sha1 |
+			      eval "$filter_blob" |
+			      git hash-object -w --stdin)
+			printf $new >$cachedir/$sha1
+		else
+			read new <"$cachedir/$sha1"
 		fi
 		printf "%s %s\t%s\n" \
 			"$mode" \
-			$(cat "$workdir/../blob-cache/$sha1") \
+			"$new" \
 			"$path"
 	done
 }
@@ -108,6 +110,7 @@ USAGE="[--env-filter <command>] [--tree-filter <command>] \
 [--index-filter <command>] [--parent-filter <command>] \
 [--msg-filter <command>] [--commit-filter <command>] \
 [--tag-name-filter <command>] [--subdirectory-filter <directory>] \
+[--blob-filter <command>] \
 [--original <namespace>] [-d <directory>] [-f | --force] \
 [<rev-list options>...]"
 
@@ -249,7 +252,8 @@ ret=0
 mkdir ../map || die "Could not create map/ directory"
 
 # cache rewritten blobs for blob filter
-mkdir ../blob-cache || die "Could not create blob-cache/ directory"
+cachedir="$workdir/../blob-cache"
+mkdir "$cachedir" || die "Could not create blob-cache/ directory"
 
 case "$filter_subdir" in
 "")
-- 
1.5.4.3

[PATCH 3/3] Update documentation to describe git filter-branch --blob-filter.

[Avery Pennarun]

Signed-off-by: Avery Pennarun <apenwarr@gmail.com>
---
 Documentation/git-filter-branch.txt |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 2a78549..367f119 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -12,6 +12,7 @@ SYNOPSIS
 	[--index-filter <command>] [--parent-filter <command>]
 	[--msg-filter <command>] [--commit-filter <command>]
 	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
+	[--blob-filter <command]
 	[--original <namespace>] [-d <directory>] [-f | --force]
 	[<rev-list options>...]
 
@@ -143,6 +144,15 @@ definition impossible to preserve signatures at any rate.)
 	The result will contain that directory (and only that) as its
 	project root.
 
+--blob-filter <command>::
+	This is the filter for modifying the contents of each file (blob)
+	in the tree.  The contents of a file are provided on stdin, and
+	the new file contents should be provided on stdout.  For efficiency,
+	the before/after results of a given blob are only calculated once
+	and then cached, so your filter must always return the same output
+	blob for any given input blob.  You might use this filter for
+	converting CRLF to LF in all your files, for example.
+
 --original <namespace>::
 	Use this option to set the namespace where the original commits
 	will be stored. The default value is 'refs/original'.
@@ -185,6 +195,13 @@ git filter-branch --index-filter 'git update-index --remove filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+To convert CRLF to LF in all your files using the "fromdos" program (be
+careful: this will attempt to modify binary files too!):
+
+----------------------------------------------
+git filter-branch --blob-filter 'fromdos' HEAD
+----------------------------------------------
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:
-- 
1.5.4.3

Re: [PATCH 1/3] Add --blob-filter option to filter-branch.

[Johannes Schindelin]

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

> From: Jeff King <peff@peff.net>
> 
> On Tue, Apr 22, 2008 at 12:51:14PM -0400, Avery Pennarun wrote:
> 
> > Do you think git would benefit from having a generalized version of
> > this script?  Basically, the user provides a "munge" script on the
> > command line, and there's a git-filter-branch mode for auto-munging
> > (with a cache) every file in every checkin.  Even if it's *only* ever
> > used for CRLF, I can imagine this being useful to a lot of people.
> 
> It was easy enough to work up the patch below, which allows
> 
>   git filter-branch --blob-filter 'tr a-z A-Z'
> 
> However, it's _still_ horribly slow. Shell script is nice and flexible,
> but running a tight loop like this is just painful. I suspect
> filter-branch in something like perl would be a lot faster and just as
> flexible (you could even do it in C, but you'd probably have to invent a
> little domain-specific scripting language).
> 
> It is still much better performance than a tree filter, though:
> 
>   $ cd git && time git filter-branch --tree-filter '
>       find . -type f | while read f; do
>         tr a-z A-Z <"$f" >tmp
>         mv tmp "$f"
>       done
>     ' HEAD~10..HEAD
> 
>   real    4m38.626s
>   user    1m32.726s
>   sys     2m51.163s
> 
>   $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
>   real    1m40.809s
>   user    0m36.822s
>   sys     1m14.273s
> 
> Lots of system time in both. I'm sure we spend a fair bit of time
> hitting our very large map and blob-cache directories, which would be
> much more nicely implemented as associative arrays in memory (if we were
> using a more featureful language).
> 
> Anyway, here is the patch. I don't know if it is even worth applying,
> since it is still painfully slow.

Not all of this belongs in the commit messaage.

> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This does.

A good general rule is: if you think it would be funny/strange to read 
this message in the output of "git log", it should be changed.

Ciao,
Dscho

Re: [PATCH 1/3] Add --blob-filter option to filter-branch.

[Avery Pennarun]

On 4/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Not all of this belongs in the commit messaage.
>
>  > Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> This does.
>
>  A good general rule is: if you think it would be funny/strange to read
>  this message in the output of "git log", it should be changed.

I felt uncomfortable modifying the message attached to Jeff's patch
(which actually does pretty clearly explain what's going on), since
it's not mine.  If he wants to send me a better commit message,
that'll be fine too.

Have fun,

Avery

Re: [PATCH 1/3] Add --blob-filter option to filter-branch.

[Johannes Schindelin]

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

> On 4/23/08, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > Not all of this belongs in the commit messaage.
> >
> >  > Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > This does.
> >
> >  A good general rule is: if you think it would be funny/strange to 
> >  read this message in the output of "git log", it should be changed.
> 
> I felt uncomfortable modifying the message attached to Jeff's patch
> (which actually does pretty clearly explain what's going on), since
> it's not mine.  If he wants to send me a better commit message,
> that'll be fine too.

Well, I think that you would do a good job rephrasing this as a proper 
commit message :-)

Ciao,
Dscho

[PATCH 1/3 v2] Add --blob-filter option to filter-branch.

[Avery Pennarun]

From: Jeff King <peff@peff.net>

This patch allows

  git filter-branch --blob-filter 'tr a-z A-Z'

However, it's _still_ horribly slow. Shell script is nice and flexible,
but running a tight loop like this is just painful. I suspect
filter-branch in something like perl would be a lot faster and just as
flexible.

It is still much better performance than a tree filter, though:

  $ cd git && time git filter-branch --tree-filter '
      find . -type f | while read f; do
        tr a-z A-Z <"$f" >tmp
        mv tmp "$f"
      done
    ' HEAD~10..HEAD

  real    4m38.626s
  user    1m32.726s
  sys     2m51.163s

  $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
  real    1m40.809s
  user    0m36.822s
  sys     1m14.273s

Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 git-filter-branch.sh |   30 ++++++++++++++++++++++++++++++
 1 files changed, 30 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ea59015..980c431 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -54,6 +54,23 @@ EOF
 
 eval "$functions"
 
+munge_blobs() {
+	while read mode sha1 stage path
+	do
+		if ! test -r "$workdir/../blob-cache/$sha1"
+		then
+			new=`git cat-file blob $sha1 |
+			     eval "$filter_blob" |
+			     git hash-object -w --stdin`
+			printf $new >$workdir/../blob-cache/$sha1
+		fi
+		printf "%s %s\t%s\n" \
+			"$mode" \
+			$(cat "$workdir/../blob-cache/$sha1") \
+			"$path"
+	done
+}
+
 # When piped a commit, output a script to set the ident of either
 # "author" or "committer
 
@@ -105,6 +122,7 @@ tempdir=.git-rewrite
 filter_env=
 filter_tree=
 filter_index=
+filter_blob=
 filter_parent=
 filter_msg=cat
 filter_commit='git commit-tree "$@"'
@@ -150,6 +168,9 @@ do
 	--index-filter)
 		filter_index="$OPTARG"
 		;;
+	--blob-filter)
+		filter_blob="$OPTARG"
+		;;
 	--parent-filter)
 		filter_parent="$OPTARG"
 		;;
@@ -227,6 +248,9 @@ ret=0
 # map old->new commit ids for rewriting parents
 mkdir ../map || die "Could not create map/ directory"
 
+# cache rewritten blobs for blob filter
+mkdir ../blob-cache || die "Could not create blob-cache/ directory"
+
 case "$filter_subdir" in
 "")
 	git rev-list --reverse --topo-order --default HEAD \
@@ -295,6 +319,12 @@ while read commit parents; do
 	eval "$filter_index" < /dev/null ||
 		die "index filter failed: $filter_index"
 
+	if test -n "$filter_blob"; then
+		git ls-files --stage |
+		munge_blobs |
+		git update-index --index-info
+	fi
+
 	parentstr=
 	for parent in $parents; do
 		for reparent in $(map "$parent"); do
-- 
1.5.4.3

Re: [PATCH 1/3 v2] Add --blob-filter option to filter-branch.

[Johannes Schindelin]

Hi,

On Wed, 23 Apr 2008, Avery Pennarun wrote:

> From: Jeff King <peff@peff.net>
> 
> This patch allows
> 
>   git filter-branch --blob-filter 'tr a-z A-Z'
> 
> However, it's _still_ horribly slow. Shell script is nice and flexible,
> but running a tight loop like this is just painful. I suspect
> filter-branch in something like perl would be a lot faster and just as
> flexible.
> 
> It is still much better performance than a tree filter, though:
> 
>   $ cd git && time git filter-branch --tree-filter '
>       find . -type f | while read f; do
>         tr a-z A-Z <"$f" >tmp
>         mv tmp "$f"
>       done
>     ' HEAD~10..HEAD
> 
>   real    4m38.626s
>   user    1m32.726s
>   sys     2m51.163s
> 
>   $ cd git && git filter-branch --blob-filter 'tr a-z A-Z' HEAD~10..HEAD
>   real    1m40.809s
>   user    0m36.822s
>   sys     1m14.273s
> 
> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---

Thanks, I really appreciate it.

Ciao,
Dscho

Re: [PATCH 1/3 v2] Add --blob-filter option to filter-branch.

[Jeff King]

On Wed, Apr 23, 2008 at 04:18:10PM -0400, Avery Pennarun wrote:

> From: Jeff King <peff@peff.net>
> 
> This patch allows
> 
>   git filter-branch --blob-filter 'tr a-z A-Z'

The commit message munging you did is fine.

However, I think Johannes Sixt's question about providing the pathname
needs to be resolved. As it is now, the blob-filter is impossible to use
in a mixed binary/text repository, short of the undocumented $path magic
that you described. And I am a little uncomfortable just adding the
$path as he suggested because of the subtle bug it introduces.

-Peff

Re: [PATCH 1/3 v2] Add --blob-filter option to filter-branch.

[Avery Pennarun]

On 4/23/08, Jeff King <peff@peff.net> wrote:
> On Wed, Apr 23, 2008 at 04:18:10PM -0400, Avery Pennarun wrote:
>
>  > From: Jeff King <peff@peff.net>
>  >
>  > This patch allows
>  >
>  >   git filter-branch --blob-filter 'tr a-z A-Z'
>
> The commit message munging you did is fine.
>
>  However, I think Johannes Sixt's question about providing the pathname
>  needs to be resolved. As it is now, the blob-filter is impossible to use
>  in a mixed binary/text repository, short of the undocumented $path magic
>  that you described. And I am a little uncomfortable just adding the
>  $path as he suggested because of the subtle bug it introduces.

It is indeed a very subtle bug; so subtle, in fact, that I never
expect to experience it myself :)

I think it would be fine to index into the cache using $path$sha1,
which would seem to resolve this issue.  The catch is that $path isn't
a very good cachefile name.  I'd suggest doing an md5sum or something
on it, but that would result in an extra fork for every file, which
brings us back to our original level of slowness (or worse).

Hmm, I gues using a cachefile like $sha1/$path would work; it requires
a "mkdir -p", but only when *filling* the cache.

Avery

Re: [PATCH 1/3 v2] Add --blob-filter option to filter-branch.

[Jeff King]

On Wed, Apr 23, 2008 at 06:07:03PM -0400, Avery Pennarun wrote:

> I think it would be fine to index into the cache using $path$sha1,
> which would seem to resolve this issue.  The catch is that $path isn't
> a very good cachefile name.  I'd suggest doing an md5sum or something
> on it, but that would result in an extra fork for every file, which
> brings us back to our original level of slowness (or worse).
> 
> Hmm, I gues using a cachefile like $sha1/$path would work; it requires
> a "mkdir -p", but only when *filling* the cache.

Keep in mind that $path can have slashes. So you actually need to:

   mkdir -p `dirname $sha1/$path`
   echo $new >$sha1/$path

-Peff