[PATCH] Teach filter-branch about subdirectory filtering

[PATCH] Teach filter-branch about subdirectory filtering

[Johannes Schindelin]

With git-filter-branch --subdirectory-filter <subdirectory> you can
get at the history, as seen by a certain subdirectory. The history
of the rewritten branch will only contain commits that touched that
subdirectory, and the subdirectory will be rewritten to be the new
project root.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 git-filter-branch.sh     |   33 ++++++++++++++++++++++++++++++---
 t/t7003-filter-branch.sh |   24 ++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 3 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index e6ed7b9..4990729 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -122,6 +122,10 @@
 #	attached, the rewritten tag won't have it. Sorry. (It is by
 #	definition impossible to preserve signatures at any rate, though.)
 #
+# --subdirectory-filter DIRECTORY:: Only regard the history, as seen by
+#	the given subdirectory. The result will contain that directory as
+#	its project root.
+#
 # EXAMPLE USAGE
 # -------------
 # Suppose you want to remove a file (containing confidential information
@@ -227,7 +231,13 @@ set_ident () {
 
 # list all parent's object names for a given commit
 get_parents () {
-	git-rev-list -1 --parents "$1" | sed "s/^[0-9a-f]*//"
+	case "$filter_subdir" in
+	"")
+		git-rev-list -1 --parents "$1"
+		;;
+	*)
+		git-rev-list -1 --parents "$1" -- "$filter_subdir"
+	esac | sed "s/^[0-9a-f]*//"
 }
 
 tempdir=.git-rewrite
@@ -238,6 +248,7 @@ filter_parent=
 filter_msg=cat
 filter_commit='git-commit-tree "$@"'
 filter_tag_name=
+filter_subdir=
 while case "$#" in 0) usage;; esac
 do
 	case "$1" in
@@ -283,6 +294,9 @@ do
 	--tag-name-filter)
 		filter_tag_name="$OPTARG"
 		;;
+	--subdirectory-filter)
+		filter_subdir="$OPTARG"
+		;;
 	*)
 		usage
 		;;
@@ -316,7 +330,14 @@ ret=0
 
 mkdir ../map # map old->new commit ids for rewriting parents
 
-git-rev-list --reverse --topo-order --default HEAD "$@" >../revs
+case "$filter_subdir" in
+"")
+	git-rev-list --reverse --topo-order --default HEAD "$@"
+	;;
+*)
+	git-rev-list --reverse --topo-order --default HEAD "$@" \
+		-- "$filter_subdir"
+esac > ../revs
 commits=$(cat ../revs | wc -l | tr -d " ")
 
 test $commits -eq 0 && die "Found nothing to rewrite"
@@ -326,7 +347,13 @@ while read commit; do
 	i=$(($i+1))
 	printf "\rRewriting commits... ($i/$commits)"
 
-	git-read-tree -i -m $commit
+	case "$filter_subdir" in
+	"")
+		git-read-tree -i -m $commit
+		;;
+	*)
+		git-read-tree -i -m $commit:"$filter_subdir"
+	esac
 
 	export GIT_COMMIT=$commit
 	git-cat-file commit "$commit" >../commit
diff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh
index 3739cb1..292b837 100755
--- a/t/t7003-filter-branch.sh
+++ b/t/t7003-filter-branch.sh
@@ -54,4 +54,28 @@ test_expect_success 'common ancestor is still common (unchanged)' '
 	test "$(git-merge-base modD D)" = "$(git-rev-parse B)"
 '
 
+test_expect_success 'filter subdirectory only' '
+	mkdir subdir &&
+	touch subdir/new &&
+	git add subdir/new &&
+	test_tick &&
+	git commit -m "subdir" &&
+	echo H > a &&
+	test_tick &&
+	git commit -m "not subdir" a &&
+	echo A > subdir/new &&
+	test_tick &&
+	git commit -m "again subdir" subdir/new &&
+	git rm a &&
+	test_tick &&
+	git commit -m "again not subdir" &&
+	git-filter-branch --subdirectory-filter subdir sub
+'
+
+test_expect_success 'subdirectory filter result looks okay' '
+	test 2 = $(git-rev-list sub | wc -l) &&
+	git show sub:new &&
+	! git show sub:subdir
+'
+
 test_done
-- 
1.5.2.1.2689.gaf768-dirty

Re: [PATCH] Teach filter-branch about subdirectory filtering

[Junio C Hamano]

Good thing you folded git-split into this, which is an obviously
right thing to do.

But it makes me wonder, if it is common to want to split a
combined project into two (or more) sets, taking more than one
directories at a time.  Contents of directory A and B (without
hoisting them up) are filtered into one history and remaining
bits (again without hoisting them up) are filtered into another.

I do not think that is common, and you can easily do that with
the index filter, I think, so it probably is pointless to make
this new filter to try supporting that mode of operation.

We might however want to make sure that we detect an attempt to
give more than one --subdirectory-filter parameter.

Re: [PATCH] Teach filter-branch about subdirectory filtering

[Johannes Schindelin]

Hi,

On Thu, 7 Jun 2007, Junio C Hamano wrote:

> Good thing you folded git-split into this, which is an obviously right 
> thing to do.

So my secritt evvil plann did not stay secritt that long.

> But it makes me wonder, if it is common to want to split a combined 
> project into two (or more) sets, taking more than one directories at a 
> time.

It's just a guess, but now that we come really, really close to having a 
concise implementation of git-subproject which will probably soon 
propagate to master, and then maint, I gather that more and more people 
come and want to split their projects (which they maintained as one big 
project) into several subprojects (which they should have been from the 
start, but the tool did not easily allow for that).

> We might however want to make sure that we detect an attempt to give 
> more than one --subdirectory-filter parameter.

Yeah. But then, I think that rev-list actually does not like to be passed 
a file/directory which does not exist currently, so that is probably a 
sensible safeguard against that usage.

Unless you think -- as I fear you do -- that people could think 
git-filter-branch (its name not withstanding) filters _multiple_ branches 
_at once_.

That case I did not foresee, and you're right, I should output a big fat 
error for the case when somebody says "filter-branch --subdirectory-filter 
a/ --subdirectory-filter b/". It does not make any sense. But humans are 
known for their unique feeling that things should work, even if they make 
no sense at all, and their brain should have warned them (that it is 
unlikely, at best, to work), but failed to do so.

Ciao,
Dscho

Re: [PATCH] Teach filter-branch about subdirectory filtering

[Johannes Sixt]

Johannes Schindelin wrote:
> It's just a guess, but now that we come really, really close to having a
> concise implementation of git-subproject which will probably soon
> propagate to master, and then maint, I gather that more and more people
> come and want to split their projects (which they maintained as one big
> project) into several subprojects (which they should have been from the
> start, but the tool did not easily allow for that).

I think that --subdirectry-filter needs to become a bit smarter to be
really useful for splitting a big project into sub-projects plus a
super-project. The reason is that once you have extracted the
sub-project(s), you have a hard time to find out which commits to
gitlink into the super-project. I don't have a plan how to make it
smarter, though.

-- Hannes

Re: [PATCH] Teach filter-branch about subdirectory filtering

[Johannes Schindelin]

Hi,

On Fri, 8 Jun 2007, Johannes Sixt wrote:

> Johannes Schindelin wrote:
> > It's just a guess, but now that we come really, really close to having 
> > a concise implementation of git-subproject which will probably soon 
> > propagate to master, and then maint, I gather that more and more 
> > people come and want to split their projects (which they maintained as 
> > one big project) into several subprojects (which they should have been 
> > from the start, but the tool did not easily allow for that).
> 
> I think that --subdirectry-filter needs to become a bit smarter to be 
> really useful for splitting a big project into sub-projects plus a 
> super-project. The reason is that once you have extracted the 
> sub-project(s), you have a hard time to find out which commits to 
> gitlink into the super-project. I don't have a plan how to make it 
> smarter, though.

Yes, it seems a good direction to follow.

I will have to think about it, but I guess that with yet another set of 
files in ../map/ (probably prefixed by the subdirectory name), we should 
even be able to smartly filter a huge project into a super-project 
together with its subprojects.

The command line option would be something like "--split-superproject 
dir1,dir2,dir3", where dir<n> are the subdirectories that are to become 
the subprojects.

Ciao,
Dscho

Re: [PATCH] Teach filter-branch about subdirectory filtering

[Johannes Sixt]

Johannes Schindelin wrote:
>  # list all parent's object names for a given commit
>  get_parents () {
> -       git-rev-list -1 --parents "$1" | sed "s/^[0-9a-f]*//"
> +       case "$filter_subdir" in
> +       "")
> +               git-rev-list -1 --parents "$1"
> +               ;;
> +       *)
> +               git-rev-list -1 --parents "$1" -- "$filter_subdir"
> +       esac | sed "s/^[0-9a-f]*//"
>  }

I think --full-history in the subdir case would make a lot of sense.

-- Hannes

Re: [PATCH] Teach filter-branch about subdirectory filtering

[Alex Riesen]

Johannes Schindelin, Fri, Jun 08, 2007 02:30:35 +0200:
> 
> With git-filter-branch --subdirectory-filter <subdirectory> you can
> get at the history, as seen by a certain subdirectory. The history
> of the rewritten branch will only contain commits that touched that
> subdirectory, and the subdirectory will be rewritten to be the new
> project root.
> 

Maybe it could be made more general

    --subdirectory-filter <from> [--subdirectory-filter-to <to>]

<to> meaning root by default, and allow multiple filters to be set?
You would have to deal with file name conflicts in case some filters
refer to the same target directory, though.

It has already been asked how to split a project into many projects
based on directory boundaries. That'd help...