[RFC PATCH 0/3] Submodule filtering for filter-branch

[RFC PATCH 0/3] Submodule filtering for filter-branch

[Thomas Rast]

Jeffrey Phillips Freeman aka tty1 asked on IRC about a way to fix the
super-project after rewriting submodules.  This quick series provides
some extra tricks in the filter-branch toolbox that can be plumbed
together to achieve this (see the last commit's message), and should
hopefully be general enough to be of other use too.

Comes without docs and tests at this point, hence RFC.  It's also only
lightly tested, but since Jeffrey has a use-case, I'll leave it up to
him to bend it a bit and see if anything breaks.


Thomas Rast (3):
  filter-branch: optionally dump all mappings at the end
  filter-branch: optionally load existing mappings prior to filtering
  filter-branch: support --submodule-filter

 git-filter-branch.sh |   64 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 64 insertions(+), 0 deletions(-)

-- 
1.7.4.rc0.240.g44e61

[RFC PATCH 1/3] filter-branch: optionally dump all mappings at the end

[Thomas Rast]

In some cases, like the sub/super-project filtering attempted in this
series, it may be necessary to carry over the mappings from one or
more filter-branch run to another.

As the first part to this, make a --dump-map option that dumps a flat
text file with lines of the form

  $sha1 $(map $sha1)

after it has established all mappings.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
 git-filter-branch.sh |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 962a93b..020b076 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -126,6 +126,7 @@ orig_namespace=refs/original/
 force=
 prune_empty=
 remap_to_ancestor=
+dump_map=
 while :
 do
 	case "$1" in
@@ -194,6 +195,16 @@ do
 	--original)
 		orig_namespace=$(expr "$OPTARG/" : '\(.*[^/]\)/*$')/
 		;;
+	--dump-map)
+		case "$OPTARG" in
+		/*)
+			dump_map="$OPTARG"
+			;;
+		*)
+			dump_map="$(pwd)/$OPTARG"
+			;;
+		esac
+		;;
 	*)
 		usage
 		;;
@@ -385,6 +396,16 @@ then
 	done < "$tempdir"/heads
 fi
 
+# At this point the mappings are stable so we can dump them if
+# requested
+
+if test -n "$dump_map"; then
+	( cd "$workdir"/../map/; ls ) |
+	while read sha1; do
+		echo $sha1 $(map $sha1)
+	done > "$dump_map"
+fi
+
 # Finally update the refs
 
 _x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
-- 
1.7.4.rc0.240.g44e61

Re: [RFC PATCH 1/3] filter-branch: optionally dump all mappings at the end

[Johannes Sixt]

On Freitag, 31. Dezember 2010, Thomas Rast wrote:
> In some cases, like the sub/super-project filtering attempted in this
> series, it may be necessary to carry over the mappings from one or
> more filter-branch run to another.
>
> As the first part to this, make a --dump-map option that dumps a flat
> text file with lines of the form
>
>   $sha1 $(map $sha1)

Despite the alternative implementation that I suggest below, the commit 
message (and also the documentation to be written) should state the output 
format in this way.

> @@ -194,6 +195,16 @@ do
>  	--original)
>  		orig_namespace=$(expr "$OPTARG/" : '\(.*[^/]\)/*$')/
>  		;;
> +	--dump-map)
> +		case "$OPTARG" in
> +		/*)

Please use

	if is_absolute_path "$OPTARG"
	then
	...

> +# At this point the mappings are stable so we can dump them if
> +# requested
> +
> +if test -n "$dump_map"; then
> +	( cd "$workdir"/../map/; ls ) |
> +	while read sha1; do
> +		echo $sha1 $(map $sha1)

There must be a more efficient way than to spawn an additional process for 
each mapped SHA1. Since this excercises only one branch in the map() function 
anyway, how about this:

	( cd "$workdir"/../map/ && ls -1 ) |
	while read sha1; do
		read mapped << $sha1 &&
		echo $sha1 $mapped
	done > "$dump_map"

Though, there is some mechanism that maps one SHA1s to more than one, and my 
approach might not cover that case.

> +	done > "$dump_map"

-- Hannes

[RFC PATCH 2/3] filter-branch: optionally load existing mappings prior to filtering

[Thomas Rast]

In the previous commit filter-branch learned to dump its mappings.
Introduce a --load-map option that lets it load map files in the same
format.

This option does not accumulate; only the last one counts.  Letting it
accumulate would require two argument-parsing passes since we need to
know the $tempdir before we can establish any maps.

Also, the maps add to all filtered maps.  This is a bit of a tradeoff.
As it stands, it could be useful to remap refs in another repository
using the map file from elsewere.  On the other hand, it does mean
that it may rewrite more refs than the user intended.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
 git-filter-branch.sh |   19 +++++++++++++++++++
 1 files changed, 19 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 020b076..9feeb26 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -127,6 +127,7 @@ force=
 prune_empty=
 remap_to_ancestor=
 dump_map=
+load_map=
 while :
 do
 	case "$1" in
@@ -205,6 +206,16 @@ do
 			;;
 		esac
 		;;
+	--load-map)
+		case "$OPTARG" in
+		/*)
+			load_map="$OPTARG"
+			;;
+		*)
+			load_map="$(pwd)/$OPTARG"
+			;;
+		esac
+		;;
 	*)
 		usage
 		;;
@@ -275,6 +286,14 @@ export GIT_INDEX_FILE
 # map old->new commit ids for rewriting parents
 mkdir ../map || die "Could not create map/ directory"
 
+if test -n "$load_map"
+then
+	while read pre post
+	do
+		echo $post > ../map/$pre
+	done < "$load_map"
+fi
+
 # we need "--" only if there are no path arguments in $@
 nonrevs=$(git rev-parse --no-revs "$@") || exit
 if test -z "$nonrevs"
-- 
1.7.4.rc0.240.g44e61

Re: [RFC PATCH 2/3] filter-branch: optionally load existing mappings prior to filtering

[Johannes Sixt]

On Freitag, 31. Dezember 2010, Thomas Rast wrote:
> +	--load-map)
> +		case "$OPTARG" in
> +		/*)

Please use is_absolute_path here as well.

-- Hannes

[RFC PATCH 3/3] filter-branch: support --submodule-filter

[Thomas Rast]

This new filter gets each submodule's current sha1 and path on stdin,
separated by a tab.  It can then emit a new submodule sha1 and/or
path, and filter-branch will:

* if the path differs, remove the submodule at the old path;

* add/replace the new sha1 at the new path.

Additionally, returning an empty new sha1 deletes the submodule.

You can tie this together with the last two commits to filter the
super-project after a subproject filtering as follows:

  ( cd sub1 && git filter-branch --dump-map map <filters|args> )
  ( cd sub2 && git filter-branch --dump-map map <filters|args> )
  cat sub1/map sub2/map > map
  git filter-branch --load-map map \
  	--submodule-filter "map $(cut -f1)" \
	<args>

Other use-cases should also be covered since we also pass through the
path.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---
 git-filter-branch.sh |   24 ++++++++++++++++++++++++
 1 files changed, 24 insertions(+), 0 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 9feeb26..4a321c4 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -122,6 +122,7 @@ filter_msg=cat
 filter_commit=
 filter_tag_name=
 filter_subdir=
+filter_submodule=
 orig_namespace=refs/original/
 force=
 prune_empty=
@@ -193,6 +194,9 @@ do
 		filter_subdir="$OPTARG"
 		remap_to_ancestor=t
 		;;
+	--submodule-filter)
+		filter_submodule="$OPTARG"
+		;;
 	--original)
 		orig_namespace=$(expr "$OPTARG/" : '\(.*[^/]\)/*$')/
 		;;
@@ -379,6 +383,26 @@ while read commit parents; do
 	eval "$filter_index" < /dev/null ||
 		die "index filter failed: $filter_index"
 
+	if [ "$filter_submodule" ]; then
+		git ls-files -s |
+		grep '^160000' |
+		while read mode sha1 stage path; do
+			printf "$sha1\t$path\n" |
+			{ eval "$filter_submodule" ||
+				die "submodule filter failed: $filter_submodule"; } |
+			read newsha1 newpath
+			if [ -z "$newsha1" ] || [ "$path" != "$newpath" ]; then
+				git update-index --remove "$path" ||
+					die "failed to remove submodule $path"
+			fi
+			if [ -n "$newsha1" ] && [ "$sha1" != "$newsha1" ]; then
+				git update-index --add --replace --cacheinfo \
+					160000 "$newsha1" "$newpath" ||
+					die "failed to add submodule $newpath"
+			fi
+		done
+	fi
+
 	parentstr=
 	for parent in $parents; do
 		for reparent in $(map "$parent"); do
-- 
1.7.4.rc0.240.g44e61

Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter

[Johannes Sixt]

On Freitag, 31. Dezember 2010, Thomas Rast wrote:
> This new filter gets each submodule's current sha1 and path on stdin,
> separated by a tab.  It can then emit a new submodule sha1 and/or
> path, and filter-branch will:
>
> * if the path differs, remove the submodule at the old path;
>
> * add/replace the new sha1 at the new path.
>
> Additionally, returning an empty new sha1 deletes the submodule.
>
> You can tie this together with the last two commits to filter the
> super-project after a subproject filtering as follows:
>
>   ( cd sub1 && git filter-branch --dump-map map <filters|args> )
>   ( cd sub2 && git filter-branch --dump-map map <filters|args> )
>   cat sub1/map sub2/map > map
>   git filter-branch --load-map map \
>   	--submodule-filter "map $(cut -f1)" \
> 	<args>
>
> Other use-cases should also be covered since we also pass through the
> path.

As I said, I'm not particularly fond of a new --submodule-filter because it is 
just a special --index-filter.

Your implementation is highly problematic:

> +	if [ "$filter_submodule" ]; then
> +		git ls-files -s |
> +		grep '^160000' |
> +		while read mode sha1 stage path; do
> +			printf "$sha1\t$path\n" |
> +			{ eval "$filter_submodule" ||
> +				die "submodule filter failed: $filter_submodule"; } |

This 'die' will not do anything useful except to write an error message.

> +			read newsha1 newpath

This 'read' is part of a pipe, and as such many shells run it in a sub-shell; 
the values that it reads do not survive the pipe, hence, the subsequent code 
does not do what you intend it.

In this case, you can put everything from 'read' to the last 'fi' below inside 
a block { } because there are no process states that need to survive the 
pipe.

> +			if [ -z "$newsha1" ] || [ "$path" != "$newpath" ]; then
> +				git update-index --remove "$path" ||
> +					die "failed to remove submodule $path"
> +			fi
> +			if [ -n "$newsha1" ] && [ "$sha1" != "$newsha1" ]; then
> +				git update-index --add --replace --cacheinfo \
> +					160000 "$newsha1" "$newpath" ||
> +					die "failed to add submodule $newpath"
> +			fi
> +		done

The whole if-branch is a pipe, and it's parts are run in a sub-shell (although 
shells are allowed to optimize this). This means that the 'die' calls will 
only exit the pipe, but not terminate filter-branch. You at least need to 
have '|| exit' behind the last 'fi' and &&-join the if-statements.

> +	fi
> +
>  	parentstr=
>  	for parent in $parents; do
>  		for reparent in $(map "$parent"); do

-- Hannes

Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter

[Jeffrey Phillips Freeman]

So im kinda new with all this so bare with me guys. I wanted to figure 
out how to apply these patches, now i know i can use git to do this with 
its patch command and such. However i was curious does this exist as a 
branch somewhere official or semi-official? I currently seem to be using 
--split-submodule which is itself in a git repo i have (which i want to 
also find its source repo so i can keep up to date with it). So applying 
the patch itself might be somewhat troubling due to conflicts, therefore 
id like to actually merge in a remote branch to keep things easy. So can 
you guys point me to which repo would be best for me to keep up to date 
with this would be?

On 12/31/2010 10:29 AM, Thomas Rast wrote:
> This new filter gets each submodule's current sha1 and path on stdin,
> separated by a tab.  It can then emit a new submodule sha1 and/or
> path, and filter-branch will:
>
> * if the path differs, remove the submodule at the old path;
>
> * add/replace the new sha1 at the new path.
>
> Additionally, returning an empty new sha1 deletes the submodule.
>
> You can tie this together with the last two commits to filter the
> super-project after a subproject filtering as follows:
>
>    ( cd sub1&&  git filter-branch --dump-map map<filters|args>  )
>    ( cd sub2&&  git filter-branch --dump-map map<filters|args>  )
>    cat sub1/map sub2/map>  map
>    git filter-branch --load-map map \
>    	--submodule-filter "map $(cut -f1)" \
> 	<args>
>
> Other use-cases should also be covered since we also pass through the
> path.
>
> Signed-off-by: Thomas Rast<trast@student.ethz.ch>
> ---
>   git-filter-branch.sh |   24 ++++++++++++++++++++++++
>   1 files changed, 24 insertions(+), 0 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index 9feeb26..4a321c4 100755
> --- a/git-filter-branch.sh
> +++ b/git-filter-branch.sh
> @@ -122,6 +122,7 @@ filter_msg=cat
>   filter_commit=
>   filter_tag_name=
>   filter_subdir=
> +filter_submodule=
>   orig_namespace=refs/original/
>   force=
>   prune_empty=
> @@ -193,6 +194,9 @@ do
>   		filter_subdir="$OPTARG"
>   		remap_to_ancestor=t
>   		;;
> +	--submodule-filter)
> +		filter_submodule="$OPTARG"
> +		;;
>   	--original)
>   		orig_namespace=$(expr "$OPTARG/" : '\(.*[^/]\)/*$')/
>   		;;
> @@ -379,6 +383,26 @@ while read commit parents; do
>   	eval "$filter_index"<  /dev/null ||
>   		die "index filter failed: $filter_index"
>
> +	if [ "$filter_submodule" ]; then
> +		git ls-files -s |
> +		grep '^160000' |
> +		while read mode sha1 stage path; do
> +			printf "$sha1\t$path\n" |
> +			{ eval "$filter_submodule" ||
> +				die "submodule filter failed: $filter_submodule"; } |
> +			read newsha1 newpath
> +			if [ -z "$newsha1" ] || [ "$path" != "$newpath" ]; then
> +				git update-index --remove "$path" ||
> +					die "failed to remove submodule $path"
> +			fi
> +			if [ -n "$newsha1" ]&&  [ "$sha1" != "$newsha1" ]; then
> +				git update-index --add --replace --cacheinfo \
> +					160000 "$newsha1" "$newpath" ||
> +					die "failed to add submodule $newpath"
> +			fi
> +		done
> +	fi
> +
>   	parentstr=
>   	for parent in $parents; do
>   		for reparent in $(map "$parent"); do

Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter

[Thomas Rast]

Please don't top-post.  (There's nothing wrong with snipping the whole
message if it does not really relate, as in this case.)

Ccs for the patch at the end; I don't really care much but I could
roll all of it into a sort of "submodule tools" series for g-f-b, so
if you like it, speak up.

Jeffrey Phillips Freeman wrote:
> So im kinda new with all this so bare with me guys. I wanted to figure 
> out how to apply these patches, now i know i can use git to do this with 
> its patch command and such. However i was curious does this exist as a 
> branch somewhere official or semi-official?

Not really.

You can dig through the mailing list archives and also e.g.
gitworkflows to see how Junio handles incoming patches, but for series
like this you usually have to apply them yourself.  I deliberately
flagged it RFC because I wanted to get some feedback ... and because I
would have had to spend more time on it for docs&tests.

> I currently seem to be using
> --split-submodule which is itself in a git repo i have (which i want to 
> also find its source repo so i can keep up to date with it).

For others reading this, I wrote --split-submodule also based on an
IRC request from Jeffrey.  The patch is at the end.  But it's way less
thought through than the --{dump,load}-map feature.  In particular
I've been wondering whether it's possible to use the latter to
implement --split-submodule as a fairly concise index filter.

> So applying 
> the patch itself might be somewhat troubling due to conflicts, therefore 
> id like to actually merge in a remote branch to keep things easy. So can 
> you guys point me to which repo would be best for me to keep up to date 
> with this would be?

You'll get the same conflicts from merging two little branches with
the features on them as with a 'git am -3'.

Many patches are never pushed to a public repo (there are some notable
exceptions in longer-running work).  The hassle of sending and
applying email is far outweighed by the ease of review.  Many reviews
happen without applying the series at all.  That I pushed both of them
to a public repo was an exception for your convenience.


--- 8< ---
Subject: TOY PATCH: filter-branch --split-submodule

Sometimes it makes sense to split out a path not as a subdirectory
(that would be merged by subtree-merge), but as a submodule.  Since
git objects are just shaped in the right way, this is actually quite
easy to do in a way that maintains the correct history relations:

When splitting out DIR in commit C, the submodule commit has tree
C:DIR and the rewritten superproject commit C' gets a submodule entry
at C:DIR instead.

Parent rewriting is done in the obvious way: submodule commits have
their corresponding submodule-changing ancestors as parents.  These
are easy to fetch since we can basically use $(map C^):DIR.

This is a toy patch because of its terrible interface: you will still
have to put the submodule in place.  As a start, you can

  git clone . DIR
  ( cd DIR && git reset --hard $(git rev-parse :DIR) )

to get a sub-repo set to the correct commit.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 962a93b..329d85c 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -191,6 +191,9 @@ do
 		filter_subdir="$OPTARG"
 		remap_to_ancestor=t
 		;;
+	--split-submodule)
+		split_submodule="$OPTARG"
+		;;
 	--original)
 		orig_namespace=$(expr "$OPTARG/" : '\(.*[^/]\)/*$')/
 		;;
@@ -349,6 +352,43 @@ while read commit parents; do
 	eval "$filter_index" < /dev/null ||
 		die "index filter failed: $filter_index"
 
+	if test -n "$split_submodule"; then
+		sub_differs=
+		sub_parents=
+		sub_commit=
+		submodule="$(git rev-parse --verify $commit:$split_submodule 2>/dev/null)"
+		if test -z "$parents"; then
+			if test -n "$submodule"; then
+				sub_differs=t
+			fi
+		fi
+		for parent in $parents; do
+			if ! test "$(git rev-parse --verify $parent:$split_submodule 2>/dev/null)" = "$submodule"; then
+				sub_differs=t
+			fi
+			for reparent in $(map "$parent"); do
+				p="$(git rev-parse --verify $reparent:$split_submodule 2>/dev/null)"
+				if test -n "$p"; then
+					sub_parents="$sub_parents -p $p"
+				fi
+			done
+		done
+		if test -n "$sub_differs"; then
+			sub_commit="$(sed -e '1,/^$/d' <../commit |
+				      git commit-tree $submodule $sub_parents)" || exit
+		else
+			for parent in $parents; do
+				sub_commit="$(git rev-parse --verify "$(map "$parent")":$split_submodule 2>/dev/null)"
+				break
+			done
+		fi
+		if test -n "$sub_commit"; then
+			git update-index --add --replace --cacheinfo 160000 $sub_commit "$split_submodule" || exit
+		else
+			git update-index --remove "$split_submodule"
+		fi
+	fi
+
 	parentstr=
 	for parent in $parents; do
 		for reparent in $(map "$parent"); do

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: [RFC PATCH 3/3] filter-branch: support --submodule-filter

[Junio C Hamano]

Thomas Rast <trast@student.ethz.ch> writes:

> Subject: TOY PATCH: filter-branch --split-submodule
>
> Sometimes it makes sense to split out a path not as a subdirectory
> (that would be merged by subtree-merge), but as a submodule.  Since
> git objects are just shaped in the right way, this is actually quite
> easy to do in a way that maintains the correct history relations:

The patch from a cursory look feels sane.

> @@ -349,6 +352,43 @@ while read commit parents; do
>  	eval "$filter_index" < /dev/null ||
>  		die "index filter failed: $filter_index"
>  
> +	if test -n "$split_submodule"; then
> +		sub_differs=
> +		sub_parents=
> +		sub_commit=

Just a style, but I find

	if test -n "$split_submodule"
        then
        	sub_differs= sub_parents= sub_commit=

easier to read.  Not a biggie, as the neighbourhood in the script already
is infested in the other style, but I thought I'd mention it.

> +		submodule="$(git rev-parse --verify $commit:$split_submodule 2>/dev/null)"

Do we need double quotes around it?

> +		if test -z "$parents"; then
> +			if test -n "$submodule"; then
> +				sub_differs=t
> +			fi
> +		fi

	if test -z "$parents" && test -n "$submodule"
        then
        	sub_differs=t
	fi

> +		for parent in $parents; do
> +			if ! test "$(git rev-parse --verify $parent:$split_submodule 2>/dev/null)" = "$submodule"; then
> +				sub_differs=t
> +			fi

If even one of the parents is different, we say "differs"...

> +		if test -n "$sub_differs"; then
> +			sub_commit="$(sed -e '1,/^$/d' <../commit |
> +				      git commit-tree $submodule $sub_parents)" || exit
> +		else
> +			for parent in $parents; do
> +				sub_commit="$(git rev-parse --verify "$(map "$parent")":$split_submodule 2>/dev/null)"
> +				break

... so we can just pick from the first parent and know all of them are the
same (could be empty which also is fine).  Good.

Re: [RFC PATCH 0/3] Submodule filtering for filter-branch

[Johannes Sixt]

On Freitag, 31. Dezember 2010, Thomas Rast wrote:
> Jeffrey Phillips Freeman aka tty1 asked on IRC about a way to fix the
> super-project after rewriting submodules.  This quick series provides
> some extra tricks in the filter-branch toolbox that can be plumbed
> together to achieve this (see the last commit's message), and should
> hopefully be general enough to be of other use too.

I like the --dump-map and --load-map options. But I think that 
a --submodule-filter is not necessary; it is just a 
particular --index-filter, and IMO should be part of the documentation in the 
form of an example that illustrates how to use the new options.

-- Hannes