[PATCH 1/2] add setup step to filter-branch

[PATCH 1/2] add setup step to filter-branch

[Andreas Heiduk]

A specific `--setup` step in `git filter-branch` makes it much easier
to define the initial values of variables used in the real filters.
Also sourcing/defining utility functions here instead of
`--env-filter` improves performance and minimizes clogging the output
in case of errors.

Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
---
 Documentation/git-filter-branch.txt | 16 +++++++++++-----
 git-filter-branch.sh                | 18 +++++++++++++-----
 2 files changed, 24 insertions(+), 10 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 6e4bb0220..45c849d8c 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -8,11 +8,11 @@ git-filter-branch - Rewrite branches
 SYNOPSIS
 --------
 [verse]
-'git filter-branch' [--env-filter <command>] [--tree-filter <command>]
-	[--index-filter <command>] [--parent-filter <command>]
-	[--msg-filter <command>] [--commit-filter <command>]
-	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
-	[--prune-empty]
+'git filter-branch' [--setup <command>] [--env-filter <command>]
+	[--tree-filter <command>] [--index-filter <command>]
+	[--parent-filter <command>] [--msg-filter <command>]
+	[--commit-filter <command>] [--tag-name-filter <command>]
+	[--subdirectory-filter <directory>] [--prune-empty]
 	[--original <namespace>] [-d <directory>] [-f | --force]
 	[--] [<rev-list options>...]
 
@@ -82,6 +82,12 @@ multiple commits.
 OPTIONS
 -------
 
+--setup <command>::
+	This is not a real filter executed for each commit but a one
+	time setup just before the loop. Therefore no commit-specific
+	variables are defined yet.  Functions or variables defined here
+	can be used or modified in the following filter steps.
+
 --env-filter <command>::
 	This filter may be used if you only need to modify the environment
 	in which the commit will be performed.  Specifically, you might
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index aafaf708d..2758ae5eb 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -81,11 +81,12 @@ set_ident () {
 	finish_ident COMMITTER
 }
 
-USAGE="[--env-filter <command>] [--tree-filter <command>]
-	[--index-filter <command>] [--parent-filter <command>]
-	[--msg-filter <command>] [--commit-filter <command>]
-	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
-	[--original <namespace>] [-d <directory>] [-f | --force]
+USAGE="[--setup <command>] [--env-filter <command>]
+	[--tree-filter <command>] [--index-filter <command>]
+	[--parent-filter <command>] [--msg-filter <command>]
+	[--commit-filter <command>] [--tag-name-filter <command>]
+	[--subdirectory-filter <directory>] [--original <namespace>]
+	[-d <directory>] [-f | --force]
 	[<rev-list options>...]"
 
 OPTIONS_SPEC=
@@ -96,6 +97,7 @@ if [ "$(is_bare_repository)" = false ]; then
 fi
 
 tempdir=.git-rewrite
+filter_setup=
 filter_env=
 filter_tree=
 filter_index=
@@ -148,6 +150,9 @@ do
 	-d)
 		tempdir="$OPTARG"
 		;;
+	--setup)
+		filter_setup="$OPTARG"
+		;;
 	--env-filter)
 		filter_env="$OPTARG"
 		;;
@@ -317,6 +322,9 @@ else
 	need_index=
 fi
 
+eval "$filter_setup" < /dev/null ||
+	die "filter setup failed: $filter_setup"
+
 while read commit parents; do
 	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))
 
-- 
2.13.0

[PATCH 2/2] add [--] to usage of filter-branch

[Andreas Heiduk]

Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
---
 Documentation/git-filter-branch.txt | 3 ++-
 git-filter-branch.sh                | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 45c849d8c..1efdda804 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -86,7 +86,8 @@ OPTIONS
 	This is not a real filter executed for each commit but a one
 	time setup just before the loop. Therefore no commit-specific
 	variables are defined yet.  Functions or variables defined here
-	can be used or modified in the following filter steps.
+	can be used or modified in the following filter steps except
+	the commit filter, for technical reasons.
 
 --env-filter <command>::
 	This filter may be used if you only need to modify the environment
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 2758ae5eb..3a74602ef 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -87,7 +87,7 @@ USAGE="[--setup <command>] [--env-filter <command>]
 	[--commit-filter <command>] [--tag-name-filter <command>]
 	[--subdirectory-filter <directory>] [--original <namespace>]
 	[-d <directory>] [-f | --force]
-	[<rev-list options>...]"
+	[--] [<rev-list options>...]"
 
 OPTIONS_SPEC=
 . git-sh-setup
-- 
2.13.0

Re: [PATCH 2/2] add [--] to usage of filter-branch

[Andreas Heiduk]

Am 03.06.2017 um 12:17 schrieb Andreas Heiduk:
> Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
> ---
>  Documentation/git-filter-branch.txt | 3 ++-
>  git-filter-branch.sh                | 2 +-
>  2 files changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
> index 45c849d8c..1efdda804 100644
> --- a/Documentation/git-filter-branch.txt
> +++ b/Documentation/git-filter-branch.txt
> @@ -86,7 +86,8 @@ OPTIONS
>  	This is not a real filter executed for each commit but a one
>  	time setup just before the loop. Therefore no commit-specific
>  	variables are defined yet.  Functions or variables defined here
> -	can be used or modified in the following filter steps.
> +	can be used or modified in the following filter steps except
> +	the commit filter, for technical reasons.

I'll move that into the previous commit.

Re: [PATCH 2/2] add [--] to usage of filter-branch

[Junio C Hamano]

Andreas Heiduk <asheiduk@gmail.com> writes:

> Am 03.06.2017 um 12:17 schrieb Andreas Heiduk:
>> Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
>> ---
>>  Documentation/git-filter-branch.txt | 3 ++-
>>  git-filter-branch.sh                | 2 +-
>>  2 files changed, 3 insertions(+), 2 deletions(-)
>> 
>> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
>> index 45c849d8c..1efdda804 100644
>> --- a/Documentation/git-filter-branch.txt
>> +++ b/Documentation/git-filter-branch.txt
>> @@ -86,7 +86,8 @@ OPTIONS
>>  	This is not a real filter executed for each commit but a one
>>  	time setup just before the loop. Therefore no commit-specific
>>  	variables are defined yet.  Functions or variables defined here
>> -	can be used or modified in the following filter steps.
>> +	can be used or modified in the following filter steps except
>> +	the commit filter, for technical reasons.
>
> I'll move that into the previous commit.

Yeah, the description of "technical limitation" is different from
clarifying the disambiguating "--" in the documentation.

I am curious what the "technical reason" really is, though ;-)

Re: [PATCH 2/2] add [--] to usage of filter-branch

[Andreas Heiduk]

Am 09.06.2017 um 15:14 schrieb Junio C Hamano:
> Andreas Heiduk <asheiduk@gmail.com> writes:
> 
>> Am 03.06.2017 um 12:17 schrieb Andreas Heiduk:
>>> Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
>>> ---
>>>  Documentation/git-filter-branch.txt | 3 ++-
>>>  git-filter-branch.sh                | 2 +-
>>>  2 files changed, 3 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
>>> index 45c849d8c..1efdda804 100644
>>> --- a/Documentation/git-filter-branch.txt
>>> +++ b/Documentation/git-filter-branch.txt
>>> @@ -86,7 +86,8 @@ OPTIONS
>>>  	This is not a real filter executed for each commit but a one
>>>  	time setup just before the loop. Therefore no commit-specific
>>>  	variables are defined yet.  Functions or variables defined here
>>> -	can be used or modified in the following filter steps.
>>> +	can be used or modified in the following filter steps except
>>> +	the commit filter, for technical reasons.
>>
>> I'll move that into the previous commit.
> 
> Yeah, the description of "technical limitation" is different from
> clarifying the disambiguating "--" in the documentation.
> 
> I am curious what the "technical reason" really is, though ;-)
> 

Well, I just picked up the wording from the "Filter" section a 
couple paragraphs above:

> The filters are applied in the order as listed below.  The <command>
> argument is always evaluated in the shell context using the 'eval' command
> (with the notable exception of the commit filter, for technical reasons).

Because these reasons exist independently from my change I think I
can get away with just that snappy reference :-]

Re: [PATCH 1/2] add setup step to filter-branch

[Junio C Hamano]

Andreas Heiduk <asheiduk@gmail.com> writes:

> A specific `--setup` step in `git filter-branch` makes it much easier
> to define the initial values of variables used in the real filters.
> Also sourcing/defining utility functions here instead of
> `--env-filter` improves performance and minimizes clogging the output
> in case of errors.
>
> Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
> ---

I was placed on To: line, but I do not have a strong opinion on this
change, either for or against.

"filter-branch" program itself may probably already be hard to port
to C, but I need to point out that this makes it even harder than it
currently is [*1*], and it is likely that it has to stay implemented
in shell forever, though.  I do not mind that future myself, but
those on platforms with weaker implementation of shells might.

[Footnote]

 *1* The issue is *not* that these individual filter commands expect
     <command> written as a shell scriptlet; it is that these
     scriptlets expect to be evaled inside a single shell process,
     making an update to a shell variable in one command visible to
     the next command that runs.


>  Documentation/git-filter-branch.txt | 16 +++++++++++-----
>  git-filter-branch.sh                | 18 +++++++++++++-----
>  2 files changed, 24 insertions(+), 10 deletions(-)
>
> diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
> index 6e4bb0220..45c849d8c 100644
> --- a/Documentation/git-filter-branch.txt
> +++ b/Documentation/git-filter-branch.txt
> @@ -8,11 +8,11 @@ git-filter-branch - Rewrite branches
>  SYNOPSIS
>  --------
>  [verse]
> -'git filter-branch' [--env-filter <command>] [--tree-filter <command>]
> -	[--index-filter <command>] [--parent-filter <command>]
> -	[--msg-filter <command>] [--commit-filter <command>]
> -	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
> -	[--prune-empty]
> +'git filter-branch' [--setup <command>] [--env-filter <command>]
> +	[--tree-filter <command>] [--index-filter <command>]
> +	[--parent-filter <command>] [--msg-filter <command>]
> +	[--commit-filter <command>] [--tag-name-filter <command>]
> +	[--subdirectory-filter <directory>] [--prune-empty]
>  	[--original <namespace>] [-d <directory>] [-f | --force]
>  	[--] [<rev-list options>...]
>  
> @@ -82,6 +82,12 @@ multiple commits.
>  OPTIONS
>  -------
>  
> +--setup <command>::
> +	This is not a real filter executed for each commit but a one
> +	time setup just before the loop. Therefore no commit-specific
> +	variables are defined yet.  Functions or variables defined here
> +	can be used or modified in the following filter steps.
> +
>  --env-filter <command>::
>  	This filter may be used if you only need to modify the environment
>  	in which the commit will be performed.  Specifically, you might
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index aafaf708d..2758ae5eb 100755
> --- a/git-filter-branch.sh
> +++ b/git-filter-branch.sh
> @@ -81,11 +81,12 @@ set_ident () {
>  	finish_ident COMMITTER
>  }
>  
> -USAGE="[--env-filter <command>] [--tree-filter <command>]
> -	[--index-filter <command>] [--parent-filter <command>]
> -	[--msg-filter <command>] [--commit-filter <command>]
> -	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
> -	[--original <namespace>] [-d <directory>] [-f | --force]
> +USAGE="[--setup <command>] [--env-filter <command>]
> +	[--tree-filter <command>] [--index-filter <command>]
> +	[--parent-filter <command>] [--msg-filter <command>]
> +	[--commit-filter <command>] [--tag-name-filter <command>]
> +	[--subdirectory-filter <directory>] [--original <namespace>]
> +	[-d <directory>] [-f | --force]
>  	[<rev-list options>...]"
>  
>  OPTIONS_SPEC=
> @@ -96,6 +97,7 @@ if [ "$(is_bare_repository)" = false ]; then
>  fi
>  
>  tempdir=.git-rewrite
> +filter_setup=
>  filter_env=
>  filter_tree=
>  filter_index=
> @@ -148,6 +150,9 @@ do
>  	-d)
>  		tempdir="$OPTARG"
>  		;;
> +	--setup)
> +		filter_setup="$OPTARG"
> +		;;
>  	--env-filter)
>  		filter_env="$OPTARG"
>  		;;
> @@ -317,6 +322,9 @@ else
>  	need_index=
>  fi
>  
> +eval "$filter_setup" < /dev/null ||
> +	die "filter setup failed: $filter_setup"
> +
>  while read commit parents; do
>  	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))

Re: [PATCH 1/2] add setup step to filter-branch

[Jeff King]

On Mon, Jun 05, 2017 at 11:15:18AM +0900, Junio C Hamano wrote:

> Andreas Heiduk <asheiduk@gmail.com> writes:
> 
> > A specific `--setup` step in `git filter-branch` makes it much easier
> > to define the initial values of variables used in the real filters.
> > Also sourcing/defining utility functions here instead of
> > `--env-filter` improves performance and minimizes clogging the output
> > in case of errors.
> >
> > Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
> > ---
> 
> I was placed on To: line, but I do not have a strong opinion on this
> change, either for or against.
> 
> "filter-branch" program itself may probably already be hard to port
> to C, but I need to point out that this makes it even harder than it
> currently is [*1*], and it is likely that it has to stay implemented
> in shell forever, though.  I do not mind that future myself, but
> those on platforms with weaker implementation of shells might.

I have a feeling that if we were ever to rewrite filter-branch, it would
probably be worth allowing people to write snippets in a better language
(possibly even a domain-specific language). I'm sure that most of the
program being written in shell doesn't help, but if we're spawning one
or more shell instances per commit (plus the Git programs they spawn!),
it's always going to be slow.

But I suspect that would be an uphill battle, as our only stable API
involves starting external processes anyway. You'd probably do better to
pick a language you like and rewrite it in using libgit2's bindings to
that language. It's not feature complete, but basic stuff like "put this
entry in the tree" is certainly mature.

>  *1* The issue is *not* that these individual filter commands expect
>      <command> written as a shell scriptlet; it is that these
>      scriptlets expect to be evaled inside a single shell process,
>      making an update to a shell variable in one command visible to
>      the next command that runs.

I think you'd need a shell "helper" that's a single long-running process
and just reads "eval the index snippet now" instructions from the C
controller. At which point I don't think Andreas's "setup" feature is
any harder to support. We just send an "eval the setup snippet"
instruction first.

I guess one way to think of that is that the C program is itself
generating a shell script.

-Peff

Re: [PATCH 1/2] add setup step to filter-branch

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I have a feeling that if we were ever to rewrite filter-branch, it would
> probably be worth allowing people to write snippets in a better language
> (possibly even a domain-specific language). I'm sure that most of the
> program being written in shell doesn't help, but if we're spawning one
> or more shell instances per commit (plus the Git programs they spawn!),
> it's always going to be slow.
>
> But I suspect that would be an uphill battle, as our only stable API
> involves starting external processes anyway. You'd probably do better to
> pick a language you like and rewrite it in using libgit2's bindings to
> that language. It's not feature complete, but basic stuff like "put this
> entry in the tree" is certainly mature.
>
>>  *1* The issue is *not* that these individual filter commands expect
>>      <command> written as a shell scriptlet; it is that these
>>      scriptlets expect to be evaled inside a single shell process,
>>      making an update to a shell variable in one command visible to
>>      the next command that runs.
>
> I think you'd need a shell "helper" that's a single long-running process
> and just reads "eval the index snippet now" instructions from the C
> controller. At which point I don't think Andreas's "setup" feature is
> any harder to support. We just send an "eval the setup snippet"
> instruction first.

Yes.  I do not think this particular one makes things any worse than
it already is.  As I said, I do not have a strong opinion against
the topic; as long as people find the feature useful, I do not mind
applying it.

Thanks.

[PATCH v2 1/2] add setup step to filter-branch

[Andreas Heiduk]

A specific `--setup` step in `git filter-branch` makes it much easier
to define the initial values of variables used in the real filters.
Also sourcing/defining utility functions here instead of
`--env-filter` improves performance and minimizes clogging the output
in case of errors.

Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
---
 Documentation/git-filter-branch.txt | 17 ++++++++++++-----
 git-filter-branch.sh                | 18 +++++++++++++-----
 2 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 7b695dbb7..9e5169aa6 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -8,11 +8,11 @@ git-filter-branch - Rewrite branches
 SYNOPSIS
 --------
 [verse]
-'git filter-branch' [--env-filter <command>] [--tree-filter <command>]
-	[--index-filter <command>] [--parent-filter <command>]
-	[--msg-filter <command>] [--commit-filter <command>]
-	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
-	[--prune-empty]
+'git filter-branch' [--setup <command>] [--env-filter <command>]
+	[--tree-filter <command>] [--index-filter <command>]
+	[--parent-filter <command>] [--msg-filter <command>]
+	[--commit-filter <command>] [--tag-name-filter <command>]
+	[--subdirectory-filter <directory>] [--prune-empty]
 	[--original <namespace>] [-d <directory>] [-f | --force]
 	[--] [<rev-list options>...]
 
@@ -82,6 +82,13 @@ multiple commits.
 OPTIONS
 -------
 
+--setup <command>::
+	This is not a real filter executed for each commit but a one
+	time setup just before the loop. Therefore no commit-specific
+	variables are defined yet.  Functions or variables defined here
+	can be used or modified in the following filter steps except
+	the commit filter, for technical reasons.
+
 --env-filter <command>::
 	This filter may be used if you only need to modify the environment
 	in which the commit will be performed.  Specifically, you might
diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index aafaf708d..2758ae5eb 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -81,11 +81,12 @@ set_ident () {
 	finish_ident COMMITTER
 }
 
-USAGE="[--env-filter <command>] [--tree-filter <command>]
-	[--index-filter <command>] [--parent-filter <command>]
-	[--msg-filter <command>] [--commit-filter <command>]
-	[--tag-name-filter <command>] [--subdirectory-filter <directory>]
-	[--original <namespace>] [-d <directory>] [-f | --force]
+USAGE="[--setup <command>] [--env-filter <command>]
+	[--tree-filter <command>] [--index-filter <command>]
+	[--parent-filter <command>] [--msg-filter <command>]
+	[--commit-filter <command>] [--tag-name-filter <command>]
+	[--subdirectory-filter <directory>] [--original <namespace>]
+	[-d <directory>] [-f | --force]
 	[<rev-list options>...]"
 
 OPTIONS_SPEC=
@@ -96,6 +97,7 @@ if [ "$(is_bare_repository)" = false ]; then
 fi
 
 tempdir=.git-rewrite
+filter_setup=
 filter_env=
 filter_tree=
 filter_index=
@@ -148,6 +150,9 @@ do
 	-d)
 		tempdir="$OPTARG"
 		;;
+	--setup)
+		filter_setup="$OPTARG"
+		;;
 	--env-filter)
 		filter_env="$OPTARG"
 		;;
@@ -317,6 +322,9 @@ else
 	need_index=
 fi
 
+eval "$filter_setup" < /dev/null ||
+	die "filter setup failed: $filter_setup"
+
 while read commit parents; do
 	git_filter_branch__commit_count=$(($git_filter_branch__commit_count+1))
 
-- 
2.13.0

[PATCH v2 2/2] add [--] to usage of filter-branch

[Andreas Heiduk]

Signed-off-by: Andreas Heiduk <asheiduk@gmail.com>
---
 git-filter-branch.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index 2758ae5eb..3a74602ef 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -87,7 +87,7 @@ USAGE="[--setup <command>] [--env-filter <command>]
 	[--commit-filter <command>] [--tag-name-filter <command>]
 	[--subdirectory-filter <directory>] [--original <namespace>]
 	[-d <directory>] [-f | --force]
-	[<rev-list options>...]"
+	[--] [<rev-list options>...]"
 
 OPTIONS_SPEC=
 . git-sh-setup
-- 
2.13.0