[PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.

[PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.

[Lee Carver]

As noted in several forums, a recommended way to move trees between
repositories
is to use git-filter-branch to revise the history for a single tree:

http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
er-preserving-history/
http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
epo-to-another-not-a-clone-preserving-history

However, this can lead to argument list too long errors when the original
repository has many retained branches (>6k)

/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits

Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
---
 git-filter-branch.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..60d239b 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
 	remap_to_ancestor=t
 fi
 
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
 
 case "$filter_subdir" in
 "")
@@ -267,8 +267,9 @@ case "$filter_subdir" in
 	;;
 esac
 
+cat ../parse | \
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" > ../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")
 
-- 
1.8.3.2

Re: [PATCH-v2] Allow git-filter-branch to process large repositories with lots of branches.

[Stefano Lattarini]

On 07/09/13 22:03, Lee Carver wrote:
> As noted in several forums, a recommended way to move trees between
> repositories
> is to use git-filter-branch to revise the history for a single tree:
>
> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
> er-preserving-history/
> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
> epo-to-another-not-a-clone-preserving-history
>
> However, this can lead to argument list too long errors when the original
> repository has many retained branches (>6k)
>
> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
> /usr/local/git/libexec/git-core/git: Argument list too long
> Could not get the commits
>
> Piping the saved output from git rev-parse into git rev-list avoids this
> problem, since the rev-parse output is not processed as a command line
> argument.
> ---
>   git-filter-branch.sh | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index ac2a005..60d239b 100755
> --- a/git-filter-branch.sh
> +++ b/git-filter-branch.sh
> @@ -255,7 +255,7 @@ else
>   	remap_to_ancestor=t
>   fi
>
> -rev_args=$(git rev-parse --revs-only "$@")
> +git rev-parse --revs-only "$@" > ../parse
>
>   case "$filter_subdir" in
>   "")
> @@ -267,8 +267,9 @@ case "$filter_subdir" in
>   	;;
>   esac
>
> +cat ../parse | \
>   git rev-list --reverse --topo-order --default HEAD \
> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
> +	--parents --simplify-merges --stdin "$@" > ../revs ||
 >
Useless use of cat IMO.  I'd suggest using a redirection instead:

   git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" > ../revs < ../parse ||

>   	die "Could not get the commits"
>   commits=$(wc -l <../revs | tr -d " ")
>
>

Regards,
   Stefano

[PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

[Lee Carver]

As noted in several forums, a recommended way to move trees between
repositories
is to use git-filter-branch to revise the history for a single tree:

http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
er-preserving-history/
http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
epo-to-another-not-a-clone-preserving-history

However, this can lead to argument list too long errors when the original
repository has many retained branches (>6k)

/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits

Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.
---
 git-filter-branch.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..2091885 100755
--- a/git-filter-branch.shgit
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
 	remap_to_ancestor=t
 fi
 
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
 
 case "$filter_subdir" in
 "")
@@ -268,7 +268,7 @@ case "$filter_subdir" in
 esac
 
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")
 
-- 
1.8.3.2

Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

[Junio C Hamano]

Lee Carver <Lee.Carver@servicenow.com> writes:

> As noted in several forums, a recommended way to move trees between
> repositories
> is to use git-filter-branch to revise the history for a single tree:
>
> http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
> er-preserving-history/
> http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
> epo-to-another-not-a-clone-preserving-history
>
> However, this can lead to argument list too long errors when the original
> repository has many retained branches (>6k)
>
> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
> /usr/local/git/libexec/git-core/git: Argument list too long
> Could not get the commits
>
> Piping the saved output from git rev-parse into git rev-list avoids this
> problem, since the rev-parse output is not processed as a command line
> argument.
> ---
>  git-filter-branch.sh | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
> index ac2a005..2091885 100755
> --- a/git-filter-branch.shgit
> +++ b/git-filter-branch.sh
> @@ -255,7 +255,7 @@ else
>  	remap_to_ancestor=t
>  fi
>  
> -rev_args=$(git rev-parse --revs-only "$@")
> +git rev-parse --revs-only "$@" > ../parse

Where is this "rev-parse" command running?  Is it always safe to
clobber the file "../parse" like this?

>  
>  case "$filter_subdir" in
>  "")
> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>  esac
>  
>  git rev-list --reverse --topo-order --default HEAD \
> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
> +	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>  	die "Could not get the commits"
>  commits=$(wc -l <../revs | tr -d " ")

Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

[Lee Carver]

On 9/10/13 4:20 PM, "Junio C Hamano" <gitster@pobox.com> wrote:

>Lee Carver <Lee.Carver@servicenow.com> writes:
>
>> As noted in several forums, a recommended way to move trees between
>> repositories
>> is to use git-filter-branch to revise the history for a single tree:
>>
>> 
>>http://gbayer.com/development/moving-files-from-one-git-repository-to-ano
>>th
>> er-preserving-history/
>> 
>>http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git
>>-r
>> epo-to-another-not-a-clone-preserving-history
>>
>> However, this can lead to argument list too long errors when the
>>original
>> repository has many retained branches (>6k)
>>
>> /usr/local/git/libexec/git-core/git-filter-branch: line 270:
>> /usr/local/git/libexec/git-core/git: Argument list too long
>> Could not get the commits
>>
>> Piping the saved output from git rev-parse into git rev-list avoids this
>> problem, since the rev-parse output is not processed as a command line
>> argument.
>> ---
>>  git-filter-branch.sh | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/git-filter-branch.sh b/git-filter-branch.sh
>> index ac2a005..2091885 100755
>> --- a/git-filter-branch.shgit
>> +++ b/git-filter-branch.sh
>> @@ -255,7 +255,7 @@ else
>>  	remap_to_ancestor=t
>>  fi
>>  
>> -rev_args=$(git rev-parse --revs-only "$@")
>> +git rev-parse --revs-only "$@" > ../parse
>
>Where is this "rev-parse" command running?  Is it always safe to
>clobber the file "../parse" like this?

It is using the same ${tempdir} working directory that git rev-list uses
below for the ../revs file

It's normally .git-rewrite/t, following the normal working directory setup
near line 205.

>
>>  
>>  case "$filter_subdir" in
>>  "")
>> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>>  esac
>>  
>>  git rev-list --reverse --topo-order --default HEAD \
>> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
>> +	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>>  	die "Could not get the commits"
>>  commits=$(wc -l <../revs | tr -d " ")

Re: [PATCH-v3] Allow git-filter-branch to process large repositories with lots of branches.

[Junio C Hamano]

Lee Carver <Lee.Carver@servicenow.com> writes:

> It is using the same ${tempdir} working directory that git rev-list uses
> below for the ../revs file

Ah, I missed that; then that should be safe.  The patch looks sane.

Can we have your sign-off, too, please?

>
> It's normally .git-rewrite/t, following the normal working directory setup
> near line 205.
>
>>
>>>  
>>>  case "$filter_subdir" in
>>>  "")
>>> @@ -268,7 +268,7 @@ case "$filter_subdir" in
>>>  esac
>>>  
>>>  git rev-list --reverse --topo-order --default HEAD \
>>> -	--parents --simplify-merges $rev_args "$@" > ../revs ||
>>> +	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
>>>  	die "Could not get the commits"
>>>  commits=$(wc -l <../revs | tr -d " ")

[PATCH-v4] Allow git-filter-branch to process large repositories with lots of branches.

[Lee Carver]

As noted in several forums, a recommended way to move trees between
repositories
is to use git-filter-branch to revise the history for a single tree:

http://gbayer.com/development/moving-files-from-one-git-repository-to-anoth
er-preserving-history/
http://stackoverflow.com/questions/1365541/how-to-move-files-from-one-git-r
epo-to-another-not-a-clone-preserving-history

However, this can lead to argument list too long errors when the original
repository has many retained branches (>6k)

/usr/local/git/libexec/git-core/git-filter-branch: line 270:
/usr/local/git/libexec/git-core/git: Argument list too long
Could not get the commits

Piping the saved output from git rev-parse into git rev-list avoids this
problem, since the rev-parse output is not processed as a command line
argument.

Signed-off-by: Lee Carver <lee.carver@servicenow.com>
---
 git-filter-branch.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ac2a005..2091885 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,7 @@ else
 	remap_to_ancestor=t
 fi
 
-rev_args=$(git rev-parse --revs-only "$@")
+git rev-parse --revs-only "$@" > ../parse
 
 case "$filter_subdir" in
 "")
@@ -268,7 +268,7 @@ case "$filter_subdir" in
 esac
 
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges $rev_args "$@" > ../revs ||
+	--parents --simplify-merges --stdin "$@" < ../parse > ../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")
 
-- 
1.8.3.2