Passing rev-list options in git-filter-branch broken

Passing rev-list options in git-filter-branch broken

[Felix Eckhofer]

Hey.

When trying to rewrite svn revisions in commit messages to corresponding 
git commit hashes, I came across the following problem (using git 
1.9.1):

   $ git filter-branch --msg-filter svnrev2git.py -- --date-order --all
   fatal: options not supported in --stdin mode
   Could not get the commits

This seems to have been caused by 3361a548db. After reverting this 
commit, using --date-order appears to work again.


Regards
felix

Re: Passing rev-list options in git-filter-branch broken

[Junio C Hamano]

Felix Eckhofer <felix@tribut.de> writes:

> When trying to rewrite svn revisions in commit messages to
> corresponding git commit hashes, I came across the following problem
> (using git 1.9.1):
>
>   $ git filter-branch --msg-filter svnrev2git.py -- --date-order --all
>   fatal: options not supported in --stdin mode
>   Could not get the commits
>
> This seems to have been caused by 3361a548db. After reverting this
> commit, using --date-order appears to work again.

Hmph, unfortunate.

3361a548 (Allow git-filter-branch to process large repositories with
lots of branches., 2013-09-10) has this change:

 
    -rev_args=$(git rev-parse --revs-only "$@")
    +git rev-parse --revs-only "$@" >../parse

and then later feeds ../parse from the standard input of rev-list.

The --revs-only option, because --date-order *is* a rev-list related
argument, is emitted by the above rev-parse, along with the tip of
refs (which come from --all).  But --stdin mode of rev-list is meant
to receive *only* revisions, not options.  When it gets to the point
to accept the list of tips to start traversing from, it is too late
to give it an option.

Changing the above to something like:

	git rev-parse --revs-only --no-flags "$@" >../parse

would be a better workaround that would not break repositories with
large number of references, but it obviously will lose --date-order
option (why would it be even necessary, though?  I suspect that
forcing the date-order will make the resulting pack a lot worse by
robbing the data stream of locality).

Re: Passing rev-list options in git-filter-branch broken

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Felix Eckhofer <felix@tribut.de> writes:
>
>> This seems to have been caused by 3361a548db. After reverting this
>> commit, using --date-order appears to work again.
>
> Hmph, unfortunate.
>
> 3361a548 (Allow git-filter-branch to process large repositories with
> lots of branches., 2013-09-10) has this change:
>
>  
>     -rev_args=$(git rev-parse --revs-only "$@")
>     +git rev-parse --revs-only "$@" >../parse
>
> and then later feeds ../parse from the standard input of rev-list.
>
> The --revs-only option, because --date-order *is* a rev-list related
> argument, is emitted by the above rev-parse, along with the tip of
> refs (which come from --all).  But --stdin mode of rev-list is meant
> to receive *only* revisions, not options.  When it gets to the point
> to accept the list of tips to start traversing from, it is too late
> to give it an option.
>
> Changing the above to something like:
>
> 	git rev-parse --revs-only --no-flags "$@" >../parse
>
> would be a better workaround that would not break repositories with
> large number of references, but it obviously will lose --date-order
> option (why would it be even necessary, though?  I suspect that
> forcing the date-order will make the resulting pack a lot worse by
> robbing the data stream of locality).

As the original "../parse" file will have mixture of revs (i.e. the
object names) and whatever rev-list options (e.g. "--date-order")
you gave from the command line, here is a band-aid to sift them into
two, to be applied on top of 3361a548 (Allow git-filter-branch to
process large repositories with lots of branches., 2013-09-10).

I wish there was a way to tell rev-parse "give us only flags
relevant to rev-list, don't include revs in the output", but I do
not think there was ever any such option (we can get revs without
any flags with --no-flags, though).


 git-filter-branch.sh | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/git-filter-branch.sh b/git-filter-branch.sh
index ca3d539..ed55c01 100755
--- a/git-filter-branch.sh
+++ b/git-filter-branch.sh
@@ -255,7 +255,9 @@ else
 	remap_to_ancestor=t
 fi
 
-git rev-parse --revs-only "$@" >../parse
+git rev-parse --revs-only "$@" >../parse.revs.flags
+sed -ne '/^[0-9a-f][0-9a-f][0-9a-f]*[0-9a-f]$/p' <../parse.revs.flags >../parse.revs
+sed -e '/^[0-9a-f][0-9a-f][0-9a-f]*[0-9a-f]$/d' <../parse.revs.flags >../parse.flags
 
 case "$filter_subdir" in
 "")
@@ -268,7 +270,8 @@ case "$filter_subdir" in
 esac
 
 git rev-list --reverse --topo-order --default HEAD \
-	--parents --simplify-merges --stdin "$@" <../parse >../revs ||
+	--parents --simplify-merges --stdin "$@" \
+	$(cat ../parse.flags) <../parse.revs >../revs ||
 	die "Could not get the commits"
 commits=$(wc -l <../revs | tr -d " ")
 

Re: Passing rev-list options in git-filter-branch broken

[Felix Eckhofer]

Junio,

thanks for your reply and your patch.

Am 27.06.2014 20:31, schrieb Junio C Hamano:
> [...]
> would be a better workaround that would not break repositories with
> large number of references, but it obviously will lose --date-order
> option (why would it be even necessary, though?  I suspect that
> forcing the date-order will make the resulting pack a lot worse by
> robbing the data stream of locality).

We're migrating some svn repositories to git. The use case is to replace 
mentions of svn revisions in commit messages by the corresponding commit 
hash. Therefore, the commits must be ordered by svn revision number, 
which is not guaranteed for commits on different branches with 
--topo-order.


Regards
felix