Bug in git-filter-branch example

Bug in git-filter-branch example

[Ivo Anjo]

I've been working on importing my svn repo into git, and while moving
some things around using git-filter-branch I ran into a bug in the
example provided on the manpage:

    To move the whole tree into a subdirectory, or remove it from there:

    git filter-branch --index-filter \
            'git ls-files -s | sed "s-\t-&newsubdir/-" |
                    GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                            git update-index --index-info &&
             mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD


The problem is with filenames that use complex utf8 (non-ascii?) chars:

> ls -lah Papers/*Lea*
-rw-r--r-- 1 knuckles users 109K 2010-01-31 18:57 Papers/A Java Fork ⁄
Join Framework -- Lea.pdf

as you can see, I'm using '⁄' as a replacement for '/' on that
filename, since I obviously can't use '/' on a filename. The problem
is, git ls-files -s lists that file wrapped in quotes and with the
char value escaped

100644 c09309342037fa7d91f37651e2f16e981e4d739a 0    "Papers/A Java
Fork \342\201\204 Join Framework -- Lea.pdf"

while other files aren't, so the simple sed line provided won't work,
as it will output the new filename as newsubdir/"Papers...

I ended up using git filter-branch to remove the offending file ("if
all you have is a hammer..."), and re-adding it after the move.

I'm also left wondering how many uses of git ls-files out there on the
"internets" survive working on a filename such as this one. Maybe git
ls-files should have an option to output the real utf8 file name
instead of escaping the bytes...?

Ivo Anjo

Re: Bug in git-filter-branch example

[Jeff King]

On Mon, Feb 01, 2010 at 11:48:27AM +0000, Ivo Anjo wrote:

>     To move the whole tree into a subdirectory, or remove it from there:
> 
>     git filter-branch --index-filter \
>             'git ls-files -s | sed "s-\t-&newsubdir/-" |
>                     GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                             git update-index --index-info &&
>              mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
> 
> 
> The problem is with filenames that use complex utf8 (non-ascii?) chars:

Yes, that is definitely the problem. update-index unquotes its input, as
you would expect, but the sed munging does not take the quote into
account. You can fix it by doing:

  sed "s-\t\"*-&newsubdir/-"

instead. Here is a patch to fix the documentation. I am slightly unsure
of whether it should be applied. These examples are supposed to be
simple and readable to help the user understand what the filters can do.
And this makes it somewhat less simple for the sake of a special case.
But at the same time, users are going to cut-and-paste these examples (I
know I have), and the special case is not _that_ special, especially for
non-English speakers. And having it in the example helps make people
aware that quoted paths are a reality.

So I think on balance it is probably better to fix it.

Note also that another way of "fixing" this would be to set
core.quotepath to false (which is something you probably want to do
anyway if you are using utf8 characters in your filenames). And I put
"fix" in quotes because you still may run across quoted paths, but they
will be much less common; you will only see them if you have control
characters or other insanity in your paths.

-- >8 --
Subject: [PATCH] docs: fix filter-branch example for quoted paths

If there is a quoted path, update-index will correctly
unquote it. However, we must take care to put our new prefix
inside the double-quote.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-filter-branch.txt |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index cfaba2a..020028c 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -358,7 +358,7 @@ To move the whole tree into a subdirectory, or remove it from there:
 
 ---------------------------------------------------------------
 git filter-branch --index-filter \
-	'git ls-files -s | sed "s-\t-&newsubdir/-" |
+	'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
 		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
 			git update-index --index-info &&
 	 mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
-- 
1.7.0.rc1.16.g21332.dirty

Re: Bug in git-filter-branch example

[Johannes Sixt]

Ivo Anjo schrieb:
> I've been working on importing my svn repo into git, and while moving
> some things around using git-filter-branch I ran into a bug in the
> example provided on the manpage:

Thanks for the report, but: Examples are just that: examples. IMO, it is
OK that there are implicit assumptions (such as that only "common" file
names are used in the repo).

Making the example universally applicable would greatly obfuscate the
important messages.

> I ended up using git filter-branch to remove the offending file ("if
> all you have is a hammer..."), and re-adding it after the move.

It seems that the example has met its objective: It showed you how to use
the hammer, and with your creativity mixed in, it enabled you to use the
hammer in new ways.

-- Hannes