Using filter-branch to move repo contents in subdirectory

Using filter-branch to move repo contents in subdirectory

[Adam Mercer]

Hi

I'm trying to use git filter-branch to move all contents of a repo
into a subdirectory of the repo root. The git filter-branch man page
has the following example:

       To move the whole tree into a subdirectory, or remove it from there:

           git filter-branch --index-filter \
                   'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
                           GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                                   git update-index --index-info &&
                    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

So I've been trying to use this, the original structure of the repo is:

$ ls
00boot                 SFTdumpall.c     debian
Makefile.am            SFTdumpheader.c  ligodcc.cls
README                 SFTvalidate.c    pdfdraftcopy.sty
SCCB_2004_09_09        SFTwrite.c       run_validate.sh
SFTReferenceLibrary.c  T040164.tex      sftreferencelibrary-2.2.tar.gz
SFTReferenceLibrary.h  configure.in     sftreferencelibrary-2.3.tar.gz
$

and I want all this in a subdirectory call "sftlib", so using the
example from the manpage I ran:

$ git filter-branch --index-filter \
    'git ls-files -s | sed "s-\t\"*-&sftlib/-" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
            git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
Rewrite 223651f2ebd5d5d9341bcfc9e7cb6caaa3f4d171 (56/65)Ignoring path
00bootsftlib/
Rewrite c979ee835ebb89408e2f5f345ddefbb4d8cd75f5 (57/65)Ignoring path
00bootsftlib/
Rewrite 10037389e41fb4b4d04234ca2e302a7a491616d5 (58/65)Ignoring path
00bootsftlib/
Rewrite 21b4c31a02c6b7718145d471ba20e6fa70eb30d0 (59/65)Ignoring path
00bootsftlib/
Rewrite dc5cebbe14050f52f2effaa5eac4eb579dd440c3 (60/65)Ignoring path
00bootsftlib/
Rewrite 3b83fdd19b5d674445f5263df8cb2fc35c3c0092 (61/65)Ignoring path
00bootsftlib/
Rewrite 5246038ceeb2d28fba462076b62f3cf1b2b7e9ab (62/65)Ignoring path
00bootsftlib/
Rewrite 0fd3256ff401b405c677e83db350ad41ee74ef0d (63/65)Ignoring path
00bootsftlib/
Ignoring path debian/compatsftlib/
Ignoring path debian/copyrightsftlib/
Rewrite 9aca227c807658b951e81a6ca4460115b2fe8ccc (64/65)Ignoring path
00bootsftlib/
Ignoring path debian/compatsftlib/
Ignoring path debian/copyrightsftlib/
Rewrite 00274be10e10920d84db145268d85faf1b06539f (65/65)Ignoring path
00bootsftlib/
Ignoring path debian/compatsftlib/
Ignoring path debian/copyrightsftlib/

Ref 'refs/heads/to_merge' was rewritten
$

now the directory structure is:

$ ls
Makefile.am  SCCB_2004_09_09        SFTReferenceLibrary.h
SFTdumpheader.c   SFTwritsftlib    configure.in  ligodcc.cls
run_validatsftlib
README       SFTReferenceLibrary.c  SFTdumpall.c
SFTvalidatsftlib  T040164.tsftlib  debian        pdfdraftsftlib
sftsftlib
$

which is clearly not what I'm wanting. What I am doing wrong here? As
far as I can I'm using the example from the man-page?

Cheers

Adam

Re: Using filter-branch to move repo contents in subdirectory

[Thomas Rast]

Adam Mercer wrote:
> $ git filter-branch --index-filter \
>     'git ls-files -s | sed "s-\t\"*-&sftlib/-" |
>         GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>             git update-index --index-info &&
>     mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
> Rewrite 223651f2ebd5d5d9341bcfc9e7cb6caaa3f4d171 (56/65)Ignoring path
> 00bootsftlib/

What OS is this?  You seem to have a 'sed' version where \t does not
match the horizontal tab character, or a shell that replaces the '\t'
inside a double-quoted string with simply 't'.

As a short-term fix, you can try to replace it with $(printf '\t')
which should always give a tab character.  You need to quote the '
though, so that's

  $ git filter-branch --index-filter \
      'git ls-files -s | sed "s-$(printf '\''\t'\'')\"*-&sftlib/-" |
          GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
              git update-index --index-info &&
      mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD


As for the actual issue: 'man 1p sed' on my system claims

        * The escape sequence '\n' shall match a <newline> embedded in the  pat-
          tern space. A literal <newline> shall not be used in the BRE of a con-
          text address or in the substitute function.

but does not mention \t at all, so I guess either that manpage is
wrong or GNU sed is not POSIX compliant even with --posix (where it
still treats \t as a tab).

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Using filter-branch to move repo contents in subdirectory

[Adam Mercer]

On Thu, Jul 29, 2010 at 14:08, Thomas Rast <trast@student.ethz.ch> wrote:
> Adam Mercer wrote:
>> $ git filter-branch --index-filter \
>>     'git ls-files -s | sed "s-\t\"*-&sftlib/-" |
>>         GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>>             git update-index --index-info &&
>>     mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
>> Rewrite 223651f2ebd5d5d9341bcfc9e7cb6caaa3f4d171 (56/65)Ignoring path
>> 00bootsftlib/
>
> What OS is this?

This was on Mac OS X 10.6. Where sed is BSD sed.

> You seem to have a 'sed' version where \t does not
> match the horizontal tab character, or a shell that replaces the '\t'
> inside a double-quoted string with simply 't'.
>
> As a short-term fix, you can try to replace it with $(printf '\t')
> which should always give a tab character.  You need to quote the '
> though, so that's
>
>  $ git filter-branch --index-filter \
>      'git ls-files -s | sed "s-$(printf '\''\t'\'')\"*-&sftlib/-" |
>          GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>              git update-index --index-info &&
>      mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Thanks, that did the trick!

> As for the actual issue: 'man 1p sed' on my system claims
>
>        * The escape sequence '\n' shall match a <newline> embedded in the  pat-
>          tern space. A literal <newline> shall not be used in the BRE of a con-
>          text address or in the substitute function.
>
> but does not mention \t at all, so I guess either that manpage is
> wrong or GNU sed is not POSIX compliant even with --posix (where it
> still treats \t as a tab).

Also using the original command, but specifying GNU sed also does the trick.

Cheers

Adam

[PATCH] filter-branch tests/docs: avoid \t in sed regexes

[Thomas Rast]

Using \t to represent a tab character is not portable beyond GNU sed
(see e.g. GNU sed's info pages).  Use printf to generate the tab
instead.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

 Adam Mercer wrote:
> > What OS is this?
> This was on Mac OS X 10.6. Where sed is BSD sed.

That's very funny however, since it means that nobody should ever have
had a successful test run on OS X with the preinstalled tools.  What
gives?


 Documentation/git-filter-branch.txt |    3 ++-
 t/t7003-filter-branch.sh            |    3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 020028c..7e3ff8e 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -357,8 +357,9 @@ git filter-branch ... D..H --not C
 To move the whole tree into a subdirectory, or remove it from there:
 
 ---------------------------------------------------------------
+export TAB="$(printf '\t')"
 git filter-branch --index-filter \
-	'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
+	'git ls-files -s | sed "s-$TAB\"*-&newsubdir/-" |
 		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
 			git update-index --index-info &&
 	 mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
diff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh
index 0da13a8..e90da6d 100755
--- a/t/t7003-filter-branch.sh
+++ b/t/t7003-filter-branch.sh
@@ -140,10 +140,11 @@ test_expect_success 'more setup' '
 	git merge branch
 '
 
+export TAB="$(printf '\t')"
 test_expect_success 'use index-filter to move into a subdirectory' '
 	git branch directorymoved &&
 	git filter-branch -f --index-filter \
-		 "git ls-files -s | sed \"s-\\t-&newsubdir/-\" |
+		 "git ls-files -s | sed \"s-$TAB-&newsubdir/-\" |
 	          GIT_INDEX_FILE=\$GIT_INDEX_FILE.new \
 			git update-index --index-info &&
 		  mv \"\$GIT_INDEX_FILE.new\" \"\$GIT_INDEX_FILE\"" directorymoved &&
-- 
1.7.2.1.342.g676a4

Re: [PATCH] filter-branch tests/docs: avoid \t in sed regexes

[Tomas Carnecky]

On 7/29/10 3:24 PM, Thomas Rast wrote:
> Using \t to represent a tab character is not portable beyond GNU sed
> (see e.g. GNU sed's info pages).  Use printf to generate the tab
> instead.
> 
> Signed-off-by: Thomas Rast <trast@student.ethz.ch>
> ---
> 
>  Adam Mercer wrote:
>>> What OS is this?
>> This was on Mac OS X 10.6. Where sed is BSD sed.
> 
> That's very funny however, since it means that nobody should ever have
> had a successful test run on OS X with the preinstalled tools.  What
> gives?

v1.7.2-rc1 on 10.6:

fixed   3
success 6328
failed  0
broken  21
total   6355

*** t7003-filter-branch.sh ***
*   ok 1: setup
*   ok 2: rewrite identically
*   ok 3: result is really identical
*   ok 4: rewrite bare repository identically
*   ok 5: result is really identical
*   ok 6: correct GIT_DIR while using -d
*   ok 7: Fail if commit filter fails
*   ok 8: rewrite, renaming a specific file
*   ok 9: test that the file was renamed
*   ok 10: rewrite, renaming a specific directory
*   ok 11: test that the directory was renamed
*   ok 12: rewrite one branch, keeping a side branch
*   ok 13: common ancestor is still common (unchanged)
*   ok 14: filter subdirectory only
*   ok 15: subdirectory filter result looks okay
*   ok 16: more setup
*   ok 17: use index-filter to move into a subdirectory
*   ok 18: stops when msg filter fails
*   ok 19: author information is preserved
*   ok 20: remove a certain author's commits
*   ok 21: barf on invalid name
*   ok 22: "map" works in commit filter
*   ok 23: Name needing quotes
*   ok 24: Subdirectory filter with disappearing trees
*   ok 25: Tag name filtering retains tag message
*   ok 26: Tag name filtering strips gpg signature
*   ok 27: Tag name filtering allows slashes in tag names
*   ok 28: Prune empty commits
*   ok 29: --remap-to-ancestor with filename filters
*   ok 30: setup submodule
*   ok 31: rewrite submodule with another content
*   ok 32: replace submodule revision

Re: [PATCH] filter-branch tests/docs: avoid \t in sed regexes

[Thomas Rast]

Tomas Carnecky wrote:
> On 7/29/10 3:24 PM, Thomas Rast wrote:
> > Using \t to represent a tab character is not portable beyond GNU sed
> > (see e.g. GNU sed's info pages).  Use printf to generate the tab
> > instead.
[...]
> > That's very funny however, since it means that nobody should ever have
> > had a successful test run on OS X with the preinstalled tools.  What
> > gives?
[...]
> *** t7003-filter-branch.sh ***
> *   ok 17: use index-filter to move into a subdirectory

Oh, I know.  The test isn't correct, it should be something like the
patch below.  Otherwise a failure in git-diff (namely that
directorymoved:newsubdir is invalid) is never detected, and since the
output in this case is empty, also the test -z succeeds.  Can you
apply that and see if it makes the test fail for you?

diff --git i/t/t7003-filter-branch.sh w/t/t7003-filter-branch.sh
index e90da6d..a8e0c48 100755
--- i/t/t7003-filter-branch.sh
+++ w/t/t7003-filter-branch.sh
@@ -148,7 +148,9 @@ test_expect_success 'use index-filter to move into a subdirectory' '
 	          GIT_INDEX_FILE=\$GIT_INDEX_FILE.new \
 			git update-index --index-info &&
 		  mv \"\$GIT_INDEX_FILE.new\" \"\$GIT_INDEX_FILE\"" directorymoved &&
-	test -z "$(git diff HEAD directorymoved:newsubdir)"'
+	git diff HEAD directorymoved:newsubdir > actual &&
+	test ! -s actual
+'
 
 test_expect_success 'stops when msg filter fails' '
 	old=$(git rev-parse HEAD) &&


-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: [PATCH] filter-branch tests/docs: avoid \t in sed regexes

[Tomas Carnecky]

On 7/29/10 4:52 PM, Thomas Rast wrote:
> Tomas Carnecky wrote:
>> *** t7003-filter-branch.sh ***
>> *   ok 17: use index-filter to move into a subdirectory
> 
> Oh, I know.  The test isn't correct, it should be something like the
> patch below.  Otherwise a failure in git-diff (namely that
> directorymoved:newsubdir is invalid) is never detected, and since the
> output in this case is empty, also the test -z succeeds.  Can you
> apply that and see if it makes the test fail for you?
> 
> diff --git i/t/t7003-filter-branch.sh w/t/t7003-filter-branch.sh
> index e90da6d..a8e0c48 100755
> --- i/t/t7003-filter-branch.sh
> +++ w/t/t7003-filter-branch.sh
> @@ -148,7 +148,9 @@ test_expect_success 'use index-filter to move into a subdirectory' '
>  	          GIT_INDEX_FILE=\$GIT_INDEX_FILE.new \
>  			git update-index --index-info &&
>  		  mv \"\$GIT_INDEX_FILE.new\" \"\$GIT_INDEX_FILE\"" directorymoved &&
> -	test -z "$(git diff HEAD directorymoved:newsubdir)"'
> +	git diff HEAD directorymoved:newsubdir > actual &&
> +	test ! -s actual
> +'
>  
>  test_expect_success 'stops when msg filter fails' '
>  	old=$(git rev-parse HEAD) &&

Now the test fails. And with the other patch applied on top of it, the
test passes again.

tom

[PATCH] t7005: fix subdirectory-filter test

[Thomas Rast]

The test would not fail if the filtering failed to do anything, since
in

  test -z "$(git diff HEAD directorymoved:newsubdir)"'

'directorymoved:newsubdir' is not valid, so git-diff fails without
printing anything on stdout.  But then the exit status of git-diff is
lost, whereas test -z "" succeeds.

Use 'git diff --exit-code' instead, which does the right thing and has
the added bonus of showing the differences if there are any.

Signed-off-by: Thomas Rast <trast@student.ethz.ch>
---

Tomas Carnecky wrote:
> Now the test fails. And with the other patch applied on top of it, the
> test passes again.

Ok.  Here's a slightly nicer solution, let's add this on top of the
earlier patch.

 t/t7003-filter-branch.sh |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/t/t7003-filter-branch.sh b/t/t7003-filter-branch.sh
index e90da6d..84c2612 100755
--- a/t/t7003-filter-branch.sh
+++ b/t/t7003-filter-branch.sh
@@ -148,7 +148,8 @@ test_expect_success 'use index-filter to move into a subdirectory' '
 	          GIT_INDEX_FILE=\$GIT_INDEX_FILE.new \
 			git update-index --index-info &&
 		  mv \"\$GIT_INDEX_FILE.new\" \"\$GIT_INDEX_FILE\"" directorymoved &&
-	test -z "$(git diff HEAD directorymoved:newsubdir)"'
+	git diff --exit-code HEAD directorymoved:newsubdir
+'
 
 test_expect_success 'stops when msg filter fails' '
 	old=$(git rev-parse HEAD) &&
-- 
1.7.2.1.342.g676a4

Re: [PATCH] t7005: fix subdirectory-filter test

[Thomas Rast]

Thomas Rast wrote:
> t7005: fix subdirectory-filter test

Bah, this subject is wrong on two counts.  It should be something like

  t7003: fix move-to-subdirectory test

instead.  Sorry for the noise.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: Using filter-branch to move repo contents in subdirectory

[Adam Mercer]

On Thu, Jul 29, 2010 at 07:37, Adam Mercer <ramercer@gmail.com> wrote:

> I'm trying to use git filter-branch to move all contents of a repo
> into a subdirectory of the repo root. The git filter-branch man page
> has the following example:
>
>       To move the whole tree into a subdirectory, or remove it from there:
>
>           git filter-branch --index-filter \
>                   'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
>                           GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                                   git update-index --index-info &&
>                    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

I'm trying to do this with another repo now and am getting a different
error message:

$ git filter-branch --index-filter \
    'git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
            git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
Rewrite b3100fb24a7eb2f349801f3b87cf995dc3d14bba (1/44)mv:
/Users/ram/git/eah2/.git-rewrite/t/../index.new: No such file or
directory
index filter failed: git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv
$GIT_INDEX_FILE.new $GIT_INDEX_FILE
$

Any ideas?

This is on Mac OS X 10.6.4, with git-1.7.2.1, and GNU sed 4.2.1

Cheers

Adam

Re: Using filter-branch to move repo contents in subdirectory

[Adam Mercer]

On Mon, Aug 9, 2010 at 14:36, Adam Mercer <ramercer@gmail.com> wrote:

> I'm trying to do this with another repo now and am getting a different
> error message:
>
> $ git filter-branch --index-filter \
>    'git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
>        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>            git update-index --index-info &&
>    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
> Rewrite b3100fb24a7eb2f349801f3b87cf995dc3d14bba (1/44)mv:
> /Users/ram/git/eah2/.git-rewrite/t/../index.new: No such file or
> directory
> index filter failed: git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
> GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv
> $GIT_INDEX_FILE.new $GIT_INDEX_FILE
> $
>
> Any ideas?
>
> This is on Mac OS X 10.6.4, with git-1.7.2.1, and GNU sed 4.2.1

Anyone?

Cheers

Adam

Re: Using filter-branch to move repo contents in subdirectory

[Jeff King]

On Wed, Aug 11, 2010 at 10:01:15AM -0500, Adam Mercer wrote:

> On Mon, Aug 9, 2010 at 14:36, Adam Mercer <ramercer@gmail.com> wrote:
> 
> > I'm trying to do this with another repo now and am getting a different
> > error message:
> >
> > $ git filter-branch --index-filter \
> >    'git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
> >        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
> >            git update-index --index-info &&
> >    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
> > Rewrite b3100fb24a7eb2f349801f3b87cf995dc3d14bba (1/44)mv:
> > /Users/ram/git/eah2/.git-rewrite/t/../index.new: No such file or
> > directory
> > index filter failed: git ls-files -s | gsed "s-\t\"*-&sftlib/-" |
> > GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info && mv
> > $GIT_INDEX_FILE.new $GIT_INDEX_FILE
> > $
> >
> > Any ideas?
> >
> > This is on Mac OS X 10.6.4, with git-1.7.2.1, and GNU sed 4.2.1
> 
> Anyone?

It looks like git-update-index will not create $GIT_INDEX_FILE at all if
you have no actual input lines to --index-info. So perhaps you have some
commit in your repo that has no actual content in it. Either that or
for some reason your "ls-files -s | gsed" invocation is producing no
output.

Try adding a "touch $GIT_INDEX_FILE.new" at the beginning of your index
filter and see if that helps. Then the file will always exist to be
moved, even if we write nothing to it. If it works, then problem solved.
If you get a repo full of empty commits, then something is wrong with
your gsed pipeline.

-Peff

Re: Using filter-branch to move repo contents in subdirectory

[Adam Mercer]

On Wed, Aug 11, 2010 at 14:32, Jeff King <peff@peff.net> wrote:

> It looks like git-update-index will not create $GIT_INDEX_FILE at all if
> you have no actual input lines to --index-info. So perhaps you have some
> commit in your repo that has no actual content in it. Either that or
> for some reason your "ls-files -s | gsed" invocation is producing no
> output.

Yep, that was the problem. I'd been modifying the history with several
git filter-branch calls and the initial commit in the repo was empty.
Getting rid of this initial empty commit allowed the command to run.

Cheers

Adam