Filter-branch's "move tree to subdirectory" example fails with BSD sed?

Filter-branch's "move tree to subdirectory" example fails with BSD sed?

[Christopher Tiwald]

The "Move tree to subdirectory" example in the 'git filter-branch'
manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04. I'm
using git version 1.7.0.4, which happened to be the version installed
on the Ubuntu VM I had laying around. I think it's a difference between
'sed' on the two systems.

The example:
git filter-branch --index-filter \
        'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

Demonstrating the problem is with sed:
git init "test"
cd "test"
mkdir -p subdirA/subdirB
echo content > subdirA/subdirB/file
git add .
git commit -m "initial commit"
git ls-files -s | sed "s-\t\"*-&newsubdir/-"

On Mac 10.7.3 the final command outputs:
100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0	subdirA/subdirB/file

On Ubuntu 10.04:
100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0	newsubdir/subdirA/subdirB/file

I can solve my immediate problem using Ubuntu, but is there a way we
could rewrite the example to work on both systems? I'm afraid differences
between BSD and GNU 'sed' are a bit beyond me.

--
Christopher Tiwald

Re: Filter-branch's "move tree to subdirectory" example fails with BSD sed?

[Johannes Sixt]

Am 14.04.2012 18:00, schrieb Christopher Tiwald:
> The "Move tree to subdirectory" example in the 'git filter-branch'
> manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04.

> git init "test"
> cd "test"
> mkdir -p subdirA/subdirB
> echo content > subdirA/subdirB/file
> git add .
> git commit -m "initial commit"
> git ls-files -s | sed "s-\t\"*-&newsubdir/-"
> 
> On Mac 10.7.3 the final command outputs:
> 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0	subdirA/subdirB/file
> 
> On Ubuntu 10.04:
> 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0	newsubdir/subdirA/subdirB/file

Perhaps a literal TAB instead of \t makes the example work?

It would be difficult, though, to write this down in the manual in an
unambiguous way.

-- Hannes

Re: Filter-branch's "move tree to subdirectory" example fails with BSD sed?

[Jeff King]

On Sat, Apr 14, 2012 at 09:25:05PM +0200, Johannes Sixt wrote:

> Am 14.04.2012 18:00, schrieb Christopher Tiwald:
> > The "Move tree to subdirectory" example in the 'git filter-branch'
> > manpage fails on Mac OSX 10.7.3, but succeeds on Ubuntu 10.04.
> 
> > git init "test"
> > cd "test"
> > mkdir -p subdirA/subdirB
> > echo content > subdirA/subdirB/file
> > git add .
> > git commit -m "initial commit"
> > git ls-files -s | sed "s-\t\"*-&newsubdir/-"
> > 
> > On Mac 10.7.3 the final command outputs:
> > 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0	subdirA/subdirB/file
> > 
> > On Ubuntu 10.04:
> > 100644 d95f3ad14dee633a758d2e331151e950dd13e4ed 0	newsubdir/subdirA/subdirB/file
> 
> Perhaps a literal TAB instead of \t makes the example work?

Yes, I suspect that is the problem, too.

> It would be difficult, though, to write this down in the manual in an
> unambiguous way.

Maybe it would be simpler to just use perl:

 git filter-branch --index-filter '
   git ls-files -s |
     perl -pe "s{\t\"?}{$&newsubdir/}" |
     GIT_INDEX_FILE=$GIT_INDEX_FILE.new git update-index --index-info &&
   mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE
 ' HEAD

I also think tweaking the indentation and line breaks as I did above
makes it a little more readable. If we are using perl, we could
also just use "-z" to get rid of the funny quote handling.

I wish we could also get rid of the temporary index. It exists solely
for the side effect of removing the existing entries (otherwise, you get
both "foo" and "newsubdir/foo" in the resulting history). It would be
nice if there was some flag to update-index to say "before you do
anything, clear the existing index" (or I guess, "don't bother reading
the existing index"). And then result could look like:

  git filter-branch --index-filter '
    git ls-files -sz |
    perl -0pe "s{\t}{\tnewsubdir/}" |
    git update-index --from-scratch -z --index-info
  ' HEAD

which is IMHO much easier to read and understand.

-Peff

[RFC/PATCH 1/2] update-index: add --clear option

[Jeff King]

This just discards existing entries from the index, which
can be useful if you are rewriting all entries with
"--index-info" or similar.

Signed-off-by: Jeff King <peff@peff.net>
---
I tried to make something like:

  git update-index --from-scratch --index-info

work by avoiding reading all entries in the first place. However,
update-index actually processes its arguments sequentially, so we _must_
read the index before we start processing arguments. But because it's
sequential, a "clear" operation makes sense, since you clear, then add
new entries.

 Documentation/git-update-index.txt |    3 +++
 builtin/update-index.c             |   10 ++++++++++
 2 files changed, 13 insertions(+)

diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt
index a3081f4..47f0ae6 100644
--- a/Documentation/git-update-index.txt
+++ b/Documentation/git-update-index.txt
@@ -71,6 +71,9 @@ OPTIONS
 --cacheinfo <mode> <object> <path>::
 	Directly insert the specified info into the index.
 
+--clear::
+	Discard all existing entries from the index.
+
 --index-info::
         Read index information from stdin.
 
diff --git a/builtin/update-index.c b/builtin/update-index.c
index a6a23fa..559dfae 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -645,6 +645,13 @@ static int cacheinfo_callback(struct parse_opt_ctx_t *ctx,
 	return 0;
 }
 
+static int clear_callback(const struct option *opt,
+			  const char *arg, int unset)
+{
+	discard_cache();
+	return 0;
+}
+
 static int stdin_cacheinfo_callback(struct parse_opt_ctx_t *ctx,
 			      const struct option *opt, int unset)
 {
@@ -774,6 +781,9 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 			"add entries from standard input to the index",
 			PARSE_OPT_NONEG | PARSE_OPT_NOARG,
 			(parse_opt_cb *) stdin_cacheinfo_callback},
+		{OPTION_CALLBACK, 0, "clear", NULL, NULL,
+			"drop all index entries", PARSE_OPT_NONEG | PARSE_OPT_NOARG,
+			clear_callback},
 		{OPTION_LOWLEVEL_CALLBACK, 0, "unresolve", &has_errors, NULL,
 			"repopulate stages #2 and #3 for the listed paths",
 			PARSE_OPT_NONEG | PARSE_OPT_NOARG,
-- 
1.7.9.6.8.g992e5

Re: [RFC/PATCH 1/2] update-index: add --clear option

[Christopher Tiwald]

On Mon, Apr 16, 2012 at 12:02:32PM -0400, Jeff King wrote:
> I tried to make something like:
> 
>   git update-index --from-scratch --index-info
> 
> work by avoiding reading all entries in the first place. However,
> update-index actually processes its arguments sequentially, so we _must_
> read the index before we start processing arguments. But because it's
> sequential, a "clear" operation makes sense, since you clear, then add
> new entries.

At the very least I can give this patch series a go this evening. I'm in
the process of combining 40 highly-intertwined, nested repositories into
one, and should be able to test it under the conditions outlined in the
updated example.

--
Christopher Tiwald

Re: [RFC/PATCH 1/2] update-index: add --clear option

[Christopher Tiwald]

On Mon, Apr 16, 2012 at 5:48 PM, Christopher Tiwald
<christiwald@gmail.com> wrote:
> At the very least I can give this patch series a go this evening. I'm in
> the process of combining 40 highly-intertwined, nested repositories into
> one, and should be able to test it under the conditions outlined in the
> updated example.

This worked great for me, at least in the case of adding trees to
subdirectories.
Fwiw, I found the newer, patched example a little easier to parse, but
I also had
spent a while figuring out what the original version did.

--
Christopher Tiwald

[RFC/PATCH 2/2] docs/filter-branch: clean up newsubdir example

[Jeff King]

Over the years, this simple example has ended up quite hard
to read because of the number of special cases that must be
handled. Let's simplify it a bit:

  1. Use the new "index-info --clear" to avoid the need for
     a temporary index.

  2. Use "-z" and "perl -0" to avoid dealing with quoting
     issues. As a bonus, using perl means that "\t" will
     work consistently in regexps (the previous example
     using sed was reported to fail on OS X).

  3. Change the indentation to keep one logical unit per
     line and avoid extra backslash-escaping.

Signed-off-by: Jeff King <peff@peff.net>
---
 Documentation/git-filter-branch.txt |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 0f2f117..3df138f 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -358,11 +358,11 @@ git filter-branch ... D..H --not C
 To move the whole tree into a subdirectory, or remove it from there:
 
 ---------------------------------------------------------------
-git filter-branch --index-filter \
-	'git ls-files -s | sed "s-\t\"*-&newsubdir/-" |
-		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
-			git update-index --index-info &&
-	 mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"' HEAD
+git filter-branch --index-filter '
+	git ls-files -sz |
+	perl -0pe "s{\t}{\tnewsubdir/}" |
+	git update-index -z --clear --index-info
+' HEAD
 ---------------------------------------------------------------
 
 
-- 
1.7.9.6.8.g992e5

Re: Filter-branch's "move tree to subdirectory" example fails with BSD sed?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> I wish we could also get rid of the temporary index. It exists solely
> for the side effect of removing the existing entries (otherwise, you get
> both "foo" and "newsubdir/foo" in the resulting history). It would be
> nice if there was some flag to update-index to say "before you do
> anything, clear the existing index" (or I guess, "don't bother reading
> the existing index"). And then result could look like:
>
>   git filter-branch --index-filter '
>     git ls-files -sz |
>     perl -0pe "s{\t}{\tnewsubdir/}" |
>     git update-index --from-scratch -z --index-info
>   ' HEAD
>
> which is IMHO much easier to read and understand.

Wouldn't "git read-tree --prefix=newsubdir/" suffice without the pipeline?
I.e.

        git filter-branch --index-filter '
		rm -f "$GIT_INDEX_FILE"
                git read-tree --prefix=newsubdir/ "$GIT_COMMIT"
	' HEAD

Re: Filter-branch's "move tree to subdirectory" example fails with BSD sed?

[Jeff King]

On Mon, Apr 16, 2012 at 10:03:03AM -0700, Junio C Hamano wrote:

> >   git filter-branch --index-filter '
> >     git ls-files -sz |
> >     perl -0pe "s{\t}{\tnewsubdir/}" |
> >     git update-index --from-scratch -z --index-info
> >   ' HEAD
> 
> Wouldn't "git read-tree --prefix=newsubdir/" suffice without the pipeline?
> I.e.
> 
>         git filter-branch --index-filter '
> 		rm -f "$GIT_INDEX_FILE"
>                 git read-tree --prefix=newsubdir/ "$GIT_COMMIT"
> 	' HEAD

Yeah, I think it does (at least it makes sense to me, and worked on a
simple test case). I think I might use "git read-tree --empty" instead
of "rm", as it is a little more obvious what is going on. But other than
that, it is much more readable.

It is a slight shame not to show an "ls-files | update-index" example,
because that is the most general form. But since this form is quite
often cut-and-paste by people, I think simple is better here. If we want
to show a more complex example, then we should add it separately.

-Peff