Truncating and cleaning a imported git repositary to make it more usable

Truncating and cleaning a imported git repositary to make it more usable

[Alex Bennee]

Hi,

I've succeeded in porting our whole CVS repository into git (with plenty
of swap, cvsps is very memory hungry). Now we have a reference
historical repository I need to think about creating one we actually use
for day to day use (the .git is currently in the order of 650Mb which is
a little heavy for pushing around DSL links willy nilly).

What I need to achieve is fairly simple.

1. Find the common ancestor point for all the product branches I want to
keep in the repo.

Can anyone point me to the magic invocation that would allow this?
git-merge-base can find the common point for two commits, do I just have
to iteratively use that?

I'm currently working over a terminal link so it would be dead handy if
there was a visual way of showing what gitk --all does on the command
line.

2. Create a new repo, starting at the common ancestor point

  2(a). I'd need to remove the orphaned old CVS modules which are still
around but not referenced any more.

3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
them

I'm not sure if I could pull directly into the new repo here. I could do
it by doing a:

  old_repo: git-format-patch ANCESTOR..BRANCH_HEAD
  new_repo: git-am ../old_repo/0*

But I'm open to newer cleaner ways of dealing with this.

Has anyone done this before? Are there cleaner ways of dealing with
this?

--
Alex, homepage: http://www.bennee.com/~alex/
When one knows women one pities men, but when one studies men, one
excuses women. -- Horne Tooke

Re: Truncating and cleaning a imported git repositary to make it more usable

[Avery Pennarun]

On 3/28/08, Alex Bennee <kernel-hacker@bennee.com> wrote:
>  I've succeeded in porting our whole CVS repository into git (with plenty
>  of swap, cvsps is very memory hungry). Now we have a reference
>  historical repository I need to think about creating one we actually use
>  for day to day use (the .git is currently in the order of 650Mb which is
>  a little heavy for pushing around DSL links willy nilly).

You used git-repack and git-gc, right?  If not, the repo will be
bigger than it should be.

>  1. Find the common ancestor point for all the product branches I want to
>  keep in the repo.
>
>  Can anyone point me to the magic invocation that would allow this?
>  git-merge-base can find the common point for two commits, do I just have
>  to iteratively use that?
>
>  I'm currently working over a terminal link so it would be dead handy if
>  there was a visual way of showing what gitk --all does on the command
>  line.

"git show-branch -a" probably does what you want.

>  2. Create a new repo, starting at the common ancestor point
>
>   2(a). I'd need to remove the orphaned old CVS modules which are still
>  around but not referenced any more.
>
>  3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
>  them

You probably want to look at git-filter-branch, particularly the
--parent-filter option.

Note that 2(a) would be easiest if you just did it before importing
from CVS at all, since it's easy to remove old modules from CVS just
by moving the files out of the way.

Have fun,

Avery

Re: Truncating and cleaning a imported git repositary to make it more usable

[Alex Bennee]

On Fri, Mar 28, 2008 at 12:50 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On 3/28/08, Alex Bennee <kernel-hacker@bennee.com> wrote:
>  >  I've succeeded in porting our whole CVS repository into git (with plenty
>  >  of swap, cvsps is very memory hungry). Now we have a reference
>  >  historical repository I need to think about creating one we actually use
>  >  for day to day use (the .git is currently in the order of 650Mb which is
>  >  a little heavy for pushing around DSL links willy nilly).
>
>  You used git-repack and git-gc, right?  If not, the repo will be
>  bigger than it should be.

Oh yes. Unfortunately for 7 years of development of an embedded
platform there is a lot of cruft in the CVS repo :-(.

However as it needs to be a drop in replacement for the current repo
we can't just drop all the history and start from a fresh repo.

>  >  1. Find the common ancestor point for all the product branches I want to
>  >  keep in the repo.

>  "git show-branch -a" probably does what you want.

Cool. Or in fact git-show-branch --merge-base seems to be the baby.

I'm sure it's been mentioned before but the plethora of git commands
can make it hard to find the commands you want even
if your sure there is some way of doing it :-)

>  >  2. Create a new repo, starting at the common ancestor point
>  >
>  >   2(a). I'd need to remove the orphaned old CVS modules which are still
>  >  around but not referenced any more.
>  >
>  >  3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
>  >  them
>
>  You probably want to look at git-filter-branch, particularly the
>  --parent-filter option.
>
>  Note that 2(a) would be easiest if you just did it before importing
>  from CVS at all, since it's easy to remove old modules from CVS just
>  by moving the files out of the way.

Trouble is looking at the current repo I'm not sure which modules are
orphans (at the common anscestor point) and which have been added
since.

Of course now I can easily go back in time (easier than in CVS) I can
build up this list.

>  Have fun,

For certain values of fun ;-)

-- 
Alex, homepage: http://www.bennee.com/~alex/

Re: Truncating and cleaning a imported git repositary to make it more usable

[Alex Bennee]

On Fri, Mar 28, 2008 at 12:50 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On 3/28/08, Alex Bennee <kernel-hacker@bennee.com> wrote:
>  >  2. Create a new repo, starting at the common ancestor point
>  >
>  >   2(a). I'd need to remove the orphaned old CVS modules which are still
>  >  around but not referenced any more.
>  >
>  >  3. Create the product branches and import ANCESTOR..BRANCH_HEAD into
>  >  them
>
>  You probably want to look at git-filter-branch, particularly the
>  --parent-filter option.
>
>  Note that 2(a) would be easiest if you just did it before importing
>  from CVS at all, since it's easy to remove old modules from CVS just
>  by moving the files out of the way.

Ahh, having investigated further it seems the repo contains two CVS
modules which account for about 1/3 of the checkout and
are not needed (as they are standalone and separate). When I do the
final import from CVS I can skip them. Having said that I'll
need to run that over the weekend as cvsps takes up so much memory and
swaps a lot.

I tried what I think is the correct usage of git-filter-branch but I get:

ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
'rm -rf big_dira big_dirb' dev-branch
warning: refname 'dev-branch' is ambiguous.
Which ref do you want to rewrite?


Any ideas?

-- 
Alex, homepage: http://www.bennee.com/~alex/

Re: Truncating and cleaning a imported git repositary to make it more usable

[Alex Bennee]

On Fri, Mar 28, 2008 at 3:52 PM, Alex Bennee <kernel-hacker@bennee.com> wrote:

>  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
>  'rm -rf big_dira big_dirb' dev-branch
>  warning: refname 'dev-branch' is ambiguous.
>  Which ref do you want to rewrite?

Ignore me. Not sure why but:

git-checkout dev-branch
git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD

works


-- 
Alex, homepage: http://www.bennee.com/~alex/

Re: Truncating and cleaning a imported git repositary to make it more usable

[Johannes Sixt]

Alex Bennee schrieb:
> On Fri, Mar 28, 2008 at 3:52 PM, Alex Bennee <kernel-hacker@bennee.com> wrote:
> 
>>  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
>>  'rm -rf big_dira big_dirb' dev-branch
>>  warning: refname 'dev-branch' is ambiguous.
>>  Which ref do you want to rewrite?
> 
> Ignore me. Not sure why but:
> 
> git-checkout dev-branch
> git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD

You really shouldn't do it this way, unless you do it on a ramdisk. Better
use an --index-filter. This is modeled after the last example in the man
page (and, of course, untested):

git filter-branch --index-filter \
        'git ls-files -s |
	 grep -v "	big_dira" |
	 grep -v "	big_dirb" |
                GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
                        git update-index --index-info &&
         mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

where the space before big_dir is actually a literal TAB!

-- Hannes

[PATCH] Documentation: Another example for git-filter-branch

[Alex Bennee]

On Fri, 2008-03-28 at 17:43 +0100, Johannes Sixt wrote:
> Alex Bennee schrieb:
> > git-checkout dev-branch
> > git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD
> 
> You really shouldn't do it this way, unless you do it on a ramdisk. Better
> use an --index-filter. This is modeled after the last example in the man
> page (and, of course, untested):

I missed the implications of the last example...

> git filter-branch --index-filter \
>         'git ls-files -s |
> 	 grep -v "	big_dira" |
> 	 grep -v "	big_dirb" |
>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                         git update-index --index-info &&
>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
> 
> where the space before big_dir is actually a literal TAB!

Good example. Maybe we should add the following (because plenty of
examples is always a good thing IMHO)?

>From 341b480bad4ed9f99a54dc66ba20b0cead4594b5 Mon Sep 17 00:00:00 2001
From: Alex Bennee <Alex.Bennee@cambridgebroadband.com>
Date: Fri, 28 Mar 2008 18:00:20 +0000
Subject: [PATCH] Update git filter-branch examples for cleaning directories

I had to do this to a tree I imported from CVS which had a lot of cruft in it
which was taking ages. Luckily Johannes Sixt submitted this scriplet on the list
before I finished which makes things clearer.
---
 Documentation/git-filter-branch.txt |   21 +++++++++++++++++++++
 1 files changed, 21 insertions(+), 0 deletions(-)

diff --git a/Documentation/git-filter-branch.txt b/Documentation/git-filter-branch.txt
index 543a1cf..0e4b581 100644
--- a/Documentation/git-filter-branch.txt
+++ b/Documentation/git-filter-branch.txt
@@ -184,6 +184,27 @@ git filter-branch --index-filter 'git update-index --remove filename' HEAD
 
 Now, you will get the rewritten history saved in HEAD.
 
+If you need to remove whole directories it is tempting to use "rm -rf"
+in the --tree-filter form, however this will generate a lot of disk IO
+on big trees. Hence it's more preferable to use the index filter. For
+example:
+
+--------------------------------------------------------------------------
+git filter-branch --index-filter \
+       'git ls-files -s |
+        grep -v "      big_dira" |
+	grep -v "      big_dirb" |
+		GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
+                git update-index --index-info &&
+	mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
+--------------------------------------------------------------------------
+
+NB: The grep is matching literal TABs from the output of "git
+ls-files". Try <ctrl-v><tab> if your shell keeps giving you a directory
+listing every time you hit tab.
+
+
+
 To set a commit (which typically is at the tip of another
 history) to be the parent of the current initial commit, in
 order to paste the other history behind the current history:
-- 
1.5.5.rc0.6.gdeda.dirty





--
Alex, homepage: http://www.bennee.com/~alex/
We reject: kings, presidents, and voting. We believe in: rough consensus
and working code. -- Dave Clark

Re: Truncating and cleaning a imported git repositary to make it more usable

[Jeff King]

On Fri, Mar 28, 2008 at 04:03:18PM +0000, Alex Bennee wrote:

> >  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
> >  'rm -rf big_dira big_dirb' dev-branch
> >  warning: refname 'dev-branch' is ambiguous.
> >  Which ref do you want to rewrite?

You have two refs called dev-branch. Probably both a tag and a branch
(i.e., refs/heads/dev-branch and refs/tags/dev-branch). You can
disambiguate by saying "heads/dev-branch").

> Ignore me. Not sure why but:
> 
> git-checkout dev-branch
> git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD

git-checkout prefers branches to tags[1], so there is no ambiguity. It
puts the full refname (refs/heads/dev-branch) into your HEAD, so there
is no ambiguity when saying "HEAD".

[1] Actually, the upcoming builtin-checkout no longer behaves this way,
and will complain about the ambiguity.

-Peff

Re: Truncating and cleaning a imported git repositary to make it more usable

[Jeff King]

On Fri, Mar 28, 2008 at 02:08:49PM -0400, Jeff King wrote:

> [1] Actually, the upcoming builtin-checkout no longer behaves this way,
> and will complain about the ambiguity.

To clarify: it is simply a warning; it will still switch to the branch.

-Peff

Re: Truncating and cleaning a imported git repositary to make it more usable

[Mike Hommey]

On Fri, Mar 28, 2008 at 05:43:58PM +0100, Johannes Sixt wrote:
> Alex Bennee schrieb:
> > On Fri, Mar 28, 2008 at 3:52 PM, Alex Bennee <kernel-hacker@bennee.com> wrote:
> > 
> >>  ajb@pitcairn:/export/git/fresh.git$ git-filter-branch --tree-filter
> >>  'rm -rf big_dira big_dirb' dev-branch
> >>  warning: refname 'dev-branch' is ambiguous.
> >>  Which ref do you want to rewrite?
> > 
> > Ignore me. Not sure why but:
> > 
> > git-checkout dev-branch
> > git-filter-branch --tree-filter  'rm -rf big_dira big_dirb' HEAD
> 
> You really shouldn't do it this way, unless you do it on a ramdisk. Better
> use an --index-filter. This is modeled after the last example in the man
> page (and, of course, untested):
> 
> git filter-branch --index-filter \
>         'git ls-files -s |
> 	 grep -v "	big_dira" |
> 	 grep -v "	big_dirb" |
>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                         git update-index --index-info &&
>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

git filter-branch --index-filter 'git rm -r --cached big_dira big_dirb'
should be working nowadays.

Mike

Re: Truncating and cleaning a imported git repositary to make it more usable

[Alex Bennee]

On Fri, Mar 28, 2008 at 4:43 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:
> Alex Bennee schrieb:
>  You really shouldn't do it this way, unless you do it on a ramdisk. Better
>  use an --index-filter. This is modeled after the last example in the man
>  page (and, of course, untested):
>
>  git filter-branch --index-filter \
>         'git ls-files -s |
>          grep -v "      big_dira" |
>          grep -v "      big_dirb" |
>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>                         git update-index --index-info &&
>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
>
>  where the space before big_dir is actually a literal TAB!

Hmm thats odd. Despite having successfully run the filter my repo is
still the same size. git-fsck --full shows loads of dangling commits
but I'm guessing because the repo is packed they don't get dropped. Is
it possible to repack the whole repo or do you have to manually unpack
the packs and re-pack?


-- 
Alex, homepage: http://www.bennee.com/~alex/

Re: Truncating and cleaning a imported git repositary to make it more usable

[Jakub Narebski]

Alex Bennee wrote:

> On Fri, Mar 28, 2008 at 4:43 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:
>> Alex Bennee schrieb:
>>
>>  You really shouldn't do it this way, unless you do it on a ramdisk. Better
>>  use an --index-filter. This is modeled after the last example in the man
>>  page (and, of course, untested):
>>
>>  git filter-branch --index-filter \
>>         'git ls-files -s |
>>          grep -v "      big_dira" |
>>          grep -v "      big_dirb" |
>>                 GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
>>                         git update-index --index-info &&
>>          mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
>>
>>  where the space before big_dir is actually a literal TAB!
> 
> Hmm thats odd. Despite having successfully run the filter my repo is
> still the same size. git-fsck --full shows loads of dangling commits
> but I'm guessing because the repo is packed they don't get dropped. Is
> it possible to repack the whole repo or do you have to manually unpack
> the packs and re-pack?

Did you pruned the repo ("git gc --prune" or "git prune")?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: Truncating and cleaning a imported git repositary to make it more usable

[Michael Haggerty]

Alex Bennee wrote:
> I've succeeded in porting our whole CVS repository into git (with plenty
> of swap, cvsps is very memory hungry). [...]

It sounds like you are doing a one-time conversion.  If so, please
consider using cvs2svn [1], which can also convert to git [2].
cvsps-based tools are known to have problems with most nontrivial CVS
repositories (the contents of the output repository simply don't agree
with those of the CVS original).  IMO, the only reason to use a
cvsps-based tool is if you really need incremental conversions *and* are
willing to live with data loss.

Before you trust your new repository, please make some checks.  As a
minimum, make sure that the contents of each tag and branch tip revision
is the same as the contents that you get if you check out of CVS.  (I
bet they don't!)

Michael

[1] http://cvs2svn.tigris.org
[2] http://cvs2svn.tigris.org/cvs2git.html

Re: Truncating and cleaning a imported git repositary to make it more usable

[Johannes Sixt]

Alex Bennee schrieb:
> Hmm thats odd. Despite having successfully run the filter my repo is
> still the same size. git-fsck --full shows loads of dangling commits
> but I'm guessing because the repo is packed they don't get dropped. Is
> it possible to repack the whole repo or do you have to manually unpack
> the packs and re-pack?

Yes. Run git repack -a -d -f or create a local clone using the file protocol:

   git clone file://$(pwd)/this that

(If you don't use the file protocol, git clone just hard-links the
repository, and you again don't get the reduced size.)

-- Hannes