Is there a way to trim old SHAs from a git tree (so it's not so large)?

Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

After doing a "make mrproper" in my Linux git tree, the result is still 1.1GB 
of files.  Compare that with just the tarball, which is just one-forth the size.

Is there a way to "trim away" old commits from the repository, so that it just 
doesn't take up that much space?  I don't care about any commits made in 2005. 
  As long as I can still do "git pull" from the source repo to update mine, 
that's good enough.

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Shawn Pearce]

Timur Tabi <timur@freescale.com> wrote:
> After doing a "make mrproper" in my Linux git tree, the result is still 
> 1.1GB of files.  Compare that with just the tarball, which is just 
> one-forth the size.
> 
> Is there a way to "trim away" old commits from the repository, so that it 
> just doesn't take up that much space?  I don't care about any commits made 
> in 2005. As long as I can still do "git pull" from the source repo to 
>  update mine, that's good enough.

No.  However...

Have you tried "git repack -a -d" to repack the loose objects into
a pack file?  Doing this every so often should reduce your disk
space consumed by a HUGE amount.

-- 

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Shawn Pearce wrote:

> Have you tried "git repack -a -d" to repack the loose objects into
> a pack file?  Doing this every so often should reduce your disk
> space consumed by a HUGE amount.

Woah!  It shrunk that bad boy down to 420M!  That should do it, thanks!

I wonder why the powerpc tree shrank so much.  Do you think the maintainer 
just needs to run git-repack on his tree?

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Shawn Pearce]

Timur Tabi <timur@freescale.com> wrote:
> Shawn Pearce wrote:
> 
> >Have you tried "git repack -a -d" to repack the loose objects into
> >a pack file?  Doing this every so often should reduce your disk
> >space consumed by a HUGE amount.
> 
> Woah!  It shrunk that bad boy down to 420M!  That should do it, thanks!
> 
> I wonder why the powerpc tree shrank so much.  Do you think the maintainer 
> just needs to run git-repack on his tree?

Possible, yes.

However published repositories don't tend to repack as often as
it makes things harder for people who clone/fetch over HTTP rather
than the native git protocol.

The reason is that HTTP can fetch individual loose objects that
you don't have yet, but if the object is only available in a pack
file then you need to fetch the entire pack file.  But you might
already have most of that pack file, so now you are downloading
lots of data you already have.  :-(

-- 

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Thomas Kolejka]

-------- Original-Nachricht --------
Datum: Wed, 15 Nov 2006 16:11:57 -0600
Von: Timur Tabi <timur@freescale.com>
An: git@vger.kernel.org
Betreff: Is there a way to trim old SHAs from a git tree (so it\'s not so large)?

> After doing a "make mrproper" in my Linux git tree, the result is still
> 1.1GB 
> of files.  Compare that with just the tarball, which is just one-forth the
> size.
> 
> Is there a way to "trim away" old commits from the repository, so that it
> just 
> doesn't take up that much space?  I don't care about any commits made in
> 2005. 
>   As long as I can still do "git pull" from the source repo to update
> mine, 
> that's good enough.
> 
> -- 
> Timur Tabi
> Linux Kernel Developer @ Freescale

Is it possible to do this with shallow clone? 
Shallow clone the local repository my.git (which should be trimmed) starting from the last needed commit to a new local repository my_trimmed.git. And then remove my.git (with something like rm -rf my.git) and rename my_trimmed.git to my.git?

Thomas
-- 
Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen! 

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Thomas Kolejka wrote:

> Is it possible to do this with shallow clone? 

Maybe.  How do you do a shallow clone?  I tried "git clone" followed by 
"git-repack", and that helped a lot, but the result was still twice the size of 
a normal tarball.  I don't see any "shallow" option to the clone command, and 
git-shallow-pack doesn't exist on my installation.

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Shawn Pearce]

Timur Tabi <timur@freescale.com> wrote:
> Thomas Kolejka wrote:
> 
> >Is it possible to do this with shallow clone? 
> 
> Maybe.  How do you do a shallow clone?  I tried "git clone" followed by 
> "git-repack", and that helped a lot, but the result was still twice the 
> size of a normal tarball.  I don't see any "shallow" option to the clone 
> command, and git-shallow-pack doesn't exist on my installation.

Twice the size of a normal tarball isn't too bad, considering that
you have the _complete_ history in the pack and yet the normal
tarball has no history at all.

Shallow clone is a development feature still being working on in
Junio's 'pu' branch of git.git.  It has a few issues still to be
worked out so it hasn't been made part of one of the more stable
branches yet (like 'next', 'master', or 'maint').

-- 

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Shawn Pearce wrote:

> Twice the size of a normal tarball isn't too bad, considering that
> you have the _complete_ history in the pack and yet the normal
> tarball has no history at all.

That is true.  However, for the particular project I'm working on, double the 
size is not really acceptable.

My goal is to provide a source tree that is "git enabled", so that user can use 
git command to fetch, apply, and create patches.  Currently, we're just handing 
out tarballs, so we want to move people to the 21st century.

> Shallow clone is a development feature still being working on in
> Junio's 'pu' branch of git.git.  It has a few issues still to be
> worked out so it hasn't been made part of one of the more stable
> branches yet (like 'next', 'master', or 'maint').

Well, until it's available on an official git release, it doesn't help me.

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Shawn Pearce]

Timur Tabi <timur@freescale.com> wrote:
> >Shallow clone is a development feature still being working on in
> >Junio's 'pu' branch of git.git.  It has a few issues still to be
> >worked out so it hasn't been made part of one of the more stable
> >branches yet (like 'next', 'master', or 'maint').
> 
> Well, until it's available on an official git release, it doesn't help me.

One of the reasons its hanging out in 'pu' still is that there is
a lack of people who are interested in the feature, and thus not
enough people are testing it.  Perhaps you might be able to lend
a hand in that regard?

-- 

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Shawn Pearce wrote:
> Timur Tabi <timur@freescale.com> wrote:
>>> Shallow clone is a development feature still being working on in
>>> Junio's 'pu' branch of git.git.  It has a few issues still to be
>>> worked out so it hasn't been made part of one of the more stable
>>> branches yet (like 'next', 'master', or 'maint').
>> Well, until it's available on an official git release, it doesn't help me.
> 
> One of the reasons its hanging out in 'pu' still is that there is
> a lack of people who are interested in the feature, and thus not
> enough people are testing it.  Perhaps you might be able to lend
> a hand in that regard?

Sure, I can do that!  Can you give me some pointers?  I've never done 
development on git itself, so I don't know Junio or his pu (sorry, I couldn't 
resist :-)).

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Shawn Pearce]

Timur Tabi <timur@freescale.com> wrote:
> Shawn Pearce wrote:
> >Timur Tabi <timur@freescale.com> wrote:
> >>>Shallow clone is a development feature still being working on in
> >>>Junio's 'pu' branch of git.git.  It has a few issues still to be
> >>>worked out so it hasn't been made part of one of the more stable
> >>>branches yet (like 'next', 'master', or 'maint').
> >>Well, until it's available on an official git release, it doesn't help me.
> >
> >One of the reasons its hanging out in 'pu' still is that there is
> >a lack of people who are interested in the feature, and thus not
> >enough people are testing it.  Perhaps you might be able to lend
> >a hand in that regard?
> 
> Sure, I can do that!  Can you give me some pointers?  I've never done 
> development on git itself, so I don't know Junio or his pu (sorry, I 
> couldn't resist :-)).

Junio C Hamano is the Git maintainer.  His published Git repository
is here:

	http://www.kernel.org/pub/scm/git/git.git/

though it oddly has pack files from Jun 2006 in the wrong directory.
Weird.  Anyway...

If you clone from that URL, or better though the native Git protocol:

	git://www.kernel.org/pub/scm/git/git.git

you will get a branch called `pu`, which is the set of "Proposed
Updates" to Git that Junio and others are currently working on.
You can then checkout a branch off that, and build it:

	git checkout -b pu-build pu
	make

finally you can either run from that directory (see INSTALL file)
or you can install the binary somewhere else.  We don't really
recommend using `pu` for production level work, so make sure you
have a backup of any repository you run it on.  :)

-- 

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Shawn Pearce wrote:

> finally you can either run from that directory (see INSTALL file)
> or you can install the binary somewhere else.  We don't really
> recommend using `pu` for production level work, so make sure you
> have a backup of any repository you run it on.  :)

So how do I make a shallow clone?  I've set it all up, but there is no 
git-shallow-clone command, and git help clone doesn't have anything either.


-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Johannes Schindelin]

Hi,

On Tue, 21 Nov 2006, Timur Tabi wrote:

> Shawn Pearce wrote:
> 
> > finally you can either run from that directory (see INSTALL file)
> > or you can install the binary somewhere else.  We don't really
> > recommend using `pu` for production level work, so make sure you
> > have a backup of any repository you run it on.  :)
> 
> So how do I make a shallow clone?  I've set it all up, but there is no
> git-shallow-clone command, and git help clone doesn't have anything either.

Try "git clone --depth 1 <url>". This will cut each ancestor chain after 
one ancestor (IIRC).

Ciao,
Dscho

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Johannes Schindelin wrote:
> Hi,
> 
> On Tue, 21 Nov 2006, Timur Tabi wrote:
> 
>> Shawn Pearce wrote:
>>
>>> finally you can either run from that directory (see INSTALL file)
>>> or you can install the binary somewhere else.  We don't really
>>> recommend using `pu` for production level work, so make sure you
>>> have a backup of any repository you run it on.  :)
>> So how do I make a shallow clone?  I've set it all up, but there is no
>> git-shallow-clone command, and git help clone doesn't have anything either.
> 
> Try "git clone --depth 1 <url>". This will cut each ancestor chain after 
> one ancestor (IIRC).

I think you mean git-clone.sh instead of git-clone.  If I do the above command, 
I get:

$ ./git clone --depth 1 git://127.0.0.1/temp/u-boot-83xx/
Usage: /home/b04825/bin/git-clone [--template=<template_directory>] 
[--use-separate-remote] [--reference <reference-repo>] [--bare] [-l [-s]] [-q] 
[-u <upload-pack>] [--origin <name>] [-n] <repo> [<dir>]

However, git-clone.sh is not quite working either.  I had to run git-daemon on 
my machine, because git-clone.sh doesn't like the http protocol, and my firewall 
blocks everything but that.  So I cloned a repo, started git-daemon, and I tried 
this:

$ ./git-clone.sh --depth 1 git://127.0.0.1/temp/u-boot-83xx
usage: git-fetch-pack [--all] [-q] [-v] [-k] [--thin] [--exec=upload-pack] 
[host:]directory <refs>...
fetch-pack from 'git://127.0.0.1/temp/u-boot-83xx/' failed.

A regular git-clone of git://127.0.0.1/temp/u-boot-83xx works, so I think 
there's something wrong with git-clone.sh or my invocation thereof.

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Jakub Narebski]

Timur Tabi wrote:

> Johannes Schindelin wrote:

>> Try "git clone --depth 1 <url>". This will cut each ancestor chain after 
>> one ancestor (IIRC).
> 
> I think you mean git-clone.sh instead of git-clone.  If I do the above command, 
> I get:
> 
> $ ./git clone --depth 1 git://127.0.0.1/temp/u-boot-83xx/
> Usage: /home/b04825/bin/git-clone [--template=<template_directory>] 
> [--use-separate-remote] [--reference <reference-repo>] [--bare] [-l [-s]] [-q] 
> [-u <upload-pack>] [--origin <name>] [-n] <repo> [<dir>]
> 
> However, git-clone.sh is not quite working either.  I had to run git-daemon on 
> my machine, because git-clone.sh doesn't like the http protocol, and my firewall 
> blocks everything but that.  So I cloned a repo, started git-daemon, and I tried 
> this:
> 
> $ ./git-clone.sh --depth 1 git://127.0.0.1/temp/u-boot-83xx
> usage: git-fetch-pack [--all] [-q] [-v] [-k] [--thin] [--exec=upload-pack] 
> [host:]directory <refs>...
> fetch-pack from 'git://127.0.0.1/temp/u-boot-83xx/' failed.
> 
> A regular git-clone of git://127.0.0.1/temp/u-boot-83xx works, so I think 
> there's something wrong with git-clone.sh or my invocation thereof.

Erm, you have to compile git with "make bindir=$(pwd)" to run it from
working directory. Or just install it somewhere not over git, like
/usr/local/ or /home/local/.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Johannes Schindelin]

Hi,

On Tue, 21 Nov 2006, Timur Tabi wrote:

> I think you mean git-clone.sh instead of git-clone.

No. If "git clone" does not work for you, the compilation failed. Do you 
run git in-place? Then you _have_ to set

	bindir=$(pwd)

in config.mak. Otherwise, the git wrapper (you wrote "git-clone", but then 
you used "git clone") will look in $(HOME)/bin for your executables, and I 
bet that you have the other (official) git installed there.

> However, git-clone.sh is not quite working either.  I had to run 
> git-daemon on my machine, because git-clone.sh doesn't like the http 
> protocol, and my firewall blocks everything but that.

For now, shallow clones only work with the git protocol, i.e. via 
git-daemon and ssh.

Hth,

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Timur Tabi]

Johannes Schindelin wrote:
> Hi,
> 
> On Tue, 21 Nov 2006, Timur Tabi wrote:
> 
>> I think you mean git-clone.sh instead of git-clone.
> 
> No. If "git clone" does not work for you, the compilation failed. Do you 
> run git in-place? Then you _have_ to set
> 
> 	bindir=$(pwd)
> 
> in config.mak.

When I tried that, make spits out this:

GIT_VERSION = 1.4.4.ge1173-dirty
     * new build flags or prefix
(cd perl && /usr/bin/perl Makefile.PL \
         PREFIX='/home/b04825')
Writing Makefile for Git

So I tried this:

make configure
./configure --prefix=$PWD
make

and got this:

     * new build flags or prefix
(cd perl && /usr/bin/perl Makefile.PL \
         PREFIX='/temp/git')
Writing Makefile for Git

which is better, but git-clone still doesn't work:

$ ./git clone --depth 1 git://127.0.0.1/temp/u-boot-83xx
Failed to run command 'clone': Success

$ ./git-clone --depth 1 git://127.0.0.1/temp/u-boot-83xx
usage: git-fetch-pack [--all] [-q] [-v] [-k] [--thin] [--exec=upload-pack] 
[host:]directory <refs>...
fetch-pack from 'git://127.0.0.1/temp/u-boot-83xx' failed.

In both of these cases, the git-daemon process doesn't log anything.

I even tried "export GIT_DIR=$PWD", but that didn't do anything.

-- 
Timur Tabi

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Jakub Narebski]

Timur Tabi wrote:

> make configure
> ./configure --prefix=$PWD
> make

make configure
./configure --prefix=$PWD --bindir=$PWD
make

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: Is there a way to trim old SHAs from a git tree (so it's not so large)?

[Johannes Schindelin]

Hi,

On Tue, 21 Nov 2006, Timur Tabi wrote:

> Shawn Pearce wrote:
> 
> > Shallow clone is a development feature still being working on in
> > Junio's 'pu' branch of git.git.  It has a few issues still to be
> > worked out so it hasn't been made part of one of the more stable
> > branches yet (like 'next', 'master', or 'maint').
> 
> Well, until it's available on an official git release, it doesn't help 
> me.

Well, if you help it, it will be available on an official git release 
soon!

Ciao,
Dscho