how to reduce disk usage for large .git dirs?

how to reduce disk usage for large .git dirs?

[Olaf Hering]

How can I reduce the disk usage for multiple copies of the same repo?

Up to now I just made copies like this, but since .git alone is already
2GB it becomes expensive:

 # git clone git://host/repo.git repo-master
 # cp -a repo-master repo-branchA
 # cd repo-branchA
 # git checkout -b branchA origin/branchA
 # cd -
 # cp -a repo-master repo-branchB
 # cd repo-branchB
 # git checkout -b branchB origin/branchB
 # cd -
 # cp -a repo-master repo-branchB-feature
 # cd repo-branchB-feature
 # git checkout -b branchB-feature origin/branchB
 # cd -


Since each .git is almost identical I wonder if there is a reliable way
to "share" it. The "git clone" man page mentions --shared as a dangerous
way to do things. It does not give an advice how to manage such cloned
trees.

So how can I reduce the disk usage needed for the four .git dirs above?
I looked around in the docs that came with my git-2.1.3 package, but
found nothing that answers my question. Maybe we can workout something
and add it to one of the existing docs.

Thanks!

Olaf

Re: how to reduce disk usage for large .git dirs?

[Fredrik Gustafsson]

On Thu, Nov 13, 2014 at 12:14:44PM +0100, Olaf Hering wrote:
> 
> How can I reduce the disk usage for multiple copies of the same repo?

You can use --local och --shared. As you say --shared can be dangerous.
If you don't understand the man page enough to know how you should
manage your clones you should probably not use it.

--local seems to be what you're looking for.

However as a side note I'm curious about what your use case is. Why do
you need this many repos?

Your setup looks familiar to me for a subversion user switching to git
and trying to use git as subversion. The common usecase is not to have
multiple worktrees but to do a checkout to the worktree you need to work
on. This is possible with git since it's very fast and I recommend you
to try to use one worktree.

-- 
Med vänlig hälsning
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Thu, Nov 13, Fredrik Gustafsson wrote:

> On Thu, Nov 13, 2014 at 12:14:44PM +0100, Olaf Hering wrote:
> > 
> > How can I reduce the disk usage for multiple copies of the same repo?
> 
> You can use --local och --shared. As you say --shared can be dangerous.
> If you don't understand the man page enough to know how you should
> manage your clones you should probably not use it.

Perhaps its more a lack of doc how to use the result.

> --local seems to be what you're looking for.

I will try this.

> However as a side note I'm curious about what your use case is. Why do
> you need this many repos?

Because I do work in each copy, poke around, or do commits and push
them.

> Your setup looks familiar to me for a subversion user switching to git
> and trying to use git as subversion. The common usecase is not to have
> multiple worktrees but to do a checkout to the worktree you need to work
> on. This is possible with git since it's very fast and I recommend you
> to try to use one worktree.

Switching branches will invalidate timestamps, causing a full rebuild.

Olaf

Re: how to reduce disk usage for large .git dirs?

[Jakub Narębski]

W dniu 2014-11-13 13:03, Olaf Hering pisze:
 > On Thu, Nov 13, Fredrik Gustafsson wrote:
[...]
>> Your setup looks familiar to me for a subversion user switching to git
>> and trying to use git as subversion. The common usecase is not to have
>> multiple worktrees but to do a checkout to the worktree you need to work
>> on. This is possible with git since it's very fast and I recommend you
>> to try to use one worktree.

> Switching branches will invalidate timestamps, causing a full rebuild.

Wouldn't a better way of solving "full rebuild" issue be to use ccache 
or similar solution?

Anyway, switching branches invalidates timestamps only on those files 
that change between branches -- it is to avoid unnecessary rebuilds.

You can always clone with --reference, and use alternates (alternate 
object store). Just don't delete objects in repository that other 
repositories borrow from; GitHub uses refs/borrowers/ namespace for 
that, IIRC.

HTH
-- 
Jakub Narębski

Re: how to reduce disk usage for large .git dirs?

[Roger Gammans]

On Thu, 2014-11-13 at 12:14 +0100, Olaf Hering wrote:
> How can I reduce the disk usage for multiple copies of the same repo?
> 
> Up to now I just made copies like this, but since .git alone is already
> 2GB it becomes expensive:
> 
>  # git clone git://host/repo.git repo-master
>  # cp -a repo-master repo-branchA
>  # cd repo-branchA
>  # git checkout -b branchA origin/branchA
>  # cd -
>  # cp -a repo-master repo-branchB
>  # cd repo-branchB
>  # git checkout -b branchB origin/branchB
>  # cd -
>  # cp -a repo-master repo-branchB-feature
>  # cd repo-branchB-feature
>  # git checkout -b branchB-feature origin/branchB
>  # cd -
> 
> 
> Since each .git is almost identical I wonder if there is a reliable way
> to "share" it. The "git clone" man page mentions --shared as a dangerous
> way to do things. It does not give an advice how to manage such cloned
> trees.

But you're not using clone you are using cp .

The clone man page also says this:-
      --local, -l
           When the repository to clone from is on a local machine, this flag bypasses the normal "Git aware" transport
           mechanism and clones the repository by making a copy of HEAD and everything under objects and refs directories. The
           files under .git/objects/ directory are hardlinked to save space when possible.

           If the repository is specified as a local path (e.g., /path/to/repo), this is the default, and --local is essentially
           a no-op. If the repository is specified as a URL, then this flag is ignored (and we never use the local
           optimizations). Specifying --no-local will override the default when /path/to/repo is given, using the regular Git
           transport instead.


Note the first sentence of the second paragraph.
 eg:
 # git clone git://host/repo.git repo-master
 # git clone repo-master repo-branchA
 # cd repo-branchA
 # git checkout -b branchA origin/branchA
 # cd -
 # git clone repo-master repo-branchB
 # cd repo-branchB
 # git checkout -b branchB origin/branchB
 # cd -
 # git clone repo-master repo-branchB-feature
 # cd repo-branchB-feature
 # git checkout -b branchB-feature origin/branchB
 # cd -

Should work better for you. And there is probably a way to do it less
commands too.

-- 
Roger Gammans <roger@gammascience.co.uk>

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Thu, Nov 13, Roger Gammans wrote:

> Note the first sentence of the second paragraph.
>  eg:
>  # git clone git://host/repo.git repo-master
>  # git clone repo-master repo-branchA
>  # cd repo-branchA
>  # git checkout -b branchA origin/branchA

It fails right here because in this dir only "master" exists, but
branchA is expected.

So far the sequence of commands is:

# git clone git://host/repo.git repo-master
# cd repo-master
# git checkout -b branchA origin/branchA
# git checkout -b branchB origin/branchB
# cd -
# git clone -l -b branchA repo-master repo-branchA
# git clone -l -b branchB repo-master repo-branchB

Next step will be:
# $do_work ; git commit -avs ; git push 

Will that work as expected? Will find out after lunch..


Olaf

Re: how to reduce disk usage for large .git dirs?

[Duy Nguyen]

On Thu, Nov 13, 2014 at 6:14 PM, Olaf Hering <olaf@aepfle.de> wrote:
> Since each .git is almost identical I wonder if there is a reliable way
> to "share" it. The "git clone" man page mentions --shared as a dangerous
> way to do things. It does not give an advice how to manage such cloned
> trees.

If you know what you are doing, you can try git-new-workdir in
contrib/workdir. A safe and reliable version of that is being worked
on, hopefully it'll be released in 2.3.0.
-- 
Duy

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Thu, Nov 13, Olaf Hering wrote:

> So how can I reduce the disk usage needed for the four .git dirs above?
> I looked around in the docs that came with my git-2.1.3 package, but
> found nothing that answers my question. Maybe we can workout something
> and add it to one of the existing docs.


While playing around with this I made some notes, this is the result:


Manage multiple branches as separate copies


To preserve disk space for each clone, use a master copy of the reop and do
local clones from such copy of a remote repository.

First clone the remote repository as usual. Then create a local branch for
each remote branch that is supposed to be worked on:
# git clone git://host/repo.git repo-master                                                                                                                                                                                                                            
# cd repo-master                                                                                                                                                                                                                                                       
# git checkout -b branchA origin/branchA                                                                                                                                                                                                                               
# git checkout -b branchB origin/branchB                                                                                                                                                                                                                               
# cd -                                                                                                                                                                                                                                                                 

Now clone each work branch into its own directory. The work dir references the
master repo. All changes come from and go into this repo, instead of the
remote repo.
# git clone -l -b branchA repo-master repo-branchA                                                                                                                                                                                                                     
# git clone -l -b branchB repo-master repo-branchB                                                                                                                                                                                                                     

To make changs in a work dir, commit as usual. The changes will be pushed from
the work copy into the local master repo. Its required to have some other
branch than branchA active in repo-master, or push from work copy to
repo-master will fail.
# cd repo-master
# git checkout master
# cd -
# cd repo-branchA
# git commit -avs
# git push origin branchA
# cd -

To publish the outstanding changes its required to do this from the master
repo. First checkout the work branch, then pull the local changes and finally
push them to the remote repo.
# cd repo-master
# git checkout branchA
# git pull
# git push origin branchA
# cd -

To receive changes from the remote repo its required to do this from the
master repo. First checkout the work branch, then pull the outstanding remote
changes into the local branch. And finally pull them into the work dir.
# cd repo-master
# git fetch --all (optional)
# git checkout branchB
# git pull
# cd -
# cd repo-branchB
# git pull
# cd -


# vim: set tw=72 et

Re: how to reduce disk usage for large .git dirs?

[Fredrik Gustafsson]

Thanks for sharing your notes! A few comments:

On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:
> First clone the remote repository as usual. Then create a local branch for
> each remote branch that is supposed to be worked on:
> # git clone git://host/repo.git repo-master
> # cd repo-master
> # git checkout -b branchA origin/branchA
> # git checkout -b branchB origin/branchB
> # cd -
>
> Now clone each work branch into its own directory. The work dir references the
> master repo. All changes come from and go into this repo, instead of the
> remote repo.
> # git clone -l -b branchA repo-master repo-branchA
> # git clone -l -b branchB repo-master repo-branchB
>
> To make changs in a work dir, commit as usual. The changes will be pushed from
> the work copy into the local master repo. Its required to have some other
> branch than branchA active in repo-master, or push from work copy to
> repo-master will fail.

That's one of the reason it's not recommended to push into a non-bare
repository. You should clone your repo-master with the --bare option to
avoid having a work dir there.

> To publish the outstanding changes its required to do this from the master
> repo. First checkout the work branch, then pull the local changes and finally
> push them to the remote repo.
> # cd repo-master
> # git checkout branchA
> # git pull
> # git push origin branchA
> # cd -

It's not. You could just add your remote repository as a remote to each
of your clones of your master repo and push directly from them. It
would be much simplier and it would allow you to directly fetch changes
from your remote into your branches as well.

(however, I'm not sure but I think, that this will slowly increase the
difference between your repositories when you develop. So that they
won't change any new data since to local clone was made).

-- 
Med vänlig hälsning
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: how to reduce disk usage for large .git dirs?

[Johan Herland]

On Thu, Nov 13, 2014 at 5:03 PM, Fredrik Gustafsson <iveqy@iveqy.com> wrote:
> Thanks for sharing your notes! A few comments:
>
> On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:
>> First clone the remote repository as usual. Then create a local branch for
>> each remote branch that is supposed to be worked on:
>> # git clone git://host/repo.git repo-master
>> # cd repo-master
>> # git checkout -b branchA origin/branchA
>> # git checkout -b branchB origin/branchB
>> # cd -
>>
>> Now clone each work branch into its own directory. The work dir references the
>> master repo. All changes come from and go into this repo, instead of the
>> remote repo.
>> # git clone -l -b branchA repo-master repo-branchA
>> # git clone -l -b branchB repo-master repo-branchB
>>
>> To make changs in a work dir, commit as usual. The changes will be pushed from
>> the work copy into the local master repo. Its required to have some other
>> branch than branchA active in repo-master, or push from work copy to
>> repo-master will fail.
>
> That's one of the reason it's not recommended to push into a non-bare
> repository. You should clone your repo-master with the --bare option to
> avoid having a work dir there.
>
>> To publish the outstanding changes its required to do this from the master
>> repo. First checkout the work branch, then pull the local changes and finally
>> push them to the remote repo.
>> # cd repo-master
>> # git checkout branchA
>> # git pull
>> # git push origin branchA
>> # cd -
>
> It's not. You could just add your remote repository as a remote to each
> of your clones of your master repo and push directly from them. It
> would be much simplier and it would allow you to directly fetch changes
> from your remote into your branches as well.
>
> (however, I'm not sure but I think, that this will slowly increase the
> difference between your repositories when you develop. So that they
> won't change any new data since to local clone was made).

Can you not do this much simpler with --reference? Like this:

  $ git clone --bare git://host/repo.git repo-master
  $ git clone -b branchA --reference repo-master git://host/repo.git
repo-branchA
  $ git clone -b branchB --reference repo-master git://host/repo.git
repo-branchB

All three repos now push/pull directly to/from git://host/repo.git,
but repo-branchA and repo-branchB reference objects from within the
bare repo-master. You have to make use to never delete objects from
repo-master (if those objects happen to be referenced from
repo-branchA|B). If you want to prevent the repos growing in size, you
must devise a way to add new objects into repo-master before
repo-branchA|B. (e.g. a nightly cron-job in repo-master that fetches
from origin), so that when repo-branchA|B pulls, they will find most
objects are already present in repo-master.


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: how to reduce disk usage for large .git dirs?

[Jeff King]

On Thu, Nov 13, 2014 at 05:08:19PM +0100, Johan Herland wrote:

> Can you not do this much simpler with --reference? Like this:
> 
>   $ git clone --bare git://host/repo.git repo-master
>   $ git clone -b branchA --reference repo-master git://host/repo.git
> repo-branchA
>   $ git clone -b branchB --reference repo-master git://host/repo.git
> repo-branchB
> 
> All three repos now push/pull directly to/from git://host/repo.git,
> but repo-branchA and repo-branchB reference objects from within the
> bare repo-master. You have to make use to never delete objects from
> repo-master

I think the "never delete" part is why we usually warn people off of
using alternates. I think at the least you would have to "git config
gc.auto 0" in the bare repository (otherwise your nightly fetches risk
pruning). Of course you'd probably want to repack eventually for
performance reasons. So maybe setting gc.pruneExpire is a better option
(to something like "20.years.ago").

> If you want to prevent the repos growing in size, you must devise a
> way to add new objects into repo-master before repo-branchA|B. (e.g. a
> nightly cron-job in repo-master that fetches from origin), so that
> when repo-branchA|B pulls, they will find most objects are already
> present in repo-master.

You can also fetch from the children into repo-master periodically.
Like:

  cd repo-master &&
  for i in branchA branchB; do
    git fetch ../$i +refs/*:refs/remotes/$i/*
  done

after which it is actually safe to run "git gc" in the master (assuming
there isn't simultaneous activity in the children). This is how we
manage fork networks on GitHub (we take in objects to individual forks
via push, and then migrate them to the master repo via fetch).

-Peff

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Thu, Nov 13, Fredrik Gustafsson wrote:

> Thanks for sharing your notes! A few comments:
> 
> On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:
> > First clone the remote repository as usual. Then create a local branch for
> > each remote branch that is supposed to be worked on:
> > # git clone git://host/repo.git repo-master
> > # cd repo-master
> > # git checkout -b branchA origin/branchA
> > # git checkout -b branchB origin/branchB
> > # cd -
> >
> > Now clone each work branch into its own directory. The work dir references the
> > master repo. All changes come from and go into this repo, instead of the
> > remote repo.
> > # git clone -l -b branchA repo-master repo-branchA
> > # git clone -l -b branchB repo-master repo-branchB
> >
> > To make changs in a work dir, commit as usual. The changes will be pushed from
> > the work copy into the local master repo. Its required to have some other
> > branch than branchA active in repo-master, or push from work copy to
> > repo-master will fail.
> 
> That's one of the reason it's not recommended to push into a non-bare
> repository. You should clone your repo-master with the --bare option to
> avoid having a work dir there.

So my repo-master is now "bare". I pushed from repo-branchA into
repo-master and see my commits in both repos. But pushing from
repo-master to the remote fails because repo-master does not have
outstanding remote commits. However, git fetch doesnt do anything:

Fetching origin
From host:/remote/dir
* branch            HEAD       -> FETCH_HEAD


Obviously I miss something. The man page of git clone or fetch does not
mention how "bare" is supposed to be handled.

Olaf

Re: how to reduce disk usage for large .git dirs?

[Fredrik Gustafsson]

On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:
> So my repo-master is now "bare". I pushed from repo-branchA into
> repo-master and see my commits in both repos. But pushing from
> repo-master to the remote fails because repo-master does not have
> outstanding remote commits. However, git fetch doesnt do anything:

Are you mixing up your branches? So that you're updating one branch in
your master repo but trying to push an other branch to your remote repo?

-- 
Med vänlig hälsning
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Fri, Nov 14, Fredrik Gustafsson wrote:

> On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:
> > So my repo-master is now "bare". I pushed from repo-branchA into
> > repo-master and see my commits in both repos. But pushing from
> > repo-master to the remote fails because repo-master does not have
> > outstanding remote commits. However, git fetch doesnt do anything:
> 
> Are you mixing up your branches? So that you're updating one branch in
> your master repo but trying to push an other branch to your remote repo?

I dont think so. I have branchA in repo-branchA, and a 'git push origin
branchA' puts it into repo-master. 
How is a bare repo supposed to be updated? Is a simple 'git fetch --all'
supposed to work?

Olaf

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Fri, Nov 14, Olaf Hering wrote:

> On Fri, Nov 14, Fredrik Gustafsson wrote:
> 
> > On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:
> > > So my repo-master is now "bare". I pushed from repo-branchA into
> > > repo-master and see my commits in both repos. But pushing from
> > > repo-master to the remote fails because repo-master does not have
> > > outstanding remote commits. However, git fetch doesnt do anything:
> > 
> > Are you mixing up your branches? So that you're updating one branch in
> > your master repo but trying to push an other branch to your remote repo?
> 
> I dont think so. I have branchA in repo-branchA, and a 'git push origin
> branchA' puts it into repo-master. 
> How is a bare repo supposed to be updated? Is a simple 'git fetch --all'
> supposed to work?

Is there s a slim chance that I have to fetch in repo-master before
doing a git push in repo-branchA? git remote show origin shows that some
branches are out of date.

Olaf

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Thu, Nov 13, Fredrik Gustafsson wrote:

> That's one of the reason it's not recommended to push into a non-bare
> repository. You should clone your repo-master with the --bare option to
> avoid having a work dir there.

Even if I do a fresh clone with --bare, the result can not be updated
anymore with git fetch. What I'm doing wrong?

Olaf

Re: how to reduce disk usage for large .git dirs?

[Andreas Schwab]

Olaf Hering <olaf@aepfle.de> writes:

> Even if I do a fresh clone with --bare, the result can not be updated
> anymore with git fetch. What I'm doing wrong?

A --bare clone has no connection to its origin (there are no remotes).
You want a --mirror.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: how to reduce disk usage for large .git dirs?

[Olaf Hering]

On Fri, Nov 14, Andreas Schwab wrote:

> Olaf Hering <olaf@aepfle.de> writes:
> 
> > Even if I do a fresh clone with --bare, the result can not be updated
> > anymore with git fetch. What I'm doing wrong?
> 
> A --bare clone has no connection to its origin (there are no remotes).
> You want a --mirror.

Using --mirror for this purpose is dangerous because it will most likely
overwrite changes on the remote side. Fortunately I used 'git push --dry-run
origin' and the output was like:

To git://host/repo.git
 + abbrev1..abbrev2 branchA -> branchA (forced update)

Before that I pushed already to the remote repo without realizing where the
push goes to.

Looks like using the --bare option for cloning the master and then doing
something like "git clone --origin local_bare -b branchA --reference
repo-master git://host/repo.git repo-branchA" will work better.

Olaf