Limiting disk usage

Limiting disk usage

[Piotr Krukowiecki]

Hi, 

What can be done to limit amount of space occupied on clone and checkout? I know about shallow clone and sparse checkout, anything else? 

Preferably I would split repository into multiple repositories but I worry that working with multiple directories would be much more troublesome. But maybe I am wrong? I don't have much experience with submodules, repo tool (what else?)

Thanks 
-- 
Piotr Krukowiecki 

Re: Limiting disk usage

[Matthieu Moy]

Piotr Krukowiecki <piotr.krukowiecki@gmail.com> writes:

> Hi, 
>
> What can be done to limit amount of space occupied on clone and
> checkout? I know about shallow clone and sparse checkout, anything
> else?

Sorry if I'm stating the obvious, but "git gc" is a good start. If you
did something wrong with your repo before, maybe "git gc --aggressive"
can help.

-- 
Matthieu Moy
http://www-verimag.imag.fr/~moy/

Re: Limiting disk usage

[Piotr Krukowiecki]

[resending as sent previous version as HTML, sorry about that]

On Tue, Oct 29, 2013 at 9:21 AM, Matthieu Moy
<Matthieu.Moy@grenoble-inp.fr> wrote:
>
> Piotr Krukowiecki <piotr.krukowiecki@gmail.com> writes:
>
> > Hi,
> >
> > What can be done to limit amount of space occupied on clone and
> > checkout? I know about shallow clone and sparse checkout, anything
> > else?
>
> Sorry if I'm stating the obvious, but "git gc" is a good start. If you
> did something wrong with your repo before, maybe "git gc --aggressive"
> can help.

Yes, I put a lot of code into the repository. I know it is wrong from git POV ;)
That's why I wrote about splitting repository into multiple smaller
repositories, but AFAIK git is not very good at managing that too...

(Background: we're thinking about migrating svn repository into git.
Currently there is one svn repository containing many related
directories. They are branched/merged/tagged together. Often work is
done on all/many of them, but there are some use cases where we need
only some of them)


-- 
Piotr Krukowiecki

Re: Limiting disk usage

[Jakub Narębski]

On 2013-10-29 14:50, Piotr Krukowiecki wrote:

> Yes, I put a lot of code into the repository. I know it is wrong from git POV ;)
> That's why I wrote about splitting repository into multiple smaller
> repositories, but AFAIK git is not very good at managing that too...

Lot of code isn't problem, see Linux kernel or GCC, or LibreOffice.org

> (Background: we're thinking about migrating svn repository into git.
> Currently there is one svn repository containing many related
> directories. They are branched/merged/tagged together. Often work is
> done on all/many of them, but there are some use cases where we need
> only some of them)

If they are independent projects, they should get independent repositories;
you can stitch them back together using git-submodule (or git-subtree).
reposurgeon can hel you with that.

-- 
Jakub Narębski

Re: Limiting disk usage

[Piotr Krukowiecki]

On Tue, Oct 29, 2013 at 3:45 PM, Jakub Narębski <jnareb@gmail.com> wrote:
> On 2013-10-29 14:50, Piotr Krukowiecki wrote:
>>
> Lot of code isn't problem, see Linux kernel or GCC, or LibreOffice.org

OK, it depends on how much code is too much for you. I'm not talking
about performance of git, just disk usage. In my case I have too much
code.


> If they are independent projects, they should get independent repositories;
> you can stitch them back together using git-submodule (or git-subtree).
> reposurgeon can hel you with that.

They are not totally independent projects (nor totally dependent).
Normally you want them all, you want to create a branch/tag on all of
them, if new directory is created you want to automatically have it
too etc. But there are use cases when you need only some of them and
then they can be seen as independent .

git-submodule have several disadvantages for my use:
- does not track branches (you have to specify hardcoded SHA1 instead
of a branch name)
- additional complexity / command layer (you have to do things like
"submodule init", "submodule update")
- you can't simply work on all submodules (you can't simply branch,
merge, commit, log, diff, etc in all submodules using one command in
top-level project)

git-subtree looks much better in that regard, but for example it needs
an explicit prefix (name of "subproject") on which it operates. So you
can't say "update everything" or "commit everything" etc. With
hundreds of subprojects you need another layer to automate this...

-- 
Piotr Krukowiecki

Re: Limiting disk usage

[Jakub Narębski]

On 2013-10-30 08:44, Piotr Krukowiecki wrote:
> On Tue, Oct 29, 2013 at 3:45 PM, Jakub Narębski<jnareb@gmail.com>  wrote:
>> On 2013-10-29 14:50, Piotr Krukowiecki wrote:

>> If they are independent projects, they should get independent repositories;
>> you can stitch them back together using git-submodule (or git-subtree).
>> reposurgeon can hel you with that.
>
> They are not totally independent projects (nor totally dependent).
> Normally you want them all, you want to create a branch/tag on all of
> them, if new directory is created you want to automatically have it
> too etc. But there are use cases when you need only some of them and
> then they can be seen as independent .
>
> git-submodule have several disadvantages for my use:
> - does not track branches (you have to specify hardcoded SHA1 instead
> of a branch name)
> - additional complexity / command layer (you have to do things like
> "submodule init", "submodule update")
> - you can't simply work on all submodules (you can't simply branch,
> merge, commit, log, diff, etc in all submodules using one command in
> top-level project)

git-submodule was created (I think) to manage loosely coupled fairly
independent projects, where you need nevertheless to ensure that
both work together (so you use specific version of subproject for
a given version of superproject).  It looks like this is not what
you have.

 From time to time you can find there ideas about adding "floating"
mode to git-submodule, but as far as I know up till now without
effect...

> git-subtree looks much better in that regard, but for example it needs
> an explicit prefix (name of "subproject") on which it operates. So you
> can't say "update everything" or "commit everything" etc. With
> hundreds of subprojects you need another layer to automate this...

git-subtree is about helping merging / embedding (or re-embedding)
of subproject history in superproject history, and extracting subproject 
history from superproject history.  But embedded subproject is just 
subdirectory to git.  It looks like it isn't what you want.

There are many projects[1] that try to solve problem of managing 
interdependent repositories, e.g.: gitslave, repo, fgit.  Maybe one
of them would be good fit for your problem.

[1]: https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools

-- 
Jakub Narębski

Re: Limiting disk usage

[Fredrik Gustafsson]

On Wed, Oct 30, 2013 at 04:39:02PM +0100, Jakub Narębski wrote:
> From time to time you can find there ideas about adding "floating"
> mode to git-submodule, but as far as I know up till now without
> effect...

How about git submodule update --remote ?

-- 
Med vänliga hälsningar
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com