Git and Very Large Codebases

Git and Very Large Codebases

[Brian Herman]

Can git support codebases on the order of magnitude of Gigabytes?
For example, a game developer might want to use git as their SCM
software but their codebases can be in gigabytes in scale.
If git cannot support this how would one begin to create this support?

--
_________________________________
Thank You,
Brian Herman

Re: Git and Very Large Codebases

[Andreas Ericsson]

On 04/20/2010 11:00 AM, Brian Herman wrote:
> Can git support codebases on the order of magnitude of Gigabytes?
> For example, a game developer might want to use git as their SCM
> software but their codebases can be in gigabytes in scale.
> If git cannot support this how would one begin to create this support?
> 

It depends more on the nature of the files than the size of the total
code. With large binary files and such, git will not perform very well.
On a code-tree of 1-30KiB source-code files it will perform
fantastically (compared to alternatives, that is; Nothing in the entire
world will handle multiple gigabytes faster than it can read it from
the disk).

I'd suggest splitting the repository in several different ones to
make git handle it well. Movies go in one repository, audio go in
another, pictures and whatnot in a third, and then you tie it all
together with a super-repo that holds the code and the build-system.

In repositories storing mainly binary files you go through the
gitattributes(5) man-page and set the necessary attributes for
the files in question. In your case, I'd suspect the .gitattributes
file would look something like this:

*.avi -diff -delta -crlf -merge
*.jpg -diff -delta -crlf -merge
*.mp3 -diff -delta -crlf -merge

This will prevent git from attempting delta-compression on the
files. It will also not try to create diffs from them, or perform
crlf-conversion or merge them. The gitattributes(5) man page has
more information regarding this.

Some trial and error with a small test-repository will help you
sort this out and understand what it means.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Git and Very Large Codebases

[Jakub Narebski]

Brian Herman <brianherman@gmail.com> writes:

> Can git support codebases on the order of magnitude of Gigabytes?
> For example, a game developer might want to use git as their SCM
> software but their codebases can be in gigabytes in scale.
> If git cannot support this how would one begin to create this support?

Take a look at git-bigfiles project.

-- 
Jakub Narebski
Poland
ShadeHawk on #git