Big project, slow access!

Big project, slow access!

[Toan Pham]

Hi,

I use git to maintain a project that is at least 8 gigs in size.
The project is a Linux from Scratch repository that includes source
codes to approximately 2000 open source projects,
gcc tool-chain, 1000+ configurations for different software packages,
source code for different kernel versions,
and many linux distributions/flavors resulted from this LFS build environment.

The git's object repository is now 4.6 gigs and consists of approx.
610,000 files and folders.
The speed of git is now terribly slow.  Each time I use basic commands
like 'git status' or 'git diff',
it would take at least 5 minutes for git to give me back a result.
Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.

would  someone please recommend on how i can optimize git's performance?
Git is so slow, are there better ways to manage a project like this?


Thank you.


Toan

Re: Big project, slow access!

[Nicolas Pitre]

On Fri, 18 Sep 2009, Toan Pham wrote:

> Hi,
> 
> I use git to maintain a project that is at least 8 gigs in size.
> The project is a Linux from Scratch repository that includes source
> codes to approximately 2000 open source projects,
> gcc tool-chain, 1000+ configurations for different software packages,
> source code for different kernel versions,
> and many linux distributions/flavors resulted from this LFS build environment.

How did you organize things in your repository?

> The git's object repository is now 4.6 gigs and consists of approx.
> 610,000 files and folders.
> The speed of git is now terribly slow.  Each time I use basic commands
> like 'git status' or 'git diff',
> it would take at least 5 minutes for git to give me back a result.

Did you repack your repository?  If so, what parameter did you use?

> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.

If so you'll have problems repacking your repository.  You'd need a 
64-bit machine with enough RAM to be able to make the repository fully 
packed, so it might then be tight enough to fit more confortably on a 
32-bit machine afterwards.

> would  someone please recommend on how i can optimize git's performance?
> Git is so slow, are there better ways to manage a project like this?

You could have a look at submodule support.


Nicolas

Re: Big project, slow access!

[Thiago Farina]

On Fri, Sep 18, 2009 at 3:38 PM, Toan Pham <tpham3783@gmail.com> wrote:
> Hi,
>
> I use git to maintain a project that is at least 8 gigs in size.
> The project is a Linux from Scratch repository that includes source
> codes to approximately 2000 open source projects,
> gcc tool-chain, 1000+ configurations for different software packages,
> source code for different kernel versions,
> and many linux distributions/flavors resulted from this LFS build environment.
>
> The git's object repository is now 4.6 gigs and consists of approx.
> 610,000 files and folders.
> The speed of git is now terribly slow.  Each time I use basic commands
> like 'git status' or 'git diff',
> it would take at least 5 minutes for git to give me back a result.
> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.
>
> would  someone please recommend on how i can optimize git's performance?
> Git is so slow, are there better ways to manage a project like this?
Git is so slow? What you expect with 4.6 gigs? It take some time to do
the things. And it is not slow. It is very fast, and it was created
with this goal, to be more fast than others VCS.
>
>
> Thank you.
>
>
> Toan
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Big project, slow access!

[Nicolas Pitre]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1384 bytes --]

On Fri, 18 Sep 2009, Thiago Farina wrote:

> On Fri, Sep 18, 2009 at 3:38 PM, Toan Pham <tpham3783@gmail.com> wrote:
> > Hi,
> >
> > I use git to maintain a project that is at least 8 gigs in size.
> > The project is a Linux from Scratch repository that includes source
> > codes to approximately 2000 open source projects,
> > gcc tool-chain, 1000+ configurations for different software packages,
> > source code for different kernel versions,
> > and many linux distributions/flavors resulted from this LFS build environment.
> >
> > The git's object repository is now 4.6 gigs and consists of approx.
> > 610,000 files and folders.
> > The speed of git is now terribly slow.  Each time I use basic commands
> > like 'git status' or 'git diff',
> > it would take at least 5 minutes for git to give me back a result.
> > Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.
> >
> > would  someone please recommend on how i can optimize git's performance?
> > Git is so slow, are there better ways to manage a project like this?
> Git is so slow? What you expect with 4.6 gigs? It take some time to do
> the things. And it is not slow. It is very fast, and it was created
> with this goal, to be more fast than others VCS.

I wrote some piece of the code in Git and I do know how it is 
possible for git to become very slow.  Denying it is not the way to go.


Nicolas

Re: Big project, slow access!

[R. Tyler Ballance]

[-- Attachment #1: Type: text/plain, Size: 1766 bytes --]


On Fri, 18 Sep 2009, Toan Pham wrote:

> Hi,
> 
> I use git to maintain a project that is at least 8 gigs in size.
> The project is a Linux from Scratch repository that includes source
> codes to approximately 2000 open source projects,
> gcc tool-chain, 1000+ configurations for different software packages,
> source code for different kernel versions,
> and many linux distributions/flavors resulted from this LFS build environment.
> 
> The git's object repository is now 4.6 gigs and consists of approx.
> 610,000 files and folders.
> The speed of git is now terribly slow.  Each time I use basic commands
> like 'git status' or 'git diff',
> it would take at least 5 minutes for git to give me back a result.
> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.

Howdy Toan, we have a similarly large repository ~405k files, the .git
folder fully packed is ~6GB. 

The advise to fully-pack your repository is likely going to have the
greatest impact on your performance in the short term, in the long term
however you might want to consider using git-filter-branch(1) or other
tools available to separate our the components of your current Git
reposotory into a series of repos.

The performance hit you're seeing likely has nothing to do with your
processor speed either, but rather your disk search speed (i'm waiting
for a new fancy SSD to help alleviate my issues ;))

> would  someone please recommend on how i can optimize git's performance?
> Git is so slow, are there better ways to manage a project like this?

Rethink how your project is laid out, and whether certain binaries files
need to sit in the tree, or can be build on a need-by-need basis.



Cheers
-R. Tyler Ballance
Slide, Inc.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: Big project, slow access!

[Toan Pham]

Dear Tyler


Thank you for your valuable feedback.

I'll research into git-filter-branch and also dividing a big project
into several sub-repositories.
This seems to increase the performance very much; however, there is a
draw-back that I am a little bit
concern with.  When we use several sub-repos option, we would probably
do manual book-keeping as to
which repo commits are compatible/built-able with other repo. commits.
 How did you manage to track
dependencies and their versions between different depos?

>>i'm waiting for a new fancy SSD to help alleviate my issues.

Please report the performance increase after you tested on your SS Drive.








On Fri, Sep 18, 2009 at 5:32 PM, R. Tyler Ballance <tyler@slide.com> wrote:
>
> On Fri, 18 Sep 2009, Toan Pham wrote:
>
>> Hi,
>>
>> I use git to maintain a project that is at least 8 gigs in size.
>> The project is a Linux from Scratch repository that includes source
>> codes to approximately 2000 open source projects,
>> gcc tool-chain, 1000+ configurations for different software packages,
>> source code for different kernel versions,
>> and many linux distributions/flavors resulted from this LFS build environment.
>>
>> The git's object repository is now 4.6 gigs and consists of approx.
>> 610,000 files and folders.
>> The speed of git is now terribly slow.  Each time I use basic commands
>> like 'git status' or 'git diff',
>> it would take at least 5 minutes for git to give me back a result.
>> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.
>
> Howdy Toan, we have a similarly large repository ~405k files, the .git
> folder fully packed is ~6GB.
>
> The advise to fully-pack your repository is likely going to have the
> greatest impact on your performance in the short term, in the long term
> however you might want to consider using git-filter-branch(1) or other
> tools available to separate our the components of your current Git
> reposotory into a series of repos.
>
> The performance hit you're seeing likely has nothing to do with your
> processor speed either, but rather your disk search speed (i'm waiting
> for a new fancy SSD to help alleviate my issues ;))
>
>> would  someone please recommend on how i can optimize git's performance?
>> Git is so slow, are there better ways to manage a project like this?
>
> Rethink how your project is laid out, and whether certain binaries files
> need to sit in the tree, or can be build on a need-by-need basis.
>
>
>
> Cheers
> -R. Tyler Ballance
> Slide, Inc.
>

Re: Big project, slow access!

[R. Tyler Ballance]

[-- Attachment #1: Type: text/plain, Size: 3523 bytes --]


On Tue, 22 Sep 2009, Toan Pham wrote:

> Dear Tyler
> 
> 
> Thank you for your valuable feedback.
> 
> I'll research into git-filter-branch and also dividing a big project
> into several sub-repositories.
> This seems to increase the performance very much; however, there is a
> draw-back that I am a little bit
> concern with.  When we use several sub-repos option, we would probably
> do manual book-keeping as to
> which repo commits are compatible/built-able with other repo. commits.
>  How did you manage to track
> dependencies and their versions between different depos?

This wholly depends on how your project laid out, for example, if your
sub-repositories are on different project timelines (shared components
come to mind) where you would want to update that submodule's HEAD in
the super-project. It's worth making sure you're aware of what you're
getting yourself into with git-submodule(1) before you dive in and start
splitting your project into submodules:

	http://www.kernel.org/pub/software/scm/git/docs/user-manual.html#submodules
	http://github.com/guides/developing-with-submodules

You could also look at using submodules and Android's repo command:
	http://source.android.com/download/using-repo



> >>i'm waiting for a new fancy SSD to help alleviate my issues.
> 
> Please report the performance increase after you tested on your SS Drive.

This was tongue-in-cheek, I can't afford an SSD in my laptop just yet ;)



Cheers

> On Fri, Sep 18, 2009 at 5:32 PM, R. Tyler Ballance <tyler@slide.com> wrote:
> >
> > On Fri, 18 Sep 2009, Toan Pham wrote:
> >
> >> Hi,
> >>
> >> I use git to maintain a project that is at least 8 gigs in size.
> >> The project is a Linux from Scratch repository that includes source
> >> codes to approximately 2000 open source projects,
> >> gcc tool-chain, 1000+ configurations for different software packages,
> >> source code for different kernel versions,
> >> and many linux distributions/flavors resulted from this LFS build environment.
> >>
> >> The git's object repository is now 4.6 gigs and consists of approx.
> >> 610,000 files and folders.
> >> The speed of git is now terribly slow.  Each time I use basic commands
> >> like 'git status' or 'git diff',
> >> it would take at least 5 minutes for git to give me back a result.
> >> Again, the machine that i run git on is a P4 3.2 gig-hertz with HT.
> >
> > Howdy Toan, we have a similarly large repository ~405k files, the .git
> > folder fully packed is ~6GB.
> >
> > The advise to fully-pack your repository is likely going to have the
> > greatest impact on your performance in the short term, in the long term
> > however you might want to consider using git-filter-branch(1) or other
> > tools available to separate our the components of your current Git
> > reposotory into a series of repos.
> >
> > The performance hit you're seeing likely has nothing to do with your
> > processor speed either, but rather your disk search speed (i'm waiting
> > for a new fancy SSD to help alleviate my issues ;))
> >
> >> would  someone please recommend on how i can optimize git's performance?
> >> Git is so slow, are there better ways to manage a project like this?
> >
> > Rethink how your project is laid out, and whether certain binaries files
> > need to sit in the tree, or can be build on a need-by-need basis.
> >
> >
> >
> > Cheers
> > -R. Tyler Ballance
> > Slide, Inc.
> >
-R. Tyler Ballance
Slide, Inc.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]