Inexplicably deteriorating performance of Git repositories on Windows

Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

Hey,

We have a bunch of Windows users, unfortunately, and they're using the
latest msysGit release (Git-1.7.3.1-preview20101002).

An interesting issue we've noticed is that the Time To Complete of
their common operations start deteriorating inexplicably, and
severely, some time after the clone.

For instance, immediately after a clone, `git status` takes about
5-6s. Which is slow compared to Linux (consistent 1-2s), but still
usable (it's a BIG repo).

However, after a reboot (of all things), `git status` latency
skyrockets to 14-15s, making the repo unusable.

Any idea what's going on?  We just recently switched from SVN, and
those users are getting really frustrated. BTW, the only real
alternative I'm aware of, Cygwin's git, is even slower.

Thanks, D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Wilbert van Dolleweerd]

> We have a bunch of Windows users, unfortunately, and they're using the
> latest msysGit release (Git-1.7.3.1-preview20101002).
>
> An interesting issue we've noticed is that the Time To Complete of
> their common operations start deteriorating inexplicably, and
> severely, some time after the clone.
>
> For instance, immediately after a clone, `git status` takes about
> 5-6s. Which is slow compared to Linux (consistent 1-2s), but still
> usable (it's a BIG repo).
>
> However, after a reboot (of all things), `git status` latency
> skyrockets to 14-15s, making the repo unusable.
>
> Any idea what's going on?  We just recently switched from SVN, and
> those users are getting really frustrated. BTW, the only real
> alternative I'm aware of, Cygwin's git, is even slower.

How big is your repository? We're using some fairly big repositories
over here but I haven't seen this behavior with the latest version of
msysgit.

Kind regards,

Wilbert van Dolleweerd
Blog: http://walkingthestack.wordpress.com/
Twitter: http://www.twitter.com/wvandolleweerd

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
<wilbert@arentheym.com> wrote:
> How big is your repository? We're using some fairly big repositories
> over here but I haven't seen this behavior with the latest version of
> msysgit.

The working copy totals about 4GB. The .git directory, tightly packed, is 1GB.

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Wilbert van Dolleweerd]

2010/11/23 Dun Peal <dunpealer@gmail.com>:

> On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
> <wilbert@arentheym.com> wrote:
>> How big is your repository? We're using some fairly big repositories
>> over here but I haven't seen this behavior with the latest version of
>> msysgit.
>
> The working copy totals about 4GB. The .git directory, tightly packed, is 1GB.

I'll see what size our largest repository is at my company first thing
in the morning.

kind regards,

Wilbert van Dolleweerd
Blog: http://walkingthestack.wordpress.com/
Twitter: http://www.twitter.com/wvandolleweerd

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Stephen Bash]

----- Original Message -----
> From: "Dun Peal" <dunpealer@gmail.com>
> To: "Wilbert van Dolleweerd" <wilbert@arentheym.com>
> Cc: "Git ML" <git@vger.kernel.org>
> Sent: Tuesday, November 23, 2010 2:59:17 PM
> Subject: Re: Inexplicably deteriorating performance of Git repositories on Windows
>
> On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
> <wilbert@arentheym.com> wrote:
> > How big is your repository? We're using some fairly big repositories
> > over here but I haven't seen this behavior with the latest version
> > of msysgit.
> 
> The working copy totals about 4GB. The .git directory, tightly packed,
> is 1GB.

We're working with about a 1.5GB repository, and while I haven't seen an specific msysgit slow downs, I did run into build issues due to Windows anti-virus programs (on-access scans, new files scans, etc).  I had to add my development directory to the anti-virus exception list to speed things back up.

That being said, I do most of my development on Mac and Linux, and msysgit is noticeably slower across the board for me...

Thanks,
Stephen

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Tue, Nov 23, 2010 at 8:25 PM, Stephen Bash <bash@genarts.com> wrote:
> ----- Original Message -----
> We're working with about a 1.5GB repository, and while I haven't seen an specific msysgit slow downs, I did run into build issues due to Windows anti-virus programs (on-access scans, new files scans, etc). I had to add my development directory to the anti-virus exception list to speed things back up.

Yeah, the performance numbers I mentioned are *after* excluding our AV
software from that directory.

> That being said, I do most of my development on Mac and Linux, and msysgit is noticeably slower across the board for me...

Yup, as mentioned, even under the best case Windows scenario (freshly
cloned repo) I'm still seeing `git status` latencies that are x2-3
times those of Linux machines.

I don't hope to get Windows' git to be as fast as on Linux; at this
point, just making it fast enough to be usable would be an
achievement: I can't tell developers who just switched from a fast SVN
setup to wait for 15s for `git status`, an operation they perform
dozens of times per day (other operations, like stash, also take
involve long, unworkable waits).

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Martin Langhoff]

On Tue, Nov 23, 2010 at 2:08 PM, Dun Peal <dunpealer@gmail.com> wrote:
> However, after a reboot (of all things), `git status` latency
> skyrockets to 14-15s, making the repo unusable.

A reboot clears your cache. Reboot and time the first and second run
of git status.

The first one should be slow (due to cold cache), the second one
should be roughly as fast as right after a clone.

If it's not, more debugging may be neeed. But always take your timings
on a well seeded ('hot') cache.

cheers,


m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Tue, Nov 23, 2010 at 9:13 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> A reboot clears your cache. Reboot and time the first and second run
> of git status.

Thanks, after all this benchmarking I'm well aware of cold and warm
caches. The 14-15s timing is unfortunately stable across multiple
runs, after the cache was warm (without it, we got times much longer
than that).

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Ferry Huberts]

Are your users using the 'show my status in the prompt' feature?
If so, then disable all but showing the current branch, it makes a whole
lot of difference :-)

On 11/23/2010 08:08 PM, Dun Peal wrote:
> Hey,
> 
> We have a bunch of Windows users, unfortunately, and they're using the
> latest msysGit release (Git-1.7.3.1-preview20101002).
> 
> An interesting issue we've noticed is that the Time To Complete of
> their common operations start deteriorating inexplicably, and
> severely, some time after the clone.
> 
> For instance, immediately after a clone, `git status` takes about
> 5-6s. Which is slow compared to Linux (consistent 1-2s), but still
> usable (it's a BIG repo).
> 
> However, after a reboot (of all things), `git status` latency
> skyrockets to 14-15s, making the repo unusable.
> 
> Any idea what's going on?  We just recently switched from SVN, and
> those users are getting really frustrated. BTW, the only real
> alternative I'm aware of, Cygwin's git, is even slower.
> 
> Thanks, D
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

grtz

-- 
Ferry Huberts

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Tue, Nov 23, 2010 at 9:49 PM, Ferry Huberts <mailings@hupie.com> wrote:
> Are your users using the 'show my status in the prompt' feature?
> If so, then disable all but showing the current branch, it makes a whole
> lot of difference :-)

Thanks, we're not :-) .D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Andreas Ericsson]

On 11/23/2010 08:08 PM, Dun Peal wrote:
> Hey,
> 
> We have a bunch of Windows users, unfortunately, and they're using the
> latest msysGit release (Git-1.7.3.1-preview20101002).
> 
> An interesting issue we've noticed is that the Time To Complete of
> their common operations start deteriorating inexplicably, and
> severely, some time after the clone.
> 
> For instance, immediately after a clone, `git status` takes about
> 5-6s. Which is slow compared to Linux (consistent 1-2s), but still
> usable (it's a BIG repo).
> 

How many refs (tags and branches) do you have?
Are the refs packed or loose?
If they are loose, does packing them resolve the issue?
Are you using network-mounted or local storage?
What does the .git/config file look like for a user where git status
is excruciatingly slow?
Does copying the config file from a windows user to a linux user make
timings somewhat consistent between various systems?
Do older version of git perform as poorly?
How is the repository laid out (ie, are there any directories with
a ton of files in, or are they spread across multiple directories)?
How many .gitignore files are you using, and what do they look like?

> However, after a reboot (of all things), `git status` latency
> skyrockets to 14-15s, making the repo unusable.
> 

That's just plain weird, and is almost certainly a system issue.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Nguyen Thai Ngoc Duy]

On Wed, Nov 24, 2010 at 2:08 AM, Dun Peal <dunpealer@gmail.com> wrote:
> Hey,
>
> We have a bunch of Windows users, unfortunately, and they're using the
> latest msysGit release (Git-1.7.3.1-preview20101002).
>
> An interesting issue we've noticed is that the Time To Complete of
> their common operations start deteriorating inexplicably, and
> severely, some time after the clone.
>
> For instance, immediately after a clone, `git status` takes about
> 5-6s. Which is slow compared to Linux (consistent 1-2s), but still
> usable (it's a BIG repo).
>
> However, after a reboot (of all things), `git status` latency
> skyrockets to 14-15s, making the repo unusable.

Does assume-unchanged bit (see "git update-index") help? I'm not
suggesting to use it but it would help determine if the slowdown is
worktree-related.
-- 
Duy

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Tay Ray Chuan]

On Wed, Nov 24, 2010 at 3:59 AM, Dun Peal <dunpealer@gmail.com> wrote:
> On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
> <wilbert@arentheym.com> wrote:
>> How big is your repository? We're using some fairly big repositories
>> over here but I haven't seen this behavior with the latest version of
>> msysgit.
>
> The working copy totals about 4GB. The .git directory, tightly packed, is 1GB.

What does the structure of your working tree look like? I think the
depth might be affecting performance.

-- 
Cheers,
Ray Chuan

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Joshua Jensen]

----- Original Message -----
From: Tay Ray Chuan
Date: 11/24/2010 7:16 AM
> On Wed, Nov 24, 2010 at 3:59 AM, Dun Peal<dunpealer@gmail.com>  wrote:
>> On Tue, Nov 23, 2010 at 7:12 PM, Wilbert van Dolleweerd
>> <wilbert@arentheym.com>  wrote:
>> The working copy totals about 4GB. The .git directory, tightly 
>> packed, is 1GB.
> What does the structure of your working tree look like? I think the
> depth might be affecting performance
Whenever I want to know exactly what is going on with disk access, I 
download Process Monitor from http://sysinternals.com/.

In order to just show disk access, I filter entries that begin with TCP, 
UDP, and Reg out.

Josh

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Wed, Nov 24, 2010 at 11:34 AM, Andreas Ericsson <ae@op5.se> wrote:
> How many refs (tags and branches) do you have?

We have about 15 branches, but thousands of tags (we tag each release,
and we make those often). However, I'm not sure how that could make
`git status` slow, since it shouldn't care about tags at all?

> Are the refs packed or loose?

Packed.

> If they are loose, does packing them resolve the issue?
> Are you using network-mounted or local storage?

Local storage on all workstations. One machine with those performance
issues actually utilizes an incredibly fast, cutting edge SSD. It's
twice as fast as the other slow machine, but 8s for `git status` is
still slow.

> What does the .git/config file look like for a user where git status
> is excruciatingly slow?

It's the normal .git/config you get by default on Git 1.7.x when
cloning from a remote. I can paste it if you want, but doubt it will
interest anyone: we didn't modify it.

> Does copying the config file from a windows user to a linux user make
> timings somewhat consistent between various systems?

No.

> Do older version of git perform as poorly?

Yes.

> How is the repository laid out (ie, are there any directories with
> a ton of files in, or are they spread across multiple directories)?

Generally spread across multiple directories.

> How many .gitignore files are you using, and what do they look like?

We're using 25 of those,

>> However, after a reboot (of all things), `git status` latency
>> skyrockets to 14-15s, making the repo unusable.
> That's just plain weird, and is almost certainly a system issue.

Yes, it's weird. I'm not sure it's a "system" issue since we're seeing
it across a diverse set of Windows systems: Vista, Windows 7, pretty
sure I saw XP as well, different hardware and software setups
(different programs installed and running).

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Wed, Nov 24, 2010 at 1:32 PM, Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:
> Does assume-unchanged bit (see "git update-index") help? I'm not
> suggesting to use it but it would help determine if the slowdown is
> worktree-related.

I will try that. Another possibility, perhaps a bit cleaner and more
appropriate as a steady-state solution, is to use sparse checkouts.

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Wed, Nov 24, 2010 at 2:16 PM, Tay Ray Chuan <rctay89@gmail.com> wrote:
> What does the structure of your working tree look like? I think the
> depth might be affecting performance.

I don't think there's anything special about it; it's a big tree, so
naturally we have some deep (15-level) directory branches. Any
particular reason you suspect the depth?

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Dun Peal]

On Wed, Nov 24, 2010 at 5:16 PM, Joshua Jensen
<jjensen@workspacewhiz.com> wrote:
> Whenever I want to know exactly what is going on with disk access, I
> download Process Monitor from http://sysinternals.com/.
>
> In order to just show disk access, I filter entries that begin with TCP,
> UDP, and Reg out.
>
> Josh

Thanks, we tried that and we don't see a whole lot of disk activity on
the "fast" machines.

One emerging theory is that the "slow" Windows machines differ from
the "fast" ones by how their disk cache works.

So `git status` on a large tree heavily depends on caching. Without
it, it would be slow; with it, it's much faster.

We verified that part since when we reboot a fast Windows machine, the
first run of `git status` is slow (~30s) but the next one is much
faster (~5s).

We see a similar phenomenon on Linux: the first run is always
significantly slower than the others.

On slow Windows machines, this difference is much less pronounced.

On a typical "slow" machine, if you clone the repo, the first run of
`git status` on it would already be fast (5s). But then your reboot,
and the first run is slow, but then it only gets up to 14s. And you
can't get back the 5s latency unless you re-clone the repo and status
the fresh clone.

So my theory is that there's a cache that on the "fast" machines
aggressively caches the entire tree on a regular `git status` run. On
such a machine, it's enough to run `git status` once, and after that
initial cold run, the rest will be warm... until you reboot the
machine, rinse, repeat.

On a slow machine, however, cache isn't so aggressive. It might be
write-oriented. So when you write out a whole new working tree, that
tree gets cached as it is written. And for the remainder of the
lifetime of that cache, you get the fully-cached performance you see
on the "fast" machines. But then you reboot the machine, and lose the
cache. And since the caching process isn't aggressive, any number of
`git status` runs won't get you back to the fully cached state. You
will only get that on a newly written working copy.

What do you think?

.D

Re: Inexplicably deteriorating performance of Git repositories on Windows

[A Large Angry SCM]

On 11/24/2010 04:00 PM, Dun Peal wrote:
> On Wed, Nov 24, 2010 at 5:16 PM, Joshua Jensen
> <jjensen@workspacewhiz.com>  wrote:
>> Whenever I want to know exactly what is going on with disk access, I
>> download Process Monitor from http://sysinternals.com/.
>>
>> In order to just show disk access, I filter entries that begin with TCP,
>> UDP, and Reg out.
>>
>> Josh
>
> Thanks, we tried that and we don't see a whole lot of disk activity on
> the "fast" machines.
>
> One emerging theory is that the "slow" Windows machines differ from
> the "fast" ones by how their disk cache works.
>
> So `git status` on a large tree heavily depends on caching. Without
> it, it would be slow; with it, it's much faster.
>
> We verified that part since when we reboot a fast Windows machine, the
> first run of `git status` is slow (~30s) but the next one is much
> faster (~5s).
>
> We see a similar phenomenon on Linux: the first run is always
> significantly slower than the others.
>
> On slow Windows machines, this difference is much less pronounced.
>
> On a typical "slow" machine, if you clone the repo, the first run of
> `git status` on it would already be fast (5s). But then your reboot,
> and the first run is slow, but then it only gets up to 14s. And you
> can't get back the 5s latency unless you re-clone the repo and status
> the fresh clone.
>
> So my theory is that there's a cache that on the "fast" machines
> aggressively caches the entire tree on a regular `git status` run. On
> such a machine, it's enough to run `git status` once, and after that
> initial cold run, the rest will be warm... until you reboot the
> machine, rinse, repeat.
>
> On a slow machine, however, cache isn't so aggressive. It might be
> write-oriented. So when you write out a whole new working tree, that
> tree gets cached as it is written. And for the remainder of the
> lifetime of that cache, you get the fully-cached performance you see
> on the "fast" machines. But then you reboot the machine, and lose the
> cache. And since the caching process isn't aggressive, any number of
> `git status` runs won't get you back to the fully cached state. You
> will only get that on a newly written working copy.
>
> What do you think?

How much memory do the fast and slow machines have? How much memory will 
windows use for disk caching? Is it possible that your normal work flow 
between status' are forcing the caches to pruned due to memory pressure?

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Johannes Sixt]

On Mittwoch, 24. November 2010, Dun Peal wrote:
> So my theory is that there's a cache that on the "fast" machines
> aggressively caches the entire tree on a regular `git status` run. On
> such a machine, it's enough to run `git status` once, and after that
> initial cold run, the rest will be warm... until you reboot the
> machine, rinse, repeat.
>
> On a slow machine, however, cache isn't so aggressive. It might be
> write-oriented. So when you write out a whole new working tree, that
> tree gets cached as it is written. And for the remainder of the
> lifetime of that cache, you get the fully-cached performance you see
> on the "fast" machines. But then you reboot the machine, and lose the
> cache. And since the caching process isn't aggressive, any number of
> `git status` runs won't get you back to the fully cached state. You
> will only get that on a newly written working copy.

You can test this theory on a slow machine: You have cloned a repository, 
rebooted, and you are now at 14s per 'git status'. Now:

 $ git rm -r .
 $ git reset --hard

erases the worktree and writes it out again (do this only on a clean 
checkout!). Are you now at 5s per 'git status'?

-- Hannes

Re: Inexplicably deteriorating performance of Git repositories on Windows

[Robin Rosenberg]

23 nov 2010 kl. 20:08 skrev Dun Peal:

> Hey,
> 
> We have a bunch of Windows users, unfortunately, and they're using the
> latest msysGit release (Git-1.7.3.1-preview20101002).
> 
> An interesting issue we've noticed is that the Time To Complete of
> their common operations start deteriorating inexplicably, and
> severely, some time after the clone.
> 
> For instance, immediately after a clone, `git status` takes about
> 5-6s. Which is slow compared to Linux (consistent 1-2s), but still
> usable (it's a BIG repo).
> 
> However, after a reboot (of all things), `git status` latency
> skyrockets to 14-15s, making the repo unusable.
> 
> Any idea what's going on?  We just recently switched from SVN, and
> those users are getting really frustrated. BTW, the only real
> alternative I'm aware of, Cygwin's git, is even slower.
> 

Is the file system badly fragmented? The worst kind is when the MFT is fragmented,
which may give you very bad performance and the defrag that comes with Windows does
not fix MFT fragmentation.

Is NTFS compression enabled? I doubt its helpfulness with Git.

-- robin