Why is git submodule slow under windows

Why is git submodule slow under windows

[Armbruster Joachim (BEG/EMS1)]

Hello,

We split a monolithic repository into ~50 submodules. The stored data has the same size. In the 1:1 comparison to the monolithic repository, the submodule handling is very slow. Under Linux everything remains fast, but windows is slow.

So, why is git getting slow when it has to deal with a lot of submodules?
I read something about the lack of the underlying cygwin to handle NTFS in a efficient way. Is this the root cause, or are there other causes also?

Re: Why is git submodule slow under windows

[Thomas Braun]

Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):
> Hello,
> 
> We split a monolithic repository into ~50 submodules. The stored data
> has the same size. In the 1:1 comparison to the monolithic
> repository, the submodule handling is very slow. Under Linux
> everything remains fast, but windows is slow.
> 
> So, why is git getting slow when it has to deal with a lot of
> submodules? I read something about the lack of the underlying cygwin
> to handle NTFS in a efficient way. Is this the root cause, or are
> there other causes also?
> 

Hi,

I assume you are using the latetst git from https://msysgit.github.io on
windows.

I would guess that submodules on windows are slow because
git-submodules.sh is a shell script, and bash on windows is not really
that fast.

There has been some (albeit older) discussion on the msysgit
mailinglist, see [1].

You can play around with core.fscache [2] maybe that helps.

Thomas

[1]: https://groups.google.com/d/msg/msysgit/AuPA4EbwchU/N42tsb6GousJ
[2]:
https://github.com/msysgit/msysgit/releases/tag/Git-1.8.5.2-preview20131230

Re: Why is git submodule slow under windows

[Fredrik Gustafsson]

On Fri, Jul 18, 2014 at 02:08:36PM +0200, Thomas Braun wrote:
> Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):
> > Hello,
> > 
> > We split a monolithic repository into ~50 submodules. The stored data
> > has the same size. In the 1:1 comparison to the monolithic
> > repository, the submodule handling is very slow. Under Linux
> > everything remains fast, but windows is slow.
> > 
> > So, why is git getting slow when it has to deal with a lot of
> > submodules? I read something about the lack of the underlying cygwin
> > to handle NTFS in a efficient way. Is this the root cause, or are
> > there other causes also?
> > 
> 
> Hi,
> 
> I assume you are using the latetst git from https://msysgit.github.io on
> windows.
> 
> I would guess that submodules on windows are slow because
> git-submodules.sh is a shell script, and bash on windows is not really
> that fast.

My guess is that because the shell script uses fork() heavily and fork()
is an expensive operation on Windows, that alone causes the slowddown.

I did a quick test a while back when I rewrote part of git-submodule.sh
in lua and runned it on my repo with ~45 submodules. The speedup was
significant and should be even bigger on windows.

-- 
Med vänlig hälsning
Fredrik Gustafsson

tel: 0733-608274
e-post: iveqy@iveqy.com

Re: Why is git submodule slow under windows

[Jens Lehmann]

Am 18.07.2014 15:02, schrieb Fredrik Gustafsson:
> On Fri, Jul 18, 2014 at 02:08:36PM +0200, Thomas Braun wrote:
>> Am 18.07.2014 12:14, schrieb Armbruster Joachim (BEG/EMS1):
>>> Hello,
>>>
>>> We split a monolithic repository into ~50 submodules. The stored data
>>> has the same size. In the 1:1 comparison to the monolithic
>>> repository, the submodule handling is very slow. Under Linux
>>> everything remains fast, but windows is slow.
>>>
>>> So, why is git getting slow when it has to deal with a lot of
>>> submodules? I read something about the lack of the underlying cygwin
>>> to handle NTFS in a efficient way. Is this the root cause, or are
>>> there other causes also?
>>>
>>
>> Hi,
>>
>> I assume you are using the latetst git from https://msysgit.github.io on
>> windows.
>>
>> I would guess that submodules on windows are slow because
>> git-submodules.sh is a shell script, and bash on windows is not really
>> that fast.
> 
> My guess is that because the shell script uses fork() heavily and fork()
> is an expensive operation on Windows, that alone causes the slowddown.
> 
> I did a quick test a while back when I rewrote part of git-submodule.sh
> in lua and runned it on my repo with ~45 submodules. The speedup was
> significant and should be even bigger on windows.

Without having looked at your Lua rewrite I suspect my recursive
submodule checkout series could speed that up even more, as it
only needs to fork a "git status" followed by a "git checkout" or
"git read-tree" for each submodule. Now that my submodule test
harness and Heiko's submodule config lookup API did hit next,
this is the thing I'm currently working on.