slow "stg push" on NFS

slow "stg push" on NFS

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 654 bytes --]

hi catalin-

still looking at this problem.

thanks to the several recent patches that add fast-forward support, CPU 
utilization seems to no longer be a problem in general.  what appears to 
be happening is that each push operation deletes and recreates 
everything in the working directory.

for shared file systems, creating and deleting files will always be 
synchronous.  anything we can do to eliminate the need for deleting and 
recreating files that are not changed by a commit would be great.

(i've also noticed that an "stg pop / stg push" sequence causes 'make' 
to rebuild everything -- possibly due to the same reason?)

am i all wet here?

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: slow "stg push" on NFS

[Catalin Marinas]

On Mon, 2005-09-19 at 15:01 -0400, Chuck Lever wrote:
> thanks to the several recent patches that add fast-forward support, CPU 
> utilization seems to no longer be a problem in general.

That's a good improvement but this doesn't work when the base of the
stack is changed.

> what appears to 
> be happening is that each push operation deletes and recreates 
> everything in the working directory.

The push operation only does a 'git-read-tree -u -m' which updates the
filesystem. I haven't looked at the git-read-tree code but I suspect it
only updates the files modified as a result of the merge operation. As
Blaisorblade reported, if there are big changes between the new HEAD and
the old bottom of the patch, like file removals and additions, it will
indeed take more time since git-read-tree cannot deal with them and
gitmergeonefile.py will be invoked for each file.

> for shared file systems, creating and deleting files will always be 
> synchronous.  anything we can do to eliminate the need for deleting and 
> recreating files that are not changed by a commit would be great.

I did some profiling with pushing 15 patches with 'git-read-tree
-m' (without -u for updating the working directory) over NFS and only
updating the working files after pushing all the patches. It didn't make
much difference and decided that it is not worth fully implementing it.
That's mainly because my patches were relatively small (few files
changed) and the base of the stack was already checked out, hence just
the patch files were being updated in the working directory.

> (i've also noticed that an "stg pop / stg push" sequence causes 'make' 
> to rebuild everything -- possibly due to the same reason?)

The same as 'git checkout branch'. I haven't checked whether the
modification time is preserved.

-- 
Catalin

Re: slow "stg push" on NFS

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

Catalin Marinas wrote:
> On Mon, 2005-09-19 at 15:01 -0400, Chuck Lever wrote:
>>what appears to 
>>be happening is that each push operation deletes and recreates 
>>everything in the working directory.
> 
> 
> The push operation only does a 'git-read-tree -u -m' which updates the
> filesystem. I haven't looked at the git-read-tree code but I suspect it
> only updates the files modified as a result of the merge operation. As
> Blaisorblade reported, if there are big changes between the new HEAD and
> the old bottom of the patch, like file removals and additions, it will
> indeed take more time since git-read-tree cannot deal with them and
> gitmergeonefile.py will be invoked for each file.

ja, indeed, gitmergeonefile.py is doing some heavy lifting in my case. 
however, i think the bogosity stems from something my new cache API is 
doing.  i can't reproduce this behavior with stock GIT.

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 439 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Linux NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard