git svn forking an awful lot of "git config"s

git svn forking an awful lot of "git config"s

[Johannes Schindelin]

Hi,

we are approaching our first release of msysGit's installer with git-svn.  
However, I am experiencing a very bad performance, and an error:

$ git svn fetch
trace: exec: 'git-svn' 'fetch'
trace: built-in: git 'config' '--bool' '--get' 'svn.fetchall'
trace: built-in: git 'config' '--get' 'svn.username'
trace: built-in: git 'config' '--get' 'svn.repackflags'
trace: built-in: git 'config' '--bool' '--get' 'svn.quiet'
trace: built-in: git 'config' '--bool' '--get' 'svn.noauthcache'
trace: built-in: git 'config' '--get' 'svn.revision'
trace: built-in: git 'config' '--int' '--get' 'svn.repack'
trace: built-in: git 'config' '--int' '--get' 'svn.logwindowsize'
trace: built-in: git 'config' '--bool' '--get' 'svn.nocheckout'
trace: built-in: git 'config' '--get' 'svn.configdir'
trace: built-in: git 'config' '--bool' '--get' 'svn.noMetadata'
trace: built-in: git 'config' '--bool' '--get' 'svn.useSvnsyncProps'
trace: built-in: git 'config' '--bool' '--get' 'svn.followparent'
trace: built-in: git 'config' '--get' 'svn.authorsfile'
trace: built-in: git 'config' '--bool' '--get' 'svn.useSvmProps'
trace: built-in: git 'config' '--bool' '--get' 'svn.uselogauthor'
trace: built-in: git 'rev-parse' '--symbolic' '--all'
trace: built-in: git 'config' '-l'
trace: built-in: git 'config' '-l'
trace: built-in: git 'config' '-l'
trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.branches-maxRev'
trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.tags-maxRev'
trace: built-in: git 'config' '--get' 'svn-remote.svn.url'
trace: built-in: git 'config' '--get' 'svn-remote.svn.uuid'
trace: built-in: git 'config' 'svn-remote.svn.branches-maxRev' '8'
could not lock config file
config svn-remote.svn.branches-maxRev 8: command returned error: 255

I suspect that the locking problem is due to some strange anti-virus 
interaction, because issuing the same command on the command line 
succeeds.

However, did you notice the many calls to "git config"?  Especially the 
three ones which list all values anyway?

I am not really sure if that is the single reason of the slowness -- 
remember, Windows is mightily spawn()-challenged -- but it sure would help 
to have git-svn read the config once at the beginning, probably with "-z", 
too, and then just read from the cached values, no?

Ciao,
Dscho

P.S.: how far is the svn:external->submodule stuff?

Re: git svn forking an awful lot of "git config"s

[Eric Wong]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
> 
> we are approaching our first release of msysGit's installer with git-svn.  
> However, I am experiencing a very bad performance, and an error:
> 
> $ git svn fetch
> trace: exec: 'git-svn' 'fetch'
> trace: built-in: git 'config' '--bool' '--get' 'svn.fetchall'
> trace: built-in: git 'config' '--get' 'svn.username'
> trace: built-in: git 'config' '--get' 'svn.repackflags'
> trace: built-in: git 'config' '--bool' '--get' 'svn.quiet'
> trace: built-in: git 'config' '--bool' '--get' 'svn.noauthcache'
> trace: built-in: git 'config' '--get' 'svn.revision'
> trace: built-in: git 'config' '--int' '--get' 'svn.repack'
> trace: built-in: git 'config' '--int' '--get' 'svn.logwindowsize'
> trace: built-in: git 'config' '--bool' '--get' 'svn.nocheckout'
> trace: built-in: git 'config' '--get' 'svn.configdir'
> trace: built-in: git 'config' '--bool' '--get' 'svn.noMetadata'
> trace: built-in: git 'config' '--bool' '--get' 'svn.useSvnsyncProps'
> trace: built-in: git 'config' '--bool' '--get' 'svn.followparent'
> trace: built-in: git 'config' '--get' 'svn.authorsfile'
> trace: built-in: git 'config' '--bool' '--get' 'svn.useSvmProps'
> trace: built-in: git 'config' '--bool' '--get' 'svn.uselogauthor'
> trace: built-in: git 'rev-parse' '--symbolic' '--all'
> trace: built-in: git 'config' '-l'
> trace: built-in: git 'config' '-l'
> trace: built-in: git 'config' '-l'
> trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.branches-maxRev'
> trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.tags-maxRev'
> trace: built-in: git 'config' '--get' 'svn-remote.svn.url'
> trace: built-in: git 'config' '--get' 'svn-remote.svn.uuid'
> trace: built-in: git 'config' 'svn-remote.svn.branches-maxRev' '8'
> could not lock config file
> config svn-remote.svn.branches-maxRev 8: command returned error: 255
> 
> I suspect that the locking problem is due to some strange anti-virus 
> interaction, because issuing the same command on the command line 
> succeeds.

I believe somebody on the list also had the same problem with file
locking in Windows.  Unfortunately, I have little idea as to what could
be wrong.  Could Windows not be releasing file locks properly?

It could be worth it to write a standalone Perl script that replicates
the git-config calls without the rest of git-svn getting in the way...

> However, did you notice the many calls to "git config"?  Especially the 
> three ones which list all values anyway?
> 
> I am not really sure if that is the single reason of the slowness -- 
> remember, Windows is mightily spawn()-challenged -- but it sure would help 
> to have git-svn read the config once at the beginning, probably with "-z", 
> too, and then just read from the cached values, no?

Many months ago, I thought about implementing a transparent caching layer
in Git.pm to work with git configs.  Of course, that requires
cooperation from all readers/writers within the process...  Done
correctly, it would help more than just git-svn. too.

I think I had this idea around the time we made git-config output Perl
hashes and arrays.

> Ciao,
> Dscho
> 
> P.S.: how far is the svn:external->submodule stuff?

Yikes.  I've let other work pile up on my ever-growing todo-list :/
I'll see if I can dig it out and wrap it up this weekend or next...

-- 
Eric Wong

Re: git svn forking an awful lot of "git config"s

[Jakub Narebski]

Eric Wong <normalperson@yhbt.net> writes:

> Many months ago, I thought about implementing a transparent caching layer
> in Git.pm to work with git configs.  Of course, that requires
> cooperation from all readers/writers within the process...  Done
> correctly, it would help more than just git-svn. too.

Take a look how gitweb uses "git config -z -l" to read all config
in one go, and save it to hash for later use, lazily.
 
> I think I had this idea around the time we made git-config output Perl
> hashes and arrays.

This is an alternate solution, better for simple scripts and one-off
scripts (you don't need to write "git config -z -l" parser), but
I think that additional eval might be not good for performance.

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: git svn forking an awful lot of "git config"s

[Nagy Balázs]

Jakub Narebski wrote:
> Take a look how gitweb uses "git config -z -l" to read all config
> in one go, and save it to hash for later use, lazily.
...
> This is an alternate solution, better for simple scripts and one-off
> scripts (you don't need to write "git config -z -l" parser), but
> I think that additional eval might be not good for performance.
>   


Maybe I'll finish my git-depot proof of concept script someday.  Here's 
the configuration reader method I use (please note git-depot doesn't use 
repositories -- yet):

sub readConfig
{
	my $self = shift;
	my ($cfg, @cfg, $cat, $key, $val);

	$cfg = Git::command(qw/config -z -l/);
	@cfg = split(/\000/m, $cfg);
	%{$self->{cfg}} = ();
	foreach (@cfg) {
		if (($cat, $key, $val) = (/\A(\S+)\.(\S+)[\n\r]+^(.+)\Z/ms)) {
			next unless $cat eq 'depot';
			$self->{cfg}->{$key} = $val;
		}
	}
	return 1;
}

Works like a charm.
-- 
-jul-

Re: git svn forking an awful lot of "git config"s

[Johannes Schindelin]

Him

On Wed, 20 Feb 2008, Eric Wong wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> 
> > we are approaching our first release of msysGit's installer with 
> > git-svn.  However, I am experiencing a very bad performance, and an 
> > error:
> > 
> > $ git svn fetch
> > trace: exec: 'git-svn' 'fetch'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.fetchall'
> > trace: built-in: git 'config' '--get' 'svn.username'
> > trace: built-in: git 'config' '--get' 'svn.repackflags'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.quiet'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.noauthcache'
> > trace: built-in: git 'config' '--get' 'svn.revision'
> > trace: built-in: git 'config' '--int' '--get' 'svn.repack'
> > trace: built-in: git 'config' '--int' '--get' 'svn.logwindowsize'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.nocheckout'
> > trace: built-in: git 'config' '--get' 'svn.configdir'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.noMetadata'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.useSvnsyncProps'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.followparent'
> > trace: built-in: git 'config' '--get' 'svn.authorsfile'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.useSvmProps'
> > trace: built-in: git 'config' '--bool' '--get' 'svn.uselogauthor'
> > trace: built-in: git 'rev-parse' '--symbolic' '--all'
> > trace: built-in: git 'config' '-l'
> > trace: built-in: git 'config' '-l'
> > trace: built-in: git 'config' '-l'
> > trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.branches-maxRev'
> > trace: built-in: git 'config' '--int' '--get' 'svn-remote.svn.tags-maxRev'
> > trace: built-in: git 'config' '--get' 'svn-remote.svn.url'
> > trace: built-in: git 'config' '--get' 'svn-remote.svn.uuid'
> > trace: built-in: git 'config' 'svn-remote.svn.branches-maxRev' '8'
> > could not lock config file
> > config svn-remote.svn.branches-maxRev 8: command returned error: 255
> > 
> > I suspect that the locking problem is due to some strange anti-virus 
> > interaction, because issuing the same command on the command line 
> > succeeds.
> 
> I believe somebody on the list also had the same problem with file 
> locking in Windows.  Unfortunately, I have little idea as to what could 
> be wrong.  Could Windows not be releasing file locks properly?

I am not half as concerned with locks on Windows (since they occur 
regularly, thanks to _two_ Antiviruses, go figure), as with unnecessary 
config file reading.

> > However, did you notice the many calls to "git config"?  Especially 
> > the three ones which list all values anyway?
> > 
> > I am not really sure if that is the single reason of the slowness -- 
> > remember, Windows is mightily spawn()-challenged -- but it sure would 
> > help to have git-svn read the config once at the beginning, probably 
> > with "-z", too, and then just read from the cached values, no?
> 
> Many months ago, I thought about implementing a transparent caching 
> layer in Git.pm to work with git configs.  Of course, that requires 
> cooperation from all readers/writers within the process...  Done 
> correctly, it would help more than just git-svn. too.

Yes, but I am really scared there.  Last time, Pasky tried to put a _lot_ 
into Git.pm, and it did not work here.  As a consequence, I put aside a 
_lot_ of my time to turn the most important (to me!) perl scripts into 
builtins.

> I think I had this idea around the time we made git-config output Perl 
> hashes and arrays.

You mean -z?

> > P.S.: how far is the svn:external->submodule stuff?
> 
> Yikes.  I've let other work pile up on my ever-growing todo-list :/ I'll 
> see if I can dig it out and wrap it up this weekend or next...

Actually, I can understand what you're talking about.  But just let me 
tell this: I (amongst others) are _excited_ to wait until you have that 
particular feature working...

Ciao,
Dscho