gitk performance questions/issues

gitk performance questions/issues

[David Tweed]

Since the gitk is being discussed, I'd like to see if I'm
understanding correctly and hitting problems or just
not doing things correctly. (This is using a
self-compiled "git version 1.5.2.2".)

[Caveat: this is on a repository that is evolves using
my own scripts which mimic the git shell scripts in
calling the low-level programs. I'm 99.999% sure it's
generating a fully valid repo, but I might be missing
something. Actual generating script is at
http://www.personal.rdg.ac.uk/~sis05dst/chronoversion.htm
]

1.  With gitk --all -n 256 on my repo it consistently
takes 12s for the window to appear and 21s for the
"I'm working" cursor to change to a normal cursor.
This is on a dual Xeon machine, /proc/cpuinfo
excerpt

model name      :   Intel(R) Xeon(TM) CPU 3.00GHz
stepping        : 3
cpu MHz         : 2992.496
cache size      : 2048 KB

running x86-64 Fedora 7, packed repo has a 12M
.git dir. Is this the expected start-up time for such a
configuration? The output of
time git-rev-list --parents --topo-order --all>/dev/null
is

real    0m0.048s
user    0m0.044s
sys     0m0.005s

2. Startup time isn't an excessive problem. However,
I often don't seem to be getting the "diff with parent"
being displayed once I click on the another commit. If
I click on several in sequence I get the error
message

"error getting diffs: couldn't fork child process: not enough memory"

The diffs are relatively small (consecutive commits
are generated hourly automatically) and a command
line git-diff generates them instantly.

3. Is "gitk --all -n 256" the best way to say "show me
relatively recent stuff" or should I be using different
options for limiting things?

Many thanks for any insight,

-- 
cheers, dave tweed__________________________
david.tweed@gmail.com
Rm 124, School of Systems Engineering, University of Reading.
"we had no idea that when we added templates we were adding a Turing-
complete compile-time language." -- C++ standardisation committee

Re: gitk performance questions/issues

[Linus Torvalds]

On Mon, 13 Aug 2007, David Tweed wrote:
> 
> 1.  With gitk --all -n 256 on my repo it consistently
> takes 12s for the window to appear and 21s for the
> "I'm working" cursor to change to a normal cursor.

There is something seriously wrong there.

For me, on the kernel archive (which is much bigger than yours), the 
window appears immediately, and the "working cursor" takes about two 
seconds to disappear.

> This is on a dual Xeon machine, /proc/cpuinfo
> excerpt
> 
> model name      :   Intel(R) Xeon(TM) CPU 3.00GHz
> stepping        : 3
> cpu MHz         : 2992.496
> cache size      : 2048 KB

Your CPU is fast enough..

> running x86-64 Fedora 7, packed repo has a 12M
> .git dir.

.. and you're apparently fully packed, and do not even have all that big a 
repository.

So:

> Is this the expected start-up time for such a
> configuration?

Absolutely not.

> The output of
>
> time git-rev-list --parents --topo-order --all>/dev/null
>
> is
> 
> real    0m0.048s
> user    0m0.044s
> sys     0m0.005s

Ok, so it's not the revision generation that takes time. 

> 2. Startup time isn't an excessive problem. However,
> I often don't seem to be getting the "diff with parent"
> being displayed once I click on the another commit. If
> I click on several in sequence I get the error
> message
> 
> "error getting diffs: couldn't fork child process: not enough memory"

You have some *serious* problem somewhere. I think your tcl/tk is screwed 
up somehow. 

Alternatively, maybe your repo has some odd structure (do you have tons 
and tons of branches or tags?) and "gitk" ends up having some problem with 
that. What does

	git rev-parse --all | wc -l

say? Various git tools have had performance problems with thousands of 
branches or tags in the past.

One thing to try: I think that "git gc" back in the 1.5.2.2 days didn't 
pack branches by default - and if you have lots and lots of branches, you 
should try to make sure that they are packed, using

	git pack-refs --all --prune

(newer git versions will do this automatically in "git gc" - while 
v1.5.2.2 will only pack the tags by default, iirc).

> The diffs are relatively small (consecutive commits
> are generated hourly automatically) and a command
> line git-diff generates them instantly.

Yeah, it sounds like git itself has no problems, and it's literally gitk 
that shows some strange behaviour.

> 3. Is "gitk --all -n 256" the best way to say "show me
> relatively recent stuff" or should I be using different
> options for limiting things?

That's a perfectly good thing to do. You don't need the "-n", you can 
just do "-256". That said, if you actually want commits to be in date 
order, use

	gitk -d --all -256

instead, which tells gitk to use "--date-order" instead, which is often 
nicer (the graph ends up being a bit messier, but it gives you a more 
correct view of relative dates when you have multiple branches!

		Linus

Re: gitk performance questions/issues

[David Tweed]

On 8/13/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> Alternatively, maybe your repo has some odd structure (do you have tons
> and tons of branches or tags?) and "gitk" ends up having some problem with
> that. What does
>
>         git rev-parse --all | wc -l
>
> say? Various git tools have had performance problems with thousands of
> branches or tags in the past.

Ah, that's it: I've got
$ wc .git/packed-refs
  1915   3832 130178 .git/packed-refs
and 35 currently unpacked tags and temporarily moving that
file away, gitk now starts up virtually instantly. My usage generates
a tag per commit which is probably excessive. For completeness,
$ git rev-parse --all | wc -l
1957

Thinking about it, I suppose even if you're only looking at the last
256 commits you've got to look through all the tags to see whether
or not they refer to something within that window, so it's not
unreasonable for it to affect startup time.

The bit that is a quite surprising to me is that once the window has
finished initialising, clicking on one of the blue dots now (ie
without .git/packed-refs file) now generates the diffs instantly
as well. It's not obvious to me why the existence/non-existence
of tags should affect displaying-diff performance.

Anyway, now I know I can rearrange things not to need anything
like as many tags.

Many thanks for the help,

-- 
cheers, dave tweed__________________________
david.tweed@gmail.com
Rm 124, School of Systems Engineering, University of Reading.
"we had no idea that when we added templates we were adding a Turing-
complete compile-time language." -- C++ standardisation committee

Re: gitk performance questions/issues

[Paul Mackerras]

David Tweed writes:

> Thinking about it, I suppose even if you're only looking at the last
> 256 commits you've got to look through all the tags to see whether
> or not they refer to something within that window, so it's not
> unreasonable for it to affect startup time.

Gitk uses a Tcl associative array for that, which should be very
fast.

Could you try this: go to the Edit->Preferences window and turn off
"Display nearby tags".  Then with your ~2000 refs in place, see how
long gitk takes to start up and to display a diff.

If doing that makes it fast, then I suspect that my topology caching
patch will also help.  That's waiting on an answer from Junio or Linus
to my question about making git rev-list ignore negated non-existent
refs.

Also, if it's possible to give me a copy of your repo, that would
help.

Paul.

Re: gitk performance questions/issues

[David Tweed]

On 8/14/07, Paul Mackerras <paulus@samba.org> wrote:
> David Tweed writes:
> Could you try this: go to the Edit->Preferences window and turn off
> "Display nearby tags".  Then with your ~2000 refs in place, see how
> long gitk takes to start up and to display a diff.

Right, I've restored the file of 1915 packed tags (but not the 35 loose
tags). Using DNT for "display nearby tags"

DNT on: consistently 11s to window appear, 20s to normal cursor
DNT off: consistently 11s to window appear but normal cursor immediately

Regarding getting the diffs, I think that was "user misunderstanding" and
I'm doubting if I definitely did click on the blue circles yesterday. Clicking
on the any part yellow tag marker displays the SHA of the tag object
(which I'd confused with the start of an unfinished commit diff);
for some reason I'd thought it would give the diff-with-parent. Clicking on
either the blue circle or "commit headline text" does bring up the diff
instantly even with the packed refs file. Sorry for the confusion.

> Also, if it's possible to give me a copy of your repo, that would
> help.

I'll privately mail Paul about getting it; if anyone else wants a copy
of the repository
in order look at startup performance issues mail me. (The most embarassing
thing in there is just some astonishingly crap code, but I'd prefer not to
blanket publish it.)

-- 
cheers, dave tweed__________________________
david.tweed@gmail.com
Rm 124, School of Systems Engineering, University of Reading.
"we had no idea that when we added templates we were adding a Turing-
complete compile-time language." -- C++ standardisation committee