Thoughts on gitk's memory footprint over linux-2.6.git

Thoughts on gitk's memory footprint over linux-2.6.git

[Martin Langhoff]

In an odd turn of affairs, I am managing a bit of kernel development,
and even writing the odd patch myself. As usual, gitk is excellent to
visualize what's new in the tree when developers in the team commit
and push new stuff.

However, I find it extremely annoying over the kernel tree, due to its
memory footprint. It is not the only thing I am running, (Chrome
Browser, Gnome3, Firefox, many gnome Terminal windows, emacs), and
given that I am looking at "just a couple of commits" I don't feel
opening a few gitk instances should be problematic... except that it
is.

Each gitk instance wants to gobble up >1GB of memory! It really wants
to show me the "Linux-2.6.12-rc2(...)Initial git repository build."
commit.

Aren't kernel hackers hurting with this? This box blazes through a
kernel compile, but only has 2GB RAM. Regardless of amount of RAM,
firing up gitk gobbles up >1G and untold CPU cycles.

My tk isn't really up to par, but... here's a stupid idea.

 - In start_rev_list, run git log into a tempfile, and concurrently
open the same file for reading, as fd

 - Read in a bound number of commits, say 10000 ("just" 200MB)

 - If we haven't made it to the end of the file, remember the offset
and keep reading the file, counting matches of /^commit/ to get a
commit count and size the scroll back

 - re-activate the reading as required...

There are of course tricky aspects of concurrent programming at hand.
I am not sure what concurrency model gitk has now, but it does a great
job of not blocking and not getting confused.

Tricky Tk programming, or go buy some RAM? Yeah, I know.

Now why didn't I pack that Tk book when I moved here?




m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: Thoughts on gitk's memory footprint over linux-2.6.git

[Elijah Newren]

On Mon, Sep 26, 2011 at 1:38 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> In an odd turn of affairs, I am managing a bit of kernel development,
> and even writing the odd patch myself. As usual, gitk is excellent to
> visualize what's new in the tree when developers in the team commit
> and push new stuff.
>
> However, I find it extremely annoying over the kernel tree, due to its
> memory footprint. It is not the only thing I am running, (Chrome
> Browser, Gnome3, Firefox, many gnome Terminal windows, emacs), and
> given that I am looking at "just a couple of commits" I don't feel
> opening a few gitk instances should be problematic... except that it
> is.

If you only want to look at a couple commits, you could tell gitk that:
  gitk -N
where N is some small integer -- or even 10000, as you suggest.  Or
use other rev-list arguments, such as
  gitk v2.6.38..master

gitk and 'git log' accept basically the same arguments, so look at the
help for git log if you want to find other example ways to slice and
dice what pieces of history to view.


Hope that helps,
Elijah

Re: Thoughts on gitk's memory footprint over linux-2.6.git

[Martin Langhoff]

On Mon, Sep 26, 2011 at 6:02 PM, Elijah Newren <newren@gmail.com> wrote:
> If you only want to look at a couple commits, you could tell gitk that:
>  gitk -N

I know I can -- I've done my bit of git hacking but it was long ago so
if you use your -N parameter you won't see it ;-)

I argue that -n 10000 should be the default -- most of the time you
open git to see recent commits, not that 2.6.12-rc3 commit. Of course,
it cannot actually be the default, because currently there's no way to
scroll past that limiter.

It's a hard limit, and that's not useful.

Now, git internally is very smart about using sliding windows over
large datasets. The porcelain tools don't try to be so smart, but
we're at a point where, as a user, I really wish gitk had some of that
magic.

cheers,



m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- Software Architect - OLPC
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: Thoughts on gitk's memory footprint over linux-2.6.git

[Felipe Contreras]

On Mon, Sep 26, 2011 at 10:38 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> However, I find it extremely annoying over the kernel tree, due to its
> memory footprint. It is not the only thing I am running, (Chrome
> Browser, Gnome3, Firefox, many gnome Terminal windows, emacs), and
> given that I am looking at "just a couple of commits" I don't feel
> opening a few gitk instances should be problematic... except that it
> is.

Sometimes I do this:
 % gitk master..branch_1 master..branch_2 ...

But as I visualize more branches, it becomes tedious.

It would be nice to have --base option, and show only the commits
<base>..<branch>.

Cheers.

-- 
Felipe Contreras

Re: Thoughts on gitk's memory footprint over linux-2.6.git

[Johannes Sixt]

Am 11/11/2011 13:44, schrieb Felipe Contreras:
> On Mon, Sep 26, 2011 at 10:38 PM, Martin Langhoff
> <martin.langhoff@gmail.com> wrote:
>> However, I find it extremely annoying over the kernel tree, due to its
>> memory footprint. It is not the only thing I am running, (Chrome
>> Browser, Gnome3, Firefox, many gnome Terminal windows, emacs), and
>> given that I am looking at "just a couple of commits" I don't feel
>> opening a few gitk instances should be problematic... except that it
>> is.
> 
> Sometimes I do this:
>  % gitk master..branch_1 master..branch_2 ...
> 
> But as I visualize more branches, it becomes tedious.
> 
> It would be nice to have --base option, and show only the commits
> <base>..<branch>.

What's wrong with

     gitk master..branch_1 branch_2 branch_3 branch_4 branch_5
or
     gitk --branches --not master

? (I do that all the time.) Recall that 'master..branch_1' is short for
'^master branch_1'. It is sufficient to specify the negative ref, ^master,
only once.

-- Hannes

Re: Thoughts on gitk's memory footprint over linux-2.6.git

[Felipe Contreras]

On Fri, Nov 11, 2011 at 4:19 PM, Johannes Sixt <j.sixt@viscovery.net> wrote:
> Am 11/11/2011 13:44, schrieb Felipe Contreras:
>> It would be nice to have --base option, and show only the commits
>> <base>..<branch>.
>
> What's wrong with
>
>     gitk master..branch_1 branch_2 branch_3 branch_4 branch_5
> or
>     gitk --branches --not master
>
> ? (I do that all the time.) Recall that 'master..branch_1' is short for
> '^master branch_1'. It is sufficient to specify the negative ref, ^master,
> only once.

Ah, nice! Thanks a lot :)

-- 
Felipe Contreras