[PATCH 0/3] gitk: memory consumption improvements

[PATCH 0/3] gitk: memory consumption improvements

[Markus Hitter]

List, Paul,

after searching for a while on why Gitk sometimes consumes exorbitant amounts of memory I found a pair of minor issues and also a big one: the text widget comes with an unlimited undo manager, which is turned on be default. Considering that each line is inserted seperately, this piles up a huuuge undo stack ... for a read-only text widget. Simply turning off this undo manager saves about 95% of memory when viewing large commits (with tens of thousands of diff lines).

3 patches are about to follow:

 - turn off the undo manager,

 - forget already closed file descriptors and

 - forget the 'commitinfo' array on a reload to enforce reloading it.

I hope this finds you appreciation.


Markus

-- 
- - - - - - - - - - - - - - - - - - -
Dipl. Ing. (FH) Markus Hitter
http://www.jump-ing.de/

[PATCH 1/3] gitk: turn off undo manager in the text widget

[Markus Hitter]

From e965e1deb9747bbc2b40dc2de95afb65aee9f7fd Mon Sep 17 00:00:00 2001
From: Markus Hitter <mah@jump-ing.de>
Date: Sun, 6 Nov 2016 20:38:03 +0100
Subject: [PATCH 1/3] gitk: turn off undo manager in the text widget

The diff text widget is read-only, so there's zero point in
building an undo stack. This change reduces memory consumption of
this widget by about 95%.

Memory usage of the whole program for viewing a reference commit
before; 579'692'744 bytes, after: 32'724'446 bytes.

Test procedure:

 - Choose a largish commit and check it out. In this case one with
   90'802 lines, 5'006'902 bytes.

 - Have a Tcl version with memory debugging enabled. This is,
   build one with --enable-symbols=mem passed to configure.

 - Instrument Gitk to regularly show a memory dump. E.g. by adding
   these code lines at the very bottom:

     proc memDump {} {
         catch {
             set output [memory info]
             puts $output
         }

         after 3000 memDump
     }

     memDump

 - Start Gitk, it'll load this largish commit into the diff text
   field automatically (because it's the current commit).

 - Wait until memory consumption levels out and note the numbers.

Note that the numbers reported by [memory info] are much smaller
than the ones reported in 'top' (1.75 GB vs. 105 MB in this case),
likely due to all the instrumentation coming with the debug
version of Tcl.

Signed-off-by: Markus Hitter <mah@jump-ing.de>
---
 gitk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gitk b/gitk
index 805a1c7..8654e29 100755
--- a/gitk
+++ b/gitk
@@ -2403,7 +2403,7 @@ proc makewindow {} {
 
     set ctext .bleft.bottom.ctext
     text $ctext -background $bgcolor -foreground $fgcolor \
-	-state disabled -font textfont \
+	-state disabled -undo 0 -font textfont \
 	-yscrollcommand scrolltext -wrap none \
 	-xscrollcommand ".bleft.bottom.sbhorizontal set"
     if {$have_tk85} {
-- 
2.9.3

[PATCH 2/3] gitk: remove closed file descriptors from $blobdifffd

[Markus Hitter]

From 0a463fcd977dc9558835c373e24a095e35ca3c82 Mon Sep 17 00:00:00 2001
From: Markus Hitter <mah@jump-ing.de>
Date: Mon, 7 Nov 2016 16:01:17 +0100
Subject: [PATCH 2/3] gitk: remove closed file descriptors from $blobdifffd

One shouldn't have descriptors of already closed files around.

The first idea to deal with this (previously) ever growing array
was to remove it entirely, but it's needed to detect start of a
new diff with ths old diff not yet done. This happens when a user
clicks on the same commit in the commit list repeatedly without
delay.

Signed-off-by: Markus Hitter <mah@jump-ing.de>
---
 gitk | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/gitk b/gitk
index 8654e29..518a4ce 100755
--- a/gitk
+++ b/gitk
@@ -8069,7 +8069,11 @@ proc getblobdiffline {bdf ids} {
     $ctext conf -state normal
     while {[incr nr] <= 1000 && [gets $bdf line] >= 0} {
 	if {$ids != $diffids || $bdf != $blobdifffd($ids)} {
+	    # Older diff read. Abort it.
 	    catch {close $bdf}
+	    if {$ids != $diffids} {
+		array unset blobdifffd $ids
+	    }
 	    return 0
 	}
 	parseblobdiffline $ids $line
@@ -8078,6 +8082,7 @@ proc getblobdiffline {bdf ids} {
     blobdiffmaybeseehere [eof $bdf]
     if {[eof $bdf]} {
 	catch {close $bdf}
+	array unset blobdifffd $ids
 	return 0
     }
     return [expr {$nr >= 1000? 2: 1}]
-- 
2.9.3

[PATCH 3/3] gitk: clear array 'commitinfo' on reload

[Markus Hitter]

From 8359452f426c68cc02250f25f20eaaacd2ddd001 Mon Sep 17 00:00:00 2001
From: Markus Hitter <mah@jump-ing.de>
Date: Mon, 7 Nov 2016 19:02:51 +0100
Subject: [PATCH 3/3] gitk: clear array 'commitinfo' on reload

After a reload we might have an entirely different set of commits,
so keeping all of them leaks memory. Remove them all because
re-creating them is not more expensive than testing wether they're
still valid. Lazy (re-)creation is already well established, so
a missing entry can't cause harm.

Signed-off-by: Markus Hitter <mah@jump-ing.de>
---
 gitk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/gitk b/gitk
index 518a4ce..aef6db6 100755
--- a/gitk
+++ b/gitk
@@ -588,7 +588,7 @@ proc updatecommits {} {
 proc reloadcommits {} {
     global curview viewcomplete selectedline currentid thickerline
     global showneartags treediffs commitinterest cached_commitrow
-    global targetid
+    global targetid commitinfo
 
     set selid {}
     if {$selectedline ne {}} {
@@ -609,6 +609,7 @@ proc reloadcommits {} {
 	getallcommits
     }
     clear_display
+    unset -nocomplain commitinfo
     unset -nocomplain commitinterest
     unset -nocomplain cached_commitrow
     unset -nocomplain targetid
-- 
2.9.3

Re: [PATCH 1/3] gitk: turn off undo manager in the text widget

[Jacob Keller]

On Mon, Nov 7, 2016 at 10:57 AM, Markus Hitter <mah@jump-ing.de> wrote:
> From e965e1deb9747bbc2b40dc2de95afb65aee9f7fd Mon Sep 17 00:00:00 2001
> From: Markus Hitter <mah@jump-ing.de>
> Date: Sun, 6 Nov 2016 20:38:03 +0100
> Subject: [PATCH 1/3] gitk: turn off undo manager in the text widget
>
> The diff text widget is read-only, so there's zero point in
> building an undo stack. This change reduces memory consumption of
> this widget by about 95%.
>
> Memory usage of the whole program for viewing a reference commit
> before; 579'692'744 bytes, after: 32'724'446 bytes.
>

Wow. Nice find!

> Test procedure:
>
>  - Choose a largish commit and check it out. In this case one with
>    90'802 lines, 5'006'902 bytes.
>
>  - Have a Tcl version with memory debugging enabled. This is,
>    build one with --enable-symbols=mem passed to configure.
>
>  - Instrument Gitk to regularly show a memory dump. E.g. by adding
>    these code lines at the very bottom:
>
>      proc memDump {} {
>          catch {
>              set output [memory info]
>              puts $output
>          }
>
>          after 3000 memDump
>      }
>
>      memDump
>
>  - Start Gitk, it'll load this largish commit into the diff text
>    field automatically (because it's the current commit).
>
>  - Wait until memory consumption levels out and note the numbers.
>
> Note that the numbers reported by [memory info] are much smaller
> than the ones reported in 'top' (1.75 GB vs. 105 MB in this case),
> likely due to all the instrumentation coming with the debug
> version of Tcl.
>

Still, this is definitely the lions share of the memory issue.
Additionally, this fix seems much better overall and does not harm any
other aspects of gitk, because we only read the widget so there is as
you mentioned, zero reason to build an undo stack.

Thanks for taking the extra time to find a proper solution to this! I
think it makes perfect sense.

> Signed-off-by: Markus Hitter <mah@jump-ing.de>
> ---
>  gitk | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/gitk b/gitk
> index 805a1c7..8654e29 100755
> --- a/gitk
> +++ b/gitk
> @@ -2403,7 +2403,7 @@ proc makewindow {} {
>
>      set ctext .bleft.bottom.ctext
>      text $ctext -background $bgcolor -foreground $fgcolor \
> -       -state disabled -font textfont \
> +       -state disabled -undo 0 -font textfont \
>         -yscrollcommand scrolltext -wrap none \
>         -xscrollcommand ".bleft.bottom.sbhorizontal set"
>      if {$have_tk85} {
> --
> 2.9.3
>

Nice that such a simple change results in a huge gain. I think this
makes perfect sense.

Regards,
Jake

Re: [PATCH 0/3] gitk: memory consumption improvements

[Junio C Hamano]

Markus Hitter <mah@jump-ing.de> writes:

> List, Paul,
>
> after searching for a while on why Gitk sometimes consumes
> exorbitant amounts of memory I found a pair of minor issues and
> also a big one: the text widget comes with an unlimited undo
> manager, which is turned on be default. Considering that each line
> is inserted seperately, this piles up a huuuge undo stack ... for
> a read-only text widget. Simply turning off this undo manager
> saves about 95% of memory when viewing large commits (with tens of
> thousands of diff lines).

You made me laugh crazy hard while in a waiting room in a clinic
yesterday with the cover letter; people around gave me a strange
look but I couldn't help.

This is a single liner with the largest gain in the history of this
project.  Very well spotted.

Are all semi-modern Tcl/Tk in service have this -undo thing so that
we can pass unconditionally to the text widget like the patch does?

If we had to do this conditionally, that robs the fun of "just
adding 8 bytes to the source to reduce 1+GB memory consumption", but
even if we had to go conditional, this is a great find.

Well done.

Re: [PATCH 0/3] gitk: memory consumption improvements

[Markus Hitter]

Am 08.11.2016 um 22:37 schrieb Junio C Hamano:
> Are all semi-modern Tcl/Tk in service have this -undo thing so that
> we can pass unconditionally to the text widget like the patch does?

Good point. As far as my research goes, this flag was introduced in Nov. 2001:

http://core.tcl.tk/tk/info/5265df93d207cec0


To defend Gitk developers, the Tk guys apparently change their mind on the default value from time to time. Official documentation says nothing about a default, the proposal from 2001 talks about -undo 0 as default and there are recent commits changing this default:

http://core.tcl.tk/tk/info/549d2f56757408f3


Markus

-- 
- - - - - - - - - - - - - - - - - - -
Dipl. Ing. (FH) Markus Hitter
http://www.jump-ing.de/

Re: [PATCH 0/3] gitk: memory consumption improvements

[Junio C Hamano]

Markus Hitter <mah@jump-ing.de> writes:

> Am 08.11.2016 um 22:37 schrieb Junio C Hamano:
>> Are all semi-modern Tcl/Tk in service have this -undo thing so that
>> we can pass unconditionally to the text widget like the patch does?
>
> Good point. As far as my research goes, this flag was introduced in Nov. 2001:
>
> http://core.tcl.tk/tk/info/5265df93d207cec0

Sounds safe enough then ;-)

Thanks for an additional research.

Re: [PATCH 0/3] gitk: memory consumption improvements

[Paul Mackerras]

On Mon, Nov 07, 2016 at 07:54:28PM +0100, Markus Hitter wrote:
> 
> List, Paul,
> 
> after searching for a while on why Gitk sometimes consumes exorbitant amounts of memory I found a pair of minor issues and also a big one: the text widget comes with an unlimited undo manager, which is turned on be default. Considering that each line is inserted seperately, this piles up a huuuge undo stack ... for a read-only text widget. Simply turning off this undo manager saves about 95% of memory when viewing large commits (with tens of thousands of diff lines).
> 
> 3 patches are about to follow:
> 
>  - turn off the undo manager,
> 
>  - forget already closed file descriptors and
> 
>  - forget the 'commitinfo' array on a reload to enforce reloading it.
> 
> I hope this finds you appreciation.

Thanks for the good work in tracking this down and making the patches.
I have applied the series.  Apologies for slow response (life has been
extremely busy for me this year).

Paul.