[PATCH] Correct pack memory leak causing git gc to try to exceed ulimit

[PATCH] Correct pack memory leak causing git gc to try to exceed ulimit

[Shawn O. Pearce]

When recursing to unpack a delta base we must unuse_pack() so that
the pack window for the current object does not remain pinned in
memory while the delta base is itself being unpacked and materialized
for our use.

On a long delta chain of 50 objects we may need to access 6 different
windows from a very large (>3G) pack file in order to obtain all
of the delta base content.  If the process ulimit permits us to
map/allocate only 1.5G we must release windows during this recursion
to ensure we stay within the ulimit and transition memory from pack
cache to standard malloc, or other mmap needs.

Inserting an unuse_pack() call prior to the recursion allows us to
avoid pinning the current window, making it available for garbage
collection if memory runs low.

This has been broken since at least before 1.5.1-rc1, and very
likely earlier than that.  Its fixed now.  :)

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 sha1_file.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index d1c4060..10346b6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1609,6 +1609,7 @@ static void *unpack_delta_entry(struct packed_git *p,
 	off_t base_offset;
 
 	base_offset = get_delta_base(p, w_curs, &curpos, *type, obj_offset);
+	unuse_pack(w_curs);
 	base = cache_or_unpack_entry(p, base_offset, &base_size, type, 0);
 	if (!base)
 		die("failed to read delta base object"
-- 
1.5.6.74.g8a5e

Re: [PATCH] Correct pack memory leak causing git gc to try to exceed ulimit

[Nicolas Pitre]

On Wed, 9 Jul 2008, Shawn O. Pearce wrote:

> When recursing to unpack a delta base we must unuse_pack() so that
> the pack window for the current object does not remain pinned in
> memory while the delta base is itself being unpacked and materialized
> for our use.
> 
> On a long delta chain of 50 objects we may need to access 6 different
> windows from a very large (>3G) pack file in order to obtain all
> of the delta base content.  If the process ulimit permits us to
> map/allocate only 1.5G we must release windows during this recursion
> to ensure we stay within the ulimit and transition memory from pack
> cache to standard malloc, or other mmap needs.
> 
> Inserting an unuse_pack() call prior to the recursion allows us to
> avoid pinning the current window, making it available for garbage
> collection if memory runs low.
> 
> This has been broken since at least before 1.5.1-rc1, and very
> likely earlier than that.  Its fixed now.  :)

Well well...  I suspect this might have been the cause of our strange 
out-of-memory issue when trying to repack the gcc repository a while 
ago.

I updated my copy of git://git.infradead.org/gcc.git and re-attempted a 
full repack with window=500 and depth=500, and this time it worked for 
me on a 32-bit machine!  Took about 6h30 single-threaded, and produced a 
mere 331MB pack file containing 1254664 objects across 142328 commits.


Nicolas