Git exhausts memory.

[Alif Wahid <alif.wahid@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1471 bytes --]

Hi there,

I'm using Git v1.7.1 on Ubuntu v10.10 and unfortunately Git seems to
regularly exhaust the memory on my machine and fails to compress loose
objects and/or collect garbage.

My Intel based dual-core machine has 2 GB of RAM and 4 GB of swap
space. I need to track a working tree with a handful of really large
tarballs that rarely change and loads of really small text files that
change frequently. What I'm seeing is that over time whenever "git gc"
runs automatically it fails with the message "fatal: Out of memory,
malloc failed". So I've been trying to manually run "git repack -ad
--window-memory=1g --max-pack-size=1g" in the hope that Git will not
exceed the physical memory. But I still get the same error message :(

As I can't make my repository public, I've attached a simple Python
script that generates a ~1.3 GB file containing random integers (takes
roughly 10 min. on my machine). Then I run the following four commands
and get the out-of-memory failure from "git repack". This is
effectively emulating the scenario I have with my repository.

$ git init
$ git add ./test_data.dat
$ git commit ./test_data.dat -m "Test data."
$ git repack -ad --window-memory=1g --max-pack-size=1g
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
fatal: Out of memory, malloc failed

I can't find anything on the wiki about out-of-memory failures. Any
info/help would be much appreciated.

Regards

Alif

[-- Attachment #2: test_data.py --]
[-- Type: text/x-python, Size: 1048 bytes --]

#! /usr/bin/env python

import sys, os, random

if __name__ == '__main__':

    fbuff = 2**28   # file buffer size in bytes
    fints = 10**8   # number of integers to write
    fsize = 12      # number of digits to write per integer
    
    fname = os.path.splitext(os.path.basename(sys.argv[0]))[0]
    try:
        os.mkdir(fname)
    except OSError:
        pass    # directory already exists
        
    fpath = os.path.join(fname, fname+'.dat')
    fhand = open(fpath, 'w', fbuff)
    
    sys.stdout.write('Writing %d MB to file \'%s\'\n' % (fints*(fsize+1)/10**6, fpath))
    sys.stdout.write('This will take some time, please wait.\n')
    sys.stdout.write('Progress:')
    sys.stdout.flush()

    random.seed(-1)
    fform = '%'+str(fsize)+'d\n'
    for x in xrange(fints): 
        fhand.write(fform % (random.randint(0,fints)))
        if x % (fints/10) == 0 and x > 0:
            sys.stdout.write('%3d/10' % (x/(fints/10))) 
            sys.stdout.flush()

    sys.stdout.write(' 10/10\n')
    sys.stdout.flush()
    fhand.close()