Re: Git server eats all memory

[Ivan Kanis <expire-by-2010-08-11@kanis.fr>]

Nguyen Thai Ngoc Duy <pclouds@gmail.com> wrote:

> Naah, git pack-objects needs list of commit tips. Try
> git for-each-ref|cut -c 1-40|git pack-objects --all --stdout > /dev/null

Jared Hance <jaredhance@gmail.com> wrote:

> I would look in the code for malloc calls that don't have a free call,
> or spots where free calls might not be hit.

Hello Jared and Nguyen,

Thank you Nguyen for your command. I can now reproduce the problem
without needing the network. I have been following Jared lead today on
a potential memory leak. Here is what I found out.

I downloaded the latest release of git 1.7.2.1 and compiled it with
debugging support. I ran valgrind on the command and found two memory
leaks. I put the output at the bottom of the e-mail as it's not very
interesting. I patched one of the leak in pack_objects.c but got the
same problem: over 4G of memory consumption for a 4G repository.

I've come to the conclusion that it's not a memory leak. 

This afternoon I put macro around the following functions: xmalloc
xmallocz, xrealloc, xcalloc and xmmap. It reported the line of code and
size passed in each functions. I then run the result through a script
that totaled the amount used by each bit of code.

Here are the top 3 consumers:

| function | source                     | size in M |
|----------+----------------------------+-----------|
| xrealloc | builtin/pack-objects.c:690 |        86 |
| xmallocz | patch-delta.c:36           |       301 |
| xmmap    | sha1_file.c:772            |      4393 |

I expected the malloc to take 4G but was surprised it didn't. It seems
to be mmap taking all the memory. I am not familiar with that function,
it looks like it's mapping memory to a file... Is it reasonable to mmap
so much memory?

Today I chatted with someone on freenode #git and he reported the same
problem on his 2G repository, I am glad I am not the only one seeing
this ;)

I tried reading the code but it's going over my head. I'll look at is
some more next monday.

If anyone is familiar with the code source of git I would love to have
some insight into this.

Take care,

Ivan Kanis

PS: output of valgrind --leak-check=full

65 bytes in 1 blocks are definitely lost in loss record 4 of 7
   at 0x4C2260E: malloc (vg_replace_malloc.c:207)
   by 0x4C22797: realloc (vg_replace_malloc.c:429)
   by 0x4C600D: xrealloc (wrapper.c:80)
   by 0x4B7939: strbuf_grow (strbuf.c:70)
   by 0x4B80BA: strbuf_addf (strbuf.c:201)
   by 0x4832EF: system_path (exec_cmd.c:37)
   by 0x483411: setup_path (exec_cmd.c:104)
   by 0x404AF2: main (git.c:536)

512 bytes in 1 blocks are definitely lost in loss record 5 of 8
   at 0x4C203E4: calloc (vg_replace_malloc.c:397)
   by 0x4C5F9D: xcalloc (wrapper.c:96)
   by 0x445741: cmd_pack_objects (pack-objects.c:2117)
   by 0x4048EE: handle_internal_command (git.c:270)
   by 0x404B03: main (git.c:470)
-- 
http://kanis.fr

Everything should be made as simple as possible, but not simpler.
    -- Albert Einstein