Re: Excessive mmap [was Git server eats all memory]

[Ivan Kanis <expire-by-2010-08-14@kanis.fr>]

Dmitry Potapov <dpotapov@gmail.com> wrote:

> Though Git uses MAP_PRIVATE with mmap, this only marks mapped pages as
> copy-on-write. Because cloning does not change the pack file, all those
> pages should be shared.

I reran the test today. One client is receiving while the other one is
compressing. I have to interrupt both client because the server is
becoming unusable. I am sure you are right about sharing the pages but I
can't test it.

> On 64-bit architecture, you have plenty virtual space, and mapping
> a file to memory should not take much physical memory (only space
> needed for system tables). 

What I can tell from the mmap man page is that it should map memory to a
file. I assume it shouldn't take up physical memory. However I am seeing
physical memory being consumed. It might be a feature of the kernel. Is
there a way to turn it off?

> You can reduce core.packedGitWindowSize in the Git configuration to
> see if it helps, but I doubt that it will have any noticeable effect.

It was worth a shot, it didn't help...

Looking some more into it today the bulk of the memory allocation
happens in write_pack_file in the following loop.

for (; i < nr_objects; i++) {
    if (!write_one(f, objects + i, &offset))
        break;
    display_progress(progress_state, written);
}

This eventually calls write_object, here I am wondering if the
unuse_pack function is doing its job. As far as I can tell it writes a
null in memory, that I think is not enough to reclaim memory.

I also looked at the use_pack function where the mmap is
happening. Would it be worth refactoring this function so that it uses
an index withing a file instead of mmap?

Unless I hear of a better idea I'll be trying that tomorrow...

Take care,
-- 
Ivan Kanis

I can stand brute force, but brute reason is quite unbearable.  There
is something unfair about its use. It is hitting below the intellect.
    -- Oscar Wilde