git tag triggers out-of-memory killer

git tag triggers out-of-memory killer

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 423 bytes --]

What knobs exist inside git to restrict the amount of memory for each individual git process?

Running a large number of "git tag --sort=taggerdate --contains " processes in parallel triggers the OOM killer because each one allocates more than one gigabyte resident memory.

I tried to set all knobs from git-config(1) that appear to be related to memory usage, but nothing seems to have an effect.


Thanks,
Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: git tag triggers out-of-memory killer

[Jeff King]

On Mon, Jul 11, 2022 at 03:31:35PM +0200, Olaf Hering wrote:

> What knobs exist inside git to restrict the amount of memory for each
> individual git process?

You can set GIT_ALLOC_LIMIT in the environment to limit the size of a
single heap allocation. But there's no overall limit for all
allocations. You'd have to use an OS-level tool like ulimit or cgroups
there.

> Running a large number of "git tag --sort=taggerdate --contains "
> processes in parallel triggers the OOM killer because each one
> allocates more than one gigabyte resident memory.

How did you measure?  Are you sure that each one is allocating a
gigabyte itself, or might some of it be shared between the processes?
Git will mmap the packfiles on disk, which will count against RSS
(assuming the memory even gets faulted in). But multiple processes on
the same repository will share those read-only pages.

Are you running the command against a large number of distinct
repositories? In that case, the pages obviously wouldn't be shared. You
_might_ have some success with core.packedGitLimit, which would lower
the amount that each process will mmap. But in general I'd expect the OS
to happily evict read-only mmap'd pages rather than OOM.

Is there anything about your repository that might be unusual? A large
number of commits, or tags?  If I run "git tag --sort=taggerdate
--contains" against linux.git, measuring with massif shows it uses about
2MB of heap. Digging for a much older commit, like 1da177e4c3, uses
about 16MB (there's some per-commit internal bookkeeping for each commit
we actually have to traverse).

Those are both with commit-graphs enabled (i.e., running "git
commit-graph write --reachable"). Without them, it looks like the heap
is closer to 100MB. Probably git-tag should be disabling
save_commit_buffer internally. Unlike git-log, it won't end up
pretty-printing the commits it walks during the --contains traversal.

So maybe try with this patch:

diff --git a/builtin/tag.c b/builtin/tag.c
index 75dece0e4f..8f3a6dffb4 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -486,6 +486,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
 	int ret = 0;
 	const char *only_in_list = NULL;
 
+	save_commit_buffer = 0;
+
 	setup_ref_filter_porcelain_msg();
 
 	git_config(git_tag_config, &sorting_options);

or with commit graphs written, and see if that improves things for you.
Those numbers aren't anywhere near the 1GB you mentioned, but perhaps a
sufficient number of 100MB heaps is enough to cause problems for you.

-Peff

Re: git tag triggers out-of-memory killer

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 875 bytes --]

Mon, 11 Jul 2022 10:06:32 -0400 Jeff King <peff@peff.net>:

> How did you measure?  Are you sure that each one is allocating a
> gigabyte itself, or might some of it be shared between the processes?
> Git will mmap the packfiles on disk, which will count against RSS
> (assuming the memory even gets faulted in). But multiple processes on
> the same repository will share those read-only pages.


I ran top(1), all the git processes were competing for memory.
There was most of the time no memory shared, according to top.

This tool to process a single repository exists since a few years.
In the past I ran it on a 12cpu/64GB machine. Today it was running the
first time on a 96cpu/64GB machine. There is a slim chance the issue did
not show up because the 12 processes always had access to enough memory.


I will try your suggestions.

Thanks.
Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: git tag triggers out-of-memory killer

[Jeff King]

On Mon, Jul 11, 2022 at 05:15:37PM +0200, Olaf Hering wrote:

> I ran top(1), all the git processes were competing for memory.
> There was most of the time no memory shared, according to top.

OK, that makes sense if they really are allocating a lot of heap. They
_would_ be sharing mmap'd pages, but the memory pressure causes the OS
to evict as many of those as possible.

> This tool to process a single repository exists since a few years.
> In the past I ran it on a 12cpu/64GB machine. Today it was running the
> first time on a 96cpu/64GB machine. There is a slim chance the issue did
> not show up because the 12 processes always had access to enough memory.
> 
> I will try your suggestions.

In case you haven't seen it, read the end of:

  https://lore.kernel.org/git/YsxiSwQGvLhzNQrt@coredump.intra.peff.net/

I realized my reproduction on linux.git was not traversing a wide enough
chunk of history. Fixing that, I do see ~1GB of heap allocation. My
patch drops that substantially, but you will still be much better off
building a commit-graph file.

-Peff

Re: git tag triggers out-of-memory killer

[Olaf Hering]

[-- Attachment #1: Type: text/plain, Size: 680 bytes --]

Mon, 11 Jul 2022 10:06:32 -0400 Jeff King <peff@peff.net>:

> --- a/builtin/tag.c
> +++ b/builtin/tag.c
> @@ -486,6 +486,8 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
>  	int ret = 0;
>  	const char *only_in_list = NULL;
>  
> +	save_commit_buffer = 0;
> +
>  	setup_ref_filter_porcelain_msg();
>  
>  	git_config(git_tag_config, &sorting_options);
> 


Thanks, this patch helps.

The individual git processes start with a maximum of ~320M, and slowly reach around ~600M as maximum.

I was not aware of commit-graph. After writing a commit-graph the processing is much faster, and an unmodified git 2.35.3 uses ~320M as a maximum.

Olaf

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 833 bytes --]