git checkout crashes after server being updated to Debian X86_64

git checkout crashes after server being updated to Debian X86_64

[Raffael Reichelt]

Hello! 

I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.

This are the error-messages:

fatal: Out of memory, realloc failed
fatal: recursion detected in die handler
fatal: recursion detected in die handler

or
fatal: unable to create threaded lstat
fatal: recursion detected in die handler
or
fatal: unable to create threaded lstat
*** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***
fatal: recursion detected in die handler
Aborted

It’s obviously not a problem of the repository - happens with all of them. I think it is also not a question of size - happens with a 80M Repository as well as with a 500M one.

Any way: did a 

git fsck
Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.
Prüfe Objekte: 100% (56305/56305), Fertig.

git gc --auto --prune=today —aggressive
git repack

Additionally I played around some config parameters  so my config now looks like:
[http]
    postbuffer = 524288000
[pack]
    threads = 1
    deltaCacheSize = 128m
    packSizeLimit = 128m
    windowMemory = 128m
[core]
    packedGitLimit = 128m
    packedGitWindowSize = 128m
    repositoryformatversion = 0
    filemode = true
    bare = true

I am running 
git version 2.1.4
 
on
Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux

Anyone out there to help me getting out of this trouble?

Regards,
Raffael

Re: git checkout crashes after server being updated to Debian X86_64

[René Scharfe]

Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:
> Hello!
>
> I have a serious problem with git, After my provider had updated to a
> X86_64 architecture git crashes with various memory-related errors.
> This is happening remote when pushing to the repository from my local
> machine as well as trying it on a shell on the server itself.
>
> This are the error-messages:
>
> fatal: Out of memory, realloc failed
> fatal: recursion detected in die handler
> fatal: recursion detected in die handler
>
> or
> fatal: unable to create threaded lstat
> fatal: recursion detected in die handler
> or
> fatal: unable to create threaded lstat
> *** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***
> fatal: recursion detected in die handler
> Aborted
>
> It’s obviously not a problem of the repository - happens with all of
> them. I think it is also not a question of size - happens with a 80M
> Repository as well as with a 500M one.
>
> Any way: did a
>
> git fsck
> Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.
> Prüfe Objekte: 100% (56305/56305), Fertig.
>
> git gc --auto --prune=today —aggressive
> git repack
>
> Additionally I played around some config parameters  so my config now looks like:
> [http]
>     postbuffer = 524288000
> [pack]
>     threads = 1
>     deltaCacheSize = 128m
>     packSizeLimit = 128m
>     windowMemory = 128m
> [core]
>     packedGitLimit = 128m
>     packedGitWindowSize = 128m
>     repositoryformatversion = 0
>     filemode = true
>     bare = true
>
> I am running
> git version 2.1.4
>
> on
> Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux
>
> Anyone out there to help me getting out of this trouble?

Git 2.1.4 is the version that comes with Debian stable according to 
https://packages.debian.org/jessie/git, so I guess using a more recent 
version is not a reasonable option.

What do "file $(which git)" and "ulimit -a" return?  Do you have an 
x86-64 binary and no unnecessarily low limits set?

René

Re: git checkout crashes after server being updated to Debian X86_64

[Raffael Reichelt]

Hello Renè!
file is returning 

/usr/bin/git: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32,  BuildID[sha1]=ee62e538d6fe6673d3ba49f0e66bfec784cc32bc, stripped

and ulimit is:
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 1
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 512
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) 1800
max user processes              (-u) 42
virtual memory          (kbytes, -v) 786432
file locks                      (-x) unlimited

Support told me git is limited to 600M

Regrads,
Rafael

> Am 18.10.2016 um 18:42 schrieb René Scharfe <l.s.r@web.de>:
> 
> Am 18.10.2016 um 17:17 schrieb Raffael Reichelt:
>> Hello!
>> 
>> I have a serious problem with git, After my provider had updated to a
>> X86_64 architecture git crashes with various memory-related errors.
>> This is happening remote when pushing to the repository from my local
>> machine as well as trying it on a shell on the server itself.
>> 
>> This are the error-messages:
>> 
>> fatal: Out of memory, realloc failed
>> fatal: recursion detected in die handler
>> fatal: recursion detected in die handler
>> 
>> or
>> fatal: unable to create threaded lstat
>> fatal: recursion detected in die handler
>> or
>> fatal: unable to create threaded lstat
>> *** Error in `git': double free or corruption (fasttop): 0x0000000000a8ade0 ***
>> fatal: recursion detected in die handler
>> Aborted
>> 
>> It’s obviously not a problem of the repository - happens with all of
>> them. I think it is also not a question of size - happens with a 80M
>> Repository as well as with a 500M one.
>> 
>> Any way: did a
>> 
>> git fsck
>> Prüfe Objekt-Verzeichnisse: 100% (256/256), Fertig.
>> Prüfe Objekte: 100% (56305/56305), Fertig.
>> 
>> git gc --auto --prune=today —aggressive
>> git repack
>> 
>> Additionally I played around some config parameters  so my config now looks like:
>> [http]
>>    postbuffer = 524288000
>> [pack]
>>    threads = 1
>>    deltaCacheSize = 128m
>>    packSizeLimit = 128m
>>    windowMemory = 128m
>> [core]
>>    packedGitLimit = 128m
>>    packedGitWindowSize = 128m
>>    repositoryformatversion = 0
>>    filemode = true
>>    bare = true
>> 
>> I am running
>> git version 2.1.4
>> 
>> on
>> Linux infongp-de65 3.14.0-ui16196-uiabi1-infong-amd64 #1 SMP Debian 3.14.73-2~ui80+4 (2016-07-13) x86_64 GNU/Linux
>> 
>> Anyone out there to help me getting out of this trouble?
> 
> Git 2.1.4 is the version that comes with Debian stable according to https://packages.debian.org/jessie/git, so I guess using a more recent version is not a reasonable option.
> 
> What do "file $(which git)" and "ulimit -a" return?  Do you have an x86-64 binary and no unnecessarily low limits set?
> 
> René

Re: git checkout crashes after server being updated to Debian X86_64

[Duy Nguyen]

On Tue, Oct 18, 2016 at 10:17 PM, Raffael Reichelt
<raffael.reichelt@gmail.com> wrote:
> Hello!
>
> I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.
>
> This are the error-messages:
>
> fatal: Out of memory, realloc failed
> fatal: recursion detected in die handler
> fatal: recursion detected in die handler

You other mail said memory is capped at 600MB, which should be a lot
for normal repositories. If you set the environment variable
GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes
first) and git attempts to allocate more than that (just that one
time, not total mem) then it's caught and we get a glimpse of how much
memory git may need. Unfortunately we can't get a stack trace or
anything like that unless you rebuild Git from source.

> or
> fatal: unable to create threaded lstat
> fatal: recursion detected in die handler

Hmm.. with "max user processes (-u) 42" we should be fine because we
only create 20 threads max. What happens if you set core.preloadindex
to false? Can it run until the end or hit some other fatal errors?

There's room for improvement in preload_index(). If we hit resource
limit like this, it's not the end of the world and we should be able
to keep going. But threaded lstat has been available for a long time
and this is the first time I see a report like this, not sure if it's
worth fixing.
-- 
Duy

[PATCH] Add a knob to abort on die() (was Re: git checkout crashes after ...)

[Duy Nguyen]

On Wed, Oct 19, 2016 at 08:27:43PM +0700, Duy Nguyen wrote:
> If you set the environment variable GIT_ALLOC_LIMIT ...  git
> attempts to allocate more than that ... then it's caught and we get
> a glimpse of how much memory git may need. Unfortunately we can't
> get a stack trace or anything like that unless you rebuild Git from
> source.

It's moments like this that I wish we had a knob to force core
dumps. And I often modify die_builtin() to add '*(char*)0 = 1;' to
force a core dump when I can't figure out some problem based on the
error message alone.

So.. how about we do something like this? We could extend it to abort
on error() as well as die(). Aborting on warning() may be a bit too
much though. On glibc systems we could even print the back trace
without aborting, which helps in some cases.

The long variable name and value are on purpose to hopefully not
trigger this by mistake.

diff --git a/git.c b/git.c
index ab5c99c..5fea224 100644
--- a/git.c
+++ b/git.c
@@ -622,15 +622,34 @@ static int run_argv(int *argcp, const char ***argv)
 	return done_alias;
 }
 
+static NORETURN void die_by_aborting(const char *err, va_list params)
+{
+	vreportf("fatal: ", err, params);
+	abort();
+}
+
+static NORETURN void die_silently_by_aborting(const char *err, va_list params)
+{
+	abort();
+}
+
 int cmd_main(int argc, const char **argv)
 {
 	const char *cmd;
 	int done_help = 0;
+	const char *die_abort_env = getenv("GIT_ABORT_ON_FATAL_ERRORS");
 
 	cmd = argv[0];
 	if (!cmd)
 		cmd = "git-help";
 
+	if (die_abort_env) {
+		if (!strcmp(die_abort_env, "yes please"))
+			set_die_routine(die_by_aborting);
+		else if (!strcmp(die_abort_env, "just die"))
+			set_die_routine(die_silently_by_aborting);
+	}
+
 	trace_command_performance(argv);
 
 	/*
--
Duy

Re: [PATCH] Add a knob to abort on die() (was Re: git checkout crashes after ...)

[Jeff King]

On Wed, Oct 19, 2016 at 08:47:50PM +0700, Duy Nguyen wrote:

> On Wed, Oct 19, 2016 at 08:27:43PM +0700, Duy Nguyen wrote:
> > If you set the environment variable GIT_ALLOC_LIMIT ...  git
> > attempts to allocate more than that ... then it's caught and we get
> > a glimpse of how much memory git may need. Unfortunately we can't
> > get a stack trace or anything like that unless you rebuild Git from
> > source.
> 
> It's moments like this that I wish we had a knob to force core
> dumps. And I often modify die_builtin() to add '*(char*)0 = 1;' to
> force a core dump when I can't figure out some problem based on the
> error message alone.
> 
> So.. how about we do something like this? We could extend it to abort
> on error() as well as die(). Aborting on warning() may be a bit too
> much though. On glibc systems we could even print the back trace
> without aborting, which helps in some cases.

I have been tempted by something like this, too, and have occasionally
resorted to patching in an abort() call to get cores from sub-processes.

I'm not sure how useful it would be in practice in deployed versions of
git, though. You'd have to coach the user into finding the core file and
generating the backtrace from it. Something that called backtrace(3)
automatically would be more seamless (but provides worse information
than gdb), and otherwise it is probably just as easy to ship the user
cut-and-paste instructions to use gdb.

See the previous discussion in this subthread:

  http://public-inbox.org/git/20150424201734.GA4747@peff.net/T/#u

-Peff

Re: git checkout crashes after server being updated to Debian X86_64

[Raffael Reichelt]

> Am 19.10.2016 um 15:27 schrieb Duy Nguyen <pclouds@gmail.com>:
> 
> On Tue, Oct 18, 2016 at 10:17 PM, Raffael Reichelt
> <raffael.reichelt@gmail.com> wrote:
>> Hello!
>> 
>> I have a serious problem with git, After my provider had updated to a X86_64 architecture git crashes with various memory-related errors. This is happening remote when pushing to the repository from my local machine as well as trying it on a shell on the server itself.
>> 
>> This are the error-messages:
>> 
>> fatal: Out of memory, realloc failed
>> fatal: recursion detected in die handler
>> fatal: recursion detected in die handler
> 
> You other mail said memory is capped at 600MB, which should be a lot
> for normal repositories. If you set the environment variable
> GIT_ALLOC_LIMIT to maybe 500MB or lower (convert it to kilobytes
> first) and git attempts to allocate more than that (just that one
> time, not total mem) then it's caught and we get a glimpse of how much
> memory git may need. Unfortunately we can't get a stack trace or
> anything like that unless you rebuild Git from source.

This was no change: crashed with the same errors …

> 
>> or
>> fatal: unable to create threaded lstat
>> fatal: recursion detected in die handler
> 
> Hmm.. with "max user processes (-u) 42" we should be fine because we
> only create 20 threads max. What happens if you set core.preloadindex
> to false? Can it run until the end or hit some other fatal errors?
> 

This did the trick :) I just repeatedly did a forced checkout and it went until the end without errors
THX a lot!

Raffael