workaround for git maintenance systemd services not exiting when git is built with address sanitizer

workaround for git maintenance systemd services not exiting when git is built with address sanitizer

[Jacob Keller]

Hi,

I recently tried to enable git maintenance for some repositories on one 
of my systems.

The services appeared to execute once but then the git processes got 
stuck the services never exited.

I tried various things to figure out what was going on including strace 
on the processes. This showed that the git process was constantly 
calling sched_yield().

I then connected to the process with gdb and took a backtrace:

>    $ gdb -p 1344308
>    (gdb) bt
>    #0  0x00007fe6058e9beb in sched_yield () from /lib64/libc.so.6
>    #1  0x00007fe606312fe5 in __sanitizer::StopTheWorld(void (*)(__sanitizer::SuspendedThreadsList const&, void*), void*) () from /lib64/libasan.so.8
>    #2  0x00007fe60632685e in __lsan::LockStuffAndStopTheWorldCallback(dl_phdr_info*, unsigned long, void*) () from /lib64/libasan.so.8
>    #3  0x00007fe605950197 in dl_iterate_phdr () from /lib64/libc.so.6
>    #4  0x00007fe606326bb0 in __lsan::LockStuffAndStopTheWorld(void (*)(__sanitizer::SuspendedThreadsList const&, void*), __lsan::CheckForLeaksParam*) ()
>       from /lib64/libasan.so.8
>    #5  0x00007fe6063235d2 in __lsan::CheckForLeaksOnce() () from /lib64/libasan.so.8
>    #6  0x00007fe606323835 in __lsan::DoLeakCheck() () from /lib64/libasan.so.8
>    #7  0x00007fe60582943c in __cxa_finalize () from /lib64/libc.so.6
>    #8  0x00007fe606205c87 in __do_global_dtors_aux () from /lib64/libasan.so.8
>    #9  0x00007ffd86656ba0 in ?? ()
>    #10 0x00007fe606a6d2d2 in _dl_call_fini (closure_map=0x7fe606a620c0) at dl-call_fini.c:43

This shows that we appear to get stuck inside of the leak sanitizer for 
some reason.

I build git locally and enable the sanitizer, which helps me catch 
issues both when developing and running git.

I don't really understand why, but some google searching shows that it 
could be related to some of the way that systemd sets up and isolates 
the process.

The end result of this setup is that my git maintenance processes run 
once then never exit, so systemd service never stops, so the timer never 
restarts git maintenance on the next hour/day/etc.

I was able to work around this by adding an override.conf with

[Service]
Environment="ASAN_OPTIONS=detect_leaks=0"

Which disables the asan leak detection, and prevents this particular issue.

I thought I'd post this so that anyone else running into this issue 
might hopefully find the fix or workaround instead of wondering why git 
was getting stuck doing nothing for days on end.