[BUG] Halt during fetch on MacOS

[BUG] Halt during fetch on MacOS

[Conley Owens]

$ git --version  # This is just the git from MacPorts
git version 1.8.5.5
$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.8.5
BuildVersion:   12F45

test.sh
"""""""""""""""""""""""""""""""""""""
#!/bin/bash
rungit() {
    mkdir $1
    GIT_DIR=$1 git init --bare
    echo '[remote "aosp"]' > $1/config
    echo '    url =
https://android.googlesource.com/platform/external/tinyxml2' >>
$1/config
    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
    rm -rf $1
}

for i in $(seq 1 100)
do
    rungit testdir$i &
done
"""""""""""""""""""""""""""""""""""""""
$ ./test.sh  # Warning! This script fetches ~40MB of data

When everything cools, you can see that there are some fetches hanging
(typically).
$ ps | grep 'git fetch'
...
63310 ttys004    0:00.01 git fetch aosp
+refs/heads/master:refs/remotes/aosp/master
63314 ttys004    0:00.01 git fetch aosp
+refs/heads/master:refs/remotes/aosp/master
63319 ttys004    0:00.01 git fetch aosp
+refs/heads/master:refs/remotes/aosp/master
63407 ttys004    0:00.00 git fetch aosp
+refs/heads/master:refs/remotes/aosp/master
63414 ttys004    0:00.00 git fetch aosp
+refs/heads/master:refs/remotes/aosp/master
63420 ttys004    0:00.00 git fetch aosp
+refs/heads/master:refs/remotes/aosp/master
...

You can look at the parent process of each and see that one half
spawned the other half, or you can look at the environment variables
for each to see that there are two processes operating in the same
directory for each directory where there's an issue.
$ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
GIT_DIR=testdir14
GIT_DIR=testdir14
GIT_DIR=testdir32
GIT_DIR=testdir32
GIT_DIR=testdir47
GIT_DIR=testdir47

I've searched through the mailing list, but this doesn't seem to be a
known issue.  I've only seen this occur on macs (and with a good deal
of regularity).  It doesn't occur on my Ubuntu box.

~cco3

Re: [BUG] Halt during fetch on MacOS

[Jeff King]

On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:

> test.sh
> """""""""""""""""""""""""""""""""""""
> #!/bin/bash
> rungit() {
>     mkdir $1
>     GIT_DIR=$1 git init --bare
>     echo '[remote "aosp"]' > $1/config
>     echo '    url =
> https://android.googlesource.com/platform/external/tinyxml2' >>
> $1/config
>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master

I don't think this is affecting your test, but you probably want ">>" to
append to the config for the first line, too. Otherwise you are
overwriting some of git's default settings.

> When everything cools, you can see that there are some fetches hanging
> (typically).
> $ ps | grep 'git fetch'
> ...
> 63310 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> [...]

I can't reproduce here on Linux. Can you find out what the processes are
doing with something like strace?

> You can look at the parent process of each and see that one half
> spawned the other half, or you can look at the environment variables
> for each to see that there are two processes operating in the same
> directory for each directory where there's an issue.
> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
> GIT_DIR=testdir14
> GIT_DIR=testdir14
> GIT_DIR=testdir32
> GIT_DIR=testdir32
> GIT_DIR=testdir47
> GIT_DIR=testdir47

A fetch will start many sub-processes. Try:

  GIT_TRACE=1 git fetch \
    https://android.googlesource.com/platform/external/tinyxml2

which shows git-fetch starting the git-remote-https helper, which in
turn starts git-fetch-pack to do the actual protocol, which uses
git-unpack-objects to manage the incoming pack.

-Peff

Re: [BUG] Halt during fetch on MacOS

[Kyle J. McKay]

On Feb 28, 2014, at 22:15, Jeff King wrote:
> On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:
>
>> test.sh
>> """""""""""""""""""""""""""""""""""""
>> #!/bin/bash
>> rungit() {
>>    mkdir $1
>>    GIT_DIR=$1 git init --bare
>>    echo '[remote "aosp"]' > $1/config
>>    echo '    url =
>> https://android.googlesource.com/platform/external/tinyxml2' >>
>> $1/config
>>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/ 
>> master
>
> I don't think this is affecting your test, but you probably want  
> ">>" to
> append to the config for the first line, too. Otherwise you are
> overwriting some of git's default settings.

I replaced it with a call to git config in my version.

>> When everything cools, you can see that there are some fetches  
>> hanging
>> (typically).
>> $ ps | grep 'git fetch'
>> ...
>> 63310 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> [...]
>
> I can't reproduce here on Linux. Can you find out what the processes  
> are
> doing with something like strace?

I can't reproduce, mostly, on Mac OS X 10.5.8 or 10.6.8.

What I mean by mostly is that the very first time I ran the test  
script I got approximately 36 of these errors:

fatal: unable to access 'https://android.googlesource.com/platform/external/tinyxml2/' 
: Unknown SSL protocol error in connection to android.googlesource.com: 
443

The rest of the fetches completed.  That was with Git 1.8.5.1.

However, I was never able to reproduce those errors again.  All the  
subsequent runs completed all fetches successfully using that same Git  
version so I also tried Git 1.8.5.2, 1.8.5.5 and Git 1.7.6.1 on the  
original and another machine.

I am, however NAT'd, so it's possible the NAT was somehow responsible  
for the initial 36 failures.

Perhaps you are seeing a similar issue.

You might try setting these sysctl variables:

# Timeout new TCP connections after 30 seconds instead of 75
net.inet.tcp.keepinit=30000
# Always keep alive TCP connections
net.inet.tcp.always_keepalive=1
# Start keep alive checks after 30 seconds instead of 2 hours
net.inet.tcp.keepidle=30000
# Wait 5 seconds between probes instead of 75
# Note that 8 probes in a row must fail to drop the connection
net.inet.tcp.keepintvl=5000

then running your test again and see if the hanging git fetch  
processes die with some kind of failed connection error within about  
70 seconds or so.  With the default sysctl settings, even with Git  
enabling keep alives, it would likely take a bit over two hours for a  
dead connection to be noticed.

--Kyle

Re: [BUG] Halt during fetch on MacOS

[Max Horn]

[-- Attachment #1: Type: text/plain, Size: 2890 bytes --]


On 01.03.2014, at 00:26, Conley Owens <cco3@android.com> wrote:

> $ git --version  # This is just the git from MacPorts
> git version 1.8.5.5
> $ sw_vers
> ProductName:    Mac OS X
> ProductVersion: 10.8.5
> BuildVersion:   12F45

I cannot reproduce this, neither with 1.8.5.5 nor with 1.9.0. I am also running Mac OS X 10.8.5.

Note: I tried this both with you original test.sh, and also with a version were I replaced the ">" by ">>", as per Jeff's suggestion. It (as predicted) didn't make any difference).

If this is a race condition, it might be easier to trigger it on slower hardware. I am running this on a 2012 MBP with 2.3 Ghz i7 and an SSD. What is your test machine?


Cheers,
Max

> 
> test.sh
> """""""""""""""""""""""""""""""""""""
> #!/bin/bash
> rungit() {
>    mkdir $1
>    GIT_DIR=$1 git init --bare
>    echo '[remote "aosp"]' > $1/config
>    echo '    url =
> https://android.googlesource.com/platform/external/tinyxml2' >>
> $1/config
>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>    rm -rf $1
> }
> 
> for i in $(seq 1 100)
> do
>    rungit testdir$i &
> done
> """""""""""""""""""""""""""""""""""""""
> $ ./test.sh  # Warning! This script fetches ~40MB of data
> 
> When everything cools, you can see that there are some fetches hanging
> (typically).
> $ ps | grep 'git fetch'
> ...
> 63310 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63314 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63319 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63407 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63414 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63420 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> ...
> 
> You can look at the parent process of each and see that one half
> spawned the other half, or you can look at the environment variables
> for each to see that there are two processes operating in the same
> directory for each directory where there's an issue.
> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
> GIT_DIR=testdir14
> GIT_DIR=testdir14
> GIT_DIR=testdir32
> GIT_DIR=testdir32
> GIT_DIR=testdir47
> GIT_DIR=testdir47
> 
> I've searched through the mailing list, but this doesn't seem to be a
> known issue.  I've only seen this occur on macs (and with a good deal
> of regularity).  It doesn't occur on my Ubuntu box.
> 
> ~cco3
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 235 bytes --]

Re: [BUG] Halt during fetch on MacOS

[Conley Owens]

On Sun, Mar 2, 2014 at 6:36 AM, Max Horn <max@quendi.de> wrote:
>
> On 01.03.2014, at 00:26, Conley Owens <cco3@android.com> wrote:
>
>> $ git --version  # This is just the git from MacPorts
>> git version 1.8.5.5
>> $ sw_vers
>> ProductName:    Mac OS X
>> ProductVersion: 10.8.5
>> BuildVersion:   12F45
>
> I cannot reproduce this, neither with 1.8.5.5 nor with 1.9.0. I am also running Mac OS X 10.8.5.
>
> Note: I tried this both with you original test.sh, and also with a version were I replaced the ">" by ">>", as per Jeff's suggestion. It (as predicted) didn't make any difference).
>
> If this is a race condition, it might be easier to trigger it on slower hardware. I am running this on a 2012 MBP with 2.3 Ghz i7 and an SSD. What is your test machine?

Mac mini i7 with 1TB fusion drive and 16GB ram.

It also seemed to trigger more when grabbing larger repositories.
Replace"external/tinyxml2" with "docs/source.android.com" and you
might be more likely to trigger the issue.  You can also try
increasing the number of processes to spawn from 100 to 150.

>
>
> Cheers,
> Max
>
>>
>> test.sh
>> """""""""""""""""""""""""""""""""""""
>> #!/bin/bash
>> rungit() {
>>    mkdir $1
>>    GIT_DIR=$1 git init --bare
>>    echo '[remote "aosp"]' > $1/config
>>    echo '    url =
>> https://android.googlesource.com/platform/external/tinyxml2' >>
>> $1/config
>>    GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>>    rm -rf $1
>> }
>>
>> for i in $(seq 1 100)
>> do
>>    rungit testdir$i &
>> done
>> """""""""""""""""""""""""""""""""""""""
>> $ ./test.sh  # Warning! This script fetches ~40MB of data
>>
>> When everything cools, you can see that there are some fetches hanging
>> (typically).
>> $ ps | grep 'git fetch'
>> ...
>> 63310 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63314 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63319 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63407 ttys004    0:00.00 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63414 ttys004    0:00.00 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> 63420 ttys004    0:00.00 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> ...
>>
>> You can look at the parent process of each and see that one half
>> spawned the other half, or you can look at the environment variables
>> for each to see that there are two processes operating in the same
>> directory for each directory where there's an issue.
>> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
>> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
>> GIT_DIR=testdir14
>> GIT_DIR=testdir14
>> GIT_DIR=testdir32
>> GIT_DIR=testdir32
>> GIT_DIR=testdir47
>> GIT_DIR=testdir47
>>
>> I've searched through the mailing list, but this doesn't seem to be a
>> known issue.  I've only seen this occur on macs (and with a good deal
>> of regularity).  It doesn't occur on my Ubuntu box.
>>
>> ~cco3
>> --
>> To unsubscribe from this list: send the line "unsubscribe git" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>

Re: [BUG] Halt during fetch on MacOS

[Conley Owens]

On Fri, Feb 28, 2014 at 10:15 PM, Jeff King <peff@peff.net> wrote:
> On Fri, Feb 28, 2014 at 03:26:28PM -0800, Conley Owens wrote:
>
>> test.sh
>> """""""""""""""""""""""""""""""""""""
>> #!/bin/bash
>> rungit() {
>>     mkdir $1
>>     GIT_DIR=$1 git init --bare
>>     echo '[remote "aosp"]' > $1/config
>>     echo '    url =
>> https://android.googlesource.com/platform/external/tinyxml2' >>
>> $1/config
>>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>
> I don't think this is affecting your test, but you probably want ">>" to
> append to the config for the first line, too. Otherwise you are
> overwriting some of git's default settings.
>
>> When everything cools, you can see that there are some fetches hanging
>> (typically).
>> $ ps | grep 'git fetch'
>> ...
>> 63310 ttys004    0:00.01 git fetch aosp
>> +refs/heads/master:refs/remotes/aosp/master
>> [...]
>
> I can't reproduce here on Linux. Can you find out what the processes are
> doing with something like strace?

Yes, none of my Linux users have had any problems with this, but many
of my Mac users have.

I'm trying to run it under dtruss, but it's slowing the entire system
down to a halt.

>
>> You can look at the parent process of each and see that one half
>> spawned the other half, or you can look at the environment variables
>> for each to see that there are two processes operating in the same
>> directory for each directory where there's an issue.
>> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
>> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
>> GIT_DIR=testdir14
>> GIT_DIR=testdir14
>> GIT_DIR=testdir32
>> GIT_DIR=testdir32
>> GIT_DIR=testdir47
>> GIT_DIR=testdir47
>
> A fetch will start many sub-processes. Try:
>
>   GIT_TRACE=1 git fetch \
>     https://android.googlesource.com/platform/external/tinyxml2
>
> which shows git-fetch starting the git-remote-https helper, which in
> turn starts git-fetch-pack to do the actual protocol, which uses
> git-unpack-objects to manage the incoming pack.

This is the full output of a fetch that ends up hanging:
trace: built-in: git 'fetch' 'aosp'
'+refs/heads/master:refs/remotes/aosp/master'
trace: run_command: 'git-remote-https' 'aosp'
'https://android.googlesource.com/platform/external/tinyxml2'

>
> -Peff

Re: [BUG] Halt during fetch on MacOS

[Jacopo Notarstefano]

On Sun, Mar 2, 2014 at 3:02 AM, Kyle J. McKay <mackyle@gmail.com> wrote:
> I can't reproduce, mostly, on Mac OS X 10.5.8 or 10.6.8.
>
> What I mean by mostly is that the very first time I ran the test script I
> got approximately 36 of these errors:
>
> fatal: unable to access
> 'https://android.googlesource.com/platform/external/tinyxml2/': Unknown SSL
> protocol error in connection to android.googlesource.com:443
>
> The rest of the fetches completed.  That was with Git 1.8.5.1.
>
> However, I was never able to reproduce those errors again.  All the
> subsequent runs completed all fetches successfully using that same Git
> version so I also tried Git 1.8.5.2, 1.8.5.5 and Git 1.7.6.1 on the original
> and another machine.

With Git 1.9.0.138.g2de3478 (latest build from source) on Mac OS X
10.9.2 I report similar results. A first run yielded several  "fatal:
unable to access" errors, while a second run yielded just one.

Re: [BUG] Halt during fetch on MacOS

[Conley Owens]

On Fri, Feb 28, 2014 at 3:26 PM, Conley Owens <cco3@android.com> wrote:
> $ git --version  # This is just the git from MacPorts
> git version 1.8.5.5
> $ sw_vers
> ProductName:    Mac OS X
> ProductVersion: 10.8.5
> BuildVersion:   12F45

OK, I've tried using my own build from master, and I still get the same results.

I've done a little more investigation and discovered it always hangs at:
`atexit(notify_parent);` in `run-command.c:start_command`
when running:
trace: run_command: 'git-remote-https' 'aosp'
'https://android.googlesource.com/platform/external/tinyxml2'

Could this have to do with the atexit implementation?  (eg. limit on
the number of functions that can be registered, etc)

$ cc -v
Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
Target: x86_64-apple-darwin12.5.0
Thread model: posix

>
> test.sh
> """""""""""""""""""""""""""""""""""""
> #!/bin/bash
> rungit() {
>     mkdir $1
>     GIT_DIR=$1 git init --bare
>     echo '[remote "aosp"]' > $1/config
>     echo '    url =
> https://android.googlesource.com/platform/external/tinyxml2' >>
> $1/config
>     GIT_DIR=$1 git fetch aosp +refs/heads/master:refs/remotes/aosp/master
>     rm -rf $1
> }
>
> for i in $(seq 1 100)
> do
>     rungit testdir$i &
> done
> """""""""""""""""""""""""""""""""""""""
> $ ./test.sh  # Warning! This script fetches ~40MB of data
>
> When everything cools, you can see that there are some fetches hanging
> (typically).
> $ ps | grep 'git fetch'
> ...
> 63310 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63314 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63319 ttys004    0:00.01 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63407 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63414 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> 63420 ttys004    0:00.00 git fetch aosp
> +refs/heads/master:refs/remotes/aosp/master
> ...
>
> You can look at the parent process of each and see that one half
> spawned the other half, or you can look at the environment variables
> for each to see that there are two processes operating in the same
> directory for each directory where there's an issue.
> $ echo "$(for pid in $(ps | grep 'git fetch' | grep -o '^[0-9]*'); do
> ps -p $pid -wwwE | grep 'GIT_DIR=[^ ]*' -o; done)" | sort
> GIT_DIR=testdir14
> GIT_DIR=testdir14
> GIT_DIR=testdir32
> GIT_DIR=testdir32
> GIT_DIR=testdir47
> GIT_DIR=testdir47
>
> I've searched through the mailing list, but this doesn't seem to be a
> known issue.  I've only seen this occur on macs (and with a good deal
> of regularity).  It doesn't occur on my Ubuntu box.
>
> ~cco3

Re: [BUG] Halt during fetch on MacOS

[Junio C Hamano]

Conley Owens <cco3@android.com> writes:

> On Fri, Feb 28, 2014 at 3:26 PM, Conley Owens <cco3@android.com> wrote:
>> $ git --version  # This is just the git from MacPorts
>> git version 1.8.5.5
>> $ sw_vers
>> ProductName:    Mac OS X
>> ProductVersion: 10.8.5
>> BuildVersion:   12F45
>
> OK, I've tried using my own build from master, and I still get the same results.
>
> I've done a little more investigation and discovered it always hangs at:
> `atexit(notify_parent);` in `run-command.c:start_command`
> when running:
> trace: run_command: 'git-remote-https' 'aosp'
> 'https://android.googlesource.com/platform/external/tinyxml2'
>
> Could this have to do with the atexit implementation?  (eg. limit on
> the number of functions that can be registered, etc)

Thanks.

An interesting theory indeed.  I read that an implementation is
supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while
I do think we register functions with atexit(3) from multiple places
in our code, I doubt we would be making that many.

> $ cc -v
> Apple clang version 4.1 (tags/Apple/clang-421.11.66) (based on LLVM 3.1svn)
> Target: x86_64-apple-darwin12.5.0
> Thread model: posix

Re: [BUG] Halt during fetch on MacOS

[Jeff King]

On Thu, Mar 06, 2014 at 10:24:49AM -0800, Junio C Hamano wrote:

> > OK, I've tried using my own build from master, and I still get the same results.
> >
> > I've done a little more investigation and discovered it always hangs at:
> > `atexit(notify_parent);` in `run-command.c:start_command`
> > when running:
> > trace: run_command: 'git-remote-https' 'aosp'
> > 'https://android.googlesource.com/platform/external/tinyxml2'
> >
> > Could this have to do with the atexit implementation?  (eg. limit on
> > the number of functions that can be registered, etc)
> 
> Thanks.
> 
> An interesting theory indeed.  I read that an implementation is
> supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while
> I do think we register functions with atexit(3) from multiple places
> in our code, I doubt we would be making that many.

It seems awfully weird that it would _hang_ in such a case, though. That
sounds more like hitting a mutex that's internal to atexit(), or
something similar.

Conley, can you see if dropping that atexit clears up the problem (you
should be OK without it; git will just fail to notice the child's
exec failure with as much detail).

-Peff

Re: [BUG] Halt during fetch on MacOS

[Conley Owens]

On Thu, Mar 6, 2014 at 1:16 PM, Jeff King <peff@peff.net> wrote:
> On Thu, Mar 06, 2014 at 10:24:49AM -0800, Junio C Hamano wrote:
>
>> > OK, I've tried using my own build from master, and I still get the same results.
>> >
>> > I've done a little more investigation and discovered it always hangs at:
>> > `atexit(notify_parent);` in `run-command.c:start_command`
>> > when running:
>> > trace: run_command: 'git-remote-https' 'aosp'
>> > 'https://android.googlesource.com/platform/external/tinyxml2'
>> >
>> > Could this have to do with the atexit implementation?  (eg. limit on
>> > the number of functions that can be registered, etc)
>>
>> Thanks.
>>
>> An interesting theory indeed.  I read that an implementation is
>> supposed to take at least ATEXIT_MAX (32) calls to atexit(3); while
>> I do think we register functions with atexit(3) from multiple places
>> in our code, I doubt we would be making that many.
>
> It seems awfully weird that it would _hang_ in such a case, though. That
> sounds more like hitting a mutex that's internal to atexit(), or
> something similar.

You are correct that it's a mutex internal to atexit.  The crazy thing
is that there is only one thread sitting and waiting for it.
====================
(gdb) thread apply all bt

Thread 1 (process 71053):
#0  0x00007fff8cb67122 in __psynch_mutexwait ()
#1  0x00007fff833d0dcd in pthread_mutex_lock ()
#2  0x00007fff8d4a77e0 in LockHelper::LockHelper ()
#3  0x00007fff8d4a8d0a in dladdr ()
#4  0x00007fff83412260 in atexit ()
#5  0x000000010597b35e in start_command (cmd=0x7fe1cd801b30) at
run-command.c:374
#6  0x0000000105998959 in get_helper (transport=<value temporarily
unavailable, due to optimizations>) at transport-helper.c:142
#7  0x00000001059970bd in get_refs_list (transport=0x7fe1cd801370,
for_push=0) at transport-helper.c:954
#8  0x000000010599635d in transport_get_remote_refs
(transport=0x7fe1cd801370) at transport.c:1227
#9  0x00000001058b7469 in get_ref_map [inlined] () at
/Users/android-build/cco3/master/git/builtin/fetch.c:278
#10 0x00000001058b7469 in fetch_one (remote=<value temporarily
unavailable, due to optimizations>, argc=<value temporarily
unavailable, due to optimizations>, argv=0x7fff00000000) at
builtin/fetch.c:862
#11 0x00000001058b6f22 in cmd_fetch (argc=<value temporarily
unavailable, due to optimizations>, argv=<value temporarily
unavailable, due to optimizations>, prefix=0x0) at
builtin/fetch.c:1158
#12 0x0000000105890acc in run_builtin [inlined] () at
/Users/android-build/cco3/master/git/git.c:314
#13 0x0000000105890acc in handle_builtin (argc=3, argv=0x7fff5a370648)
at git.c:487
#14 0x00000001058906c1 in main (argc=3, av=<value temporarily
unavailable, due to optimizations>) at git.c:533
===================

>
> Conley, can you see if dropping that atexit clears up the problem (you
> should be OK without it; git will just fail to notice the child's
> exec failure with as much detail).

Yes, I'm unable to reproduce the issue after dropping atexit.

>
> -Peff