Puzzling Git backtrace

Puzzling Git backtrace

[Bryan Turner]

I'm trying to assist a Bitbucket Server customer who is seeing some
"git-upload-pack" processes "hang" on their server.

While investigating, we had them connect gdb to their processes (which
are 2.10.0 built from source using a simple unzip-and-run-make
approach) and get the backtraces for them. The output that they're
seeing makes no sense to me, though, so I'm throwing this out to the
list just to see if anyone has any idea how the processes could end up
like this.

One "family" of "hung" processes looks like this in ps:

32432  7811 atlbitb+ Mar22 /bin/git upload-pack /path/to/repository
32433 32432 atlbitb+ Mar22 git-upload-pack /path/to/repository

Looks normal enough. The backtrace for process 32432 also looks like I'd expect:

#0  0x00007fa39842fc9c in __libc_waitpid (pid=pid@entry=32433,
stat_loc=stat_loc@entry=0x7fffb42a4e4c, options=options@entry=0) at
../sysdeps/unix/sysv/linux/waitpid.c:31
#1  0x00000000005098f2 in wait_or_whine (pid=32433, argv0=0xd6e0c0
"git-upload-pack", in_signal=in_signal@entry=0) at run-command.c:229
#2  0x000000000050a949 in finish_command
(cmd=cmd@entry=0x7fffb42a4ea0) at run-command.c:537
#3  0x000000000050a9c1 in run_command (cmd=cmd@entry=0x7fffb42a4ea0)
at run-command.c:558
#4  0x000000000050aa5e in run_command_v_opt_cd_env
(argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40, dir=dir@entry=0x0,
env=env@entry=0x0) at run-command.c:578
#5  0x000000000050aa79 in run_command_v_opt
(argv=argv@entry=0x7fffb42a5140, opt=opt@entry=40) at
run-command.c:563
#6  0x0000000000405ad0 in execv_dashed_external (argv=0x7fffb42a5140)
at git.c:569
#7  run_argv (argv=0x7fffb42a4f10, argcp=0x7fffb42a4f1c) at git.c:596
#8  cmd_main (argc=2, argc@entry=3, argv=0x7fffb42a5140,
argv@entry=0x7fffb42a5138) at git.c:665
#9  0x0000000000404d9d in main (argc=3, argv=0x7fffb42a5138) at common-main.c:40

It's in "wait_or_whine" for "git-upload-pack".

When they attached to 32433 and printed its backtrace, though, things
go a little sideways:

(gdb) attach 32433
Attaching to program: /usr/bin/git, process 32433
Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols
from /usr/lib/debug/usr/lib64/ld-2.17.so.debug...done.
done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x00007f79d1aca240 in ?? ()
(gdb) bt
#0  0x00007f79d1aca240 in ?? ()
#1  0x000000000045179e in mktree_line (allow_missing=4,
nul_term_line=0, len=<optimized out>, buf=<optimized out>) at
builtin/mktree.c:103
#2  cmd_mktree (ac=<optimized out>, av=<optimized out>,
prefix=<optimized out>) at builtin/mktree.c:173
#3  0x0000000000000000 in ?? ()

So ps shows "git-upload-pack", and the parent process shows it's
waiting for "git-upload-pack", but gdb shows it's attaching to "git",
not "git-upload-pack", and the stack trace on the child shows it's
running "git mktree". I've searched through the source code and I
don't see any path through upload-pack.c that could result in it
essentially exec'ing "git mktree" over the top of itself, or even
spawning a "git mktree" of its own.

Has anyone seen anything like this before? Any thoughts on how what
should be a "git-upload-pack" could possibly end up being a "git
mktree" instead?

Best regards,
Bryan

Re: Puzzling Git backtrace

[Jeff King]

On Wed, Mar 27, 2019 at 05:49:27PM -0700, Bryan Turner wrote:

> I'm trying to assist a Bitbucket Server customer who is seeing some
> "git-upload-pack" processes "hang" on their server.
> 
> While investigating, we had them connect gdb to their processes (which
> are 2.10.0 built from source using a simple unzip-and-run-make
> approach) and get the backtraces for them. The output that they're
> seeing makes no sense to me, though, so I'm throwing this out to the
> list just to see if anyone has any idea how the processes could end up
> like this.

upload-pack didn't become a builtin until v2.18, so...

> When they attached to 32433 and printed its backtrace, though, things
> go a little sideways:
> 
> (gdb) attach 32433
> Attaching to program: /usr/bin/git, process 32433

The debugger needs to be using git-upload-pack as its executable, not
"git".

> (gdb) bt
> #0  0x00007f79d1aca240 in ?? ()
> #1  0x000000000045179e in mktree_line (allow_missing=4,
> nul_term_line=0, len=<optimized out>, buf=<optimized out>) at
> builtin/mktree.c:103
> #2  cmd_mktree (ac=<optimized out>, av=<optimized out>,
> prefix=<optimized out>) at builtin/mktree.c:173
> #3  0x0000000000000000 in ?? ()

And all of this is just trash, because it's matching the core with the
wrong binary.

I'm sure there's a way to switch binaries within gdb, but I don't know
it offhand. I'd just start a new gdb like:

  gdb git-upload-pack 32433

-Peff

Re: Puzzling Git backtrace

[Bryan Turner]

On Wed, Mar 27, 2019 at 6:50 PM Jeff King <peff@peff.net> wrote:
>
> On Wed, Mar 27, 2019 at 05:49:27PM -0700, Bryan Turner wrote:
>
> > I'm trying to assist a Bitbucket Server customer who is seeing some
> > "git-upload-pack" processes "hang" on their server.
> >
> > While investigating, we had them connect gdb to their processes (which
> > are 2.10.0 built from source using a simple unzip-and-run-make
> > approach) and get the backtraces for them. The output that they're
> > seeing makes no sense to me, though, so I'm throwing this out to the
> > list just to see if anyone has any idea how the processes could end up
> > like this.
>
> upload-pack didn't become a builtin until v2.18, so...
>
> > When they attached to 32433 and printed its backtrace, though, things
> > go a little sideways:
> >
> > (gdb) attach 32433
> > Attaching to program: /usr/bin/git, process 32433
>
> The debugger needs to be using git-upload-pack as its executable, not
> "git".

Derp. Of course. Sorry, clearly it's been too long since I used C in anger!

Thanks for taking the time to point out what should have been obvious,
and for doing it so kindly!

Bryan