git fetch pack freezes

git fetch pack freezes

[Ivan Kanis]

Hello,

We are using git with the smart HTTP protocol. We are seeing the client
freeze with Jenkins poll. It stops the continuous build which is a bit
of a pain. It happens with the latest release 1.8.0 on the client side.
The server is using 1.7.0.4.

On the server we are seeing the following error message:

var/log/apache2/error.log.1:[Mon Nov 19 15:17:12 2012] [error] [client 192.168.50.15] fatal: git upload-pack: not our ref aa92ca2076c2421433f89ee0b0ae89cd36d27481 multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta

The error is located in upload-pack.c:

                 * Hmph.  We may later want to allow "want" line that
                 * asks for something like "master~10" (symbolic)...
                 * would it make sense?  I don't know.
                 */
                o = lookup_object(sha1_buf);
                if (!o || !(o->flags & OUR_REF))
                        die("git upload-pack: not our ref %s", line+5);

It calls die with I believe is doing the right thing(TM).

On the client side I compiled git with debugging symbols. I then
attached the debugger to the process that gets hosed. I get the
following backtrace:

#0  0x00007fff85389af0 in read ()
#1  0x000000010f900913 in xread (fd=0, buf=0x7fff6f376e2c, len=4) at wrapper.c:142
#2  0x000000010f900a11 in read_in_full (fd=0, buf=0x7fff6f376e2c, count=4) at wrapper.c:171
#3  0x000000010f8a14e4 in safe_read (fd=0, buffer=0x7fff6f376e2c, size=4, return_line_fail=0) at pkt-line.c:140
#4  0x000000010f8a110e in packet_read_internal (fd=0, buffer=0x10f981eb0 "", size=1000, return_line_fail=0) at pkt-line.c:183
#5  0x000000010f8a1246 in packet_read_line (fd=0, buffer=0x10f981eb0 "", size=1000) at pkt-line.c:211
#6  0x000000010f7b5196 in get_ack (fd=0, result_sha1=0x7fff6f377450 "") at builtin/fetch-pack.c:227
#7  0x000000010f7b4074 in find_common (fd=0x7fff6f377668, result_sha1=0x7fff6f377450 "", refs=0x0) at builtin/fetch-pack.c:416
#8  0x000000010f7b32a5 in do_fetch_pack (fd=0x7fff6f377668, orig_ref=0x10fb007a0, sought=0x7fff6f377670, pack_lockfile=0x7fff6f377660)
    at builtin/fetch-pack.c:832
#9  0x000000010f7b2cbd in fetch_pack (my_args=0x10f973fe8, fd=0x7fff6f377668, conn=0x0, ref=0x10fb007a0, 
    dest=0x7fff6f377a78 "https://foo.visionobjects.com/git/a-project-name/", sought=0x7fff6f377670, pack_lockfile=0x7fff6f377660)
    at builtin/fetch-pack.c:1069
#10 0x000000010f7b2a25 in cmd_fetch_pack (argc=7, argv=0x7fff6f377890, prefix=0x0) at builtin/fetch-pack.c:1012
#11 0x000000010f77aa5b in run_builtin (p=0x10f971990, argc=7, argv=0x7fff6f377890) at git.c:306
#12 0x000000010f779ac7 in handle_internal_command (argc=7, argv=0x7fff6f377890) at git.c:467
#13 0x000000010f77a299 in run_argv (argcp=0x7fff6f377858, argv=0x7fff6f377850) at git.c:513
#14 0x000000010f7798fd in main (argc=7, argv=0x7fff6f377890) at git.c:588

The file descriptor being 0 in #6 seems a bit suspicious. I am not
familiar with the git protocol. Could I get some guidance on how to
troubleshoot some more so that I come up with a patch?
-- 
Ivan Kanis, Release Manager, Vision Objects,
Tel +33 2 28 01 49 50,  Fax +33 2 40 25 89 20
http://www.visionobjects.com

The essence of science: ask an impertinent question, and you are on
the way to a pertinent answer.
    -- Jacob Bronowski

Re: git fetch pack freezes

[Shawn Pearce]

On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis <ivan.kanis@googlemail.com> wrote:
> We are using git with the smart HTTP protocol. We are seeing the client
> freeze with Jenkins poll. It stops the continuous build which is a bit
> of a pain. It happens with the latest release 1.8.0 on the client side.
> The server is using 1.7.0.4.
>
> On the server we are seeing the following error message:
>
> var/log/apache2/error.log.1:[Mon Nov 19 15:17:12 2012] [error] [client 192.168.50.15] fatal: git upload-pack: not our ref aa92ca2076c2421433f89ee0b0ae89cd36d27481 multi_ack_detailed side-band-64k thin-pack no-progress ofs-delta

Upgrade your server. What happened here is a race condition that is
only possible in the smart HTTP protocol. The client makes an initial
request to ask where the branches are. Then it makes a second (and
possibly more) subsequent request to negotiate a common ancestor and
actually obtain the necessary objects. If a branch is updated between
that initial request and any subsequent request, the server fails with
this error message.

More recent servers handle this more gracefully by looking to see if
the object the client wants is reachable within some reasonable
distance from a current branch. If so, the request is still allowed to
continue, rather than die() with a failure.

> On the client side I compiled git with debugging symbols. I then
> attached the debugger to the process that gets hosed. I get the
> following backtrace:
>
> #5  0x000000010f8a1246 in packet_read_line (fd=0, buffer=0x10f981eb0 "", size=1000) at pkt-line.c:211
> #6  0x000000010f7b5196 in get_ack (fd=0, result_sha1=0x7fff6f377450 "") at builtin/fetch-pack.c:227
> #7  0x000000010f7b4074 in find_common (fd=0x7fff6f377668, result_sha1=0x7fff6f377450 "", refs=0x0) at builtin/fetch-pack.c:416
>
> The file descriptor being 0 in #6 seems a bit suspicious. I am not
> familiar with the git protocol. Could I get some guidance on how to
> troubleshoot some more so that I come up with a patch?

There is actually a parent process in front of this one called
git-remote-http (or git-remote-https depending on protocol). That
parent is doing the HTTP transport and is sending whatever the server
provides into this child's stdin. I'm not surprised there is yet
another deadlock condition in here, its a rather heinous process setup
and way to handle the protocol.

fd in frame #7 is an address of an int array. fd in frame #6 is an
actual fd, the one to read from, which in this case should be 0 to
read from stdin, because its trying to get data from the HTTP server
by way of reading it from the pipe the parent supplied to it on stdin.

So the stack frames are correct. Its just a problem that the parent
didn't identify the server crashing and closing its side of the pipe
on stdin to force it to EOF to prevent the child from getting hung
here in a read.

Re: git fetch pack freezes

[Ivan Kanis]

Hi Shawn,

Thanks for the quick reply.

Shawn Pearce <spearce@spearce.org> wrote:

> On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis <ivan.kanis@googlemail.com> wrote:
>>
>> On the server we are seeing the following error message:
>
> Upgrade your server.

OK we'll look into it. I have a question: will a 1.8 server still serve
1.7 git client?

> So the stack frames are correct. Its just a problem that the parent
> didn't identify the server crashing and closing its side of the pipe
> on stdin to force it to EOF to prevent the child from getting hung
> here in a read.

It sounds like a bug on the client, doesn't it?
-- 
Ivan Kanis, Release Manager, Vision Objects,
Tel +33 2 28 01 49 50,  Fax +33 2 40 25 89 20
http://www.visionobjects.com

Let a fool hold his tongue and he will pass for a sage.
    -- Publilius Syrus

Re: git fetch pack freezes

[Shawn Pearce]

On Wed, Nov 28, 2012 at 9:25 AM, Ivan Kanis <ivan.kanis@googlemail.com> wrote:
> Shawn Pearce <spearce@spearce.org> wrote:
>
>> On Wed, Nov 28, 2012 at 6:12 AM, Ivan Kanis <ivan.kanis@googlemail.com> wrote:
>>>
>>> On the server we are seeing the following error message:
>>
>> Upgrade your server.
>
> OK we'll look into it. I have a question: will a 1.8 server still serve
> 1.7 git client?

Yes.

>> So the stack frames are correct. Its just a problem that the parent
>> didn't identify the server crashing and closing its side of the pipe
>> on stdin to force it to EOF to prevent the child from getting hung
>> here in a read.
>
> It sounds like a bug on the client, doesn't it?

Yes.