[PATCH] Fix potential local deadlock during fetch-pack

[PATCH] Fix potential local deadlock during fetch-pack

[Junio C Hamano]

The fetch-pack/upload-pack protocol relies on the underlying transport
(local pipe or TCP socket) to have enough slack to allow one window worth
of data in flight without blocking the writer.  Traditionally we always
relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
bytes) to stream.

The recent "progressive-stride" change allows "fetch-pack" to send up to
1024 "have"s without reading any response from "upload-pack".  The
outgoing pipe of "upload-pack" can be clogged with many ACK and NAK that
are unread, while "fetch-pack" is still stuffing its outgoing pike with
more "have"s, leading to a deadlock.

Revert the change unless we are in stateless rpc (aka smart-http) mode, as
using a large window full of "have"s is still a good way to help reduce
the number of back-and-forth, and there is no buffering issue there (it is
strictly "ping-pong" without an overlap).

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 builtin/fetch-pack.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c
index 1724b76..e5fdacd 100644
--- a/builtin/fetch-pack.c
+++ b/builtin/fetch-pack.c
@@ -229,16 +229,17 @@ static void insert_alternate_refs(void)
 }
 
 #define INITIAL_FLUSH 16
+#define PIPESAFE_FLUSH 32
 #define LARGE_FLUSH 1024
 
 static int next_flush(int count)
 {
-	if (count < INITIAL_FLUSH * 2)
-		count += INITIAL_FLUSH;
-	else if (count < LARGE_FLUSH)
+	int flush_limit = args.stateless_rpc ? LARGE_FLUSH : PIPESAFE_FLUSH;
+
+	if (count < flush_limit)
 		count <<= 1;
 	else
-		count += LARGE_FLUSH;
+		count += flush_limit;
 	return count;
 }
 

Re: [PATCH] Fix potential local deadlock during fetch-pack

[Shawn Pearce]

On Tue, Mar 29, 2011 at 10:06, Junio C Hamano <gitster@pobox.com> wrote:
> The fetch-pack/upload-pack protocol relies on the underlying transport
> (local pipe or TCP socket) to have enough slack to allow one window worth
> of data in flight without blocking the writer.  Traditionally we always
> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k

Its 64. Because the client "races ahead" one window before ever
reading. Which is closer to 3K of data in flight.

> The recent "progressive-stride" change allows "fetch-pack" to send up to
> 1024 "have"s without reading any response from "upload-pack".  The
> outgoing pipe of "upload-pack" can be clogged with many ACK and NAK that
> are unread, while "fetch-pack" is still stuffing its outgoing pike with

s/pike/pipe/

> @@ -229,16 +229,17 @@ static void insert_alternate_refs(void)
>  }
>
>  #define INITIAL_FLUSH 16
> +#define PIPESAFE_FLUSH 32
>  #define LARGE_FLUSH 1024
>
>  static int next_flush(int count)
>  {
> -       if (count < INITIAL_FLUSH * 2)
> -               count += INITIAL_FLUSH;
> -       else if (count < LARGE_FLUSH)
> +       int flush_limit = args.stateless_rpc ? LARGE_FLUSH : PIPESAFE_FLUSH;
> +
> +       if (count < flush_limit)
>                count <<= 1;
>        else
> -               count += LARGE_FLUSH;
> +               count += flush_limit;

Nak. You still deadlock because when count reaches PIPESAFE_FLUSH you
still double it to 2*PIPESAFE_FLUSH here. Instead I think you mean:

  if (args.stateless_rpc) {
    if (count < LARGE_FLUSH)
      count <<= 1;
    else
      count += LARGE_FLUSH;
  } else {
    if (count * 2 < PIPESAFE_FLUSH)
      count <<= 1;
  }

-- 
Shawn.

Re: [PATCH] Fix potential local deadlock during fetch-pack

[Shawn Pearce]

On Tue, Mar 29, 2011 at 10:22, Shawn Pearce <spearce@spearce.org> wrote:
> On Tue, Mar 29, 2011 at 10:06, Junio C Hamano <gitster@pobox.com> wrote:
>> The fetch-pack/upload-pack protocol relies on the underlying transport
>> (local pipe or TCP socket) to have enough slack to allow one window worth
>> of data in flight without blocking the writer.  Traditionally we always
>> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
>
>> +               count += flush_limit;
>
> Nak. You still deadlock because when count reaches PIPESAFE_FLUSH you
> still double it to 2*PIPESAFE_FLUSH here. Instead I think you mean:

I take this comment back. Re-reading fetch-pack.c the next_flush()
method is accepting as input a running counter of how many have lines
have already been sent to the remote peer, and is never reset to 0.
Therefore it is necessary to add the next round size to count and
return it.

-- 
Shawn.

Re: [PATCH] Fix potential local deadlock during fetch-pack

[Erik Faye-Lund]

On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:
> The fetch-pack/upload-pack protocol relies on the underlying transport
> (local pipe or TCP socket) to have enough slack to allow one window worth
> of data in flight without blocking the writer.  Traditionally we always
> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
> bytes) to stream.
>

Hmm, this explanation makes me wonder: Could this be related to the
deadlock we're experiencing with git-push over the git-protocol on
Windows when side-band-64k is enabled?

Re: [PATCH] Fix potential local deadlock during fetch-pack

[Erik Faye-Lund]

On Wed, Mar 30, 2011 at 11:42 AM, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> The fetch-pack/upload-pack protocol relies on the underlying transport
>> (local pipe or TCP socket) to have enough slack to allow one window worth
>> of data in flight without blocking the writer.  Traditionally we always
>> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
>> bytes) to stream.
>>
>
> Hmm, this explanation makes me wonder: Could this be related to the
> deadlock we're experiencing with git-push over the git-protocol on
> Windows when side-band-64k is enabled?
>

No, It doesn't seem like that's it. The socket buffers appears to be
8k by default on Windows, which should be plenty, right?

---8<---
diff --git a/compat/mingw.c b/compat/mingw.c
index d527562..985c271 100644
--- a/compat/mingw.c
+++ b/compat/mingw.c
@@ -1404,7 +1404,7 @@ int mingw_getnameinfo(const struct sockaddr *sa,
socklen_t salen,

 int mingw_socket(int domain, int type, int protocol)
 {
-	int sockfd;
+	int sockfd, val, len;
 	SOCKET s;

 	ensure_socket_initialization();
@@ -1428,6 +1428,12 @@ int mingw_socket(int domain, int type, int protocol)
 		return error("unable to make a socket file descriptor: %s",
 			strerror(errno));
 	}
+
+	len = sizeof(val);
+	if (!getsockopt(s, SOL_SOCKET, SO_RCVBUF, (char *)&val, &len))
+		fprintf(stderr, "SO_RCVBUF: %d\n", val);
+	len = sizeof(val);
+	if (!getsockopt(s, SOL_SOCKET, SO_SNDBUF, (char *)&val, &len))
+		fprintf(stderr, "SO_SNDBUF: %d\n", val);
 	return sockfd;
 }

---8<---

$ git push git://localhost
SO_RCVBUF: 8192
SO_SNDBUF: 8192
...

Re: [PATCH] Fix potential local deadlock during fetch-pack

[Shawn Pearce]

On Wed, Mar 30, 2011 at 02:42, Erik Faye-Lund <kusmabite@gmail.com> wrote:
> On Tue, Mar 29, 2011 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> The fetch-pack/upload-pack protocol relies on the underlying transport
>> (local pipe or TCP socket) to have enough slack to allow one window worth
>> of data in flight without blocking the writer.  Traditionally we always
>> relied on being able to have a batch of 32 "have"s in flight (roughly 1.5k
>> bytes) to stream.
>
> Hmm, this explanation makes me wonder: Could this be related to the
> deadlock we're experiencing with git-push over the git-protocol on
> Windows when side-band-64k is enabled?

I think its unrelated. It might also be a deadlock, but the push
protocol is quite a bit different. As far as I remember, there is no
risk of deadlock in the push protocol.

-- 
Shawn.