[PATCH] upload-pack: fix timeout in create_pack_file

[PATCH] upload-pack: fix timeout in create_pack_file

[Matthias Lederhofer]

Signed-off-by: Matthias Lederhofer <matled@gmx.net>
---
<yacc> fatal: packfile '../linux-2.6/.git/objects/pack/tmp-7iPJo5'
       SHA1 mismatch
<yacc> error: git-fetch-pack: unable to read from git-index-pack
<yacc> error: git-index-pack died with error code 128
<yacc> Any idea what this means?
This happens after ~12 minutes.  The problem is that the loop in
upload-pack.c actually sending the pack does not reset the timeout.
I'd guess --timeout is 600 or a bit more on git.kernel.org :)

This does not help for low timeouts with slow clients.  If a client is
slow enough so the server is blocked for more time than specified by
timeout the connection will be closed too (e.g. 15kb/s with a timeout
of 30 (git adds 10 extra) is not enough).  We should either add a
warning to the man page or try to fix this.  I don't know if this can
be fixed not using non-blocking sockets.

Perhaps support for resume would be quite useful too but I've no idea
how hard this is to implement.

A workaround for is to pull a part of the repository first and the
rest later.  For example using this:
$ git init-db
$ git fetch URL refs/tags/old-tag:refs/tags/old-tag
$ git fetch URL refs/tags/newer-tag:refs/tags/newer-tag
..
---
 upload-pack.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/upload-pack.c b/upload-pack.c
index f6f5a7e..07ecdb4 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -182,6 +182,8 @@ static void create_pack_file(void)
 		ssize_t sz;
 		int pe, pu, pollsize;
 
+		reset_timeout();
+
 		pollsize = 0;
 		pe = pu = -1;
 
-- 
1.4.2.rc1.ge7a0

Re: [PATCH] upload-pack: fix timeout in create_pack_file

[Junio C Hamano]

Matthias Lederhofer <matled@gmx.net> writes:

> Signed-off-by: Matthias Lederhofer <matled@gmx.net>
> ---
> <yacc> fatal: packfile '../linux-2.6/.git/objects/pack/tmp-7iPJo5'
>        SHA1 mismatch
> <yacc> error: git-fetch-pack: unable to read from git-index-pack
> <yacc> error: git-index-pack died with error code 128
> <yacc> Any idea what this means?
> This happens after ~12 minutes.  The problem is that the loop in
> upload-pack.c actually sending the pack does not reset the timeout.
> I'd guess --timeout is 600 or a bit more on git.kernel.org :)
>
> This does not help for low timeouts with slow clients.  If a client is
> slow enough so the server is blocked for more time than specified by
> timeout the connection will be closed too (e.g. 15kb/s with a timeout
> of 30 (git adds 10 extra) is not enough).

Where do we add 10 extra?

> We should either add a
> warning to the man page or try to fix this.  I don't know if this can
> be fixed not using non-blocking sockets.

I think the intent of "timeout" was to protect us from funny
clients by avoiding talking with the ones that take too much
time doing something that should not take too long.  When we are
in create_pack_file(), we are already committed to the heaviest
operation anyway, so one possibility might be to stop doing
timeout at that point.  I am not sure if that is acceptable,
though -- it opens up the daemon to even easier DoS than it
currently is.

My gut feeling is that your patch would be fine as is (have you
tried and confirmed that it helps cases other than slow
clients?)

> Perhaps support for resume would be quite useful too but I've no idea
> how hard this is to implement.

That would be _very_ hard.

Re: [PATCH] upload-pack: fix timeout in create_pack_file

[Matthias Lederhofer]

Junio C Hamano <junkio@cox.net> wrote:
> > This does not help for low timeouts with slow clients.  If a client is
> > slow enough so the server is blocked for more time than specified by
> > timeout the connection will be closed too (e.g. 15kb/s with a timeout
> > of 30 (git adds 10 extra) is not enough).
> 
> Where do we add 10 extra?
Oops, never mind, I misread the source.

> > We should either add a
> > warning to the man page or try to fix this.  I don't know if this can
> > be fixed not using non-blocking sockets.
> 
> I think the intent of "timeout" was to protect us from funny
> clients by avoiding talking with the ones that take too much
> time doing something that should not take too long.  When we are
> in create_pack_file(), we are already committed to the heaviest
> operation anyway, so one possibility might be to stop doing
> timeout at that point.  I am not sure if that is acceptable,
> though -- it opens up the daemon to even easier DoS than it
> currently is.
> 
> My gut feeling is that your patch would be fine as is (have you
> tried and confirmed that it helps cases other than slow
> clients?)
What else?  The problem solved by this patch is the missing
reset_timeout while downloading pack files and this affects all
clients that are too slow (cannot download the packfile within
--timeout seconds).  I tried a server with --timeout=600 and a client
that cannot download the whole pack file in 10 minutes.  Without the
patch the server closes the connection, with this patch it works.
Additionally I tried it with a client reading so slow that the server
was blocked for more than --timeout seconds, then the server closes
the connection too (this is more or less the DoS case).

> > Perhaps support for resume would be quite useful too but I've no idea
> > how hard this is to implement.
> 
> That would be _very_ hard.
I don't know how much demand there is for resuming but there is always rsync
(at least on kernel.org and other repositories that need it can install it too)
to get the new objects with support for resuming.

Re: [PATCH] upload-pack: fix timeout in create_pack_file

[Junio C Hamano]

Matthias Lederhofer <matled@gmx.net> writes:

> Junio C Hamano <junkio@cox.net> wrote:
>
>> My gut feeling is that your patch would be fine as is (have you
>> tried and confirmed that it helps cases other than slow
>> clients?)
> ... I tried a server with --timeout=600 and a client
> that cannot download the whole pack file in 10 minutes.  Without the
> patch the server closes the connection, with this patch it works.
> Additionally I tried it with a client reading so slow that the server
> was blocked for more than --timeout seconds, then the server closes
> the connection too (this is more or less the DoS case).

Thanks, that's all I wanted to know.  Let's have it.