[RFC] Timeouts on HTTP requests

[RFC] Timeouts on HTTP requests

[Nick Hengeveld]

Our QA department today checked what would happen if the network connection
went away completely in the middle of an HTTP transfer.  It looks as though
the answer is that git-http-fetch sits there forever waiting for CURL to
return something.

I'm thinking of taking advantage of CURL's capability of aborting a request
if the transfer rate drops below a threshold for a specified length of time
using a new pair of environment variables and/or config file settings:

GIT_HTTP_LOW_SPEED_LIMIT/http.lowspeedlimit
GIT_HTTP_LOW_SPEED_TIME/http.lowspeedtime

Does this make sense, and if so should there be defaults if nothing is
specified?

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: [RFC] Timeouts on HTTP requests

[Junio C Hamano]

Nick Hengeveld <nickh@reactrix.com> writes:

> Our QA department today checked what would happen if the network connection
> went away completely in the middle of an HTTP transfer.  It looks as though
> the answer is that git-http-fetch sits there forever waiting for CURL to
> return something.

Ouch.

> I'm thinking of taking advantage of CURL's capability of aborting a request
> if the transfer rate drops below a threshold for a specified length of time
> using a new pair of environment variables and/or config file settings:
>
> GIT_HTTP_LOW_SPEED_LIMIT/http.lowspeedlimit
> GIT_HTTP_LOW_SPEED_TIME/http.lowspeedtime
>
> Does this make sense, and if so should there be defaults if nothing is
> specified?

I suspect these would be quite different between DSL and
localnet, so I doubt if there is a reasonable default value to
quick give-up.

On the other hand, having _no_ activity for say 30 seconds would
indicate a dead link on either modem or localnet.

BTW, I've been thinking about giving defaults by shipping
templates/config (i.e. no compile-time defaults).  One trick I
found cute is to have "clone.keeppack = 1" in the templates to
be applied for any newly built repository, especially now
kernel.org has git-daemon enabled.

Re: [RFC] Timeouts on HTTP requests

[Petr Baudis]

Dear diary, on Wed, Oct 19, 2005 at 08:02:14AM CEST, I got a letter
where Junio C Hamano <junkio@cox.net> told me that...
> Nick Hengeveld <nickh@reactrix.com> writes:
> 
> > Our QA department today checked what would happen if the network connection
> > went away completely in the middle of an HTTP transfer.  It looks as though
> > the answer is that git-http-fetch sits there forever waiting for CURL to
> > return something.
> 
> Ouch.
> 
> > I'm thinking of taking advantage of CURL's capability of aborting a request
> > if the transfer rate drops below a threshold for a specified length of time
> > using a new pair of environment variables and/or config file settings:
> >
> > GIT_HTTP_LOW_SPEED_LIMIT/http.lowspeedlimit
> > GIT_HTTP_LOW_SPEED_TIME/http.lowspeedtime
> >
> > Does this make sense, and if so should there be defaults if nothing is
> > specified?
> 
> I suspect these would be quite different between DSL and
> localnet, so I doubt if there is a reasonable default value to
> quick give-up.
> 
> On the other hand, having _no_ activity for say 30 seconds would
> indicate a dead link on either modem or localnet.

I agree that we should definitely use timeout instead of some low speed
limit - can't curl do that?

But 30 seconds is too little (on bad links, I've seen TCP connections
stalled for much longer), I would use at least 120 seconds. Or perhaps
30 seconds, but retry three times or so.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: [RFC] Timeouts on HTTP requests

[Nick Hengeveld]

On Wed, Oct 19, 2005 at 01:09:10PM +0200, Petr Baudis wrote:

> > On the other hand, having _no_ activity for say 30 seconds would
> > indicate a dead link on either modem or localnet.
> 
> I agree that we should definitely use timeout instead of some low speed
> limit - can't curl do that?

Curl lets you set timeouts for the connect and the overall request, but
neither of these are related to activity.  In our case, some of the files
we transfer can be quite large and need to transfer over a 56K modem line.
As long as data is moving, we're happy; but we need the transfer to abort
if the connection drops and data stops moving completely.

> But 30 seconds is too little (on bad links, I've seen TCP connections
> stalled for much longer), I would use at least 120 seconds. Or perhaps
> 30 seconds, but retry three times or so.

I've been testing with low speed limit of 1 and low speed time of 300 - if
data transfer stays below 1 byte/sec for 5 minutes, curl aborts the transfer.

I like Junio's suggestion regarding defaults though, don't compile them in
but let default config templates take care of it.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.