Errors from http-fetch

Errors from http-fetch

[Daniel Barkalow]

I've now started getting errors from http-fetch; my guess for the output 
is that it has problems if an object is not available loose, and is only 
available in a pack file which is already being downloaded for a different 
object, possibly only if the first request to be made for the loose 
object is not the first one to fail.

On the other hand, it looks like just trying again works fine (or, at 
least, makes progress), so there's no need to fall back to rsync or such.

But that also means that, in order to try to make a repeatable test, you 
need to copy your state before you run it and get the error; I'll probably 
try this evening to work it out.

	-Daniel
*This .sig left intentionally blank*

Re: Errors from http-fetch

[Nick Hengeveld]

On Wed, Oct 19, 2005 at 03:46:10PM -0400, Daniel Barkalow wrote:

> I've now started getting errors from http-fetch; my guess for the output 
> is that it has problems if an object is not available loose, and is only 
> available in a pack file which is already being downloaded for a different 
> object, possibly only if the first request to be made for the loose 
> object is not the first one to fail.

Are you getting a curl return code of 52/CURLE_GOT_NOTHING?  Any chance
you have access to the associated server logs?

In the situation you describe, fetch() won't be called for the second
object until fetch() for the first object is finished - at which point
the pack will have been fully downloaded for the first object (or
failed, causing fetch() for the first object to fail as well.)

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: Errors from http-fetch

[Daniel Barkalow]

On Wed, 19 Oct 2005, Nick Hengeveld wrote:

> On Wed, Oct 19, 2005 at 03:46:10PM -0400, Daniel Barkalow wrote:
> 
> > I've now started getting errors from http-fetch; my guess for the output 
> > is that it has problems if an object is not available loose, and is only 
> > available in a pack file which is already being downloaded for a different 
> > object, possibly only if the first request to be made for the loose 
> > object is not the first one to fail.
> 
> Are you getting a curl return code of 52/CURLE_GOT_NOTHING?  Any chance
> you have access to the associated server logs?

I'm getting 404 for the object, which it reports. This was pulling from 
kernel.org.

> In the situation you describe, fetch() won't be called for the second
> object until fetch() for the first object is finished - at which point
> the pack will have been fully downloaded for the first object (or
> failed, causing fetch() for the first object to fail as well.)

My theory is that you have: prefetch(#2), prefetch(#1), (other stuff), #1 
gives an error, start downloading the pack, fetch(#2), #2 gives an error, 
and there are no more packs to try.

That is, it was a later object that prompted getting the pack, because 
either the server decided to send that error message sooner, some packet 
got dropped and retransmitted, the connection got a lower-numbered file 
descriptor and they were ready at the same time, or something like that.

Another thing I noticed was that it was in the middle of downloading the 
pack when it suddenly exitted due to not being able to find the object; it 
would probably be worth having a call to finish up the active transfers 
after giving up on completing the whole thing, since we probably do 
actually want to finish downloading a big pack if we've started.

Consider the situation where you happen to catch the server in the middle 
of mirroring, and you're halfway through a big pack; restarting is nice, 
but it might as well let you finish downloading the pack, so you have less 
to do when you try again later.

	-Daniel
*This .sig left intentionally blank*

Re: Errors from http-fetch

[Nick Hengeveld]

On Wed, Oct 19, 2005 at 06:10:52PM -0400, Daniel Barkalow wrote:

> I'm getting 404 for the object, which it reports. This was pulling from 
> kernel.org.

Something similar was reported yesterday, and when I investigated I
found that one of the kernel.org servers was returning a 404 for the
object in question and the other wasn't.  Vger bounced my reply because
it contained Content-type: headers...

> That is, it was a later object that prompted getting the pack, because 
> either the server decided to send that error message sooner, some packet 
> got dropped and retransmitted, the connection got a lower-numbered file 
> descriptor and they were ready at the same time, or something like that.

I could see that happen if the pack appeared on the repo after #1 404ed
and subsequently decided there were no packs, and before #2 404ed.  Or,
as was the case I saw yesterday, servers with the same DNS name aren't
in sync.  However, the pack download for #2 should never start in that
case because fetch() for #1 would fail and exit.

> Another thing I noticed was that it was in the middle of downloading the 
> pack when it suddenly exitted due to not being able to find the object; it 
> would probably be worth having a call to finish up the active transfers 
> after giving up on completing the whole thing, since we probably do 
> actually want to finish downloading a big pack if we've started.

I'm not sure how that could happen - once a pack download request
starts, it has to finish before any fetch() calls can fail.  However,
there could certainly be other object requests in process when one
fails, and it would be polite to let them finish.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: Errors from http-fetch

[Daniel Barkalow]

On Wed, 19 Oct 2005, Nick Hengeveld wrote:

> > Another thing I noticed was that it was in the middle of downloading the 
> > pack when it suddenly exitted due to not being able to find the object; it 
> > would probably be worth having a call to finish up the active transfers 
> > after giving up on completing the whole thing, since we probably do 
> > actually want to finish downloading a big pack if we've started.
> 
> I'm not sure how that could happen - once a pack download request
> starts, it has to finish before any fetch() calls can fail.  However,
> there could certainly be other object requests in process when one
> fails, and it would be polite to let them finish.

Hmm; it definitely cut off a pack download in progress. When I ran it 
again, it resumed that download, which contained the missing object. Last 
night, I didn't see any problems, though. I'd certainly believe that 
server disagreement could be a factor, though.

	-Daniel
*This .sig left intentionally blank*