git http-push and MKCOL error (22/409)

git http-push and MKCOL error (22/409)

[Sean Davis]

I built git from source for mac os and after updating curl to the
latest version and installing the latest expat (both from source), I
have git-http-push installed.  I have a bare repository in the webDAV
directory (which was placed there over a webDAV mount, so it appears
that webDAV is working).  When I try to push to it, I get the
following:

sdavis$ git push http://sdavis@example.com/git/stuff.git master
Fetching remote heads...
  refs/
  refs/heads/
updating 'refs/heads/master'
  from 0000000000000000000000000000000000000000
  to   f1ea50f8f60497ea3eca2355c10c141734db6451
    sending 3070 objects
MKCOL d7a940ff6c7bbc924f6e2d2c319aea22b8b0fdac failed, aborting (22/409)
MKCOL c9ac4c17871308e520a9944e12d846be9f24f91d failed, aborting (22/409)
MKCOL 1ac1bf55510deade68b14debd71357e3d929fe1c failed, aborting (22/409)
MKCOL da5e8ff9649c2b6ee0fde2280163ab4b399a1f1a failed, aborting (22/409)
MKCOL 87fc8ecfa98c36e0f9e44ed772d49d02378fc191 failed, aborting (22/409)
UNLOCK HTTP error 500
error: failed to push some refs to 'http://sdavis@example.com/git/stuff.git'

The apache error log shows:

[Mon Sep 22 19:48:53 2008] [error] [client 156.40.148.25] File does
not exist: /home/www/git/stuff.git/info
[Mon Sep 22 19:48:53 2008] [error] [client 156.40.148.25] File does
not exist: /home/www/git/stuff.git/objects
[Mon Sep 22 19:49:47 2008] [error] [client 156.40.148.25] (2)No such
file or directory: Cannot create collection; intermediate collection
does not exist.  [409, #0]
[Mon Sep 22 19:49:47 2008] [error] [client 156.40.148.25] (2)No such
file or directory: Cannot create collection; intermediate collection
does not exist.  [409, #0]
[Mon Sep 22 19:49:47 2008] [error] [client 156.40.148.25] (2)No such
file or directory: Cannot create collection; intermediate collection
does not exist.  [409, #0]
[Mon Sep 22 19:49:47 2008] [error] [client 156.40.148.25] (2)No such
file or directory: Cannot create collection; intermediate collection
does not exist.  [409, #0]
[Mon Sep 22 19:49:47 2008] [error] [client 156.40.148.25] (2)No such
file or directory: Cannot create collection; intermediate collection
does not exist.  [409, #0]

Any ideas?

Thanks,
Sean

git http-push and MKCOL error (22/409)

[Thomas Schlichter]

Hello,

I was using git version 1.6.4 and a remote WebDAV repository. But 
unfortunately, git-http-push always returns following error:

schlicht@netbook:~/dummy$ git push
Fetching remote heads...
  refs
  refs/heads
  refs/tags
  refs/heads
  refs/tags
updating 'refs/heads/master'
  from 980385b1032efc0db665dff3ad54068f762af9aa
  to   98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d
    sending 1 objects
MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting (22/409)
Updating remote server info
error: failed to push some refs to 
'https://webdav.smartdrive.web.de/dummy.git'

Current "master" and "next" trees also have this problem. But as git version 
1.6.4 does not have this problem, I was able to bisect it down to commit:

   5424bc557fc6414660830b470dd45774b8f5f281
   http*: add helper methods for fetching objects (loose)

I can always reproduce this problem, so I am willing to test patches to fix 
this regression.

Kind regards,
Thomas Schlichter

Re: git http-push and MKCOL error (22/409)

[Tay Ray Chuan]

Hi,

On Sun, Aug 16, 2009 at 9:57 PM, Thomas
Schlichter<thomas.schlichter@web.de> wrote:
> Current "master" and "next" trees also have this problem. But as git version
> 1.6.4 does not have this problem, I was able to bisect it down to commit:
>
>   5424bc557fc6414660830b470dd45774b8f5f281
>   http*: add helper methods for fetching objects (loose)

Interesting. Please do provide:

 -steps to reproduce,
 -your server's access log.


-- 
Cheers,
Ray Chuan

Re: git http-push and MKCOL error (22/409)

[Thomas Schlichter]

Hi,
Am Sonntag 16 August 2009 16:27:40 schrieb Tay Ray Chuan:
> Interesting. Please do provide:
>
>  -steps to reproduce,
>  -your server's access log.

Unfortunately I cannot provide the server's access log. It is a server 
provided by the German E-Mail provider web.de. It allows to save files via a 
web-interface and via WebDAV.

Steps to reproduce:
  1. locally set up a git archive:
     mkdir dummy.git
     cd dummy.git
     git init --bare
  2. Upload this directory to the server.
     I did do this using KDE's dolphin via WebDAV.
  3. Clone this remote repository:
     git clone https://webdav.smartdrive.web.de/dummy.git my_dummy
  4. Create a local commit:
     cd my_dummy
     touch dummy.c
     git commit -a
  5. Push this commit up to the remote repository:
     git push origin master

The last step fails es explained. During bisecting git I was able to commit 
several changes with versions before commit 
5424bc557fc6414660830b470dd45774b8f5f281.

Kind regards,
Thomas Schlichter

P.S.: Maybe it is important that the server uses the secure https protocol, 
therefore I set my username and password combination in my local ~/.netrc 
file.

Re: git http-push and MKCOL error (22/409)

[Junio C Hamano]

Tay Ray Chuan <rctay89@gmail.com> writes:

> On Sun, Aug 16, 2009 at 9:57 PM, Thomas
> Schlichter<thomas.schlichter@web.de> wrote:
>> Current "master" and "next" trees also have this problem. But as git version
>> 1.6.4 does not have this problem, I was able to bisect it down to commit:
>>
>>   5424bc557fc6414660830b470dd45774b8f5f281
>>   http*: add helper methods for fetching objects (loose)
>
> Interesting. Please do provide:
>
>  -steps to reproduce,
>  -your server's access log.

The report said:

    MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting (22/409)

As far as I can see you are trying (in http-push.c::start_mkcol()) to
create the two-hexdigit fan-out directory (i.e. "98" for this example); it
is strange to see a request to create the full 40-hexdigit collection in
the first place.

In another thread you responded to earlier:

    http://thread.gmane.org/gmane.comp.version-control.git/125933/focus=125972

the original report did not give the exact error message, but in that one,
instead of failing to create 40-hexdigit collection like this, I am
guessing that it fails with something like "MKCOL 'refs' failed".

So are these unrelated "breakages"[1]?

[Foornote]

*1* Not necessarily in the sense the client is broken but in the sense
that the server-client combination does not work for the reporter.

Re: git http-push and MKCOL error (22/409)

[Tay Ray Chuan]

Hi,

On Sun, Aug 16, 2009 at 10:52 PM, Thomas
Schlichter<thomas.schlichter@web.de> wrote:
> Steps to reproduce:

before I try this out, I have a few queries:

>  1. locally set up a git archive:
>     mkdir dummy.git
>     cd dummy.git
>     git init --bare
>  2. Upload this directory to the server.
>     I did do this using KDE's dolphin via WebDAV.
>  3. Clone this remote repository:
>     git clone https://webdav.smartdrive.web.de/dummy.git my_dummy

You didn't run git update-server-info before doing the clone; cloning
wouldn't work cos git can't find a info/refs file. Perhaps you did do
it and unintentionally left it out from your procedure listed here?

>  4. Create a local commit:
>     cd my_dummy
>     touch dummy.c
>     git commit -a

I assume you want to commit dummy.c? But you didn't tell git to track
it in the first place (git add dummy.c).

>  5. Push this commit up to the remote repository:
>     git push origin master

I also find it strange that git only pushed 1 object from your git
push output, when usually there would be a tree, a blob and a commit
to push for a new commit.

-- 
Cheers,
Ray Chuan

Re: git http-push and MKCOL error (22/409)

[Tay Ray Chuan]

Hi,

On Mon, Aug 17, 2009 at 3:34 AM, Junio C Hamano<gitster@pobox.com> wrote:
> The report said:
>
>    MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting (22/409)
>
> As far as I can see you are trying (in http-push.c::start_mkcol()) to
> create the two-hexdigit fan-out directory (i.e. "98" for this example); it
> is strange to see a request to create the full 40-hexdigit collection in
> the first place.

Yes, you're right, but git prints out the full SHA-1 hash even though
it is actually referring to the 2-hexdigit directory that it failed to
create/verify, for whatever reason.

> In another thread you responded to earlier:
>
>    http://thread.gmane.org/gmane.comp.version-control.git/125933/focus=125972
>
> the original report did not give the exact error message, but in that one,
> instead of failing to create 40-hexdigit collection like this, I am
> guessing that it fails with something like "MKCOL 'refs' failed".

I guess by "original report" you refer to Thomas' initial email: yes,
he (unwittingly) did provide theerror code in the subject line, the
part which says (22/409). It actually means (<curl return code>/<http
return code>). Referring again to
http://www.webdav.org/specs/rfc4918.html#rfc.section.9.3.1:

  409 (Conflict) - A collection cannot be made at the Request-URI until
  one or more intermediate collections have been created. The server
  must not create those intermediate collections automatically.

meaning one or more parent directories weren't created.

Based on this and his procedure, I'm guessing that there's something
wrong with his setup.

> *1* Not necessarily in the sense the client is broken but in the sense
> that the server-client combination does not work for the reporter.

Indeed.

-- 
Cheers,
Ray Chuan

Re: git http-push and MKCOL error (22/409)

[Thomas Schlichter]

Hi,

Am Montag 17 August 2009 06:52:32 schrieb Tay Ray Chuan:
> Hi,
>
> On Sun, Aug 16, 2009 at 10:52 PM, Thomas
>
> Schlichter<thomas.schlichter@web.de> wrote:
> > Steps to reproduce:
>
> before I try this out, I have a few queries:
> >  1. locally set up a git archive:
> >     mkdir dummy.git
> >     cd dummy.git
> >     git init --bare
> >  2. Upload this directory to the server.
> >     I did do this using KDE's dolphin via WebDAV.
> >  3. Clone this remote repository:
> >     git clone https://webdav.smartdrive.web.de/dummy.git my_dummy
>
> You didn't run git update-server-info before doing the clone; cloning
> wouldn't work cos git can't find a info/refs file. Perhaps you did do
> it and unintentionally left it out from your procedure listed here?

Oh, yes, sorry.
I did run it but unintentionally left it out. At least I had no problem with 
clone, it did clone an "empty repository" without a problem. Even during 
bisecting, I never had a problem with pulling from the server, only with 
pushing to it.

> >  4. Create a local commit:
> >     cd my_dummy
> >     touch dummy.c
> >     git commit -a
>
> I assume you want to commit dummy.c?

Yes, indeed...

> But you didn't tell git to track
> it in the first place (git add dummy.c).

I think that was not neccessary,  the parameter "-a" should do it for me...

> >  5. Push this commit up to the remote repository:
> >     git push origin master
>
> I also find it strange that git only pushed 1 object from your git
> push output, when usually there would be a tree, a blob and a commit
> to push for a new commit.

Yes, most of the times git tries to push three objects, and fails with all 
three. As I wrote, during bisecting I was able to commit several times. The 
reported message was the last one failing. Unfortunately git only needed to 
push one object that time. I didn't think it was important if it were three or 
just one object?!

Kind regards,
Thomas Schlichter

Re: git http-push and MKCOL error (22/409)

[Thomas Schlichter]

Hi,

Am Montag 17 August 2009 06:58:25 schrieb Tay Ray Chuan:
> Hi,
>
> On Mon, Aug 17, 2009 at 3:34 AM, Junio C Hamano<gitster@pobox.com> wrote:
> > The report said:
> >
> >    MKCOL 98fd7fb8f32843c1bb40bd195a2f1cd6cab0751d failed, aborting
> > (22/409)
> >
> > As far as I can see you are trying (in http-push.c::start_mkcol()) to
> > create the two-hexdigit fan-out directory (i.e. "98" for this example);
> > it is strange to see a request to create the full 40-hexdigit collection
> > in the first place.
>
> Yes, you're right, but git prints out the full SHA-1 hash even though
> it is actually referring to the 2-hexdigit directory that it failed to
> create/verify, for whatever reason.
>
> > In another thread you responded to earlier:
> >
> >  
> >  http://thread.gmane.org/gmane.comp.version-control.git/125933/focus=1259
> >72
> >
> > the original report did not give the exact error message, but in that
> > one, instead of failing to create 40-hexdigit collection like this, I am
> > guessing that it fails with something like "MKCOL 'refs' failed".
>
> I guess by "original report" you refer to Thomas' initial email: yes,
> he (unwittingly) did provide theerror code in the subject line, the
> part which says (22/409). It actually means (<curl return code>/<http
> return code>). Referring again to
> http://www.webdav.org/specs/rfc4918.html#rfc.section.9.3.1:
>
>   409 (Conflict) - A collection cannot be made at the Request-URI until
>   one or more intermediate collections have been created. The server
>   must not create those intermediate collections automatically.
>
> meaning one or more parent directories weren't created.
>
> Based on this and his procedure, I'm guessing that there's something
> wrong with his setup.
>
> > *1* Not necessarily in the sense the client is broken but in the sense
> > that the server-client combination does not work for the reporter.
>
> Indeed.

Hmm. I hope you won't take this as an argument for not fixing it, because it 
is a clear regression! I tried git version 1.6.3.4, and it works flawlessly 
with exactly this server! Even during bisecting, some versions worked, some 
didn't (these after the mentioned commit...)

So I think this commit didn't only refactor things, but unintentionally 
changed the behavior. And this must be the problem. As I'm not into this code, 
and the refactoring was not completely trivial, I was not able to quickly find 
the place where the behavior was changed...

Kind regards,
Thomas Schlichter

Re: git http-push and MKCOL error (22/409)

[Thomas Schlichter]

Hi,

Am Montag 17 August 2009 07:18:15 schrieb Thomas Schlichter:
> Hi,
>
> Am Montag 17 August 2009 06:52:32 schrieb Tay Ray Chuan:
> > I assume you want to commit dummy.c?
>
> Yes, indeed...
>
> > But you didn't tell git to track
> > it in the first place (git add dummy.c).
>
> I think that was not neccessary,  the parameter "-a" should do it for me...

OK, I tried it again, and the parameter "-a" was not sufficient. So I 
obviously did run a "git add dummy.c". Sorry for missing that.

Now I do also have the error message for the first commit:

schlicht@netbook:~/dummy2$ git push origin master
Fetching remote heads...
  refs
  refs/heads
  refs/tags
  refs/heads
  refs/tags
updating 'refs/heads/master'
  from 0000000000000000000000000000000000000000
  to   60157fdfdca4693f703c744fed95e1b31743bb4e
    sending 3 objects
MKCOL e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 failed, aborting (22/409)
MKCOL 4aefccd66b8e2bfa58079910045c30bc792877dc failed, aborting (22/409)
MKCOL 60157fdfdca4693f703c744fed95e1b31743bb4e failed, aborting (22/409)
Updating remote server info
error: failed to push some refs to 
'https://webdav.smartdrive.web.de/dummy2.git'

Kind regards,
Thomas Schlichter

Re: git http-push and MKCOL error (22/409)

[Junio C Hamano]

Thomas Schlichter <thomas.schlichter@web.de> writes:

> Hmm. I hope you won't take this as an argument for not fixing it, because it 
> is a clear regression! I tried git version 1.6.3.4, and it works flawlessly 
> with exactly this server! Even during bisecting, some versions worked, some 
> didn't (these after the mentioned commit...)

If it works (without changing anything else) with one version and with a
problematic patch it doesn't, it is a clear regression.

If you revert 5424bc5 (http*: add helper methods for fetching objects
(loose), 2009-06-06) from the tip of 'master', does it make the problem go
away?  

The patch should look like something like the attached.

diff --git a/http-push.c b/http-push.c
index 00e83dc..d1ea702 100644
--- a/http-push.c
+++ b/http-push.c
@@ -115,10 +115,18 @@ struct transfer_request
 	struct remote_lock *lock;
 	struct curl_slist *headers;
 	struct buffer buffer;
+	char filename[PATH_MAX];
+	char tmpfile[PATH_MAX];
+	int local_fileno;
 	enum transfer_state state;
 	CURLcode curl_result;
 	char errorstr[CURL_ERROR_SIZE];
 	long http_code;
+	unsigned char real_sha1[20];
+	git_SHA_CTX c;
+	z_stream stream;
+	int zret;
+	int rename;
 	void *userData;
 	struct active_request_slot *slot;
 	struct transfer_request *next;
@@ -226,6 +234,15 @@ static struct curl_slist *get_dav_token_headers(struct remote_lock *lock, enum d
 	return dav_headers;
 }
 
+static void append_remote_object_url(struct strbuf *buf, const char *url,
+				     const char *hex,
+				     int only_two_digit_prefix)
+{
+	strbuf_addf(buf, "%sobjects/%.*s/", url, 2, hex);
+	if (!only_two_digit_prefix)
+		strbuf_addf(buf, "%s", hex+2);
+}
+
 static void finish_request(struct transfer_request *request);
 static void release_request(struct transfer_request *request);
 
@@ -239,29 +256,169 @@ static void process_response(void *callback_data)
 
 #ifdef USE_CURL_MULTI
 
+static char *get_remote_object_url(const char *url, const char *hex,
+				   int only_two_digit_prefix)
+{
+	struct strbuf buf = STRBUF_INIT;
+	append_remote_object_url(&buf, url, hex, only_two_digit_prefix);
+	return strbuf_detach(&buf, NULL);
+}
+
+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
+			       void *data)
+{
+	unsigned char expn[4096];
+	size_t size = eltsize * nmemb;
+	int posn = 0;
+	struct transfer_request *request = (struct transfer_request *)data;
+	do {
+		ssize_t retval = xwrite(request->local_fileno,
+					(char *) ptr + posn, size - posn);
+		if (retval < 0)
+			return posn;
+		posn += retval;
+	} while (posn < size);
+
+	request->stream.avail_in = size;
+	request->stream.next_in = ptr;
+	do {
+		request->stream.next_out = expn;
+		request->stream.avail_out = sizeof(expn);
+		request->zret = git_inflate(&request->stream, Z_SYNC_FLUSH);
+		git_SHA1_Update(&request->c, expn,
+				sizeof(expn) - request->stream.avail_out);
+	} while (request->stream.avail_in && request->zret == Z_OK);
+	data_received++;
+	return size;
+}
+
 static void start_fetch_loose(struct transfer_request *request)
 {
+	char *hex = sha1_to_hex(request->obj->sha1);
+	char *filename;
+	char prevfile[PATH_MAX];
+	char *url;
+	int prevlocal;
+	unsigned char prev_buf[PREV_BUF_SIZE];
+	ssize_t prev_read = 0;
+	long prev_posn = 0;
+	char range[RANGE_HEADER_SIZE];
+	struct curl_slist *range_header = NULL;
 	struct active_request_slot *slot;
-	struct http_object_request *obj_req;
 
-	obj_req = new_http_object_request(repo->url, request->obj->sha1);
-	if (obj_req == NULL) {
+	filename = sha1_file_name(request->obj->sha1);
+	snprintf(request->filename, sizeof(request->filename), "%s", filename);
+	snprintf(request->tmpfile, sizeof(request->tmpfile),
+		 "%s.temp", filename);
+
+	snprintf(prevfile, sizeof(prevfile), "%s.prev", request->filename);
+	unlink_or_warn(prevfile);
+	rename(request->tmpfile, prevfile);
+	unlink_or_warn(request->tmpfile);
+
+	if (request->local_fileno != -1)
+		error("fd leakage in start: %d", request->local_fileno);
+	request->local_fileno = open(request->tmpfile,
+				     O_WRONLY | O_CREAT | O_EXCL, 0666);
+	/*
+	 * This could have failed due to the "lazy directory creation";
+	 * try to mkdir the last path component.
+	 */
+	if (request->local_fileno < 0 && errno == ENOENT) {
+		char *dir = strrchr(request->tmpfile, '/');
+		if (dir) {
+			*dir = 0;
+			mkdir(request->tmpfile, 0777);
+			*dir = '/';
+		}
+		request->local_fileno = open(request->tmpfile,
+					     O_WRONLY | O_CREAT | O_EXCL, 0666);
+	}
+
+	if (request->local_fileno < 0) {
 		request->state = ABORTED;
+		error("Couldn't create temporary file %s for %s: %s",
+		      request->tmpfile, request->filename, strerror(errno));
 		return;
 	}
 
-	slot = obj_req->slot;
+	memset(&request->stream, 0, sizeof(request->stream));
+
+	git_inflate_init(&request->stream);
+
+	git_SHA1_Init(&request->c);
+
+	url = get_remote_object_url(repo->url, hex, 0);
+	request->url = xstrdup(url);
+
+	/*
+	 * If a previous temp file is present, process what was already
+	 * fetched.
+	 */
+	prevlocal = open(prevfile, O_RDONLY);
+	if (prevlocal != -1) {
+		do {
+			prev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);
+			if (prev_read>0) {
+				if (fwrite_sha1_file(prev_buf,
+						     1,
+						     prev_read,
+						     request) == prev_read)
+					prev_posn += prev_read;
+				else
+					prev_read = -1;
+			}
+		} while (prev_read > 0);
+		close(prevlocal);
+	}
+	unlink_or_warn(prevfile);
+
+	/*
+	 * Reset inflate/SHA1 if there was an error reading the previous temp
+	 * file; also rewind to the beginning of the local file.
+	 */
+	if (prev_read == -1) {
+		memset(&request->stream, 0, sizeof(request->stream));
+		git_inflate_init(&request->stream);
+		git_SHA1_Init(&request->c);
+		if (prev_posn>0) {
+			prev_posn = 0;
+			lseek(request->local_fileno, 0, SEEK_SET);
+			ftruncate(request->local_fileno, 0);
+		}
+	}
+
+	slot = get_active_slot();
 	slot->callback_func = process_response;
 	slot->callback_data = request;
 	request->slot = slot;
-	request->userData = obj_req;
+
+	curl_easy_setopt(slot->curl, CURLOPT_FILE, request);
+	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
+	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, request->errorstr);
+	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+
+	/*
+	 * If we have successfully processed data from a previous fetch
+	 * attempt, only fetch the data we don't already have.
+	 */
+	if (prev_posn>0) {
+		if (push_verbosely)
+			fprintf(stderr,
+				"Resuming fetch of object %s at byte %ld\n",
+				hex, prev_posn);
+		sprintf(range, "Range: bytes=%ld-", prev_posn);
+		range_header = curl_slist_append(range_header, range);
+		curl_easy_setopt(slot->curl,
+				 CURLOPT_HTTPHEADER, range_header);
+	}
 
 	/* Try to get the request started, abort the request on error */
 	request->state = RUN_FETCH_LOOSE;
 	if (!start_active_slot(slot)) {
 		fprintf(stderr, "Unable to start GET request\n");
 		repo->can_update_info_refs = 0;
-		release_http_object_request(obj_req);
 		release_request(request);
 	}
 }
@@ -520,14 +677,16 @@ static void release_request(struct transfer_request *request)
 			entry->next = entry->next->next;
 	}
 
+	if (request->local_fileno != -1)
+		close(request->local_fileno);
 	free(request->url);
 	free(request);
 }
 
 static void finish_request(struct transfer_request *request)
 {
+	struct stat st;
 	struct http_pack_request *preq;
-	struct http_object_request *obj_req;
 
 	request->curl_result = request->slot->curl_result;
 	request->http_code = request->slot->http_code;
@@ -582,17 +741,39 @@ static void finish_request(struct transfer_request *request)
 			aborted = 1;
 		}
 	} else if (request->state == RUN_FETCH_LOOSE) {
-		obj_req = (struct http_object_request *)request->userData;
-
-		if (finish_http_object_request(obj_req) == 0)
-			if (obj_req->rename == 0)
-				request->obj->flags |= (LOCAL | REMOTE);
+		close(request->local_fileno);
+		request->local_fileno = -1;
+
+		if (request->curl_result != CURLE_OK &&
+		    request->http_code != 416) {
+			if (stat(request->tmpfile, &st) == 0) {
+				if (st.st_size == 0)
+					unlink_or_warn(request->tmpfile);
+			}
+		} else {
+			if (request->http_code == 416)
+				warning("requested range invalid; we may already have all the data.");
+
+			git_inflate_end(&request->stream);
+			git_SHA1_Final(request->real_sha1, &request->c);
+			if (request->zret != Z_STREAM_END) {
+				unlink_or_warn(request->tmpfile);
+			} else if (hashcmp(request->obj->sha1, request->real_sha1)) {
+				unlink_or_warn(request->tmpfile);
+			} else {
+				request->rename =
+					move_temp_to_file(
+						request->tmpfile,
+						request->filename);
+				if (request->rename == 0)
+					request->obj->flags |= (LOCAL | REMOTE);
+			}
+		}
 
 		/* Try fetching packed if necessary */
-		if (request->obj->flags & LOCAL) {
-			release_http_object_request(obj_req);
+		if (request->obj->flags & LOCAL)
 			release_request(request);
-		} else
+		else
 			start_fetch_packed(request);
 
 	} else if (request->state == RUN_FETCH_PACKED) {
@@ -664,6 +845,7 @@ static void add_fetch_request(struct object *obj)
 	request->url = NULL;
 	request->lock = NULL;
 	request->headers = NULL;
+	request->local_fileno = -1;
 	request->state = NEED_FETCH;
 	request->next = request_queue_head;
 	request_queue_head = request;
@@ -702,6 +884,7 @@ static int add_send_request(struct object *obj, struct remote_lock *lock)
 	request->url = NULL;
 	request->lock = lock;
 	request->headers = NULL;
+	request->local_fileno = -1;
 	request->state = NEED_PUSH;
 	request->next = request_queue_head;
 	request_queue_head = request;
diff --git a/http-walker.c b/http-walker.c
index 700bc13..8f7a975 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -3,6 +3,8 @@
 #include "walker.h"
 #include "http.h"
 
+#define PREV_BUF_SIZE 4096
+
 struct alt_base
 {
 	char *base;
@@ -23,8 +25,20 @@ struct object_request
 	struct walker *walker;
 	unsigned char sha1[20];
 	struct alt_base *repo;
+	char *url;
+	char filename[PATH_MAX];
+	char tmpfile[PATH_MAX];
+	int local;
 	enum object_request_state state;
-	struct http_object_request *req;
+	CURLcode curl_result;
+	char errorstr[CURL_ERROR_SIZE];
+	long http_code;
+	unsigned char real_sha1[20];
+	git_SHA_CTX c;
+	z_stream stream;
+	int zret;
+	int rename;
+	struct active_request_slot *slot;
 	struct object_request *next;
 };
 
@@ -45,6 +59,34 @@ struct walker_data {
 
 static struct object_request *object_queue_head;
 
+static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
+			       void *data)
+{
+	unsigned char expn[4096];
+	size_t size = eltsize * nmemb;
+	int posn = 0;
+	struct object_request *obj_req = (struct object_request *)data;
+	do {
+		ssize_t retval = xwrite(obj_req->local,
+					(char *) ptr + posn, size - posn);
+		if (retval < 0)
+			return posn;
+		posn += retval;
+	} while (posn < size);
+
+	obj_req->stream.avail_in = size;
+	obj_req->stream.next_in = ptr;
+	do {
+		obj_req->stream.next_out = expn;
+		obj_req->stream.avail_out = sizeof(expn);
+		obj_req->zret = git_inflate(&obj_req->stream, Z_SYNC_FLUSH);
+		git_SHA1_Update(&obj_req->c, expn,
+				sizeof(expn) - obj_req->stream.avail_out);
+	} while (obj_req->stream.avail_in && obj_req->zret == Z_OK);
+	data_received++;
+	return size;
+}
+
 static void fetch_alternates(struct walker *walker, const char *base);
 
 static void process_object_response(void *callback_data);
@@ -52,35 +94,172 @@ static void process_object_response(void *callback_data);
 static void start_object_request(struct walker *walker,
 				 struct object_request *obj_req)
 {
+	char *hex = sha1_to_hex(obj_req->sha1);
+	char prevfile[PATH_MAX];
+	char *url;
+	char *posn;
+	int prevlocal;
+	unsigned char prev_buf[PREV_BUF_SIZE];
+	ssize_t prev_read = 0;
+	long prev_posn = 0;
+	char range[RANGE_HEADER_SIZE];
+	struct curl_slist *range_header = NULL;
 	struct active_request_slot *slot;
-	struct http_object_request *req;
 
-	req = new_http_object_request(obj_req->repo->base, obj_req->sha1);
-	if (req == NULL) {
+	snprintf(prevfile, sizeof(prevfile), "%s.prev", obj_req->filename);
+	unlink_or_warn(prevfile);
+	rename(obj_req->tmpfile, prevfile);
+	unlink_or_warn(obj_req->tmpfile);
+
+	if (obj_req->local != -1)
+		error("fd leakage in start: %d", obj_req->local);
+	obj_req->local = open(obj_req->tmpfile,
+			      O_WRONLY | O_CREAT | O_EXCL, 0666);
+	/*
+	 * This could have failed due to the "lazy directory creation";
+	 * try to mkdir the last path component.
+	 */
+	if (obj_req->local < 0 && errno == ENOENT) {
+		char *dir = strrchr(obj_req->tmpfile, '/');
+		if (dir) {
+			*dir = 0;
+			mkdir(obj_req->tmpfile, 0777);
+			*dir = '/';
+		}
+		obj_req->local = open(obj_req->tmpfile,
+				      O_WRONLY | O_CREAT | O_EXCL, 0666);
+	}
+
+	if (obj_req->local < 0) {
 		obj_req->state = ABORTED;
+		error("Couldn't create temporary file %s for %s: %s",
+		      obj_req->tmpfile, obj_req->filename, strerror(errno));
 		return;
 	}
-	obj_req->req = req;
 
-	slot = req->slot;
+	memset(&obj_req->stream, 0, sizeof(obj_req->stream));
+
+	git_inflate_init(&obj_req->stream);
+
+	git_SHA1_Init(&obj_req->c);
+
+	url = xmalloc(strlen(obj_req->repo->base) + 51);
+	obj_req->url = xmalloc(strlen(obj_req->repo->base) + 51);
+	strcpy(url, obj_req->repo->base);
+	posn = url + strlen(obj_req->repo->base);
+	strcpy(posn, "/objects/");
+	posn += 9;
+	memcpy(posn, hex, 2);
+	posn += 2;
+	*(posn++) = '/';
+	strcpy(posn, hex + 2);
+	strcpy(obj_req->url, url);
+
+	/*
+	 * If a previous temp file is present, process what was already
+	 * fetched.
+	 */
+	prevlocal = open(prevfile, O_RDONLY);
+	if (prevlocal != -1) {
+		do {
+			prev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);
+			if (prev_read>0) {
+				if (fwrite_sha1_file(prev_buf,
+						     1,
+						     prev_read,
+						     obj_req) == prev_read)
+					prev_posn += prev_read;
+				else
+					prev_read = -1;
+			}
+		} while (prev_read > 0);
+		close(prevlocal);
+	}
+	unlink_or_warn(prevfile);
+
+	/*
+	 * Reset inflate/SHA1 if there was an error reading the previous temp
+	 * file; also rewind to the beginning of the local file.
+	 */
+	if (prev_read == -1) {
+		memset(&obj_req->stream, 0, sizeof(obj_req->stream));
+		git_inflate_init(&obj_req->stream);
+		git_SHA1_Init(&obj_req->c);
+		if (prev_posn>0) {
+			prev_posn = 0;
+			lseek(obj_req->local, 0, SEEK_SET);
+			ftruncate(obj_req->local, 0);
+		}
+	}
+
+	slot = get_active_slot();
 	slot->callback_func = process_object_response;
 	slot->callback_data = obj_req;
+	obj_req->slot = slot;
+
+	curl_easy_setopt(slot->curl, CURLOPT_FILE, obj_req);
+	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
+	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, obj_req->errorstr);
+	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+
+	/*
+	 * If we have successfully processed data from a previous fetch
+	 * attempt, only fetch the data we don't already have.
+	 */
+	if (prev_posn>0) {
+		if (walker->get_verbosely)
+			fprintf(stderr,
+				"Resuming fetch of object %s at byte %ld\n",
+				hex, prev_posn);
+		sprintf(range, "Range: bytes=%ld-", prev_posn);
+		range_header = curl_slist_append(range_header, range);
+		curl_easy_setopt(slot->curl,
+				 CURLOPT_HTTPHEADER, range_header);
+	}
 
 	/* Try to get the request started, abort the request on error */
 	obj_req->state = ACTIVE;
 	if (!start_active_slot(slot)) {
 		obj_req->state = ABORTED;
-		release_http_object_request(req);
+		obj_req->slot = NULL;
+		close(obj_req->local);
+		obj_req->local = -1;
+		free(obj_req->url);
 		return;
 	}
 }
 
 static void finish_object_request(struct object_request *obj_req)
 {
-	if (finish_http_object_request(obj_req->req))
+	struct stat st;
+
+	close(obj_req->local);
+	obj_req->local = -1;
+
+	if (obj_req->http_code == 416) {
+		fprintf(stderr, "Warning: requested range invalid; we may already have all the data.\n");
+	} else if (obj_req->curl_result != CURLE_OK) {
+		if (stat(obj_req->tmpfile, &st) == 0)
+			if (st.st_size == 0)
+				unlink_or_warn(obj_req->tmpfile);
 		return;
+	}
+
+	git_inflate_end(&obj_req->stream);
+	git_SHA1_Final(obj_req->real_sha1, &obj_req->c);
+	if (obj_req->zret != Z_STREAM_END) {
+		unlink_or_warn(obj_req->tmpfile);
+		return;
+	}
+	if (hashcmp(obj_req->sha1, obj_req->real_sha1)) {
+		unlink_or_warn(obj_req->tmpfile);
+		return;
+	}
+	obj_req->rename =
+		move_temp_to_file(obj_req->tmpfile, obj_req->filename);
 
-	if (obj_req->req->rename == 0)
+	if (obj_req->rename == 0)
 		walker_say(obj_req->walker, "got %s\n", sha1_to_hex(obj_req->sha1));
 }
 
@@ -92,16 +271,19 @@ static void process_object_response(void *callback_data)
 	struct walker_data *data = walker->data;
 	struct alt_base *alt = data->alt;
 
-	process_http_object_request(obj_req->req);
+	obj_req->curl_result = obj_req->slot->curl_result;
+	obj_req->http_code = obj_req->slot->http_code;
+	obj_req->slot = NULL;
 	obj_req->state = COMPLETE;
 
 	/* Use alternates if necessary */
-	if (missing_target(obj_req->req)) {
+	if (missing_target(obj_req)) {
 		fetch_alternates(walker, alt->base);
 		if (obj_req->repo->next != NULL) {
 			obj_req->repo =
 				obj_req->repo->next;
-			release_http_object_request(obj_req->req);
+			close(obj_req->local);
+			obj_req->local = -1;
 			start_object_request(walker, obj_req);
 			return;
 		}
@@ -114,8 +296,8 @@ static void release_object_request(struct object_request *obj_req)
 {
 	struct object_request *entry = object_queue_head;
 
-	if (obj_req->req !=NULL && obj_req->req->localfile != -1)
-		error("fd leakage in release: %d", obj_req->req->localfile);
+	if (obj_req->local != -1)
+		error("fd leakage in release: %d", obj_req->local);
 	if (obj_req == object_queue_head) {
 		object_queue_head = obj_req->next;
 	} else {
@@ -125,6 +307,7 @@ static void release_object_request(struct object_request *obj_req)
 			entry->next = entry->next->next;
 	}
 
+	free(obj_req->url);
 	free(obj_req);
 }
 
@@ -152,13 +335,19 @@ static void prefetch(struct walker *walker, unsigned char *sha1)
 	struct object_request *newreq;
 	struct object_request *tail;
 	struct walker_data *data = walker->data;
+	char *filename = sha1_file_name(sha1);
 
 	newreq = xmalloc(sizeof(*newreq));
 	newreq->walker = walker;
 	hashcpy(newreq->sha1, sha1);
 	newreq->repo = data->alt;
+	newreq->url = NULL;
+	newreq->local = -1;
 	newreq->state = WAITING;
-	newreq->req = NULL;
+	snprintf(newreq->filename, sizeof(newreq->filename), "%s", filename);
+	snprintf(newreq->tmpfile, sizeof(newreq->tmpfile),
+		 "%s.temp", filename);
+	newreq->slot = NULL;
 	newreq->next = NULL;
 
 	http_is_verbose = walker->get_verbosely;
@@ -449,6 +638,15 @@ abort:
 
 static void abort_object_request(struct object_request *obj_req)
 {
+	if (obj_req->local >= 0) {
+		close(obj_req->local);
+		obj_req->local = -1;
+	}
+	unlink_or_warn(obj_req->tmpfile);
+	if (obj_req->slot) {
+		release_active_slot(obj_req->slot);
+		obj_req->slot = NULL;
+	}
 	release_object_request(obj_req);
 }
 
@@ -457,7 +655,6 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c
 	char *hex = sha1_to_hex(sha1);
 	int ret = 0;
 	struct object_request *obj_req = object_queue_head;
-	struct http_object_request *req;
 
 	while (obj_req != NULL && hashcmp(obj_req->sha1, sha1))
 		obj_req = obj_req->next;
@@ -465,8 +662,6 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c
 		return error("Couldn't find request for %s in the queue", hex);
 
 	if (has_sha1_file(obj_req->sha1)) {
-		if (obj_req->req != NULL)
-			abort_http_object_request(obj_req->req);
 		abort_object_request(obj_req);
 		return 0;
 	}
@@ -478,42 +673,34 @@ static int fetch_object(struct walker *walker, struct alt_base *repo, unsigned c
 	start_object_request(walker, obj_req);
 #endif
 
-	/*
-	 * obj_req->req might change when fetching alternates in the callback
-	 * process_object_response; therefore, the "shortcut" variable, req,
-	 * is used only after we're done with slots.
-	 */
 	while (obj_req->state == ACTIVE)
-		run_active_slot(obj_req->req->slot);
-
-	req = obj_req->req;
+		run_active_slot(obj_req->slot);
 
-	if (req->localfile != -1) {
-		close(req->localfile);
-		req->localfile = -1;
+	if (obj_req->local != -1) {
+		close(obj_req->local);
+		obj_req->local = -1;
 	}
 
 	if (obj_req->state == ABORTED) {
 		ret = error("Request for %s aborted", hex);
-	} else if (req->curl_result != CURLE_OK &&
-		   req->http_code != 416) {
-		if (missing_target(req))
+	} else if (obj_req->curl_result != CURLE_OK &&
+		   obj_req->http_code != 416) {
+		if (missing_target(obj_req))
 			ret = -1; /* Be silent, it is probably in a pack. */
 		else
 			ret = error("%s (curl_result = %d, http_code = %ld, sha1 = %s)",
-				    req->errorstr, req->curl_result,
-				    req->http_code, hex);
-	} else if (req->zret != Z_STREAM_END) {
+				    obj_req->errorstr, obj_req->curl_result,
+				    obj_req->http_code, hex);
+	} else if (obj_req->zret != Z_STREAM_END) {
 		walker->corrupt_object_found++;
-		ret = error("File %s (%s) corrupt", hex, req->url);
-	} else if (hashcmp(obj_req->sha1, req->real_sha1)) {
+		ret = error("File %s (%s) corrupt", hex, obj_req->url);
+	} else if (hashcmp(obj_req->sha1, obj_req->real_sha1)) {
 		ret = error("File %s has bad hash", hex);
-	} else if (req->rename < 0) {
+	} else if (obj_req->rename < 0) {
 		ret = error("unable to write sha1 filename %s",
-			    req->filename);
+			    obj_req->filename);
 	}
 
-	release_http_object_request(req);
 	release_object_request(obj_req);
 	return ret;
 }
diff --git a/http.c b/http.c
index 14d5357..ad7b6f8 100644
--- a/http.c
+++ b/http.c
@@ -12,10 +12,6 @@ static CURLM *curlm;
 #ifndef NO_CURL_EASY_DUPHANDLE
 static CURL *curl_default;
 #endif
-
-#define PREV_BUF_SIZE 4096
-#define RANGE_HEADER_SIZE 30
-
 char curl_errorstr[CURL_ERROR_SIZE];
 
 static int curl_ssl_verify = -1;
@@ -45,7 +41,8 @@ static char *ssl_cert_password;
 static int ssl_cert_password_required;
 
 static struct curl_slist *pragma_header;
-static struct curl_slist *no_pragma_header;
+
+struct curl_slist *no_pragma_header;
 
 static struct active_request_slot *active_queue_head;
 
@@ -715,23 +712,6 @@ static char *quote_ref_url(const char *base, const char *ref)
 	return strbuf_detach(&buf, NULL);
 }
 
-void append_remote_object_url(struct strbuf *buf, const char *url,
-			      const char *hex,
-			      int only_two_digit_prefix)
-{
-	strbuf_addf(buf, "%s/objects/%.*s/", url, 2, hex);
-	if (!only_two_digit_prefix)
-		strbuf_addf(buf, "%s", hex+2);
-}
-
-char *get_remote_object_url(const char *url, const char *hex,
-			    int only_two_digit_prefix)
-{
-	struct strbuf buf = STRBUF_INIT;
-	append_remote_object_url(&buf, url, hex, only_two_digit_prefix);
-	return strbuf_detach(&buf, NULL);
-}
-
 /* http_request() targets */
 #define HTTP_REQUEST_STRBUF	0
 #define HTTP_REQUEST_FILE	1
@@ -1061,231 +1041,3 @@ abort:
 	free(preq);
 	return NULL;
 }
-
-/* Helpers for fetching objects (loose) */
-static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
-			       void *data)
-{
-	unsigned char expn[4096];
-	size_t size = eltsize * nmemb;
-	int posn = 0;
-	struct http_object_request *freq =
-		(struct http_object_request *)data;
-	do {
-		ssize_t retval = xwrite(freq->localfile,
-					(char *) ptr + posn, size - posn);
-		if (retval < 0)
-			return posn;
-		posn += retval;
-	} while (posn < size);
-
-	freq->stream.avail_in = size;
-	freq->stream.next_in = ptr;
-	do {
-		freq->stream.next_out = expn;
-		freq->stream.avail_out = sizeof(expn);
-		freq->zret = git_inflate(&freq->stream, Z_SYNC_FLUSH);
-		git_SHA1_Update(&freq->c, expn,
-				sizeof(expn) - freq->stream.avail_out);
-	} while (freq->stream.avail_in && freq->zret == Z_OK);
-	data_received++;
-	return size;
-}
-
-struct http_object_request *new_http_object_request(const char *base_url,
-	unsigned char *sha1)
-{
-	char *hex = sha1_to_hex(sha1);
-	char *filename;
-	char prevfile[PATH_MAX];
-	int prevlocal;
-	unsigned char prev_buf[PREV_BUF_SIZE];
-	ssize_t prev_read = 0;
-	long prev_posn = 0;
-	char range[RANGE_HEADER_SIZE];
-	struct curl_slist *range_header = NULL;
-	struct http_object_request *freq;
-
-	freq = xmalloc(sizeof(*freq));
-	hashcpy(freq->sha1, sha1);
-	freq->localfile = -1;
-
-	filename = sha1_file_name(sha1);
-	snprintf(freq->filename, sizeof(freq->filename), "%s", filename);
-	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
-		 "%s.temp", filename);
-
-	snprintf(prevfile, sizeof(prevfile), "%s.prev", filename);
-	unlink_or_warn(prevfile);
-	rename(freq->tmpfile, prevfile);
-	unlink_or_warn(freq->tmpfile);
-
-	if (freq->localfile != -1)
-		error("fd leakage in start: %d", freq->localfile);
-	freq->localfile = open(freq->tmpfile,
-			       O_WRONLY | O_CREAT | O_EXCL, 0666);
-	/*
-	 * This could have failed due to the "lazy directory creation";
-	 * try to mkdir the last path component.
-	 */
-	if (freq->localfile < 0 && errno == ENOENT) {
-		char *dir = strrchr(freq->tmpfile, '/');
-		if (dir) {
-			*dir = 0;
-			mkdir(freq->tmpfile, 0777);
-			*dir = '/';
-		}
-		freq->localfile = open(freq->tmpfile,
-				       O_WRONLY | O_CREAT | O_EXCL, 0666);
-	}
-
-	if (freq->localfile < 0) {
-		error("Couldn't create temporary file %s for %s: %s",
-		      freq->tmpfile, freq->filename, strerror(errno));
-		goto abort;
-	}
-
-	memset(&freq->stream, 0, sizeof(freq->stream));
-
-	git_inflate_init(&freq->stream);
-
-	git_SHA1_Init(&freq->c);
-
-	freq->url = get_remote_object_url(base_url, hex, 0);
-
-	/*
-	 * If a previous temp file is present, process what was already
-	 * fetched.
-	 */
-	prevlocal = open(prevfile, O_RDONLY);
-	if (prevlocal != -1) {
-		do {
-			prev_read = xread(prevlocal, prev_buf, PREV_BUF_SIZE);
-			if (prev_read>0) {
-				if (fwrite_sha1_file(prev_buf,
-						     1,
-						     prev_read,
-						     freq) == prev_read) {
-					prev_posn += prev_read;
-				} else {
-					prev_read = -1;
-				}
-			}
-		} while (prev_read > 0);
-		close(prevlocal);
-	}
-	unlink_or_warn(prevfile);
-
-	/*
-	 * Reset inflate/SHA1 if there was an error reading the previous temp
-	 * file; also rewind to the beginning of the local file.
-	 */
-	if (prev_read == -1) {
-		memset(&freq->stream, 0, sizeof(freq->stream));
-		git_inflate_init(&freq->stream);
-		git_SHA1_Init(&freq->c);
-		if (prev_posn>0) {
-			prev_posn = 0;
-			lseek(freq->localfile, 0, SEEK_SET);
-			if (ftruncate(freq->localfile, 0) < 0) {
-				error("Couldn't truncate temporary file %s for %s: %s",
-					  freq->tmpfile, freq->filename, strerror(errno));
-				goto abort;
-			}
-		}
-	}
-
-	freq->slot = get_active_slot();
-
-	curl_easy_setopt(freq->slot->curl, CURLOPT_FILE, freq);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_ERRORBUFFER, freq->errorstr);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_URL, freq->url);
-	curl_easy_setopt(freq->slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
-
-	/*
-	 * If we have successfully processed data from a previous fetch
-	 * attempt, only fetch the data we don't already have.
-	 */
-	if (prev_posn>0) {
-		if (http_is_verbose)
-			fprintf(stderr,
-				"Resuming fetch of object %s at byte %ld\n",
-				hex, prev_posn);
-		sprintf(range, "Range: bytes=%ld-", prev_posn);
-		range_header = curl_slist_append(range_header, range);
-		curl_easy_setopt(freq->slot->curl,
-				 CURLOPT_HTTPHEADER, range_header);
-	}
-
-	return freq;
-
-abort:
-	free(filename);
-	free(freq->url);
-	free(freq);
-	return NULL;
-}
-
-void process_http_object_request(struct http_object_request *freq)
-{
-	if (freq->slot == NULL)
-		return;
-	freq->curl_result = freq->slot->curl_result;
-	freq->http_code = freq->slot->http_code;
-	freq->slot = NULL;
-}
-
-int finish_http_object_request(struct http_object_request *freq)
-{
-	struct stat st;
-
-	close(freq->localfile);
-	freq->localfile = -1;
-
-	process_http_object_request(freq);
-
-	if (freq->http_code == 416) {
-		fprintf(stderr, "Warning: requested range invalid; we may already have all the data.\n");
-	} else if (freq->curl_result != CURLE_OK) {
-		if (stat(freq->tmpfile, &st) == 0)
-			if (st.st_size == 0)
-				unlink_or_warn(freq->tmpfile);
-		return -1;
-	}
-
-	git_inflate_end(&freq->stream);
-	git_SHA1_Final(freq->real_sha1, &freq->c);
-	if (freq->zret != Z_STREAM_END) {
-		unlink_or_warn(freq->tmpfile);
-		return -1;
-	}
-	if (hashcmp(freq->sha1, freq->real_sha1)) {
-		unlink_or_warn(freq->tmpfile);
-		return -1;
-	}
-	freq->rename =
-		move_temp_to_file(freq->tmpfile, freq->filename);
-
-	return freq->rename;
-}
-
-void abort_http_object_request(struct http_object_request *freq)
-{
-	unlink_or_warn(freq->tmpfile);
-
-	release_http_object_request(freq);
-}
-
-void release_http_object_request(struct http_object_request *freq)
-{
-	if (freq->localfile != -1) {
-		close(freq->localfile);
-		freq->localfile = -1;
-	}
-	if (freq->url != NULL) {
-		free(freq->url);
-		freq->url = NULL;
-	}
-	freq->slot = NULL;
-}
diff --git a/http.h b/http.h
index 4c4e99c..511c0c4 100644
--- a/http.h
+++ b/http.h
@@ -88,6 +88,10 @@ extern void add_fill_function(void *data, int (*fill)(void *));
 extern void step_active_slots(void);
 #endif
 
+extern struct curl_slist *no_pragma_header;
+
+#define RANGE_HEADER_SIZE 30
+
 extern void http_init(struct remote *remote);
 extern void http_cleanup(void);
 
@@ -110,13 +114,6 @@ static inline int missing__target(int code, int result)
 
 #define missing_target(a) missing__target((a)->http_code, (a)->curl_result)
 
-/* Helpers for modifying and creating URLs */
-extern void append_remote_object_url(struct strbuf *buf, const char *url,
-				     const char *hex,
-				     int only_two_digit_prefix);
-extern char *get_remote_object_url(const char *url, const char *hex,
-				   int only_two_digit_prefix);
-
 /* Options for http_request_*() */
 #define HTTP_NO_CACHE		1
 
@@ -170,30 +167,4 @@ extern struct http_pack_request *new_http_pack_request(
 extern int finish_http_pack_request(struct http_pack_request *preq);
 extern void release_http_pack_request(struct http_pack_request *preq);
 
-/* Helpers for fetching object */
-struct http_object_request
-{
-	char *url;
-	char filename[PATH_MAX];
-	char tmpfile[PATH_MAX];
-	int localfile;
-	CURLcode curl_result;
-	char errorstr[CURL_ERROR_SIZE];
-	long http_code;
-	unsigned char sha1[20];
-	unsigned char real_sha1[20];
-	git_SHA_CTX c;
-	z_stream stream;
-	int zret;
-	int rename;
-	struct active_request_slot *slot;
-};
-
-extern struct http_object_request *new_http_object_request(
-	const char *base_url, unsigned char *sha1);
-extern void process_http_object_request(struct http_object_request *freq);
-extern int finish_http_object_request(struct http_object_request *freq);
-extern void abort_http_object_request(struct http_object_request *freq);
-extern void release_http_object_request(struct http_object_request *freq);
-
 #endif /* HTTP_H */

Re: git http-push and MKCOL error (22/409)

[Tay Ray Chuan]

Hi,

On Mon, Aug 17, 2009 at 1:25 PM, Thomas Schlichter<thomas.schlichter@web.de> wrote:
> Hmm. I hope you won't take this as an argument for not fixing it, because it
> is a clear regression! I tried git version 1.6.3.4, and it works flawlessly
> with exactly this server! Even during bisecting, some versions worked, some
> didn't (these after the mentioned commit...)
>
> So I think this commit didn't only refactor things, but unintentionally
> changed the behavior. And this must be the problem. As I'm not into this code,
> and the refactoring was not completely trivial, I was not able to quickly find
> the place where the behavior was changed...

Thomas, indeed you're right about this, the commit you noted in your
earlier email (5424bc5) was indeed at fault.

The commit changed how urls was handled, and resulted (incorrectly) in
an extra slash being added to the url. In other words, urls now look
like this:

  http://server/repo.git//objects/2a/

This problem occurs only in http-push.c, because repo urls
(http://server/repo.git) are always made to end with a slash there.

Btw, on my Apache 2.2.x server, this didn't crop up when I tested,
while Thomas worked with a 2.0.x.

I've attached a patch below that hopefully fixes it.

-- >8 --

Subject: [PATCH] http.c: don't assume that urls don't end with slash

Make append_remote_object_url() (and by implication,
get_remote_object_url) use end_url_with_slash() to ensure that the url
ends with a slash.

Previously, they assumed that the url did not end with a slash and
as a result appended a slash, sometimes errorneously.

This fixes an issue introduced in 5424bc5 ("http*: add helper methods
for fetching objects (loose)"), where the append_remote_object_url()
implementation in http-push.c, which assumed that urls end with a
slash, was replaced by another one in http.c, which assumed urls did
not end with a slash.

The above issue was raised by Thomas Schlichter:

  http://marc.info/?l=git&m=125043105231327

Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>
---
 http.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/http.c b/http.c
index 14d5357..eb0c669 100644
--- a/http.c
+++ b/http.c
@@ -719,7 +719,9 @@ void append_remote_object_url(struct strbuf *buf, const char *url,
 			      const char *hex,
 			      int only_two_digit_prefix)
 {
-	strbuf_addf(buf, "%s/objects/%.*s/", url, 2, hex);
+	end_url_with_slash(buf, url);
+
+	strbuf_addf(buf, "objects/%.*s/", 2, hex);
 	if (!only_two_digit_prefix)
 		strbuf_addf(buf, "%s", hex+2);
 }
--
1.6.4.dirty

Re: git http-push and MKCOL error (22/409)

[Thomas Schlichter]

Hi,

Am Montag 17 August 2009 11:09:43 schrieb Tay Ray Chuan:
> Subject: [PATCH] http.c: don't assume that urls don't end with slash
>
> Make append_remote_object_url() (and by implication,
> get_remote_object_url) use end_url_with_slash() to ensure that the url
> ends with a slash.
>
> Previously, they assumed that the url did not end with a slash and
> as a result appended a slash, sometimes errorneously.
>
> This fixes an issue introduced in 5424bc5 ("http*: add helper methods
> for fetching objects (loose)"), where the append_remote_object_url()
> implementation in http-push.c, which assumed that urls end with a
> slash, was replaced by another one in http.c, which assumed urls did
> not end with a slash.
>
> The above issue was raised by Thomas Schlichter:
>
>   http://marc.info/?l=git&m=125043105231327
>
> Signed-off-by: Tay Ray Chuan <rctay89@gmail.com>

Thank you very much!
This patch indeed fixes my problem.

You may add my

   Tested-by: Thomas Schlichter <thomas.schlichter@web.de>

if you'd like to. I don't know how this is handled in the git-tree...

Is there a plan in which version this patch will be released? Will there be a 
1.6.4.1 or is 1.6.5 next?

Kind regards,
Thomas Schlichter