Problem cloning packed-and-pruned http repository

Problem cloning packed-and-pruned http repository

[Panagiotis Issaris]

Hi

I've been having trouble setting up a public repository using GIT. After
I have pushed my repository to a directory within ~/public_html, I can
clone it. But the repository is _big_ (261M).

So, I use "git-repack" on it and a "git-prune-packed". This makes it
nicely fit in 14MiB. If I try to clone this pruned/packed repository
again both cg-clone hangs on it (as does git-clone).

Here's two outputs demonstrating this. One repository was a "cp -lr"
clone [1] of the other, and one was packed/pruned, the other wasn't:
takis@poseidon:/tmp$ cg-clone http://lumumba.uhasselt.be/takis/git/ffmpeg-h264.git
defaulting to local storage area
Fetching head...
Fetching objects...
progress: 38 objects, 159434 bytes
cg-clone: interrupted
takis@poseidon:/tmp$ cg-clone http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git
defaulting to local storage area
Fetching head...
Fetching objects...
Getting alternates list for http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git/
Getting pack list for http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git/
progress: 0 objects, 0 bytes
cg-clone: interrupted

So, I tried tracing it, to see what was going on:
takis@poseidon:/tmp/a$ ps x|grep git
18386 pts/9    S+     0:00 /bin/sh /home/takis/bin/git-clone http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git
18400 pts/9    S+     0:00 git-http-fetch -v -a -w heads/master heads/master http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git/
18416 pts/10   S+     0:00 grep git

takis@poseidon:/tmp/a$ strace -f -p 18400
Process 18400 attached - interrupt to quit
select(0, [], [], [], {0, 48000})       = 0 (Timeout)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0
gettimeofday({1160169868, 454932}, NULL) = 0
gettimeofday({1160169868, 454989}, NULL) = 0
select(0, [], [], [], {0, 50000})       = 0 (Timeout)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0
gettimeofday({1160169868, 506226}, NULL) = 0
gettimeofday({1160169868, 506277}, NULL) = 0
select(0, [], [], [], {0, 50000})       = 0 (Timeout)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0
gettimeofday({1160169868, 558245}, NULL) = 0
gettimeofday({1160169868, 558296}, NULL) = 0
select(0, [], [], [], {0, 50000})       = 0 (Timeout)
poll([{fd=4, events=POLLIN}], 1, 0)     = 0
gettimeofday({1160169868, 610227}, NULL) = 0
gettimeofday({1160169868, 610277}, NULL) = 0
select(0, [], [], [], {0, 50000})       = 0 (Timeout)
...

And this keeps going on... so I can't see any data getting in :-(

Any hints what I might be doing wrong?

Thanks in advance for any replies! :)

With friendly regards,
Takis

[1] I am aware of "git-clone -l -s" but wanted to make fully
independent copy and wasn't sure about any links between them if only
using "-l".
-- 
OpenPGP key: http://lumumba.uhasselt.be/takis/takis_public_key.txt
fingerprint: 6571 13A3 33D9 3726 F728  AA98 F643 B12E ECF3 E029

Re: Problem cloning packed-and-pruned http repository

[Panagiotis Issaris]

Hi

On Fri, Oct 06, 2006 at 11:26:16PM +0200 or thereabouts,  wrote:
> [...]
> I've been having trouble setting up a public repository using GIT. After
> I have pushed my repository to a directory within ~/public_html, I can
> clone it. But the repository is _big_ (261M).
> 
> So, I use "git-repack" on it and a "git-prune-packed". This makes it
> nicely fit in 14MiB. If I try to clone this pruned/packed repository
> again both cg-clone hangs on it (as does git-clone).
> [...] 
> takis@poseidon:/tmp$ cg-clone http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git
> defaulting to local storage area
> Fetching head...
> Fetching objects...
> Getting alternates list for http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git/
> Getting pack list for http://lumumba.uhasselt.be/takis/git/ffmpeg-h264-test.git/
> progress: 0 objects, 0 bytes
> cg-clone: interrupted
> 
Apparently, it does work :-/ After a _long_ time I noticed that the
repository indeed got cloned... I am not sure if this is normal behavior
or not, it seemed to take a _really_ long. I would have thought
downloading 14MiB should not take a long time on my ADSL line.


With friendly regards,
Takis

-- 
OpenPGP key: http://lumumba.uhasselt.be/takis/takis_public_key.txt
fingerprint: 6571 13A3 33D9 3726 F728  AA98 F643 B12E ECF3 E029

Re: Problem cloning packed-and-pruned http repository

[Sean]

On Sat, 7 Oct 2006 00:05:42 +0200
takis@lumumba.uhasselt.be (Panagiotis Issaris) wrote:

> Apparently, it does work :-/ After a _long_ time I noticed that the
> repository indeed got cloned... I am not sure if this is normal behavior
> or not, it seemed to take a _really_ long. I would have thought
> downloading 14MiB should not take a long time on my ADSL line.

It's not normal.  There's something odd going on.  I can clone your
repo with wget in about two minutes, while Git still hadn't downloaded
anything after 12 minutes when I killed it.

Poked around a bit, and found that if I comment out these lines from
http-fetch.c:

#ifndef NO_EXPAT
        if (remote_ls(repo, "objects/pack/", PROCESS_FILES,
                      process_ls_pack, NULL) == 0)
                return 0;
#endif

Then everything downloads nice and fast.  Does anyone have a guess
why that would be?

Sean

[RFC PATCH] Add WEBDAV timeout to http-fetch.

[Sean]

If a server is having problems delivering the Git repo over WEBDAV,
timeout after two minutes so that a regular http transfer can
be tried.

---

 http-fetch.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

Not sure if this is the correct fix, but it should improve the situation
for cloning and fetching from servers like Takis's.  When connecting to
his server WEBDAV doesn't respond after the initial connection.  Nothing
proceeds until the OS connection times out many minutes later.

This patch sets the CURL timeout to two minutes so that things proceed
sooner.  Even with this patch it takes two extra minutes of "dead time"
to complete all operations; obivously this still sucks.

However, I don't know if the two minute timeout is long enough for
all cases with a server where WEBDAV is functioning properly.
Hopefully someone who knows more about Curl can comment and perhaps
offer another solution.

Maybe the real solution is just to figure out and fix whatever is
going on with the WEBDAV server and forget this patch.

Sean


diff --git a/http-fetch.c b/http-fetch.c
index bc74f30..046245a 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -861,6 +861,7 @@ static int remote_ls(struct alt_base *re
 	curl_easy_setopt(slot->curl, CURLOPT_UPLOAD, 1);
 	curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, DAV_PROPFIND);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
+	curl_easy_setopt(slot->curl, CURLOPT_TIMEOUT, 120);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
-- 
1.4.2.3.gabd697

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Junio C Hamano]

Sean <seanlkml@sympatico.ca> writes:

> If a server is having problems delivering the Git repo over WEBDAV,
> timeout after two minutes so that a regular http transfer can
> be tried.
>
> ---
>
>  http-fetch.c |    1 +
>  1 files changed, 1 insertions(+), 0 deletions(-)
>
> Not sure if this is the correct fix, but it should improve the situation
> for cloning and fetching from servers like Takis's.  When connecting to
> his server WEBDAV doesn't respond after the initial connection.  Nothing
> proceeds until the OS connection times out many minutes later.
>
> This patch sets the CURL timeout to two minutes so that things proceed
> sooner.  Even with this patch it takes two extra minutes of "dead time"
> to complete all operations; obivously this still sucks.
>
> However, I don't know if the two minute timeout is long enough for
> all cases with a server where WEBDAV is functioning properly.
> Hopefully someone who knows more about Curl can comment and perhaps
> offer another solution.
>
> Maybe the real solution is just to figure out and fix whatever is
> going on with the WEBDAV server and forget this patch.

I think it is prudent to protect the client from a broken server
and it is independent from "fixing" the server side.  It would
perhaps make sense to make this overridable somehow but I am not
sure how -- .git/config is too global (the problem would be per
remote site), and having the user to set environment variable
only when going to specific server is too cumbersome to be
useful.  This ideally should be a per-remote configuration
item.

> diff --git a/http-fetch.c b/http-fetch.c
> index bc74f30..046245a 100644
> --- a/http-fetch.c
> +++ b/http-fetch.c
> @@ -861,6 +861,7 @@ static int remote_ls(struct alt_base *re
>  	curl_easy_setopt(slot->curl, CURLOPT_UPLOAD, 1);
>  	curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, DAV_PROPFIND);
>  	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
> +	curl_easy_setopt(slot->curl, CURLOPT_TIMEOUT, 120);
>  
>  	if (start_active_slot(slot)) {
>  		run_active_slot(slot);
> -- 
> 1.4.2.3.gabd697

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Jakub Narebski]

Junio C Hamano wrote:

> I think it is prudent to protect the client from a broken server
> and it is independent from "fixing" the server side.  It would
> perhaps make sense to make this overridable somehow but I am not
> sure how -- .git/config is too global (the problem would be per
> remote site), and having the user to set environment variable
> only when going to specific server is too cumbersome to be
> useful.  This ideally should be a per-remote configuration
> item.

Perhaps use undocumented (hint, hint! for whoever did code this)
per-user ~/.gitconfig?
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Johannes Schindelin]

Hi,

On Sat, 7 Oct 2006, Jakub Narebski wrote:

> Perhaps use undocumented (hint, hint! for whoever did code this)
> per-user ~/.gitconfig?

A good idea to use this (hint, hint! whoever finds out how it works can 
document it as well) feature.

HOWEVER, Junio pointed out that he'd like a finer grain than per-repo, and 
.gitconfig is a coarser one! (BTW why do you strip Junio from your Cc: 
when you respond directly to his email?)

Ciao,
Dscho

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Petr Baudis]

(This post's gonna be really strange.)

Dear diary, on Sat, Oct 07, 2006 at 07:32:25PM CEST, I got a letter
where Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...
> On Sat, 7 Oct 2006, Jakub Narebski wrote:
> 
> > Perhaps use undocumented (hint, hint! for whoever did code this)
> > per-user ~/.gitconfig?
> 
> A good idea to use this (hint, hint! whoever finds out how it works can 
> document it as well) feature.

(Which is no excuse for the initial implementor not documenting it,
though.)

> HOWEVER, Junio pointed out that he'd like a finer grain than per-repo, and 
> .gitconfig is a coarser one!

(I honestly don't think it's worth it at all to make this configurable.
What's the point?)

> (BTW why do you strip Junio from your Cc: when you respond directly to
> his email?)

Jakub, what about putting a mini-FAQ to your signature? ;-)

It would be nice if gmane supported honoring mail-followup-to when
gatewaying posts back to the mailing list (of course it would need to do
some smart anti-spam protection).

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Petr Baudis]

Dear diary, on Sat, Oct 07, 2006 at 07:32:25PM CEST, I got a letter
where Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...
> Hi,
> 
> On Sat, 7 Oct 2006, Jakub Narebski wrote:
> 
> > Perhaps use undocumented (hint, hint! for whoever did code this)
> > per-user ~/.gitconfig?
> 
> A good idea to use this (hint, hint! whoever finds out how it works can 
> document it as well) feature.
> 
> HOWEVER, Junio pointed out that he'd like a finer grain than per-repo, and 
> .gitconfig is a coarser one!

Actually, that doesn't matter. The point is that it is of _different_
shape than this division. It's per remote server, even spanning several
repositories. So you want to be able to set it up per the server, at the
most global place possible for a regular user, so ~/.gitconfig might be
good idea.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
#!/bin/perl -sp0777i<X+d*lMLa^*lN%0]dsXx++lMlN/dsM0<j]dsj
$/=unpack('H*',$_);$_=`echo 16dio\U$k"SK$/SM$n\EsN0p[lN*1
lK[d2%Sa2/d0$^Ixp"|dc`;s/\W//g;$_=pack('H*',/((..)*)$/)

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Johannes Schindelin]

Hi,

On Sun, 8 Oct 2006, Petr Baudis wrote:

> Dear diary, on Sat, Oct 07, 2006 at 07:32:25PM CEST, I got a letter
> where Johannes Schindelin <Johannes.Schindelin@gmx.de> said that...
>
> > HOWEVER, Junio pointed out that he'd like a finer grain than per-repo, and 
> > .gitconfig is a coarser one!
> 
> Actually, that doesn't matter. The point is that it is of _different_
> shape than this division. It's per remote server, even spanning several
> repositories. So you want to be able to set it up per the server, at the
> most global place possible for a regular user, so ~/.gitconfig might be
> good idea.

Actually, I do not think that anybody in her right mind would set this to 
different values for different repos or servers.

I _know_ that if I hit that very problem, the next thing I'd do is set the 
timeout to 5 seconds _globally_.

Ciao,
Dscho

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Actually, I do not think that anybody in her right mind would set this to 
> different values for different repos or servers.
>
> I _know_ that if I hit that very problem, the next thing I'd do is set the 
> timeout to 5 seconds _globally_.

Let's step back a bit.

The DAV request in question the one to remote_ls() in
http-fetch.c, which tries to read directly from objects/pack/
instead of using objects/info/packs, and it does not matter if
DAV request fails because it would fall back to non-DAV anyway.

Using DAV, if it works with the server, has the advantage of not
having to keep objects/info/packs up-to-date from repository
owner's point of view.  But the repository owner ends up keeping
up-to-date as a side effect of keeping info/refs up-to-date
anyway (as I do not see a code to read that information over
DAV), so there is no point doing this over DAV in practice.

Perhaps we should remove call to remote_ls() from
fetch_indices() unconditionally, not just protected with
NO_EXPAT and be done with it?

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Johannes Schindelin]

Hi,

On Sat, 7 Oct 2006, Junio C Hamano wrote:

> [...] But the repository owner ends up keeping up-to-date as a side 
> effect of keeping info/refs up-to-date anyway (as I do not see a code to 
> read that information over DAV), so there is no point doing this over 
> DAV in practice.

Makes sense to me.

Ciao,
Dscho

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Sean]

On Sat, 07 Oct 2006 21:52:02 -0700
Junio C Hamano <junkio@cox.net> wrote:

> Using DAV, if it works with the server, has the advantage of not
> having to keep objects/info/packs up-to-date from repository
> owner's point of view.  But the repository owner ends up keeping
> up-to-date as a side effect of keeping info/refs up-to-date
> anyway (as I do not see a code to read that information over
> DAV), so there is no point doing this over DAV in practice.
> 
> Perhaps we should remove call to remote_ls() from
> fetch_indices() unconditionally, not just protected with
> NO_EXPAT and be done with it?

That makes a lot of sense.  A server really has to always provide
a objects/info/packs anyway, just to be fetchable today by clients
that are compiled with NO_EXPAT.

+1

Sean

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Junio C Hamano]

Sean <seanlkml@sympatico.ca> writes:

> On Sat, 07 Oct 2006 21:52:02 -0700
> Junio C Hamano <junkio@cox.net> wrote:
>
>> Using DAV, if it works with the server, has the advantage of not
>> having to keep objects/info/packs up-to-date from repository
>> owner's point of view.  But the repository owner ends up keeping
>> up-to-date as a side effect of keeping info/refs up-to-date
>> anyway (as I do not see a code to read that information over
>> DAV), so there is no point doing this over DAV in practice.
>> 
>> Perhaps we should remove call to remote_ls() from
>> fetch_indices() unconditionally, not just protected with
>> NO_EXPAT and be done with it?
>
> That makes a lot of sense.  A server really has to always provide
> a objects/info/packs anyway, just to be fetchable today by clients
> that are compiled with NO_EXPAT.

And even for an isolated group where everybody knows that
everybody else runs DAV-enabled clients, they need info/refs
prepared for ls-remote and git-fetch script, which means you
will run update-server-info to keep objects/info/packs up to
date.

Nick, do you see holes in my logic?

-- >8 --
http-fetch.c: drop remote_ls()

While doing remote_ls() over DAV potentially allows the server
side not to keep objects/info/pack up-to-date, misconfigured or
buggy servers can silently ignore or not to respond to DAV
requests and makes the client hang.

The server side (unfortunately) needs to run git-update-server-info
even if remote_ls() removes the need to keep objects/info/pack file
up-to-date, because the caller of git-http-fetch (git-fetch) and other
clients that interact with the repository (e.g. git-ls-remote) need to
read from info/refs file (there is no code to make that unnecessary by
using DAV yet).

Perhaps the right solution in the longer-term is to make info/refs
also unnecessary by using DAV, and we would want to resurrect the
code this patch removes when we do so, but let's drop remote_ls()
implementation for now.  It is causing problems without really
helping anything yet.

git will keep it for us until we need it next time.

Signed-off-by: Junio C Hamano <junkio@cox.net>
---

diff --git a/http-fetch.c b/http-fetch.c
index 8f251e7..396552d 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -4,35 +4,6 @@ #include "pack.h"
 #include "fetch.h"
 #include "http.h"
 
-#ifndef NO_EXPAT
-#include <expat.h>
-
-/* Definitions for DAV requests */
-#define DAV_PROPFIND "PROPFIND"
-#define DAV_PROPFIND_RESP ".multistatus.response"
-#define DAV_PROPFIND_NAME ".multistatus.response.href"
-#define DAV_PROPFIND_COLLECTION ".multistatus.response.propstat.prop.resourcetype.collection"
-#define PROPFIND_ALL_REQUEST "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n<D:propfind xmlns:D=\"DAV:\">\n<D:allprop/>\n</D:propfind>"
-
-/* Definitions for processing XML DAV responses */
-#ifndef XML_STATUS_OK
-enum XML_Status {
-  XML_STATUS_OK = 1,
-  XML_STATUS_ERROR = 0
-};
-#define XML_STATUS_OK    1
-#define XML_STATUS_ERROR 0
-#endif
-
-/* Flags that control remote_ls processing */
-#define PROCESS_FILES (1u << 0)
-#define PROCESS_DIRS  (1u << 1)
-#define RECURSIVE     (1u << 2)
-
-/* Flags that remote_ls passes to callback functions */
-#define IS_DIR (1u << 0)
-#endif
-
 #define PREV_BUF_SIZE 4096
 #define RANGE_HEADER_SIZE 30
 
@@ -90,30 +61,6 @@ struct alternates_request {
 	int http_specific;
 };
 
-#ifndef NO_EXPAT
-struct xml_ctx
-{
-	char *name;
-	int len;
-	char *cdata;
-	void (*userFunc)(struct xml_ctx *ctx, int tag_closed);
-	void *userData;
-};
-
-struct remote_ls_ctx
-{
-	struct alt_base *repo;
-	char *path;
-	void (*userFunc)(struct remote_ls_ctx *ls);
-	void *userData;
-	int flags;
-	char *dentry_name;
-	int dentry_flags;
-	int rc;
-	struct remote_ls_ctx *parent;
-};
-#endif
-
 static struct object_request *object_queue_head;
 
 static size_t fwrite_sha1_file(void *ptr, size_t eltsize, size_t nmemb,
@@ -714,193 +661,6 @@ #endif
 	free(url);
 }
 
-#ifndef NO_EXPAT
-static void
-xml_start_tag(void *userData, const char *name, const char **atts)
-{
-	struct xml_ctx *ctx = (struct xml_ctx *)userData;
-	const char *c = strchr(name, ':');
-	int new_len;
-
-	if (c == NULL)
-		c = name;
-	else
-		c++;
-
-	new_len = strlen(ctx->name) + strlen(c) + 2;
-
-	if (new_len > ctx->len) {
-		ctx->name = xrealloc(ctx->name, new_len);
-		ctx->len = new_len;
-	}
-	strcat(ctx->name, ".");
-	strcat(ctx->name, c);
-
-	free(ctx->cdata);
-	ctx->cdata = NULL;
-
-	ctx->userFunc(ctx, 0);
-}
-
-static void
-xml_end_tag(void *userData, const char *name)
-{
-	struct xml_ctx *ctx = (struct xml_ctx *)userData;
-	const char *c = strchr(name, ':');
-	char *ep;
-
-	ctx->userFunc(ctx, 1);
-
-	if (c == NULL)
-		c = name;
-	else
-		c++;
-
-	ep = ctx->name + strlen(ctx->name) - strlen(c) - 1;
-	*ep = 0;
-}
-
-static void
-xml_cdata(void *userData, const XML_Char *s, int len)
-{
-	struct xml_ctx *ctx = (struct xml_ctx *)userData;
-	free(ctx->cdata);
-	ctx->cdata = xmalloc(len + 1);
-	strlcpy(ctx->cdata, s, len + 1);
-}
-
-static int remote_ls(struct alt_base *repo, const char *path, int flags,
-		     void (*userFunc)(struct remote_ls_ctx *ls),
-		     void *userData);
-
-static void handle_remote_ls_ctx(struct xml_ctx *ctx, int tag_closed)
-{
-	struct remote_ls_ctx *ls = (struct remote_ls_ctx *)ctx->userData;
-
-	if (tag_closed) {
-		if (!strcmp(ctx->name, DAV_PROPFIND_RESP) && ls->dentry_name) {
-			if (ls->dentry_flags & IS_DIR) {
-				if (ls->flags & PROCESS_DIRS) {
-					ls->userFunc(ls);
-				}
-				if (strcmp(ls->dentry_name, ls->path) &&
-				    ls->flags & RECURSIVE) {
-					ls->rc = remote_ls(ls->repo,
-							   ls->dentry_name,
-							   ls->flags,
-							   ls->userFunc,
-							   ls->userData);
-				}
-			} else if (ls->flags & PROCESS_FILES) {
-				ls->userFunc(ls);
-			}
-		} else if (!strcmp(ctx->name, DAV_PROPFIND_NAME) && ctx->cdata) {
-			ls->dentry_name = xmalloc(strlen(ctx->cdata) -
-						  ls->repo->path_len + 1);
-			strcpy(ls->dentry_name, ctx->cdata + ls->repo->path_len);
-		} else if (!strcmp(ctx->name, DAV_PROPFIND_COLLECTION)) {
-			ls->dentry_flags |= IS_DIR;
-		}
-	} else if (!strcmp(ctx->name, DAV_PROPFIND_RESP)) {
-		free(ls->dentry_name);
-		ls->dentry_name = NULL;
-		ls->dentry_flags = 0;
-	}
-}
-
-static int remote_ls(struct alt_base *repo, const char *path, int flags,
-		     void (*userFunc)(struct remote_ls_ctx *ls),
-		     void *userData)
-{
-	char *url = xmalloc(strlen(repo->base) + strlen(path) + 1);
-	struct active_request_slot *slot;
-	struct slot_results results;
-	struct buffer in_buffer;
-	struct buffer out_buffer;
-	char *in_data;
-	char *out_data;
-	XML_Parser parser = XML_ParserCreate(NULL);
-	enum XML_Status result;
-	struct curl_slist *dav_headers = NULL;
-	struct xml_ctx ctx;
-	struct remote_ls_ctx ls;
-
-	ls.flags = flags;
-	ls.repo = repo;
-	ls.path = xstrdup(path);
-	ls.dentry_name = NULL;
-	ls.dentry_flags = 0;
-	ls.userData = userData;
-	ls.userFunc = userFunc;
-	ls.rc = 0;
-
-	sprintf(url, "%s%s", repo->base, path);
-
-	out_buffer.size = strlen(PROPFIND_ALL_REQUEST);
-	out_data = xmalloc(out_buffer.size + 1);
-	snprintf(out_data, out_buffer.size + 1, PROPFIND_ALL_REQUEST);
-	out_buffer.posn = 0;
-	out_buffer.buffer = out_data;
-
-	in_buffer.size = 4096;
-	in_data = xmalloc(in_buffer.size);
-	in_buffer.posn = 0;
-	in_buffer.buffer = in_data;
-
-	dav_headers = curl_slist_append(dav_headers, "Depth: 1");
-	dav_headers = curl_slist_append(dav_headers, "Content-Type: text/xml");
-
-	slot = get_active_slot();
-	slot->results = &results;
-	curl_easy_setopt(slot->curl, CURLOPT_INFILE, &out_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_INFILESIZE, out_buffer.size);
-	curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, fread_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_FILE, &in_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
-	curl_easy_setopt(slot->curl, CURLOPT_URL, url);
-	curl_easy_setopt(slot->curl, CURLOPT_UPLOAD, 1);
-	curl_easy_setopt(slot->curl, CURLOPT_CUSTOMREQUEST, DAV_PROPFIND);
-	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, dav_headers);
-
-	if (start_active_slot(slot)) {
-		run_active_slot(slot);
-		if (results.curl_result == CURLE_OK) {
-			ctx.name = xcalloc(10, 1);
-			ctx.len = 0;
-			ctx.cdata = NULL;
-			ctx.userFunc = handle_remote_ls_ctx;
-			ctx.userData = &ls;
-			XML_SetUserData(parser, &ctx);
-			XML_SetElementHandler(parser, xml_start_tag,
-					      xml_end_tag);
-			XML_SetCharacterDataHandler(parser, xml_cdata);
-			result = XML_Parse(parser, in_buffer.buffer,
-					   in_buffer.posn, 1);
-			free(ctx.name);
-
-			if (result != XML_STATUS_OK) {
-				ls.rc = error("XML error: %s",
-					      XML_ErrorString(
-						      XML_GetErrorCode(parser)));
-			}
-		} else {
-			ls.rc = -1;
-		}
-	} else {
-		ls.rc = error("Unable to start PROPFIND request");
-	}
-
-	free(ls.path);
-	free(url);
-	free(out_data);
-	free(in_buffer.buffer);
-	curl_slist_free_all(dav_headers);
-
-	return ls.rc;
-}
-
-#endif
-
 static int fetch_indices(struct alt_base *repo)
 {
 	unsigned char sha1[20];

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Panagiotis Issaris]

Hi,

Junio C Hamano <junkio <at> cox.net> writes:
>[...]
> And even for an isolated group where everybody knows that
> everybody else runs DAV-enabled clients, they need info/refs
> prepared for ls-remote and git-fetch script, which means you
> will run update-server-info to keep objects/info/packs up to
> date.
This patch worked excellent for me. Thanks! :) Any chance this might make it
into your tree? I recommended GIT to the FFmpeg developers and would like to be
able to recommend them to use GIT without any extra patches.

> Nick, do you see holes in my logic?

With friendly regards,
Takis 

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Johannes Schindelin]

Hi,

On Wed, 11 Oct 2006, Panagiotis Issaris wrote:

> Hi,
> 
> Junio C Hamano <junkio <at> cox.net> writes:
> >[...]
> > And even for an isolated group where everybody knows that
> > everybody else runs DAV-enabled clients, they need info/refs
> > prepared for ls-remote and git-fetch script, which means you
> > will run update-server-info to keep objects/info/packs up to
> > date.
> This patch worked excellent for me. Thanks! :) Any chance this might 
> make it into your tree?

It already is: adc446fe5d5f6dc1fb5edeaa9aa016ef94e70da1 from Sun Oct 8 
12:56:19 2006 -0700 in branch 'next'.

Hth,
Dscho

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Panagiotis Issaris]

Hi,

On Sat, Oct 07, 2006 at 03:15:58AM -0700, Junio C Hamano wrote:
> > This patch sets the CURL timeout to two minutes so that things proceed
> > sooner.  Even with this patch it takes two extra minutes of "dead time"
> > to complete all operations; obivously this still sucks.
> >
> > However, I don't know if the two minute timeout is long enough for
> > all cases with a server where WEBDAV is functioning properly.
> > Hopefully someone who knows more about Curl can comment and perhaps
> > offer another solution.
> >
> > Maybe the real solution is just to figure out and fix whatever is
> > going on with the WEBDAV server and forget this patch.
> 
> I think it is prudent to protect the client from a broken server
> and it is independent from "fixing" the server side.  It would
>[...]
Wouldn't most users ctrl-c the program before the two minute timeout occurs?
Especially since their appears to be nothing happening?

With friendly regards,
Takis
 

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Junio C Hamano]

Panagiotis Issaris <takis.issaris@uhasselt.be> writes:

>> > Maybe the real solution is just to figure out and fix whatever is
>> > going on with the WEBDAV server and forget this patch.
>> 
>> I think it is prudent to protect the client from a broken server
>> and it is independent from "fixing" the server side.  It would
>>[...]
> Wouldn't most users ctrl-c the program before the two minute timeout occurs?
> Especially since their appears to be nothing happening?

I think we are talking about the same thing -- after you kill it
with C-c you would want to work it around.  The question is how?

You would want to re-run with timeout value of 0 second (or
WEBDAV disabled, if we can have such an option); if it works
then you would want the tool to remember that you want to use
that particular settings but probably only when talking to that
remote.

Re: [RFC PATCH] Add WEBDAV timeout to http-fetch.

[Panagiotis Issaris]

Hi,

Thanks for your quick reply!

On Fri, Oct 06, 2006 at 10:04:01PM -0400 or thereabouts, Sean wrote:
> If a server is having problems delivering the Git repo over WEBDAV,
> timeout after two minutes so that a regular http transfer can
> be tried.
> [...]
> Maybe the real solution is just to figure out and fix whatever is
> going on with the WEBDAV server and forget this patch.
I had a quick glance at the Apache2 config on this server:
grep -i dav /etc/apache2/apache2.conf
# redirects for folders with DAV methods.
BrowserMatch "^WebDAVFS/1.[012]" redirect-carefully

Google showed me some bugs occurring on other software where they said
the problems were happening because the redirect weren't handled
correctly. I haven't got a clue about this, but thought mentioning it
might ring someone's bell :)

With friendly regards,
Takis
-- 
OpenPGP key: http://lumumba.luc.ac.be/takis/takis_public_key.txt
fingerprint: 6571 13A3 33D9 3726 F728  AA98 F643 B12E ECF3 E029