* Re: [PATCH/RFC] Allow curl to rewind the RPC read buffer
From: Shawn O. Pearce @ 2009-12-01 16:01 UTC (permalink / raw)
To: Martin Storsj?
Cc: Tay Ray Chuan, git, Nicholas Miell, gsky51, Clemens Buchacher,
Mark Lodato, Johannes Schindelin
In-Reply-To: <alpine.DEB.2.00.0912011232450.5582@cone.home.martin.st>
Martin Storsj? <martin@martin.st> wrote:
> When using multi-pass authentication methods, the curl library may need
> to rewind the read buffers used for providing data to HTTP POST, if data
> has been output before a 401 error is received.
In theory, since the cURL session stays active, we would have
received the 401 authentication error during the initial
"GET $GIT_DIR/info/refs?service=git-$service" request, and the subsequent
"POST $GIT_DIR/git-$service" requests would automatically include the
authentication data.
That's theory. Reality doesn't always agree with my theories. :-)
> remote-curl.c | 30 ++++++++++++++++++++++++++++++
> 1 files changed, 30 insertions(+), 0 deletions(-)
Acked-by: Shawn O. Pearce <spearce@spearce.org>
--
Shawn.
^ permalink raw reply
* Re: [PATCH] get_ref_states: strdup entries and free util in stale list
From: Bert Wesarg @ 2009-12-01 15:53 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: Junio C Hamano, Jay Soffian, git
In-Reply-To: <36ca99e90912010105r428a7bfdw63928e8a5515bd1d@mail.gmail.com>
On Tue, Dec 1, 2009 at 10:05, Bert Wesarg <bert.wesarg@googlemail.com> wrote:
> On Tue, Dec 1, 2009 at 09:35, Johannes Schindelin
>> Thanks. I trust you ran the test suite with valgrind just to make sure?
> Not the test suite. But my use case where I found the problem (Ie.
> cut-off branch names) which was 'git remote show <remote>'.
> There are still invalid reads of size 4. I think the problem is the
> flex array member of 'struct ref' and strlen(). If its worth I can
> look into this.
I need this new suppression to run the test suite:
diff --git i/t/valgrind/default.supp w/t/valgrind/default.supp
index 9e013fa..39b080a 100644
--- i/t/valgrind/default.supp
+++ w/t/valgrind/default.supp
@@ -43,3 +43,10 @@
fun:write_buffer
fun:write_loose_object
}
+
+{
+ writing-data-from-zlib-triggers-even-more-errors-2
+ Memcheck:Param
+ write(buf)
+ obj:*libpthread-*.so
+}
Bert
^ permalink raw reply related
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Erik Faye-Lund @ 2009-12-01 15:50 UTC (permalink / raw)
To: Thomas Singer; +Cc: Johannes Sixt, git
In-Reply-To: <4B15391C.5090302@syntevo.com>
On Tue, Dec 1, 2009 at 4:41 PM, Thomas Singer <thomas.singer@syntevo.com> wrote:
> Johannes Sixt wrote:
>> Thomas Singer schrieb:
>>> I'm quite surprised, that -- as I
>>> understand you -- msys-Git (or Git at all?) is not able to handle all
>>> characters (aka unicode) at the same time. I expected it would be better
>>> than older tools, e.g. SVN.
>>
>> This has been discussed at length here and in the msysgit mailing list.
>> Git expects that the file system returns file names with the same byte
>> sequence that git used to create a file. On Windows, this works only as
>> long as you do not switch the codepage.
>
> Now you confuse me: is this a problem of Windows, Git using a less capable
> Windows-API call or is there no unicode-capable API call to list file names
> on Windows? I ask myself how Java does it in its internals, finally it
> (also) consists of a C-base, I guess.
>
Git uses the 8-bit file APIs, and Windows doesn't support setting
UTF-8 as the locale. Some work have been done in msysGit to use
_wopen() and friends instead, but AFAIK it's not completed. See the
branch called "work/utf-filenames" in
git://repo.or.cz/git/mingw/4msysgit.git if you are interested in
helping to complete it.
--
Erik "kusma" Faye-Lund
^ permalink raw reply
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Thomas Singer @ 2009-12-01 15:41 UTC (permalink / raw)
To: Johannes Sixt; +Cc: git
In-Reply-To: <4B151782.8050309@viscovery.net>
Johannes Sixt wrote:
> Thomas Singer schrieb:
>> I'm quite surprised, that -- as I
>> understand you -- msys-Git (or Git at all?) is not able to handle all
>> characters (aka unicode) at the same time. I expected it would be better
>> than older tools, e.g. SVN.
>
> This has been discussed at length here and in the msysgit mailing list.
> Git expects that the file system returns file names with the same byte
> sequence that git used to create a file. On Windows, this works only as
> long as you do not switch the codepage.
Now you confuse me: is this a problem of Windows, Git using a less capable
Windows-API call or is there no unicode-capable API call to list file names
on Windows? I ask myself how Java does it in its internals, finally it
(also) consists of a C-base, I guess.
--
Tom
^ permalink raw reply
* Re: Umlaut in filename makes troubles
From: Michael J Gruber @ 2009-12-01 14:10 UTC (permalink / raw)
To: Jochen; +Cc: git
In-Reply-To: <hf2pbp$rss$1@ger.gmane.org>
Jochen venit, vidit, dixit 01.12.2009 10:58:
> I found another strange effect...
>
> I made a file with "touch aöäü.txt" and from "git status" I get
> # "a\303\266\303\244\303\274.txt"
> reported as untracked. But when I start "git gui" I get file displayed with
> it's correct name...
>
Does this change after
git config core.quotepath false
Michael
^ permalink raw reply
* [RFC PATCH 4/8] Support remote helpers implementing smart transports
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
Documentation/git-remote-helpers.txt | 30 ++++++++++-
transport-helper.c | 94 +++++++++++++++++++++++++++++++--
transport.c | 21 ++++++++
transport.h | 5 ++
4 files changed, 143 insertions(+), 7 deletions(-)
diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index 5cfdc0c..adf815c 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -90,6 +90,28 @@ Supported if the helper has the "push" capability.
+
Supported if the helper has the "import" capability.
+'connect-r' <service>::
+ Connects to given service. Stdin and stdout of helper are
+ connected to specified service (no git or git- prefixes are used,
+ so e.g. fetching uses 'upload-pack' as service) on remote side.
+ Valid replies to this command are 'OK' (connection established),
+ 'FALLBACK' (no smart transport support, fall back to dumb
+ transports) and 'ERROR' (can't connect, don't bother trying to
+ fall back). After line feed terminating the OK response, the
+ output of service starts. After the connection ends, the remote
+ helper exits. Note that to prevent deadlocking, all read data
+ should be immediately flushed to outgoing connection.
++
+Supported if the helper has the "connect-r" capability. Not used if
+helper has the "invoke-r" capability, as invoke is preferred to connect.
+
+'invoke-r' <cmdlength> <cmd>::
+ Like connect-r command, but instead of service name, command
+ line is given. The length of command field is given in command
+ length field.
++
+Supported if the helper has the "invoke-r" capability.
+
If a fatal error occurs, the program writes the error message to
stderr and exits. The caller should expect that a suitable error
message has been printed if the child closes the connection without
@@ -123,6 +145,12 @@ CAPABILITIES
all, it must cover all refs reported by the list command; if
it is not used, it is effectively "*:*"
+'connect-r'::
+ This helper supports the 'connect-r' command.
+
+'invoke-r'::
+ This helper supports the 'invoke-r' command.
+
REF LIST ATTRIBUTES
-------------------
@@ -167,7 +195,7 @@ OPTIONS
Documentation
-------------
-Documentation by Daniel Barkalow.
+Documentation by Daniel Barkalow and Ilari Liusvaara
GIT
---
diff --git a/transport-helper.c b/transport-helper.c
index 777ecbb..0e4da79 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -16,7 +16,9 @@ struct helper_data
unsigned fetch : 1,
import : 1,
option : 1,
- push : 1;
+ push : 1,
+ connect_r : 1,
+ invoke_r : 1;
/* These go from remote name (as in "list") to private name */
struct refspec *refspecs;
int refspec_nr;
@@ -75,7 +77,10 @@ static struct child_process *get_helper(struct transport *transport)
would run and print totally inapporiate error message. */
strbuf_addf(&buf, "git-remote-%s", data->name);
helper->argv[0] = strbuf_detach(&buf, NULL);
- helper->argv[1] = transport->remote->name;
+ if(transport->remote)
+ helper->argv[1] = transport->remote->name;
+ else
+ helper->argv[1] = "";
helper->argv[2] = remove_ext_force(transport->url);
helper->git_cmd = 0;
if (start_command(helper))
@@ -125,6 +130,10 @@ static struct child_process *get_helper(struct transport *transport)
refspec_alloc);
refspecs[refspec_nr++] = strdup(buf.buf + strlen("refspec "));
}
+ if (!strcmp(buf.buf, "connect-r"))
+ data->connect_r = 1;
+ if (!strcmp(buf.buf, "invoke-r"))
+ data->invoke_r = 1;
}
if (refspecs) {
int i;
@@ -344,12 +353,83 @@ static int fetch_with_import(struct transport *transport,
return 0;
}
+static int _process_connect_or_invoke(struct transport *transport,
+ const char *name, const char *exec)
+{
+ struct helper_data *data = transport->data;
+ struct strbuf cmdbuf = STRBUF_INIT;
+ struct child_process *helper;
+
+ helper = get_helper(transport);
+
+ if(data->invoke_r) {
+ strbuf_addf(&cmdbuf, "invoke-r %i %s\n",
+ strlen(exec), exec);
+ } else if(data->connect_r) {
+ strbuf_addf(&cmdbuf, "connect-r %s\n", name);
+ } else
+ return 0;
+
+ write_in_full(helper->in, cmdbuf.buf, cmdbuf.len);
+ strbuf_reset(&cmdbuf);
+ if (strbuf_getline(&cmdbuf, data->out, '\n') == EOF)
+ exit(128); /* child died, message supplied already */
+ if(!strcmp(cmdbuf.buf, "OK"))
+ return 1;
+ else if(!strcmp(cmdbuf.buf, "FALLBACK"))
+ return 0;
+ else if(!strcmp(cmdbuf.buf, "ERROR"))
+ exit(128); /* Error already suppiled. */
+ else
+ die("Unknown response to invoke/connect: %s",
+ cmdbuf.buf);
+
+ return 0; /* Shouldn't be here. */
+}
+
+static int process_connect_or_invoke(struct transport* transport,
+ int for_push)
+{
+ struct helper_data *data = transport->data;
+ const char *name;
+ const char *exec;
+
+ name = for_push ? "receive-pack" : "upload-pack";
+ if(for_push)
+ exec = data->gitoptions.receivepack;
+ else
+ exec = data->gitoptions.uploadpack;
+
+ return _process_connect_or_invoke(transport, name, exec);
+}
+
+static int connect_helper(struct transport *transport, const char *name,
+ const char *exec, int fd[2])
+{
+ struct helper_data *data = transport->data;
+
+ /* Get_helper so invoke_r and connect_r are inited. */
+ get_helper(transport);
+ if(!data->invoke_r && !data->connect_r)
+ die("Operation not supported by protocol.");
+
+ if(!_process_connect_or_invoke(transport, name, exec))
+ die("Can't connect to subservice %s.", name);
+
+ fd[0] = data->helper->out;
+ fd[1] = data->helper->in;
+ return 0;
+}
+
static int fetch(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct helper_data *data = transport->data;
int i, count;
+ if(process_connect_or_invoke(transport, 0))
+ return TRANSPORT_LAYER6_READY;
+
count = 0;
for (i = 0; i < nr_heads; i++)
if (!(to_fetch[i]->status & REF_STATUS_UPTODATE))
@@ -377,6 +457,9 @@ static int push_refs(struct transport *transport,
struct child_process *helper;
struct ref *ref;
+ if(process_connect_or_invoke(transport, 1))
+ return TRANSPORT_LAYER6_READY;
+
if (!remote_refs)
return 0;
@@ -520,10 +603,8 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)
helper = get_helper(transport);
- if (data->push && for_push)
- write_str_in_full(helper->in, "list for-push\n");
- else
- write_str_in_full(helper->in, "list\n");
+ if(process_connect_or_invoke(transport, for_push))
+ return &special_transport_layer6_ready;
while (1) {
char *eov, *eon;
@@ -572,6 +653,7 @@ int transport_helper_init(struct transport *transport, const char *name)
transport->fetch = fetch;
transport->push_refs = push_refs;
transport->disconnect = release_helper;
+ transport->connect = connect_helper;
transport->disown = helper_disown;
transport->smart_options = &(data->gitoptions);
return 0;
diff --git a/transport.c b/transport.c
index 09e4c97..a32f405 100644
--- a/transport.c
+++ b/transport.c
@@ -762,6 +762,17 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
return ret;
}
+static int connect_git(struct transport *transport, const char* name,
+ const char* executable, int fd[2])
+{
+ struct git_transport_data *data = transport->data;
+ data->conn = git_connect(data->fd, transport->url,
+ executable, 0);
+ fd[0] = data->fd[0];
+ fd[1] = data->fd[1];
+ return 0;
+}
+
static int disconnect_git(struct transport *transport)
{
struct git_transport_data *data = transport->data;
@@ -926,6 +937,7 @@ struct transport *transport_get(struct remote *remote, const char *url)
ret->get_refs_list = get_refs_via_connect;
ret->fetch = fetch_refs_via_pack;
ret->push_refs = git_transport_push;
+ ret->connect = connect_git;
ret->disconnect = disconnect_git;
ret->smart_options = &(data->options);
ret->disown = NULL;
@@ -1109,6 +1121,15 @@ void transport_unlock_pack(struct transport *transport)
}
}
+int transport_connect(struct transport *transport, const char *name,
+ const char* exec, int fd[2])
+{
+ if(transport->connect) {
+ return transport->connect(transport, name, exec, fd);
+ } else
+ die("Operation not supported by protocol");
+}
+
int transport_disconnect(struct transport *transport)
{
int ret = 0;
diff --git a/transport.h b/transport.h
index f3ee890..c86329a 100644
--- a/transport.h
+++ b/transport.h
@@ -64,6 +64,8 @@ struct transport {
**/
int (*push_refs)(struct transport *transport, struct ref *refs, int flags);
int (*push)(struct transport *connection, int refspec_nr, const char **refspec, int flags);
+ int (*connect)(struct transport *connection, const char* name, const char* executable,
+ int fd[2]);
/**
* Disown the transport helper. Releases all resources used
@@ -143,6 +145,9 @@ void transport_unlock_pack(struct transport *transport);
int transport_disconnect(struct transport *transport);
char *transport_anonymize_url(const char *url);
+int transport_connect(struct transport *transport, const char *name,
+ const char* exec, int fd[2]);
+
/* Transport methods defined outside transport.c */
int transport_helper_init(struct transport *transport, const char *name);
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 7/8] Add remote helper debug mode
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Remote helpers deadlock easily, so support debug mode which shows the
interaction steps.
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
transport-helper.c | 110 ++++++++++++++++++++++++++++++++++------------------
1 files changed, 72 insertions(+), 38 deletions(-)
diff --git a/transport-helper.c b/transport-helper.c
index 0e4da79..697f026 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -8,6 +8,8 @@
#include "quote.h"
#include "remote.h"
+static int debug = 0;
+
struct helper_data
{
const char *name;
@@ -25,6 +27,47 @@ struct helper_data
struct git_transport_options gitoptions;
};
+static void sendline(struct helper_data *helper, struct strbuf *buffer)
+{
+ if(debug)
+ fprintf(stderr, "Debug: Remote helper: -> %s", buffer->buf);
+ if(write_in_full(helper->helper->in, buffer->buf, buffer->len)
+ != buffer->len) {
+ die_errno("Full write to remote helper failed");
+ }
+}
+
+static int recvline(struct helper_data *helper, struct strbuf *buffer,
+ int supress_finish)
+{
+ strbuf_reset(buffer);
+ if (strbuf_getline(buffer, helper->out, '\n') == EOF) {
+ if(debug)
+ fprintf(stderr, "Debug: Remote helper quit.\n");
+ if(!supress_finish)
+ finish_command(helper->helper);
+ exit(128);
+ }
+
+ if(debug)
+ fprintf(stderr, "Debug: Remote helper: <- %s\n", buffer->buf);
+ return 0;
+}
+
+static void xchgline(struct helper_data *helper, struct strbuf *buffer)
+{
+ sendline(helper, buffer);
+ recvline(helper, buffer, 1);
+}
+
+static void write_constant(int fd, const char *str)
+{
+ if(debug)
+ fprintf(stderr, "Debug: Remote helper: -> %s", str);
+ if(write_in_full(fd, str, strlen(str)) != strlen(str))
+ die_errno("Full write to remote helper failed");
+}
+
static struct child_process* helper_disown(struct transport *transport)
{
struct helper_data *data = transport->data;
@@ -99,23 +142,15 @@ static struct child_process *get_helper(struct transport *transport)
data->out = xfdopen(duped, "r");
setvbuf(data->out, NULL, _IONBF, 0);
- write_str_in_full(helper->in, "capabilities\n");
+ write_constant(helper->in, "capabilities\n");
while (1) {
- if (strbuf_getline(&buf, data->out, '\n') == EOF) {
- /* If we haven't seen line yet, try to finish the
- command so we get error message about failed
- execution. */
- if(!seen_line)
- finish_command(helper);
-
- exit(128); /* child died, message supplied already */
- }
-
+ recvline(data, &buf, seen_line);
seen_line = 1;
if (!*buf.buf)
break;
+ if(debug) fprintf(stderr, "Debug: Got cap %s\n", buf.buf);
if (!strcmp(buf.buf, "fetch"))
data->fetch = 1;
if (!strcmp(buf.buf, "option"))
@@ -145,14 +180,19 @@ static struct child_process *get_helper(struct transport *transport)
free(refspecs);
}
strbuf_release(&buf);
+ if(debug) fprintf(stderr, "Debug: Capabilities complete.\n");
return data->helper;
}
static int disconnect_helper(struct transport *transport)
{
struct helper_data *data = transport->data;
+ struct strbuf buf = STRBUF_INIT;
+
if (data->helper) {
- write_str_in_full(data->helper->in, "\n");
+ if(debug) fprintf(stderr, "Debug: Disconnecting.\n");
+ strbuf_addf(&buf, "\n");
+ sendline(data, &buf);
close(data->helper->in);
close(data->helper->out);
fclose(data->out);
@@ -182,10 +222,11 @@ static int set_helper_option(struct transport *transport,
const char *name, const char *value)
{
struct helper_data *data = transport->data;
- struct child_process *helper = get_helper(transport);
struct strbuf buf = STRBUF_INIT;
int i, ret, is_bool = 0;
+ get_helper(transport);
+
if (!data->option)
return 1;
@@ -208,12 +249,7 @@ static int set_helper_option(struct transport *transport,
quote_c_style(value, &buf, NULL, 0);
strbuf_addch(&buf, '\n');
- if (write_in_full(helper->in, buf.buf, buf.len) != buf.len)
- die_errno("cannot send option to %s", data->name);
-
- strbuf_reset(&buf);
- if (strbuf_getline(&buf, data->out, '\n') == EOF)
- exit(128); /* child died, message supplied already */
+ xchgline(data, &buf);
if (!strcmp(buf.buf, "ok"))
ret = 0;
@@ -273,13 +309,10 @@ static int fetch_with_fetch(struct transport *transport,
}
strbuf_addch(&buf, '\n');
- if (write_in_full(data->helper->in, buf.buf, buf.len) != buf.len)
- die_errno("cannot send fetch to %s", data->name);
+ sendline(data, &buf);
while (1) {
- strbuf_reset(&buf);
- if (strbuf_getline(&buf, data->out, '\n') == EOF)
- exit(128); /* child died, message supplied already */
+ recvline(data, &buf, 1);
if (!prefixcmp(buf.buf, "lock ")) {
const char *name = buf.buf + 5;
@@ -314,12 +347,13 @@ static int fetch_with_import(struct transport *transport,
int nr_heads, struct ref **to_fetch)
{
struct child_process fastimport;
- struct child_process *helper = get_helper(transport);
struct helper_data *data = transport->data;
int i;
struct ref *posn;
struct strbuf buf = STRBUF_INIT;
+ get_helper(transport);
+
if (get_importer(transport, &fastimport))
die("Couldn't run fast-import");
@@ -329,7 +363,7 @@ static int fetch_with_import(struct transport *transport,
continue;
strbuf_addf(&buf, "import %s\n", posn->name);
- write_in_full(helper->in, buf.buf, buf.len);
+ sendline(data, &buf);
strbuf_reset(&buf);
}
disconnect_helper(transport);
@@ -370,12 +404,12 @@ static int _process_connect_or_invoke(struct transport *transport,
} else
return 0;
- write_in_full(helper->in, cmdbuf.buf, cmdbuf.len);
- strbuf_reset(&cmdbuf);
- if (strbuf_getline(&cmdbuf, data->out, '\n') == EOF)
- exit(128); /* child died, message supplied already */
- if(!strcmp(cmdbuf.buf, "OK"))
+ xchgline(data, &cmdbuf);
+ if(!strcmp(cmdbuf.buf, "OK")) {
+ if(debug) fprintf(stderr, "Debug: Layer 6 link ready, "
+ "starting layer 7...\n");
return 1;
+ }
else if(!strcmp(cmdbuf.buf, "FALLBACK"))
return 0;
else if(!strcmp(cmdbuf.buf, "ERROR"))
@@ -508,17 +542,14 @@ static int push_refs(struct transport *transport,
}
strbuf_addch(&buf, '\n');
- if (write_in_full(helper->in, buf.buf, buf.len) != buf.len)
- exit(128);
+ sendline(data, &buf);
ref = remote_refs;
while (1) {
char *refname, *msg;
int status;
- strbuf_reset(&buf);
- if (strbuf_getline(&buf, data->out, '\n') == EOF)
- exit(128); /* child died, message supplied already */
+ recvline(data, &buf, 1);
if (!buf.len)
break;
@@ -608,8 +639,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)
while (1) {
char *eov, *eon;
- if (strbuf_getline(&buf, data->out, '\n') == EOF)
- exit(128); /* child died, message supplied already */
+ recvline(data, &buf, 1);
if (!*buf.buf)
break;
@@ -634,6 +664,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push)
}
tail = &((*tail)->next);
}
+ if(debug) fprintf(stderr, "Debug: Read ref listing.\n");
strbuf_release(&buf);
for (posn = ret; posn; posn = posn->next)
@@ -647,6 +678,9 @@ int transport_helper_init(struct transport *transport, const char *name)
struct helper_data *data = xcalloc(sizeof(*data), 1);
data->name = name;
+ if(getenv("GIT_TRANSPORT_HELPER_DEBUG"))
+ debug = 1;
+
transport->data = data;
transport->set_option = set_helper_option;
transport->get_refs_list = get_refs_list;
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 8/8] Support mandatory capabilities
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Add support for marking capability as mandatory for hosting git version
to understand. This is useful for helpers which require various types
of assistance from main git binary.
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
Documentation/git-remote-helpers.txt | 5 ++++-
transport-helper.c | 31 +++++++++++++++++++++++--------
2 files changed, 27 insertions(+), 9 deletions(-)
diff --git a/Documentation/git-remote-helpers.txt b/Documentation/git-remote-helpers.txt
index adf815c..eab9c03 100644
--- a/Documentation/git-remote-helpers.txt
+++ b/Documentation/git-remote-helpers.txt
@@ -25,7 +25,10 @@ Commands are given by the caller on the helper's standard input, one per line.
'capabilities'::
Lists the capabilities of the helper, one per line, ending
- with a blank line.
+ with a blank line. Each capability may be preceeded with '*'.
+ This marks them mandatory for git version using the remote
+ helper to understand (unknown mandatory capability is fatal
+ error).
'list'::
Lists the refs, one per line, in the format "<value> <name>
diff --git a/transport-helper.c b/transport-helper.c
index 697f026..a128560 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -145,30 +145,45 @@ static struct child_process *get_helper(struct transport *transport)
write_constant(helper->in, "capabilities\n");
while (1) {
+ const char* capname;
+ int mandatory = 0;
recvline(data, &buf, seen_line);
seen_line = 1;
if (!*buf.buf)
break;
- if(debug) fprintf(stderr, "Debug: Got cap %s\n", buf.buf);
- if (!strcmp(buf.buf, "fetch"))
+
+ if(*buf.buf == '*') {
+ capname = buf.buf + 1;
+ mandatory = 1;
+ } else
+ capname = buf.buf;
+
+ if(debug) fprintf(stderr, "Debug: Got cap %s\n", capname);
+ if (!strcmp(capname, "fetch"))
data->fetch = 1;
- if (!strcmp(buf.buf, "option"))
+ else if (!strcmp(capname, "option"))
data->option = 1;
- if (!strcmp(buf.buf, "push"))
+ else if (!strcmp(capname, "push"))
data->push = 1;
- if (!strcmp(buf.buf, "import"))
+ else if (!strcmp(capname, "import"))
data->import = 1;
- if (!data->refspecs && !prefixcmp(buf.buf, "refspec ")) {
+ else if (!data->refspecs && !prefixcmp(capname, "refspec ")) {
ALLOC_GROW(refspecs,
refspec_nr + 1,
refspec_alloc);
refspecs[refspec_nr++] = strdup(buf.buf + strlen("refspec "));
}
- if (!strcmp(buf.buf, "connect-r"))
+ else if (!strcmp(capname, "connect-r"))
data->connect_r = 1;
- if (!strcmp(buf.buf, "invoke-r"))
+ else if (!strcmp(capname, "invoke-r"))
data->invoke_r = 1;
+ else if (mandatory) {
+ fflush(stderr);
+ die("Unknown madatory capability %s. This remote "
+ "helper probably needs newer version of Git.\n",
+ capname);
+ }
}
if (refspecs) {
int i;
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 6/8] Remove special casing of http, https and ftp
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
HTTP, HTTPS and FTP are no longer special to transport code. Also
add support for FTPS (curl supports it so it is easy).
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
.gitignore | 5 ++++-
Makefile | 16 ++++++++++++++--
transport.c | 8 --------
3 files changed, 18 insertions(+), 11 deletions(-)
diff --git a/.gitignore b/.gitignore
index 7cc54b4..65508ea 100644
--- a/.gitignore
+++ b/.gitignore
@@ -106,7 +106,10 @@
/git-reflog
/git-relink
/git-remote
-/git-remote-curl
+/git-remote-http
+/git-remote-https
+/git-remote-ftp
+/git-remote-ftps
/git-repack
/git-replace
/git-repo-config
diff --git a/Makefile b/Makefile
index 42744a4..be0be87 100644
--- a/Makefile
+++ b/Makefile
@@ -1097,7 +1097,7 @@ else
else
CURL_LIBCURL = -lcurl
endif
- PROGRAMS += git-remote-curl$X git-http-fetch$X
+ PROGRAMS += git-remote-http$X git-remote-https$X git-remote-ftp$X git-remote-ftps$X git-http-fetch$X
curl_check := $(shell (echo 070908; curl-config --vernum) | sort -r | sed -ne 2p)
ifeq "$(curl_check)" "070908"
ifndef NO_EXPAT
@@ -1676,7 +1676,19 @@ git-http-push$X: revision.o http.o http-push.o $(GITLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)
-git-remote-curl$X: remote-curl.o http.o http-walker.o $(GITLIBS)
+git-remote-http$X: remote-curl.o http.o http-walker.o $(GITLIBS)
+ $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
+ $(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)
+
+git-remote-https$X: remote-curl.o http.o http-walker.o $(GITLIBS)
+ $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
+ $(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)
+
+git-remote-ftp$X: remote-curl.o http.o http-walker.o $(GITLIBS)
+ $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
+ $(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)
+
+git-remote-ftps$X: remote-curl.o http.o http-walker.o $(GITLIBS)
$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
$(LIBS) $(CURL_LIBCURL) $(EXPAT_LIBEXPAT)
diff --git a/transport.c b/transport.c
index a32f405..872cc30 100644
--- a/transport.c
+++ b/transport.c
@@ -944,14 +944,6 @@ struct transport *transport_get(struct remote *remote, const char *url)
data->conn = NULL;
data->virtual_connected = 0;
- } else if (!prefixcmp(url, "http://")
- || !prefixcmp(url, "https://")
- || !prefixcmp(url, "ftp://")) {
- /* These three are just plain special. */
- transport_helper_init(ret, "curl");
-#ifdef NO_CURL
- error("git was compiled without libcurl support.");
-#else
} else {
/* Unknown protocol in URL. Pass to external handler. */
int len = external_specification_len(url);
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 3/8] Support taking over transports
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Add support for taking over transports that turn out to be smart.
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
transport-helper.c | 12 +++++++
transport.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++----
transport.h | 15 +++++++++
3 files changed, 109 insertions(+), 7 deletions(-)
diff --git a/transport-helper.c b/transport-helper.c
index a499751..777ecbb 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -20,8 +20,18 @@ struct helper_data
/* These go from remote name (as in "list") to private name */
struct refspec *refspecs;
int refspec_nr;
+ struct git_transport_options gitoptions;
};
+static struct child_process* helper_disown(struct transport *transport)
+{
+ struct helper_data *data = transport->data;
+ struct child_process *child = data->helper;
+ fclose(data->out);
+ free(data);
+ return child;
+}
+
const char* remove_ext_force(const char* url)
{
const char* url2 = url;
@@ -562,5 +572,7 @@ int transport_helper_init(struct transport *transport, const char *name)
transport->fetch = fetch;
transport->push_refs = push_refs;
transport->disconnect = release_helper;
+ transport->disown = helper_disown;
+ transport->smart_options = &(data->gitoptions);
return 0;
}
diff --git a/transport.c b/transport.c
index 7956892..09e4c97 100644
--- a/transport.c
+++ b/transport.c
@@ -9,6 +9,8 @@
#include "dir.h"
#include "refs.h"
+struct ref special_transport_layer6_ready;
+
/* rsync support */
/*
@@ -398,6 +400,8 @@ struct git_transport_data {
struct git_transport_options options;
struct child_process *conn;
int fd[2];
+ /* Connection is fully up. */
+ unsigned virtual_connected : 1;
struct extra_have_objects extra_have;
};
@@ -432,10 +436,21 @@ static int set_git_option(struct git_transport_options *opts,
static int connect_setup(struct transport *transport, int for_push, int verbose)
{
struct git_transport_data *data = transport->data;
+
+ if(!data->virtual_connected && data->conn) {
+ /* Just mark it connected. */
+ data->virtual_connected = 1;
+ return 0;
+ }
+
data->conn = git_connect(data->fd, transport->url,
for_push ? data->options.receivepack :
data->options.uploadpack,
verbose ? CONNECT_VERBOSE : 0);
+
+ if(data->conn)
+ data->virtual_connected = 1;
+
return 0;
}
@@ -477,7 +492,7 @@ static int fetch_refs_via_pack(struct transport *transport,
for (i = 0; i < nr_heads; i++)
origh[i] = heads[i] = xstrdup(to_fetch[i]->name);
- if (!data->conn) {
+ if (!data->virtual_connected) {
connect_setup(transport, 0, 0);
get_remote_heads(data->fd[0], &refs_tmp, 0, NULL, 0, NULL);
}
@@ -490,6 +505,7 @@ static int fetch_refs_via_pack(struct transport *transport,
if (finish_connect(data->conn))
refs = NULL;
data->conn = NULL;
+ data->virtual_connected = 0;
free_refs(refs_tmp);
@@ -718,7 +734,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
struct send_pack_args args;
int ret;
- if (!data->conn) {
+ if (!data->virtual_connected) {
struct ref *tmp_refs;
connect_setup(transport, 1, 0);
@@ -741,6 +757,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
close(data->fd[0]);
ret |= finish_connect(data->conn);
data->conn = NULL;
+ data->virtual_connected = 0;
return ret;
}
@@ -749,7 +766,8 @@ static int disconnect_git(struct transport *transport)
{
struct git_transport_data *data = transport->data;
if (data->conn) {
- packet_flush(data->fd[1]);
+ if(data->virtual_connected)
+ packet_flush(data->fd[1]);
close(data->fd[0]);
close(data->fd[1]);
finish_connect(data->conn);
@@ -759,6 +777,35 @@ static int disconnect_git(struct transport *transport)
return 0;
}
+static void git_take_over_transport(struct transport *transport)
+{
+ struct git_transport_data *data;
+
+ if(!transport->disown)
+ die("Bug detected: Taking over transport requires non-NULL "
+ "disown method.");
+ if(!transport->smart_options)
+ die("Bug detected: Taking over transport requires non-NULL "
+ "smart_options field.");
+
+ data = xcalloc(1, sizeof(*data));
+ data->options = *transport->smart_options;
+ data->conn = transport->disown(transport);
+ data->fd[0] = data->conn->out;
+ data->fd[1] = data->conn->in;
+ data->virtual_connected = 0;
+ transport->data = data;
+
+ transport->set_option = NULL;
+ transport->get_refs_list = get_refs_via_connect;
+ transport->fetch = fetch_refs_via_pack;
+ transport->push = NULL;
+ transport->push_refs = git_transport_push;
+ transport->disconnect = disconnect_git;
+ transport->smart_options = &(data->options);
+ transport->disown = NULL;
+}
+
static int is_local(const char *url)
{
const char *colon = strchr(url, ':');
@@ -857,6 +904,7 @@ struct transport *transport_get(struct remote *remote, const char *url)
ret->fetch = fetch_objs_via_rsync;
ret->push = rsync_transport_push;
ret->smart_options = NULL;
+ ret->disown = NULL;
} else if (is_local(url) && is_file(url)) {
struct bundle_transport_data *data = xcalloc(1, sizeof(*data));
ret->data = data;
@@ -864,6 +912,7 @@ struct transport *transport_get(struct remote *remote, const char *url)
ret->fetch = fetch_refs_from_bundle;
ret->disconnect = close_bundle;
ret->smart_options = NULL;
+ ret->disown = NULL;
} else if(!is_url(url)
|| !prefixcmp(url, "file://")
|| !prefixcmp(url, "git://")
@@ -879,8 +928,10 @@ struct transport *transport_get(struct remote *remote, const char *url)
ret->push_refs = git_transport_push;
ret->disconnect = disconnect_git;
ret->smart_options = &(data->options);
+ ret->disown = NULL;
data->conn = NULL;
+ data->virtual_connected = 0;
} else if (!prefixcmp(url, "http://")
|| !prefixcmp(url, "https://")
|| !prefixcmp(url, "ftp://")) {
@@ -938,14 +989,25 @@ int transport_push(struct transport *transport,
int refspec_nr, const char **refspec, int flags,
int *nonfastforward)
{
+ int rc = 0;
*nonfastforward = 0;
verify_remote_names(refspec_nr, refspec);
- if (transport->push)
- return transport->push(transport, refspec_nr, refspec, flags);
- if (transport->push_refs) {
+retry:
+ if (transport->push) {
+ rc = transport->push(transport, refspec_nr, refspec, flags);
+ if(rc == TRANSPORT_LAYER6_READY) {
+ git_take_over_transport(transport);
+ goto retry;
+ }
+ return rc;
+ } else if (transport->push_refs) {
struct ref *remote_refs =
transport->get_refs_list(transport, 1);
+ if(remote_refs == &special_transport_layer6_ready) {
+ git_take_over_transport(transport);
+ goto retry;
+ }
struct ref *local_refs = get_local_heads();
int match_flags = MATCH_REFS_NONE;
int verbose = flags & TRANSPORT_PUSH_VERBOSE;
@@ -985,8 +1047,15 @@ int transport_push(struct transport *transport,
const struct ref *transport_get_remote_refs(struct transport *transport)
{
- if (!transport->remote_refs)
+ if (!transport->remote_refs) {
+retry:
transport->remote_refs = transport->get_refs_list(transport, 0);
+ if(transport->remote_refs == &special_transport_layer6_ready) {
+ git_take_over_transport(transport);
+ goto retry;
+ }
+ }
+
return transport->remote_refs;
}
@@ -1020,7 +1089,13 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs)
heads[nr_heads++] = rm;
}
+retry:
rc = transport->fetch(transport, nr_heads, heads);
+ if(rc == TRANSPORT_LAYER6_READY) {
+ git_take_over_transport(transport);
+ goto retry;
+ }
+
free(heads);
return rc;
}
diff --git a/transport.h b/transport.h
index 5949132..f3ee890 100644
--- a/transport.h
+++ b/transport.h
@@ -65,6 +65,15 @@ struct transport {
int (*push_refs)(struct transport *transport, struct ref *refs, int flags);
int (*push)(struct transport *connection, int refspec_nr, const char **refspec, int flags);
+ /**
+ * Disown the transport helper. Releases all resources used
+ * by field pointed by member data, except that the child
+ * process is not released but returned and whatever is pointed
+ * by smart transport options structure is not freed (but the
+ * smart transport options structure itself is).
+ **/
+ struct child_process* (*disown)(struct transport* connection);
+
/** get_refs_list(), fetch(), and push_refs() can keep
* resources (such as a connection) reserved for futher
* use. disconnect() releases these resources.
@@ -79,6 +88,12 @@ struct transport {
struct git_transport_options* smart_options;
};
+/* Returned by get_refs_list, fetch or push methods of struct transport: Layer 6 is ready,
+ take over the transport and retry operation. */
+#define TRANSPORT_LAYER6_READY -42
+extern struct ref special_transport_layer6_ready;
+
+
#define TRANSPORT_PUSH_ALL 1
#define TRANSPORT_PUSH_FORCE 2
#define TRANSPORT_PUSH_DRY_RUN 4
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 5/8] Support remote archive from external protocol helpers
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Helpers which support invoke/connect also should support remote archive
snapshot (or at least there's only one way to attempt it). So support
remote snapshotting for protocol helpers.
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
builtin-archive.c | 17 ++++++++++-------
1 files changed, 10 insertions(+), 7 deletions(-)
diff --git a/builtin-archive.c b/builtin-archive.c
index 12351e9..3c053b4 100644
--- a/builtin-archive.c
+++ b/builtin-archive.c
@@ -5,6 +5,7 @@
#include "cache.h"
#include "builtin.h"
#include "archive.h"
+#include "transport.h"
#include "parse-options.h"
#include "pkt-line.h"
#include "sideband.h"
@@ -25,12 +26,16 @@ static void create_output_file(const char *output_file)
static int run_remote_archiver(int argc, const char **argv,
const char *remote, const char *exec)
{
- char *url, buf[LARGE_PACKET_MAX];
+ char buf[LARGE_PACKET_MAX];
int fd[2], i, len, rv;
- struct child_process *conn;
+ struct transport *transport;
+ struct remote *_remote;
- url = xstrdup(remote);
- conn = git_connect(fd, url, exec, 0);
+ _remote = remote_get(remote);
+ if (!_remote->url[0])
+ die("git archive: Remote with no URL");
+ transport = transport_get(_remote, _remote->url[0]);
+ transport_connect(transport, "upload-archive", exec, fd);
for (i = 1; i < argc; i++)
packet_write(fd[1], "argument %s\n", argv[i]);
@@ -53,9 +58,7 @@ static int run_remote_archiver(int argc, const char **argv,
/* Now, start reading from fd[0] and spit it out to stdout */
rv = recv_sideband("archive", fd[0], 1);
- close(fd[0]);
- close(fd[1]);
- rv |= finish_connect(conn);
+ rv |= transport_disconnect(transport);
return !!rv;
}
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 2/8] Refactor git transport options parsing
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Refactor the transport options parsing so that protocols that aren't
directly smart transports (file://, git://, ssh:// & co) can record
the smart transport options for the case if it turns that transport
can actually be smart.
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
transport.c | 78 +++++++++++++++++++++++++++++++++++-----------------------
transport.h | 12 +++++++++
2 files changed, 59 insertions(+), 31 deletions(-)
diff --git a/transport.c b/transport.c
index 148b9e1..7956892 100644
--- a/transport.c
+++ b/transport.c
@@ -395,41 +395,35 @@ static int close_bundle(struct transport *transport)
}
struct git_transport_data {
- unsigned thin : 1;
- unsigned keep : 1;
- unsigned followtags : 1;
- int depth;
+ struct git_transport_options options;
struct child_process *conn;
int fd[2];
- const char *uploadpack;
- const char *receivepack;
struct extra_have_objects extra_have;
};
-static int set_git_option(struct transport *connection,
+static int set_git_option(struct git_transport_options *opts,
const char *name, const char *value)
{
- struct git_transport_data *data = connection->data;
if (!strcmp(name, TRANS_OPT_UPLOADPACK)) {
- data->uploadpack = value;
+ opts->uploadpack = value;
return 0;
} else if (!strcmp(name, TRANS_OPT_RECEIVEPACK)) {
- data->receivepack = value;
+ opts->receivepack = value;
return 0;
} else if (!strcmp(name, TRANS_OPT_THIN)) {
- data->thin = !!value;
+ opts->thin = !!value;
return 0;
} else if (!strcmp(name, TRANS_OPT_FOLLOWTAGS)) {
- data->followtags = !!value;
+ opts->followtags = !!value;
return 0;
} else if (!strcmp(name, TRANS_OPT_KEEP)) {
- data->keep = !!value;
+ opts->keep = !!value;
return 0;
} else if (!strcmp(name, TRANS_OPT_DEPTH)) {
if (!value)
- data->depth = 0;
+ opts->depth = 0;
else
- data->depth = atoi(value);
+ opts->depth = atoi(value);
return 0;
}
return 1;
@@ -439,7 +433,8 @@ static int connect_setup(struct transport *transport, int for_push, int verbose)
{
struct git_transport_data *data = transport->data;
data->conn = git_connect(data->fd, transport->url,
- for_push ? data->receivepack : data->uploadpack,
+ for_push ? data->options.receivepack :
+ data->options.uploadpack,
verbose ? CONNECT_VERBOSE : 0);
return 0;
}
@@ -469,15 +464,15 @@ static int fetch_refs_via_pack(struct transport *transport,
struct ref *refs_tmp = NULL;
memset(&args, 0, sizeof(args));
- args.uploadpack = data->uploadpack;
- args.keep_pack = data->keep;
+ args.uploadpack = data->options.uploadpack;
+ args.keep_pack = data->options.keep;
args.lock_pack = 1;
- args.use_thin_pack = data->thin;
- args.include_tag = data->followtags;
+ args.use_thin_pack = data->options.thin;
+ args.include_tag = data->options.followtags;
args.verbose = (transport->verbose > 0);
args.quiet = (transport->verbose < 0);
args.no_progress = args.quiet || (!transport->progress && !isatty(1));
- args.depth = data->depth;
+ args.depth = data->options.depth;
for (i = 0; i < nr_heads; i++)
origh[i] = heads[i] = xstrdup(to_fetch[i]->name);
@@ -734,7 +729,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re
memset(&args, 0, sizeof(args));
args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);
args.force_update = !!(flags & TRANSPORT_PUSH_FORCE);
- args.use_thin_pack = data->thin;
+ args.use_thin_pack = data->options.thin;
args.verbose = !!(flags & TRANSPORT_PUSH_VERBOSE);
args.quiet = !!(flags & TRANSPORT_PUSH_QUIET);
args.dry_run = !!(flags & TRANSPORT_PUSH_DRY_RUN);
@@ -861,12 +856,14 @@ struct transport *transport_get(struct remote *remote, const char *url)
ret->get_refs_list = get_refs_via_rsync;
ret->fetch = fetch_objs_via_rsync;
ret->push = rsync_transport_push;
+ ret->smart_options = NULL;
} else if (is_local(url) && is_file(url)) {
struct bundle_transport_data *data = xcalloc(1, sizeof(*data));
ret->data = data;
ret->get_refs_list = get_refs_from_bundle;
ret->fetch = fetch_refs_from_bundle;
ret->disconnect = close_bundle;
+ ret->smart_options = NULL;
} else if(!is_url(url)
|| !prefixcmp(url, "file://")
|| !prefixcmp(url, "git://")
@@ -876,20 +873,14 @@ struct transport *transport_get(struct remote *remote, const char *url)
/* These are builtin smart transports. */
struct git_transport_data *data = xcalloc(1, sizeof(*data));
ret->data = data;
- ret->set_option = set_git_option;
+ ret->set_option = NULL;
ret->get_refs_list = get_refs_via_connect;
ret->fetch = fetch_refs_via_pack;
ret->push_refs = git_transport_push;
ret->disconnect = disconnect_git;
+ ret->smart_options = &(data->options);
- data->thin = 1;
data->conn = NULL;
- data->uploadpack = "git-upload-pack";
- if (remote->uploadpack)
- data->uploadpack = remote->uploadpack;
- data->receivepack = "git-receive-pack";
- if (remote->receivepack)
- data->receivepack = remote->receivepack;
} else if (!prefixcmp(url, "http://")
|| !prefixcmp(url, "https://")
|| !prefixcmp(url, "ftp://")) {
@@ -907,14 +898,39 @@ struct transport *transport_get(struct remote *remote, const char *url)
transport_helper_init(ret, handler);
}
+ if(ret->smart_options) {
+ ret->smart_options->thin = 1;
+ ret->smart_options->uploadpack = "git-upload-pack";
+ if (remote->uploadpack)
+ ret->smart_options->uploadpack = remote->uploadpack;
+ ret->smart_options->receivepack = "git-receive-pack";
+ if (remote->receivepack)
+ ret->smart_options->receivepack = remote->receivepack;
+ }
+
return ret;
}
int transport_set_option(struct transport *transport,
const char *name, const char *value)
{
+ int git_reports = 1, protocol_reports = 1;
+
+ if (transport->smart_options)
+ git_reports = set_git_option(transport->smart_options,
+ name, value);
+
if (transport->set_option)
- return transport->set_option(transport, name, value);
+ protocol_reports = transport->set_option(transport, name,
+ value);
+
+ /* If either report is 0, report 0 (success). */
+ if(!git_reports || !protocol_reports)
+ return 0;
+ /* If either reports -1 (invalid value), report -1. */
+ if((git_reports == -1) || (protocol_reports == -1))
+ return -1;
+ /* Otherwise if both report unknown, report unknown. */
return 1;
}
diff --git a/transport.h b/transport.h
index 9e74406..5949132 100644
--- a/transport.h
+++ b/transport.h
@@ -4,6 +4,15 @@
#include "cache.h"
#include "remote.h"
+struct git_transport_options {
+ unsigned thin : 1;
+ unsigned keep : 1;
+ unsigned followtags : 1;
+ int depth;
+ const char *uploadpack;
+ const char *receivepack;
+};
+
struct transport {
struct remote *remote;
const char *url;
@@ -65,6 +74,9 @@ struct transport {
signed verbose : 3;
/* Force progress even if the output is not a tty */
unsigned progress : 1;
+ /* If transport is at least potentially smart, this points to git_transport_options
+ structure to use in case transport actually turns out to be smart. */
+ struct git_transport_options* smart_options;
};
#define TRANSPORT_PUSH_ALL 1
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 1/8] Pass unknown protocols to external protocol handlers
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
In-Reply-To: <1259675838-14692-1-git-send-email-ilari.liusvaara@elisanet.fi>
Change URL handling to allow external protocol handlers to provde
new protocols without the '::' syntax if helper name does not conflict
with any builtin protocol.
foo:// now invokes git-remote-foo with foo:// URL.
Signed-off-by: Ilari Liusvaara <ilari.liusvaara@elisanet.fi>
---
transport-helper.c | 57 ++++++++++++++++++++++++++++++++----
transport.c | 82 +++++++++++++++++++++++++++++++++++++++++++++-------
2 files changed, 122 insertions(+), 17 deletions(-)
diff --git a/transport-helper.c b/transport-helper.c
index 6182413..a499751 100644
--- a/transport-helper.c
+++ b/transport-helper.c
@@ -22,6 +22,26 @@ struct helper_data
int refspec_nr;
};
+const char* remove_ext_force(const char* url)
+{
+ const char* url2 = url;
+ const char* first_colon = NULL;
+
+ if(!url)
+ return NULL;
+
+ while(*url2 && !first_colon)
+ if(*url2 == ':')
+ first_colon = url2;
+ else
+ url2++;
+
+ if(first_colon && first_colon[1] == ':')
+ return first_colon + 2;
+ else
+ return url;
+}
+
static struct child_process *get_helper(struct transport *transport)
{
struct helper_data *data = transport->data;
@@ -30,6 +50,8 @@ static struct child_process *get_helper(struct transport *transport)
const char **refspecs = NULL;
int refspec_nr = 0;
int refspec_alloc = 0;
+ int duped;
+ int seen_line = 0;
if (data->helper)
return data->helper;
@@ -39,21 +61,43 @@ static struct child_process *get_helper(struct transport *transport)
helper->out = -1;
helper->err = 0;
helper->argv = xcalloc(4, sizeof(*helper->argv));
- strbuf_addf(&buf, "remote-%s", data->name);
+ /* We use the dashed form because git <unknown helper>
+ would run and print totally inapporiate error message. */
+ strbuf_addf(&buf, "git-remote-%s", data->name);
helper->argv[0] = strbuf_detach(&buf, NULL);
helper->argv[1] = transport->remote->name;
- helper->argv[2] = transport->url;
- helper->git_cmd = 1;
+ helper->argv[2] = remove_ext_force(transport->url);
+ helper->git_cmd = 0;
if (start_command(helper))
- die("Unable to run helper: git %s", helper->argv[0]);
+ die("Unable to run helper: %s", helper->argv[0]);
data->helper = helper;
+ /* Open the output as FILE* so strbuf_getline() can be used.
+ Do this with duped fd because fclose() will close the fd,
+ and stuff like disowning will require the fd to remain.
+
+ Set the stream to unbuffered because some reads are critical
+ in sense that any overreading will cause deadlocks.
+ */
+ if((duped = dup(helper->out)) < 0)
+ die_errno("Can't dup helper output fd");
+ data->out = xfdopen(duped, "r");
+ setvbuf(data->out, NULL, _IONBF, 0);
+
write_str_in_full(helper->in, "capabilities\n");
- data->out = xfdopen(helper->out, "r");
while (1) {
- if (strbuf_getline(&buf, data->out, '\n') == EOF)
+ if (strbuf_getline(&buf, data->out, '\n') == EOF) {
+ /* If we haven't seen line yet, try to finish the
+ command so we get error message about failed
+ execution. */
+ if(!seen_line)
+ finish_command(helper);
+
exit(128); /* child died, message supplied already */
+ }
+
+ seen_line = 1;
if (!*buf.buf)
break;
@@ -91,6 +135,7 @@ static int disconnect_helper(struct transport *transport)
if (data->helper) {
write_str_in_full(data->helper->in, "\n");
close(data->helper->in);
+ close(data->helper->out);
fclose(data->out);
finish_command(data->helper);
free((char *)data->helper->argv[0]);
diff --git a/transport.c b/transport.c
index 3eea836..148b9e1 100644
--- a/transport.c
+++ b/transport.c
@@ -780,6 +780,55 @@ static int is_file(const char *url)
return S_ISREG(buf.st_mode);
}
+static const char* strchrc(const char* str, int c)
+{
+ while(*str)
+ if(*str == c)
+ return str;
+ else
+ str++;
+ return NULL;
+}
+
+static int is_url(const char* url)
+{
+ if(!url)
+ return 0;
+
+ const char* url2 = url;
+ const char* first_slash = strchrc(url, '/');
+
+ /* Input with no slash at all or slash first can't be URL. */
+ if(!first_slash || first_slash == url)
+ return 0;
+ /* Character before must be : and next must be /. */
+ if(first_slash[-1] != ':' || first_slash[1] != '/')
+ return 0;
+ /* There must be something before the :// */
+ if(first_slash == url + 1)
+ return 0;
+ /* Check all characters up to first slash. Only alpha, num and
+ : are allowed. : must be followed by : or / */
+ url2 = url;
+ while(url2 < first_slash) {
+ if(*url2 != ':' && !isalnum((unsigned char)*url2))
+ return 0;
+ if(*url2 == ':' && url2[1] != ':' && url2[1] != '/')
+ return 0;
+ if(*url2 == ':')
+ url2++; /* Skip second : */
+ url2++;
+ }
+
+ /* Valid enough. */
+ return 1;
+}
+
+static int external_specification_len(const char* url)
+{
+ return strchrc(url, ':') - url;
+}
+
struct transport *transport_get(struct remote *remote, const char *url)
{
struct transport *ret = xcalloc(1, sizeof(*ret));
@@ -812,23 +861,19 @@ struct transport *transport_get(struct remote *remote, const char *url)
ret->get_refs_list = get_refs_via_rsync;
ret->fetch = fetch_objs_via_rsync;
ret->push = rsync_transport_push;
-
- } else if (!prefixcmp(url, "http://")
- || !prefixcmp(url, "https://")
- || !prefixcmp(url, "ftp://")) {
- transport_helper_init(ret, "curl");
-#ifdef NO_CURL
- error("git was compiled without libcurl support.");
-#endif
-
} else if (is_local(url) && is_file(url)) {
struct bundle_transport_data *data = xcalloc(1, sizeof(*data));
ret->data = data;
ret->get_refs_list = get_refs_from_bundle;
ret->fetch = fetch_refs_from_bundle;
ret->disconnect = close_bundle;
-
- } else {
+ } else if(!is_url(url)
+ || !prefixcmp(url, "file://")
+ || !prefixcmp(url, "git://")
+ || !prefixcmp(url, "ssh://")
+ || !prefixcmp(url, "git+ssh://")
+ || !prefixcmp(url, "ssh+git://")) {
+ /* These are builtin smart transports. */
struct git_transport_data *data = xcalloc(1, sizeof(*data));
ret->data = data;
ret->set_option = set_git_option;
@@ -845,6 +890,21 @@ struct transport *transport_get(struct remote *remote, const char *url)
data->receivepack = "git-receive-pack";
if (remote->receivepack)
data->receivepack = remote->receivepack;
+ } else if (!prefixcmp(url, "http://")
+ || !prefixcmp(url, "https://")
+ || !prefixcmp(url, "ftp://")) {
+ /* These three are just plain special. */
+ transport_helper_init(ret, "curl");
+#ifdef NO_CURL
+ error("git was compiled without libcurl support.");
+#else
+ } else {
+ /* Unknown protocol in URL. Pass to external handler. */
+ int len = external_specification_len(url);
+ char* handler = xmalloc(len + 1);
+ handler[len] = 0;
+ strncpy(handler, url, len);
+ transport_helper_init(ret, handler);
}
return ret;
--
1.6.6.rc0.64.g5593e
^ permalink raw reply related
* [RFC PATCH 0/8] Git remote helpers to implement smart transports.
From: Ilari Liusvaara @ 2009-12-01 13:57 UTC (permalink / raw)
To: git
This series implements extensions to remote helpers for carrying smary
transports. It is against next, because master doesn't contain necressary
patches (the allow specifying remote helper in url one).
First patch reworks URL handling so that unknown protocols are passed
to remote helpers. This allows having remote helpers implement git
transports without duplicating the protocol part.
Second patch refactors git transport option parsing to split smart
transport option to own structure and keep this structure up to date
with encountered options. This is needed if transport turns out to
be smart transport.
Third patch adds capabilty to have git smart transport code take
over connection, replacing "layer 7" with git smart transport protocols.
Fourth patch actually adds the extensions to external transport code to
allow helpers signal that transport should be taken over (become smart
transport).
Fifth patch extends 'git archive' to allow snapshotting off any transport
that uses git smart transport code, not just file://, git:// and ssh://
Sixth patch removes special casing of http, https and ftp. And while
at it, adds ftps, since CURL supports it.
Seventh patch adds debug mode for remote helpers. Might be useful for
debugging deadlocks by showing command traffic between git executable
and remote helper.
Eighth patch adds support for remote helper to signal that it requires
some capability and have git complain if it doesn't know it.
Misc remarks:
Underlying network link is assumed to be full-duplex since most of the
time if the underlying link isn't HTTP, it will be full-duplex (most of the
time even TCP).
Simplest deadlock-free buffering is just to read incoming pipe from git
when there's no data to send to remote end. This gives adequate performance
in all cases except when sending large initial ref adverts (and those are
ended by flush anyway, so those can be safely buffered). So no extensions
to add missing flushes are needed.
Ilari Liusvaara (8):
Pass unknown protocols to external protocol handlers
Refactor git transport options parsing
Support taking over transports
Support remote helpers implementing smart transports
Support remote archive from external protocol helpers
Remove special casing of http, https and ftp
Add remote helper debug mode
Support mandatory capabilities
.gitignore | 5 +-
Documentation/git-remote-helpers.txt | 35 ++++-
Makefile | 16 ++-
builtin-archive.c | 17 ++-
transport-helper.c | 270 ++++++++++++++++++++++++++++-----
transport.c | 258 ++++++++++++++++++++++++++------
transport.h | 32 ++++
7 files changed, 533 insertions(+), 100 deletions(-)
^ permalink raw reply
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Johannes Sixt @ 2009-12-01 13:17 UTC (permalink / raw)
To: Thomas Singer; +Cc: git
In-Reply-To: <4B150747.2030900@syntevo.com>
Thomas Singer schrieb:
> I'm quite surprised, that -- as I
> understand you -- msys-Git (or Git at all?) is not able to handle all
> characters (aka unicode) at the same time. I expected it would be better
> than older tools, e.g. SVN.
This has been discussed at length here and in the msysgit mailing list.
Git expects that the file system returns file names with the same byte
sequence that git used to create a file. On Windows, this works only as
long as you do not switch the codepage.
> BTW, we are invoking the Git executable from Java. Is there automatically a
> console "around" Git?
I don't think so. In this case, the codepage that Java has set up will
apply. I guess that Java doesn't mess with the codepage at all, and then
on German Windows git would operate in cp1252.
> Should we invoke a shell-script (which sets the
> console's code page) instead of the Git executable directly?
I don't think that is necessary.
-- Hannes
^ permalink raw reply
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Thomas Singer @ 2009-12-01 12:11 UTC (permalink / raw)
To: kusmabite; +Cc: Maximilien Noal, git
In-Reply-To: <40aa078e0912010112u4205452as3627ba019544c5fe@mail.gmail.com>
Erik Faye-Lund wrote:
> Did you try to make sure your console window used a Unicode font on
> your German Windows installation? Asian Windows installations might do
> this by default, something at least neither English nor Norwegian
> Windows installations seems to do...
>
> You can change the console window font through the properties-menu
> that appears when you right click the title-bar.
I've tried to change the console font (there is just one alternative), but
without any change.
--
Tom
^ permalink raw reply
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Thomas Singer @ 2009-12-01 12:08 UTC (permalink / raw)
To: Johannes Sixt; +Cc: git
In-Reply-To: <4B14E934.9090304@viscovery.net>
Johannes Sixt wrote:
> Thomas Singer schrieb:
>> Is it a German Windows limitation, that far-east characters are not
>> supported on it (but work fine on a Japanese Windows), are there different
>> (mysys)Git versions available or is this a configuration issue?
>
> It is a matter of configuration.
>
> Since 8 bits are not sufficient to support Japanese alphabet in addition
> to the German alphabet, programs that are not Unicode aware -- such as git
> -- have to make a decision which alphabet they support. The decision is
> made by picking a "codepage".
>
> On German Windows, you are in codepage 850 (in the console). The filenames
> (that actually are in Unicode) are converted to bytes according to
> codepage 850 *before* git sees them. If your filenames contain Hiragana,
> they are substituted by the "unknown character" marker because there is no
> place for them in codepage 850.
>
> However, you can install Japanese language support on German Windows. Then
> you can change your console to codepage 932:
>
> chcp 932
>
> When you run git from *this* console, Hiragana in the filenames are
> converted to cp932 before git sees them. The resulting byte sequence is
> different from the one in cp850, but git will be able to see that the file
> exists and was modified, and you can 'git add' it.
>
> But if you have files with umlauts, they will not be recognized anymore
> because umlauts have no place in cp932.
>
> In neither case can you exchange the repository with Linux if you have
> your locale set to UTF-8 on Linux, because neither byte sequence (umlauts
> from cp850 or Hiragana from cp932) are valid UTF-8 sequences, let alone
> result in the expected glyphs.
>
> Corollary: Stick to ASCII file names.
>
> There have been suggestions to switch the console to codepage 65001
> (UTF-8), but I have never heard of success reports. I'm not saying it does
> not work, though.
Thanks for the detailed explanation. I know the differences between bytes
and characters and the needed *encoding* to convert from one to another, but
I did not know how Git handles it. I'm quite surprised, that -- as I
understand you -- msys-Git (or Git at all?) is not able to handle all
characters (aka unicode) at the same time. I expected it would be better
than older tools, e.g. SVN.
BTW, we are invoking the Git executable from Java. Is there automatically a
console "around" Git? Should we invoke a shell-script (which sets the
console's code page) instead of the Git executable directly?
--
Tom
^ permalink raw reply
* Re: Newbie
From: Mikko Oksalahti @ 2009-12-01 12:00 UTC (permalink / raw)
To: git
In-Reply-To: <26ae428a0912010145k61dbfc41l8243363493918445@mail.gmail.com>
Howard Miller <howardsmiller <at> googlemail.com> writes:
>
> 2009/12/1 Mikko Oksalahti <mikko <at> azila.fi>:
> > How do I now get the accidentally deleted files back from the repository
without
> > losing local changes made to 10 files?
>
> 'git status' should show you what files you have deleted. 'git
> checkout filename' should get them back. I can't think of a way of
> recovering every file you have just deleted although - I suspect it
> might be tricky. Thinks like 'git pull' only apply to remote
> repositories and you don't have one of those.
>
> Howard
>
Ok. That helps. I just assumed the 'git pull' would work same way on local and
remote repositories but I guess not then...
> ....You're not thinking of it the right way (yet)
I hate when you say that :P
/Mikko
^ permalink raw reply
* Re: equal-tree-merges as way to make rebases fast-forward-able
From: Michael Haggerty @ 2009-12-01 11:50 UTC (permalink / raw)
To: Bernhard R. Link; +Cc: git
In-Reply-To: <cover.1259524136.git.brlink@debian.org>
Bernhard R. Link wrote:
> Example 1:
>
> Let's assume you maintain such a regularily-rebased branch that you
> want to be able to publish (or pull from other repositories for example
> on your laptop):
>
> o=m=o=o=master
> \
> a=b=c=d=e=feature
>
> with this patch you can do "git rebase -eqt master" and get:
>
> a'=b'=c'=d'=e'=feature'=eqt
> / /
> o=m=o=o=master-------- /
> \ \ /
> a=b=c=d=e=feature--merge-------
Actually, there is more information that can be retained about this
rebase operation. Your scheme records the fact that (a+b+c+d+e+merge)
== (o+o+a'+b'+c'+d'+e'), which is certainly true. But in the process of
rebasing, the user has (implicitly or explicitly) resolved conflicts in
transforming each of the patches a -> a', b -> b', etc. In fact, the
patch a' is itself a merge between a and master; b' is a merge between b
and a'; etc. If you record each of these merges individually, the
result looks like this:
o=m=o=o=master
\ \
\ a'=b'=c'=d'=e'=feature'
\ / / / / /
---a==b==c==d==e==feature
There are advantages to retaining all of this history:
* It faithfully represents intermediate steps of the rebase.
* There is no need for special "merge" and "eqt" merge commits affecting
an arbitrary group of feature patches; each of the rebased patches is
treated identically.
* There is a direct ancestry connection from the "new version" to the
"old version" of each patch; for example, it is easy to see that c' is a
new version of c and to compute the corresponding interdiffs.
* There are situations where the additional info can help git choose
better merge bases in the case of merge/rebases across three or more
repositories. For example, somebody who is developing a subfeature
based on the feature branch can merge/rebase changes from both feature
and master without causing utter chaos.
The "historical" version of the feature branch should be omitted from
most git output as you have suggested, but this would be best
implemented by marking the "historical" ancestor with some extra flag in
each merge commit.
> Example 2:
>
> Let's assume you have a feature branch like
>
> o=master
> \
> a=b=c=d=e=f
>
> Assume you just commited "f" which fixes a bug introduced by "b". [...]
>
> So with this patches you can do "git rebase -i --eqt" and squash f into b
> and get:
>
> o=master
> \
> a=b=c=d=e=f---
> \ \
> b+f=c'=d'=e'=eqt
This case can also record additional information:
o=master
\
a=b===c==d=e=f
\ \ \ \
b+f=c'=d'==e'
Here the new DAG cannot represent *all* ancestry information (namely,
that b+f, c', and d' also include the original patch f), but it does
accurately reflect useful information such as that c' includes c and
that e' includes e and f.
I wrote some blog entries about rebasing-with-history that might be
interesting [1-3].
Michael
[1]
http://softwareswirl.blogspot.com/2009/04/truce-in-merge-vs-rebase-war.html
[2]
http://softwareswirl.blogspot.com/2009/08/upstream-rebase-just-works-if-history.html
[3]
http://softwareswirl.blogspot.com/2009/08/rebase-with-history-implementation.html
^ permalink raw reply
* Transplant branch from another repository
From: Jeenu V @ 2009-12-01 11:01 UTC (permalink / raw)
To: git
Hi,
Say, I have two repositories A and B (local, independent, but similar
- they are for content tracking and not collaboration purposes). A has
a branch 'a', which I want to have in B. What I mean is that I'd like
to have the sequence of changes in the branch 'a' to be present in B,
thus creating an independent branch 'b' in B.
Is there any way to achieve this? One thing that I could think of is
to use 'format-patch' to generate the list of patch files from A. But
I don't see how to convert those patches to a sequence of commits in
repo B. I could do a 'git apply patches/*' but then all patches
collapse to one single commit. If format-patch is a/the way, could
somebody tell me how to get this done? Or are there any alternatives?
FWIW: I'm running Git under Cygwin, and sendmail isn't configured.
Thanks
Jeenu
^ permalink raw reply
* [PATCH/RFC] Allow curl to rewind the RPC read buffer at any time
From: Martin Storsjö @ 2009-12-01 10:37 UTC (permalink / raw)
To: Tay Ray Chuan
Cc: git, Nicholas Miell, gsky51, Clemens Buchacher, Mark Lodato,
Johannes Schindelin
In-Reply-To: <alpine.DEB.2.00.0912011208160.5582@cone.home.martin.st>
When using multi-pass authentication methods, the curl library may
need to rewind the read buffers used for providing data to HTTP POST,
if data has been output before a 401 error is received.
This solution buffers all data read by the curl library, in order to allow
it to rewind the reading buffer at any time later.
If communicating with a HTTP server that doesn't support the
Expect: 100-continue headers, all HTTP POST data will be sent before
the server replies with the 401 error containing the authentication
challenge.
The buffering is enabled only if the rpc_service function allocates a
buffer - this should perhaps be limited to the cases where http.authAny
is enabled.
Signed-off-by: Martin Storsjo <martin@martin.st>
---
remote-curl.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 files changed, 53 insertions(+), 0 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index a331bae..c1c5ccd 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -286,6 +286,10 @@ struct rpc_state {
size_t alloc;
size_t len;
size_t pos;
+ char *rewind_buf;
+ size_t rewind_buf_size;
+ size_t rewind_buf_write_pos;
+ size_t rewind_buf_read_pos;
int in;
int out;
struct strbuf result;
@@ -299,12 +303,28 @@ static size_t rpc_out(void *ptr, size_t eltsize,
struct rpc_state *rpc = buffer_;
size_t avail = rpc->len - rpc->pos;
+ if (rpc->rewind_buf && rpc->rewind_buf_read_pos < rpc->rewind_buf_write_pos) {
+ avail = rpc->rewind_buf_write_pos - rpc->rewind_buf_read_pos;
+ if (max < avail)
+ avail = max;
+ memcpy(ptr, rpc->rewind_buf + rpc->rewind_buf_read_pos, avail);
+ rpc->rewind_buf_read_pos += avail;
+ return avail;
+ }
+
if (!avail) {
avail = packet_read_line(rpc->out, rpc->buf, rpc->alloc);
if (!avail)
return 0;
rpc->pos = 0;
rpc->len = avail;
+
+ if (rpc->rewind_buf) {
+ ALLOC_GROW(rpc->rewind_buf, rpc->rewind_buf_write_pos + avail, rpc->rewind_buf_size);
+ memcpy(rpc->rewind_buf + rpc->rewind_buf_write_pos, rpc->buf, avail);
+ rpc->rewind_buf_write_pos += avail;
+ rpc->rewind_buf_read_pos += avail;
+ }
}
if (max < avail)
@@ -314,6 +334,26 @@ static size_t rpc_out(void *ptr, size_t eltsize,
return avail;
}
+#ifndef NO_CURL_IOCTL
+curlioerr rpc_ioctl(CURL *handle, int cmd, void *clientp)
+{
+ struct rpc_state *rpc = clientp;
+
+ switch (cmd) {
+ case CURLIOCMD_NOP:
+ return CURLIOE_OK;
+
+ case CURLIOCMD_RESTARTREAD:
+ rpc->rewind_buf_read_pos = 0;
+ rpc->pos = rpc->len;
+ return CURLIOE_OK;
+
+ default:
+ return CURLIOE_UNKNOWNCMD;
+ }
+}
+#endif
+
static size_t rpc_in(const void *ptr, size_t eltsize,
size_t nmemb, void *buffer_)
{
@@ -370,8 +410,18 @@ static int post_rpc(struct rpc_state *rpc)
*/
headers = curl_slist_append(headers, "Expect: 100-continue");
headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
+ if (rpc->rewind_buf) {
+ ALLOC_GROW(rpc->rewind_buf, rpc->rewind_buf_write_pos + rpc->len, rpc->rewind_buf_size);
+ memcpy(rpc->rewind_buf, rpc->buf, rpc->len);
+ rpc->rewind_buf_read_pos = rpc->len;
+ rpc->rewind_buf_write_pos = rpc->len;
+ }
curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);
curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);
+#ifndef NO_CURL_IOCTL
+ curl_easy_setopt(slot->curl, CURLOPT_IOCTLFUNCTION, rpc_ioctl);
+ curl_easy_setopt(slot->curl, CURLOPT_IOCTLDATA, rpc);
+#endif
if (options.verbosity > 1) {
fprintf(stderr, "POST %s (chunked)\n", rpc->service_name);
fflush(stderr);
@@ -472,6 +522,8 @@ static int rpc_service(struct rpc_state *rpc, struct discovery *heads)
rpc->buf = xmalloc(rpc->alloc);
rpc->in = client.in;
rpc->out = client.out;
+ rpc->rewind_buf_size = 100;
+ rpc->rewind_buf = xmalloc(rpc->rewind_buf_size);
strbuf_init(&rpc->result, 0);
strbuf_addf(&buf, "%s/%s", url, svc);
@@ -503,6 +555,7 @@ static int rpc_service(struct rpc_state *rpc, struct discovery *heads)
free(rpc->hdr_content_type);
free(rpc->hdr_accept);
free(rpc->buf);
+ free(rpc->rewind_buf);
strbuf_release(&buf);
return err;
}
--
1.6.4.4
^ permalink raw reply related
* [PATCH/RFC] Allow curl to rewind the RPC read buffer
From: Martin Storsjö @ 2009-12-01 10:33 UTC (permalink / raw)
To: Tay Ray Chuan
Cc: git, Nicholas Miell, gsky51, Clemens Buchacher, Mark Lodato,
Johannes Schindelin
In-Reply-To: <alpine.DEB.2.00.0912011208160.5582@cone.home.martin.st>
When using multi-pass authentication methods, the curl library may need
to rewind the read buffers used for providing data to HTTP POST, if data
has been output before a 401 error is received.
This is needed only when the first request (when the multi-pass
authentication method isn't initialized and hasn't received its challenge
yet) for a certain curl session is a chunked HTTP POST.
As long as the current rpc read buffer is the first one, we're able to
rewind without need for additional buffering.
The curl library currently starts sending data without waiting for a
response to the Expect: 100-continue header, due to a bug in curl that
exists up to curl version 7.19.7.
If the HTTP server doesn't handle Expect: 100-continue headers properly
(e.g. Lighttpd), the library has to start sending data without knowing
if the request will be successfully authenticated. In this case, this
rewinding solution is not sufficient - the whole request will be sent
before the 401 error is received.
Signed-off-by: Martin Storsjo <martin@martin.st>
---
The curl bug is yet unconfirmed upstream, discussed here:
http://article.gmane.org/gmane.comp.web.curl.library/25991
remote-curl.c | 30 ++++++++++++++++++++++++++++++
1 files changed, 30 insertions(+), 0 deletions(-)
diff --git a/remote-curl.c b/remote-curl.c
index a331bae..28b2a31 100644
--- a/remote-curl.c
+++ b/remote-curl.c
@@ -290,6 +290,7 @@ struct rpc_state {
int out;
struct strbuf result;
unsigned gzip_request : 1;
+ unsigned initial_buffer : 1;
};
static size_t rpc_out(void *ptr, size_t eltsize,
@@ -300,6 +301,7 @@ static size_t rpc_out(void *ptr, size_t eltsize,
size_t avail = rpc->len - rpc->pos;
if (!avail) {
+ rpc->initial_buffer = 0;
avail = packet_read_line(rpc->out, rpc->buf, rpc->alloc);
if (!avail)
return 0;
@@ -314,6 +316,29 @@ static size_t rpc_out(void *ptr, size_t eltsize,
return avail;
}
+#ifndef NO_CURL_IOCTL
+curlioerr rpc_ioctl(CURL *handle, int cmd, void *clientp)
+{
+ struct rpc_state *rpc = clientp;
+
+ switch (cmd) {
+ case CURLIOCMD_NOP:
+ return CURLIOE_OK;
+
+ case CURLIOCMD_RESTARTREAD:
+ if (rpc->initial_buffer) {
+ rpc->pos = 0;
+ return CURLIOE_OK;
+ }
+ fprintf(stderr, "Unable to rewind rpc post data - try increasing http.postBuffer\n");
+ return CURLIOE_FAILRESTART;
+
+ default:
+ return CURLIOE_UNKNOWNCMD;
+ }
+}
+#endif
+
static size_t rpc_in(const void *ptr, size_t eltsize,
size_t nmemb, void *buffer_)
{
@@ -370,8 +395,13 @@ static int post_rpc(struct rpc_state *rpc)
*/
headers = curl_slist_append(headers, "Expect: 100-continue");
headers = curl_slist_append(headers, "Transfer-Encoding: chunked");
+ rpc->initial_buffer = 1;
curl_easy_setopt(slot->curl, CURLOPT_READFUNCTION, rpc_out);
curl_easy_setopt(slot->curl, CURLOPT_INFILE, rpc);
+#ifndef NO_CURL_IOCTL
+ curl_easy_setopt(slot->curl, CURLOPT_IOCTLFUNCTION, rpc_ioctl);
+ curl_easy_setopt(slot->curl, CURLOPT_IOCTLDATA, rpc);
+#endif
if (options.verbosity > 1) {
fprintf(stderr, "POST %s (chunked)\n", rpc->service_name);
fflush(stderr);
--
1.6.4.4
^ permalink raw reply related
* Re: [PATCH 0/2] http: allow multi-pass authentication
From: Martin Storsjö @ 2009-12-01 10:28 UTC (permalink / raw)
To: Tay Ray Chuan
Cc: git, Nicholas Miell, gsky51, Clemens Buchacher, Mark Lodato,
Johannes Schindelin
In-Reply-To: <20091127234110.7b7e9993.rctay89@gmail.com>
On Fri, 27 Nov 2009, Tay Ray Chuan wrote:
> This patch series applies on top of master. It enables fetching and
> pushing over http with the most suitable authentication scheme chosen
> by curl when the http.authAny or GIT_HTTP_AUTH_ANY is set.
I also tested this, and things generally seem to work fine.
Thanks to the "maintain curl sessions" patch, only the first request needs
to be redone after getting the 401 error containing the authentication
challenge, the later ones work fine on the first try. However,
theoretically, I guess we can't be certain that the curl session really is
initialized for the later requests (we could be given a new fresh curl
session for some reason), or the first request could perhaps be a large,
(currently) non-rewindable POST.
Avoiding redoing large POST requests is generally accomplished by adding a
Expect: 100-continue header, and then waiting for a reply (either 100
continue or 401 unauthorized) to that header before actually sending the
POST body data. If the server doesn't support the Expect header (e.g.
Lighttpd doesn't support it), the client starts sending the POST body
after a timeout (1 second in libcurl).
(As a side note, chunked POST requests without a content-length header
isn't supported by lighttpd at all at the moment, neither in the stable
1.4 version nor in the new upcoming 1.5 branch.)
Normally, libcurl should add the Expect: 100-continue header
automatically, but for some reason
(http://article.gmane.org/gmane.comp.web.curl.library/25992) it doesn't,
so that's probably why we're manually adding that header in
remote-curl.c:371 at the moment. libcurl doesn't detect this at the moment
(http://article.gmane.org/gmane.comp.web.curl.library/25991) so it won't
wait for the 100 continue response before starting to send the body data.
So, with a server supporting Expect, the 401 error response may come after
sending a few KB of POST data (corresponding to the roundtrip delay for
the server to respond to the header) - if the server doesn't support
Expect at all, the whole request will be sent and may need to be rewound.
To clarify - this only happens if the curl authentication isn't
initialized yet, for the first request of every curl session. The
"maintain curl sessions" patch makes sure this isn't needed in the normal
case.
I've experimented with two solutions to this, which add partial and full
rewind solutions to the chunked POST requests - I'll send them as
follow-ups to this mail.
// Martin
^ permalink raw reply
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Johannes Sixt @ 2009-12-01 10:08 UTC (permalink / raw)
To: Thomas Singer; +Cc: git
In-Reply-To: <4B14DC20.6040808@syntevo.com>
Thomas Singer schrieb:
> To be more precise: Who is interpreting the bytes in the file names as
> characters? Windows, Git or Java?
In the case of git: Windows does it, using the console's codepage to
convert between bytes and Unicode.
I don't know about Java, but I guess that no conversion is necessary
because Java is Unicode-aware.
-- Hannes
^ permalink raw reply
* Re: non-US-ASCII file names (e.g. Hiragana) on Windows
From: Johannes Sixt @ 2009-12-01 10:00 UTC (permalink / raw)
To: Thomas Singer; +Cc: git
In-Reply-To: <4B14DA1A.4060505@syntevo.com>
Thomas Singer schrieb:
> Is it a German Windows limitation, that far-east characters are not
> supported on it (but work fine on a Japanese Windows), are there different
> (mysys)Git versions available or is this a configuration issue?
It is a matter of configuration.
Since 8 bits are not sufficient to support Japanese alphabet in addition
to the German alphabet, programs that are not Unicode aware -- such as git
-- have to make a decision which alphabet they support. The decision is
made by picking a "codepage".
On German Windows, you are in codepage 850 (in the console). The filenames
(that actually are in Unicode) are converted to bytes according to
codepage 850 *before* git sees them. If your filenames contain Hiragana,
they are substituted by the "unknown character" marker because there is no
place for them in codepage 850.
However, you can install Japanese language support on German Windows. Then
you can change your console to codepage 932:
chcp 932
When you run git from *this* console, Hiragana in the filenames are
converted to cp932 before git sees them. The resulting byte sequence is
different from the one in cp850, but git will be able to see that the file
exists and was modified, and you can 'git add' it.
But if you have files with umlauts, they will not be recognized anymore
because umlauts have no place in cp932.
In neither case can you exchange the repository with Linux if you have
your locale set to UTF-8 on Linux, because neither byte sequence (umlauts
from cp850 or Hiragana from cp932) are valid UTF-8 sequences, let alone
result in the expected glyphs.
Corollary: Stick to ASCII file names.
There have been suggestions to switch the console to codepage 65001
(UTF-8), but I have never heard of success reports. I'm not saying it does
not work, though.
-- Hannes
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox