* Re: [PATCH] Add support for parallel HTTP transfers
From: Nick Hengeveld @ 2005-10-07 4:56 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7virwa5ety.fsf@assigned-by-dhcp.cox.net>
On Thu, Oct 06, 2005 at 05:51:53PM -0700, Junio C Hamano wrote:
> Well, I'd suggest just to hardcode a reasonable value to be a
> good net citizen, and not make it configurable. Four, perhaps?
>
> OTOH, we may want to have an option to disable parallel from the
> command line (I think -r 1 would mean that with yours).
I'd prefer to keep it configurable - for our purposes we'll be hitting
a single server from several clients and will probably want to limit
concurrent connections to something like two per client, but when doing
a fetch from a big server farm more connections would make sense.
> If we really want to have the number of parallel configurable,
> and -r implies recursive as you say, maybe '-j' to mimic
> parallel make?
Not that I'm a huge fan of using environment variables, but it might make
sense to use one here. That would allow the setting to work whether
git-http-fetch is run directly or via git-fetch. GIT_HTTP_MAX_REQUESTS?
--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
^ permalink raw reply
* Re: [PATCH] Show original and resulting blob object info in diff output.
From: Linus Torvalds @ 2005-10-07 4:47 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Eric W. Biederman, git
In-Reply-To: <7vachm5a4w.fsf_-_@assigned-by-dhcp.cox.net>
On Thu, 6 Oct 2005, Junio C Hamano wrote:
>
> diff --git a/apply.c b/apply.c
> old index 7be50413538868412a87c847f8fa184cadd0fa2a 100644
> new index 83660822fcbb9edac523634999e30d65c2790cae 100644
Hmm.. It strikes me that if we don't have the SHA1 object, then the SHA1
is largely useless except as a verification of the thing (ie we can still
_calculate_ the blob sha of whatever the current file is).
And if we _do_ have the SHA1, then we don't actually need all of it.
So the above is just very ugly, but it would be equally useful if it just
had a partial SHA1 - still plenty good to identify the thing.
IO, why not make it shorter ans prettier and just make it be
index 7be50413538868412a87..83660822fcbb9edac523
which is basically the first 20 hex digits of each SHA1.
Still plenty good to look up the object if we have it, and _certainly_
good enough as a quick verification. And much less "imposing".
The thing is, I actually tend to look at diffs when I send stuff out or
receive them, and your one is just very ugly:
> applies-to: 8cf0bb652ffde36e57bca5d6f33287305a61bb74
> 2c48a811a36ee6e12aeb044b3d1ff459b8d958c2
> diff --git a/apply.c b/apply.c
> old index 7be50413538868412a87c847f8fa184cadd0fa2a 100644
> new index ed2eac2c67dd003c8a0eaa7325e1543d1472777d 100644
> --- a/apply.c
> +++ b/apply.c
> @@ -14,6 +14,7 @@
That's a _lot_ of very "solid" unreadable stuff. It makes my eyes water.
I'd really like to minimize it.
If you want the "applies-to" etc, then _all_ of it could be just crammed
onto just one line (just cut each SHA1 down to 10 hex characters: 40 bits
is still _way_ enough to indicate a unique version of a particular file if
we have it, even if it's "just" a quarter of the full name. It's not
like this is meant to be cryptographic. In fact, I bet 20 bits would be
plenty).
Linus
^ permalink raw reply
* [PATCH] Show original and resulting blob object info in diff output.
From: Junio C Hamano @ 2005-10-07 2:33 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Eric W. Biederman, git
In-Reply-To: <Pine.LNX.4.64.0510060756060.31407@g5.osdl.org>
This adds more cruft to diff --git header to record the blob SHA1 and
the mode the patch/diff is intended to be applied against, to help the
receiving end fall back on a three-way merge. The new header looks
like this:
diff --git a/apply.c b/apply.c
old index 7be50413538868412a87c847f8fa184cadd0fa2a 100644
new index 83660822fcbb9edac523634999e30d65c2790cae 100644
--- a/apply.c
+++ b/apply.c
@@ -14,6 +14,7 @@
// files that are being modified, but doesn't apply the patch
// --stat does just a diffstat, and doesn't actually apply
+// --show-index-info shows the old and new index info for...
...
There is a counterpart option --show-index-info to git-apply command
to summarize this:
- 7be5041... 100644 apply.c
+ 8366082... 100644 apply.c
- ec2a161... 100644 cache.h
+ 514adb8... 100644 cache.h
- ...
Upon receiving such a patch, if the patch did not apply cleanly to the
target tree, the recipient can try to find the matching old objects in
her object database and create a temporary tree, apply the patch to
that temporary tree, and attempt a 3-way merge between the patched
temporary tree and the target tree using the original temporary tree
as the common ancestor.
The patch lifts the code to compute the hash for an on-filesystem
object from update-index.c and makes it available to the diff output
routine.
Signed-off-by: Junio C Hamano <junkio@cox.net>
---
Linus Torvalds <torvalds@osdl.org> writes:
> On Thu, 6 Oct 2005, Eric W. Biederman wrote:
>>
>> After thinking about it I don't think you need to look through the
>> history to use it for a merge3 operation. As I recall merge3 only
>> looks at the base and the two derived versions of the file. If we
>> have the sha1 of the original in the git repository I think all we
>> need to compute is the diff between that sha1 the current version
>> file. And then apply the merge3 algorithm to combine the two sets of
>> changes.
>
> Ahh, that I can definitely agree with. In fact, it makes perfect sense.
And this is a sample implementation to do so. Will not
graduate "pu" until the receiving end gets ready and unless
people agree that the additional cruft is not too
distracting.
apply.c | 64 +++++++++++++++++++++++++++++++++-
cache.h | 1 +
diff.c | 76 +++++++++++++++++++++++++++++-----------
sha1_file.c | 39 +++++++++++++++++++++
t/diff-lib.sh | 10 ++++-
t/t4000-diff-format.sh | 6 +++
t/t4001-diff-rename.sh | 3 +-
t/t4004-diff-rename-symlink.sh | 3 +-
update-index.c | 33 ++---------------
9 files changed, 179 insertions(+), 56 deletions(-)
applies-to: 8cf0bb652ffde36e57bca5d6f33287305a61bb74
2c48a811a36ee6e12aeb044b3d1ff459b8d958c2
diff --git a/apply.c b/apply.c
old index 7be50413538868412a87c847f8fa184cadd0fa2a 100644
new index ed2eac2c67dd003c8a0eaa7325e1543d1472777d 100644
--- a/apply.c
+++ b/apply.c
@@ -14,6 +14,7 @@
// files that are being modified, but doesn't apply the patch
// --stat does just a diffstat, and doesn't actually apply
// --show-files shows the directory changes
+// --show-index-info shows the old and new index info for paths if available.
//
static int check_index = 0;
static int write_index = 0;
@@ -22,8 +23,9 @@ static int summary = 0;
static int check = 0;
static int apply = 1;
static int show_files = 0;
+static int show_index_info = 0;
static const char apply_usage[] =
-"git-apply [--stat] [--summary] [--check] [--index] [--apply] [--show-files] <patch>...";
+"git-apply [--stat] [--summary] [--check] [--index] [--apply] [--show-files] [--show-index-info] <patch>...";
/*
* For "diff-stat" like behaviour, we keep track of the biggest change
@@ -56,6 +58,8 @@ struct patch {
struct fragment *fragments;
char *result;
unsigned long resultsize;
+ unsigned char old_sha1[20];
+ unsigned char new_sha1[20];
struct patch *next;
};
@@ -334,6 +338,24 @@ static int gitdiff_dissimilarity(const c
return 0;
}
+static int gitdiff_old_index(const char *line, struct patch *patch)
+{
+ if (get_sha1_hex(line, patch->old_sha1))
+ memcpy(patch->old_sha1, null_sha1, 20);
+ else
+ gitdiff_oldmode(line + 41, patch);
+ return 0;
+}
+
+static int gitdiff_new_index(const char *line, struct patch *patch)
+{
+ if (get_sha1_hex(line, patch->new_sha1))
+ memcpy(patch->new_sha1, null_sha1, 20);
+ else
+ gitdiff_newmode(line + 41, patch);
+ return 0;
+}
+
/*
* This is normal for a diff that doesn't change anything: we'll fall through
* into the next diff. Tell the parser to break out.
@@ -438,6 +460,8 @@ static int parse_git_header(char *line,
{ "rename to ", gitdiff_renamedst },
{ "similarity index ", gitdiff_similarity },
{ "dissimilarity index ", gitdiff_dissimilarity },
+ { "old index ", gitdiff_old_index },
+ { "new index ", gitdiff_new_index },
{ "", gitdiff_unrecognized },
};
int i;
@@ -1136,6 +1160,36 @@ static void show_file_list(struct patch
}
}
+static inline int is_null_sha1(const unsigned char *sha1)
+{
+ return !memcmp(sha1, null_sha1, 20);
+}
+
+static void show_index_list(struct patch *list)
+{
+ struct patch *patch;
+
+ for (patch = list; patch; patch = patch->next) {
+ if ( (!patch->is_delete && is_null_sha1(patch->new_sha1)) ||
+ (!patch->is_new && is_null_sha1(patch->old_sha1)) )
+ die("patch does not record sha1 information");
+ }
+
+ for (patch = list; patch; patch = patch->next) {
+ if (!is_null_sha1(patch->old_sha1))
+ printf("- %s %06o %s\n",
+ sha1_to_hex(patch->old_sha1),
+ patch->old_mode,
+ patch->old_name);
+
+ if (!is_null_sha1(patch->new_sha1))
+ printf("+ %s %06o %s\n",
+ sha1_to_hex(patch->new_sha1),
+ patch->new_mode,
+ patch->new_name);
+ }
+}
+
static void stat_patch_list(struct patch *patch)
{
int files, adds, dels;
@@ -1476,6 +1530,9 @@ static int apply_patch(int fd)
if (show_files)
show_file_list(list);
+ if (show_index_info)
+ show_index_list(list);
+
if (diffstat)
stat_patch_list(list);
@@ -1534,6 +1591,11 @@ int main(int argc, char **argv)
show_files = 1;
continue;
}
+ if (!strcmp(arg, "--show-index-info")) {
+ apply = 0;
+ show_index_info = 1;
+ continue;
+ }
fd = open(arg, O_RDONLY);
if (fd < 0)
usage(apply_usage);
diff --git a/cache.h b/cache.h
old index ec2a1610b2fd6edec6c95847d4377f9c0241b738 100644
new index 514adb8f8ed621d98175bea0d701576de13eb620 100644
--- a/cache.h
+++ b/cache.h
@@ -165,6 +165,7 @@ extern int ce_match_stat(struct cache_en
extern int ce_modified(struct cache_entry *ce, struct stat *st);
extern int ce_path_match(const struct cache_entry *ce, const char **pathspec);
extern int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, const char *type);
+extern int index_path(unsigned char *sha1, const char *path, struct stat *st, int write_object);
extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);
struct cache_file {
diff --git a/diff.c b/diff.c
old index 7d06b035ae8b6a53f10a7f94251a24d911baa509 100644
new index b602e25ccb2b4cf56dbfb5f2797651b1e44d4585 100644
--- a/diff.c
+++ b/diff.c
@@ -596,15 +596,31 @@ static void run_external_diff(const char
remove_tempfile();
}
+static void diff_fill_sha1_info(struct diff_filespec *one)
+{
+ if (DIFF_FILE_VALID(one)) {
+ if (!one->sha1_valid) {
+ struct stat st;
+ if (stat(one->path, &st) < 0)
+ die("stat %s", one->path);
+ if (index_path(one->sha1, one->path, &st, 0))
+ die("cannot hash %s\n", one->path);
+ }
+ }
+ else
+ memset(one->sha1, 0, 20);
+}
+
static void run_diff(struct diff_filepair *p)
{
const char *pgm = external_diff();
- char msg_[PATH_MAX*2+200], *xfrm_msg;
+ char msg[PATH_MAX*2+300], *xfrm_msg;
struct diff_filespec *one;
struct diff_filespec *two;
const char *name;
const char *other;
int complete_rewrite = 0;
+ int len;
if (DIFF_PAIR_UNMERGED(p)) {
/* unmerged */
@@ -616,38 +632,58 @@ static void run_diff(struct diff_filepai
name = p->one->path;
other = (strcmp(name, p->two->path) ? p->two->path : NULL);
one = p->one; two = p->two;
+
+ diff_fill_sha1_info(one);
+ diff_fill_sha1_info(two);
+
+ len = 0;
+ if (memcmp(one->sha1, two->sha1, 20)) {
+ if (memcmp(one->sha1, null_sha1, 20))
+ len += snprintf(msg + len, sizeof(msg) - len,
+ "old index %s %06o\n",
+ sha1_to_hex(one->sha1),
+ one->mode);
+ if (memcmp(two->sha1, null_sha1, 20))
+ len += snprintf(msg + len, sizeof(msg) - len,
+ "new index %s %06o\n",
+ sha1_to_hex(two->sha1),
+ two->mode);
+ }
+
switch (p->status) {
case DIFF_STATUS_COPIED:
- sprintf(msg_,
- "similarity index %d%%\n"
- "copy from %s\n"
- "copy to %s",
- (int)(0.5 + p->score * 100.0/MAX_SCORE),
- name, other);
- xfrm_msg = msg_;
+ len += snprintf(msg + len, sizeof(msg) - len,
+ "similarity index %d%%\n"
+ "copy from %s\n"
+ "copy to %s\n",
+ (int)(0.5 + p->score * 100.0/MAX_SCORE),
+ name, other);
break;
case DIFF_STATUS_RENAMED:
- sprintf(msg_,
- "similarity index %d%%\n"
- "rename from %s\n"
- "rename to %s",
- (int)(0.5 + p->score * 100.0/MAX_SCORE),
- name, other);
- xfrm_msg = msg_;
+ len += snprintf(msg + len, sizeof(msg) - len,
+ "similarity index %d%%\n"
+ "rename from %s\n"
+ "rename to %s\n",
+ (int)(0.5 + p->score * 100.0/MAX_SCORE),
+ name, other);
break;
case DIFF_STATUS_MODIFIED:
if (p->score) {
- sprintf(msg_,
- "dissimilarity index %d%%",
- (int)(0.5 + p->score * 100.0/MAX_SCORE));
- xfrm_msg = msg_;
+ len += snprintf(msg + len, sizeof(msg) - len,
+ "dissimilarity index %d%%\n",
+ (int)(0.5 + p->score *
+ 100.0/MAX_SCORE));
complete_rewrite = 1;
break;
}
/* fallthru */
default:
- xfrm_msg = NULL;
+ /* nothing */
+ ;
}
+ if (len)
+ msg[--len] = 0;
+ xfrm_msg = len ? msg : NULL;
if (!pgm &&
DIFF_FILE_VALID(one) && DIFF_FILE_VALID(two) &&
diff --git a/sha1_file.c b/sha1_file.c
old index 895c1fab6fc00b8131e44851f33b79b1d2310b12 100644
new index 287f618827d9b2fd472c06add42ead65de291822 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1545,3 +1545,42 @@ int index_fd(unsigned char *sha1, int fd
munmap(buf, size);
return ret;
}
+
+int index_path(unsigned char *sha1, const char *path, struct stat *st, int write_object)
+{
+ int fd;
+ char *target;
+
+ switch (st->st_mode & S_IFMT) {
+ case S_IFREG:
+ fd = open(path, O_RDONLY);
+ if (fd < 0)
+ return error("open(\"%s\"): %s", path,
+ strerror(errno));
+ if (index_fd(sha1, fd, st, write_object, NULL) < 0)
+ return error("%s: failed to insert into database",
+ path);
+ break;
+ case S_IFLNK:
+ target = xmalloc(st->st_size+1);
+ if (readlink(path, target, st->st_size+1) != st->st_size) {
+ char *errstr = strerror(errno);
+ free(target);
+ return error("readlink(\"%s\"): %s", path,
+ errstr);
+ }
+ if (!write_object) {
+ unsigned char hdr[50];
+ int hdrlen;
+ write_sha1_file_prepare(target, st->st_size, "blob",
+ sha1, hdr, &hdrlen);
+ } else if (write_sha1_file(target, st->st_size, "blob", sha1))
+ return error("%s: failed to insert into database",
+ path);
+ free(target);
+ break;
+ default:
+ return error("%s: unsupported file type", path);
+ }
+ return 0;
+}
diff --git a/t/diff-lib.sh b/t/diff-lib.sh
old index a912f435aa67d7b2cde09f0b8c9855442c6c2377 100755
new index a4095db2ed3b57d5e042c5ca49dd912c1c534cf8 100755
--- a/t/diff-lib.sh
+++ b/t/diff-lib.sh
@@ -29,7 +29,13 @@ compare_diff_raw_z () {
compare_diff_patch () {
# When heuristics are improved, the score numbers would change.
# Ignore them while comparing.
- sed -e '/^[dis]*imilarity index [0-9]*%$/d' <"$1" >.tmp-1
- sed -e '/^[dis]*imilarity index [0-9]*%$/d' <"$2" >.tmp-2
+ sed -e '
+ /^[dis]*imilarity index [0-9]*%$/d
+ /^[no][el][dw] index [0-9a-f]* [0-7]*$/d
+ ' <"$1" >.tmp-1
+ sed -e '
+ /^[dis]*imilarity index [0-9]*%$/d
+ /^[no][el][dw] index [0-9a-f]* [0-7]*$/d
+ ' <"$2" >.tmp-2
diff -u .tmp-1 .tmp-2 && rm -f .tmp-1 .tmp-2
}
diff --git a/t/t4000-diff-format.sh b/t/t4000-diff-format.sh
old index f3b6330a9b4af0e68d9e402ab7d82e600c939ccc 100755
new index fa387cc24b8e5e6c92df9d1d504b805c92b94e36 100755
--- a/t/t4000-diff-format.sh
+++ b/t/t4000-diff-format.sh
@@ -7,6 +7,7 @@ test_description='Test built-in diff out
'
. ./test-lib.sh
+. ../diff-lib.sh
echo >path0 'Line 1
Line 2
@@ -29,6 +30,8 @@ cat >expected <<\EOF
diff --git a/path0 b/path0
old mode 100644
new mode 100755
+old index 85d7fd2fbbe9564fe6ecc5e72416adb36e62d658 100644
+new index 6ad36e52f0002937ed2de6a1c15d8a0ae5df056a 100755
--- a/path0
+++ b/path0
@@ -1,3 +1,3 @@
@@ -38,6 +41,7 @@ new mode 100755
+Line 3
diff --git a/path1 b/path1
deleted file mode 100755
+old index 85d7fd2fbbe9564fe6ecc5e72416adb36e62d658 100755
--- a/path1
+++ /dev/null
@@ -1,3 +0,0 @@
@@ -48,6 +52,6 @@ EOF
test_expect_success \
'validate git-diff-files -p output.' \
- 'cmp -s current expected'
+ 'compare_diff_patch current expected'
test_done
diff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh
old index be474856824f8e6659151590ed5acff38187e97d 100755
new index 2e3c20d6b9468bf413e97d422e7dbe13ac4238cd 100755
--- a/t/t4001-diff-rename.sh
+++ b/t/t4001-diff-rename.sh
@@ -7,6 +7,7 @@ test_description='Test rename detection
'
. ./test-lib.sh
+. ../diff-lib.sh
echo >path0 'Line 1
Line 2
@@ -61,6 +62,6 @@ EOF
test_expect_success \
'validate the output.' \
- 'diff -I "similarity.*" >/dev/null current expected'
+ 'compare_diff_patch current expected'
test_done
diff --git a/t/t4004-diff-rename-symlink.sh b/t/t4004-diff-rename-symlink.sh
old index f59614ae255b33f450a784200716c9fd63b0a054 100755
new index a23aaa0a9471c68b233480cf34c7115d1f40e154 100755
--- a/t/t4004-diff-rename-symlink.sh
+++ b/t/t4004-diff-rename-symlink.sh
@@ -10,6 +10,7 @@ copy of symbolic links, but should not p
by an edit for them.
'
. ./test-lib.sh
+. ../diff-lib.sh
test_expect_success \
'prepare reference tree' \
@@ -61,6 +62,6 @@ EOF
test_expect_success \
'validate diff output' \
- 'diff -u current expected'
+ 'compare_diff_patch current expected'
test_done
diff --git a/update-index.c b/update-index.c
old index b825a11d2f6d8a53f5e42e4f157ba036a13edb59 100644
new index 9dc518877b67bedced29ac91ecd8fafbb2a60135 100644
--- a/update-index.c
+++ b/update-index.c
@@ -37,8 +37,6 @@ static int add_file_to_cache(const char
int size, namelen, option, status;
struct cache_entry *ce;
struct stat st;
- int fd;
- char *target;
status = lstat(path, &st);
if (status < 0 || S_ISDIR(st.st_mode)) {
@@ -77,34 +75,9 @@ static int add_file_to_cache(const char
fill_stat_cache_info(ce, &st);
ce->ce_mode = create_ce_mode(st.st_mode);
ce->ce_flags = htons(namelen);
- switch (st.st_mode & S_IFMT) {
- case S_IFREG:
- fd = open(path, O_RDONLY);
- if (fd < 0)
- return error("open(\"%s\"): %s", path, strerror(errno));
- if (index_fd(ce->sha1, fd, &st, !info_only, NULL) < 0)
- return error("%s: failed to insert into database", path);
- break;
- case S_IFLNK:
- target = xmalloc(st.st_size+1);
- if (readlink(path, target, st.st_size+1) != st.st_size) {
- char *errstr = strerror(errno);
- free(target);
- return error("readlink(\"%s\"): %s", path,
- errstr);
- }
- if (info_only) {
- unsigned char hdr[50];
- int hdrlen;
- write_sha1_file_prepare(target, st.st_size, "blob",
- ce->sha1, hdr, &hdrlen);
- } else if (write_sha1_file(target, st.st_size, "blob", ce->sha1))
- return error("%s: failed to insert into database", path);
- free(target);
- break;
- default:
- return error("%s: unsupported file type", path);
- }
+
+ if (index_path(ce->sha1, path, &st, !info_only))
+ return -1;
option = allow_add ? ADD_CACHE_OK_TO_ADD : 0;
option |= allow_replace ? ADD_CACHE_OK_TO_REPLACE : 0;
if (add_cache_entry(ce, option))
---
0.99.8.GIT
^ permalink raw reply
* Re: [PATCH] Add support for parallel HTTP transfers
From: Junio C Hamano @ 2005-10-07 0:51 UTC (permalink / raw)
To: Nick Hengeveld; +Cc: git
In-Reply-To: <20051007000041.GH15593@reactrix.com>
Nick Hengeveld <nickh@reactrix.com> writes:
> About that "-r" arg - seems like it should be something else as -r is
> used elsewhere in git to enable recursion. "-c" was my first thought,
> but that's used to fetch commit objects.
Well, I'd suggest just to hardcode a reasonable value to be a
good net citizen, and not make it configurable. Four, perhaps?
OTOH, we may want to have an option to disable parallel from the
command line (I think -r 1 would mean that with yours).
If we really want to have the number of parallel configurable,
and -r implies recursive as you say, maybe '-j' to mimic
parallel make?
^ permalink raw reply
* [PATCH] Only compile parallel HTTP support with CURL >= 7.9.8
From: Nick Hengeveld @ 2005-10-07 0:04 UTC (permalink / raw)
To: git
Only compile parallel HTTP support with CURL >= 7.9.8
Signed-off-by: Nick Hengeveld <nickh@reactrix.com>
---
http-fetch.c | 62 +++++++++++++++++++++++++++++++++++++++++++++++++++++-----
1 files changed, 57 insertions(+), 5 deletions(-)
2677e4eab199eb43f5e3a865e4008d3db79708a5
diff --git a/http-fetch.c b/http-fetch.c
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -6,7 +6,10 @@
#include <curl/curl.h>
#include <curl/easy.h>
+#if LIBCURL_VERSION_NUM >= 0x070908
+#define USE_CURL_MULTI
#define DEFAULT_MAX_REQUESTS 5
+#endif
#if LIBCURL_VERSION_NUM < 0x070704
#define curl_global_cleanup() do { /* nothing */ } while(0)
@@ -18,11 +21,13 @@
#define PREV_BUF_SIZE 4096
#define RANGE_HEADER_SIZE 30
-static int max_requests = DEFAULT_MAX_REQUESTS;
static int active_requests = 0;
static int data_received;
+#ifdef USE_CURL_MULTI
+static int max_requests = DEFAULT_MAX_REQUESTS;
static CURLM *curlm;
+#endif
static CURL *curl_default;
static struct curl_slist *no_pragma_header;
static struct curl_slist *no_range_header;
@@ -151,13 +156,17 @@ int relink_or_rename(char *old, char *ne
return 0;
}
+#ifdef USE_CURL_MULTI
void process_curl_messages();
void process_request_queue();
+#endif
struct active_request_slot *get_active_slot()
{
struct active_request_slot *slot = active_queue_head;
struct active_request_slot *newslot;
+
+#ifdef USE_CURL_MULTI
int num_transfers;
/* Wait for a slot to open up if the queue is full */
@@ -167,6 +176,7 @@ struct active_request_slot *get_active_s
process_curl_messages();
}
}
+#endif
while (slot != NULL && slot->in_use) {
slot = slot->next;
@@ -202,6 +212,7 @@ struct active_request_slot *get_active_s
int start_active_slot(struct active_request_slot *slot)
{
+#ifdef USE_CURL_MULTI
CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);
if (curlm_result != CURLM_OK &&
@@ -210,12 +221,13 @@ int start_active_slot(struct active_requ
slot->in_use = 0;
return 0;
}
-
+#endif
return 1;
}
void run_active_slot(struct active_request_slot *slot)
{
+#ifdef USE_CURL_MULTI
int num_transfers;
long last_pos = 0;
long current_pos;
@@ -255,6 +267,10 @@ void run_active_slot(struct active_reque
&excfds, &select_timeout);
}
}
+#else
+ slot->curl_result = curl_easy_perform(slot->curl);
+ active_requests--;
+#endif
}
void start_request(struct transfer_request *request)
@@ -356,7 +372,7 @@ void start_request(struct transfer_reque
CURLOPT_HTTPHEADER, range_header);
}
- /* Try to add to multi handle, abort the request on error */
+ /* Try to get the request started, abort the request on error */
if (!start_active_slot(slot)) {
request->state = ABORTED;
close(request->local);
@@ -413,6 +429,7 @@ void release_request(struct transfer_req
free(request);
}
+#ifdef USE_CURL_MULTI
void process_curl_messages()
{
int num_messages;
@@ -478,6 +495,7 @@ void process_request_queue()
request = request->next;
}
}
+#endif
void prefetch(unsigned char *sha1)
{
@@ -505,8 +523,10 @@ void prefetch(unsigned char *sha1)
}
tail->next = newreq;
}
+#ifdef USE_CURL_MULTI
process_request_queue();
process_curl_messages();
+#endif
}
static int got_alternates = 0;
@@ -864,13 +884,14 @@ static int fetch_object(struct alt_base
char *hex = sha1_to_hex(sha1);
int ret;
struct transfer_request *request = request_queue_head;
- int num_transfers;
while (request != NULL && memcmp(request->sha1, sha1, 20))
request = request->next;
if (request == NULL)
return error("Couldn't find request for %s in the queue", hex);
+#ifdef USE_CURL_MULTI
+ int num_transfers;
while (request->state == WAITING) {
curl_multi_perform(curlm, &num_transfers);
if (num_transfers < active_requests) {
@@ -878,9 +899,30 @@ static int fetch_object(struct alt_base
process_request_queue();
}
}
+#else
+ start_request(request);
+#endif
- if (request->state == ACTIVE)
+ while (request->state == ACTIVE) {
run_active_slot(request->slot);
+#ifndef USE_CURL_MULTI
+ request->curl_result = request->slot->curl_result;
+ curl_easy_getinfo(request->slot->curl,
+ CURLINFO_HTTP_CODE,
+ &request->http_code);
+ request->slot = NULL;
+
+ /* Use alternates if necessary */
+ if (request->http_code == 404 &&
+ request->repo->next != NULL) {
+ request->repo = request->repo->next;
+ start_request(request);
+ } else {
+ finish_request(request);
+ request->state = COMPLETE;
+ }
+#endif
+ }
if (request->state == ABORTED) {
release_request(request);
@@ -992,16 +1034,22 @@ int main(int argc, char **argv)
arg++;
} else if (!strcmp(argv[arg], "--recover")) {
get_recover = 1;
+#ifdef USE_CURL_MULTI
} else if (argv[arg][1] == 'r') {
max_requests = atoi(argv[arg + 1]);
if (max_requests < 1)
max_requests = DEFAULT_MAX_REQUESTS;
arg++;
+#endif
}
arg++;
}
if (argc < arg + 2) {
+#ifdef USE_CURL_MULTI
+ usage("git-http-fetch [-c] [-t] [-a] [-d] [-v] [-r concurrent-request-limit] [--recover] [-w ref] commit-id url");
+#else
usage("git-http-fetch [-c] [-t] [-a] [-d] [-v] [--recover] [-w ref] commit-id url");
+#endif
return 1;
}
commit_id = argv[arg];
@@ -1009,11 +1057,13 @@ int main(int argc, char **argv)
curl_global_init(CURL_GLOBAL_ALL);
+#ifdef USE_CURL_MULTI
curlm = curl_multi_init();
if (curlm == NULL) {
fprintf(stderr, "Error creating curl multi handle.\n");
return 1;
}
+#endif
no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
no_range_header = curl_slist_append(no_range_header, "Range:");
@@ -1061,7 +1111,9 @@ int main(int argc, char **argv)
curl_easy_cleanup(slot->curl);
slot = slot->next;
}
+#ifdef USE_CURL_MULTI
curl_multi_cleanup(curlm);
+#endif
curl_global_cleanup();
return 0;
}
^ permalink raw reply
* Re: [PATCH] Add support for parallel HTTP transfers
From: Nick Hengeveld @ 2005-10-07 0:00 UTC (permalink / raw)
To: Daniel Barkalow; +Cc: git
In-Reply-To: <Pine.LNX.4.63.0510061550510.23242@iabervon.org>
On Thu, Oct 06, 2005 at 04:07:07PM -0400, Daniel Barkalow wrote:
> Somewhat weirdly, the version of curl on my desktop doesn't actually have
> an implementation of curl_multi_info_read, although it's in the header
> file and documentation. So you'll want a version check somewhere, I think,
> which should probably just disable parallel transfers.
I was afraid that was going to happen... From the archived versions on the
CURL download site, it looks as though multi support was added in 7.9.8 -
which version do you have installed on your desktop?
I'll follow this up with a patch that works as you describe to disable
building with parallel transfer support on versions < 7.9.8.
> It should be fine to download objects and a pack that contains them at the
> same time, although there's currently a check in fetch.c which should be
> removed, so that it will call fetch() for an object if the object appears
> between the prefetch() and the fetch().
Can you provide a patch, or point me toward the right place to make that
change?
> I should be able to review this over the weekend. What sort of performance
> are you getting at this point (in terms of bandwidth utilization)?
I've done limited testing by using the time command to track real/user/sys
taken to run 'git fetch http://kernel.org/pub/scm/git/git.git master',
and have seen performance improve by a factor of ~2-10:
0.99.8:
real 2m48.800s, user 0m2.540s, sys 0m0.470s
real 2m40.316s, user 0m2.850s, sys 0m0.500s
real 2m8.543s, user 0m2.910s, sys 0m0.600s
real 2m18.009s, user 0m2.440s, sys 0m0.580s
real 1m55.354s, user 0m2.520s, sys 0m0.430s
Parallel: -r 5 (default)
real 0m49.499s, user 0m3.220s, sys 0m0.370s
real 1m0.177s, user 0m3.310s, sys 0m0.740s
real 0m52.936s, user 0m2.680s, sys 0m0.230s
real 1m0.158s, user 0m2.870s, sys 0m0.770s
real 0m52.780s, user 0m2.970s, sys 0m0.600s
Parallel: -r 10
real 0m28.338s, user 0m2.940s, sys 0m0.630s
real 0m35.944s, user 0m3.030s, sys 0m0.570s
real 0m18.019s, user 0m3.050s, sys 0m0.530s
real 0m21.539s, user 0m2.960s, sys 0m0.520s
real 0m31.405s, user 0m3.080s, sys 0m0.610s
Parallel: -r 20
real 0m25.810s, user 0m3.070s, sys 0m0.490s
real 0m16.265s, user 0m2.880s, sys 0m0.370s
real 0m28.536s, user 0m2.890s, sys 0m0.650s
real 0m16.889s, user 0m2.770s, sys 0m0.460s
real 0m23.125s, user 0m2.800s, sys 0m0.450s
Parallel: multi disabled
real 3m3.833s, user 0m12.080s, sys 0m3.240s
real 2m15.454s, user 0m12.130s, sys 0m2.820s
real 2m23.011s, user 0m12.690s, sys 0m3.030s
real 2m38.720s, user 0m12.300s, sys 0m2.850s
real 2m42.025s, usre 0m12.310s, sys 0m2.880s
That's all running on a CentOS 3.5 desktop with CURL 7.10.6.
About that "-r" arg - seems like it should be something else as -r is
used elsewhere in git to enable recursion. "-c" was my first thought,
but that's used to fetch commit objects.
--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
^ permalink raw reply
* Re: Allow "-u" flag to tag signing
From: Chris Wright @ 2005-10-06 23:32 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0510060952410.31407@g5.osdl.org>
* Linus Torvalds (torvalds@osdl.org) wrote:
> So rather than force my tagger ID to have to match the gpg key name, just
> support the "-u" flag to "git tag" instead. It implicitly enables signing,
> since it doesn't make any sense without it. Thus:
>
> git tag -u <gpg-key-name> <tag-name> [<tagged-object>]
>
> will use the named gpg key for signing.
Nice, I've had a hack something like this locally here as well, and been
meaning to cleanup and submit.
thanks,
-chris
^ permalink raw reply
* Create object subdirectories on demand
From: Linus Torvalds @ 2005-10-06 23:23 UTC (permalink / raw)
To: Git Mailing List, Junio C Hamano
This makes it possible to have a "sparse" git object subdirectory
structure, something that has become much more attractive now that people
use pack-files all the time.
As a result of pack-files, a git object directory doesn't necessarily have
any individual objects lying around, and in that case it's just wasting
space to keep the empty first-level object directories around: on many
filesystems the 256 empty directories will be aboue 1MB of diskspace.
Even more importantly, after you re-pack a project that _used_ to be
unpacked, you could be left with huge directories that no longer contain
anything, but that waste space and take time to look through.
With this change, "git prune-packed" can just do an rmdir() on the
directories, and they'll get removed if empty, and re-created on demand.
This patch also tries to fix up "write_sha1_from_fd()" to use the new
common infrastructure for creating the object files, closing a hole where
we might otherwise leave half-written objects in the object database.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
This has _not_ gotten a lot of testing, but I verified that basic things
seem to work, and that packing an archive properly removes the unnecessary
subdirectories.
I'd suggest it sit in "pu" for a while.
The primary reason for this was that a big git archive (the historic one)
ends up with lots of objects, and the object subdirectories end up being
about 150kB in size. Even repacking the archive did nothing for it - you
still ended up with 256 of these monster directories, never mind that they
were empty. Walking through them in the cold-cache case was actually
really painful - it would walk through about 40MB of disk very very
inefficiently, for no actual gain.
However, somebody should really check my code carefully before merging
this. In particular, I didn't test "git-ssh-pull" at all, so I'm not sure
I actually fixed the "write temp-file" thing properly.
diff --git a/daemon.c b/daemon.c
--- a/daemon.c
+++ b/daemon.c
@@ -142,7 +142,7 @@ static int upload(char *dir, int dirlen)
* is ok with us doing this.
*/
if ((!export_all_trees && access("git-daemon-export-ok", F_OK)) ||
- access("objects/00", X_OK) ||
+ access("objects/", X_OK) ||
access("HEAD", R_OK)) {
logerror("Not a valid git-daemon-enabled repository: '%s'", dir);
return -1;
diff --git a/fsck-objects.c b/fsck-objects.c
--- a/fsck-objects.c
+++ b/fsck-objects.c
@@ -329,9 +329,8 @@ static int fsck_dir(int i, char *path)
DIR *dir = opendir(path);
struct dirent *de;
- if (!dir) {
- return error("missing sha1 directory '%s'", path);
- }
+ if (!dir)
+ return 0;
while ((de = readdir(dir)) != NULL) {
char name[100];
diff --git a/git-rename.perl b/git-rename.perl
--- a/git-rename.perl
+++ b/git-rename.perl
@@ -15,7 +15,7 @@ sub usage($);
my $GIT_DIR = $ENV{'GIT_DIR'} || ".git";
unless ( -d $GIT_DIR && -d $GIT_DIR . "/objects" &&
- -d $GIT_DIR . "/objects/00" && -d $GIT_DIR . "/refs") {
+ -d $GIT_DIR . "/objects/" && -d $GIT_DIR . "/refs") {
usage("Git repository not found.");
}
diff --git a/git-sh-setup.sh b/git-sh-setup.sh
--- a/git-sh-setup.sh
+++ b/git-sh-setup.sh
@@ -22,4 +22,4 @@ refs/*) : ;;
*) false ;;
esac &&
[ -d "$GIT_DIR/refs" ] &&
-[ -d "$GIT_OBJECT_DIRECTORY/00" ]
+[ -d "$GIT_OBJECT_DIRECTORY/" ]
diff --git a/init-db.c b/init-db.c
--- a/init-db.c
+++ b/init-db.c
@@ -244,10 +244,6 @@ int main(int argc, char **argv)
memcpy(path, sha1_dir, len);
safe_create_dir(sha1_dir);
- for (i = 0; i < 256; i++) {
- sprintf(path+len, "/%02x", i);
- safe_create_dir(path);
- }
strcpy(path+len, "/pack");
safe_create_dir(path);
strcpy(path+len, "/info");
diff --git a/prune-packed.c b/prune-packed.c
--- a/prune-packed.c
+++ b/prune-packed.c
@@ -26,6 +26,8 @@ static void prune_dir(int i, DIR *dir, c
else if (unlink(pathname) < 0)
error("unable to unlink %s", pathname);
}
+ pathname[len] = 0;
+ rmdir(pathname);
}
static void prune_packed_objects(void)
@@ -46,7 +48,7 @@ static void prune_packed_objects(void)
sprintf(pathname + len, "%02x/", i);
d = opendir(pathname);
if (!d)
- die("unable to open %s", pathname);
+ continue;
prune_dir(i, d, pathname, len + 3);
closedir(d);
}
diff --git a/sha1_file.c b/sha1_file.c
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1248,6 +1248,73 @@ char *write_sha1_file_prepare(void *buf,
return sha1_file_name(sha1);
}
+/*
+ * Link the tempfile to the final place, possibly creating the
+ * last directory level as you do so.
+ *
+ * Returns the errno on failure, 0 on success.
+ */
+static int link_temp_to_file(const char *tmpfile, char *filename)
+{
+ int ret;
+
+ if (!link(tmpfile, filename))
+ return 0;
+
+ /*
+ * Try to mkdir the last path component if that failed
+ * with an ENOENT.
+ *
+ * Re-try the "link()" regardless of whether the mkdir
+ * succeeds, since a race might mean that somebody
+ * else succeeded.
+ */
+ ret = errno;
+ if (ret == ENOENT) {
+ char *dir = strrchr(filename, '/');
+ if (dir) {
+ *dir = 0;
+ mkdir(filename, 0777);
+ *dir = '/';
+ if (!link(tmpfile, filename))
+ return 0;
+ ret = errno;
+ }
+ }
+ return ret;
+}
+
+/*
+ * Move the just written object into its final resting place
+ */
+static int move_temp_to_file(const char *tmpfile, char *filename)
+{
+ int ret = link_temp_to_file(tmpfile, filename);
+ if (ret) {
+ /*
+ * Coda hack - coda doesn't like cross-directory links,
+ * so we fall back to a rename, which will mean that it
+ * won't be able to check collisions, but that's not a
+ * big deal.
+ *
+ * When this succeeds, we just return 0. We have nothing
+ * left to unlink.
+ */
+ if (ret == EXDEV && !rename(tmpfile, filename))
+ return 0;
+ }
+ unlink(tmpfile);
+ if (ret) {
+ if (ret != EEXIST) {
+ fprintf(stderr, "unable to write sha1 filename %s: %s", filename, strerror(ret));
+ return -1;
+ }
+ /* FIXME!!! Collision check here ? */
+ }
+
+ return 0;
+}
+
int write_sha1_file(void *buf, unsigned long len, const char *type, unsigned char *returnsha1)
{
int size;
@@ -1257,7 +1324,7 @@ int write_sha1_file(void *buf, unsigned
char *filename;
static char tmpfile[PATH_MAX];
unsigned char hdr[50];
- int fd, hdrlen, ret;
+ int fd, hdrlen;
/* Normally if we have it in the pack then we do not bother writing
* it out into .git/objects/??/?{38} file.
@@ -1320,32 +1387,7 @@ int write_sha1_file(void *buf, unsigned
close(fd);
free(compressed);
- ret = link(tmpfile, filename);
- if (ret < 0) {
- ret = errno;
-
- /*
- * Coda hack - coda doesn't like cross-directory links,
- * so we fall back to a rename, which will mean that it
- * won't be able to check collisions, but that's not a
- * big deal.
- *
- * When this succeeds, we just return 0. We have nothing
- * left to unlink.
- */
- if (ret == EXDEV && !rename(tmpfile, filename))
- return 0;
- }
- unlink(tmpfile);
- if (ret) {
- if (ret != EEXIST) {
- fprintf(stderr, "unable to write sha1 filename %s: %s", filename, strerror(ret));
- return -1;
- }
- /* FIXME!!! Collision check here ? */
- }
-
- return 0;
+ return move_temp_to_file(tmpfile, filename);
}
int write_sha1_to_fd(int fd, const unsigned char *sha1)
@@ -1420,8 +1462,7 @@ int write_sha1_to_fd(int fd, const unsig
int write_sha1_from_fd(const unsigned char *sha1, int fd, char *buffer,
size_t bufsize, size_t *bufposn)
{
- char *filename = sha1_file_name(sha1);
-
+ char tmpfile[PATH_MAX];
int local;
z_stream stream;
unsigned char real_sha1[20];
@@ -1429,10 +1470,11 @@ int write_sha1_from_fd(const unsigned ch
int ret;
SHA_CTX c;
- local = open(filename, O_WRONLY | O_CREAT | O_EXCL, 0666);
+ snprintf(tmpfile, sizeof(tmpfile), "%s/obj_XXXXXX", get_object_directory());
+ local = mkstemp(tmpfile);
if (local < 0)
- return error("Couldn't open %s\n", filename);
+ return error("Couldn't open %s for %s\n", tmpfile, sha1_to_hex(sha1));
memset(&stream, 0, sizeof(stream));
@@ -1462,7 +1504,7 @@ int write_sha1_from_fd(const unsigned ch
size = read(fd, buffer + *bufposn, bufsize - *bufposn);
if (size <= 0) {
close(local);
- unlink(filename);
+ unlink(tmpfile);
if (!size)
return error("Connection closed?");
perror("Reading from connection");
@@ -1475,15 +1517,15 @@ int write_sha1_from_fd(const unsigned ch
close(local);
SHA1_Final(real_sha1, &c);
if (ret != Z_STREAM_END) {
- unlink(filename);
+ unlink(tmpfile);
return error("File %s corrupted", sha1_to_hex(sha1));
}
if (memcmp(sha1, real_sha1, 20)) {
- unlink(filename);
+ unlink(tmpfile);
return error("File %s has bad hash\n", sha1_to_hex(sha1));
}
-
- return 0;
+
+ return move_temp_to_file(tmpfile, sha1_file_name(sha1));
}
int has_pack_index(const unsigned char *sha1)
^ permalink raw reply
* [PATCH] Reduce memory usage in git-update-server-info.
From: robfitz @ 2005-10-06 22:49 UTC (permalink / raw)
To: git; +Cc: Robert Fitzsimons
Modify parse_object_cheap() to also free all the entries from the tree
data structures.
Signed-off-by: Robert Fitzsimons <robfitz@273k.net>
---
server-info.c | 10 ++++++++++
1 files changed, 10 insertions(+), 0 deletions(-)
applies-to: fc843c89d12160dfe9704e7a0dd678cc729459cb
67be887f195205e36c7eec39b9caea249a435e95
diff --git a/server-info.c b/server-info.c
--- a/server-info.c
+++ b/server-info.c
@@ -59,6 +59,16 @@ static struct object *parse_object_cheap
struct commit *commit = (struct commit *)o;
free(commit->buffer);
commit->buffer = NULL;
+ } else if (o->type == tree_type) {
+ struct tree *tree = (struct tree *)o;
+ struct tree_entry_list *e, *n;
+ for (e = tree->entries; e; e = n) {
+ free(e->name);
+ e->name = NULL;
+ n = e->next;
+ free(e);
+ }
+ tree->entries = NULL;
}
return o;
}
---
0.99.8.GIT
^ permalink raw reply
* Re: Still unsafe: write_sha1_from_fd()
From: Junio C Hamano @ 2005-10-06 22:37 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0510061520400.31407@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> The only user is apparently ssh-fetch, but especially since the input-fd
> is a network connection, it looks like it is not at all unlikely that this
> case will trigger - all it takes is somebody impatient waiting for a large
> object.
>
> Ugh,
Ugh indeed.
I'd vote for renaming it to a less generic name, moving it out
of sha1_file.c -- make it static in ssh-fetch.c -- and slowly
deprecate the use of ssh-fetch/ssh-upload pair.
I do not see much point prefering it over the pack transfer
protocol anyway, if you already have ssh access to the box.
^ permalink raw reply
* Still unsafe: write_sha1_from_fd()
From: Linus Torvalds @ 2005-10-06 22:26 UTC (permalink / raw)
To: Git Mailing List
I was looking over the object writing functions, and noticed that
"write_sha1_from_fd()" writes its object directly into the final object
name, which means that if it is interrupted with ^C at an inopportune
time, it will leave a corrupt object behind.
The only user is apparently ssh-fetch, but especially since the input-fd
is a network connection, it looks like it is not at all unlikely that this
case will trigger - all it takes is somebody impatient waiting for a large
object.
Ugh,
Linus
^ permalink raw reply
* [PATCH] Fix wrong filename listing bug in git-ls-tree.
From: robfitz @ 2005-10-06 21:52 UTC (permalink / raw)
To: git; +Cc: Robert Fitzsimons
This patch fixes a bug in git-ls-tee in which the wrong filenames are
listed if the exact same file and directory contents are present in
another location in the tree.
Added a new series of test cases for directory and filename handling.
Signed-off-by: Robert Fitzsimons <robfitz@273k.net>
---
ls-tree.c | 55 ++++++++-------
t/t3101-ls-tree-dirname.sh | 160 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 189 insertions(+), 26 deletions(-)
create mode 100644 t/t3101-ls-tree-dirname.sh
applies-to: 9eec9ff9326032bca405a12a265918725edf4ac7
223dbed3ac931657f88466fec22e30e10197403e
diff --git a/ls-tree.c b/ls-tree.c
--- a/ls-tree.c
+++ b/ls-tree.c
@@ -54,11 +54,13 @@ static int prepare_children(struct tree_
return 0;
}
-static struct tree_entry_list *find_entry(const char *path)
+static struct tree_entry_list *find_entry(const char *path, char *pathbuf)
{
const char *next, *slash;
int len;
- struct tree_entry_list *elem = &root_entry;
+ struct tree_entry_list *elem = &root_entry, *oldelem = NULL;
+
+ *(pathbuf) = '\0';
/* Find tree element, descending from root, that
* corresponds to the named path, lazily expanding
@@ -86,6 +88,10 @@ static struct tree_entry_list *find_entr
len = slash - path;
}
if (len) {
+ if (oldelem) {
+ pathbuf += sprintf(pathbuf, "%s/", oldelem->name);
+ }
+
/* (len == 0) if the original path was "drivers/char/"
* and we have run already two rounds, having elem
* pointing at the drivers/char directory.
@@ -101,6 +107,8 @@ static struct tree_entry_list *find_entr
}
if (!elem)
return NULL;
+
+ oldelem = elem;
}
path = next;
}
@@ -108,19 +116,6 @@ static struct tree_entry_list *find_entr
return elem;
}
-static void show_entry_name(struct tree_entry_list *e)
-{
- /* This is yucky. The root level is there for
- * our convenience but we really want to do a
- * forest.
- */
- if (e->parent && e->parent != &root_entry) {
- show_entry_name(e->parent);
- putchar('/');
- }
- printf("%s", e->name);
-}
-
static const char *entry_type(struct tree_entry_list *e)
{
return (e->directory ? "tree" : "blob");
@@ -134,28 +129,35 @@ static const char *entry_hex(struct tree
}
/* forward declaration for mutually recursive routines */
-static int show_entry(struct tree_entry_list *, int);
+static int show_entry(struct tree_entry_list *, int, char *pathbuf);
-static int show_children(struct tree_entry_list *e, int level)
+static int show_children(struct tree_entry_list *e, int level, char *pathbuf)
{
+ int oldlen = strlen(pathbuf);
+
+ if (e != &root_entry)
+ sprintf(pathbuf + oldlen, "%s/", e->name);
+
if (prepare_children(e))
die("internal error: ls-tree show_children called with non tree");
e = e->item.tree->entries;
while (e) {
- show_entry(e, level);
+ show_entry(e, level, pathbuf);
e = e->next;
}
+
+ pathbuf[oldlen] = '\0';
+
return 0;
}
-static int show_entry(struct tree_entry_list *e, int level)
+static int show_entry(struct tree_entry_list *e, int level, char *pathbuf)
{
int err = 0;
if (e != &root_entry) {
- printf("%06o %s %s ", e->mode, entry_type(e),
- entry_hex(e));
- show_entry_name(e);
+ printf("%06o %s %s %s%s", e->mode, entry_type(e),
+ entry_hex(e), pathbuf, e->name);
putchar(line_termination);
}
@@ -176,10 +178,10 @@ static int show_entry(struct tree_entry_
*/
if (level == 0 && !(ls_options & LS_TREE_ONLY))
/* case (1)-a and (1)-b */
- err = err | show_children(e, level+1);
+ err = err | show_children(e, level+1, pathbuf);
else if (level && ls_options & LS_RECURSIVE)
/* case (2)-b */
- err = err | show_children(e, level+1);
+ err = err | show_children(e, level+1, pathbuf);
}
return err;
}
@@ -187,7 +189,8 @@ static int show_entry(struct tree_entry_
static int list_one(const char *path)
{
int err = 0;
- struct tree_entry_list *e = find_entry(path);
+ char pathbuf[MAXPATHLEN + 1];
+ struct tree_entry_list *e = find_entry(path, pathbuf);
if (!e) {
/* traditionally ls-tree does not complain about
* missing path. We may change this later to match
@@ -195,7 +198,7 @@ static int list_one(const char *path)
*/
return err;
}
- err = err | show_entry(e, 0);
+ err = err | show_entry(e, 0, pathbuf);
return err;
}
diff --git a/t/t3101-ls-tree-dirname.sh b/t/t3101-ls-tree-dirname.sh
new file mode 100644
--- /dev/null
+++ b/t/t3101-ls-tree-dirname.sh
@@ -0,0 +1,160 @@
+#!/bin/sh
+#
+# Copyright (c) 2005 Junio C Hamano
+# Copyright (c) 2005 Robert Fitzsimons
+#
+
+test_description='git-ls-tree directory and filenames handling.
+
+This test runs git-ls-tree with the following in a tree.
+
+ 1.txt - a file
+ 2.txt - a file
+ path0/a/b/c/1.txt - a file in a directory
+ path1/b/c/1.txt - a file in a directory
+ path2/1.txt - a file in a directory
+ path3/1.txt - a file in a directory
+ path3/2.txt - a file in a directory
+
+Test the handling of mulitple directories which have matching file
+entries. Also test odd filename and missing entries handling.
+'
+. ./test-lib.sh
+
+test_expect_success \
+ 'setup' \
+ 'echo 111 >1.txt &&
+ echo 222 >2.txt &&
+ mkdir path0 path0/a path0/a/b path0/a/b/c &&
+ echo 111 >path0/a/b/c/1.txt &&
+ mkdir path1 path1/b path1/b/c &&
+ echo 111 >path1/b/c/1.txt &&
+ mkdir path2 &&
+ echo 111 >path2/1.txt &&
+ mkdir path3 &&
+ echo 111 >path3/1.txt &&
+ echo 222 >path3/2.txt &&
+ find *.txt path* \( -type f -o -type l \) -print |
+ xargs git-update-index --add &&
+ tree=`git-write-tree` &&
+ echo $tree'
+
+_x40='[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f]'
+_x40="$_x40$_x40$_x40$_x40$_x40$_x40$_x40$_x40"
+test_output () {
+ sed -e "s/ $_x40 / X /" <current >check
+ diff -u expected check
+}
+
+test_expect_success \
+ 'ls-tree plain' \
+ 'git-ls-tree $tree >current &&
+ cat >expected <<\EOF &&
+100644 blob X 1.txt
+100644 blob X 2.txt
+040000 tree X path0
+040000 tree X path1
+040000 tree X path2
+040000 tree X path3
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree recursive' \
+ 'git-ls-tree -r $tree >current &&
+ cat >expected <<\EOF &&
+100644 blob X 1.txt
+100644 blob X 2.txt
+040000 tree X path0
+040000 tree X path0/a
+040000 tree X path0/a/b
+040000 tree X path0/a/b/c
+100644 blob X path0/a/b/c/1.txt
+040000 tree X path1
+040000 tree X path1/b
+040000 tree X path1/b/c
+100644 blob X path1/b/c/1.txt
+040000 tree X path2
+100644 blob X path2/1.txt
+040000 tree X path3
+100644 blob X path3/1.txt
+100644 blob X path3/2.txt
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree filter 1.txt' \
+ 'git-ls-tree $tree 1.txt >current &&
+ cat >expected <<\EOF &&
+100644 blob X 1.txt
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree filter path1/b/c/1.txt' \
+ 'git-ls-tree $tree path1/b/c/1.txt >current &&
+ cat >expected <<\EOF &&
+100644 blob X path1/b/c/1.txt
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree filter all 1.txt files' \
+ 'git-ls-tree $tree 1.txt path0/a/b/c/1.txt path1/b/c/1.txt path2/1.txt path3/1.txt >current &&
+ cat >expected <<\EOF &&
+100644 blob X 1.txt
+100644 blob X path0/a/b/c/1.txt
+100644 blob X path1/b/c/1.txt
+100644 blob X path2/1.txt
+100644 blob X path3/1.txt
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree filter directories' \
+ 'git-ls-tree $tree path3 path2 path0/a/b/c path1/b/c path0/a >current &&
+ cat >expected <<\EOF &&
+040000 tree X path3
+100644 blob X path3/1.txt
+100644 blob X path3/2.txt
+040000 tree X path2
+100644 blob X path2/1.txt
+040000 tree X path0/a/b/c
+100644 blob X path0/a/b/c/1.txt
+040000 tree X path1/b/c
+100644 blob X path1/b/c/1.txt
+040000 tree X path0/a
+040000 tree X path0/a/b
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree filter odd names' \
+ 'git-ls-tree $tree 1.txt /1.txt //1.txt path3/1.txt /path3/1.txt //path3//1.txt path3 /path3/ path3// >current &&
+ cat >expected <<\EOF &&
+100644 blob X 1.txt
+100644 blob X 1.txt
+100644 blob X 1.txt
+100644 blob X path3/1.txt
+100644 blob X path3/1.txt
+100644 blob X path3/1.txt
+040000 tree X path3
+100644 blob X path3/1.txt
+100644 blob X path3/2.txt
+040000 tree X path3
+100644 blob X path3/1.txt
+100644 blob X path3/2.txt
+040000 tree X path3
+100644 blob X path3/1.txt
+100644 blob X path3/2.txt
+EOF
+ test_output'
+
+test_expect_success \
+ 'ls-tree filter missing files and extra slashes' \
+ 'git-ls-tree $tree 1.txt/ abc.txt path3//23.txt path3/2.txt/// >current &&
+ cat >expected <<\EOF &&
+EOF
+ test_output'
+
+test_done
---
0.99.8.GIT
^ permalink raw reply
* Re: Some ASCII Art
From: Junio C Hamano @ 2005-10-06 20:38 UTC (permalink / raw)
To: Jon Loeliger; +Cc: git
In-Reply-To: <1128621923.29904.30.camel@cashmere.sps.mot.com>
Jon Loeliger <jdl@freescale.com> writes:
> Fundamental Git Index Operations
> Git Index Operations
These two look almost the same.
I find the label "commit-tree" on index -> odb in the second
picture a bit misleading. "commit-tree" takes a tree and zero
or more commit objects to create a new commit object, so it
works solely inside odb.
It is good that you mention that "read-tree -u" form updates
working tree in the second picture. In the same spirit, you
might also want to mention that "checkout-index -u" updates the
index (i.e. matches the stat information) in the same picture.
Diff drawing looks good. If you are aiming for completeness,
diff-stages is lacking -- it compares between stages in the
index file. But diff-stages is really a specialized operation
whose usefulness is still unproven, so not drawing it for the
sake of simplicity is also OK.
> Commit DAG Revision Naming
> ==========================
>
> Both node B and C are a commit parents of node A.
I assume that parents are left to right in this picture, that
is, B's first parent is D, second E, and third F.
>
> G H I J
> \ / \ /
> D E F
> \ | /
> \ | /
> \|/
> B C
> \ /
> \ /
> A
>
>
> A = rev = rev^0
> B = rev^ = rev^1 = rev~1
> D = rev^^ = rev^1^1 = rev~2
> G = rev^^^ = rev^1^1^1 = rev~3
> Is there a way to name node C, E, F, H, I or J?
C = A^2
E = B^2 = A^^2
F = B^3 = A^^3
H = D^2 = B^^2 = A^^^2 = A~2^2
I = F^ = B^3^ = A^^3^
J = F^2 = B^3^2 = A^^3^2
They look like line noise ;-)
> Is A also called rev~0?
If you are talking about only commit objects, then REV^0 is
always the same as REV. You could write REV^0^0^0^0 if you
want but that is rather pointless ;-).
^ permalink raw reply
* Re: [PATCH] Write .editmsg in GIT_DIR to avoid being in git-status.
From: Junio C Hamano @ 2005-10-06 20:19 UTC (permalink / raw)
To: Santi Bejar; +Cc: git
In-Reply-To: <8aa486160510060630m2afb18ffx@mail.gmail.com>
Santi Bejar <sbejar@gmail.com> writes:
>> If a working tree is read-only, I wonder what you are committing.
>
> I keep my /etc modifications in git, and I want to commit as a
> user not as root.
OK. Your /etc/.git is writable by you but /etc is not.
Although I personally would do it a bit differently (my /etc
sources would live in a different area which is a source tree
and 'make install' as root would install the latest to /etc; no
need for anything user writable under /etc, including
/etc/.git/), I do not think your set-up is wrong -- we should
support it.
>> ... If the working tree root level is not writable for you,
>> is it valid/reasonable to assume that $GIT_DIR is?
I changed my mind about this. $GIT_DIR by definition should be
writable by you since we are talking about committing -- I think
moving them to $GIT_DIR/ is a good change.
^ permalink raw reply
* Re: Allow "-u" flag to tag signing
From: Junio C Hamano @ 2005-10-06 20:17 UTC (permalink / raw)
To: H. Peter Anvin; +Cc: Linus Torvalds, Git Mailing List
In-Reply-To: <43455BBC.6020908@zytor.com>
"H. Peter Anvin" <hpa@zytor.com> writes:
>> ... It implicitly enables signing, since it doesn't make any
>> sense without it. Thus:
>> git tag -u <gpg-key-name> <tag-name> [<tagged-object>]
>> will use the named gpg key for signing.
>
> This is important for another reason as well: a lot of people have
> multiple keys.
Agreed. Thanks both. Will apply.
^ permalink raw reply
* Re: clone: I'm only doing a max of 256 requests
From: Junio C Hamano @ 2005-10-06 20:16 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Alex Riesen, git, Andy Isaacson
In-Reply-To: <Pine.LNX.4.64.0510060737160.31407@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> Junio, please apply.
I've been considering about this myself for quite a while, but
haven't done so only because I suspected the removal of the
checks are probably not good enough. I'll apply the patch, and
we will see what happens.
^ permalink raw reply
* Re: [PATCH] Add support for parallel HTTP transfers
From: Daniel Barkalow @ 2005-10-06 20:07 UTC (permalink / raw)
To: Nick Hengeveld; +Cc: git
In-Reply-To: <20051005214447.GF15593@reactrix.com>
On Wed, 5 Oct 2005, Nick Hengeveld wrote:
> Add support for parallel HTTP transfers. Prefetch populates a queue of
> objects to transfer and starts feeding requests to an active request
> queue for processing; fetch_object keeps the active queue moving
> while the specified object is being transferred. The size of the active
> queue can be restricted using -r and defaults to 5 concurrent transfers.
Somewhat weirdly, the version of curl on my desktop doesn't actually have
an implementation of curl_multi_info_read, although it's in the header
file and documentation. So you'll want a version check somewhere, I think,
which should probably just disable parallel transfers.
> ---
>
> I could use extra eyes on this patch - it seems to be stable although I have
> seen periodic cases during testing where it detects an empty server response
> (from kernel.org?) Are there implications to downloading a (potentially large)
> pack while objects contained in that pack have been prefetched and are in the
> transfer and/or active queue?
It should be fine to download objects and a pack that contains them at the
same time, although there's currently a check in fetch.c which should be
removed, so that it will call fetch() for an object if the object appears
between the prefetch() and the fetch().
I should be able to review this over the weekend. What sort of performance
are you getting at this point (in terms of bandwidth utilization)?
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* Re: Question on git clone
From: Luben Tuikov @ 2005-10-06 19:56 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7v64sbyk5e.fsf@assigned-by-dhcp.cox.net>
On 10/05/05 12:59, Junio C Hamano wrote:
> Luben Tuikov <luben_tuikov@adaptec.com> writes:
>
>
>>Cannot get remote repository information.
>>Perhaps git-update-server-info needs to be run there?
>
>
> Because HTTP support in git-fetch chooses not to trust the
> directory index the HTTP server may spit out, and relies solely
> on having info/refs file available there for discovering refs.
>
> It is debatable if HTTP support in git-fetch should fall back on
> discovery using "wget -r" like Cogito does, when the info/refs
> file is not found. I've written about this once on this list --
> I demonstrated what you would see if you do "wget -r" against
> git.git/refs/ on kernel.org; you will see why I do not think it
> is necessarily a better approach.
Thanks Junio for the reply.
I looked and indeed info/refs was ug+r while it should've been
a+r. That fixed it.
Thanks again,
Luben
>
> Not doing refs discovery using directory index forces the owner
> of an HTTP reachable repository to create info/refs by running
> update-server-info. This is a good thing -- it trains him to
> behave.
>
> update-server-info does not just create info/refs. It also
> creates another file objects/info/packs, which is needed for
> fetching over a commit walker if the repository is packed.
> AFAIK, even if you are using Cogito, you would not succeed
> pulling from a repository that is packed and does not have this
> file.
>
> There was a discussion about helping Cogito's tag-tracking. The
> downloading side needs to know if any new tag available on the
> other side refers to a commit on the branches the downloader
> tracks, without pulling everything that tag object refers to
> first. One way to help achieving this has been offered, which
> would involve adding a bit more information to info/refs, to say
> what object each tag refers to. It could be done on the client
> side, but it is far simpler if this kind of help is given on the
> server side.
>
> I anticipate in the future we may need to have more auxiliary
> files, or to add more information to existing auxiliary files,
> that summarize what the repository has for downloaders, and
> code to do so would be added to update-server-info, so the
> repository owner needs to learn to run only one command.
>
>
>
^ permalink raw reply
* [PATCH] Add support for parallel HTTP transfers
From: Nick Hengeveld @ 2005-10-06 18:54 UTC (permalink / raw)
To: git
Add support for parallel HTTP transfers. Prefetch populates a queue of
objects to transfer and starts feeding requests to an active request
queue for processing; fetch_object keeps the active queue moving
while the specified object is being transferred. The size of the active
queue can be restricted using -r and defaults to 5 concurrent transfers.
Requests for objects that are not prefetched are also processed via the
active queue.
Signed-off-by: Nick Hengeveld <nickh@reactrix.com>
---
This patch replaces the previous version. It seems that the instability
I experienced with empty server responses was related to request timeouts
because the active queue wasn't being processed during a non-object fetch.
http-fetch.c | 810 ++++++++++++++++++++++++++++++++++++++++++----------------
1 files changed, 587 insertions(+), 223 deletions(-)
8dcb9f1bdfa5d4229ef031f8d01cc8cde0d54bbc
diff --git a/http-fetch.c b/http-fetch.c
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -6,6 +6,8 @@
#include <curl/curl.h>
#include <curl/easy.h>
+#define DEFAULT_MAX_REQUESTS 5
+
#if LIBCURL_VERSION_NUM < 0x070704
#define curl_global_cleanup() do { /* nothing */ } while(0)
#endif
@@ -16,13 +18,16 @@
#define PREV_BUF_SIZE 4096
#define RANGE_HEADER_SIZE 30
-static CURL *curl;
+static int max_requests = DEFAULT_MAX_REQUESTS;
+static int active_requests = 0;
+static int data_received;
+
+static CURLM *curlm;
+static CURL *curl_default;
static struct curl_slist *no_pragma_header;
static struct curl_slist *no_range_header;
static char curl_errorstr[CURL_ERROR_SIZE];
-static char *initial_base;
-
struct alt_base
{
char *base;
@@ -33,11 +38,46 @@ struct alt_base
static struct alt_base *alt = NULL;
-static SHA_CTX c;
-static z_stream stream;
+enum transfer_state {
+ WAITING,
+ ABORTED,
+ ACTIVE,
+ COMPLETE,
+};
+
+struct transfer_request
+{
+ unsigned char sha1[20];
+ struct alt_base *repo;
+ char *url;
+ char filename[PATH_MAX];
+ char tmpfile[PATH_MAX];
+ int local;
+ enum transfer_state state;
+ CURLcode curl_result;
+ char errorstr[CURL_ERROR_SIZE];
+ long http_code;
+ unsigned char real_sha1[20];
+ SHA_CTX c;
+ z_stream stream;
+ int zret;
+ int rename;
+ struct active_request_slot *slot;
+ struct transfer_request *next;
+};
+
+struct active_request_slot
+{
+ CURL *curl;
+ FILE *local;
+ int in_use;
+ int done;
+ CURLcode curl_result;
+ struct active_request_slot *next;
+};
-static int local;
-static int zret;
+static struct transfer_request *request_queue_head = NULL;
+static struct active_request_slot *active_queue_head = NULL;
static int curl_ssl_verify;
static char *ssl_cert;
@@ -60,6 +100,7 @@ static size_t fwrite_buffer(void *ptr, s
size = buffer->size - buffer->posn;
memcpy(buffer->buffer + buffer->posn, ptr, size);
buffer->posn += size;
+ data_received++;
return size;
}
@@ -69,28 +110,28 @@ static size_t fwrite_sha1_file(void *ptr
unsigned char expn[4096];
size_t size = eltsize * nmemb;
int posn = 0;
+ struct transfer_request *request = (struct transfer_request *)data;
do {
- ssize_t retval = write(local, ptr + posn, size - posn);
+ ssize_t retval = write(request->local,
+ ptr + posn, size - posn);
if (retval < 0)
return posn;
posn += retval;
} while (posn < size);
- stream.avail_in = size;
- stream.next_in = ptr;
+ request->stream.avail_in = size;
+ request->stream.next_in = ptr;
do {
- stream.next_out = expn;
- stream.avail_out = sizeof(expn);
- zret = inflate(&stream, Z_SYNC_FLUSH);
- SHA1_Update(&c, expn, sizeof(expn) - stream.avail_out);
- } while (stream.avail_in && zret == Z_OK);
+ request->stream.next_out = expn;
+ request->stream.avail_out = sizeof(expn);
+ request->zret = inflate(&request->stream, Z_SYNC_FLUSH);
+ SHA1_Update(&request->c, expn,
+ sizeof(expn) - request->stream.avail_out);
+ } while (request->stream.avail_in && request->zret == Z_OK);
+ data_received++;
return size;
}
-void prefetch(unsigned char *sha1)
-{
-}
-
int relink_or_rename(char *old, char *new) {
int ret;
@@ -110,10 +151,369 @@ int relink_or_rename(char *old, char *ne
return 0;
}
+void process_curl_messages();
+void process_request_queue();
+
+struct active_request_slot *get_active_slot()
+{
+ struct active_request_slot *slot = active_queue_head;
+ struct active_request_slot *newslot;
+ int num_transfers;
+
+ /* Wait for a slot to open up if the queue is full */
+ while (active_requests >= max_requests) {
+ curl_multi_perform(curlm, &num_transfers);
+ if (num_transfers < active_requests) {
+ process_curl_messages();
+ }
+ }
+
+ while (slot != NULL && slot->in_use) {
+ slot = slot->next;
+ }
+ if (slot == NULL) {
+ newslot = xmalloc(sizeof(*newslot));
+ newslot->curl = curl_easy_duphandle(curl_default);
+ newslot->in_use = 0;
+ newslot->next = NULL;
+
+ slot = active_queue_head;
+ if (slot == NULL) {
+ active_queue_head = newslot;
+ } else {
+ while (slot->next != NULL) {
+ slot = slot->next;
+ }
+ slot->next = newslot;
+ }
+ slot = newslot;
+ }
+
+ active_requests++;
+ slot->in_use = 1;
+ slot->done = 0;
+ slot->local = NULL;
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_pragma_header);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_range_header);
+ curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
+
+ return slot;
+}
+
+int start_active_slot(struct active_request_slot *slot)
+{
+ CURLMcode curlm_result = curl_multi_add_handle(curlm, slot->curl);
+
+ if (curlm_result != CURLM_OK &&
+ curlm_result != CURLM_CALL_MULTI_PERFORM) {
+ active_requests--;
+ slot->in_use = 0;
+ return 0;
+ }
+
+ return 1;
+}
+
+void run_active_slot(struct active_request_slot *slot)
+{
+ int num_transfers;
+ long last_pos = 0;
+ long current_pos;
+ fd_set readfds;
+ fd_set writefds;
+ fd_set excfds;
+ int max_fd;
+ struct timeval select_timeout;
+ CURLMcode curlm_result;
+
+ while (!slot->done) {
+ data_received = 0;
+ do {
+ curlm_result = curl_multi_perform(curlm,
+ &num_transfers);
+ } while (curlm_result == CURLM_CALL_MULTI_PERFORM);
+ if (num_transfers < active_requests) {
+ process_curl_messages();
+ process_request_queue();
+ }
+
+ if (!data_received && slot->local != NULL) {
+ current_pos = ftell(slot->local);
+ if (current_pos > last_pos)
+ data_received++;
+ last_pos = current_pos;
+ }
+
+ if (!slot->done && !data_received) {
+ max_fd = 0;
+ FD_ZERO(&readfds);
+ FD_ZERO(&writefds);
+ FD_ZERO(&excfds);
+ select_timeout.tv_sec = 0;
+ select_timeout.tv_usec = 50000;
+ select(max_fd, &readfds, &writefds,
+ &excfds, &select_timeout);
+ }
+ }
+}
+
+void start_request(struct transfer_request *request)
+{
+ char *hex = sha1_to_hex(request->sha1);
+ char prevfile[PATH_MAX];
+ char *url;
+ char *posn;
+ int prevlocal;
+ unsigned char prev_buf[PREV_BUF_SIZE];
+ ssize_t prev_read = 0;
+ long prev_posn = 0;
+ char range[RANGE_HEADER_SIZE];
+ struct curl_slist *range_header = NULL;
+ struct active_request_slot *slot;
+
+ snprintf(prevfile, sizeof(prevfile), "%s.prev", request->filename);
+ unlink(prevfile);
+ rename(request->tmpfile, prevfile);
+ unlink(request->tmpfile);
+
+ request->local = open(request->tmpfile,
+ O_WRONLY | O_CREAT | O_EXCL, 0666);
+ if (request->local < 0) {
+ request->state = ABORTED;
+ error("Couldn't create temporary file %s for %s: %s\n",
+ request->tmpfile, request->filename, strerror(errno));
+ return;
+ }
+
+ memset(&request->stream, 0, sizeof(request->stream));
+
+ inflateInit(&request->stream);
+
+ SHA1_Init(&request->c);
+
+ url = xmalloc(strlen(request->repo->base) + 50);
+ request->url = xmalloc(strlen(request->repo->base) + 50);
+ strcpy(url, request->repo->base);
+ posn = url + strlen(request->repo->base);
+ strcpy(posn, "objects/");
+ posn += 8;
+ memcpy(posn, hex, 2);
+ posn += 2;
+ *(posn++) = '/';
+ strcpy(posn, hex + 2);
+ strcpy(request->url, url);
+
+ /* If a previous temp file is present, process what was already
+ fetched. */
+ prevlocal = open(prevfile, O_RDONLY);
+ if (prevlocal != -1) {
+ do {
+ prev_read = read(prevlocal, prev_buf, PREV_BUF_SIZE);
+ if (prev_read>0) {
+ if (fwrite_sha1_file(prev_buf,
+ 1,
+ prev_read,
+ request) == prev_read) {
+ prev_posn += prev_read;
+ } else {
+ prev_read = -1;
+ }
+ }
+ } while (prev_read > 0);
+ close(prevlocal);
+ }
+ unlink(prevfile);
+
+ /* Reset inflate/SHA1 if there was an error reading the previous temp
+ file; also rewind to the beginning of the local file. */
+ if (prev_read == -1) {
+ memset(&request->stream, 0, sizeof(request->stream));
+ inflateInit(&request->stream);
+ SHA1_Init(&request->c);
+ if (prev_posn>0) {
+ prev_posn = 0;
+ lseek(request->local, SEEK_SET, 0);
+ ftruncate(request->local, 0);
+ }
+ }
+
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, request);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
+ curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, request->errorstr);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+
+ /* If we have successfully processed data from a previous fetch
+ attempt, only fetch the data we don't already have. */
+ if (prev_posn>0) {
+ if (get_verbosely)
+ fprintf(stderr,
+ "Resuming fetch of object %s at byte %ld\n",
+ hex, prev_posn);
+ sprintf(range, "Range: bytes=%ld-", prev_posn);
+ range_header = curl_slist_append(range_header, range);
+ curl_easy_setopt(slot->curl,
+ CURLOPT_HTTPHEADER, range_header);
+ }
+
+ /* Try to add to multi handle, abort the request on error */
+ if (!start_active_slot(slot)) {
+ request->state = ABORTED;
+ close(request->local);
+ free(request->url);
+ return;
+ }
+
+ request->slot = slot;
+ request->state = ACTIVE;
+}
+
+void finish_request(struct transfer_request *request)
+{
+ fchmod(request->local, 0444);
+ close(request->local);
+
+ if (request->http_code == 416) {
+ fprintf(stderr, "Warning: requested range invalid; we may already have all the data.\n");
+ } else if (request->curl_result != CURLE_OK) {
+ return;
+ }
+
+ inflateEnd(&request->stream);
+ SHA1_Final(request->real_sha1, &request->c);
+ if (request->zret != Z_STREAM_END) {
+ unlink(request->tmpfile);
+ return;
+ }
+ if (memcmp(request->sha1, request->real_sha1, 20)) {
+ unlink(request->tmpfile);
+ return;
+ }
+ request->rename =
+ relink_or_rename(request->tmpfile, request->filename);
+
+ if (request->rename == 0)
+ pull_say("got %s\n", sha1_to_hex(request->sha1));
+}
+
+void release_request(struct transfer_request *request)
+{
+ struct transfer_request *entry = request_queue_head;
+
+ if (request == request_queue_head) {
+ request_queue_head = request->next;
+ } else {
+ while (entry->next != NULL && entry->next != request)
+ entry = entry->next;
+ if (entry->next == request)
+ entry->next = entry->next->next;
+ }
+
+ free(request->url);
+ free(request);
+}
+
+void process_curl_messages()
+{
+ int num_messages;
+ struct active_request_slot *slot;
+ struct transfer_request *request = NULL;
+ CURLMsg *curl_message = curl_multi_info_read(curlm, &num_messages);
+
+ while (curl_message != NULL) {
+ if (curl_message->msg == CURLMSG_DONE) {
+ slot = active_queue_head;
+ while (slot != NULL &&
+ slot->curl != curl_message->easy_handle)
+ slot = slot->next;
+ if (slot != NULL) {
+ curl_multi_remove_handle(curlm, slot->curl);
+ active_requests--;
+ slot->done = 1;
+ slot->in_use = 0;
+ slot->curl_result = curl_message->data.result;
+ request = request_queue_head;
+ while (request != NULL &&
+ request->slot != slot)
+ request = request->next;
+ } else {
+ fprintf(stderr, "Received DONE message for unknown request!\n");
+ }
+ if (request != NULL) {
+ request->curl_result =
+ curl_message->data.result;
+ curl_easy_getinfo(slot->curl,
+ CURLINFO_HTTP_CODE,
+ &request->http_code);
+ request->slot = NULL;
+
+ /* Use alternates if necessary */
+ if (request->http_code == 404 &&
+ request->repo->next != NULL) {
+ request->repo = request->repo->next;
+ start_request(request);
+ } else {
+ finish_request(request);
+ request->state = COMPLETE;
+ }
+ }
+ } else {
+ fprintf(stderr, "Unknown CURL message received: %d\n",
+ (int)curl_message->msg);
+ }
+ curl_message = curl_multi_info_read(curlm, &num_messages);
+ }
+}
+
+void process_request_queue()
+{
+ struct transfer_request *request = request_queue_head;
+ int num_transfers;
+
+ while (active_requests < max_requests && request != NULL) {
+ if (request->state == WAITING) {
+ start_request(request);
+ curl_multi_perform(curlm, &num_transfers);
+ }
+ request = request->next;
+ }
+}
+
+void prefetch(unsigned char *sha1)
+{
+ struct transfer_request *newreq;
+ struct transfer_request *tail;
+ char *filename = sha1_file_name(sha1);
+
+ newreq = xmalloc(sizeof(*newreq));
+ memcpy(newreq->sha1, sha1, 20);
+ newreq->repo = alt;
+ newreq->url = NULL;
+ newreq->local = -1;
+ newreq->state = WAITING;
+ snprintf(newreq->filename, sizeof(newreq->filename), "%s", filename);
+ snprintf(newreq->tmpfile, sizeof(newreq->tmpfile),
+ "%s.temp", filename);
+ newreq->next = NULL;
+
+ if (request_queue_head == NULL) {
+ request_queue_head = newreq;
+ } else {
+ tail = request_queue_head;
+ while (tail->next != NULL) {
+ tail = tail->next;
+ }
+ tail->next = newreq;
+ }
+ process_request_queue();
+ process_curl_messages();
+}
+
static int got_alternates = 0;
static int fetch_index(struct alt_base *repo, unsigned char *sha1)
{
+ char *hex = sha1_to_hex(sha1);
char *filename;
char *url;
char tmpfile[PATH_MAX];
@@ -121,20 +521,18 @@ static int fetch_index(struct alt_base *
long prev_posn = 0;
char range[RANGE_HEADER_SIZE];
struct curl_slist *range_header = NULL;
- CURLcode curl_result;
FILE *indexfile;
+ struct active_request_slot *slot;
if (has_pack_index(sha1))
return 0;
if (get_verbosely)
- fprintf(stderr, "Getting index for pack %s\n",
- sha1_to_hex(sha1));
+ fprintf(stderr, "Getting index for pack %s\n", hex);
url = xmalloc(strlen(repo->base) + 64);
- sprintf(url, "%s/objects/pack/pack-%s.idx",
- repo->base, sha1_to_hex(sha1));
+ sprintf(url, "%s/objects/pack/pack-%s.idx", repo->base, hex);
filename = sha1_pack_index_name(sha1);
snprintf(tmpfile, sizeof(tmpfile), "%s.temp", filename);
@@ -143,12 +541,12 @@ static int fetch_index(struct alt_base *
return error("Unable to open local file %s for pack index",
filename);
- curl_easy_setopt(curl, CURLOPT_FILE, indexfile);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
- curl_easy_setopt(curl, CURLOPT_URL, url);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);
- curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);
-
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, indexfile);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ slot->local = indexfile;
+
/* If there is data present from a previous transfer attempt,
resume where it left off */
prev_posn = ftell(indexfile);
@@ -156,20 +554,21 @@ static int fetch_index(struct alt_base *
if (get_verbosely)
fprintf(stderr,
"Resuming fetch of index for pack %s at byte %ld\n",
- sha1_to_hex(sha1), prev_posn);
+ hex, prev_posn);
sprintf(range, "Range: bytes=%ld-", prev_posn);
range_header = curl_slist_append(range_header, range);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, range_header);
}
- /* Clear out the Range: header after performing the request, so
- other curl requests don't inherit inappropriate header data */
- curl_result = curl_easy_perform(curl);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);
- if (curl_result != 0) {
- fclose(indexfile);
- return error("Unable to get pack index %s\n%s", url,
- curl_errorstr);
+ if (start_active_slot(slot)) {
+ run_active_slot(slot);
+ if (slot->curl_result != CURLE_OK) {
+ fclose(indexfile);
+ return error("Unable to get pack index %s\n%s", url,
+ curl_errorstr);
+ }
+ } else {
+ return error("Unable to start request");
}
fclose(indexfile);
@@ -205,6 +604,9 @@ static int fetch_alternates(char *base)
char *data;
int i = 0;
int http_specific = 1;
+ struct alt_base *tail = alt;
+
+ struct active_request_slot *slot;
if (got_alternates)
return 0;
data = xmalloc(4096);
@@ -218,22 +620,31 @@ static int fetch_alternates(char *base)
url = xmalloc(strlen(base) + 31);
sprintf(url, "%s/objects/info/http-alternates", base);
- curl_easy_setopt(curl, CURLOPT_FILE, &buffer);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
- curl_easy_setopt(curl, CURLOPT_URL, url);
-
- if (curl_easy_perform(curl) || !buffer.posn) {
- http_specific = 0;
-
- sprintf(url, "%s/objects/info/alternates", base);
-
- curl_easy_setopt(curl, CURLOPT_FILE, &buffer);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
- curl_easy_setopt(curl, CURLOPT_URL, url);
-
- if (curl_easy_perform(curl)) {
- return 0;
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ if (start_active_slot(slot)) {
+ run_active_slot(slot);
+ if (slot->curl_result != CURLE_OK || !buffer.posn) {
+ http_specific = 0;
+
+ sprintf(url, "%s/objects/info/alternates", base);
+
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,
+ fwrite_buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ if (start_active_slot(slot)) {
+ run_active_slot(slot);
+ if (slot->curl_result != CURLE_OK) {
+ return 0;
+ }
+ }
}
+ } else {
+ return 0;
}
data[buffer.posn] = '\0';
@@ -283,11 +694,13 @@ static int fetch_alternates(char *base)
fprintf(stderr,
"Also look at %s\n", target);
newalt = xmalloc(sizeof(*newalt));
- newalt->next = alt;
+ newalt->next = NULL;
newalt->base = target;
newalt->got_indices = 0;
newalt->packs = NULL;
- alt = newalt;
+ while (tail->next != NULL)
+ tail = tail->next;
+ tail->next = newalt;
ret++;
}
}
@@ -306,6 +719,8 @@ static int fetch_indices(struct alt_base
char *data;
int i = 0;
+ struct active_request_slot *slot;
+
if (repo->got_indices)
return 0;
@@ -320,14 +735,18 @@ static int fetch_indices(struct alt_base
url = xmalloc(strlen(repo->base) + 21);
sprintf(url, "%s/objects/info/packs", repo->base);
- curl_easy_setopt(curl, CURLOPT_FILE, &buffer);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
- curl_easy_setopt(curl, CURLOPT_URL, url);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);
- curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);
-
- if (curl_easy_perform(curl))
- return error("%s", curl_errorstr);
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);
+ if (start_active_slot(slot)) {
+ run_active_slot(slot);
+ if (slot->curl_result != CURLE_OK)
+ return error("%s", curl_errorstr);
+ } else {
+ return error("Unable to start request");
+ }
while (i < buffer.posn) {
switch (data[i]) {
@@ -364,7 +783,8 @@ static int fetch_pack(struct alt_base *r
long prev_posn = 0;
char range[RANGE_HEADER_SIZE];
struct curl_slist *range_header = NULL;
- CURLcode curl_result;
+
+ struct active_request_slot *slot;
if (fetch_indices(repo))
return -1;
@@ -390,11 +810,11 @@ static int fetch_pack(struct alt_base *r
return error("Unable to open local file %s for pack",
filename);
- curl_easy_setopt(curl, CURLOPT_FILE, packfile);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite);
- curl_easy_setopt(curl, CURLOPT_URL, url);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);
- curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, packfile);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ slot->local = packfile;
/* If there is data present from a previous transfer attempt,
resume where it left off */
@@ -406,17 +826,18 @@ static int fetch_pack(struct alt_base *r
sha1_to_hex(target->sha1), prev_posn);
sprintf(range, "Range: bytes=%ld-", prev_posn);
range_header = curl_slist_append(range_header, range);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, range_header);
}
- /* Clear out the Range: header after performing the request, so
- other curl requests don't inherit inappropriate header data */
- curl_result = curl_easy_perform(curl);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);
- if (curl_result != 0) {
- fclose(packfile);
- return error("Unable to get pack file %s\n%s", url,
- curl_errorstr);
+ if (start_active_slot(slot)) {
+ run_active_slot(slot);
+ if (slot->curl_result != CURLE_OK) {
+ fclose(packfile);
+ return error("Unable to get pack file %s\n%s", url,
+ curl_errorstr);
+ }
+ } else {
+ return error("Unable to start request");
}
fclose(packfile);
@@ -441,155 +862,73 @@ static int fetch_pack(struct alt_base *r
static int fetch_object(struct alt_base *repo, unsigned char *sha1)
{
char *hex = sha1_to_hex(sha1);
- char *filename = sha1_file_name(sha1);
- unsigned char real_sha1[20];
- char tmpfile[PATH_MAX];
- char prevfile[PATH_MAX];
int ret;
- char *url;
- char *posn;
- int prevlocal;
- unsigned char prev_buf[PREV_BUF_SIZE];
- ssize_t prev_read = 0;
- long prev_posn = 0;
- char range[RANGE_HEADER_SIZE];
- struct curl_slist *range_header = NULL;
- CURLcode curl_result;
-
- snprintf(tmpfile, sizeof(tmpfile), "%s.temp", filename);
- snprintf(prevfile, sizeof(prevfile), "%s.prev", filename);
-
- if (unlink(prevfile) && (errno != ENOENT))
- return error("Failed to unlink %s (%s)",
- prevfile, strerror(errno));
- if (rename(tmpfile, prevfile) && (errno != ENOENT))
- return error("Failed to rename %s to %s (%s)",
- tmpfile, prevfile, strerror(errno));
-
- local = open(tmpfile, O_WRONLY | O_CREAT | O_EXCL, 0666);
-
- /* Note: if another instance starts now, it will turn our new
- tmpfile into its prevfile. */
-
- if (local < 0)
- return error("Couldn't create temporary file %s for %s: %s\n",
- tmpfile, filename, strerror(errno));
+ struct transfer_request *request = request_queue_head;
+ int num_transfers;
- memset(&stream, 0, sizeof(stream));
-
- inflateInit(&stream);
-
- SHA1_Init(&c);
-
- curl_easy_setopt(curl, CURLOPT_FAILONERROR, 1);
- curl_easy_setopt(curl, CURLOPT_FILE, NULL);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_pragma_header);
- curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);
-
- url = xmalloc(strlen(repo->base) + 50);
- strcpy(url, repo->base);
- posn = url + strlen(repo->base);
- strcpy(posn, "objects/");
- posn += 8;
- memcpy(posn, hex, 2);
- posn += 2;
- *(posn++) = '/';
- strcpy(posn, hex + 2);
+ while (request != NULL && memcmp(request->sha1, sha1, 20))
+ request = request->next;
+ if (request == NULL)
+ return error("Couldn't find request for %s in the queue", hex);
+
+ while (request->state == WAITING) {
+ curl_multi_perform(curlm, &num_transfers);
+ if (num_transfers < active_requests) {
+ process_curl_messages();
+ process_request_queue();
+ }
+ }
- curl_easy_setopt(curl, CURLOPT_URL, url);
+ if (request->state == ACTIVE)
+ run_active_slot(request->slot);
- /* If a previous temp file is present, process what was already
- fetched. */
- prevlocal = open(prevfile, O_RDONLY);
- if (prevlocal != -1) {
- do {
- prev_read = read(prevlocal, prev_buf, PREV_BUF_SIZE);
- if (prev_read>0) {
- if (fwrite_sha1_file(prev_buf,
- 1,
- prev_read,
- NULL) == prev_read) {
- prev_posn += prev_read;
- } else {
- prev_read = -1;
- }
- }
- } while (prev_read > 0);
- close(prevlocal);
+ if (request->state == ABORTED) {
+ release_request(request);
+ return error("Request for %s aborted", hex);
}
- unlink(prevfile);
- /* Reset inflate/SHA1 if there was an error reading the previous temp
- file; also rewind to the beginning of the local file. */
- if (prev_read == -1) {
- memset(&stream, 0, sizeof(stream));
- inflateInit(&stream);
- SHA1_Init(&c);
- if (prev_posn>0) {
- prev_posn = 0;
- lseek(local, SEEK_SET, 0);
- ftruncate(local, 0);
- }
+ if (request->curl_result != CURLE_OK && request->http_code != 416) {
+ ret = error("%s", request->errorstr);
+ release_request(request);
+ return ret;
}
- /* If we have successfully processed data from a previous fetch
- attempt, only fetch the data we don't already have. */
- if (prev_posn>0) {
- if (get_verbosely)
- fprintf(stderr,
- "Resuming fetch of object %s at byte %ld\n",
- hex, prev_posn);
- sprintf(range, "Range: bytes=%ld-", prev_posn);
- range_header = curl_slist_append(range_header, range);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, range_header);
+ if (request->zret != Z_STREAM_END) {
+ ret = error("File %s (%s) corrupt\n", hex, request->url);
+ release_request(request);
+ return ret;
}
- /* Clear out the Range: header after performing the request, so
- other curl requests don't inherit inappropriate header data */
- curl_result = curl_easy_perform(curl);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, no_range_header);
- if (curl_result != 0) {
- return error("%s", curl_errorstr);
- }
-
- fchmod(local, 0444);
- close(local);
- inflateEnd(&stream);
- SHA1_Final(real_sha1, &c);
- if (zret != Z_STREAM_END) {
- unlink(tmpfile);
- return error("File %s (%s) corrupt\n", hex, url);
- }
- if (memcmp(sha1, real_sha1, 20)) {
- unlink(tmpfile);
+ if (memcmp(request->sha1, request->real_sha1, 20)) {
+ release_request(request);
return error("File %s has bad hash\n", hex);
}
- ret = relink_or_rename(tmpfile, filename);
- if (ret)
- return error("unable to write sha1 filename %s: %s",
- filename, strerror(ret));
- pull_say("got %s\n", hex);
+ if (request->rename < 0) {
+ ret = error("unable to write sha1 filename %s: %s",
+ request->filename,
+ strerror(request->rename));
+ release_request(request);
+ return ret;
+ }
+
+ release_request(request);
return 0;
}
int fetch(unsigned char *sha1)
{
struct alt_base *altbase = alt;
+
+ if (!fetch_object(altbase, sha1))
+ return 0;
while (altbase) {
- if (!fetch_object(altbase, sha1))
- return 0;
if (!fetch_pack(altbase, sha1))
return 0;
- if (fetch_alternates(altbase->base) > 0) {
- altbase = alt;
- continue;
- }
altbase = altbase->next;
}
return error("Unable to find %s under %s\n", sha1_to_hex(sha1),
- initial_base);
+ alt->base);
}
int fetch_ref(char *ref, unsigned char *sha1)
@@ -597,17 +936,13 @@ int fetch_ref(char *ref, unsigned char *
char *url, *posn;
char hex[42];
struct buffer buffer;
- char *base = initial_base;
+ char *base = alt->base;
+ struct active_request_slot *slot;
buffer.size = 41;
buffer.posn = 0;
buffer.buffer = hex;
hex[41] = '\0';
- curl_easy_setopt(curl, CURLOPT_FILE, &buffer);
- curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
- curl_easy_setopt(curl, CURLOPT_HTTPHEADER, NULL);
- curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errorstr);
-
url = xmalloc(strlen(base) + 6 + strlen(ref));
strcpy(url, base);
posn = url + strlen(base);
@@ -615,11 +950,19 @@ int fetch_ref(char *ref, unsigned char *
posn += 5;
strcpy(posn, ref);
- curl_easy_setopt(curl, CURLOPT_URL, url);
-
- if (curl_easy_perform(curl))
- return error("Couldn't get %s for %s\n%s",
- url, ref, curl_errorstr);
+ slot = get_active_slot();
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ if (start_active_slot(slot)) {
+ run_active_slot(slot);
+ if (slot->curl_result != CURLE_OK)
+ return error("Couldn't get %s for %s\n%s",
+ url, ref, curl_errorstr);
+ } else {
+ return error("Unable to start request");
+ }
hex[40] = '\0';
get_sha1_hex(hex, sha1);
@@ -631,6 +974,7 @@ int main(int argc, char **argv)
char *commit_id;
char *url;
int arg = 1;
+ struct active_request_slot *slot;
while (arg < argc && argv[arg][0] == '-') {
if (argv[arg][1] == 't') {
@@ -648,6 +992,11 @@ int main(int argc, char **argv)
arg++;
} else if (!strcmp(argv[arg], "--recover")) {
get_recover = 1;
+ } else if (argv[arg][1] == 'r') {
+ max_requests = atoi(argv[arg + 1]);
+ if (max_requests < 1)
+ max_requests = DEFAULT_MAX_REQUESTS;
+ arg++;
}
arg++;
}
@@ -660,44 +1009,59 @@ int main(int argc, char **argv)
curl_global_init(CURL_GLOBAL_ALL);
- curl = curl_easy_init();
+ curlm = curl_multi_init();
+ if (curlm == NULL) {
+ fprintf(stderr, "Error creating curl multi handle.\n");
+ return 1;
+ }
no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
no_range_header = curl_slist_append(no_range_header, "Range:");
+ curl_default = curl_easy_init();
+
curl_ssl_verify = getenv("GIT_SSL_NO_VERIFY") ? 0 : 1;
- curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);
+ curl_easy_setopt(curl_default, CURLOPT_SSL_VERIFYPEER, curl_ssl_verify);
#if LIBCURL_VERSION_NUM >= 0x070907
- curl_easy_setopt(curl, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
+ curl_easy_setopt(curl_default, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);
#endif
if ((ssl_cert = getenv("GIT_SSL_CERT")) != NULL) {
- curl_easy_setopt(curl, CURLOPT_SSLCERT, ssl_cert);
+ curl_easy_setopt(curl_default, CURLOPT_SSLCERT, ssl_cert);
}
#if LIBCURL_VERSION_NUM >= 0x070902
if ((ssl_key = getenv("GIT_SSL_KEY")) != NULL) {
- curl_easy_setopt(curl, CURLOPT_SSLKEY, ssl_key);
+ curl_easy_setopt(curl_default, CURLOPT_SSLKEY, ssl_key);
}
#endif
#if LIBCURL_VERSION_NUM >= 0x070908
if ((ssl_capath = getenv("GIT_SSL_CAPATH")) != NULL) {
- curl_easy_setopt(curl, CURLOPT_CAPATH, ssl_capath);
+ curl_easy_setopt(curl_default, CURLOPT_CAPATH, ssl_capath);
}
#endif
if ((ssl_cainfo = getenv("GIT_SSL_CAINFO")) != NULL) {
- curl_easy_setopt(curl, CURLOPT_CAINFO, ssl_cainfo);
+ curl_easy_setopt(curl_default, CURLOPT_CAINFO, ssl_cainfo);
}
+ curl_easy_setopt(curl_default, CURLOPT_FAILONERROR, 1);
alt = xmalloc(sizeof(*alt));
alt->base = url;
alt->got_indices = 0;
alt->packs = NULL;
alt->next = NULL;
- initial_base = url;
+ fetch_alternates(alt->base);
if (pull(commit_id))
return 1;
curl_slist_free_all(no_pragma_header);
+ curl_slist_free_all(no_range_header);
+ curl_easy_cleanup(curl_default);
+ slot = active_queue_head;
+ while (slot != NULL) {
+ curl_easy_cleanup(slot->curl);
+ slot = slot->next;
+ }
+ curl_multi_cleanup(curlm);
curl_global_cleanup();
return 0;
}
^ permalink raw reply
* Some ASCII Art
From: Jon Loeliger @ 2005-10-06 18:05 UTC (permalink / raw)
To: Git List
Guys,
I tend to be a visual learner. So I made up some ASCII
art and annotated it with some Git operations and stuff.
First, I'd love to have any feedback on these drawings,
especially if it actually misrepresents things. If there
are obvious additions to these, that'd be nice to know too.
Second, if you think they are generally useful, please
feel free to add them to whatever documentation seems
appropriate. (If you want a patch to a certain file
and a Sign-Off, let me know!)
Thanks,
jdl
Fundamental Git Index Operations
================================
+-----------+
| Object DB |
| Backing |
| Store |
+-----------+
^
write-tree | |
tree obj | |
| | read-tree
| | tree obj
V
+-----------+
| Index |
| "cache" |
+-----------+
^
update-index | |
blob obj | |
| | checkout-index
| | blob obj
V
+-----------+
| Working |
| Directory |
+-----------+
Git Index Operations
====================
+-----------+
| Object DB |
| Backing |
| Store | -----------+
+-----------+ |
^ |
commit-tree | | |
commit obj | | read-tree -m |
| | tree obj |
| | |
V |
+-----------+ |
| Index | <- - - - - +
| "cache" | - - - - - >+
+-----------+ |
^ |
| |
update-index | read-tree -m -u |
blob obj | tree obj |
| |
|
+-----------+ |
| Working |<-----------+
| Directory |
+-----------+
Git Diff Types
==============
diff-tree
+----+
| |
| |
V V
+-----------+
| Object DB |
| Backing |
| Store |
+-----------+
^ ^
| |
| | diff-index --cached
| |
diff-index | V
| +-----------+
| | Index |
| | "cache" |
| +-----------+
| ^
| |
| | diff-files
| |
V V
+-----------+
| Working |
| Directory |
+-----------+
Commit DAG Revision Naming
==========================
Both node B and C are a commit parents of node A.
G H I J
\ / \ /
D E F
\ | /
\ | /
\|/
B C
\ /
\ /
A
A = rev = rev^0
B = rev^ = rev^1 = rev~1
D = rev^^ = rev^1^1 = rev~2
G = rev^^^ = rev^1^1^1 = rev~3
Is there a way to name node C, E, F, H, I or J?
Is A also called rev~0?
^ permalink raw reply
* Re: Allow "-u" flag to tag signing
From: H. Peter Anvin @ 2005-10-06 17:15 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, Git Mailing List
In-Reply-To: <Pine.LNX.4.64.0510060952410.31407@g5.osdl.org>
Linus Torvalds wrote:
> The current "git tag -s" thing always uses the tagger name as the signing
> user key, which is very irritating, since my key is under my email
> address, but the tagger key obviously contains the actual machine name
> too.
>
> Now, I could just use "GIT_COMMITTER_EMAIL" and force it to be my real
> email, but I actually think that it's nice to see which machine I use for
> my work.
>
> So rather than force my tagger ID to have to match the gpg key name, just
> support the "-u" flag to "git tag" instead. It implicitly enables signing,
> since it doesn't make any sense without it. Thus:
>
> git tag -u <gpg-key-name> <tag-name> [<tagged-object>]
>
> will use the named gpg key for signing.
This is important for another reason as well: a lot of people have
multiple keys.
-hpa
^ permalink raw reply
* Re: [PATCH] Fall back to three-way merge when applying a patch.
From: Eric W. Biederman @ 2005-10-06 17:07 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0510060756060.31407@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> On Thu, 6 Oct 2005, Eric W. Biederman wrote:
>>
>> After thinking about it I don't think you need to look through the
>> history to use it for a merge3 operation. As I recall merge3 only
>> looks at the base and the two derived versions of the file. If we
>> have the sha1 of the original in the git repository I think all we
>> need to compute is the diff between that sha1 the current version
>> file. And then apply the merge3 algorithm to combine the two sets of
>> changes.
>
> Ahh, that I can definitely agree with. In fact, it makes perfect sense.
>
> However, it assumes that everybody is a git user, which isn't actually
> true.
Agreed. The question is the subset of everyone large enough to
make it a useful technique. Given that a standalone diff utility can
be taught how to generate the extra information, and a standalone
patch could use it to verify you are at least patching the version
of the file the patch was intended for I suspect the subset of
everyone is large enough to be interesting.
> Also, I'm wondering whether the advantages outweigh the
> disadvantages: it would make the diff uglier. We'd have to add that SHA1
> there somewhere (either on the "diff" line itself, or as anothe rextended
> git line like the "rename from/to" lines - a "original <sha1>" line).
I don't think an extra line in the header is going to be much of a problem.
Just more header noise.
Eric
^ permalink raw reply
* Allow "-u" flag to tag signing
From: Linus Torvalds @ 2005-10-06 16:58 UTC (permalink / raw)
To: Junio C Hamano, Git Mailing List
The current "git tag -s" thing always uses the tagger name as the signing
user key, which is very irritating, since my key is under my email
address, but the tagger key obviously contains the actual machine name
too.
Now, I could just use "GIT_COMMITTER_EMAIL" and force it to be my real
email, but I actually think that it's nice to see which machine I use for
my work.
So rather than force my tagger ID to have to match the gpg key name, just
support the "-u" flag to "git tag" instead. It implicitly enables signing,
since it doesn't make any sense without it. Thus:
git tag -u <gpg-key-name> <tag-name> [<tagged-object>]
will use the named gpg key for signing.
Signed-off-by: Linus Torvalds <torvalds@osdl.org>
---
Not very well tested, but I re-did my v2.6.14-rc3 tag with this, since I'd
messed up the comments (and called it v2.6.14-rc2 in there ;)
So it may even work.
diff --git a/git-tag.sh b/git-tag.sh
--- a/git-tag.sh
+++ b/git-tag.sh
@@ -12,6 +12,7 @@ annotate=
signed=
force=
message=
+username=
while case "$#" in 0) break ;; esac
do
case "$1" in
@@ -30,6 +31,12 @@ do
shift
message="$1"
;;
+ -u)
+ annotate=1
+ signed=1
+ shift
+ username="$1"
+ ;;
-*)
usage
;;
@@ -70,8 +77,11 @@ if [ "$annotate" ]; then
( echo -e "object $object\ntype $type\ntag $name\ntagger $tagger\n"; cat .tagmsg ) > .tmp-tag
rm -f .tmp-tag.asc .tagmsg
if [ "$signed" ]; then
- me=$(expr "$tagger" : '\(.*>\)') &&
- gpg -bsa -u "$me" .tmp-tag &&
+ {
+ [ "$username" ] ||
+ username=$(expr "$tagger" : '\(.*>\)')
+ } &&
+ gpg -bsa -u "$username" .tmp-tag &&
cat .tmp-tag.asc >>.tmp-tag ||
die "failed to sign the tag with GPG."
fi
^ permalink raw reply
* Re: [PATCH] Fall back to three-way merge when applying a patch.
From: Linus Torvalds @ 2005-10-06 14:59 UTC (permalink / raw)
To: Eric W. Biederman; +Cc: Junio C Hamano, git
In-Reply-To: <m1wtkqk89a.fsf@ebiederm.dsl.xmission.com>
On Thu, 6 Oct 2005, Eric W. Biederman wrote:
>
> After thinking about it I don't think you need to look through the
> history to use it for a merge3 operation. As I recall merge3 only
> looks at the base and the two derived versions of the file. If we
> have the sha1 of the original in the git repository I think all we
> need to compute is the diff between that sha1 the current version
> file. And then apply the merge3 algorithm to combine the two sets of
> changes.
Ahh, that I can definitely agree with. In fact, it makes perfect sense.
However, it assumes that everybody is a git user, which isn't actually
true. Also, I'm wondering whether the advantages outweigh the
disadvantages: it would make the diff uglier. We'd have to add that SHA1
there somewhere (either on the "diff" line itself, or as anothe rextended
git line like the "rename from/to" lines - a "original <sha1>" line).
Linus
^ permalink raw reply
* Re: [PATCH] Fall back to three-way merge when applying a patch.
From: Eric W. Biederman @ 2005-10-06 14:52 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Junio C Hamano, git
In-Reply-To: <Pine.LNX.4.64.0510060731560.31407@g5.osdl.org>
Linus Torvalds <torvalds@osdl.org> writes:
> On Wed, 5 Oct 2005, Eric W. Biederman wrote:
>>
>> Ah. I had missed that git-whatchanged can be given a filename that is
>> nice. One of the better kept secrets of git. That makes my whole
>> question worthwhile :)
>
> That's definitely not a secret - it was part of the whole point of
> git-whatchanged. It's the native git version of "annotate", and I
> personally find it much more useful.
>
> It's not even just a filename. You can do
>
> git-whatchanged -p drivers/scsi/ include/scsi
>
> to limit the set to those two subdirectories. IOW, you can give
> git-whatchanged an arbitrary list of individual pathnames or directory
> names.
Which probably means it's time for me to generate a patch to the
git-whatchanged documentation.
>> But if you happen to have at least the file level sha1 you can
>> actually know if the patch was against what you think it is against.
>
> Yes, a file-level SHA1 may be useful. On the other hand, I suspect that by
> that time (since you have to search for the version anyway) you might as
> well have the "just try to apply the patch" approach. It's basically the
> same search space.
Maybe.
After thinking about it I don't think you need to look through the
history to use it for a merge3 operation. As I recall merge3 only
looks at the base and the two derived versions of the file. If we
have the sha1 of the original in the git repository I think all we
need to compute is the diff between that sha1 the current version
file. And then apply the merge3 algorithm to combine the two sets of
changes.
Eric
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox