* Re: Balanced packing strategy
From: Craig Schlenter @ 2005-11-12 15:14 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20051112135947.GC30496@pasky.or.cz>
On 12 Nov 2005, at 3:59 PM, Petr Baudis wrote:
> Dear diary, on Sat, Nov 12, 2005 at 02:40:50PM CET, I got a letter
> where Craig Schlenter <craig@codefountain.com> said that...
>> The 100MB situation is not cool for those of us on a tight bandwidth
>> budget or slow links. Can anyone tell me if the native git protocol is
>> any better at this stuff please?
>
> Yes, the native GIT protocol transfers only the objects you need.
Ah, magic, thanks!
> But the 100MB situation is still bad. FWIW, this is my proposal I sent
> about a month ago to some packs-related discussion at the kernel.org
> mailing list (ok, I updated it a little):
It would be nice if there was some meaningful automatic packing that
didn't hurt "non-git-aware protocol" users.
Does the pack index file contain enough information to enable a client
to send http byte range requests to grab individual objects from a pack?
It does seem to store object offsets but maybe I'm missing something ...
Thank you,
--Craig
^ permalink raw reply
* Re: [PATCH] Building git on NetBSD
From: Andreas Ericsson @ 2005-11-12 15:32 UTC (permalink / raw)
To: walt; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0511120649430.2695@x2.ybpnyarg>
walt wrote:
>
> I'm no software developer so I'm not sure about the syntax of the -rpath
> flag. It does what I wanted, but it results in lots of warnings about
> not being used because no linking is taking place. Should this maybe
> go in LDFLAGS instead of CFLAGS? Dunno.
>
Yes it should. Currently the git makefile doesn't distinguish between
CFLAGS and LDFLAGS which results in a fair few warnings.
> Oh, one person was concerned about the cogito scripts running on the
> NetBSD version of sh, which isn't bash. So far I've tried cg-clone,
> cg-update, and gitk, and all worked just like on linux. Any chance
> that some of the other scripts may not work as well on a non-bash sh?
>
gitk is in tcl so I'd be surprised if that didn't work.
I don't think there's been a whole lot of testing for other shells but
the friendly folks on this list usually takes care of fixing such things
fairly quickly so long as someone tells them where to look or,
preferrably, sends a patch that does the trick.
--
Andreas Ericsson andreas.ericsson@op5.se
OP5 AB www.op5.se
Tel: +46 8-230225 Fax: +46 8-230231
^ permalink raw reply
* Cloning speed comparison, round II
From: Petr Baudis @ 2005-11-12 15:53 UTC (permalink / raw)
To: git
Hello,
here comes another round of the cloning speed comparison. For the
previous round, see
6395 F Aug 13 Petr Baudis ( 2.6K) Cloning speed comparison
Message-ID: <20050813015402.GC20812@pasky.ji.cz>
What do we have here:
git.git: 10267 objects, mostly packed
cogito.git: 99903903 objects, no packs
(But please note that compared to the previous round, there are
probably more unpacked objects lying around in the git.git repository
- actually roughly 1000.)
Latest GIT, latest Cogito, performed by cg-clone to the official
kernel.org hostnames for each transport. 'real' medians from several
tries (number of tries usually at least three, based on their variance):
rsync git+ssh(*) git(**) http
git.git 0m45s 0m34s 5m30s 4m01s (++)
cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)
(*) git+ssh was to master.kernel.org, which was under significant load
from some seemingly runaway gzip process, so that slowed things
down.
(**) The unpacking was slooooooooow yet the load was quite low. This
should be investigated, the native git fetching is much slower
than even HTTP.
(+) There was unusually high variance here - the results range from
0m27s to 2m30s. In all other cases the results for all runs were
very similiar.
(++) Messy - it stalls at 72 objects, then does seemingly nothing for
a long while, then requests a pack. Huh? Also, I saw the
Getting alternates list for http://www.kernel.org/pub/scm/git/git.git/
line about 100 times for the 72 objects (but never after the pack
is fetched, at least).
Conclusion: The GIT protocol has some major problem which is dragging
it down to ridiculous slowness (isn't it supposed to be as fast as git+ssh
or even slightly faster?). rsync unfortunately still provides the fastest
anonymous access. HTTP is better than it was, but especially when fetching
packs, there is a room for improvement - the mysterious stalls should be
eliminated, and it shouldn't get bogged down by repeated silly requests
for alternates lists etc.
Compared to the last round, HTTP is much faster for unpacked object
stores (thanks to parallel fetching), but much slower for packed object
stores - however that can be caused by more objects lying unpacked now
in the git.git repository. git+ssh (back then clone-pack:ssh) is as
fast as usual - the 0m27s case suggests that it would be much faster
for Cogito if master.kernel.org wouldn't be under load.
GIT protocol is the great disappointment. And I kind of hoped that I could
push the rsync deprecation further because of the GIT protocol.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply
* Re: cg-admin-setuprepo fails to make a cg-clone'able repo
From: Randal L. Schwartz @ 2005-11-12 16:51 UTC (permalink / raw)
To: Petr Baudis; +Cc: Git Mailing List
In-Reply-To: <20051111001333.GH30496@pasky.or.cz>
>>>>> "Petr" == Petr Baudis <pasky@suse.cz> writes:
Petr> But this is actually an entirely unrelated bug, it seems - pretty
Petr> strange. I revived another few months old thread because of this, since
Petr> it seems the 'b' was actually a typo and we don't want to pass 'cp' this
Petr> argument at all, but something different. Does cloning locally otherwise
Petr> work?
When I tried "local cloning", I first tried to make an empty archive
with cgi-init, but it dumped me into an editor. I couldn't find a
switch to turn that off. Perhaps the manpage should say "you'll
be dumped in to an editor", and perhaps provide a way to pass that
message with "-m".
So, on a whim, I realize that it's performing an initial commit, so
I try cg-init -I. And here's the result:
localhost:~/Projects/Git % cg-init -I my_master
defaulting to local storage area
localhost:~/Projects/Git % cg-clone my_master my_slave
defaulting to local storage area
find: /Users/merlyn/Projects/Git/my_master: No such file or directory
Hard links don't work - using copy
Fetching head...
cp: illegal option -- b
usage: cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src target
cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src1 ... srcN directory
cg-fetch: unable to get the HEAD branch
cg-clone: fetch failed
So, I guess your answer is no. What is "-b" for cp. That's not
on my manpage.
And, after reading the manpage closer, I see my first "my_master"
was created wrong. But no error message. :( So, repeating this
using a subdirectory...
localhost:~/Projects/Git % rm -rf .git
localhost:~/Projects/Git % mkdir my_master
localhost:~/Projects/Git % cd my_master
localhost:..ects/Git/my_master % cg-init -I
defaulting to local storage area
localhost:..ects/Git/my_master % cd ..
localhost:~/Projects/Git % cg-clone my_master my_slave
defaulting to local storage area
Hard links don't work - using copy
Fetching head...
cp: illegal option -- b
usage: cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src target
cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src1 ... srcN directory
cg-fetch: unable to get the HEAD branch
cg-clone: fetch failed
Again, lots of "no" answers there.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
^ permalink raw reply
* [PATCH] Fix for multiple alternates requests in http-fetch
From: Nick Hengeveld @ 2005-11-12 17:11 UTC (permalink / raw)
To: git
Stop additional alternates requests from starting if one is already in
progress. This adds an optional callback which is processed after a slot
has finished running.
Signed-off-by: Nick Hengeveld <nickh@reactrix.com>
---
http-fetch.c | 178 ++++++++++++++++++++++++++++++++++++++--------------------
1 files changed, 117 insertions(+), 61 deletions(-)
applies-to: 9a0f0c748316751fbf593a21f2b16bcdd975095a
6a34e305a8c718b69604e416e0766ab60e5c1944
diff --git a/http-fetch.c b/http-fetch.c
index f39e748..99b6cc7 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -25,7 +25,7 @@
#define PREV_BUF_SIZE 4096
#define RANGE_HEADER_SIZE 30
-static int got_alternates = 0;
+static int got_alternates = -1;
static int active_requests = 0;
static int data_received;
@@ -87,9 +87,19 @@ struct active_request_slot
int done;
CURLcode curl_result;
long http_code;
+ void *callback_data;
+ void (*callback_func)(void *data);
struct active_request_slot *next;
};
+struct alt_request {
+ char *base;
+ char *url;
+ struct buffer *buffer;
+ struct active_request_slot *slot;
+ int http_specific;
+};
+
static struct transfer_request *request_queue_head = NULL;
static struct active_request_slot *active_queue_head = NULL;
@@ -237,7 +247,7 @@ static size_t fwrite_sha1_file(void *ptr
static void process_curl_messages(void);
static void process_request_queue(void);
#endif
-static int fetch_alternates(char *base);
+static void fetch_alternates(char *base);
static CURL* get_curl_handle(void)
{
@@ -324,6 +334,8 @@ static struct active_request_slot *get_a
slot->in_use = 1;
slot->done = 0;
slot->local = NULL;
+ slot->callback_data = NULL;
+ slot->callback_func = NULL;
curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);
curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_range_header);
curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
@@ -601,6 +613,12 @@ static void process_curl_messages(void)
} else {
fprintf(stderr, "Received DONE message for unknown request!\n");
}
+
+ /* Process slot callback if appropriate */
+ if (slot->callback_func != NULL) {
+ slot->callback_func(slot->callback_data);
+ }
+
if (request != NULL) {
request->curl_result = curl_result;
request->http_code = slot->http_code;
@@ -766,72 +784,51 @@ static int setup_index(struct alt_base *
return 0;
}
-static int fetch_alternates(char *base)
+static void process_alternates(void *callback_data)
{
- int ret = 0;
- struct buffer buffer;
- char *url;
- char *data;
- int i = 0;
- int http_specific = 1;
+ struct alt_request *alt_req = (struct alt_request *)callback_data;
+ struct active_request_slot *slot = alt_req->slot;
struct alt_base *tail = alt;
+ char *base = alt_req->base;
static const char null_byte = '\0';
+ char *data;
+ int i = 0;
- struct active_request_slot *slot;
-
- if (got_alternates)
- return 0;
-
- data = xmalloc(4096);
- buffer.size = 4096;
- buffer.posn = 0;
- buffer.buffer = data;
-
- if (get_verbosely)
- fprintf(stderr, "Getting alternates list for %s\n", base);
-
- url = xmalloc(strlen(base) + 31);
- sprintf(url, "%s/objects/info/http-alternates", base);
-
- slot = get_active_slot();
- curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
- curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,
- fwrite_buffer_dynamic);
- curl_easy_setopt(slot->curl, CURLOPT_URL, url);
- if (start_active_slot(slot)) {
- run_active_slot(slot);
- if (slot->curl_result != CURLE_OK || !buffer.posn) {
- http_specific = 0;
-
- sprintf(url, "%s/objects/info/alternates", base);
-
- slot = get_active_slot();
- curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
- curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,
- fwrite_buffer_dynamic);
- curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+ if (alt_req->http_specific) {
+ if (slot->curl_result != CURLE_OK ||
+ !alt_req->buffer->posn) {
+
+ /* Try reusing the slot to get non-http alternates */
+ alt_req->http_specific = 0;
+ sprintf(alt_req->url, "%s/objects/info/alternates",
+ base);
+ curl_easy_setopt(slot->curl, CURLOPT_URL,
+ alt_req->url);
+ active_requests++;
+ slot->in_use = 1;
+ slot->done = 0;
if (start_active_slot(slot)) {
- run_active_slot(slot);
- if (slot->curl_result != CURLE_OK) {
- free(buffer.buffer);
- if (slot->http_code == 404)
- got_alternates = 1;
- return 0;
- }
+ return;
+ } else {
+ got_alternates = -1;
+ slot->done = 1;
+ return;
}
}
- } else {
- free(buffer.buffer);
- return 0;
+ } else if (slot->curl_result != CURLE_OK) {
+ if (slot->http_code != 404) {
+ got_alternates = -1;
+ return;
+ }
}
- fwrite_buffer_dynamic(&null_byte, 1, 1, &buffer);
- buffer.posn--;
- data = buffer.buffer;
+ fwrite_buffer_dynamic(&null_byte, 1, 1, alt_req->buffer);
+ alt_req->buffer->posn--;
+ data = alt_req->buffer->buffer;
- while (i < buffer.posn) {
+ while (i < alt_req->buffer->posn) {
int posn = i;
- while (posn < buffer.posn && data[posn] != '\n')
+ while (posn < alt_req->buffer->posn && data[posn] != '\n')
posn++;
if (data[posn] == '\n') {
int okay = 0;
@@ -855,7 +852,7 @@ static int fetch_alternates(char *base)
// If the server got removed, give up.
okay = strchr(base, ':') - base + 3 <
serverlen;
- } else if (http_specific) {
+ } else if (alt_req->http_specific) {
char *colon = strchr(data + i, ':');
char *slash = strchr(data + i, '/');
if (colon && slash && colon < data + posn &&
@@ -881,15 +878,74 @@ static int fetch_alternates(char *base)
while (tail->next != NULL)
tail = tail->next;
tail->next = newalt;
- ret++;
}
}
i = posn + 1;
}
got_alternates = 1;
- free(buffer.buffer);
- return ret;
+}
+
+static void fetch_alternates(char *base)
+{
+ struct buffer buffer;
+ char *url;
+ char *data;
+ struct active_request_slot *slot;
+ static struct alt_request alt_req;
+ int num_transfers;
+
+ /* If another request has already started fetching alternates,
+ wait for them to arrive and return to processing this request's
+ curl message */
+ while (got_alternates == 0) {
+ curl_multi_perform(curlm, &num_transfers);
+ process_curl_messages();
+ process_request_queue();
+ }
+
+ /* Nothing to do if they've already been fetched */
+ if (got_alternates == 1)
+ return;
+
+ /* Start the fetch */
+ got_alternates = 0;
+
+ data = xmalloc(4096);
+ buffer.size = 4096;
+ buffer.posn = 0;
+ buffer.buffer = data;
+
+ if (get_verbosely)
+ fprintf(stderr, "Getting alternates list for %s\n", base);
+
+ url = xmalloc(strlen(base) + 31);
+ sprintf(url, "%s/objects/info/http-alternates", base);
+
+ /* Use a callback to process the result, since another request
+ may fail and need to have alternates loaded before continuing */
+ slot = get_active_slot();
+ slot->callback_func = process_alternates;
+ slot->callback_data = &alt_req;
+
+ curl_easy_setopt(slot->curl, CURLOPT_FILE, &buffer);
+ curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION,
+ fwrite_buffer_dynamic);
+ curl_easy_setopt(slot->curl, CURLOPT_URL, url);
+
+ alt_req.base = base;
+ alt_req.url = url;
+ alt_req.buffer = &buffer;
+ alt_req.http_specific = 1;
+ alt_req.slot = slot;
+
+ if (start_active_slot(slot))
+ run_active_slot(slot);
+ else
+ got_alternates = -1;
+
+ free(data);
+ free(url);
}
static int fetch_indices(struct alt_base *repo)
---
0.99.9.GIT
^ permalink raw reply related
* Re: [PATCH] Show URL in the "Getting <foo> list" http-fetch messages
From: Nick Hengeveld @ 2005-11-12 17:22 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Petr Baudis, git
In-Reply-To: <7vwtjeis09.fsf@assigned-by-dhcp.cox.net>
On Fri, Nov 11, 2005 at 05:17:42PM -0800, Junio C Hamano wrote:
> > Therefore, can I somehow stall fetching of an object? The idea is:
> >
> > I want to check the alternates list!
> > I don't have it yet...
> > ...but a request for it is already queued. Ok, I'll wait.
>
> I agree, and probably this applies when index request is in
> transit as well.
I just posted a patch to prevent another alternate request from starting
if one is already in progress.
This should not be an issue with index requests because they are only
initiated from fetch(). The previous patch to load alternates on demand
added alternate handling to process_curl_messages() so that a 404 for an
object can be handled immediately rather than waiting for the fetch()
call for that object to notice.
--
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.
^ permalink raw reply
* Re: [PATCH] Fix bunch of fd leaks in http-fetch
From: Nick Hengeveld @ 2005-11-12 17:38 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Petr Baudis, git
In-Reply-To: <7vk6feiflx.fsf@assigned-by-dhcp.cox.net>
On Fri, Nov 11, 2005 at 09:45:30PM -0800, Junio C Hamano wrote:
> It strikes me somewhat odd that these close() are not tied to
> the lifetime rule of the transfer_request structure. When the
> program falls back from an individual object to alternates, the
> same request structure is reused, but in that case ->local stays
> the same. Otherwise, the original request structure is released
> so I wonder if would make things cleaner to close ->local inside
> request_release()...
That is the intent of the fd close in finish_request() - but that isn't
called if the server returns a 404 and there are no alternates left to
try.
The following patch should fix it.
Added a call to finish_request to clean up resources if the server
returned a 404 and there are no alternates left to try.
Signed-off-by: Nick Hengeveld <nickh@reactrix.com>
---
http-fetch.c | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
applies-to: 8bae950cd42c1d615fafdf63f4c96f6b665f1e0e
fe26837d08627fbb2f5f57879ebb573474680c4a
diff --git a/http-fetch.c b/http-fetch.c
index cbb9690..78becce 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -632,6 +632,8 @@ static void process_curl_messages(void)
request->repo =
request->repo->next;
start_request(request);
+ } else {
+ finish_request(request);
}
} else {
finish_request(request);
---
0.99.9.GIT
^ permalink raw reply related
* [Patch] gitweb added site css, header and footer
From: Alan Chandler @ 2005-11-12 17:50 UTC (permalink / raw)
To: kay.sievers; +Cc: git
I wanted to include gitweb on my site - but with the ability to keep my own
site header and footer. I have therefore made a few changes to enable this
to happen. I also had a problem in that I had installed gitweb.cgi
in /usr/lib/cgi-bin, and it was attempting to read indextext.html from this
directory rather than the projectroot.
I have therefore changed it to pick up this file from projectroot, and also
two other files header.html and footer.html which contain the html for the
header and footer of each page.
Lastly, I have made a paramter at the top of the file for the the url to my
site wide css file.
Signed-off-by: Alan Chandler <alan@chandlerfamily.org.uk>
I am not sure what the netiquette is with regards these changes.
I have published them as the master branch at
http://www.chandlerfamily.org.uk/git/gitweb.git
but I have also attached a patch below (and someone can tell me if kmail does
the "right thing")
What should one do about the changes to the variables at the top of the file,
you will see I have changed $projectroot and $gitbin
---------
diff --git a/gitweb.cgi b/gitweb.cgi
index b8f1537..d16def4 100755
--- a/gitweb.cgi
+++ b/gitweb.cgi
@@ -22,10 +22,10 @@ my $rss_link = "";
# absolute fs-path which will be prepended to the project path
#my $projectroot = "/pub/scm";
-my $projectroot = "/home/kay/public_html/pub/scm";
+my $projectroot = "/var/lib/git";
# location of the git-core binaries
-my $gitbin = "/usr/bin";
+my $gitbin = "/usr/local/bin";
# location for temporary files needed for diffs
my $git_temp = "/tmp/gitweb";
@@ -33,12 +33,25 @@ my $git_temp = "/tmp/gitweb";
# target of the home link on top of all pages
my $home_link = $my_uri;
+#css url for overall site
+
+my $site_css_url = "/style/site.css";
+
+#page header and footer for all pages on this site
+
+my $site_header= $projectroot."/header.html";
+my $site_footer= $projectroot."/footer.html";
+
# html text to include at home page
-my $home_text = "indextext.html";
+my $home_text = $projectroot."/indextext.html";
# source of projects list
-#my $projects_list = $projectroot;
-my $projects_list = "index/index.aux";
+my $projects_list = $projectroot;
+#my $projects_list = "index/index.aux";
+
+
+
+
# input validation and dispatch
my $action = $cgi->param('a');
@@ -217,6 +230,12 @@ sub git_header_html {
$title .= "/$action";
}
}
+
+ my $site_css = "";
+ if (defined $site_css_url) {
+ $site_css="<link rel=\"stylesheet\" text=\"text/css\"
href=\"$site_css_url\"/>";
+ }
+
print $cgi->header(-type=>'text/html', -charset => 'utf-8', -status=>
$status, -expires => $expires);
print <<EOF;
<?xml version="1.0" encoding="utf-8"?>
@@ -228,6 +247,7 @@ sub git_header_html {
<meta name="robots" content="index, nofollow"/>
<title>$title</title>
$rss_link
+$site_css
<style type="text/css">
body { font-family: sans-serif; font-size: 12px; margin:0px; border:solid
#d9d8d1; border-width:1px; margin:10px; }
a { color:#0000cc; }
@@ -284,6 +304,13 @@ a.rss_logo:hover { background-color:#ee5
</head>
<body>
EOF
+
+ if (-f $site_header) {
+ open (my $fd, $site_header);
+ print <$fd>;
+ close $fd;
+ }
+
print "<div class=\"page_header\">\n" .
"<a href=\"http://www.kernel.org/pub/software/scm/git/docs/\"
title=\"git documentation\">" .
"<img src=\"$my_uri?a=git-logo.png\" width=\"72\" height=\"27\"
alt=\"git\" style=\"float:right; border-width:0px;\"/>" .
@@ -329,8 +356,15 @@ sub git_footer_html {
} else {
print $cgi->a({-href => "$my_uri?a=opml", -class => "rss_logo"}, "OPML") .
"\n";
}
- print "</div>\n" .
- "</body>\n" .
+ print "</div>\n";
+ if (-f $site_footer) {
+ open (my $fd, $site_footer);
+ print <$fd>;
+ close $fd;
+ }
+
+
+ print "</body>\n" .
"</html>";
}
--
Alan Chandler
http://www.chandlerfamily.org.uk
Open Source. It's the difference between trust and antitrust.
^ permalink raw reply related
* Re: [PATCH] GIT commit statistics.
From: Johannes Schindelin @ 2005-11-12 19:04 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git
In-Reply-To: <46a038f90511120419v70166c60t93d58b7544e03e3b@mail.gmail.com>
Hi,
On Sun, 13 Nov 2005, Martin Langhoff wrote:
> [...] I've been wondering whether it'd be possible to teach git to
> rebase local patches, even if that means rewriting local history. When
> you are dealing with team shared repo, the sequences of pull/push end up
> being quite messy, full of little meaningless merges. Similarly, when
> dealing with an upstream, my tree gets slowly out of sync and slightly
> messy. Eventually I get a new checkout, and rebase any pending patches
> with git-format-patch and git-am.
I thought about this problem for a while. You described a typical use case
(in a small team with one central repo), where pushes are done only from a
cleaned up branch, but the work branches stay.
If the tree corresponding to the central HEAD matches the tree
corresponding of your private merge branch at a given stage, a simple
graft should be your solution.
Example: You pull and push from/to origin on a central server. Your
(dirty) work branch is master. Now, at a given time,
origin^{tree}==master~15^{tree}. You could then generate a graft which
tells git that all parents of origin are also parents of master~15.
If at a given stage, origin^{tree}==master^{tree}, that graft would make
your next merge a fast forward, effectively cleaning up your branch
without loosing your history.
The graft could be generated by a simple script.
Would this help you?
Ciao,
Dscho
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Linus Torvalds @ 2005-11-12 19:40 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20051112155302.GD30496@pasky.or.cz>
On Sat, 12 Nov 2005, Petr Baudis wrote:
>
> rsync git+ssh(*) git(**) http
>
> git.git 0m45s 0m34s 5m30s 4m01s (++)
>
> cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)
>
>
> (*) git+ssh was to master.kernel.org, which was under significant load
> from some seemingly runaway gzip process, so that slowed things
> down.
>
> (**) The unpacking was slooooooooow yet the load was quite low. This
> should be investigated, the native git fetching is much slower
> than even HTTP.
git:// and git+ssh:// should be the exact same protocol, the main
difference in this case being the server they go to.
In the case of (**), the unpacking itself is fast, but it's done as the
stream of data comes in, so it will appear slow if the server at the other
end is slow (or the network to that server is slow).
So I think the difference between your git+ssh and git tests are purely
due to the fact that master.kernel.org sees a lot less load (both in CPU
and in networking) than the public sites, and the time differences have
nothing to do with the protocol per se.
I suspect master.kernel.org also has a beefier machine with more memory
(but even if that's not the case, it's simply true that the public
machines obviously do mirroring to a lot of other machines, and run things
like webgit and just basic serving too).
If anything, git:// as a protocol is theoretically a bit faster, since the
login procedure is faster and there's no encryption overhead.
Linus
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Petr Baudis @ 2005-11-12 19:46 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0511121134170.3263@g5.osdl.org>
Dear diary, on Sat, Nov 12, 2005 at 08:40:11PM CET, I got a letter
where Linus Torvalds <torvalds@osdl.org> said that...
>
>
> On Sat, 12 Nov 2005, Petr Baudis wrote:
> >
> > rsync git+ssh(*) git(**) http
> >
> > git.git 0m45s 0m34s 5m30s 4m01s (++)
> >
> > cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)
> >
> >
> > (*) git+ssh was to master.kernel.org, which was under significant load
> > from some seemingly runaway gzip process, so that slowed things
> > down.
> >
> > (**) The unpacking was slooooooooow yet the load was quite low. This
> > should be investigated, the native git fetching is much slower
> > than even HTTP.
(BTW, it's obviously not much slower than HTTP - I wrote that after the
first git run, but I screwed up myself at that one.)
> So I think the difference between your git+ssh and git tests are purely
> due to the fact that master.kernel.org sees a lot less load (both in CPU
> and in networking) than the public sites, and the time differences have
> nothing to do with the protocol per se.
Well, at the time of fetching, master.kernel.org with git+ssh had load
about ~3.5 and some wild gzip was eating most of the CPU there. So if
the git protocol still manages to be TEN times slower while rsync goes
full speed from that machine, I would say that this means the git server
requires way too much CPU.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply
* Re: [PATCH] Fix bunch of fd leaks in http-fetch
From: Petr Baudis @ 2005-11-12 19:55 UTC (permalink / raw)
To: Nick Hengeveld; +Cc: Junio C Hamano, git
In-Reply-To: <20051112173828.GG4051@reactrix.com>
Dear diary, on Sat, Nov 12, 2005 at 06:38:28PM CET, I got a letter
where Nick Hengeveld <nickh@reactrix.com> said that...
> On Fri, Nov 11, 2005 at 09:45:30PM -0800, Junio C Hamano wrote:
>
> > It strikes me somewhat odd that these close() are not tied to
> > the lifetime rule of the transfer_request structure. When the
> > program falls back from an individual object to alternates, the
> > same request structure is reused, but in that case ->local stays
> > the same. Otherwise, the original request structure is released
> > so I wonder if would make things cleaner to close ->local inside
> > request_release()...
>
> That is the intent of the fd close in finish_request() - but that isn't
> called if the server returns a 404 and there are no alternates left to
> try.
>
> The following patch should fix it.
What about the rest of the leaks?
Specifically, the one around release_request(), and the one caused by
re-open()ing local in start_request() when re-calling it on existing
request.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Linus Torvalds @ 2005-11-12 20:13 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20051112194646.GE30496@pasky.or.cz>
On Sat, 12 Nov 2005, Petr Baudis wrote:
> Dear diary, on Sat, Nov 12, 2005 at 08:40:11PM CET, I got a letter
> where Linus Torvalds <torvalds@osdl.org> said that...
> >
> >
> > On Sat, 12 Nov 2005, Petr Baudis wrote:
> > >
> > > rsync git+ssh(*) git(**) http
> > >
> > > git.git 0m45s 0m34s 5m30s 4m01s (++)
> > >
> > > cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)
>
> Well, at the time of fetching, master.kernel.org with git+ssh had load
> about ~3.5 and some wild gzip was eating most of the CPU there. So if
> the git protocol still manages to be TEN times slower while rsync goes
> full speed from that machine, I would say that this means the git server
> requires way too much CPU.
Look again.
master.kernel.org was _faster_ than rsync using the native git protocol,
despite being under a load of 3.5.
Look at the numbers: 45 secs for rsync, 34 secs for git protocol to
master.
Now, I don't know which rsync machine you used (you can rsync both from
master and from rync.kernel.org), since you don't say.
Now, it's unquestionably true that rsync can be faster under many
circumstances. Most notably when disk IO is really slow, since the native
git protocol will do a lot more synchronous operations, since it actually
tests what it is doing.
But I _guarantee_ you that rsync is at least ten times slower than the
native git protocol in many circumstances. It can't handle repacking
(which is critical for good server performance).
And in fact it can't handle totally unpacked directories and small updates
well either (the reason I totally stopped doing rsync was because it took
minutes to go through the whole list of unpacked objects for a small
update, while the native protocol would just fetch the needed objects and
be done with it.
So sometimes rsync is faster, sometimes the git protocol is faster. But
the git protocol is _always_ better from a sanity standpoint.
The things you get with the native git protocol:
- you don't have to trust the other end. If the other end lies about the
SHA1's of its objects, rsync will never know. It will just download the
thing, and you may have a corrupt database.
With the git protocol, we just get the objects, and recompute their
names. The other end can't lie about what their SHA is.
- the rsync protocol totally breaks down with multiple branches. It
fetches stuff it shouldn't because it doesn't know better.
- the rsync protocol scales with project size, not with change size. This
works well for small projects, where the changes are usually not all
that hugely different from the total size of the project, but it really
sucks for big projects.
- the rsync protocol fundamentally cannot handle two differently packed
trees well. That doesn't matter if you only track one tree, but it
matters _hugely_ for people (like me) who pull from tens of different
trees.
So the fact is: rsync is often slower, and _always_ less capable.
Linus
^ permalink raw reply
* Re: [PATCH] archimport improvements
From: Eric Wong @ 2005-11-12 20:21 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list
In-Reply-To: <46a038f90511120354n4584aedfhb1f2928ac41478ab@mail.gmail.com>
Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Eric,
>
>
> On 11/12/05, Eric Wong <normalperson@yhbt.net> wrote:
> > I'm another Arch-user trying out git. Unfortunately, I encountered
> > several problems with git-archimport that I needed fixed before my
> > development trees could be imported into git.
>
> Welcome and good stuff! I'll give your patches a try when I sober up.
> In the meantime, some notes after having read the patches a bit...
>
> > Bug Fixes:
> >
> > * Support for '--branch'-less Arch version names.
> > Encoding '/' to '--' (as was previously done) is not 100% reversable
> > because the "--branch" portion of an fully-qualified Arch version name
> > is optional (though not many people or Arch-related tools know this).
> >
> > * I'm encoding the '/' in the fully-qualified name as ',' to not confuse
> > other porcelains, but leaving '/' in branch names may be alright
> > provided porcelains can support them.
> >
> > * Identify git branches as an Arch "archive,category<--branch>--version"
> > Anything less than that is ambiguous as far as history and patch
> > relationships go.
>
> These bug/sanity fixes are _good_. As you mention, I wasn't aware that
> patchnames could show up not having a --branch part. Tricky...
Thanks. I got lazy one day and started ignoring --branch on some of my
personal projects to save my fingers :)
> > * Renamed directories containing renamed/moved files inside didn't get
> > tracked properly. The original code was inadequate for this, and
> > making it support all rename cases that Arch supports is too much
> > work. Instead, I maintain full-blown Arch trees in the temp dir and
> > replay patches + rsync based on that. Performance is slightly slower
> > than before, but accuracy is more important to me.
> >
> > * Permission (execute bit only because of git) tracking as a side effect
> > of the above.
>
> Hmmm. I understand what you are doing, but I'm not sure we'd want to
> replace the current code with this strategy. Importing large trees
> with hundreds (thousands) of commits is so slow it is just a no go.
> Renames are described quite well in the 'commit log', and the current
> code does handle file renames...
Untouched files inside renamed directories aren't explicitly tracked.
Renamed directories are especially a pain when a renamed one contains
sub-directories that are also renamed.
> > * Tracking changes from branches that are only cherry-picked now works
>
> Can you elaborate a bit more on this?
Basically, don't die when merge-base fails, look a few lines down.
> > * Pika-escaped filenames unhandled. This seems fixed in the latest
> > git, but I fixed it more generally and removed the ShellQuote module
> > dependency along the way.
>
> Yes, this got fixed recently. Your change here goes together with the
> 'tla get' + rsync strategy which I'm not sure about.
>
> > * Don't die() when a merge-base can't be found. Arch supports
> > merging between unrelated trees.
>
> Fair enough. Does it result on a good graft in git?
Right now I end up with separate branches that are imported (according
to git-branch) but the git-log and gitk don't seem to to show
relationships between the unrelated trees. I think find_parents()
may need to use an alternate strategy instead of warning and skipping
if a merge-base can't be found.
> > Usability enhancements:
> >
> > * Optionally detect merged branches and attempt to import their history,
> > too. Use the -D <depth> option for this. Specifying a <depth>
> > greater than 1 is usually not needed unless the tree you're tracking
> > has had history pruned.
> >
> > * Optionally attempt to auto-register unknown Arch archives from
> > mirrors.sourcecontrol.net to pull their history with the -a (boolean)
> > switch. Not sure how useful users will find this.
>
> Those two are interesting!
>
> > * Removed -A <archive> usage (unnecessary in all cases) and made all
> > Arch calls and output parsing to be compatible with both tla (tested
> > 1.3.3) and baz (1.4.2). Default is still tla, but the ARCH_CLIENT
> > environment variable may be changed to baz.
>
> That's excellent -- thanks!
>
> > Current weaknesses:
> >
> > * (Present in the original code as well).
> > The code still assumes that dates in commit logs can be trusted, which is
> > fine in most cases, but a wayward branch can screw up git-archimport and
> > cause parents to be missed.
>
> Fair enough. You mention an alternative strategy (tla ancestry) --
> have you tried it at all?
No, not yet.
--
Eric Wong
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Linus Torvalds @ 2005-11-12 20:31 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0511121159470.3263@g5.osdl.org>
On Sat, 12 Nov 2005, Linus Torvalds wrote:
>
> So the fact is: rsync is often slower, and _always_ less capable.
Side note: a lot of the rsync problems are non-issues for the initial
clone.
That initial clone is in fact the only time I think rsync can be a good
idea, especially if the server end is repacking regularly.
Then the only downside of cloning using rsync is that it will obviously
also set up the "origin" branch to be updated using rsync, which is sad if
there are alternatives.
But it may be a good idea to first clone using rsync, and then edit the
origin file to change the rsync into a git-native protocol if one is
available.
Linus
^ permalink raw reply
* git-merge-recursive: documentation and implementation notes
From: Fredrik Kuivinen @ 2005-11-12 20:34 UTC (permalink / raw)
To: git; +Cc: torvalds
Hi,
This mail contains the promised documentation for the recursive merge
strategy. I don't really know where or even if this belongs in the Git
repository, so I am not posting it as a patch. If it is understandable
it might be appropriate for Documentaion/technical.
This note is supposed to describe how the recursive merge strategy
works. As I am not a native English speaker comments regarding
spelling and grammar are more than welcome. Comments on the content
is, of course, even more welcome :)
The fundamental merge problem that the recursive strategy tries to
solve is when there are more than one common ancestor of the branches
we want to merge. This happens, for example, in the "criss-cross"
merge case:
A
|\
| \
| \
| \
| \
| \
| \
B8 C3
|\ /|
| \ / |
| \ / |
| X |
| / \ |
| / \ |
|/ \|
D8 E3
\ |
\ |
\ |
\ |
\ |
\ |
\|
M
(This example and the ASCII graphic are from Bram Cohen's excellent
description of the criss-cross case in
http://marc.theaimsgroup.com/?l=git&m=111463358500362&w=2)
We have a common base A, at commit B and D line 8 was changed and at
commit C and E line 3 was changed. There is only one file, so the same
file is modified in B, C, D and E. The result of the merge is called
M.
If we do a classical three-way merge of D and E with either A, B or C
as the common ancestor we will get conflicts. Theoretically this
shouldn't happen as both the D branch and the E branch knows about the
changes made at both B and C, the theoretical clean merge should
therefore contain line number 3 from E and line number 8 from D.
The recursive merge strategy solves this problem by recursively
merging any common ancestors. Hence, in the criss-cross scenario above
the code will detect that D and E has both B and C as their common
ancestor. It will then recursively merge B with C. The common ancestor
of B and C is A. When there is only one common ancestor an ordinary
three-way merge is done (modulo renames, will talk about those
below). So in this case we will merge B and C with A as the common
ancestor to create a temporary tree. This new tree will then be used
as the common ancestor for a merge of D and E.
In some cases there are more than two common ancestor, for example
44583d380d189095fa959ec8ba87f0cb6deb15f5 in Thomas Gleixner's
historical Linux kernel repository has no less than six common
ancestors.
The recursive strategy also has some support for renamed objects.
Consider the following ancestry graph:
O
/ \
/ \
A B
Let the file 'foo' be renamed to 'bar' in A. In the B branch 'foo' is
modified. The result of a merge of A and B will contain a file 'bar'
with whatever contents merge(1) produced given A/bar and B/foo as the
files to be merged and O/foo as the ancestor. Renames are detected by
executing 'git-diff-tree -M --diff-filter=R <branch> <ancestor>' for
both branches. The rename processing may be seen as a pre-processing
step to the three-way merge procedure in the sense that renames are
processed first and then the ordinary file-level merging is done.
There are, essentially, only two merge scenarios which involves
renames that are cleanly merged. The first one is the case described
above when a file is renamed (and possibly modified) in one branch and
modified in the other branch. The other scenario is when the same file
is renamed to the same new name in both branches. However, there are
several scenarios which causes conflicts, if for example we are going
to merge the branches A and B then the following table contain a few
of the conflict cases:
In A In B
-------------------------------------
rename a -> b rename a -> c
rename a -> b rename c -> b
add a rename b -> a
delete a rename a -> b
(Each line should be interpreted as an individual scenario, file names
on different lines do not have anything in common.)
There is currently no support for directory renames. If a directory is
renamed it will be processed as if every file in the directory was
renamed. This will cause clean merges of certain cases which probably
should be conflicts. (Such as: directory 'foo' is renamed to 'bar' in
one branch, in the other branch a new file 'foo/baz' is added. With
the current code we will end up with both a 'foo' directory,
containing 'foo/baz', and a 'bar' directory containing whatever 'foo'
contained in the common ancestor. A better outcome would probably be
to flag this as a conflict.)
The code tries, as far as possible, to let the user know what happens
in the conflict cases. The idea is that the user shouldn't have to
guess what the algorithm has done when she stumbles upon some strange
corner case. For example, if a file 'foo/bar' is added in the branch
'testing' and 'baz' is renamed to 'foo' in the branch 'release' then
the user will be given the message:
CONFLICT (rename/directory): Rename baz => foo in branch release directory foo added in testing
Renaming baz to foo~release instead
(The file name 'foo~release' is carefully constructed to not overwrite
any existing files.)
What follows is pseudo code which describes the algorithm from a
high-level point of view. Text within [brackets] is either a name of a
Python function where the real code can be found or a short
description of where the code can be found. Currently, the code is
contained in two files, git-merge-recursive.py and gitMergeCommon.py.
[merge]
merge(A, B)
{
[getCommonAncestors]
commonAncestors = getCommonAncestors(A, B)
mergeCommonAncestor = commonAncestors[0]
for(i = 1; i < length(commonAncestors); i++) {
mergedCommonAncestor = merge(mergedCommonAncestor, commonAncestors[i])
}
[in mergeTrees]
git-read-tree -m mergeCommonAncestor A B
[loop in mergeTrees, git-diff-tree execution in getRenames]
for each file in either
'git-diff-tree -M --diff-filter=R -r mergeCommonAncestor A' or
'git-diff-tree -M --diff-filter=R -r mergeCommonAncestor B':
[processRenames]
All rename processing is done here. This includes detection of
various conflict cases and merge the 'rename in one branch and
modified in the other'-case
[processRenames]
For every file which is merged cleanly update the index
[loop in mergeTrees]
for each file which is still 'unmerged' in the index:
[processEntry]
Do a file-level merge. This is more or less the same thing as
git-merge-one-file does, but it is implemented in Python.
Return the new merged tree.
}
Please note that the pseudo code above just is a sketch of the real
code. One important difference is that in the real code a resulting
tree is always returned from the 'merge'-function as soon as the
'merge'-function has been called recursively.
Lets assume there is a merge conflict when we merge B and C in the
criss-cross case above. Then both D and E must resolve this
conflict. If they have done it in the same way we wont get a merge
conflict at M, if they have resolved it differently we will get a
merge conflict. In the first case there is no merge conflict at M, in
the second case the user has to pick which one of the two different
resolutions she wants.
Note that the algorithm will happily write non-clean merge results to
the object database during the "merge common ancestors" stage. Hence,
when we are merging B and C "internally" we will _not_ ask the user to
resolve any eventual merge conflicts.
The temporary trees and non-cleanly merged files which are written to
the object database are not removed when the merge is finished. Hence,
in some cases, there will be unreachable objects in the database after
a merge.
- Fredrik
^ permalink raw reply
* Re: [PATCH 4/5] Overhaul of changeset application
From: Eric Wong @ 2005-11-12 20:49 UTC (permalink / raw)
To: Martin Langhoff; +Cc: git list
In-Reply-To: <46a038f90511120407y68031071v15244175f6458c54@mail.gmail.com>
Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Eric,
>
> I'd actually like to improve the script to handle directory renames
> and file modes correctly so we don't need to ever call the glacially
> slow `tla get` -- I don't think it's that much work, all I need is a
> sample repo. OTOH, if you think (or can convince me) that there are
> more serious problems ahead, perhaps we can have this as an
> alternative import mechanism?
tla get is hardly ever called, I don't think it's called any more than
before, even. tla replay by far the most common case and still
reasonably fast. I had to add the -I flag to rsync because it was going
at > 1 patch per second, plenty fast enough for me.
I also had an alternate implementation for using the revision library,
but that was slower than the current strategy because it had to do
two full Arch tree integrity checks for each patch applied.
Even on a hot (fully filled) revlib and all I had to do was tla
library-find + rsync, it ran more slowly, probably because rsync
couldn't take advantage of kernel/fs-level caching when it had to
work on a different directory each time.
Tracking renamed directories (especially when nested subdirectories are
also renamed) is very, very far from pleasant.
> On 11/12/05, Eric Wong <normalperson@yhbt.net> wrote:
> > - Correctly parse multi-line summary text in patch-logs
>
> Was this broken!? I'm sure I've imported multiline summaries!
It only got the first summary line when I tried it. Also, it's possible
for hand-made message bodies to fool archimport if it has "headers"
after the first \n\n. IIRC, some old tools copied entire logs of merged
changesets into the message body.
--
Eric Wong
^ permalink raw reply
* Re: git-merge-recursive: documentation and implementation notes
From: Petr Baudis @ 2005-11-12 21:03 UTC (permalink / raw)
To: Fredrik Kuivinen; +Cc: git
In-Reply-To: <20051112203457.GA5234@c165.ib.student.liu.se>
Hi,
Dear diary, on Sat, Nov 12, 2005 at 09:34:57PM CET, I got a letter
where Fredrik Kuivinen <freku045@student.liu.se> said that...
> Lets assume there is a merge conflict when we merge B and C in the
> criss-cross case above. Then both D and E must resolve this
> conflict. If they have done it in the same way we wont get a merge
> conflict at M, if they have resolved it differently we will get a
> merge conflict. In the first case there is no merge conflict at M, in
> the second case the user has to pick which one of the two different
> resolutions she wants.
>
> Note that the algorithm will happily write non-clean merge results to
> the object database during the "merge common ancestors" stage. Hence,
> when we are merging B and C "internally" we will _not_ ask the user to
> resolve any eventual merge conflicts.
I find this part somewhat unclear. So how does the user pick which of
the two different resolution she wants if we will not ask the user to
resolve any conflicts?
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Petr Baudis @ 2005-11-12 21:20 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0511121159470.3263@g5.osdl.org>
Dear diary, on Sat, Nov 12, 2005 at 09:13:20PM CET, I got a letter
where Linus Torvalds <torvalds@osdl.org> said that...
> On Sat, 12 Nov 2005, Petr Baudis wrote:
>
> > Dear diary, on Sat, Nov 12, 2005 at 08:40:11PM CET, I got a letter
> > where Linus Torvalds <torvalds@osdl.org> said that...
> > >
> > >
> > > On Sat, 12 Nov 2005, Petr Baudis wrote:
> > > >
> > > > rsync git+ssh(*) git(**) http
> > > >
> > > > git.git 0m45s 0m34s 5m30s 4m01s (++)
> > > >
> > > > cogito.git 2m09s 1m54s (+) 4m30s 15m11s (only single run)
> >
> > Well, at the time of fetching, master.kernel.org with git+ssh had load
> > about ~3.5 and some wild gzip was eating most of the CPU there. So if
> > the git protocol still manages to be TEN times slower while rsync goes
> > full speed from that machine, I would say that this means the git server
> > requires way too much CPU.
>
> Look again.
>
> master.kernel.org was _faster_ than rsync using the native git protocol,
> despite being under a load of 3.5.
>
> Look at the numbers: 45 secs for rsync, 34 secs for git protocol to
> master.
That's git+ssh (not git transport protocol - this DID make a difference
at first, I know it's hard to believe) on machine A vs. rsync on
machine B.
> Now, I don't know which rsync machine you used (you can rsync both from
> master and from rync.kernel.org), since you don't say.
rsync.kernel.org ("... official kernel.org hostnames for each transport.")
So I had this length more detailed benchmark which re-examined git+ssh
vs. git on master.kernel.org (good that git-daemon binds to
non-priviledged port). However, an embarrassing thing turned out - this
seems to be a network problem, actually.
From kernel.org to machine X, it takes 20s, and from machine X to my
home machine, it takes 20s. From kernel.org to my home machine, it takes
5 minutes. This concerns only the git protocols, HTTP and SSH from my
home machine to kernel.org is fast as usual. Even stranger, this is the
same for both hera.kernel.org and zeus.kernel.org, while each is at
totally different network. Tried many times. Ugh.
Well, so sorry for false alarm.
Your totally puzzled,
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Petr Baudis @ 2005-11-12 21:30 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0511121229340.3263@g5.osdl.org>
Dear diary, on Sat, Nov 12, 2005 at 09:31:32PM CET, I got a letter
where Linus Torvalds <torvalds@osdl.org> said that...
> On Sat, 12 Nov 2005, Linus Torvalds wrote:
> >
> > So the fact is: rsync is often slower, and _always_ less capable.
>
> Side note: a lot of the rsync problems are non-issues for the initial
> clone.
But based on my "machine X" tests, git is as fast as git+ssh (which is
as it should be), which means it is even slightly faster than rsync,
even for the initial commit.
> That initial clone is in fact the only time I think rsync can be a good
> idea, especially if the server end is repacking regularly.
Yes, the only advantage of rsync I can see is that you get it all
packed. Well, I think this only means we are doing something wrong,
and perhaps we should automatically pack as well.
--
Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
^ permalink raw reply
* Re: GIT character codecs
From: Junio C Hamano @ 2005-11-12 22:00 UTC (permalink / raw)
To: Marco Costalba; +Cc: git
In-Reply-To: <20051112123712.75182.qmail@web26301.mail.ukl.yahoo.com>
Marco Costalba <mcostalba@yahoo.it> writes:
> I modified qgit to force the use of utf-8 codec instead of the local one,
> in my case ISO-8859-15.
I suspect that trying to have a globa single encoding is a wrong
approach overall. There are a handful issues to think about.
First the easiest one -- the commit log messages. We encourage
use of UTF-8, because they hold people's names which are the
significant source of i18n needs. Even if your project starts
out with just a group of people whose names can be spelled in
ASCII just fine, or all western european in which case you might
be tempted to do latin-1 or 8859-15, or all Japanese with
EUC-JP, your project _might_ someday have a member whose name
cannot be spelled if you pick one of these encodings [*2*]. So
we encourage use of UTF-8 and help people by having -u flag in
git-mailinfo, for example.
But we do not _enforce_ UTF-8. I've never worked actively with
non-English free software projects, and I do not know if people
in, say, Japan write their commit log messages in Japanese. I
however think it is a reasonable thing to do if the project is
local and exclusive (e.g. a company internal project). And if
local encoding is more convenient to handle for users
(e.g. their "less" is configured to expect something non UTF-8),
it is perfectly reasonable to do all their commit log messages
in their local encoding, as long as the members understand that
is the project's policy. And it is probably reasonable to
assume that within one project, a single encoding is used for
commit log messages, although it may not be UTF-8.
What this means for qgit is that it is often sufficient for its
log browser to support one encoding at a time, provided if it
allows the user to switch which encoding to use depending on
what project is being viewed.
The trouble you had is however not with commit messages, but
with the blob contents (i.e. user data). First of all, the user
data is just any binary goo for git, and the interpretation of
it is left to the Porcelains and the users. There are two
things to talk about here.
I imagine qgit has, in addition to diff-between-revisions
viewer, a blob viewer that lets your users browse a whole file
in a given revision. The files in the Documentation directory
in git.git/ right now happens to be encoded in UTF-8 because
they are asciidoc sources. But other projects may use different
encodings, and even a single project can have its i18n message
files in different encodings and charsets in different files.
Users probably want to be able to view all of them, even if they
only understand a couple of languages and not others.
What this means for qgit is that at least you should be able to
show a whole file in a single encoding, but if you show more
than two files at the same time, one in each window, these
windows may be showing its contents in different encodings and
charsets. So you would need to give a way to your users to tell
you what encoding each file is in. Using global locale as the
default and having a way to override that per file basis would
be sufficient.
A more interesting thing about user data is the diff between
revisions. How should you show the diff between
git-pack-redundant documentation that changes Lukas' name
encoded from latin-1 to UTF-8?
First, git-diff ("diff" it calls) is encoding agnostic. The
lines that come out from it are taken from its input files and
if two revisions you compared both store your words in Italian
encoded in iso-8859-15, the output would be iso-8859-15. IOW,
diff would end up doing the right thing but without knowing it.
The Lukas' name encoding patch would be handled the same way.
Diff output would just say "these lines have different bytes",
and spit out '-' line encoded in latin-1 and '+' in UTF-8.
This is a mess in one sense but not necessarily so. If we could
say "Mr diff, the first file is in latin-1 and the second one is
in UTF-8, so compare them accordingly, and by the way we would
like the output in UTF-8 please", then it would be interesting
and may be sometimes useful, but in this particular case it is
useless -- there would not be any diff. And the reason _I_ ran
diff before committing that change was to make sure I did the
encoding conversion correctly, so noticing the byte-level
differences was the right thing to do.
What should qgit do about this? This "encoding gotcha fix
patch" is a special case, so I think it is perfectly reasonable
if qgit did not do anything special about it. So the "per-file
encoding override" I suggested for whole-file viewer would be
adequate. For the special "encoding fix" case, as I discussed
above, the most useful output is "raw bytes", so maybe throw in
a special encoding "raw" as one of the choices and I think you
are done.
[Footnote]
*1* Some people may argue unicode cannot spell all names, but
let's pretend it can -- or at least it can cover wider character
sets than any single local encodings. Some people further may
argue that iso-2022 does better, but let's not go there.
*2* We do not forbid you to store arbitrary binary blob in your
commit objects.
^ permalink raw reply
* Re: Remove unneeded packs
From: Lukas Sandström @ 2005-11-12 22:02 UTC (permalink / raw)
To: git; +Cc: Marcel Holtmann
In-Reply-To: <1131802238.29461.18.camel@blade>
Marcel Holtmann wrote:
> you are right. It is exactly what I was looking for. I just saw it some
> minutes ago, when I pulled the latest git tree. However to make an old
> GCC 2.95 happy, the attached patch is needed.
>
> I am not sure if it is fully working. It deletes a lot of old packs, but
> in case of the linux-2.6 tree it leaves on additional behind.
>
> .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.idx
> .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.pack
> .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.pack
> .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.idx
>
> The 4d76... is the current pack, but the b3c6... is an old one that is
> not needed anymore.
>
> Regards
>
> Marcel
This is most likley because the pack b3c6... contains unreachable objects.
git-pack-redundant only makes sure that all objects present in packfiles
still are present in packfiles after the redundant packs have been removed.
Thus, unreachable objects will also be considered as required.
Note that I haven't checked if this is the cause in this particular case,
but I have the same packfiles (I use the HTTP transport too).
I'm thinking of the possibility passing a list of objects to be ignored
on stdin to git-pack-redundant. This would hopefully solve this problem.
/Lukas Sandström
^ permalink raw reply
* Re: Cloning speed comparison, round II
From: Linus Torvalds @ 2005-11-12 22:03 UTC (permalink / raw)
To: Petr Baudis; +Cc: git
In-Reply-To: <20051112212030.GH30496@pasky.or.cz>
On Sat, 12 Nov 2005, Petr Baudis wrote:
>
> From kernel.org to machine X, it takes 20s, and from machine X to my
> home machine, it takes 20s. From kernel.org to my home machine, it takes
> 5 minutes.
That's a twilight zone moment ;)
The pack-file writer is even trying to be good about not doing lots of
small writes (it should chunk things up into 8kB chunks) so it should
actually be a nice network app and send full-sized TCP frames from the
very beginning (and nagle should mean that it continues to do so even
around the chunk boundaries, assuming the server is fast enough at
generating the data).
But it's entirely possible that one of the paths between the machine has
some logic that prioritizes known stream types - gives higher priority to
http/ssh over "unknown" protocols. That's a bad thing to do (the whole
point of the internet is that the smarts is in the end-points, not in the
network), but with so much of the packets whizzing by being
virus-generated crap, some places apparently do things like that.
Linus
^ permalink raw reply
* Re: Remove unneeded packs
From: Marcel Holtmann @ 2005-11-12 22:13 UTC (permalink / raw)
To: Lukas Sandström; +Cc: git
In-Reply-To: <43766687.2000007@etek.chalmers.se>
Hi Lukas,
> > you are right. It is exactly what I was looking for. I just saw it some
> > minutes ago, when I pulled the latest git tree. However to make an old
> > GCC 2.95 happy, the attached patch is needed.
> >
> > I am not sure if it is fully working. It deletes a lot of old packs, but
> > in case of the linux-2.6 tree it leaves on additional behind.
> >
> > .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.idx
> > .git/objects/pack/pack-4d7682fb8230fef33eb518fa8e53885ec675795e.pack
> > .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.pack
> > .git/objects/pack/pack-b3c6fbdfa36a326815de6358885c7a570a986b1b.idx
> >
> > The 4d76... is the current pack, but the b3c6... is an old one that is
> > not needed anymore.
>
> This is most likley because the pack b3c6... contains unreachable objects.
> git-pack-redundant only makes sure that all objects present in packfiles
> still are present in packfiles after the redundant packs have been removed.
>
> Thus, unreachable objects will also be considered as required.
>
> Note that I haven't checked if this is the cause in this particular case,
> but I have the same packfiles (I use the HTTP transport too).
maybe these packs are from a previous bad update. The cloned repository
I found it, is actually quite old. When I checked it with some others it
seems that it works perfect.
> I'm thinking of the possibility passing a list of objects to be ignored
> on stdin to git-pack-redundant. This would hopefully solve this problem.
Sounds good, but I don't even know what objects are involved in this
case and stops it from being marked as redundant.
Regards
Marcel
^ permalink raw reply
* RPM report on git and cogito
From: Horst von Brand @ 2005-11-12 13:53 UTC (permalink / raw)
To: git
Fedora rawhide, x86_64
Latest git and cogito build fine, and git-core-arch now has the correct tla
dependency. But git-core-email still asks for perl(Email::Valid) which is
/not/ provided by the Fedora repositories. I don't understand enough of
what it tries to do with that to go in and hack it out.
--
Dr. Horst H. von Brand User #22616 counter.li.org
Departamento de Informatica Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria +56 32 654239
Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox