* Re: [PATCH] Invert numbers and names in the git-shortlog summary mode.
From: Junio C Hamano @ 2007-12-11 18:09 UTC (permalink / raw)
To: Nicolas Pitre
Cc: Pierre Habouzit, Ingo Molnar, Jeff King, Jakub Narebski,
Christian Couder, git
In-Reply-To: <alpine.LFD.0.99999.0712111032530.555@xanadu.home>
Nicolas Pitre <nico@cam.org> writes:
> On Tue, 11 Dec 2007, Pierre Habouzit wrote:
> ...
>> Sometimes history hurts :) I don't think there is much we can do on a
>> short timescale. Maybe the old way can be slowly deprecated, and then
>> git-shortlog will be able to act like git-log.
>
> At least, HEAD could be assumed by default when stdin is a tty.
I'd say I agree.
-- >8 --
shortlog: default ot HEAD when standard input is a tty
Instead of warning the user that it is expecting git log output from the
standard input (and instructing to type the log from the keyboard),
default to traverse from HEAD when there is no rev parameter given.
This factors out a useful helper "add_head()" from builtin-diff.c to a
more appropriate place revision.c while renaming it to more descriptive
name add_head_to_pending(), as that is what the function is about.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
builtin-shortlog.c | 5 +++--
builtin-diff.c | 14 +-------------
builtin-log.c | 5 +----
revision.c | 12 ++++++++++++
revision.h | 2 ++
5 files changed, 19 insertions(+), 19 deletions(-)
diff --git a/builtin-shortlog.c b/builtin-shortlog.c
index 90666cb..3d8d709 100644
--- a/builtin-shortlog.c
+++ b/builtin-shortlog.c
@@ -249,9 +249,10 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix)
read_mailmap(&mailmap, ".mailmap", &common_repo_prefix);
+ /* assume HEAD if from a tty */
+ if (!rev.pending.nr && isatty(0))
+ add_head_to_pending(&rev);
if (rev.pending.nr == 0) {
- if (isatty(0))
- fprintf(stderr, "(reading log to summarize from standard input)\n");
read_from_stdin(&list);
}
else
diff --git a/builtin-diff.c b/builtin-diff.c
index 1b61599..55fb84c 100644
--- a/builtin-diff.c
+++ b/builtin-diff.c
@@ -176,18 +176,6 @@ static int builtin_diff_combined(struct rev_info *revs,
return 0;
}
-void add_head(struct rev_info *revs)
-{
- unsigned char sha1[20];
- struct object *obj;
- if (get_sha1("HEAD", sha1))
- return;
- obj = parse_object(sha1);
- if (!obj)
- return;
- add_pending_object(revs, obj, "HEAD");
-}
-
static void refresh_index_quietly(void)
{
struct lock_file *lock_file;
@@ -272,7 +260,7 @@ int cmd_diff(int argc, const char **argv, const char *prefix)
if (!strcmp(arg, "--"))
break;
else if (!strcmp(arg, "--cached")) {
- add_head(&rev);
+ add_head_to_pending(&rev);
if (!rev.pending.nr)
die("No HEAD commit to compare with (yet)");
break;
diff --git a/builtin-log.c b/builtin-log.c
index e1f1cf6..d375c9d 100644
--- a/builtin-log.c
+++ b/builtin-log.c
@@ -18,9 +18,6 @@
static int default_show_root = 1;
static const char *fmt_patch_subject_prefix = "PATCH";
-/* this is in builtin-diff.c */
-void add_head(struct rev_info *revs);
-
static void add_name_decoration(const char *prefix, const char *name, struct object *obj)
{
int plen = strlen(prefix);
@@ -746,7 +743,7 @@ int cmd_format_patch(int argc, const char **argv, const char *prefix)
* does not have.
*/
rev.pending.objects[0].item->flags |= UNINTERESTING;
- add_head(&rev);
+ add_head_to_pending(&rev);
}
/*
* Otherwise, it is "format-patch -22 HEAD", and/or
diff --git a/revision.c b/revision.c
index 2a59035..7e2f4f1 100644
--- a/revision.c
+++ b/revision.c
@@ -139,6 +139,18 @@ void add_pending_object(struct rev_info *revs, struct object *obj, const char *n
add_pending_object_with_mode(revs, obj, name, S_IFINVALID);
}
+void add_head_to_pending(struct rev_info *revs)
+{
+ unsigned char sha1[20];
+ struct object *obj;
+ if (get_sha1("HEAD", sha1))
+ return;
+ obj = parse_object(sha1);
+ if (!obj)
+ return;
+ add_pending_object(revs, obj, "HEAD");
+}
+
static struct object *get_reference(struct rev_info *revs, const char *name, const unsigned char *sha1, unsigned int flags)
{
struct object *object;
diff --git a/revision.h b/revision.h
index 992e1e9..8572315 100644
--- a/revision.h
+++ b/revision.h
@@ -130,6 +130,8 @@ extern void add_object(struct object *obj,
extern void add_pending_object(struct rev_info *revs, struct object *obj, const char *name);
+extern void add_head_to_pending(struct rev_info *);
+
enum commit_action {
commit_ignore,
commit_show,
^ permalink raw reply related
* Re: [PATCH] Better errors when trying to merge a submodule
From: Finn Arne Gangstad @ 2007-12-11 18:11 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vsl2al5ia.fsf@gitster.siamese.dyndns.org>
On Mon, Dec 10, 2007 at 11:22:05AM -0800, Junio C Hamano wrote:
> Finn Arne Gangstad <finnag@pvv.org> writes:
>
> > Instead of dying with weird errors when trying to merge submodules from a
> > supermodule, emit errors that show what the problem is.
>
> Thanks.
>
> Your change to merge-one-file.sh is Ok, although I'd reword the message
> a bit, and fold it as a new case arm to the existing case statement
> immediately above.
> [...]
> merge-recursive did not support merging trees that have conflicting
> changes in submodules they contain, and died. Support it exactly the
> same way as how it handles conflicting symbolic link changes --- mark it
> as a conflict, take the tentative result from the current side, and
> letting the caller resolve the conflict, without dying in merge_file()
> function.
> [...]
Your patch is obviously much nicer than the one I sent in, so please
put it in if/when convenient!
On another note, has there been any though to get merge to support
sub-module merging properly? It seems like it should be possible (and
it would make submodules a lot more useful)
- Finn Arne
^ permalink raw reply
* Re: [ANNOUNCE] ugit: a pyqt-based git gui // was: Re: If you would write git from scratch now, what would you change?
From: Marco Costalba @ 2007-12-11 18:20 UTC (permalink / raw)
To: David; +Cc: Andy Parkins, git
In-Reply-To: <402731c90712110548k67f28b64w5afa93ee908ce73b@mail.gmail.com>
On Dec 11, 2007 2:48 PM, David <davvid@gmail.com> wrote:
>
> Of course there are some notable things missing (such as proper i18n),
> but it's not too bad for a first draft.
>
Cannot start the thing...
$ python bin/ugit.py
Traceback (most recent call last):
File "bin/ugit.py", line 6, in <module>
from ugitlibs.models import GitModel
ImportError: No module named ugitlibs.models
$
Some hints?
Thanks
Marco
^ permalink raw reply
* Re: [PATCH] Fix clone not to ignore depth when performing a local clone
From: Charles Bailey @ 2007-12-11 18:26 UTC (permalink / raw)
To: Johannes Schindelin; +Cc: git
In-Reply-To: <Pine.LNX.4.64.0712111154070.27959@racer.site>
On Tue, Dec 11, 2007 at 11:54:54AM +0000, Johannes Schindelin wrote:
> Hi,
>
> On Tue, 11 Dec 2007, Charles Bailey wrote:
>
> > @@ -205,7 +205,10 @@ fi
> > # it is local
> > if base=$(get_repo_base "$repo"); then
> > repo="$base"
> > - local=yes
> > + if test -z "$depth"
> > + then
> > + local=yes
> > + fi
> > fi
>
> Or shorter:
>
> - local=yes
> + test -z "$depth" && local=yes
>
> Ciao,
> Dscho
>
Yes, I have no particular preference. I was just following the
'convention' of the -z "$origin" test a few lines above although
I could see uses of both alternatives as well as instances of [...]
instead of test ... .
Charles.
^ permalink raw reply
* Re: git annotate runs out of memory
From: Marco Costalba @ 2007-12-11 18:32 UTC (permalink / raw)
To: Daniel Berlin; +Cc: git
In-Reply-To: <4aca3dc20712110933i636342fbifb15171d3e3cafb3@mail.gmail.com>
On Dec 11, 2007 6:33 PM, Daniel Berlin <dberlin@dberlin.org> wrote:
>
> Annotate is treasured by gcc developers (this was a key sticking point
> in svn conversion).
> Having an annotate that is 2x slower and takes 15x memory would not
> fly (regardless of how good the results are).
>
Speed of annotation is mainly due to getting the file history more
then calculating the actual annotation.
I don't know *how* file history is stored in the others scm, perhaps
is easier to retrieve, i.e. without a full walk across the
revisions...
In case you have qgit (especially the 2.0 version that is much faster
in this feature) I would be very interested to have annotation times
on this file. Indeed annotation times are shown splitted between file
history retrieval, based on something along the lines of "git log -p
-- <path>", and actual annotation calculation (fully internal at
qgit).
I would be interested in cold start and warm cache start (close the
annotation tab and start annotation again).
Thanks (a lot)
Marco
^ permalink raw reply
* Re: git annotate runs out of memory
From: Linus Torvalds @ 2007-12-11 18:40 UTC (permalink / raw)
To: Daniel Berlin; +Cc: git
In-Reply-To: <4aca3dc20712110933i636342fbifb15171d3e3cafb3@mail.gmail.com>
On Tue, 11 Dec 2007, Daniel Berlin wrote:
>
> This seems to be a common problem with git. It seems to use a lot of
> memory to perform common operations on the gcc repository (even though
> it is faster in some cases than hg).
The thing is, git has a very different notion of "common operations" than
you do.
To git, "git annotate" is just about the *last* thing you ever want to do.
It's not a common operation, it's a "last resort" operation. In git, the
whole workflow is designed for "git log -p <pathnamepattern>" rather than
annotate/blame.
In fact, we didn't support annotate at all for the first year or so of
git.
The reason for git being relatively slow is exactly that git doesn't have
"file history" at all, and only tracks full snapshots. So "git blame" is
really a very complex operation that basically looks at the global history
(because nothing else exists) and will basically generate a totally
different "view" of local history from that one.
The disadvantage is that it's much slower and much more costly than just
having a local history view to begin with.
However, the absolutely *huge* advantage is that it isn't then limited to
local history.
So where git shines is when you actually use the global history, and do
merges or when you track more than one file (which others find hard, but
git finds much more natural).
An examples of this is content that actually comes from multiple files.
File-based systems simply cannot do this at all. They aren't just slower,
they are totally unable to do it sanely. For git, it's all the same: it
never really cares about file boundaries in the first place.
The other example is doing things like "git log -p drivers/char", where
you don't ask for the log of a single file, but a general file pattern,
and get (still atomic!) commits as the result.
And perhaps the best example is just tracking code when you have two files
that merge into one (possibly because the "same" file was created
independently in two different branches). git gets things like that right
without even thinking about it. Others tend to just flounder about and
can't do anything at all about it.
That said, I'll see if I can speed up "git blame" on the gcc repository.
It _is_ a fundamentally much more expensive operation than it is for
systems that do single-file things.
Linus
^ permalink raw reply
* Re: Something is broken in repack
From: Jon Smirl @ 2007-12-11 18:43 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Nicolas Pitre, Junio C Hamano, gcc, Git Mailing List
In-Reply-To: <alpine.LFD.0.9999.0712110806540.25032@woody.linux-foundation.org>
On 12/11/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Tue, 11 Dec 2007, Jon Smirl wrote:
> >
> > So why does our threaded code take 20 CPU minutes longer (12%) to run
> > than the same code with a single thread?
>
> Threaded code *always* takes more CPU time. The only thing you can hope
> for is a wall-clock reduction. You're seeing probably a combination of
> (a) more cache misses
> (b) bigger dataset active at a time
> and a probably fairly miniscule
> (c) threading itself tends to have some overheads.
>
> > Q6600 is just two E6600s in the same package, the caches are not shared.
>
> Sure they are shared. They're just not *entirely* shared. But they are
> shared between each two cores, so each thread essentially has only half
> the cache they had with the non-threaded version.
>
> Threading is *not* a magic solution to all problems. It gives you
> potentially twice the CPU power, but there are real downsides that you
> should keep in mind.
>
> > Why does the threaded code need 2.24GB (google allocator, 2.85GB gcc)
> > with 4 threads? But only need 950MB with one thread? Where's the extra
> > gigabyte going?
>
> I suspect that it's really simple: you have a few rather big files in the
> gcc history, with deep delta chains. And what happens when you have four
> threads running at the same time is that they all need to keep all those
> objects that they are working on - and their hash state - in memory at the
> same time!
>
> So if you want to use more threads, that _forces_ you to have a bigger
> memory footprint, simply because you have more "live" objects that you
> work on. Normally, that isn't much of a problem, since most source files
> are small, but if you have a few deep delta chains on big files, both the
> delta chain itself is going to use memory (you may have limited the size
> of the cache, but it's still needed for the actual delta generation, so
> it's not like the memory usage went away).
This makes sense. Those runs that blew up to 4.5GB were a combination
of this effect and fragmentation in the gcc allocator. Google
allocator appears to be much better at controlling fragmentation.
Is there a reasonable scheme to force the chains to only be loaded
once and then shared between worker threads? The memory blow up
appears to be directly correlated with chain length.
>
> That said, I suspect there are a few things fighting you:
>
> - threading is hard. I haven't looked a lot at the changes Nico did to do
> a threaded object packer, but what I've seen does not convince me it is
> correct. The "trg_entry" accesses are *mostly* protected with
> "cache_lock", but nothing else really seems to be, so quite frankly, I
> wouldn't trust the threaded version very much. It's off by default, and
> for a good reason, I think.
>
> For example: the packing code does this:
>
> if (!src->data) {
> read_lock();
> src->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);
> read_unlock();
> ...
>
> and that's racy. If two threads come in at roughly the same time and
> see a NULL src->data, theÿ́'ll both get the lock, and they'll both
> (serially) try to fill it in. It will all *work*, but one of them will
> have done unnecessary work, and one of them will have their result
> thrown away and leaked.
That may account for the threaded version needing an extra 20 minutes
CPU time. An extra 12% of CPU seems like too much overhead for
threading. Just letting a couple of those long chain compressions be
done twice
>
> Are you hitting issues like this? I dunno. The object sorting means
> that different threads normally shouldn't look at the same objects (not
> even the sources), so probably not, but basically, I wouldn't trust the
> threading 100%. It needs work, and it needs to stay off by default.
>
> - you're working on a problem that isn't really even worth optimizing
> that much. The *normal* case is to re-use old deltas, which makes all
> of the issues you are fighting basically go away (because you only have
> a few _incremental_ objects that need deltaing).
I agree, this problem only occurs when people import giant
repositories. But every time someone hits these problems they declare
git to be screwed up and proceed to thrash it in their blogs.
> In other words: the _real_ optimizations have already been done, and
> are done elsewhere, and are much smarter (the best way to optimize X is
> not to make X run fast, but to avoid doing X in the first place!). The
> thing you are trying to work with is the one-time-only case where you
> explicitly disable that big and important optimization, and then you
> complain about the end result being slow!
>
> It's like saying that you're compiling with extreme debugging and no
> optimizations, and then complaining that the end result doesn't run as
> fast as if you used -O2. Except this is a hundred times worse, because
> you literally asked git to do the really expensive thing that it really
> really doesn't want to do ;)
>
> > Is there another allocator to try? One that combines Google's
> > efficiency with gcc's speed?
>
> See above: I'd look around at threading-related bugs and check the way we
> lock (or don't) accesses.
>
> Linus
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Something is broken in repack
From: Nicolas Pitre @ 2007-12-11 18:57 UTC (permalink / raw)
To: Jon Smirl; +Cc: Linus Torvalds, Junio C Hamano, gcc, Git Mailing List
In-Reply-To: <9e4733910712111043h6a361996x740f4dba3d742da5@mail.gmail.com>
[-- Attachment #1: Type: TEXT/PLAIN, Size: 3040 bytes --]
On Tue, 11 Dec 2007, Jon Smirl wrote:
> This makes sense. Those runs that blew up to 4.5GB were a combination
> of this effect and fragmentation in the gcc allocator.
I disagree. This is insane.
> Google allocator appears to be much better at controlling fragmentation.
Indeed. And if fragmentation is indeed wasting half of Git's memory
usage then we'll have to come with a custom memory allocator.
> Is there a reasonable scheme to force the chains to only be loaded
> once and then shared between worker threads? The memory blow up
> appears to be directly correlated with chain length.
No. That would be the equivalent of holding each revision of all files
uncompressed all at once in memory.
> > That said, I suspect there are a few things fighting you:
> >
> > - threading is hard. I haven't looked a lot at the changes Nico did to do
> > a threaded object packer, but what I've seen does not convince me it is
> > correct. The "trg_entry" accesses are *mostly* protected with
> > "cache_lock", but nothing else really seems to be, so quite frankly, I
> > wouldn't trust the threaded version very much. It's off by default, and
> > for a good reason, I think.
> >
> > For example: the packing code does this:
> >
> > if (!src->data) {
> > read_lock();
> > src->data = read_sha1_file(src_entry->idx.sha1, &type, &sz);
> > read_unlock();
> > ...
> >
> > and that's racy. If two threads come in at roughly the same time and
> > see a NULL src->data, theÿ́'ll both get the lock, and they'll both
> > (serially) try to fill it in. It will all *work*, but one of them will
> > have done unnecessary work, and one of them will have their result
> > thrown away and leaked.
>
> That may account for the threaded version needing an extra 20 minutes
> CPU time. An extra 12% of CPU seems like too much overhead for
> threading. Just letting a couple of those long chain compressions be
> done twice
No it may not. This theory is wrong as explained before.
> >
> > Are you hitting issues like this? I dunno. The object sorting means
> > that different threads normally shouldn't look at the same objects (not
> > even the sources), so probably not, but basically, I wouldn't trust the
> > threading 100%. It needs work, and it needs to stay off by default.
> >
> > - you're working on a problem that isn't really even worth optimizing
> > that much. The *normal* case is to re-use old deltas, which makes all
> > of the issues you are fighting basically go away (because you only have
> > a few _incremental_ objects that need deltaing).
>
> I agree, this problem only occurs when people import giant
> repositories. But every time someone hits these problems they declare
> git to be screwed up and proceed to thrash it in their blogs.
It's not only for repack. Someone just reported git-blame being
unusable too due to insane memory usage, which I suspect is due to the
same issue.
Nicolas
^ permalink raw reply
* Re: git annotate runs out of memory
From: Matthieu Moy @ 2007-12-11 19:01 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Daniel Berlin, git
In-Reply-To: <alpine.LFD.0.9999.0712111018540.25032@woody.linux-foundation.org>
Linus Torvalds <torvalds@linux-foundation.org> writes:
> The other example is doing things like "git log -p drivers/char", where
> you don't ask for the log of a single file, but a general file pattern,
> and get (still atomic!) commits as the result.
I've seen you pointing this kind of examples many times, but is that
really different from what even SVN does? "svn log drivers/char" will
also list atomic commits, and give me a filtered view of the global
log.
So, yes, that's cool, but I don't see a real difference between git
and almost anything else (except CVS which really got this wrong, no
big surprise).
--
Matthieu
^ permalink raw reply
* Re: git annotate runs out of memory
From: Daniel Berlin @ 2007-12-11 19:03 UTC (permalink / raw)
To: Marco Costalba; +Cc: git
In-Reply-To: <e5bfff550712111032p60fedbedu304cab834ce86eb9@mail.gmail.com>
On 12/11/07, Marco Costalba <mcostalba@gmail.com> wrote:
> On Dec 11, 2007 6:33 PM, Daniel Berlin <dberlin@dberlin.org> wrote:
> >
> > Annotate is treasured by gcc developers (this was a key sticking point
> > in svn conversion).
> > Having an annotate that is 2x slower and takes 15x memory would not
> > fly (regardless of how good the results are).
> >
>
> Speed of annotation is mainly due to getting the file history more
> then calculating the actual annotation.
>
Yes, i figured as much.
> I don't know *how* file history is stored in the others scm, perhaps
> is easier to retrieve, i.e. without a full walk across the
> revisions...
It is stored in an easier format. However, can you not simply provide
side-indexes to do the annotation?
I guess that own't work in git because you can change history (in
other scm's, history is readonly so you could know the results for
committed revisions will never change).
> I would be interested in cold start and warm cache start (close the
> annotation tab and start annotation again).
I will try to do this.
^ permalink raw reply
* Re: git annotate runs out of memory
From: Nicolas Pitre @ 2007-12-11 19:06 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Daniel Berlin, git
In-Reply-To: <alpine.LFD.0.9999.0712111018540.25032@woody.linux-foundation.org>
On Tue, 11 Dec 2007, Linus Torvalds wrote:
> That said, I'll see if I can speed up "git blame" on the gcc repository.
> It _is_ a fundamentally much more expensive operation than it is for
> systems that do single-file things.
It has no excuse for eating up to 1.6GB or RAM though. That's plainly
wrong.
Nicolas
^ permalink raw reply
* Re: git annotate runs out of memory
From: Daniel Berlin @ 2007-12-11 19:09 UTC (permalink / raw)
To: Linus Torvalds; +Cc: git
In-Reply-To: <alpine.LFD.0.9999.0712111018540.25032@woody.linux-foundation.org>
On 12/11/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Tue, 11 Dec 2007, Daniel Berlin wrote:
> >
> > This seems to be a common problem with git. It seems to use a lot of
> > memory to perform common operations on the gcc repository (even though
> > it is faster in some cases than hg).
>
> The thing is, git has a very different notion of "common operations" than
> you do.
>
> To git, "git annotate" is just about the *last* thing you ever want to do.
> It's not a common operation, it's a "last resort" operation. In git, the
> whole workflow is designed for "git log -p <pathnamepattern>" rather than
> annotate/blame.
>
I understand this, and completely agree with you.
However, I cannot force GCC people to adopt completely new workflow in
this regard.
The changelog's are not useful enough (and we've had huge fights over
this) to do git log -p and figure out the info we want.
Looking through thousands of diffs to find the one that happened to
your line is also pretty annoying.
Annotate is a major use for gcc developers as a result
I wish I could fix this silliness, but i can't :)
> That said, I'll see if I can speed up "git blame" on the gcc repository.
> It _is_ a fundamentally much more expensive operation than it is for
> systems that do single-file things.
SVN had the same problem (the file retrieval was the most expensive op
on FSFS). One of the things i did to speed it up tremendously was to
do the annotate from newest to oldest (IE in reverse), and stop
annotating when we had come up with annotate info for all the lines.
If you can't speed up file retrieval itself, you can make it need less
files :)
In GCC history, it is likely you will be able to cut off at least 30%
of the time if you do this, because files often have changed entirely
multiple times.
^ permalink raw reply
* Re: v1.5.4 plans
From: Junio C Hamano @ 2007-12-11 19:13 UTC (permalink / raw)
To: Kristian Høgsberg; +Cc: git
In-Reply-To: <1197386675.5001.4.camel@hinata.boston.redhat.com>
Kristian Høgsberg <krh@redhat.com> writes:
> On Mon, 2007-12-10 at 14:37 -0800, Junio C Hamano wrote:
>> * We have a handful deprecation notices in the draft release notes, but
>> if I recall correctly, Nico wanted to add a few more. We need to
>> come up with a wording that is easy to understand for the end users
>> to decide which ancient versions will be affected.
>
> Can we deprecate .git/branches?
We deprecated it from the very beginning, in the sense that we never
wrote into it ourselves, but merely tried to pay attention to what
others wrote there. So I am open to a bullet point that officially
declares the deprecation.
But we need to come up with a removal schedule. Say, the first feature
release after June 2008?
^ permalink raw reply
* Re: git annotate runs out of memory
From: Marco Costalba @ 2007-12-11 19:14 UTC (permalink / raw)
To: Daniel Berlin; +Cc: git
In-Reply-To: <4aca3dc20712111103s1af3b045h484ea749378c6282@mail.gmail.com>
On Dec 11, 2007 8:03 PM, Daniel Berlin <dberlin@dberlin.org> wrote:
>
> > I don't know *how* file history is stored in the others scm, perhaps
> > is easier to retrieve, i.e. without a full walk across the
> > revisions...
>
> It is stored in an easier format. However, can you not simply provide
> side-indexes to do the annotation?
>
> I guess that own't work in git because you can change history (in
> other scm's, history is readonly so you could know the results for
> committed revisions will never change).
>
As Linus pointed out annotation in git is "much slower and much more
costly than just
having a local history view to begin with".
Indeed to annotate say kernel/sched.c
the time is spent by git while executing
git log -p -- kernel/sched.c
could be also 10X higher the the following annotation processing time
starting from the git log output.
Unfortunately my knowledge of git internals falls far far shorter then
guessing what could be done to increase the *one file* history case
that _seems_ to be the common one.
> > I would be interested in cold start and warm cache start (close the
> > annotation tab and start annotation again).
>
> I will try to do this.
>
Thanks. Very appreciated.
^ permalink raw reply
* Re: [ANNOUNCE] ugit: a pyqt-based git gui // was: Re: If you would write git from scratch now, what would you change?
From: Jason Sewall @ 2007-12-11 19:14 UTC (permalink / raw)
To: Marco Costalba; +Cc: David, Andy Parkins, git
In-Reply-To: <e5bfff550712111020k51829c03n5d64a94ce7c7ac2a@mail.gmail.com>
On Dec 11, 2007 1:20 PM, Marco Costalba <mcostalba@gmail.com> wrote:
> Cannot start the thing...
>
> $ python bin/ugit.py
> Traceback (most recent call last):
> File "bin/ugit.py", line 6, in <module>
> from ugitlibs.models import GitModel
> ImportError: No module named ugitlibs.models
> $
>
> Some hints?
Where did you install it?
Because I had the same problem when I did
./configure
because it puts the python stuff in
/usr/local/python2.5/site-packages/... and python doesn't look there
by default.
./configure --prefix=/usr fixed that for me (and I'm sure there's a
way to tell python to look in /usr/local... too, but I can't be
bothered with that.
I re-installed without the prefix and that error disappeared, but now I get
Traceback (most recent call last):
File "/usr/local/bin/ugit", line 12, in <module>
view = GitView (app.activeWindow())
File "../py/views.py", line 15, in __init__
File "default/ui/Window.py", line 43, in setupUi
AttributeError: setLeftMargin
I'm too busy to poke around and see why that's happening, but
hopefully someone can.
This with Python 2.5.1 and PyQt4 4.2-8 (from an up-to-date Fedora 8 install)
Jason
^ permalink raw reply
* Re: Something is broken in repack
From: Linus Torvalds @ 2007-12-11 19:17 UTC (permalink / raw)
To: Jon Smirl; +Cc: Nicolas Pitre, Junio C Hamano, gcc, Git Mailing List
In-Reply-To: <9e4733910712111043h6a361996x740f4dba3d742da5@mail.gmail.com>
On Tue, 11 Dec 2007, Jon Smirl wrote:
> >
> > So if you want to use more threads, that _forces_ you to have a bigger
> > memory footprint, simply because you have more "live" objects that you
> > work on. Normally, that isn't much of a problem, since most source files
> > are small, but if you have a few deep delta chains on big files, both the
> > delta chain itself is going to use memory (you may have limited the size
> > of the cache, but it's still needed for the actual delta generation, so
> > it's not like the memory usage went away).
>
> This makes sense. Those runs that blew up to 4.5GB were a combination
> of this effect and fragmentation in the gcc allocator. Google
> allocator appears to be much better at controlling fragmentation.
Yes. I think we do have some case where we simply keep a lot of objects
around, and if we are talking reasonably large deltas, we'll have the
whole delta-chain in memory just to unpack one single object.
The delta cache size limits kick in only when we explicitly cache old
delta results (in case they will be re-used, which is rather common), it
doesn't affect the normal "I'm using this data right now" case at all.
And then fragmentation makes it much much worse. Since the allocation
patterns aren't nice (they are pretty random and depend on just the sizes
of the objects), and the lifetimes aren't always nicely nested _either_
(they become more so when you disable the cache entirely, but that's just
death for performance), I'm not surprised that there can be memory
allocators that end up having some issues.
> Is there a reasonable scheme to force the chains to only be loaded
> once and then shared between worker threads? The memory blow up
> appears to be directly correlated with chain length.
The worker threads explicitly avoid touching the same objects, and no, you
definitely don't want to explode the chains globally once, because the
whole point is that we do fit 15 years worth of history into 300MB of
pack-file thanks to having a very dense representation. The "loaded once"
part is the mmap'ing of the pack-file into memory, but if you were to
actually then try to expand the chains, you'd be talking about many *many*
more gigabytes of memory than you already see used ;)
So what you actually want to do is to just re-use already packed delta
chains directly, which is what we normally do. But you are explicitly
looking at the "--no-reuse-delta" (aka "git repack -f") case, which is why
it then blows up.
I'm sure we can find places to improve. But I would like to re-iterate the
statement that you're kind of doing a "don't do that then" case which is
really - by design - meant to be done once and never again, and is using
resources - again, pretty much by design - wildly inappropriately just to
get an initial packing done.
> That may account for the threaded version needing an extra 20 minutes
> CPU time. An extra 12% of CPU seems like too much overhead for
> threading. Just letting a couple of those long chain compressions be
> done twice
Well, Nico pointed out that those things should all be thread-private
data, so no, the race isn't there (unless there's some other bug there).
> I agree, this problem only occurs when people import giant
> repositories. But every time someone hits these problems they declare
> git to be screwed up and proceed to thrash it in their blogs.
Sure. I'd love to do global packing without paying the cost, but it really
was a design decision. Thanks to doing off-line packing ("let it run
overnight on some beefy machine") we can get better results. It's
expensive, yes. But it was pretty much meant to be expensive. It's a very
efficient compression algorithm, after all, and you're turning it up to
eleven ;)
I also suspect that the gcc archive makes things more interesting thanks
to having some rather large files. The ChangeLog is probably the worst
case (large file with *lots* of edits), but I suspect the *.po files
aren't wonderful either.
Linus
^ permalink raw reply
* Re: git annotate runs out of memory
From: Linus Torvalds @ 2007-12-11 19:22 UTC (permalink / raw)
To: Matthieu Moy; +Cc: Daniel Berlin, git
In-Reply-To: <vpq4pepcaz5.fsf@bauges.imag.fr>
On Tue, 11 Dec 2007, Matthieu Moy wrote:
>
> I've seen you pointing this kind of examples many times, but is that
> really different from what even SVN does? "svn log drivers/char" will
> also list atomic commits, and give me a filtered view of the global
> log.
Ok, BK and CVS both got this horribly wrong, which is why I care. Maybe
this is one of the things SVN gets right.
I seriously doubt it, though. Do you get *history* right, or do you just
get a random list of commits?
Of course, to see the difference, you need to do "gitk drivers/char" or
use another of the log viewers that actually show you history too. A plain
"git log" won't make it obvious (unless you actually ask for parent
information and then just track the history in your head, in which case
you don't really need an SCM in the first place ;)
Linus
^ permalink raw reply
* Re: git annotate runs out of memory
From: Daniel Berlin @ 2007-12-11 19:24 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Matthieu Moy, git
In-Reply-To: <alpine.LFD.0.9999.0712111119310.25032@woody.linux-foundation.org>
On 12/11/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
>
>
> On Tue, 11 Dec 2007, Matthieu Moy wrote:
> >
> > I've seen you pointing this kind of examples many times, but is that
> > really different from what even SVN does? "svn log drivers/char" will
> > also list atomic commits, and give me a filtered view of the global
> > log.
>
> Ok, BK and CVS both got this horribly wrong, which is why I care. Maybe
> this is one of the things SVN gets right.
>
> I seriously doubt it, though. Do you get *history* right, or do you just
> get a random list of commits?
No, it will get actual history (IE not just things that happen to have
that path in the repository)
^ permalink raw reply
* Re: git annotate runs out of memory
From: Daniel Barkalow @ 2007-12-11 19:26 UTC (permalink / raw)
To: Daniel Berlin; +Cc: Linus Torvalds, git
In-Reply-To: <4aca3dc20712111109y5d74a292rf29be6308932393c@mail.gmail.com>
On Tue, 11 Dec 2007, Daniel Berlin wrote:
> On 12/11/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> > On Tue, 11 Dec 2007, Daniel Berlin wrote:
> > >
> > > This seems to be a common problem with git. It seems to use a lot of
> > > memory to perform common operations on the gcc repository (even though
> > > it is faster in some cases than hg).
> >
> > The thing is, git has a very different notion of "common operations" than
> > you do.
> >
> > To git, "git annotate" is just about the *last* thing you ever want to do.
> > It's not a common operation, it's a "last resort" operation. In git, the
> > whole workflow is designed for "git log -p <pathnamepattern>" rather than
> > annotate/blame.
> >
> I understand this, and completely agree with you.
> However, I cannot force GCC people to adopt completely new workflow in
> this regard.
> The changelog's are not useful enough (and we've had huge fights over
> this) to do git log -p and figure out the info we want.
> Looking through thousands of diffs to find the one that happened to
> your line is also pretty annoying.
> Annotate is a major use for gcc developers as a result
> I wish I could fix this silliness, but i can't :)
>
> > That said, I'll see if I can speed up "git blame" on the gcc repository.
> > It _is_ a fundamentally much more expensive operation than it is for
> > systems that do single-file things.
>
> SVN had the same problem (the file retrieval was the most expensive op
> on FSFS). One of the things i did to speed it up tremendously was to
> do the annotate from newest to oldest (IE in reverse), and stop
> annotating when we had come up with annotate info for all the lines.
> If you can't speed up file retrieval itself, you can make it need less
> files :)
> In GCC history, it is likely you will be able to cut off at least 30%
> of the time if you do this, because files often have changed entirely
> multiple times.
Unfortunately, we're doing that already. One improvement that is already
available is that we can do progressive annotate: we can output lines we
find in the order we find them, such that lines that changed recently
(which are usually the more interesting ones) get annotated quicker.
Obviously, you need a GUI-ish thing to do this, because pagers don't like
having stuff written out of order, but there's a good chance that a user
annotating fold-const.c will have the info for the interesting lines in a
few seconds, and go on while git is still trying to find where the boring
old lines came from.
There's also the possibility of generating caches of commit:file pairs
you've annotated, which would make generating the annotation for something
you'd annotated for a recent commit blindingly fast.
-Daniel
*This .sig left intentionally blank*
^ permalink raw reply
* [PATCH] builtin-clone: Implement git clone as a builtin command.
From: Kristian Høgsberg @ 2007-12-11 19:57 UTC (permalink / raw)
To: gister; +Cc: git
Still work-in-progress, local clones and --reference not fully functional.
---
Ok, don't flame me, I know this isn't appropriate at the moment with
stabilization for 1.5.4 going on, but I just wanted to post a heads up
on this work to avoid duplicate effort. It's one big patch at this point
and I haven't even run the test suite yet, but that will change.
cheers,
Kristian
Makefile | 2 +-
builtin-clone.c | 504 +++++++++++++++++++++++++
builtin-init-db.c | 119 +++----
builtin-rerere.c | 19 +-
builtin.h | 1 +
cache.h | 5 +
git-clone.sh => contrib/examples/git-clone.sh | 0
copy.c | 21 +
diff.c | 8 +-
git.c | 1 +
unpack-trees.c | 3 +-
unpack-trees.h | 1 +
12 files changed, 595 insertions(+), 89 deletions(-)
create mode 100644 builtin-clone.c
rename git-clone.sh => contrib/examples/git-clone.sh (100%)
diff --git a/Makefile b/Makefile
index cb1cbb1..ca42ed1 100644
--- a/Makefile
+++ b/Makefile
@@ -213,7 +213,6 @@ BASIC_LDFLAGS =
SCRIPT_SH = \
git-bisect.sh git-checkout.sh \
- git-clone.sh \
git-merge-one-file.sh git-mergetool.sh git-parse-remote.sh \
git-pull.sh git-rebase.sh git-rebase--interactive.sh \
git-repack.sh git-request-pull.sh \
@@ -327,6 +326,7 @@ BUILTIN_OBJS = \
builtin-checkout-index.o \
builtin-check-ref-format.o \
builtin-clean.o \
+ builtin-clone.o \
builtin-commit.o \
builtin-commit-tree.o \
builtin-count-objects.o \
diff --git a/builtin-clone.c b/builtin-clone.c
new file mode 100644
index 0000000..acd7beb
--- /dev/null
+++ b/builtin-clone.c
@@ -0,0 +1,504 @@
+/*
+ * Builtin "git clone"
+ *
+ * Copyright (c) 2007 Kristian Høgsberg <krh@redhat.com>
+ * Based on git-commit.sh by Junio C Hamano and Linus Torvalds
+ *
+ * Clone a repository into a different directory that does not yet exist.
+ */
+
+#include "cache.h"
+#include "parse-options.h"
+#include "fetch-pack.h"
+#include "refs.h"
+#include "tree.h"
+#include "tree-walk.h"
+#include "unpack-trees.h"
+
+/*
+ * Overall FIXMEs:
+ * - respect DB_ENVIRONMENT for .git/objects.
+ * - error path cleanup of dirs+files.
+ *
+ * Implementation notes:
+ * - dropping use-separate-remote and no-separate-remote compatibility
+ *
+ */
+static const char * const builtin_clone_usage[] = {
+ "git-clone [options] [--] <repo> [<dir>]",
+ NULL
+};
+
+static int option_quiet, option_no_checkout, option_bare;
+static int option_local, option_no_hardlinks, option_shared, option_depth;
+static char *option_template, *option_reference;
+static char *option_origin = "origin";
+static char *option_upload_pack = "git-upload-pack";
+
+static struct option builtin_clone_options[] = {
+ OPT__QUIET(&option_quiet),
+ OPT_BOOLEAN('n', "no-checkout", &option_no_checkout,
+ "don't create a checkout"),
+ OPT_BOOLEAN(0, "bare", &option_bare, "create a bare repository"),
+ OPT_BOOLEAN(0, "naked", &option_bare, "create a bare repository"),
+ OPT_BOOLEAN('l', "local", &option_local,
+ "to clone from a local repository"),
+ OPT_BOOLEAN(0, "no-hardlinks", &option_no_hardlinks,
+ "don't use local hardlinks, always copy"),
+ OPT_BOOLEAN('s', "shared", &option_shared,
+ "setup as shared repository"),
+ OPT_STRING(0, "template", &option_template, "path",
+ "path the template repository"),
+ OPT_STRING(0, "reference", &option_reference, "repo",
+ "reference repository"),
+ OPT_STRING('o', "origin", &option_origin, "branch",
+ "use <branch> instead or 'origin' to track upstream"),
+ OPT_STRING('u', "upload-pack", &option_upload_pack, "path",
+ "path to git-upload-pack on the remote"),
+ OPT_INTEGER(0, "depth", &option_depth,
+ "create a shallow clone of that depth"),
+
+ OPT_END()
+};
+
+static char *get_repo_path(const char *repo)
+{
+ const char *path;
+ struct stat buf;
+
+ path = mkpath("%s/.git", repo);
+ if (!stat(path, &buf) && S_ISDIR(buf.st_mode))
+ return xstrdup(make_absolute_path(path));
+
+ path = mkpath("%s.git", repo);
+ if (!stat(path, &buf) && S_ISDIR(buf.st_mode))
+ return xstrdup(make_absolute_path(path));
+
+ if (!stat(repo, &buf) && S_ISDIR(buf.st_mode))
+ return xstrdup(make_absolute_path(repo));
+
+ return NULL;
+}
+
+static char *guess_dir_name(const char *repo)
+{
+ const char *p, *start, *end, *limit;
+ int after_slash_or_colon;
+
+ /* Guess dir name from repository: strip trailing '/',
+ * strip trailing '[:/]*git', strip leading '.*[/:]'. */
+
+ after_slash_or_colon = 1;
+ limit = repo + strlen(repo);
+ start = repo;
+ end = limit;
+ for (p = repo; p < limit; p++) {
+ if (!prefixcmp(p, ".git")) {
+ if (!after_slash_or_colon)
+ end = p;
+ p += 3;
+ } else if (*p == '/' || *p == ':') {
+ if (end == limit)
+ end = p;
+ after_slash_or_colon = 1;
+ } else if (after_slash_or_colon) {
+ start = p;
+ end = limit;
+ after_slash_or_colon = 0;
+ }
+ }
+
+ return xstrndup(start, end - start);
+}
+
+static void
+write_alternates_file(const char *repo, const char *reference)
+{
+ char *file;
+ char *alternates;
+ int fd;
+
+ file = mkpath("%s/objects/info/alternates", repo);
+ fd = open(file, O_CREAT | O_WRONLY, 0666);
+ if (fd < 0)
+ die("failed to create %s", file);
+ alternates = mkpath("%s/objects\n", reference);
+ write_or_die(fd, alternates, strlen(alternates));
+ if (close(fd))
+ die("could not close %s", file);
+}
+
+static int
+setup_tmp_ref(const char *refname,
+ const unsigned char *sha1, int flags, void *cb_data)
+{
+ const char *ref_temp = cb_data;
+ char *path;
+ struct lock_file lk;
+ struct ref_lock *rl;
+
+ /*
+
+ echo "$ref_git/objects" >"$GIT_DIR/objects/info/alternates"
+ (
+ GIT_DIR="$ref_git" git for-each-ref \
+ --format='%(objectname) %(*objectname)'
+ ) |
+ while read a b
+ do
+ test -z "$a" ||
+ git update-ref "refs/reference-tmp/$a" "$a"
+ test -z "$b" ||
+ git update-ref "refs/reference-tmp/$b" "$b"
+ done
+
+ */
+
+ /* We go a bit out of way to use write_ref_sha1() here. We
+ * could just write the ref file directly, since neither
+ * locking or reflog really matters here. However, let's use
+ * the standard interface for writing refs as much as is
+ * possible given that get_git_dir() != the repo we're writing
+ * the refs in. */
+
+ printf("%s -> %s/%s\n",
+ sha1_to_hex(sha1), ref_temp, sha1_to_hex(sha1));
+
+ path = mkpath("%s/%s", ref_temp, sha1_to_hex(sha1));
+ rl = xmalloc(sizeof *rl);
+ rl->force_write = 1;
+ rl->lk = &lk;
+ rl->ref_name = xstrdup(sha1_to_hex(sha1));
+ rl->orig_ref_name = xstrdup(rl->ref_name);
+ rl->lock_fd = hold_lock_file_for_update(rl->lk, path, 1);
+ if (write_ref_sha1(rl, sha1, NULL) < 0)
+ die("failed to write temporary ref %s", lk.filename);
+
+ return 0;
+}
+
+static char *
+setup_reference(const char *repo)
+{
+ struct stat buf;
+ const char *ref_git;
+ char *ref_temp;
+
+ if (!option_reference)
+ return NULL;
+
+ ref_git = make_absolute_path(option_reference);
+
+ if (!stat(mkpath("%s/.git/objects", ref_git), &buf) &&
+ S_ISDIR(buf.st_mode))
+ ref_git = mkpath("%s/.git", ref_git);
+ else if (stat(mkpath("%s/objects", ref_git), &buf) ||
+ !S_ISDIR(buf.st_mode))
+ die("reference repository '%s' is not a local directory.",
+ option_reference);
+
+ set_git_dir(ref_git);
+
+ write_alternates_file(repo, ref_git);
+
+ ref_temp = xstrdup(mkpath("%s/refs/reference-tmp", repo));
+ if (mkdir(ref_temp, 0777))
+ die("could not create directory %s", ref_temp);
+ for_each_ref(setup_tmp_ref, (void *) ref_temp);
+
+ return ref_temp;
+}
+
+static void
+cleanup_reference(char *ref_temp)
+{
+ struct dirent *de;
+ DIR *dir;
+
+ if (!ref_temp)
+ return;
+ dir = opendir(ref_temp);
+ if (!dir) {
+ if (errno == ENOENT)
+ return;
+ die("failed to open directory %s", ref_temp);
+ }
+
+ while ((de = readdir(dir)) != NULL) {
+ if (de->d_name[0] == '.')
+ continue;
+ unlink(mkpath("%s/%s", ref_temp, de->d_name));
+ }
+
+ unlink(ref_temp);
+ free(ref_temp);
+}
+
+static void
+walk_objects(char *src, char *dest)
+{
+ struct dirent *de;
+ struct stat buf;
+ int src_len, dest_len;
+ DIR *dir;
+
+ dir = opendir(src);
+ if (!dir)
+ die("failed to open %s\n", src);
+
+ if (mkdir(dest, 0777)) {
+ if (errno != EEXIST)
+ die("failed to create directory %s\n", dest);
+ else if (stat(dest, &buf))
+ die("failed to stat %s\n", dest);
+ else if (!S_ISDIR(buf.st_mode))
+ die("%s exists and is not a directory\n", dest);
+ }
+
+ src_len = strlen(src);
+ src[src_len] = '/';
+ dest_len = strlen(dest);
+ dest[dest_len] = '/';
+
+ while ((de = readdir(dir)) != NULL) {
+ strcpy(src + src_len + 1, de->d_name);
+ strcpy(dest + dest_len + 1, de->d_name);
+ if (stat(src, &buf)) {
+ fprintf(stderr, "failed to stat %s, ignoring\n", src);
+ continue;
+ }
+ if (S_ISDIR(buf.st_mode)) {
+ if (de->d_name[0] != '.')
+ walk_objects(src, dest);
+ continue;
+ }
+
+ if (unlink(dest) && errno != ENOENT)
+ die("failed to unlink %s\n", dest);
+ if (option_no_hardlinks) {
+ if (copy_file(dest, src, 0666))
+ die("failed to copy file to %s\n", dest);
+ } else {
+ if (link(src, dest))
+ die("failed to create link %s\n", dest);
+ }
+ }
+}
+
+static struct ref *
+clone_local(const char *src_repo, const char *dest_repo)
+{
+ char src[PATH_MAX];
+ char dest[PATH_MAX];
+
+ if (option_shared) {
+ write_alternates_file(dest_repo, src_repo);
+ } else {
+ snprintf(src, PATH_MAX, "%s/objects", src_repo);
+ snprintf(dest, PATH_MAX, "%s/objects", dest_repo);
+ walk_objects(src, dest);
+ }
+
+ /* FIXME: Return list of refs for src repo. */
+
+ return NULL;
+}
+
+int cmd_clone(int argc, const char **argv, const char *prefix)
+{
+ int use_local_hardlinks = 1;
+ int use_separate_remote = 1;
+ struct stat buf;
+ const char *repo, *work_tree, *git_dir;
+ char *path, *dir, *head, *ref_temp;
+ struct ref *refs, *r, *remote_head, *head_points_at;
+ char branch_top[256], key[256], refname[256], value[256];
+
+ argc = parse_options(argc, argv, builtin_clone_options,
+ builtin_clone_usage, 0);
+
+ if (argc == 0)
+ die("You must specify a repository to clone.");
+
+ if (option_no_hardlinks)
+ use_local_hardlinks = 0;
+
+ if (option_bare) {
+ if (option_origin)
+ die("--bare and --origin %s options are incompatible.",
+ option_origin);
+ option_no_checkout = 1;
+ use_separate_remote = 0;
+ }
+
+ repo = argv[0];
+ path = get_repo_path(repo);
+
+ if (argc == 2) {
+ dir = xstrdup(argv[1]);
+ } else {
+ dir = guess_dir_name(repo);
+ }
+
+ if (!stat(dir, &buf))
+ die("destination directory '%s' already exists.", dir);
+
+ if (option_bare)
+ work_tree = NULL;
+ else {
+ work_tree = getenv("GIT_WORK_TREE");
+ if (work_tree && !stat(work_tree, &buf))
+ die("working tree '%s' already exists.", work_tree);
+ }
+
+ if (mkdir(dir, 0755))
+ die("could not create repository dir '%s'.", dir);
+ if (work_tree && mkdir(work_tree, 0755))
+ die("could not create work tree dir '%s'.", work_tree);
+
+ if (option_bare || work_tree)
+ git_dir = xstrdup(dir);
+ else
+ git_dir = xstrdup(mkpath("%s/.git", dir));
+
+ init_db(git_dir, option_template, option_quiet ? INIT_DB_QUIET : 0);
+
+ /* This calls set_git_dir for the reference repo so we can get
+ * the refs there. Thus, call this before calling
+ * set_git_dir() on the repo we're setting up. */
+ ref_temp = setup_reference(git_dir);
+
+ set_git_dir(make_absolute_path(git_dir));
+
+ if (option_bare)
+ git_config_set("core.bare", "true");
+
+ if (path != NULL) {
+ refs = clone_local(path, git_dir);
+ } else {
+ struct fetch_pack_args args;
+
+ args.uploadpack = option_upload_pack;
+ args.quiet = option_quiet;
+ args.fetch_all = 1;
+ args.lock_pack = 0;
+ args.keep_pack = 1;
+ args.depth = option_depth;
+ args.no_progress = 1;
+
+ refs = fetch_pack(&args, argv[0], 0, NULL, NULL);
+ }
+
+ cleanup_reference(ref_temp);
+
+ if (option_bare)
+ strcpy(branch_top, "heads");
+ else
+ snprintf(branch_top, sizeof branch_top,
+ "refs/remotes/%s", option_origin);
+
+ remote_head = NULL;
+ for (r = refs; r; r = r->next) {
+ if (!strcmp(r->name, "HEAD")) {
+ remote_head = r;
+ continue;
+ }
+
+ if (!prefixcmp(r->name, "refs/heads/"))
+ snprintf(refname, sizeof refname,
+ "%s/%s", branch_top, r->name + 11);
+ else if (!prefixcmp(r->name, "refs/tags/"))
+ snprintf(refname, sizeof refname,
+ "refs/tags/%s", r->name + 10);
+ else
+ continue;
+
+ update_ref("clone from $repo",
+ refname, r->old_sha1, NULL, 0, DIE_ON_ERR);
+ }
+
+ if (option_bare)
+ return 0;
+
+ /* Is HEAD always first? If so, we could do this in the loop above. */
+ head_points_at = NULL;
+ for (r = refs; r; r = r->next) {
+ if (r != remote_head &&
+ !hashcmp(r->old_sha1, remote_head->old_sha1)) {
+ head_points_at = r;
+ printf("head points at %s\n", r->name);
+ break;
+ }
+ }
+
+ if (strrchr(head_points_at->name, '/'))
+ head = strrchr(head_points_at->name, '/') + 1;
+ else
+ head = head_points_at->name;
+
+ /* FIXME: What about the "Uh-oh, the remote told us..." case? */
+
+ snprintf(key, sizeof key, "remote.%s.remote", option_origin);
+ git_config_set(key, repo);
+ snprintf(key, sizeof key, "remote.%s.fetch", option_origin);
+ snprintf(value, sizeof value, "+refs/heads/*:%s/*", branch_top);
+
+ git_config_set_multivar(key, value, "^$", 0);
+
+ if (head_points_at) {
+ /* Local default branch */
+ create_symref("HEAD", head_points_at->name, NULL);
+
+ /* Tracking branch for the primary branch at the remote. */
+ update_ref(NULL, "HEAD", head_points_at->old_sha1,
+ NULL, 0, DIE_ON_ERR);
+ /*
+ rm -f "refs/remotes/$origin/HEAD"
+ git symbolic-ref "refs/remotes/$origin/HEAD" \
+ "refs/remotes/$origin/$head_points_at" &&
+ */
+
+ snprintf(key, sizeof key, "branch.%s.remote", head);
+ git_config_set(key, option_origin);
+ snprintf(key, sizeof key, "branch.%s.merge", head);
+ git_config_set(key, head_points_at->name);
+ } else {
+ /* Source had detached HEAD pointing nowhere. */
+ update_ref("clone from $repo", "HEAD", remote_head->old_sha1,
+ NULL, REF_NODEREF, DIE_ON_ERR);
+ }
+
+ if (!option_no_checkout) {
+ char base_dir[PATH_MAX];
+ struct lock_file lock_file;
+ struct unpack_trees_options opts;
+ struct tree *tree;
+ struct tree_desc t[2];
+ int fd;
+
+ fd = hold_locked_index(&lock_file, 1);
+
+ memset(&opts, 0, sizeof opts);
+ opts.update = 1;
+ opts.verbose_update = !option_quiet;
+ opts.merge = 1;
+ opts.fn = twoway_merge;
+
+ /* FIXME: Handle basedir ends in '/' */
+ snprintf(base_dir, sizeof base_dir, "%s/",
+ work_tree ? work_tree : dir);
+ opts.base_dir = base_dir;
+
+ tree = parse_tree_indirect(remote_head->old_sha1);
+ parse_tree(tree);
+ init_tree_desc(&t[0], tree->buffer, tree->size);
+ init_tree_desc(&t[1], tree->buffer, tree->size);
+ unpack_trees(2, t, &opts);
+
+ if (write_cache(fd, active_cache, active_nr) ||
+ close(fd) || commit_locked_index(&lock_file))
+ die("unable to write new index file");
+ }
+
+ return 0;
+}
diff --git a/builtin-init-db.c b/builtin-init-db.c
index e1393b8..18abc43 100644
--- a/builtin-init-db.c
+++ b/builtin-init-db.c
@@ -29,27 +29,6 @@ static void safe_create_dir(const char *dir, int share)
die("Could not make %s writable by group\n", dir);
}
-static int copy_file(const char *dst, const char *src, int mode)
-{
- int fdi, fdo, status;
-
- mode = (mode & 0111) ? 0777 : 0666;
- if ((fdi = open(src, O_RDONLY)) < 0)
- return fdi;
- if ((fdo = open(dst, O_WRONLY | O_CREAT | O_EXCL, mode)) < 0) {
- close(fdi);
- return fdo;
- }
- status = copy_fd(fdi, fdo);
- if (close(fdo) != 0)
- return error("%s: write error: %s", dst, strerror(errno));
-
- if (!status && adjust_shared_perm(dst))
- return -1;
-
- return status;
-}
-
static void copy_templates_1(char *path, int baselen,
char *template, int template_baselen,
DIR *dir)
@@ -330,49 +309,11 @@ static void guess_repository_type(const char *git_dir)
return;
}
-static const char init_db_usage[] =
-"git-init [-q | --quiet] [--template=<template-directory>] [--shared]";
-
-/*
- * If you want to, you can share the DB area with any number of branches.
- * That has advantages: you can save space by sharing all the SHA1 objects.
- * On the other hand, it might just make lookup slower and messier. You
- * be the judge. The default case is to have one DB per managed directory.
- */
-int cmd_init_db(int argc, const char **argv, const char *prefix)
+int init_db(const char *git_dir, const char *template_dir, unsigned int flags)
{
- const char *git_dir;
const char *sha1_dir;
- const char *template_dir = NULL;
char *path;
- int len, i, reinit;
- int quiet = 0;
-
- for (i = 1; i < argc; i++, argv++) {
- const char *arg = argv[1];
- if (!prefixcmp(arg, "--template="))
- template_dir = arg+11;
- else if (!strcmp(arg, "--shared"))
- shared_repository = PERM_GROUP;
- else if (!prefixcmp(arg, "--shared="))
- shared_repository = git_config_perm("arg", arg+9);
- else if (!strcmp(arg, "-q") || !strcmp(arg, "--quiet"))
- quiet = 1;
- else
- usage(init_db_usage);
- }
-
- /*
- * GIT_WORK_TREE makes sense only in conjunction with GIT_DIR
- * without --bare. Catch the error early.
- */
- git_dir = getenv(GIT_DIR_ENVIRONMENT);
- if ((!git_dir || is_bare_repository_cfg == 1)
- && getenv(GIT_WORK_TREE_ENVIRONMENT))
- die("%s (or --work-tree=<directory>) not allowed without "
- "specifying %s (or --git-dir=<directory>)",
- GIT_WORK_TREE_ENVIRONMENT,
- GIT_DIR_ENVIRONMENT);
+ int len, reinit;
guess_repository_type(git_dir);
@@ -388,7 +329,6 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)
/*
* Set up the default .git directory contents
*/
- git_dir = getenv(GIT_DIR_ENVIRONMENT);
if (!git_dir)
git_dir = DEFAULT_GIT_DIR_ENVIRONMENT;
safe_create_dir(git_dir, 0);
@@ -403,9 +343,13 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)
reinit = create_default_files(git_dir, template_dir);
/*
- * And set up the object store.
+ * And set up the object store. Don't use
+ * get_object_directory() here, since we're initializing
+ * relative to git_dir, not $GIT_DIR.
*/
- sha1_dir = get_object_directory();
+ sha1_dir = getenv(DB_ENVIRONMENT);
+ if (!sha1_dir)
+ sha1_dir = mkpath("%s/objects", git_dir);
len = strlen(sha1_dir);
path = xmalloc(len + 40);
memcpy(path, sha1_dir, len);
@@ -427,7 +371,7 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)
git_config_set("receive.denyNonFastforwards", "true");
}
- if (!quiet)
+ if (!(flags & INIT_DB_QUIET))
printf("%s%s Git repository in %s/\n",
reinit ? "Reinitialized existing" : "Initialized empty",
shared_repository ? " shared" : "",
@@ -435,3 +379,48 @@ int cmd_init_db(int argc, const char **argv, const char *prefix)
return 0;
}
+
+static const char init_db_usage[] =
+"git-init [-q | --quiet] [--template=<template-directory>] [--shared]";
+
+/*
+ * If you want to, you can share the DB area with any number of branches.
+ * That has advantages: you can save space by sharing all the SHA1 objects.
+ * On the other hand, it might just make lookup slower and messier. You
+ * be the judge. The default case is to have one DB per managed directory.
+ */
+int cmd_init_db(int argc, const char **argv, const char *prefix)
+{
+ const char *git_dir;
+ const char *template_dir = NULL;
+ unsigned int flags = 0;
+ int i;
+
+ for (i = 1; i < argc; i++, argv++) {
+ const char *arg = argv[1];
+ if (!prefixcmp(arg, "--template="))
+ template_dir = arg+11;
+ else if (!strcmp(arg, "--shared"))
+ shared_repository = PERM_GROUP;
+ else if (!prefixcmp(arg, "--shared="))
+ shared_repository = git_config_perm("arg", arg+9);
+ else if (!strcmp(arg, "-q") || !strcmp(arg, "--quiet"))
+ flags |= INIT_DB_QUIET;
+ else
+ usage(init_db_usage);
+ }
+
+ /*
+ * GIT_WORK_TREE makes sense only in conjunction with GIT_DIR
+ * without --bare. Catch the error early.
+ */
+ git_dir = getenv(GIT_DIR_ENVIRONMENT);
+ if ((!git_dir || is_bare_repository_cfg == 1)
+ && getenv(GIT_WORK_TREE_ENVIRONMENT))
+ die("%s (or --work-tree=<directory>) not allowed without "
+ "specifying %s (or --git-dir=<directory>)",
+ GIT_WORK_TREE_ENVIRONMENT,
+ GIT_DIR_ENVIRONMENT);
+
+ return init_db(git_dir, template_dir, flags);
+}
diff --git a/builtin-rerere.c b/builtin-rerere.c
index 7449323..2d83524 100644
--- a/builtin-rerere.c
+++ b/builtin-rerere.c
@@ -267,23 +267,6 @@ static int diff_two(const char *file1, const char *label1,
return 0;
}
-static int copy_file(const char *src, const char *dest)
-{
- FILE *in, *out;
- char buffer[32768];
- int count;
-
- if (!(in = fopen(src, "r")))
- return error("Could not open %s", src);
- if (!(out = fopen(dest, "w")))
- return error("Could not open %s", dest);
- while ((count = fread(buffer, 1, sizeof(buffer), in)))
- fwrite(buffer, 1, count, out);
- fclose(in);
- fclose(out);
- return 0;
-}
-
static int do_plain_rerere(struct path_list *rr, int fd)
{
struct path_list conflict = { NULL, 0, 0, 1 };
@@ -343,7 +326,7 @@ static int do_plain_rerere(struct path_list *rr, int fd)
continue;
fprintf(stderr, "Recorded resolution for '%s'.\n", path);
- copy_file(path, rr_path(name, "postimage"));
+ copy_file(path, rr_path(name, "postimage"), 0666);
tail_optimization:
if (i < rr->nr - 1)
memmove(rr->items + i,
diff --git a/builtin.h b/builtin.h
index cb675c4..1b9da64 100644
--- a/builtin.h
+++ b/builtin.h
@@ -24,6 +24,7 @@ extern int cmd_check_attr(int argc, const char **argv, const char *prefix);
extern int cmd_check_ref_format(int argc, const char **argv, const char *prefix);
extern int cmd_cherry(int argc, const char **argv, const char *prefix);
extern int cmd_cherry_pick(int argc, const char **argv, const char *prefix);
+extern int cmd_clone(int argc, const char **argv, const char *prefix);
extern int cmd_clean(int argc, const char **argv, const char *prefix);
extern int cmd_commit(int argc, const char **argv, const char *prefix);
extern int cmd_commit_tree(int argc, const char **argv, const char *prefix);
diff --git a/cache.h b/cache.h
index 4e59646..1e29e70 100644
--- a/cache.h
+++ b/cache.h
@@ -230,6 +230,10 @@ extern const char *prefix_filename(const char *prefix, int len, const char *path
extern void verify_filename(const char *prefix, const char *name);
extern void verify_non_filename(const char *prefix, const char *name);
+#define INIT_DB_QUIET 0x0001
+
+extern int init_db(const char *git_dir, const char *template_dir, unsigned int flags);
+
#define alloc_nr(x) (((x)+16)*3/2)
/*
@@ -588,6 +592,7 @@ extern const char *git_log_output_encoding;
/* IO helper functions */
extern void maybe_flush_or_die(FILE *, const char *);
extern int copy_fd(int ifd, int ofd);
+extern int copy_file(const char *dst, const char *src, int mode);
extern int read_in_full(int fd, void *buf, size_t count);
extern int write_in_full(int fd, const void *buf, size_t count);
extern void write_or_die(int fd, const void *buf, size_t count);
diff --git a/git-clone.sh b/contrib/examples/git-clone.sh
similarity index 100%
rename from git-clone.sh
rename to contrib/examples/git-clone.sh
diff --git a/copy.c b/copy.c
index c225d1b..afc4fbf 100644
--- a/copy.c
+++ b/copy.c
@@ -34,3 +34,24 @@ int copy_fd(int ifd, int ofd)
close(ifd);
return 0;
}
+
+int copy_file(const char *dst, const char *src, int mode)
+{
+ int fdi, fdo, status;
+
+ mode = (mode & 0111) ? 0777 : 0666;
+ if ((fdi = open(src, O_RDONLY)) < 0)
+ return fdi;
+ if ((fdo = open(dst, O_WRONLY | O_CREAT | O_EXCL, mode)) < 0) {
+ close(fdi);
+ return fdo;
+ }
+ status = copy_fd(fdi, fdo);
+ if (close(fdo) != 0)
+ return error("%s: write error: %s", dst, strerror(errno));
+
+ if (!status && adjust_shared_perm(dst))
+ return -1;
+
+ return status;
+}
diff --git a/diff.c b/diff.c
index 5175950..0af5b81 100644
--- a/diff.c
+++ b/diff.c
@@ -258,8 +258,8 @@ static void print_line_count(int count)
}
}
-static void copy_file(int prefix, const char *data, int size,
- const char *set, const char *reset)
+static void copy_file_with_prefix(int prefix, const char *data, int size,
+ const char *set, const char *reset)
{
int ch, nl_just_seen = 1;
while (0 < size--) {
@@ -310,9 +310,9 @@ static void emit_rewrite_diff(const char *name_a,
print_line_count(lc_b);
printf(" @@%s\n", reset);
if (lc_a)
- copy_file('-', one->data, one->size, old, reset);
+ copy_file_with_prefix('-', one->data, one->size, old, reset);
if (lc_b)
- copy_file('+', two->data, two->size, new, reset);
+ copy_file_with_prefix('+', two->data, two->size, new, reset);
}
static int fill_mmfile(mmfile_t *mf, struct diff_filespec *one)
diff --git a/git.c b/git.c
index f406c4b..c8dfb6d 100644
--- a/git.c
+++ b/git.c
@@ -302,6 +302,7 @@ static void handle_internal_command(int argc, const char **argv)
{ "check-attr", cmd_check_attr, RUN_SETUP | NEED_WORK_TREE },
{ "cherry", cmd_cherry, RUN_SETUP },
{ "cherry-pick", cmd_cherry_pick, RUN_SETUP | NEED_WORK_TREE },
+ { "clone", cmd_clone },
{ "clean", cmd_clean, RUN_SETUP | NEED_WORK_TREE },
{ "commit", cmd_commit, RUN_SETUP | NEED_WORK_TREE },
{ "commit-tree", cmd_commit_tree, RUN_SETUP },
diff --git a/unpack-trees.c b/unpack-trees.c
index e9eb795..752278a 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -338,7 +338,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options
memset(&df_conflict_list, 0, sizeof(df_conflict_list));
df_conflict_list.next = &df_conflict_list;
memset(&state, 0, sizeof(state));
- state.base_dir = "";
+ state.base_dir = o->base_dir ? o->base_dir : "";
+ state.base_dir_len = strlen(state.base_dir);
state.force = 1;
state.quiet = 1;
state.refresh_cache = 1;
diff --git a/unpack-trees.h b/unpack-trees.h
index 5517faa..15b2ed9 100644
--- a/unpack-trees.h
+++ b/unpack-trees.h
@@ -17,6 +17,7 @@ struct unpack_trees_options {
int verbose_update;
int aggressive;
const char *prefix;
+ const char *base_dir;
int pos;
struct dir_struct *dir;
merge_fn_t fn;
--
1.5.3.4
^ permalink raw reply related
* Re: git annotate runs out of memory
From: Jason Sewall @ 2007-12-11 19:27 UTC (permalink / raw)
To: Daniel Berlin; +Cc: Marco Costalba, git
In-Reply-To: <4aca3dc20712111103s1af3b045h484ea749378c6282@mail.gmail.com>
On Dec 11, 2007 2:03 PM, Daniel Berlin <dberlin@dberlin.org> wrote:
> It is stored in an easier format. However, can you not simply provide
> side-indexes to do the annotation?
>
> I guess that own't work in git because you can change history (in
> other scm's, history is readonly so you could know the results for
> committed revisions will never change).
>
I don't know how other scms work, but history is definitely readonly
in git - whatever sha1 you have that describes a commit was calculated
based on its ancestor commits.
If you have a commit's id, it will *always* refer to the same thing -
a tree state and its complete ancestry.
^ permalink raw reply
* Re: git annotate runs out of memory
From: Steven Grimm @ 2007-12-11 19:29 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Daniel Berlin, git
In-Reply-To: <alpine.LFD.0.9999.0712111018540.25032@woody.linux-foundation.org>
On Dec 11, 2007, at 10:40 AM, Linus Torvalds wrote:
> To git, "git annotate" is just about the *last* thing you ever want
> to do.
> It's not a common operation, it's a "last resort" operation. In git,
> the
> whole workflow is designed for "git log -p <pathnamepattern>" rather
> than
> annotate/blame.
My use of "git blame" is perhaps not typical, but I use it fairly
often when I'm looking at a part of my company's code base that I'm
not terribly familiar with. I've found it's the fastest way to figure
out who to go ask about a particular block of code that I think is
responsible for a bug, or more commonly, who to ask to review a change
I'm making.
"git log" is too coarse-grained to be useful for that purpose; it
usually doesn't tell me which of the 500 revisions to the file I'm
looking at introduced the actual line of code I want to change.
To me that really has nothing whatsoever to do with git workflow or
svn workflow; it happens well before I'm ready to do any kind of
integration or commit or even, sometimes, before I've made any changes
to any code at all.
Given infinite spare time, one of the things I'd be strongly tempted
to try to build would be some kind of blame cache. You could
theoretically make blame pretty much instantaneous by doing something
as simple as caching the per-line revision ID for each file in each
revision in a shadow repository (or a shadow branch in the main repo)
and keeping a map between shadow-repo revisions and real-repo ones. If
the cache was of the form "one SHA1 hash per line in the original
file" it would delta-compress pretty well. It'd be easy to update
incrementally since you only need to walk back in history until you
get to the most recently cached revision for each file, at which point
you use the cached value for all the lines that haven't changed.
Yeah, I know, code talks louder than words...
-Steve
^ permalink raw reply
* Re: [ANNOUNCE] ugit: a pyqt-based git gui // was: Re: If you would write git from scratch now, what would you change?
From: Marco Costalba @ 2007-12-11 19:33 UTC (permalink / raw)
To: Jason Sewall; +Cc: David, Andy Parkins, git
In-Reply-To: <31e9dd080712111114t2bbdba60m18b7d6210f3f9174@mail.gmail.com>
On Dec 11, 2007 8:14 PM, Jason Sewall <jasonsewall@gmail.com> wrote:
> On Dec 11, 2007 1:20 PM, Marco Costalba <mcostalba@gmail.com> wrote:
> ./configure --prefix=/usr fixed that for me (and I'm sure there's a
> way to tell python to look in /usr/local... too, but I can't be
> bothered with that.
>
./configure --prefix=$HOME/bin
(half) worked thanks.
> I re-installed without the prefix and that error disappeared, but now I get
> Traceback (most recent call last):
> File "/usr/local/bin/ugit", line 12, in <module>
> view = GitView (app.activeWindow())
> File "../py/views.py", line 15, in __init__
> File "default/ui/Window.py", line 43, in setupUi
> AttributeError: setLeftMargin
>
> I'm too busy to poke around and see why that's happening, but
> hopefully someone can.
>
I have only $HOME/bin in path so I manually moved main 4 files and
directory ugitlibs under bin/
$HOME/bin -> ugit ugit.py uit.pyc ugit.pyo
-> ugitlibs/ -> with remaining files
And it worked.
Marco
^ permalink raw reply
* Re: git annotate runs out of memory
From: Pierre Habouzit @ 2007-12-11 19:34 UTC (permalink / raw)
To: Daniel Berlin; +Cc: Linus Torvalds, git
In-Reply-To: <4aca3dc20712111109y5d74a292rf29be6308932393c@mail.gmail.com>
[-- Attachment #1: Type: text/plain, Size: 2787 bytes --]
On Tue, Dec 11, 2007 at 07:09:03PM +0000, Daniel Berlin wrote:
> On 12/11/07, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> >
> >
> > On Tue, 11 Dec 2007, Daniel Berlin wrote:
> > >
> > > This seems to be a common problem with git. It seems to use a lot of
> > > memory to perform common operations on the gcc repository (even though
> > > it is faster in some cases than hg).
> >
> > The thing is, git has a very different notion of "common operations" than
> > you do.
> >
> > To git, "git annotate" is just about the *last* thing you ever want to do.
> > It's not a common operation, it's a "last resort" operation. In git, the
> > whole workflow is designed for "git log -p <pathnamepattern>" rather than
> > annotate/blame.
> >
> I understand this, and completely agree with you.
> However, I cannot force GCC people to adopt completely new workflow in
> this regard.
> The changelog's are not useful enough (and we've had huge fights over
> this) to do git log -p and figure out the info we want.
> Looking through thousands of diffs to find the one that happened to
> your line is also pretty annoying.
If the question you want to answer is "what happened to that line"
then using git annotate is using a big hammer for no good reason.
git log -S'<put the content of the line here>' -- path/to/file.c
will give you the very same answer, pointing you to the changes that
added or removed that line directly. It's not a fast command either, but
it should be less resource hungry than annotate that has to do roughly
the same for all lines whereas you're interested in one only.
The direct plus here, is that git log output is incremental, so you have
answers about the first diffs quite quick, which let you examine the
first answers while the rest is still being computed.
Unlike git annotate, this also allow you to restrict the revisions
where it searches to a range where you know this happened, which makes
it almost instantaneous in most cases.
Of course, if the line is ' free(p);\n' then you will probably have
quite a few false positives, but with the path restriction, I assume
this will still be quite accurate.
What is important here is to know what is the real question the GCC
programmers want to answer to. It seems to me that `blame` is an
overkill for the underlying issue.
Note that it does not justifies the current memory consumption that just
looks bad and wrong to me, but this aims at finding a way to answer your
question doing just what you need to answer it and not gazillions of
other things :)
--
·O· Pierre Habouzit
··O madcoder@debian.org
OOO http://www.madism.org
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply
* Re: Something is broken in repack
From: Junio C Hamano @ 2007-12-11 19:40 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jon Smirl, Nicolas Pitre, gcc, Git Mailing List
In-Reply-To: <alpine.LFD.0.9999.0712111055590.25032@woody.linux-foundation.org>
Linus Torvalds <torvalds@linux-foundation.org> writes:
> On Tue, 11 Dec 2007, Jon Smirl wrote:
>> >
>> > So if you want to use more threads, that _forces_ you to have a bigger
>> > memory footprint, simply because you have more "live" objects that you
>> > work on. Normally, that isn't much of a problem, since most source files
>> > are small, but if you have a few deep delta chains on big files, both the
>> > delta chain itself is going to use memory (you may have limited the size
>> > of the cache, but it's still needed for the actual delta generation, so
>> > it's not like the memory usage went away).
>>
>> This makes sense. Those runs that blew up to 4.5GB were a combination
>> of this effect and fragmentation in the gcc allocator. Google
>> allocator appears to be much better at controlling fragmentation.
>
> Yes. I think we do have some case where we simply keep a lot of objects
> around, and if we are talking reasonably large deltas, we'll have the
> whole delta-chain in memory just to unpack one single object.
Eh, excuse me. unpack_delta_entry()
- first unpacks the base object (this goes recursive);
- uncompresses the delta;
- applies the delta to the base to obtain the target object;
- frees delta;
- frees (but allows it to be cached) the base object;
- returns the result
So no matter how deep a chain is, you keep only one delta at a time in
core, not whole delta-chain in core.
> So what you actually want to do is to just re-use already packed delta
> chains directly, which is what we normally do. But you are explicitly
> looking at the "--no-reuse-delta" (aka "git repack -f") case, which is why
> it then blows up.
While that does not explain, as Nico pointed out, the huge difference
between the two repack runs that have different starting pack, I would
say it is a fair thing to say. If you have a suboptimal pack (i.e. not
enough reusable deltas, as in the 2.1GB pack case), do run "repack -f",
but if you have a good pack (i.e. 300MB pack), don't.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox