* Re: Something is broken in repack
From: Morten Welinder @ 2007-12-10 20:16 UTC (permalink / raw)
To: Jon Smirl; +Cc: Nicolas Pitre, Git Mailing List
In-Reply-To: <9e4733910712101205q218152a2td14a8931e63d2610@mail.gmail.com>
> Here's another observation, the gcc objects are larger. Kernel has
> 650K objects in 190MB, gcc has 870K objects in 330MB. Average gcc
> object is 30% larger. How should the average kernel developer
> interpret this?
Could this be explained by the ChangeLog file? It's large; it has tons of
revisions; it is a prime candidate for delta compression.
Morten
^ permalink raw reply
* Re: Performance problem, long run of identical hashes
From: Nicolas Pitre @ 2007-12-10 20:11 UTC (permalink / raw)
To: David Kastrup; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <85tzmqnxua.fsf@lola.goethe.zz>
On Mon, 10 Dec 2007, David Kastrup wrote:
> Nicolas Pitre <nico@cam.org> writes:
>
> > On Mon, 10 Dec 2007, Jon Smirl wrote:
> >
> >> I added some debug printfs which show that I
> >> have a 100,000+ run of identical hash entries. Processing the 100,000
> >> entries also causes RAM consumption to explode.
> >
> > That is impossible. If you look at the code where those hash entries
> > are created in create_delta_index(), you'll notice a hard limit of
> > HASH_LIMIT (currently 64) is imposed on the number of identical hash
> > entries.
>
> On the other hand, if we have, say, laaaaaarge streaks of zeros, what
> happens is that we have 64 hashes seeing them. Now about 4096 bytes are
> compared, and then the comparison stops.
No.
What would happen in that case is that the first hash entry pointing
somewhere in the beginning of the zero chunk will match _at least_ 4096
bytes, probably much more, up to the end of the buffer if that is
possible.
> Then it scans backwards to
> seek for more zeros (and finds about 64k of them before it stops)
The first hash entry for those zeroes is going to point at an offset not
greater than 15 bytes from the start of the zero area. So, unless both
buffers are going to match even before that zero area, the backward scan
will stop quickly.
> folds them into the current compacted form. Each of these backward
> scans (of which we have 64 in the worst case) is in a different memory
> area. So since we scan/compare areas of 64k for each advance of 4k, we
> have an overscanning factor of 16 (for a worst case scenario).
Actually, if we matched 100MB of zeroes, we'll just encode that in a
suite of 64KB copy operations, and continue scanning the buffer after
that 100MB.
So I don't see where your scan/compare areas of 64k for each advance of
4k comes from.
> Not sure whether this is what we are seeing here. It would still not
> explain exploding memory usage I think.
Right.
Nicolas
^ permalink raw reply
* Re: Something is broken in repack
From: Jon Smirl @ 2007-12-10 20:05 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Git Mailing List
In-Reply-To: <alpine.LFD.0.99999.0712101434560.555@xanadu.home>
On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Fri, 7 Dec 2007, Jon Smirl wrote:
>
> > Using this config:
> > [pack]
> > threads = 4
> > deltacachesize = 256M
> > deltacachelimit = 0
> >
> > And the 330MB gcc pack for input
> > git repack -a -d -f --depth=250 --window=250
> >
> > complete seconds RAM
> > 10% 47 1GB
> > 20% 29 1Gb
> > 30% 24 1Gb
> > 40% 18 1GB
> > 50% 110 1.2GB
> > 60% 85 1.4GB
> > 70% 195 1.5GB
> > 80% 186 2.5GB
> > 90% 489 3.8GB
> > 95% 800 4.8GB
> > I killed it because it started swapping
> >
> > The mmaps are only about 400MB in this case.
> > At the end the git process had 4.4GB of physical RAM allocated.
> >
> > Starting from a highly compressed pack greatly aggravates the problem.
> > Starting with a 2GB pack of the same data my process size only grew to
> > 3GB with 2GB of mmaps.
>
> You said having reproduced the issue, albeit not as severe, with the
> Linux kernel repo. I did just that:
>
> # to get the default pack:
> $ git repack -a -f -d
>
> # first measurement with a repack from a default pack
> $ /usr/bin/time git repack -a -f --window=256 --depth=256
> 2572.17user 5.87system 22:46.80elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k
> 15720inputs+356640outputs (71major+264376minor)pagefaults 0swaps
>
> # do it again to start from a highly packed pack
> $ /usr/bin/time git repack -a -f --window=256 --depth=256
> 2573.53user 5.62system 22:45.60elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k
> 29176inputs+356664outputs (210major+274887minor)pagefaults 0swaps
>
> This is with pack.threads=2 on a P4 with HT, and I'm using the machine
> for other tasks as well, but all measured time is sensibly the same for
> both cases. Virtual memory allocation never reached 700MB in both cases
> either.
>
This is the mail about the kernel pack, the one you quoted is a gcc run.
The kernel repo has the same problem but not nearly as bad.
Starting from a default pack
git repack -a -d -f --depth=1000 --window=1000
Uses 1GB of physical memory
Now do the command again.
git repack -a -d -f --depth=1000 --window=1000
Uses 1.3GB of physical memory
I suspect the gcc repo has much longer revision chains than the kernel
one since the kernel repo is only a few years old. The Mozilla repo
contained revision chains with over 2,000 revisions. Longer revision
chains result in longer delta chains.
So what is allocating the extra memory? Either a function of the
number of entries in the chain, or related to accessing the chain
since a chain with more entries will need to be accessed more times.
I have a 168MB kernel pack now after 15 minutes of four cores at 100%.
Here's another observation, the gcc objects are larger. Kernel has
650K objects in 190MB, gcc has 870K objects in 330MB. Average gcc
object is 30% larger. How should the average kernel developer
interpret this?
>
> Nicolas
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Something is broken in repack
From: Nicolas Pitre @ 2007-12-10 19:56 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910712071505y6834f040k37261d65a2d445c4@mail.gmail.com>
On Fri, 7 Dec 2007, Jon Smirl wrote:
> Using this config:
> [pack]
> threads = 4
> deltacachesize = 256M
> deltacachelimit = 0
>
> And the 330MB gcc pack for input
> git repack -a -d -f --depth=250 --window=250
>
> complete seconds RAM
> 10% 47 1GB
> 20% 29 1Gb
> 30% 24 1Gb
> 40% 18 1GB
> 50% 110 1.2GB
> 60% 85 1.4GB
> 70% 195 1.5GB
> 80% 186 2.5GB
> 90% 489 3.8GB
> 95% 800 4.8GB
> I killed it because it started swapping
>
> The mmaps are only about 400MB in this case.
> At the end the git process had 4.4GB of physical RAM allocated.
>
> Starting from a highly compressed pack greatly aggravates the problem.
> Starting with a 2GB pack of the same data my process size only grew to
> 3GB with 2GB of mmaps.
You said having reproduced the issue, albeit not as severe, with the
Linux kernel repo. I did just that:
# to get the default pack:
$ git repack -a -f -d
# first measurement with a repack from a default pack
$ /usr/bin/time git repack -a -f --window=256 --depth=256
2572.17user 5.87system 22:46.80elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k
15720inputs+356640outputs (71major+264376minor)pagefaults 0swaps
# do it again to start from a highly packed pack
$ /usr/bin/time git repack -a -f --window=256 --depth=256
2573.53user 5.62system 22:45.60elapsed 188%CPU (0avgtext+0avgdata 0maxresident)k
29176inputs+356664outputs (210major+274887minor)pagefaults 0swaps
This is with pack.threads=2 on a P4 with HT, and I'm using the machine
for other tasks as well, but all measured time is sensibly the same for
both cases. Virtual memory allocation never reached 700MB in both cases
either.
Nicolas
^ permalink raw reply
* Re: Performance problem, long run of identical hashes
From: David Kastrup @ 2007-12-10 19:39 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <alpine.LFD.0.99999.0712101037270.555@xanadu.home>
Nicolas Pitre <nico@cam.org> writes:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
>> Running oprofile during my gcc repack shows this loop as the hottest
>> place in the code by far.
>
> Well, that is kind of expected.
>
>> I added some debug printfs which show that I
>> have a 100,000+ run of identical hash entries. Processing the 100,000
>> entries also causes RAM consumption to explode.
>
> That is impossible. If you look at the code where those hash entries
> are created in create_delta_index(), you'll notice a hard limit of
> HASH_LIMIT (currently 64) is imposed on the number of identical hash
> entries.
On the other hand, if we have, say, laaaaaarge streaks of zeros, what
happens is that we have 64 hashes seeing them. Now about 4096 bytes are
compared, and then the comparison stops. Then it scans backwards to
seek for more zeros (and finds about 64k of them before it stops) and
folds them into the current compacted form. Each of these backward
scans (of which we have 64 in the worst case) is in a different memory
area. So since we scan/compare areas of 64k for each advance of 4k, we
have an overscanning factor of 16 (for a worst case scenario).
Not sure whether this is what we are seeing here. It would still not
explain exploding memory usage I think.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
^ permalink raw reply
* Re: [PATCH] Don't cache DESTDIR in perl/perl.mak.
From: Junio C Hamano @ 2007-12-10 19:24 UTC (permalink / raw)
To: Gerrit Pape; +Cc: git, Eric Wong
In-Reply-To: <20071210093102.3050.qmail@06e91d20307a62.315fe32.mid.smarden.org>
Hmph. That's reverting this:
commit 4c5cf8c44ce06a79da5bafd4a92e6d6f598cea2e
Author: Eric Wong <normalperson@yhbt.net>
Date: Sun Aug 13 04:13:25 2006 -0700
pass DESTDIR to the generated perl/Makefile
Makes life for binary packagers easier, as the Perl modules will
be installed inside DESTDIR.
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Signed-off-by: Junio C Hamano <junkio@cox.net>
Eric, care to comment?
^ permalink raw reply
* Re: [PATCH] Better errors when trying to merge a submodule
From: Junio C Hamano @ 2007-12-10 19:22 UTC (permalink / raw)
To: Finn Arne Gangstad; +Cc: git
In-Reply-To: <20071210124435.GA4788@pvv.org>
Finn Arne Gangstad <finnag@pvv.org> writes:
> Instead of dying with weird errors when trying to merge submodules from a
> supermodule, emit errors that show what the problem is.
Thanks.
Your change to merge-one-file.sh is Ok, although I'd reword the message
a bit, and fold it as a new case arm to the existing case statement
immediately above.
Your change to merge-recursive is not quite right, although you spotted
correctly what codepath needs to be fixed. merge_file() should not die
in such a case but set result.clean to 0 to signal that the result has
conflicts and cannot be merged, pick the sha1 from the current tree
(i.e. side A), and let the caller deal with the conflict. If you die
there, the user cannot resolve a merge if this happens while building a
virtual ancestor commit during a recursive merge of two crisscrossing
histories.
Perhaps something like this...
-- >8 --
Support a merge with conflicting gitlink change
merge-recursive did not support merging trees that have conflicting
changes in submodules they contain, and died. Support it exactly the
same way as how it handles conflicting symbolic link changes --- mark it
as a conflict, take the tentative result from the current side, and
letting the caller resolve the conflict, without dying in merge_file()
function.
Also reword the error message issued when merge_file() has to die
because it sees a tree entry of type it does not support yet.
---
git-merge-one-file.sh | 4 ++++
merge-recursive.c | 10 ++++++----
2 files changed, 10 insertions(+), 4 deletions(-)
diff --git a/git-merge-one-file.sh b/git-merge-one-file.sh
index 1e7727d..9ee3f80 100755
--- a/git-merge-one-file.sh
+++ b/git-merge-one-file.sh
@@ -80,6 +80,10 @@ case "${1:-.}${2:-.}${3:-.}" in
echo "ERROR: $4: Not merging symbolic link changes."
exit 1
;;
+ *,160000,*)
+ echo "ERROR: $4: Not merging conflicting submodule changes."
+ exit 1
+ ;;
esac
src2=`git-unpack-file $3`
diff --git a/merge-recursive.c b/merge-recursive.c
index 9a1e2f2..2a58dad 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1046,14 +1046,16 @@ static struct merge_file_info merge_file(struct diff_filespec *o,
free(result_buf.ptr);
result.clean = (merge_status == 0);
- } else {
- if (!(S_ISLNK(a->mode) || S_ISLNK(b->mode)))
- die("cannot merge modes?");
-
+ } else if (S_ISGITLINK(a->mode)) {
+ result.clean = 0;
+ hashcpy(result.sha, a->sha1);
+ } else if (S_ISLNK(a->mode)) {
hashcpy(result.sha, a->sha1);
if (!sha_eq(a->sha1, b->sha1))
result.clean = 0;
+ } else {
+ die("unsupported object type in the tree");
}
}
^ permalink raw reply related
* [PATCH] pack-objects: more threaded load balancing fix with often changed paths
From: Nicolas Pitre @ 2007-12-10 19:19 UTC (permalink / raw)
To: Junio C Hamano, Jon Smirl; +Cc: git
In-Reply-To: <alpine.LFD.0.99999.0712101308150.555@xanadu.home>
The code that splits the object list amongst work threads tries to do so
on "path" boundaries not to prevent good delta matches. However, in
some cases, a few paths may largely dominate the hash distribution and
it is not possible to have good load balancing without ignoring those
boundaries.
Signed-off-by: Nicolas Pitre <nico@cam.org>
---
On Mon, 10 Dec 2007, Nicolas Pitre wrote:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
> > On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> >
> > > The hash in the code above has to do with the file names the
> > > corresponding objects are coming from.
> >
> > So can we change this loop to exit after a max of window_size * 10 or
> > something like that iterations? Without capping it the threads become
> > way unbalanced in the end. In the gcc case one thread is continuing
> > 30+ minutes past the others exiting.
>
> Indeed, some more tweaking are needed.
>
> The object path distribution goes like this for the gcc repo:
>
> 105557 gcc
> 42537 gcc/ChangeLog
> 25210 gcc/config
> 20690 gcc/testsuite
> 13434 gcc/testsuite/ChangeLog
> 12363 libstdc++-v3
> 9346 gcc/cp
> 8757 libstdc++-v3/include
> 8186 gcc/version.c
> 7867 gcc/cp/ChangeLog
> 7737 libstdc++-v3/include/bits
> 7653 libjava
> 6577 gcc/testsuite/gcc.dg
> 5942 libjava/ChangeLog
> 5351 gcc/config/i386
> 5260 gcc/testsuite/g++.dg
> 4451 gcc/f
> 4330 libstdc++-v3/ChangeLog
> 4321 libstdc++-v3/include/bits/c++config
> 4316 gcc/doc
> [...]
>
> So... the top entries are most certainly going to create load balancing
> issues if their path hash clustering isn't broken.
Here's the fix.
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 250dc56..7dd0d7f 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -1709,6 +1709,16 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,
list++;
sub_size--;
}
+ if (!sub_size) {
+ /*
+ * It is possible for some "paths" to have
+ * so many objects that no hash boundary
+ * might be found. Let's just steal the
+ * exact half in that case.
+ */
+ sub_size = victim->remaining / 2;
+ list -= sub_size;
+ }
target->list = list;
victim->list_size -= sub_size;
victim->remaining -= sub_size;
^ permalink raw reply related
* Re: There is not summary of kernel 2.4.x repo.
From: J.H. @ 2007-12-10 17:57 UTC (permalink / raw)
To: Andreas Ericsson; +Cc: J.C. Pizarro, Linus Torvalds, git
In-Reply-To: <475D39A3.1040401@op5.se>
To further this the Linus Torvalds 2.4 tree does not actually exist (at
least not where your looking and not in my quick glance on the server),
so gitweb not showing you anything is a bit expected.
- John 'Warthog9' Hawley
On Mon, 2007-12-10 at 14:05 +0100, Andreas Ericsson wrote:
> J.C. Pizarro wrote:
> > It exists http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=summary
> > but http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.4.git;a=summary
> > is empty. :)
>
> That's what you get for hand-hacking URL's when there's a list to choose
> from.
>
> http://git.kernel.org/?p=linux/kernel/git/wtarreau/linux-2.4.git;a=summary
> works though.
>
^ permalink raw reply
* Re: [PATCH 2/2] pack-objects: fix threaded load balancing
From: Nicolas Pitre @ 2007-12-10 18:21 UTC (permalink / raw)
To: Jon Smirl; +Cc: Junio C Hamano, git
In-Reply-To: <9e4733910712100906g6794e326qf18da4be146f3667@mail.gmail.com>
On Mon, 10 Dec 2007, Jon Smirl wrote:
> On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
>
> > The hash in the code above has to do with the file names the
> > corresponding objects are coming from.
>
> So can we change this loop to exit after a max of window_size * 10 or
> something like that iterations? Without capping it the threads become
> way unbalanced in the end. In the gcc case one thread is continuing
> 30+ minutes past the others exiting.
Indeed, some more tweaking are needed.
The object path distribution goes like this for the gcc repo:
105557 gcc
42537 gcc/ChangeLog
25210 gcc/config
20690 gcc/testsuite
13434 gcc/testsuite/ChangeLog
12363 libstdc++-v3
9346 gcc/cp
8757 libstdc++-v3/include
8186 gcc/version.c
7867 gcc/cp/ChangeLog
7737 libstdc++-v3/include/bits
7653 libjava
6577 gcc/testsuite/gcc.dg
5942 libjava/ChangeLog
5351 gcc/config/i386
5260 gcc/testsuite/g++.dg
4451 gcc/f
4330 libstdc++-v3/ChangeLog
4321 libstdc++-v3/include/bits/c++config
4316 gcc/doc
[...]
So... the top entries are most certainly going to create load balancing
issues if their path hash clustering isn't broken.
Nicolas
^ permalink raw reply
* Re: How-to combine several separate git repos?
From: Wink Saville @ 2007-12-10 17:55 UTC (permalink / raw)
To: Alex Riesen; +Cc: git
In-Reply-To: <20071210075231.GA2984@steel.home>
Alex Riesen wrote:
> Wink Saville, Sun, Dec 09, 2007 20:12:37 +0100:
>
> <snip>
Thanks it was my mistake,
Wink
^ permalink raw reply
* Re: [RFC] Configurable name(s) for .gitmodules
From: Ping Yin @ 2007-12-10 17:35 UTC (permalink / raw)
To: Sven Verdoolaege; +Cc: Git Mailing List
In-Reply-To: <46dff0320712100934o86af6femaaf563cf40295333@mail.gmail.com>
On Dec 11, 2007 1:34 AM, Ping Yin <pkufranky@gmail.com> wrote:
>
> Also I found Lars's following reply, i still don't understand quite
> well why this idea is not accepted.
Sorry for the typo. It's 'Although', not 'Also'
>
>
> --
> Ping Yin
>
--
Ping Yin
^ permalink raw reply
* Re: [RFC] Configurable name(s) for .gitmodules
From: Ping Yin @ 2007-12-10 17:34 UTC (permalink / raw)
To: Sven Verdoolaege; +Cc: Git Mailing List
In-Reply-To: <20071210165200.GF3490MdfPADPa@greensroom.kotnet.org>
On Dec 11, 2007 12:52 AM, Sven Verdoolaege <skimo@kotnet.org> wrote:
> On Sat, Dec 08, 2007 at 12:00:12AM +0800, Ping Yin wrote:
> > I have a super project with many submodules. Each kind of role may
> > check out different set of submodules. There are some common modules
> > which are almost checked out by every role.
> >
> > Here comes my question: how to implement this elegantly? If all
> > submodules are put in the same .gitmodules, every role has to in the
> > command line manually designate all submodules to be checked out.
>
> You could have submodules (roles in your case) containing
> other submodules, as proposed in the thread that ends in
> http://thread.gmane.org/gmane.comp.version-control.git/48643/focus=48939
>
> skimo
>
Thanks for the suggestion. Linus's idea is what i want. As linus said,
one level of indirect will give much more flexibility. It's wonderful
to put something like following to .gitmodules.
[module "infrastructure"]
submodule = lib
submodule = build
[submodule "lib"]
url = git://xyzzy/lib-1.2.3
[submodule "build"]
url = git://xyzzy/build-0.61
Also I found Lars's following reply, i still don't understand quite
well why this idea is not accepted.
> The reason I wanted to use
>
> [path "lib"]
> submodule=xyzzylib
> url=git://xyzzy/lib-1.2.3
>
> is that the git-submodule command always starts out with the submodule
> path, trying to map that to an url or a submodule name. My proposal
> allows
>
> $ name=$(GIT_CONFIG=.gitmodules git-config path."$path".submodule)
> $ url=$(GIT_CONFIG=.gitmodules git-config path."$path".url)
>
--
Ping Yin
^ permalink raw reply
* Re: Adding Git to Better SCM Initiative : Comparison
From: Jakub Narebski @ 2007-12-10 17:21 UTC (permalink / raw)
To: Chris Shoemaker; +Cc: Florian Weimer, Shlomi Fish, Linus Torvalds, git
In-Reply-To: <20071210165052.GA22327@pe.Belkin>
Chris Shoemaker <c.shoemaker@cox.net> writes:
> On Mon, Dec 10, 2007 at 03:49:39PM +0100, Florian Weimer wrote:
> > * Jakub Narebski:
> >
> > > + <s id="git">
> > > + Yes (or no depending on interpretation). Git
> >
> > This should be "No." (same for copies below).
>
> ISTM that people are stuck using less than helpful criteria for
> judging whether renames are supported. Namely, in effect, they ask:
> "Does the user get to do extra work in order to get rename-detection?"
>
> Let me humbly suggest an alternate, two-fold, very practical criteria
> that I actually care about as a user:
>
> 1) If I edit file A, while another developer renames file A to B, and
> I merge my work with his, do I have to clean things up myself, or does
> everything Just Work?
The only thing Git doesn't implement _yet_ is when you have renamed
a directory, and another developer created a new file in the old
directory name. Currently Git creates new files in old directory.
Note however that moving files to other directory might need changes
in files: for example Java, or header files includes in C/C++. This
is not very common, though.
BTW. this issue is in TODO for "Better SCM : Comparison"
* Add intelligent merging of renamed paths.
> 2) If I'm browsing the history of some code in a renamed file, does
> the history continue through the rename?
And Git does support it in both "git blame" (or "git gui blame"),
and in "git log" thanks to --follow option.
Note however that --follow cannot be used (yet?) with directories or
pathspecs. Not that other SCMs support wildcard pathspec limiting...
> By these criteria, git certainly does support renames.
That's why I wrote "Yes", adding "or no" (as suggested by Robin
Rosenberg) because it does it dofferently than other SCMs.
--
Jakub Narebski
Poland
ShadeHawk on #git
^ permalink raw reply
* Re: [PATCH 2/2] pack-objects: fix threaded load balancing
From: Jon Smirl @ 2007-12-10 17:06 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, git
In-Reply-To: <alpine.LFD.0.99999.0712101104320.555@xanadu.home>
On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
> > On 12/10/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > I just deleted the section looking for identical hashes.
> > >
> > > + while (sub_size && list[0]->hash &&
> > > + list[0]->hash == list[-1]->hash) {
> > > + list++;
> > > + sub_size--;
> > > + }
> > >
> > > Doing that allows the long chains to be split over the cores.
> > >
> > > My last 5% of objects is taking over 50% of the total CPU time in the
> > > repack. I think these objects are the ones from that 103,817 entry
> > > chain. It is also causing the explosion in RAM consumption.
> > >
> > > At the end I can only do 20 objects per clock second on four cores. It
> > > takes 30 clock minutes (120 CPU minutes) to do the last 3% of objects.
> >
> > It's all in create_delta...
>
> Here you're mixing two different hashes with no relation what so ever
> with each other.
>
> The hash in create_delta corresponds to chunk of data in a reference
> buffer that we try to match in a source buffer.
>
> The hash in the code above has to do with the file names the
> corresponding objects are coming from.
So can we change this loop to exit after a max of window_size * 10 or
something like that iterations? Without capping it the threads become
way unbalanced in the end. In the gcc case one thread is continuing
30+ minutes past the others exiting.
> And again, both hash uses are deterministic i.e. they will be the same
> when repacking with -f regardless if the source pack is the 2.1GB or the
> 300MB one, so they may not explain the huge performance and memory usage
> discrepency you see between those two packs.
There is a correlation here but I don't know what it is. The memory
blow up occurs near the end of repacking. At the same time I move from
processing hundreds of objects per second to just a few per second.
And the threads are getting unbalanced.
I did notice that when I removed the above loop and evened things out
memory consumption did not spike as bad as it previously did. I maxed
out at 3GB instead of 4.5GB.
Linus suggested memory fragmentation could be a culprit. Evening the
threads out changed the allocation pattern. It is possible that it
avoided a fragmentation problem. It is also possible that evening
things out split the work so that less memory needed to be allocated.
Don't hold any of these numbers to be gospel. I am using the machine
for other things while I run these tests and there may be
interactions.
>
> The code that do get influenced by the source pack, though, is all
> concentrated in sha1_file.c.
>
>
> Nicolas
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Adding Git to Better SCM Initiative : Comparison
From: Chris Shoemaker @ 2007-12-10 16:50 UTC (permalink / raw)
To: Florian Weimer; +Cc: git
In-Reply-To: <87ve76mwos.fsf@mid.deneb.enyo.de>
On Mon, Dec 10, 2007 at 03:49:39PM +0100, Florian Weimer wrote:
> * Jakub Narebski:
>
> > + <s id="git">
> > + Yes (or no depending on interpretation). Git
>
> This should be "No." (same for copies below).
ISTM that people are stuck using less than helpful criteria for
judging whether renames are supported. Namely, in effect, they ask:
"Does the user get to do extra work in order to get rename-detection?"
Let me humbly suggest an alternate, two-fold, very practical criteria
that I actually care about as a user:
1) If I edit file A, while another developer renames file A to B, and
I merge my work with his, do I have to clean things up myself, or does
everything Just Work?
2) If I'm browsing the history of some code in a renamed file, does
the history continue through the rename?
By these criteria, git certainly does support renames.
-chris
^ permalink raw reply
* Re: [RFC] Configurable name(s) for .gitmodules
From: Sven Verdoolaege @ 2007-12-10 16:54 UTC (permalink / raw)
To: Ping Yin; +Cc: Git Mailing List
In-Reply-To: <46dff0320712070800j6dd83714j265b3f7b291b855c@mail.gmail.com>
On Sat, Dec 08, 2007 at 12:00:12AM +0800, Ping Yin wrote:
> I have a super project with many submodules. Each kind of role may
> check out different set of submodules. There are some common modules
> which are almost checked out by every role.
>
> Here comes my question: how to implement this elegantly? If all
> submodules are put in the same .gitmodules, every role has to in the
> command line manually designate all submodules to be checked out.
You could have submodules (roles in your case) containing
other submodules, as proposed in the thread that ends in
http://thread.gmane.org/gmane.comp.version-control.git/48643/focus=48939
skimo
^ permalink raw reply
* Re: dmalloc and leaks in git
From: Nicolas Pitre @ 2007-12-10 16:54 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <alpine.LFD.0.9999.0712100828590.12046@woody.linux-foundation.org>
On Mon, 10 Dec 2007, Linus Torvalds wrote:
>
>
> On Sun, 9 Dec 2007, Linus Torvalds wrote:
> >
> > For example, if you use
> >
> > valgrind --tool=massif --heap=yes ...
>
> I tried this on my copy of the gcc thing, but I didn't do the extreme
> packing thing, so I never saw the 3.4GB usage. Massif just reported a 200M
> heap, and about half of that was "add_object_entry".
So far, it seems that the problem occurs much more severely when you run
'git repack -a -f' while using the already highly packed gcc repo as a
starting point.
Remains to determine if it occurs only when the repack is threaded, or
if that has no significance.
Nicolas
^ permalink raw reply
* Re: Performance problem, long run of identical hashes
From: David Kastrup @ 2007-12-10 16:14 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <alpine.LFD.0.99999.0712101037270.555@xanadu.home>
Nicolas Pitre <nico@cam.org> writes:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
>> Running oprofile during my gcc repack shows this loop as the hottest
>> place in the code by far.
>
> Well, that is kind of expected.
>
>> I added some debug printfs which show that I
>> have a 100,000+ run of identical hash entries. Processing the 100,000
>> entries also causes RAM consumption to explode.
>
> That is impossible. If you look at the code where those hash entries
> are created in create_delta_index(), you'll notice a hard limit of
> HASH_LIMIT (currently 64) is imposed on the number of identical hash
> entries.
Well, impossible is a strong word to use with respect to code: bugs are
possible. However, we have the assertion
assert(packed_entry - (struct index_entry *)mem == entries);
in create_delta_index and that makes a pretty strong guarantee that the
culling of hash entries should be effective. So at least the overall
_number_ of entries should be consistent. If there is a bug, it might
be that they get garbled or something.
It is also not clear how this could cause an explosion of RAM
consumption in the loop.
--
David Kastrup, Kriemhildstr. 15, 44793 Bochum
^ permalink raw reply
* Re: Adding Git to Better SCM Initiative : Comparison
From: Linus Torvalds @ 2007-12-10 16:38 UTC (permalink / raw)
To: Florian Weimer; +Cc: git
In-Reply-To: <87ve76mwos.fsf@mid.deneb.enyo.de>
On Mon, 10 Dec 2007, Florian Weimer wrote:
> * Jakub Narebski:
>
> > + <s id="git">
> > + Yes (or no depending on interpretation). Git
>
> This should be "No." (same for copies below).
No.
Git handles renames and copies better than most SCM's that _claim_ to
handle them. Saying we don't do them is just stupid. We do them *right*.
The fact that inferior systems do it differently is _their_ problem, and
not a cause for saying that git wouldn't do them.
Linus
^ permalink raw reply
* Re: dmalloc and leaks in git
From: Linus Torvalds @ 2007-12-10 16:34 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <alpine.LFD.0.9999.0712091251330.12046@woody.linux-foundation.org>
On Sun, 9 Dec 2007, Linus Torvalds wrote:
>
> For example, if you use
>
> valgrind --tool=massif --heap=yes ...
I tried this on my copy of the gcc thing, but I didn't do the extreme
packing thing, so I never saw the 3.4GB usage. Massif just reported a 200M
heap, and about half of that was "add_object_entry".
Of course, that doesn't report any mmap usage at all, so it totally
ignores the mapping of the original pack-file itself (which will obviously
be totally dense by the end, since we look at all objects).
It also doesn't take into account various secondary effects. For example,
I don't think it looks at heap fragmentation issues etc, which normally
aren't a noticeable thing, but maybe some particular allocation pattern
can make the glibc allocator waste horrid amounts of memory or something
like that.
Linus
^ permalink raw reply
* Re: Performance problem, long run of identical hashes
From: Nicolas Pitre @ 2007-12-10 16:30 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910712100820k1bd0959fsdfae92727826c6db@mail.gmail.com>
On Mon, 10 Dec 2007, Jon Smirl wrote:
> On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> > On Mon, 10 Dec 2007, Jon Smirl wrote:
> >
> > > Running oprofile during my gcc repack shows this loop as the hottest
> > > place in the code by far.
> >
> > Well, that is kind of expected.
> >
> > > I added some debug printfs which show that I
> > > have a 100,000+ run of identical hash entries. Processing the 100,000
> > > entries also causes RAM consumption to explode.
> >
> > That is impossible. If you look at the code where those hash entries
> > are created in create_delta_index(), you'll notice a hard limit of
> > HASH_LIMIT (currently 64) is imposed on the number of identical hash
> > entries.
>
> On 12/10/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 12/9/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > > + if (victim) {
> > > > + sub_size = victim->remaining / 2;
> > > > + list = victim->list + victim->list_size - sub_size;
> > > > + while (sub_size && list[0]->hash &&
> > > > + list[0]->hash == list[-1]->hash) {
> > > > + list++;
> > >
> > > I think you needed to copy sub_size to another variable for this loop
> >
> > Copying sub_size was wrong. I believe you are checking for deltas on
> > the same file. It's probably that chain of 103,817 deltas that can't
> > be broken up.
>
> At the end of multi-threaded repack one thread ends up with 45 minutes
> of work after all the other threads have exited. That's because it
> hits this loop and can't spit the list any more.
>
> If the lists can't be over 64 identical entries, why do I get caught
> in this loop for 50,000+ iterations? If remove this loop the threads
> are balanced right to the end.
Completely different issue.
Please read my other answers.
Nicolas
^ permalink raw reply
* Re: Adding Git to Better SCM Initiative : Comparison
From: Florian Weimer @ 2007-12-10 16:28 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git, Shlomi Fish
In-Reply-To: <m34peqtuuj.fsf@roke.D-201>
* Jakub Narebski:
> Florian Weimer <fw@deneb.enyo.de> writes:
>
>> * Jakub Narebski:
>> > @@ -214,6 +225,13 @@ <title>File and Directories Copies</title>
> [...]
>> > + <s id="git">
>> > + Yes (or no depending on interpretation). Git
>>
>> This should be "No." (same for copies below).
>
> I would agree to "N/A" or "Partial", but with 'git log --follow'
> implemented at least for single file I wouldn't say that that git
> doesn't support file and directories renames (copies). It does, in
> it's own fashion, using rename (copy) detection instead of rename
> (copy) tracking.
It's undoubtly a difficult question. In my experience, developers tend
to not mark renames properly, so rename detection in log/diff/annotate
is still helpful even if renames are encoded explicitly. But this was a
learning process; I used to think that explicit rename support was
essential.
>> > + <s id="git">
>> > + Partial (?). It is possible to lock down repository
>> > + (access to branches and tags) using hooks.
>> > + </s>
>>
>> I doubt this works reliably. You still can access data once you've got
>> its SHA1 hash, for instance.
>
> So what? The data is not visible, so it is as if it didn't
> exist.
Uhm, I'd commit something that references some SHA-1, and voilà, I can
read the object with that SHA-1.
>> > + <s id="git">
>> > + Yes. Changesets are supported.<br />
>> > + Actually Git is snapshot based which means Git records
>> > + the full state in every commit. This means that any two
>> > + commits can be compared directly very quickly, although the
>> > + repository is typically browsed as a series of changesets.
>> > + </s>
>>
>> I don't think this explanation is necessary. What does Subversion say?
>
> Subversion has the following currently:
>
> Partial support. There are implicit changeset that are generated on
> each commit.
Hmm.
> Well, we could follow Mercurial, Monotone and Darcs and simply write
>
> + <s id="git">
> + Yes. Changesets are supported.
> + </s>
Makes sense, especially since there's "git bundle" nowadays.
>> I don't think this is true. Is there any command that closely matches
>> what CVS does?
>
> Yes: init, add, annotate (alias to blame), checkout, commit, diff,
> status, log, version. At least in principle, if not in output format.
I think we disagree on the meaning of "close" here. 8-/
In my experience, it's hard to see the parallels between GIT and CVS
because the semantics are so different. This is, to some extent,
unavoidable. But I'm not sure if knowing your way around CVS actually
helps learning GIT (the old sayings about BASIC come to my mind).
^ permalink raw reply
* Re: [RFC] Configurable name(s) for .gitmodules
From: Ping Yin @ 2007-12-10 16:20 UTC (permalink / raw)
To: Git Mailing List
In-Reply-To: <46dff0320712070800j6dd83714j265b3f7b291b855c@mail.gmail.com>
On Dec 8, 2007 12:00 AM, Ping Yin <pkufranky@gmail.com> wrote:
> I have a super project with many submodules. Each kind of role may
> check out different set of submodules. There are some common modules
> which are almost checked out by every role.
>
> Here comes my question: how to implement this elegantly? If all
> submodules are put in the same .gitmodules, every role has to in the
> command line manually designate all submodules to be checked out.
> However, it's hard to remember which submodules is required by which
> role, not to mention the so huge .gitmodules. Maybe a script for each
> role can help, but it's a little ugly.
>
> If the name for '.gitmodules' is configurable, we can check in
> modules.roleA and modules.roleB, etc. Then role A can designate the
> corresponding modules.roleA as .gitmodules. Furtherly, if mutiple
> names are allowed, for common modules, we can have modules.common and
> then submodule.defaultnames="modules.common, modules.roleA" to avoid
> duplicated module name entires in both modules.roleA and
> modules.roleB.
>
> I have gone deeply into git-submodule.sh and see
> "GIT_CONFIG=.gitmodules" is used. However, configuable multiple names
> can't be implemented in this way.
>
> Recap:
> 1. configurable name for '.gitmodules'
> 2. better to allow mutiple names
> 3. the implementation issue for mutiple names
>
Can anyone give me some suggestion? Does it make sense to make
GIT_CONFIG record multiple config files, such as
"GIT_CONFIG=modules.roleA:modules.roleB"? Or any better way?
> --
> Ping Yin
>
--
Ping Yin
^ permalink raw reply
* Re: Performance problem, long run of identical hashes
From: Jon Smirl @ 2007-12-10 16:20 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Git Mailing List
In-Reply-To: <alpine.LFD.0.99999.0712101037270.555@xanadu.home>
On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
> > Running oprofile during my gcc repack shows this loop as the hottest
> > place in the code by far.
>
> Well, that is kind of expected.
>
> > I added some debug printfs which show that I
> > have a 100,000+ run of identical hash entries. Processing the 100,000
> > entries also causes RAM consumption to explode.
>
> That is impossible. If you look at the code where those hash entries
> are created in create_delta_index(), you'll notice a hard limit of
> HASH_LIMIT (currently 64) is imposed on the number of identical hash
> entries.
On 12/10/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 12/9/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > + if (victim) {
> > > + sub_size = victim->remaining / 2;
> > > + list = victim->list + victim->list_size - sub_size;
> > > + while (sub_size && list[0]->hash &&
> > > + list[0]->hash == list[-1]->hash) {
> > > + list++;
> >
> > I think you needed to copy sub_size to another variable for this loop
>
> Copying sub_size was wrong. I believe you are checking for deltas on
> the same file. It's probably that chain of 103,817 deltas that can't
> be broken up.
At the end of multi-threaded repack one thread ends up with 45 minutes
of work after all the other threads have exited. That's because it
hits this loop and can't spit the list any more.
If the lists can't be over 64 identical entries, why do I get caught
in this loop for 50,000+ iterations? If remove this loop the threads
are balanced right to the end.
--
Jon Smirl
jonsmirl@gmail.com
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox