* Re: Something is broken in repack
From: Jon Smirl @ 2007-12-11 2:25 UTC (permalink / raw)
To: Git Mailing List, Nicolas Pitre
In-Reply-To: <9e4733910712071505y6834f040k37261d65a2d445c4@mail.gmail.com>
New run using same configuration. With the addition of the more
efficient load balancing patches and delta cache accounting.
Seconds are wall clock time. They are lower since the patch made
threading better at using all four cores. I am stuck at 380-390% CPU
utilization for the git process.
complete seconds RAM
10% 60 900M (includes counting)
20% 15 900M
30% 15 900M
40% 50 1.2G
50% 80 1.3G
60% 70 1.7G
70% 140 1.8G
80% 180 2.0G
90% 280 2.2G
95% 530 2.8G - 1,420 total to here, previous was 1,983
100% 1390 2.85G
During the writing phase RAM fell to 1.6G
What is being freed in the writing phase??
I have no explanation for the change in RAM usage. Two guesses come to
mind. Memory fragmentation. Or the change in the way the work was
split up altered RAM usage.
Total CPU time was 195 minutes in 70 minutes clock time. About 70%
efficient. During the compress phase all four cores were active until
the last 90 seconds. Writing the objects took over 23 minutes CPU
bound on one core.
New pack file is: 270,594,853
Old one was: 344,543,752
It still has 828,660 objects
On 12/7/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> Using this config:
> [pack]
> threads = 4
> deltacachesize = 256M
> deltacachelimit = 0
>
> And the 330MB gcc pack for input
> git repack -a -d -f --depth=250 --window=250
>
> complete seconds RAM
> 10% 47 1GB
> 20% 29 1Gb
> 30% 24 1Gb
> 40% 18 1GB
> 50% 110 1.2GB
> 60% 85 1.4GB
> 70% 195 1.5GB
> 80% 186 2.5GB
> 90% 489 3.8GB
> 95% 800 4.8GB
> I killed it because it started swapping
>
> The mmaps are only about 400MB in this case.
> At the end the git process had 4.4GB of physical RAM allocated.
>
> Starting from a highly compressed pack greatly aggravates the problem.
> Starting with a 2GB pack of the same data my process size only grew to
> 3GB with 2GB of mmaps.
>
> --
> Jon Smirl
> jonsmirl@gmail.com
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Using git with Eclipse
From: Shawn O. Pearce @ 2007-12-11 2:44 UTC (permalink / raw)
To: Wink Saville; +Cc: git
In-Reply-To: <475DC0CE.9070109@saville.com>
Wink Saville <wink@saville.com> wrote:
> I'm trying to use git on an Eclipse workspace and the .metadata
> directory is chock full of files and was wondering what, if anything,
> should be ignored. At the moment .history looks like a candidate for
> ignoring there are probably others.
Ignore all of .metadata; its Eclipse private state that you don't
want to version. I'd add it to .git/info/exclude so its ignored only
in the repository that is using Eclipse, rather than in .gitignore
(which is published).
--
Shawn.
^ permalink raw reply
* Re: [PATCH (amend)] diff: Make numstat machine friendly also for renames (and copies)
From: Junio C Hamano @ 2007-12-11 2:50 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <200712110226.35343.jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> Junio C Hamano wrote:
>> Jakub Narebski <jnareb@gmail.com> writes:
>>
>> > Previous version of this patch (from 7 May 2007) used instead of current
>> > only "to_name" format similar to git-diff-tree raw format for renames:
>> >
>> > added deleted TAB path for "src" TAB path for "dst" LF
That's perfectly fine (that is without -z).
>> > ... Previous version
>> > of patch used (or was to use actually, because of error in the code)
>> >
>> > added deleted TAB path for "src" NUL NUL path for "dst" NUL
>> >
>> > when -z option was used.
I think your "previous" one:
<added> <deleted> <src> NUL (no rename)
<added> <deleted> <src> NUL NUL <dst> NUL (with rename)
would be unambiguously parsable, but it would be cleaner and easier for
the parser if the latter were like this instead:
<added> <deleted> NUL <src> NUL <dst> NUL (with rename)
The reader can expect to find a NUL terminated src, finds the length is
zero and notices it cannot be src and then reads two paths from that
point.
Without having the status letter, we cannot do "optional two paths"
without breaking existing --numstat -z readers that do not care about
renames and copies, and I agree that existing --numstat -z readers that
pass -M or -C are already broken, so I think a format extension along
the above line (your "previous" and the above clean-up proposal have the
same power of expressiveness, I just think the latter is syntactically
cleaner) would be reasonable. And at that point we probably would not
need --numstat-enhanced at all ;-)
^ permalink raw reply
* Re: Something is broken in repack
From: Junio C Hamano @ 2007-12-11 2:55 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List, Nicolas Pitre
In-Reply-To: <9e4733910712101825l33cdc2c0mca2ddbfd5afdb298@mail.gmail.com>
"Jon Smirl" <jonsmirl@gmail.com> writes:
> 95% 530 2.8G - 1,420 total to here, previous was 1,983
> 100% 1390 2.85G
> During the writing phase RAM fell to 1.6G
> What is being freed in the writing phase??
entry->delta_data is the only thing I can think of that are freed
in the function that have been allocated much earlier before entering
the function.
^ permalink raw reply
* Re: [PATCH] autoconf: Check autoconf version (ASCIIDOC8)
From: Junio C Hamano @ 2007-12-11 3:11 UTC (permalink / raw)
To: Jakub Narebski; +Cc: git
In-Reply-To: <1197205059-16742-1-git-send-email-jnareb@gmail.com>
Jakub Narebski <jnareb@gmail.com> writes:
> Subject: Re: [PATCH] autoconf: Check autoconf version (ASCIIDOC8)
That's "Check asciidoc version".
> Check for asciidoc, and if it exists check asciidoc version, setting
> ASCIIDOC8 when needed. Currently it just runs asciidoc in asciidoc7
> compatibility mode (see: Documentation/Makefile).
>
> Signed-off-by: Jakub Narebski <jnareb@gmail.com>
> ---
> diff --git a/config.mak.in b/config.mak.in
> index 7d5df9b..759470a 100644
> --- a/config.mak.in
> +++ b/config.mak.in
> @@ -7,6 +7,7 @@ AR = @AR@
> TAR = @TAR@
> #INSTALL = @INSTALL@ # needs install-sh or install.sh in sources
> TCLTK_PATH = @TCLTK_PATH@
> +ASCIIDOC=@ASCIIDOC@
And this is "ASCIIDOC8=@ASCIIDOC8@", I think.
Will apply with the said fixups.
Thanks.
^ permalink raw reply
* Re: Something is broken in repack
From: Nicolas Pitre @ 2007-12-11 3:27 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jon Smirl, Git Mailing List
In-Reply-To: <7vlk82hrdt.fsf@gitster.siamese.dyndns.org>
On Mon, 10 Dec 2007, Junio C Hamano wrote:
> "Jon Smirl" <jonsmirl@gmail.com> writes:
>
> > 95% 530 2.8G - 1,420 total to here, previous was 1,983
> > 100% 1390 2.85G
> > During the writing phase RAM fell to 1.6G
> > What is being freed in the writing phase??
>
> entry->delta_data is the only thing I can think of that are freed
> in the function that have been allocated much earlier before entering
> the function.
Yet all ->delta-data instances are limited to 256MB according to Jon's
config.
Nicolas
>
Nicolas
^ permalink raw reply
* Re: Something is broken in repack
From: Nicolas Pitre @ 2007-12-11 3:49 UTC (permalink / raw)
To: Jon Smirl; +Cc: Git Mailing List
In-Reply-To: <9e4733910712101825l33cdc2c0mca2ddbfd5afdb298@mail.gmail.com>
On Mon, 10 Dec 2007, Jon Smirl wrote:
> New run using same configuration. With the addition of the more
> efficient load balancing patches and delta cache accounting.
>
> Seconds are wall clock time. They are lower since the patch made
> threading better at using all four cores. I am stuck at 380-390% CPU
> utilization for the git process.
>
> complete seconds RAM
> 10% 60 900M (includes counting)
> 20% 15 900M
> 30% 15 900M
> 40% 50 1.2G
> 50% 80 1.3G
> 60% 70 1.7G
> 70% 140 1.8G
> 80% 180 2.0G
> 90% 280 2.2G
> 95% 530 2.8G - 1,420 total to here, previous was 1,983
> 100% 1390 2.85G
> During the writing phase RAM fell to 1.6G
> What is being freed in the writing phase??
The cached delta results, but you put a cap of 256MB for them.
Could you try again with that cache disabled entirely, with
pack.deltacachesize = 1 (don't use 0 as that means unbounded).
And then, while still keeping the delta cache disabled, could you try
with pack.threads = 2, and pack.threads = 1 ?
I'm sorry to ask you to do this but I don't have enough ram to even
complete a repack with threads=2 so I'm reattempting single threaded at
the moment. But I really wonder if the threading has such an effect on
memory usage.
>
> I have no explanation for the change in RAM usage. Two guesses come to
> mind. Memory fragmentation. Or the change in the way the work was
> split up altered RAM usage.
>
> Total CPU time was 195 minutes in 70 minutes clock time. About 70%
> efficient. During the compress phase all four cores were active until
> the last 90 seconds. Writing the objects took over 23 minutes CPU
> bound on one core.
>
> New pack file is: 270,594,853
> Old one was: 344,543,752
> It still has 828,660 objects
You mean the pack for the gcc repo is now less than 300MB? Wow.
Nicolas
^ permalink raw reply
* Re: v1.5.4 plans
From: Nicolas Pitre @ 2007-12-11 3:53 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vmysijhwq.fsf_-_@gitster.siamese.dyndns.org>
On Mon, 10 Dec 2007, Junio C Hamano wrote:
> There are outstanding issues that need to be resolved:
>
> * We need to do something about "gc --aggressive". The documentation
> removal patch from Linus, if it ever materializes, would be better
> than nothing, but I have this nagging suspicion that the explosion is
> merely a bad interation between -A and -f option to the repack, which
> are not meant to be used together.
Well, with the gcc repo, simply using 'git repack -a -f' with current
default window size does produce a 2.1GB pack, while changing the window
size to 100 (keeping default delta depth) produces a 400MB pack for me.
So this is really a matter of not having a sufficiently large window for
some data sets.
> * We have a handful deprecation notices in the draft release notes, but
> if I recall correctly, Nico wanted to add a few more. We need to
> come up with a wording that is easy to understand for the end users
> to decide which ancient versions will be affected.
This is still on my todo list.
Nicolas
^ permalink raw reply
* "git tag -u keyname" broken
From: Linus Torvalds @ 2007-12-11 4:08 UTC (permalink / raw)
To: Junio C Hamano, Carlos Rica, Git Mailing List
Commit 396865859918e9c7bf8ce74aae137c57da134610 broke signed tags using
the "-u" flag when it made builtin-tag.c use parse_options() to parse its
arguments (but it quite possibly was broken even before that, by the
builtin rewrite).
It used to be that passing the signing ID with the -u parameter also
(obviously!) implied that you wanted to sign and annotate the tag, but
that logic got dropped. It also totally ignored the actual key ID that was
passed in.
This reinstates it all.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
---
This has gotten only very minimal testing. Somebody should double-check it
all, and adding a test would probably be great too.
builtin-tag.c | 14 +++++++++++---
1 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/builtin-tag.c b/builtin-tag.c
index 517419f..274901a 100644
--- a/builtin-tag.c
+++ b/builtin-tag.c
@@ -236,14 +236,18 @@ static const char tag_template[] =
"# Write a tag message\n"
"#\n";
+static void set_signingkey(const char *value)
+{
+ if (strlcpy(signingkey, value, sizeof(signingkey)) >= sizeof(signingkey))
+ die("signing key value too long (%.10s...)", value);
+}
+
static int git_tag_config(const char *var, const char *value)
{
if (!strcmp(var, "user.signingkey")) {
if (!value)
die("user.signingkey without value");
- if (strlcpy(signingkey, value, sizeof(signingkey))
- >= sizeof(signingkey))
- die("user.signingkey value too long");
+ set_signingkey(value);
return 0;
}
@@ -396,6 +400,10 @@ int cmd_tag(int argc, const char **argv, const char *prefix)
argc = parse_options(argc, argv, options, git_tag_usage, 0);
+ if (keyid) {
+ sign = 1;
+ set_signingkey(keyid);
+ }
if (sign)
annotate = 1;
^ permalink raw reply related
* [PATCH] t7004: test that "git-tag -u" implies "-s"
From: Jeff King @ 2007-12-11 4:24 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, Carlos Rica, Git Mailing List
In-Reply-To: <alpine.LFD.0.9999.0712101950110.28293@woody.linux-foundation.org>
This was lost in the C conversion, but recently fixed by
Linus.
Signed-off-by: Jeff King <peff@peff.net>
---
On Mon, Dec 10, 2007 at 08:08:06PM -0800, Linus Torvalds wrote:
> It used to be that passing the signing ID with the -u parameter also
> (obviously!) implied that you wanted to sign and annotate the tag, but
> that logic got dropped. It also totally ignored the actual key ID that was
> passed in.
> [...]
> This has gotten only very minimal testing. Somebody should double-check it
> all, and adding a test would probably be great too.
Looks good to me, and here's a test. It was trivial to whip up, since I
wrote a test for almost the identical bug a few days ago (that one was
"-s implies -a").
t/t7004-tag.sh | 8 ++++++++
1 files changed, 8 insertions(+), 0 deletions(-)
diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh
index c7130c4..5393d35 100755
--- a/t/t7004-tag.sh
+++ b/t/t7004-tag.sh
@@ -682,6 +682,14 @@ test_expect_success '-s implies annotated tag' '
get_tag_msg implied-annotate >actual &&
git diff expect actual
'
+get_tag_header implied-sign $commit commit $time >expect
+./fakeeditor >>expect
+echo '-----BEGIN PGP SIGNATURE-----' >>expect
+test_expect_success '-u implies signed tag' '
+ GIT_EDITOR=./fakeeditor git-tag -u CDDE430D implied-sign &&
+ get_tag_msg implied-sign >actual &&
+ git diff expect actual
+'
test_expect_success \
'trying to create a signed tag with non-existing -F file should fail' '
--
1.5.3.7.2229.gd040e-dirty
^ permalink raw reply related
* Re: [PATCH] t7004: test that "git-tag -u" implies "-s"
From: Junio C Hamano @ 2007-12-11 4:38 UTC (permalink / raw)
To: Jeff King; +Cc: Linus Torvalds, Carlos Rica, Git Mailing List
In-Reply-To: <20071211042436.GA23805@coredump.intra.peff.net>
Jeff King <peff@peff.net> writes:
> Looks good to me, and here's a test. It was trivial to whip up, since I
> wrote a test for almost the identical bug a few days ago (that one was
> "-s implies -a").
Looks good to me. We should verify both common forms of key-id and also
check a negative case. This squashes a few of mine on top of yours.
---
t/t7004-tag.sh | 26 ++++++++++++++++++++++++++
1 files changed, 26 insertions(+), 0 deletions(-)
diff --git a/t/t7004-tag.sh b/t/t7004-tag.sh
index c7130c4..a502286 100755
--- a/t/t7004-tag.sh
+++ b/t/t7004-tag.sh
@@ -640,6 +640,32 @@ test_expect_success 'creating a signed tag with -m message should succeed' '
git diff expect actual
'
+get_tag_header implied-sign $commit commit $time >expect
+echo 'Another signed tag' >>expect
+echo '-----BEGIN PGP SIGNATURE-----' >>expect
+test_expect_success '-u implies signed tag' '
+ git-tag -u CDDE430D -m "Another signed tag" implied-sign &&
+ get_tag_msg implied-sign >actual &&
+ git diff expect actual
+'
+
+get_tag_header u-signed-tag $commit commit $time >expect
+echo 'Another message' >>expect
+echo '-----BEGIN PGP SIGNATURE-----' >>expect
+test_expect_success 'sign with a given key id' '
+
+ git tag -u committer@example.com -m "Another message" u-signed-tag &&
+ get_tag_msg u-signed-tag >actual &&
+ git diff expect actual
+
+'
+
+test_expect_success 'sign with an unknown id' '
+
+ ! git tag -u author@example.com -m "Another message" o-signed-tag
+
+'
+
cat >sigmsgfile <<EOF
Another signed tag
message in a file.
^ permalink raw reply related
* Re: v1.5.4 plans
From: Junio C Hamano @ 2007-12-11 5:02 UTC (permalink / raw)
To: Jeff King; +Cc: git
In-Reply-To: <7vtzmqhvgq.fsf@gitster.siamese.dyndns.org>
Junio C Hamano <gitster@pobox.com> writes:
> Jeff King <peff@peff.net> writes:
>
>> A few regressions that you did not mention, but I think should be
>> addressed before 1.5.4:
>>
>> - extra newline in builtin-commit output. You found a case that
>> needs it, but fixing it is non-trivial, and I wanted to get your
>> input before preparing a patch. See
>>
>> http://mid.gmane.org/20071203075357.GB3614@sigill.intra.peff.net
>
> I am actually becoming somewhat fond of the newline that makes the end
> of a session that led to a commit stand out ;-). IOW, I was wondering if
> we can have another for a merge commit case.
>
> But I suspect that it amounts to the change in the same area and of
> similar complexity.
-- >8 --
[PATCH] commit: do not add extra LF at the end of the summary.
The scripted version relied on the nice "auto-strip the terminating LF"
behaviour shell gives to "var=$(cmd)" construct, but we have to roll
that ourselves.
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
builtin-commit.c | 11 ++++++++---
1 files changed, 8 insertions(+), 3 deletions(-)
diff --git a/builtin-commit.c b/builtin-commit.c
index b6b81d5..e5a972c 100644
--- a/builtin-commit.c
+++ b/builtin-commit.c
@@ -660,12 +660,17 @@ static void print_summary(const char *prefix, const unsigned char *sha1)
rev.verbose_header = 1;
rev.show_root_diff = 1;
rev.commit_format = get_commit_format("format:%h: %s");
- rev.always_show_header = 1;
+ rev.always_show_header = 0;
printf("Created %scommit ", initial_commit ? "initial " : "");
- log_tree_commit(&rev, commit);
- printf("\n");
+ if (!log_tree_commit(&rev, commit)) {
+ struct strbuf buf = STRBUF_INIT;
+ pretty_print_commit(rev.commit_format, commit, &buf,
+ 0, NULL, NULL, 0, 0);
+ printf("%s\n", buf.buf);
+ strbuf_release(&buf);
+ }
}
int git_commit_config(const char *k, const char *v)
--
1.5.3.7-1149-g591a
^ permalink raw reply related
* Re: [PATCH 6/5] Move fetch_ref from http-push.c and http-walker.c to http.c
From: Junio C Hamano @ 2007-12-11 5:09 UTC (permalink / raw)
To: Mike Hommey; +Cc: git
In-Reply-To: <1197328105-4843-1-git-send-email-mh@glandium.org>
Mike Hommey <mh@glandium.org> writes:
> Make the necessary changes to be ok with their difference, and rename the
> function http_fetch_ref.
Sorry, but I cannot parse "ok with their difference" part...
^ permalink raw reply
* Re: [PATCH 0/3] git-help -i (info): finishing touches
From: Christian Couder @ 2007-12-11 5:22 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git, David Kastrup
In-Reply-To: <7v8x42n9s4.fsf@gitster.siamese.dyndns.org>
Le lundi 10 décembre 2007, Junio C Hamano a écrit :
> Junio C Hamano <gitster@pobox.com> writes:
> > This series attempts to add finishing touches to "git help -i" (info)
> > support.
It looks like the patch series you talk about was not sent to the list.
In the last patch there is:
diff --git a/help.c b/help.c
index 9d7ad6f..c96b167 100644
--- a/help.c
+++ b/help.c
@@ -286,6 +286,7 @@ static void show_man_page(const char *git_cmd)
static void show_info_page(const char *git_cmd)
{
const char *page = cmd_to_page(git_cmd);
+ setenv("INFOPATH", GIT_INFO_PATH, 1);
execlp("info", "info", "gitman", page, NULL);
}
But I wonder if something like this would be better:
diff --git a/help.c b/help.c
index c96b167..933d850 100644
--- a/help.c
+++ b/help.c
@@ -255,23 +255,23 @@ static const char *cmd_to_page(const char *git_cmd)
}
}
-static void setup_man_path(void)
+static void prepend_to_env_path(const char *name, const char *value)
{
struct strbuf new_path;
- const char *old_path = getenv("MANPATH");
+ const char *old_path = getenv(name);
strbuf_init(&new_path, 0);
/* We should always put ':' after our path. If there is no
- * old_path, the ':' at the end will let 'man' to try
- * system-wide paths after ours to find the manual page. If
+ * old_path, the ':' at the end will let some commands
+ * like 'man' to try system-wide paths after ours. If
* there is old_path, we need ':' as delimiter. */
- strbuf_addstr(&new_path, GIT_MAN_PATH);
+ strbuf_addstr(&new_path, value);
strbuf_addch(&new_path, ':');
if (old_path)
strbuf_addstr(&new_path, old_path);
- setenv("MANPATH", new_path.buf, 1);
+ setenv(name, new_path.buf, 1);
strbuf_release(&new_path);
}
@@ -279,14 +279,14 @@ static void setup_man_path(void)
static void show_man_page(const char *git_cmd)
{
const char *page = cmd_to_page(git_cmd);
- setup_man_path();
+ prepend_to_env_path("MANPATH", GIT_MAN_PATH);
execlp("man", "man", page, NULL);
}
static void show_info_page(const char *git_cmd)
{
const char *page = cmd_to_page(git_cmd);
- setenv("INFOPATH", GIT_INFO_PATH, 1);
+ prepend_to_env_path("INFOPATH", GIT_INFO_PATH);
execlp("info", "info", "gitman", page, NULL);
}
Tell me if you want a proper path.
Christian.
^ permalink raw reply related
* Re: [PATCH] t7004: test that "git-tag -u" implies "-s"
From: Jeff King @ 2007-12-11 5:22 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Linus Torvalds, Carlos Rica, Git Mailing List
In-Reply-To: <7v63z5j16g.fsf@gitster.siamese.dyndns.org>
On Mon, Dec 10, 2007 at 08:38:31PM -0800, Junio C Hamano wrote:
> +get_tag_header implied-sign $commit commit $time >expect
> +echo 'Another signed tag' >>expect
> +echo '-----BEGIN PGP SIGNATURE-----' >>expect
> +test_expect_success '-u implies signed tag' '
> + git-tag -u CDDE430D -m "Another signed tag" implied-sign &&
> + get_tag_msg implied-sign >actual &&
> + git diff expect actual
> +'
One thing that my original did test but this version doesn't: not only
does -u imply -s, but -u implies -s which implies -a. IOW, using "-u"
will put you in an editor (and a poorly done fix might have gotten this
wrong, but apparently Linus is a super-genius).
Your other two tests look good.
-Peff
^ permalink raw reply
* Re: Something is broken in repack
From: Jon Smirl @ 2007-12-11 5:25 UTC (permalink / raw)
To: Nicolas Pitre, Junio C Hamano; +Cc: Git Mailing List
In-Reply-To: <alpine.LFD.0.99999.0712102231570.555@xanadu.home>
On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> On Mon, 10 Dec 2007, Jon Smirl wrote:
>
> > New run using same configuration. With the addition of the more
> > efficient load balancing patches and delta cache accounting.
> >
> > Seconds are wall clock time. They are lower since the patch made
> > threading better at using all four cores. I am stuck at 380-390% CPU
> > utilization for the git process.
> >
> > complete seconds RAM
> > 10% 60 900M (includes counting)
> > 20% 15 900M
> > 30% 15 900M
> > 40% 50 1.2G
> > 50% 80 1.3G
> > 60% 70 1.7G
> > 70% 140 1.8G
> > 80% 180 2.0G
> > 90% 280 2.2G
> > 95% 530 2.8G - 1,420 total to here, previous was 1,983
> > 100% 1390 2.85G
> > During the writing phase RAM fell to 1.6G
> > What is being freed in the writing phase??
>
> The cached delta results, but you put a cap of 256MB for them.
>
> Could you try again with that cache disabled entirely, with
> pack.deltacachesize = 1 (don't use 0 as that means unbounded).
>
> And then, while still keeping the delta cache disabled, could you try
> with pack.threads = 2, and pack.threads = 1 ?
>
> I'm sorry to ask you to do this but I don't have enough ram to even
> complete a repack with threads=2 so I'm reattempting single threaded at
> the moment. But I really wonder if the threading has such an effect on
> memory usage.
I already have a threads = 1 running with this config. Binary and
config were same from threads=4 run.
10% 28min 950M
40% 135min 950M
50% 157min 900M
60% 160min 830M
100% 170min 830M
Something is hurting bad with threads. 170 CPU minutes with one
thread, versus 195 CPU minutes with four threads.
Is there a different memory allocator that can be used when
multithreaded on gcc? This whole problem may be coming from the memory
allocation function. git is hardly interacting at all on the thread
level so it's likely a problem in the C run-time.
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[pack]
threads = 1
deltacachesize = 256M
windowmemory = 256M
deltacachelimit = 0
[remote "origin"]
url = git://git.infradead.org/gcc.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "trunk"]
remote = origin
merge = refs/heads/trunk
>
>
>
> >
> > I have no explanation for the change in RAM usage. Two guesses come to
> > mind. Memory fragmentation. Or the change in the way the work was
> > split up altered RAM usage.
> >
> > Total CPU time was 195 minutes in 70 minutes clock time. About 70%
> > efficient. During the compress phase all four cores were active until
> > the last 90 seconds. Writing the objects took over 23 minutes CPU
> > bound on one core.
> >
> > New pack file is: 270,594,853
> > Old one was: 344,543,752
> > It still has 828,660 objects
>
> You mean the pack for the gcc repo is now less than 300MB? Wow.
>
>
> Nicolas
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [PATCH 0/3] git-help -i (info): finishing touches
From: Junio C Hamano @ 2007-12-11 5:27 UTC (permalink / raw)
To: Christian Couder; +Cc: git, David Kastrup
In-Reply-To: <200712110622.51583.chriscool@tuxfamily.org>
Christian Couder <chriscool@tuxfamily.org> writes:
> Le lundi 10 décembre 2007, Junio C Hamano a écrit :
>> Junio C Hamano <gitster@pobox.com> writes:
>> > This series attempts to add finishing touches to "git help -i" (info)
>> > support.
>
> It looks like the patch series you talk about was not sent to the list.
Hmph, it appears that there was a misconfiguration on my part to drive
send-email. Look into 'next' branch.
> In the last patch there is:
>
> diff --git a/help.c b/help.c
> index 9d7ad6f..c96b167 100644
> --- a/help.c
> +++ b/help.c
> @@ -286,6 +286,7 @@ static void show_man_page(const char *git_cmd)
> static void show_info_page(const char *git_cmd)
> {
> const char *page = cmd_to_page(git_cmd);
> + setenv("INFOPATH", GIT_INFO_PATH, 1);
> execlp("info", "info", "gitman", page, NULL);
> }
>
> But I wonder if something like this would be better:
I do not think so, as MANPATH has that funky semantics of "appending the
system path where a single colon separates an empty path component from
the rest." but I do not think there is anything like that in INFOPATH.
^ permalink raw reply
* Re: Something is broken in repack
From: Jon Smirl @ 2007-12-11 5:29 UTC (permalink / raw)
To: Nicolas Pitre, Junio C Hamano, gcc; +Cc: Git Mailing List
In-Reply-To: <9e4733910712102125w56c70c0cxb8b00a060b62077@mail.gmail.com>
I added the gcc people to the CC, it's their repository. Maybe they
can help up sort this out.
On 12/11/07, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 12/10/07, Nicolas Pitre <nico@cam.org> wrote:
> > On Mon, 10 Dec 2007, Jon Smirl wrote:
> >
> > > New run using same configuration. With the addition of the more
> > > efficient load balancing patches and delta cache accounting.
> > >
> > > Seconds are wall clock time. They are lower since the patch made
> > > threading better at using all four cores. I am stuck at 380-390% CPU
> > > utilization for the git process.
> > >
> > > complete seconds RAM
> > > 10% 60 900M (includes counting)
> > > 20% 15 900M
> > > 30% 15 900M
> > > 40% 50 1.2G
> > > 50% 80 1.3G
> > > 60% 70 1.7G
> > > 70% 140 1.8G
> > > 80% 180 2.0G
> > > 90% 280 2.2G
> > > 95% 530 2.8G - 1,420 total to here, previous was 1,983
> > > 100% 1390 2.85G
> > > During the writing phase RAM fell to 1.6G
> > > What is being freed in the writing phase??
> >
> > The cached delta results, but you put a cap of 256MB for them.
> >
> > Could you try again with that cache disabled entirely, with
> > pack.deltacachesize = 1 (don't use 0 as that means unbounded).
> >
> > And then, while still keeping the delta cache disabled, could you try
> > with pack.threads = 2, and pack.threads = 1 ?
> >
> > I'm sorry to ask you to do this but I don't have enough ram to even
> > complete a repack with threads=2 so I'm reattempting single threaded at
> > the moment. But I really wonder if the threading has such an effect on
> > memory usage.
>
> I already have a threads = 1 running with this config. Binary and
> config were same from threads=4 run.
>
> 10% 28min 950M
> 40% 135min 950M
> 50% 157min 900M
> 60% 160min 830M
> 100% 170min 830M
>
> Something is hurting bad with threads. 170 CPU minutes with one
> thread, versus 195 CPU minutes with four threads.
>
> Is there a different memory allocator that can be used when
> multithreaded on gcc? This whole problem may be coming from the memory
> allocation function. git is hardly interacting at all on the thread
> level so it's likely a problem in the C run-time.
>
> [core]
> repositoryformatversion = 0
> filemode = true
> bare = false
> logallrefupdates = true
> [pack]
> threads = 1
> deltacachesize = 256M
> windowmemory = 256M
> deltacachelimit = 0
> [remote "origin"]
> url = git://git.infradead.org/gcc.git
> fetch = +refs/heads/*:refs/remotes/origin/*
> [branch "trunk"]
> remote = origin
> merge = refs/heads/trunk
>
>
>
>
> >
> >
> >
> > >
> > > I have no explanation for the change in RAM usage. Two guesses come to
> > > mind. Memory fragmentation. Or the change in the way the work was
> > > split up altered RAM usage.
> > >
> > > Total CPU time was 195 minutes in 70 minutes clock time. About 70%
> > > efficient. During the compress phase all four cores were active until
> > > the last 90 seconds. Writing the objects took over 23 minutes CPU
> > > bound on one core.
> > >
> > > New pack file is: 270,594,853
> > > Old one was: 344,543,752
> > > It still has 828,660 objects
> >
> > You mean the pack for the gcc repo is now less than 300MB? Wow.
> >
> >
> > Nicolas
> >
>
>
> --
> Jon Smirl
> jonsmirl@gmail.com
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: Using git with Eclipse
From: Wink Saville @ 2007-12-11 5:55 UTC (permalink / raw)
To: Shawn O. Pearce; +Cc: git
In-Reply-To: <20071211024442.GJ14735@spearce.org>
Shawn O. Pearce wrote:
> Wink Saville <wink@saville.com> wrote:
>
>> I'm trying to use git on an Eclipse workspace and the .metadata
>> directory is chock full of files and was wondering what, if anything,
>> should be ignored. At the moment .history looks like a candidate for
>> ignoring there are probably others.
>>
>
> Ignore all of .metadata; its Eclipse private state that you don't
> want to version. I'd add it to .git/info/exclude so its ignored only
> in the repository that is using Eclipse, rather than in .gitignore
> (which is published).
>
>
Shawn,
I added .metadata to exclude then used git rm to remove
.metadata from the repository. I then cloned that
repository to see how Eclipse would work. (As part of my
workflow I use git as a backup so I wanted to see what would
happen when I "restored".)
As I'm sure you know with the metadata gone my existing projects
in the Ui were gone and they have to be recreated as well as
some Eclipse and plugin specific configuration.
I understand you and others are working on an Eclipse plugin
for git, will it also ignore . metadata?
Do you need any testing done or is it too early? I'd be glad to
test if you feel its solid enough that I won't lose data or if it
uses a separate different repo then I could use both.
Regards,
Wink Saville
^ permalink raw reply
* Re: Something is broken in repack
From: Sean @ 2007-12-11 6:01 UTC (permalink / raw)
To: Jon Smirl; +Cc: Nicolas Pitre, Junio C Hamano, Git Mailing List
In-Reply-To: <9e4733910712102125w56c70c0cxb8b00a060b62077@mail.gmail.com>
On Tue, 11 Dec 2007 00:25:55 -0500
"Jon Smirl" <jonsmirl@gmail.com> wrote:
> Something is hurting bad with threads. 170 CPU minutes with one
> thread, versus 195 CPU minutes with four threads.
>
> Is there a different memory allocator that can be used when
> multithreaded on gcc? This whole problem may be coming from the memory
> allocation function. git is hardly interacting at all on the thread
> level so it's likely a problem in the C run-time.
You might want to try Google's malloc, it's basically a drop in replacement
with some optional built-in performance monitoring capabilities. It is said
to be much faster and better at threading than glibc's:
http://code.google.com/p/google-perftools/wiki/GooglePerformanceTools
http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html
You can LD_PRELOAD it or link directly.
Cheers,
Sean
^ permalink raw reply
* Re: [Resend PATCH 2/4] Use strbuf in http code
From: Junio C Hamano @ 2007-12-11 6:04 UTC (permalink / raw)
To: Mike Hommey; +Cc: git
In-Reply-To: <1197228659-19459-1-git-send-email-mh@glandium.org>
Mike Hommey <mh@glandium.org> writes:
> Also, replace whitespaces with tabs in some places
>
> Signed-off-by: Mike Hommey <mh@glandium.org>
> ---
>
> While this doesn't fix the problem with symbolic refs, it does fail more
> cleanly.
>
> http-push.c | 187 +++++++++++++++++++--------------------------------------
> http-walker.c | 60 +++++++------------
> http.c | 34 ++++------
> http.h | 11 ++--
> transport.c | 18 ++----
> 5 files changed, 109 insertions(+), 201 deletions(-)
I like this code reduction, but
> diff --git a/http.h b/http.h
> index fe1b0d1..bf3f12c 100644
> --- a/http.h
> +++ b/http.h
> @@ -6,6 +6,8 @@
> #include <curl/curl.h>
> #include <curl/easy.h>
>
> +#include "strbuf.h"
> +
> #if LIBCURL_VERSION_NUM >= 0x071000
> #define USE_CURL_MULTI
> #define DEFAULT_MAX_REQUESTS 5
> @@ -48,18 +50,17 @@ struct active_request_slot
>
> struct buffer
> {
> - size_t posn;
> - size_t size;
> - void *buffer;
> + struct strbuf buf;
> + size_t posn;
> };
With this definition of "struct buffer", I do not think this can be correct.
> @@ -1267,10 +1257,8 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
> {
> struct active_request_slot *slot;
> struct slot_results results;
> - struct buffer out_buffer;
> - struct buffer in_buffer;
> - char *out_data;
> - char *in_data;
> + struct buffer out_buffer = { 0, STRBUF_INIT };
How seriously have you proofread and tested this series before sending
it out?
^ permalink raw reply
* Re: [Resend PATCH 2/4] Use strbuf in http code
From: Mike Hommey @ 2007-12-11 6:16 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vlk81him3.fsf@gitster.siamese.dyndns.org>
On Mon, Dec 10, 2007 at 10:04:52PM -0800, Junio C Hamano wrote:
> Mike Hommey <mh@glandium.org> writes:
> > struct buffer
> > {
> > - size_t posn;
> > - size_t size;
> > - void *buffer;
> > + struct strbuf buf;
> > + size_t posn;
> > };
>
> With this definition of "struct buffer", I do not think this can be correct.
>
> > @@ -1267,10 +1257,8 @@ static struct remote_lock *lock_remote(const char *path, long timeout)
> > {
> > struct active_request_slot *slot;
> > struct slot_results results;
> > - struct buffer out_buffer;
> > - struct buffer in_buffer;
> > - char *out_data;
> > - char *in_data;
> > + struct buffer out_buffer = { 0, STRBUF_INIT };
>
> How seriously have you proofread and tested this series before sending
> it out?
Damn ! I did fix this on sunday and was pretty sure to have sent it :-/
Mike
^ permalink raw reply
* Re: v1.5.4 plans
From: Jeff King @ 2007-12-11 6:17 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Eric Wong, git
In-Reply-To: <7vtzmqhvgq.fsf@gitster.siamese.dyndns.org>
On Mon, Dec 10, 2007 at 05:27:17PM -0800, Junio C Hamano wrote:
> > And perhaps not a regression, but I think we should bring git-svn's
> > handling of color.* in line with the changes to the rest of the code
> > before 1.5.4. I posted a "last resort" patch, but I think with your
> > changes to "git config --colorbool" it might be possible to use that.
> > I'll try to work up a new patch.
>
> Thanks for a reminder. Anything else?
2-patch series will follow momentarily. 1/2 gives --get-colorbool the
necessary information for implementing color.pager, and 2/2 fixes
git-svn.
Very light testing by me, since I'm not actually a git-svn user, but it
does pass the test scripts. Acks from svn-using people would be nice.
-Peff
^ permalink raw reply
* Re: Something is broken in repack
From: Jon Smirl @ 2007-12-11 6:20 UTC (permalink / raw)
To: Sean; +Cc: Nicolas Pitre, Junio C Hamano, Git Mailing List
In-Reply-To: <BAYC1-PASMTP08CFB6F824B1282649E5EAAE640@CEZ.ICE>
On 12/11/07, Sean <seanlkml@sympatico.ca> wrote:
> On Tue, 11 Dec 2007 00:25:55 -0500
> "Jon Smirl" <jonsmirl@gmail.com> wrote:
>
> > Something is hurting bad with threads. 170 CPU minutes with one
> > thread, versus 195 CPU minutes with four threads.
> >
> > Is there a different memory allocator that can be used when
> > multithreaded on gcc? This whole problem may be coming from the memory
> > allocation function. git is hardly interacting at all on the thread
> > level so it's likely a problem in the C run-time.
>
> You might want to try Google's malloc, it's basically a drop in replacement
> with some optional built-in performance monitoring capabilities. It is said
> to be much faster and better at threading than glibc's:
>
> http://code.google.com/p/google-perftools/wiki/GooglePerformanceTools
> http://google-perftools.googlecode.com/svn/trunk/doc/tcmalloc.html
>
>
> You can LD_PRELOAD it or link directly.
I'm 45 minutes into a run using it. It doesn't seem to be any faster
but it is reducing memory consumption significantly. The run should be
done in another 20 minutes or so.
>
> Cheers,
> Sean
>
--
Jon Smirl
jonsmirl@gmail.com
^ permalink raw reply
* Re: [PATCH 6/5] Move fetch_ref from http-push.c and http-walker.c to http.c
From: Mike Hommey @ 2007-12-11 6:21 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
In-Reply-To: <7vve75hl72.fsf@gitster.siamese.dyndns.org>
On Mon, Dec 10, 2007 at 09:09:05PM -0800, Junio C Hamano wrote:
> Mike Hommey <mh@glandium.org> writes:
>
> > Make the necessary changes to be ok with their difference, and rename the
> > function http_fetch_ref.
>
> Sorry, but I cannot parse "ok with their difference" part...
What about "to fit with their difference" ?
Mike
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox