* Re: [PATCH 4/2] grep: turn off threading for non-worktree
From: René Scharfe @ 2011-12-07 17:11 UTC (permalink / raw)
To: Jeff King; +Cc: Thomas Rast, git, Eric Herman, Junio C Hamano
In-Reply-To: <20111207044242.GB10765@sigill.intra.peff.net>
Am 07.12.2011 05:42, schrieb Jeff King:
> On Wed, Dec 07, 2011 at 12:01:37AM +0100, René Scharfe wrote:
>
>> Reading of git objects needs to be protected by an exclusive lock
>> and cannot be parallelized. Searching the read buffers can be done
>> in parallel, but for simple expressions threading is a net loss due
>> to its overhead, as measured by Thomas. Turn it off unless we're
>> searching in the worktree.
>
> Based on my earlier numbers, I was going to complain that we should
> also be checking the "simple expressions" assumption here, as time spent
> in the actual regex might be important.
>
> However, after trying to repeat my experiment, I think the numbers I
> posted earlier were misleading. For example, using my "more complex"
> regex of 'a.*b':
>
> $ time git grep --threads=8 'a.*b' HEAD >/dev/null
> real 0m8.655s
> user 0m23.817s
> sys 0m0.480s
>
> Look at that sweet, sweet parallelism. It's a quad-core with
> hyperthreading, so we're not getting the 8x speedup we might hope for
> (presumably due to lock contention on extracting blobs), but hey, 3x
> isn't bad. Except, wait:
>
> $ time git grep --threads=0 'a.*b' HEAD >/dev/null
> real 0m7.651s
> user 0m7.600s
> sys 0m0.048s
>
> We can get 1x on a single core, but the total time is lower! This
> processor is an i7 with "turbo boost", which means it clocks higher in
> single-core mode than when multiple cores are active. So the numbers I
> posted earlier were misleading. Yes, we got parallelism, but at the cost
> of knocking the clock speed down for a net loss.
Ugh, right, Turbo Boost complicates matters.
I don't understand the multiplied user time in the threaded case,
though. Is it caused by busy-waiting? Thomas reported similar numbers
earlier.
> The sweet spot for me seems to be:
>
> $ time git grep --threads=2 'a.*b' HEAD >/dev/null
> real 0m6.303s
> user 0m11.129s
> sys 0m0.220s
>
> I'd be curious to see results from somebody with a quad-core (or more)
> without turbo boost; I suspect that threading may have more benefit
> there, even though we have some lock contention for blobs.
It would be nice if we could come up with simple rules to calculate
defaults for the number of threads on a given run. Users shouldn't have
to specify this option normally. And it would be good if these rules
didn't require a list of all CPUs known to git. :)
>> --- a/builtin/grep.c
>> +++ b/builtin/grep.c
>> @@ -1048,7 +1048,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)
>> nr_threads = 0;
>> #else
>> if (nr_threads == -1)
>> - nr_threads = (online_cpus() > 1) ? THREADS : 0;
>> + nr_threads = (online_cpus() > 1 && !list.nr) ? THREADS : 0;
>>
>> if (nr_threads > 0) {
>> opt.use_threads = 1;
>
> This doesn't kick in for "--cached", which has the same performance
> characteristics as grepping a tree. I think you want to add "&& !cached" to
> the conditional.
Oh, yes.
René
^ permalink raw reply
* Re: Odd issue - The Diffs That WILL NOT DIE.
From: Philippe Vaucher @ 2011-12-07 17:20 UTC (permalink / raw)
To: Chris Patti; +Cc: Carlos Martín Nieto, git
In-Reply-To: <CAJ8P3RCPt9Kwi1F7_TEkZQhkm1mwR_TFKhYszS5LL50kXU8oNQ@mail.gmail.com>
> 11:35][admin@Hiram-Abiff-2:~/src]$ rm -rf framework/
> [11:44][admin@Hiram-Abiff-2:~/src]$ git clone
...snip...
> [11:51][admin@Hiram-Abiff-2:~/src]$ cd framework/
> [11:51][admin@Hiram-Abiff-2:~/src/framework(master)]$ git diff
...snip...
Can you paste the output of "git status" right after the "cd framework"?
Looks like you have some external process that goes and touches your
file after the git clone, or that git fails to check out the files
after cloning but isn't able to work out it failed to checkout those
files.
Philippe
^ permalink raw reply
* [PATCH] index-pack: eliminate unlimited recursion in get_delta_base()
From: Nguyễn Thái Ngọc Duy @ 2011-12-07 17:50 UTC (permalink / raw)
To: git; +Cc: Junio C Hamano, Nguyễn Thái Ngọc Duy
In-Reply-To: <7vvcpthh97.fsf@alter.siamese.dyndns.org>
Revert the order of delta applying so that by the time a delta is
applied, its base is either non-delta or already inflated.
get_delta_base() is still recursive, but because base's data is always
ready, the inner get_delta_base() call never has any chance to call
itself again.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/index-pack.c | 30 +++++++++++++++++++++---------
1 files changed, 21 insertions(+), 9 deletions(-)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 0945adb..103e19c 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -508,10 +508,25 @@ static void *get_base_data(struct base_data *c)
{
if (!c->data) {
struct object_entry *obj = c->obj;
+ struct base_data **delta = NULL;
+ int delta_nr = 0, delta_alloc = 0;
- if (is_delta_type(obj->type)) {
- void *base = get_base_data(c->base);
- void *raw = get_data_from_pack(obj);
+ for (; is_delta_type(c->obj->type); c = c->base) {
+ ALLOC_GROW(delta, delta_nr + 1, delta_alloc);
+ delta[delta_nr++] = c;
+ }
+ if (!delta_nr) {
+ c->data = get_data_from_pack(obj);
+ c->size = obj->size;
+ base_cache_used += c->size;
+ prune_base_data(c);
+ }
+ for (; delta_nr > 0; delta_nr--) {
+ void *base, *raw;
+ c = delta[delta_nr - 1];
+ obj = c->obj;
+ base = get_base_data(c->base);
+ raw = get_data_from_pack(obj);
c->data = patch_delta(
base, c->base->size,
raw, obj->size,
@@ -519,13 +534,10 @@ static void *get_base_data(struct base_data *c)
free(raw);
if (!c->data)
bad_object(obj->idx.offset, "failed to apply delta");
- } else {
- c->data = get_data_from_pack(obj);
- c->size = obj->size;
+ base_cache_used += c->size;
+ prune_base_data(c);
}
-
- base_cache_used += c->size;
- prune_base_data(c);
+ free(delta);
}
return c->data;
}
--
1.7.8.36.g69ee2
^ permalink raw reply related
* Re: Odd issue - The Diffs That WILL NOT DIE.
From: Chris Patti @ 2011-12-07 18:05 UTC (permalink / raw)
To: Philippe Vaucher; +Cc: Carlos Martín Nieto, git
In-Reply-To: <CAGK7Mr7z6hqoda=pr3syzC5mO1+ZExMeD7sReeyRaYL5OMhemw@mail.gmail.com>
On Wed, Dec 7, 2011 at 12:20 PM, Philippe Vaucher
<philippe.vaucher@gmail.com> wrote:
>> 11:35][admin@Hiram-Abiff-2:~/src]$ rm -rf framework/
>> [11:44][admin@Hiram-Abiff-2:~/src]$ git clone
> ...snip...
>> [11:51][admin@Hiram-Abiff-2:~/src]$ cd framework/
>> [11:51][admin@Hiram-Abiff-2:~/src/framework(master)]$ git diff
> ...snip...
>
> Can you paste the output of "git status" right after the "cd framework"?
>
> Looks like you have some external process that goes and touches your
> file after the git clone, or that git fails to check out the files
> after cloning but isn't able to work out it failed to checkout those
> files.
>
> Philippe
Sure.
--
[13:04][admin@Hiram-Abiff-2:~/src/framework(master)]$ git status
# On branch master
# Changes not staged for commit:
# (use "git add <file>..." to update what will be committed)
# (use "git checkout -- <file>..." to discard changes in working directory)
#
# modified: app/modules/Core/controllers/CloudSponge.php
#
no changes added to commit (use "git add" and/or "git commit -a")
[13:04][admin@Hiram-Abiff-2:~/src/framework(master)]$
I'll try rebooting the Mac to see if there's some rogue process afoot :)
-Chris
--
Christopher Patti - Geek At Large | GTalk: cpatti@gmail.com | AIM:
chrisfeohpatti | P: (260) 54PATTI
"Technology challenges art, art inspires technology." - John Lasseter, Pixar
^ permalink raw reply
* Re: [PATCH v2 3/3] grep: disable threading in all but worktree case
From: Jeff King @ 2011-12-07 18:10 UTC (permalink / raw)
To: René Scharfe; +Cc: git, Eric Herman, Junio C Hamano
In-Reply-To: <4EDF99BA.3040006@lsrfire.ath.cx>
On Wed, Dec 07, 2011 at 05:52:10PM +0100, René Scharfe wrote:
> > As a user, I would do:
> >
> > git grep --threads=1 ...
> >
> > if I wanted a single-threaded process. Instead, we actually spawn a
> > sub-thread and do all of the locking, which has a measurable cost:
>
> Yes, the difference is measurable, and that's exactly how I like it to
> be. :) A user can turn off threading with --threads=0 or (more
> intuitively) --no-threads. And we can quantify the overhead.
That seems acceptable to me if --threads is for speed-testing, but a
horrible interface if it is meant for end users who just want git to be
fast.
-Peff
^ permalink raw reply
* Re: Odd issue - The Diffs That WILL NOT DIE.
From: Chris Patti @ 2011-12-07 18:16 UTC (permalink / raw)
To: Philippe Vaucher; +Cc: Carlos Martín Nieto, git
In-Reply-To: <CAGK7Mr7z6hqoda=pr3syzC5mO1+ZExMeD7sReeyRaYL5OMhemw@mail.gmail.com>
On Wed, Dec 7, 2011 at 12:20 PM, Philippe Vaucher
<philippe.vaucher@gmail.com> wrote:
>> 11:35][admin@Hiram-Abiff-2:~/src]$ rm -rf framework/
>> [11:44][admin@Hiram-Abiff-2:~/src]$ git clone
> ...snip...
>> [11:51][admin@Hiram-Abiff-2:~/src]$ cd framework/
>> [11:51][admin@Hiram-Abiff-2:~/src/framework(master)]$ git diff
> ...snip...
>
> Can you paste the output of "git status" right after the "cd framework"?
>
> Looks like you have some external process that goes and touches your
> file after the git clone, or that git fails to check out the files
> after cloning but isn't able to work out it failed to checkout those
> files.
>
> Philippe
FYI this behavior persists across reboots so this rogue process is
also restarting itself :)
Thanks,
-Chris
--
Christopher Patti - Geek At Large | GTalk: cpatti@gmail.com | AIM:
chrisfeohpatti | P: (260) 54PATTI
"Technology challenges art, art inspires technology." - John Lasseter, Pixar
^ permalink raw reply
* Re: [PATCH 4/2] grep: turn off threading for non-worktree
From: Jeff King @ 2011-12-07 18:28 UTC (permalink / raw)
To: René Scharfe; +Cc: Thomas Rast, git, Eric Herman, Junio C Hamano
In-Reply-To: <4EDF9E53.7090702@lsrfire.ath.cx>
On Wed, Dec 07, 2011 at 06:11:47PM +0100, René Scharfe wrote:
> > $ time git grep --threads=8 'a.*b' HEAD >/dev/null
> > real 0m8.655s
> > user 0m23.817s
> > sys 0m0.480s
> [...]
>
> Ugh, right, Turbo Boost complicates matters.
>
> I don't understand the multiplied user time in the threaded case,
> though. Is it caused by busy-waiting? Thomas reported similar numbers
> earlier.
I think it's mostly the clock speed. This processor runs at 1.86GHz but
boosts to 3.2GHz. So we'd expect just the actual work to take close to
twice as long. Plus it's a quad-core with hyperthreading, so 8 threads
is going to mean two threads sharing each core, including cache (i.e.,
hyperthreading a core does not let you double performance, even though
it presents itself as an extra core).
And then you have context switching and lock overhead. So I can believe
that it takes 3x the CPU time to accomplish the task. In an ideal world,
it would be mitigated by having 8x the threads, but in this case, lock
contention brings us down to less than 3x, and it's a slight net loss.
-Peff
^ permalink raw reply
* Re: &&-chaining tester
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <20111207100858.GB13374@elie.hsd1.il.comcast.net>
Hi,
Jonathan Nieder wrote:
> Ramkumar Ramachandra wrote:
>
>> Missing the chaining '&&' seems to be quite a common
>> error: I vaguely remember seeing a patch to detect this sometime ago.
>> Jonathan?
>
> [...]
> - fix all cases of broken &&-chaining. For extra bonus points,
> send out those patches, respond to reviews, and gently usher
> them into Junio's tree
Hm, involves a huge amount of janitorial work. Anyway, thanks for
pointing me in the right direction- here's a small start.
[What I used; for reference]
--
t/test-lib.sh | 10 ++++++++++
1 files changed, 10 insertions(+), 0 deletions(-)
diff --git a/t/test-lib.sh b/t/test-lib.sh
index bdd9513..3bf75ac 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -472,7 +472,17 @@ test_eval_ () {
eval >&3 2>&4 "$*"
}
+check_command_chaining_ () {
+ eval >&3 2>&4 "(exit 189) && $1"
+ eval_chain_ret=$?
+ if test "$eval_chain_ret" != 189
+ then
+ echo '[CHAIN_BUG] in the test script: missing "&&" in test commands'
+ fi
+}
+
test_run_ () {
+ check_command_chaining_ "$1"
test_cleanup=:
expecting_failure=$2
test_eval_ "$1"
--
Thanks.
-- Ram
Ramkumar Ramachandra (15):
t1013 (loose-object-format): fix && chaining
t1300 (repo-config): fix && chaining
t1412 (reflog-loop): fix && chaining
t1007 (hash-object): fix && chaining
t1510 (repo-setup): fix && chaining
t1511 (rev-parse-caret): fix && chaining
t1510 (worktree): fix && chaining
t3200 (branch): fix && chaining
t3418 (rebase-continue): fix && chaining
t3400 (rebase): fix && chaining
t3310 (notes-merge-manual-resolve): fix && chaining
t3419 (rebase-patch-id): fix && chaining
t3030 (merge-recursive): use test_expect_code
t1006 (cat-file): use test_cmp
t3040 (subprojects-basic): modernize style
t/t1006-cat-file.sh | 53 +++---------
t/t1007-hash-object.sh | 16 ++--
t/t1013-loose-object-format.sh | 2 +-
t/t1300-repo-config.sh | 2 +-
t/t1412-reflog-loop.sh | 2 +-
t/t1501-worktree.sh | 6 +-
t/t1510-repo-setup.sh | 4 +-
t/t1511-rev-parse-caret.sh | 2 +-
t/t3030-merge-recursive.sh | 72 ++---------------
t/t3040-subprojects-basic.sh | 144 ++++++++++++++++----------------
t/t3200-branch.sh | 4 +-
t/t3310-notes-merge-manual-resolve.sh | 10 +-
t/t3400-rebase.sh | 4 +-
t/t3418-rebase-continue.sh | 4 +-
t/t3419-rebase-patch-id.sh | 2 +-
15 files changed, 119 insertions(+), 208 deletions(-)
--
1.7.7.3
^ permalink raw reply related
* [PATCH 01/15] t1013 (loose-object-format): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1013-loose-object-format.sh | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/t/t1013-loose-object-format.sh b/t/t1013-loose-object-format.sh
index 0a9cedd..fbf5f2f 100755
--- a/t/t1013-loose-object-format.sh
+++ b/t/t1013-loose-object-format.sh
@@ -34,7 +34,7 @@ assert_blob_equals() {
}
test_expect_success setup '
- cp -R "$TEST_DIRECTORY/t1013/objects" .git/
+ cp -R "$TEST_DIRECTORY/t1013/objects" .git/ &&
git --version
'
--
1.7.7.3
^ permalink raw reply related
* [PATCH 02/15] t1300 (repo-config): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1300-repo-config.sh | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/t/t1300-repo-config.sh b/t/t1300-repo-config.sh
index 51caff0..0690e0e 100755
--- a/t/t1300-repo-config.sh
+++ b/t/t1300-repo-config.sh
@@ -38,7 +38,7 @@ cat > expect << EOF
WhatEver = Second
EOF
test_expect_success 'similar section' '
- git config Cores.WhatEver Second
+ git config Cores.WhatEver Second &&
test_cmp expect .git/config
'
--
1.7.7.3
^ permalink raw reply related
* [PATCH 03/15] t1412 (reflog-loop): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1412-reflog-loop.sh | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/t/t1412-reflog-loop.sh b/t/t1412-reflog-loop.sh
index 647d888..3acd895 100755
--- a/t/t1412-reflog-loop.sh
+++ b/t/t1412-reflog-loop.sh
@@ -20,7 +20,7 @@ test_expect_success 'setup reflog with alternating commits' '
'
test_expect_success 'reflog shows all entries' '
- cat >expect <<-\EOF
+ cat >expect <<-\EOF &&
topic@{0} reset: moving to two
topic@{1} reset: moving to one
topic@{2} reset: moving to two
--
1.7.7.3
^ permalink raw reply related
* [PATCH 04/15] t1007 (hash-object): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1007-hash-object.sh | 16 ++++++++--------
1 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/t/t1007-hash-object.sh b/t/t1007-hash-object.sh
index 6d52b82..316c60a 100755
--- a/t/t1007-hash-object.sh
+++ b/t/t1007-hash-object.sh
@@ -154,13 +154,13 @@ test_expect_success 'check that --no-filters option works with --stdin-paths' '
pop_repo
for args in "-w --stdin" "--stdin -w"; do
- push_repo
+ push_repo &&
test_expect_success "hash from stdin and write to database ($args)" '
test $example_sha1 = $(git hash-object $args < example)
- '
+ ' &&
- test_blob_exists $example_sha1
+ test_blob_exists $example_sha1 &&
pop_repo
done
@@ -176,20 +176,20 @@ test_expect_success "hash two files with names on stdin" '
'
for args in "-w --stdin-paths" "--stdin-paths -w"; do
- push_repo
+ push_repo &&
test_expect_success "hash two files with names on stdin and write to database ($args)" '
test "$sha1s" = "$(echo_without_newline "$filenames" | git hash-object $args)"
- '
+ ' &&
- test_blob_exists $hello_sha1
- test_blob_exists $example_sha1
+ test_blob_exists $hello_sha1 &&
+ test_blob_exists $example_sha1 &&
pop_repo
done
test_expect_success 'corrupt tree' '
- echo abc >malformed-tree
+ echo abc >malformed-tree &&
test_must_fail git hash-object -t tree malformed-tree
'
--
1.7.7.3
^ permalink raw reply related
* [PATCH 05/15] t1510 (repo-setup): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1510-repo-setup.sh | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/t1510-repo-setup.sh b/t/t1510-repo-setup.sh
index ec50a9a..80aedfc 100755
--- a/t/t1510-repo-setup.sh
+++ b/t/t1510-repo-setup.sh
@@ -603,7 +603,7 @@ test_expect_success '#22a: core.worktree = GIT_DIR = .git dir' '
# like case #6.
setup_repo 22a "$here/22a/.git" "" unset &&
- setup_repo 22ab . "" unset
+ setup_repo 22ab . "" unset &&
mkdir -p 22a/.git/sub 22a/sub &&
mkdir -p 22ab/.git/sub 22ab/sub &&
try_case 22a/.git unset . \
@@ -742,7 +742,7 @@ test_expect_success '#28: core.worktree and core.bare conflict (gitfile case)' '
# Case #29: GIT_WORK_TREE(+core.worktree) overrides core.bare (gitfile case).
test_expect_success '#29: setup' '
setup_repo 29 non-existent gitfile true &&
- mkdir -p 29/sub/sub 29/wt/sub
+ mkdir -p 29/sub/sub 29/wt/sub &&
(
cd 29 &&
GIT_WORK_TREE="$here/29" &&
--
1.7.7.3
^ permalink raw reply related
* [PATCH 06/15] t1511 (rev-parse-caret): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1511-rev-parse-caret.sh | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/t/t1511-rev-parse-caret.sh b/t/t1511-rev-parse-caret.sh
index e043cb7..eaefc77 100755
--- a/t/t1511-rev-parse-caret.sh
+++ b/t/t1511-rev-parse-caret.sh
@@ -6,7 +6,7 @@ test_description='tests for ref^{stuff}'
test_expect_success 'setup' '
echo blob >a-blob &&
- git tag -a -m blob blob-tag `git hash-object -w a-blob`
+ git tag -a -m blob blob-tag `git hash-object -w a-blob` &&
mkdir a-tree &&
echo moreblobs >a-tree/another-blob &&
git add . &&
--
1.7.7.3
^ permalink raw reply related
* [PATCH 07/15] t1510 (worktree): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1501-worktree.sh | 6 +++---
1 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/t/t1501-worktree.sh b/t/t1501-worktree.sh
index 6384983..d9761bd 100755
--- a/t/t1501-worktree.sh
+++ b/t/t1501-worktree.sh
@@ -48,7 +48,7 @@ test_expect_success 'setup: helper for testing rev-parse' '
'
test_expect_success 'setup: core.worktree = relative path' '
- unset GIT_WORK_TREE;
+ unset GIT_WORK_TREE &&
GIT_DIR=repo.git &&
GIT_CONFIG="$(pwd)"/$GIT_DIR/config &&
export GIT_DIR GIT_CONFIG &&
@@ -89,7 +89,7 @@ test_expect_success 'subdir of work tree' '
'
test_expect_success 'setup: core.worktree = absolute path' '
- unset GIT_WORK_TREE;
+ unset GIT_WORK_TREE &&
GIT_DIR=$(pwd)/repo.git &&
GIT_CONFIG=$GIT_DIR/config &&
export GIT_DIR GIT_CONFIG &&
@@ -334,7 +334,7 @@ test_expect_success 'absolute pathspec should fail gracefully' '
'
test_expect_success 'make_relative_path handles double slashes in GIT_DIR' '
- >dummy_file
+ >dummy_file &&
echo git --git-dir="$(pwd)//repo.git" --work-tree="$(pwd)" add dummy_file &&
git --git-dir="$(pwd)//repo.git" --work-tree="$(pwd)" add dummy_file
'
--
1.7.7.3
^ permalink raw reply related
* [PATCH 08/15] t3200 (branch): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3200-branch.sh | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/t3200-branch.sh b/t/t3200-branch.sh
index bc73c20..7877290 100755
--- a/t/t3200-branch.sh
+++ b/t/t3200-branch.sh
@@ -22,7 +22,7 @@ test_expect_success \
test_expect_success \
'git branch --help should not have created a bogus branch' '
- git branch --help </dev/null >/dev/null 2>/dev/null;
+ git branch --help </dev/null >/dev/null 2>/dev/null &&
test_path_is_missing .git/refs/heads/--help
'
@@ -88,7 +88,7 @@ test_expect_success \
test_expect_success \
'git branch -m n/n n should work' \
'git branch -l n/n &&
- git branch -m n/n n
+ git branch -m n/n n &&
test_path_is_file .git/logs/refs/heads/n'
test_expect_success 'git branch -m o/o o should fail when o/p exists' '
--
1.7.7.3
^ permalink raw reply related
* [PATCH 09/15] t3418 (rebase-continue): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3418-rebase-continue.sh | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/t3418-rebase-continue.sh b/t/t3418-rebase-continue.sh
index 1e855cd..2680375 100755
--- a/t/t3418-rebase-continue.sh
+++ b/t/t3418-rebase-continue.sh
@@ -51,7 +51,7 @@ test_expect_success 'rebase --continue remembers merge strategy and options' '
test_commit "commit-new-file-F3-on-topic-branch" F3 32 &&
test_when_finished "rm -fr test-bin funny.was.run" &&
mkdir test-bin &&
- cat >test-bin/git-merge-funny <<-EOF
+ cat >test-bin/git-merge-funny <<-EOF &&
#!$SHELL_PATH
case "\$1" in --opt) ;; *) exit 2 ;; esac
shift &&
@@ -77,7 +77,7 @@ test_expect_success 'rebase --continue remembers merge strategy and options' '
test_expect_success 'rebase --continue remembers --rerere-autoupdate' '
rm -fr .git/rebase-* &&
git reset --hard commit-new-file-F3-on-topic-branch &&
- git checkout master
+ git checkout master &&
test_commit "commit-new-file-F3" F3 3 &&
git config rerere.enabled true &&
test_must_fail git rebase -m master topic &&
--
1.7.7.3
^ permalink raw reply related
* [PATCH 10/15] t3400 (rebase): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3400-rebase.sh | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/t/t3400-rebase.sh b/t/t3400-rebase.sh
index 6eaecec..c355533 100755
--- a/t/t3400-rebase.sh
+++ b/t/t3400-rebase.sh
@@ -172,8 +172,8 @@ test_expect_success 'fail when upstream arg is missing and not configured' '
test_expect_success 'default to @{upstream} when upstream arg is missing' '
git checkout -b default topic &&
- git config branch.default.remote .
- git config branch.default.merge refs/heads/master
+ git config branch.default.remote . &&
+ git config branch.default.merge refs/heads/master &&
git rebase &&
test "$(git rev-parse default~1)" = "$(git rev-parse master)"
'
--
1.7.7.3
^ permalink raw reply related
* [PATCH 11/15] t3310 (notes-merge-manual-resolve): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3310-notes-merge-manual-resolve.sh | 10 +++++-----
1 files changed, 5 insertions(+), 5 deletions(-)
diff --git a/t/t3310-notes-merge-manual-resolve.sh b/t/t3310-notes-merge-manual-resolve.sh
index 4ec4d11..4367197 100755
--- a/t/t3310-notes-merge-manual-resolve.sh
+++ b/t/t3310-notes-merge-manual-resolve.sh
@@ -389,7 +389,7 @@ test_expect_success 'abort notes merge' '
test_must_fail ls .git/NOTES_MERGE_* >output 2>/dev/null &&
test_cmp /dev/null output &&
# m has not moved (still == y)
- test "$(git rev-parse refs/notes/m)" = "$(cat pre_merge_y)"
+ test "$(git rev-parse refs/notes/m)" = "$(cat pre_merge_y)" &&
# Verify that other notes refs has not changed (w, x, y and z)
verify_notes w &&
verify_notes x &&
@@ -525,9 +525,9 @@ EOF
test -f .git/NOTES_MERGE_WORKTREE/$commit_sha3 &&
test -f .git/NOTES_MERGE_WORKTREE/$commit_sha4 &&
# Refs are unchanged
- test "$(git rev-parse refs/notes/m)" = "$(git rev-parse refs/notes/w)"
- test "$(git rev-parse refs/notes/y)" = "$(git rev-parse NOTES_MERGE_PARTIAL^1)"
- test "$(git rev-parse refs/notes/m)" != "$(git rev-parse NOTES_MERGE_PARTIAL^1)"
+ test "$(git rev-parse refs/notes/m)" = "$(git rev-parse refs/notes/w)" &&
+ test "$(git rev-parse refs/notes/y)" = "$(git rev-parse NOTES_MERGE_PARTIAL^1)" &&
+ test "$(git rev-parse refs/notes/m)" != "$(git rev-parse NOTES_MERGE_PARTIAL^1)" &&
# Mention refs/notes/m, and its current and expected value in output
grep -q "refs/notes/m" output &&
grep -q "$(git rev-parse refs/notes/m)" output &&
@@ -545,7 +545,7 @@ test_expect_success 'resolve situation by aborting the notes merge' '
test_must_fail ls .git/NOTES_MERGE_* >output 2>/dev/null &&
test_cmp /dev/null output &&
# m has not moved (still == w)
- test "$(git rev-parse refs/notes/m)" = "$(git rev-parse refs/notes/w)"
+ test "$(git rev-parse refs/notes/m)" = "$(git rev-parse refs/notes/w)" &&
# Verify that other notes refs has not changed (w, x, y and z)
verify_notes w &&
verify_notes x &&
--
1.7.7.3
^ permalink raw reply related
* [PATCH 12/15] t3419 (rebase-patch-id): fix && chaining
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3419-rebase-patch-id.sh | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/t/t3419-rebase-patch-id.sh b/t/t3419-rebase-patch-id.sh
index bd8efaf..e70ac10 100755
--- a/t/t3419-rebase-patch-id.sh
+++ b/t/t3419-rebase-patch-id.sh
@@ -39,7 +39,7 @@ run()
}
test_expect_success 'setup' '
- git commit --allow-empty -m initial
+ git commit --allow-empty -m initial &&
git tag root
'
--
1.7.7.3
^ permalink raw reply related
* [PATCH 13/15] t3030 (merge-recursive): use test_expect_code
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Use test_expect_code in preference to repeatedly checking exit codes
by hand.
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3030-merge-recursive.sh | 72 ++++----------------------------------------
1 files changed, 6 insertions(+), 66 deletions(-)
diff --git a/t/t3030-merge-recursive.sh b/t/t3030-merge-recursive.sh
index 55ef189..a5e3da7 100755
--- a/t/t3030-merge-recursive.sh
+++ b/t/t3030-merge-recursive.sh
@@ -285,17 +285,7 @@ test_expect_success 'merge-recursive simple' '
rm -fr [abcd] &&
git checkout -f "$c2" &&
- git merge-recursive "$c0" -- "$c2" "$c1"
- status=$?
- case "$status" in
- 1)
- : happy
- ;;
- *)
- echo >&2 "why status $status!!!"
- false
- ;;
- esac
+ test_expect_code 1 git merge-recursive "$c0" -- "$c2" "$c1"
'
test_expect_success 'merge-recursive result' '
@@ -334,17 +324,7 @@ test_expect_success 'merge-recursive remove conflict' '
rm -fr [abcd] &&
git checkout -f "$c1" &&
- git merge-recursive "$c0" -- "$c1" "$c5"
- status=$?
- case "$status" in
- 1)
- : happy
- ;;
- *)
- echo >&2 "why status $status!!!"
- false
- ;;
- esac
+ test_expect_code 1 git merge-recursive "$c0" -- "$c1" "$c5"
'
test_expect_success 'merge-recursive remove conflict' '
@@ -388,17 +368,7 @@ test_expect_success 'merge-recursive d/f conflict' '
git reset --hard &&
git checkout -f "$c1" &&
- git merge-recursive "$c0" -- "$c1" "$c4"
- status=$?
- case "$status" in
- 1)
- : happy
- ;;
- *)
- echo >&2 "why status $status!!!"
- false
- ;;
- esac
+ test_expect_code 1 git merge-recursive "$c0" -- "$c1" "$c4"
'
test_expect_success 'merge-recursive d/f conflict result' '
@@ -422,17 +392,7 @@ test_expect_success 'merge-recursive d/f conflict the other way' '
git reset --hard &&
git checkout -f "$c4" &&
- git merge-recursive "$c0" -- "$c4" "$c1"
- status=$?
- case "$status" in
- 1)
- : happy
- ;;
- *)
- echo >&2 "why status $status!!!"
- false
- ;;
- esac
+ test_expect_code 1 git merge-recursive "$c0" -- "$c4" "$c1"
'
test_expect_success 'merge-recursive d/f conflict result the other way' '
@@ -456,17 +416,7 @@ test_expect_success 'merge-recursive d/f conflict' '
git reset --hard &&
git checkout -f "$c1" &&
- git merge-recursive "$c0" -- "$c1" "$c6"
- status=$?
- case "$status" in
- 1)
- : happy
- ;;
- *)
- echo >&2 "why status $status!!!"
- false
- ;;
- esac
+ test_expect_code 1 git merge-recursive "$c0" -- "$c1" "$c6"
'
test_expect_success 'merge-recursive d/f conflict result' '
@@ -490,17 +440,7 @@ test_expect_success 'merge-recursive d/f conflict' '
git reset --hard &&
git checkout -f "$c6" &&
- git merge-recursive "$c0" -- "$c6" "$c1"
- status=$?
- case "$status" in
- 1)
- : happy
- ;;
- *)
- echo >&2 "why status $status!!!"
- false
- ;;
- esac
+ test_expect_code 1 git merge-recursive "$c0" -- "$c6" "$c1"
'
test_expect_success 'merge-recursive d/f conflict result' '
--
1.7.7.3
^ permalink raw reply related
* [PATCH 14/15] t1006 (cat-file): use test_cmp
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Use test_cmp in preference to repeatedly comparing command outputs by
hand.
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t1006-cat-file.sh | 53 +++++++++++---------------------------------------
1 files changed, 12 insertions(+), 41 deletions(-)
diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh
index d8b7f2f..af6cacc 100755
--- a/t/t1006-cat-file.sh
+++ b/t/t1006-cat-file.sh
@@ -45,57 +45,28 @@ $content"
test -z "$content" ||
test_expect_success "Content of $type is correct" '
- expect="$(maybe_remove_timestamp "$content" $no_ts)"
- actual="$(maybe_remove_timestamp "$(git cat-file $type $sha1)" $no_ts)"
-
- if test "z$expect" = "z$actual"
- then
- : happy
- else
- echo "Oops: expected $expect"
- echo "but got $actual"
- false
- fi
+ maybe_remove_timestamp "$content" $no_ts >expect &&
+ maybe_remove_timestamp "$(git cat-file $type $sha1)" $no_ts >actual &&
+ test_cmp expect actual
'
test_expect_success "Pretty content of $type is correct" '
- expect="$(maybe_remove_timestamp "$pretty_content" $no_ts)"
- actual="$(maybe_remove_timestamp "$(git cat-file -p $sha1)" $no_ts)"
- if test "z$expect" = "z$actual"
- then
- : happy
- else
- echo "Oops: expected $expect"
- echo "but got $actual"
- false
- fi
+ maybe_remove_timestamp "$pretty_content" $no_ts >expect &&
+ maybe_remove_timestamp "$(git cat-file -p $sha1)" $no_ts >actual &&
+ test_cmp expect actual
'
test -z "$content" ||
test_expect_success "--batch output of $type is correct" '
- expect="$(maybe_remove_timestamp "$batch_output" $no_ts)"
- actual="$(maybe_remove_timestamp "$(echo $sha1 | git cat-file --batch)" $no_ts)"
- if test "z$expect" = "z$actual"
- then
- : happy
- else
- echo "Oops: expected $expect"
- echo "but got $actual"
- false
- fi
+ maybe_remove_timestamp "$batch_output" $no_ts >expect &&
+ maybe_remove_timestamp "$(echo $sha1 | git cat-file --batch)" $no_ts >actual &&
+ test_cmp expect actual
'
test_expect_success "--batch-check output of $type is correct" '
- expect="$sha1 $type $size"
- actual="$(echo_without_newline $sha1 | git cat-file --batch-check)"
- if test "z$expect" = "z$actual"
- then
- : happy
- else
- echo "Oops: expected $expect"
- echo "but got $actual"
- false
- fi
+ echo "$sha1 $type $size" >expect &&
+ echo_without_newline $sha1 | git cat-file --batch-check >actual &&
+ test_cmp expect actual
'
}
--
1.7.7.3
^ permalink raw reply related
* [PATCH 15/15] t3040 (subprojects-basic): modernize style
From: Ramkumar Ramachandra @ 2011-12-07 19:36 UTC (permalink / raw)
To: Jonathan Nieder; +Cc: Junio C Hamano, Git List
In-Reply-To: <1323286611-4806-1-git-send-email-artagnon@gmail.com>
Put the opening quote starting each test on the same line as the
test_expect_* invocation. While at it:
- Indent the file with tabs, not spaces.
- Guard commands that prepare test input for individual tests in the
same test_expect_success, so that their scope is clearer and errors
at that stage can be caught.
- Use <<\-EOF in preference to <<EOF to save readers the trouble of
looking for variable interpolations.
- Include "setup" in the titles of test assertions that prepare for
later ones to make it more obvious which tests can be skipped.
- Chain commands with &&. Breaks in a test assertion's && chain can
potentially hide failures from earlier commands in the chain.
- Use test_expect_code() in preference to checking the exit status of
various statements by hand.
Inspired-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
t/t3040-subprojects-basic.sh | 144 +++++++++++++++++++++---------------------
1 files changed, 72 insertions(+), 72 deletions(-)
diff --git a/t/t3040-subprojects-basic.sh b/t/t3040-subprojects-basic.sh
index f6973e9..0a4ff6d 100755
--- a/t/t3040-subprojects-basic.sh
+++ b/t/t3040-subprojects-basic.sh
@@ -3,81 +3,81 @@
test_description='Basic subproject functionality'
. ./test-lib.sh
-test_expect_success 'Super project creation' \
- ': >Makefile &&
- git add Makefile &&
- git commit -m "Superproject created"'
-
-
-cat >expected <<EOF
-:000000 160000 00000... A sub1
-:000000 160000 00000... A sub2
-EOF
-test_expect_success 'create subprojects' \
- 'mkdir sub1 &&
- ( cd sub1 && git init && : >Makefile && git add * &&
- git commit -q -m "subproject 1" ) &&
- mkdir sub2 &&
- ( cd sub2 && git init && : >Makefile && git add * &&
- git commit -q -m "subproject 2" ) &&
- git update-index --add sub1 &&
- git add sub2 &&
- git commit -q -m "subprojects added" &&
- git diff-tree --abbrev=5 HEAD^ HEAD |cut -d" " -f-3,5- >current &&
- test_cmp expected current'
-
-git branch save HEAD
-
-test_expect_success 'check if fsck ignores the subprojects' \
- 'git fsck --full'
-
-test_expect_success 'check if commit in a subproject detected' \
- '( cd sub1 &&
- echo "all:" >>Makefile &&
- echo " true" >>Makefile &&
- git commit -q -a -m "make all" ) && {
- git diff-files --exit-code
- test $? = 1
- }'
-
-test_expect_success 'check if a changed subproject HEAD can be committed' \
- 'git commit -q -a -m "sub1 changed" && {
- git diff-tree --exit-code HEAD^ HEAD
- test $? = 1
- }'
-
-test_expect_success 'check if diff-index works for subproject elements' \
- 'git diff-index --exit-code --cached save -- sub1
- test $? = 1'
-
-test_expect_success 'check if diff-tree works for subproject elements' \
- 'git diff-tree --exit-code HEAD^ HEAD -- sub1
- test $? = 1'
-
-test_expect_success 'check if git diff works for subproject elements' \
- 'git diff --exit-code HEAD^ HEAD
- test $? = 1'
-
-test_expect_success 'check if clone works' \
- 'git ls-files -s >expected &&
- git clone -l -s . cloned &&
- ( cd cloned && git ls-files -s ) >current &&
- test_cmp expected current'
-
-test_expect_success 'removing and adding subproject' \
- 'git update-index --force-remove -- sub2 &&
- mv sub2 sub3 &&
- git add sub3 &&
- git commit -q -m "renaming a subproject" && {
- git diff -M --name-status --exit-code HEAD^ HEAD
- test $? = 1
- }'
+test_expect_success 'setup: create superproject' '
+ : >Makefile &&
+ git add Makefile &&
+ git commit -m "Superproject created"
+'
+
+test_expect_success 'setup: create subprojects' '
+ mkdir sub1 &&
+ ( cd sub1 && git init && : >Makefile && git add * &&
+ git commit -q -m "subproject 1" ) &&
+ mkdir sub2 &&
+ ( cd sub2 && git init && : >Makefile && git add * &&
+ git commit -q -m "subproject 2" ) &&
+ git update-index --add sub1 &&
+ git add sub2 &&
+ git commit -q -m "subprojects added" &&
+ git diff-tree --abbrev=5 HEAD^ HEAD |cut -d" " -f-3,5- >current &&
+ git branch save HEAD &&
+ cat >expected <<-\EOF &&
+ :000000 160000 00000... A sub1
+ :000000 160000 00000... A sub2
+ EOF
+ test_cmp expected current
+'
+
+test_expect_success 'check if fsck ignores the subprojects' '
+ git fsck --full
+'
+
+test_expect_success 'check if commit in a subproject detected' '
+ ( cd sub1 &&
+ echo "all:" >>Makefile &&
+ echo " true" >>Makefile &&
+ git commit -q -a -m "make all" ) &&
+ test_expect_code 1 git diff-files --exit-code
+'
+
+test_expect_success 'check if a changed subproject HEAD can be committed' '
+ git commit -q -a -m "sub1 changed" &&
+ test_expect_code 1 git diff-tree --exit-code HEAD^ HEAD
+'
+
+test_expect_success 'check if diff-index works for subproject elements' '
+ test_expect_code 1 git diff-index --exit-code --cached save -- sub1
+'
+
+test_expect_success 'check if diff-tree works for subproject elements' '
+ test_expect_code 1 git diff-tree --exit-code HEAD^ HEAD -- sub1
+'
+
+test_expect_success 'check if git diff works for subproject elements' '
+ test_expect_code 1 git diff --exit-code HEAD^ HEAD
+'
+
+test_expect_success 'check if clone works' '
+ git ls-files -s >expected &&
+ git clone -l -s . cloned &&
+ ( cd cloned && git ls-files -s ) >current &&
+ test_cmp expected current
+'
+
+test_expect_success 'removing and adding subproject' '
+ git update-index --force-remove -- sub2 &&
+ mv sub2 sub3 &&
+ git add sub3 &&
+ git commit -q -m "renaming a subproject" &&
+ test_expect_code 1 git diff -M --name-status --exit-code HEAD^ HEAD
+'
# the index must contain the object name the HEAD of the
# subproject sub1 was at the point "save"
-test_expect_success 'checkout in superproject' \
- 'git checkout save &&
- git diff-index --exit-code --raw --cached save -- sub1'
+test_expect_success 'checkout in superproject' '
+ git checkout save &&
+ git diff-index --exit-code --raw --cached save -- sub1
+'
# just interesting what happened...
# git diff --name-status -M save master
--
1.7.7.3
^ permalink raw reply related
* [linux@arm.linux.org.uk: Re: [GIT PULL] Identity mapping changes for 3.3]
From: Uwe Kleine-König @ 2011-12-07 20:06 UTC (permalink / raw)
To: git; +Cc: Russell King - ARM Linux, Nicolas Pitre, Will Deacon
Hello,
Just FYI ...
Best regards
Uwe
----- Forwarded message from Russell King - ARM Linux <linux@arm.linux.org.uk> -----
Date: Wed, 7 Dec 2011 19:48:26 +0000
From: Russell King - ARM Linux <linux@arm.linux.org.uk>
To: Nicolas Pitre <nico@fluxnic.net>
Cc: Will Deacon <will.deacon@arm.com>, "linux-arm-kernel@lists.infradead.org" <linux-arm-kernel@lists.infradead.org>
Subject: Re: [GIT PULL] Identity mapping changes for 3.3
Message-ID: <20111207194826.GB14542@n2100.arm.linux.org.uk>
On Tue, Dec 06, 2011 at 11:25:47PM -0500, Nicolas Pitre wrote:
> Make sure the repo on that machine is nicely packed. Running "git gc"
> (gc as in garbage collect) once in a while is a good thing to do,
> especially that you now have the smart HTTP protocol enabled. That will
> bring the memory usage way down, and serving requests will be much
> faster too. It is safe to put that in a cron job once a week or so,
> even if concurrent requests are being serviced.
Well, I tried an experiment. On my laptop, if I run git fsck, it takes
around about 20 minutes to complete.
On ZenIV, I started this, this morning:
$ GIT_DIR=linux-2.6-arm.git git fsck
and it's now (this evening) some 10 hours after, its still going. This
is the exact same repository (as it's an rsync'd copy of the git objects
and packs which are on the laptop.)
$ ps aux | grep git
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
rmk 26731 1.4 1.0 382716 22036 pts/71 D+ 08:37 9:13 git fsck
apache 29810 1.0 10.0 793288 206564 ? Sl 09:24 6:09 git pack-objects --revs --all --stdout --progress --delta-base-offset
apache 30923 1.0 10.0 790560 206260 ? Sl 09:39 5:53 git pack-objects --revs --all --stdout --progress --delta-base-offset
So, it's consumed 9h of CPU time, the git pack-objects have taken some
5-6 hours...
$ vmstat 10
procs ------------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 4 2502200 45568 8252 125912 50 16 37 62 1 1 1 1 91 7 0
0 4 2502068 46056 8176 126016 44 0 23858 9 2231 1216 1 4 1 94 0
0 4 2501724 45100 8188 126764 76 0 22160 6 2269 1196 1 4 4 91 0
$ iostat 10
...
avg-cpu: %user %nice %system %iowait %steal %idle
0.88 0.00 4.10 91.71 0.00 3.32
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sdf 1.20 10.40 4.80 104 48
sde 0.70 16.00 0.00 160 0
sdd 16.00 432.00 90.40 4320 904
sdg 126.00 12614.40 0.00 126144 0
md0 3.90 26.40 4.80 264 48
sda 139.90 12292.80 0.00 122928 0
sdb 136.10 12393.60 0.00 123936 0
sdc 113.70 12238.40 0.00 122384 0
md1 628.20 49643.20 0.00 496432 0
The repository resides on md1 (sdg, sda, sdb, sdc), the swap partially
on sdd and partially on md0 (sdf and sde).
If I kill all git processes, the disks fall back to about 2MB/s over 10
seconds of analysis.
As you can see, git fsck seems to be pulling data at around 50MB/s,
presumably for 9 hours - this is rediculous because there's only 500MB
of git data for it to read!
What are the top six users of system memory?
rmk 26731 1.4 1.0 380600 21348 pts/71 D+ 08:37 9:17 git fsck
named 30776 0.4 1.4 130804 29212 ? Ssl Oct08 386:55 /usr/sbin/named-sdb
root 2475 0.6 2.0 58940 41376 ? S 10:40 3:38 spamd child
exim 2148 0.0 4.4 160228 91100 ? Ssl Jun24 183:50 clamd.exim
apache 29810 1.0 10.2 794312 210816 ? Sl 09:24 6:27 git pack-objects
apache 30923 1.0 10.3 794656 213536 ? Sl 09:39 6:10 git pack-objects
and lower down there's the bunch of apache httpd processes.
What this is saying to me is that git can't run sensibly on a dual-core
P4, 3GHz machine with 2G of RAM and 4G swap, with a disk IO subsystem
capable of about 50MB/s - basically, git is driving ZenIV into the ground
(and I believe git was also responsible for ZenIV having a load average
hitting a few hundred several months ago which resulted in us having to
have it rebooted.)
Last bit of information - from /proc/29810/maps:
0012b000-00148000 r-xp 00000000 08:33 179255 /lib/ld-2.13.so
00148000-00149000 r--p 0001c000 08:33 179255 /lib/ld-2.13.so
00149000-0014a000 rw-p 0001d000 08:33 179255 /lib/ld-2.13.so
0014c000-002cf000 r-xp 00000000 08:33 179409 /lib/libc-2.13.so
002cf000-002d0000 ---p 00183000 08:33 179409 /lib/libc-2.13.so
002d0000-002d2000 r--p 00183000 08:33 179409 /lib/libc-2.13.so
002d2000-002d3000 rw-p 00185000 08:33 179409 /lib/libc-2.13.so
002d3000-002d6000 rw-p 00000000 00:00 0
00304000-0031b000 r-xp 00000000 08:33 179711 /lib/libpthread-2.13.so
0031b000-0031c000 r--p 00016000 08:33 179711 /lib/libpthread-2.13.so
0031c000-0031d000 rw-p 00017000 08:33 179711 /lib/libpthread-2.13.so
0031d000-0031f000 rw-p 00000000 00:00 0
00328000-0033c000 r-xp 00000000 08:33 179755 /lib/libz.so.1.2.5
0033c000-0033d000 rw-p 00013000 08:33 179755 /lib/libz.so.1.2.5
006da000-006db000 r-xp 00000000 00:00 0 [vdso]
08047000-08171000 r-xp 00000000 08:32 600804 /usr/libexec/git-core/git
08171000-08176000 rw-p 0012a000 08:32 600804 /usr/libexec/git-core/git
08176000-081be000 rw-p 00000000 00:00 0
0a06a000-0f1b0000 rw-p 00000000 00:00 0 [heap]
80a74000-90836000 rw-p 00000000 00:00 0
986bc000-9a6bc000 r--p 00000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
9a6bc000-9c6bc000 r--p 0b000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
9c6bc000-9e6bd000 rw-p 00000000 00:00 0
a0330000-a0331000 ---p 00000000 00:00 0
a0331000-a0d31000 rw-p 00000000 00:00 0
a0d31000-a0d32000 ---p 00000000 00:00 0
a0d32000-a1732000 rw-p 00000000 00:00 0
a1e72000-a3e72000 r--p 08000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a3e72000-a5e72000 r--p 12000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a5e72000-a7e72000 r--p 02000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a8732000-a9e72000 r--p 1a000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
a9e72000-abe72000 r--p 05000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
abe72000-ac6fd000 rw-p 00000000 00:00 0
ac6fd000-ae6fd000 r--p 06000000 09:01 106316459 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.pack
ae6fd000-af3fb000 rw-p 00000000 00:00 0
af946000-af947000 rw-p 00000000 00:00 0
afb00000-afbb3000 rw-p 00000000 00:00 0
afbb3000-afc00000 ---p 00000000 00:00 0
afcc1000-b1652000 rw-p 00000000 00:00 0
b1800000-b1900000 rw-p 00000000 00:00 0
b1ae8000-b1af0000 r--p 00000000 09:01 88555537 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-60ee04b368ada02ba788d17a537c71bad7471a7a.idx
b1af0000-b1af3000 r--p 00000000 09:01 106318826 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-87f106a487c423809976c6868ec7f1e9fc12e676.idx
b1af3000-b1af6000 r--p 00000000 09:01 88555528 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-1a10fd87008d0e06d2ce83efff9bf4ba7abb12e8.idx
b1af6000-b1b03000 r--p 00000000 09:01 88555566 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-6ed1c32d2ec8e0cf20c9ef7d8b813e1f63f67203.idx
b1d00000-b1de8000 rw-p 00000000 00:00 0
b1de8000-b1e00000 ---p 00000000 00:00 0
b1e00000-b1e51000 rw-p 00000000 00:00 0
b1e51000-b1f00000 ---p 00000000 00:00 0
b1f00000-b1ffe000 rw-p 00000000 00:00 0
b1ffe000-b2000000 ---p 00000000 00:00 0
b2000000-b20e5000 rw-p 00000000 00:00 0
b20e5000-b2100000 ---p 00000000 00:00 0
b2100000-b21fc000 rw-p 00000000 00:00 0
b21fc000-b2200000 ---p 00000000 00:00 0
b2200000-b22fd000 rw-p 00000000 00:00 0
b22fd000-b2300000 ---p 00000000 00:00 0
b2300000-b23d7000 rw-p 00000000 00:00 0
b23d7000-b2400000 ---p 00000000 00:00 0
b2400000-b24fd000 rw-p 00000000 00:00 0
b24fd000-b2500000 ---p 00000000 00:00 0
b2500000-b25ea000 rw-p 00000000 00:00 0
b25ea000-b2600000 ---p 00000000 00:00 0
b2600000-b26e4000 rw-p 00000000 00:00 0
b26e4000-b2700000 ---p 00000000 00:00 0
b2700000-b27ee000 rw-p 00000000 00:00 0
b27ee000-b2800000 ---p 00000000 00:00 0
b2800000-b28ec000 rw-p 00000000 00:00 0
b28ec000-b2900000 ---p 00000000 00:00 0
b2900000-b29f3000 rw-p 00000000 00:00 0
b29f3000-b2a00000 ---p 00000000 00:00 0
b2b00000-b2cfe000 rw-p 00000000 00:00 0
b2cfe000-b2d00000 ---p 00000000 00:00 0
b2d00000-b2dfd000 rw-p 00000000 00:00 0
b2dfd000-b2e00000 ---p 00000000 00:00 0
b2f00000-b2ff4000 rw-p 00000000 00:00 0
b2ff4000-b3000000 ---p 00000000 00:00 0
b3100000-b31fe000 rw-p 00000000 00:00 0
b31fe000-b3200000 ---p 00000000 00:00 0
b3300000-b33f1000 rw-p 00000000 00:00 0
b33f1000-b3400000 ---p 00000000 00:00 0
b3500000-b35f6000 rw-p 00000000 00:00 0
b35f6000-b3600000 ---p 00000000 00:00 0
b3600000-b36f3000 rw-p 00000000 00:00 0
b36f3000-b3700000 ---p 00000000 00:00 0
b3700000-b37f9000 rw-p 00000000 00:00 0
b37f9000-b3800000 ---p 00000000 00:00 0
b3900000-b39f3000 rw-p 00000000 00:00 0
b39f3000-b3a00000 ---p 00000000 00:00 0
b3b03000-b76ab000 r--p 00000000 09:01 106316443 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-895bab46ff2e2e3d88eb2ebb8919a86104e6f59e.idx
b76ab000-b76b1000 r--p 00000000 09:01 106316389 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-742507456b81724d4b1b7c62a9c7b1a08dea788c.idx
b76b1000-b76b3000 r--p 00000000 09:01 88555543 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-01d8e6a80f43c483b170f252a63549c3f3177435.idx
b76b3000-b76b5000 r--p 00000000 09:01 106316378 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-22db475e3babd502f5462dbd53b0083f63054da8.idx
b76b5000-b76b7000 r--p 00000000 09:01 106319097 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-81e9156c16956b1ca13d0786f26c57da8a8c4d00.idx
b76b7000-b76b9000 r--p 00000000 09:01 106316404 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-02bd0e7dcc0d4f9ba1faef6cb90d3080451223ab.idx
b76b9000-b76be000 r--p 00000000 09:01 106320451 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-64a9ef29c0dd71c0ada102c21a6d3297715b5ec0.idx
b76be000-b76c0000 r--p 00000000 09:01 88555532 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-cf138203ddca32052979b5eadc8b03fa8d9f6313.idx
b76c0000-b76c2000 r--p 00000000 09:01 88555522 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-08c44c7f188425373f8ecb95e4bb342ee40d340e.idx
b76c2000-b76cb000 r--p 00000000 09:01 88555534 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-1fd3ce5f1145636ad725dec83f5ef91b80ec1cf3.idx
b76cb000-b76cf000 r--p 00000000 09:01 88555545 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-0e5768a969e7c4ef4f3406a129b1271971ee5342.idx
b76cf000-b76d4000 r--p 00000000 09:01 88555547 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-92592ad40e58a75d3d1b30a0a46e7ad63817637b.idx
b76d4000-b76d6000 r--p 00000000 09:01 88555549 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-5b26c4eb6fed3ce5b26e921aee548a7f3259a11e.idx
b76d6000-b76d9000 r--p 00000000 09:01 88555551 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-4a9b526f37540b18dd2ff4e97ec94fcb49c0aeb6.idx
b76d9000-b76dd000 r--p 00000000 09:01 88555553 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-1a7d19bf1f4aa63bb54b0f7e77f5df8889099e3f.idx
b76dd000-b76e0000 r--p 00000000 09:01 88555558 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-7da553217dc96aa89cdcf50fe45615143aa01089.idx
b76e0000-b76e6000 r--p 00000000 09:01 88555555 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-e795f1594ea70cca88a97ad23d2d62da06941038.idx
b76e6000-b76ea000 r--p 00000000 09:01 88555560 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-4c79da67fb0435e3a4a7440a714fb9d617995387.idx
b76e6000-b76ea000 r--p 00000000 09:01 88555560 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-4c79da67fb0435e3a4a7440a714fb9d617995387.idx
b76ea000-b76ef000 r--p 00000000 09:01 88555563 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-63e7f343a0aa5f8f6841c81211db00ce95adbe76.idx
b76ef000-b76f1000 rw-p 00000000 00:00 0
b76f5000-b76fa000 r--p 00000000 09:01 88555565 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-b37609403c59702e384e42a9df4bc16e2b3c1072.idx
b76fa000-b76fc000 r--p 00000000 09:01 88555542 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-23f0d9e30ada6f829f6743a395bcb6f229c1453b.idx
b76fc000-b7700000 r--p 00000000 09:01 88555571 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-d073eba6bd4143e405ec9db4172d4406fa08ace8.idx
b7700000-b7702000 r--p 00000000 09:01 88555574 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-b2481dc10cc72226ff396e864e4a6d7e2c26137d.idx
b7702000-b7704000 r--p 00000000 09:01 88555577 /archive/ftp/pub/linux/arm/kernel/git-cur/linux-2.6-arm.git/objects/pack/pack-866787d1e38a3c7c9ac0ab7057935bf5970bf319.idx
b7704000-b7705000 rw-p 00000000 00:00 0
bf8dc000-bf8fd000 rw-p 00000000 00:00 0 [stack]
It's worth noting that Linus tree currently has 19 pack files, my tree
has an additional 9 on that.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
----- End forwarded message -----
--
Pengutronix e.K. | Uwe Kleine-König |
Industrial Linux Solutions | http://www.pengutronix.de/ |
^ permalink raw reply
* Re: [PATCH 4/2] grep: turn off threading for non-worktree
From: J. Bruce Fields @ 2011-12-07 20:11 UTC (permalink / raw)
To: Jeff King
Cc: René Scharfe, Thomas Rast, git, Eric Herman, Junio C Hamano
In-Reply-To: <20111207044242.GB10765@sigill.intra.peff.net>
On Tue, Dec 06, 2011 at 11:42:42PM -0500, Jeff King wrote:
> On Wed, Dec 07, 2011 at 12:01:37AM +0100, René Scharfe wrote:
>
> > Reading of git objects needs to be protected by an exclusive lock
> > and cannot be parallelized. Searching the read buffers can be done
> > in parallel, but for simple expressions threading is a net loss due
> > to its overhead, as measured by Thomas. Turn it off unless we're
> > searching in the worktree.
>
> Based on my earlier numbers, I was going to complain that we should
> also be checking the "simple expressions" assumption here, as time spent
> in the actual regex might be important.
>
> However, after trying to repeat my experiment, I think the numbers I
> posted earlier were misleading. For example, using my "more complex"
> regex of 'a.*b':
>
> $ time git grep --threads=8 'a.*b' HEAD >/dev/null
> real 0m8.655s
> user 0m23.817s
> sys 0m0.480s
Dumb question (I missed the beginning of the conversation): what kind of
storage are you using, and is the data already cached?
I seem to recall part of the motivation for the multithreading being
NFS, where the goal isn't so much to keep CPU's busy as it is to keep
the network busy.
Probably a bigger problem for something like "git status" which I think
ends up doing a series of stat's (which can each require a round trip to
the server in the NFS case), as it is a problem for something like
git-grep that's also doing reads.
Just a plea for considering the IO cost as well when making these kinds
of decisions....
(Which maybe you already do, apologies again for just naively dropping
into the middle of a thread.)
--b.
>
> Look at that sweet, sweet parallelism. It's a quad-core with
> hyperthreading, so we're not getting the 8x speedup we might hope for
> (presumably due to lock contention on extracting blobs), but hey, 3x
> isn't bad. Except, wait:
>
> $ time git grep --threads=0 'a.*b' HEAD >/dev/null
> real 0m7.651s
> user 0m7.600s
> sys 0m0.048s
>
> We can get 1x on a single core, but the total time is lower! This
> processor is an i7 with "turbo boost", which means it clocks higher in
> single-core mode than when multiple cores are active. So the numbers I
> posted earlier were misleading. Yes, we got parallelism, but at the cost
> of knocking the clock speed down for a net loss.
>
> The sweet spot for me seems to be:
>
> $ time git grep --threads=2 'a.*b' HEAD >/dev/null
> real 0m6.303s
> user 0m11.129s
> sys 0m0.220s
>
> I'd be curious to see results from somebody with a quad-core (or more)
> without turbo boost; I suspect that threading may have more benefit
> there, even though we have some lock contention for blobs.
>
> > --- a/builtin/grep.c
> > +++ b/builtin/grep.c
> > @@ -1048,7 +1048,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)
> > nr_threads = 0;
> > #else
> > if (nr_threads == -1)
> > - nr_threads = (online_cpus() > 1) ? THREADS : 0;
> > + nr_threads = (online_cpus() > 1 && !list.nr) ? THREADS : 0;
> >
> > if (nr_threads > 0) {
> > opt.use_threads = 1;
>
> This doesn't kick in for "--cached", which has the same performance
> characteristics as grepping a tree. I think you want to add "&& !cached" to
> the conditional.
>
> -Peff
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox