* [PATCH] autodetect number of CPUs by default when using threads
@ 2008-12-11 20:36 Nicolas Pitre
2008-12-11 22:25 ` Junio C Hamano
2008-12-13 13:32 ` Jeff King
0 siblings, 2 replies; 11+ messages in thread
From: Nicolas Pitre @ 2008-12-11 20:36 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
... and display the actual number of threads used when locally
repacking. A remote server still won't tell you how many threads it
uses during a fetch though.
Signed-off-by: Nicolas Pitre <nico@cam.org>
---
I've spent quite a while wondering why repacking in one repo was faster
than repacking in a clone of that repo on the same machine. So let's
display how many threads are actually used.
We have comprehensive test in Makefile to determine if threads are
available, just to not use them by default. I think that code has
proven itself for long enough now not to let people benefit from it.
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index cedef52..619e597 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -78,7 +78,7 @@ static int progress = 1;
static int window = 10;
static uint32_t pack_size_limit, pack_size_limit_cfg;
static int depth = 50;
-static int delta_search_threads = 1;
+static int delta_search_threads;
static int pack_to_stdout;
static int num_preferred_base;
static struct progress *progress_state;
@@ -1612,6 +1612,9 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,
find_deltas(list, &list_size, window, depth, processed);
return;
}
+ if (progress > pack_to_stdout)
+ fprintf(stderr, "Delta compression using %d threads.\n",
+ delta_search_threads);
/* Partition the work amongst work threads. */
for (i = 0; i < delta_search_threads; i++) {
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] autodetect number of CPUs by default when using threads
2008-12-11 20:36 [PATCH] autodetect number of CPUs by default when using threads Nicolas Pitre
@ 2008-12-11 22:25 ` Junio C Hamano
2008-12-12 0:55 ` Nicolas Pitre
2008-12-13 13:32 ` Jeff King
1 sibling, 1 reply; 11+ messages in thread
From: Junio C Hamano @ 2008-12-11 22:25 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: git
Nicolas Pitre <nico@cam.org> writes:
> I've spent quite a while wondering why repacking in one repo was faster
> than repacking in a clone of that repo on the same machine. So let's
> display how many threads are actually used.
>
> We have comprehensive test in Makefile to determine if threads are
> available, just to not use them by default. I think that code has
> proven itself for long enough now not to let people benefit from it.
Hmm, it does appear that distros compile with THREADED_DELTA_SEARCH turned
on (I only checked Fedora and Debian), and I tend to agree with "long
enough" but "proven itself" feels bit too strong a statement.
I think defaulting to autodetect is a good change. I do not like to add
new output to stderr deep in -rc, though.
Can we park this in 'next' and move it to 'master' after 1.6.1?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] autodetect number of CPUs by default when using threads
2008-12-11 22:25 ` Junio C Hamano
@ 2008-12-12 0:55 ` Nicolas Pitre
0 siblings, 0 replies; 11+ messages in thread
From: Nicolas Pitre @ 2008-12-12 0:55 UTC (permalink / raw)
To: Junio C Hamano; +Cc: git
On Thu, 11 Dec 2008, Junio C Hamano wrote:
> Nicolas Pitre <nico@cam.org> writes:
>
> > I've spent quite a while wondering why repacking in one repo was faster
> > than repacking in a clone of that repo on the same machine. So let's
> > display how many threads are actually used.
> >
> > We have comprehensive test in Makefile to determine if threads are
> > available, just to not use them by default. I think that code has
> > proven itself for long enough now not to let people benefit from it.
>
> Hmm, it does appear that distros compile with THREADED_DELTA_SEARCH turned
> on (I only checked Fedora and Debian), and I tend to agree with "long
> enough" but "proven itself" feels bit too strong a statement.
>
> I think defaulting to autodetect is a good change. I do not like to add
> new output to stderr deep in -rc, though.
>
> Can we park this in 'next' and move it to 'master' after 1.6.1?
Sure. This is probably best for next.
Nicolas
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] autodetect number of CPUs by default when using threads
2008-12-11 20:36 [PATCH] autodetect number of CPUs by default when using threads Nicolas Pitre
2008-12-11 22:25 ` Junio C Hamano
@ 2008-12-13 13:32 ` Jeff King
2008-12-13 19:27 ` Nicolas Pitre
2008-12-13 20:06 ` [PATCH] pack-objects: don't use too many threads with few objects Nicolas Pitre
1 sibling, 2 replies; 11+ messages in thread
From: Jeff King @ 2008-12-13 13:32 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, git
On Thu, Dec 11, 2008 at 03:36:47PM -0500, Nicolas Pitre wrote:
> ... and display the actual number of threads used when locally
> repacking. A remote server still won't tell you how many threads it
> uses during a fetch though.
Hrm. I have no idea how, but this patch reliably causes t5300 to fail on
my FreeBSD test box ("next" is broken, bisection pointed to 43cc2b42).
Sample verbose output is below.
-- >8 --
Initialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/
* expecting success: rm -f .git/index*
for i in a b c
do
dd if=/dev/zero bs=4k count=1 | perl -pe "y/\\000/$i/" >$i &&
git update-index --add $i || return 1
done &&
cat c >d && echo foo >>d && git update-index --add d &&
tree=`git write-tree` &&
commit=`git commit-tree $tree </dev/null` && {
echo $tree &&
echo $commit &&
git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\) .*/\\1/"
} >obj-list && {
git diff-tree --root -p $commit &&
while read object
do
t=`git cat-file -t $object` &&
git cat-file $t $object || return 1
done <obj-list
} >expect
1+0 records in
1+0 records out
4096 bytes transferred in 0.000059 secs (69273666 bytes/sec)
1+0 records in
1+0 records out
4096 bytes transferred in 0.000053 secs (77386798 bytes/sec)
1+0 records in
1+0 records out
4096 bytes transferred in 0.000055 secs (74695083 bytes/sec)
* ok 1: setup
* expecting success: packname_1=$(git pack-objects --window=0 test-1 <obj-list)
* ok 2: pack without delta
* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&
export GIT_OBJECT_DIRECTORY &&
git init &&
git unpack-objects -n <test-1-20f53d0edce6980dfa341c034ae365aef2616e8d.pack &&
git unpack-objects <test-1-20f53d0edce6980dfa341c034ae365aef2616e8d.pack
Reinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/
* ok 3: unpack without delta
* expecting success: (cd ../.git && find objects -type f -print) |
while read path
do
cmp $path ../.git/$path || {
echo $path differs.
return 1
}
done
* ok 4: check unpack without delta
* expecting success: pwd &&
packname_2=$(git pack-objects test-2 <obj-list)
/home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object
* ok 5: pack with REF_DELTA
* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&
export GIT_OBJECT_DIRECTORY &&
git init &&
git unpack-objects -n <test-2-${packname_2}.pack &&
git unpack-objects <test-2-${packname_2}.pack
Reinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/
* ok 6: unpack with REF_DELTA
* expecting success: (cd ../.git && find objects -type f -print) |
while read path
do
cmp $path ../.git/$path || {
echo $path differs.
return 1
}
done
* ok 7: check unpack with REF_DELTA
* expecting success: pwd &&
packname_3=$(git pack-objects --delta-base-offset test-3 <obj-list)
/home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object
* ok 8: pack with OFS_DELTA
* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&
export GIT_OBJECT_DIRECTORY &&
git init &&
git unpack-objects -n <test-3-${packname_3}.pack &&
git unpack-objects <test-3-${packname_3}.pack
Reinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/
* ok 9: unpack with OFS_DELTA
* expecting success: (cd ../.git && find objects -type f -print) |
while read path
do
cmp $path ../.git/$path || {
echo $path differs.
return 1
}
done
* ok 10: check unpack with OFS_DELTA
* expecting success:
perl -e '
defined($_ = -s $_) or die for @ARGV;
exit 1 if $ARGV[0] <= $ARGV[1];
' test-2-$packname_2.pack test-3-$packname_3.pack
* FAIL 11: compare delta flavors
perl -e '
defined($_ = -s $_) or die for @ARGV;
exit 1 if $ARGV[0] <= $ARGV[1];
' test-2-$packname_2.pack test-3-$packname_3.pack
* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&
export GIT_OBJECT_DIRECTORY &&
git init &&
cp test-1-${packname_1}.pack test-1-${packname_1}.idx .git2/objects/pack && {
git diff-tree --root -p $commit &&
while read object
do
t=`git cat-file -t $object` &&
git cat-file $t $object || return 1
done <obj-list
} >current &&
diff expect current
Reinitialized existing Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/.git/
* ok 12: use packed objects
* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&
export GIT_OBJECT_DIRECTORY &&
rm -f .git2/objects/pack/test-* &&
cp test-2-${packname_2}.pack test-2-${packname_2}.idx .git2/objects/pack && {
git diff-tree --root -p $commit &&
while read object
do
t=`git cat-file -t $object` &&
git cat-file $t $object || return 1
done <obj-list
} >current &&
diff expect current
* ok 13: use packed deltified (REF_DELTA) objects
* expecting success: GIT_OBJECT_DIRECTORY=.git2/objects &&
export GIT_OBJECT_DIRECTORY &&
rm -f .git2/objects/pack/test-* &&
cp test-3-${packname_3}.pack test-3-${packname_3}.idx .git2/objects/pack && {
git diff-tree --root -p $commit &&
while read object
do
t=`git cat-file -t $object` &&
git cat-file $t $object || return 1
done <obj-list
} >current &&
diff expect current
* ok 14: use packed deltified (OFS_DELTA) objects
* expecting success: git verify-pack test-1-${packname_1}.idx \
test-2-${packname_2}.idx \
test-3-${packname_3}.idx
* ok 15: verify pack
* expecting success: git verify-pack -v test-1-${packname_1}.idx \
test-2-${packname_2}.idx \
test-3-${packname_3}.idx
0be779221aca65277fd447c8207e1b3c2706ae20 blob 4096 31 311
2f22df7b2a002e7e84bd7c124483f0df7f7bc1ef commit 163 121 128
890b180ae96f7abd6a1917dee506bceb188003b6 tree 116 116 12
9d235ed07cd19811a6ceb342de82f190e49c9f68 blob 4096 31 249
b010fe5253f7dc59c6605dacb92fcea00d199d4e blob 4100 36 342
c82de19312b6c3695c0c18f70709a6c535682a67 blob 4096 31 280
test-1-20f53d0edce6980dfa341c034ae365aef2616e8d.pack: ok
0be779221aca65277fd447c8207e1b3c2706ae20 blob 4096 31 311
2f22df7b2a002e7e84bd7c124483f0df7f7bc1ef commit 163 121 128
890b180ae96f7abd6a1917dee506bceb188003b6 tree 116 116 12
9d235ed07cd19811a6ceb342de82f190e49c9f68 blob 4096 31 249
b010fe5253f7dc59c6605dacb92fcea00d199d4e blob 4100 36 342
c82de19312b6c3695c0c18f70709a6c535682a67 blob 4096 31 280
test-2-20f53d0edce6980dfa341c034ae365aef2616e8d.pack: ok
0be779221aca65277fd447c8207e1b3c2706ae20 blob 4096 31 311
2f22df7b2a002e7e84bd7c124483f0df7f7bc1ef commit 163 121 128
890b180ae96f7abd6a1917dee506bceb188003b6 tree 116 116 12
9d235ed07cd19811a6ceb342de82f190e49c9f68 blob 4096 31 249
b010fe5253f7dc59c6605dacb92fcea00d199d4e blob 4100 36 342
c82de19312b6c3695c0c18f70709a6c535682a67 blob 4096 31 280
test-3-20f53d0edce6980dfa341c034ae365aef2616e8d.pack: ok
* ok 16: verify pack -v
* expecting success: cat test-1-${packname_1}.idx >test-3.idx &&
cat test-2-${packname_2}.pack >test-3.pack &&
if git verify-pack test-3.idx
then false
else :;
fi
* FAIL 17: verify-pack catches mismatched .idx and .pack files
cat test-1-${packname_1}.idx >test-3.idx &&
cat test-2-${packname_2}.pack >test-3.pack &&
if git verify-pack test-3.idx
then false
else :;
fi
* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&
dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=2 &&
if git verify-pack test-3.idx
then false
else :;
fi
1+0 records in
1+0 records out
1 bytes transferred in 0.000042 secs (23831 bytes/sec)
error: file test-3.pack is not a GIT packfile
fatal: packfile test-3.pack cannot be accessed
* ok 18: verify-pack catches a corrupted pack signature
* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&
dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=7 &&
if git verify-pack test-3.idx
then false
else :;
fi
1+0 records in
1+0 records out
1 bytes transferred in 0.000042 secs (23831 bytes/sec)
error: packfile test-3.pack is version 0 and not supported (try upgrading GIT to a newer version)
fatal: packfile test-3.pack cannot be accessed
* ok 19: verify-pack catches a corrupted pack version
* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&
dd if=/dev/zero of=test-3.pack count=1 bs=1 conv=notrunc seek=12 &&
if git verify-pack test-3.idx
then false
else :;
fi
1+0 records in
1+0 records out
1 bytes transferred in 0.000046 secs (21732 bytes/sec)
error: test-3.pack SHA1 checksum mismatch
error: index CRC mismatch for object 890b180ae96f7abd6a1917dee506bceb188003b6 from test-3.pack at offset 12
error: unknown object type 0 at offset 12 in test-3.pack
error: cannot unpack 890b180ae96f7abd6a1917dee506bceb188003b6 from test-3.pack at offset 12
* ok 20: verify-pack catches a corrupted type/size of the 1st packed object data
* expecting success: l=`wc -c <test-3.idx` &&
l=`expr $l - 20` &&
cat test-1-${packname_1}.pack >test-3.pack &&
dd if=/dev/zero of=test-3.idx count=20 bs=1 conv=notrunc seek=$l &&
if git verify-pack test-3.pack
then false
else :;
fi
20+0 records in
20+0 records out
20 bytes transferred in 0.000123 secs (162570 bytes/sec)
error: Packfile index for test-3.pack SHA1 mismatch
* ok 21: verify-pack catches a corrupted sum of the index file itself
* expecting success: cat test-1-${packname_1}.pack >test-3.pack &&
git index-pack -o tmp.idx test-3.pack &&
cmp tmp.idx test-1-${packname_1}.idx &&
git index-pack test-3.pack &&
cmp test-3.idx test-1-${packname_1}.idx &&
cat test-2-${packname_2}.pack >test-3.pack &&
git index-pack -o tmp.idx test-2-${packname_2}.pack &&
cmp tmp.idx test-2-${packname_2}.idx &&
git index-pack test-3.pack &&
cmp test-3.idx test-2-${packname_2}.idx &&
cat test-3-${packname_3}.pack >test-3.pack &&
git index-pack -o tmp.idx test-3-${packname_3}.pack &&
cmp tmp.idx test-3-${packname_3}.idx &&
git index-pack test-3.pack &&
cmp test-3.idx test-3-${packname_3}.idx &&
:
20f53d0edce6980dfa341c034ae365aef2616e8d
20f53d0edce6980dfa341c034ae365aef2616e8d
20f53d0edce6980dfa341c034ae365aef2616e8d
20f53d0edce6980dfa341c034ae365aef2616e8d
20f53d0edce6980dfa341c034ae365aef2616e8d
20f53d0edce6980dfa341c034ae365aef2616e8d
* ok 22: build pack index for an existing pack
* expecting success: test -f .git/objects/c8/2de19312b6c3695c0c18f70709a6c535682a67 &&
cp -f .git/objects/9d/235ed07cd19811a6ceb342de82f190e49c9f68 \
.git/objects/c8/2de19312b6c3695c0c18f70709a6c535682a67
* ok 23: fake a SHA1 hash collision
* expecting success: test_must_fail git index-pack -o bad.idx test-3.pack 2>msg &&
grep "SHA1 COLLISION FOUND" msg
fatal: SHA1 COLLISION FOUND WITH c82de19312b6c3695c0c18f70709a6c535682a67 !
* ok 24: make sure index-pack detects the SHA1 collision
* expecting success: git config pack.packSizeLimit 200 &&
packname_4=$(git pack-objects test-4 <obj-list) &&
test 3 = $(ls test-4-*.pack | wc -l)
* ok 25: honor pack.packSizeLimit
* expecting success:
git config --unset pack.packsizelimit &&
for j in a b c d e f g
do
for i in 0 1 2 3 4 5 6 7 8 9
do
o=$(echo $j$i | git hash-object -w --stdin) &&
echo "100644 $o 0 $j$i"
done
done >LIST &&
rm -f .git/index &&
git update-index --index-info <LIST &&
LIST=$(git write-tree) &&
rm -f .git/index &&
head -n 10 LIST | git update-index --index-info &&
LI=$(git write-tree) &&
rm -f .git/index &&
tail -n 10 LIST | git update-index --index-info &&
ST=$(git write-tree) &&
PACK5=$( git rev-list --objects "$LIST" "$LI" "$ST" | \
git pack-objects test-5 ) &&
PACK6=$( (
echo "$LIST"
echo "$LI"
echo "$ST"
) | git pack-objects test-6 ) &&
test_create_repo test-5 &&
(
cd test-5 &&
git unpack-objects --strict <../test-5-$PACK5.pack &&
git ls-tree -r $LIST &&
git ls-tree -r $LI &&
git ls-tree -r $ST
) &&
test_create_repo test-6 &&
(
# tree-only into empty repo -- many unreachables
cd test-6 &&
test_must_fail git unpack-objects --strict <../test-6-$PACK6.pack
) &&
(
# already populated -- no unreachables
cd test-5 &&
git unpack-objects --strict <../test-6-$PACK6.pack
)
Initialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-5/.git/
100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02 0 a0
100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c 0 a1
100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c 0 a2
100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3 0 a3
100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1 0 a4
100644 blob fafec68b313bde633804c37b46810f136ee12ea6 0 a5
100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f 0 a6
100644 blob 6bac181dd20780870c30d69308f5a90b59a15102 0 a7
100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d 0 a8
100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc 0 a9
100644 blob 2270a80fbb71d7109b85e7489f3f307d0141a559 0 b0
100644 blob c9c6af7f78bc47490dbf3e822cf2f3c24d4b9061 0 b1
100644 blob e6bfff5c1d0f0ecd501552b43a1e13d8008abc31 0 b2
100644 blob 6d0875c82a99774922eab07fbc08510fc85d9e0a 0 b3
100644 blob 8e953e84d803f13fd06416a1bd8161dcd93cfd00 0 b4
100644 blob 90a5159bf020296276ea5ca1bcd292a9b1de9947 0 b5
100644 blob 07eb61d36f49569a2b0649af299f9f00013d0969 0 b6
100644 blob 6e6d33cfc7b1581413a88ae0759202b25ac4cad1 0 b7
100644 blob 5d453c66b522d8de5d99c03bd2bf8b65ca0bcf33 0 b8
100644 blob 9a16e0d1a643983973291127012d2694a0b17fe8 0 b9
100644 blob caecf05cdbb03e144f113ecab2b99e5ee74df706 0 c0
100644 blob ae9304576a6ec3419b231b2b9c8e33a06f97f9fb 0 c1
100644 blob 16f9ec009e5568c435f473ba3a1df732d49ce8c3 0 c2
100644 blob 0771aea884dd394a7b12783d049f05b5599f41a4 0 c3
100644 blob a103f673dd1b7c5727aa0ae0100419adc50e1b76 0 c4
100644 blob c36357109ce20af8f90df3c0dd0784e89408d707 0 c5
100644 blob 86a716504cbdc40a135faa5ab199d15e364c416d 0 c6
100644 blob 20be68787e59f4cb8983b641d170c4baf8ea25dd 0 c7
100644 blob bb420bc8fb9e91530fdc25e363f6b822d5195309 0 c8
100644 blob f899bd1761a5ca5978799bc3189a04d3c52d8435 0 c9
100644 blob 79a5e858be4a3a969e3b71406fef0d3d36e1d387 0 d0
100644 blob 6f1852975b9306ae5d8dfdf0d4cb1f5cb36ac229 0 d1
100644 blob fd3671590780b645e1bef030d550191f6cdf1c95 0 d2
100644 blob 69c77a9ea746edd27b46107142fc2c5288c1daf5 0 d3
100644 blob d7b30ee07d4b9819a77a3b122282b4c04528ec21 0 d4
100644 blob fa590f3f8bfbef6095cf6525e8497b2b2b46445c 0 d5
100644 blob fef4390d19cb98fa97efd44c09d4e2eb7b084b40 0 d6
100644 blob a5adf29249757619b8890c86f2cfcfeee611b5d8 0 d7
100644 blob 1859919d049c078ba63fcecbce60e697c6da8e01 0 d8
100644 blob b6148c82443a60d1d5ce07872a85e3c544b5f4b0 0 d9
100644 blob 4fe4106b50b257f3d6ffd23b750d2d1945c2608b 0 e0
100644 blob 5ff91639494693b1f238b5dc9baf8be95142c3ab 0 e1
100644 blob 3811af3ca744c2fb44077a8025c23b4d4166a449 0 e2
100644 blob 9338c3fa1f74d202651154257572d32aed57cc16 0 e3
100644 blob c5195815f83aa9ef1ebac9e11f14d62b26406c99 0 e4
100644 blob 0c44c89ef9c29077cceebc6dd9ca0b3a950c95ee 0 e5
100644 blob f17172bc54cfa55a9e531cc845dff39855cd8df0 0 e6
100644 blob 29870e9ccff84f86938ee588a47b39cf3a802e6b 0 e7
100644 blob e8d01d007452cede5c6eafc3343fc80675b1e2f9 0 e8
100644 blob 0c1945ad576ffc445ad69a3fafd1d3c04cc0bd40 0 e9
100644 blob 844dc808a1d3d769ddcf5430eac933d4d87ff606 0 f0
100644 blob 8e1e71d5ce34c01b6fe83bc5051545f2918c8c2b 0 f1
100644 blob 9de77c18733ab8009a956c25e28c85fe203a17d7 0 f2
100644 blob 45d9e0e9fc8859787c33081dffdf12f41b54fcf3 0 f3
100644 blob c48ac4847d3abae8e5899dcc5cf490a98df55755 0 f4
100644 blob 14c61ecf25961ce674048103c5fa5ea3c535a5cd 0 f5
100644 blob 59ce90029dfd9e224701a040a26153e9f2feb74c 0 f6
100644 blob 0a7bd52ffb819609cf65f40eb91be62000db47e5 0 f7
100644 blob c7e617f6b429a1f3b462df6add2b4ea2cf702faf 0 f8
100644 blob a720d8c8d277ad31ee5b180bef57379b5f62517c 0 f9
100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1 0 g0
100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126 0 g1
100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677 0 g2
100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da 0 g3
100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd 0 g4
100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039 0 g5
100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf 0 g6
100644 blob e19089689a05907b359f52c972ee7d2294fa96ab 0 g7
100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d 0 g8
100644 blob 09827d7e43d81089e868e9f0f06502b865939c05 0 g9
100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02 0 a0
100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c 0 a1
100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c 0 a2
100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3 0 a3
100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1 0 a4
100644 blob fafec68b313bde633804c37b46810f136ee12ea6 0 a5
100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f 0 a6
100644 blob 6bac181dd20780870c30d69308f5a90b59a15102 0 a7
100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d 0 a8
100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc 0 a9
100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1 0 g0
100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126 0 g1
100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677 0 g2
100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da 0 g3
100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd 0 g4
100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039 0 g5
100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf 0 g6
100644 blob e19089689a05907b359f52c972ee7d2294fa96ab 0 g7
100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d 0 g8
100644 blob 09827d7e43d81089e868e9f0f06502b865939c05 0 g9
Initialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-6/.git/
error: unable to find 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02
fatal: object of unexpected type
* ok 26: unpacking with --strict
* expecting success:
for j in a b c d e f g
do
for i in 0 1 2 3 4 5 6 7 8 9
do
o=$(echo $j$i | git hash-object -w --stdin) &&
echo "100644 $o 0 $j$i"
done
done >LIST &&
rm -f .git/index &&
git update-index --index-info <LIST &&
LIST=$(git write-tree) &&
rm -f .git/index &&
head -n 10 LIST | git update-index --index-info &&
LI=$(git write-tree) &&
rm -f .git/index &&
tail -n 10 LIST | git update-index --index-info &&
ST=$(git write-tree) &&
PACK5=$( git rev-list --objects "$LIST" "$LI" "$ST" | \
git pack-objects test-5 ) &&
PACK6=$( (
echo "$LIST"
echo "$LI"
echo "$ST"
) | git pack-objects test-6 ) &&
test_create_repo test-7 &&
(
cd test-7 &&
git index-pack --strict --stdin <../test-5-$PACK5.pack &&
git ls-tree -r $LIST &&
git ls-tree -r $LI &&
git ls-tree -r $ST
) &&
test_create_repo test-8 &&
(
# tree-only into empty repo -- many unreachables
cd test-8 &&
test_must_fail git index-pack --strict --stdin <../test-6-$PACK6.pack
) &&
(
# already populated -- no unreachables
cd test-7 &&
git index-pack --strict --stdin <../test-6-$PACK6.pack
)
Initialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-7/.git/
pack df7e2e394127cb487b2a6904a9a3fa45be4d4199
100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02 0 a0
100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c 0 a1
100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c 0 a2
100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3 0 a3
100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1 0 a4
100644 blob fafec68b313bde633804c37b46810f136ee12ea6 0 a5
100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f 0 a6
100644 blob 6bac181dd20780870c30d69308f5a90b59a15102 0 a7
100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d 0 a8
100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc 0 a9
100644 blob 2270a80fbb71d7109b85e7489f3f307d0141a559 0 b0
100644 blob c9c6af7f78bc47490dbf3e822cf2f3c24d4b9061 0 b1
100644 blob e6bfff5c1d0f0ecd501552b43a1e13d8008abc31 0 b2
100644 blob 6d0875c82a99774922eab07fbc08510fc85d9e0a 0 b3
100644 blob 8e953e84d803f13fd06416a1bd8161dcd93cfd00 0 b4
100644 blob 90a5159bf020296276ea5ca1bcd292a9b1de9947 0 b5
100644 blob 07eb61d36f49569a2b0649af299f9f00013d0969 0 b6
100644 blob 6e6d33cfc7b1581413a88ae0759202b25ac4cad1 0 b7
100644 blob 5d453c66b522d8de5d99c03bd2bf8b65ca0bcf33 0 b8
100644 blob 9a16e0d1a643983973291127012d2694a0b17fe8 0 b9
100644 blob caecf05cdbb03e144f113ecab2b99e5ee74df706 0 c0
100644 blob ae9304576a6ec3419b231b2b9c8e33a06f97f9fb 0 c1
100644 blob 16f9ec009e5568c435f473ba3a1df732d49ce8c3 0 c2
100644 blob 0771aea884dd394a7b12783d049f05b5599f41a4 0 c3
100644 blob a103f673dd1b7c5727aa0ae0100419adc50e1b76 0 c4
100644 blob c36357109ce20af8f90df3c0dd0784e89408d707 0 c5
100644 blob 86a716504cbdc40a135faa5ab199d15e364c416d 0 c6
100644 blob 20be68787e59f4cb8983b641d170c4baf8ea25dd 0 c7
100644 blob bb420bc8fb9e91530fdc25e363f6b822d5195309 0 c8
100644 blob f899bd1761a5ca5978799bc3189a04d3c52d8435 0 c9
100644 blob 79a5e858be4a3a969e3b71406fef0d3d36e1d387 0 d0
100644 blob 6f1852975b9306ae5d8dfdf0d4cb1f5cb36ac229 0 d1
100644 blob fd3671590780b645e1bef030d550191f6cdf1c95 0 d2
100644 blob 69c77a9ea746edd27b46107142fc2c5288c1daf5 0 d3
100644 blob d7b30ee07d4b9819a77a3b122282b4c04528ec21 0 d4
100644 blob fa590f3f8bfbef6095cf6525e8497b2b2b46445c 0 d5
100644 blob fef4390d19cb98fa97efd44c09d4e2eb7b084b40 0 d6
100644 blob a5adf29249757619b8890c86f2cfcfeee611b5d8 0 d7
100644 blob 1859919d049c078ba63fcecbce60e697c6da8e01 0 d8
100644 blob b6148c82443a60d1d5ce07872a85e3c544b5f4b0 0 d9
100644 blob 4fe4106b50b257f3d6ffd23b750d2d1945c2608b 0 e0
100644 blob 5ff91639494693b1f238b5dc9baf8be95142c3ab 0 e1
100644 blob 3811af3ca744c2fb44077a8025c23b4d4166a449 0 e2
100644 blob 9338c3fa1f74d202651154257572d32aed57cc16 0 e3
100644 blob c5195815f83aa9ef1ebac9e11f14d62b26406c99 0 e4
100644 blob 0c44c89ef9c29077cceebc6dd9ca0b3a950c95ee 0 e5
100644 blob f17172bc54cfa55a9e531cc845dff39855cd8df0 0 e6
100644 blob 29870e9ccff84f86938ee588a47b39cf3a802e6b 0 e7
100644 blob e8d01d007452cede5c6eafc3343fc80675b1e2f9 0 e8
100644 blob 0c1945ad576ffc445ad69a3fafd1d3c04cc0bd40 0 e9
100644 blob 844dc808a1d3d769ddcf5430eac933d4d87ff606 0 f0
100644 blob 8e1e71d5ce34c01b6fe83bc5051545f2918c8c2b 0 f1
100644 blob 9de77c18733ab8009a956c25e28c85fe203a17d7 0 f2
100644 blob 45d9e0e9fc8859787c33081dffdf12f41b54fcf3 0 f3
100644 blob c48ac4847d3abae8e5899dcc5cf490a98df55755 0 f4
100644 blob 14c61ecf25961ce674048103c5fa5ea3c535a5cd 0 f5
100644 blob 59ce90029dfd9e224701a040a26153e9f2feb74c 0 f6
100644 blob 0a7bd52ffb819609cf65f40eb91be62000db47e5 0 f7
100644 blob c7e617f6b429a1f3b462df6add2b4ea2cf702faf 0 f8
100644 blob a720d8c8d277ad31ee5b180bef57379b5f62517c 0 f9
100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1 0 g0
100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126 0 g1
100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677 0 g2
100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da 0 g3
100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd 0 g4
100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039 0 g5
100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf 0 g6
100644 blob e19089689a05907b359f52c972ee7d2294fa96ab 0 g7
100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d 0 g8
100644 blob 09827d7e43d81089e868e9f0f06502b865939c05 0 g9
100644 blob 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02 0 a0
100644 blob da0f8ed91a8f2f0f067b3bdf26265d5ca48cf82c 0 a1
100644 blob c1827f07e114c20547dc6a7296588870a4b5b62c 0 a2
100644 blob d616f7380ad325123fed6f628d02fa76e1ce77c3 0 a3
100644 blob 88ba23dca8c8529f8165539c369925a99391a4d1 0 a4
100644 blob fafec68b313bde633804c37b46810f136ee12ea6 0 a5
100644 blob 5b521d0587015867ac1a23fa00be3f3fce080b9f 0 a6
100644 blob 6bac181dd20780870c30d69308f5a90b59a15102 0 a7
100644 blob f1914bb68f8515ba7d1ab0f0cddcdf960731773d 0 a8
100644 blob dbe2bc1151e5d5d469644f20189f923f006ed0cc 0 a9
100644 blob 2e7d2f0b106eb8823e449a020497e26b86dc3eb1 0 g0
100644 blob d8a17fff13638d804e8bf7f9f357c174db98f126 0 g1
100644 blob 247c4abd244a429297a4c4b091592ae13bbf4677 0 g2
100644 blob affaa5ba8c43cfc71469226e0d57cb6823c843da 0 g3
100644 blob 51bc00f93e60aeef7dccfcf646e3f615056112bd 0 g4
100644 blob 02b4fdd23d495769852f1550c10eb015f0fe0039 0 g5
100644 blob e65aa9d64b596da2ba61e0662b1173f1e16dcbcf 0 g6
100644 blob e19089689a05907b359f52c972ee7d2294fa96ab 0 g7
100644 blob 30c2af9542d38e1f248390632612e0fb944fd27d 0 g8
100644 blob 09827d7e43d81089e868e9f0f06502b865939c05 0 g9
Initialized empty Git repository in /home/peff/compile/gitbuild/jk/freebsd/project/t/trash directory.t5300-pack-object/test-8/.git/
error: unable to find 0042f6c56d8fc1896f3efc2cdc5060e5b5e44e02
fatal: object of unexpected type
pack f67e682280757d6079b87511ea590015d0e26770
* ok 27: index-pack with --strict
* expecting success:
git config pack.packSizeLimit 2 &&
packname_9=$(git pack-objects test-9 <obj-list) &&
test $(wc -l <obj-list) = $(ls test-9-*.pack | wc -l)
* ok 28: tolerate absurdly small packsizelimit
* failed 2 among 28 test(s)
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] autodetect number of CPUs by default when using threads
2008-12-13 13:32 ` Jeff King
@ 2008-12-13 19:27 ` Nicolas Pitre
2008-12-13 20:06 ` [PATCH] pack-objects: don't use too many threads with few objects Nicolas Pitre
1 sibling, 0 replies; 11+ messages in thread
From: Nicolas Pitre @ 2008-12-13 19:27 UTC (permalink / raw)
To: Jeff King; +Cc: Junio C Hamano, git
On Sat, 13 Dec 2008, Jeff King wrote:
> On Thu, Dec 11, 2008 at 03:36:47PM -0500, Nicolas Pitre wrote:
>
> > ... and display the actual number of threads used when locally
> > repacking. A remote server still won't tell you how many threads it
> > uses during a fetch though.
>
> Hrm. I have no idea how, but this patch reliably causes t5300 to fail on
> my FreeBSD test box ("next" is broken, bisection pointed to 43cc2b42).
> Sample verbose output is below.
Hmmm... Interesting.
Nicolas
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH] pack-objects: don't use too many threads with few objects
2008-12-13 13:32 ` Jeff King
2008-12-13 19:27 ` Nicolas Pitre
@ 2008-12-13 20:06 ` Nicolas Pitre
2008-12-14 2:20 ` Jeff King
2008-12-15 18:38 ` Mike Ralphson
1 sibling, 2 replies; 11+ messages in thread
From: Nicolas Pitre @ 2008-12-13 20:06 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Jeff King, git
If there are few objects to deltify, they might be split amongst threads
so that there is simply no other objects left to delta against within
the same thread. Let's use the same 2*window treshold as used for the
final load balancing to allow extra threads to be created.
This fixes the benign t5300 test failure.
Signed-off-by: Nicolas Pitre <nico@cam.org>
---
On Sat, 13 Dec 2008, Jeff King wrote:
> On Thu, Dec 11, 2008 at 03:36:47PM -0500, Nicolas Pitre wrote:
>
> > ... and display the actual number of threads used when locally
> > repacking. A remote server still won't tell you how many threads it
> > uses during a fetch though.
>
> Hrm. I have no idea how, but this patch reliably causes t5300 to fail on
> my FreeBSD test box ("next" is broken, bisection pointed to 43cc2b42).
diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c
index 619e597..e851534 100644
--- a/builtin-pack-objects.c
+++ b/builtin-pack-objects.c
@@ -1620,6 +1620,10 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size,
for (i = 0; i < delta_search_threads; i++) {
unsigned sub_size = list_size / (delta_search_threads - i);
+ /* don't use too small segments or no deltas will be found */
+ if (sub_size < 2*window && i+1 < delta_search_threads)
+ sub_size = 0;
+
p[i].window = window;
p[i].depth = depth;
p[i].processed = processed;
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] pack-objects: don't use too many threads with few objects
2008-12-13 20:06 ` [PATCH] pack-objects: don't use too many threads with few objects Nicolas Pitre
@ 2008-12-14 2:20 ` Jeff King
2008-12-15 18:38 ` Mike Ralphson
1 sibling, 0 replies; 11+ messages in thread
From: Jeff King @ 2008-12-14 2:20 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, git
On Sat, Dec 13, 2008 at 03:06:40PM -0500, Nicolas Pitre wrote:
> If there are few objects to deltify, they might be split amongst threads
> so that there is simply no other objects left to delta against within
> the same thread. Let's use the same 2*window treshold as used for the
> final load balancing to allow extra threads to be created.
>
> This fixes the benign t5300 test failure.
I can confirm this fixes my t5300 failure. Thanks.
Tested-by: Jeff King <peff@peff.net>
-Peff
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] pack-objects: don't use too many threads with few objects
2008-12-13 20:06 ` [PATCH] pack-objects: don't use too many threads with few objects Nicolas Pitre
2008-12-14 2:20 ` Jeff King
@ 2008-12-15 18:38 ` Mike Ralphson
2008-12-16 1:15 ` Junio C Hamano
1 sibling, 1 reply; 11+ messages in thread
From: Mike Ralphson @ 2008-12-15 18:38 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Junio C Hamano, Jeff King, git
2008/12/13 Nicolas Pitre <nico@cam.org>
>
> If there are few objects to deltify, they might be split amongst threads
> so that there is simply no other objects left to delta against within
> the same thread. Let's use the same 2*window treshold as used for the
> final load balancing to allow extra threads to be created.
>
> This fixes the benign t5300 test failure.
Even with this I'm seeing failures in t5302 which I think are probably
related to 43cc2b42
I was trying to bisect it (test runs take forever on this platform -
AIX 5.3) but I hit the t5300 failure, which I hadn't seen on the last
automatic build.
Please shout if I've snipped the vital lines (...)
Initialized empty Git repository in /usr/local/src/gitbuild/t/trash
directory.t5302-pack-index/.git/
* expecting success: rm -rf .git
git init &&
i=1 &&
while test $i -le 100
do
iii=`printf %03i $i`
test-genrandom "bar" 200 > wide_delta_$iii &&
test-genrandom "baz $iii" 50 >> wide_delta_$iii &&
test-genrandom "foo"$i 100 > deep_delta_$iii &&
test-genrandom "foo"`expr $i + 1` 100 >> deep_delta_$iii &&
test-genrandom "foo"`expr $i + 2` 100 >> deep_delta_$iii &&
echo $iii >file_$iii &&
test-genrandom "$iii" 8192 >>file_$iii &&
git update-index --add file_$iii deep_delta_$iii wide_delta_$iii &&
i=`expr $i + 1` || return 1
done &&
{ echo 101 && test-genrandom 100 8192; } >file_101 &&
git update-index --add file_101 &&
tree=`git write-tree` &&
commit=`git commit-tree $tree </dev/null` && {
echo $tree &&
git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\) .*/\\1/"
} >obj-list &&
git update-ref HEAD $commit
Initialized empty Git repository in /usr/local/src/gitbuild/t/trash
directory.t5302-pack-index/.git/
* ok 1: setup
* expecting success: pack1=$(git pack-objects --index-version=1 test-1
<obj-list) &&
git verify-pack -v "test-1-${pack1}.pack"
0106e17481932f5c223fafadc1d26abc6adf40d6 blob 57 90 850703 1
69e0b8ef8cda369575b6801c6ed47daf09aa3c62
...
fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob 8196 8210 187576
chain length = 1: 110 objects
chain length = 2: 11 objects
chain length = 3: 11 objects
chain length = 4: 10 objects
chain length = 5: 9 objects
chain length = 6: 9 objects
chain length = 7: 9 objects
chain length = 8: 9 objects
chain length = 9: 9 objects
test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok
* ok 2: pack-objects with index version 1
* expecting success: pack2=$(git pack-objects --index-version=2 test-2
<obj-list) &&
git verify-pack -v "test-2-${pack2}.pack"
0106e17481932f5c223fafadc1d26abc6adf40d6 blob 57 90 850531 1
69e0b8ef8cda369575b6801c6ed47daf09aa3c62
...
fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob 8196 8210 187404
chain length = 1: 109 objects
chain length = 2: 10 objects
chain length = 3: 10 objects
chain length = 4: 10 objects
chain length = 5: 10 objects
chain length = 6: 10 objects
chain length = 7: 10 objects
chain length = 8: 10 objects
chain length = 9: 9 objects
test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok
* ok 3: pack-objects with index version 2
* expecting success: cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack differ: char
10236, line 32
* FAIL 4: both packs should be identical
cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
* expecting success: ! cmp "test-1-${pack1}.idx" "test-2-${pack2}.idx"
test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx
test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx differ: char 1,
line 1
* ok 5: index v1 and index v2 should be different
* expecting success: git index-pack --index-version=1 -o 1.idx
"test-1-${pack1}.pack"
9108b6dfd43bf36ccbfc2839fe62d1503bf84292
* ok 6: index-pack with index version 1
* expecting success: git index-pack --index-version=2 -o 2.idx
"test-1-${pack1}.pack"
9108b6dfd43bf36ccbfc2839fe62d1503bf84292
* ok 7: index-pack with index version 2
* expecting success: cmp "test-1-${pack1}.idx" "1.idx" &&
cmp "test-2-${pack2}.idx" "2.idx"
test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx 2.idx differ: char
7273, line 23
* FAIL 8: index-pack results should match pack-objects ones
cmp "test-1-${pack1}.idx" "1.idx" &&
cmp "test-2-${pack2}.idx" "2.idx"
* expecting success: pack3=$(git pack-objects
--index-version=2,0x40000 test-3 <obj-list)
* ok 9: index v2: force some 64-bit offsets with pack-objects
* expecting success: git verify-pack -v "test-3-${pack3}.pack"
0106e17481932f5c223fafadc1d26abc6adf40d6 blob 57 90 850531 1
69e0b8ef8cda369575b6801c6ed47daf09aa3c62
...
fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob 8196 8210 187404
chain length = 1: 109 objects
chain length = 2: 10 objects
chain length = 3: 10 objects
chain length = 4: 10 objects
chain length = 5: 10 objects
chain length = 6: 10 objects
chain length = 7: 10 objects
chain length = 8: 10 objects
chain length = 9: 9 objects
test-3-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok
* ok 10: index v2: verify a pack with some 64-bit offsets
* expecting success: ! cmp "test-2-${pack2}.idx" "test-3-${pack3}.idx"
test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx
test-3-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx differ: char 8281,
line 24
* ok 11: 64-bit offsets: should be different from previous index v2 results
* expecting success: git index-pack --index-version=2,0x40000 -o 3.idx
"test-1-${pack1}.pack"
9108b6dfd43bf36ccbfc2839fe62d1503bf84292
* ok 12: index v2: force some 64-bit offsets with index-pack
* expecting success: cmp "test-3-${pack3}.idx" "3.idx"
test-3-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx 3.idx differ: char
7273, line 23
* FAIL 13: 64-bit offsets: index-pack result should match pack-objects one
cmp "test-3-${pack3}.idx" "3.idx"
* expecting success: git index-pack --index-version=1 --stdin <
"test-1-${pack1}.pack" &&
git prune-packed &&
git count-objects | ( read nr rest && test "$nr" -eq 1 ) &&
cmp "test-1-${pack1}.pack" ".git/objects/pack/pack-${pack1}.pack" &&
cmp "test-1-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
pack 9108b6dfd43bf36ccbfc2839fe62d1503bf84292
Removing duplicate objects: 25% (65/256)
Removing duplicate objects: 26% (68/256)
Removing duplicate objects: 27% (71/256)
Removing duplicate objects: 28% (74/256)
Removing duplicate objects: 30% (79/256)
Removing duplicate objects: 31% (81/256)
Removing duplicate objects: 32% (83/256)
Removing duplicate objects: 33% (85/256)
Removing duplicate objects: 34% (88/256)
Removing duplicate objects: 35% (90/256)
Removing duplicate objects: 36% (93/256)
Removing duplicate objects: 37% (95/256)
Removing duplicate objects: 38% (98/256)
Removing duplicate objects: 39% (100/256)
Removing duplicate objects: 40% (103/256)
Removing duplicate objects: 41% (106/256)
Removing duplicate objects: 42% (108/256)
Removing duplicate objects: 43% (112/256)
Removing duplicate objects: 44% (114/256)
Removing duplicate objects: 45% (116/256)
Removing duplicate objects: 46% (118/256)
Removing duplicate objects: 47% (121/256)
Removing duplicate objects: 48% (124/256)
Removing duplicate objects: 50% (130/256)
Removing duplicate objects: 51% (132/256)
Removing duplicate objects: 52% (134/256)
Removing duplicate objects: 53% (136/256)
Removing duplicate objects: 54% (140/256)
Removing duplicate objects: 55% (141/256)
Removing duplicate objects: 56% (144/256)
Removing duplicate objects: 57% (146/256)
Removing duplicate objects: 57% (147/256)
Removing duplicate objects: 58% (149/256)
Removing duplicate objects: 60% (154/256)
Removing duplicate objects: 61% (157/256)
Removing duplicate objects: 62% (159/256)
Removing duplicate objects: 63% (163/256)
Removing duplicate objects: 64% (164/256)
Removing duplicate objects: 65% (167/256)
Removing duplicate objects: 66% (169/256)
Removing duplicate objects: 67% (172/256)
Removing duplicate objects: 68% (175/256)
Removing duplicate objects: 69% (177/256)
Removing duplicate objects: 70% (180/256)
Removing duplicate objects: 71% (182/256)
Removing duplicate objects: 72% (185/256)
Removing duplicate objects: 73% (187/256)
Removing duplicate objects: 74% (190/256)
Removing duplicate objects: 75% (192/256)
Removing duplicate objects: 76% (195/256)
Removing duplicate objects: 77% (198/256)
Removing duplicate objects: 78% (201/256)
Removing duplicate objects: 79% (204/256)
Removing duplicate objects: 80% (206/256)
Removing duplicate objects: 81% (208/256)
Removing duplicate objects: 82% (210/256)
Removing duplicate objects: 83% (215/256)
Removing duplicate objects: 84% (217/256)
Removing duplicate objects: 85% (218/256)
Removing duplicate objects: 86% (222/256)
Removing duplicate objects: 87% (223/256)
Removing duplicate objects: 88% (226/256)
Removing duplicate objects: 89% (228/256)
Removing duplicate objects: 90% (231/256)
Removing duplicate objects: 91% (233/256)
Removing duplicate objects: 92% (236/256)
Removing duplicate objects: 93% (240/256)
Removing duplicate objects: 94% (241/256)
Removing duplicate objects: 95% (245/256)
Removing duplicate objects: 96% (246/256)
Removing duplicate objects: 97% (249/256)
Removing duplicate objects: 98% (251/256)
Removing duplicate objects: 99% (254/256)
Removing duplicate objects: 100% (256/256)
Removing duplicate objects: 100% (256/256), done.
* ok 14: [index v1] 1) stream pack to repository
* expecting success: # This test assumes file_101 is a delta smaller
than 16 bytes.
# It should be against file_100 but we substitute its base for file_099
sha1_101=`git hash-object file_101` &&
sha1_099=`git hash-object file_099` &&
offs_101=`index_obj_offset 1.idx $sha1_101` &&
nr_099=`index_obj_nr 1.idx $sha1_099` &&
chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
if=".git/objects/pack/pack-${pack1}.idx" \
skip=$((4 + 256 * 4 + $nr_099 * 24)) \
bs=1 count=20 conv=notrunc &&
git cat-file blob $sha1_101 > file_101_foo1
20+0 records in
20+0 records out
* ok 15: [index v1] 2) create a stealth corruption in a delta base reference
* expecting success: test -f file_101_foo1 && ! cmp file_101 file_101_foo1
file_101 file_101_foo1 differ: char 5, line 2
* ok 16: [index v1] 3) corrupted delta happily returned wrong data
* expecting success: test_must_fail git fsck --full $commit
error: .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
SHA1 checksum mismatch
error: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
is corrupt
error: sha1 mismatch 0b968294884af115eb5a1b941b3dabd48470c3bb
error: 0b968294884af115eb5a1b941b3dabd48470c3bb: object corrupt or missing
* ok 17: [index v1] 4) confirm that the pack is actually corrupted
* expecting success: pack4=$(git pack-objects test-4 <obj-list) &&
test -f "test-4-${pack1}.pack"
* ok 18: [index v1] 5) pack-objects happily reuses corrupted data
* expecting success: test_must_fail git verify-pack -v "test-4-${pack1}.pack"
error: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from
test-4-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack is corrupt
test-4-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: bad
* ok 19: [index v1] 6) newly created pack is BAD !
* expecting success: rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
git prune-packed &&
git count-objects | ( read nr rest && test "$nr" -eq 1 ) &&
cmp "test-1-${pack1}.pack" ".git/objects/pack/pack-${pack1}.pack" &&
cmp "test-2-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
pack 9108b6dfd43bf36ccbfc2839fe62d1503bf84292
test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.idx
differ: char 7273, line 23
* FAIL 20: [index v2] 1) stream pack to repository
rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
git prune-packed &&
git count-objects | ( read nr rest && test "$nr" -eq 1 ) &&
cmp "test-1-${pack1}.pack" ".git/objects/pack/pack-${pack1}.pack" &&
cmp "test-2-${pack1}.idx" ".git/objects/pack/pack-${pack1}.idx"
* expecting success: # This test assumes file_101 is a delta smaller
than 16 bytes.
# It should be against file_100 but we substitute its base for file_099
sha1_101=`git hash-object file_101` &&
sha1_099=`git hash-object file_099` &&
offs_101=`index_obj_offset 1.idx $sha1_101` &&
nr_099=`index_obj_nr 1.idx $sha1_099` &&
chmod +w ".git/objects/pack/pack-${pack1}.pack" &&
dd of=".git/objects/pack/pack-${pack1}.pack" seek=$(($offs_101 + 1)) \
if=".git/objects/pack/pack-${pack1}.idx" \
skip=$((8 + 256 * 4 + $nr_099 * 20)) \
bs=1 count=20 conv=notrunc &&
git cat-file blob $sha1_101 > file_101_foo2
20+0 records in
20+0 records out
* ok 21: [index v2] 2) create a stealth corruption in a delta base reference
* expecting success: test -f file_101_foo2 && ! cmp file_101 file_101_foo2
file_101 file_101_foo2 differ: char 5, line 2
* ok 22: [index v2] 3) corrupted delta happily returned wrong data
* expecting success: test_must_fail git fsck --full $commit
error: .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
SHA1 checksum mismatch
error: index CRC mismatch for object
0b968294884af115eb5a1b941b3dabd48470c3bb from
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
at offset 844376
error: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
is corrupt
error: sha1 mismatch 0b968294884af115eb5a1b941b3dabd48470c3bb
error: 0b968294884af115eb5a1b941b3dabd48470c3bb: object corrupt or missing
* ok 23: [index v2] 4) confirm that the pack is actually corrupted
* expecting success: test_must_fail git pack-objects test-5 <obj-list &&
test_must_fail git pack-objects --no-reuse-object test-6 <obj-list
error: bad packed object CRC for 0b968294884af115eb5a1b941b3dabd48470c3bb
error: bad packed object CRC for 0b968294884af115eb5a1b941b3dabd48470c3bb
error: failed to read object 0b968294884af115eb5a1b941b3dabd48470c3bb
at offset 844376 from
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
fatal: object 0b968294884af115eb5a1b941b3dabd48470c3bb is corrupted
error: bad packed object CRC for 0b968294884af115eb5a1b941b3dabd48470c3bb
error: failed to read object 0b968294884af115eb5a1b941b3dabd48470c3bb
at offset 844376 from
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
fatal: object 0b968294884af115eb5a1b941b3dabd48470c3bb is corrupted
* ok 24: [index v2] 5) pack-objects refuses to reuse corrupted data
* expecting success: rm -f .git/objects/pack/* &&
git index-pack --index-version=2 --stdin < "test-1-${pack1}.pack" &&
git verify-pack ".git/objects/pack/pack-${pack1}.pack" &&
obj=`git hash-object file_001` &&
nr=`index_obj_nr ".git/objects/pack/pack-${pack1}.idx" $obj` &&
chmod +w ".git/objects/pack/pack-${pack1}.idx" &&
dd if=/dev/zero of=".git/objects/pack/pack-${pack1}.idx" conv=notrunc \
bs=1 count=4 seek=$((8 + 256 * 4 + `wc -l <obj-list` * 20 + $nr * 4)) &&
( while read obj
do git cat-file -p $obj >/dev/null || exit 1
done <obj-list ) &&
err=$(test_must_fail git verify-pack \
".git/objects/pack/pack-${pack1}.pack" 2>&1) &&
echo "$err" | grep "CRC mismatch"
pack 9108b6dfd43bf36ccbfc2839fe62d1503bf84292
4+0 records in
4+0 records out
error: index CRC mismatch for object
1297547aab30f952af805d38f6f1c1930cb3117b from
.git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
at offset 23376
* ok 25: [index v2] 6) verify-pack detects CRC mismatch
* expecting success:
rm -f .git/objects/pack/* &&
cp test-1-${pack1}.pack .git/objects/pack/pack-${pack1}.pack &&
(
cd .git/objects/pack
git index-pack pack-${pack1}.pack
) &&
test -f .git/objects/pack/pack-${pack1}.idx
9108b6dfd43bf36ccbfc2839fe62d1503bf84292
* ok 26: running index-pack in the object store
* failed 4 among 26 test(s)
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] pack-objects: don't use too many threads with few objects
2008-12-15 18:38 ` Mike Ralphson
@ 2008-12-16 1:15 ` Junio C Hamano
2008-12-16 3:20 ` Jeff King
2008-12-16 5:21 ` Nicolas Pitre
0 siblings, 2 replies; 11+ messages in thread
From: Junio C Hamano @ 2008-12-16 1:15 UTC (permalink / raw)
To: Nicolas Pitre; +Cc: Mike Ralphson, Jeff King, git
"Mike Ralphson" <mike.ralphson@gmail.com> writes:
> 2008/12/13 Nicolas Pitre <nico@cam.org>
>>
>> If there are few objects to deltify, they might be split amongst threads
>> so that there is simply no other objects left to delta against within
>> the same thread. Let's use the same 2*window treshold as used for the
>> final load balancing to allow extra threads to be created.
>>
>> This fixes the benign t5300 test failure.
>
> Even with this I'm seeing failures in t5302 which I think are probably
> related to 43cc2b42
> ...
> * ok 2: pack-objects with index version 1
>
> * expecting success: pack2=$(git pack-objects --index-version=2 test-2
> <obj-list) &&
> git verify-pack -v "test-2-${pack2}.pack"
> 0106e17481932f5c223fafadc1d26abc6adf40d6 blob 57 90 850531 1
> 69e0b8ef8cda369575b6801c6ed47daf09aa3c62
> ...
> fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob 8196 8210 187404
> chain length = 1: 109 objects
> chain length = 2: 10 objects
> chain length = 3: 10 objects
> chain length = 4: 10 objects
> chain length = 5: 10 objects
> chain length = 6: 10 objects
> chain length = 7: 10 objects
> chain length = 8: 10 objects
> chain length = 9: 9 objects
> test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok
> * ok 3: pack-objects with index version 2
>
> * expecting success: cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
> test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
> test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack differ: char
> 10236, line 32
> * FAIL 4: both packs should be identical
> cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
This is interesting. The same set of objects are packed with the
identical list of objects stored in obj-list and the only difference
between test #2 and test #3 are the format of the .idx file they produce.
Yet they are producing different .pack file?
And this does not fail with 'master' on Mike's AIX, but with 'next' that
has the threaded delta search.
Since the list of objects packed (obj-list in the test script) contains
302 objects, which is way bigger than the window size, I can understand
that packing with or without threading would produce different results.
Even though depending on the timings of how worker threads complete
assigned work and grab more work for them, two otherwise identical runs,
even with the same set of parameters, could assign different set of
objects to different threads. But as far as I can see, the same set of
objects should go to the same chunk, and the only difference should be
which thread gets which chunk, and I do not see how that could affect the
outcome. Puzzled...
Perhaps the two runs are seeing different number of CPUs (hence threads)
available? That would then change the distribution of the work itself
(i.e. what slice of obj-list goes as a single chunk to be processed) and
would affect the outcome.
Does the second test this patch adds fail?
t/t5302-pack-index.sh | 11 +++++++++++
1 files changed, 11 insertions(+), 0 deletions(-)
diff --git i/t/t5302-pack-index.sh w/t/t5302-pack-index.sh
index 884e242..bd5db03 100755
--- i/t/t5302-pack-index.sh
+++ w/t/t5302-pack-index.sh
@@ -39,6 +39,17 @@ test_expect_success \
git verify-pack -v "test-1-${pack1}.pack"'
test_expect_success \
+ 'pack-objects with index version 1 (again)' \
+ 'pack1=$(git pack-objects --index-version=1 test-1again <obj-list) &&
+ git verify-pack -v "test-1again-${pack1}.pack"'
+
+test_expect_success 'both should match' '
+ cmp "test-1-${pack1}.pack" "test-1again-${pack1}.pack"
+'
+
+exit
+
+test_expect_success \
'pack-objects with index version 2' \
'pack2=$(git pack-objects --index-version=2 test-2 <obj-list) &&
git verify-pack -v "test-2-${pack2}.pack"'
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH] pack-objects: don't use too many threads with few objects
2008-12-16 1:15 ` Junio C Hamano
@ 2008-12-16 3:20 ` Jeff King
2008-12-16 5:21 ` Nicolas Pitre
1 sibling, 0 replies; 11+ messages in thread
From: Jeff King @ 2008-12-16 3:20 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Nicolas Pitre, Mike Ralphson, git
On Mon, Dec 15, 2008 at 05:15:44PM -0800, Junio C Hamano wrote:
> Perhaps the two runs are seeing different number of CPUs (hence threads)
> available? That would then change the distribution of the work itself
> (i.e. what slice of obj-list goes as a single chunk to be processed) and
> would affect the outcome.
I am seeing the same failure as Mike on one of my boxen. Each run sees
the same number of threads (4 in my case).
> Does the second test this patch adds fail?
With your patch, the added "both should match" test fails. So it seems
to be about doing a second run at all, not the difference in index
versions.
-Peff
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH] pack-objects: don't use too many threads with few objects
2008-12-16 1:15 ` Junio C Hamano
2008-12-16 3:20 ` Jeff King
@ 2008-12-16 5:21 ` Nicolas Pitre
1 sibling, 0 replies; 11+ messages in thread
From: Nicolas Pitre @ 2008-12-16 5:21 UTC (permalink / raw)
To: Junio C Hamano; +Cc: Mike Ralphson, Jeff King, git
On Mon, 15 Dec 2008, Junio C Hamano wrote:
> "Mike Ralphson" <mike.ralphson@gmail.com> writes:
>
> > 2008/12/13 Nicolas Pitre <nico@cam.org>
> >>
> >> If there are few objects to deltify, they might be split amongst threads
> >> so that there is simply no other objects left to delta against within
> >> the same thread. Let's use the same 2*window treshold as used for the
> >> final load balancing to allow extra threads to be created.
> >>
> >> This fixes the benign t5300 test failure.
> >
> > Even with this I'm seeing failures in t5302 which I think are probably
> > related to 43cc2b42
> > ...
> > * ok 2: pack-objects with index version 1
> >
> > * expecting success: pack2=$(git pack-objects --index-version=2 test-2
> > <obj-list) &&
> > git verify-pack -v "test-2-${pack2}.pack"
> > 0106e17481932f5c223fafadc1d26abc6adf40d6 blob 57 90 850531 1
> > 69e0b8ef8cda369575b6801c6ed47daf09aa3c62
> > ...
> > fff3a3a92d2268a464dbdcd00fc055885ee3cba9 blob 8196 8210 187404
> > chain length = 1: 109 objects
> > chain length = 2: 10 objects
> > chain length = 3: 10 objects
> > chain length = 4: 10 objects
> > chain length = 5: 10 objects
> > chain length = 6: 10 objects
> > chain length = 7: 10 objects
> > chain length = 8: 10 objects
> > chain length = 9: 9 objects
> > test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack: ok
> > * ok 3: pack-objects with index version 2
> >
> > * expecting success: cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
> > test-1-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack
> > test-2-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack differ: char
> > 10236, line 32
> > * FAIL 4: both packs should be identical
> > cmp "test-1-${pack1}.pack" "test-2-${pack2}.pack"
>
> This is interesting. The same set of objects are packed with the
> identical list of objects stored in obj-list and the only difference
> between test #2 and test #3 are the format of the .idx file they produce.
> Yet they are producing different .pack file?
Indeed, this is really strange.
> Even though depending on the timings of how worker threads complete
> assigned work and grab more work for them, two otherwise identical runs,
> even with the same set of parameters, could assign different set of
> objects to different threads. But as far as I can see, the same set of
> objects should go to the same chunk, and the only difference should be
> which thread gets which chunk, and I do not see how that could affect the
> outcome. Puzzled...
You probably have the explanation here: worker threads are not
necessarily completing their own chunks alwais at the same time relative
with each other. Hence, a thread stealing more work from another thread
might or might not interrupt that other thread always at the same point
between runs, and therefore the final delta chain might get shifted a
bit.
I think the best "fix" is actually to configure only one thread for this
test. The point here is to make sure different index versions don't
introduce variations in pack generation, but multiple threads do defeat
that because of the inner randomness in thread scheduling.
I'm therefore giving my ACK to Johannes' patch posted earlier.
Nicolas
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2008-12-16 5:22 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-12-11 20:36 [PATCH] autodetect number of CPUs by default when using threads Nicolas Pitre
2008-12-11 22:25 ` Junio C Hamano
2008-12-12 0:55 ` Nicolas Pitre
2008-12-13 13:32 ` Jeff King
2008-12-13 19:27 ` Nicolas Pitre
2008-12-13 20:06 ` [PATCH] pack-objects: don't use too many threads with few objects Nicolas Pitre
2008-12-14 2:20 ` Jeff King
2008-12-15 18:38 ` Mike Ralphson
2008-12-16 1:15 ` Junio C Hamano
2008-12-16 3:20 ` Jeff King
2008-12-16 5:21 ` Nicolas Pitre
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).