From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Mike Ralphson" Subject: Re: [PATCH] pack-objects: don't use too many threads with few objects Date: Mon, 15 Dec 2008 18:38:18 +0000 Message-ID: References: <20081213133238.GA6718@sigill.intra.peff.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: "Junio C Hamano" , "Jeff King" , git@vger.kernel.org To: "Nicolas Pitre" X-From: git-owner@vger.kernel.org Mon Dec 15 19:39:44 2008 Return-path: Envelope-to: gcvg-git-2@gmane.org Received: from vger.kernel.org ([209.132.176.167]) by lo.gmane.org with esmtp (Exim 4.50) id 1LCIM0-0007xo-9y for gcvg-git-2@gmane.org; Mon, 15 Dec 2008 19:39:41 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752343AbYLOSiV (ORCPT ); Mon, 15 Dec 2008 13:38:21 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754062AbYLOSiV (ORCPT ); Mon, 15 Dec 2008 13:38:21 -0500 Received: from qw-out-2122.google.com ([74.125.92.26]:27773 "EHLO qw-out-2122.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752337AbYLOSiU (ORCPT ); Mon, 15 Dec 2008 13:38:20 -0500 Received: by qw-out-2122.google.com with SMTP id 3so624602qwe.37 for ; Mon, 15 Dec 2008 10:38:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=JzMsOm8JajGXUpIYYhKERuci3a0Zr7PvmBqXf05Sa28=; b=wH98VFHdIWVr5b6x802XgNJPzGF8ELGz0sOja7O/zoZsqH18/r9IEmF7qfCQrOyy/5 RgWNxqiBWYvqrP5HQWmlmtdOHlrIgiZT4Q2q7ivQRhi07wlrMz0pLzGg5UlSKNhuOT2w OHUs+GtPEbv0woP4F/iPrI6UWM6o8J6zEydx4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=n4o3tvzcZcXCIfLP6SWSy3vpMT7wudI2IVhEWuHghlJHqNhwlzRJacdN86xjej2LuO GRVp+WT3yD4fjCzVc7JY0Wc3IXqLbfTJ0/QH2N+2VLjjklsYV4ATbJEcfn/ZNAqP6Tos /y5RUPeVikQr7/I8R7KgtSvR5sCGT3fIbIU2g= Received: by 10.215.100.13 with SMTP id c13mr8018416qam.353.1229366298931; Mon, 15 Dec 2008 10:38:18 -0800 (PST) Received: by 10.214.150.9 with HTTP; Mon, 15 Dec 2008 10:38:18 -0800 (PST) In-Reply-To: Content-Disposition: inline Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Archived-At: 2008/12/13 Nicolas Pitre > > If there are few objects to deltify, they might be split amongst threads > so that there is simply no other objects left to delta against within > the same thread. Let's use the same 2*window treshold as used for the > final load balancing to allow extra threads to be created. > > This fixes the benign t5300 test failure. Even with this I'm seeing failures in t5302 which I think are probably related to 43cc2b42 I was trying to bisect it (test runs take forever on this platform - AIX 5.3) but I hit the t5300 failure, which I hadn't seen on the last automatic build. Please shout if I've snipped the vital lines (...) Initialized empty Git repository in /usr/local/src/gitbuild/t/trash directory.t5302-pack-index/.git/ * expecting success: rm -rf .git git init && i=1 && while test $i -le 100 do iii=`printf %03i $i` test-genrandom "bar" 200 > wide_delta_$iii && test-genrandom "baz $iii" 50 >> wide_delta_$iii && test-genrandom "foo"$i 100 > deep_delta_$iii && test-genrandom "foo"`expr $i + 1` 100 >> deep_delta_$iii && test-genrandom "foo"`expr $i + 2` 100 >> deep_delta_$iii && echo $iii >file_$iii && test-genrandom "$iii" 8192 >>file_$iii && git update-index --add file_$iii deep_delta_$iii wide_delta_$iii && i=`expr $i + 1` || return 1 done && { echo 101 && test-genrandom 100 8192; } >file_101 && git update-index --add file_101 && tree=`git write-tree` && commit=`git commit-tree $tree obj-list && git update-ref HEAD $commit Initialized empty Git repository in /usr/local/src/gitbuild/t/trash directory.t5302-pack-index/.git/ * ok 1: setup * expecting success: pack1=$(git pack-objects --index-version=1 test-1 file_101_foo1 20+0 records in 20+0 records out * ok 15: [index v1] 2) create a stealth corruption in a delta base reference * expecting success: test -f file_101_foo1 && ! cmp file_101 file_101_foo1 file_101 file_101_foo1 differ: char 5, line 2 * ok 16: [index v1] 3) corrupted delta happily returned wrong data * expecting success: test_must_fail git fsck --full $commit error: .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack SHA1 checksum mismatch error: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack is corrupt error: sha1 mismatch 0b968294884af115eb5a1b941b3dabd48470c3bb error: 0b968294884af115eb5a1b941b3dabd48470c3bb: object corrupt or missing * ok 17: [index v1] 4) confirm that the pack is actually corrupted * expecting success: pack4=$(git pack-objects test-4 file_101_foo2 20+0 records in 20+0 records out * ok 21: [index v2] 2) create a stealth corruption in a delta base reference * expecting success: test -f file_101_foo2 && ! cmp file_101 file_101_foo2 file_101 file_101_foo2 differ: char 5, line 2 * ok 22: [index v2] 3) corrupted delta happily returned wrong data * expecting success: test_must_fail git fsck --full $commit error: .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack SHA1 checksum mismatch error: index CRC mismatch for object 0b968294884af115eb5a1b941b3dabd48470c3bb from .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack at offset 844376 error: packed 0b968294884af115eb5a1b941b3dabd48470c3bb from .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack is corrupt error: sha1 mismatch 0b968294884af115eb5a1b941b3dabd48470c3bb error: 0b968294884af115eb5a1b941b3dabd48470c3bb: object corrupt or missing * ok 23: [index v2] 4) confirm that the pack is actually corrupted * expecting success: test_must_fail git pack-objects test-5 /dev/null || exit 1 done &1) && echo "$err" | grep "CRC mismatch" pack 9108b6dfd43bf36ccbfc2839fe62d1503bf84292 4+0 records in 4+0 records out error: index CRC mismatch for object 1297547aab30f952af805d38f6f1c1930cb3117b from .git/objects/pack/pack-9108b6dfd43bf36ccbfc2839fe62d1503bf84292.pack at offset 23376 * ok 25: [index v2] 6) verify-pack detects CRC mismatch * expecting success: rm -f .git/objects/pack/* && cp test-1-${pack1}.pack .git/objects/pack/pack-${pack1}.pack && ( cd .git/objects/pack git index-pack pack-${pack1}.pack ) && test -f .git/objects/pack/pack-${pack1}.idx 9108b6dfd43bf36ccbfc2839fe62d1503bf84292 * ok 26: running index-pack in the object store * failed 4 among 26 test(s)