From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Jon Smirl" Subject: Re: [PATCH] RFC: git lazy clone proof-of-concept Date: Sun, 10 Feb 2008 12:01:47 -0500 Message-ID: <9e4733910802100901m729b0cdfg85ccc0ca77011249@mail.gmail.com> References: <200802081828.43849.kendy@suse.cz> <200802091627.25913.kendy@suse.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "Nicolas Pitre" , "Jan Holesovsky" , "Jakub Narebski" , git@vger.kernel.org, "Junio C Hamano" To: "Johannes Schindelin" X-From: git-owner@vger.kernel.org Sun Feb 10 18:02:41 2008 Return-path: Envelope-to: gcvg-git-2@gmane.org Received: from vger.kernel.org ([209.132.176.167]) by lo.gmane.org with esmtp (Exim 4.50) id 1JOFZe-0008FK-Ti for gcvg-git-2@gmane.org; Sun, 10 Feb 2008 18:02:39 +0100 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751926AbYBJRBv (ORCPT ); Sun, 10 Feb 2008 12:01:51 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1751903AbYBJRBv (ORCPT ); Sun, 10 Feb 2008 12:01:51 -0500 Received: from wa-out-1112.google.com ([209.85.146.182]:32996 "EHLO wa-out-1112.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751855AbYBJRBu (ORCPT ); Sun, 10 Feb 2008 12:01:50 -0500 Received: by wa-out-1112.google.com with SMTP id v27so1655282wah.23 for ; Sun, 10 Feb 2008 09:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; bh=gOio2+Cm8eImNV5cFOWzew7jUwgEiXyM269xS86qh5s=; b=tQPqQpGdVpeDWC10hAtNIm01YY0AYHl4fSbaZU1RtBpPsMtPvYJgXNWnUmZMg/nuaPRn5gKmisxGpO1kp4XLpdI/RoEA+x/GlSlcFjZYqkh83mtRaw/WWDW4Lg41bByeKcDiiW5GRftbA4HHtGIjdHJMJruzXrESnJKiad4uopg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=v24a4XRhntVZIivqaBX6TaOv58OwbY9lJ2qrXV+GyqmoyU6O4yW7HKl7EG7ig3pPEHL/1L1fDWgaT9xT7s/Kx25B8TuO52XKVSuPDh+mGwplrt1wQxWZRZ+sjCcdfkjVn/I3YasWcM4CqnIjPt+KEpVQ92zKPMWSyOIEHDQNi2E= Received: by 10.114.81.1 with SMTP id e1mr9036517wab.11.1202662908045; Sun, 10 Feb 2008 09:01:48 -0800 (PST) Received: by 10.114.200.7 with HTTP; Sun, 10 Feb 2008 09:01:47 -0800 (PST) In-Reply-To: Content-Disposition: inline Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Archived-At: On 2/10/08, Johannes Schindelin wrote: > Hi, > > On Sat, 9 Feb 2008, Nicolas Pitre wrote: > > > On Sat, 9 Feb 2008, Jan Holesovsky wrote: > > > > > On Friday 08 February 2008 20:00, Jakub Narebski wrote: > > > > > > > Both Mozilla import, and GCC import were packed below 0.5 GB. Warning: > > > > you would need machine with large amount of memory to repack it > > > > tightly in sensible time! > > > > > > As I answered elsewhere, unfortunately it goes out of memory even on 8G > > > machine (x86-64), so... But still trying. > > > > Try setting the following config variables as follows: > > > > git config pack.deltaCacheLimit 1 > > git config pack.deltaCacheSize 1 > > git config pack.windowMemory 1g > > > > That should help keeping memory usage somewhat bounded. > > I tried that: > > $ git config pack.deltaCacheLimit 1 > $ git config pack.deltaCacheSize 1 > $ git config pack.windowMemory 2g > $ #/usr/bin/time git repack -a -d -f --window=250 --depth=250 > $ du -s objects/ > 2548137 objects/ > $ /usr/bin/time git repack -a -d -f --window=250 --depth=250 > Counting objects: 2477715, done. > fatal: Out of memory, malloc failed411764) > Command exited with non-zero status 1 > 9356.95user 53.33system 2:38:58elapsed 98%CPU (0avgtext+0avgdata > 0maxresident)k > 0inputs+0outputs (31929major+18088744minor)pagefaults 0swaps > > Note that this is on a 2.4GHz Quadcode CPU with 3.5GB RAM. Turning on multi-core support greatly increases the memory consumption; at least double the single thread case. Going over the original repository and deleting (get all copies out of the history) those giant i18n files generated by programs than Sean refers to would be my first step. If you have 5,000 revisions of a 10MB file I suspect it would take a huge amount of memory to pack. Plus you have to copy all of that pointless history around. > > I'm retrying with smaller values, but at over 2.5 hours per try, this is > getting tedious. > > Ciao, > Dscho > > - > To unsubscribe from this list: send the line "unsubscribe git" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Jon Smirl jonsmirl@gmail.com