From mboxrd@z Thu Jan 1 00:00:00 1970 From: Shawn Pearce Subject: Re: how to keep git-fetch from running out of memory? Date: Sun, 15 May 2011 13:37:27 -0700 Message-ID: References: <7vboz3pv7x.fsf@alter.siamese.dyndns.org> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Cc: Kartik Agaram , git@vger.kernel.org To: Junio C Hamano X-From: git-owner@vger.kernel.org Sun May 15 22:37:56 2011 Return-path: Envelope-to: gcvg-git-2@lo.gmane.org Received: from vger.kernel.org ([209.132.180.67]) by lo.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1QLi4a-0003ZZ-2S for gcvg-git-2@lo.gmane.org; Sun, 15 May 2011 22:37:56 +0200 Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753475Ab1EOUht (ORCPT ); Sun, 15 May 2011 16:37:49 -0400 Received: from mail-vw0-f46.google.com ([209.85.212.46]:55074 "EHLO mail-vw0-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753372Ab1EOUhs (ORCPT ); Sun, 15 May 2011 16:37:48 -0400 Received: by vws1 with SMTP id 1so2755123vws.19 for ; Sun, 15 May 2011 13:37:48 -0700 (PDT) Received: by 10.52.176.3 with SMTP id ce3mr5123516vdc.313.1305491867100; Sun, 15 May 2011 13:37:47 -0700 (PDT) Received: by 10.52.157.73 with HTTP; Sun, 15 May 2011 13:37:27 -0700 (PDT) In-Reply-To: <7vboz3pv7x.fsf@alter.siamese.dyndns.org> Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Archived-At: On Sun, May 15, 2011 at 13:25, Junio C Hamano wrote: > Shawn Pearce writes: > >> However, that may still be insufficient. A big object must still be >> allocated in memory in order to compute its SHA-1. > > I wonder if we can stream this? If its a non-delta, yes. We already do this in JGit for blobs over 50M. (Trees, commits and tags are not streamed because we might need to do fsck on them and the fsck code for these object types requires the entire object in a single buffer.) Its probably not a huge change to index-pack.c. But I haven't dug around in there in a while. I wish I could say I will look at this on Monday, but I don't have the time. -- Shawn.