From: "Shawn O. Pearce" <spearce@spearce.org>
To: Nicolas Pitre <nico@fluxnic.net>
Cc: Junio C Hamano <gitster@pobox.com>, git <git@vger.kernel.org>
Subject: Re: [PATCH] fast-import: Stream very large blobs directly to pack
Date: Thu, 28 Jan 2010 18:37:01 -0800 [thread overview]
Message-ID: <20100129023701.GE20488@spearce.org> (raw)
In-Reply-To: <alpine.LFD.2.00.1001282125410.1681@xanadu.home>
Nicolas Pitre <nico@fluxnic.net> wrote:
> On Thu, 28 Jan 2010, Shawn O. Pearce wrote:
>
> > If a blob is larger than the configured big-file-threshold, instead
> > of reading it into a single buffer obtained from malloc, stream it
> > onto the end of the current pack file. Streaming the larger objects
> > into the pack avoids the 4+ GiB memory footprint that occurs when
> > fast-import is processing 2+ GiB blobs.
>
> Yeah. I've had that item on my todo list for ages now. This
> big-file-threshold principle has to be applied to 'git add' too so a big
> blob is stored in pack file form right away, and used to bypass delta
> searching in 'git pack-objects', used to skip the diff machinery, and so
> on.
Yea, there are a lot of places we should improve for bigger files.
gfi just happened to be the first one I got a bug report on from
a user...
--
Shawn.
next prev parent reply other threads:[~2010-01-29 2:37 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-01-29 1:23 [PATCH] fast-import: Stream very large blobs directly to pack Shawn O. Pearce
2010-01-29 2:33 ` Nicolas Pitre
2010-01-29 2:37 ` Shawn O. Pearce [this message]
2010-01-29 5:29 ` Junio C Hamano
2010-01-29 15:22 ` Shawn O. Pearce
2010-01-29 16:38 ` [PATCH v2] " Shawn O. Pearce
2010-01-29 18:29 ` Jakub Narebski
2010-01-29 18:30 ` Shawn O. Pearce
2010-01-29 23:02 ` A Large Angry SCM
2010-01-30 7:17 ` Junio C Hamano
2010-01-29 18:35 ` [PATCH] " Sverre Rabbelier
2010-01-29 18:37 ` Shawn O. Pearce
2010-01-29 18:41 ` Sverre Rabbelier
2010-01-29 18:44 ` Shawn O. Pearce
2010-01-30 3:41 ` Junio C Hamano
2010-01-30 6:19 ` Junio C Hamano
2010-01-30 7:33 ` Junio C Hamano
2010-02-01 15:28 ` Shawn O. Pearce
2010-02-01 20:14 ` Junio C Hamano
2010-02-04 2:01 ` Junio C Hamano
2010-02-04 2:07 ` Shawn O. Pearce
2010-02-04 2:25 ` Junio C Hamano
2010-02-04 2:27 ` Junio C Hamano
2010-02-04 2:30 ` Shawn O. Pearce
2010-02-04 2:28 ` Nicolas Pitre
2010-02-01 15:41 ` [PATCH] fast-import: Document the core.bigFileThreshold configuration setting Shawn O. Pearce
2010-02-01 15:23 ` [PATCH] fast-import: Stream very large blobs directly to pack Shawn O. Pearce
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100129023701.GE20488@spearce.org \
--to=spearce@spearce.org \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=nico@fluxnic.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.