From: Shawn Pearce <spearce@spearce.org>
To: weigelt@metux.de, git@vger.kernel.org
Subject: Re: large files and low memory
Date: Mon, 4 Oct 2010 11:05:54 -0700 [thread overview]
Message-ID: <AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com> (raw)
In-Reply-To: <20101004092046.GA4382@nibiru.local>
On Mon, Oct 4, 2010 at 2:20 AM, Enrico Weigelt <weigelt@metux.de> wrote:
>
> when adding files which are larger than available physical memory,
> git performs very slow. Perhaps it has to do with git's mmap()ing
> the whole file. Is there any way to do it w/o mmap (hoping that
> might perform a bit better) ?
The mmap() isn't the problem. Its the allocation of a buffer that is
larger than the file in order to hold the result of deflating the file
before it gets written to disk. When the file is bigger than physical
memory, the kernel has to page in parts of the file as well as swap in
and out parts of that allocated buffer to hold the deflated file.
This is a known area in Git where big files aren't handled well.
--
Shawn.
next prev parent reply other threads:[~2010-10-04 18:08 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-10-04 9:20 large files and low memory Enrico Weigelt
2010-10-04 18:05 ` Shawn Pearce [this message]
2010-10-04 18:24 ` Joshua Jensen
2010-10-04 18:57 ` Shawn Pearce
2010-10-05 0:59 ` Enrico Weigelt
2010-10-05 7:41 ` Enrico Weigelt
2010-10-05 8:01 ` Matthieu Moy
2010-10-05 8:17 ` Enrico Weigelt
2010-10-05 11:29 ` Alex Riesen
2010-10-05 11:38 ` Matthieu Moy
2010-10-05 11:55 ` Nguyen Thai Ngoc Duy
2010-10-05 16:42 ` Junio C Hamano
2010-10-05 10:13 ` Nguyen Thai Ngoc Duy
2010-10-05 19:12 ` Nicolas Pitre
2010-10-04 18:58 ` Jonathan Nieder
2010-10-04 19:11 ` Shawn Pearce
2010-10-04 19:16 ` Jonathan Nieder
2010-10-05 10:59 ` Nguyen Thai Ngoc Duy
2010-10-05 20:17 ` Nicolas Pitre
2010-10-05 20:34 ` Jonathan Nieder
2010-10-05 21:11 ` Nicolas Pitre
2010-10-05 0:57 ` Enrico Weigelt
2010-10-05 1:07 ` Ævar Arnfjörð Bjarmason
2010-10-05 1:10 ` Jonathan Nieder
2010-10-05 7:35 ` Enrico Weigelt
2010-10-05 13:47 ` Jonathan Nieder
2010-10-05 0:50 ` Enrico Weigelt
2010-10-05 19:06 ` Nicolas Pitre
2010-10-05 22:51 ` Enrico Weigelt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=AANLkTimbdrAqoWMxiteT5zNYmwHp8M698BEv1FLuiAxx@mail.gmail.com \
--to=spearce@spearce.org \
--cc=git@vger.kernel.org \
--cc=weigelt@metux.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).