From: Christopher Jefferson <caj@cs.st-andrews.ac.uk>
To: git@vger.kernel.org
Subject: Problem with large files on different OSes
Date: Wed, 27 May 2009 11:52:12 +0100 [thread overview]
Message-ID: <submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk> (raw)
I recently came across a very annoying problem, characterised by the
following example:
On a recent ubuntu install:
dd if=/dev/zero of=file bs=1300k count=1k
git commit file -m "Add huge file"
The repository can be pulled and pushed successfully to other ubuntu
installs, but on Mac OS X, 10.5.7 machine with 4GB ram git pull
produces:
remote: Counting objects: 6, done.
remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed
(error code=12)
remote: *** error: can't allocate region
remote: *** set a breakpoint in malloc_error_break to debug
remote: git(1533,0xb0081000) malloc: *** mmap(size=1363152896) failed
(error code=12)
remote: *** error: can't allocate region
remote: *** set a breakpoint in malloc_error_break to debug
remote: fatal: Out of memory, malloc failed
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository corruption
on the remote side.
remote: aborting due to possible repository corruption on the remote
side.
fatal: protocol error: bad pack header
The problem appears to be the different maximum mmap sizes available
on different OSes. Whic I don't really mind the maximum file size
restriction git imposes, this restriction varying from OS to OS is
very annoying, fixing this required rewriting history to remove the
commit, which caused problems as the commit had already been pulled,
and built on, by a number of developers.
If the requirement that all files can be mmapped cannot be easily
removed, would be it perhaps be acceptable to impose a (soft?)
1GB(ish) file size limit? I suggest 1GB as all the OSes I can get hold
of easily (freeBSD, windows, Mac OS X, linux) support a mmap of size >
1GB.
next reply other threads:[~2009-05-27 11:20 UTC|newest]
Thread overview: 31+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-05-27 10:52 Christopher Jefferson [this message]
2009-05-27 11:37 ` Problem with large files on different OSes Andreas Ericsson
2009-05-27 13:02 ` Christopher Jefferson
2009-05-27 13:28 ` John Tapsell
2009-05-27 13:30 ` Christopher Jefferson
2009-05-27 13:32 ` John Tapsell
2009-05-27 14:01 ` Tomas Carnecky
2009-05-27 14:09 ` Christopher Jefferson
2009-05-27 14:22 ` Andreas Ericsson
2009-05-27 14:37 ` Jakub Narebski
2009-05-27 16:30 ` Linus Torvalds
2009-05-27 16:59 ` Linus Torvalds
2009-05-27 17:22 ` Christopher Jefferson
2009-05-27 17:30 ` Jakub Narebski
2009-05-27 17:37 ` Nicolas Pitre
2009-05-27 21:53 ` Jeff King
2009-05-27 22:07 ` Linus Torvalds
2009-05-27 23:09 ` Alan Manuel Gloria
2009-05-28 1:56 ` Linus Torvalds
2009-05-28 3:26 ` Nicolas Pitre
2009-05-28 4:21 ` Eric Raible
2009-05-28 4:30 ` Shawn O. Pearce
2009-05-28 5:52 ` Eric Raible
2009-05-28 8:52 ` Andreas Ericsson
2009-05-28 17:41 ` Nicolas Pitre
2009-05-28 19:43 ` Jeff King
2009-05-28 19:49 ` Linus Torvalds
2009-05-27 23:29 ` Nicolas Pitre
2009-05-28 20:00 ` Jeff King
2009-05-28 20:54 ` Nicolas Pitre
2009-05-28 21:21 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=submission.1M9Gk0-0000N8-MQ@mail.cs.st-andrews.ac.uk \
--to=caj@cs.st-andrews.ac.uk \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).