From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>,
worley@alum.mit.edu, "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
Subject: [PATCH v3 3/6] unpack-objects: continue when fail to malloc due to large objects
Date: Wed, 13 Aug 2014 17:57:31 +0700 [thread overview]
Message-ID: <1407927454-9268-4-git-send-email-pclouds@gmail.com> (raw)
In-Reply-To: <1407927454-9268-1-git-send-email-pclouds@gmail.com>
As a recovery tool, unpack-objects should go on unpacking as many
objects as it can.
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
builtin/unpack-objects.c | 42 +++++++++++++++++++++++++++++++++++++++++-
t/t1050-large.sh | 7 +++++++
2 files changed, 48 insertions(+), 1 deletion(-)
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index 99cde45..8b5c67e 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -88,10 +88,50 @@ static void use(int bytes)
consumed_bytes += bytes;
}
+static void inflate_and_throw_away(unsigned long size)
+{
+ git_zstream stream;
+ char buf[8192];
+
+ memset(&stream, 0, sizeof(stream));
+ stream.next_out = (unsigned char *)buf;
+ stream.avail_out = sizeof(buf);
+ stream.next_in = fill(1);
+ stream.avail_in = len;
+ git_inflate_init(&stream);
+
+ for (;;) {
+ int ret = git_inflate(&stream, 0);
+ use(len - stream.avail_in);
+ if (stream.total_out == size && ret == Z_STREAM_END)
+ break;
+ if (ret != Z_OK) {
+ error("inflate returned %d", ret);
+ if (!recover)
+ exit(1);
+ has_errors = 1;
+ break;
+ }
+ stream.next_out = (unsigned char *)buf;
+ stream.avail_out = sizeof(buf);
+ stream.next_in = fill(1);
+ stream.avail_in = len;
+ }
+ git_inflate_end(&stream);
+}
+
static void *get_data(unsigned long size)
{
git_zstream stream;
- void *buf = xmalloc(size);
+ void *buf = xmalloc_gentle(size);
+
+ if (!buf) {
+ if (!recover)
+ exit(1);
+ has_errors = 1;
+ inflate_and_throw_away(size);
+ return NULL;
+ }
memset(&stream, 0, sizeof(stream));
diff --git a/t/t1050-large.sh b/t/t1050-large.sh
index 5642f84..eec2cca 100755
--- a/t/t1050-large.sh
+++ b/t/t1050-large.sh
@@ -169,4 +169,11 @@ test_expect_success 'fsck' '
test "$n" -gt 1
'
+test_expect_success 'unpack-objects' '
+ P=`ls .git/objects/pack/*.pack` &&
+ git unpack-objects -n -r <$P 2>err
+ test $? = 1 &&
+ grep "error: attempting to allocate .* over limit" err
+'
+
test_done
--
2.1.0.rc0.78.gc0d8480
next prev parent reply other threads:[~2014-08-13 10:58 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-05-27 16:47 Git chokes on large file Dale R. Worley
2014-05-28 13:32 ` Duy Nguyen
2014-05-28 17:10 ` Junio C Hamano
2014-05-28 18:18 ` Dale R. Worley
2014-05-28 18:15 ` Dale R. Worley
2014-05-28 18:23 ` David Lang
2014-05-28 18:47 ` Dale R. Worley
2014-05-28 19:05 ` David Lang
2014-05-29 19:12 ` Dale R. Worley
2014-05-28 18:54 ` Junio C Hamano
2014-05-28 19:09 ` David Lang
2014-05-29 12:57 ` [PATCH 1/4] wrapper.c: introduce gentle xmallocz that does not die() Nguyễn Thái Ngọc Duy
2014-05-29 12:57 ` [PATCH 2/4] fsck: do not die when not enough memory to examine a pack entry Nguyễn Thái Ngọc Duy
2014-05-29 12:57 ` [PATCH 3/4] diff.c: allow to pass more flags to diff_populate_filespec Nguyễn Thái Ngọc Duy
2014-05-29 12:57 ` [PATCH 4/4] diff: mark any file larger than core.bigfilethreshold binary Nguyễn Thái Ngọc Duy
2014-06-19 12:27 ` Thomas Braun
2014-06-23 12:18 ` Duy Nguyen
2014-06-23 19:21 ` Thomas Braun
2014-06-24 11:45 ` [PATCH v2 1/4] wrapper.c: introduce gentle xmallocz that does not die() Nguyễn Thái Ngọc Duy
2014-06-24 11:45 ` [PATCH v2 2/4] fsck: do not die when not enough memory to examine a pack entry Nguyễn Thái Ngọc Duy
2014-06-26 18:09 ` Junio C Hamano
2014-06-29 0:40 ` Duy Nguyen
2014-06-24 11:45 ` [PATCH v2 3/4] diff.c: allow to pass more flags to diff_populate_filespec Nguyễn Thái Ngọc Duy
2014-06-24 11:45 ` [PATCH v2 4/4] diff: mark any file larger than core.bigfilethreshold binary Nguyễn Thái Ngọc Duy
2014-06-26 17:55 ` Junio C Hamano
2014-06-27 18:56 ` Thomas Braun
2014-06-29 1:11 ` Duy Nguyen
2014-08-13 10:57 ` [PATCH v3 0/6] Large file improvements Nguyễn Thái Ngọc Duy
2014-08-13 10:57 ` [PATCH v3 1/6] wrapper.c: introduce gentle xmalloc(z) that does not die() Nguyễn Thái Ngọc Duy
2014-08-14 16:38 ` Junio C Hamano
2014-08-13 10:57 ` [PATCH v3 2/6] sha1_file.c: do not die failing to malloc in unpack_compressed_entry Nguyễn Thái Ngọc Duy
2014-08-13 21:13 ` Junio C Hamano
2014-08-13 10:57 ` Nguyễn Thái Ngọc Duy [this message]
2014-08-14 16:58 ` [PATCH v3 3/6] unpack-objects: continue when fail to malloc due to large objects Junio C Hamano
2014-08-15 5:24 ` Duy Nguyen
2014-08-13 10:57 ` [PATCH v3 4/6] diff.c: allow to pass more flags to diff_populate_filespec Nguyễn Thái Ngọc Duy
2014-08-13 10:57 ` [PATCH v3 5/6] diff --stat: mark any file larger than core.bigfilethreshold binary Nguyễn Thái Ngọc Duy
2014-08-13 19:32 ` Eric Sunshine
2014-08-13 10:57 ` [PATCH v3 6/6] diff: shortcut for diff'ing two binary SHA-1 objects Nguyễn Thái Ngọc Duy
2014-08-14 17:00 ` Junio C Hamano
2014-08-15 12:11 ` Duy Nguyen
2014-08-14 17:17 ` Junio C Hamano
2014-08-16 3:08 ` [PATCH v4 0/5] Large file improvements Nguyễn Thái Ngọc Duy
2014-08-16 3:08 ` [PATCH v4 1/5] wrapper.c: introduce gentle xmallocz that does not die() Nguyễn Thái Ngọc Duy
2014-08-16 3:08 ` [PATCH v4 2/5] sha1_file.c: do not die failing to malloc in unpack_compressed_entry Nguyễn Thái Ngọc Duy
2014-08-16 3:08 ` [PATCH v4 3/5] diff.c: allow to pass more flags to diff_populate_filespec Nguyễn Thái Ngọc Duy
2014-08-16 3:08 ` [PATCH v4 4/5] diff --stat: mark any file larger than core.bigfilethreshold binary Nguyễn Thái Ngọc Duy
2014-08-16 3:08 ` [PATCH v4 5/5] diff: shortcut for diff'ing two binary SHA-1 objects Nguyễn Thái Ngọc Duy
2014-05-28 15:05 ` Git chokes on large file Thomas Braun
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1407927454-9268-4-git-send-email-pclouds@gmail.com \
--to=pclouds@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=worley@alum.mit.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).