git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Junio C Hamano <gitster@pobox.com>
To: git@vger.kernel.org
Subject: [PATCH 0/6] zlib only processes 4GB at a time
Date: Fri, 10 Jun 2011 13:15:42 -0700	[thread overview]
Message-ID: <1307736948-16956-1-git-send-email-gitster@pobox.com> (raw)

This is a re-roll from yesterday's series, taking a very different
approach. Instead of fixing call sites one by one, the goal is to define
our own API that can give an illusion to the callers that they can feed
and receive data whose size can fit within "unsigned long", so that they
can set up the avail_in field of z_stream (now git_zstream) to the total
size, and expect it to fall down to zero when everything is consumed by
the library (vice versa for avail_out).

The first 5 patches clean up the existing callers that directly call into
zlib to call git_inflate and git_deflate wrappers.

Patch 6 introduces git_zstream that is a thin wrapper around z_stream,
but barfs when the caller gives a buffer larger than 4GB that would fit
the underlying zlib calling convention.

The next step would be to tweak zlib_post_call(), git_inflate() and
git_deflate() functions to internally loop and call underlying inflate()
and deflate() when the incoming buffers are larger than 4GB, but that
part is not done in this series (yet).

Junio C Hamano (6):
  zlib: refactor error message formatter
  zlib: wrap remaining calls to direct inflate/inflateEnd
  zlib: wrap inflateInit2 used to accept only for gzip format
  zlib: wrap deflate side of the API
  zlib: wrap deflateBound() too
  zlib: zlib can only process 4GB at a time

 archive-zip.c            |   10 +-
 builtin/apply.c          |    2 +-
 builtin/index-pack.c     |   12 ++--
 builtin/pack-objects.c   |   18 ++--
 builtin/unpack-objects.c |    2 +-
 cache.h                  |   30 +++++--
 diff.c                   |   10 +-
 fast-import.c            |   30 +++---
 http-backend.c           |   11 +--
 http-push.c              |   16 ++--
 http.h                   |    2 +-
 pack-check.c             |    4 +-
 remote-curl.c            |   14 +--
 sha1_file.c              |   28 +++---
 zlib.c                   |  218 +++++++++++++++++++++++++++++++++++++++-------
 15 files changed, 283 insertions(+), 124 deletions(-)

-- 
1.7.6.rc1.118.ge175b4a

             reply	other threads:[~2011-06-10 20:15 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-06-10 20:15 Junio C Hamano [this message]
2011-06-10 20:15 ` [PATCH 1/6] zlib: refactor error message formatter Junio C Hamano
2011-06-10 20:15 ` [PATCH 2/6] zlib: wrap remaining calls to direct inflate/inflateEnd Junio C Hamano
2011-06-10 20:15 ` [PATCH 3/6] zlib: wrap inflateInit2 used to accept only for gzip format Junio C Hamano
2011-06-10 20:15 ` [PATCH 4/6] zlib: wrap deflate side of the API Junio C Hamano
2011-06-10 22:23   ` Thiago Farina
2011-06-10 23:00     ` Junio C Hamano
2011-06-10 20:15 ` [PATCH 5/6] zlib: wrap deflateBound() too Junio C Hamano
2011-06-10 20:15 ` [PATCH 6/6] zlib: zlib can only process 4GB at a time Junio C Hamano
2011-06-12 20:43   ` Erik Faye-Lund
2011-06-12 21:33     ` Junio C Hamano
2011-06-12 21:46       ` Matthieu Moy
2011-06-13 11:17       ` Erik Faye-Lund
2011-06-13 11:52         ` Jonathan Nieder
2011-06-13 11:56         ` Junio C Hamano
2011-06-10 23:47 ` [PATCH 7/6] zlib: allow feeding more than 4GB in one go Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=1307736948-16956-1-git-send-email-gitster@pobox.com \
    --to=gitster@pobox.com \
    --cc=git@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).