All of lore.kernel.org
 help / color / mirror / Atom feed
From: Elia Pinto <gitter.spiros@gmail.com>
To: git@vger.kernel.org
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Elia Pinto" <gitter.spiros@gmail.com>,
	"Justin Tobler" <jltobler@gmail.com>,
	"Han Xin" <hanxin.hx@alibaba-inc.com>,
	"Han Xin" <chiyutianyi@gmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Patrick Steinhardt" <ps@pks.im>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Jonathan Tan" <jonathantanmy@fastmail.com>,
	"Kristofer Karlsson" <krka@spotify.com>
Subject: [PATCH v2 0/4] use size_t when receiving and verifying packs
Date: Wed, 18 Nov 2026 07:56:34 +0100	[thread overview]
Message-ID: <cover.1794984184.git.gitter.spiros@gmail.com> (raw)
In-Reply-To: <cover.1793446357.git.gitter.spiros@gmail.com>

This series fixes object sizes that are truncated when receiving and
verifying packs. Most of them are kept in an "unsigned long", which is
a problem where "long" is 32 bits while size_t is 64, such as on 64-bit
Windows; 2/4 is about an "unsigned int", which is 32 bits everywhere.

 1/4 unpack-objects fails with "inflate returned -5" on an object of
     4GiB or more that is not streamed, dies on a delta base that
     large, and silently writes a truncated object for a delta result
     that large. With 1/4, "unpack-objects -n" and large delta bases
     work, and writing such an object, unless it is streamed, dies
     with a clear error.

 2/4 "index-pack --fix-thin" with a local base of 4GiB or more
     compresses only part of it and dies later with "serious inflate
     inconsistency" (all platforms).

 3/4 index-pack reports a false "SHA1 COLLISION FOUND" for a large
     object that already exists locally.

 4/4 "git fsck" checks only a prefix of large objects in packs.

None of them comes with a test, as each needs an object of 4GiB or
more; 1/4 and 2/4 describe how I checked them with the size truncated
to 16 bits.

Changes since v1:

 - Rebased onto master at e1ec09b23c (The 22nd batch). v1's 1/4 no
   longer applied: e07ac16d1c (unpack-objects: pass the size of a
   streamed blob as size_t, 2026-11-06) by Kristofer Karlsson, now in
   master, makes the same change to stream_blob(). 1/4 drops that hunk;
   Kristofer is Cc'd.

 - The message of 1/4 used stream_blob() as its only example, so it is
   rewritten around the paths that are still affected: objects that
   are not streamed, all blobs with "git unpack-objects -n", and
   deltas.

 - 1/4 no longer drops the cast_size_t_to_ulong() in
   unpack_delta_entry(), but moves it to write_object(), where the size
   is passed to odb_write_object() and parse_object_buffer(). v1 let
   objects of 4GiB or more reach these functions and be written
   truncated.

 - 4/4 says that parse_object_buffer() and fsck_object() still see a
   truncated size until they take a size_t.

 - The messages of 1/4, 3/4 and 4/4 now say that the problem is "long"
   being 32 bits while size_t is 64. The code of 2/4, 3/4 and 4/4 is
   unchanged.

There were no comments on v1.

v1 left pack-objects for a later series. That fix has since been
merged as ep/pack-objects-streamed-large-blobs-fix (ed0aed1e17); it
covers the streamed path and explains why the other truncations in
pack-objects cannot be reached.

The callees odb_write_object(), parse_object_buffer() and
fsck_object() still take an "unsigned long" on master. They are changed
by two other series of mine, which are not queued yet:

  https://lore.kernel.org/git/cover.1793444420.git.gitter.spiros@gmail.com/
  https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/

The series is based on master at e1ec09b23c.

Elia Pinto (4):
  unpack-objects: use size_t for object sizes
  index-pack: use size_t in write_compressed()
  index-pack: use size_t for object sizes
  pack-check: use size_t for the size passed to verify callbacks

 builtin/fsck.c           |  2 +-
 builtin/index-pack.c     | 14 +++++++-------
 builtin/unpack-objects.c | 42 ++++++++++++++++++++--------------------
 odb.h                    |  2 +-
 pack.h                   |  2 +-
 5 files changed, 31 insertions(+), 31 deletions(-)

Range-diff against v1:
1:  f259eeb6a6 ! 1:  b2c3d349a4 unpack-objects: use size_t for object sizes
    @@ Metadata
      ## Commit message ##
         unpack-objects: use size_t for object sizes
     
    -    builtin/unpack-objects.c uses "unsigned long" to track object sizes
    -    throughout the file, while sizes read from packs are handled as size_t.
    -    On platforms where "long" is 32 bits, this truncates the size for
    -    objects of 4GiB or more.
    +    Since f2063855fb (index-pack, unpack-objects: use size_t for object
    +    size, 2026-05-08), unpack_one() decodes the size of an object into a
    +    size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed
    +    blob as size_t, 2026-11-06) passes it on as such to stream_blob(),
    +    which handles blobs larger than core.bigFileThreshold. The rest of
    +    builtin/unpack-objects.c still takes the size as an "unsigned long":
     
    -    When handling large objects on those platforms, unpack-objects fails
    -    rather than corrupting data. For instance, streaming large blobs
    -    (exceeding core.bigFileThreshold) passes a truncated size to
    -    stream_blob(), causing the stream write to fail with an error.
    +     - unpack_non_delta_entry() and get_data(), used for commits, trees,
    +       tags and blobs below core.bigFileThreshold, and for all blobs with
    +       "git unpack-objects -n", which never streams;
     
    -    Use size_t across the board in builtin/unpack-objects.c and remove the
    -    now-unnecessary size cast in unpack_delta_entry(). Downstream consumers
    -    like odb_write_object() and fsck_object() are updated separately.
    +     - unpack_delta_entry() and the functions that keep and resolve
    +       deltas;
     
    -    Triggering this requires processing 4GiB or more of input on a platform
    -    where "long" is 32 bits, so no test is added.
    +     - struct obj_buffer, which keeps objects around for --strict.
    +
    +    On platforms where "long" is 32 bits while size_t is 64, such as 64-bit
    +    Windows, the size of an object of 4GiB or more is therefore truncated:
    +
    +     - For an object that is not streamed, get_data() allocates a buffer
    +       of the truncated size, inflate runs out of room in it, and
    +       unpack-objects fails with "inflate returned -5". Truncating the
    +       size to 16 bits in get_data() and unpack_non_delta_entry()
    +       reproduces this with a pack holding a 70000-byte blob, both with
    +       and without -n.
    +
    +     - unpack_delta_entry() reads the base of a delta through
    +       cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even
    +       when the result would be smaller.
    +
    +     - The result of a delta reaches write_object() as a size_t, but
    +       write_object() takes it as an "unsigned long". A result of 4GiB or
    +       more is silently truncated there, and an object with the wrong
    +       contents, hence the wrong name, is written.
    +
    +    Use size_t across builtin/unpack-objects.c. The functions that
    +    write_object() hands objects to, odb_write_object() and
    +    parse_object_buffer(), still take an "unsigned long", as does
    +    fsck_object(); they are updated separately. Until then, move the size
    +    cast from unpack_delta_entry() to write_object(), where the size is
    +    passed to them. fsck_object() only sees objects that went through
    +    parse_object_buffer() first, and so does write_cached_object(), which
    +    writes them out later; neither needs a cast of its own.
    +
    +    As a result, on those platforms:
    +
    +     - "git unpack-objects -n" handles objects of any size;
    +
    +     - a delta with a base of 4GiB or more and a smaller result is
    +       resolved;
    +
    +     - an object of 4GiB or more that is not streamed and has to be
    +       written, including the result of a delta, makes unpack-objects die
    +       with "object too large to read on this platform" instead of failing
    +       with "inflate returned -5" or being written truncated.
    +
    +    Truncating the size to 16 bits where write_object() passes it on
    +    confirms the last point for a 70000-byte blob: with the cast,
    +    unpack-objects dies with "object too large to read on this platform:
    +    70000 is cut off to 4464"; without it, it exits with 0 and writes a
    +    truncated blob under another name.
    +
    +    Triggering this requires processing an object of 4GiB or more on a
    +    platform where "long" is 32 bits while size_t is 64, so no test is
    +    added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
    @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
     +			 void *buf, size_t size)
      {
      	if (!strict) {
    - 		if (odb_write_object(the_repository->objects, buf, size, type,
    +-		if (odb_write_object(the_repository->objects, buf, size, type,
    ++		if (odb_write_object(the_repository->objects, buf,
    ++				     cast_size_t_to_ulong(size), type,
    + 				     &obj_list[nr].oid) < 0)
    + 			die("failed to write object");
    + 		added_object(nr, type, buf, size);
    +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
    + 		obj_list[nr].obj = NULL;
    + 	} else if (type == OBJ_BLOB) {
    + 		struct blob *blob;
    +-		if (odb_write_object(the_repository->objects, buf, size, type,
    ++		if (odb_write_object(the_repository->objects, buf,
    ++				     cast_size_t_to_ulong(size), type,
    + 				     &obj_list[nr].oid) < 0)
    + 			die("failed to write object");
    + 		added_object(nr, type, buf, size);
    +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
    + 				 &obj_list[nr].oid);
    + 		added_object(nr, type, buf, size);
    + 		obj = parse_object_buffer(the_repository, &obj_list[nr].oid,
    +-					  type, size, buf,
    ++					  type, cast_size_t_to_ulong(size), buf,
    + 					  &eaten);
    + 		if (!obj)
    + 			die("invalid %s", type_name(type));
     @@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
      }
      
    @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
      				   unsigned nr)
      {
      	void *buf = get_data(size);
    -@@ builtin/unpack-objects.c: static ssize_t zlib_stream_read(struct odb_stream *in_stream,
    - 	return buf_len - zstream->avail_out;
    - }
    - 
    --static void stream_blob(unsigned long size, unsigned nr)
    -+static void stream_blob(size_t size, unsigned nr)
    - {
    - 	git_zstream zstream = { 0 };
    - 	struct zlib_stream in_stream = {
    -@@ builtin/unpack-objects.c: static void stream_blob(unsigned long size, unsigned nr)
    +@@ builtin/unpack-objects.c: static void stream_blob(size_t size, unsigned nr)
      }
      
      static int resolve_against_held(unsigned nr, const struct object_id *base,
2:  438f7c6ba2 = 2:  9952f0fb10 index-pack: use size_t in write_compressed()
3:  0258282745 ! 3:  e66860d274 index-pack: use size_t for object sizes
    @@ Commit message
         Functions like sha1_object() and append_obj_to_pack() in
         builtin/index-pack.c take object sizes as an "unsigned long", even
         though struct object_entry already tracks them as size_t. On platforms
    -    where "long" is 32 bits, this truncates the size for objects of 4GiB or
    -    more.
    +    where "long" is 32 bits while size_t is 64, such as 64-bit Windows,
    +    this truncates the size of objects of 4GiB or more.
     
         When receiving, on those platforms, a large object that already exists
         locally, sha1_object() compares the real size (size_t) against the
    @@ Commit message
         Use size_t for the size parameter across these functions.
     
         Triggering this requires processing 4GiB or more of input on a platform
    -    where "long" is 32 bits, so no test is added.
    +    where "long" is 32 bits while size_t is 64, so no test is added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
4:  87b0d2bd96 ! 4:  00939c7f1b pack-check: use size_t for the size passed to verify callbacks
    @@ Commit message
         replicated in struct odb_fsck_options (odb.h) and implemented by
         fsck_obj_buffer() in builtin/fsck.c.
     
    -    On platforms where "long" is 32 bits, running "git fsck" on packs with
    -    objects of 4GiB or more passes a truncated size to fsck_obj_buffer(),
    -    causing parse_object_buffer() and fsck_object() to inspect only a
    -    prefix.
    +    On platforms where "long" is 32 bits while size_t is 64, such as
    +    64-bit Windows, running "git fsck" on packs with objects of 4GiB or
    +    more passes a truncated size to fsck_obj_buffer(), causing
    +    parse_object_buffer() and fsck_object() to inspect only a prefix.
     
         Use size_t for the size parameter in verify_fn,
         struct odb_fsck_options.object_cb, and fsck_obj_buffer().
    +    parse_object_buffer() and fsck_object() themselves still take an
    +    "unsigned long"; until they are updated separately, they keep
    +    inspecting only a prefix.
     
         Triggering this requires processing objects of 4GiB or more on a
    -    platform where "long" is 32 bits, so no test is added.
    +    platform where "long" is 32 bits while size_t is 64, so no test is
    +    added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     

base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
-- 
2.53.0


  parent reply	other threads:[~2026-11-18  6:56 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-31 11:39 [PATCH 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-10-31 11:39 ` [PATCH 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-10-31 11:39 ` [PATCH 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18  6:56 ` Elia Pinto [this message]
2026-11-18  6:56   ` [PATCH v2 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-11-18  6:56   ` [PATCH v2 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-11-18  6:56   ` [PATCH v2 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-11-18  6:56   ` [PATCH v2 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1794984184.git.gitter.spiros@gmail.com \
    --to=gitter.spiros@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=chiyutianyi@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=hanxin.hx@alibaba-inc.com \
    --cc=jltobler@gmail.com \
    --cc=jonathantanmy@fastmail.com \
    --cc=krka@spotify.com \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.