Git development
 help / color / mirror / Atom feed
From: Elia Pinto <gitter.spiros@gmail.com>
To: git@vger.kernel.org
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Elia Pinto" <gitter.spiros@gmail.com>,
	"Justin Tobler" <jltobler@gmail.com>,
	"Han Xin" <hanxin.hx@alibaba-inc.com>,
	"Han Xin" <chiyutianyi@gmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Patrick Steinhardt" <ps@pks.im>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Jonathan Tan" <jonathantanmy@fastmail.com>,
	"Kristofer Karlsson" <krka@spotify.com>
Subject: [PATCH v2 0/4] use size_t when receiving and verifying packs
Date: Wed, 18 Nov 2026 07:56:34 +0100	[thread overview]
Message-ID: <cover.1794984184.git.gitter.spiros@gmail.com> (raw)
In-Reply-To: <cover.1793446357.git.gitter.spiros@gmail.com>

This series fixes object sizes that are truncated when receiving and
verifying packs. Most of them are kept in an "unsigned long", which is
a problem where "long" is 32 bits while size_t is 64, such as on 64-bit
Windows; 2/4 is about an "unsigned int", which is 32 bits everywhere.

 1/4 unpack-objects fails with "inflate returned -5" on an object of
     4GiB or more that is not streamed, dies on a delta base that
     large, and silently writes a truncated object for a delta result
     that large. With 1/4, "unpack-objects -n" and large delta bases
     work, and writing such an object, unless it is streamed, dies
     with a clear error.

 2/4 "index-pack --fix-thin" with a local base of 4GiB or more
     compresses only part of it and dies later with "serious inflate
     inconsistency" (all platforms).

 3/4 index-pack reports a false "SHA1 COLLISION FOUND" for a large
     object that already exists locally.

 4/4 "git fsck" checks only a prefix of large objects in packs.

None of them comes with a test, as each needs an object of 4GiB or
more; 1/4 and 2/4 describe how I checked them with the size truncated
to 16 bits.

Changes since v1:

 - Rebased onto master at e1ec09b23c (The 22nd batch). v1's 1/4 no
   longer applied: e07ac16d1c (unpack-objects: pass the size of a
   streamed blob as size_t, 2026-11-06) by Kristofer Karlsson, now in
   master, makes the same change to stream_blob(). 1/4 drops that hunk;
   Kristofer is Cc'd.

 - The message of 1/4 used stream_blob() as its only example, so it is
   rewritten around the paths that are still affected: objects that
   are not streamed, all blobs with "git unpack-objects -n", and
   deltas.

 - 1/4 no longer drops the cast_size_t_to_ulong() in
   unpack_delta_entry(), but moves it to write_object(), where the size
   is passed to odb_write_object() and parse_object_buffer(). v1 let
   objects of 4GiB or more reach these functions and be written
   truncated.

 - 4/4 says that parse_object_buffer() and fsck_object() still see a
   truncated size until they take a size_t.

 - The messages of 1/4, 3/4 and 4/4 now say that the problem is "long"
   being 32 bits while size_t is 64. The code of 2/4, 3/4 and 4/4 is
   unchanged.

There were no comments on v1.

v1 left pack-objects for a later series. That fix has since been
merged as ep/pack-objects-streamed-large-blobs-fix (ed0aed1e17); it
covers the streamed path and explains why the other truncations in
pack-objects cannot be reached.

The callees odb_write_object(), parse_object_buffer() and
fsck_object() still take an "unsigned long" on master. They are changed
by two other series of mine, which are not queued yet:

  https://lore.kernel.org/git/cover.1793444420.git.gitter.spiros@gmail.com/
  https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/

The series is based on master at e1ec09b23c.

Elia Pinto (4):
  unpack-objects: use size_t for object sizes
  index-pack: use size_t in write_compressed()
  index-pack: use size_t for object sizes
  pack-check: use size_t for the size passed to verify callbacks

 builtin/fsck.c           |  2 +-
 builtin/index-pack.c     | 14 +++++++-------
 builtin/unpack-objects.c | 42 ++++++++++++++++++++--------------------
 odb.h                    |  2 +-
 pack.h                   |  2 +-
 5 files changed, 31 insertions(+), 31 deletions(-)

Range-diff against v1:
1:  f259eeb6a6 ! 1:  b2c3d349a4 unpack-objects: use size_t for object sizes
    @@ Metadata
      ## Commit message ##
         unpack-objects: use size_t for object sizes
     
    -    builtin/unpack-objects.c uses "unsigned long" to track object sizes
    -    throughout the file, while sizes read from packs are handled as size_t.
    -    On platforms where "long" is 32 bits, this truncates the size for
    -    objects of 4GiB or more.
    +    Since f2063855fb (index-pack, unpack-objects: use size_t for object
    +    size, 2026-05-08), unpack_one() decodes the size of an object into a
    +    size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed
    +    blob as size_t, 2026-11-06) passes it on as such to stream_blob(),
    +    which handles blobs larger than core.bigFileThreshold. The rest of
    +    builtin/unpack-objects.c still takes the size as an "unsigned long":
     
    -    When handling large objects on those platforms, unpack-objects fails
    -    rather than corrupting data. For instance, streaming large blobs
    -    (exceeding core.bigFileThreshold) passes a truncated size to
    -    stream_blob(), causing the stream write to fail with an error.
    +     - unpack_non_delta_entry() and get_data(), used for commits, trees,
    +       tags and blobs below core.bigFileThreshold, and for all blobs with
    +       "git unpack-objects -n", which never streams;
     
    -    Use size_t across the board in builtin/unpack-objects.c and remove the
    -    now-unnecessary size cast in unpack_delta_entry(). Downstream consumers
    -    like odb_write_object() and fsck_object() are updated separately.
    +     - unpack_delta_entry() and the functions that keep and resolve
    +       deltas;
     
    -    Triggering this requires processing 4GiB or more of input on a platform
    -    where "long" is 32 bits, so no test is added.
    +     - struct obj_buffer, which keeps objects around for --strict.
    +
    +    On platforms where "long" is 32 bits while size_t is 64, such as 64-bit
    +    Windows, the size of an object of 4GiB or more is therefore truncated:
    +
    +     - For an object that is not streamed, get_data() allocates a buffer
    +       of the truncated size, inflate runs out of room in it, and
    +       unpack-objects fails with "inflate returned -5". Truncating the
    +       size to 16 bits in get_data() and unpack_non_delta_entry()
    +       reproduces this with a pack holding a 70000-byte blob, both with
    +       and without -n.
    +
    +     - unpack_delta_entry() reads the base of a delta through
    +       cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even
    +       when the result would be smaller.
    +
    +     - The result of a delta reaches write_object() as a size_t, but
    +       write_object() takes it as an "unsigned long". A result of 4GiB or
    +       more is silently truncated there, and an object with the wrong
    +       contents, hence the wrong name, is written.
    +
    +    Use size_t across builtin/unpack-objects.c. The functions that
    +    write_object() hands objects to, odb_write_object() and
    +    parse_object_buffer(), still take an "unsigned long", as does
    +    fsck_object(); they are updated separately. Until then, move the size
    +    cast from unpack_delta_entry() to write_object(), where the size is
    +    passed to them. fsck_object() only sees objects that went through
    +    parse_object_buffer() first, and so does write_cached_object(), which
    +    writes them out later; neither needs a cast of its own.
    +
    +    As a result, on those platforms:
    +
    +     - "git unpack-objects -n" handles objects of any size;
    +
    +     - a delta with a base of 4GiB or more and a smaller result is
    +       resolved;
    +
    +     - an object of 4GiB or more that is not streamed and has to be
    +       written, including the result of a delta, makes unpack-objects die
    +       with "object too large to read on this platform" instead of failing
    +       with "inflate returned -5" or being written truncated.
    +
    +    Truncating the size to 16 bits where write_object() passes it on
    +    confirms the last point for a 70000-byte blob: with the cast,
    +    unpack-objects dies with "object too large to read on this platform:
    +    70000 is cut off to 4464"; without it, it exits with 0 and writes a
    +    truncated blob under another name.
    +
    +    Triggering this requires processing an object of 4GiB or more on a
    +    platform where "long" is 32 bits while size_t is 64, so no test is
    +    added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
    @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
     +			 void *buf, size_t size)
      {
      	if (!strict) {
    - 		if (odb_write_object(the_repository->objects, buf, size, type,
    +-		if (odb_write_object(the_repository->objects, buf, size, type,
    ++		if (odb_write_object(the_repository->objects, buf,
    ++				     cast_size_t_to_ulong(size), type,
    + 				     &obj_list[nr].oid) < 0)
    + 			die("failed to write object");
    + 		added_object(nr, type, buf, size);
    +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
    + 		obj_list[nr].obj = NULL;
    + 	} else if (type == OBJ_BLOB) {
    + 		struct blob *blob;
    +-		if (odb_write_object(the_repository->objects, buf, size, type,
    ++		if (odb_write_object(the_repository->objects, buf,
    ++				     cast_size_t_to_ulong(size), type,
    + 				     &obj_list[nr].oid) < 0)
    + 			die("failed to write object");
    + 		added_object(nr, type, buf, size);
    +@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
    + 				 &obj_list[nr].oid);
    + 		added_object(nr, type, buf, size);
    + 		obj = parse_object_buffer(the_repository, &obj_list[nr].oid,
    +-					  type, size, buf,
    ++					  type, cast_size_t_to_ulong(size), buf,
    + 					  &eaten);
    + 		if (!obj)
    + 			die("invalid %s", type_name(type));
     @@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
      }
      
    @@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
      				   unsigned nr)
      {
      	void *buf = get_data(size);
    -@@ builtin/unpack-objects.c: static ssize_t zlib_stream_read(struct odb_stream *in_stream,
    - 	return buf_len - zstream->avail_out;
    - }
    - 
    --static void stream_blob(unsigned long size, unsigned nr)
    -+static void stream_blob(size_t size, unsigned nr)
    - {
    - 	git_zstream zstream = { 0 };
    - 	struct zlib_stream in_stream = {
    -@@ builtin/unpack-objects.c: static void stream_blob(unsigned long size, unsigned nr)
    +@@ builtin/unpack-objects.c: static void stream_blob(size_t size, unsigned nr)
      }
      
      static int resolve_against_held(unsigned nr, const struct object_id *base,
2:  438f7c6ba2 = 2:  9952f0fb10 index-pack: use size_t in write_compressed()
3:  0258282745 ! 3:  e66860d274 index-pack: use size_t for object sizes
    @@ Commit message
         Functions like sha1_object() and append_obj_to_pack() in
         builtin/index-pack.c take object sizes as an "unsigned long", even
         though struct object_entry already tracks them as size_t. On platforms
    -    where "long" is 32 bits, this truncates the size for objects of 4GiB or
    -    more.
    +    where "long" is 32 bits while size_t is 64, such as 64-bit Windows,
    +    this truncates the size of objects of 4GiB or more.
     
         When receiving, on those platforms, a large object that already exists
         locally, sha1_object() compares the real size (size_t) against the
    @@ Commit message
         Use size_t for the size parameter across these functions.
     
         Triggering this requires processing 4GiB or more of input on a platform
    -    where "long" is 32 bits, so no test is added.
    +    where "long" is 32 bits while size_t is 64, so no test is added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
4:  87b0d2bd96 ! 4:  00939c7f1b pack-check: use size_t for the size passed to verify callbacks
    @@ Commit message
         replicated in struct odb_fsck_options (odb.h) and implemented by
         fsck_obj_buffer() in builtin/fsck.c.
     
    -    On platforms where "long" is 32 bits, running "git fsck" on packs with
    -    objects of 4GiB or more passes a truncated size to fsck_obj_buffer(),
    -    causing parse_object_buffer() and fsck_object() to inspect only a
    -    prefix.
    +    On platforms where "long" is 32 bits while size_t is 64, such as
    +    64-bit Windows, running "git fsck" on packs with objects of 4GiB or
    +    more passes a truncated size to fsck_obj_buffer(), causing
    +    parse_object_buffer() and fsck_object() to inspect only a prefix.
     
         Use size_t for the size parameter in verify_fn,
         struct odb_fsck_options.object_cb, and fsck_obj_buffer().
    +    parse_object_buffer() and fsck_object() themselves still take an
    +    "unsigned long"; until they are updated separately, they keep
    +    inspecting only a prefix.
     
         Triggering this requires processing objects of 4GiB or more on a
    -    platform where "long" is 32 bits, so no test is added.
    +    platform where "long" is 32 bits while size_t is 64, so no test is
    +    added.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     

base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
-- 
2.53.0


  parent reply	other threads:[~2026-11-18  6:56 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-31 11:39 [PATCH 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-10-31 11:39 ` [PATCH 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-10-31 11:39 ` [PATCH 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18  6:56 ` Elia Pinto [this message]
2026-11-18  6:56   ` [PATCH v2 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-11-18  6:56   ` [PATCH v2 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-11-18  6:56   ` [PATCH v2 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-11-18  6:56   ` [PATCH v2 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1794984184.git.gitter.spiros@gmail.com \
    --to=gitter.spiros@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=chiyutianyi@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=hanxin.hx@alibaba-inc.com \
    --cc=jltobler@gmail.com \
    --cc=jonathantanmy@fastmail.com \
    --cc=krka@spotify.com \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox