Git development
 help / color / mirror / Atom feed
From: Patrick Steinhardt <ps@pks.im>
To: Elia Pinto <gitter.spiros@gmail.com>
Cc: git@vger.kernel.org,
	"Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Kristofer Karlsson" <krka@spotify.com>,
	"Han Xin" <hanxin.hx@alibaba-inc.com>,
	"Han Xin" <chiyutianyi@gmail.com>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: Re: [PATCH v2 1/4] unpack-objects: use size_t for object sizes
Date: Wed, 18 Nov 2026 14:33:12 +0100	[thread overview]
Message-ID: <av2pBh7vLj5DVDha@pks.im> (raw)
In-Reply-To: <b2c3d349a4ac73da7e1d066ac43754d3e6939b9f.1794984184.git.gitter.spiros@gmail.com>

On Wed, Nov 18, 2026 at 07:56:35AM +0100, Elia Pinto wrote:
> Since f2063855fb (index-pack, unpack-objects: use size_t for object
> size, 2026-05-08), unpack_one() decodes the size of an object into a
> size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed
> blob as size_t, 2026-11-06) passes it on as such to stream_blob(),
> which handles blobs larger than core.bigFileThreshold. The rest of
> builtin/unpack-objects.c still takes the size as an "unsigned long":
> 
>  - unpack_non_delta_entry() and get_data(), used for commits, trees,
>    tags and blobs below core.bigFileThreshold, and for all blobs with
>    "git unpack-objects -n", which never streams;
> 
>  - unpack_delta_entry() and the functions that keep and resolve
>    deltas;
> 
>  - struct obj_buffer, which keeps objects around for --strict.
> 
> On platforms where "long" is 32 bits while size_t is 64, such as 64-bit
> Windows, the size of an object of 4GiB or more is therefore truncated:
> 
>  - For an object that is not streamed, get_data() allocates a buffer
>    of the truncated size, inflate runs out of room in it, and
>    unpack-objects fails with "inflate returned -5". Truncating the
>    size to 16 bits in get_data() and unpack_non_delta_entry()
>    reproduces this with a pack holding a 70000-byte blob, both with
>    and without -n.
> 
>  - unpack_delta_entry() reads the base of a delta through
>    cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even
>    when the result would be smaller.
> 
>  - The result of a delta reaches write_object() as a size_t, but
>    write_object() takes it as an "unsigned long". A result of 4GiB or
>    more is silently truncated there, and an object with the wrong
>    contents, hence the wrong name, is written.
> 
> Use size_t across builtin/unpack-objects.c. The functions that
> write_object() hands objects to, odb_write_object() and
> parse_object_buffer(), still take an "unsigned long", as does
> fsck_object(); they are updated separately. Until then, move the size
> cast from unpack_delta_entry() to write_object(), where the size is
> passed to them. fsck_object() only sees objects that went through
> parse_object_buffer() first, and so does write_cached_object(), which
> writes them out later; neither needs a cast of its own.
> 
> As a result, on those platforms:
> 
>  - "git unpack-objects -n" handles objects of any size;
> 
>  - a delta with a base of 4GiB or more and a smaller result is
>    resolved;
> 
>  - an object of 4GiB or more that is not streamed and has to be
>    written, including the result of a delta, makes unpack-objects die
>    with "object too large to read on this platform" instead of failing
>    with "inflate returned -5" or being written truncated.
> 
> Truncating the size to 16 bits where write_object() passes it on
> confirms the last point for a 70000-byte blob: with the cast,
> unpack-objects dies with "object too large to read on this platform:
> 70000 is cut off to 4464"; without it, it exits with 0 and writes a
> truncated blob under another name.
> 
> Triggering this requires processing an object of 4GiB or more on a
> platform where "long" is 32 bits while size_t is 64, so no test is
> added.

Phew, this commit message really is extremely verbose, and I eventually
lost steam reading through it halfways. It's good to document what
you're doing, but this here might have gone off the rails a bit with
that.

What's worth documenting is the unexpected parts in your patch, and
things that may not be immediately obvious to a reviewer. But you also
document what specifically you are doing to what function, and that's
quite obvious from the diff itself. So overall I bet that this message
could be condensed to half or even a third of its size without losing
much information.

> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
> index d2f25bed49..9bd355b2c4 100644
> --- a/builtin/unpack-objects.c
> +++ b/builtin/unpack-objects.c
> @@ -269,10 +269,11 @@ static void added_object(unsigned nr, enum object_type type,
>   * to be checked at the end.
>   */
>  static void write_object(unsigned nr, enum object_type type,
> -			 void *buf, unsigned long size)
> +			 void *buf, size_t size)
>  {
>  	if (!strict) {
> -		if (odb_write_object(the_repository->objects, buf, size, type,
> +		if (odb_write_object(the_repository->objects, buf,
> +				     cast_size_t_to_ulong(size), type,
>  				     &obj_list[nr].oid) < 0)
>  			die("failed to write object");
>  		added_object(nr, type, buf, size);

These casts here make me a bit unhappy, as you also have a patch series
in flight that fix `odb_write_object()`. So you'll have to send another
patch series after both have landed to remove those again.

Would it make more sense to defer this series here until the other one
lands?

> @@ -299,7 +301,7 @@ static void write_object(unsigned nr, enum object_type type,
>  				 &obj_list[nr].oid);
>  		added_object(nr, type, buf, size);
>  		obj = parse_object_buffer(the_repository, &obj_list[nr].oid,
> -					  type, size, buf,
> +					  type, cast_size_t_to_ulong(size), buf,
>  					  &eaten);
>  		if (!obj)
>  			die("invalid %s", type_name(type));

Likewise, even though I don't know whether you also have a patch series
in flight fixing this one from the top of my head.

Patrick

  reply	other threads:[~2026-11-18 13:33 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-31 11:39 [PATCH 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-10-31 11:39 ` [PATCH 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-10-31 11:39 ` [PATCH 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18  6:56 ` [PATCH v2 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-11-18  6:56   ` [PATCH v2 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-11-18 13:33     ` Patrick Steinhardt [this message]
2026-11-18 16:59       ` Junio C Hamano
2026-11-18  6:56   ` [PATCH v2 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-11-18 13:33     ` Patrick Steinhardt
2026-11-18  6:56   ` [PATCH v2 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-11-18  6:56   ` [PATCH v2 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18 13:34     ` Patrick Steinhardt

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=av2pBh7vLj5DVDha@pks.im \
    --to=ps@pks.im \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=chiyutianyi@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=gitter.spiros@gmail.com \
    --cc=hanxin.hx@alibaba-inc.com \
    --cc=krka@spotify.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox