From: Patrick Steinhardt <ps@pks.im>
To: Elia Pinto <gitter.spiros@gmail.com>
Cc: git@vger.kernel.org,
"Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
"Kristofer Karlsson" <krka@spotify.com>,
"Han Xin" <hanxin.hx@alibaba-inc.com>,
"Han Xin" <chiyutianyi@gmail.com>,
"Junio C Hamano" <gitster@pobox.com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: Re: [PATCH v2 1/4] unpack-objects: use size_t for object sizes
Date: Wed, 18 Nov 2026 14:33:12 +0100 [thread overview]
Message-ID: <av2pBh7vLj5DVDha@pks.im> (raw)
In-Reply-To: <b2c3d349a4ac73da7e1d066ac43754d3e6939b9f.1794984184.git.gitter.spiros@gmail.com>
On Wed, Nov 18, 2026 at 07:56:35AM +0100, Elia Pinto wrote:
> Since f2063855fb (index-pack, unpack-objects: use size_t for object
> size, 2026-05-08), unpack_one() decodes the size of an object into a
> size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed
> blob as size_t, 2026-11-06) passes it on as such to stream_blob(),
> which handles blobs larger than core.bigFileThreshold. The rest of
> builtin/unpack-objects.c still takes the size as an "unsigned long":
>
> - unpack_non_delta_entry() and get_data(), used for commits, trees,
> tags and blobs below core.bigFileThreshold, and for all blobs with
> "git unpack-objects -n", which never streams;
>
> - unpack_delta_entry() and the functions that keep and resolve
> deltas;
>
> - struct obj_buffer, which keeps objects around for --strict.
>
> On platforms where "long" is 32 bits while size_t is 64, such as 64-bit
> Windows, the size of an object of 4GiB or more is therefore truncated:
>
> - For an object that is not streamed, get_data() allocates a buffer
> of the truncated size, inflate runs out of room in it, and
> unpack-objects fails with "inflate returned -5". Truncating the
> size to 16 bits in get_data() and unpack_non_delta_entry()
> reproduces this with a pack holding a 70000-byte blob, both with
> and without -n.
>
> - unpack_delta_entry() reads the base of a delta through
> cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even
> when the result would be smaller.
>
> - The result of a delta reaches write_object() as a size_t, but
> write_object() takes it as an "unsigned long". A result of 4GiB or
> more is silently truncated there, and an object with the wrong
> contents, hence the wrong name, is written.
>
> Use size_t across builtin/unpack-objects.c. The functions that
> write_object() hands objects to, odb_write_object() and
> parse_object_buffer(), still take an "unsigned long", as does
> fsck_object(); they are updated separately. Until then, move the size
> cast from unpack_delta_entry() to write_object(), where the size is
> passed to them. fsck_object() only sees objects that went through
> parse_object_buffer() first, and so does write_cached_object(), which
> writes them out later; neither needs a cast of its own.
>
> As a result, on those platforms:
>
> - "git unpack-objects -n" handles objects of any size;
>
> - a delta with a base of 4GiB or more and a smaller result is
> resolved;
>
> - an object of 4GiB or more that is not streamed and has to be
> written, including the result of a delta, makes unpack-objects die
> with "object too large to read on this platform" instead of failing
> with "inflate returned -5" or being written truncated.
>
> Truncating the size to 16 bits where write_object() passes it on
> confirms the last point for a 70000-byte blob: with the cast,
> unpack-objects dies with "object too large to read on this platform:
> 70000 is cut off to 4464"; without it, it exits with 0 and writes a
> truncated blob under another name.
>
> Triggering this requires processing an object of 4GiB or more on a
> platform where "long" is 32 bits while size_t is 64, so no test is
> added.
Phew, this commit message really is extremely verbose, and I eventually
lost steam reading through it halfways. It's good to document what
you're doing, but this here might have gone off the rails a bit with
that.
What's worth documenting is the unexpected parts in your patch, and
things that may not be immediately obvious to a reviewer. But you also
document what specifically you are doing to what function, and that's
quite obvious from the diff itself. So overall I bet that this message
could be condensed to half or even a third of its size without losing
much information.
> diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
> index d2f25bed49..9bd355b2c4 100644
> --- a/builtin/unpack-objects.c
> +++ b/builtin/unpack-objects.c
> @@ -269,10 +269,11 @@ static void added_object(unsigned nr, enum object_type type,
> * to be checked at the end.
> */
> static void write_object(unsigned nr, enum object_type type,
> - void *buf, unsigned long size)
> + void *buf, size_t size)
> {
> if (!strict) {
> - if (odb_write_object(the_repository->objects, buf, size, type,
> + if (odb_write_object(the_repository->objects, buf,
> + cast_size_t_to_ulong(size), type,
> &obj_list[nr].oid) < 0)
> die("failed to write object");
> added_object(nr, type, buf, size);
These casts here make me a bit unhappy, as you also have a patch series
in flight that fix `odb_write_object()`. So you'll have to send another
patch series after both have landed to remove those again.
Would it make more sense to defer this series here until the other one
lands?
> @@ -299,7 +301,7 @@ static void write_object(unsigned nr, enum object_type type,
> &obj_list[nr].oid);
> added_object(nr, type, buf, size);
> obj = parse_object_buffer(the_repository, &obj_list[nr].oid,
> - type, size, buf,
> + type, cast_size_t_to_ulong(size), buf,
> &eaten);
> if (!obj)
> die("invalid %s", type_name(type));
Likewise, even though I don't know whether you also have a patch series
in flight fixing this one from the top of my head.
Patrick
next prev parent reply other threads:[~2026-11-18 13:33 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-31 11:39 [PATCH 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-10-31 11:39 ` [PATCH 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-10-31 11:39 ` [PATCH 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18 6:56 ` [PATCH v2 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-11-18 6:56 ` [PATCH v2 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-11-18 13:33 ` Patrick Steinhardt [this message]
2026-11-18 16:59 ` Junio C Hamano
2026-11-19 10:09 ` Patrick Steinhardt
2026-11-19 6:05 ` Elia Pinto
2026-11-19 10:12 ` Patrick Steinhardt
2026-11-19 17:48 ` Elia Pinto
2026-11-18 6:56 ` [PATCH v2 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-11-18 13:33 ` Patrick Steinhardt
2026-11-19 6:05 ` Elia Pinto
2026-11-18 6:56 ` [PATCH v2 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-11-18 6:56 ` [PATCH v2 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18 13:34 ` Patrick Steinhardt
2026-11-19 6:05 ` Elia Pinto
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=av2pBh7vLj5DVDha@pks.im \
--to=ps@pks.im \
--cc=Johannes.Schindelin@gmx.de \
--cc=avarab@gmail.com \
--cc=chiyutianyi@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=gitter.spiros@gmail.com \
--cc=hanxin.hx@alibaba-inc.com \
--cc=krka@spotify.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.