From: Elia Pinto <gitter.spiros@gmail.com>
To: git@vger.kernel.org
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
"Elia Pinto" <gitter.spiros@gmail.com>,
"Justin Tobler" <jltobler@gmail.com>,
"Han Xin" <hanxin.hx@alibaba-inc.com>,
"Han Xin" <chiyutianyi@gmail.com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Patrick Steinhardt" <ps@pks.im>,
"Junio C Hamano" <gitster@pobox.com>,
"Jonathan Tan" <jonathantanmy@fastmail.com>,
"Kristofer Karlsson" <krka@spotify.com>
Subject: [PATCH v2 0/4] use size_t when receiving and verifying packs
Date: Wed, 18 Nov 2026 07:56:34 +0100 [thread overview]
Message-ID: <cover.1794984184.git.gitter.spiros@gmail.com> (raw)
In-Reply-To: <cover.1793446357.git.gitter.spiros@gmail.com>
This series fixes object sizes that are truncated when receiving and
verifying packs. Most of them are kept in an "unsigned long", which is
a problem where "long" is 32 bits while size_t is 64, such as on 64-bit
Windows; 2/4 is about an "unsigned int", which is 32 bits everywhere.
1/4 unpack-objects fails with "inflate returned -5" on an object of
4GiB or more that is not streamed, dies on a delta base that
large, and silently writes a truncated object for a delta result
that large. With 1/4, "unpack-objects -n" and large delta bases
work, and writing such an object, unless it is streamed, dies
with a clear error.
2/4 "index-pack --fix-thin" with a local base of 4GiB or more
compresses only part of it and dies later with "serious inflate
inconsistency" (all platforms).
3/4 index-pack reports a false "SHA1 COLLISION FOUND" for a large
object that already exists locally.
4/4 "git fsck" checks only a prefix of large objects in packs.
None of them comes with a test, as each needs an object of 4GiB or
more; 1/4 and 2/4 describe how I checked them with the size truncated
to 16 bits.
Changes since v1:
- Rebased onto master at e1ec09b23c (The 22nd batch). v1's 1/4 no
longer applied: e07ac16d1c (unpack-objects: pass the size of a
streamed blob as size_t, 2026-11-06) by Kristofer Karlsson, now in
master, makes the same change to stream_blob(). 1/4 drops that hunk;
Kristofer is Cc'd.
- The message of 1/4 used stream_blob() as its only example, so it is
rewritten around the paths that are still affected: objects that
are not streamed, all blobs with "git unpack-objects -n", and
deltas.
- 1/4 no longer drops the cast_size_t_to_ulong() in
unpack_delta_entry(), but moves it to write_object(), where the size
is passed to odb_write_object() and parse_object_buffer(). v1 let
objects of 4GiB or more reach these functions and be written
truncated.
- 4/4 says that parse_object_buffer() and fsck_object() still see a
truncated size until they take a size_t.
- The messages of 1/4, 3/4 and 4/4 now say that the problem is "long"
being 32 bits while size_t is 64. The code of 2/4, 3/4 and 4/4 is
unchanged.
There were no comments on v1.
v1 left pack-objects for a later series. That fix has since been
merged as ep/pack-objects-streamed-large-blobs-fix (ed0aed1e17); it
covers the streamed path and explains why the other truncations in
pack-objects cannot be reached.
The callees odb_write_object(), parse_object_buffer() and
fsck_object() still take an "unsigned long" on master. They are changed
by two other series of mine, which are not queued yet:
https://lore.kernel.org/git/cover.1793444420.git.gitter.spiros@gmail.com/
https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/
The series is based on master at e1ec09b23c.
Elia Pinto (4):
unpack-objects: use size_t for object sizes
index-pack: use size_t in write_compressed()
index-pack: use size_t for object sizes
pack-check: use size_t for the size passed to verify callbacks
builtin/fsck.c | 2 +-
builtin/index-pack.c | 14 +++++++-------
builtin/unpack-objects.c | 42 ++++++++++++++++++++--------------------
odb.h | 2 +-
pack.h | 2 +-
5 files changed, 31 insertions(+), 31 deletions(-)
Range-diff against v1:
1: f259eeb6a6 ! 1: b2c3d349a4 unpack-objects: use size_t for object sizes
@@ Metadata
## Commit message ##
unpack-objects: use size_t for object sizes
- builtin/unpack-objects.c uses "unsigned long" to track object sizes
- throughout the file, while sizes read from packs are handled as size_t.
- On platforms where "long" is 32 bits, this truncates the size for
- objects of 4GiB or more.
+ Since f2063855fb (index-pack, unpack-objects: use size_t for object
+ size, 2026-05-08), unpack_one() decodes the size of an object into a
+ size_t, and e07ac16d1c (unpack-objects: pass the size of a streamed
+ blob as size_t, 2026-11-06) passes it on as such to stream_blob(),
+ which handles blobs larger than core.bigFileThreshold. The rest of
+ builtin/unpack-objects.c still takes the size as an "unsigned long":
- When handling large objects on those platforms, unpack-objects fails
- rather than corrupting data. For instance, streaming large blobs
- (exceeding core.bigFileThreshold) passes a truncated size to
- stream_blob(), causing the stream write to fail with an error.
+ - unpack_non_delta_entry() and get_data(), used for commits, trees,
+ tags and blobs below core.bigFileThreshold, and for all blobs with
+ "git unpack-objects -n", which never streams;
- Use size_t across the board in builtin/unpack-objects.c and remove the
- now-unnecessary size cast in unpack_delta_entry(). Downstream consumers
- like odb_write_object() and fsck_object() are updated separately.
+ - unpack_delta_entry() and the functions that keep and resolve
+ deltas;
- Triggering this requires processing 4GiB or more of input on a platform
- where "long" is 32 bits, so no test is added.
+ - struct obj_buffer, which keeps objects around for --strict.
+
+ On platforms where "long" is 32 bits while size_t is 64, such as 64-bit
+ Windows, the size of an object of 4GiB or more is therefore truncated:
+
+ - For an object that is not streamed, get_data() allocates a buffer
+ of the truncated size, inflate runs out of room in it, and
+ unpack-objects fails with "inflate returned -5". Truncating the
+ size to 16 bits in get_data() and unpack_non_delta_entry()
+ reproduces this with a pack holding a 70000-byte blob, both with
+ and without -n.
+
+ - unpack_delta_entry() reads the base of a delta through
+ cast_size_t_to_ulong(), so a base of 4GiB or more makes it die even
+ when the result would be smaller.
+
+ - The result of a delta reaches write_object() as a size_t, but
+ write_object() takes it as an "unsigned long". A result of 4GiB or
+ more is silently truncated there, and an object with the wrong
+ contents, hence the wrong name, is written.
+
+ Use size_t across builtin/unpack-objects.c. The functions that
+ write_object() hands objects to, odb_write_object() and
+ parse_object_buffer(), still take an "unsigned long", as does
+ fsck_object(); they are updated separately. Until then, move the size
+ cast from unpack_delta_entry() to write_object(), where the size is
+ passed to them. fsck_object() only sees objects that went through
+ parse_object_buffer() first, and so does write_cached_object(), which
+ writes them out later; neither needs a cast of its own.
+
+ As a result, on those platforms:
+
+ - "git unpack-objects -n" handles objects of any size;
+
+ - a delta with a base of 4GiB or more and a smaller result is
+ resolved;
+
+ - an object of 4GiB or more that is not streamed and has to be
+ written, including the result of a delta, makes unpack-objects die
+ with "object too large to read on this platform" instead of failing
+ with "inflate returned -5" or being written truncated.
+
+ Truncating the size to 16 bits where write_object() passes it on
+ confirms the last point for a 70000-byte blob: with the cast,
+ unpack-objects dies with "object too large to read on this platform:
+ 70000 is cut off to 4464"; without it, it exits with 0 and writes a
+ truncated blob under another name.
+
+ Triggering this requires processing an object of 4GiB or more on a
+ platform where "long" is 32 bits while size_t is 64, so no test is
+ added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
@@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
+ void *buf, size_t size)
{
if (!strict) {
- if (odb_write_object(the_repository->objects, buf, size, type,
+- if (odb_write_object(the_repository->objects, buf, size, type,
++ if (odb_write_object(the_repository->objects, buf,
++ cast_size_t_to_ulong(size), type,
+ &obj_list[nr].oid) < 0)
+ die("failed to write object");
+ added_object(nr, type, buf, size);
+@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
+ obj_list[nr].obj = NULL;
+ } else if (type == OBJ_BLOB) {
+ struct blob *blob;
+- if (odb_write_object(the_repository->objects, buf, size, type,
++ if (odb_write_object(the_repository->objects, buf,
++ cast_size_t_to_ulong(size), type,
+ &obj_list[nr].oid) < 0)
+ die("failed to write object");
+ added_object(nr, type, buf, size);
+@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
+ &obj_list[nr].oid);
+ added_object(nr, type, buf, size);
+ obj = parse_object_buffer(the_repository, &obj_list[nr].oid,
+- type, size, buf,
++ type, cast_size_t_to_ulong(size), buf,
+ &eaten);
+ if (!obj)
+ die("invalid %s", type_name(type));
@@ builtin/unpack-objects.c: static void write_object(unsigned nr, enum object_type type,
}
@@ builtin/unpack-objects.c: static void added_object(unsigned nr, enum object_type
unsigned nr)
{
void *buf = get_data(size);
-@@ builtin/unpack-objects.c: static ssize_t zlib_stream_read(struct odb_stream *in_stream,
- return buf_len - zstream->avail_out;
- }
-
--static void stream_blob(unsigned long size, unsigned nr)
-+static void stream_blob(size_t size, unsigned nr)
- {
- git_zstream zstream = { 0 };
- struct zlib_stream in_stream = {
-@@ builtin/unpack-objects.c: static void stream_blob(unsigned long size, unsigned nr)
+@@ builtin/unpack-objects.c: static void stream_blob(size_t size, unsigned nr)
}
static int resolve_against_held(unsigned nr, const struct object_id *base,
2: 438f7c6ba2 = 2: 9952f0fb10 index-pack: use size_t in write_compressed()
3: 0258282745 ! 3: e66860d274 index-pack: use size_t for object sizes
@@ Commit message
Functions like sha1_object() and append_obj_to_pack() in
builtin/index-pack.c take object sizes as an "unsigned long", even
though struct object_entry already tracks them as size_t. On platforms
- where "long" is 32 bits, this truncates the size for objects of 4GiB or
- more.
+ where "long" is 32 bits while size_t is 64, such as 64-bit Windows,
+ this truncates the size of objects of 4GiB or more.
When receiving, on those platforms, a large object that already exists
locally, sha1_object() compares the real size (size_t) against the
@@ Commit message
Use size_t for the size parameter across these functions.
Triggering this requires processing 4GiB or more of input on a platform
- where "long" is 32 bits, so no test is added.
+ where "long" is 32 bits while size_t is 64, so no test is added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
4: 87b0d2bd96 ! 4: 00939c7f1b pack-check: use size_t for the size passed to verify callbacks
@@ Commit message
replicated in struct odb_fsck_options (odb.h) and implemented by
fsck_obj_buffer() in builtin/fsck.c.
- On platforms where "long" is 32 bits, running "git fsck" on packs with
- objects of 4GiB or more passes a truncated size to fsck_obj_buffer(),
- causing parse_object_buffer() and fsck_object() to inspect only a
- prefix.
+ On platforms where "long" is 32 bits while size_t is 64, such as
+ 64-bit Windows, running "git fsck" on packs with objects of 4GiB or
+ more passes a truncated size to fsck_obj_buffer(), causing
+ parse_object_buffer() and fsck_object() to inspect only a prefix.
Use size_t for the size parameter in verify_fn,
struct odb_fsck_options.object_cb, and fsck_obj_buffer().
+ parse_object_buffer() and fsck_object() themselves still take an
+ "unsigned long"; until they are updated separately, they keep
+ inspecting only a prefix.
Triggering this requires processing objects of 4GiB or more on a
- platform where "long" is 32 bits, so no test is added.
+ platform where "long" is 32 bits while size_t is 64, so no test is
+ added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
--
2.53.0
next prev parent reply other threads:[~2026-11-18 6:56 UTC|newest]
Thread overview: 10+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-31 11:39 [PATCH 0/4] use size_t when receiving and verifying packs Elia Pinto
2026-10-31 11:39 ` [PATCH 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-10-31 11:39 ` [PATCH 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-10-31 11:39 ` [PATCH 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
2026-11-18 6:56 ` Elia Pinto [this message]
2026-11-18 6:56 ` [PATCH v2 1/4] unpack-objects: use size_t for object sizes Elia Pinto
2026-11-18 6:56 ` [PATCH v2 2/4] index-pack: use size_t in write_compressed() Elia Pinto
2026-11-18 6:56 ` [PATCH v2 3/4] index-pack: use size_t for object sizes Elia Pinto
2026-11-18 6:56 ` [PATCH v2 4/4] pack-check: use size_t for the size passed to verify callbacks Elia Pinto
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1794984184.git.gitter.spiros@gmail.com \
--to=gitter.spiros@gmail.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=avarab@gmail.com \
--cc=chiyutianyi@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=hanxin.hx@alibaba-inc.com \
--cc=jltobler@gmail.com \
--cc=jonathantanmy@fastmail.com \
--cc=krka@spotify.com \
--cc=ps@pks.im \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox