From: Elia Pinto <gitter.spiros@gmail.com>
To: git@vger.kernel.org
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
"Elia Pinto" <gitter.spiros@gmail.com>,
"Junio C Hamano" <gitster@pobox.com>,
"Toon Claes" <toon@iotcl.com>, "Patrick Steinhardt" <ps@pks.im>,
"Jeff King" <peff@peff.net>, "René Scharfe" <l.s.r@web.de>,
"Jonathan Tan" <jonathantanmy@fastmail.com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v3 0/5] loose object size checks and a few size truncations
Date: Thu, 19 Nov 2026 07:21:45 +0100 [thread overview]
Message-ID: <cover.1795069177.git.gitter.spiros@gmail.com> (raw)
In-Reply-To: <cover.1794914210.git.gitter.spiros@gmail.com>
Changes since v2:
- 1/5: a size mismatch now gets its own error message, "loose object
'<oid>' has N bytes, but its header says M", as Patrick suggested,
and the check is written as "total_out != hdrlen + size", as Junio
preferred. The commit message explains the case of an object that
fits entirely into the header buffer better.
- 3/5: the new check no longer prints an error. Like the other
failures in read_istream_loose() it returns -1 and leaves the
message to the caller, as Junio suggested, so the oid member it
needed is gone.
- 2/5 now says why no st_add() is needed: the size is parsed with
st_mult() and st_add(), and allocated before any other arithmetic
is done on it.
- Shorter commit messages, 4/5 in particular.
Not done: Patrick's suggestion to use memchr() instead of strlen() in
unpack_loose_rest(). parse_loose_header() has already made sure that
the header ends with a NUL, so I would leave that for a separate
patch.
The series, for those who missed v1 and v2:
1/5 A loose object whose stream is shorter than the size in its header
hands out uninitialized memory; a longer one is silently cut (all
platforms).
2/5 Where "long" has 32 bits while size_t has 64, the size from the
loose object header is truncated and callers read past the buffer.
3/5 The same check as 1/5 for the streaming reader.
4/5 "git cat-file --batch" writes memory past the end of objects
between 2GiB and 4GiB (all platforms).
5/5 On 64-bit Windows, concatenating notes of 4GiB or more overflows a
heap buffer.
All but 3/5 went to the git-security list first; they do not consider
them security issues. 1/5 and 3/5 have tests; the others would need
objects of several GiB, so their messages say how to reproduce them.
5/5 passes the size to odb_write_object() through
cast_size_t_to_ulong(), as its length is still an "unsigned long" on
master; with 1/4 of [1] the cast becomes unnecessary but harmless.
Based on master at e1ec09b23c (The 22nd batch).
[1] https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/
Elia Pinto (5):
object-file: check size of inflated loose object against its header
object-file: use size_t for loose object sizes
odb/source-loose: check object size when streaming loose objects
cat-file: use size_t for the length passed to batch_write()
notes: avoid truncating the size of concatenated notes
builtin/cat-file.c | 2 +-
notes.c | 7 +++----
object-file.c | 31 +++++++++++++++++++++----------
object-file.h | 6 +++---
odb/source-loose.c | 19 +++++++++++++++----
t/lib-loose.sh | 5 ++++-
t/t1006-cat-file.sh | 18 ++++++++++++++++++
t/t1060-object-corruption.sh | 28 ++++++++++++++++++++++++++++
8 files changed, 93 insertions(+), 23 deletions(-)
Range-diff against v2:
1: edb5f34498 ! 1: 0d748dbd8e object-file: check size of inflated loose object against its header
@@ Commit message
object-file: check size of inflated loose object against its header
unpack_loose_rest() allocates a buffer of the size given in the object
- header and inflates the rest of the stream into it, but it never checks
- that the stream actually produced that many bytes. If the compressed
- data ends early we return success with the tail of the buffer never
- written, and callers like odb_read_object() hand it out as object
- content. For example, "git cat-file commit" on a loose object whose
- header says "commit 200" but which contains only 44 bytes prints 156
- bytes of uninitialized heap memory.
+ header, but never checks that the stream produced that many bytes:
- The opposite case slips through, too: when the whole object fits into
- the buffer filled by unpack_loose_header(), we copy only "size" bytes
- out of it, the stream has already ended, and the extra data is silently
- dropped.
+ - If the stream is shorter, the unwritten tail of the buffer is handed
+ out as object content: "git cat-file commit" on an object whose
+ header says "commit 200" but which holds 44 bytes prints 156 bytes
+ of uninitialized heap memory.
- The packfile reader in unpack_compressed_entry() and index-pack already
- compare total_out with the expected size. Do the same here and report
- the object as corrupt if they differ. Today fsck notices the first case
- only by accident, because it hashes the uninitialized bytes and gets a
- mismatch.
+ - If the whole object fits into the MAX_HEADER_LEN bytes that
+ unpack_loose_header() already inflated, the stream has ended before
+ we get here; we copy only "size" bytes out of that buffer and
+ silently drop the rest.
- Teach loose_obj() in lib-loose.sh to write a bogus size, so that the
+ Compare total_out with the expected size, as unpack_compressed_entry()
+ and index-pack already do, and say how many bytes we got when they
+ differ.
+
+ Teach loose_obj() in t/lib-loose.sh to write a bogus size, so that the
tests can create such objects.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
@@ object-file.c: void *unpack_loose_rest(git_zstream *stream,
if (bytes <= size) {
/*
@@ object-file.c: void *unpack_loose_rest(git_zstream *stream,
- }
- }
-
-- if (status != Z_STREAM_END) {
-+ /*
-+ * Make sure that the stream produced exactly as many bytes as the
-+ * header claims. Otherwise we would hand back a buffer whose tail
-+ * was never written (if the stream is short), or silently drop data
-+ * (if the stream is long but ended before we noticed).
-+ */
-+ if (status != Z_STREAM_END ||
-+ stream->total_out - hdrlen != size) {
+ if (status != Z_STREAM_END) {
error(_("corrupt loose object '%s'"), oid_to_hex(oid));
FREE_AND_NULL(buf);
++ } else if (stream->total_out != hdrlen + size) {
++ /*
++ * A short stream would leave the tail of buf[] unwritten;
++ * a long one can end inside the header buffer, so that
++ * only "size" bytes were copied and the rest dropped.
++ */
++ error(_("loose object '%s' has %"PRIuMAX" bytes, "
++ "but its header says %"PRIuMAX),
++ oid_to_hex(oid), (uintmax_t)(stream->total_out - hdrlen),
++ (uintmax_t)size);
++ FREE_AND_NULL(buf);
} else if (stream->avail_in) {
+ error(_("garbage at end of loose object '%s'"),
+ oid_to_hex(oid));
## t/lib-loose.sh ##
@@
@@ t/t1060-object-corruption.sh: test_expect_success 'corrupted loose commit can be
+
+test_expect_success 'reading loose object shorter than its header fails' '
+ test_must_fail git -C bogus-size cat-file commit $(cat oid-too-big) 2>err &&
-+ test_grep "corrupt loose object" err
++ test_grep "has 5 bytes, but its header says 200" err
+'
+
+test_expect_success 'reading loose object longer than its header fails' '
+ test_must_fail git -C bogus-size cat-file commit $(cat oid-too-small) 2>err &&
-+ test_grep "corrupt loose object" err
++ test_grep "has 11 bytes, but its header says 3" err
+'
+
+test_expect_success 'fsck reports loose objects with bogus header size' '
+ test_must_fail git -C bogus-size fsck 2>err &&
-+ test_grep "corrupt loose object .$(cat oid-too-big)" err &&
-+ test_grep "corrupt loose object .$(cat oid-too-small)" err
++ test_grep "loose object .$(cat oid-too-big). has 5 bytes" err &&
++ test_grep "loose object .$(cat oid-too-small). has 11 bytes" err
+'
+
test_done
2: eca5d3ddb8 ! 2: c9facd8b3b object-file: use size_t for loose object sizes
@@ Commit message
Since c6a4629e32 (odb: use size_t for object_info.sizep and the size
APIs, 2026-06-15), parse_loose_header() reports the object size as a
- size_t, but unpack_loose_rest() still takes it as an "unsigned long".
- On 64-bit Windows, where "long" has 32 bits while size_t has 64, the
- size is truncated on the way. A header claiming 4GiB + 5 bytes thus
- makes us allocate a 6-byte buffer, the size check added by the
- previous commit is happy because the stream really has 5 bytes, and
- the caller gets that buffer back together with *oi->sizep set to
- 4GiB + 5. Any user of the object then reads far beyond the end of the
- buffer; read_loose_object() already does so when it hashes the
- contents.
+ size_t, but unpack_loose_rest() takes it as an "unsigned long". Where
+ "long" has 32 bits while size_t has 64, as on 64-bit Windows, a header
+ claiming 4GiB + 5 bytes makes us allocate 6 bytes; the check added by
+ the previous commit passes, as the stream does hold 5 bytes, and the
+ caller then reads the object as 4GiB + 5 bytes long.
- Use size_t for the size in unpack_loose_rest() and check_stream_oid(),
- and while at it for the size of the mapped file in map_fd(),
- odb_source_loose_map_object() and unpack_loose_header(), so that the
- whole loose object reading path agrees on the type.
+ Use size_t along the whole loose object reading path. No st_add() is
+ needed on the way: the size is parsed with st_mult() and st_add(), and
+ a buffer of that size is allocated with xmallocz() before anything is
+ added to it or subtracted from it.
- There is no test, as the reader allocates a buffer of the size claimed
- in the header before it can notice that the stream is short, so a test
- would need 4GiB of memory. The problem can be reproduced on Windows
- with the loose_obj() helper from the previous commit:
+ No test, as the reader allocates the claimed size before it notices
+ the short stream. On Windows,
printf short |
loose_obj .git/objects commit $((4 * 1024 * 1024 * 1024 + 5))
- followed by "git cat-file commit" on the resulting object ID.
+ and "git cat-file commit" on the resulting object reproduce it.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
3: a8572aa836 ! 3: c694d3e128 odb/source-loose: check object size when streaming loose objects
@@ Metadata
## Commit message ##
odb/source-loose: check object size when streaming loose objects
- read_istream_loose() inflates a loose object until zlib reports the end
- of the stream, but never compares the number of bytes it produced with
- the size given in the object header. A corrupt object whose stream is
- shorter than its header claims is thus streamed out without any error,
- e.g. "git cat-file blob" exits with 0 after writing fewer bytes than
- "git cat-file -s" reports. The same happens if the stream is longer than
- the header claims.
+ read_istream_loose() streams a loose object until zlib reports the end
+ of the stream, but never compares the number of bytes with the size in
+ the object header. "git cat-file blob" on an object whose stream is
+ shorter or longer than its header claims thus exits with 0, writing a
+ different number of bytes than "git cat-file -s" reports.
- Keep track of how many bytes we handed out and report the object as
- corrupt if that does not match the size from the header once the stream
- ends.
-
- The tests create such objects with the loose_obj() helper, which can
- already write a bogus size into the header.
+ Count the bytes handed out and fail the read if they do not match once
+ the stream ends. Like the other errors in this function, this returns
+ -1 silently and leaves the message to the caller, e.g. "unable to
+ stream <oid> to stdout" from cat-file.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
@@ odb/source-loose.c: struct odb_loose_read_stream {
int hdr_avail;
int hdr_used;
+ size_t total_read;
-+ struct object_id oid;
};
static ssize_t read_istream_loose(struct odb_stream *_st, char *buf, size_t sz)
@@ odb/source-loose.c: static ssize_t read_istream_loose(struct odb_stream *_st, ch
git_inflate_end(&st->z);
+ /*
+ * The object must be exactly as large as its
-+ * header claims.
++ * header claims; like the other errors below,
++ * this is reported by our caller.
+ */
+ if (st->total_read + total_read != st->base.size) {
-+ error(_("corrupt loose object '%s'"),
-+ oid_to_hex(&st->oid));
+ st->z_state = ODB_LOOSE_READ_STREAM_ERROR;
+ return -1;
+ }
@@ odb/source-loose.c: static ssize_t read_istream_loose(struct odb_stream *_st, ch
return total_read;
}
-@@ odb/source-loose.c: static int odb_source_loose_read_object_stream(struct odb_stream **out,
-
- st->mapped = mapped;
- st->mapsize = mapsize;
-+ oidcpy(&st->oid, oid);
- st->hdr_used = strlen(st->hdr) + 1;
- st->hdr_avail = st->z.total_out;
- st->z_state = ODB_LOOSE_READ_STREAM_INUSE;
## t/t1006-cat-file.sh ##
@@ t/t1006-cat-file.sh: test_expect_success PIPE 'concurrently written alternates are loaded' '
@@ t/t1006-cat-file.sh: test_expect_success PIPE 'concurrently written alternates a
+
+test_expect_success 'streaming blob shorter than its header fails' '
+ test_must_fail git -C bogus-size cat-file blob $(cat oid-too-big) >out 2>err &&
-+ test_grep "corrupt loose object" err
++ test_grep "unable to stream" err
+'
+
+test_expect_success 'streaming blob longer than its header fails' '
+ test_must_fail git -C bogus-size cat-file blob $(cat oid-too-small) >out 2>err &&
-+ test_grep "corrupt loose object" err
++ test_grep "unable to stream" err
+'
+
test_done
4: 5b61a1b089 ! 4: 99bf45c606 cat-file: use size_t for the length passed to batch_write()
@@ Commit message
cat-file: use size_t for the length passed to batch_write()
batch_write() takes the length of the data as an "int", but its callers
- pass the size of whole objects as returned by odb_read_object() or
- filter_object(), which is a size_t. For an object between 2GiB and
- 4GiB the length wraps to a negative value, which fwrite() and
- write_or_die() then see as an enormous size_t. We happily write the
- object and then keep going past the end of the buffer, sending whatever
- memory follows to stdout, until we hit an unmapped page and die with
- "write error: Bad address". Objects of 4GiB and more are truncated
- instead, which desynchronizes the --batch output.
-
- This affects all platforms, not only those with a 32-bit "long". It can
- be triggered with "--batch" on any non-blob object, or on blobs with
- --filters or --textconv, as blobs are otherwise streamed. For example:
-
- blob=$(test-tool genzeros $((3 * 1024 * 1024 * 1024)) |
- git hash-object -w --stdin) &&
- echo "$blob big" |
- git cat-file --batch="%(objectname) %(rest)" --filters >out
-
- fails with "write error: Bad address"; with stdout going to /dev/null,
- strace shows write() being called on addresses far beyond the end of the
- 3GiB buffer. There is no test, as it would need several GiB of memory
- and disk.
-
- Both fwrite() and write_or_die() take a size_t already, so just change
- the parameter type.
+ pass the size of whole objects, a size_t. On all platforms, an object
+ between 2GiB and 4GiB gets a negative length, which fwrite() and
+ write_or_die() see as an enormous size_t: "git cat-file --batch" writes
+ memory past the end of the object to stdout until it dies with "write
+ error: Bad address". Larger objects are truncated instead. Blobs are
+ affected only with --filters or --textconv, as they are otherwise
+ streamed.
+
+ Both callees take a size_t, so just change the parameter type.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
5: 04f320d1de = 5: d0cc63a8ed notes: avoid truncating the size of concatenated notes
base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
--
2.53.0
next prev parent reply other threads:[~2026-11-19 6:22 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-30 10:28 [PATCH 0/6] loose object size checks and a few size truncations Elia Pinto
2026-10-30 10:28 ` [PATCH 1/6] object-file: check size of inflated loose object against its header Elia Pinto
2026-10-30 10:28 ` [PATCH 2/6] object-file: use size_t for loose object sizes Elia Pinto
2026-10-30 10:28 ` [PATCH 3/6] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-10-30 10:28 ` [PATCH 4/6] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-10-30 10:28 ` [PATCH 5/6] notes: avoid truncating the size of concatenated notes Elia Pinto
2026-10-30 10:28 ` [PATCH 6/6] diff: use size_t in count_trailing_blank() Elia Pinto
2026-11-17 11:38 ` [PATCH v2 0/5] loose object size checks and a few size truncations Elia Pinto
2026-11-17 11:38 ` [PATCH v2 1/5] object-file: check size of inflated loose object against its header Elia Pinto
2026-11-17 19:33 ` Junio C Hamano
2026-11-18 12:21 ` Patrick Steinhardt
2026-11-18 18:17 ` Junio C Hamano
2026-11-17 11:38 ` [PATCH v2 2/5] object-file: use size_t for loose object sizes Elia Pinto
2026-11-17 22:34 ` Junio C Hamano
2026-11-17 11:38 ` [PATCH v2 3/5] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-11-17 22:55 ` Junio C Hamano
2026-11-17 11:38 ` [PATCH v2 4/5] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-11-17 22:59 ` Junio C Hamano
2026-11-17 11:38 ` [PATCH v2 5/5] notes: avoid truncating the size of concatenated notes Elia Pinto
2026-11-17 23:01 ` Junio C Hamano
2026-11-19 6:21 ` Elia Pinto [this message]
2026-11-19 6:21 ` [PATCH v3 1/5] object-file: check size of inflated loose object against its header Elia Pinto
2026-11-19 6:21 ` [PATCH v3 2/5] object-file: use size_t for loose object sizes Elia Pinto
2026-11-19 6:21 ` [PATCH v3 3/5] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-11-19 6:21 ` [PATCH v3 4/5] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-11-19 6:21 ` [PATCH v3 5/5] notes: avoid truncating the size of concatenated notes Elia Pinto
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1795069177.git.gitter.spiros@gmail.com \
--to=gitter.spiros@gmail.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jonathantanmy@fastmail.com \
--cc=l.s.r@web.de \
--cc=peff@peff.net \
--cc=ps@pks.im \
--cc=toon@iotcl.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox