Git development
 help / color / mirror / Atom feed
From: Elia Pinto <gitter.spiros@gmail.com>
To: git@vger.kernel.org
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Elia Pinto" <gitter.spiros@gmail.com>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Toon Claes" <toon@iotcl.com>, "Patrick Steinhardt" <ps@pks.im>,
	"Jeff King" <peff@peff.net>, "René Scharfe" <l.s.r@web.de>,
	"Jonathan Tan" <jonathantanmy@fastmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v3 0/5] loose object size checks and a few size truncations
Date: Thu, 19 Nov 2026 07:21:45 +0100	[thread overview]
Message-ID: <cover.1795069177.git.gitter.spiros@gmail.com> (raw)
In-Reply-To: <cover.1794914210.git.gitter.spiros@gmail.com>

Changes since v2:

 - 1/5: a size mismatch now gets its own error message, "loose object
   '<oid>' has N bytes, but its header says M", as Patrick suggested,
   and the check is written as "total_out != hdrlen + size", as Junio
   preferred. The commit message explains the case of an object that
   fits entirely into the header buffer better.

 - 3/5: the new check no longer prints an error. Like the other
   failures in read_istream_loose() it returns -1 and leaves the
   message to the caller, as Junio suggested, so the oid member it
   needed is gone.

 - 2/5 now says why no st_add() is needed: the size is parsed with
   st_mult() and st_add(), and allocated before any other arithmetic
   is done on it.

 - Shorter commit messages, 4/5 in particular.

Not done: Patrick's suggestion to use memchr() instead of strlen() in
unpack_loose_rest(). parse_loose_header() has already made sure that
the header ends with a NUL, so I would leave that for a separate
patch.

The series, for those who missed v1 and v2:

 1/5 A loose object whose stream is shorter than the size in its header
     hands out uninitialized memory; a longer one is silently cut (all
     platforms).

 2/5 Where "long" has 32 bits while size_t has 64, the size from the
     loose object header is truncated and callers read past the buffer.

 3/5 The same check as 1/5 for the streaming reader.

 4/5 "git cat-file --batch" writes memory past the end of objects
     between 2GiB and 4GiB (all platforms).

 5/5 On 64-bit Windows, concatenating notes of 4GiB or more overflows a
     heap buffer.

All but 3/5 went to the git-security list first; they do not consider
them security issues. 1/5 and 3/5 have tests; the others would need
objects of several GiB, so their messages say how to reproduce them.

5/5 passes the size to odb_write_object() through
cast_size_t_to_ulong(), as its length is still an "unsigned long" on
master; with 1/4 of [1] the cast becomes unnecessary but harmless.

Based on master at e1ec09b23c (The 22nd batch).

[1] https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/

Elia Pinto (5):
  object-file: check size of inflated loose object against its header
  object-file: use size_t for loose object sizes
  odb/source-loose: check object size when streaming loose objects
  cat-file: use size_t for the length passed to batch_write()
  notes: avoid truncating the size of concatenated notes

 builtin/cat-file.c           |  2 +-
 notes.c                      |  7 +++----
 object-file.c                | 31 +++++++++++++++++++++----------
 object-file.h                |  6 +++---
 odb/source-loose.c           | 19 +++++++++++++++----
 t/lib-loose.sh               |  5 ++++-
 t/t1006-cat-file.sh          | 18 ++++++++++++++++++
 t/t1060-object-corruption.sh | 28 ++++++++++++++++++++++++++++
 8 files changed, 93 insertions(+), 23 deletions(-)

Range-diff against v2:
1:  edb5f34498 ! 1:  0d748dbd8e object-file: check size of inflated loose object against its header
    @@ Commit message
         object-file: check size of inflated loose object against its header
     
         unpack_loose_rest() allocates a buffer of the size given in the object
    -    header and inflates the rest of the stream into it, but it never checks
    -    that the stream actually produced that many bytes. If the compressed
    -    data ends early we return success with the tail of the buffer never
    -    written, and callers like odb_read_object() hand it out as object
    -    content. For example, "git cat-file commit" on a loose object whose
    -    header says "commit 200" but which contains only 44 bytes prints 156
    -    bytes of uninitialized heap memory.
    +    header, but never checks that the stream produced that many bytes:
     
    -    The opposite case slips through, too: when the whole object fits into
    -    the buffer filled by unpack_loose_header(), we copy only "size" bytes
    -    out of it, the stream has already ended, and the extra data is silently
    -    dropped.
    +     - If the stream is shorter, the unwritten tail of the buffer is handed
    +       out as object content: "git cat-file commit" on an object whose
    +       header says "commit 200" but which holds 44 bytes prints 156 bytes
    +       of uninitialized heap memory.
     
    -    The packfile reader in unpack_compressed_entry() and index-pack already
    -    compare total_out with the expected size. Do the same here and report
    -    the object as corrupt if they differ. Today fsck notices the first case
    -    only by accident, because it hashes the uninitialized bytes and gets a
    -    mismatch.
    +     - If the whole object fits into the MAX_HEADER_LEN bytes that
    +       unpack_loose_header() already inflated, the stream has ended before
    +       we get here; we copy only "size" bytes out of that buffer and
    +       silently drop the rest.
     
    -    Teach loose_obj() in lib-loose.sh to write a bogus size, so that the
    +    Compare total_out with the expected size, as unpack_compressed_entry()
    +    and index-pack already do, and say how many bytes we got when they
    +    differ.
    +
    +    Teach loose_obj() in t/lib-loose.sh to write a bogus size, so that the
         tests can create such objects.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
    @@ object-file.c: void *unpack_loose_rest(git_zstream *stream,
      	if (bytes <= size) {
      		/*
     @@ object-file.c: void *unpack_loose_rest(git_zstream *stream,
    - 		}
    - 	}
    - 
    --	if (status != Z_STREAM_END) {
    -+	/*
    -+	 * Make sure that the stream produced exactly as many bytes as the
    -+	 * header claims. Otherwise we would hand back a buffer whose tail
    -+	 * was never written (if the stream is short), or silently drop data
    -+	 * (if the stream is long but ended before we noticed).
    -+	 */
    -+	if (status != Z_STREAM_END ||
    -+	    stream->total_out - hdrlen != size) {
    + 	if (status != Z_STREAM_END) {
      		error(_("corrupt loose object '%s'"), oid_to_hex(oid));
      		FREE_AND_NULL(buf);
    ++	} else if (stream->total_out != hdrlen + size) {
    ++		/*
    ++		 * A short stream would leave the tail of buf[] unwritten;
    ++		 * a long one can end inside the header buffer, so that
    ++		 * only "size" bytes were copied and the rest dropped.
    ++		 */
    ++		error(_("loose object '%s' has %"PRIuMAX" bytes, "
    ++			"but its header says %"PRIuMAX),
    ++		      oid_to_hex(oid), (uintmax_t)(stream->total_out - hdrlen),
    ++		      (uintmax_t)size);
    ++		FREE_AND_NULL(buf);
      	} else if (stream->avail_in) {
    + 		error(_("garbage at end of loose object '%s'"),
    + 		      oid_to_hex(oid));
     
      ## t/lib-loose.sh ##
     @@
    @@ t/t1060-object-corruption.sh: test_expect_success 'corrupted loose commit can be
     +
     +test_expect_success 'reading loose object shorter than its header fails' '
     +	test_must_fail git -C bogus-size cat-file commit $(cat oid-too-big) 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "has 5 bytes, but its header says 200" err
     +'
     +
     +test_expect_success 'reading loose object longer than its header fails' '
     +	test_must_fail git -C bogus-size cat-file commit $(cat oid-too-small) 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "has 11 bytes, but its header says 3" err
     +'
     +
     +test_expect_success 'fsck reports loose objects with bogus header size' '
     +	test_must_fail git -C bogus-size fsck 2>err &&
    -+	test_grep "corrupt loose object .$(cat oid-too-big)" err &&
    -+	test_grep "corrupt loose object .$(cat oid-too-small)" err
    ++	test_grep "loose object .$(cat oid-too-big). has 5 bytes" err &&
    ++	test_grep "loose object .$(cat oid-too-small). has 11 bytes" err
     +'
     +
      test_done
2:  eca5d3ddb8 ! 2:  c9facd8b3b object-file: use size_t for loose object sizes
    @@ Commit message
     
         Since c6a4629e32 (odb: use size_t for object_info.sizep and the size
         APIs, 2026-06-15), parse_loose_header() reports the object size as a
    -    size_t, but unpack_loose_rest() still takes it as an "unsigned long".
    -    On 64-bit Windows, where "long" has 32 bits while size_t has 64, the
    -    size is truncated on the way. A header claiming 4GiB + 5 bytes thus
    -    makes us allocate a 6-byte buffer, the size check added by the
    -    previous commit is happy because the stream really has 5 bytes, and
    -    the caller gets that buffer back together with *oi->sizep set to
    -    4GiB + 5. Any user of the object then reads far beyond the end of the
    -    buffer; read_loose_object() already does so when it hashes the
    -    contents.
    +    size_t, but unpack_loose_rest() takes it as an "unsigned long". Where
    +    "long" has 32 bits while size_t has 64, as on 64-bit Windows, a header
    +    claiming 4GiB + 5 bytes makes us allocate 6 bytes; the check added by
    +    the previous commit passes, as the stream does hold 5 bytes, and the
    +    caller then reads the object as 4GiB + 5 bytes long.
     
    -    Use size_t for the size in unpack_loose_rest() and check_stream_oid(),
    -    and while at it for the size of the mapped file in map_fd(),
    -    odb_source_loose_map_object() and unpack_loose_header(), so that the
    -    whole loose object reading path agrees on the type.
    +    Use size_t along the whole loose object reading path. No st_add() is
    +    needed on the way: the size is parsed with st_mult() and st_add(), and
    +    a buffer of that size is allocated with xmallocz() before anything is
    +    added to it or subtracted from it.
     
    -    There is no test, as the reader allocates a buffer of the size claimed
    -    in the header before it can notice that the stream is short, so a test
    -    would need 4GiB of memory. The problem can be reproduced on Windows
    -    with the loose_obj() helper from the previous commit:
    +    No test, as the reader allocates the claimed size before it notices
    +    the short stream. On Windows,
     
                 printf short |
                 loose_obj .git/objects commit $((4 * 1024 * 1024 * 1024 + 5))
     
    -    followed by "git cat-file commit" on the resulting object ID.
    +    and "git cat-file commit" on the resulting object reproduce it.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
3:  a8572aa836 ! 3:  c694d3e128 odb/source-loose: check object size when streaming loose objects
    @@ Metadata
      ## Commit message ##
         odb/source-loose: check object size when streaming loose objects
     
    -    read_istream_loose() inflates a loose object until zlib reports the end
    -    of the stream, but never compares the number of bytes it produced with
    -    the size given in the object header. A corrupt object whose stream is
    -    shorter than its header claims is thus streamed out without any error,
    -    e.g. "git cat-file blob" exits with 0 after writing fewer bytes than
    -    "git cat-file -s" reports. The same happens if the stream is longer than
    -    the header claims.
    +    read_istream_loose() streams a loose object until zlib reports the end
    +    of the stream, but never compares the number of bytes with the size in
    +    the object header. "git cat-file blob" on an object whose stream is
    +    shorter or longer than its header claims thus exits with 0, writing a
    +    different number of bytes than "git cat-file -s" reports.
     
    -    Keep track of how many bytes we handed out and report the object as
    -    corrupt if that does not match the size from the header once the stream
    -    ends.
    -
    -    The tests create such objects with the loose_obj() helper, which can
    -    already write a bogus size into the header.
    +    Count the bytes handed out and fail the read if they do not match once
    +    the stream ends. Like the other errors in this function, this returns
    +    -1 silently and leaves the message to the caller, e.g. "unable to
    +    stream <oid> to stdout" from cat-file.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
    @@ odb/source-loose.c: struct odb_loose_read_stream {
      	int hdr_avail;
      	int hdr_used;
     +	size_t total_read;
    -+	struct object_id oid;
      };
      
      static ssize_t read_istream_loose(struct odb_stream *_st, char *buf, size_t sz)
    @@ odb/source-loose.c: static ssize_t read_istream_loose(struct odb_stream *_st, ch
      			git_inflate_end(&st->z);
     +			/*
     +			 * The object must be exactly as large as its
    -+			 * header claims.
    ++			 * header claims; like the other errors below,
    ++			 * this is reported by our caller.
     +			 */
     +			if (st->total_read + total_read != st->base.size) {
    -+				error(_("corrupt loose object '%s'"),
    -+				      oid_to_hex(&st->oid));
     +				st->z_state = ODB_LOOSE_READ_STREAM_ERROR;
     +				return -1;
     +			}
    @@ odb/source-loose.c: static ssize_t read_istream_loose(struct odb_stream *_st, ch
      	return total_read;
      }
      
    -@@ odb/source-loose.c: static int odb_source_loose_read_object_stream(struct odb_stream **out,
    - 
    - 	st->mapped = mapped;
    - 	st->mapsize = mapsize;
    -+	oidcpy(&st->oid, oid);
    - 	st->hdr_used = strlen(st->hdr) + 1;
    - 	st->hdr_avail = st->z.total_out;
    - 	st->z_state = ODB_LOOSE_READ_STREAM_INUSE;
     
      ## t/t1006-cat-file.sh ##
     @@ t/t1006-cat-file.sh: test_expect_success PIPE 'concurrently written alternates are loaded' '
    @@ t/t1006-cat-file.sh: test_expect_success PIPE 'concurrently written alternates a
     +
     +test_expect_success 'streaming blob shorter than its header fails' '
     +	test_must_fail git -C bogus-size cat-file blob $(cat oid-too-big) >out 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "unable to stream" err
     +'
     +
     +test_expect_success 'streaming blob longer than its header fails' '
     +	test_must_fail git -C bogus-size cat-file blob $(cat oid-too-small) >out 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "unable to stream" err
     +'
     +
      test_done
4:  5b61a1b089 ! 4:  99bf45c606 cat-file: use size_t for the length passed to batch_write()
    @@ Commit message
         cat-file: use size_t for the length passed to batch_write()
     
         batch_write() takes the length of the data as an "int", but its callers
    -    pass the size of whole objects as returned by odb_read_object() or
    -    filter_object(), which is a size_t. For an object between 2GiB and
    -    4GiB the length wraps to a negative value, which fwrite() and
    -    write_or_die() then see as an enormous size_t. We happily write the
    -    object and then keep going past the end of the buffer, sending whatever
    -    memory follows to stdout, until we hit an unmapped page and die with
    -    "write error: Bad address". Objects of 4GiB and more are truncated
    -    instead, which desynchronizes the --batch output.
    -
    -    This affects all platforms, not only those with a 32-bit "long". It can
    -    be triggered with "--batch" on any non-blob object, or on blobs with
    -    --filters or --textconv, as blobs are otherwise streamed. For example:
    -
    -            blob=$(test-tool genzeros $((3 * 1024 * 1024 * 1024)) |
    -                   git hash-object -w --stdin) &&
    -            echo "$blob big" |
    -            git cat-file --batch="%(objectname) %(rest)" --filters >out
    -
    -    fails with "write error: Bad address"; with stdout going to /dev/null,
    -    strace shows write() being called on addresses far beyond the end of the
    -    3GiB buffer. There is no test, as it would need several GiB of memory
    -    and disk.
    -
    -    Both fwrite() and write_or_die() take a size_t already, so just change
    -    the parameter type.
    +    pass the size of whole objects, a size_t. On all platforms, an object
    +    between 2GiB and 4GiB gets a negative length, which fwrite() and
    +    write_or_die() see as an enormous size_t: "git cat-file --batch" writes
    +    memory past the end of the object to stdout until it dies with "write
    +    error: Bad address". Larger objects are truncated instead. Blobs are
    +    affected only with --filters or --textconv, as they are otherwise
    +    streamed.
    +
    +    Both callees take a size_t, so just change the parameter type.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
5:  04f320d1de = 5:  d0cc63a8ed notes: avoid truncating the size of concatenated notes

base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
-- 
2.53.0


  parent reply	other threads:[~2026-11-19  6:22 UTC|newest]

Thread overview: 26+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-30 10:28 [PATCH 0/6] loose object size checks and a few size truncations Elia Pinto
2026-10-30 10:28 ` [PATCH 1/6] object-file: check size of inflated loose object against its header Elia Pinto
2026-10-30 10:28 ` [PATCH 2/6] object-file: use size_t for loose object sizes Elia Pinto
2026-10-30 10:28 ` [PATCH 3/6] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-10-30 10:28 ` [PATCH 4/6] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-10-30 10:28 ` [PATCH 5/6] notes: avoid truncating the size of concatenated notes Elia Pinto
2026-10-30 10:28 ` [PATCH 6/6] diff: use size_t in count_trailing_blank() Elia Pinto
2026-11-17 11:38 ` [PATCH v2 0/5] loose object size checks and a few size truncations Elia Pinto
2026-11-17 11:38   ` [PATCH v2 1/5] object-file: check size of inflated loose object against its header Elia Pinto
2026-11-17 19:33     ` Junio C Hamano
2026-11-18 12:21     ` Patrick Steinhardt
2026-11-18 18:17       ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 2/5] object-file: use size_t for loose object sizes Elia Pinto
2026-11-17 22:34     ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 3/5] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-11-17 22:55     ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 4/5] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-11-17 22:59     ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 5/5] notes: avoid truncating the size of concatenated notes Elia Pinto
2026-11-17 23:01     ` Junio C Hamano
2026-11-19  6:21   ` Elia Pinto [this message]
2026-11-19  6:21     ` [PATCH v3 1/5] object-file: check size of inflated loose object against its header Elia Pinto
2026-11-19  6:21     ` [PATCH v3 2/5] object-file: use size_t for loose object sizes Elia Pinto
2026-11-19  6:21     ` [PATCH v3 3/5] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-11-19  6:21     ` [PATCH v3 4/5] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-11-19  6:21     ` [PATCH v3 5/5] notes: avoid truncating the size of concatenated notes Elia Pinto

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1795069177.git.gitter.spiros@gmail.com \
    --to=gitter.spiros@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jonathantanmy@fastmail.com \
    --cc=l.s.r@web.de \
    --cc=peff@peff.net \
    --cc=ps@pks.im \
    --cc=toon@iotcl.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox