All of lore.kernel.org
 help / color / mirror / Atom feed
From: Elia Pinto <gitter.spiros@gmail.com>
To: git@vger.kernel.org
Cc: "Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Elia Pinto" <gitter.spiros@gmail.com>,
	"Junio C Hamano" <gitster@pobox.com>,
	"Toon Claes" <toon@iotcl.com>, "Patrick Steinhardt" <ps@pks.im>,
	"Jeff King" <peff@peff.net>, "René Scharfe" <l.s.r@web.de>,
	"Jonathan Tan" <jonathantanmy@fastmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v3 0/5] loose object size checks and a few size truncations
Date: Thu, 19 Nov 2026 07:21:45 +0100	[thread overview]
Message-ID: <cover.1795069177.git.gitter.spiros@gmail.com> (raw)
In-Reply-To: <cover.1794914210.git.gitter.spiros@gmail.com>

Changes since v2:

 - 1/5: a size mismatch now gets its own error message, "loose object
   '<oid>' has N bytes, but its header says M", as Patrick suggested,
   and the check is written as "total_out != hdrlen + size", as Junio
   preferred. The commit message explains the case of an object that
   fits entirely into the header buffer better.

 - 3/5: the new check no longer prints an error. Like the other
   failures in read_istream_loose() it returns -1 and leaves the
   message to the caller, as Junio suggested, so the oid member it
   needed is gone.

 - 2/5 now says why no st_add() is needed: the size is parsed with
   st_mult() and st_add(), and allocated before any other arithmetic
   is done on it.

 - Shorter commit messages, 4/5 in particular.

Not done: Patrick's suggestion to use memchr() instead of strlen() in
unpack_loose_rest(). parse_loose_header() has already made sure that
the header ends with a NUL, so I would leave that for a separate
patch.

The series, for those who missed v1 and v2:

 1/5 A loose object whose stream is shorter than the size in its header
     hands out uninitialized memory; a longer one is silently cut (all
     platforms).

 2/5 Where "long" has 32 bits while size_t has 64, the size from the
     loose object header is truncated and callers read past the buffer.

 3/5 The same check as 1/5 for the streaming reader.

 4/5 "git cat-file --batch" writes memory past the end of objects
     between 2GiB and 4GiB (all platforms).

 5/5 On 64-bit Windows, concatenating notes of 4GiB or more overflows a
     heap buffer.

All but 3/5 went to the git-security list first; they do not consider
them security issues. 1/5 and 3/5 have tests; the others would need
objects of several GiB, so their messages say how to reproduce them.

5/5 passes the size to odb_write_object() through
cast_size_t_to_ulong(), as its length is still an "unsigned long" on
master; with 1/4 of [1] the cast becomes unnecessary but harmless.

Based on master at e1ec09b23c (The 22nd batch).

[1] https://lore.kernel.org/git/cover.1793439137.git.gitter.spiros@gmail.com/

Elia Pinto (5):
  object-file: check size of inflated loose object against its header
  object-file: use size_t for loose object sizes
  odb/source-loose: check object size when streaming loose objects
  cat-file: use size_t for the length passed to batch_write()
  notes: avoid truncating the size of concatenated notes

 builtin/cat-file.c           |  2 +-
 notes.c                      |  7 +++----
 object-file.c                | 31 +++++++++++++++++++++----------
 object-file.h                |  6 +++---
 odb/source-loose.c           | 19 +++++++++++++++----
 t/lib-loose.sh               |  5 ++++-
 t/t1006-cat-file.sh          | 18 ++++++++++++++++++
 t/t1060-object-corruption.sh | 28 ++++++++++++++++++++++++++++
 8 files changed, 93 insertions(+), 23 deletions(-)

Range-diff against v2:
1:  edb5f34498 ! 1:  0d748dbd8e object-file: check size of inflated loose object against its header
    @@ Commit message
         object-file: check size of inflated loose object against its header
     
         unpack_loose_rest() allocates a buffer of the size given in the object
    -    header and inflates the rest of the stream into it, but it never checks
    -    that the stream actually produced that many bytes. If the compressed
    -    data ends early we return success with the tail of the buffer never
    -    written, and callers like odb_read_object() hand it out as object
    -    content. For example, "git cat-file commit" on a loose object whose
    -    header says "commit 200" but which contains only 44 bytes prints 156
    -    bytes of uninitialized heap memory.
    +    header, but never checks that the stream produced that many bytes:
     
    -    The opposite case slips through, too: when the whole object fits into
    -    the buffer filled by unpack_loose_header(), we copy only "size" bytes
    -    out of it, the stream has already ended, and the extra data is silently
    -    dropped.
    +     - If the stream is shorter, the unwritten tail of the buffer is handed
    +       out as object content: "git cat-file commit" on an object whose
    +       header says "commit 200" but which holds 44 bytes prints 156 bytes
    +       of uninitialized heap memory.
     
    -    The packfile reader in unpack_compressed_entry() and index-pack already
    -    compare total_out with the expected size. Do the same here and report
    -    the object as corrupt if they differ. Today fsck notices the first case
    -    only by accident, because it hashes the uninitialized bytes and gets a
    -    mismatch.
    +     - If the whole object fits into the MAX_HEADER_LEN bytes that
    +       unpack_loose_header() already inflated, the stream has ended before
    +       we get here; we copy only "size" bytes out of that buffer and
    +       silently drop the rest.
     
    -    Teach loose_obj() in lib-loose.sh to write a bogus size, so that the
    +    Compare total_out with the expected size, as unpack_compressed_entry()
    +    and index-pack already do, and say how many bytes we got when they
    +    differ.
    +
    +    Teach loose_obj() in t/lib-loose.sh to write a bogus size, so that the
         tests can create such objects.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
    @@ object-file.c: void *unpack_loose_rest(git_zstream *stream,
      	if (bytes <= size) {
      		/*
     @@ object-file.c: void *unpack_loose_rest(git_zstream *stream,
    - 		}
    - 	}
    - 
    --	if (status != Z_STREAM_END) {
    -+	/*
    -+	 * Make sure that the stream produced exactly as many bytes as the
    -+	 * header claims. Otherwise we would hand back a buffer whose tail
    -+	 * was never written (if the stream is short), or silently drop data
    -+	 * (if the stream is long but ended before we noticed).
    -+	 */
    -+	if (status != Z_STREAM_END ||
    -+	    stream->total_out - hdrlen != size) {
    + 	if (status != Z_STREAM_END) {
      		error(_("corrupt loose object '%s'"), oid_to_hex(oid));
      		FREE_AND_NULL(buf);
    ++	} else if (stream->total_out != hdrlen + size) {
    ++		/*
    ++		 * A short stream would leave the tail of buf[] unwritten;
    ++		 * a long one can end inside the header buffer, so that
    ++		 * only "size" bytes were copied and the rest dropped.
    ++		 */
    ++		error(_("loose object '%s' has %"PRIuMAX" bytes, "
    ++			"but its header says %"PRIuMAX),
    ++		      oid_to_hex(oid), (uintmax_t)(stream->total_out - hdrlen),
    ++		      (uintmax_t)size);
    ++		FREE_AND_NULL(buf);
      	} else if (stream->avail_in) {
    + 		error(_("garbage at end of loose object '%s'"),
    + 		      oid_to_hex(oid));
     
      ## t/lib-loose.sh ##
     @@
    @@ t/t1060-object-corruption.sh: test_expect_success 'corrupted loose commit can be
     +
     +test_expect_success 'reading loose object shorter than its header fails' '
     +	test_must_fail git -C bogus-size cat-file commit $(cat oid-too-big) 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "has 5 bytes, but its header says 200" err
     +'
     +
     +test_expect_success 'reading loose object longer than its header fails' '
     +	test_must_fail git -C bogus-size cat-file commit $(cat oid-too-small) 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "has 11 bytes, but its header says 3" err
     +'
     +
     +test_expect_success 'fsck reports loose objects with bogus header size' '
     +	test_must_fail git -C bogus-size fsck 2>err &&
    -+	test_grep "corrupt loose object .$(cat oid-too-big)" err &&
    -+	test_grep "corrupt loose object .$(cat oid-too-small)" err
    ++	test_grep "loose object .$(cat oid-too-big). has 5 bytes" err &&
    ++	test_grep "loose object .$(cat oid-too-small). has 11 bytes" err
     +'
     +
      test_done
2:  eca5d3ddb8 ! 2:  c9facd8b3b object-file: use size_t for loose object sizes
    @@ Commit message
     
         Since c6a4629e32 (odb: use size_t for object_info.sizep and the size
         APIs, 2026-06-15), parse_loose_header() reports the object size as a
    -    size_t, but unpack_loose_rest() still takes it as an "unsigned long".
    -    On 64-bit Windows, where "long" has 32 bits while size_t has 64, the
    -    size is truncated on the way. A header claiming 4GiB + 5 bytes thus
    -    makes us allocate a 6-byte buffer, the size check added by the
    -    previous commit is happy because the stream really has 5 bytes, and
    -    the caller gets that buffer back together with *oi->sizep set to
    -    4GiB + 5. Any user of the object then reads far beyond the end of the
    -    buffer; read_loose_object() already does so when it hashes the
    -    contents.
    +    size_t, but unpack_loose_rest() takes it as an "unsigned long". Where
    +    "long" has 32 bits while size_t has 64, as on 64-bit Windows, a header
    +    claiming 4GiB + 5 bytes makes us allocate 6 bytes; the check added by
    +    the previous commit passes, as the stream does hold 5 bytes, and the
    +    caller then reads the object as 4GiB + 5 bytes long.
     
    -    Use size_t for the size in unpack_loose_rest() and check_stream_oid(),
    -    and while at it for the size of the mapped file in map_fd(),
    -    odb_source_loose_map_object() and unpack_loose_header(), so that the
    -    whole loose object reading path agrees on the type.
    +    Use size_t along the whole loose object reading path. No st_add() is
    +    needed on the way: the size is parsed with st_mult() and st_add(), and
    +    a buffer of that size is allocated with xmallocz() before anything is
    +    added to it or subtracted from it.
     
    -    There is no test, as the reader allocates a buffer of the size claimed
    -    in the header before it can notice that the stream is short, so a test
    -    would need 4GiB of memory. The problem can be reproduced on Windows
    -    with the loose_obj() helper from the previous commit:
    +    No test, as the reader allocates the claimed size before it notices
    +    the short stream. On Windows,
     
                 printf short |
                 loose_obj .git/objects commit $((4 * 1024 * 1024 * 1024 + 5))
     
    -    followed by "git cat-file commit" on the resulting object ID.
    +    and "git cat-file commit" on the resulting object reproduce it.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
3:  a8572aa836 ! 3:  c694d3e128 odb/source-loose: check object size when streaming loose objects
    @@ Metadata
      ## Commit message ##
         odb/source-loose: check object size when streaming loose objects
     
    -    read_istream_loose() inflates a loose object until zlib reports the end
    -    of the stream, but never compares the number of bytes it produced with
    -    the size given in the object header. A corrupt object whose stream is
    -    shorter than its header claims is thus streamed out without any error,
    -    e.g. "git cat-file blob" exits with 0 after writing fewer bytes than
    -    "git cat-file -s" reports. The same happens if the stream is longer than
    -    the header claims.
    +    read_istream_loose() streams a loose object until zlib reports the end
    +    of the stream, but never compares the number of bytes with the size in
    +    the object header. "git cat-file blob" on an object whose stream is
    +    shorter or longer than its header claims thus exits with 0, writing a
    +    different number of bytes than "git cat-file -s" reports.
     
    -    Keep track of how many bytes we handed out and report the object as
    -    corrupt if that does not match the size from the header once the stream
    -    ends.
    -
    -    The tests create such objects with the loose_obj() helper, which can
    -    already write a bogus size into the header.
    +    Count the bytes handed out and fail the read if they do not match once
    +    the stream ends. Like the other errors in this function, this returns
    +    -1 silently and leaves the message to the caller, e.g. "unable to
    +    stream <oid> to stdout" from cat-file.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
    @@ odb/source-loose.c: struct odb_loose_read_stream {
      	int hdr_avail;
      	int hdr_used;
     +	size_t total_read;
    -+	struct object_id oid;
      };
      
      static ssize_t read_istream_loose(struct odb_stream *_st, char *buf, size_t sz)
    @@ odb/source-loose.c: static ssize_t read_istream_loose(struct odb_stream *_st, ch
      			git_inflate_end(&st->z);
     +			/*
     +			 * The object must be exactly as large as its
    -+			 * header claims.
    ++			 * header claims; like the other errors below,
    ++			 * this is reported by our caller.
     +			 */
     +			if (st->total_read + total_read != st->base.size) {
    -+				error(_("corrupt loose object '%s'"),
    -+				      oid_to_hex(&st->oid));
     +				st->z_state = ODB_LOOSE_READ_STREAM_ERROR;
     +				return -1;
     +			}
    @@ odb/source-loose.c: static ssize_t read_istream_loose(struct odb_stream *_st, ch
      	return total_read;
      }
      
    -@@ odb/source-loose.c: static int odb_source_loose_read_object_stream(struct odb_stream **out,
    - 
    - 	st->mapped = mapped;
    - 	st->mapsize = mapsize;
    -+	oidcpy(&st->oid, oid);
    - 	st->hdr_used = strlen(st->hdr) + 1;
    - 	st->hdr_avail = st->z.total_out;
    - 	st->z_state = ODB_LOOSE_READ_STREAM_INUSE;
     
      ## t/t1006-cat-file.sh ##
     @@ t/t1006-cat-file.sh: test_expect_success PIPE 'concurrently written alternates are loaded' '
    @@ t/t1006-cat-file.sh: test_expect_success PIPE 'concurrently written alternates a
     +
     +test_expect_success 'streaming blob shorter than its header fails' '
     +	test_must_fail git -C bogus-size cat-file blob $(cat oid-too-big) >out 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "unable to stream" err
     +'
     +
     +test_expect_success 'streaming blob longer than its header fails' '
     +	test_must_fail git -C bogus-size cat-file blob $(cat oid-too-small) >out 2>err &&
    -+	test_grep "corrupt loose object" err
    ++	test_grep "unable to stream" err
     +'
     +
      test_done
4:  5b61a1b089 ! 4:  99bf45c606 cat-file: use size_t for the length passed to batch_write()
    @@ Commit message
         cat-file: use size_t for the length passed to batch_write()
     
         batch_write() takes the length of the data as an "int", but its callers
    -    pass the size of whole objects as returned by odb_read_object() or
    -    filter_object(), which is a size_t. For an object between 2GiB and
    -    4GiB the length wraps to a negative value, which fwrite() and
    -    write_or_die() then see as an enormous size_t. We happily write the
    -    object and then keep going past the end of the buffer, sending whatever
    -    memory follows to stdout, until we hit an unmapped page and die with
    -    "write error: Bad address". Objects of 4GiB and more are truncated
    -    instead, which desynchronizes the --batch output.
    -
    -    This affects all platforms, not only those with a 32-bit "long". It can
    -    be triggered with "--batch" on any non-blob object, or on blobs with
    -    --filters or --textconv, as blobs are otherwise streamed. For example:
    -
    -            blob=$(test-tool genzeros $((3 * 1024 * 1024 * 1024)) |
    -                   git hash-object -w --stdin) &&
    -            echo "$blob big" |
    -            git cat-file --batch="%(objectname) %(rest)" --filters >out
    -
    -    fails with "write error: Bad address"; with stdout going to /dev/null,
    -    strace shows write() being called on addresses far beyond the end of the
    -    3GiB buffer. There is no test, as it would need several GiB of memory
    -    and disk.
    -
    -    Both fwrite() and write_or_die() take a size_t already, so just change
    -    the parameter type.
    +    pass the size of whole objects, a size_t. On all platforms, an object
    +    between 2GiB and 4GiB gets a negative length, which fwrite() and
    +    write_or_die() see as an enormous size_t: "git cat-file --batch" writes
    +    memory past the end of the object to stdout until it dies with "write
    +    error: Bad address". Larger objects are truncated instead. Blobs are
    +    affected only with --filters or --textconv, as they are otherwise
    +    streamed.
    +
    +    Both callees take a size_t, so just change the parameter type.
     
         Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
     
5:  04f320d1de = 5:  d0cc63a8ed notes: avoid truncating the size of concatenated notes

base-commit: e1ec09b23c8e0deb21a0b95ac5996422277395d4
-- 
2.53.0


  parent reply	other threads:[~2026-11-19  6:22 UTC|newest]

Thread overview: 27+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-10-30 10:28 [PATCH 0/6] loose object size checks and a few size truncations Elia Pinto
2026-10-30 10:28 ` [PATCH 1/6] object-file: check size of inflated loose object against its header Elia Pinto
2026-10-30 10:28 ` [PATCH 2/6] object-file: use size_t for loose object sizes Elia Pinto
2026-10-30 10:28 ` [PATCH 3/6] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-10-30 10:28 ` [PATCH 4/6] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-10-30 10:28 ` [PATCH 5/6] notes: avoid truncating the size of concatenated notes Elia Pinto
2026-10-30 10:28 ` [PATCH 6/6] diff: use size_t in count_trailing_blank() Elia Pinto
2026-11-17 11:38 ` [PATCH v2 0/5] loose object size checks and a few size truncations Elia Pinto
2026-11-17 11:38   ` [PATCH v2 1/5] object-file: check size of inflated loose object against its header Elia Pinto
2026-11-17 19:33     ` Junio C Hamano
2026-11-18 12:21     ` Patrick Steinhardt
2026-11-18 18:17       ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 2/5] object-file: use size_t for loose object sizes Elia Pinto
2026-11-17 22:34     ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 3/5] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-11-17 22:55     ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 4/5] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-11-17 22:59     ` Junio C Hamano
2026-11-17 11:38   ` [PATCH v2 5/5] notes: avoid truncating the size of concatenated notes Elia Pinto
2026-11-17 23:01     ` Junio C Hamano
2026-11-19  6:21   ` Elia Pinto [this message]
2026-11-19  6:21     ` [PATCH v3 1/5] object-file: check size of inflated loose object against its header Elia Pinto
2026-11-19  6:21     ` [PATCH v3 2/5] object-file: use size_t for loose object sizes Elia Pinto
2026-11-20 13:26       ` Patrick Steinhardt
2026-11-19  6:21     ` [PATCH v3 3/5] odb/source-loose: check object size when streaming loose objects Elia Pinto
2026-11-19  6:21     ` [PATCH v3 4/5] cat-file: use size_t for the length passed to batch_write() Elia Pinto
2026-11-19  6:21     ` [PATCH v3 5/5] notes: avoid truncating the size of concatenated notes Elia Pinto

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1795069177.git.gitter.spiros@gmail.com \
    --to=gitter.spiros@gmail.com \
    --cc=Johannes.Schindelin@gmx.de \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jonathantanmy@fastmail.com \
    --cc=l.s.r@web.de \
    --cc=peff@peff.net \
    --cc=ps@pks.im \
    --cc=toon@iotcl.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.