* [PATCH 0/3] use size_t for a few sizes that are 32 bits everywhere
@ 2026-11-01 15:36 Elia Pinto
2026-11-01 15:36 ` [PATCH 1/3] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
` (3 more replies)
0 siblings, 4 replies; 16+ messages in thread
From: Elia Pinto @ 2026-11-01 15:36 UTC (permalink / raw)
To: git
Cc: Johannes Schindelin, Elia Pinto, Patrick Steinhardt, Neeraj Singh,
brian m. carlson, Junio C Hamano, Michael Montalbo
Unlike the previous series, these aren't about "long" on Windows: the
types involved ("uint32_t", "int", "unsigned int") are 32 bits on every
platform. They were found by scanning a native 64-bit Linux build with
-Wshorten-64-to-32. That scan sees the "unsigned long" -> 32-bit
conversions that a Windows target can't see, because there "long" is
already 32 bits.
- 1/3 hashwrite(): a corrupt pack when pack-objects writes a compressed
buffer of 4GiB or more in one call;
- 2/3 count_lines(): a wrong diff -B --stat for files of 2GiB or more;
- 3/3 hash_chars(): renames of files of 4GiB or more aren't recognised.
No tests, because they need GiB-sized inputs; each was verified by
simulating the truncation at 16 bits.
The other results of that scan are bounded by the format or by the input
(options, masks, line lengths, xdiff limited to 1023MiB). It doesn't
depend on the other series in flight and merges cleanly with them. Based
on master 50fe5cc63f.
Elia Pinto (3):
csum-file: use size_t for the count passed to hashwrite()
diff: use size_t for the size passed to count_lines()
diffcore-delta: use size_t for the size of the buffer to hash
csum-file.c | 2 +-
csum-file.h | 2 +-
diff.c | 2 +-
diffcore-delta.c | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
base-commit: 50fe5cc63f9cd6e89b87a2c9ccf96a444f4cfdb7
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 1/3] csum-file: use size_t for the count passed to hashwrite()
2026-11-01 15:36 [PATCH 0/3] use size_t for a few sizes that are 32 bits everywhere Elia Pinto
@ 2026-11-01 15:36 ` Elia Pinto
2026-11-01 15:36 ` [PATCH 2/3] diff: use size_t for the size passed to count_lines() Elia Pinto
` (2 subsequent siblings)
3 siblings, 0 replies; 16+ messages in thread
From: Elia Pinto @ 2026-11-01 15:36 UTC (permalink / raw)
To: git
Cc: Johannes Schindelin, Elia Pinto, Neeraj Singh, Junio C Hamano,
Patrick Steinhardt, brian m. carlson
hashwrite() takes the byte count as "uint32_t" on all platforms. When
pack-objects writes the entire compressed buffer of an object in a
single call, which can exceed 4GiB if the object is in memory, poorly
compressible and core.bigFileThreshold is raised accordingly, it writes
only "count mod 2^32" bytes, corrupting the pack.
This was verified using a 16-bit simulation, where "git cat-file" and
"git fsck" failed after "git repack -a -d" with "offset beyond end of
packfile (truncated pack?)".
Make the count parameter of hashwrite() a "size_t". The internal loop
already handles writing in buffer-sized blocks.
Triggering this requires processing an object of 4GiB or more, so no
test is added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
csum-file.c | 2 +-
csum-file.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/csum-file.c b/csum-file.c
index fe18ee1de3..26aa662637 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -102,7 +102,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
return fd;
}
-void hashwrite(struct hashfile *f, const void *buf, uint32_t count)
+void hashwrite(struct hashfile *f, const void *buf, size_t count)
{
while (count) {
unsigned left = f->buffer_len - f->offset;
diff --git a/csum-file.h b/csum-file.h
index 6ed74d1637..f1c8a9df13 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -75,7 +75,7 @@ void free_hashfile(struct hashfile *f);
* Finalize the hashfile by flushing data to disk and free'ing it.
*/
int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
-void hashwrite(struct hashfile *, const void *, uint32_t);
+void hashwrite(struct hashfile *, const void *, size_t);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
uint32_t crc32_end(struct hashfile *);
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 2/3] diff: use size_t for the size passed to count_lines()
2026-11-01 15:36 [PATCH 0/3] use size_t for a few sizes that are 32 bits everywhere Elia Pinto
2026-11-01 15:36 ` [PATCH 1/3] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
@ 2026-11-01 15:36 ` Elia Pinto
2026-11-04 13:58 ` Phillip Wood
2026-11-01 15:36 ` [PATCH 3/3] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
3 siblings, 1 reply; 16+ messages in thread
From: Elia Pinto @ 2026-11-01 15:36 UTC (permalink / raw)
To: git; +Cc: Johannes Schindelin, Elia Pinto, Junio C Hamano, Michael Montalbo
count_lines() takes the size as an "int", whereas callers have "unsigned
long" or "size_t". It is used for the diffstat of complete rewrites
("git diff -B --stat") and in emit_rewrite_diff().
With a file of 2GiB or more, the "int" becomes negative (or truncated
above 4GiB), leading to an incorrect line count. This only affects the
diffstat output, with no memory access issues.
This was verified using a 16-bit simulation, where "git diff -B --stat"
reported 0 changed lines instead of the actual count.
Make the size parameter a size_t. The line counter remains "int".
Triggering this requires processing a file of 2GiB or more, so no test
is added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
diff.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/diff.c b/diff.c
index b4ac17f8ef..ed7a931c41 100644
--- a/diff.c
+++ b/diff.c
@@ -655,7 +655,7 @@ struct line_range_callback {
int ret; /* latched error from orig_line_fn */
};
-static int count_lines(const char *data, int size)
+static int count_lines(const char *data, size_t size)
{
int count, ch, completely_empty = 1, nl_just_seen = 0;
count = 0;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH 3/3] diffcore-delta: use size_t for the size of the buffer to hash
2026-11-01 15:36 [PATCH 0/3] use size_t for a few sizes that are 32 bits everywhere Elia Pinto
2026-11-01 15:36 ` [PATCH 1/3] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
2026-11-01 15:36 ` [PATCH 2/3] diff: use size_t for the size passed to count_lines() Elia Pinto
@ 2026-11-01 15:36 ` Elia Pinto
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
3 siblings, 0 replies; 16+ messages in thread
From: Elia Pinto @ 2026-11-01 15:36 UTC (permalink / raw)
To: git; +Cc: Johannes Schindelin, Elia Pinto, Junio C Hamano
hash_chars() in diffcore-delta.c takes the buffer size as an "unsigned
int". With a file of 4GiB or more, it hashes only a prefix; because the
similarity score is calculated against the full size, similarity is
undervalued, so true renames are missed, which also affects
"git log --follow" and merges.
This was verified using a 16-bit simulation, where
"git diff --cached -M" reported a delete and add instead of a rename
(e.g., "D a" and "A b" instead of "R099 a b").
Make the size variable "size_t".
Triggering this requires processing a file of 4GiB or more, so no test
is added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
diffcore-delta.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/diffcore-delta.c b/diffcore-delta.c
index 2b7db39983..7f10f79ffe 100644
--- a/diffcore-delta.c
+++ b/diffcore-delta.c
@@ -127,7 +127,7 @@ static struct spanhash_top *hash_chars(struct repository *r,
unsigned int accum1, accum2, hashval;
struct spanhash_top *hash;
unsigned char *buf = one->data;
- unsigned int sz = one->size;
+ size_t sz = one->size;
int is_text = !diff_filespec_is_binary(r, one);
i = INITIAL_HASH_SIZE;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] diff: use size_t for the size passed to count_lines()
2026-11-01 15:36 ` [PATCH 2/3] diff: use size_t for the size passed to count_lines() Elia Pinto
@ 2026-11-04 13:58 ` Phillip Wood
2026-11-04 15:45 ` Phillip Wood
0 siblings, 1 reply; 16+ messages in thread
From: Phillip Wood @ 2026-11-04 13:58 UTC (permalink / raw)
To: Elia Pinto, git; +Cc: Johannes Schindelin, Junio C Hamano, Michael Montalbo
Hi Elia
On 01/11/2026 15:36, Elia Pinto wrote:
> count_lines() takes the size as an "int", whereas callers have "unsigned
> long" or "size_t". It is used for the diffstat of complete rewrites
> ("git diff -B --stat") and in emit_rewrite_diff().
>
> With a file of 2GiB or more, the "int" becomes negative (or truncated
> above 4GiB), leading to an incorrect line count. This only affects the
> diffstat output, with no memory access issues.
>
> This was verified using a 16-bit simulation, where "git diff -B --stat"
> reported 0 changed lines instead of the actual count.
>
> Make the size parameter a size_t. The line counter remains "int".
That means the counter is subject to overflow as a file can contain as
many lines as its size.
I don't know what the simulation entailed, but this patch changes the
return type without changing either the type of the returned value which
can now overflow, or the types where the return values is stored by callers.
I'm concerned that the preparation of these patches and the others
you've sent has relied too heavily on a flawed mechanical process, and
have not enough on manual review reading the code and reasoning about it.
Thanks
Phillip
> Triggering this requires processing a file of 2GiB or more, so no test
> is added.
>
> Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
> ---
> diff.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/diff.c b/diff.c
> index b4ac17f8ef..ed7a931c41 100644
> --- a/diff.c
> +++ b/diff.c
> @@ -655,7 +655,7 @@ struct line_range_callback {
> int ret; /* latched error from orig_line_fn */
> };
>
> -static int count_lines(const char *data, int size)
> +static int count_lines(const char *data, size_t size)
> {
> int count, ch, completely_empty = 1, nl_just_seen = 0;
> count = 0;
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 2/3] diff: use size_t for the size passed to count_lines()
2026-11-04 13:58 ` Phillip Wood
@ 2026-11-04 15:45 ` Phillip Wood
0 siblings, 0 replies; 16+ messages in thread
From: Phillip Wood @ 2026-11-04 15:45 UTC (permalink / raw)
To: Elia Pinto, git; +Cc: Johannes Schindelin, Junio C Hamano, Michael Montalbo
On 04/11/2026 13:58, Phillip Wood wrote:
> On 01/11/2026 15:36, Elia Pinto wrote:
>> count_lines() takes the size as an "int", whereas callers have "unsigned
>> long" or "size_t". It is used for the diffstat of complete rewrites
>> ("git diff -B --stat") and in emit_rewrite_diff().
>>
>> With a file of 2GiB or more, the "int" becomes negative (or truncated
>> above 4GiB), leading to an incorrect line count. This only affects the
>> diffstat output, with no memory access issues.
>>
>> This was verified using a 16-bit simulation, where "git diff -B --stat"
>> reported 0 changed lines instead of the actual count.
>>
> > Make the size parameter a size_t. The line counter remains "int".
>
> That means the counter is subject to overflow as a file can contain as
> many lines as its size.
>
> I don't know what the simulation entailed, but this patch changes the
> return type without changing either the type of the returned value which
> can now overflow, or the types where the return values is stored by
> callers.
Sorry I seem to have misread the patch when I wrote that - the return
type is not changed. We're still left with the possibility of the count
overflowing now that "size" is a size_t though I think.
Phillip
> I'm concerned that the preparation of these patches and the others
> you've sent has relied too heavily on a flawed mechanical process, and
> have not enough on manual review reading the code and reasoning about it.
>
> Thanks
>
> Phillip
>
>
>> Triggering this requires processing a file of 2GiB or more, so no test
>> is added.
>>
>> Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
>> ---
>> diff.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/diff.c b/diff.c
>> index b4ac17f8ef..ed7a931c41 100644
>> --- a/diff.c
>> +++ b/diff.c
>> @@ -655,7 +655,7 @@ struct line_range_callback {
>> int ret; /* latched error from orig_line_fn */
>> };
>> -static int count_lines(const char *data, int size)
>> +static int count_lines(const char *data, size_t size)
>> {
>> int count, ch, completely_empty = 1, nl_just_seen = 0;
>> count = 0;
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere
2026-11-01 15:36 [PATCH 0/3] use size_t for a few sizes that are 32 bits everywhere Elia Pinto
` (2 preceding siblings ...)
2026-11-01 15:36 ` [PATCH 3/3] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
@ 2026-11-17 10:37 ` Elia Pinto
2026-11-17 10:37 ` [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
` (3 more replies)
3 siblings, 4 replies; 16+ messages in thread
From: Elia Pinto @ 2026-11-17 10:37 UTC (permalink / raw)
To: git
Cc: Johannes Schindelin, Elia Pinto, Patrick Steinhardt, Neeraj Singh,
brian m. carlson, Junio C Hamano, Jeff King, Michael Montalbo,
Torsten Bögershausen, Alexander Meshcheryakov, Elijah Newren,
Phillip Wood
This is v2 of the series that widens a few sizes and line counts held
in 32-bit types even on 64-bit platforms.
Changes since v1:
- 2/4 is rewritten after Phillip's review [1]. v1 only made the size
parameter of count_lines() a size_t; as Phillip pointed out, that
removes the bound that kept the "int" line counter from overflowing.
The patch now uses size_t for the sizes and line counts along the
whole complete-rewrite path of "git diff -B". This includes
emit_rewrite_lines(), which with v1 still truncated the size, so
that "git diff -B -p" printed a correct hunk header above an empty
body. It also adds two EXPENSIVE tests to t4022.
- 4/4 is new. It was found by looking at the totals of "--stat" after
2/4: they are added up in "int" and overflow past INT_MAX lines,
which 2/4 makes possible with a single file. The same code also
showed that scale_linear() overflows for a small file when the graph
is wide enough; a cheap test in t4052 covers that case.
- 1/4 and 3/4 are unchanged. As in v1, they need objects of 4GiB or
more and were verified by simulating the truncation at 16 bits.
2/4 and 4/4 come with tests instead.
2/4 now also changes count_trailing_blank(), whose size is also
changed by patch 6/6 of my series "loose object size checks and a few
size truncations" [2]. That patch will be dropped from v2 of that
series, so the two series will not conflict.
The series is based on master at 50fe5cc63f (The 12th batch).
[1] https://lore.kernel.org/git/0cb42395-3e62-42e0-bfd3-a4265ed7cc33@gmail.com/
[2] https://lore.kernel.org/git/cover.1793355390.git.gitter.spiros@gmail.com/
Elia Pinto (4):
csum-file: use size_t for the count passed to hashwrite()
diff: use size_t for sizes and line counts of complete rewrites
diffcore-delta: use size_t for the size of the buffer to hash
diff: use uintmax_t for diffstat totals and graph scaling
csum-file.c | 2 +-
csum-file.h | 2 +-
diff.c | 90 +++++++++++++++++++++++------------------
diff.h | 2 +-
diffcore-delta.c | 2 +-
t/t4022-diff-rewrite.sh | 51 ++++++++++++++++++++++-
t/t4052-stat-output.sh | 10 +++++
7 files changed, 114 insertions(+), 45 deletions(-)
Range-diff against v1:
1: 07baad46b1 = 1: 07baad46b1 csum-file: use size_t for the count passed to hashwrite()
2: 2011b90d80 < -: ---------- diff: use size_t for the size passed to count_lines()
-: ---------- > 2: e63ed0c49c diff: use size_t for sizes and line counts of complete rewrites
3: 0c5442280c = 3: 3427c3d2c8 diffcore-delta: use size_t for the size of the buffer to hash
-: ---------- > 4: 692c503e97 diff: use uintmax_t for diffstat totals and graph scaling
--
2.53.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite()
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
@ 2026-11-17 10:37 ` Elia Pinto
2026-11-17 17:52 ` Junio C Hamano
2026-11-17 10:37 ` [PATCH v2 2/4] diff: use size_t for sizes and line counts of complete rewrites Elia Pinto
` (2 subsequent siblings)
3 siblings, 1 reply; 16+ messages in thread
From: Elia Pinto @ 2026-11-17 10:37 UTC (permalink / raw)
To: git
Cc: Johannes Schindelin, Elia Pinto, brian m. carlson, Neeraj Singh,
Junio C Hamano, Patrick Steinhardt
hashwrite() takes the byte count as "uint32_t" on all platforms. When
pack-objects writes the entire compressed buffer of an object in a
single call, which can exceed 4GiB if the object is in memory, poorly
compressible and core.bigFileThreshold is raised accordingly, it writes
only "count mod 2^32" bytes, corrupting the pack.
This was verified using a 16-bit simulation, where "git cat-file" and
"git fsck" failed after "git repack -a -d" with "offset beyond end of
packfile (truncated pack?)".
Make the count parameter of hashwrite() a "size_t". The internal loop
already handles writing in buffer-sized blocks.
Triggering this requires processing an object of 4GiB or more, so no
test is added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
csum-file.c | 2 +-
csum-file.h | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/csum-file.c b/csum-file.c
index fe18ee1de3..26aa662637 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -102,7 +102,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
return fd;
}
-void hashwrite(struct hashfile *f, const void *buf, uint32_t count)
+void hashwrite(struct hashfile *f, const void *buf, size_t count)
{
while (count) {
unsigned left = f->buffer_len - f->offset;
diff --git a/csum-file.h b/csum-file.h
index 6ed74d1637..f1c8a9df13 100644
--- a/csum-file.h
+++ b/csum-file.h
@@ -75,7 +75,7 @@ void free_hashfile(struct hashfile *f);
* Finalize the hashfile by flushing data to disk and free'ing it.
*/
int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
-void hashwrite(struct hashfile *, const void *, uint32_t);
+void hashwrite(struct hashfile *, const void *, size_t);
void hashflush(struct hashfile *f);
void crc32_begin(struct hashfile *);
uint32_t crc32_end(struct hashfile *);
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 2/4] diff: use size_t for sizes and line counts of complete rewrites
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
2026-11-17 10:37 ` [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
@ 2026-11-17 10:37 ` Elia Pinto
2026-11-17 10:37 ` [PATCH v2 3/4] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
2026-11-17 10:37 ` [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling Elia Pinto
3 siblings, 0 replies; 16+ messages in thread
From: Elia Pinto @ 2026-11-17 10:37 UTC (permalink / raw)
To: git
Cc: Johannes Schindelin, Elia Pinto, Phillip Wood, Jeff King,
Junio C Hamano, Michael Montalbo
When "git diff -B" finds a complete rewrite, the diffstat comes from
count_lines() and the patch from emit_rewrite_diff(). Neither goes
through xdiff, so MAX_XDIFF_SIZE does not apply to them.
Without -B, a file this large does not get that far. If its contents
have not been loaded yet and it is larger than core.bigFileThreshold
(512MiB by default), it is treated as binary; otherwise xdiff refuses
it, as it does not accept more than MAX_XDIFF_SIZE (just under 1GiB).
With -B, should_break() in diffcore-break.c loads the contents without
asking for the binary check, so core.bigFileThreshold is never looked
at and a file too large for an "int" reaches count_lines().
count_lines() and emit_rewrite_lines() take the size as an "int". For
a file of 2GiB or more the size is truncated, typically to a negative
value, and both functions then see no data. As a result:
- "git diff -B --stat" reports 0 changed lines;
- "git diff -B -p" prints "@@ -0,0 +0,0 @@" with no body, a patch
that "git apply" rejects as corrupt, and still exits with status 0.
Passing the size as a size_t is not enough. The size no longer bounds
the "int" line counter in count_lines(), which overflows on a file of
more than INT_MAX lines, and emit_rewrite_lines() would still truncate
the size, leaving a correct hunk header above an empty body.
Use size_t throughout this code path instead:
- count_lines() takes a size_t and returns a size_t count, which
struct diffstat_file already stores in uintmax_t;
- count_trailing_blank() uses size_t for its counter, which overflows
on a file ending in more than INT_MAX blank lines, and for the size,
which it stored in a "long". On LLP64 platforms such as 64-bit
Windows, where "long" has 32 bits while size_t has 64, a size of
2GiB or more would move the pointer before the start of the buffer.
This was found by inspection and has not been tested on such a
platform;
- emit_rewrite_diff(), add_line_count() and check_blank_at_eof() hold
and print these counts as size_t, as do the line number and blank
line fields of struct emit_callback;
- emit_rewrite_lines() takes the size as a size_t. The length of one
line remains an "int" because emit_del_line() and emit_add_line()
take one, so use cast_size_t_to_int() to die instead of truncating
it when a single line is longer than INT_MAX;
- builtin_checkdiff() follows the type of the field it reads. It is
still protected by the xdiff limit.
To reproduce, using about 2.2GiB of working tree data:
yes old | head -c 2200M >big &&
git add big &&
git commit -m old &&
yes new | head -c 2200M >big &&
git diff -B --stat
Before this patch this reports 0 insertions and 0 deletions. With it,
it reports 576716800 of each.
Add two tests to t4022, marked EXPENSIVE because they need several GiB
of disk space and take minutes to run. The first checks the numstat
and the patch for a file of 2GiB. The second checks the numstat for a
file of 2^31 lines. Both fail before this patch, and also with only the
size parameter of count_lines() changed.
The tests run only where size_t and "long" have 64 bits. struct
diff_filespec still holds the size of the file in an "unsigned long",
and the tests have not been run on a platform where "long" has 32
bits, such as Windows, whose CI would run them.
The count_trailing_blank() overflow is not covered by these tests. It
only affects whitespace highlighting, so it is not visible in the
uncoloured output. It was checked instead with a build using
"make SANITIZE=signed-integer-overflow", by running "git diff -B -p"
on a complete rewrite to a file of 2200MiB of newlines: the sanitizer
stops in count_trailing_blank() without this change and reports nothing
with it.
Helped-by: Phillip Wood <phillip.wood@dunelm.org.uk>
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
Changes since v1:
- v1 only changed the size parameter of count_lines(). Phillip pointed
out that this removes the bound that kept the "int" counter from
overflowing, so count_lines() now counts and returns a size_t.
- emit_rewrite_lines() now takes a size_t. With v1, "git diff -B -p"
printed a correct hunk header above an empty body.
- count_trailing_blank() and the callers and fields that hold line
counts now use size_t. This also fixes a "long" size that is too
small on LLP64 platforms.
- Add EXPENSIVE tests to t4022. Both fail on master and with v1.
- The commit message is rewritten. v1 said that only the diffstat was
affected and that there were no memory access issues; neither claim
holds. The 16-bit simulation is replaced by a reproduction on a real
file.
---
diff.c | 44 ++++++++++++++++++-----------------
t/t4022-diff-rewrite.sh | 51 ++++++++++++++++++++++++++++++++++++++++-
2 files changed, 73 insertions(+), 22 deletions(-)
diff --git a/diff.c b/diff.c
index b4ac17f8ef..ee92a8ac4d 100644
--- a/diff.c
+++ b/diff.c
@@ -598,10 +598,10 @@ static struct diff_tempfile {
struct emit_callback {
int color_diff;
unsigned ws_rule;
- int blank_at_eof_in_preimage;
- int blank_at_eof_in_postimage;
- int lno_in_preimage;
- int lno_in_postimage;
+ size_t blank_at_eof_in_preimage;
+ size_t blank_at_eof_in_postimage;
+ size_t lno_in_preimage;
+ size_t lno_in_postimage;
int last_line_kind;
const char **label_path;
struct diff_words_data *diff_words;
@@ -655,9 +655,10 @@ struct line_range_callback {
int ret; /* latched error from orig_line_fn */
};
-static int count_lines(const char *data, int size)
+static size_t count_lines(const char *data, size_t size)
{
- int count, ch, completely_empty = 1, nl_just_seen = 0;
+ size_t count;
+ int ch, completely_empty = 1, nl_just_seen = 0;
count = 0;
while (0 < size--) {
ch = *data++;
@@ -708,11 +709,11 @@ static unsigned long diff_filespec_size(struct repository *r,
return one->size;
}
-static int count_trailing_blank(mmfile_t *mf)
+static size_t count_trailing_blank(mmfile_t *mf)
{
char *ptr = mf->ptr;
- long size = mf->size;
- int cnt = 0;
+ size_t size = mf->size;
+ size_t cnt = 0;
if (!size)
return cnt;
@@ -737,7 +738,7 @@ static int count_trailing_blank(mmfile_t *mf)
static void check_blank_at_eof(mmfile_t *mf1, mmfile_t *mf2,
struct emit_callback *ecbdata)
{
- int l1, l2, at;
+ size_t l1, l2, at;
l1 = count_trailing_blank(mf1);
l2 = count_trailing_blank(mf2);
if (l2 <= l1) {
@@ -1816,7 +1817,7 @@ static void remove_tempfile(void)
}
}
-static void add_line_count(struct strbuf *out, int count)
+static void add_line_count(struct strbuf *out, size_t count)
{
switch (count) {
case 0:
@@ -1826,29 +1827,30 @@ static void add_line_count(struct strbuf *out, int count)
strbuf_addstr(out, "1");
break;
default:
- strbuf_addf(out, "1,%d", count);
+ strbuf_addf(out, "1,%"PRIuMAX, (uintmax_t)count);
break;
}
}
static void emit_rewrite_lines(struct emit_callback *ecbdata,
- int prefix, const char *data, int size)
+ int prefix, const char *data, size_t size)
{
const char *endp = NULL;
while (0 < size) {
- int len, plen;
+ size_t len;
+ int plen;
char *pdata = NULL;
endp = memchr(data, '\n', size);
if (endp) {
len = endp - data + 1;
- plen = len;
+ plen = cast_size_t_to_int(len);
} else {
len = size;
- plen = len + 1;
- pdata = xmalloc(plen + 2);
+ plen = cast_size_t_to_int(len + 1);
+ pdata = xmalloc(len + 3);
memcpy(pdata, data, len);
pdata[len] = '\n';
pdata[len + 1] = '\0';
@@ -1879,7 +1881,7 @@ static void emit_rewrite_diff(const char *name_a,
struct userdiff_driver *textconv_two,
struct diff_options *o)
{
- int lc_a, lc_b;
+ size_t lc_a, lc_b;
static struct strbuf a_name = STRBUF_INIT, b_name = STRBUF_INIT;
const char *a_prefix, *b_prefix;
char *data_one, *data_two;
@@ -4308,7 +4310,7 @@ static void builtin_checkdiff(const char *name_a, const char *name_b,
if (data.ws_rule & WS_BLANK_AT_EOF) {
struct emit_callback ecbdata;
- int blank_at_eof;
+ size_t blank_at_eof;
ecbdata.ws_rule = data.ws_rule;
check_blank_at_eof(&mf1, &mf2, &ecbdata);
@@ -4318,8 +4320,8 @@ static void builtin_checkdiff(const char *name_a, const char *name_b,
static char *err;
if (!err)
err = whitespace_error_string(WS_BLANK_AT_EOF);
- fprintf(o->file, "%s:%d: %s.\n",
- data.filename, blank_at_eof, err);
+ fprintf(o->file, "%s:%"PRIuMAX": %s.\n",
+ data.filename, (uintmax_t)blank_at_eof, err);
data.status = 1; /* report errors */
}
}
diff --git a/t/t4022-diff-rewrite.sh b/t/t4022-diff-rewrite.sh
index b5c1b6868f..08e5533437 100755
--- a/t/t4022-diff-rewrite.sh
+++ b/t/t4022-diff-rewrite.sh
@@ -97,5 +97,54 @@ test_expect_success 'no newline at eof is on its own line with -B' '
test_grep ! "^..*\\\\ " res
'
-test_done
+# Write "$1" repeated 2^$2 times to "$3".
+repeat_doubling () {
+ printf "$1" >"$3" &&
+ for i in $(test_seq "$2")
+ do
+ cat "$3" "$3" >"$3.tmp" &&
+ mv "$3.tmp" "$3" || return 1
+ done
+}
+
+test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,LONG_IS_64BIT 'rewrite of a 2GiB file' '
+ test_when_finished "rm -rf rewrite-2gb" &&
+ git init rewrite-2gb &&
+ (
+ cd rewrite-2gb &&
+ test_seq 1000 >big &&
+ git add big &&
+ git commit -m small &&
+ repeat_doubling "new\n" 29 big &&
+
+ printf "536870912\t1000\tbig\n" >expect &&
+ git diff -B --numstat >actual &&
+ test_cmp expect actual &&
+
+ git diff -B -p >diff &&
+ echo "@@ -1,1000 +1,536870912 @@" >expect &&
+ grep "^@@" diff >actual &&
+ test_cmp expect actual &&
+ echo 536870912 >expect &&
+ grep -c "^+new$" diff >actual &&
+ test_cmp expect actual
+ )
+'
+test_expect_success EXPENSIVE,SIZE_T_IS_64BIT,LONG_IS_64BIT 'rewrite with 2^31 lines' '
+ test_when_finished "rm -rf rewrite-2g-lines" &&
+ git init rewrite-2g-lines &&
+ (
+ cd rewrite-2g-lines &&
+ test_seq 1000 >big &&
+ git add big &&
+ git commit -m small &&
+ repeat_doubling "\n" 31 big &&
+
+ printf "2147483648\t1000\tbig\n" >expect &&
+ git diff -B --numstat >actual &&
+ test_cmp expect actual
+ )
+'
+
+test_done
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 3/4] diffcore-delta: use size_t for the size of the buffer to hash
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
2026-11-17 10:37 ` [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
2026-11-17 10:37 ` [PATCH v2 2/4] diff: use size_t for sizes and line counts of complete rewrites Elia Pinto
@ 2026-11-17 10:37 ` Elia Pinto
2026-11-17 23:21 ` Junio C Hamano
2026-11-17 10:37 ` [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling Elia Pinto
3 siblings, 1 reply; 16+ messages in thread
From: Elia Pinto @ 2026-11-17 10:37 UTC (permalink / raw)
To: git; +Cc: Johannes Schindelin, Elia Pinto, Junio C Hamano
hash_chars() in diffcore-delta.c takes the buffer size as an "unsigned
int". With a file of 4GiB or more, it hashes only a prefix; because the
similarity score is calculated against the full size, similarity is
undervalued, so true renames are missed, which also affects
"git log --follow" and merges.
This was verified using a 16-bit simulation, where
"git diff --cached -M" reported a delete and add instead of a rename
(e.g., "D a" and "A b" instead of "R099 a b").
Make the size variable "size_t".
Triggering this requires processing a file of 4GiB or more, so no test
is added.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
diffcore-delta.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/diffcore-delta.c b/diffcore-delta.c
index 2b7db39983..7f10f79ffe 100644
--- a/diffcore-delta.c
+++ b/diffcore-delta.c
@@ -127,7 +127,7 @@ static struct spanhash_top *hash_chars(struct repository *r,
unsigned int accum1, accum2, hashval;
struct spanhash_top *hash;
unsigned char *buf = one->data;
- unsigned int sz = one->size;
+ size_t sz = one->size;
int is_text = !diff_filespec_is_binary(r, one);
i = INITIAL_HASH_SIZE;
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
` (2 preceding siblings ...)
2026-11-17 10:37 ` [PATCH v2 3/4] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
@ 2026-11-17 10:37 ` Elia Pinto
2026-11-17 16:36 ` Torsten Bögershausen
2026-11-17 23:13 ` Junio C Hamano
3 siblings, 2 replies; 16+ messages in thread
From: Elia Pinto @ 2026-11-17 10:37 UTC (permalink / raw)
To: git
Cc: Johannes Schindelin, Elia Pinto, Junio C Hamano,
Alexander Meshcheryakov, Torsten Bögershausen, Elijah Newren
struct diffstat_file stores the number of added and deleted lines of
each file in uintmax_t, but show_stats() and show_shortstats() add them
up in "int" variables. When the total goes over INT_MAX, the summary
line overflows. With the previous patch, a single file is enough: for
a complete rewrite into a file of 2^31 lines, "git diff -B --stat"
ends with
1 file changed, -2147483648 insertions(+), 1000 deletions(-)
Several files whose changes add up to more than INT_MAX lines lead to
the same overflow even without -B.
Use uintmax_t for these totals and for the parameters of
print_stat_summary() and print_stat_summary_inserts_deletes(), which
printed them with "%d". print_stat_summary() is also called by
"git apply --stat", hence the change of its prototype in diff.h.
apply.c still adds up its own totals in "int"; it is left alone here.
The graph has a similar problem. scale_linear() takes the number of
changed lines and max_change as "int", although show_stats() holds
them in uintmax_t, and computes "it * (width - 1)" in "int". The
product overflows even for a small file when the graph is wide enough.
For a new file of 50000 lines,
git diff --stat=46100 --stat-graph-width=46000
prints no "+" at all, because 50000 * 45999 is larger than INT_MAX.
Make scale_linear() take uintmax_t, so that the product is computed in
uintmax_t. Its result is at most "width", so it still fits in an
"int".
show_stats() also assigned max_change to the "int" graph_width. Cap it
at "width" instead: a wider graph is cut down to fit "width" right
after, so the output does not change when max_change fits in an "int".
When the graph is not scaled, the numbers of added and deleted lines
are copied as they are into "add" and "del", which stay "int": in that
case both are smaller than graph_width.
Add the graph case to t4052. The test fails without this patch and is
cheap enough to run by default. The totals are not tested, because
going over INT_MAX lines needs a file of 2GiB or more.
Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
---
diff.c | 46 +++++++++++++++++++++++++-----------------
diff.h | 2 +-
t/t4052-stat-output.sh | 10 +++++++++
3 files changed, 38 insertions(+), 20 deletions(-)
diff --git a/diff.c b/diff.c
index ee92a8ac4d..0d2ea21280 100644
--- a/diff.c
+++ b/diff.c
@@ -2838,7 +2838,7 @@ static int diffstat_consume(void *priv, char *line, unsigned long len)
const char mime_boundary_leader[] = "------------";
-static int scale_linear(int it, int width, int max_change)
+static int scale_linear(uintmax_t it, int width, uintmax_t max_change)
{
if (!it)
return 0;
@@ -2880,7 +2880,7 @@ static void fill_print_name(struct diffstat_file *file)
}
static void print_stat_summary_inserts_deletes(struct diff_options *options,
- int files, int insertions, int deletions)
+ int files, uintmax_t insertions, uintmax_t deletions)
{
struct strbuf sb = STRBUF_INIT;
@@ -2905,13 +2905,15 @@ static void print_stat_summary_inserts_deletes(struct diff_options *options,
*/
if (insertions || deletions == 0) {
strbuf_addf(&sb,
- (insertions == 1) ? ", %d insertion(+)" : ", %d insertions(+)",
+ (insertions == 1) ? ", %"PRIuMAX" insertion(+)" :
+ ", %"PRIuMAX" insertions(+)",
insertions);
}
if (deletions || insertions == 0) {
strbuf_addf(&sb,
- (deletions == 1) ? ", %d deletion(-)" : ", %d deletions(-)",
+ (deletions == 1) ? ", %"PRIuMAX" deletion(-)" :
+ ", %"PRIuMAX" deletions(-)",
deletions);
}
strbuf_addch(&sb, '\n');
@@ -2921,7 +2923,7 @@ static void print_stat_summary_inserts_deletes(struct diff_options *options,
}
void print_stat_summary(FILE *fp, int files,
- int insertions, int deletions)
+ uintmax_t insertions, uintmax_t deletions)
{
struct diff_options o;
memset(&o, 0, sizeof(o));
@@ -2954,8 +2956,8 @@ static int utf8_ish_width(const char **start)
static void show_stats(struct diffstat_t *data, struct diff_options *options)
{
- int i, len, add, del, adds = 0, dels = 0;
- uintmax_t max_change = 0, max_len = 0;
+ int i, len, add, del;
+ uintmax_t adds = 0, dels = 0, max_change = 0, max_len = 0;
int total_files = data->nr, count;
int width, name_width, graph_width, number_width = 0, bin_width = 0;
const char *reset, *add_c, *del_c;
@@ -3062,7 +3064,11 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
* strlen("Bin XXX -> YYY bytes") == bin_width, and the part
* starting from "XXX" should fit in graph_width.
*/
- graph_width = max_change + 4 > bin_width ? max_change : bin_width - 4;
+ if (max_change + 4 > bin_width)
+ /* anything wider than "width" is cut down below anyway */
+ graph_width = max_change < width ? max_change : width;
+ else
+ graph_width = bin_width - 4;
if (options->stat_graph_width &&
options->stat_graph_width < graph_width)
graph_width = options->stat_graph_width;
@@ -3164,21 +3170,22 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
/*
* scale the add/delete
*/
- add = added;
- del = deleted;
-
if (graph_width <= max_change) {
- int total = scale_linear(add + del, graph_width, max_change);
- if (total < 2 && add && del)
+ int total = scale_linear(added + deleted, graph_width, max_change);
+ if (total < 2 && added && deleted)
/* width >= 2 due to the sanity check */
total = 2;
- if (add < del) {
- add = scale_linear(add, graph_width, max_change);
+ if (added < deleted) {
+ add = scale_linear(added, graph_width, max_change);
del = total - add;
} else {
- del = scale_linear(del, graph_width, max_change);
+ del = scale_linear(deleted, graph_width, max_change);
add = total - del;
}
+ } else {
+ /* both fit in the graph, hence in an int */
+ add = added;
+ del = deleted;
}
strbuf_addf(&out, " %s%s%*s | %*"PRIuMAX"%s",
prefix, name, padding, "",
@@ -3222,14 +3229,15 @@ static void show_stats(struct diffstat_t *data, struct diff_options *options)
static void show_shortstats(struct diffstat_t *data, struct diff_options *options)
{
- int i, adds = 0, dels = 0, total_files = data->nr;
+ int i, total_files = data->nr;
+ uintmax_t adds = 0, dels = 0;
if (data->nr == 0)
return;
for (i = 0; i < data->nr; i++) {
- int added = data->files[i]->added;
- int deleted = data->files[i]->deleted;
+ uintmax_t added = data->files[i]->added;
+ uintmax_t deleted = data->files[i]->deleted;
if (data->files[i]->is_unmerged ||
(!data->files[i]->is_interesting && (added + deleted == 0))) {
diff --git a/diff.h b/diff.h
index bb5cddaf34..d51507d192 100644
--- a/diff.h
+++ b/diff.h
@@ -764,7 +764,7 @@ int parse_rename_score(const char **cp_p);
long parse_algorithm_value(const char *value);
void print_stat_summary(FILE *fp, int files,
- int insertions, int deletions);
+ uintmax_t insertions, uintmax_t deletions);
void setup_diff_pager(struct diff_options *);
extern int diff_auto_refresh_index;
diff --git a/t/t4052-stat-output.sh b/t/t4052-stat-output.sh
index 62d3d2604c..d1cca6ab8e 100755
--- a/t/t4052-stat-output.sh
+++ b/t/t4052-stat-output.sh
@@ -470,4 +470,14 @@ test_expect_success FUNNYNAMES 'diffstat truncation with control chars does not
git commit -m "remove test file"
'
+test_expect_success 'diffstat graph of a large change with a wide graph' '
+ test_when_finished "git rm -f --cached big && rm -f big" &&
+ test_seq 50000 >big &&
+ git add -N big &&
+ git diff --stat=46100 --stat-graph-width=46000 -- big >output &&
+ grep " | " output >line &&
+ tr -cd + <line >plus &&
+ test $(wc -c <plus) -eq 46000
+'
+
test_done
--
2.53.0
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling
2026-11-17 10:37 ` [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling Elia Pinto
@ 2026-11-17 16:36 ` Torsten Bögershausen
2026-11-17 23:13 ` Junio C Hamano
1 sibling, 0 replies; 16+ messages in thread
From: Torsten Bögershausen @ 2026-11-17 16:36 UTC (permalink / raw)
To: Elia Pinto
Cc: git, Johannes Schindelin, Junio C Hamano, Alexander Meshcheryakov,
Elijah Newren
Looks good to me.
Test case passed under MacOs.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite()
2026-11-17 10:37 ` [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
@ 2026-11-17 17:52 ` Junio C Hamano
2026-11-17 23:50 ` brian m. carlson
0 siblings, 1 reply; 16+ messages in thread
From: Junio C Hamano @ 2026-11-17 17:52 UTC (permalink / raw)
To: Elia Pinto
Cc: git, Johannes Schindelin, brian m. carlson, Neeraj Singh,
Patrick Steinhardt
Elia Pinto <gitter.spiros@gmail.com> writes:
> hashwrite() takes the byte count as "uint32_t" on all platforms. When
> pack-objects writes the entire compressed buffer of an object in a
> single call, which can exceed 4GiB if the object is in memory, poorly
> compressible and core.bigFileThreshold is raised accordingly, it writes
> only "count mod 2^32" bytes, corrupting the pack.
> ...
> diff --git a/csum-file.h b/csum-file.h
> index 6ed74d1637..f1c8a9df13 100644
> --- a/csum-file.h
> +++ b/csum-file.h
> @@ -75,7 +75,7 @@ void free_hashfile(struct hashfile *f);
> * Finalize the hashfile by flushing data to disk and free'ing it.
> */
> int finalize_hashfile(struct hashfile *, unsigned char *, enum fsync_component, unsigned int);
> -void hashwrite(struct hashfile *, const void *, uint32_t);
> +void hashwrite(struct hashfile *, const void *, size_t);
> void hashflush(struct hashfile *f);
> void crc32_begin(struct hashfile *);
> uint32_t crc32_end(struct hashfile *);
src/csum_file.rs still thinks that hashwrite() takes u32, though.
If a caller of hashwrite() has a contiguous region of memory that
exceeds 4GiB in length, shouldn't it be looping to call hashwrite()
in 4GiB (or less) chunks, without changing the API contract, I would
have to wonder.
pub mod c {
use std::os::raw::{c_char, c_int, c_void};
extern "C" {
pub fn hashfd(algop: *const c_void, fd: i32, name: *const c_char) -> *mut c_void;
pub fn hashwrite(f: *mut c_void, data: *const c_void, len: u32);
pub fn hashflush(f: *mut c_void);
pub fn finalize_hashfile(
f: *mut c_void,
data: *mut u8,
component: u32,
flags: u32,
) -> c_int;
}
}
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling
2026-11-17 10:37 ` [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling Elia Pinto
2026-11-17 16:36 ` Torsten Bögershausen
@ 2026-11-17 23:13 ` Junio C Hamano
1 sibling, 0 replies; 16+ messages in thread
From: Junio C Hamano @ 2026-11-17 23:13 UTC (permalink / raw)
To: Elia Pinto
Cc: git, Johannes Schindelin, Alexander Meshcheryakov,
Torsten Bögershausen, Elijah Newren
Elia Pinto <gitter.spiros@gmail.com> writes:
> struct diffstat_file stores the number of added and deleted lines of
> each file in uintmax_t, but show_stats() and show_shortstats() add them
> up in "int" variables. When the total goes over INT_MAX, the summary
> line overflows. With the previous patch, a single file is enough: for
> a complete rewrite into a file of 2^31 lines, "git diff -B --stat"
> ends with
>
> 1 file changed, -2147483648 insertions(+), 1000 deletions(-)
>
> Several files whose changes add up to more than INT_MAX lines lead to
> the same overflow even without -B.
>
> Use uintmax_t for these totals and for the parameters of
> print_stat_summary() and print_stat_summary_inserts_deletes(), which
> printed them with "%d". print_stat_summary() is also called by
> "git apply --stat", hence the change of its prototype in diff.h.
> apply.c still adds up its own totals in "int"; it is left alone here.
The above is super detailed write up but still leaves readers
confused because a crucial question is left unanswered.
So we use a wider type to accumulate changes to avoid overflowing
the accumulator of a narrower type. But then what else is done to
deal with the accumulator of the winder type that overflows after
enough numbers are added to it?
The remainder of the proposed log message shares exactly the same
issue. It points out different areas that use narrow accumulator
causing trouble by overflowing, talks about replacing them with
accumulator of wider type, but stops there without answering the
same crucial question.
> - graph_width = max_change + 4 > bin_width ? max_change : bin_width - 4;
> + if (max_change + 4 > bin_width)
> + /* anything wider than "width" is cut down below anyway */
> + graph_width = max_change < width ? max_change : width;
> + else
> + graph_width = bin_width - 4;
This change looks more interesting than a mere type widening, but I
do not think we saw it described in the proposed log message that
was a wall of text.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 3/4] diffcore-delta: use size_t for the size of the buffer to hash
2026-11-17 10:37 ` [PATCH v2 3/4] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
@ 2026-11-17 23:21 ` Junio C Hamano
0 siblings, 0 replies; 16+ messages in thread
From: Junio C Hamano @ 2026-11-17 23:21 UTC (permalink / raw)
To: Elia Pinto; +Cc: git, Johannes Schindelin
Elia Pinto <gitter.spiros@gmail.com> writes:
> hash_chars() in diffcore-delta.c takes the buffer size as an "unsigned
> int". With a file of 4GiB or more, it hashes only a prefix; because the
> similarity score is calculated against the full size, similarity is
> undervalued, so true renames are missed, which also affects
> "git log --follow" and merges.
>
> This was verified using a 16-bit simulation, where
> "git diff --cached -M" reported a delete and add instead of a rename
> (e.g., "D a" and "A b" instead of "R099 a b").
>
> Make the size variable "size_t".
>
> Triggering this requires processing a file of 4GiB or more, so no test
> is added.
>
> Signed-off-by: Elia Pinto <gitter.spiros@gmail.com>
> ---
> diffcore-delta.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/diffcore-delta.c b/diffcore-delta.c
> index 2b7db39983..7f10f79ffe 100644
> --- a/diffcore-delta.c
> +++ b/diffcore-delta.c
> @@ -127,7 +127,7 @@ static struct spanhash_top *hash_chars(struct repository *r,
> unsigned int accum1, accum2, hashval;
> struct spanhash_top *hash;
> unsigned char *buf = one->data;
> - unsigned int sz = one->size;
> + size_t sz = one->size;
The cnt member of spanhash is still uint, and can wrap down to 0
after 4 billion collisions and makes it appear the hash bucket is
empty. Should it be widened at the same time, or are there reasons
why it does not need to (like hashval member that can stay uint)?
> int is_text = !diff_filespec_is_binary(r, one);
>
> i = INITIAL_HASH_SIZE;
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite()
2026-11-17 17:52 ` Junio C Hamano
@ 2026-11-17 23:50 ` brian m. carlson
0 siblings, 0 replies; 16+ messages in thread
From: brian m. carlson @ 2026-11-17 23:50 UTC (permalink / raw)
To: Junio C Hamano
Cc: Elia Pinto, git, Johannes Schindelin, Neeraj Singh,
Patrick Steinhardt
[-- Attachment #1: Type: text/plain, Size: 573 bytes --]
On 2026-11-17 at 17:52:57, Junio C Hamano wrote:
> src/csum_file.rs still thinks that hashwrite() takes u32, though.
>
> If a caller of hashwrite() has a contiguous region of memory that
> exceeds 4GiB in length, shouldn't it be looping to call hashwrite()
> in 4GiB (or less) chunks, without changing the API contract, I would
> have to wonder.
Yes, I think that would be the right thing to do. If we're not going to
do that, then we need to update the Rust code accordingly.
--
brian m. carlson (they/them)
Toronto, Ontario, CA
Speaking for myself only
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-11-17 23:50 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-01 15:36 [PATCH 0/3] use size_t for a few sizes that are 32 bits everywhere Elia Pinto
2026-11-01 15:36 ` [PATCH 1/3] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
2026-11-01 15:36 ` [PATCH 2/3] diff: use size_t for the size passed to count_lines() Elia Pinto
2026-11-04 13:58 ` Phillip Wood
2026-11-04 15:45 ` Phillip Wood
2026-11-01 15:36 ` [PATCH 3/3] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
2026-11-17 10:37 ` [PATCH v2 0/4] widen a few sizes and counts that are 32 bits everywhere Elia Pinto
2026-11-17 10:37 ` [PATCH v2 1/4] csum-file: use size_t for the count passed to hashwrite() Elia Pinto
2026-11-17 17:52 ` Junio C Hamano
2026-11-17 23:50 ` brian m. carlson
2026-11-17 10:37 ` [PATCH v2 2/4] diff: use size_t for sizes and line counts of complete rewrites Elia Pinto
2026-11-17 10:37 ` [PATCH v2 3/4] diffcore-delta: use size_t for the size of the buffer to hash Elia Pinto
2026-11-17 23:21 ` Junio C Hamano
2026-11-17 10:37 ` [PATCH v2 4/4] diff: use uintmax_t for diffstat totals and graph scaling Elia Pinto
2026-11-17 16:36 ` Torsten Bögershausen
2026-11-17 23:13 ` Junio C Hamano
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.