* [PATCH 1/2] read-cache: do not trust a size change when conversion is active
2026-10-08 20:45 [PATCH 0/2] status: agree with diff and add when conversion is active Curtis Allen Smith
@ 2026-10-08 20:45 ` Curtis Allen Smith
2026-10-09 5:37 ` Junio C Hamano
2026-10-08 20:45 ` [PATCH 2/2] core: add core.convertAwareStatus to opt out of the content check Curtis Allen Smith
1 sibling, 1 reply; 4+ messages in thread
From: Curtis Allen Smith @ 2026-10-08 20:45 UTC (permalink / raw)
To: git; +Cc: Curtis Allen Smith, Torsten Bögershausen
"git status" can report a file as modified while "git diff" and
"git add", which both run the clean filter, agree its contents are
unchanged:
git init t && cd t
printf '* text eol=lf\n' >.gitattributes
printf 'one\ntwo\nthree\n' >file.txt
git add . && git commit -m init
printf 'one\r\ntwo\r\nthree\r\n' >file.txt
git status --short # -> M file.txt
git diff # -> empty
git add file.txt # -> stages nothing
Three commands that answer the same question answer it differently,
and the two that consult the conversion are the ones that get it
right.
ie_modified() declares a path modified as soon as ie_match_stat()
reports DATA_CHANGED, which it does whenever the size of the file in
the working tree differs from the size recorded in the index, and it
returns without ever reading the file. That shortcut is sound only
while the working tree file and the blob are the same bytes. When it
was written in 2005 they were, and a size mismatch really was a proof
of a content change. Conversion removed that premise: core.autocrlf
arrived in 2007 and the "text" and "eol" attributes in 2010, and
making the two representations differ in their bytes while agreeing on
their content is precisely what they are for. The shortcut was never
re-examined against the feature layered on top of it.
The same function already handles the comparable case correctly. When
only the mtime changed, it falls through to ce_modified_check_fs(),
reads the file, applies the conversion, and answers "unchanged" when
that is the truth. Git is therefore already willing to pay for a
conversion-aware check here; the size branch is the only place where a
difference in the bytes on disk is taken to be a difference in
content.
So fall through to that same check when the size changed and the path
is subject to conversion. Paths without conversion take the early
return exactly as before, and a repository that uses no conversion is
unaffected.
Hashing is the expensive part of that check -- Git's collision
detecting SHA-1 runs at about 800 MB/s on the machine used below --
and it is avoidable most of the time. Contents that are equal
necessarily have equal length, so ce_compare_data() now compares the
length of the converted file against the size of the blob, which costs
an object header lookup, and hashes only when the two agree. A file
that was really edited almost always changes length and is rejected
without being hashed. The file this commit is about has exactly the
length of its blob, so it is hashed, found equal, and the index then
records its new size, after which it is not read again.
When the end-of-line conversion is the only one that applies, even
the conversion can be skipped. "git add" either keeps such a file as
it is or turns each CRLF into LF, so the converted length is one of
two numbers, and a scan for CR gives both. If neither is the size of
the blob, the file is modified, and it is neither converted nor
hashed.
In a repository of 200 files of 1 MB each under "* text=auto" with
every file modified, these take "git status" from 517ms to 26ms; with
10000 files of 2.6 KB, from 142ms to 49ms.
The inconsistency is a chronic annoyance for anyone sharing a tree
between Windows and Unix with normalized line endings. Any tool that
rewrites unchanged files in native line endings -- javadoc, code
generators, formatters, a good number of editors -- changes the size
of every file it touches and flags the whole output tree as modified
with empty diffs. The documented remedy, "git add --renormalize",
does not stick: the next checkout, stash or branch switch rewrites the
checkout-form bytes and the recorded sizes along with them, and the
next run of the tool flags everything again.
Signed-off-by: Curtis Allen Smith <curtis.allen.smith@gmail.com>
---
read-cache.c | 129 ++++++++++++++++++++++++++++++++++++++++++++++--
t/t0020-crlf.sh | 45 +++++++++++++++++
2 files changed, 171 insertions(+), 3 deletions(-)
diff --git a/read-cache.c b/read-cache.c
index c4cf08a3a..8875706d8 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -9,6 +9,7 @@
#include "git-compat-util.h"
#include "config.h"
+#include "convert.h"
#include "date.h"
#include "diff.h"
#include "diffcore.h"
@@ -228,6 +229,99 @@ int fake_lstat(const struct cache_entry *ce, struct stat *st)
return 0;
}
+/*
+ * Count the CRs in "buf" that are immediately followed by an LF.
+ */
+static size_t count_crlf(const char *buf, size_t len)
+{
+ const char *end = buf + len;
+ size_t n = 0;
+
+ while ((buf = memchr(buf, '\r', end - buf))) {
+ if (++buf < end && *buf == '\n')
+ n++;
+ }
+ return n;
+}
+
+/*
+ * Compare an open file to the blob recorded for it, converting the file
+ * the way "git add" would. Contents that are equal necessarily have
+ * equal length, so when the converted length differs from the size of
+ * the blob the file is modified and there is no need to hash it, and
+ * hashing is by far the most expensive part of this comparison.
+ *
+ * Only the case this can help is handled here: a regular file whose
+ * conversion Git performs itself. A path driven by an external filter
+ * is left to index_fd(), which streams it into the filter.
+ *
+ * Returns 1 if the file differs, 0 if it matches, -1 if it could not be
+ * read. Does not close "fd".
+ */
+static int ce_compare_converted_data(struct index_state *istate,
+ const struct cache_entry *ce,
+ struct stat *st, int fd)
+{
+ struct strbuf raw = STRBUF_INIT;
+ struct strbuf converted = STRBUF_INIT;
+ struct object_info oi = OBJECT_INFO_INIT;
+ struct object_id oid;
+ struct conv_attrs ca;
+ enum object_type type;
+ size_t blob_size;
+ const char *buf;
+ size_t len;
+ int have_size, match = -1;
+
+ oi.typep = &type;
+ oi.sizep = &blob_size;
+ have_size = (odb_read_object_info_extended(istate->repo->objects,
+ &ce->oid, &oi,
+ OBJECT_INFO_SKIP_FETCH_OBJECT |
+ OBJECT_INFO_QUICK) == ODB_READ_OK &&
+ type == OBJ_BLOB);
+
+ if (strbuf_read(&raw, fd, st->st_size) < 0)
+ goto out;
+
+ /*
+ * When the end-of-line conversion is the only one, "git add"
+ * either keeps the file as it is or turns every CRLF into LF
+ * ("text=auto" refuses to convert a file with a lone CR, so
+ * stripping all CRs comes to the same thing). The converted
+ * length is therefore one of two values, and if neither is the
+ * size of the blob the file is modified without converting it.
+ */
+ convert_attrs(istate, &ca, ce->name);
+ if (have_size && !ca.drv && !ca.ident &&
+ !ca.working_tree_encoding &&
+ raw.len != blob_size &&
+ raw.len - count_crlf(raw.buf, raw.len) != blob_size) {
+ match = 1;
+ goto out;
+ }
+
+ buf = raw.buf;
+ len = raw.len;
+ if (convert_to_git(istate, ce->name, raw.buf, raw.len, &converted, 0)) {
+ buf = converted.buf;
+ len = converted.len;
+ }
+
+ if (have_size && len != blob_size) {
+ match = 1;
+ goto out;
+ }
+
+ hash_object_file(istate->repo->hash_algo, buf, len, OBJ_BLOB, &oid);
+ match = !oideq(&oid, &ce->oid);
+
+out:
+ strbuf_release(&raw);
+ strbuf_release(&converted);
+ return match;
+}
+
static int ce_compare_data(struct index_state *istate,
const struct cache_entry *ce,
struct stat *st)
@@ -237,9 +331,16 @@ static int ce_compare_data(struct index_state *istate,
if (fd >= 0) {
struct object_id oid;
- if (!index_fd(istate, &oid, fd, st, OBJ_BLOB, ce->name, 0))
+
+ if (S_ISREG(st->st_mode) &&
+ would_convert_to_git(istate, ce->name) &&
+ !would_convert_to_git_filter_fd(istate, ce->name)) {
+ match = ce_compare_converted_data(istate, ce, st, fd);
+ close(fd);
+ } else if (!index_fd(istate, &oid, fd, st, OBJ_BLOB, ce->name, 0)) {
match = !oideq(&oid, &ce->oid);
- /* index_fd() closed the file descriptor already */
+ /* index_fd() closed the file descriptor already */
+ }
}
return match;
}
@@ -438,6 +539,27 @@ int ie_match_stat(struct index_state *istate,
return changed;
}
+/*
+ * A difference between the size of the file in the working tree and the
+ * size recorded for it in the index proves that the contents changed
+ * only as long as the two are byte-for-byte comparable. That stops
+ * being true as soon as the path is run through a clean filter:
+ * rewriting a file with CRLF endings under "text eol=lf", for example,
+ * changes its size in the working tree without changing the blob Git
+ * would record for it. For such a path the only way to tell is to read
+ * the contents and convert them, which is what we already do when only
+ * the mtime changed.
+ */
+static int size_change_is_conclusive(struct index_state *istate,
+ const struct cache_entry *ce,
+ struct stat *st)
+{
+ if (!S_ISREG(st->st_mode))
+ return 1;
+
+ return !would_convert_to_git(istate, ce->name);
+}
+
int ie_modified(struct index_state *istate,
const struct cache_entry *ce,
struct stat *st, unsigned int options)
@@ -480,7 +602,8 @@ int ie_modified(struct index_state *istate,
*/
(!S_ISLNK(st->st_mode) || ce->ce_stat_data.sd_size != MAX_PATH) &&
#endif
- (S_ISGITLINK(ce->ce_mode) || ce->ce_stat_data.sd_size != 0))
+ (S_ISGITLINK(ce->ce_mode) || ce->ce_stat_data.sd_size != 0) &&
+ size_change_is_conclusive(istate, ce, st))
return changed;
changed_fs = ce_modified_check_fs(istate, ce, st);
diff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh
index fd1cae09e..88b728d50 100755
--- a/t/t0020-crlf.sh
+++ b/t/t0020-crlf.sh
@@ -397,4 +397,49 @@ test_expect_success 'New CRLF file gets LF in repo' '
test_cmp alllf alllf2
'
+test_expect_success 'status does not report a CRLF-only rewrite as modified' '
+ git init eol-status &&
+ (
+ cd eol-status &&
+ echo "* text eol=lf" >.gitattributes &&
+ printf "one\ntwo\nthree\n" >file.txt &&
+ git add .gitattributes file.txt &&
+ git commit -m initial &&
+
+ # a generator rewrites the file with CRLF, same content
+ printf "one\r\ntwo\r\nthree\r\n" >file.txt &&
+ git status --porcelain -uno >actual &&
+ test_must_be_empty actual &&
+ git diff --exit-code &&
+
+ # a real change is still reported
+ printf "one\r\ntwo\r\nfour\r\n" >file.txt &&
+ git status --porcelain -uno >actual &&
+ echo " M file.txt" >expect &&
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'status sizes a text file by its CRLF pairs, not its CRs' '
+ git init eol-status-lone-cr &&
+ (
+ cd eol-status-lone-cr &&
+ echo "* text eol=lf" >.gitattributes &&
+ printf "one\rtwo\nthree\n" >file.txt &&
+ git add .gitattributes file.txt &&
+ git commit -m initial &&
+
+ # "git add" keeps the lone CR and drops the others
+ printf "one\rtwo\r\nthree\r\n" >file.txt &&
+ git status --porcelain -uno >actual &&
+ test_must_be_empty actual &&
+
+ # the converted length matches the blob, the content does not
+ printf "one\rtwo\nthrEE\r\n" >file.txt &&
+ git status --porcelain -uno >actual &&
+ echo " M file.txt" >expect &&
+ test_cmp expect actual
+ )
+'
+
test_done
--
2.53.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH 2/2] core: add core.convertAwareStatus to opt out of the content check
2026-10-08 20:45 [PATCH 0/2] status: agree with diff and add when conversion is active Curtis Allen Smith
2026-10-08 20:45 ` [PATCH 1/2] read-cache: do not trust a size change " Curtis Allen Smith
@ 2026-10-08 20:45 ` Curtis Allen Smith
1 sibling, 0 replies; 4+ messages in thread
From: Curtis Allen Smith @ 2026-10-08 20:45 UTC (permalink / raw)
To: git; +Cc: Curtis Allen Smith, Torsten Bögershausen
The previous commit makes an index refresh read and convert a path
whose size changed when conversion is active for it, so that "git
status" agrees with "git diff" and "git add". Reading costs more than
trusting the size, and when the path has a clean filter configured the
cost includes running that filter -- Git LFS on a large file, say.
Reading is cheap enough on current hardware that agreeing with
"git diff" is the better default, but nobody should be stuck with it
if their filters are expensive. Add core.convertAwareStatus:
true (default) consult the conversion for any path that has one,
including paths with a clean filter
no-filter consult only the conversions Git performs itself,
and decide a path with a clean filter on its size
false always treat a size change as a modification, as
Git did before
Being ordinary configuration, it can equally be given for a single
command:
git -c core.convertAwareStatus=no-filter status
Paths that are not subject to conversion are decided on their size
alone in every mode, so this costs nothing in a repository that does
not use conversion.
Signed-off-by: Curtis Allen Smith <curtis.allen.smith@gmail.com>
---
Documentation/config/core.adoc | 22 ++++++++++++++++
environment.c | 14 ++++++++++
environment.h | 7 +++++
read-cache.c | 12 +++++++++
t/t0020-crlf.sh | 47 ++++++++++++++++++++++++++++++++++
5 files changed, 102 insertions(+)
diff --git a/Documentation/config/core.adoc b/Documentation/config/core.adoc
index 0b697f53f..9737c804f 100644
--- a/Documentation/config/core.adoc
+++ b/Documentation/config/core.adoc
@@ -156,6 +156,28 @@ some fields (e.g. JGit); by excluding these fields from the
comparison, the `minimal` mode may help interoperability when the
same repository is used by these other systems at the same time.
+core.convertAwareStatus::
+ When a path is subject to content conversion -- the `text` and
+ `eol` attributes, `core.autocrlf`, a `working-tree-encoding`,
+ or a clean filter -- the size of the file in the working tree
+ is not determined by its contents alone, so a change in size
+ does not prove that the contents changed. When this variable
+ is missing or set to `true`, Git reads and converts such a
+ file before reporting it as modified, which keeps 'git status'
+ in agreement with 'git diff' and 'git add'. When set to
+ `no-filter`, Git does this only for the conversions it
+ performs itself, and a path with a clean filter configured
+ (Git LFS, for example) is reported as modified on a size
+ change without running the filter. When set to `false`, a
+ size change is always taken as a modification, which is what
+ Git did before this variable existed.
++
+Reading the file costs more than trusting its size, so `no-filter`
+and `false` trade this consistency for speed in repositories where
+running the filter, or reading the file at all, is too expensive.
+Paths that are not subject to conversion are decided on the size
+alone in every mode.
+
core.quotePath::
Commands that output paths (e.g. 'ls-files', 'diff'), will
quote "unusual" characters in the pathname by enclosing the
diff --git a/environment.c b/environment.c
index c83cf4483..b079262c8 100644
--- a/environment.c
+++ b/environment.c
@@ -344,6 +344,19 @@ int git_default_core_config(const char *var, const char *value,
var, value);
}
+ if (!strcmp(var, "core.convertawarestatus")) {
+ int b = git_parse_maybe_bool(value);
+ if (0 <= b)
+ cfg->convert_aware_status = b ? CONVERT_AWARE_STATUS_ALL
+ : CONVERT_AWARE_STATUS_NEVER;
+ else if (value && !strcasecmp(value, "no-filter"))
+ cfg->convert_aware_status = CONVERT_AWARE_STATUS_IN_PROCESS;
+ else
+ return error(_("invalid value for '%s': '%s'"),
+ var, value);
+ return 0;
+ }
+
if (!strcmp(var, "core.quotepath")) {
quote_path_fully = git_config_bool(var, value);
return 0;
@@ -766,6 +779,7 @@ void repo_config_values_init(struct repo_config_values *cfg)
cfg->apply_sparse_checkout = 0;
cfg->trust_ctime = 1;
cfg->check_stat = 1;
+ cfg->convert_aware_status = CONVERT_AWARE_STATUS_ALL;
cfg->zlib_compression_level = Z_BEST_SPEED;
cfg->pack_compression_level = Z_DEFAULT_COMPRESSION;
cfg->precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */
diff --git a/environment.h b/environment.h
index b336459e9..49a88de27 100644
--- a/environment.h
+++ b/environment.h
@@ -115,6 +115,12 @@ enum object_creation_mode {
OBJECT_CREATION_USES_RENAMES = 1
};
+enum convert_aware_status {
+ CONVERT_AWARE_STATUS_NEVER = 0,
+ CONVERT_AWARE_STATUS_IN_PROCESS,
+ CONVERT_AWARE_STATUS_ALL
+};
+
struct repo_config_values {
/* section "core" config values */
char *attributes_file;
@@ -130,6 +136,7 @@ struct repo_config_values {
int apply_sparse_checkout;
int trust_ctime;
int check_stat;
+ enum convert_aware_status convert_aware_status;
int zlib_compression_level;
int pack_compression_level;
int precomposed_unicode;
diff --git a/read-cache.c b/read-cache.c
index 8875706d8..2bb27a388 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -554,9 +554,21 @@ static int size_change_is_conclusive(struct index_state *istate,
const struct cache_entry *ce,
struct stat *st)
{
+ struct repo_config_values *cfg = repo_config_values(the_repository);
+ struct conv_attrs ca;
+
+ if (cfg->convert_aware_status == CONVERT_AWARE_STATUS_NEVER)
+ return 1;
+
if (!S_ISREG(st->st_mode))
return 1;
+ if (cfg->convert_aware_status == CONVERT_AWARE_STATUS_IN_PROCESS) {
+ convert_attrs(istate, &ca, ce->name);
+ if (ca.drv)
+ return 1;
+ }
+
return !would_convert_to_git(istate, ce->name);
}
diff --git a/t/t0020-crlf.sh b/t/t0020-crlf.sh
index 88b728d50..4c127207e 100755
--- a/t/t0020-crlf.sh
+++ b/t/t0020-crlf.sh
@@ -442,4 +442,51 @@ test_expect_success 'status sizes a text file by its CRLF pairs, not its CRs' '
)
'
+test_expect_success 'core.convertAwareStatus=false restores the size shortcut' '
+ git init convert-aware &&
+ (
+ cd convert-aware &&
+ echo "* text eol=lf" >.gitattributes &&
+ printf "one\ntwo\nthree\n" >file.txt &&
+ git add .gitattributes file.txt &&
+ git commit -m initial &&
+ printf "one\r\ntwo\r\nthree\r\n" >file.txt &&
+
+ git -c core.convertAwareStatus=false status --porcelain -uno >actual &&
+ echo " M file.txt" >expect &&
+ test_cmp expect actual &&
+
+ git -c core.convertAwareStatus=true status --porcelain -uno >actual &&
+ test_must_be_empty actual
+ )
+'
+
+test_expect_success 'core.convertAwareStatus=no-filter leaves clean filters alone' '
+ git init convert-aware-filter &&
+ (
+ cd convert-aware-filter &&
+ write_script stripcr <<-\EOF &&
+ tr -d "\015"
+ EOF
+ echo "file.txt filter=stripcr" >.gitattributes &&
+ git config filter.stripcr.clean ./stripcr &&
+ printf "one\ntwo\nthree\n" >file.txt &&
+ git add .gitattributes file.txt &&
+ git commit -m initial &&
+ printf "one\r\ntwo\r\nthree\r\n" >file.txt &&
+
+ git -c core.convertAwareStatus=no-filter status --porcelain -uno >actual &&
+ echo " M file.txt" >expect &&
+ test_cmp expect actual &&
+
+ git status --porcelain -uno >actual &&
+ test_must_be_empty actual
+ )
+'
+
+test_expect_success 'core.convertAwareStatus rejects an unknown value' '
+ test_must_fail git -c core.convertAwareStatus=bogus status 2>err &&
+ test_grep "invalid value" err
+'
+
test_done
--
2.53.0
^ permalink raw reply related [flat|nested] 4+ messages in thread