From: Tamir Duberstein <tamird@gmail.com>
To: git@vger.kernel.org
Cc: "Tao Klerks" <tao@klerks.biz>,
"Junio C Hamano" <gitster@pobox.com>,
"Elijah Newren" <newren@gmail.com>, "Jeff King" <peff@peff.net>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Tamir Duberstein" <tamird@gmail.com>
Subject: [PATCH v3 0/3] ls-files: reuse and update the untracked cache
Date: Fri, 23 Oct 2026 16:03:03 -0400 [thread overview]
Message-ID: <20261023-ls-files-untracked-cache-v3-0-292260e95f53@gmail.com> (raw)
In-Reply-To: <20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com>
Repeated queries such as
git ls-files --cached --others --exclude-standard -z -- "**/pyproject.toml"
walk the working tree even when status has populated an untracked cache.
This series lets ls-files reuse those listings and save the work for later
commands through Git's existing optional index writes.
The first patch fixes inconsistent ignore-file hashes that invalidate an
unchanged cache. The second lets 'git status -unormal' and
'git status -uall' reuse the same cache. 'git status -unormal' still stops
scanning an untracked directory after finding an untracked file; a later
command requesting all untracked files completes the listing as needed.
The third lets ls-files use the cache and write pending untracked cache
and fsmonitor updates back to the index. --no-optional-locks suppresses
those writes.
For wildcard pathspecs without a fixed prefix, apply the pathspec after
reading complete directory listings. A first query can therefore scan
directories that its pathspec would otherwise skip. Fixed-prefix,
attribute and exclude pathspecs retain their existing traversal.
A new dir_flags value makes older versions of Git rebuild the cache before
using it to list untracked files. Changes invalidate cached entries in
parent directories as well, so updates can reopen more directories than
a cache populated with --untracked-files=all before this series.
On macOS, with 100,000 files in 5,000 leaf directories, half tracked.
Each group of 100 leaf directories has 50 tracked and 50 untracked.
Each binary populated its own cache using the indicated status mode.
hyperfine --warmup 3 --runs 7 (mean ± standard deviation, milliseconds):
cache fsmonitor command upstream v1 v2
normal off query 241.3 ± 14.8 132.7 ± 20.0 33.3 ± 2.4
normal off query + status 292.7 ± 11.2 187.0 ± 10.8 92.6 ± 3.3
all off query 231.3 ± 14.6 29.3 ± 1.3 34.7 ± 2.0
normal on query 223.1 ± 17.8 111.0 ± 13.4 33.8 ± 1.5
all on query 216.6 ± 12.0 28.0 ± 5.1 34.6 ± 0.8
"query" is the ls-files command above; "status" is
"git status -unormal --porcelain". These are repeated queries with warm
filesystem and untracked caches.
Assisted-by: LLM
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Changes in v3:
- Release the index lock after reading the directory, before printing any
paths. A slow reader of ls-files output can no longer block index writers.
- Skip the optional write with --debug so the listing shows the stat data
read from the index.
- Link to v2: https://patch.msgid.link/20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com
Changes in v2:
- Share the cache between --untracked-files=normal and --untracked-files=all.
- Write ls-files untracked cache and fsmonitor updates back to the index.
- Test alternating modes, cache invalidation and optional index writes.
- Replace the v1 measurements with results for the revised implementation.
- Link to v1: https://patch.msgid.link/20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com
---
Tamir Duberstein (3):
dir: hash ignore files before appending newline
dir: share untracked caches across output modes
ls-files: use and update the untracked cache
Documentation/git-ls-files.adoc | 4 +
Documentation/gitformat-index.adoc | 13 +-
builtin/ls-files.c | 59 +++++---
dir.c | 251 +++++++++++++++++---------------
dir.h | 16 +--
t/perf/p3010-ls-files.sh | 15 ++
t/t3001-ls-files-others-exclude.sh | 20 +++
t/t7063-status-untracked-cache.sh | 288 +++++++++++++++++++++++++++++--------
t/t7519-status-fsmonitor.sh | 44 ++++++
9 files changed, 506 insertions(+), 204 deletions(-)
---
Range-diff versus v2:
1: d45e8c7ea9 = 1: fb0b53b2cc dir: hash ignore files before appending newline
2: b2ea9e163c = 2: ca78501354 dir: share untracked caches across output modes
3: a84c08dabe ! 3: 4622b804a6 ls-files: use and update the untracked cache
@@ Commit message
options can be processed in the same order before reading the index.
A pathspec prefix can prune the in-memory index, and --with-tree can add
- entries to it. Do not write the index in those cases. Respect
- --no-optional-locks and continue without writing when the index lock is
- busy.
+ entries to it. Do not write the index in those cases or with --debug,
+ which prints stat data that an index write can change. Respect
+ --no-optional-locks and continue without writing when the lock is busy.
+
+ Finish the optional update after the directory walk and before printing
+ paths, so a slow reader cannot block another index writer.
Assisted-by: LLM
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
@@ builtin/ls-files.c
#include "object-name.h"
#include "strbuf.h"
#include "parse-options.h"
+@@ builtin/ls-files.c: static void show_files(struct repository *repo, struct dir_struct *dir)
+ int i;
+ struct strbuf fullname = STRBUF_INIT;
+
+- /* For cached/deleted files we don't need to even do the readdir */
+- if (show_others || show_killed) {
+- if (!show_others)
+- dir->flags |= DIR_COLLECT_KILLED_ONLY;
+- fill_directory(dir, repo->index, &pathspec);
+- if (show_others)
+- show_other_files(repo->index, dir);
+- if (show_killed)
+- show_killed_files(repo->index, dir);
+- }
++ if (show_others)
++ show_other_files(repo->index, dir);
++ if (show_killed)
++ show_killed_files(repo->index, dir);
+
+ if (!(show_cached || show_stage || show_deleted || show_modified))
+ return;
@@ builtin/ls-files.c: int cmd_ls_files(int argc,
struct dir_struct dir = DIR_INIT;
struct pattern_list *pl;
@@ builtin/ls-files.c: int cmd_ls_files(int argc,
+ /*
+ * Do not save an index pruned by a pathspec or changed by --with-tree.
++ * --debug shows stat data that writing the index can change.
+ * With index.skipHash, the checksum cannot detect concurrent index
+ * changes. Lock the index before reading it.
+ */
+ if (show_others && !dir.flags && dir.exclude_per_dir &&
-+ !max_prefix && !with_tree && use_optional_locks() &&
++ !max_prefix && !with_tree && !debug_mode && use_optional_locks() &&
+ !dir.internal.unmanaged_exclude_files && !exclude_list.nr &&
+ repo->settings.core_untracked_cache != UNTRACKED_CACHE_REMOVE)
+ index_fd = repo_hold_locked_index(repo, &index_lock, 0);
@@ builtin/ls-files.c: int cmd_ls_files(int argc,
/* Treat unmatching pathspec elements as errors */
@@ builtin/ls-files.c: int cmd_ls_files(int argc,
- ret = 1;
+ overlay_tree_on_index(repo->index, with_tree, max_prefix);
}
-+ if (index_fd >= 0) {
-+ if (!ret && dir.untracked)
-+ repo_update_index_if_able(repo, &index_lock);
-+ else
-+ rollback_lock_file(&index_lock);
++ if (show_others || show_killed) {
++ if (!show_others)
++ dir.flags |= DIR_COLLECT_KILLED_ONLY;
++ fill_directory(&dir, repo->index, &pathspec);
+ }
+
- string_list_clear(&exclude_list, 0);
- dir_clear(&dir);
- free(max_prefix);
++ /* Don't hold the index lock while writing paths to stdout. */
++ if (index_fd >= 0 && dir.untracked)
++ repo_update_index_if_able(repo, &index_lock);
++ rollback_lock_file(&index_lock);
++
+ show_files(repo, &dir);
+
+ if (show_resolve_undo)
## dir.c ##
@@ dir.c: static int add_patterns(const char *fname, const char *base, int baselen,
@@ t/t7063-status-untracked-cache.sh: test_expect_success 'filtered status retains
+ test_cmp_bin ../saved-index .git/index
+ )
+'
++
++test_expect_success 'ls-files --debug shows the stat data read from the index' '
++ (
++ cd persistent-cache &&
++ git config core.trustctime false &&
++ printf one >tracked/racy &&
++ test-tool chmtime =1000000000 tracked/racy &&
++ git add tracked/racy &&
++ printf two >tracked/racy &&
++ test-tool chmtime =1000000000 tracked/racy .git/index &&
++
++ git --no-optional-locks ls-files -co --exclude-standard --debug >../expect &&
++ git ls-files -co --exclude-standard --debug >../actual &&
++ test_cmp ../expect ../actual
++ )
++'
+
test_done
---
base-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7
change-id: 20260923-ls-files-untracked-cache-3559bed01a3e
next prev parent reply other threads:[~2026-10-23 20:03 UTC|newest]
Thread overview: 13+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-23 17:10 [PATCH 0/2] ls-files: reuse the untracked cache Tamir Duberstein
2026-09-23 17:10 ` [PATCH 1/2] dir: hash ignore files before adding parser LF Tamir Duberstein
2026-09-23 17:10 ` [PATCH 2/2] ls-files: reuse cached untracked listings Tamir Duberstein
2026-09-23 18:38 ` [PATCH 0/2] ls-files: reuse the untracked cache Tamir Duberstein
2026-09-23 22:26 ` [PATCH v2 0/3] ls-files: reuse and update " Tamir Duberstein
2026-09-23 22:26 ` [PATCH v2 1/3] dir: hash ignore files before appending newline Tamir Duberstein
2026-09-23 22:26 ` [PATCH v2 2/3] dir: share untracked caches across output modes Tamir Duberstein
2026-09-23 22:27 ` [PATCH v2 3/3] ls-files: use and update the untracked cache Tamir Duberstein
2026-10-23 2:13 ` [PATCH v2 0/3] ls-files: reuse " Tamir Duberstein
2026-10-23 20:03 ` Tamir Duberstein [this message]
2026-10-23 20:03 ` [PATCH v3 1/3] dir: hash ignore files before appending newline Tamir Duberstein
2026-10-23 20:03 ` [PATCH v3 2/3] dir: share untracked caches across output modes Tamir Duberstein
2026-10-23 20:03 ` [PATCH v3 3/3] ls-files: use and update the untracked cache Tamir Duberstein
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261023-ls-files-untracked-cache-v3-0-292260e95f53@gmail.com \
--to=tamird@gmail.com \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=newren@gmail.com \
--cc=peff@peff.net \
--cc=tao@klerks.biz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox