[RFC PATCH 0/3] Parallel submodule status

[RFC PATCH 0/3] Parallel submodule status

[Atneya Nair]

This set tries to speed up git status (and other commands which examine
the tree state) for repositories with many submodules.

Currently, submodules, unlike regular files, are examined serially
despite the expense of this operation. Fsmonitor also doesn't support
submodules, so there isn't a great alternative here.

Atneya Nair (3):
  Make read_gitfile and resolve_gitfile thread safe
  Make ce_compare_gitlink thread-safe
  Preload submodule state in refresh_index

 builtin/init-db.c   |  7 ++++---
 builtin/rev-parse.c |  4 +++-
 config.c            |  3 ++-
 config.h            |  2 +-
 preload-index.c     | 25 ++++++++++++++++++++++---
 read-cache-ll.h     |  1 +
 read-cache.c        |  3 +++
 refs.c              |  9 +++++++++
 repository.c        |  9 +++++----
 setup.c             | 36 +++++++++++++++++++++++++-----------
 setup.h             |  7 +++----
 submodule.c         | 32 +++++++++++++++++++++++---------
 worktree.c          | 27 +++++++++++++--------------
 13 files changed, 114 insertions(+), 51 deletions(-)


base-commit: f41f85c9ec8d4d46de0fd5fded88db94d3ec8c11
-- 
2.44.0.rc1.240.g4c46232300-goog

Re: [RFC PATCH 0/3] Parallel submodule status

[Benjamin]

> This set tries to speed up git status (and other commands which examine the tree state) for repositories with many submodules.

I built these patches on top of master from d6fd04375f (The twelfth batch, 2024-03-28), and that went smoothly. However, I used hyperfine to benchmark the changes in a repo of mine with many submodules (which I have used as a case study for the performance before [1]), and I don’t notice any change:

λ hyperfine 'git status' ~c/git/git-status 'git status --ignore-submodules=all' '~/code/git/git-status --ignore-submodules=all'
Benchmark 1: git status
  Time (mean ± σ):     977.8 ms ±   3.4 ms    [User: 415.4 ms, System: 460.2 ms]
  Range (min … max):   971.5 ms … 983.4 ms    10 runs

Benchmark 2: /Users/Knoble/code/git/git-status
  Time (mean ± σ):     987.0 ms ±   6.0 ms    [User: 419.5 ms, System: 461.5 ms]
  Range (min … max):   980.5 ms … 1000.1 ms    10 runs

Benchmark 3: git status --ignore-submodules=all
  Time (mean ± σ):      29.2 ms ±   0.5 ms    [User: 8.5 ms, System: 19.1 ms]
  Range (min … max):    27.8 ms …  30.4 ms    90 runs

Benchmark 4: ~/code/git/git-status --ignore-submodules=all
  Time (mean ± σ):      32.3 ms ±   0.6 ms    [User: 10.5 ms, System: 19.1 ms]
  Range (min … max):    31.0 ms …  35.0 ms    81 runs

Summary
  git status --ignore-submodules=all ran
    1.11 ± 0.03 times faster than ~/code/git/git-status --ignore-submodules=all
   33.47 ± 0.63 times faster than git status
   33.78 ± 0.66 times faster than /Users/Knoble/code/git/git-status


Is there something I needed to enable to see the speedup (am I « holding it wrong »)? Or is this series not actually implementing the parallelization yet—perhaps it is a series of preparatory commits?

I look forward to an implementation of this feature; searching the mailing list reveals some candidates which don’t appear to have ever made it to master [2].

[1]: https://lore.kernel.org/git/CALnO6CCoXOZTsfag6yN_Ffn+H7KE-KTzm+P-GqLKnDMg8j_Qmg@mail.gmail.com/
[2]: https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/