Empty directory showing as untracked with untrackedCache (with reproduction script)

[Nikita Leshenko <nikita@island.io>]

Summary: Enabling untrackedCache sometimes causes an empty directory to show as
untracked (as if the directory contains content). This is racy but there's a
script to reproduce it deterministically.

What did you do before the bug happened? (Steps to reproduce your issue)

- Create repo with some content
- Enable untrackedCache
- git status, shows nothing
- create aa/bb
- git status, shows '?? aa/'
- rm aa/bb (leaving aa/ empty)
- git status

What did you expect to happen? (Expected behavior)

git status should show nothing.

What happened instead? (Actual behavior)

git status shows aa/ as having untracked content.

Anything else you want to add:

Here's a Python script to reproduce:
------------------------ >8 ------------------------
import os
from subprocess import check_output, check_call

check_call(['git', 'init'])

# Populate directory for some load, makes reproduction faster.
for a in range(40):
  os.mkdir(f'z_{a}')
  for b in range(40):
    os.mkdir(f'z_{a}/{b}')
    for c in range(40):
      with open(f'z_{a}/{b}/{c}', 'w') as f:
        pass
check_call(['git', 'add', '.'])
check_call(['git', 'commit', '-m', '.', '-q'])

# Enable untracked cache.
check_call(['git', 'update-index', '--test-untracked-cache'])
check_call(['git', 'config', 'core.untrackedCache', 'true'])

# Repeatedly create/delete same file.
for i in range(1, 1000):

  s = check_output(['git', 'status', '--porcelain'])
  assert not s, "Expected clean git status"

  os.mkdir(f'aa')
  with open('aa/bb', 'w') as f:
    pass
  s = check_output(['git', 'status', '--porcelain'])
  assert s == b'?? aa/\n', "Expected aa dir to show up"

  os.unlink('aa/bb')

  s = check_output(['git', 'status', '--porcelain'])
  if s:
    print("\n\n=============== REPRODUCED! ===============\n")
    print(f'Reproduced in {i} attempts')

    print(f'\n$ ls -la aa/')
    check_call(['ls', '-la', 'aa/'])

    print(f'\n$ git status')
    check_call(['git', 'status'])

    print(f'\n$ git --version')
    check_call(['git', '--version'])
    break

  os.rmdir(f'aa')

else:
  print('Failed to reproduce')
------------------------ >8 ------------------------

Example output of this script:
------------------------ >8 ------------------------
Initialized empty Git repository in /home/user/tmp/repro/.git/
Testing mtime in '/home/user/tmp/repro' ...... OK


=============== REPRODUCED! ===============

Reproduced in 90 attempts

$ ls -la aa/
total 8
drwxr-xr-x.  2 user user 4096 Oct 21 19:46 .
drwxr-xr-x. 44 user user 4096 Oct 21 19:46 ..

$ git status
On branch master
Untracked files:
  (use "git add <file>..." to include in what will be committed)
aa/

nothing added to commit but untracked files present (use "git add" to track)

$ git --version
git version 2.47.0.107.g34b6ce9b30
------------------------ >8 ------------------------

This reproduced in the following Git versions:
- Latest master (34b6ce9b30)
- Latest shipping in Fedora (2.47.0)
- Ubuntu 22 LTS (2.34.1)

[System Info]
git version:
git version 2.47.0.107.g34b6ce9b30
cpu: x86_64
built from commit: 34b6ce9b30747131b6e781ff718a45328aa887d0
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
libcurl: 8.6.0
OpenSSL: OpenSSL 3.2.2 4 Jun 2024
zlib: 1.3.1.zlib-ng
uname: Linux 6.11.3-200.fc40.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Oct 10
22:31:19 UTC 2024 x86_64
compiler info: gnuc: 14.2
libc info: glibc: 2.39
$SHELL (typically, interactive shell): /bin/zsh


[Enabled Hooks]