[PATCH v3 0/4] worktree: copy-on-write creation and ephemeral branches

["Jason Newton via GitGitGadget" <gitgitgadget@gmail.com>]

When many worktrees share one repository -- for example a fleet of agents
each needing an isolated checkout -- "git worktree add" is costly at scale.
Objects are shared via the common directory, but the working tree is not:
each add rewrites every tracked file, so N worktrees cost N full checkouts
of disk and I/O. A branch created only to back one of those worktrees is
also left behind when the worktree is removed.

Patch 1 adds "git worktree add --reflink". On a copy-on-write filesystem it
populates the new worktree by reflinking the current worktree's files and
index, then "git reset --hard" rewrites only the paths that differ from .
reflink_file() in copy.c uses FICLONE (Linux), clonefile() (macOS), and ReFS
block cloning (FSCTL_DUPLICATE_EXTENTS_TO_FILE, as Git LFS does). Elsewhere
it is probed up front and falls back to a normal checkout. worktree.reflink
(true/false/auto) sets the default; --no-reflink overrides.

Uncommitted edits to tracked files in the source do not leak into the new
worktree. The copied index is refreshed before reset, and because every
reflinked file is a new inode the refresh re-validates by content. Only
untracked files are carried over. The option stays opt-in for that reason:
it copies the source's untracked and ignored files, which a plain checkout
does not.

For many worktrees at one commit, the series does not share a branch.
git-worktree(1) shows a single update-ref transaction that creates a ref per
worktree (or a detached HEAD). A shared branch still lets a stale index in
one worktree silently revert a commit made in another.

Patch 2 treats a branch that "git worktree add " created itself as owned by
that worktree. "git worktree remove" deletes it. Branches named with -b/-B,
or that already existed, are left alone. --keep-branch keeps an owned
branch.

Changes since v1:

 * Dropped the patch that allowed one branch to be checked out in several
   worktrees. Documented the per-ref / detached-HEAD pattern instead (brian
   m. carlson, Junio).
 * Explained, in the commit message and the manual, why a dirty source
   worktree does not leak tracked content (brian m. carlson).
 * Added the ReFS block-cloning arm (brian m. carlson).
 * Noted overlayfs as a supported setup when the lower filesystem supports
   reflinks (brian m. carlson).
 * Kept --reflink opt-in. A whole-tree reflink is not invisible: it carries
   untracked files, and cloning every file can be slower than checkout when
   the files are small (Junio; qub1n on the ReFS work).
 * Rebased onto 47fdf2208c ("The 7th batch"). reflink_file() now takes a
   repository argument, matching copy_file().
 * Added ephemeral-branch reaping for the auto-named branch that "worktree
   add " creates.

ReFS measurements (32 GiB Dev Drive, Windows build 26300). The USB disk
itself would not format as ReFS, so the volume is a Dev Drive VHD on an NTFS
partition of that disk. NTFS rejects the clone ioctl (win32 error 1 there,
error 50 on C:) and falls back to a full write.

The clone shares the source clusters and does not flush. Four thousand
clones of one 8 MiB file, 31.3 GiB of logical names, were read back
unbuffered and SHA-256'd in full: 4000 of 4000 matched the source. The
source was then overwritten with a different pattern. A read-only
FlushFileBuffers failed with win32 error 5; a writable reopen succeeded.
After that flush, 4000 of 4000 clones still hashed to the original contents
and none hashed to the new source. While the clones were created the
free-space counter fell to about 1 GiB, then returned to 28.38 GiB with the
clones still present. The backing VHD did not grow. Deleting the clones
freed nothing further.

Jason Newton (4):
  worktree: add --reflink for copy-on-write worktree creation
  worktree: reap ephemeral branches created by "worktree add"
  worktree: disable reflink on Windows
  worktree: keep the reflink probe referenced on Windows

 Documentation/config/worktree.adoc |  12 ++
 Documentation/git-worktree.adoc    |  75 ++++++-
 builtin/branch.c                   |  28 +++
 builtin/worktree.c                 | 315 ++++++++++++++++++++++++++++-
 copy.c                             |  83 ++++++++
 copy.h                             |  14 ++
 t/t2400-worktree-add.sh            | 166 +++++++++++++++
 t/t3200-branch.sh                  |   2 +-
 t/t3400-rebase.sh                  |   2 +-
 t/t5516-fetch-push.sh              |   2 +-
 10 files changed, 690 insertions(+), 9 deletions(-)


base-commit: 47fdf2208c5b94c8cc01c106b8f7c85efb91c926
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2317%2Fnevion%2Fworktree-reflink-cow-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2317/nevion/worktree-reflink-cow-v3
Pull-Request: https://github.com/git/git/pull/2317

Range-diff vs v2:

 1:  83841020f4 ! 1:  a163f737af worktree: add --reflink for copy-on-write worktree creation
     @@ Commit message
          and bcachefs implement, and the one overlayfs forwards when the lower
          filesystem can clone. This path has been tested on btrfs.
      
     -    On macOS it uses clonefile(), the APFS primitive. That path is
     -    implemented and has not been run here.
     +    On macOS it uses clonefile(), the APFS primitive. The t2400
     +    REFLINK prerequisite probes that with "cp -c", because BSD cp has no
     +    --reflink=always, so the macOS CI jobs exercise this path on APFS.
      
          On Windows it uses ReFS block cloning,
          FSCTL_DUPLICATE_EXTENTS_TO_FILE, as Git LFS does. The destination is
     @@ t/t2400-worktree-add.sh: test_expect_success 'relative worktree sets extension c
      +# Reflinks are only available on copy-on-write filesystems (btrfs, XFS,
      +# bcachefs, APFS, ...). Where they are unavailable, --reflink transparently
      +# falls back to a regular checkout, which the test above already covers.
     ++#
     ++# GNU cp probes with --reflink=always. macOS cp has no such option; -c
     ++# is clonefile(2), which is what the APFS CI jobs exercise.
      +test_lazy_prereq REFLINK '
      +	echo probe >reflink-src &&
     -+	cp --reflink=always reflink-src reflink-dst 2>/dev/null
     ++	if test_have_prereq MACOS
     ++	then
     ++		cp -c reflink-src reflink-dst 2>/dev/null
     ++	else
     ++		cp --reflink=always reflink-src reflink-dst 2>/dev/null
     ++	fi
      +'
      +
      +test_expect_success REFLINK '--reflink carries untracked files and reconciles changes' '
 2:  6e2acdc5e2 = 2:  042485f0e9 worktree: reap ephemeral branches created by "worktree add"
 -:  ---------- > 3:  91806975a6 worktree: disable reflink on Windows
 -:  ---------- > 4:  876fb541a8 worktree: keep the reflink probe referenced on Windows

-- 
gitgitgadget