[PATCH v13 0/6] fetch: avoid fetching every branch of a new remote in a shallow repo

["Harald Nordgren via GitGitGadget" <gitgitgadget@gmail.com>]

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v13:

 * Added a new commit so @{push} resolves correctly for a --limited-fetch
   remote, falling back to remote.<name>.fetchRefmap the same way fetch
   already does when remote.<name>.fetch doesn't cover the branch being
   pushed.
 * Added a regression test to fetch: add remote.<name>.fetchRefmap covering
   a remote.<name>.fetch entry with an explicit destination, confirming it's
   stored in both that destination and the refmap's.

Changes in v12:

 * Renamed remote.<name>.refmap to remote.<name>.fetchRefmap throughout
   code, docs, and tests, so it reads unambiguously as fetch-only.
 * Added the missing code path in get_ref_map() so a branch fetched by name
   via remote.<name>.fetch is also stored through remote.<name>.fetchRefmap,
   with a new test (the "cabal" example: fetch = master/next/seen plus a
   refmap storing them all under one namespace).
 * Reworded the fetchRefmap commit message and doc entry to drop a redundant
   sentence, fix a confusing paragraph, and spell out that
   remote.<name>.fetch, when configured, still decides what's fetched while
   the refmap only remaps it.
 * Added a prefetch coverage test to t5586-fetch-refmap.sh.

Changes in v11:

 * remote_find_tracking() now falls back to remote.<name>.refmap when
   remote.<name>.fetch doesn't match. Negative refspecs in fetch still
   exclude refs, and both cases have tests
 * The helpers write straight into a strvec, take the output parameter last
   and the remote as const
 * The followRemoteHEAD lookup is done once, not duplicated in two branches
 * Clearer commit messages and docs: a working example plus a reference to
   49ca2fba39, the goal stated first, an incorrect "used to die" claim
   dropped, and an explanation of why a refmap turns on the inference
 * use_limited_fetch is a bool

Changes in v10:

 * Resolve merge conflict after master integrated
   ch/fetch-followremotehead-lazy-validation.

Changes in v9:

 * Moved the do_fetch() if/else-if/else split out of the preliminary
   helper-extraction commit and into the refmap-inference commit that needs
   it. Preliminary commit is now a simpler refactor.

Changes in v8:

 * Split into a preliminary collect_upstream_from_remote() helper commit
   plus a simplified refmap-inference commit.
 * Fixed a regression that could drop the current branch's @{upstream} when
   remote.<name>.fetch was narrowed.
 * Threaded struct repository * through instead of hardcoding
   the_repository.
 * Minor naming and comment cleanups.

Changes in v7:

 * Fixed fatal: --refmap option is only meaningful with command-line
   refspec(s) firing just from configuring remote.<name>.refmap alone, with
   nothing else to fetch.
 * When both remote.<name>.fetch and remote.<name>.refmap are configured, a
   refspec-less fetch follows .fetch as usual, .refmap only remaps something
   already being fetched by name. Added tests.

Changes in v6:

 * Remove leftover reference to deleted default-branch logic in commit
   message.

Changes in v5:

 * Renumber test file from t5585 to t5586.

Changes in v4:

 * Removed the automatic default-branch fetch. A fresh remote fetches
   nothing until you track a branch explicitly.
 * Fixed fetch report showing "new ref HEAD" instead of "new branch "
 * Reworded remote..refmap docs and commit message.

Changes in v3:

 * Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap,
   reusing git's existing --refmap mechanism instead of inventing new
   refspec syntax.
 * Split the change into 4 commits.

Changes in v2:

 * Replaced the opt-in fetch.shallow config entirely with a new special
   fetch refspec (+:) that git remote add now defaults new remotes to in a
   shallow repository. The new refspec fetches whichever branches any local
   branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (6):
  fetch: add remote.<name>.fetchRefmap
  fetch: extract collect_upstream_from_remote() helper
  fetch: infer branches to fetch from a refmap-only remote
  remote: add "git remote add --limited-fetch"
  remote: default to --limited-fetch in a shallow repository
  remote: resolve @{push} through remote.<name>.fetchRefmap

 Documentation/config/remote.adoc |   9 ++
 Documentation/fetch-options.adoc |  10 ++
 Documentation/git-remote.adoc    |  14 ++-
 builtin/fetch.c                  | 100 +++++++++++++++-----
 builtin/remote.c                 |  31 +++++--
 remote.c                         |  62 ++++++++++++-
 remote.h                         |  20 ++++
 t/meson.build                    |   1 +
 t/t1514-rev-parse-push.sh        |  19 ++++
 t/t5505-remote.sh                |  76 ++++++++++++++++
 t/t5510-fetch.sh                 | 151 +++++++++++++++++++++++++++++++
 t/t5586-fetch-refmap.sh          | 133 +++++++++++++++++++++++++++
 12 files changed, 593 insertions(+), 33 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh


base-commit: b9e5cff1803ab24582201ef059516eb16014b9d9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v13
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v13
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v12:

 1:  a1f5e81fdc ! 1:  07081828d1 fetch: add remote.<name>.fetchRefmap
     @@ t/t5510-fetch.sh: test_expect_success 'explicit --refmap option overrides remote
      +	)
      +'
      +
     ++test_expect_success 'remote.<name>.fetchRefmap stores an explicit-dst remote.<name>.fetch entry in both places' '
     ++	test_when_finished "rm -fr fetch-refmap-upstream downstream" &&
     ++	git init -b master fetch-refmap-upstream &&
     ++	test_commit -C fetch-refmap-upstream base &&
     ++	git init downstream &&
     ++	(
     ++		cd downstream &&
     ++		git remote add cabal ../fetch-refmap-upstream &&
     ++		git config --unset-all remote.cabal.fetch &&
     ++		git config --add remote.cabal.fetch \
     ++			"+refs/heads/master:refs/remotes/custom/master" &&
     ++		git config remote.cabal.fetchRefmap "+refs/heads/*:refs/remotes/cabal/*" &&
     ++		git fetch cabal &&
     ++		check_fetched_refs <<-\EOF
     ++		refs/remotes/cabal/HEAD
     ++		refs/remotes/cabal/master
     ++		refs/remotes/custom/master
     ++		EOF
     ++	)
     ++'
     ++
      +test_expect_success 'remote.<name>.fetchRefmap tracks a branch outside remote.<name>.fetch' '
      +	test_when_finished "rm -fr fetch-refmap-upstream downstream" &&
      +	git init -b main fetch-refmap-upstream &&
 2:  e8af5bc76b = 2:  3f5808cbf3 fetch: extract collect_upstream_from_remote() helper
 3:  aad6e5f66f = 3:  b02e0a5397 fetch: infer branches to fetch from a refmap-only remote
 4:  97650da870 = 4:  3ee427de87 remote: add "git remote add --limited-fetch"
 5:  a802c9f82f = 5:  2cba4623ee remote: default to --limited-fetch in a shallow repository
 -:  ---------- > 6:  25eec8d967 remote: resolve @{push} through remote.<name>.fetchRefmap

-- 
gitgitgadget