[PATCH v14 0/5] fetch: avoid fetching every branch of a new remote in a shallow repo

["Harald Nordgren via GitGitGadget" <gitgitgadget@gmail.com>]

In a shallow single-branch clone of a large repository, git remote add
upstream <url> sets up the default +refs/heads/*:refs/remotes/upstream/*
refspec. The next git fetch upstream then downloads every branch of that
remote, which defeats the point of a shallow clone.

This series lets a remote fetch only the branches that local branches track:

git remote add --limited-fetch upstream <url>
git fetch upstream main
git branch --set-upstream-to=upstream/main
git fetch upstream    # fetches only main from now on


 * Patches 1 and 2 add remote.<name>.fetchRefmap, a configured default for
   --refmap, and a small helper refactor.
 * Patch 3 makes a remote with a fetchRefmap and no fetch refspec work on
   its own. A plain git fetch fetches the branches whose @{upstream} is on
   that remote, and @{push} resolves through the refmap.
 * Patch 4 adds git remote add --limited-fetch to set up such a remote.
 * Patch 5 makes --limited-fetch the default for git remote add in a shallow
   repository.

Changes in v14:

 * Squashed the @{push} fix into "fetch: infer branches to fetch from a
   refmap-only remote", so the series no longer adds the bug in one patch
   and fixes it in a later one
 * Moved its test to t5586 using the refmap-only remote, so it no longer
   depends on --limited-fetch
 * Dropped the separate "remote: resolve @{push} through
   remote..fetchRefmap" commit

Changes in v13:

 * Added a new commit so @{push} resolves correctly for a --limited-fetch
   remote, falling back to remote.<name>.fetchRefmap the same way fetch
   already does when remote.<name>.fetch doesn't cover the branch being
   pushed.
 * Added a regression test to fetch: add remote.<name>.fetchRefmap covering
   a remote.<name>.fetch entry with an explicit destination, confirming it's
   stored in both that destination and the refmap's.

Changes in v12:

 * Renamed remote.<name>.refmap to remote.<name>.fetchRefmap throughout
   code, docs, and tests, so it reads unambiguously as fetch-only.
 * Added the missing code path in get_ref_map() so a branch fetched by name
   via remote.<name>.fetch is also stored through remote.<name>.fetchRefmap,
   with a new test (the "cabal" example: fetch = master/next/seen plus a
   refmap storing them all under one namespace).
 * Reworded the fetchRefmap commit message and doc entry to drop a redundant
   sentence, fix a confusing paragraph, and spell out that
   remote.<name>.fetch, when configured, still decides what's fetched while
   the refmap only remaps it.
 * Added a prefetch coverage test to t5586-fetch-refmap.sh.

Changes in v11:

 * remote_find_tracking() now falls back to remote.<name>.refmap when
   remote.<name>.fetch doesn't match. Negative refspecs in fetch still
   exclude refs, and both cases have tests
 * The helpers write straight into a strvec, take the output parameter last
   and the remote as const
 * The followRemoteHEAD lookup is done once, not duplicated in two branches
 * Clearer commit messages and docs: a working example plus a reference to
   49ca2fba39, the goal stated first, an incorrect "used to die" claim
   dropped, and an explanation of why a refmap turns on the inference
 * use_limited_fetch is a bool

Changes in v10:

 * Resolve merge conflict after master integrated
   ch/fetch-followremotehead-lazy-validation.

Changes in v9:

 * Moved the do_fetch() if/else-if/else split out of the preliminary
   helper-extraction commit and into the refmap-inference commit that needs
   it. Preliminary commit is now a simpler refactor.

Changes in v8:

 * Split into a preliminary collect_upstream_from_remote() helper commit
   plus a simplified refmap-inference commit.
 * Fixed a regression that could drop the current branch's @{upstream} when
   remote.<name>.fetch was narrowed.
 * Threaded struct repository * through instead of hardcoding
   the_repository.
 * Minor naming and comment cleanups.

Changes in v7:

 * Fixed fatal: --refmap option is only meaningful with command-line
   refspec(s) firing just from configuring remote.<name>.refmap alone, with
   nothing else to fetch.
 * When both remote.<name>.fetch and remote.<name>.refmap are configured, a
   refspec-less fetch follows .fetch as usual, .refmap only remaps something
   already being fetched by name. Added tests.

Changes in v6:

 * Remove leftover reference to deleted default-branch logic in commit
   message.

Changes in v5:

 * Renumber test file from t5585 to t5586.

Changes in v4:

 * Removed the automatic default-branch fetch. A fresh remote fetches
   nothing until you track a branch explicitly.
 * Fixed fetch report showing "new ref HEAD" instead of "new branch "
 * Reworded remote..refmap docs and commit message.

Changes in v3:

 * Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap,
   reusing git's existing --refmap mechanism instead of inventing new
   refspec syntax.
 * Split the change into 4 commits.

Changes in v2:

 * Replaced the opt-in fetch.shallow config entirely with a new special
   fetch refspec (+:) that git remote add now defaults new remotes to in a
   shallow repository. The new refspec fetches whichever branches any local
   branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (5):
  fetch: add remote.<name>.fetchRefmap
  fetch: extract collect_upstream_from_remote() helper
  fetch: infer branches to fetch from a refmap-only remote
  remote: add "git remote add --limited-fetch"
  remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   9 ++
 Documentation/fetch-options.adoc |  10 ++
 Documentation/git-remote.adoc    |  14 ++-
 builtin/fetch.c                  | 100 +++++++++++++++-----
 builtin/remote.c                 |  31 +++++--
 remote.c                         |  62 ++++++++++++-
 remote.h                         |  20 ++++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++
 t/t5510-fetch.sh                 | 151 +++++++++++++++++++++++++++++++
 t/t5586-fetch-refmap.sh          | 147 ++++++++++++++++++++++++++++++
 11 files changed, 588 insertions(+), 33 deletions(-)
 create mode 100755 t/t5586-fetch-refmap.sh


base-commit: 7fb6055e678f800320d8ab8cd8f201d9493d1ba0
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v14
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v14
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v13:

 1:  07081828d1 = 1:  c63699b402 fetch: add remote.<name>.fetchRefmap
 2:  3f5808cbf3 = 2:  4ad94eefe9 fetch: extract collect_upstream_from_remote() helper
 3:  b02e0a5397 ! 3:  6724152e69 fetch: infer branches to fetch from a refmap-only remote
     @@ Commit message
          already on that remote. The refmap itself still only controls where
          those fetched refs land, not which branches are selected.
      
     +    Let @{push} use the refmap too, so a branch pushed to such a remote
     +    resolves to the remote-tracking branch the push updated, instead of
     +    failing with "has no local tracking branch".
     +
          Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
      
       ## Documentation/config/remote.adoc ##
     @@ remote.c: void collect_upstream_from_remote(struct repository *repo,
       __attribute__((format (printf,2,3)))
       static char *error_buf(struct strbuf *err, const char *fmt, ...)
       {
     +@@ remote.c: static char *tracking_for_push_dest(struct repository *repo,
     + 
     + 	remote = repo_remote_for_push_tracking(repo, remote);
     + 	ret = apply_refspecs(&remote->fetch, refname);
     ++	if (!ret)
     ++		ret = apply_refspecs(&remote->refmap, refname);
     + 	if (!ret)
     + 		return error_buf(err,
     + 				 _("push destination '%s' on remote '%s' has no local tracking branch"),
      
       ## remote.h ##
      @@ remote.h: void collect_upstream_from_remote(struct repository *repo,
     @@ t/t5586-fetch-refmap.sh (new)
      +	)
      +'
      +
     ++test_expect_success '@{push} resolves for a branch pushed to a refmap-only remote' '
     ++	(
     ++		cd client &&
     ++		git config push.default current &&
     ++		git checkout -b topic upstream/main &&
     ++		test_commit topic-1 &&
     ++		git push &&
     ++		echo refs/remotes/upstream/topic >expect &&
     ++		git rev-parse --symbolic-full-name topic@{push} >actual &&
     ++		test_cmp expect actual &&
     ++		test_cmp_rev topic topic@{push}
     ++	)
     ++'
     ++
      +test_done
 4:  3ee427de87 = 4:  6cff6786ba remote: add "git remote add --limited-fetch"
 5:  2cba4623ee = 5:  00f9d44a88 remote: default to --limited-fetch in a shallow repository
 6:  25eec8d967 < -:  ---------- remote: resolve @{push} through remote.<name>.fetchRefmap

-- 
gitgitgadget