[PATCH v4 0/4] fetch: avoid fetching every branch of a new remote in a shallow repo

["Harald Nordgren via GitGitGadget" <gitgitgadget@gmail.com>]

Avoid fetching every branch of a new remote in a shallow repo.

Changes in v4:

 * Removed the automatic default-branch fetch. A fresh remote fetches
   nothing until you track a branch explicitly.
 * Fixed fetch report showing "new ref HEAD" instead of "new branch "
 * Reworded remote..refmap docs and commit message.

Changes in v3:

 * Replace the special ":"/"+:" fetch refspec with remote.<name>.refmap,
   reusing git's existing --refmap mechanism instead of inventing new
   refspec syntax.
 * Split the change into 4 commits.

Changes in v2:

 * Replaced the opt-in fetch.shallow config entirely with a new special
   fetch refspec (+:) that git remote add now defaults new remotes to in a
   shallow repository. The new refspec fetches whichever branches any local
   branch tracks at that remote, plus the remote's default branch.

Harald Nordgren (4):
  fetch: add remote.<name>.refmap
  fetch: infer branches to fetch from a refmap-only remote
  remote: add "git remote add --limited-fetch"
  remote: default to --limited-fetch in a shallow repository

 Documentation/config/remote.adoc |   8 +++
 Documentation/fetch-options.adoc |   5 ++
 Documentation/git-remote.adoc    |  14 +++-
 builtin/fetch.c                  |  60 +++++++++++++---
 builtin/remote.c                 |  31 ++++++--
 remote.c                         |  41 ++++++++++-
 remote.h                         |   9 +++
 t/meson.build                    |   1 +
 t/t5505-remote.sh                |  76 ++++++++++++++++++++
 t/t5510-fetch.sh                 |  17 +++++
 t/t5585-fetch-refmap.sh          | 119 +++++++++++++++++++++++++++++++
 11 files changed, 363 insertions(+), 18 deletions(-)
 create mode 100755 t/t5585-fetch-refmap.sh


base-commit: a018953688f1b10bddf91bff8747068f5f4746a4
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-2412%2FHaraldNordgren%2Ffetch-shallow-narrow-refspec-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-2412/HaraldNordgren/fetch-shallow-narrow-refspec-v4
Pull-Request: https://github.com/git/git/pull/2412

Range-diff vs v3:

 1:  b04c00b974 ! 1:  d48a7004e4 fetch: add remote.<name>.refmap
     @@ Commit message
      
          Add a per-remote config variable, remote.<name>.refmap, that provides
          the default value for --refmap the same way remote.<name>.fetch
     -    already provides the default refspecs to fetch. It only takes effect
     -    when there is something explicit to fetch, on the command line or via
     -    remote.<name>.fetch, matching how --refmap itself already behaves.
     +    already provides the default refspecs to fetch. Like --refmap itself,
     +    it only maps refs that are actually being fetched, so it has nothing
     +    to do when there is nothing explicit to fetch, on the command line or
     +    via remote.<name>.fetch.
      
          Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
      
     @@ Documentation/config/remote.adoc: remote.<name>.fetch::
       
      +remote.<name>.refmap::
      +	The default value of the `--refmap` option for linkgit:git-fetch[1].
     -+	Only takes effect when the fetch names what to fetch explicitly,
     -+	either on the command line or via `remote.<name>.fetch`. See the
     -+	`--refmap` entry in linkgit:git-fetch[1].
     ++	Used to map remote refs being fetched to remote-tracking refs to
     ++	store. See the `--refmap` entry in linkgit:git-fetch[1].
      +
       remote.<name>.push::
       	The default set of "refspec" for linkgit:git-push[1]. See
 2:  4ec508a223 ! 2:  45b26e2bb2 fetch: infer branches to fetch from a refmap-only remote
     @@ Commit message
          says where to put fetched refs, not what to fetch.
      
          Make that case infer what to fetch: the local branches whose
     -    @{upstream} is already on that remote, plus the remote's default
     -    branch, which is always included so it is available even before
     -    anything is set up to track it. This lets a remote be configured to
     -    fetch only the branches actually in use, without listing them by
     -    hand in remote.<name>.fetch, and without needing to touch the
     -    command line every time.
     +    @{upstream} is already on that remote. This lets a remote be
     +    configured to fetch only the branches actually in use, without
     +    listing them by hand in remote.<name>.fetch, and without needing to
     +    touch the command line every time.
      
          Signed-off-by: Harald Nordgren <haraldnordgren@gmail.com>
      
       ## Documentation/config/remote.adoc ##
      @@ Documentation/config/remote.adoc: remote.<name>.fetch::
     - 
       remote.<name>.refmap::
       	The default value of the `--refmap` option for linkgit:git-fetch[1].
     --	Only takes effect when the fetch names what to fetch explicitly,
     --	either on the command line or via `remote.<name>.fetch`. See the
     --	`--refmap` entry in linkgit:git-fetch[1].
     -+	If `remote.<name>.fetch` is not set either, a refspec-less fetch
     -+	infers what to fetch from local branches built on this remote,
     -+	instead of fetching every branch it has. See the `--refmap` entry
     -+	in linkgit:git-fetch[1].
     + 	Used to map remote refs being fetched to remote-tracking refs to
     +-	store. See the `--refmap` entry in linkgit:git-fetch[1].
     ++	store. If `remote.<name>.fetch` is not set either, a refspec-less
     ++	fetch infers what to fetch from local branches built on this
     ++	remote, instead of fetching every branch it has. See the
     ++	`--refmap` entry in linkgit:git-fetch[1].
       
       remote.<name>.push::
       	The default set of "refspec" for linkgit:git-push[1]. See
     @@ Documentation/fetch-options.adoc: endif::git-pull[]
      +When a refmap is active (from `--refmap` or `remote.<name>.refmap`) but
      +there is nothing to fetch, neither on the command line nor from
      +`remote.<name>.fetch`, Git infers what to fetch from the local branches
     -+whose `@{upstream}` is on that remote, plus the remote's default branch,
     -+which is always included so that it is available even before anything
     -+is set up to track it.
     ++whose `@{upstream}` is on that remote.
       
       `-t`::
       `--tags`::
      
       ## builtin/fetch.c ##
     -@@ builtin/fetch.c: static void filter_prefetch_refspec(struct refspec *rs)
     - static struct ref *get_ref_map(struct remote *remote,
     - 			       const struct ref *remote_refs,
     - 			       struct refspec *rs,
     --			       int tags, int *autotags)
     -+			       int tags, int *autotags,
     -+			       char **bootstrap_head_branch)
     - {
     - 	int i;
     - 	struct ref *rm;
      @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       	struct ref **tail = &ref_map;
       	struct refspec *effective_refmap =
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
      +	    effective_refmap && effective_refmap->nr) {
      +		struct string_list tracked = STRING_LIST_INIT_DUP;
      +		struct string_list_item *item;
     -+		struct ref *head;
     -+		char *default_branch = NULL;
     -+		char *default_branch_dst = NULL;
      +
      +		branches_tracking_remote(remote, &tracked);
      +		for_each_string_list_item(item, &tracked)
      +			refspec_append(&inferred_rs, item->string);
     -+
     -+		/* Always fetch the default branch too, as "HEAD". */
     -+		head = get_remote_ref(remote_refs, "HEAD");
     -+		if (head && head->symref && *head->symref)
     -+			default_branch = xstrdup(head->symref);
     -+		free_one_ref(head);
     -+
     -+		for (i = 0; default_branch && !default_branch_dst &&
     -+			    !string_list_has_string(&tracked, default_branch) &&
     -+			    i < effective_refmap->nr; i++) {
     -+			struct refspec_item *map = &effective_refmap->items[i];
     -+
     -+			if (map->pattern)
     -+				match_refname_with_pattern(map->src, default_branch,
     -+							    map->dst, &default_branch_dst);
     -+			else if (!strcmp(map->src, default_branch))
     -+				default_branch_dst = xstrdup(map->dst);
     -+		}
     -+
     -+		if (default_branch_dst) {
     -+			struct refspec_item head_item = { .force = 1 };
     -+
     -+			head_item.src = xstrdup("HEAD");
     -+			head_item.dst = default_branch_dst;
     -+			get_fetch_map(remote_refs, &head_item, &tail, 1);
     -+			free(head_item.src);
     -+		}
     -+
     -+		if (default_branch && bootstrap_head_branch) {
     -+			const char *branch_name = default_branch;
     -+			skip_prefix(branch_name, "refs/heads/", &branch_name);
     -+			*bootstrap_head_branch = xstrdup(branch_name);
     -+		}
     -+
     -+		free(default_branch);
     -+		free(default_branch_dst);
      +		string_list_clear(&tracked, 0);
      +
      +		rs = &inferred_rs;
     @@ builtin/fetch.c: static struct ref *get_ref_map(struct remote *remote,
       	return ref_map;
       }
       
     -@@ builtin/fetch.c: static void warn_set_head(const char *remote, const char *head_name,
     - }
     - 
     - static int set_head(const struct ref *remote_refs, struct remote *remote,
     --			int follow_remote_head)
     -+			int follow_remote_head, const char *known_head_branch)
     - {
     - 	int result = 0, create_only, baremirror, was_detached;
     - 	struct strbuf b_head = STRBUF_INIT, b_remote_head = STRBUF_INIT,
     - 		      b_local_head = STRBUF_INIT;
     - 	const char *no_warn_branch = remote->no_warn_branch;
     - 	char *head_name = NULL;
     --	struct ref *ref, *matches;
     -+	struct ref *ref, *matches = NULL;
     - 	struct ref *fetch_map = NULL, **fetch_map_tail = &fetch_map;
     - 	struct refspec_item refspec = {
     - 		.force = 0,
     -@@ builtin/fetch.c: static int set_head(const struct ref *remote_refs, struct remote *remote,
     - 	struct string_list heads = STRING_LIST_INIT_DUP;
     - 	struct ref_store *refs = get_main_ref_store(the_repository);
     - 
     --	get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
     --	matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
     --				    fetch_map, REMOTE_GUESS_HEAD_ALL);
     --	for (ref = matches; ref; ref = ref->next) {
     --		string_list_append(&heads, strip_refshead(ref->name));
     --	}
     -+	if (known_head_branch) {
     -+		head_name = xstrdup(known_head_branch);
     -+	} else {
     -+		get_fetch_map(remote_refs, &refspec, &fetch_map_tail, 0);
     -+		matches = guess_remote_head(find_ref_by_name(remote_refs, "HEAD"),
     -+					    fetch_map, REMOTE_GUESS_HEAD_ALL);
     -+		for (ref = matches; ref; ref = ref->next) {
     -+			string_list_append(&heads, strip_refshead(ref->name));
     -+		}
     - 
     --	if (!heads.nr)
     --		result = 1;
     --	else if (heads.nr > 1)
     --		result = 1;
     --	else
     --		head_name = xstrdup(heads.items[0].string);
     -+		if (!heads.nr)
     -+			result = 1;
     -+		else if (heads.nr > 1)
     -+			result = 1;
     -+		else
     -+			head_name = xstrdup(heads.items[0].string);
     -+	}
     - 
     - 	if (!head_name)
     - 		goto cleanup;
     -@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 	struct strmap rejected_refs = STRMAP_INIT;
     - 	int summary_width = 0;
     - 	int follow_remote_head;
     -+	char *bootstrap_head_branch = NULL;
     - 
     - 	if (tags == TAGS_DEFAULT) {
     - 		if (transport->remote->fetch_tags == 2)
      @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       		refspec_ref_prefixes(rs, &transport_ls_refs_options.ref_prefixes);
       	} else {
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
      +				strvec_push(&transport_ls_refs_options.ref_prefixes,
      +					    item->string);
      +			string_list_clear(&tracked, 0);
     -+			strvec_push(&transport_ls_refs_options.ref_prefixes, "HEAD");
      +		} else if (transport->remote->fetch.nr) {
       			refspec_ref_prefixes(&transport->remote->fetch,
       					     &transport_ls_refs_options.ref_prefixes);
     @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
       			for (i = 0; i < branch->merge_nr; i++) {
       				strvec_push(&transport_ls_refs_options.ref_prefixes,
      @@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 	transport_ls_refs_options_release(&transport_ls_refs_options);
       
       	ref_map = get_ref_map(transport->remote, remote_refs, rs,
     --			      tags, &autotags);
     -+			      tags, &autotags, &bootstrap_head_branch);
     + 			      tags, &autotags);
      +
       	if (!update_head_ok)
       		check_not_current_branch(ref_map);
       
     -@@ builtin/fetch.c: static int do_fetch(struct transport *transport,
     - 		 * Way too many cases where this can go wrong so let's just
     - 		 * ignore errors and fail silently for now.
     - 		 */
     --		set_head(remote_refs, transport->remote, follow_remote_head);
     -+		set_head(remote_refs, transport->remote, follow_remote_head,
     -+			 bootstrap_head_branch);
     - 	}
     - 
     - cleanup:
     -+	free(bootstrap_head_branch);
     - 	/*
     - 	 * When using batched updates, we want to commit the non-rejected
     - 	 * updates and also handle the rejections.
      
       ## remote.c ##
      @@ remote.c: int branch_merge_matches(struct branch *branch,
     @@ t/t5585-fetch-refmap.sh (new)
      +
      +When a remote has a refmap configured but no fetch refspec, a
      +refspec-less fetch infers what to fetch from the local branches whose
     -+@{upstream} is on that remote, plus the default branch of that remote,
     -+which is always included so that it is available even before anything
     -+is set up to track it.
     ++@{upstream} is on that remote.
      +'
      +
      +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
     @@ t/t5585-fetch-refmap.sh (new)
      +	)
      +'
      +
     -+test_expect_success 'fetching the new remote does not need every branch it has' '
     ++test_expect_success 'a bare fetch needs nothing until a branch is tracked' '
      +	(
      +		cd client &&
      +		git fetch upstream &&
      +		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
     -+		cat >expect <<-\EOF &&
     -+		refs/remotes/upstream/HEAD
     -+		refs/remotes/upstream/main
     -+		EOF
     -+		test_cmp expect actual
     ++		test_must_be_empty actual
      +	)
      +'
      +
     -+test_expect_success 'set-upstream-to now resolves right after that first fetch' '
     ++test_expect_success 'an explicit one-time fetch lets a branch be tracked' '
      +	(
      +		cd client &&
     -+		git branch --set-upstream-to=upstream &&
     ++		git fetch upstream main &&
     ++		git branch --set-upstream-to=upstream/main &&
      +		test_cmp_config upstream branch.main.remote &&
      +		test_cmp_config refs/heads/main branch.main.merge
      +	)
 3:  604e584956 ! 3:  e2f072c254 remote: add "git remote add --limited-fetch"
     @@ Documentation/git-remote.adoc: the `refs/remotes/<name>/` namespace, a refspec t
      +With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
      +that tracks all branches, `remote.<name>.refmap` is set up so that a
      +refspec-less `git fetch <name>` only fetches branches our local branches
     -+are built on, plus the remote's default branch. See the `--refmap` entry
     -+in linkgit:git-fetch[1] for details.
     ++are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
     ++details.
      ++
       With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
       up to point at remote's _<master>_ branch. See also the set-head command.
 4:  31462e9445 ! 4:  43b9711a2c remote: default to --limited-fetch in a shallow repository
     @@ Documentation/git-remote.adoc: Add a remote named _<name>_ for the repository at
      @@ Documentation/git-remote.adoc: With `--limited-fetch` option, instead of a `remote.<name>.fetch` refspec
       that tracks all branches, `remote.<name>.refmap` is set up so that a
       refspec-less `git fetch <name>` only fetches branches our local branches
     - are built on, plus the remote's default branch. See the `--refmap` entry
     --in linkgit:git-fetch[1] for details.
     -+in linkgit:git-fetch[1] for details. `--no-limited-fetch` explicitly
     -+disables this, overriding the shallow-repository default described above.
     + are built on. See the `--refmap` entry in linkgit:git-fetch[1] for
     +-details.
     ++details. `--no-limited-fetch` explicitly disables this, overriding the
     ++shallow-repository default described above.
       +
       With `-m <master>` option, a symbolic-ref `refs/remotes/<name>/HEAD` is set
       up to point at remote's _<master>_ branch. See also the set-head command.
     @@ t/t5505-remote.sh: test_expect_success 'filters are listed by git remote -v only
      +		test_cmp_config "+refs/heads/*:refs/remotes/upstream/*" \
      +			remote.upstream.refmap &&
      +		test_must_fail git config get remote.upstream.fetch &&
     ++		git fetch upstream main &&
     ++		git branch --set-upstream-to=upstream/main &&
     ++		test_cmp_config upstream branch.main.remote &&
     ++		test_cmp_config refs/heads/main branch.main.merge &&
      +		git fetch upstream &&
      +		git for-each-ref --format="%(refname)" refs/remotes/upstream >actual &&
      +		cat >expect <<-\EOF &&
      +		refs/remotes/upstream/HEAD
      +		refs/remotes/upstream/main
      +		EOF
     -+		test_cmp expect actual &&
     -+		git branch --set-upstream-to=upstream &&
     -+		test_cmp_config upstream branch.main.remote &&
     -+		test_cmp_config refs/heads/main branch.main.merge
     ++		test_cmp expect actual
      +	)
      +'
      +

-- 
gitgitgadget