Re: [PATCH v2] fetch: avoid fetching every branch of a new remote in a shallow repo

[Junio C Hamano <gitster@pobox.com>]

"Harald Nordgren via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Harald Nordgren <haraldnordgren@gmail.com>
>
> git remote add sets a new remote up to fetch every branch by default.
> In an already shallow repository, that turns the next plain fetch or
> pull into a slow or hanging one, even when only one or two branches
> are ever used.
>
> Add a special fetch refspec, "+:", that fetches only the branches our
> local branches are built on, plus the remote's default branch so a
> brand new remote is usable right away, without needing to first set
> anything up to track it. git remote add now uses it instead of the
> usual wildcard refspec whenever the repository is already shallow.

That's way too much for a single patch.  It needs to be split into
digestible chunks, but I offhand do not know how many pieces are
appropriate, so let's think aloud together and try to refine the
design while we do so.

The outline of our design so far should give something like this in
our configuration file.

	[remote "second"]
		fetch = :+

	[branch "topic1"]
		remote = second
		merge = refs/heads/main

	[branch "topic2"]
		remote = second
		merge = refs/heads/next

In the "fetch only what we build on" mode, we would collect local
branches $X where branch.$X.remote == second and then collect
branch.$X.merge for these local branches.  In this case, we would
decide to fetch 'main' and 'next' branches in the end.

But notice that this does not give us sufficient information.  There
is no explicit clue that tells that the remote-tracking branches for
this remote 'second' should be stored under refs/remotes/second/
hierarchy.  A normal remote that is defined like so:

	[remote "origin"]
		fetch = +refs/heads/*:refs/remotes/origin/*

does not have such a problem, as it makes it crystal clear that
their branches go under refs/remotes/origin/ hierarchy.

So using "fetch = :+" is *not* a good idea, as I said.  Let's scrap
that syntax.

One thing we could do is probably to introduce

	[remote "second"]
		refmap = +refs/heads/*:refs/remotes/second/*

instead to give this clue (see "git fetch --help" for what a refmap
is; it looks similar to refspec but only defines how their refs are
mapped to our namespace without specifying what to be fetched, which
is exactly what we need here).  We do not use remote.second.fetch at
all.

It would be an easy first step to teach that an explicit

	$ git fetch second main next

with such a remote.second.refmap should behave the same way as

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second \
		main next

in a repository without the refmote.second.refmap configuration.  As
"git fetch --refmap=... second main next" should already work, it
would be only the matter of supplementing the command line argument
with configured default.

Then teach "git fetch" to further treat

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second

i.e., fetch with refmap but without specifying what exactly to
fetch, as a request to fetch their branches we build on (and nothing
else), using the refmap, in other words, the lack of "what to fetch"
in the above command line signals "git fetch" to rewrite the above to

	$ git fetch --refmap='+refs/heads/*:refs/remotes/second/*' second \
		main next

internally.  Since we have the previous remote.X.refmap step already,
it means that with remote.second.refmap configured properly, the
user can only say

	$ git fetch second

and it would do the right thing in our scenario.

Another and final step would be to teach "git remote add" to add

        [remote "second"]
                refmap = +refs/heads/*:refs/remotes/second/*

when you want to fetch only what you build on.  I am not sure what
should trigger the decision.  Your initial message said something
about shallow and sparse and an earlier review refuted one of them
(I do not recall which offhand, but probably sparse).  It probably
is a good idea to start with an explicit command line option to "git
remote add --limited-fetch" in a single commit.

And then add heuristics (like "in a shallow clone, this mode is
turned on by default, but an explicit '--no-limited-fetch' can
countermand it") in another commit.

So far, we identified four distinct commits, each bite sized.

 - remote.X.refmap configuration acts as if --refmap=... command
   line argument was passed.

 - passing refmap without saying what to fetch enumerates what their
   branches we build on, and pretend as if the user listed these
   branches on the command line to fetch.

 - "git remote add --limited-fetch" creates remote.X.refmap instead
   of remote.X.fetch as necessary.

 - "git remote add" without explicit "--[no-]limited-fetch" uses
   heuristics to enable it.

Or something like that, perhaps?