Remotes order in "git remote update"

Remotes order in "git remote update"

[Samuel Tardieu]

It looks like remote repositories are ordered alphabetically by their
local names when doing a "git remote update". In a project I follow,
it has an unfortunate consequence: a dumb remote repository using
"http://" is fetched first, while many commits are shared between all
the upstreams I follow.

If remotes were fetched using their order in .git/config, it would
fetch from a "git://" remote repository first, limiting transfers from
the dumb remote repository. Is there a way to force the update order
short of changing the local names or issuing multiple fetch commands?

  Sam
-- 
Samuel Tardieu -- sam@rfc1149.net -- http://www.rfc1149.net/

Re: Remotes order in "git remote update"

[Junio C Hamano]

Samuel Tardieu <sam@rfc1149.net> writes:

> It looks like remote repositories are ordered alphabetically by their
> local names when doing a "git remote update".

Yeah, it is an unfortunate consequence of the Perl implementation that
uses a hash to record remote and then try to give a stable ordering by
sorting keys.  The command originally was never intended for working on
more than one remote, but addition of "remote update" could have been done
to follow the order in the configuration.

I think the reimplementaiton in 'next' that will hopefully be in 1.5.5
will keep the list of remotes in the order read from the config.

Re: Remotes order in "git remote update"

[Samuel Tardieu]

On  9/03, Junio C Hamano wrote:

| Samuel Tardieu <sam@rfc1149.net> writes:
| 
| > It looks like remote repositories are ordered alphabetically by their
| > local names when doing a "git remote update".
|
| I think the reimplementaiton in 'next' that will hopefully be in 1.5.5
| will keep the list of remotes in the order read from the config.

Indeed it does, thanks.

Re: Remotes order in "git remote update"

[Johannes Schindelin]

Hi,

On Sun, 9 Mar 2008, Samuel Tardieu wrote:

> On  9/03, Junio C Hamano wrote:
> 
> | Samuel Tardieu <sam@rfc1149.net> writes:
> | 
> | > It looks like remote repositories are ordered alphabetically by their
> | > local names when doing a "git remote update".
> |
> | I think the reimplementaiton in 'next' that will hopefully be in 1.5.5
> | will keep the list of remotes in the order read from the config.
> 
> Indeed it does, thanks.

Well, technically this is a regression.

If you really want to order your remotes, why not add something like

[remotes]
	default = my-first-remote my-second-remote [...]

to the config?  That is what the (recently fixed in builtin-remote) remote 
groups are for...

Ciao,
Dscho

Re: Remotes order in "git remote update"

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Sun, 9 Mar 2008, Samuel Tardieu wrote:
>
>> On  9/03, Junio C Hamano wrote:
>> 
>> | Samuel Tardieu <sam@rfc1149.net> writes:
>> | 
>> | > It looks like remote repositories are ordered alphabetically by their
>> | > local names when doing a "git remote update".
>> |
>> | I think the reimplementaiton in 'next' that will hopefully be in 1.5.5
>> | will keep the list of remotes in the order read from the config.
>> 
>> Indeed it does, thanks.
>
> Well, technically this is a regression.
>
> If you really want to order your remotes, why not add something like
>
> [remotes]
> 	default = my-first-remote my-second-remote [...]
>
> to the config?  That is what the (recently fixed in builtin-remote) remote 
> groups are for...

We never guaranteed there is any order the remotes are processed.  One
order is as good as another, and the only way that can give some guarantee
is with the remote groups.  So there is no regression and the suggestion
to use remote group is one good way.

We _might_ want to start guaranteeing some orders by documenting, but I do
not think it is necessary.  Does "git config" and "git remote" code even
guarantee the order the newly added ones are stored in the $GIT_DIR/config
file, and is it by design and specified in the documentation, or it is by
accident and happens to be the way the code is written?

Re: Remotes order in "git remote update"

[Samuel Tardieu]

On  9/03, Johannes Schindelin wrote:

| Well, technically this is a regression.
| 
| If you really want to order your remotes, why not add something like
| 
| [remotes]
| 	default = my-first-remote my-second-remote [...]
| 
| to the config?  That is what the (recently fixed in builtin-remote) remote 
| groups are for...

I could do that, but it means that if I add a new remote, it won't enter
the default group by itself since I defined it explicitely. I think
respecting the order given in the .git/config file when not using a
group doesn't hurt and may help.

Or maybe we should add a remotes.reference configuration variable which
lists the remote to use first if it belongs to the list of remotes we
are going to fetch from. Or at least, make origin the first fetched
remote, but I know some projects where origin uses a dumb protocol while
other contributors have up-to-date non-dumb public repositories.

Re: Remotes order in "git remote update"

[Johannes Schindelin]

Hi,

On Sun, 9 Mar 2008, Junio C Hamano wrote:

> We _might_ want to start guaranteeing some orders by documenting, but I 
> do not think it is necessary.  Does "git config" and "git remote" code 
> even guarantee the order the newly added ones are stored in the 
> $GIT_DIR/config file, and is it by design and specified in the 
> documentation, or it is by accident and happens to be the way the code 
> is written?

git_config_set() appends to the end of the config if the section it tries 
to edit did not exist yet.  However, this is just an implementation 
detail, no design decision.

Ciao,
Dscho

Re: Remotes order in "git remote update"

[Johannes Schindelin]

Hi,

On Mon, 10 Mar 2008, Samuel Tardieu wrote:

> On  9/03, Johannes Schindelin wrote:
> 
> | Well, technically this is a regression.
> | 
> | If you really want to order your remotes, why not add something like
> | 
> | [remotes]
> | 	default = my-first-remote my-second-remote [...]
> | 
> | to the config?  That is what the (recently fixed in builtin-remote) 
> | remote groups are for...
> 
> I could do that, but it means that if I add a new remote, it won't enter 
> the default group by itself since I defined it explicitely. I think 
> respecting the order given in the .git/config file when not using a 
> group doesn't hurt and may help.

Well, since the builtin remote does that, I do not see a reason to change 
it ;-)  However, I do not see a reason to guarantee that, either.

> Or maybe we should add a remotes.reference configuration variable which 
> lists the remote to use first if it belongs to the list of remotes we 
> are going to fetch from. Or at least, make origin the first fetched 
> remote, but I know some projects where origin uses a dumb protocol while 
> other contributors have up-to-date non-dumb public repositories.

I do not like the remotes.reference idea.  Too specific for one particular 
use case.

Ciao,
Dscho

Re: Remotes order in "git remote update"

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Mon, 10 Mar 2008, Samuel Tardieu wrote:
>
>> On  9/03, Johannes Schindelin wrote:
>> 
>> | Well, technically this is a regression.
>> | 
>> | If you really want to order your remotes, why not add something like
>> | 
>> | [remotes]
>> | 	default = my-first-remote my-second-remote [...]
>> | 
>> | to the config?  That is what the (recently fixed in builtin-remote) 
>> | remote groups are for...
>> 
>> I could do that, but it means that if I add a new remote, it won't enter 
>> the default group by itself since I defined it explicitely. I think 
>> respecting the order given in the .git/config file when not using a 
>> group doesn't hurt and may help.
>
> Well, since the builtin remote does that, I do not see a reason to change 
> it ;-)  However, I do not see a reason to guarantee that, either.

I think it is a reasonable expectation that, if you have these in the
configuration file (not limited to "remotes" but for a random "a"):

	[a "foo"]
        	x = "frotz"
                x = "xyzzy"
	[a "bar"]
        	x = "nitfol"
                x = "rezrov"

something that _iterates_ over a.*.* would see them in the order of
appearance (foo.frotz, foo.xyzzy, bar.nitfol and then bar.rezrov).

If you need both iterable and also quick lookup (e.g. when there can be
thousands of "foo" and "bar" for a particular "a" and it is common to ask
for "a.$name.x" for random $name), the config reader for "a" needs to
have an implementation that is better than just a naive linear list.
Maybe it can use a hashed table whose entry records the appearance order
in the configuration file, expecting that a look-up is far more common
operation than enumeration, and when somebody asks you to enumerate, you
can create a sorted list on-demand and iterate on that.  The
implementation would be different depending on the expected usage pattern,
but I do not see much to gain for us by reserving the right to enumerate
things in a random order.