[RFD] making separate-remote layout easier to use

[RFD] making separate-remote layout easier to use

[Junio C Hamano]

Now separate-remote layout is the default for newly cloned
repositories, I think it is a good time to make further effort
to make things easier to use.  Here are some of the ideas off
the top of my head.

* Do not change what branch the sole local branch is named.

In the current implementation, we find out what branch the
remote HEAD points at, and match the name of the sole local
branch to it.  HEAD of most central distribution point
repositories that use git with the default configuration have
HEAD pointing at refs/heads/master and a cloned repository would
continue to use refs/heads/master as the sole local branch when
it starts as before, but some projects may use alternate
wordings such as 'main', and a new repository would have
refs/heads/main branch when cloned from such a repository
instead.  I think this behaviour is a reasonable default, so
let's keep this.

Distribution point repositories are of interest here, because
that is what most people clone from, and this message is about
making the default easier to use by those casual cloners.

* Change the default contents of $GIT_DIR/remotes/origin

Currently we list all the remote branches that exist when the
clone is made, making sure that the branch pointed at by HEAD at
the remote is listed first.  The intent is to track every branch
at the remote but merge in the "primary" branch to us.

Two issues that have been raised about this is:

 - the intent to track every branch is good, but the current
   implementation means we would not notice new branches at the
   remote.

 - merging the "primary" branch is good only when the user is on
   the corresponding "primary" branch.  It is usually a wrong
   thing to do when on another branch.

The first issue can be solved, with the help of recent "glob
pattern refspec in fetch" from Andy.  I am thinking about making
the default contents of $GIT_DIR/remotes/origin to be:

	URL: <url of the cloned repository>
	Pull: +refs/heads/<primary>:refs/remotes/<origin>/<primary>
	Pull: +refs/heads/*:refs/remotes/<origin>/*

to address this issue.

	Side note: <primary> is what HEAD pointed at at the
        remote when the clone was made, and <origin> is usually
        'origin' but "git clone -o $origin" can override it.

Forcing with '+' is debatable, but with separate-remote layout,
remotes/*/ hierarchy is to track what the remote has, and you
cannot do much else other than noticing and warning when the
remote end does funny things to its refs anyway, so I think
having '+' might be a better default.

The right thing to do to address the second issue is less clear.
If the "upstream" has two more-or-less equally prominent
branches, say 'main' and 'test', it may make sense to use
corresponding two branches on the local side and merge 'main'
from the remote when on local 'main' and merge 'test' when on
local 'test'.  Even when dealing with a specific topic branch,
that would hold true for most of the time.  A topic branch
refs/heads/bug#2073 to work on the bug 2073 can be published at
the central distribution point.  The proposed updates to
$GIT_DIR/remotes/origin file would track it with
refs/remotes/origin/bug#2073 and interested people can create a
local branch refs/heads/bug#2073 from it and work on it, which
makes it easy to polish a topic branch in a collaborative way.

I am not sure if 'merge in corresponding branch' is the only
valid workflow, however. I am reluctant to make the system
automatically do so if the solution makes other workflows more
painful to follow.  Automatically merging remotes/origin/$foo
when on $foo branch is not good enough, in other words (also,
there may be a hierarchy under remotes/ other than origin).  It
might make sense to introduce "Merge: " in remotes/ file and if
they are present use "Pull: " only to decide what are fetched
and use "Merge: " to decide what is merged (if we were doing the
system from scratch, the former would have been named "Fetch: "
but it is too late now).

Re: [RFD] making separate-remote layout easier to use

[Jakub Narebski]

Junio C Hamano wrote:

> Now separate-remote layout is the default for newly cloned
> repositories, I think it is a good time to make further effort
> to make things easier to use.  Here are some of the ideas off
> the top of my head.
[...]
> * Change the default contents of $GIT_DIR/remotes/origin

The question is: do we continue to use remotes/ file, or do we
save remotes info in the config file: remote.<name>.url,
remote.<name>.fetch, remote.<name>.push and branch.<name>.merge
(in our case '[remote "origin"]' section)?
 
> Currently we list all the remote branches that exist when the
> clone is made, making sure that the branch pointed at by HEAD at
> the remote is listed first.  The intent is to track every branch
> at the remote but merge in the "primary" branch to us.
> 
> Two issues that have been raised about this is:
> 
>  - the intent to track every branch is good, but the current
>    implementation means we would not notice new branches at the
>    remote.
> 
>  - merging the "primary" branch is good only when the user is on
>    the corresponding "primary" branch.  It is usually a wrong
>    thing to do when on another branch.
> 
> The first issue can be solved, with the help of recent "glob
> pattern refspec in fetch" from Andy.  I am thinking about making
> the default contents of $GIT_DIR/remotes/origin to be:
> 
>       URL: <url of the cloned repository>
>       Pull: +refs/heads/<primary>:refs/remotes/<origin>/<primary>
>       Pull: +refs/heads/*:refs/remotes/<origin>/*
> 
> to address this issue.

I hope that it also works with the remote.origin config file
section instead of $GIT_DIR/remotes/origin

>       Side note: <primary> is what HEAD pointed at at the
>         remote when the clone was made, and <origin> is usually
>         'origin' but "git clone -o $origin" can override it.
> 
> Forcing with '+' is debatable, but with separate-remote layout,
> remotes/*/ hierarchy is to track what the remote has, and you
> cannot do much else other than noticing and warning when the
> remote end does funny things to its refs anyway, so I think
> having '+' might be a better default.

Perhaps, perhaps not. It would be nice to have configuration option
that would tell that history of given branch is being changed, and
the ability to ask about it remotely, so git-clone would be able
to add this + _when needed_ automatically.

But it's a fact that with separate remote the need to use fast-forward
check is lessened, and it might be more important to not confuse first
time user with having to modify $GIT_DIR/remotes/origin or remote.origin
config section to fetch from the repository he/she cloned from.

> The right thing to do to address the second issue is less clear.
> If the "upstream" has two more-or-less equally prominent
> branches, say 'main' and 'test', it may make sense to use
> corresponding two branches on the local side and merge 'main'
> from the remote when on local 'main' and merge 'test' when on
> local 'test'.  Even when dealing with a specific topic branch,
> that would hold true for most of the time.  A topic branch
> refs/heads/bug#2073 to work on the bug 2073 can be published at
> the central distribution point.  The proposed updates to
> $GIT_DIR/remotes/origin file would track it with
> refs/remotes/origin/bug#2073 and interested people can create a
> local branch refs/heads/bug#2073 from it and work on it, which
> makes it easy to polish a topic branch in a collaborative way.
> 
> I am not sure if 'merge in corresponding branch' is the only
> valid workflow, however. I am reluctant to make the system
> automatically do so if the solution makes other workflows more
> painful to follow.  Automatically merging remotes/origin/$foo
> when on $foo branch is not good enough, in other words (also,
> there may be a hierarchy under remotes/ other than origin).  It
> might make sense to introduce "Merge: " in remotes/ file and if
> they are present use "Pull: " only to decide what are fetched
> and use "Merge: " to decide what is merged (if we were doing the
> system from scratch, the former would have been named "Fetch: "
> but it is too late now).

If you add "Merge: " in remotes/, then please add it also in
remote section in config file. Config file has now 
branch.<branchname>.merge (and it would be nice if clone would
set ou this for local branches corresponding to remote branches),
but it is not the same.

-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> Junio C Hamano wrote:
>
>> Now separate-remote layout is the default for newly cloned
>> repositories, I think it is a good time to make further effort
>> to make things easier to use.  Here are some of the ideas off
>> the top of my head.
> [...]
>> * Change the default contents of $GIT_DIR/remotes/origin
>
> The question is: do we continue to use remotes/ file, or do we
> save remotes info in the config file: remote.<name>.url,
> remote.<name>.fetch, remote.<name>.push and branch.<name>.merge
> (in our case '[remote "origin"]' section)?

It is not "the question"; it is irrelevant because
$GIT_DIR/remotes/origin and [remote "origin"] are pretty much
interchangeable, and will hopefully continue to be.

        [remote "origin"]
                url = master.kernel.org:/pub/scm/git/git.git/
                fetch = +refs/heads/master:refs/remotes/origin/master
                fetch = +refs/heads/*:refs/remotes/origin/*

would be the way to express the equilvalent of the example in
the previous message.

Re: [RFD] making separate-remote layout easier to use

[Jakub Narebski]

Junio C Hamano wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
> 
>> Junio C Hamano wrote:
>>
>>> Now separate-remote layout is the default for newly cloned
>>> repositories, I think it is a good time to make further effort
>>> to make things easier to use.  Here are some of the ideas off
>>> the top of my head.
>> [...]
>>> * Change the default contents of $GIT_DIR/remotes/origin
>>
>> The question is: do we continue to use remotes/ file, or do we
>> save remotes info in the config file: remote.<name>.url,
>> remote.<name>.fetch, remote.<name>.push and branch.<name>.merge
>> (in our case '[remote "origin"]' section)?
> 
> It is not "the question"; it is irrelevant because
> $GIT_DIR/remotes/origin and [remote "origin"] are pretty much
> interchangeable, and will hopefully continue to be.
> 
>         [remote "origin"]
>                 url = master.kernel.org:/pub/scm/git/git.git/
>                 fetch = +refs/heads/master:refs/remotes/origin/master
>                 fetch = +refs/heads/*:refs/remotes/origin/*
> 
> would be the way to express the equilvalent of the example in
> the previous message.

Well, the question is if git-clone would generate appropriate config 
entries in addition to or instead of $GIT_DIR/remotes/origin file...

By the way, what do you think about extended refspec syntax, namely
in addition to <remote name>:<local name> it would allow to say
<remote name>:<local name>:<local branch to merge into>?
-- 
Jakub Narebski

Re: [RFD] making separate-remote layout easier to use

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Jakub Narebski <jnareb@gmail.com> writes:
> > The question is: do we continue to use remotes/ file, or do we
> > save remotes info in the config file: remote.<name>.url,
> > remote.<name>.fetch, remote.<name>.push and branch.<name>.merge
> > (in our case '[remote "origin"]' section)?
> 
> It is not "the question"; it is irrelevant because
> $GIT_DIR/remotes/origin and [remote "origin"] are pretty much
> interchangeable, and will hopefully continue to be.

I'm all for:

 * changing the default made git-clone to be [remote "<origin>"]
 * continuing to support parsing of $GIT_DIR/remotes/*

We moved away from the $GIT_DIR/branches directory to
$GIT_DIR/remotes, yet we still support $GIT_DIR/branches in
remote handling code.  I see no reason why we cannot start to use
remote.<name>.url by default while continuing to support the older
branches and remotes formats.

For one thing the newer remote.<name>.fetch seems to make more sense
to new users than Pull: lines do.  For another it appears to be
supported since v1.4.0, which was released June 10th.  Most users
cloning a repository with >1.4.4.1 will probably only use 1.4.0
or later on that same repository, so there is probably low risk of
breakage due to the remote not being recognized by a pre-1.4.0 Git.

-- 

Re: [RFD] making separate-remote layout easier to use

[Shawn Pearce]

Jakub Narebski <jnareb@gmail.com> wrote:
> Junio C Hamano wrote:
> >       Pull: +refs/heads/*:refs/remotes/<origin>/*
> 
> I hope that it also works with the remote.origin config file
> section instead of $GIT_DIR/remotes/origin

Yes, it does.  Fortunately Git is relatively consistent.  :-)
 
> > Forcing with '+' is debatable, but with separate-remote layout,
> > remotes/*/ hierarchy is to track what the remote has, and you
> > cannot do much else other than noticing and warning when the
> > remote end does funny things to its refs anyway, so I think
> > having '+' might be a better default.
> 
> Perhaps, perhaps not. It would be nice to have configuration option
> that would tell that history of given branch is being changed, and
> the ability to ask about it remotely, so git-clone would be able
> to add this + _when needed_ automatically.
> 
> But it's a fact that with separate remote the need to use fast-forward
> check is lessened, and it might be more important to not confuse first
> time user with having to modify $GIT_DIR/remotes/origin or remote.origin
> config section to fetch from the repository he/she cloned from.

Yes.  From an out-of-the-box-make-Git-appear-to-be-easier-to-use
point of view tossing + into the Pull:/remote.<name>.fetch setup
might seem like the right thing to do.  It lets the end-user blindly
follow the upstream repository they cloned from.  Almost.

The problem becomes when the user makes a topic branch off say
Junio's `pu` branch and later after doing a fetch and looking at
the log of `remotes/origin/pu` decide to pull the updated `pu`
into their topic branch, so they can continue testing.  But now
they are in merge hell as Junio has completely rebased `pu` and
what was there isn't, and what's there now ain't compatible...
and the new user curses at Git.

Needing to put + in front of a refspec (or needing to fetch it with
--force) is the user agreeing that something _evil_ is going on with
the upstream and that they acknowledge this may cause problems for
them locally.

I would prefer to see the upstream be able to publish a short
description of each branch, where the repository owner can describe
the policy of that branch, as well as have a machine readable
setting on each branch indicating if that branch will be rewound
from time to time, or never rewound.

git-clone should skip rewinding branches by default, unless the user
adds an option (e.g. --include-rewinding-branches).  This way new
users to git.git don't get the `pu` branch unless they really mean
to get it, at which point they have hopefully also read the upstream's
description of the `pu` branch and its rewinding policy, and can
at least start to grasp what is going to happen if they start to
work with the branch.
 
> > I am not sure if 'merge in corresponding branch' is the only
> > valid workflow, however. I am reluctant to make the system
> > automatically do so if the solution makes other workflows more
> > painful to follow.  Automatically merging remotes/origin/$foo
> > when on $foo branch is not good enough, in other words (also,
> > there may be a hierarchy under remotes/ other than origin).  It
> > might make sense to introduce "Merge: " in remotes/ file and if
> > they are present use "Pull: " only to decide what are fetched
> > and use "Merge: " to decide what is merged (if we were doing the
> > system from scratch, the former would have been named "Fetch: "
> > but it is too late now).
> 
> If you add "Merge: " in remotes/, then please add it also in
> remote section in config file. Config file has now 
> branch.<branchname>.merge (and it would be nice if clone would
> set ou this for local branches corresponding to remote branches),
> but it is not the same.

I'm against adding anything to the remotes/ file format.

We already have branch.<name>.merge to indicate what the default
source for a git-pull on the branch named <name> should be.
git-branch probably should fill that entry in when a branch is
created from a remotes ref.

-- 

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> Well, the question is if git-clone would generate appropriate config 
> entries in addition to or instead of $GIT_DIR/remotes/origin file...

What I meant by "irrelevant" is that it is purely syntax.  I
would rather nail down the desired semantics first, and then
worry about the syntax.  We have done remotes/origin file by
default and allowed the same thing specified in configuration
file for people who prefer configuration file, and I do not see
an immediate need to change that.

Only if/when the desired semantics turns out to be too complex
to be expressed in the remotes/ file scheme and at the same time
if configuration file is easier to express the same information,
we should start talking about favoring configuration file over
remotes/ file, and perhaps generating the config entries by
default without generating remotes/.

In either case, I do not think generating _both_ by default is a
bad idea.  They will become out of sync and the user either need
to delete one or the other to avoid confusion.

> By the way, what do you think about extended refspec syntax, namely
> in addition to <remote name>:<local name> it would allow to say
> <remote name>:<local name>:<local branch to merge into>?

I do not think it is reasonable to assume that one remote branch
is to be always merged into one fixed local branch, so that is
too limiting as an extension.  In the configuration file, I
think we already allow the default remote to be fetched from and
remote branches to be merged per branch statically.  A
reasonable thing to do is to see how satisfied users are with
that facility, what they find missing there. Worrying about the
syntax should wait until we know what kind of things need to be
expressed; that can be done both by extending the existing
configuration and/or introducing a new entries in remotes/
files but the point is that is mere implementation detail.

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> We moved away from the $GIT_DIR/branches directory to
> $GIT_DIR/remotes, yet we still support $GIT_DIR/branches in
> remote handling code.

IIRC, git never used "branches", so there was no "moving away".
It was merely to interpret what cg-init left in the repository.
Since the very early days remotes/ has been _the_ format git
used, and I do not see any strong reason to change it now.  The
continued support for branches is an irrelevant topic and is not
a justification to favor configuration over remotes/ nor the
other way around.

> For one thing the newer remote.<name>.fetch seems to make more sense
> to new users than Pull: lines do.

That is quite subjective.

> ..., so there is probably low risk of
> breakage ...

Again, "low risk of breakage" does not justify switching.

As I said in the other message, I think configuration vs remotes/
is futile subject and premature topic to talk about before
nailing down what semantics we would want.  Either format can
have the expressive power, once we know what we want to express.

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> Needing to put + in front of a refspec (or needing to fetch it with
> --force) is the user agreeing that something _evil_ is going on with
> the upstream and that they acknowledge this may cause problems for
> them locally.
>
> I would prefer to see the upstream be able to publish a short
> description of each branch, where the repository owner can describe
> the policy of that branch, as well as have a machine readable
> setting on each branch indicating if that branch will be rewound
> from time to time, or never rewound.
>
> git-clone should skip rewinding branches by default, unless the user
> adds an option (e.g. --include-rewinding-branches).  This way new
> users to git.git don't get the `pu` branch unless they really mean
> to get it, at which point they have hopefully also read the upstream's
> description of the `pu` branch and its rewinding policy, and can
> at least start to grasp what is going to happen if they start to
> work with the branch.

I like this approach very much.

Re: [RFD] making separate-remote layout easier to use

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> > Needing to put + in front of a refspec (or needing to fetch it with
> > --force) is the user agreeing that something _evil_ is going on with
> > the upstream and that they acknowledge this may cause problems for
> > them locally.
> >
> > I would prefer to see the upstream be able to publish a short
> > description of each branch, where the repository owner can describe
> > the policy of that branch, as well as have a machine readable
> > setting on each branch indicating if that branch will be rewound
> > from time to time, or never rewound.
> >
> > git-clone should skip rewinding branches by default, unless the user
> > adds an option (e.g. --include-rewinding-branches).  This way new
> > users to git.git don't get the `pu` branch unless they really mean
> > to get it, at which point they have hopefully also read the upstream's
> > description of the `pu` branch and its rewinding policy, and can
> > at least start to grasp what is going to happen if they start to
> > work with the branch.
> 
> I like this approach very much.

I think the idea of glob'ing off the remote during clone (and
using the glob for future fetches) is a good idea.  So I think what
I'm suggesting above is to make glob'ing by default not copy the
rewinding branches, unless the glob spec itself starts with '+'
(which is what say --include-rewriting-branches would do).

So now we're at a point of:

 * how do we get this branch information from the remote?
 * how does the remote store this branch information?

I'm leaning towards the repo config for the latter, with say:

	[branch "pu"]
		rewinds = true
		description = This branch contains some proposed updates to git.\n\
	It rewinds often, as different updates are proposed or merged into 'next'.

but looking at that think its rather horrible looking.  :-)

However for the former it may be useful if the client can download
the repo config from the remote repository, such as to fetch
the branch description data.

-- 

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> So now we're at a point of:
>
>  * how do we get this branch information from the remote?
>  * how does the remote store this branch information?
>
> I'm leaning towards the repo config for the latter, with say:
>
> 	[branch "pu"]
> 		rewinds = true
> 		description = This branch contains some proposed updates to git.\n\
> 	It rewinds often, as different updates are proposed or merged into 'next'.
>
> but looking at that think its rather horrible looking.  :-)
>
> However for the former it may be useful if the client can download
> the repo config from the remote repository, such as to fetch
> the branch description data.

For the former, I was hoping that we could do a new action
against remote repository that is "get remote information". Just
like we have "ls-remote" as a protocol neutral front-end to get
what git native protocol would give "peek-remote", git native
protocol would have get-repository-info action to transfer this
information, and http and dumb transports do a file download
from a known file, just like they download files prepared with
update-server-info.  Most likely, that get-repository-info is
just a glorified single file transfer and nothing more.

As an implementation detail, I think it is one possibility to
store this information in the config of the repository being
cloned from.  We should be able to use repo-config to
parse that file in config-like syntax to extract what we would
want, so we do not have to come up with a new syntax nor a new
parser, which is a big plus.

	Side note: repo-config, especially the updating side of
	it, is one of the programs I find the most brittle in
	the whole system, and it always makes me hesitate when
	somebody proposes to use config file in any important
	way read-write for this reason.  But this "reading out
	per-branch information from the file" is read-only so I
	would not worry too much about it.

Strictly speaking, however, the config file is a wrong place to
store it.  For one thing it has core.sharedrepository and
receive.denynonfastforwards that are true configuration to
control the behaviour of git _at_ _the_ _repository_ the
configuration is at.  The new "branch property" are primarily to
help the other end, and the "filtering rewinding ones" we want
at the clone/fetch side wants that information only and not
interested in the true configuration information at that
repository.

So my preference is:

 * store it in a fixed filename under $GIT_DIR/$something, in a
   format that is similar to the true config file.

 * Dumb transports would just do a file transfer, and we add a
   new command (like "upload-pack", "upload-archive") for git
   native transport to read that file.  We might want to have a
   protocol neutral wrapper (just like "ls-remote").

 * Cloners and fetchers would do the file transfer and use
   repo-config to read from that transferred file to find out
   which are rewinding branches.

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> In either case, I do not think generating _both_ by default is a
> bad idea.  They will become out of sync and the user either need

Eh, excuse me.  "generating both is a BAD idea" is what I meant...

Re: [RFD] making separate-remote layout easier to use

[Shawn Pearce]

Junio C Hamano <junkio@cox.net> wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> > So now we're at a point of:
> >
> >  * how do we get this branch information from the remote?
> >  * how does the remote store this branch information?
> 
> For the former, I was hoping that we could do a new action
> against remote repository that is "get remote information".

Yes, that is completely sane.  :)
 
> As an implementation detail, I think it is one possibility to
> store this information in the config of the repository being
> cloned from.  We should be able to use repo-config to
> parse that file in config-like syntax to extract what we would
> want, so we do not have to come up with a new syntax nor a new
> parser, which is a big plus.

Also quite sane.  Nobody wants to write new code if we don't really
have to. :)

> 	Side note: repo-config, especially the updating side of
> 	it, is one of the programs I find the most brittle in
> 	the whole system, and it always makes me hesitate when
> 	somebody proposes to use config file in any important
> 	way read-write for this reason.  But this "reading out
> 	per-branch information from the file" is read-only so I
> 	would not worry too much about it.

Now you tell me.  git-gui uses repo-config for all of its option
data.  It writes back to the config file to save user configured
options, including window layout at quit.  It also uses both the
user global and the local repository config files. :)

We can safely update the index, symrefs and refs.  Why can't we
also safely update the config file?  The biggest problem is its
also a hand-modified file complete with support for comments,
so users expect the file to stay intact during edits.  Aside from
that little issue its really a fairly simple format...

> Strictly speaking, however, the config file is a wrong place to
> store it.  For one thing it has core.sharedrepository and
> receive.denynonfastforwards that are true configuration to
> control the behaviour of git _at_ _the_ _repository_ the
> configuration is at.  The new "branch property" are primarily to
> help the other end, and the "filtering rewinding ones" we want
> at the clone/fetch side wants that information only and not
> interested in the true configuration information at that
> repository.

Yes, of course.  I never suggested copy the entire configuration
as is.  What we really want to do is take the branch data from the
remote and rewrite it for every branch we fetched into a ref, e.g.:

  remote:
    [branch "master"]
        rewinds = false
        description = blah blah

  local:
    [branch "remotes/origin/master"]
        rewinds = false
        description = blah blah

while skipping other configuration sections and options that have
no meaning for the local repository.  core.sharedrepository is just
one of *many* configuration options that the local repository should
not inherit.  remote.<name>.url is another...  etc.

> So my preference is:
> 
>  * store it in a fixed filename under $GIT_DIR/$something, in a
>    format that is similar to the true config file.

At which point you might as well put it into $GIT_DIR/config.
People are going to use git-repo-config to edit that file, even if
you say its brittle and maybe shouldn't be used for heavy editing.
Its just damn handy.
 
Besides of which, lets not forget that something like:

  [branch "master"]
      rewinds = false
  [branch "pu"]
      rewinds = true

is not only data for the client to examine.  It can be useful in
say git-receive-pack as a much more fine-grained alternative to
receive.denynonfastforwards.  If the server's policy is to not
rewind a branch then receive-pack shouldn't let a remote user
rewind it.  At which point its useful to have that branch data
in $GIT_DIR/config.  :-)

-- 

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

>> Strictly speaking, however, the config file is a wrong place to
>> store it.  For one thing it has core.sharedrepository and
>> receive.denynonfastforwards that are true configuration to
>> control the behaviour of git _at_ _the_ _repository_ the
>> configuration is at.  The new "branch property" are primarily to
>> help the other end, and the "filtering rewinding ones" we want
>> at the clone/fetch side wants that information only and not
>> interested in the true configuration information at that
>> repository.
>
> Yes, of course.  I never suggested copy the entire configuration
> as is.

I never misunderstood your plan as such.  I was more worried
about a case where you clone from a not-so-public repository and
the config file has some information of sensitive nature.
Currently we do not have any such variable, so I am probably
being a bit paranoid here, but using the config for the purpose
of what we are discussing right now closes the door to store
sensitive information and declare that the config file at the
remote is "none of the cloner's business".

However,...

> Besides of which, lets not forget that something like:
>
>   [branch "master"]
>       rewinds = false
>   [branch "pu"]
>       rewinds = true
>
> is not only data for the client to examine.  It can be useful in
> say git-receive-pack as a much more fine-grained alternative to
> receive.denynonfastforwards.  If the server's policy is to not
> rewind a branch then receive-pack shouldn't let a remote user
> rewind it.  At which point its useful to have that branch data
> in $GIT_DIR/config.  :-)

This is a good point.  If we want to ever have a 'not to be
leaked' configuration per repository, we can have a separate,
private, configuration file to store that, so I probably should
not worry too much about this at this point.  I'm Ok with having
this in the usual config -- the convenience outweighs the
purity.

To summarize:

 * A repository can have branch."foo".rewinds configuration to
   mark the "foo" branch to be subject to rewinding.  This
   configuration variable defaults to false.

 * Receive.denynonfastforwards is currently a boolean but a new
   value, "per-branch", is also allowed.  When it is set to
   "per-branch", non fast-forward update is allowed only when
   branch."foo".rewinds is true for the branch.

 * A cloner creates "Pull: refs/heads/*:refs/remotes/origin/*"
   (or config equivalent) upon cloning.  When it clones, it
   downloads the config from the remote in order to see which
   are marked as "rewinds".  It makes tracking branches for only
   the ones that are not marked as "rewinds" by default.  We
   might have an option to do '+' variant glob, and clone all
   branches.

 * Subsequent fetch, when the glob is not '+' variant, would
   download the config from the remote in order to see which are
   marked as "rewinds", and ignores the branches that are marked
   as such.

I am a bit unhappy that subsequent fetches have to re-read the
remote config every time.  I can sort-of-see we can cram the
"this is expected to be rewound" information as part of
peek-remote exchange to avoid the overhead, but I do not think
it is easily doable for dumb transports without breaking the
backward compatibility.

To avoid this, we need to remember what remote branches we have
seen but decided not to track, perhaps because they were marked
as "rewinds".  Subsequent fetches first learn what are on the
remote side, and only when it sees a branch that we do not track
and we have not seen, it needs to re-read the remote config to
see if that is to be tracked (in which case we would create a
new tracking branch) or ignored (in which case we would remember
that we will ignore this branch in the future fetches).

Re: [RFD] making separate-remote layout easier to use

[Jakub Narebski]

Shawn Pearce wrote:
> Jakub Narebski <jnareb@gmail.com> wrote:
>> Junio C Hamano wrote:
>>
>>> I am not sure if 'merge in corresponding branch' is the only
>>> valid workflow, however. I am reluctant to make the system
>>> automatically do so if the solution makes other workflows more
>>> painful to follow.  Automatically merging remotes/origin/$foo
>>> when on $foo branch is not good enough, in other words (also,
>>> there may be a hierarchy under remotes/ other than origin).  It
>>> might make sense to introduce "Merge: " in remotes/ file and if
>>> they are present use "Pull: " only to decide what are fetched
>>> and use "Merge: " to decide what is merged (if we were doing the
>>> system from scratch, the former would have been named "Fetch: "
>>> but it is too late now).
>> 
>> If you add "Merge: " in remotes/, then please add it also in
>> remote section in config file. Config file has now 
>> branch.<branchname>.merge (and it would be nice if clone would
>> set ou this for local branches corresponding to remote branches),
>> but it is not the same.
> 
> I'm against adding anything to the remotes/ file format.
> 
> We already have branch.<name>.merge to indicate what the default
> source for a git-pull on the branch named <name> should be.
> git-branch probably should fill that entry in when a branch is
> created from a remotes ref.

As I said, branch.<name>.merge is about something else: it just means
that if we are on <name> branch "git pull" will merge 
branch.<name>.merge branch into it.

I think the "Merge: " or remote.<repo>.merge is about changing current 
implicit rule: first branch is to be merged with current branch (if not 
specified otherwise) when pull-ing, into explicit rule: branch marked 
as "Merge: " is to be merged with current branch (unless specified 
otherwise).

Correct me if I'm wrong, Junio.
-- 
Jakub Narebski

Re: [RFD] making separate-remote layout easier to use

[Jakub Narebski]

Junio C Hamano wrote:

>  * Subsequent fetch, when the glob is not '+' variant, would
>    download the config from the remote in order to see which are
>    marked as "rewinds", and ignores the branches that are marked
>    as such.
> 
> I am a bit unhappy that subsequent fetches have to re-read the
> remote config every time.  I can sort-of-see we can cram the
> "this is expected to be rewound" information as part of
> peek-remote exchange to avoid the overhead, but I do not think
> it is easily doable for dumb transports without breaking the
> backward compatibility.

Perhaps below the "Pull: refs/heads/*:refs/remotes/origin/*" (or config
equivalent) we should list (in some format) branches which are not to be
fetched becaus they don't fast forward. One option, namely
  Pull:-refs/heads/pu:refs/remotes/origin/pu
(or config equivalent)comes to mind. It has the advantage that when we
decide to fetch this branch, it would be as easy as changing '-' to '+'.
Or we can use '!' instead of '-' if you think it is more readable (probably
is).
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git

Re: [RFD] making separate-remote layout easier to use

[Josef Weidendorfer]

On Saturday 25 November 2006 22:53, Junio C Hamano wrote:
>  - merging the "primary" branch is good only when the user is on
>    the corresponding "primary" branch.  It is usually a wrong
>    thing to do when on another branch.

I vote for a config option

 core.pull.stopWithoutDefaultMerge

which would error out on a

 git pull

when no branch.<current branch>.merge is set with a message like

 "No default remote branch set to merge into current branch <branch>.
  Set branch.<branch>.remote/merge to make this work"

It is arguable if this option should change the behavior of
"git pull <repo>". Probably not: I would assume that the
user explicitly wants to merge the primary branch of <repo>.

Additionally, set branch.<primary>.remote/merge on git-clone,
together with core.pull.stopWithoutDefaultMerge=1.

Re: [RFD] making separate-remote layout easier to use

[Josef Weidendorfer]

On Sunday 26 November 2006 04:58, Junio C Hamano wrote:
> > git-clone should skip rewinding branches by default, unless the user
> > adds an option (e.g. --include-rewinding-branches).  This way new
> > users to git.git don't get the `pu` branch unless they really mean
> > to get it, at which point they have hopefully also read the upstream's
> > description of the `pu` branch and its rewinding policy, and can
> > at least start to grasp what is going to happen if they start to
> > work with the branch.
> 
> I like this approach very much.

I am not so sure about this. IMHO, any user expects to see "pu" branch
cloned too after cloning git.git (at least me, and a newbie probably too).

I agree about the remote branch description saying that "pu" branch will
be rewinded. git-clone should set "+" before such a branch.

As said, the real problem begins when the user tries to branch off her
own local branch from "pu". At this point, "git branch" or
"git checkout -b ..." should warn the user that he has to expect troubles
when branching off from such a branch, and only allow it with a
"--force" option.

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Josef Weidendorfer <Josef.Weidendorfer@gmx.de> writes:

> I am not so sure about this. IMHO, any user expects to see "pu" branch
> cloned too after cloning git.git (at least me, and a newbie probably too).

I think you are right that we can keep our sanity without
omitting rewinding ones from the tracked set of branches.  In
fact, remotes/origin/* hierarchy are not to be checked out
directly anyway, even after we do the "bare SHA-1 stored in
$GIT_DIR/HEAD" update -- that should give you a read-only
checkout that you cannot commit on top of.  As long as we warn
users who try to merge from these rewinding branches, there is
no reason not to track them.

> As said, the real problem begins when the user tries to branch off her
> own local branch from "pu". At this point, "git branch" or
> "git checkout -b ..." should warn the user that he has to expect troubles
> when branching off from such a branch, and only allow it with a
> "--force" option.

That is true if you make "git branch my-pu remotes/origin/pu" to
somehow set up the default merge source of "my-pu" branch to be
remotes/origin/pu to encourage merging from it.

I am still not convinced it is a valid assumption that a branch
often want to merge from the branch it was forked off of, and
even less so that "git branch" and "git checkout -b" are the
places to do that configuration.

But for the sake of discussion let's pretend for now that it
were a good idea.  If we know remotes/origin/pu is expected to
be rewound, the logic that configures "my-pu" to merge from the
fork origin should be able to notice that situation, and refrain
from doing the configuration, to prevent the user from issuing
"git pull" without saying "from where", which can be done with
your merge.stopsansdefault option in your other message.

Re: [RFD] making separate-remote layout easier to use

[Jon Loeliger]

On Sat, 2006-11-25 at 15:53, Junio C Hamano wrote:

> I am not sure if 'merge in corresponding branch' is the only
> valid workflow, however. I am reluctant to make the system
> automatically do so if the solution makes other workflows more
> painful to follow.  Automatically merging remotes/origin/$foo
> when on $foo branch is not good enough, in other words (also,
> there may be a hierarchy under remotes/ other than origin).  It
> might make sense to introduce "Merge: " in remotes/ file and if
> they are present use "Pull: " only to decide what are fetched
> and use "Merge: " to decide what is merged (if we were doing the
> system from scratch, the former would have been named "Fetch: "
> but it is too late now).

Heh.  It was too late back here too!

    From: 	Jon Loeliger <jdl@freescale.com>
    To: 	Git List <git@vger.kernel.org>
    Subject: 	Re: GIT 0.99.7d, and end of week status.
    Date: 	Mon, 26 Sep 2005 15:17:32 -0500
    Message-Id:  <1127765852.5735.36.camel@cashmere.sps.mot.com>

jdl

Re: [RFD] making separate-remote layout easier to use

[Jon Loeliger]

On Sat, 2006-11-25 at 21:58, Junio C Hamano wrote:
> Shawn Pearce <spearce@spearce.org> writes:
> 
> > Needing to put + in front of a refspec (or needing to fetch it with
> > --force) is the user agreeing that something _evil_ is going on with
> > the upstream and that they acknowledge this may cause problems for
> > them locally.
> >
> > I would prefer to see the upstream be able to publish a short
> > description of each branch, where the repository owner can describe
> > the policy of that branch, as well as have a machine readable
> > setting on each branch indicating if that branch will be rewound
> > from time to time, or never rewound.
> >
> > git-clone should skip rewinding branches by default, unless the user
> > adds an option (e.g. --include-rewinding-branches).  This way new
> > users to git.git don't get the `pu` branch unless they really mean
> > to get it, at which point they have hopefully also read the upstream's
> > description of the `pu` branch and its rewinding policy, and can
> > at least start to grasp what is going to happen if they start to
> > work with the branch.
> 
> I like this approach very much.

I see how it is! :-)  When Shawn says it, it is to be liked.
But when I allude to it a _year_ ago, I'm ignored.


    From: Jon Loeliger <jdl@freescale.com>
    To: git@vger.kernel.org
    Subject: Re: Trying to Update All Heads of a Repository
    Date: Fri, 4 Nov 2005 07:49:40 -0700 
    Message-ID:  <E1EY2su-0006LW-IN@jdl.com>

    So, from the git-pull man page:

        For "git push", the local ref that matches <src> is
        used to fast forward the remote ref that matches
        <dst>. If the optional plus + is used, the remote
        ref is updated even if it does not result in a fast
        forward update.

    Ah-ha!  Wait.  But here's the conceptual missing piece:
    When might I _know_ I have the situation where a
    fast-forward update might not happen?  And as a remote
    puller, that would be "never" -- unless I know something
    about the nature of the remote end.  That is, like you
    said, "pu" is subject to wild fluctuation and non-linear
    behavior.  But any random puller can't know that a
    priori.  So far, that is out-of-band information about a
    branch that needs to be "available".

Yep, out-of-band information that needs to be available indeed...

Ah well. :-)

jdl

Re: [RFD] making separate-remote layout easier to use

[Junio C Hamano]

Jon Loeliger <jdl@freescale.com> writes:

> On Sat, 2006-11-25 at 21:58, Junio C Hamano wrote:
>> > ...
>> > I would prefer to see the upstream be able to publish a short
>> > description of each branch, where the repository owner can describe
>> > the policy of that branch,...
>> ...
>> I like this approach very much.
>
> I see how it is! :-)  When Shawn says it, it is to be liked.
> But when I allude to it a _year_ ago, I'm ignored.

Think of it not as a difference between you and Shawn, but as a
difference in me back then and me now, who finally saw the light
in your revelation after gaining a year's worth of experience ;-).

Seriously, I did not ignore it, and I think some of your other
comments in the message contributed to improved documentation.
When your message was posted, the issue Shawn raised in his
message was not so prominent, because people did not even get
that "Pull: refs/heads/pu:refs/heads/pu" line in their remotes/
file by default back then.  It was back in 0.99.9 era and a few
days after your message, a patch was posted to put more than
just the 'master' mapping in the remotes/ file clone creates.
It became an issue after that change but we lacked the resources
nor sense of urgency to do the out-of-band stuff, as we were
busy working on other things.